Both physicalism and idealism start from an assumption

Letting go of the hard problem by assuming that matter causes consciousness to arise is no more justified than assuming consciousness is fundamental everywhere; either way one is simply choosing an unproven assumption about how consciousness relates to matter, so taking consciousness as fundamental is at least as legitimate a starting point.