Unfortunately, it all matters. All components including the amp. Speakers may be the biggest determinant.
I can comment on amps. For a year and a half I had both an Audio Research Reference 160 stereo and a pair of 160 monoblocks. So, identical amplifiers… just one spread into two boxes. The most notable difference was imaging. The soundstage was wider and much deeper with the monoblocks and hence the images presented more three dimensionally in a larger sound space.
What I heard was confirmed by a couple other folks that had an opportunity to experience it (different amps). I suspect a lot of it comes from a lower noise floor. More quiet allows you to hear more subtle cues in the sound.