You have an emotional connection to the music not the system. I will tap my feet listening to a telephone.
The speaker and room have to be considered as one component. Together, they are certainly the weak link in our systems by a HUGE margin. As long as the amplifier is appropriately matched to the speakers, distortion in electronics is relatively very minor.
As for our ears yes, you can focus on a group in a crowd until the crowd reaches a certain volume. Then the crowd masks the group. We are also linear listeners. We can only focus on one thing at a time. As an example try listening critically to two instruments at once. You can't do it. You can switch back and forth quickly, but you can not do both at the same time, three people looking at each other in the eye at the same time. Our ears are relative quite accurate, the problem is that they are connected to a brain which at times actually likes distortion, just listen to any Hendrix tune.
So, what do we do? Most of us buy the equipment we can afford and put it in the rooms we have available and make it work to our preferences as best we can. You just have to be aware of the snake oil problem.
What do I do? I try to go about it in a scientifically sound manor using what is available, designing the room specifically for sound reproduction, choosing a speaker type with inherently lower distortion that has controlled directivity, so that the room has even less of an affect, use acoustic treatment where required, select an amp that matches the loudspeaker well and use the best sources I can afford.
There is no magic here, but there are a lot of theories floating around that are fun to discuss, but in most cases are not applicable to our situations.