Do distortion 's affect enjoyment of speaker?

I hope not, since all speakers produce it in one form or another.
Seriously though, from my own experience it can if it a type to which you are sensitive. When you are aware of a certain bothersome distortion or sonic anomaly of any type, it consciously or subconsciously gets in the way of your enjoyment of the music.

Matt,
It is not desirable, but we have no choice but to accept it, because it exists in all of our systems in many different forms to a greater or lesser degree. As long as it isn't a sort that bothers the person who is listening to the system, all is well.