I prefer to look at it in the context of a system, as opposed to an absolute. In any system, there is a group of components contributing/detracting from the sound. In that system, there is a "bottleneck" somewhere, that is doing the most damage. Even in the best systems, there is some component that is not as good as the rest. While it may seem likely that the speaker is the culprit, that may not be the case. You may have a great speaker, then what? It may lie in the preamp, or the amp, or a cable run, or, horror of horrors, even the source.
The reasons that I state this in a system context, is that no system is perfect, and to state an absolute about what component strays from perfection the most, misses the point. I am quite aware that in my system, my preamp is the bottleneck, even though it is quite a nice preamp. I can't afford to upgrade it right now, but my speakers, amp and TT clearly are better than my preamp, and when I upgrade next, that is where the money is going.
So I say that the component that degrades the signal the most is the one that is the bottleneck in YOUR system. There always is one. When you reach a point where you might have a contender for the best speaker in the world, then you really can't do any better there, and you have to look elsewhere for your problem. If/when you reach a point where all the components are state-of-the-art, then you have to look at even smaller bits to find the bottlenecks, like connectors, and power, and resistors and caps, and internal wiring, etc., to find your improvements. You can never really reach the end of "perfection" so it is a constant challenge. Maybe if the system is near perfect, then the room needs upgrading. It never ends.
I realize that this is a different angle than the others here took, but I think it is a valid way to look at it realistically.