It is the whole system which, of course, begins with the input and ends with the room. But like any chain every link matters. If any link fails the system fails. But in some ways the input could be thought of first. The rest of the system is basically set. Changing any part involves time, effort and money. A lousy source is easily replaced.
But let me throw out another thought. After over six decades of audiophilia I have come to believe the one characteristic of sound most important to sounding real is a system is dynamic linearity. And I don't mean the first thought of what usually comes to mind which is the ability to play loud cleanly. This is part of dynamic linearity. Dynamic linearity is the ability of the system to accurately, linearly, follow the source(which hopefully albeit too often isn't dynamically linear). It's the ability to make linear level changes whether the change is micro or mini or midi or macro. Think of a live, unamplified orchestra. Change your seat and the frequency response changes and yet it still is live. Or think about listening to acoustic music at the door outside the room. The sound is significantly affected and yet you know it's live sound. It's dynamic linearity.