Everything matters is the short answer as always. But to over amplify the importance of one aspect of audiophilia over another is not a fruitful argument imo. Room construction, size, acoustic treatments, speakers, amps, preamps, sources, cables, etc..., all that matters if getting the best sound to your ears truly matters to you. Unfortunately there is no easy short cut, not even high end speakers.
"The Ultra High-End Speaker."
My entire relatively simple high end audio system retails for approx. $70,000, with my speakers alone retailing for approx. $24,000 (Revel Salon 2 speakers). I've been around high-end audio for over 40 years. I attend audio shows and visit local and non-local high-end audio shops on a regular basis. I get to hears a lot of high-end audio speakers and gear all the time. That said, I honestly believe, along with others who've visited my home and have listened to my system, that my system (speakers) produce that ultra high-end, reference quality sound. Others would suggest that, when it comes to speakers, that the "Ultra High-End" sound can only be achieved by megabuck speakers costing 50K, 100K, 250k and beyond. I do not believe that ultra high-end ("Sound Quality") is excusive to those speakers costing a king's ransom. And, I think my own system is am example of what can be achieved at a lower (not for most people) price point. I absolutely believe in the law of diminishing returns, especially when it comes to high-end speakers. What's your definition, idea of, what you consider to be, a "Ultra High-End Speaker, and at what price point does the ultra high-end start?????