I don’t listen to $20,000 loudspeakers, so I don’t know what I am talking about. But as that never stopped me before . . . .
When I gave away my pedestrian 30+ year old stereo system to get something better, that meant giving up two ‘80’s vintage Jensen speakers for two Martin Logan towers (not the planars, the 60 XT’s). Everyone I know assumed that I wanted to blast the neighborhood with sound, but that was never my intention. I used to listen to loud music decades ago, but now I prefer low-to-medium volume and bought nicer speakers for the quality of sound, not the volume. When others here note that they prefer to listen at 80-85 dB, I only go that loud when I do housekeeping chores in the next room and want the music as “company”. When doing what others call “serious listening”, I turn the music down to maybe 50-60 dB, for what I call relaxing.
How I relate this to the OP’s question: My assumption is that very expensive loudspeakers would make the greatest difference at higher volume levels. That is where the speakers are being pushed more and where they would hopefully shine compared to lesser speakers. It would be nice to visit and listen to such high-end speakers, to hear what the fuss is about, I don’t think they would make as much difference at the low volumes I usually do my listening at.
And, I’m not a big fan of listening at the frequency extremes that high-end speakers can reproduce. I compare this to the vocal range of singers — people tend to thrill to the singers that can reach the highest notes, but that is not my priority. I would rather listen to a singer that can make beautiful sounds in a more-restricted frequency range than the singer that can pierce metal with the very high frequencies. It comes back to comfort for me, not setting records.