I've been to three audio shows in the past several years (Axpona, Tampa, PAF) and I have come to the conclusion that the concept of diminishing returns doesn't apply very well to speakers. Rather, there is just no solid correlation between price and performance. Like the OP I have heard megabuck speakers that didn't sound good and I've heard several moderately priced speakers that sounded wonderful. I don't think that I can attribute this to non-optimum amplification or source equipment because I trust the speaker manufacturer to know what components work best with their speakers. In many demonstrations the cables and power conditioning cost more than my entire system and I my system was better, at least to me.
To put it another way, I've heard six-figure speakers that sounded amazing and I've heard others that sounded mediocre. I'm a geezer - I've had this hobby for 50 years - and I've concluded that once you get into the audiophile range of gear (thousands of dollars per piece, not hundreds) the value proposition seems to break down. One would think that a million dollar system would be life changing but I have not found that to be the case. It's not that diminishing returns takes over, it's more that it's a crapshoot. My favorite speakers of all time are the MBL 101 E Mk II and while they are certainly expensive ($80K) they outperformed speakers that were multiples of their price.