Don't forget that speakers are the end of the signal path and can only deliver what the amp gives it, which can only deliver what the preamp gives it, etc.
Also a big part of the speaker cost is the material and construction cost of the cabinet, which may or may not contribute to the sound quality. But it may help you get approval from the boss, especially if the 3 feet from the wall rule encroaches on a more general purpose room/den like I have my system located.
Diminishing returns depends on how much you are willing to spend on the entire listening experience - room (see @mikelavigne's system) and then each component. Nothing exists in a vacuum. He based his entire house purchase on what he could convert into a dedicated listening room. First item - design of the room including minimizing noise from his HVAC system!
I think he is the most dedicated contributor I've heard about on the 'Gon. I don't look at any other audiophile discussion boards though - maybe there is someone else out there. Wish I was ever near Seattle so I could listen to his system, especially the reel to reel decks.