It has been worth it for me. Although early in my audiophile journey I did a lot of upgrading, about 40 years ago I assembled a core system that has proven satisfying. I've had to upgrade parts of it, and particularly I have had to upgrade the digital side of my system, but now even that is quite satisfactory.
I can simply listen and enjoy the music without having to be critical about this or that detail. I really believe that for my system the biggest shortcomings are in the recordings themselves--I have found few convincingly realistic recordings, but the difference between them and the rest have convinced me that one's reproducing gear can only go so far--you can't make a silk purse from a sow's ear.
But if the acoustics of your room are not good, you either need to address that or focus on headphone listening, which I can enjoy while realizing it misses some of the elements that loudspeakers provide.