I felt the same as the OP all the years since i had no good system to use and must create one at low cost ...
i was unsatisfied and frustrated ...I had no idea about the importance and laws of acoustics ...
The bad sound was too much for me to listening somtheting else than Bach ...
I learned then in my own way with experiments to embed any system at any price , especially with acoustics...It takes me full time for few years but i was retired ...
Now with a low cost system i succeeded... No more sound evident limitations, i listen music all day between walks ...Or reading ...
It MAY cost peanuts to create audiophile experience if you pick the right gear with the right synergy, but it is not enough at all ; you must learn your way to understand how to control the mechanical (vibrations resonance) electrical(signal-noise ratio) acoustical (not only room acoustics but psycho acoustics basic) working dimensions of the gear ...
i am very proud of my 700 bucks system headphone and speakers ...
Music is ectasy because if sound is the body of the woman you love,the music is his expressive soul and cannot be reduced to his body so beautiful could it be ...