It stops when you find out what makes sound do what it does. It is not pricing that dictates sound, it is design and parts quality. You can upgrade your existing components using better parts that the manufacturer used and get fantastic results without spending that much compared to what are considered reference components. We do this all the time probably 5 units each month. Even $20K units are improved greatly.
How good is good enough?
Most of us here cannot afford six figure prices for each component (assuming that will bring the best sound.) So how far do we want to go to improve our systems? There are always bigger fish. When does it stop? It stops when we say it stops, when our gear brings us satisfaction. To constantly strive for better sound is an endless quest, not necessarily based on the quality of our set but on our personality.