Isn't this a question every audio crazy has to answer for himself?
How good is good enough?
Most of us here cannot afford six figure prices for each component (assuming that will bring the best sound.) So how far do we want to go to improve our systems? There are always bigger fish. When does it stop? It stops when we say it stops, when our gear brings us satisfaction. To constantly strive for better sound is an endless quest, not necessarily based on the quality of our set but on our personality.