@grauerbar Good post!
I'll admit to similar feelings at certain points in building systems over now nearly thirty years. There were times when the cost in form of funds and audiophile angst seemed excessive for the value of sound quality I was experiencing. In those bad old days, excessive analytical listening mode, having to work too hard to get to music listening mode. Problem was I was looking for perfection with systems that had far too many defects.
While there were times when I felt like giving up, I persevered, managing to put together subsequent systems that provided far more of the sound qualities I desire. Keeping an audio journal since late 1990's has proven invaluable in that it allowed an orderly progression in marginal gains towards the system I have today. Without the journal and it's attention to minutiae I'd have likely gone in circles doing the equipment churn thing.
And so now onto more recent times, I've not had these ruminations for some years now. There was a point where I surpassed a certain plateau in system sound quality that allowed an easy transition from analytical mode to music enjoyment mode with virtually no effort. I still listen in analytical mode for short periods and continue to evolve my system, but I now consistently enter music enjoyment mode easily, even though I may have added new equipment or tweaks.
In my estimation, its reaching this certain plateau that's the difficult part of being an audiophile, once past this easy sailing.
I'd also posit, perhaps digital only systems or concentrating on digital only in systems with both vinyl and digital setups makes the journey more difficult. For me, resolution, transparency rather easy to achieve with digital, getting digital to sound like best quality analog/vinyl was the difficult part.
I'll only say I've finally made it to entirely happy audiophile stage, all the work and angst has paid off! I realize there's no perfection possible, but my contentment level means this pursuit not all folly.