Has it all been worth it?

Definitely yes. And you don't have to spend too much to have a very good satisfactorily sounding system. There is always something better out there, so what? Your system must be capable of conveying the emotional essence of music, the rest is less important.

Yes it has been worth it. Never second guess yourself as there are others that will do that for you. 
Enjoy the music. 

You're chasing something that is unattainable or something that don't exist in most cases especially if you have been doing this for quite some time. When you finally get a system that sounds great then leave well enough alone. Make sure you have a few extra pieces laying around and lots of cabling to change things up when you get bored. Still, this is a lot fun, no telling what awful things you might have gotten involved in if it wasn't for this hobby.

I hear what you’re saying. I currently own the best system I could put together. I also had a stroke a year ago (definitely causes me frustrations with things, including my hifi daily expectations), some days I love the Hifi some days I think what’s the next step to raise the level. I find some speaker adjustments always get me settled down, a small adjustment changes the sound I’ve become accustomed to. Not sure if it’s better but it is a different sound that reminds me I have a great system. I think I get so accustomed to my set up and sound, I get a little bit bored with it and start thinking what else do I need to buy. I’m retired so the last thing I need is a new piece of equipment, I spent the last 20 years getting my system to its current level, which I know is quite good. Don’t know if this helps you, but it’s what I’ve learned over the last few years. Going to Axpona doesn’t help me either, but it’s a fun time with my hifi buddies. 

The age old question.  For someone my age (68) i remember loving music comin out of little am radio back in the 60's.  However, I also LOVE hearing all these new high res versions of classics on my 35k rig.    To me, the quest was always worth it.  Maybe why my jaw still drops when hearing some 500k system at a show I know I will never own.