"Has the cost of HiFi gotten a bit too much."
That's a question of opinion. Certainly, the audio market tracks closely to other luxury goods markets. I'm a bit amazed you can spend 100K on a watch or any number of things, like an amplifier. But is it too much? I suppose it's not because the high end exists because there are people at the top of the pyramid that can afford to pay just about anything, for anything.
The second part of the question -- "it seems like high-end components prices have really risen more than inflation" -- is something we could find out, if someone was enterprising enough to look at the data. My guess is that prices have exceeded inflation adjusted pricing. But, the quality of materials, design and build, etc. are truly extraordinary these days--and that does cost something to achieve.
Now, do I feel a bit of shock at that $145K TechDAS Air Force One Premium turntable?
[Falls out of chair.]