As someone about to re-enter hifi (mid-fi, really) after decades of mediocre listening, I think there are several reasons the hifi market is dwindling. Most notably, hifi is not nearly so necessary for tolerable listening as it was when I bought my pioneer SA-9500 in 1976. I was not - and never have been - an audiophile (by most descriptions). But as a musician, music was important to me and I could not enjoy listening to recordings through the kind of crapy equipment my parents and other casual listeners used.
Back then I had to forgo a first high school era car to but decent sound. Today, decent sound is available at relatively low cost. If I could have had sound equivalent to that produced by a $99 Dragonfly DAC and $250 650 headphones driven by my MAC Retina, I doubt I would have sacrificed a car for better audio.
Now, my standards have increased (while my hearing has diminished) and I am quite willing to spend much more on audio than most people I know with comparable disposable income. Such are my priorities. But diminishing returns on the cost of audio seem to come much more quickly and unjustifiably now, than then.