There are several reasons why high end audio is a niche market. One of the main reasons is that the average person simply doesn't know this equipment exists. I have had to educate several of my co-workers on the fact that there really does exist sound quality which is MUCH MUCH MUCH better than mass-market. They are used to hearing the receivers from Wal Mart, Best Buy, and department stores. They don't even know that there exists stores which set all their gear up like how it would be in a house. They don't know anything about Vandersteen, Krell, or Magnepan because they don't see them at the "big box" retailers.
The other problem is the price. People get turned off when they hear about a $3,000 CD player. They would rather spend that money on something else because they read that "bit for bit all CD players sound the same because they are digital" and don't see the reason to spend big money. They also don't see the reason to buy 7 monobloc amplifiers for $7,000 when an Onkyo receiver has 7 channels in a small box. Blame defunct magazines like Stereo Review for emphasizing cheap gear while at the same time mocking the expensive equipment.
The root of this is education and advertising. I'm sure that if high end audio was marketed like diamonds, expensive cars, and wedding dresses were then we'd see Arcam and Jolida as often as we now see Sony and Pioneer. People need to know this stuff exists and they need to know that it is worth the money. Until then high end audio will remain a cottage industry.
