How large (or small) is the audiophile market?

The market probably isn't growing uniformly. HT is probably doing better than high-end 2-channel. Many don't see that as "audiophile" but the truth is that many people entering the market of audiophile products have started in HT.  More esoteric markets, tube gear and such is gaining small increases in interest mainly from lower-cost products being available from eastern Europe and China. The toeholds are new brands from China that make experimentation affordable. The costs of many high end brands sharply limit the market to later-life affluent (mostly) men. As far as competing for discretionary dollars, audio just doesn't do all that well against rising costs of other necessary and desirable goods.

Like it or not, the torch-bearer for high-grade audio, in the U.S. anyway, is Magnolia. It is a gateway to 2-channel systems. The web is  the road to other dealers representing higher-end or rarer equipment. Audio shows might offer exposure if you live convenient to regular show venues.

Finding products from even mainstream brands like Yamaha is not as easy as in the past, at least not to see and hear in a shop.

Audiophile record sales of new pressing seem to be doing pretty well. I would say that anyone that buys an audiophile record when a "regular" pressing is available is an audiophile. I don't want to start an argument with the digital only folks, but that is a good start. 

They have no problem selling 20-30,000 of $100+ records of very popular albums. Maybe one in 50 audiophiles buys them (this is global). Add on the digital audiophiles and you're probably at a couple million people worldwide. Higher percentages of the population are in they group in England, Europe and Asia.

If you widen the group to include anyone who buys $1500 speakers, you've gone too far because you've included HT people, many who have no idea what 2 channel auto sounds like.

@chenry one of 2 BestBuys in my city has closed and the one still open has Magnolia no more. So I am not sure how real it is. 

The UHQR release of Aja is limited to 30,000 copies and are still in stock so is it safe to say less than 30,000?  ; )