How large (or small) is the audiophile market?

Like it or not, the torch-bearer for high-grade audio, in the U.S. anyway, is Magnolia. It is a gateway to 2-channel systems. The web is  the road to other dealers representing higher-end or rarer equipment. Audio shows might offer exposure if you live convenient to regular show venues.

Finding products from even mainstream brands like Yamaha is not as easy as in the past, at least not to see and hear in a shop.

The market probably isn't growing uniformly. HT is probably doing better than high-end 2-channel. Many don't see that as "audiophile" but the truth is that many people entering the market of audiophile products have started in HT.  More esoteric markets, tube gear and such is gaining small increases in interest mainly from lower-cost products being available from eastern Europe and China. The toeholds are new brands from China that make experimentation affordable. The costs of many high end brands sharply limit the market to later-life affluent (mostly) men. As far as competing for discretionary dollars, audio just doesn't do all that well against rising costs of other necessary and desirable goods.