Hi-Fi can be less expensive than anything

Being an audiophile is never going to be a truly cheap pursuit. Equipment is only one element and probably the least expensive. A serious music collection can easily cost multiples of what is spent on equipment. Plus you have to consider the real estate element. At a certain point most people begin to realize the importance of the listening room. The true costs of a dedicated listening room with even a small amount of acoustic treatment will be greater than the price of the equipment. I knew I was getting serious when my real estate living decisions revolved around listening room considerations.

The whole point of a dedicated room is that it can be optimized for sound quality. As far as being a "man cave", take a look at some of the pictures in the virtual systems section where any number of people have aesthetically wonderful looking rooms. For me I don't readily allow people access to use my primary system, but my music collection is playable in six different rooms of the house. Wireless tech does make this convenient. In the end loudspeaker based systems need physical space to perform their best and whether you rent or own the cost attributable to this requirement can be substantial.