Before I retired, I remodeled my man cave. As part of that, I sold my McIntosh system and gave away my beloved 1980's system and began building my retirement system. My man cave has a dedicated listening room; a pool table area; an area for my HO train layout; a bar area; and a TV room.
During the winter months, I try to spend several hours per week just listening to music. At other times, I spend a few hours working on my HO train layout while listening to music in the background. On occasion, my wife may join me - she loves music as well, but she's OK listening to a small system in our sitting room or our whole house system.
During the summer months, I spend less time in the house thus a little less time listening to music (at least during the day). My wife is an avid golfer, I don't play golf. When she's on the course, I'm sometimes out with my photography gear taking pictures or riding my bike.
I think I've found the right balance between sitting on my butt just listening to music and enjoying my other hobbies and favorite activities.