My audiophile system is only used once a week for 'intense' listening sessions, this system demands my attention so using it for casual listening is a waste. Most of my music listening done via completely mediocre audio system at work. I much more enjoy casual listening with this system since the music can still demand my attention while the sound quality is completely moot.
I've never used an audiophile system for casual or daily listening, not enough hours in the day, between work and other hobbies I only have limited time for serious listening. The other thing for me is 'special' experiences become the usual or mundane when repeated too often, keeping them special means limited exposure for me.