Critical listening means optimizing the setup and listening closely to what the system is doing. I use it when evaluating changes. Otherwise, I just turn it on and listen more casually to the music.
If critical listening used as a tool makes my system better, then yes, it does bring some enjoyment.