For me, listening to recorded music involves three essential factors. The first is the quality of the playback system. Unless there's a new component in the system I already know what it sounds like, so at the moment I'm listening there's not much point in analyzing it whether I'm listening casually or critically.
The second factor is the quality of the recording itself. Someone else here said that listening isn't analyzing how the music is recorded, but rather analyzing how it's played. For me that's true to some extent, but if it's simply a bad recording I think it's hard to ignore no matter how casually I'm listening since it's going to affect my ability to accurately analyze the music. I may or may not put with it depending on how interested I am in that bit of music. But if it's a reasonably good recording, and one can usually tell pretty quickly, then there's no point in analyzing it further.
The last factor, of course, is the quality of the music being played. I tend to save my critical listening time for music that's at least a little more complex and that requires full concentration to "analyze" and appreciate. Then there's the less complex stuff that I might listen to while reading or doing something else that shares my attention. That music will normally be played at a lower volume, and I'm hearing it but I'm not fully engaged in it, i.e., listening casually. Of course, there have been many times when I've been hearing but not "listening" to a piece of music when something yanks my attention away from whatever else I was doing and requires me to turn it up and shift into critical listening mode. I love those moments!