While it is very common to call this "phase" inversion, calling it polarity inversion makes more sense to me but, that is a minor point. The first thing to be aware of is that not everyone is sensitive to "absolute" polarity inversion which is, both channels being inverted at the same time. I know a couple of people that say they don't hear any difference. There is also the not common issue of "relative" polarity inversion which is, just one channel being inverted, usually by an incorrectly connected speaker wire on one channel at either the amp or the speaker (red to black and black to red). I think that most people can hear that something is wrong when relative inversion exists in a given system.
Paul's explanation is a bit on the technical side. Here's what I perceive when comparing correct polarity to incorrect polarity The sound with correct polarity will project outward toward the listener and there will be more space and air around the sounds and voices. Incorrect polarity will move the soundstage back a little and will flatten the soundstage as well. There will be less space and air around the sounds. In my experience the effects of correct/incorrect polarity will vary with speaker design in particular, speaker placement in the room, and all of the other variables with room dimensions, wall treatments, etc.
If you don't pick up on the differences at first, pay attention to how the bass sounds and also high sounds such as high hat cymbals as you try each switch position.
Yes, as yogiboy said, some recordings are polarity inverted.