Correct Way to Invert Phase

I have a Mark Levinson No. 32 preamp (on loan to a friend) that allows for inverting polarity by remote control.  On most tracks, I cannot reliably hear a difference.  On some, there is a preferable polarity, on others, one instrument will sound better, but another one or the vocalist suffers (there is no guarantee with multi-track recordings that the polarity is consistent throughout the recording).  I concluded that it did not matter enough for me to drive myself nuts doing a track-by-track evaluation (different tracks on the same album may have a different preferable setting).

There are test CDs that will present the same track with  different polarity so that you can hear the effect for yourself.  The Chesky Jazz sampler is the one I have.  Chesky is very meticulous with their recordings, so, they are good for such testing of subtle effects.  

Just speculation on my part, but, I bet speakers with drivers that are time aligned on their baffle and have crossovers that also do not shift phase (1st order crossovers) might make it easier to discern changes in polarity.  Of course that would be the case too with full-range drivers, including electrostatics.