As long as your speakers are in phase with each other it will sound like it should.
Polarity DOES matter to those that can hear it.
In pop music, IF a live drummer is used, the kick is often recorded without the outer head or with a hole in the outer head for the microphone. This generates a positive pressure in phase with the beater head contact thwack.
On a system with reverse polarity the kick sucks in on the initial hit and it is clearly audible.
Electronics are less of a problem today than in the past when they were sometimes inverting.
That being said, there is no guarantee that the source material is polarity correct.
The best one can do is ensure one's system is polarity correct and hope for the best. Or flip polarity, determine which sounds better and mark the source, adjusting as necessary. All in all, a PITA