Some recordings sound better one way. Some sound better the other. If you notice a big enough difference then you could try switching for every recording and write down which way sounds best, then switch every recording. Or plan your listening by grouping tracks that sound best in phase and playing them back to back. But I found this to be too hard. So what I do now, split the difference. Wire one speaker each way.
Incoming.....!
Incoming.....!