I was always of the general impression that soundstage was a relationship between space (audio reflections - both as captured in the recording and from our space), time (frequency - and how it interacts with space and nth order reflections), and our brains attempt to make sense out of things and line them back up when information feels like it’s missing. By flipping the channels in an audio piece you are familiar with, could the brain be more preoccupied with trying to the push the left back into the right channel? The relatively small imperfections, which the brain would normally interpret as a range of frequencies being narrow / wide / up / down / forward / back, is now such a subtle discrepancy in the music compared to the glaring problem of the drums being on the wrong side of the room.
Have you tried this with a piece you are not familiar with? Do that for a few weeks, get accustomed to the soundstage as presented, and then switch back to the correct channels.
Otherwise, this makes absolutely no sense to me.