Playing louder has at least 2 affects. For one our hearing response varies in the bass and treble with loudness. Both roll off earlier at low levels. Flat is around 100 dB if I recall correctly and that's too dam loud and can cause ear damage. Fortunately most software is engineered to sound good at lower levels. But mastering varies and the correct level can vary from recording to recording.
Another problem can be dynamic linearity(my handle). This isn't the ability to play loud cleanly(although that's a part of it). It's the ability to change levels accurately from micro to medium to macro, to not compress level changes. Unfortunately many recordings compress level changes. And the affect of this compression is most at low levels and playing loudly ameliorates this affect. Many speakers are guilty of dynamic compression. For lower level listening lack of compression is essential to sort of sounding live. Picking dynamic speakers is essential.