'Loudness' is the stupidest name it could have been given, which is why it is misunderstood, then misused.
Leaving Loudness ON, at normal listening levels:
is the OPPOSITE of it's intended use, which is to BOOST Bass, for LOW LISTENING LEVELS, Progressive Boost with Diminishing Volume.
Because: our ability to hear bass relative to mids is progressively reduced at low volumes (equal at normal volumes).
https://en.wikipedia.org/wiki/Equal-loudness_contourSo, you are leaving it ON, finding it Bass Heavy, then turning the Bass Down. Creating and Solving A Problem.
I suggest, at normal listening volume, you turn Loudness OFF, and put your BASS Tone Control back to neutral. Then, use only the tone controls if you feel the frequency response needs a slight adjustment IN YOUR SPACE.
Afterall, the speaker maker's design for an imaginary average room, not your particular space. Vintage designs included L-Pads for frequency refinement, 1/2 attenuation the default, then boost for dead rooms or cut for live rooms, and your personal taste which is probably related to your hearing as well as your space.