I wouldn't be inclined to worry about it either, but maybe one other area to look at is your source. If the output voltage is high it could also have an affect on the volume level and may be adjustable depending on your unit. Just another thought.
Tim
Tim