In the latest issue of Stereophile, reviewer Jim Austin addresses this topic in the review of the Benchmark DAC3. He stated " most systems achieve their best noise performance in the top half of their volume range".I think the operative words there are *most systems* (certainly not all). Just "listen" to your amp with no source input by putting your ear up to the speaker (tweeter) and hear what happens as you raise the volume. The more efficient your speakers are, the more "sizzle" you may/can hear.
All amps distort the more their volume is raised. My vote is less amp volume and more source gain, but again, you don't want clipping and that sort of thing so it could be a fine balancing act to optimize the two.