What is your preferred listening volume in the room? Have you bought a sound level meter in order to get a real number to work with? Note that one person's "loud" is another's "medium" and so on. The size of the room is also a factor along with whether or not you use a subwoofer (deep bass takes the most power.)
Finally, keep in mind that the position of the volume knob is not an indication of whether or not you are at or near the amp's max output. Here, the voltage output of the source and the input sensitivity of the peramp/amp play a big role. For example, a while back I tried using a unity-gain preamp with a low sensitivity power amp and having the volume knob turned to max only gave me moderate volume. I had to switch to a preamp with additional gain to get the sound I wanted, even though I did not change power amps.
As you can see, lots of moving parts. But your first move is to figure out your desired listening level.