I am not suprised at your findings. I have always been advised that one should set iTunes at full volume. Then set amp such that your preamp can be run between 10-2 o'clock depending on desired volume ( therefor around 12) . This is usually the sweet spot on most variable pots used in a preamp.
One problem you have is the total gain you have between preamp and amplifier causing you to attenuate much of the available gain in the amplifier. Your compromise seems the best fit.