Does the position of attenuation matter?
The answer in general is no. If you really start splitting hairs though.... what's happening in the big picture is you're amplifying a very weak signal in the phono stage, then attenuating it in the pre-amp. Since whatever small amount of noise is in the phono stage, whatever it is there is more of it with more gain. So theoretically at least it should be better to use less gain. Especially since that extra gain isn't really needed, as evidenced by your low volume setting.
Another splitting hairs to think about is the volume control itself. If it really is a pot they are never really perfectly linear and note I said perfect your question is splitting hairs so let's split hairs. There's bound to be situations where the same measured volume output sounds better at one position than another. The same is certainly true of stepped volume where even if every resistor is perfectly matched they will still vary simply by how much signal and how often they are used.
All this crazy technical talk mainly serves to underscore what I say time and time again: Go and listen. You will see. If you hear a difference, great. You know what to do. If you don't- great! You know it doesn't matter.