Maybe get a really good preamp with a stepped attenuator. The Primaluna, while being a decent preamp, has a Alps attenuator. While, not bad, it's not stellar. Go for the preamp as if it will be the last one you buy. Everything plugs into the preamp...so you will get an improvement everywhere.
Consider a full function preamp:
You could also get a preamp (luxman or McIntosh) that includes a decent DAC, phono and some EQ functions and clean things up.