Down sizing

Looks like your primare pre and amp should do all that you want.  Plenty of IO. 2ch bal in, 2 out, 3 unbal in, line and pre 1 unbal out, you could run your phono pre thru primare pre?

I’ve run everything through the Primares in the past. Just wanted a touch of warmth added on the vinyl side which the Tandberg did. Would tubes help with that? What would be the downside of a tube preamp? Kind of dreaming of an Audio Research SP 20...

It seems to me a switch box would solve your immediate problem of switching speaker wires, just a matter of finding, or specifying, the right connections.

From what I’ve read about the Tandberg unit you have, I think you will be hard-pressed to replicate its sound.
 

As for down-sizing and changing up your system, that’s another matter. It seems like more manufacturers are offer phono stages on their integrated and pre-amps these days. I would identify manufacturers whose ‘house sound’ (voicing) appeals to you, check their product line for preamp offerings which include phono stages, and see how they’re reviewed. Contrary to some reports, I have found integrated phono stages from reputable manufacturers yield acceptable results with much easier system integration issues (one less set of cables). Unless you have a moving coil cartridge with special loading requirements, you should be fine with standard offerings. If not, you can always add a separate phono stage afterward that integrates better with your chosen power amp. 
 

I can sympathize with the itch to upgrade or ‘change-up’ to the system.* For me, it is an important part of staying engaged with the hobby. Personally, I think you chose well to develop vinyl and digital systems separately. Another ‘solution’ would be a second set of speakers. :-)

*I chose a tube power amp with solid state preamp; now I’m wondering what a solid state amp would sound like —with a tubed preamp. The grass always looks greener . . ..

My 3 cents (2 cents @ 50% inflation rate)