How to integrate stereo and multi channel?

I need advice. Here’s my problem: I listen to both stereo and 5- and 3-channel music. How can I use the same set of left and right speakers for both stereo and multi ch music?  That is, is there a way to configure the preamps/amps in such a way that I can use only one set of speakers? Note: this is about ONLY music; I don’t do home theater. 
I have a stereo preamp (Pass Labs) , a stereo amp (First Watt), a multi ch preamp and multi ch amp (both Parasound). Plus Tannoy speakers all around. For multi channel music I have a NAD SACD player, and a server hooked up to an Exasound e38 multi ch DAC—both with 5.1 RCA outputs. 

I’m happy to replace the parasound amp and preamp if needed.

I guess I can use a high quality speaker switcher like the Luxman AS 55, but I don’t know what the degradation in signal quality will be.

 Thanks very much.