There is a company named Zektor that makes very high audiophile grade switching panels among other things. I have one called a MAS7.1, so named because it can handle up to 7.1 channel analog surround system. It will also switch three unbalanced stereo inputs into one output (or vice-versa) using silver make-before-break contacts so there is absolutely no switching noise. Each contact is a straight through connection. The switch will also handle switching for 5.1 or 7.1 analog outputs in a multichannel system. It's price is I believe around $800 but they are sometimes available used for much less.
Only something of this quality design and construction should be used to do the switching you wish to do.