Bryston 2WSB switch box to choose amp that drives speakers

I really want to be able to choose between a low power amp for tv/background music and my good amp for serious music sessions. I'm tired of amp rolling, have posted recently about this and got good advice not to use two sets of speakers cables but only one live amp. So I'm thinking of a device like the Bryston, but have the speaker outputs be the amp inputs and the amp input be the speaker output. As long as only one speaker is selected at a time...electrically that should work. My only concern looking at the schematics is that the red positive leads are wired together and the black negative leads are switched. Could this still cause voltage to be applied to the unpowered amp? If so would swapping polarity work? There's a schematic on this page. 
https://reverb.com/item/10151350-bryston-2-way-speaker-switch-box-model-2wsb-sc1
Any help on this would be greatly appreciated.