Dave are you trying to use the speaker selector switch *backwards*? Selector switches are normally connected with a single source component (receiver) as the input, & then you'll have two or three pairs or speakers on the switched output sides.
It sounds here like you are trying to drive a single pair of speakers from two separate receivers? You can probably do that too, but everything will have to be setup backwards. By that, I mean the speakers would have to be wired to the input side of the switcher. Then the two receivers would be wired to the output sides (A & B respectively) of the switcher. To play receiver A you switch to A & to play receiver B you switch to B. As a precaution you should probably not simultaneously have both receivers powered up & driving their outputs at a high level, perhaps not powered up at all when switching between them. If the switcher contacts don't break before make then you could have some serious smoke.
It sounds here like you are trying to drive a single pair of speakers from two separate receivers? You can probably do that too, but everything will have to be setup backwards. By that, I mean the speakers would have to be wired to the input side of the switcher. Then the two receivers would be wired to the output sides (A & B respectively) of the switcher. To play receiver A you switch to A & to play receiver B you switch to B. As a precaution you should probably not simultaneously have both receivers powered up & driving their outputs at a high level, perhaps not powered up at all when switching between them. If the switcher contacts don't break before make then you could have some serious smoke.