"swapping the speaker cables by hand" That's the way most of us do it.
What about turning the amps around so the speaker connection are facing outward?
How often do will you be doing this?
If I had to do it again, I.would still go with a box, a bunch of dual banana jacks and a pair of dual banana jumper cables. I don't think there is a switch that is as neutral transparent as a banana plug or screw down connection. High current solid state switch? Works for line level switching. I'll throw this one at McGowen.