Can a tube amp be damaged by a speaker switch?

Help please from the tech crowd.

I had a room with two systems....one for audio and one for TV. The audio is a Leben tube amp into Altec Valencia speakers. The TV is a Marantz receiver into a group of five budget small speakers. My goal was to use just the Valencia's for both amps because the big Altec's are so much easier to hear dialogue on the tv. I also want to clear up wire clutter in a cramped cabinet. 

My solution was a Luxman AS-55 speaker switch which a very high quality unit designed to allow 3 speakers and one amp, or in reverse setup, three amps and one set of speakers. During set up, I had a question and called the Luxman US manager who picked up the call and was so gracious with his expertise. Great service. The problem is after hearing my questions he said..." we do not recommend using a switch with a tube amp as running tubes without a load can damage the amp...even break it. 

I can use the switch if the the tube amp is turned on only when running into the speakers. The problem with that is the Leben really needs a good half hour to warm up and I would like to do that while watching tv. It will make it cumbersome to switch back and forth between the two. 

Eliminating the Luxman AS-55 switch and just using two sets of speaker wires is an option, but I really do not want to have to plug and unplug the bananas on my Audience cables as that can only hurt them and its a hassle. Im also worried about potential damage to the Altec's with two amps hooked up. 

My question is for the experienced tube users. How much risk am I running if I use the Luxman switch as I intended? Can it really damage the amp or is the Luxman rep exageratting the risk for laibility reasons. What is the potential damage?