Two amps to one speaker

I use an Ayre VX-5 Twenty to drive a pair of KEF LS50s.  Occasionally I'd like to use a pair of Cary 572 SE monoblocks.  Would I create a problem by connecting cables from the Ayre with spades and those from the Cary with bananas?  Of course, only the Ayre or Cary monoblocks would be powered up at a time.

The cables from the Ayre to the KEFs is Cardas Clear Day with spades at both ends.  I suppose I could have one end converted to bananas, and that might be the proper solution.

db