no damage, but might change the sound. you should be able to sell your XLR's for the same price as buying RCA's. XLR cables tend to cost same as RCA and sometimes a little or a lot more for same model...
XLR to RCA adaptor
So I have a new integrated Decware tube amp coming and its inputs are RCA. All my current sources gear has both XLR and RCA outputs, however I have been using XLR interconnects in the current system. Because I'm cheap, and don't want to spend 2000 dollars to get the same cable in RCA format, can I just get an adaptor that will allow me to go from female xlr to male RCA. I know they make them, I'm more concerned about will I damage the equipment?