Is there any harm in using the main XLR output to send signal to the subwoofers, and using the 1, or 2 RCA output to send signal to the main drivers?Yes. You'll unbalance the signal going to your main amps.
The solution is to use a set of specialized subwoofer transformers made by Jensen, which are optimized for subwoofer operation (and so go down to 1Hz to prevent phase shift or any hint of a low frequency rolloff). They can be used to convert from balanced to single-ended. I use a set in my system for this purpose- MP-1 preamp has dual balanced outputs so one set goes to the main amplifier and the other set to the Jensen ISO-MAX boxes. From there I run a very short set of RCA connections that are only about a foot long to the subwoofer amplifier. Link below:
https://www.jensen-transformers.com/product/sub-1rr/