RCA are the limiting feature. You may add XLR adaptors if you like, but nothing will change.
In fact, and I’m happy to be corrected, the XLR may actually have to do extra work to deal with the single ended signal. I try to keep things simple.
I don’t pretend to understand your description of the setup very well, so what I said is a general statement.