I am sure it's gear dependent but in my experience, with two highly regarded sources, a Modwright Oppo and a Lumin A1 music server, the XLR outputs were far superior to the RCA's.
In a little second system I ran a variety of amps and speakers with a Lumin D2 as source....because the pre amp and most amps only accepted RCA I was forced to go all single ended. Amps were SS and tubes, speakers were Omega (2 different models) Sonus Faber, Harbeth, Sterling Broadcast and Klipsch. I always had the same issue....the midrange was awful no matter what combo I tried.....kind of hollowed out and tinny. Finally realized the one common denominator was the Lumin, thought about selling it then realized in all my Lumin experience I had never run single ended output cables in my primary system, always XLR, so I purchased Cardas XLR to RCA adapters on the preamp side with XLR cables....mid range magic returned. Pisses me off I didn't realize it earlier as I fired some really nice speaker amp combos.