Considering switching to XLR cables,

I highly recommend looking into Cerious Technologies cables. I’ve used very expensive cables until I found Cerious. Bob’s XLRs are more resolving than other cables three to four times the price. Just take a look at their website. 
 

https://cerioustech.com
 

There is really very little difference between XLR and RCA. Most of us aren't going to need a cable longer than 2m.

I've had some great RCA cables and poor sounding XLR cables and vice versa.

Many high end valve amp manufacturers only provide RCA inputs.

I would say the OP's dilema is in finding synchronicity in his system - we've all been there!

Over the years I've used Cardas as my reference, they get along with most electronics.

I am sure it's gear dependent but in my experience, with two highly regarded sources, a Modwright Oppo and a Lumin A1 music server, the XLR outputs were far superior to the RCA's. 

In a little second system I ran a variety of amps and speakers with a Lumin D2 as source....because the pre amp and most amps only accepted RCA I was forced to go all single ended.  Amps were SS and tubes, speakers were Omega (2 different models) Sonus Faber, Harbeth, Sterling Broadcast and Klipsch.  I always had the same issue....the midrange was awful no matter what combo I tried.....kind of hollowed out and tinny.  Finally realized the one common denominator was the Lumin, thought about selling it then realized in all my Lumin experience I had never run single ended output cables in my primary system, always XLR, so I purchased Cardas XLR to RCA adapters on the preamp side with XLR cables....mid range magic returned.  Pisses me off I didn't realize it earlier as I fired some really nice speaker amp combos.

Unless the equipment is actually true balanced differential there is no reason to use XLR’s . As an example I have a Mark Levinson no.585.5 amp that is balanced. But my DAC is a Chord Dave. Even though my Dave has XLR outputs it’s not balanced (per Rob Watts) so using XLR’s is pointless. BTW I use Audience FrontRow Reserve interconnects

My system is exclusively XLR, except where it’s impossible—like my turntable and tube preamp, which use RCA. For cables, I use Mogami with Neutrik connectors. You can build or buy them pre-made; the pricing is very reasonable.

My thinking is simple: if it’s good enough for professional studios, it’s good enough for me. There’s no need to spend $4500 on XLRs with carbon fiber shells and silk-wrapped cables. I’d rather use well-engineered, proven cables with top-tier commercial-grade connectors—nothing flashy, just reliable and built to perform.