For digital at RF frequency, 75 ohm BNC should be the best thing to do. 50 and 75 ohm BNC are much matured technology, already being used in high speed communication for a long time. These connections and cables can handle much higher frequency and way lower noise than audio stuff at high frequency. RCA and XLR are developed for audio frequency in the beginning. A good RCA or XLR for higher frequency digital would cost much more at this moment. Our cable TV and cable modem can run very high speed digital signals, and you don't want to use RCA/XLR on those things, you use BNC and coaxial cables....
If your tranport and DAC have good output/input design matched to 75 ohm, a farly cheap BNC and coax will have less distortion and lower noise than $$$$ RCA and XLR.