Looking at your gear there is no way this should sound bright at all. Your gear tends towards a very nice, clear sound with just a touch of warmth. Very nice set up. As a point of reference my system tends toward brightness, so I always look for warmth from my sources, and after trying many different DAC's(Benchmark DAC-1 USB, MHDT Labs Havana, Bel Canto DACIII, PS Audio dac3, Berkeley Alpha DAC, Tube Audio Design PreDac) to go from my AppleTV to my preamp I choose the Bryston, and I am finally no longer in the market.
But, you are right, you can't use an RCA interconnect for a Coax interface. It's an analog not digital cable and you are not hearing the actual output from the DAC. If you want to find out what your DAC really sounds like, go buy a cheap Toslink cable and run it from your source, you will be shocked at the difference. When your new, good, better digital cable comes, you will be even happier.
Good luck.
Good luck.