NAD M5 no sound from Digital Coax out SACD

XLR may or may not give better performance. Why not just try it both ways and listen?

SACD won't play through digital hook-up or DAC. Use a pair of analog interconnects from the M5 directly to an input on your pre.

I assume you have a pair of RCA's from the DAC to the M3 on one input. Continue to use the single digital XLR cable from the M5 to the DAC for CD's and then also run another pair of RCA's directly from the M5 to the M3, bypassing the DAC, for SACD. You will need to change the input on the M3 depending on what you are playing, CD or SACD.