nad 541i--output to interconnects or digital coax

It all just depends on what DAC would you prefer to rely: of 541i or the receiver's one. I had 541i with 761, and my opinion that DAC in 541i is far better then in the receiver, my dealer confirmed me that 541i circuitry is well refined in comparison to any of their receivers.
And, the best use of the digital out is to connect a good stand-alone DAC of better quality then 541's own. This would give you more flexibility, while I'm not sure it's money-wise. And, don't forget to get quality interconnects!

I would suggest shop good interconnect 1st.Keep in mind you can use it elsewhere in the system later if you decide to use the preamps dac.

You probably have to try it out. My guess is that using a good interconnect will give you a slightly better sound since the 762 does have bypass for analog inputs.