A low-jitter USB converter will easily beat the Toslink output from the Mac. The best way to go is I2S, but your DAC must have this input. Next to that, S/PDIF coax will be great. Whatever cable you end-up with, it should be a high quality cable. Makes a big difference. Cables in the $100-300 range will not get you there.
Firewire is no better than USB. It all depends on the implementation/design. Either can be great.
Steve N.
Empirical Audio