Marco (Jax2), the Duet apparently provided an optical output, as well as a coaxial digital output.
Shlibotnik, did you mean that you had both outputs connected simultaneously, or that you tried them one at a time, with only one of them connected at a time? If you had them both connected simultaneously, it seems conceivable to me that crosstalk between the two inputs within the DAC (after the optical input is converted to an electrical signal) could have degraded the sound due to its impact on jitter.
Also, if possible try a different cable length for the coaxial connection between the Duet and the DAC. The relation between coaxial cable length and signal risetimes and falltimes (which are almost always unspecified) can affect jitter significantly. See
this paper.
And are you sure that your rip settings in iTunes were correct, including use of error correction? Also, as far as I am aware iTunes is generally not recommended for high quality rips and/or playback, especially if it is running on a Windows-based machine.
I wouldn't absolutely rule out the possibility that the wireless link is somehow causing the poor sound quality you are hearing, but I would look elsewhere first.
Regards,
-- Al