Everything matters, but if you're looking to improve sound quality of your digital playback, the DAC is where most of the magic happens, and that's where I suggest you consider investing.
I had a Nucleus and an Aqua La Voce S3 DAC. I replaced the DAC with a Rockna Wavedream Signature DAC, which resulted in a significant improvement in everything - dynamics, clarity, detail, soundstage, etc.
Many reviews and end user testimonials said that the DAC would perform its best with the matching Wavedream NET server and connecting them via i2s. Replacing the server was honestly somewhat disappointing. The difference in sound quality, if any, was negligible. The biggest upside to the Wavedream NET is that it has a very good CD transport that is completely isolated from the server and provides excellent sound quality when fed to the Wavedream DAC.