@auronthas The reason your amp is noisier with the balanced input is that the amp itself has a single-ended input; its not internally balanced. So the balanced input goes thru a separate circuit which adds noise.
IMO/IME this is a sloppy way of doing things since the balanced operation is more of an afterthought whereas again IMO it should be the star of the show.
Normally if the internal circuitry were balanced-differential, the single-ended input would be noisier as there would be less noise cancellation.
This also explains why there is less noise when the balanced DAC is connected; the opamp that does the balanced to single-ended conversion is able to pick up noise with no input load.