If you are hearing distortion when the "music swells" and you are not playing at high volume, it could be that you are over-driving some other part of your audio chain. For example, if you have a DAC with a high maximum output level that is feeding a preamp that has a low maximum input signal, this could cause the distortion you are getting.
Also, if you are doing any kind of equalization such as boosting bass or treble, or room correction, this may be causing clipping of the DAC or preamp (depending on where it is done).
Also, if you are doing any kind of equalization such as boosting bass or treble, or room correction, this may be causing clipping of the DAC or preamp (depending on where it is done).