I always use an XLR connection because, in my experience, the sound is better (with volume matched) than the unbalanced outputs.
If the design is truly balanced, the higher output level from XLR can generally contribute to better sound (again, when volume matched) due to a higher SNR and lower noise floor in terms of THD+N. Apart from the well-known benefit of better noise rejection, handling the signal path separately for + and - significantly reduces crosstalk (see below).
The only potential adverse impact is that the output impedance of the XLR connection might be 2-3 times high. If you have an amplifier with low input impedance, such as a valve amp, you might run into impedance matching issue.