You are simply not gaining the real benefits from an XLR balanced connection. It is certainly possible that you still prefer the sound using the XLR output. If it is due to the gain, I think what you are hearing has more to do with a relative volume shift than an actual quality difference.
Are you able to use the connections in parallel and then switch back and forth and then use a dB meter to check the relative volumes. People generally prefer the louder source.
Are you able to use the connections in parallel and then switch back and forth and then use a dB meter to check the relative volumes. People generally prefer the louder source.