You likely will experiience no difference. However, instead of an adapter, I would make or have made a quality cable with XLR on one end and RCA on the other.
@soix You are assuming there is an inherent benefit to balanced cables. Many, if not most, do not agree with that assumption. Having multiple inputs=good. Having ability to accomodate both xlr and rca=good. Needing another RCA because your component doesn't have xlr out=common problem.