I've had mixed line systems too. SE sources feeding a balanced power train. No prob with (if needed, good adapters). I've also run the balanced amp as SE with adapters.... again... no prob.
For me, the XLR path is quieter. the added 3-6 db gain was inconsequential. The quieter, darker backdrop made more a diff.
Was it enough to keep me in the XLR camp? Nope. I'm all SE now. It's all on you, how you wish to proceedd. Electrically, it ain't gonna hurt mixing inputs. Sonically, if the proper cabling is used, I don't think the diffs will be great enough that I'd be making wholesale changes either.
Good luck.