Can you build a transparent XLR selector box?

I know that the answer is yes, as in every professional recording studio employs patch bays.

But have you personally tried one in your hifi signal path and were able to notice any effect on your sound?

Its important to understand that most studio equipment supports the balanced line standard also known as AES48. Most high end audio home equipment does not- although there are a few exceptions which are rare. Because if this you'll find that a patch bay or XLR switchbox may well affect the sound.