The easy answer is yes, Everything matters. But, and there can be a lot of buts. Age, hearing, room, quality of the complete system, listening habits and many more things that I’m not thinking of at the moment.
I used an USB, A/B switch behind a streamer and a hard drive on my system and I would say I heard a dulling of the sound, but maybe you won’t. Buy the splitter and see what happens. I will also say that USB cables can make a difference to, but…
All the best.