Hello @mathiasmingus You are getting good advice here from@lalitk and @ghdprentice.
I have gleaned much from these two over the past two years. I am in the camp of everything matters AND I don't like clutter.
First, the streamer is very important. How anyone can appreciate the increase in quality of more expensive amps dacs and speakers, yet say streamers don't matter is ridiculous; they do. Anyone here who has moved up the chain can corroborate that. And, as ghd says, the better the streamer the less the upstream gear matters. But I differ some here and feel it still does matter.
A quality switch, with a quality power supply inserted between your router and streamer does improve my system. Since I split my audio feeds to two nice systems, I actually need a switch. So in my office (away from my audio systems), I am now using a JCAT M12 gold switch with nice LPS to feed those systems. I also have long runs of ethernet, roughly 25 and 30 feet. I can run cables through the floor, drop ceiling and either back up into the family room or down the wall to by basement system.
I formerly used Supra Cat 8, but recently (on @lalitk's suggestion), tried the Link UP 22g Cat 8. It is reasonable from Amazon and easily returnable. I ran it on the floor and down the stairs first, and could easily swap cables. I liked the Link UP better and ran it through the wall / ceiling. Thanks @lalitk.
I also use a Network Acoustics filter, in my case the Muon Pro. It (as @ghdprentice says) makes less difference with a quality streamer, but I still think it improves and smooths things. My streamer is the Grimm MU1 and my DAC is the Tambaqui. But I keep the switch and its power supply in my office by my router (no clutter).
I think you tried a great thing with the wrong switch. I have tried many, and the sub $1000 switches did not impress. I do not know yours, but the English Electric / Bonn 8 / EtherRegen / Netgear are examples.
I would look at, save for and try the Network Acoustics Tempus. AT $4000 it includes its own power supply and definitely improved my system when I had it on trial. Or, the GTT Switch X is also fantastic, and addresses network noise by creating a separate clean network. I expect it will be back in my home soon to stay.
Routers do create noise, network traffic in your home creates noise. Anyone who says they don't hear any noise probably has not listened to a good system with these upgrades. When I put in a great switch, it is not like 'oh there is less noise'. Rather it is 'wow, the soundstage has expanded, the individual voices or instruments are clearly more separated, the image is more focused, and yes the background is blacker'.
Everything matters, and I don't like clutter!
Happy Holidays!! Ken