I'm in the camp that an audiophile grade switch can provide sonic benefits.
Your router is not an audiophile grade device.
I inserted an English 8 Switch into my setup a year and a half ago. It yielded immediate sonic benefits. A decent switch will re-clock the signal and help de-jitter and de-crapifie the signal.