Is optical mostly a waste of time versus Ethernet?

You might want to read this article and the other linked articles.

If you are convinced there is low level noise coming over your Ethernet cable, you might try one of a variety of optical isolation devices near your server, such as the converters and fiber cable discussed here, to something like the GigaFOILv4-INLINE Ethernet Filter, which is an optical filter, to something more expensive like the Sonore offerings or maybe the Network Acoustics ENO or Muon, which are unpowered, transformer-based isolation devices. I suspect you only need one of these isolator solutions. The "audiophile" switch thing seems superfluous. I am currently running the GigaFOIL unit powered by a LPS at the end of my Ethernet cable run, close to my server. It seems to sound just as good as it did with all the other stuff connected.

None of this is "one size fits all."  The results/performance can vary depending on what is being used as a "server."  With my Mojo DejaVu server (and with the Antipodes DX3 I still own), I have the option to output an ethernet cable from the server to a separate Roon endpoint, or renderer, such as the Bricasti M5, Metrum Ambre, Sonnet Hermes, or similar.  In some cases, removing the renderer from the server unit can be sonically beneficial, but not always.    

I also have the option with both of the above servers to output USB directly from the server into my DAC.  In other words, both of those servers can also function as the renderer.  In the case of the DejaVu, the USB output board is the excellent JCAT USB Card XE, which results in the USB output performing at least sonically equal to using a separate renderer (when using the renderers and DACs I have tried in my system).  I suppose, I could try inserting the ENO between the server and the renderer to evaluate whether that makes a sonic difference.  

Eno between server and renderer would be interesting

I tried that today - Ethernet from router to switch (located near router) then 45 feet from switch to GigaFOILv4 (located near the server) then short Ethernet to server and short Ethernet from server to ENO then captive Ethernet from ENO into Metrum Ambre renderer.  AES/EBU XLR cable from the Ambre into the Mystique EVO B4B21 DAC.  

Simple without the fiber converters and accompanying power supplies.  It sounds really good.  I don't know why exactly but the sound was full, rich, and musical with a wide range of music from Coltrane to Chili Peppers.  I want to next compare this to the same set-up into the server, but then running a USB cable from the USB output of the server directly into the USB input of the DAC, without using a separate renderer.    

There are several power supplies in most people's digital signal chain, including those powering the modem, the router, and any switches, as well as power supplies inside of the servers and DACs that process the digital signal.  Ethernet itself is galvanically isolated but some believe that optical isolation near the endpoint can improve on that wrt noise reduction, which is probably where the idea of using fiber came from.  The Network Acoustics stuff is unpowered and isolates using transformers, which do not break the electrical connection the same as optical.  My Gigafoilv4 is powered by an HDPlex LPS so there are no switching supplies in the chain following the optical isolation offered by the Gigafoilv4.  Using the ENO just before the Ambre endpoint sounded good yesterday so if I switch to using USB directly from the server to DAC, I may try using only the ENO just before the server, and no other boxes needing power supplies.  I guess you just need to try stuff and listen.

By using a simple Linux OS focused only on doing audio, these(and many other good streamers) eliminate lots of this noise.

I believe this to be an important point.  Many of these add-on enhancers originated in response to cleaning up EMI/RFI noise resulting from computer sources.  While today's dedicated servers are essentially still computers, many have been significantly improved by using Linux OS, upgraded power supplies, and improved connection interfaces.  This may be one reason why there is such a wide range of impact and improvements reported by users of these add-on enhancers.   My server uses a Linux OS, choke power supply, and the well-regarded JCAT USB Card XE audio output board, and the changes/improvements I have heard from a variety of this stuff (such as the ENO, Gigafoil, switches, and fiber) have been subtle at most.  

Do you really need to run fiber from the router to the server or is an optical isolator (such as Gigafoilv4)  just in front of the server sufficient to clean up any noise on the Ethernet cable?  Is a transformer isolator (such as the NA ENO or Muon) just in front of the server sufficient for the same purpose?  If your only wired connection out of the router is your audio system, why do you even need a switch and how can the switch possibly improve sound quality in that situation?  It does seem that simplifying the digital signal path and removing converters and power supplies should in general reduce noise, unless there is something amiss with the digital signal delivery.

How could any of this make a difference but it does to those who have experimented.

OTOH, that is not true for everyone.  I have tried a number of these things with results being so subtle as to question whether there is really a difference.  It's not that I summarily dismiss the possibility that this stuff can make a sonic difference, but, at least in my room with my equipment, none of this stuff so far has made an easily identifiable change or improvement.