Does ethernet switch quality matter when using fiber connection?

Not enough information about your LAN or streamer to give you a meaningful response.  But at some point you have to convert fiber to copper and/or copper to fiber.  And for that conversion, quality matters.  As does the PSU for the converter/switch.

wall > standard ethernet cable to switch > lumin x1 via fiber. basic setup

When you say "wall" you mean coming out of your router, correct?

Most routers output through Ethernet and most streamers/servers input through Ethernet.  Therefore, in most cases, people need two conversions.  If your Lumin has a direct optical input then you have an advantage by needing one less convertor.

Some folks convert the Ethernet to fiber for only a short distance just ahead of the streamer/server as sort of a galvanic break.  There are companies who make small boxes that filter and do other magical things and many of those also provide  galvanic isolation to reduce/eliminate EMI/RFI being transmitted to the streamer/server.  I convert to fiber imediately after the router and then again just before the streamer/server.  Most of my 45 foot run is in fiber.  Honestly, I cannot hear a difference whether that run is Ethernet cable or fiber but I have left the fiber in-place.

Regarding your switch, @audiom3 nailed it with the importance of using a LPS on the switch (and on the Ethernet to Fiber converters).  Regarding the purchase of an "audiophile approved" switch, only you can decide how important that is to the sound of your system.  There are people here who are pretty passionate about their switches with the English Electric 8 switch and the Bonn N8 Silent Angel switches both being popular.   OTOH, you might look at this.

Some other stuff you might want to read about includes the Network Acoustics ENO filter and the Intona Technology USB Hi-Speed Isolator.  The level of impact of certain items on the sound of your system may be affected by the design of your streamer/server, the EMI/RFI in your ambient environment, and other factors.  The best way to know for sure is to listen.

Is it still worth investing in an audiophile grade switch and if so why?

No ,There is never a reason to buy a so called "audiophile grade switch" . Learn what a switch does  instead, then you have at least furthered your knowledge. 

My recent experience with an audiophile switch disappointing. I have no filtering of any kind prior to router, Audioquest Vodka feeding router from modem and router to server. LPS on router. All my network optimization is post server. Over time I became convinced audiophile switch prior to server would improve sq. Quite the contrary, i much prefer no switch, seems my router with lps is providing everything I need.

My suspicion is the switch and its internal clock was one clock too many in my setup. My switch also had built in very nice quality lps. YMMV. I'll also second use of fiber conversion, that was a very nice additon!

I recently bought a CA streamer that after week stopped recognizing my NAS.  CA told me to change the switch and that solved it.  Not a sonic issue, but obviously an important one

I went from a Cat 4 and FO to Cat 6 alone. I didn't touch anything else, I re-ran all three hard lines in the house and shop. Cat 4 to Cat 6. 20, 50 and 100 foot runs.

It was like magic.. 15, 20, and 35.00 dollar cables. I had zero errors before and zero errors after but 10X increase in DL and 2 or 3 X in up load, bidirectional. 

100.00 upgrade, 5X increase in overall speed.. YES the sound improved..the shop more so, (the longest run and Cat 4 only before). I like the CAT 6 connections too..

Regards

@questforhifi The EitherRegen did nothing for my system that I could hear. I also use fibre near the last mile before my DAC. I do not have the X1 for direct fibre so I use the Sonore OpticalRendu (fiber-2-usb). This is was a massive improvement in sound over non-fiber (Sonore microRendu). The EtherRegen did not even improve my RJ45 streamer in my KRELL K-300i integrated. I still use it but I should sell it.

You raise a very big issue that most people are reluncant to talk about. Why does the quality of the digital stream matter before the fibre cable. In other words, why do you need a noise optimized music server ($$$), why use fancy Ethernet cable, switches. I asked this same question to Chris Connell (sp?) the big cheese on the AudiophileStyle web site. I think he will write an article about this.

For me ignorance is bliss (with great sound) using my fibre.

I use the following to get my fibre.

Amazon.com: Ubiquiti Networks Networks UniFi Switch 8-Port 150 Watts, White : Electronics