What streamer are you using and do you have a wireless component in your chain? I use SGC and Roon and have not had any problems. I also us a Bricasti M3 DAC with network card. It sounds like you are having network drops and have not buffering built into your streamer. Just a thought.
