Yes, I know exactly what this is.
Unfortunately, even today, DAC makers still don't universally use galvanically isolated USB inputs. << insert big sigh here>> and this is 100% what happens. Especially when your PC and audio are on different AC circuits.
The solution is really simple and cheap though. You need a USB isolator. Make sure it supports the bandwidth you need. Here is a good one.
This is a bad situation because there are a number of situations where a ground loop occurs and it's not this obvious. Usually listeners "only" suffer from elevated jitter, but it is detrimental to the sound quality.