Well, I finally got to hear the BitFrost with the USB input. And I didn't like it! I listened for a day and something just sounded off. Tried a lot of different music and they all seemed not quite right.
So I switched back to my coax cable and input and the magic was back! The only thing that comes to mind is that the USB cable is a DH Labs Mirage, a decent but not expensive cable. My coax cable is a Transparent Audio Premier 75ohm cable that retails for over $1k. It could be the difference between USB and coax in general, but I don't have a way to test that.
The USB version was kind of cold and clinical while the coax is more musically coherent. The sound just flows while the USB sounds like it's being forced out. Not what I expected but it's what I got.