Mitch
In my situation, improvements have mainly come from improving power.
At the time, my DAC's one review fault was with it's USB input, but with the upgraded model that I own (+), improvements were made to that. I put together the C.A.P.S. Carbon music server, so usb to usb. The two sounded pretty good, but when I hooked up my old CD player to the coaxial input, that bettered my Carbon. So, more detailed and dynamic,etc.. That's when I started looking into a converter and wound up with the Audiophilleo2 with the optional PurePower.
The dirty power from the usb cable is used to charge the PurePower battery and do some processing work, but the power to the clocks and the spdif output stage is completely clean. So, you are probably correct about some experiencing worse sound with straight usb. I'm sure, that this also depends on the gear being used and what kind of job, if any, that the gear does to handle the dirty 5v.
Power is very important to me, and it has made dramatic improvements across the board to the sound of my system in every aspect. No need for specifics, because as you remove the background noise all things improve.
I have also applied this power philosophy to my Carbon and even my Squeezebox. The Carbon is DC and fanless, and powered by a SOtM multi-rail power supply. I also built (kit) a power supply for the Squeezebox. The sound improvements, in my system, are not limited to usb vs. spdif, because I have always used spdif for my Squeezebox. So, going from the wall wart to the power supply for the Squeezebox was big.