My take on downside.
If the passive VC on the amp is up full and you use the DAC's volume control, theoretically there is no down side, just another pot wiper light contact in the signal path, that the signal has to pass through, even though it's up full.
But to me, better not to have that extra light contact in the signal path. If your anal like me take the amps passive VC out of the equation altogether.
Cheers George