Which version of the ADI-2 FS do you have? The AKM or ESS version? Despite RME stating there is virtually no difference in sound signature between the AKM and ESS versions, my experience was otherwise. I'd originally picked up the AKM (the v2 DAC) version and found it was too bright and fatiguing in the highs. Thus I stuck with the NAD M51 I had been using. Fast forward a few years and I figured I'd try the ADI-2 FS again since they switched to the ESS chipset. I ended up keeping the ESS (v3 DAC). It doesn't seem to be as musical as the M51, but the difference is minimal and the RME has far superior detail retrieval than the NAD without the fatiguing brightness of the AKM version. Note the difference in high frequencies between the AKM and ESS is very slight and the majority of people cannot differentiate between the two versions.
I do not use the headphone amp of the ADI-2 FS. Despite the abundance of options to shape the sound, I only use the bass/treble controls, and even then it's mostly to attenuate bass frequencies late at night.
Admittedly I haven't compared the ADI-2 FS to any DACs other than the NAD M51 that I previously owned. However, that will change soon as a Chord Mojo is on the way.