Best practices when conducting a DAC comparison

It is not an easy thing to do, because the differences will be tiny at most. The methodology is simple, however. First, look at measured test data to find if there any weird anomalies. Second, if you are then still inclined to do listening tests, make sure that levels are very very precisely matched (to witin 0.2 dB). The best way to do so is by using a good volt meter. Next, switching should be near instant, given humans' short audio memory of at best a few seconds. Finally, comparison should be blind, to avoid expectation bias, and should be repeated multiple times, to exclude random error.
It is the unwilingness to submit to such a rigorous test protocol that generates the audiophile fairy tales of night and day differences between DACs and other electronics. People hear differences and believe that what they hear is real. But that is not necessarily true. Good luck - it is a hard job.
I can also predict that those not trained in scientific research methodology will immediately pounce on me.

Thanks for the reponse. I know it is not easy, and it is precisely for that reason that I deplore the gratuitous praise for some components over others. We are, of course, largely let down by the audio press that mostly only publishes advertorials. They should measure with the expensive audio analyzers that individuals cannot afford to buy, do the double blind listening tests, and help us avoid spending money to upgrade from already perfect components.
I do remember participating in a blind listening tests of amplifiers by Quad's Peter Walker. At the time people were arguing that his new 405 amplifier was not as good as his 303, and that his valve amplifier was actually the best he had ever done. So he had this rig where you could compare without knowing what you were listening to. I thought I could identify them, and he greatly enjoyed demonstrating to me that I had only been guessing randomly. A more elaborate version was published here: http://www.keith-snook.info/wireless-world-magazine/Wireless-World-1978/Valves%20versus%20Transistor...

I respect the genuine effort to do this properly. Methodologically there was one error, however. You may not have changed the volume control, but the output voltage of the two units may have been different. Mind you, I am not saying that this is what made the difference, but methodologically it does matter.
Substantively the take home is, of course, that a great effort was needed to maybe, but only maybe, establish a tiny difference.

For the record, you did a great job. I don't have any measuring gear at home either, and I similarly try to make sense of my imperfect observations. However, as a research scientist I try to stick to proper protocol to shield myself from all kinds of known biases. Sometimes I hear differences, but sometimes I really don't, even on my pretty revealing Quad electrostats.

A recent blind test with a large group of listeners showed no statistically significant differences with 24/96 pcm. I fact, if anything, there was a slight preference for the 24/96 pcm. The test methodology was not perfect, but I have seen nothing better thus far.