Best practices when conducting a DAC comparison

@david_ten

Well, you may or may not know that I just went through a grueling and somewhat meticulous exercise myself when I received my Yggy.
https://forum.audiogon.com/discussions/new-yggdrasil-first-and-second-impressions

Grueling because my angst was sort of self imposed in that I had a short return window to work within.

Anyway, long story short, I truly believe I conducted proper comparative testing.

- I used the same (type and length) RCA cables connected to two different source inputs.

- I ensured the volume control on my amp never changed.

- I enlisted the help of another listener.

- I sampled many types, varieties, etc. of music.

- I sampled hi and lo rez formats, but only PCM.

- I tried (and helper) both blind and non-blind testing

- I used a playback source that allowed me to "loop" music (5-20 seconds would suffice)

- Settled on particular music that highlighted the differences. Namely, Elegant Gypsy by American jazz fusion guitarist Al Di Meola. If you can get your hands/ears on that recording, do so. Virtually every second of music on it can be used effectively to A/B among DACS.

- I consulted with other "audio-experts" here on the forum and in a private club I’m associated with, in order to better understand whatever specific nuance I should attempt to listen for.

- I conducted testing at different times of the day, with and without alcohol.

- I conducted testing with equipment cold (initial startup)

- I conducted testing with equipment warm (2 hrs plus on)

- I conducted testing at the PLP in my listening environment AND other locations throughout the room. BTW, many users report how incredibly "live" the Yggy sounds even *in another room* and I cannot agree more. 

- I did other things too, just can’t remember right now :)

I respect the genuine effort to do this properly. Methodologically there was one error, however. You may not have changed the volume control, but the output voltage of the two units may have been different. Mind you, I am not saying that this is what made the difference, but methodologically it does matter.
Substantively the take home is, of course, that a great effort was needed to maybe, but only maybe, establish a tiny difference.

Not according to the output voltage specs of the *three* components used (Yggy, ERC-3, UDP-205). But I respect your (good) point.

Look, I admit, I do not run or have access to a "scientifically exactly perfect audio laboratory".

For purposes of what I was trying to achieve, accomplish, etc. I’ll stand pat on my decision that in my case the Yggy was and is the correct purchase, and that my testing is *more than* satisfactory.

EDIT:

I agree differences in digital can be tiny and hard to discern. But it’s worth the effort (IMO) once you realize, understand and can prove with reasonable certainty that the component change was/is worthwhile. I wouldn’t go back to delta-sigma from multi-bit R2R. No way. *BIG* difference. (with the Yggy that is. My assumption is that multi-bit implementation will best delta-sigma in any case). More "authentic sounding", period.

@willemj 

Thanks!

@david_ten 

Something else I did that I forgot to mention in my earlier post herein was keep a written log of the results. Seriously. I wanted/needed to "keep score" because often will be the case - especially if when trying blind testing/comparison - that in one instance you favor device A and in another you favor device B. Do that a hundred times and it could be worthwhile to reconcile the results from a written log, in addition to whatever other methods you use.

.... I do not want to rush this. Thank you for the detailed response and guidance. Since you recently went through this, the list of all the steps you took is terrific and very helpful.....

@david_ten

You’re welcome, David.

I agree with you (and @toddverrone). I didn’t want to "rush" anything either. Thing is, at least in my case (and this is usually the case with any new component), the manufacturer may provide an "audition window of time" whereby you can return the piece if not satisfied. In the case of the Yggy this is 15 days. Would I have preferred 30 or more, sure! Point is I only "rushed" in order to maximize the return opportunity. Once you know for certain you are keeping your unit, at that point absolutely, take all the time you need/want to compare it to other pieces. Presumably you would be living with your gear for many years at least, so there is certainly plenty of time to "check" (and even re-check) it during that time.

Best of luck David. You are rather thorough, knowledgeable, etc. so I have good reason to believe you will make the right decision in any case.

The Denafrips Terminator was delivered early yesterday morning and went in system yesterday afternoon.

I’ve ordered the Gen 5 USB board from Schiit and will be doing the install myself.

I don’t understand. I thought the Schiit Gen 5 is an upgrade for any Bifrost, Gungnir, or Yggdrasil with the Gen 1, 2, or 3 USB inputs.

Your Denafrips Terminator (I assume this one http://www.denafrips.com/terminator.html at face value looks like a monster DAC and it should be at more than twice the cost of the Yggy. The website indicates it uses an Amanero USB Interface, which does appear to be of similar physical size to the Gen 5, but can you just swap the USB cards out?

@david_ten 

So may I assume that it is your intention to compare **on a per input basis** the Schiit Yggdrasil to the Denafrips Terminator ?

What inputs will you be comparing?

Will you be posting your observation results herein?

@david_ten 

I initially thought, because of the way I interpreted your 11-07-2017 8:24am post, that you were wanting to install or use the Gen 5 board with or within your Denafrips Terminator. I also assumed (because you wrote that you have the "Schiit Yggdrasil (my current primary DAC)" that *is already* Gen 5, not knowing your current Schiit Yggy has the Gen 3. This is/was my misunderstanding. Thanks for the clarification. 

........ is using the same cabling across the different DACs the best way forward?

OR

........ is it better to compare optimized systems, with the components standing on their own within them?

Both ways is technically best from a comparison and arrival at conclusion(s) perspective.

Why would anybody buy a dac without MQA support? Just a few years ago, you probably had the same naysayers about DSD as you do about MQA. A good DSD cut is far superior than the same 16/44 cut. If you purchase an expensive dac today without MQA support nor the opportunity for a future upgrade, your dac will be a paperweight in just a short period of time. So not only will you be wasting your $, you will be missing out on hearing some great music!

As far as I know, MQA uses a *lossy* compression codec. I have absolutely no desire for a lossy format, in particular because the live music I collect and listen to is *lossless*. So this is why I chose a DAC (Schiit Yggdrasil) without MQA support.

@david_ten 

As I mentioned in my 11-06-2017 1:17pm post, "I ensured the volume control on my amp never changed". So while I do not disagree with George, it could be a moot point and you may want to forgo the hassle of precise level matching. Why, because the difference among the DACs may (and likely should or will) be audible from a tonality, dimensionality, or other characteristic that can be readily detected irrespective of level matching. Personally I'd "worry" about precise level matching only when and if you're at the stage when all else is equal. 

All dacs have different analogue outputs levels even if it’s by a 1/10th of a volt, how can anyone compare dacs sound, if the levels are not matched, even a minute increase in level will favour that dac in an A/B comparision.

Hi George. You are right. I have no disagreement with regard to the different output voltages. In my case, I compared the maximum output specification of the Yggdrasil (4.0V RMS (balanced), 2.0V RMS (single-ended)) and that of the Oppo UDP205 (RCA) 2.1±0.2Vrms. (XLR) 4.2±0.4Vrms) and concluded that for my purposes the output voltages were close enough to not be a concern.

I knew my comparison would be less than scientifically exactly perfect, however, I did want/desire as accurate results as possible *and practical*. Practical being the operative word, and precise level matching may not be practical or necessary.

I disagree that a minute increase in level well *necessarily* favor a DAC in a A/B comparison. It may, or it may not. What I’m merely suggesting to the OP is that there are other factors besides "voltage output levels" that matter even more when doing the A/B. For instance, the Oppo treble is more "tinny" than that of the Yggdrasil. That is true regardless of the output voltage, and one doesn’t need to jump through hoops and level match to find out.

Obviously one does not want to listen to DAC "A" at 60db and DAC "B" at 100db. Sure, if the *result* of the output levels produce egregiously different volume then something is amiss and that could/would invalidate a test. But a 10th of a volt, all else being equal, I think not.

EDIT:

Should the OP have the proper equipment to precisely level match the output voltage, certainly there is nothing wrong in that and in fact that would be preferred. So in no way am I suggesting not to level match. Instead, I'm calling attention to the merit of doing so relative to other items and the likely hassle involved in attempting to *precisely* match.

....If you can't tell which one is better within the first 30 seconds there really isn't a significant difference....

Nor should there be if you already have a highly-decent sounding system.