Best practices when conducting a DAC comparison
@david_ten I initially thought, because of the way I interpreted your 11-07-2017 8:24am post, that you were wanting to install or use the Gen 5 board with or within your Denafrips Terminator. I also assumed (because you wrote that you have the "Schiit Yggdrasil (my current primary DAC)" that *is already* Gen 5, not knowing your current Schiit Yggy has the Gen 3. This is/was my misunderstanding. Thanks for the clarification. |
The Denafrips Terminator was delivered early yesterday morning and went in system yesterday afternoon. I don’t understand. I thought the Schiit Gen 5 is an upgrade for any Bifrost, Gungnir, or Yggdrasil with the Gen 1, 2, or 3 USB inputs. Your Denafrips Terminator (I assume this one http://www.denafrips.com/terminator.html at face value looks like a monster DAC and it should be at more than twice the cost of the Yggy. The website indicates it uses an Amanero USB Interface, which does appear to be of similar physical size to the Gen 5, but can you just swap the USB cards out? |
.... I do not want to rush this. Thank you for the detailed response and guidance. Since you recently went through this, the list of all the steps you took is terrific and very helpful..... @david_ten You’re welcome, David. I agree with you (and @toddverrone). I didn’t want to "rush" anything either. Thing is, at least in my case (and this is usually the case with any new component), the manufacturer may provide an "audition window of time" whereby you can return the piece if not satisfied. In the case of the Yggy this is 15 days. Would I have preferred 30 or more, sure! Point is I only "rushed" in order to maximize the return opportunity. Once you know for certain you are keeping your unit, at that point absolutely, take all the time you need/want to compare it to other pieces. Presumably you would be living with your gear for many years at least, so there is certainly plenty of time to "check" (and even re-check) it during that time. Best of luck David. You are rather thorough, knowledgeable, etc. so I have good reason to believe you will make the right decision in any case. |
@gdhal I have read and kept up with your thread. I also posted a few times. Luckily, I don't have to worry about the return window since I purchased outright because I believe in what @toddverrone mentions in his post. I do not want to rush this. Thank you for the detailed response and guidance. Since you recently went through this, the list of all the steps you took is terrific and very helpful. I'm not sure about the cold start one for me, but I understand why you did it when you explained it on your thread. |
@david_ten Something else I did that I forgot to mention in my earlier post herein was keep a written log of the results. Seriously. I wanted/needed to "keep score" because often will be the case - especially if when trying blind testing/comparison - that in one instance you favor device A and in another you favor device B. Do that a hundred times and it could be worthwhile to reconcile the results from a written log, in addition to whatever other methods you use. |
First compare the same interfaces, whether USB, Ethernet, AES/EBU or S/PDIF coax. You should obviously level-match using a SL meter. Tracks should all be .wav and high quality. Some should be non-music, like falling rain, tinkling glass, blowing wind, percussion, drums. Try Doors Riders on the Storm. Andreas Vollenwider. One track should have deep bass, one track should have serious dynamics. Limit it to 3 or 4 tracks. Try going direct to your amps and also through your preamp or linestage with both. You may find one blows away the other if you go direct. Steve N. Empirical Audio |
For the record, you did a great job. I don't have any measuring gear at home either, and I similarly try to make sense of my imperfect observations. However, as a research scientist I try to stick to proper protocol to shield myself from all kinds of known biases. Sometimes I hear differences, but sometimes I really don't, even on my pretty revealing Quad electrostats. |
I respect the genuine effort to do this properly. Methodologically there was one error, however. You may not have changed the volume control, but the output voltage of the two units may have been different. Mind you, I am not saying that this is what made the difference, but methodologically it does matter. Not according to the output voltage specs of the *three* components used (Yggy, ERC-3, UDP-205). But I respect your (good) point. Look, I admit, I do not run or have access to a "scientifically exactly perfect audio laboratory". For purposes of what I was trying to achieve, accomplish, etc. I’ll stand pat on my decision that in my case the Yggy was and is the correct purchase, and that my testing is *more than* satisfactory. EDIT: I agree differences in digital can be tiny and hard to discern. But it’s worth the effort (IMO) once you realize, understand and can prove with reasonable certainty that the component change was/is worthwhile. I wouldn’t go back to delta-sigma from multi-bit R2R. No way. *BIG* difference. (with the Yggy that is. My assumption is that multi-bit implementation will best delta-sigma in any case). More "authentic sounding", period. |
I respect the genuine effort to do this properly. Methodologically there was one error, however. You may not have changed the volume control, but the output voltage of the two units may have been different. Mind you, I am not saying that this is what made the difference, but methodologically it does matter. Substantively the take home is, of course, that a great effort was needed to maybe, but only maybe, establish a tiny difference. |
@david_ten Well, you may or may not know that I just went through a grueling and somewhat meticulous exercise myself when I received my Yggy. https://forum.audiogon.com/discussions/new-yggdrasil-first-and-second-impressions Grueling because my angst was sort of self imposed in that I had a short return window to work within. Anyway, long story short, I truly believe I conducted proper comparative testing. - I used the same (type and length) RCA cables connected to two different source inputs. - I ensured the volume control on my amp never changed. - I enlisted the help of another listener. - I sampled many types, varieties, etc. of music. - I sampled hi and lo rez formats, but only PCM. - I tried (and helper) both blind and non-blind testing - I used a playback source that allowed me to "loop" music (5-20 seconds would suffice) - Settled on particular music that highlighted the differences. Namely, Elegant Gypsy by American jazz fusion guitarist Al Di Meola. If you can get your hands/ears on that recording, do so. Virtually every second of music on it can be used effectively to A/B among DACS. - I consulted with other "audio-experts" here on the forum and in a private club I’m associated with, in order to better understand whatever specific nuance I should attempt to listen for. - I conducted testing at different times of the day, with and without alcohol. - I conducted testing with equipment cold (initial startup) - I conducted testing with equipment warm (2 hrs plus on) - I conducted testing at the PLP in my listening environment AND other locations throughout the room. BTW, many users report how incredibly "live" the Yggy sounds even *in another room* and I cannot agree more. - I did other things too, just can’t remember right now :) |
The thing is, getting to know an audio system takes time. Hearing its strengths and weaknesses is a process that requires time and a multitude of different recordings. Walking up to an unknown system and trying to guess what is what is bound to fail. It's funny, when I was into modding my car, these same conversations took place amongst motorheads, with some going by feel and impressions and others demanding dyno numbers.. |
Thanks for the reponse. I know it is not easy, and it is precisely for that reason that I deplore the gratuitous praise for some components over others. We are, of course, largely let down by the audio press that mostly only publishes advertorials. They should measure with the expensive audio analyzers that individuals cannot afford to buy, do the double blind listening tests, and help us avoid spending money to upgrade from already perfect components. I do remember participating in a blind listening tests of amplifiers by Quad's Peter Walker. At the time people were arguing that his new 405 amplifier was not as good as his 303, and that his valve amplifier was actually the best he had ever done. So he had this rig where you could compare without knowing what you were listening to. I thought I could identify them, and he greatly enjoyed demonstrating to me that I had only been guessing randomly. A more elaborate version was published here: http://www.keith-snook.info/wireless-world-magazine/Wireless-World-1978/Valves%20versus%20Transistor... |
@shadorne Thank you. I have a preamp that will allow for that and I will be using Roon / Tidal. What I don’t have is the extra isolation platform and footers so that variable is equal. I’ll do my best to keep levels matched. I know it’s tricky. I'll also have to get an extra set of matched interconnects. |
It is not an easy thing to do, because the differences will be tiny at most. The methodology is simple, however. First, look at measured test data to find if there any weird anomalies. Second, if you are then still inclined to do listening tests, make sure that levels are very very precisely matched (to witin 0.2 dB). The best way to do so is by using a good volt meter. Next, switching should be near instant, given humans' short audio memory of at best a few seconds. Finally, comparison should be blind, to avoid expectation bias, and should be repeated multiple times, to exclude random error. It is the unwilingness to submit to such a rigorous test protocol that generates the audiophile fairy tales of night and day differences between DACs and other electronics. People hear differences and believe that what they hear is real. But that is not necessarily true. Good luck - it is a hard job. I can also predict that those not trained in scientific research methodology will immediately pounce on me. |
