Best practices when conducting a DAC comparison


Reaching out for general advice on how Agoners have compared DACs in their own systems.....

....and how you have determined the 'better' or the 'preferred' component, based on your comparison.

This will be my first in-depth comparison.

Feel free to mention whatever you believe will help and stuff I may need to look out for / be aware of.

Thank You.
david_ten
Hey guys, my write up is posted here:

http://terminatorandyggdrasil.blogspot.com/

And here:

https://forum.audiogon.com/discussions/denafrips-terminator-and-schiit-yggdrasil-evaluation-and-comp...

This thread guided my approach, though it is not as rigorous as some suggested or how you might conduct your own evaluations.

The suggestions, however, certainly became part of the process and for that I am thankful and grateful to all who posted. I learned a lot through your feedback. Thank you.
Thanks to everyone for your guidance and to the recent helpful posts from @seventies @audioengr  and @dlcockrum 

I've learned much from each poster.

I finished the comparison between the Schiit Yggdrasil and the Denafrips Terminator this morning. The comparison got held up for a number of reasons but all-in-all the delays, in retrospect, were actually helpful and made for a stronger and more robust evaluation.

First, I was able to get to a place where all the components were well broken in.

Second, I was able to spend significant time with my system and each of the 'newer' components and cabling and I now have a very good feel and sense for them. 

Third, though the comparison is Subjective, I was able to control many of the variables to level the playing field --- as much as I reasonably could; however, the review process will not meet bench level rigor.

Fourth, the added time has allowed me to think through and reflect on the process and what I'm looking to achieve much, much more thoroughly. 

Fifth, I was able to get another audiophile's perspective on where my system is 'performance wise' as well as getting feedback on shortcomings. I also wanted a second opinion on What I 'hear' and How I 'hear' and to discuss my audio preferences 'live' and based on my system. I've also had a number of phone discussions with audiophiles active on Audiogon to answer questions and bounce thoughts and approaches off of.

I will be writing up my findings and takeaways starting tomorrow and should be able to wrap it up over the next few days. Once it's finalized, I will post a link for those who may be interested and curious.

A most excellent list, Steve.

The question that bugs me is that of optimizing each component for best results. Should David use a reclocker for both pieces in the comparison even if one benefits and the other doesn’t (assumedly because the latter has better addressed the issue of noise and jitter reduction internally)? Does that fairly represent the potential user experience of each? What about different cabling/power cords that optimize each independently?

The purists’ "apples to apples" mentality would seem to require that the exact same cabling (even using the included power cords and perhaps generic digital cabling) with no external devices in play. Sounds fair, but what if one component would benefit tremendously from a reclocker or specific cabling to the point that it significantly outperforms the other and/or changes David’s preference? Even worse if one is much less costly than the other and investing a relatively small amount (i.e. that the total investment would be significantly less) would make it equal or even to outperform the much more expensive competitor.

The Terminator lists for ~$4400 and the Yggsrasil for $2400. Would spending less than $2k to optimize the Yggdrasil with a reclocker, cabling, etc. yield a better sonic result, thus making it a better purchase decision for the same or lower investment?

Seems that in the "bang for the buck" approach that the answer would be to address this with a "same or less total cash outlay" comparison as this is the real world dilemma for many of us. But then there are those that will choose to use each without any "supplements" (reclockers or upscale cabling, etc) that would find value only from the "apples to apples" comparison. And also those seeking an "all out assault" that may have interest in an ultimate sound quality comparison with each having every enhancement specifically optimized for that particular component regardless of cost. Who, short of the most diehard professional reviewer, has a sufficient number of these ancillary components available to even begin to satisfy all possible curiosity?

David has a real challenge on his hands trying to potentially satisfy each category of reader. I wish him the best of luck.

Dave

Here are a few considerations:

1) if you plan to play FLAC files only, then make sure to do FLAC in several sample-rates

2) If you plan to use a preamp, make sure you have that preamp on-hand, not something in your future...

3) Use a test tone track to match levels using a sound level meter

4) Make sure your source has low-jitter if you plan to use S/PDIF or AES/EBU.  If one DAC has reclocking and another doesn't, the difference will be large due to that alone.  Get a reclocker if you need to lower jitter.

5) Make sure you have a really good S/PDIF cable if that is the method you plan to use.

6) If you are using USB, make sure you have a good USB cable.  Your PC, Mac or server will need to be optimized for USB.

7) If you are using USB, make sure your playback engine is a good one, like Amarra on Mac or an Antipodes or Aurender server.

Steve N.

Empirical Audio

David,
I have now owned and auditioned a Bryston BDA-3, Luxman DA-06, and the Oppo 105d with its internal DAC.  Each, as you know, uses a different DAC chip: respectively AKM, Sabre Burr- Brown.
I now use exclusively the Bryston, sourced via USB by a Bryston BDP-3 streamer, or the Oppo disc player via HDMI.
The characteristics of these DAC's has been described by others, and there may be general agreement that the Luxman is 'very smooth', the Oppo (sabre) detailed but harsh, the Bryston in-between.
My personal impressions:
1. The Luxman employed for chamber music...string quartets, piano trios and quartets...
is 'muddy'....the lack of detail increasingly becomes annoying.  Its employ for solo piano invites similar comment.  Try as I might to persuade myself that the 'more expensive sound'...ie, the Luxman's...should be better than the Oppo's, I had to throw in the towel and
avoid the Luxman for strings and piano.
2. The Luxman employed for voice is a delight.  I've read of various hi-fi components
that a 'smoother' high frequency response decreases listener fatigue.
Such comment applies to the Luxman used, for instance, for listening to an
opera DVD.
3. That virtue is such that even 'downrated' DVD audio sourced to the Luxman via the Oppo's coaxial audio output was superior to the Oppo's internal audio decoding of opera DVD's,
including blu-ray DVD's.
4. The Bryston BDA-3 is a good compromise...' 
I struggled to parse the foregoing impressions in respect of digital source format...standard CD, SACD, hi res PCM, double speed SACD.   The sound quality seems to improve as one progresses from the beginning to the end of that list, but the DAC comparisons...at least my ear...are the same. 
First, thanks for the additional thoughts shared via the previous few posts.

Second, I will try to consolidate everyone’s suggestions on approach and put together a summary.

Third, the Denafrips Terminator is broken in and is performing very well. I’ll be swapping in the Schiit Yggdrasil to get some listening time with it (and the new Gen 5 USB board).

Fourth, a major UPDATE:

I thought I’d conduct the comparison over the Thanksgiving Holiday...but other priorities got in the way. I wish I had, because I now have a new element in the system. I’ve been waiting for a speaker delivery, which was planned and scheduled for before the Denafrips DAC was delivered. The speaker was delivered yesterday.

My original plan was to have the new speaker fully broken in as well as to have a significant amount of personal listening time with it. Since I plan to keep the new speaker (and the DAC that performs / has the best synergy with it) I think the best approach is conduct the DAC comparison once the speaker is broken in and I’m familiar with the new speaker.

Thoughts in agreement or contrary to this? If otherwise, I’m interested in how you would proceed? Thanks.
GeorgeHIFI is so right about volume ,many would choose any higher level even the tiniest amount over the other component ,

.Someone mentioned MQA and why would anyone not choose a dac that didn’t feature this ?  Recording engineer and owner of AIX Records Mark Waldrep including others in the industry have plenty to say about MQA overthe past couple years , RealHD-.com is a great place to start ,

David 10 , have fun with your evaluation,.
Erik,
I do not listen to test tones or specs.  I only use red book as digital source.  There are few innovations in the digital conversion world that truly separate one product from another.  Most use the same chip sets with various number of chips deployed and specific parts implemented in the circuit.  

I have four four digital sources available in my main system.  Three of them resemble one another in that they posses a more noticeable tell tail digital signature.  The least desirable to my ears is the Oppo 105D circa  2015.  Next up an early nineties Sony followed by a mid nineties tube dac from VAC.  Then there is one that within that first 30 seconds of playback sounds that much more realistic to me that further comparison is pointless.

 I also have an analog system while not state of the art is far above an entry level setup.  If both the the analog and digital sound more similar than different this is a good thing in my experience.

However, the main goal for both is to sound like real music and convey the emotions and content to make me sit back and get lost in what's playing which is happening now as I type.  Type a bit, stop and listen, then a few more words on the page.

An example of the 30 seconds test is track 2 of Holly Coles  Don't Smoke in Bed.  It's not a complex piece but revealing of digital nasties.

OBTW, I can afford more but what I have pleases me and that is all that I require.  You're invited anytime for a listen.
I agree with bigkdz. If you can’t tell which one is better within the first 30 seconds there really isn’t a significant difference.

What????? Jesus. What exact test tones are you using? It takes several tracks just to fully exercise a DAC. Bass, air, imaging, impact which is not to mention some issues like fatigue, take much longer than 30 seconds to set in. Then how will you compare the Redbook, High Rez and DSD performance? 30 seconds is barely long enough to eliminate a DAC you will never like, assuming it is warmed up but is not a fair way to choose among good performers, unless you believe they all sound the same, and specs are all that matter. If this is you, buy the cheapest you can find.

Best,

E
....If you can't tell which one is better within the first 30 seconds there really isn't a significant difference....

Nor should there be if you already have a highly-decent sounding system.
I agree with bigkdz.  If you can't tell which one is better within the first 30 seconds there really isn't a significant difference.

Level matching may make a fair comparison if you are switching back and forth, but if at that point you are straining to hear a difference you're wasting time.

Another approach would be to play a familiar cut for a short time at different volume levels, then do the same for the competition.
I have MQA in my beloved DAC, and it's disabled. I find I prefer high resolution with a different filter than it demands. I don't hear a benefit of MQA. Sorry.

As for auditioning, make sure both components are 24 hours warm and broken in. YMMV, but that's a more fair test. Ice cold, some DAC's do better than others.

Next, compare red book as much as high resolution. Don't assume that the DAC which has the most difference is the better one. The last generation of DAC's play red book exceedingly well. The previous generations only played high rez well.

Best,

E
My way to audition a new DAC is with music (a) that I am very familiar with (b) with a mixture of recordings that I love and hate on the existing DAC. Recordings one loves should sound equally as good or better with a new DAC. Just as important is when recordings that grated on you with the old DAC no longer offend and become interesting to you ; these are signs you have fixed a broken area in the old DAC. The pop genre tends to be sprinkled with recordings which excite bad sound on lesser DAC's.

Most DAC improvement is in increased detail   and reduced digital artifacts. DAC's can have tone color signatures (lean/lush/neutral) which will also affect your preference. DAC's also cannot produce what doesn't reach them correctly, anemic bass and blurry detail for accompanying instruments can be signs of jitter issues with the source solution feeding the DAC.

@georgehifi  and @gdhal  Thanks for your thoughts on the level matching point you are making and are mainly in agreement on. 

@stfoth  I like the 'enjoyment' part of your assessment. Your primary suggestion on how to go about the evaluation is valid and has good company in that it has also been offered up (similarly) as an approach by a couple of others. 
Post removed 
All dacs have different analogue outputs levels even if it’s by a 1/10th of a volt, how can anyone compare dacs sound, if the levels are not matched, even a minute increase in level will favour that dac in an A/B comparision.

Hi George. You are right. I have no disagreement with regard to the different output voltages. In my case, I compared the maximum output specification of the Yggdrasil (4.0V RMS (balanced), 2.0V RMS (single-ended)) and that of the Oppo UDP205 (RCA) 2.1±0.2Vrms. (XLR) 4.2±0.4Vrms) and concluded that for my purposes the output voltages were close enough to not be a concern.

I knew my comparison would be less than scientifically exactly perfect, however, I did want/desire as accurate results as possible *and practical*. Practical being the operative word, and precise level matching may not be practical or necessary.

I disagree that a minute increase in level well *necessarily* favor a DAC in a A/B comparison. It may, or it may not. What I’m merely suggesting to the OP is that there are other factors besides "voltage output levels" that matter even more when doing the A/B. For instance, the Oppo treble is more "tinny" than that of the Yggdrasil. That is true regardless of the output voltage, and one doesn’t need to jump through hoops and level match to find out.

Obviously one does not want to listen to DAC "A" at 60db and DAC "B" at 100db. Sure, if the *result* of the output levels produce egregiously different volume then something is amiss and that could/would invalidate a test. But a 10th of a volt, all else being equal, I think not.

EDIT:

Should the OP have the proper equipment to precisely level match the output voltage, certainly there is nothing wrong in that and in fact that would be preferred. So in no way am I suggesting not to level match. Instead, I'm calling attention to the merit of doing so relative to other items and the likely hassle involved in attempting to *precisely* match.
"I ensured the volume control on my amp never changed"
  it could be a moot point and you may want to forgo the hassle of precise level matching.
All dacs have different analogue outputs levels even if it's by a 1/10th of a volt, how can anyone compare dacs sound, if the levels are not matched, even a minute increase in level will favour that dac in an A/B comparision.

Cheers George  
@david_ten 

As I mentioned in my 11-06-2017 1:17pm post, "I ensured the volume control on my amp never changed". So while I do not disagree with George, it could be a moot point and you may want to forgo the hassle of precise level matching. Why, because the difference among the DACs may (and likely should or will) be audible from a tonality, dimensionality, or other characteristic that can be readily detected irrespective of level matching. Personally I'd "worry" about precise level matching only when and if you're at the stage when all else is equal. 
@georgehifi Is there a ’cheater’ way to get close enough? : )

No.
If you do it by ear, you could be as much as 3db out on level matching.
Cheers George
@rbstehno   I have a long history of audio bricks as doorstops. : )

It would have been nice to have. In the end it wasn't a deal breaker for me.
@georgehifi   Is there a 'cheater' way to get close enough? : )

I can do so, if you and others believe it is critically important.
@bigkidz Thanks for your response and the details shared. I generally agree that if one doesn’t pick out differences immediately then it can be a toss up choice (having controlled and equalized the variables as best as possible).

I find this also applies to out of the box performance. If one can pick up differences at that initial point, then they become clearer as a component gets further run-in time on it. This has recently been the case with a couple of components.

I have also had the opposite experiences.

For example, I had to spend much more time A / B -ing to drill out and appreciate differences with a recent USB cable comparison. It was tough and time consuming but well worth it and very educational for me.

In some cases (where there were readily apparent differences upfront) elucidating whether they met my goals and preferences was a drawn out process. This happened with a recent speaker comparison.

I believe our level of listening experience and development of ’hearing’ expertise comes into play as well. Some may be much further along the spectrum, which would make elucidating the differences easier and quicker.
Why would anybody buy a dac without MQA support? Just a few years ago, you probably had the same naysayers about DSD as you do about MQA. A good DSD cut is far superior than the same 16/44 cut. If you purchase an expensive dac today without MQA support nor the opportunity for a future upgrade, your dac will be a paperweight in just a short period of time. So not only will you be wasting your $, you will be missing out on hearing some great music!

As far as I know, MQA uses a *lossy* compression codec. I have absolutely no desire for a lossy format, in particular because the live music I collect and listen to is *lossless*. So this is why I chose a DAC (Schiit Yggdrasil) without MQA support.
Why would anybody buy a dac without MQA support? Just a few years ago, you probably had the same naysayers about DSD as you do about MQA. A good DSD cut is far superior than the same 16/44 cut. If you purchase an expensive dac today without MQA support nor the opportunity for a future upgrade, your dac will be a paperweight in just a short period of time. So not only will you be wasting your $, you will be missing out on hearing some great music!
david_ten
Best practices when conducting a DAC comparison

Each dac will have a different gain structure (volume), don’t trust your ears to level match.
It must be done with a 1khz sine wave and measure the ac mV (millivolt) at the speaker terminal.

Cheers George
david_ten I build Direct Heated Triode DACs - most people never heard a DHT product.  First you should have the material that you are extremely familiar with so you know what you are looking for so when you swap out the product you are using, you should know if the newer product makes enough of a significant change based on your listening experience.  I know that many people will say that it has to be in the system for some time for you to get to really hear what the new product offers, well they are probably right, but IMO only, I usually get to hear what it does within 30 seconds probably because I know the material extremely well.  for example, I have a switch in my preamp where I can change resistors by turning a rotary switch.  There are 5 resistors that you can select from.  My partner could hear the changes without knowing which resistor was which simply because we have listened extensively to those resistors in the past, so he was able to know which resistor was what immediately.

I wrote the 30 second test because I manufacturer Direct Heated Tube products so I am biased.  Typically I bring my DAC over a persons home to audition or ship one to them.  So the DAC is usually already a broken-in model.  Most people want to hear it in their system before purchasing so we offer a 14 day in-home trail.  Once it is plugged in, if you don't hear an immediate change to the sound of your system or want the DAC offers within 30 seconds, send it back.  After about 20 to 30 minutes the tubes begin to warm up and the sound will change again for the better.

Happy Listening.  To the rest of you reading this, keep the comments to a minimum!

 


 
@bigkidz   Is your 30 second test after break-in and when doing a quick A / B? Thanks.

@rbstehno  I agree. We are the ones living with our gear, usually for significant periods of time. These are very personal choices based on our experience and preferences and system synergy.

@dpetri  To date, to my knowledge there is only one major review of the Denafrips Terminator DAC that I purchased. I had made the decision to do so prior to the 6Moons review, but it came out approximately the same time I placed the order.

Certainly preferable would be a number of sources, owners and reviewers, to help with the choice. Good as a guide, especially if one finds their preferences line up with a reviewer's. But as @rbstehno  posted, ultimately it's our ears, and system, and room, etc. etc.

@dumacker  Thanks. I guess that is the crux of the issue: "Is there any better way to hear the difference?"

As a general update, I have two weeks of run time on the new DAC. I still need to install the Gen 5 board in the Yggdrasil and get it running as well. Everything is still on track for an evaluation between both towards the end of November.

Please add any other thoughts on the topic and approaches on how to best go about a DAC comparison. Thanks.
I agree that the only way to compare audio equipment is a/b any contenders.  I also agree that matching volume if allowed two inputs is critical.  Is there any better way to hear the difference?

There are so many very good brands out there, reviews and comments/opinions  like these provide a guide in narrowing down selections within your budget...

jhv
Personally, I let the reviewers do the heavy lifting.  I read all of the reviews I could find on the DAC I was interested in and paid attention to the associated equipment they were using.  I was interested in the Auralic Vega nad purchased it based on the reviews of several seasoned audio/digital audio reviewers and decided to purchase the DAC based on what their observations were.  I believe I made an excellent choice.  I live in Cleveland and when I purchased the Vega, there were not many dealers that were well versed (or equipped) with digital audio gear.  Just my 2 cents.

Dan
So if a recent blind test indicated that piece ‘A’ is the best, would you go out and buy this without hearing it 1st? I don’t give a s$&# if a group or a reviewer indicates this it that, I would have to listen to it 1st before spending a dime on it.
if my ears hear a difference between pcm versions, or DSD, flac/aiff, or MQA, then I will pursue that format.
if you don't hear a significant difference in the first 30 seconds then keep what you have!  Well that is my opinion anyway.  Happy Listening.
@aberyclark   Thanks for the tip on The Beatles 'Come Together' track. I'll give it a critical listen.

@gdhal   Thanks for your ongoing thoughts and advice.
A recent blind test with a large group of listeners showed no statistically significant differences with 24/96 pcm. I fact, if anything, there was a slight preference for the 24/96 pcm. The test methodology was not perfect, but I have seen nothing better thus far.
Consumer benefits of MQA? Have you actually listened to MQA and compared this to other mediums or are you listening to other posters remarks? Again, I don’t listen to what other people say about new technologies because there are many times they are slanted. Some people are still pissed off from the sacd/dvd-audio debacle and won’t consider any new medium, or people way over the top about every new technology. The best thing to do is hear it for yourself using your own equipment in your own room. My buddy had 5 different dacs on loan to him so he could evaluate them with his own equipment, in his room, and using his music. This is the best way to evaluate a piece of equipment.
@randy-11 I’ll be installing the Gen 5 USB board which does offer galvanic isolation for the Yggdrasil. I believe there is galvanic isolation built into the Denafrips (but I need to double check this). For now, optical isolation is off the table; I am aware of the benefits.
I am not clear on the consumer benefits of MQA.  It does seem DRM capable however...
@rbstehno Thank you for your response.

Some differences: the Denafrips offers DSD; higher resolutions of PCM; Over Sampling; three I2S inputs, and an extra set of Coax, TOSLink, AES each, but no BNC --- over the Yggdrasil.

Unfortunately, probably fortunately for some : ) MQA is not an option for either DAC at this time. It would have been a nice feature to have.
there will only be tiny differences between dacs relative to most speakers, room tmts., etc.

galvanic isolation can be a biggie for a DAC -- besides an opto-isolation technique, a LPS, and/or star-quad DC power cables are good ideas
@steakster  Agreed! They definitely make a difference, including the upgraded DC cables. I've been running HDPlex to power the switch, server, etc. for quite some time now. I'll probably play around with different brands once the major components are finalized.
How can somebody say that there will only be tiny differences between dacs? Maybe this guy hasn’t heard a lot of dacs. There can be big differences between dacs. Also, I don’t agree that you have to use the same cable or input to compare different dacs. If you are only using usb, then sure you have no choice, but if you have an option of using i2s, toslink, or spdif, then I would use the cable that the dac prefers because you want the best sounding dac that you can afford. I would never go back to usb, and I would tend to use i2s if possible.
Also, ignore all the measurement specs that some reviewers post, IMO they are worthless. Who cares if 1 dac has .0001% distortion and the other has .0005%, you think you are going to hear that? That’s like asking the Porsche dealer to give you a readout of the compression of each cylinder before you buy a new Porsche. You have the best instruments to do your evaluation and that your ears.

What I haven’t seen is your requirements for the dac and this is where you can get huge differences. Do you want the dac to support MQA? DSD and which resolution? I have some MQA tracks that surpass vinyl by not a small margin.
I would make sure that your volume levels are set the same and then test each dac with your favorite music using various inputs unless you are stuck with only 1 interface.
Post removed 
@david_ten,
I am rooting for the Terminator because I have been following it since its conception, it has been tempting my wallet for some time now.

........ is using the same cabling across the different DACs the best way forward?

OR

........ is it better to compare optimized systems, with the components standing on their own within them?


Both ways is technically best from a comparison and arrival at conclusion(s) perspective.
The best measure for me to demonstrate the quality of the low end is the Beatles "Come Together" test. The first break where Paul let's his Bass "A" string linger out ("Joker do what he please"). Even if generally listening you cannot tell the differences between dacs, amps, cables....this one little passage will give it all away. You'll know it when you hit a sweet spot because you can feel growl of the bass string. You can switch components and notice that something is missing. This isolated bass snippet is copied and pasted a couple of times in the Beatles LOVE soundtrack as well.
Thanks, Todd. I think you will appreciate what follows.

My main concern is that my current system has been (mostly) optimized with the Schiit Yggdrasil within it. I struggle with this very issue in the comparisons and component evaluations I read, whether professionally done or owner based.

I’m especially concerned about cabling.

I recently completed a USB cable comparison. It was difficult yet very educational. I’m well aware that the cable I found to perform best to my preferences and goals may not match as well or deliver the same outcome with the Denafrips DAC (again to my preferences).

For example, I chose the Cardas Clear Reflection XLR ICs for my headphone system. They are performing very well in the 2ch system, but the choice was made for a different application and goals. Would another brand of cable or moving up the Cardas line-up be a better fit for my 2ch system, with the Yggdrasil in it? For me, the answer is likely.

The ’million’ dollar question (for me):

........ is using the same cabling across the different DACs the best way forward?

OR

........ is it better to compare optimized systems, with the components standing on their own within them?

I believe the second approach has a greater degree of fairness in terms of determining endpoint differences and performance preferences. It, of course, is not nearly as practical or financially feasible.

[Note: At a minimum optimizing around the power and signal cables in and out of the DAC is a somewhat more viable option]

I realize we have a number of Audiogoners who find none to minor differences between cabling. I respect this. Further, keeping the cables identical will make the process straightforward and much, much easier for me.

For those of you that do find cabling an important piece, I’d love to hear your approach to this. Thoughts? Thanks.
Post removed 
You are certainly being thorough.. I'm interested to hear what you find. Keep us posted!
@muzikmann  Yes, especially the Terminator. The manufacturer advises 300-500 hours; and yes I know how some feel about this. : )

As @toddverrone mentions in his post, spending time with the components will also help accomplish this.

I will start with the critical listening for the Terminator towards the end of November. I'll energize and run signal through it 24/7 until that time.

I'll also keep the Yggy and the Comet energized. 
@gdhal 

Got it. No, I don't want to mess with Mr. Zhao's flagship work!. : )

My main output is USB from a Sonore Signature Rendu SE.

I'll primarily be focused on comparing the USB input between all DACs.
I would also ensure all the components are broken in, especially the Terminator DAC.