Why HiFi Gear Measurements Are Misleading (yes ASR talking to you…)

@alexatpos 

I guess I predicted the future?

Wanting to be seen as smart and being intelligent are not the same thing folks.

 

Your tone and how you almost bully people into listening to you is rather rude. Hence why virtually every audio forum on the web has labelled you all kinds of silly names.

Or...many people are triggered by having their own subjective conclusions challenged.

"How dare Amir tell us what we can and can not hear!"

It's taken personally, so often the response is to go ad hominem on Amir, call him names or dogmatic etc.  When often it is the more "subjective-based" audiophile who is closing his ranks around his beliefs.

Amir has actually responded quite civilly to those engaging him likewise.

And his response was even quite measured to those implying he is dishonest or colluding.

I certainly don't mean to paint anyone as a saint, Amir included.  But there is a lot of assumptions made that I find come from a bias on the "subjective" side, where they see what they think are pushiness from ASR, while turning a blind eye to the type of ad hominem and insults from "their own side."

 

Alright..you can have your cake and eat it too! All I’m saying is....live and let live. Your tone and how you almost bully people into listening to you is rather rude. Hence why virtually every audio forum on the web has labelled you all kinds of silly names.

Uses an OP amp and tons of subtractive distortion limiting - like negative feedback in a circuit. Tons of this, much like dynamic range compression in mastering will limit perceived dynamic range in a track.

 

This is bias likely derived from reading, not knowledge or experience. Your purported expertise is recording? Mixing?  Audiophiles often write about how superior audio recording and mixing used to be, even in the 70's into the 80's. Do you think most mixing consoles used op-amps, or discrete transistors?  

Can you clearly communicate what you think op-amps and negative feedback is doing to the signal, and why this does not show up even in complex distortion measurements?

 

@alexatpos 

One of them probably has an answer to this too...

lol

Well, I am very curious to hear any system that was assembled through blind testing of its individual components....

@amir_asr

They can spend months and it wouldn’t make their reviews reliable. If you know what you are doing, including science and engineering of the gear and what the measurements show, you can zoom in and find issues. You don’t sit there listen to random track after random music for weeks. That tells you nothing.

More time spent on a task can net better results. This is not always true. However, when it comes to listening (headphones, speakers, DACs, amps etc.) spending more time evaluating a product before releasing a review can help in a few key ways:

1) Testing for Reliability

2) Features & Functionality

3) Overall sound quality analysis

4) Small important details

I’m sure that most of these people doing reviews have a standard set of reference tracks; or at least a background/strong interest in audio; enough so to make their impressions reliable. They were hired to do a task and might be very good at it. We have no way of knowing how much audio knowledge they have...

Tyll Hertsens - of innerfidelity (now defunct) was probably the best reviewer of headphones on the net. Like I said in one of my discussions, he took time to describe what each headphone sounded like with a particular track. He also did measurements and highlighted key areas in the frequency response or octaves where performance could have been better...while still quoting the measurements he took. And on top of all this, comparing it headphones in the same price-range with the same factor - open/closed back!

Since he went above and beyond, this gives a potential customer huge insights! You can go ahead and test with the same track. You can wait a week, be really busy, visit an audio shop, listen to that headphone (or headphones) and jot down your impressions on the notepad app your phone has installed.

Then you can compare and contrast your review with his - figure out if this headphone is worth the money for you. I’m sure that anyone who has bought audio products based on solid reviews like his will agree...

Now back to reviews...I bought a SABAJ A10h based on your review. I also bought a DROP THX789 based on your review. In both cases, not only did I find that output power was severely lacking; each of them also had their own sound signature. Based on your measurements and overall write-up of both units, a potential buyer would actually believe that each of them were a wire with gain!

Please see my profile for a photo that illustrates this. Looking inside one of these devices tells you it is cheap to build. Uses an OP amp and tons of subtractive distortion limiting - like negative feedback in a circuit. Tons of this, much like dynamic range compression in mastering will limit perceived dynamic range in a track. You’ve got to wonder how they put something together at that price-point and sold it. You can easily look up parts by just looking inside a unit and doing a parts inventory check...they are both not state-of-the-art ! lol

Alright..you can have your cake and eat it too! All I’m saying is....live and let live. Your tone and how you almost bully people into listening to you is rather rude. Hence why virtually every audio forum on the web has labelled you all kinds of silly names.

You need to stop listening to your lay intuition and embrace science of how to do such evaluations correctly. Formal testing shows long term listening to be much less revealing than instantaneous ones. See this published research on that:

I implore to you start paying attention to decades of research on what it takes to properly evaluate audio gear. The lay understanding and intuition stuff needs to go out the window.

@andy2 

That is because a good tube amp will cost a lot more money compared to a SS amp.  To get the same performance you need to spend quite a bit more.  If money is not an issue, most people would go with tube.

Back to the myth that money buys performance.  It can, but you only know it if you measure.

That aside, almost all content you listen to was created and approved by the talent using solid state electronics.  Is your claim that they heard it with haze?  If so, that haze must be part of the experience they want you to have!  Best to leave it just like guitar distortion.  :)

@andy2 

Amir has claimed that he personally listened to "200" pieces of equipment per years.  That is around 366/200 or 1.5 days per equipment.  I mean come on who in the world can take you seriously if you only spend such a short time evaluating an equipment.

In comparison, people at Stereophile spend weeks on an particular equipment before they publish the review.

They can spend months and it wouldn't make their reviews reliable.  If you know what you are doing, including science and engineering of the gear and what the measurements show, you can zoom in and find issues.  You don't sit there listen to random track after random music for weeks.  That tells you nothing.

Every one of my listening tests uses the same, revealing and proper tracks. I focus on what measurements say is wrong with the unit and test level of audibility.  A headphone amp that has too little power with high impedance gets tested that way.  And with content designed to find audible issues.

Same gear is tested by one of your favorite reviewers reads like a music review.  Oh listen to this album and that album.  What?  I want to know what the equipment is doing, not what music you listen to.  

Proper research has been performed to find such tracks.  See this:

You need to stop listening to your lay intuition and embrace science of how to do such evaluations correctly.  Formal testing shows long term listening to be much less revealing than instantaneous ones.  See this published research on that:

I implore to you start paying attention to decades of research on what it takes to properly evaluate audio gear.  The lay understanding and intuition stuff needs to go out the window.

 

@amir_asr

Problem is, when most of you are put to blind tests, you flunk being able to tell the source from the compressed one. And no, resolving system has nothing to do with it. The fact that you say that tells me you don’t know what it takes to hear such differences. As a trained listener in this domain, I can tell differences with just about any headphone on any system.

Yes, this is, I find, one of the most common myths among audiophiles. Whenever someone raises skeptical doubts about claims made about, say the audible character of cables, the response if the skeptic doesn’t agree is the "ears or gear" - either you don’t have the hearing acuity the Golden Eared Audiophile does, or your system just isn’t "resolving enough" like you need a "super resolving system" to hear these differences.

There are various problems with this idea:

1. Subtle sonic differences, if real, can be heard across a large range of transducers. Sure we can get to something like the worst laptop audio or whatever, but it really doesn’t take THAT much to produce speakers on which you can hear very subtle differences. I’ve worked in tons of different studio conditions, different monitors, headphones of varying quality and ALL have allowed me to hear and make the subtle changes I need to for my job. Give me an old pair of radio shack minimus 7 speakers and if I do a subtle EQ tweak to bring out the upper mids, you will hear it!

 

2. The type of sonic differences often ascribed to (for instance) cables is often fairly dramatic - the "highs opening up" the bass becoming more punchy or extended, more forward in tonal balance, more laid back etc. These are all qualities, if that obvious, should be audible on most speakers. It’s why different mastering is obvious on most speakers, from cheap to expensive.

 

3. We have audiophiles reporting these "obvious sonic differences" across a wide range of systems and speakers. It’s not just the well-heeled audiophiles with the Super Resolving Systems. Just go to the typical amazon page for some set of audiophile cables (even not expensive ones) and you’ll see audiophiles with very modest systems reporting "obvious differences" with the cables in question.

From this you have either two implications for those making the Resolving System demand:

A. It’s a red herring to demand that someone must own a Very Resolving System in order to evaluate whether a cable is making a difference.

Or:

B. If these differences aren’t obvious on less resolving systems, plenty of those audiophiles with modestly ’resolving’ systems are imagining they are hearing differences between cables. Which would only emphasize the problem variable of listener bias in the first place.

 

 

@thespeakerdude 

I think the test you gave yourself is too easy 😀  Your point is well taken in regards to trained listeners and MP3. I think a better test is to serve up 10 different tracks which may be MP3 or may be wave, and then test how well listeners do at accurately assessing if the track is compressed or not. I wonder if even the trained listeners will be challenged in that case without a reference.

I didn't give that test to myself.  I was challenged on a major forum by an objectivist to be able to tell MP3 from original with him claiming that no one could.  At the same time, there had been a challenged on that forum to tell 16 bit content from 24 bit.  Content for that was produced by AIX records which is well known for quality of its productions.  So to remove any appearance of bias in selection of material, I grabbed the clips from that test and compressed them to MP3.  And post those results.  The clip was not at all "a codec killer" where such differences are easier to hear.

On the type of test you mention, I am not a fan of them for the reason you mention.  It is harder to identify the original vs compressed that way because you have to now know what the algorithm does to create or hide sounds.  In other words, is an artifact part of the original content or was it removed.  

Our goal with listening tests should always be to try and find differences, not make it hard for people to find what is there.  Because once we know an artifact exists, we can fix it.  Making the test harder to pass goes counter to that.

That said, I and many others were challenged to such a test on the same major site above.  We were given a handful of clips and asked to find which is which.  Results were privately shared with the test conductor.  When I shared my outcome, he told me I did not do all that well!  I was surprised as I was sure two of the clips were identical and thought that was put in there as a control.

Fast forward to when the results are published and wouldn't you know it, I was "wrong."  We had a regular member with huge reputation for mixing soundtracks for major films and he got it "right."  Puzzled, I performed a binary comparison and showed that the two files were identical!  Test conductor was shocked.  He went and checked and found out that he had uploaded the same file twice!  He declared the test faulty and that was that.

Despite that, as you saw, I will repeat again, I don't want to make blind tests too hard on purpose.  We need to be interested as much in positive outcomes as negative.

 

@nyev 

All this to say, that sometimes, maybe a lesser performing component may sound better given lesser surrounding components in one’s system. Which again, judging a piece by measurements would never provide any useful context in these situations.

Thank you for another measured response.  On this point, we never foreclose such possibility.  All we ask that evidence of sonic superiority come in the form of only auditory senses.  And have statistical rigor to be reliable.  Should this evidence come about, we certainly will throw out the current measurements and investigate further what is going on.

What we face unfortunately is so and so says it sounds good.  Well, I can get you people who say the opposite.  And neither can be shown to be reliable.  Someone mentioned Robert Harley.  He raved about my Mark Levinson amplifiers.  Stereophile editors hated it.  To their credit, they showed some measurement issues although that was not normative with respect to comments from subjective reviewer (which no doubt was biased against class D amps).

So what we ask is simple: please conduct a test where all factors have been removed other than sonic fidelity of the two devices.  Match levels.  Play the same content on the same system.  And repat at least 10 times and see if you can get 8 out of 10 right.  It is not much to ask for as I perform many blind tests to add back up to my arguments.  If that is too much, and it can be, then we better run with a) measurements of the device and b) science and engineering behind how the device works and why something would or would not be audible.

I should note that when it comes to transducers, are measurements are less predictive and I have in a number of occasions liked something that didn't measure well.  Wilson Tunetot speaker review was one such speaker.  It was an expensive bookshelf speaker ($12K), so would have been easy to go with the flow of bad measurements and expensive so let's damn it.  But I could not in my listening tests and reported that.  Got heat for it from my own crowd but so be it.

 

@amir_asr ,

 

I think the test you gave yourself is too easy 😀  Your point is well taken in regards to trained listeners and MP3. I think a better test is to serve up 10 different tracks which may be MP3 or may be wave, and then test how well listeners do at accurately assessing if the track is compressed or not. I wonder if even the trained listeners will be challenged in that case without a reference.

@jayctoy

If I remember Robert Harley was warned by either Mike Moffat or Jason from Schiit audio that yagdrassil dac don’t have good measurements ? It turned out the yagdrassil got a stellar review from Robert Harley.

An outcome which he hands to any product that costs money. Why would that in any form or fashion mean anything in the context of measurements being reliable or not? I only recommend 1/3 of all products I review. What is that percentage for RH? 99%? 100%

BTW, Schiit publishes measurements of everything they build now. As noted, this came from work that we did at ASR, highlighting the value of such things. So I would not pick this as an example of anything.

 

@mastering92 

@amir_asr

mp3s are not great. Sure, you could fool someone in to thinking that 2 files are the same on a smartphone over bluetooth, but upon further inspection; in a more resolving system, you could tell the original .wav file and .mp3 file apart easily, no matter what the kbps was, even 320 kbps.

Self-bragging is common when it comes to lossy compression.  Problem is, when most of you are put to blind tests, you flunk being able to tell the source from the compressed one.  And no, resolving system has nothing to do with it.  The fact that you say that tells me you don't know what it takes to hear such differences.  As a trained listener in this domain, I can tell differences with just about any headphone on any system.  

While it is true that MP3 was not designed to be transparent, at high bitrates, especially at 320 kbps, it easily fools even the most ardent audiophiles.  I know because we have tested them.  While at Microsoft, I told my signal processing manager to recruit the large body of audiophiles we had there for testing our lossy codec.  We ran a large scale test among our self-selected audiophile group.  Results were embarrassing for me as an audiophile.  None could remotely match our trained but non-audiophile listeners.  

To hear those impairments, you need to learn to hear them.  It does not come naturally to audiophiles.  This learning also involves understanding of the algorithms and where the weak points may be.

I have lost count how many times a presenter at an audio show has whispered to me that they were playing lossy audio to audiophiles who had no idea, thinking they were uncompressed content!

You may be the exception -- there is a small percentage of audiophiles who are good at this.  To prove that, you need to provide results of a double blind test to show that and not just claim it.  Here is an example of me passing such test:

foo_abx 2.0 beta 4 report
foobar2000 v1.3.5
2015-01-05 20:26:27

File A: On_The_Street_Where_You_Live_A2.mp3
SHA1: 21f894d14e89d7176732d1bd4170e4aa39d289a3
File B: On_The_Street_Where_You_Live_A2.wav
SHA1: 3f060f9eb94eb20fc673987c631e6c57c8e7892f

Output:
DS : Primary Sound Driver

20:26:27 : Test started.
20:27:01 : 01/01
20:27:09 : 02/02
20:27:16 : 03/03
20:27:22 : 04/04
20:27:28 : 05/05
20:27:34 : 06/06
20:27:40 : 06/07
20:27:51 : 07/08
20:28:01 : 08/09
20:28:09 : 09/10
20:28:09 : Test finished.

----------
Total: 9/10
Probability that you were guessing: 1.1%

-- signature --
7a3d0c1aaaf8321306ff6cfdd1f91ff68f828a54

So please don't make such assertions unless you have evidence to back it.  Fish stories in audio are quite common.  Reliable facts, not so much....

prof,

If I remember correctly, you had stated you could not hear any difference in cables.

Yes.  So far.  Once I've controlled for sighted bias.

Without controls, I can "hear" differences like anyone else, because I'm human.

Can you demonstrate you can hear differences between cables...when you aren't allowed to peek?  (That is, when you aren't allowed to know which cable you are listening to?)    There isn't much evidence out there of people being able to do this.

As for my hearing, I've been using hearing protection for many decades and I have exceptional hearing for my age.  Plus my job depends on my being able to identify extremely minute differences - literally matching the subtle "air tone" between room recordings, or playing back 20 tracks listening for the tiniest artifact that might be flagged by a mix, etc.  So I don't get to just say on the internet that I can hear things others can't: I'm in a rubber meets the road scenario where I have to demonstrate it.

Further, if you look at my long thread in which I report on speakers I've audtitioned, the consensus is that I have been quite accurate.

So...if you mean to go the route of "prof just doesn't have sensitive enough hearing" it will not be very fruitful for you. 

If that is the case, I am not sure I can take your words seriously.

Cool.   I'll take your words more seriously when you produce some objective evidence for your claims about solid state amp "haze" or if you can show you can reliably identify between cables under conditions controlling for your imagination - that is, when you aren't allowed to peek. 

 

If I remember Robert Harley was warned by either Mike Moffat or Jason from Schiit audio that yagdrassil dac don’t have good measurements ? It turned out the yagdrassil got a stellar review from Robert Harley.

A lot of what you're posting sounds like market research.

 

Considering most of my posts are technical in nature with detailed explanations, I find that conclusion, difficult to arrive at.

prof,

If I remember correctly, you had stated you could not hear any difference in cables.  If that is the case, I am not sure I can take your words seriously.

 

 

@thespeakerdude 

A lot of what you're posting sounds like market research. (To be honest).

 

@andy2

 

You seem to be backtracking now from your previous claim which was:

 

"Why then SS amp ALWAYS has a haze where as tube amps always has a transparent sound "

(emphasis mine)

There is no evidence you’ve given to support this. The last SS amp I had in my system to compare to my CJ tube amps was a Bryston 4B3 and there was no "haze" that you mention to that amp, nor to any other SS amp I’ve ever heard.

And if anything it is more transparent than the average tube amp (which can include many expensive tube amps - see Stereophile measurements).

Whether we are talking about beginner audiophiles or audiophiles who have managed to acquire top end gear, it’s still the case solid state generally seems more popular and I see no consensus at all on this "haze."

A lot of audiophiles project and generalize their own impressions as some form of "truth" which makes these conversations a bit fraught.

 

Good luck with that!. ASR often doesn't bother listening at all to some of the components it tests. It just isn't important to them if they have their measurements to review. I think looking at numbers is more fun to them.

Amir has claimed that he personally listened to "200" pieces of equipment per years.  That is around 366/200 or 1.5 days per equipment.  I mean come on who in the world can take you seriously if you only spend such a short time evaluating an equipment.

In comparison, people at Stereophile spend weeks on an particular equipment before they publish the review.

 

 

Also if you look at a speaker freq. response and distortion, it is an order of magnitude (or even higher) higher than anything audio chain (amp, preamp), so measurement would tell you that it will dominate anything in the upstream components, it is not. You can hear the difference with different amp or preamp.

 

I have no doubt with the high output resistance of a tube amplifier that many could tell that apart from a SS amplifier because of the change in the frequency response. There have been many challenges put forth about amplifiers not run into clipping, and I assume of low output resistance sounding the same. Has anyone passed one of those tests? I can only offer my experience with active speakers, that the performance of the amplifier is indeed buried under the speakers performance assuming we are not nearing clipping. Some artifacts like noise are readily evident though, but that is to be expected. This is all determined in listening tests. These would all be SS amps with competent designs, so I don’t feel confident extending that to all amps. If I was making an amp for the high end consumer market, I think I would want to make it sound different. How else do you stand out? Otherwise you are competing with products that are much cheaper that do the same thing.

 

That is because a good tube amp will cost a lot more money compared to a SS amp.  To get the same performance you need to spend quite a bit more.  If money is not an issue, most people would go with tube.

Right now, on Audiogon, there are 113 Solid State amplifiers for sale over $5000, and 74 tube amplifiers over $5,000.

@andy2 

Nope. I’m asking because I had posed a response to him above. Also, other members on different forums claiming his measurements were done incorrectly. Just saying...will post later when I’m home.

Let's hear from @amir_asr 

Is he your God lols :-)

 

On what do you base this claim?

I myself like my tube amps, but so what?  The vast majority of audiophiles use solid state amplification and are very happy with it.  (And that includes many who have compared with tube amps, or who had previously owned tube amps).

That is because a good tube amp will cost a lot more money compared to a SS amp.  To get the same performance you need to spend quite a bit more.  If money is not an issue, most people would go with tube.

The "haze" that you keep speaking of remains completely anecdotal so it doesn't really address Amir's points about how to reliably make such determinations.

The "haze" is there.  In the old day it is very apparent in SS amp, but now SS has gotten a lot better so you don't notice it as much but a listen vs. a good tube amp will reveal it.  FET amp tends to be guilty of this more so than Bipolar.

Look if SS is so perfect than nobody would use tubes.  But SS has its flaws but people will ignore it since it can output a lot more wattage vs. tubes.

 

@speakerdude @prof lots of points to respond to. Let's hear from @amir_asr 

Tube always has inferior freq. response to SS amp and higher distortion but most people will favor tubes due to its more musical nature. 

 

On what do you base this claim?

I myself like my tube amps, but so what?  The vast majority of audiophiles use solid state amplification and are very happy with it.  (And that includes many who have compared with tube amps, or who had previously owned tube amps).

 

The "haze" that you keep speaking of remains completely anecdotal so it doesn't really address Amir's points about how to reliably make such determinations.

 

See, all explainable.

I don't think you explained anything.  Tube always has inferior freq. response to SS amp and higher distortion but most people will favor tubes due to its more musical nature.  Of course if a bad tube amp very high distortion you can tell, but most tube amps nowaday are pretty good.

FET amp with its being a square I vs. V curve sounds more tube like but its weakness is that it is less transparent vs. Bipolar and not as dynamic.

Here is the thing.  The freq. response curve and distortion will only tell you so much.  How can measurement tell you if it is a FET amp or Bipolar amp?  But a quick listening will tell you the difference between a FET vs. Bipolar.  FET amp has gotten much better now, but in the old day it was very "hazy" vs. Bipolar.  A lot of amp now uses FET as an input stage and Bipolar at the output stage.  Just like using tube as input stage then SS as the output.

 

Also if you look at a speaker freq. response and distortion, it is an order of magnitude (or even higher)  higher than anything audio chain (amp, preamp), so  measurement would tell you that it will dominate anything in the upstream components, it is not.  You can hear the difference with different amp or preamp.

@nyev ,

 

You started out with a post asserting that, and I hope I am paraphrasing correctly, that you believe we can have two things that measure identical, or close enough, but hear a difference. First, is that truly what you are asserting? 

I think measurements can provide us with significantly more information about how we will interpret how something sounds than many audiophiles give them credit to do. I think first this belief from audiophiles comes from general lack of understanding of how to interpret measurements or how to apply them. There is a lot of data in a Klippel report for a speaker. It takes some level of training, not extensive, but at least some, and definitely some experience, to read all that data and come up with a fairly good understanding of how that speaker will be perceived by most people, even more so when their room is considered.  Where this data is highly beneficial is where you have the data for the speaker you are currently using, the one you are considering, and know what you like/dislike about your current speaker.  This allows an interpretation of the measurements within a framework of the listener's preferred target sound.

I do think the most contentious thing that ASR does is make the claim for many products that the product is transparent, and not only that it is transparent, but because it is transparent, it will sound the same as this much more expensive product. I would say that question could be easily resolved with a blind format listening test, but I am now quite certain that even if that showed them to be the same, that far too many would not accept the results. I am a bit shocked by the views on blind format listening I have read here.  So I will ask you, what do you thin is an adequate and acceptable way to prove that two products sound the same?

This effect does not exist in a quiet listening environment. You go to visit an audio shop. Walk up the stairs and they’ve got a listening room. No one else there but you and the sales guy.

Yes this "effect" does happen in quiet room. The ability to pick out relevant details in a noisy environment is just one outcome of how the brain adapts. Our hearing including our ability to extract details, hear artifacts, etc. is not static. It is task dependent. At a lay level, it is called selective attention. At a neural level, our brain rapidly adapts neural weighting to the tasks on hand, which means if you are looking for discrepancies in how you think something should sound, you are far more likely to hear them as opposed to them just being "background" information.

The process whereby you adapt to a new piece of equipment is also related. Initially it is new, so you are looking for artifacts, differences, changes. If there are really changes, you are more likely to find them, because your brain has rewired to actively look for them. It will also find things that were always there that you never noticed. Over time, you/your brain settles, and you are back to listening to the music.

 

mp3s are not great. Sure, you could fool someone in to thinking that 2 files are the same on a smartphone over bluetooth, but upon further inspection; in a more resolving system, you could tell the original .wav file and .mp3 file apart easily, no matter what the kbps was, even 320 kbps.

How confident are you that if presented with only an MP3 file, 320kbps, that you could accurately state that it is MP3?

 

 

@amir_asr

mp3s are not great. Sure, you could fool someone in to thinking that 2 files are the same on a smartphone over bluetooth, but upon further inspection; in a more resolving system, you could tell the original .wav file and .mp3 file apart easily, no matter what the kbps was, even 320 kbps.

For example there is a feedback loop from the brain to the hearing system to seek out information in a noisy environment. This is the so called "cocktail party effect"

This effect does not exist in a quiet listening environment. You go to visit an audio shop. Walk up the stairs and they’ve got a listening room. No one else there but you and the sales guy.

You’re at home - in your office listening to headphones and/or speakers. The room is at probably 30 dB, perhaps even less. My office for example can be a bit lower than that. The music playing on speakers and headphones will overpower the environment in this case for 2 reasons -

First, the level is louder and...

Midrange frequencies that are louder or within the same octaves effectively cancel out other midrange frequencies. Bass and treble would also soften the resolve of background noise.

I’ve found a bunch of posts on other forums where people are refuting your measurements with their own. I will share them later today....just been busy.

 

 

@prof , like Amir’s points, you make rational points as well.  But really what we are debating is, which approach is more flawed - our senses because our biases get in the way, or measurements because while they reflect “reality”, they do not reflect on how we perceive a component’s physical performance.  For me, I believe the latter is far more flawed.  It doesn’t mean that the flaws you are pointing out in subject assessments are not valid - they totally are!  But IMO it’s the best we have, barring the day we can figure out  measure human perception of physical stimuli.  Once we can do that, we are into Bladerunner territory!

@nyev

 

But over time, and not a heck of a lot of time, say 2-3 weeks of “living with” a component, I find that I can arrive at a more stable and true subjective assessment of a component. One that personally, I find goes beyond what the measurements can tell us.

 

The thing is, if you are starting with a flawed methodology - in your case sighted listening - using the same flawed methodology over a longer time doesn’t yield more reliable results. Biases can modify, or settle in over time, and you may be attributing this to the gear rather than changes in your perception, which has been Amir’s point.

The thing is most of us really don’t want to be told...or learn...our perception isn’t reliable. We stake so much on it. But, reality doesn’t bend to our desires, human foibles are what they are.

Now in talking about blind testing like this the hackles will often go up "hold on, so you are telling us we can’t rely at all on our perception? Now we have to blind test everything or we can’t have an opinion or do this hobby?"

No. Not exactly.

First of all, clearly our perception IS relatively reliable. We get through the day using our senses. And blind testing is not easy, and depending on the gear can be utterly impractical. I don’t think any audiophile HAS to engage in blind testing or pay attention to audio science. We can all do whatever we want.

HOWEVER, the fact is we have biases and they are relevant. If we don’t acknowledge that variable, we are simply being ignorant.

So given blind testing (and often, measuring) is impractical for many audiophiles, how to navigate these problems? One way is to just say "look, I don’t care about measurements or blind testing, I’ll go just on what I seem to hear." Fine and dandy for anyone who wants to do that. But IF you are going to take your impressions and make claims to other audiophiles based on those impressions, especially if they are dubious claims in light of generally accepted technical theory and practice, then you shouldn’t be surprised if some audiophiles hold a skeptical opinion and point out they would prefer more rigorous data - e.g. measurements/listening tests controlling for biases - before they accept such a claim about said piece of gear.

So nobody has to do blind testing to conclude what they want, but IF we want to be more careful about conclusions - learning from what the scientific method has to tell us - THEN it makes sense to look for evidence that is less susceptible to run of the mill sighted bias effects.

And, though blind testing may be inconvenient for most of us, there is plenty of engineering, testing, science out there ALREADY done by competent people, that we can look to when trying to evaluate a claim about equipment.

So, the way I navigate the problem of the impracticality of blind testing everything, vs recognizing the variable of human bias is basically a heuristic like ’Extraordinary Claims Require Extraordinary Evidence.’

So, if someone tells me they auditioned some Wilson Audio speakers and some MBL speakers and they describe the differences they heard, I’m perfectly happy to provisionally accept they heard those differences. Sure, could be some bias involved, but it’s also extremely likely they heard real sonic characteristics in each.

That there are audible differences between speakers is well established by theory, practice, experience, measurable evidence and even listening tests controlled for sighted biases.

That’s not the case for some of the claims in high end audio, though. The claim that an expensive USB or AC cable will likely alter the sound from any cheap (competently designed) cable IS quite technically controversial. Many knowledgeable people (the ones not selling such stuff) will explain why this his highly unlikely given how those things work. So...for THAT type of claim I personally will raise the bar for the evidence. I’ve had the personal experience of "hearing obvious sonic differences" between cheap AC and high end AC cables, but when I performed blind tests where I didn’t know which was being used, none of the sonic differences were there at all and my guesses were random.

So when an audiophile swears up and down he heard something "so obvious" when changing a USB or AC cable, I’ve learned that we really can have very strong but erroneous impressions (something science has told us for a long time).

It won’t matter if the anecdotes pile up, because they are all using the same method that allows for sighted bias. I’ll wait until I see measurements showing actual changes in the audio signal and/or people able to reliably pass blind tests for choosing between such cables.

Neither you, nor anyone else here, needs to have the same criteria I do.  Follow your bliss.   But there ARE good reasons for having such criteria and it shouldn’t be seen as some sort of heresy for which the skeptical person is made a villain, just by giving this reasoning.

 

Cheers.

 

 

 

 

 

I have never heard a haze when listening to an SS amp,

Likewise.  I've done plenty of listening to SS and tube amps, never heard this SS "haze" of which he spoke.

Some people project their own impressions as some objective truth, as if just claiming something means it's true and "has to be explained."

Amir showing up here is like Richard Dawkins dropping in to a Church revival ;-)

 

 

 

 

Why then SS amp always has a haze where as tube amps always has a transparent sound but we all know tube always have inferior frequency response vs. ss.

In short you can't measure it.

 

I have never heard a haze when listening to an SS amp, or some startling clarity with tubes. However, one factor that may affect your personal listening experience is most tube amplifiers have high output resistance. Couple that with the right (wrong) speakers and you can accentuate some frequencies by even a few db. Maybe you interpret that as haze, maybe it even compensates for a speaker/room response issue.  Then again, it could just be what Amir stated a biased view.

 

 

 

@amir_asr , thanks for responding with a constructive perspective. It would be nice if all debates on the internet were this friendly and constructive, but it is what it is, as they say. Another way of saying it isn’t what it should be.

The problem with your argument is, I agree with everything you have said. I’ve many times been tricked by my own mind, only to realize later that my initial findings were flawed. But over time, and not a heck of a lot of time, say 2-3 weeks of “living with” a component, I find that I can arrive at a more stable and true subjective assessment of a component. One that personally, I find goes beyond what the measurements can tell us.

I don’t feel the need to win arguments, and I am comfortable with alternate perspectives coexisting with my own. I believe Audio Science Review is great for checking out how gear physically performs, but my own perspective is that too much emphasis is attributed to these measurements in terms of how much we can expect to subjectively enjoy a particular component. I think this is the fundamental point where our perspectives diverge.

I believe that other factors come into play as well. One particular example that seemed to rub some the wrong way is ASR’s review of the Mustec MA005, where ASR reported poor performance. I also see forum posts from some folks who report that the Musetec bests the more expensive Mola Mola Tambaqui which performed well in ASR’s measurements. Including one guy who sold his Tambaqui after he heard the Mustec. However, in each of these posts from people who prefer the Mustec, I’ve noticed they are feeding the DAC with an inexpensive Roon Rock NUC or a Mac Mini. I do wonder if the tables would be turned if feeding the DAC’s with a higher end streamer like the Grimm MU1. In other words, I wonder if the Tambaqui in these cases is simply exposing the flaws of the source more easily than the Musetec, or in other cases, the Holo May (which some say bests the Tambaqui) which yes, shows that it rolls off the upper frequencies more sharply that the Tambaqui. All this to say, that sometimes, maybe a lesser performing component may sound better given lesser surrounding components in one’s system. Which again, judging a piece by measurements would never provide any useful context in these situations.

@thyname 

How can you know? Objective proof? Check Ali Express. Available to your business friendly US market place. Ali express. App. There is a thread here in Audiogon. Did you send your Nordost cable to the company for authentication?

How do I know you are a real human being?  Maybe you are the output of AI/ChatGPT.  Can you prove you are not? 

Alternatively you could be a shill for high-end audio companies.  Can you prove you are not?

As to the cable, company could have reached out to me stating I tested a clone.  Why don't you write to them and ask.  Until then, I am pretty sure of what I tested as I trust the owner saying he bought all that expensive cables on the insistence of the people who sold him his Chord DAC.

You guys are really running on empty to be making these arguments....

@tcotruvo 

In the debate about science vs senses as a way to evaluate audio equipment…

??? Audio science heavily embodies listening tests and so includes our senses.  It however does that correctly by testing just what we hear, not what we pick up with other senses.  

@andy2 

Why then SS amp always has a haze where as tube amps always has a transparent sound but we all know tube always have inferior frequency response vs. ss.

In short you can't measure it.

Oh we can and do.  Measurements say that they don't have haze.  And proper listening tests confirm the same.  That you think otherwise means you are in dire need of performing a controlled test where you don't know which is which.  Otherwise, your bias as stated above will always give you what you want to hear, pun intended.

In my testing of many tube amps, they either have low enough distortion to sound just like solid state devices.  Or, they have sufficient distortion to sound muddy at lower volumes and then get quite distorted.  No way are they transparent in any form or fashion.  Now, if you are not good at hearing such non-linear impairments, then and you operate under the above bias, then your conclusion will be what you stated.

See, all explainable.

In the debate about science vs senses as a way to evaluate audio equipment…

Both work.  And neither works.

 If frequency response is flat and independent of load, has distortion and noise below threshold of hearing, then you can very confidently declare is transparent.  This analysis assumes perfect speakers.  To the extent the speaker is not, then the job gets easier and hence the reason amps with noise and distortion above threshold of hearing are also declared as transparent.

Why then SS amp always has a haze where as tube amps always has a transparent sound but we all know tube always have inferior frequency response vs. ss.

In short you can't measure it.

 

@nyev 

@amir_asr , thank you for sharing your perspectives. I have an education in computer engineering. But being an audiophile, I just don’t agree with the position that the science and measurements can totally explain our perceptions. I have a fundamental belief that science cannot explain all of the dimensions that impact our subjective interpretation of physical sound waves. Why? You have suggested folks get upset when ASR rejects a component that they subjectively praise. I think you are correct in many, many cases. It’s why people get so fired up about ASR. In my case I don’t care if ASR rejects a component that I subjectively enjoy - that doesn’t bother me in the slightest because if I enjoy it that’s all that matters to me. So why do I follow my subjective judgement over science? I simply don’t believe that science can FULLY explain, at our present level of understanding, how sound waves are subjectively interpreted by humans. 

First, thank you for kind attitude in asking this question.  :)  Much appreciated.

On your point, it is very true that we don't understand why we perceive what we perceive.  Advancing that knowledge though is domain of neurologists who want to diagnose disease.  The science we follow is psychoacoustics which is the "what" we near and don't hear.  If needed we do, we do draw from neuroscience but in general, we don't need to.

Example, we know most people can't hear above 20 kHz. The why has to do with the design of ear.  But we don't need to know that.  We simply conduct controlled tests and find out the highly non-linear frequency response of our hearing.  And then we use that to build things like lossy audio codecs (MP3, AAC, etc.) which work remarkably well in fooling people into thinking they are hearing high fidelity sound.  Again, we can look at features or of our hearing like IHC, filter banks, etc.  but we don't need to, to build a loss codec.

By the same token, we can measure a device's electrical characteristics and then determine if they fall below our threshold of hearing.  Once they do, the what is what matters, not the why.  We can declare the device transparent.

Now, I should note that listening tests play a huge rule in audio science. Every speaker and headphone I test relies on decades of psychoacoustics and controlled testing to develop the target responses.  Again, it doesn't why we are the way we are.  In speaker testing for example, we all seem to like a neutral response even though we have no idea how the music was mixed and mastered!  We have an interesting compass inside us that says deviations from flat on-axis response is not preferable.  

To be sure our hearing is complex.  For example there is a feedback loop from the brain to the hearing system to seek out information in a noisy environment.  This is the so called "cocktail party effect" where we can hear people talking to use even though there is so much background noise with others talking.  The brain dynamically creates filters to get rid of what you don't want to hear, and hear what you want to hear.

This causes problem in audio testing.  You listen to product A. Then you go and listen to product B, hoping to find a difference.  Your brain obeys your orders and will tune your hearing differently.  All of a sudden you hear a darker background.  Details become obvious that were not.  None of this is a function of device B however.  It is happening because you have knowledge of what you are trying to do, and use it to hear things differently in a comparison.

Due to understanding of above, we perform testing blindly.  Once you don't know which is which, your brain  can't bias the session.  Actually it tries but we run enough trials to find out if that is a random thing, or due to actual audible differences.

So as you see, we understand what we need to understand to determine fidelity of audio products.  Said products are not magical.  They have no intelligence  Measurements as such, powerfully tell us what they are doing. 

And no, none of his cables were "fake."  I post the review of his S/PDIF cable from Nordost.  You have a reason to claim it is fake?  And that the owner of tens of thousands of dollars in DAC electronics would buy fake $20 cables?

How can you know? Objective proof? Check Ali Express. Available to your business friendly US market place. Ali express. App. There is a thread here in Audiogon. Did you send your Nordost cable to the company for authentication?

 

Many send their gear in after coming from your camp and then watching my videos and reviews. They then wonder if they were taken for a ride

True. People who have no idea, and don’t know how good sound is like. There are many people like that. See my Original Post.

 

My camp? You have no clue what “my camp” is. Clue… just my experience. Good enough for me. Again, you do you. There is value on what you do. I am not stupid not to admit this.

 

@amir_asr : thanks for replying to me. I understand your post, and I get your point. Regardless of what this exchange sounds like, I do appreciate what you do, and I mean it. You do you. There is definitely value on what you do

@thyname 

What does that have to do with what I asked. A “member “ can send you a bunch of stuff for various reasons. Does that make you feel happy and accomplished with your life?

Also, if he spent so much money on a cable (assuming not a $20 Ali fake), don’t you think he is NOT your audience?

Sounds like you forgot your elitist argument that only poor people living in their parents home frequent ASR to find cheap Chinese gear.  So I post examples of many members sending me a ton of expensive gear indicating they have the means to purchase some.  So definitely not poor as you claimed and assumed.

And they are precisely our audience.  Many send their gear in after coming from your camp and then watching my videos and reviews.  They then wonder if they were taken for a ride or if their expensive gear really performs.  I do the testing and then they know.  In many cases their reaction is "lesson learned" and no longer follow claims by folks like you on fidelity.  Or Joe reviewer on youtube.

And no, none of his cables were "fake."  I post the review of his S/PDIF cable from Nordost.  You have a reason to claim it is fake?  And that the owner of tens of thousands of dollars in DAC electronics would buy fake $20 cables?

 

@thyname 

@amir_asr great and dandy but why anyone would pay thousands of dollars for a Mola Mola DAC if a $50 CCP DAC can measure just as good within humans’ audible thresholds of hearing? Aesthetics?

Explain this to me.

Many reasons, independent of sound.  They like the support, warranty, good looks, resale value, pride of ownership, features, etc.  I replaced my $6000 Mark Levinson DAC with a Topping.  Even though the topping is more capable (supports more formats) and has better objective fidelity, I miss the old DAC with its big and beautiful display and enclosure.

It is for the same reason I own a Reel to Reel tape deck.  It is not because it has better fidelity or cheap (tapes cost hundreds of dollars!).  But because I love looking at its reels and looks.

Net, net, there is nothing wrong in buying high-end gear that costs a lot of money.  Just make sure it doesn't sacrifice fidelity to get there.  When they do, like the Chord owner before, they will buy the much cheaper and more performant electronics and happy for it.

BTW, there is no $50 DAC that competes with Mola Mola.  The highest SINAD ranking DAC in my testing costs US $900.  Go ahead and tell me this is for out of work kids with no money.  'cause it sure as heck is not in my book.

But yes, a $9 dongle from Apple often outperorms multi-thousand dollar DACs from high-end companies.  Shame on them for shortchanging people for performance while they could have done far, far better.  Only if they properly tested their ideas of low noise, distortion, etc. before releasing the boxes to you all.

@prof : I am giving you points. Many points. A chance to redeem yourself with Master Amir. You are welcome!