Why HiFi Gear Measurements Are Misleading (yes ASR talking to you…)

We can’t trust our senses?  Really?  Science is the study of the natural world based on observation.  How do we observe the natural world?  With our senses.  Seems like a circular argument to me.

On the flip side, all you need is to buy a $1k audiophile switch, and a $5k Ethernet cable, and then your Bluetooth speaker from Best Buy will sound like a $1M system. 
 

so there is that.

+1 @celtic66 

I'd rather have a bottle in front of me than a frontal lobotomy.

Regards,

barts

+1 @celtic66 

Good point!... and we learn and move on...

Applaud legitimate scientific efforts and results.......to a point.  We need understand the world around us.  However, pick up any publication from any field of science, 50 years old, and study the practitioners' claims.  So much is so far afield it can be stunningly wrong.  Confirmation bias can afflict even the best of minds.

Practitioners of medicine used to perform frontal lobotomies in their offices.  Really? That alone should give one serious pause.

@snilf 

Thanks for your insightful remarks. I really enjoyed reading them.

In particular, the anecdote about umami, and the various factors which enter into how we hear music (including circumstances and mood). Not only is it notoriously hard to describe what we're hearing at any given time, it's also quite difficult to control our attention so that when we listen a second (or third) time, we attend to the same particulars very closely. The mind wanders even when we try to stop it — and when we do try to stop it (e.g. fix it on, say, "that cymbal crash") we are effectively in a different state of mind (we're in a controlling state of mind, not a relaxed one).

This comment bears out for me, as well:

The most important element: quality of the original recording. Number 2: room acoustics. Number 3: speakers. After that, it’s all marginal effects. The debates about power cords, interconnects, even fuses is, well....

The way I understand the situation with many (not all) audiophiles is as follows: They're into the hobby. They want to hear changes and make improvements. But they cannot or will not listen to only well-recorded material (who can blame them) or deal with their room acoustics (practical obstacles or laziness). They also want to gear shop. So, in order to pursue the hobby, they have to exaggerate the differences made in what they can actually change. And that leads to debate of what (most will not admit) are marginal differences — compared to the ones which will really change things.

@falconquest   -- 

Each one of us is therefore unique. We simply will never perceive something, visual, auditory etc. the same way. That is why we cannot agree on "what's best" or what is the best method for determining "best".

I suppose the question becomes how we can ever agree on anything or even converse. My guess is that we are not only brains but creatures sharing a language and a culture. Thus, despite our manifold differences, we find ways to agree and disagree. 

@nyev 

Our subjective sensory experience IS flawed.

I know what you mean. I would use the word "fallible" and "subject to reconsideration when used as the basis for judgments."

Like you, I've made many mistakes in characterizing and evaluating what I was hearing. Talking to others and also just taking more time before judging has been the key for me. But your point is well taken.

@nyev 

You aren’t going to find many ASR lovers here. Most of us see him for what he is and move on.

All of our sensory capabilities are not to be trusted. That is why our brain is so large...to better interpret what we sense. The foundation of science is proof and repeatability. Subjective experience is just that...subjective. Not proof of anything. Millions of people talk to their God everyday. Does not prove existence of a God. And there are so many other variables; cost, peer opinion, opponents' opinion, politics, origin, age, bias......

Discussion is valuable however, now what did I do with my blindfold? Must have misplaced it after my last duel. 

My spouse is a PhD that works with brains (while they are still in people's heads mind you) We all create neural pathways within our brains that affect our perceptions. Each one of us is therefore unique. We simply will never perceive something, visual, auditory etc. the same way. That is why we cannot agree on "what's best" or what is the best method for determining "best". Measuring audio equipment performance is a scientific analysis but becomes irrelevant when introduced to our unique brains while listening.

As far as hearing the difference between amplifiers, I can definitely hear a difference. In my main system I owned an SS amp from a particular manufacturer that was pure class A. It sounded great but at higher SPL's would go into clipping. I decided to sell it and purchased a class A/B amp from the same manufacturer. It offers a bit more power but also offers a much deeper dynamic range than the pure class A unit. Likewise in my second system I sometimes switch between a low power tube amp and a SS integrated. I can tell the difference immediately. 

By the way, I can't state this for fact because I am certainly not a scientist but genetics may play a role as well. Think about the unique situation with cilantro. Some people love it while others that have a particular gene perceive it as having a soapy taste. This is my situation. While my spouse loves it I find it completely terrible! Can genetics also affect our hearing? I can't say for sure but I would bet that it can.

For those of you who may not have read the discussion about Viv Labs Rigid Float tonearms, some of the posts made by people who insisted that you should only take notice of measurements and theories are quite unbelievable.

Many years ago I was blind auditioned 2 amplifiers, I could definitely hear the difference and the one I bought was an Audiolab and the other was Naim. What makes this to this discussion is that before I did this my heart was set on buying the Naim, and I genuinely believe that if I knew what I was listening to because of my preconception I would have convinced myself the Naim was better.

You cannot tell if you will like driving or owning a car from measurements of its performance.

As far as ASR goes:

There was a time that Amir, on many posts, would post a photo of Pink Panther asking for site donations to keep the site running.

In a thread here, not long ago, he said he has +$200,000 worth of testing equipment.

"Suddenly" becomes a chi-fi proponent?

Could have come from donations.

Just as likely it didn't.

I'm on the side of it didn't.

Measurements and specifications are important, but they don't necessarily reveal how something will sound. I find Stereophiles measurements essential in helping me evaluate equipment and company's products! What's bothersome is how misleading some companies advertised specifications are. A recent speaker review in Stereophile found that the speakers sensitivity ratings were almost 8 dB lower than what the company specified! Even worse are power ratings for tube amps. It seems the new standards for company's power ratings are at 10% distortion! I've even seen a review discover an manufacturers mistake in one piece of gear. They were putting in a lower gain modules from a cheaper model product in the more expensive product and the review alerted them of the mistake!

I am puzzled as to why anyone thinks a "blind" listening test should require a blindfold.  "Blind" means that the test subjectl/istener does not know which equipment is being used - not that the test subject/listener is blindfolded.  

Decades ago Peter Walker of Quad did a listening test of two amps

I have to completely agree with you here. Science and mankind have not made any progress since then. You should be given the Nobel award for this find.

@hilde45 wrote, "Skeptics like to call "placebo" on people who hear things, but not everyone who hears something is delusional."

Indeed, which is why double or even triple blind tests are necessary. Then you have the issue of just because you CAN hear a difference between say two cables, or whatever, which one is "best" depends on your own personal perceptions, tastes, expectations, and likes. Measurements won’t show you what is "best".

The problem with ASR is that there are a limited number of electrical parameters that can be measured with their equipment. I read one review of a $900 DAC where he admits he is more or less measuring the limits of what his test gear CAN show (measuring the noise in his own test equipment). By that admission I guess any DAC costing more than that is superfluous, right?

The thing is, no good manufacturer I know of ever designs gear by a spec sheet alone, tests it to see if it meets specs, and sends it out the door without listening to it first or at least a representative sample.

Just because there are somethings that can be measured doesn’t mean it gives us the entire picture, otherwise there probably wouldn’t be any "audiophile quality" gear in production.

It’s kind of like having only a colorimeter and judging a car by how close to "cherry red" the paint finish is when there’s an orange Maserati MC20 waiting to be tested.

Splendid post!

@jasonbourne52

re: quad 405 vs tubes.  It isn't that simple.  The 405 is well known as a "soft" sounding amp (extremes somewhat rolled off).

Two amps with different topologies may sound the same, but it is silly to infer that therefore ALL amps sound the same.  Ridiculous really...

I don't know what ASR is, but the only way to ensure someone isn't being influenced by price, size, brand name, etc is to perform blind testing. Otherwise it's 100% opinion and you know what they say about those...

Absolutes in either direction are misguided. There is some usefulness in subjective listening and some usefulness in objective measurement.

Throwing either baby out with the bathwater is not smart in my opinion.

I would add that Amir also is adjusting his measurement methods and techniques as he learns more.  And there are others on that site that make different measurements than Amir and make interesting inferences about the measurement techniques and results  and the expected sonic consequences.

Science is many time interactive, but sometimes it requires a major adjustment in measurement, theorizing and retesting. Do we really know everything that should be measured to duplicate quality sound reproduction?

We have a half century of most accepting the Big Bang as generally descriptive of the universe that we see. At some point it was suggested that black energy and black mass existed. This gave us convenient theories to explain phenomena that we don’t understand.  Now, everything is up in the air.

Uncertainty of Science. ..

nyev

Our subjective sensory experience IS flawed.

That's an absolutely outrageous claim. IS there any evidence or IS it merely your perception.

Are you claiming test equipment hears better than humans? What a load of rubbish!

jasonbourne57: thanks for the anecdote. Richard Clark and Peter Aczel are two more "experts" who maintain the relative unimportance of amplification so long as there is sufficient power never to drive the amp into clipping. Both have done some extensive "objective" testing to confirm this; Richard Clark actually issued a challenge, with a pretty big monetary prize, to anyone who could reliably tell two amps apart in blind A/B/X tests. So far, no one has won the money.

I very much take this to heart in the design of my system. The most important element: quality of the original recording. Number 2: room acoustics. Number 3: speakers. After that, it’s all marginal effects. The debates about power cords, interconnects, even fuses is, well....

Interesting thread. A few observations.

First, regarding nyev's "debate" with noske. Thanks, noske, for that bit of science history, but nyev's point is not thereby invalidated. Certainly, before Bell's discovery in 1880, science would have rejected testimonials of perceived sounds associated with meteors for the reason nyev points out. And that's enough to provide a compelling instance of the maxim that "not everything you can hear can be measured." noske's original post was not intended as a lesson in the history of science, but as a provocative instance of an experienced phenomenon that was not (at one time) explainable by "science." 

FWIW, here's another example of that sort of cautionary tale. Before scientists discovered (at the turn of the 20th century) the neural receptors for a taste now known as "umami," it was believed that all taste was analyzable into sweet, sour, salty and bitter. Does that mean no one tasted "umami" before the 20th century? Of course not.

But the other side of this controversy also has merit. Audio equipment is, after all, designed by engineers with educations in the relevant sciences. Audio components are devices that are designed according to the same scientific "facts" that ASR purports to measure; they're not naturally occurring phenomena. (By the way: when nyev writes "phenomena" [plural], he means "phenomenon" [singular].)

But, for reasons that are not well understood, the representation of sound is not always best served by "accuracy." A recent discussion on this forum of Nelson Pass's "harmonic distortion generator" is relevant here. Pass set out to introduce into solid state amplification some of the "artifacts" (that is, distortion) that many audiophiles describe as the "warmth" of tube sound. Obviously, Amir at ASR would pan any amplifier that measured in such a way. But many listeners prefer the sound. Who knows why? 

I'm convinced that we audiophiles too often overlook the vagaries of subjectivity—just as oenophiles often do when judging wines. On some occasions, one's state of being (whatever that means) is just better suited to appreciating music (or wine) than on others. A couple of days ago, an audiophile friend brought over his $5,000 PS Audio power "regenerator," something Amir has debunked as bogus on ASR. My friend swears by it. And I have to say, although I could NOT discern a difference in the sound of a given track with this device in or out of my system, he left it with me for the night—and I just couldn't stop listening to music. The music really "engaged" my attention, and I didn't want to do anything else. Was that because of the device? Frankly, I doubt it. Getting involved in the music is not such a rare delight. But it is rare enough that I had a great time that evening; listening not to the system, but to the music is what this hobby is all about. Even if this is a placebo effect, I do want more of it.

@jasonbourne52 , good one.

I can see your point, and agree with you.

Of course, it's 2023.....but well presented.

Decades ago Peter Walker of Quad did a listening test of two amps - the Quad II tube amp and the 405. Speakers were the Quad 63’s. Music sources were 30" IPS copies of EMI classical master tapes. Reviewers were audio recording engineers and members of the British HiFi press. Nobody could tell the two very different amps apart as long as clipping was avoided. So there you have it! A well-controlled subjective listening test without measurements!

@edcyn ASR = Audio Science Review.  Led by Amir, an online reviewer who suggests that subjective sensory assessments of audio gear cannot be trusted.  Therefore, he takes a measurements first approach to assess the performance of audio gear.  He ends up shooting down some equipment that are well respected by audiophiles, but fare poorly in his measurement tests. 

To his credit, he has amassed quite a following, with many people crying “snake oil”.  His followers and other audiophiles have been at odds for quite some time now…

@gs5556 , lol….  Now you’ve opened Pandora’s box….

@joshua43214 , in my non-scientist opinion you are describing the ideal mindset of a good scientist.  Scientists are supposed to be sceptical.  But at times it’s also helpful to remain at least a little bit sceptical of scientific consensus as well, especially when new data is brought forward.

What's ASR?

As a scientist, one of the really cool things I get to experience a times is an observation that does not make sense or is counter to consensus.

It is important to remember that "consensus" among scientists is science by democracy - it is not scientific at all, just our current opinion and can change in an instant when new information is gained.

When I see someone point at a measurement and state boldly "that is not audible," I am reminded of the same being said about MP3 when Apple first started pushing the format. Same thing was said about jitter for many years, engineers stating boldly that human hearing is not that sensitive to time.

@mastering92 , all valid points.  It’s a matter of figuring out, among all the flawed approaches to evaluating gear, which is the least flawed.  I agree the answer to that is unequivocally the use of our own senses.

To your point with blindfolds introducing a “probe effect”, I’m not sure if it’s in my head or not, but I feel like there could be a difference in sound when I put my glasses on.  What I know for certain is that transparency is increased when I look to the floor, as opposed to when looking straight forward.

If the researchers recorded the meteor sound and played it to audiophiles, would they all agree they heard the same thing?

@nyev

Sensory experience is all we’ve got.

Go to a costly steakhouse - order a steak. Does it taste just good or much better than a $30 steak? we’re talking 3 - 6 times the price. If it is dry, not properly cooked (medium or medium rare etc. to your liking, you will notice). Even if you were blindfolded and asked to try, you would notice.

Ask a friend to audition 2 different bluetooth headphones with the same song on their smartphone. Then once they’re done, ask them which headphone they liked more and why. You could aruge they were not perfectly volume matched - but evenso, the differences in sound quality will be obvious.

Sensory experience is certainly imperfect; like all human senses. We are only human - not gods. Because we experience music from our audio systems with our senses, that sensory experience is all-important. All of it - sight, hearing, etc.

Blind listening for audio is a flawed practice. I don’t know anyone in pro audio that uses it. For example, at AIR studios in London, a power amplifier for their main control room (I believe) was chosen based on listening sessions. They bought a Class A/B power amp from a UK-based company called ATC.

😂Those who are fond of conducting blind tests for audio believe that a certain number of successful trials is sufficient for proving whether we can hear a difference or not... how did we arrive at this number of successful trials?!

😅 8/10 or even 10/10 successful trials could be riddled with guesses and inaccurate auditory memory recall. The test subject may not admit they were unsure, because they wanted to be correct and prove their ability to be golden-eared to their peers.

Wearing a blindfold also creates problems that make an objective listening test more difficult. Blindfolds may hamper with the frequency response characteristics of speakers and headphones.

Because blindfolds are made of soft fabric with padding or a sheet of fabric, placing them over the eyes creates a sound-absorbing pocket, whereby the sound waves from speakers would not disperse as evenly with it on. And since our eardrums are levelled to our eyes, they can be a direct source of sound wave absorption/reflection from the blindfold strap or covering.

A blindfold may interfere with achieving a proper seal with over-the-ear headphones and on-ear headphones. If conducting such a test, the test subject may already be attuned to how said headphone fits and feels on or over their ears... thereby giving them a sense of pseudo-preferential vision, much like figuratively having another set of eyes to visualize the set of headphones being auditioned in their mind during the test.

Lastly (for now anyway), you must acknowledge at some point in your subconcious that a "blind test" which you believe is wholly unfallible is being conducted. And as the subject of that test, you start to believe in the testing methodology. If you haven’t been the subject of such a test before, that is blind faith being invested...no better than losing teeth as a kid and asking the tooth fairy for 20 bucks.

I will even go so far as to say that I do agree with a lot of what @amir_asr says.  Our subjective sensory experience IS flawed.  We’d be lying to ourselves if we thought that wasn’t the case.  I’ve been caught with that, with initial impressions of a new piece sounding way better, only very quickly realizing later that it wasn’t actually THAT much better.  I also agree that blind testing is valuable, whenever possible.  Even blind testing is somewhat limited.  Just because one might not be able to discern differences in a blind test does not mean that those indiscernible differences may not add up to greater or lesser enjoyment over the long term, even if we are not directly aware of those differences.

BUT.  I do believe your sensory experience, while flawed, is the best form of measurement we have and the one that ultimately matters.  The problems Amir mentions which I agree with, can be worked through, with extended listening and, as much as possible, keeping an open mind. 

As an aside.  Since @amir_asr has supported the practice and value of blind testing, why does he not do that with the gear he measures?  If it was done in a fair, controlled manner, it would be really great to see how all of those gears he trashes would fare in such a test.   It would be great to see how frequently his measurement results would be corroborated by such tests.

If I were Amir and I truly believed in my heart the value of measurements, I’d want to prove it with such tests, which Amir himself has said are useful.  It would be great to see a blind test of burnt in vs brand new gear.

That would be the ULTIMATE forum/YouTube channel.  One that specializes exclusively in controlled, double blind testing.

@decooney ✅️

When someone cannot effectively describe how something sounds in different ways, measurements alone by self-proclaimed experts are of no interest to me. 

Agree. To each their own. Spent many hours at ASR. General impression after more personal experience: you only have what you’ve heard - measurements tell only part of the story.

@tuberist , exactly.  To each their own.  No harm no foul.  Conspiracy theories notwithstanding.

For amusement I viewed ASR yesterday. Seemed like audio gear was equated with lab equipment,  not musical reproduction. To each their own. 

My original post wasn’t really meant to start a ASR bash thread, but I see that probably will happen anyways!  Conspiracies would be bad but hey bad stuff happens in this world.

My post was simply sharing a personal account that demonstrates, extremely well in my opinion, why I believe it’s critical to start with your senses, and only then apply science and measurements.  Which really what John Atkinson has done for all those years.

Does that present an alternate philosophy to that of ASR? Absolutely.  But wow I can’t bring myself to passionately hate on them as much as some do!

@noske , it’s not as if I colluded with Earth and Sky to rewrite history to prove my point in this thread. I found that 25 years ago when I experienced this phenomena that it was widely questioned and even ridiculed. I merely found an article to corroborate my experience 25 years ago. It was the very first link I clicked today that corroborated it. Sorry!

Not questioning that the science existed way before. But no one was applying it in these scenarios until a researcher, Keay, was able to record it. Remember there is a difference between scientific theory and recorded proof of the theory.

If you think that very article I found somehow is bogus, well, I’m sure you can find it corroborated elsewhere, in many, many places….

Here is one for the road, from NPR, saying Scientists doubted the accounts of those hearing meteors.  

NPR

So yeah… Sorry!

@hilde45

Long ago, there was proof of collusion between said site admin and now-popular Chi-Fi audio brands (execs and designers) on various forums.

I’m glad you think so. And it’s true. If you do enough digging online, you would be very lucky to find even 1 forum post that confirmed this as fact....like I said..it’s been paved over. Think about it - asking for donations + an overabundance of audio gear from particular audio manufacturers. No 3rd party testing or validation of his measurements. But why would he need to?! He’s a genius afterall...send it to audio precision...get them to do even more measurements and we’ll see who’s right.

That forum is where you go for digital conspiracy theories ....

Like my father used to say: "nobody gives you anything for free in this world. There is always a cost."

I actually enjoy posting here on audiogon because it’s a great community. I ask a question, people answer with useful info. I try my best to reciprocate. It’s fun being on here...

@noske , also see this paragraph, which is HIGHLY pertinent to my point with respect to HiFi:

“Even in modern times folks who reported hearing such sounds were ridiculed. It was only about 25 years ago that Keay was able to do the research and legitimize the experiences of all those generations of people. It shows there are still wonders in nature yet to be recognized and understood. We should take this experience with meteors as a reason to open our minds to what may yet be learned.”

That last line is just perfect…..

@mastering92 

Long ago, there was proof of collusion between said site admin and now-popular Chi-Fi audio brands (execs and designers) on various forums.

This is a very, very serious accusation. I had not heard it before. @amir_asr might need to defends ASR on that one.

@nyev

elementary school audio science = ASR
The measurements done for audio electronics in particular do not paint an accurate picture of what we are supposed to hear. Much less rated output power (also false in my experience) Therefore, I have to use the Sabaj A10h as a basic tool to test if amplifier inputs are working (units I’m selling) rather than listening to / enjoying it.

I would always suggest listening first; and measurements later. There are obviously reasons why such a site exists...and I don’t think the founder is out to save audiophiles from themselves and provide 100% honest reviews to save people money; all for nothing....

People make donations on ASR. Long ago, there was proof of collusion between said site admin and now-popular Chi-Fi audio brands (execs and designers) on various forums. Of course, before starting ASR, those tracks were paved over....so anyone who thinks ASR is an audio science charity is fooling themselves.

it is a cult - where noobs and audio novices gather to talk about how their latest $10 headphone dongle or $99 DAC is far superior to "audiophile jewelry" all while never having stepped foot in an audio store to listen...because those who are most influencial on that site tell them that higher THD is the only thing that can impact audio quality....lol

Very interesting straw man  argument.

@noske here is an article that explains how researchers once thought people that heard this phenomena was “fantasy”.  It’s lower in the article.

Earth & Sky

“The dismissal of these observations by suggesting that the sound perception may be psychological through “an affrighted imagination” set back the study of the phenomenon for nearly two centuries.”

So yeah.

After I heard this phenomena I was interested in what I heard, and wanted to learn more.  It’s not like I took this example and published it in a scientific journal or talked to university professors.  As I mentioned I had simply googled this subject at the time.  I had found several articles and Wikipedia posts saying that this phenomena was being reported widely and that scientists had concluded that this effect was psychological.  Are these reports that I read the actual sentiment of the wider scientific consensus?  Of course not.  It was simply what was being reported widely from what I found on the internet at the time.  Fast forward years later and that all changed…