Why HiFi Gear Measurements Are Misleading (yes ASR talking to you…)

Accuracy is a fools game, you can never achieve it no matter how good the measurements are, it will never sound identical to the actual musicians playing so why not enjoyable.

We have to test the product to see how good it really is. Its not a question of enjoyment. Its a question of TRUTH. If you dont care about how accurately your system can reproduce the signals you put in then you are in the wrong hobby. A speaker should not be enjoyable it should be neutral. The enjoyment should come from the music itself not the equipment. 

Who cares. 

If someone who bought it was able to derive some kind of enjoyment from it, why bother with testing it? 

Please stop supporting the man. Let him speak for himself.

Here is an example of the good work the folks at Audio science review group do for the community. About a year ago, a speaker company known as GR research perpetrated a hoax on the audio community and came out with a cheap little speaker which they called a Little giant killer. It was essentially a cheap wooden box with a single small cheap driver in ’em. The simplest speaker you could possibly have.

News got out and eventually ASR decided to get their hands on them and test them. The results were shockingly bad and the hoax had been exposed.

It is work like this which the audiophile community needs. ASR are the unsung heroes of the audio community.

The fireball sounds were a "long standing mystery" according to the article I read. Not some source of mockery for those who claimed to hear them, although there always are some mockers out there of absolutely everything. The problem comes from people insisting that their explanation of the sound they are perceiving is adequate when it isn't. The explanation for this one has been hard to come up with, partly because it's hard to be there on the scene with test equipment when it's happening. But if you come up with a plausible explanation and then test that idea by generating your own bright light that's sequenced to play a tune, and that tune can be audibly heard and recorded from a black surface receiving the light, then you've done something that sheds some light on the subject. 

I'm sure they could do a blind test easily with this. Just cover people's eyes and have the indicate when they hear sound emitting from the surface. If they accurately indicate the right times that correspond with the light emission then you know they are actually hearing the photo acoustic effect. Blind test - should be easy to pass. If it's not, you have to ask yourself why, and at least question whether or not it's really a sound wave that you're perceiving. 

If anything this whole story validates the usefulness of plausible hypothesis and then measurements to verify perceptions that are surprising and unexpected. When it comes to high end audio not too many serious people are doubting that audiophiles are really perceiving differences in sound. The debate is about people's unverified hypotheses about why they're perceiving those differences. 

I have found some major disappointments that I might post in a photo on my profile today.. talk about huge discrepancies!!

@p05129 

testing for all variables would be extremely expensive for manufacturers. You are talking about impossibilities...and also not practical.

@nyev i can see that, and you keep confusing things, so you will never actually understand what you don’t agree with. The health issues related to sitting is not something that changes what we know about chairs. Sure, it may influence future designs, but again, it doesn’t change what we know about chairs. 
 

since you can’t get things like this straight in your mind, you are not able to have a scientific discussion, and that is why people who believe in facts and science have very little patience with people who just vomits words.

I feel our knowledge of chairs may yet evolve.  Only recently we’ve started focusing on that fact that sitting for too long causes issues, with many employers encouraging the use of standing desks. So @fredrik222 , again, I get the logic of your argument, but I guess I just don’t agree with it, because I think we still have a lot to learn about how commercial electronics impacts our senses.

NYEV, very well said.  Several months ago I bought up a simple observation about ASR, specifically related to the number of highly rated DAC's from a certain Asian country.  I got a direct response from ASR who pointed out that he had rated one USA brand very highly which was true.  The next day the WHOLE discussion was taken down by Agon which I find ridiculous.  So thank you for bringing it up so eloquently.

I even posited that it may simply be that the engineers from said country are the best at implementing this relatively new audio segment and these new DAC chips.  It may be these engineers are the best at designing for measurements that ASR and maybe others are currently defining as important.  Especially in a new audio segment, I think we may see new ways to measure what we are actually hearing as you allude to.

Who listens to ASR? Terrible reviewer, a measurements are everything type of guy, just like stereo-hole, which has determined that something that measures very good can sound bad and vice-versa. 

I can't believe I read some of these posts here on agon. These are more appropriate for the ANA site. Somebody claims wearing a blindfold will alter the sound you hear. Ok, how about when you wear glasses, the glass/plastic will have a reflective surface that will also impact the sound. How about if you're bald? Another reflective surface. How about if you have long hair over your ears? Is your hair a defuser or an absorber? How about if you have cataracts? Sound kindof off the wall IMO.

1 more thing: buying a new better piece of equipment might trigger you to get another piece of equipment that betters something you might already have. This is not a placebo effect, it's upgrading. Say you have a great sounding system in a small room and you move into a much larger room in a new house. So you upgrade your speakers to a larger pair, then find out you need a more powerful amp to fill the room, so you upgrade the amp. This happens quite a bit, moving to a new house and your current system isn't the best for your new environment

Amir really lives rent free in a lot of heads here.

I don't understand the animus or energy directed against him or his site.

If it's as ludicrous as you think, dismiss it and ignore it. 

If it's credible, at least in part, give some credit where that is due and then explain why it needs to improve.

Currently, in physics, "spooky action from a distance" is not understood.

Until very recently, very smart people did not have a possible explanation regarding the existance of dark matter/dark energy.

We now have a possible (and controversial) explanation for dark matter/dark energy.

In audio, there are things we simply do not understand or do not fully understand.

What do I not fully understand? Why a cable made a difference in what I was hearing.

While it is certainly possible I may never understand, I think it’s silly to say I never will.

Knowing what isn’t going on? I think that’s important too.

@steve_wisc

Right on. ✔

Also testing just one speaker at a time; not both. Driving them to Nabraska. Then saying they don’t produce a linear response. Also, like @kota1 said, and he’s right; Amir’s room is untreated.

I mean, if you have non-ported speakers (small) that are right infront of you - in a nearfield setting; room treatment is not nearly as important.

For floor standing speakers; which he also measures, room treatments are signficantly more important. Especially since those speakers are meant to fill a room with sound and have way more potential/reflection points. One could suggest the machine alone negates all of this, but that is unlikely. Since of course, those machines for measurements may have also been made with human error and engineering choices that the owner is unware of, which adversely impact the results.

I could go further, but this wack-a-mole is so bruised I almost feel bad.

@nyev there is a pretty significant difference between saying that one topic is fully understood and saying that there is nothing new that can come up.
 

To make it more simple for you, the topic of a chair is fully understood, yet, there are new models of chairs using what is understood differently coming out every year.

My thoughts on ASR, Part II....My favorite ASR story.:

* Amir tests a pair of speakers and gives them a big thumbs down.

* Manufactorer posts that entire test was done beyond the max SPL specification.

* Insta-ban/thread delete.

Perhaps I'm wrong, but there are many twisty knobby switchy things on test equipment. My thinking is I can make a graph say anything I want.

But perhaps I'm wrong

@decooney thank you sir! For sharing that with me. I think the world would be a simpler place if we all let each other touch each others tone controls.

@raysmtb1 ..Threads like this make me yearn for a much simpler time. Like many years ago when we had tone controls. Everyone’s looking for a way to tweak their system these days. it was so much easier back in the day when you turn knob or push a slider.

Maybe it’s true that history repeats itself some times. You’re in Luck! No joke, a 50+ year in audio and audiophile buddy of mine, he’s had several amplifiers, solid state, many many tube amps (mainstream, custom, one-off builds), several Single-Ended-Triode amps (10+), and umpteen different tube preamplifiers (10+). He’s rotated through many speakers, and MORE TUBES and trying different coupling capacitors, more than anyone I’ve ever met in audio. To a ridiculous degree.

After buying a new tube preamp with "tone controls" last year, a few weeks ago he finally caved in and was ordering one these - now retired from rotating gear. And, does not give a hoot about measurements any more - just how it sounds to his ears.

Shared he wants to simply turn a knob, play music, change sound with a Remote Control. Tone controls, hey not a bad idea, back to "simpler times" - hahah. Out of Stock.

https://www.schiit.com/products/loki-max

@kenjit thanks for the response. I’m not sure all your metaphors work… so you’re saying that the cables stay the same but it’s my ears that are actually doing the burn in? Or do the cables burn in and your ears burn in and your sneakers burn in? What about your feet? Do your feet burn in like your ears? Or only your sneakers burn in like the cables? I’m going to have to ask one of the electricians out there if it’s true that the lighting burns in your house after it’s wired? How long does it take for the electrical wiring in your house to stabilize? I totally agree with you that you need to put your speakers in the appropriate place and that your room possibly could need a tweak or two because of what it’s constructed of but are you saying that this also is a form of burn in?

@raysmtb1 

your ears need to burn into the sound as well. Its no different than anything else that is new in your life. It takes a period of time to adjust to it. If you buy a pair of shoes they wont feel right until they have burned in. Part of this is the rubber becoming softer as you continue to wear them in. Of course eventuallly, the shoes will wear out which is a different thing. 

I never heard them say that the lighting would look better after the wires burned in

That is because the efffect would be too subtle to see. However audiophiles care about the smallest differences they can hear so that is why burn in cannot be ignored. 

More importantly, custom tuning is compulsory. If you want your speakers to suit your ears they will need to be tuned to your ears, and not the designers ears. 

@invalid i’m sure that everything was thoroughly tested, but I never heard the term burnt in until around 2020 when I got on this forum. I’ve been around, electricians my whole life, and I never heard them say that the lighting would look better after the wires burned in. I’ve heard the term used when speaking of a new clutch, motor or drag racing rear tires. Those all make sense, but electronics. I always thought once they were wired and tested they were complete. If something requires burning in to be  at its best doesn’t that imply that it changes when voltage is applied multiplied by some amount of time? How do the cables know when to stop burning in? If they change after 100 hours of voltage is applied wouldn’t they continue to change?? at some point wouldn’t that mean that they would go beyond their ideal, sweet spot and start going bad? Or do they only go to the good and then burn on into infinite perfection?

This is awesome….  I love it :)

@raysmtb1 you don't think they tested all there electronics and mechanicals for hundreds of hours?

@invalid absolutely, I’m sure that they ran tests. But burn in as I understand, it claims that things get better after hundreds of hours of usage of the circuit. If that holds true for audio electronics, why wouldn’t it hold true for any other kind of electronics? My point was that back then everything was analog and you would think that if it makes a difference in audio electronics that it would make some kind of a difference and all kind of electronics especially something as important as some thing like an Apollo mission. 

@raysmtb1 you don't think NASA ran tests several times before liftoff?

Threads like this make me yearn for a much simpler time. Like many years ago when we had tone controls. Everyone’s looking for a way to tweak their system these days.  it was so much easier back in the day when you turn knob or push a slider.

while reading through all this, I had a couple questions that came to mind. Let’s say it’s 1969 and you are an astronaut sitting on top of a rocket ship heading for the moon. Do you think any of the astronauts asked Houston if they had completed the burn in of the electronics inside the capsule? You would think something that important if burn in was some thing they would’ve known about it back then. Does anyone know? Was it something back then?

@fredrik222 , I think the logic of your argument is absolutely sound. Where I totally lose you, completely, is the notion that everything about commercial audio gear is 100% understood and therefore science can explain everything. Believing that we know everything about a topic like audio gear, in scientific and measurable terms, is what I can’t get behind. I get that you feel that this position is closed-minded, but it is thoroughly debatable as to who is closed minded in this scenario :).

There was a time when one popular theory was that human ears were not good enough to perceive the levels of jitter that exist in audio gear. I don’t even think ASR would support that belief at this point, and I think we are past that. As you stated, things evolve and so does our knowledge.

Evan @noske was taking the position that no scientist should ever take the position of believing they know everything to the extent that in their minds, it is “case closed”. Which is what your argument seems to imply - we know everything about commercial electronics so there is no need to believe we hear something that cannot be explained. Who is closed minded?

Would love for Amir to join in but totally understand and respect why he wouldn’t, considering the unnecessary nastiness going around!

We should strive to keep the discussion and debate positive and constructive.  We are discussing a subject we all care about whatever side of the fence we are on.  

@amir_asr what about madrona digital, don't you own that company, and don't you sell digital audio equipment?

@amir_asr 

As far as I am aware after having done research long ago, Topping and SMSL are owned by the same company: AOSHIDA 

Maybe I will...

@nyev  you can ask him yourself.

but, audio equipment, and almost everything else, changes over time. That’s just how the universe works.

but that is still not the point, the point you made is that we need keep our minds open, and you, yourself, keep your mind very closed, or you would have abandoned your straw man argument long time ago. You are the point of the point you were trying to make.

The funny thing about the subjective/objective divide is that it's ALL subject to scientific evaluation.

E.g. "burn-in" may be a mechanical phenomenon and measurable with tools we have or it may not yet be measurable.

Or, "burn-in" may be something which happens in the brain/mind of the listener. We certainly lack to the tools to measure brains that precisely, yet.

Or it may not be a phenomenon at all, either objectively or subjectively. But science cannot prove negatives.

If the phenomenon exists, it falls under the remit of science, which will do the best it can, be it physics, psychology, or whatever.

@fredrik222 , have you asked ASR if they agree with the link you posted?  I doubt they do!!  That’s my point.  Now that @amir_asr has joined the discussion maybe he can weigh in.  But I’m guessing he is not interested in engaging, beyond the more unfortunate topic he responded to.

@mastering92 

People make donations on ASR. Long ago, there was proof of collusion between said site admin and now-popular Chi-Fi audio brands (execs and designers) on various forums. Of course, before starting ASR, those tracks were paved over....so anyone who thinks ASR is an audio science charity is fooling themselves.

The only people fooling themselves are those believing such nonsense.  Here is the background on this.

The rumor started on SBAF forum.  Some guy found a distributor for Topping products who had the same *first name* as me, i.e. Amir.  That is like thinking anyone called John is like any other person called John in US!  He then made another preposterous claim that Topping and SMSL are the same company.  This is completely untrue.  These are fierce competitors and are not at all part of any company, holding or otherwise.

I pointed this out on ASR a few years ago when this came up and have heard anyone repeat it until now.

So to be clear, I have zero business relationship with any audio company, Chinese or otherwise.  I retired successfully from technology industry more than a decade ago and am in no need of commercial benefits in that manner.  My methodology for recommending audio products is based for the most part on objective measurements which cannot be subject to bias in that manner even if I did have a connection with them.

So please don't go spreading such rumors, much less to say there is any kind of "proof."  If you have proof, present it.  Otherwise, you are just showing how little it takes for you to believe in complete fantasy.  

@nyev again, you are wrong. If you want more on this particular topic explained, here you go : 

But that is still besides the point; your whole argument is a fallacy, and until you fix that, you can’t communicate with people who care about the facts and what science says, and to your point, your mind remains closed. 

Yeah but those plenty of explanations of burn-in are theories. Not proof. If it were proven, ASR followers would easily subscribe to the published proof, and would not have any issue in admitting that burn-in is a thing. They have suggested, strongly (please correct me if I am wrong on this but I believe this is their position) that burn-in is a figment of the listener’s imagination, that it is the listener than changes their mind as they adapt to the sound, not the gear. This precise example is just like researchers were asserting that the sizzling sounds of meteors are in the observer’s minds, until the "proof" later indicated otherwise.

@nyev but in the end, the only thing your example shows is the state of your mind, it is closed. It is a straw man argument, and I showed you why it is, and you stick to that argument.

So, you did prove your point, just not the way you thought you would. 

@nyev  there are plenty who explain why burn in works and how ir works, and it depends on the equipment, and also what you call burn in. Is there is a change in equipment sound and measurements over time, of course. 
 

but, we are back to what I said, you have a red car, therefore all cars are red; you don’t understand something, therefore no one understands it. Obviously this is completely false. 

@fredrik222, that path unfortunately won’t resolve our friendly debate. Because, I can name any or all of the subjects that ASR followers and other audiophiles disagree with, which don’t have foundations in science. There are theories why burn-in is a thing, but no one has definitively proven it or there wouldn’t be a debate.  And many other examples.  To which some may say, well, burn in is not a thing.  But then we just arrive back to my meteor analogy, which continues to apply just fine!

@nyev as a general principle, astronomy or Astro physics is not well understood, while consumer audio is well understood.

and the argument that we can’t explain everything with measurements is another false argument. We can can explain everything in consumer audio, and measurements do not explain anything, they confirm hypotheses. 
 

name just one subject in consumer audio that can’t be explained and one quick google search will immediately prove you wrong. 

@fredrik222 , I get your logic but I still fail to see how it invalidates my point, in any way.  I think it comes down to the fact that maybe there is a belief, which I disagree with, that science has fully explained all aspects of audio but not astronomy.  I don’t believe that is accurate.  I also think that if you believe that, it’s essentially the same degree of closed mindedness that the researchers who didn’t believe that people were hearing meteors exhibited.  There is so much in audio that we cannot explain with measurements.  Yet.

@nyev you got it. It is a straw man argument, meaning that it doesn’t actually address anything regarding the topic at hand (consumer hifi equipment science) but gives the false impression that it does (look, science is evolving in astronomy!).

so, if you really want to take on ASR and others, maybe don’t make straw man arguments.
 

also, you need to start with that our mind always interfere with how we experience things. Always. 

I was hoping this post would encourage some interesting and somewhat philosophical discussion and debate. Not everyone will get the point I was making, or agree with it, which is totally fine. Others will.

@fredrik222 I get your logic but I don’t see how it applies in this case. Also I could be wrong but I think your last sentence may have been worded in the opposite manner than you intended - genuinely not sure here:

“doesn’t mean that there are topics where science can fully explain everything.”

Were you trying to suggest there are topics where science CAN explain everything, and therefore my argument is invalid because there are topics within HIFi that can be fully explained by science?

Again, simply attempting to figure out how your logic applies here. Not sure if I got it right.

@tonywinga I think we are saying the same thing. Use your senses, explain with science. However, I do get your point. Ultimately it’s all sensory. But, at that point we are debating semantics. Maybe “science” is the wrong word to be using above. Maybe it is better to say “Use your senses first, then measure the results to substantiate and further explore what it is your are sensing.” Which is really the entire point of my original anecdote. Like John Atkinson does. In my case researchers dismissed anecdotal accounts of people hearing meteors, which turned out to be wrong. Likewise, I feel people who exclusively measure HiFi equipment without assessing how they sound can miss out on important qualities and assessments of the gear, or come to incorrect conclusions.

Our minds can interfere with the way we subjectively perceive our senses is the point here. But my further point is that it’s the best we have to go on, and there are processes one can go through to get closer to, for lack of better words, as objective as possible subjective assessments. Not sure everyone will get what I mean but that’s the best way I can describe it.

I know I’m sort of arguing both sides a bit which maybe some people don’t get. I’m acknowledging that the ASR camp have elements to their argument that I think are correct, but that ultimately I disagree that it fully supports their overall argument that you can exclusively measure gear to see how it performs. Because as flawed as our senses are with our minds getting in the way to trick us, it is the best we have.

I know that there is a level deeper into the rabbit hole here, where one could argue, our perceptions are all that matter so why does it matter if our minds are tricking us? Valid point, but I’m referring to the instances where we listen to a new USB cable and instantly go “that’s a component level upgrade!” For a week or so, then we later realize the difference was not actually as great as we initially thought. An actual example I ran into.

Great discussion!

It's a straw man argument, akin to one car is red, therefore all cars are red. 

Just because science has not explained everything in the universe and we learn more almost daily, doesn't mean that there are topics where science can fully explain everything.