It all comes down to what sounds good to you, who cares about measurements or blind tests. No one likes the same thing anyhow, like anything else in life that we consume.
Why Do So Many Audiophiles Reject Blind Testing Of Audio Components?
Because it was scientifically proven to be useless more than 60 years ago.
A speech scientist by the name of Irwin Pollack have conducted an experiment in the early 1950s. In a blind ABX listening test, he asked people to distinguish minimal pairs of consonants (like “r” and “l”, or “t” and “p”).
He found out that listeners had no problem telling these consonants apart when they were played back immediately one after the other. But as he increased the pause between the playbacks, the listener’s ability to distinguish between them diminished. Once the time separating the sounds exceeded 10-15 milliseconds (approximately 1/100th of a second), people had a really hard time telling obviously different sounds apart. Their answers became statistically no better than a random guess.
If you are interested in the science of these things, here’s a nice summary:
Categorical and noncategorical modes of speech perception along the voicing continuum
Since then, the experiment was repeated many times (last major update in 2000, Reliability of a dichotic consonant-vowel pairs task using an ABX procedure.)
So reliably recognizing the difference between similar sounds in an ABX environment is impossible. 15ms playback gap, and the listener’s guess becomes no better than random. This happens because humans don't have any meaningful waveform memory. We cannot exactly recall the sound itself, and rely on various mental models for comparison. It takes time and effort to develop these models, thus making us really bad at playing "spot the sonic difference right now and here" game.
Also, please note that the experimenters were using the sounds of speech. Human ears have significantly better resolution and discrimination in the speech spectrum. If a comparison method is not working well with speech, it would not work at all with music.
So the “double blind testing” crowd is worshiping an ABX protocol that was scientifically proven more than 60 years ago to be completely unsuitable for telling similar sounds apart. And they insist all the other methods are “unscientific.”
The irony seems to be lost on them.
Why do so many audiophiles reject blind testing of audio components? - Quora317 responses Add your response
True jss. Beyond that though most of the ones on here that deny blind testing is because it disputes their personal preferences or beliefs. Either towards a product in general such as cables, or a specific model that either wins or loses in such a comparison that goes against their thoughts on it. They will many times try to pick apart the testing procedure and explain why it was done wrong, no matter how it was done. There are fast earthers in our hobby just as there are in general. Most times when someone tells me they hear something as better and I do not, it is not because I do not hear a difference, it’s because of my personal preference in what I hear. When I don’t hear anything ( rarely, but it happens) I assume either it’s hearing degradation due to age, or my personal inability to distinguish what others are hearing. I don’t assume large numbers of people are making it up and fooling themselves about it as some will try to say. I have had a few tell me in private that they didn’t want to admit they hear a difference for exactly the reason you’d stated, or for a few, because they could t afford to play at a high level and didn’t want the embarrassment of saying so. Those ones just need to realize that we all have budgets, and so the best we can at what we can afford. Very few can actually afford the best of the best, and as we all know, the returns for the dollars get smaller as you climb up the ladder. |
@djones"Why do so many people reject/fear science?" There's a whole area of study to this called psychological rigidity, but in simplest terms it boils down to some people's problem of equating being wrong with feeling stupid. So they never consider that they are wrong. And they never consider any evidence or procedure that could prove they're wrong. That way they get to stay comfortable. When I was younger I used to suffer from this. Fortunately I worked my way out of it. It's mostly about whether you look at being wrong as feeling stupid or as a learning experience. |
| Post removed |
Every audiophile friend of mine (and myself) are totally OK with blind testing. It is fun, it is a good exercise, and it is also a learning tool to perfect our skills of perception. We do it once in a while, for fun. Speaking of myself, I welcome blind tests, (being a research scientist that's not at all surprising), but I cannot suffer people who force it to the nth degree. Blind test fanatics cannot trust their own honesty to not to attach bias to what they see, to be independent of their own previous judgements. What's the point of audio then, if one is so utterly insecure of what they hear, and so caught up with ego, pride, that it gets in the way of objectivity? To me, the sort of attitude that "mine has to be better" or having stakes at proving something with a blind test is utter rubbish. I am an audio pilgrim, welcome and embrace new experiences, even when they strengthen other's positions. Especially then, because that's an opportunity for me to learn. |
@artemus_5 "He found out that listeners had no problem telling these consonants apart when they were played back immediately one after the other. But as he increased the pause between the playbacks, the listener’s ability to distinguish between them diminished. Once the time separating the sounds exceeded 10-15 milliseconds" OK Mr Artemus explain this: Miller, say, (applies to many many others) has a listening session. Then he replaces, say, the footers under his loudspeakers with springs (boing, boing!). The replacement takes say 15 minutes; they are big heavy speakers. Then he listens again. He pronounces the SQ massively improved. As he would. But the experiment said he would not be able to tell the difference reliably after 15 milleseconds and Miller waited a quarter of an hour. |
There you go again cakyol with the under breath slighted remarks that spew nothing but JEALOUSY. Not because you can’t hear the difference, but because you WON’T.. you would make a horrible mechanic. I’d give you 3 months before they’d give you the boot for being too darn thick to learn HOW to listen.. The apprenticeship program I went through taught you HOW to learn too.. 101... how to learn. FIRST things FIRST. THEN to listen, and then ask questions... Not show what you haven’t learned. You ask question until you have the proof you need, not doubt, what you obviously can’t seem to grasp... Tough being an army of one... No roxy54 it’s not a word ( ordinary’ness). , BUT we can LEARN to think it is.. :-) |
Everyone sites this and that and my ears and brain are better than yours, etc., etc., but frankly none of this matters. All that matters is the simple question, “does music move you emotionally and does the system you have deliver that emotion to you in a way that maximizes your enjoyment?”. Like a long discussion in another recent thread where @millercarbon hit the nail on the head, this hobby isn’t just about your ears. You feel the music all over your body as it literally washes over you with your ears being a primary, but not only component. I look at all of the performance graphs I can take in my FIbre Channel Data Centre switch related career “day” job. I’m frankly not interested in it at all when it comes to music and my system. I recognize only emotional, visceral enjoyment is what has kept me enamored with music and high-end audio constantly for my entire life. I’ll continue to exercise the other half of my brain more when it comes to choosing the combination of gear for my other main hobby of considerable expense - Astrophotography :). Then again, I’m also much less comfortable with Astrophotography since it is a new hobby for me. Maybe all of this graph obsession some have with audio is a manifestation of insecurity, not yet feeling totally grounded in the hobby like I certainly feel currently with telescopes, Astro-cameras, myriad of hardware accessories, photo stacking/post processing software, etc. I don’t know. We all have our own paths to take in life. |
Short attention span theater intermission. The subject was the premise double blind testing is inappropriate and meaningless. The supporting evidence is a study involving speech recognition. The flaw in the reasoning, no evidence is given that speech recognition and component evaluations involve similar brain functions. Let alone skills sets. So the OP premise is fatally flawed. We now return you to our regularly scheduled blather. |
It's really a depressing question. Why do so many people reject/fear science?So who rejects science? Flat Earth people? Who fears science? Why would anyone be afraid of science? Science or equipment tests can tell us a lot of things but it is completely unable to show how it sounds compared to a live performance. Have you looked at how complex sounds look? Even a miniscule change can make a massive difference, yet it is impossible to detect with scientific equipment. In that regard it is entirely appropriate to reject science and instead actually listen, and don't believe reviewers who provide glowing reports just to stay in business and get the next piece of equipment to test. |
| Post removed |
| Post removed |
When I followed Andrew Jones’...an actual physicist by training/education... career from TAD/Pioneer to ELAC, I last heard his masterpiece TAD Reference One before he went to ELAC...where he showcased the diminutive Navis ARB-51... I was expecting diminutive sound from such relatively small drivers. It ended up like Spud Web against Shaq. Simply huge performance from such a small footprint. Was it the same visceral experience as the commanding presence of the $85,000/pair TADs to the $2,200/pair internally triamped ELAC ARB-51? No. Was it a difference worth the $81,000? Oh hell no. That would be a fun blind test demo to listen to, where you no nothing of what’s coming through the blind screen until the reveal, including pricing. Even Andrew was surprised at what he’s accomplished. |
I’ve had a few customers...musicians...that are actually blind, unsighted. The interesting thing about such people...especially musicians...is they actually listen with their ears. I’ve made repairs and adjustments for them based on whet they can hear, not what they can see. It’s rarely the most expensive or esoteric thing either. Simplicity works very well in this regard. |
By virtue of being on these forums and making recommendations, YOU ARE, and you know OHM (my new short form for you), that many on here are very forward in trying to convince others purely on sighted testing.
<<<<<<<<<<<<<<<<<<<<<<<>>>>>>>>>>>>>>>>>>>>>> Listen I didn't say some folks ears didn't have an agenda. I'm just saying my ears have to please ME.. And right after me you're first... I like to share affordable alternatives but VERY close to the same performance at sometime literally 100 X less. 100.00 vs 10,000.00 usd, same cable .. no kidding.. Might be missing a wooden block or two.. Blind testing doesn't need to be used, for 100 vs 10,000.00 LOL who cares.. Some people that is half a week of Starbucks. take the plunge.. |
Do you eat blindfolded https://www.theunconventionalroute.com/blind-taste-tests/ Strange, some accept science and can have fun with it. |
| Post removed |
| Post removed |
| Post removed |
| Post removed |
@hilde45 I'm naive about this debate, but if someone is willing to help me, IF ABX testing is useful at all, in what circumstances is ABX most useful and when is it least useful? Blind audio testing – PS Audio |
Blind testing has it’s place. Just not in a "Personal Stereo System". We as individuals are NOT trying to prove to others what we hear is good to them only US. Why try to please the masses because a VERY few want to prove a point. The issues is not the fact that the testing is good or bad, it just doesn’t have the final say or in my case ANY say in my system building.. No reason for me to use the method.. I like what I like, no need to prove it to anybody. Recommend maybe, BUT I don’t have to prove something is better to anyone. If I was trying to prove something WAS actually different, trickery by setting terms for testing really isn’t accurate. We are humans and we can be TRICKED. Our hearing is as much a part of seeing when in concert together. NOW add feeling and smell along with a little taste of popcorn, holy moly you got a full blown skewed test because someone POPED popcorn and YOU smelled it.. That same analogy is why a trained ear for some things is ABSOLUTE. There is no other way to gather data into YOUR brain.. You have to listen with your ears, and feel through your bottom, chest, face and hands. ALL are different collection points.. ALL adding to your hearing, perception. You really can’t measure it, but you can REMEMBER it.. Remember the phone call from a 50 year old friend, you haven’t heard from in 45 years. You know exactly who it is as soon as they speak.. We have the ability to hear the difference, the question is does every one have that ability? 7 billion people on the planet, one person calls you from 45 years ago. You know exactly who it is...there’s a blind test for ya. That actually has a purpose.. Go figure.. Hello good buddy long time no hear, BUT I can SEE you in my minds eye.. plane as day... Memory is memory. Have you ever remembered the tune but forgot the words, so you add your own.. Thirty years later your still mumblin’ the wrong lyrics to the same TUNE.. Wrong memory works too, you have to be able to DISCERN the difference.. Tough for some.. Actually a lot.. You have to be able to remember when your WRONG.. Some it is just impossible.. No names ay!!! Regards |
However, it has been consistently shown that trained listeners - those who were instructed in advance what to listen for - were more likely to be able to detect differences. See, to me, audiophiles are people who have already trained themselves on how to listen to equipment and observe differences. That said, if you have studies out there where audiophiles were no more effective at detecting differences in audio equipment in blind listening tests than the average Joe on the street, I'd love to take a look at them. Always game to be enlightened. |
jerkface The more discerning ears of the audiophile are far more useful in ABX tests.Maybe. But this has not been shown in any of the legitimate, scientific blind listening tests with which I'm familiar. However, it has been consistently shown that trained listeners - those who were instructed in advance what to listen for - were more likely to be able to detect differences. |
dletch2 No, there is not abundant literature that says blind testing is bad.I don’t think anyone has claimed that, and it’s interesting that you equate "frailty and limitations" with "bad." There is abundant literature that details the fallibility of blind testing, some of which has been linked in this thread. For the measurementalists here, blind testing is a religion; it is perfect and absolute. The results, oddly, are to be accepted on "blind faith." That was Kaptchuk’s point - which you’d understand if you actually read his paper. |
I'm naive about this debate, but if someone is willing to help me, IF ABX testing is useful at all, in what circumstances is ABX most useful and when is it least useful? I did a lot of A/B testing of speakers, DACs, etc. to choose equipment. Sometimes I hear a big difference that mattered and sometimes I caught myself inventing a difference so I could have something to tie-break two items. |
@cleeds I’ve had similar experiences as an ABX subject. I still think blind testing has value, even though it’s not likely to be of much use to audiophiles. Disagree. The more discerning ears of the audiophile are far more useful in ABX tests. I pointed out the criticisms I had for the way audio ABX tests are conducted. It doesn't defeat the utility value of audio ABX tests, just points toward some changes in approach that would increase their utility value. I'll grant you, there are some audiophiles out there who will never be convinced by the most perfectly conducted ABX testing. And most aspects of an audiophile's system cannot be easily ABX'ed, at least not at home. One can ABX a source, such as a CD player, fairly easily, as most preamps have multiple inputs and can easily be switched between them. Interconnects are a bit more challenging, and the nearly impossible test is ABX-ing a power cable, because now we're into having multiple amplifiers and some sort of switching device between them in order to verify a difference in sound between two power cables. Which is, again, why I do my best not to piss on people who choose to spend their money on these sorts of upgrades. Without a serious A/B, never mind A/B/X test, there's no way to prove them right or wrong. I prefer to spend my money on things that will demonstrably improve my system. Maybe once my room is as close to perfect as possible, I've swapped out the crossovers and the tweeters on my speakers, I've found the right cost/benefit balance on my speaker wire and interconnects, and am satisfied with the signal chain of DAC/preamp/amp I've installed, I'll consider playing around with last-mile stuff like that. But probably not. |
I volunteered for an ABX speaker wire test at Klipsch HQ back in '06. The first five rounds, I was perfect. 5 for 5 identifying the more expensive wire versus the lamp cord. 13/20 across a range of test subjects would be statistically significant, but this point to bad test design, and not any error in blind testing. The result actually had nothing to do with blind testing at all, but an ABX test where listener fatigue set in. Any good analysis of results would also look at grouping to determine if there was a listener fatigue element. This goes back to the opacity of testing, all results and methods should be published. |
To quote Disney, "because when everyone is super, no one is super". Bonus points if you can identify the reference without Google. |
The notion that blind testing for audio is an absolute test is absurd, and on so many levels. There is abundant literature (although not enough) on the frailty and limitations of blind testing in all matters of research. (That doesn’t mean that blind testing doesn’t have its place in audio, but it’s useless for most audiophiles. No, there is not abundant literature that says blind testing is bad. You will have a hard time finding any. There is literature that deals with bad testing that is blind, but not the basic concept of blind testing. Every example given in this thread claims to show blind testing is bad, but not one of the actually does. |
That does not even make sense. |
Gee @cleeds , nice selective posting there. You know there are AES members and people with access to research literature here ... This is a convention paper, not a journal paper, which means it does not go through the normal peer review of a formal journal paper. https://secure.aes.org/forum/pubs/conventions/?elib=11480 The conventional .05 significance level used to analyze typical listening tests can produce a much larger risk of concluding that audible differences are inaudible than concluding that inaudible differences are audible than concluding that inaudible differences are audible, resulting in strong systematic bias against those who believe differences are clearly audible between well designed components that are spectrally equated and not overdriven. This paper discusses ways to equalize error risks, introduces a quantitative measure of a listening test’s fairness, discusses implications for literature reviewers, and presents a statistical table enabling readers to conduct equal-error analyses without calculations. |
i’ve challenged blind testing advocates to show me a system that equals or exceeds the performance of my system using only blind testing as a system building method. all i heard was crickets. zero response. blind testers don’t assemble systems using blind testing. they just have pre-conceived opinions. so why even pay attention to them? i don’t. |
It's amazing that you could read this article, though I don't think you did, I think you are quoting others excerpts, and reach this conclusion! THE AUTHOR IS NOT ADVOCATING AGAINST BLIND TESTING! Can I be any more clear? What he is advocating against is poor quality of testing, such that the results are taken as absolute, without any consideration to whether test implication truly met the goals, and the opaqueness that often surrounds these tests! |
jerkface I volunteered for an ABX speaker wire test at Klipsch HQ ... My accuracy, as the test continued, began to deteriorate, as my ears desensitized to the source material and it all began to blur together ...I’ve had similar experiences as an ABX subject. I still think blind testing has value, even though it’s not likely to be of much use to audiophiles. Here’s another scholarly, objective evaluation that explores the frailty of blind testing in audio (referenced in the Stereophile article): " The conventional .05 significance level used to analyze typical listening tests can produce a much larger risk of concluding that audible differences are inaudible than concluding that inaudible differences are audible ... resulting in strong systematic bias against those who believe differences are clearly audible between well designed components that are spectrally equated and not overdriven." |
