Why blind listening tests are flawed

I think what you are describing is the process of using things other than how the gear actually sounds to form an opinion of how it sounds. Once you form your opinion you then look for justification to support your views. Scientifically, this is called the power of assumptions and it is how the human brain functions on a neurological level.

An oversimplified version of your post is that if you prefer blue speakers to red speakers and in a blind test you wouldn't have that visual cue to cloud your perspective. If you think speakers are ugly you are less likely to appreciate their sound.

The mind is a mysterious thing, no two are the same. Explains all.

I don't disagree but suspect the reasons are perhaps more extensive.

I do agree that to really get to grips with new gear, you often have to listen to it over a long period of time. Music can sound different depending on our moods - I read that our ears are not passive listening devices but rather the brain continually tunes the hairs inside to adjust its sensitivity to different frequencies. The listening process involves both the brain and the ear - and is far more complex than we give it credit for.

I also read that it is only when we are relaxed, can we hear the subtle differences between gear. In most ABX tests, we are under a lot of stress and our brains are not in the same frame of mind.

"This explains why many times you'll love how something sounds until you don't anymore? Anyone experience that?"

Yes, this year with a new amp. Great for two months, then I started hearing a raspyness in the treble. Tried some room treatments with absolutely no success. Sold the amp.

Larrybou, you are exactly right.
Some folks of a scientific bent tend to think everything can measured and/or explained.
The fact our unconscious memories may run the show distresses them.

I am with Doggie on this.

IMHO neither sighted nor blind tests are perfect.

To really get to the bottom of a bit of gear you need both.

But having participated in blind tests I know only too well how hard they are to set up and do properly, which is why you see such a dearth of them.

The thing that always amuses me about blind test aficionados is when I say I am always willing to learn about judging gear can they point me to their write up of blind tests they have done. Can't recall a single one that was able to to that. One cant help but get the feeling for them its religious zealotry.

That said, as I said at the start, you really need both. Its just such a pity blind tests are difficult to do properly.

Thanks
Bill

From your description it appears that you are listening to your equipment as opposed to listening to music via your equipment. Since no equipment is completely transparent and without fault, if you actively search out the shortcomings, you will hear them. In other words it's your mindset that is creating this situation. The sound of equipment may be a series of memories, but music is something that is completely in the moment. If your equipment is good enough to put you in the moment, then why look for faults? You will only make yourself unhappy.

We are highly sensitive and quite variable beings of auditory input; our surivival in the wilds depended on it, and especially our sense of hearing, providing information of direction and distance where sight alone often failed. I know not to always trust my hearing because, in all honesty, there are times when, depending on my emotive state, this sense can be influenced by factors that have nothing to do with the system or source material I am listening to. And quantitatively, it seems impossible to replicate or even know when one's emotive state is calibrated relative to an earlier time, when perhaps a qualitative assessment of equipment performance, was being audibly judged. We, I propose, are the greatest source of variation in the subjective enjoyment of music, and this must be taken into account when evaluating one component against another, or a system as a whole, or an artist, performance or recording. We, like the system and the room, play as much a role in the overall test by hearing process, whether sighted or blinded.

It’s simple - listen with your ears not with your eyes.

I understand the long term exposure to really get to know the sound of something. I’ve heard NS-10s thousands of times - when I hear a pair of those things, it’s a very familiar sound.

But - if you’re trying to make comparisons between two pieces of gear you have to level match, and make instantaneous comparisons - back and forth, back and forth. This is the way our senses work - our hearing is very non-linear, and our senses have a very short memory - much shorter than one would think.

I've been comparing multiple speaker wire lately and found the only way I could come to any kind of strong conclusions about their overall was to listen to each one several days at a time in my system.

It could have saved lots of time if a couple hours of blind listening tests would have declared an easy winner. But after a few days of listening to each with a wide variety of music, the winner was absolutely clear. And putting it back in my system weeks later confirmed that my opinion of them hadn't changed an iota.

Maybe it's just the way my brain works, but I can't really imagine anyone getting such a rock steady bead on various gear through blind listening tests.

Non-blind listening tests are flawed, but more time with gear is better than less time. If you could stretch a blind test out into weeks or months, that would be the most accurate way to make the decision.

A customer of ours has been using a microphone to measure the results in his system as a result of minor changes he has done. He's on to something- you can clearly see how distortion has changed due to his changes.

http://www.audioasylum.com/forums/otl/messages/3/36516.html

Funny- the objectivist camp says that if you hear something but can't measure it, its not real. Yet I never see them use this technique. Instead they might test one piece of equipment on the bench rather than in the system. Its not scientific! As a result, I don't seem them having any numbers at all to support their position- they are the pot calling the kettle black!

"Funny- the objectivist camp says that if you hear something but can't measure it, its not real."

I would say it may or may not be real and there is no objective way to determine for certain.

Maybe if teh test is done exactly the same way in a controlled environment for a large enough sample size, the results might turn out to be significant statistically.

That's different than saying its not real unless it can be measured.

There is no real basis to say any single test case is indicative of reality or not. A pure stand either way is just wrong.

I'm all for blind tests...the opposition is coming from hi-end makers, publications, etc...they have nothing to gain against lower cost competition...if they win...they were supposed to...lose, there goes bragging rights and their business...
.

When I used to sell gear years ago, blind listening tests for customers, though flawed for reasons the OP described, were the best tool available to help a customer decide what they liked best, AT THE TIME.

But Time Always tells.

From a buyers perspective, the best approach is to train your ears by listening to a lot of music as much as possible, as many ways as possible, then use that as the reference for determining where things stand with your setup. Then tweak and tune as needed to hit your target. Meanwhile, keep on listening along the way to as much as possible as many different ways as possible to know how things can and do really sound. At some point you will know the target when you hear it, and it probably will not even change very much, and only then will you be in a position to bring the hunt to an end.

SO the key is to know how your gear sounds and compares to everything else. BLind a/b listening tests are not needed.

Larrybou - what you need to do after listening to cables for days at a time and then declaring a winner is to have someone else switch the losing cable back in without you knowing. If you notice a change then maybe you conclusion was valid and if you don't the maybe the real difference was less than you perceived when you had full awareness of the changes being made.

I've experienced a sighted test that was more informative than the blind test, but I was watching the listeners instead of the equipment. I was evaluating the Aleph 0's in my home system and had different friends over to experience them. In previous sessions there would be an extended silence as they listened. But with the Aleph 0's each visitor would talk instead of listen, ofter jabbering about how they liked the amps. This behavior reinforced my own uneasiness about them, and I "Pass'ed."

To me, this was a measurement of the system capturing the attention of the listener through music. When the mind is captured, it shuts down other activity, such as jabbering and comparing.

I can't prove it, but always felt like the A/B blind test was the antithesis of having the mind captured by music. It seems like the testing/straining/comparing parts of the brain would be lit up, and the fascinated/delighted/captured part of the brain would fade out, which misses the main point of the whole shebang.

Ok, as I mentioned - there are two important considerations. Level matched components - quick A/B with no time gaps. It can even be sighted, but those two things MUST happen in order to exclude variables that will cloud our judgement.

During the time it takes to swap out a component, your memory has failed you (and even shorter time than that). There’s no possibility that you can remember the sound well enough to make an honest comparison. If you could, you would be an oddity, and very rich and famous. If I played you a test tone - a single tone, not even anything as complex as a song. Then very slightly pitched the tone, unplugged the cables, plugged them back in again then played the new tone - you’d hear no difference. But if I quickly A/B’d them, everyone here would hear the very slight pitch difference.

I used to get tired of hearing about level matching. Level match this, level match that... However, when you experience how different something sounds when you change the volume (even just the smallest amount) you quickly realize how important it is to level match. Our hearing is not built to make comparisons without first making sure the levels aren’t skewing the test.

I still don’t understand all of the pressure from an A/B test - I do it all of the time in the studio. Does A sound better than B? Well, let’s see: listen to A, listen to B - let’s go with B - next... I don’t sit in a session and live with a sound for a week, make a change and live with that one for another week. When you see the power in making quick, A/B comparisons, your ears will very easily tell you what’s right.

Sounds like you may not have trained your ears. If you need a blind test to hear the difference than in my opinion the change is nothing special. It took me three years to train my ears to really hear how a component sounds or interacts with a system. Since building my own components, and experimenting with parts swapping, I now have a pretty good understanding how various parts change the sound so I am not that quick to judge if one is better than the other but I can tell you what each part can do and what it cannot do in general. I also know that many people have not heard a component that really can make an improvement as I have found the different manufacturers (the large majority of them at least) all have a similar sound and each has a sound but nothing seems to do it all IMO. So as I read various comparisons, that is what I used to say before understanding how a better component can shape the sound. For example, I use a direct heated triode preamp design. Only a few people on Audiogon have heard something like this and what it can do. No caps in the signal path. You may prefer the sound of something different but most have not heard what a DHT component can do. I also have a switch that can change output resistors on the spot so you can hear what they sound like. Blind or not blind, you will hear the difference and then it all comes down to your preference. If you are familiar with two or three recordings than you should be able to hear what a change in your system does almost immediately. Otherwise I am not sure if you know what you are hearing or listening for.

My opinion. Happy Listening.

If you can't tell the difference with your eyes closed, then don't close your eyes when you listen.

Art Dudley has an interesting comparison in this month's stereophile.

Two of the examples are brilliant.

a. you don't ask an art expert to make determination of whether a painting is a forgery or fake using a blind ABX test.

b. a blind "sip" test showed Pepsi to be preferred in both Pepsi and Coke's own tests - which caused Coke to launch the ill fated New Coke - but there's a difference between a sip and a full can of the drink - and while some may prefer Pepsi's (and New Coke's) sweeter taste, it's different when you drink an entire can.

The Coke/Pepsi/New Coke tests are a great way to illustrate the limitations of blind comparison tests. The only way you can really tell what people prefer is what they choose over the long run.

Interestingly, after 27 years of digital dominance, analog-chain LPs have been roaring back. They're voting with their wallets, which is much more reliable than contrived short-term tests.

Short-term tests ignore the mechanism of mental schemas, whereby we build mental models of everything we sense. A short test ignores the mind's need to build schemas to understand constructions of various concepts (e.g., sonic signatures, musical values, etc.) and compare their virtues over time.

Wow i didn't know that.

Art Dudley's is a big disappointment to me. He constructs a straw man model of blind testing and flails away. I simply do not get his analogy about art forgery. Is there anyone out there saying you should examine visual art blindfolded? Plus he ignores the fact that peer reviewed scientific (non-subjectivist) testing is used in forgery investigation. Dudley's straw man model is limited to rapid switching and he is correct in how such quick switching or short samples can be misleading, but he refuses to explore blind testing with long-term sampling. Extended sample time blind testing probably is a very effective methodology for judging the quality of audio equipment.

The New Coke switch is probably the most studied business case ever. Pepsi knew what the outcome of their challenge would be. Coke knew it too. A sweeter drink is equivalent to an audio test where one sample is consistently louder than the other. The real question has always been why Coke reformulated? The answer to that question is still being debated, but what is undeniable is that Coke got more press attention and marketing buzz with the intro of New Coke and the reversion to Classic Coke then any consumer products company has ever received. The popular notion is that New Coke was a fiasco, but it actually revitalized Coke.

The need for blind tests illustrates to me just how little the bench measurements correspond to what we hear. If there was greater correlation, we would not need the blind tests at all.

Am I the only one that sees the irony?

10-22-14: Atmasphere
The need for blind tests illustrates to me just how little the bench measurements correspond to what we hear. If there was greater correlation, we would not need the blind tests at all.

I was watching one of those detective/forensic shows and they were analyzing the 911 call tape. The Forensic Audio Analyst said, "speech recognition is a science we've put a lot of money into but nothing is as good as the human ear and human perceptual system in understanding speech..." as he sat there in front of his expensive looking equipment and computer monitor displaying waveforms of the call. I'd say the same applies to music.

It doesn't matter what a reviewer "thinks", what "religion" he follows...important is, that he writes something which can be sold from a Magazine.
This is the reason that for every customer, buyer, reader is a "special reviewer."
We have one for readers who prefer
- ultra expensive units
- beginner units
- medium priced unit
- technical specs
- analog
- digital
- alternative products from the good old days because they don't trust modern units
- DIY wonders from enthusiastic Amateurs
- blind tests ?
You get what you deserve. What you believe or not, please, no one cares about. Keep the tire rolling, that's all we want :-)

"Larrybou - what you need to do after listening to cables for days at a time and then declaring a winner is to have someone else switch the losing cable back in without you knowing."

Maybe so. But I wouldn't trust the conclusion until I was able to spend a good number of hours listening to a variety of material on the losing cable. And then if I knew when the cable was switched, and knew it was the losing cable it wouldn't serve any purpose. If I didn't know when the cable was switched, then my memory is being tricked and it would be counter productive since I wouldn't know at what point I was listening to what cable.

"Short-term tests ignore the mechanism of mental schemas, whereby we build mental models of everything we sense. A short test ignores the mind's need to build schemas to understand constructions of various concepts (e.g., sonic signatures, musical values, etc.) and compare their virtues over time."

I knew there was a better way of saying it..

"It doesn't matter what a reviewer "thinks"

Syntax, are you really Dwayne "The Rock" Johnson? :^)

For me, Sebrof has hit the nail on the head here - the human ear/brain are still unsurpassed by any technology yet invented. One must develop one's ears to really hear and understand differences in pieces of equipment, and this requires ABA testing, otherwise one is metaphorically groping in the dark.

(I also disagree that Art Dudley has set up a straw man argument, I think both quoted points are logically sound.)

Mapman's advice on how to develop one's ear is good. Listen, listen, listen, to as many different pieces of music on as many different pieces of equipment as you can. This is also how we professional musicians learn.

I don't think Onhwy61 is reading the article right. He didn't say the art expert is analysing blind. He said, they don't make a conclusion based on small samples of an artwork when comparing real vs forgery.

But the conclusion is based on a long process that requires the total painting.

Thanks Doggieh, that makes more sense to me. I'll go back and look at the article again. I usually like Art D's writing, I just thought he was being shrill and wondered why he was even writing on the topic.

Lets not forget that magazines charge the equipment manufacturers for equipment reviews via advertising contracts. This is why overtly bad reviews are rarely seen (and when they happen you can be sure no such contract was signed).

What people perceive as sounding good is based on personal taste and the only important part of a panel of experienced listeners"blindly" judging/reviewing a piece of gear is what they agree on. That can be useful information. I'm not sure what Learsfool means about how "we professional musicians learn" since he' unclear about what it is they (us) are learning...how to play in tune? How to get an "early music" tone? I've imposed my personal opinion of many thousands of listeners as a live sound mixer and musician, and if enough people told me I suck at it I'd stop or simply not get hired again, or maybe I'm simply lucky. I've had hifi gear (and guitar amps) start sounding not so good, greasing the way for that component's exit due to the gear changing (it happens) or my taste changing or the gear simply sucking large chunks (a technical term) from day one. The quest for the "absolute sound" is silly if all you're looking for is validation of what might be your quirks (common among the wildly insecure), so it's always best to just hang with what feels right, and the absolutists can pound the collective sand.

10-22-14: Atmasphere
The need for blind tests illustrates to me just how little the bench measurements correspond to what we hear. If there was greater correlation, we would not need the blind tests at all.

Am I the only one that sees the irony?

I think the discrepancies between bench measurements and subjective impressions is that the bench measurements were contrived by one mindset while we listen with a different one.

Consider: When did measurements get important? I say it was in the late '60s to accompany the introduction of solid state electronics. In the '50s and early '60s, did tube products from HH Scott, Fisher, Marantz, Dynaco, Heath, Eico, etc. specify bandwidth and THD? These measurements really came into vogue when solid state components started taking over the product mix in the late '60s. I think the engineers formulated the specs and the marketing people used the specs to convince the buying public that the SS components measured better and therefore sounded better. It became such dogma that the great offerings from C-J and ARC were laughed at for daring to allow a 1% THD.

But I haven't seen much serious challenge to the validity of these specs and measurements as they relate to human perception. Yet most standard measurements are in the frequency or input/output comparison domains, and few if any are in the time domain or amplitude domain. In other words, the specs are oriented toward sound, but not musical values.

Consider THD: It stands for *Total* harmonic distortion, intentionally lumping together even order and odd order harmonic distortion. Not too surprising, as tubes' THD favors even-order, which tends to enrich the sound. And the easy way to lower THD in an SS amp is to increase the number of feedback loops.

But every time you add a loop, you slow down the rise time, which affects the timing, a value that affects the rhythmic aspect of music but doesn't show up in test tones, the stock in trade of specs and measurements.

Another aspect of sound reproduction is resolution of small differences in amplitude, something sometimes called microdynamics. This resolution conveys the finer expressivity and interpretation of a musical piece, the one that distinguishes Horowitz from a 2nd year piano major. Based on what I hear, this finer resolution is what distinguishes analog and tubes from digital and transistors. It's easy to feel the differernce on voice or cello.

Case in point: I have the CDs of Rostropovich's Bach Cello Suites. When I played it for the first time, my wife found it incredibly irritating. Later I got the Starker LP reissue of the same works. It's one of her favorite recordings as long as it's analog. It's not really "ears" (she has tinnitus) or expectations (she had none); it's how the output affects your brain waves and emotional state.

If reproduced music doesn't create the intended emotional state, it's a big fail, no matter what the product's controlled listening tests, measurements, or spec sheets tell you.

When doing A/B comparisons, its not necessary to do it blinded. What's important is to make the switches rapidly, because audible memory fades quickly. Multiple switches may be needed. If the results are not immediately obvious, it means that the differences are not great. The ability to do this improves with practice. It also helps to know what one is aiming for (sonic preferences), as what sounds better for one person may not for another. My two cents.

Learsfool - Science may not have a full understanding of human hearing, but it isn't too difficult to perform test of the limits of human hearing. The limits vary among individuals, but not significantly among healthy people. I can't understand why people dispute this.

Science has also proved that placebo is strong with all of us and impacts every area of our lives.

It is absolutely possible to train your ear to hear certain things, but you can't ever hear frequencies outside your physical limits.

The amount of nonsense audiophiles come up with...
Read Floyd Toole work. In a long blind testing environment (you need to distinguish between short quick comparative tests and long one) , people preferences were very similar, and correlated with better measured/design loudspeakers. It is that simple. Keep your "personal taste" i.e. ego, out of the equation, and suddenly, there is a consensus on what actually sounds good.

Hi Mceljo - what some studies have suggested is that although the ear has limits to "hearing" certain frequencies, the brain DOES somehow perceive them anyway. As far as I know, this has not been proven yet, but it is strongly suggested in different experiments that have been done, and certainly many musicians believe it to be true. Don't have any links, but it seems as though this was discussed pretty extensively here on these forums several years ago...

Hi Wolf - all I meant is that musicians learn by using their ears - by listening to as many different people/groups as possible, both live and recorded, and deciding what we like and what we don't like about a specific sound. Also, we hear ourselves every single time we play, which may seem obvious, even silly, to state, but the important point is - this type of listening is quite obviously NOT done blindly. One needs to know WHY a certain physical action results in a better sound, by essentially doing ABA type comparisons in the practice room. We learn something every time we hear ANYONE play - even if it is one more way not to do it.

'When doing A/B comparisons, its not necessary to do it blinded. What's important is to make the switches rapidly, because audible memory fades quickly.’

That’s really the heart of the matter. Listeners would be surprised at how short our sensory memory actually is... It can be surprising how two things might sound the same, but when put right next to each other, obvious differences emerge. I do this everyday at work, you really do learn to listen better, and trust your hearing more and more...

Sciencecop, what particular audiophile nonsense are you referring to? There's so much to choose from. Are you saying that long tests are valid and short ones are not? I think they accomplish different things.

I would differ with your opinion on personal taste. My dad and I recently did a comparison between a CD and a high rez file of the exact same performance. I thought the high rez file was superior, but my Dad preferred the CD. He has no experience with high rez, and to him it sounded too detailed and less musical.

One standard I have is if I could tell if a change had been made walking in blind. In a short a/b comparison I can tell a difference but I would be guessing if I came in and had to correctly identify the configuration. It would be a guess.

In respect to the OP, my experience parallels Larrybou's, but I find that critical listening for differences in gear (or whole system evaluation) requires at least two efforts on the part of the listener that tend to oppose to each other. On an ongoing, momentary basis, you must be ready and able to take inventory of a rather exhaustive length and breadth of various sonic categories and traits in the attempt to track and analyze them - to remain vigilant and on point enough to follow the sonic aspects you're concerned with. That is to say we must concentrate. And yet not lose sight of the role of those aspects in the overall picture, or the Gestalt. That is to say we must also be relaxed. This is not only a hard state, I feel, to both attain and to maintain, for whatever we might want to consider a useful purpose, but I find it's sometimes further complicated by an additional circumstance.

When I first heard the 25th anniv. edition of McCartney's Band On The Run CD, on it Paul spoke of an instance in which he recalled he had spent so much time and studio effort in an attempt to narrow down his choice between 2 different versions of one song (the title escapes me) to be chosen for the initial single release ahead of the album, that he found he no longer could rely on his own instincts for knowing which version he should choose for that single's release...IOW, he had lost his perspective on how to gauge the public's reaction on hearing it for the first time. After his producer learned of the choice between the two Paul had made, he called him up and told him that he thought Paul had made the wrong one. Not long after, it began to click for Paul that his producer had been right.

That actually happens often with artists and it's referred to simply as "getting too close to your work". It happens to audiophiles, too, I believe. And we're not known as audio"philes" for nothing. Listening for evaluation's sake is likewise a creative process that also requires a certain amount of passion, aesthetically, to see through to completion (however you may objectively define that completion). We must be sufficiently on point and yet also remain relaxed as possible, but there may come a point in it where you can simply get too close to the work and you are forced to back off for a while to keep from just spinning your wheels. Sometimes the best cure for it is a self-imposed absence...whether that's days, weeks, or whatever is always up to the individual. But, over the years I've found that when I've returned from such a prolonged absence as that, it is Far easier for me to get a handle on things than when I last left it - a very good 'reset button'. But, the give and take between concentration and being in a relaxed state does seem to force evaluation to be more of a long-term thing, to me. In all this, the truly relaxed state for us may end up being the most elusive.

Sciencecop, what particular audiophile nonsense are you referring to?

Well, you just pointed to one good example. More “detail” if indeed that is what you think you are hearing, will inevitably have to be “more musical”, don’t you think? You can only detract from the recording that was captured for you to listen to, not add anything to it that is worth listening to (like “musicality”). If you think you can improve the original capture, by hearing less of it, I would say you got an issue, and it’s not necessarily your stereo (-;

Having said all that, you most likely don’t know how these recording you are talking about were taken/transferred. So many so called “high-res” recording are not really high res, and even if they are, they are not always better due to the format they are converted to. All things equal, with the same quality DAC there is no real option for a 44.1 version of a higher sampling rate recording (or analog) to sound better. Not possible.

Sciencecop - one explanation, that I have heard, that makes some sense to me for why upsampling can sound better is that the digital filters used in upsampling DACs are better and not because there is high resolution.