Blind Testing is Dead - Long live My Wallet testing.

That's a pretty good summation and close to how I feel about all of this.
However, if everyone felt that way, there'd be way less acrimony, discussion and debate here except for one's taste in music. 😄

All the best,
Nonoise

That is very well said Erik we each have our own sense of value and our own ears.  We are far better off when we make decisions based on our own listening and the gear that makes us most happy.

Erik,

I agree with following statement, 

“The more you detach yourself from brands, costs and worries about measurements the more frugal, and happier you will be”. 

I took a chance on reletively unknown brand late last year and couldn’t be any happier. The mono amps continues to amaze me each day as they are close to being fully broken-in. 

Eric: It seems like you’re trying to breathe new life into these old controversies.  I’ll click.  My guess is that the Audiogon forum readers are securely in their respective corners.

Developers still use double blind testing.  IMO DB testing is too difficult for consumers and is suicidal for reviewers. 

I see you contribute to DIYaudio on occasion.  There was a developer over there lamenting the daunting task of displacing a perennial speaker favorite from the DB winners circle.  (I assumed it was a competitor’s product.)

I still read the measurements JA posts on speakers.  A speaker with a frequency response rising from 10K to 20K is something I would avoid - no need to waste time there.   

There are a number of audio products that either left the engineering domain a while back (luxury electronics) or were never in it to begin with (tweaks).   Measurements are irrelevant to these guys and their customers.

I agree that we all decide with our ears and our wallet (although some seek validation on various forums.)  

Mike

I hope I'm not breathing new life into old arguments.


My intention was to bring a synthesis of competing ideas together into one that makes us happier, and reduces cognitive dissonance. :)

Erik,
Good post; so true, hahaha!
At one time; here in South Florida, there were a few places within a 10 mile radius; where I could test different "high-end" sources, etc.., along with "high-end" speakers.
Now there's one, that I know of; where someone is looming over my shoulder.
( makes me itch). 
Understandable to an extent; but I can't relax and listen that way.

Hi @ebm

Wow another very important thread!!!!

Maybe you aren’t in the right forum for you? I don’t see you making any constructive statements, I don’t see you being helpful, or appreciative of the work anyone else here does, but I seem to live rent free in your head.

Have you thought about knitting? Maybe that’s where your heart is.

Best,

E

Blind testing by accident is valid I think.How many of us have dropped in a new cable or component and have a spouse or friend comment on the change in sound?

WADIA best brand off all audio equipments.

@mikexxyz, "IMO DB testing is too difficult for consumers and is suicidal for reviewers."

Thanks for that, I needed some cheering up after today's sad news of Scott Walker's death.

Your one sentence sums things up brilliantly. It is the elephant in the room as far as audio reviewing is concerning. They could and should do it, but know only too well the consequences.

Therefore they avoid it like the plague and squirm around making all manner of excuses.

cd318"It is the elephant in the room as far as audio reviewing is concerning. They could and should do it, but know only too well the consequences."

Of what practical use, utility, or function would a blind test undertaken by a critic, reviewer, or other audiophile possibly have to you as a reader, user, or consumer? I think smart users of Music Reproduction Systems make they're own decisions, determinations, and acquisitions using they're own methods, techniques, and processes.

Like results of any kind of testing, blind test results should always be taken with a grain of salt. Sometimes people have ulterior motives, you know, like Boeing, for example. Or, say, Harmon Kardon.

Components should have stickers like wine: 92 points- rich, fully developed acoustic space, strong wood notes but not too brassy, etc. 

Until the high-end manufacturers wake up and start using stickers the only thing we can do is trust buyers to carry only the best brands. That is why I do all my high-end shopping at Costco.

Even so I like to let most purchases age one to three years before listening.

@clearthink, if these esteemed reviewers with all of their years and years of listening and critically evaluating equipment cannot reach a clear evaluation through blind testing then we are forced to conclude that either the differences are so subliminal as to hardly be worth the bother or there aren't any. 

Let's face it, subliminal differences are of little use to most of us audiophiles, but blind evaluation must always reveal major ones, ones that we can hear 365 days a year, ones we are all interested in. I can't speak for others but I have no interest in pursuing subtle subliminal improvements when I am looking to upgrade. Many years ago I started with a Rega 3 and later upgraded to an LP12. Was it an upgrade? Yes. Was it worth the expense and trouble? No, but that's another story.

I agree that their opinion of reviewers is far from being the final word but its beginning to look as if that opinion is almost entirely worthless. Little more than subjective rambling in most cases. Unfortunately many still rely upon reviews even in 2019 instead of regarding them as no more than light, but frivolous entertainment.

HiVyNyws channel is one of the very few left that doesn't pull his punches. But then he's reviewing vinyl where real (as distinct from imagined) differences are easy to demonstrate.
https://youtu.be/E4G7hSUQoO

In cases where home auditioning is impossible perhaps we need to go back to evaluating and buying gear on specifications once more? If you know how to read them they can tell you at a glance a lot more than some lilly livered critic ever will.

millercarbon"Components should have stickers like wine: 92 points- rich, fully developed acoustic space, strong wood notes but not too brassy"

Those who are knowledgeable, educated, and experienced do not need others to tell them how equipment sounds whether it be through labels, pronouncements on the Internet, or the claims, beliefs, and convictions of other "audiophiles." 

Right on Erik!

Tom

cd318"if these esteemed reviewers with all of their years and years of listening and critically evaluating equipment cannot reach a clear evaluation through blind testing then we are forced to conclude that either the differences are so subliminal as to hardly be worth the bother or there aren't any."

I am in no way forced, compelled, or inclined to reach this conclusion that you state because I am not subject to the two logical fallacies you have committed hear in one sentence!

" subliminal differences are of little use to most of us audiophiles, but blind evaluation must always reveal major ones"

This is an opinion stated as fact yet another logical fallacy!

" In cases where home auditioning is impossible perhaps we need to go back to evaluating and buying gear on specifications once more? If you know how to read them they can tell you at a glance a lot more than some lilly livered critic ever will"

You cannot judge a book by it's cover or a Music Reproduction System component or accessory based on measurements alone this has been repeatedly, reliably, and consistently shown, demonstrated, and revealed.

Truel. true. the wallet delivers the final verdict.

but, blind listening is not dead!

trust me on that one.

my recent visit to an audio Show offered a few ways to examine, listen, investigate, and or ‘value/appraise’ its various exhibitions.

I could ask right off what is being demonstrated front to back, top to bottom, then get quite attentive and decide how to judge each demo.
 
I chose only to find out the brand of the speakers and then, which amp (s) were driving them.

if the demo was one that reached out and grabbed me, then I would get more details.

entering every room was for me, a real blind audition.

of all the folks here esthetics should not be much of or at all, a biggie for me when deciding between pieces of audio gear.

but it is….

this is mostly with racks and speakers, not so much with the electronics per se.

but there is only one way to note the influence or input of any device within an audio system and that is to know the system itself prior to exchanging ANY facet within it.

otherwise, the only known value of any outfit which can be remarked upon sanely will be the system’s loud speakers.

all other anecdotal feedback, input, articles, press, blogs are incidentals, asides, humourous or otherwise are instances of time well or poorly wasted.

the only thing I extract from published notes on this or that device or accessory might be to see what ever commonalities are contained in the various judgements.

its plain testy and difficult to perfectly setup ‘blind’ testing of a particular device to begin with, though with some effort and within some reasonableness it can be done so at least the broadest strokes of change can be unearthed by the listeners.. .

Many here have made a choice based mostly on faith when procuring their next piece for their stereo system.

so then it is quite true that at times even ‘blind wallets’ will buy the next gizmo.

@blindjim, "but there is only one way to note the influence or input of any device within an audio system and that is to know the system itself prior to exchanging ANY facet within it.

otherwise, the only known value of any outfit which can be remarked upon sanely will be the system’s loud speakers."

Many already believe that loudspeakers are where virtually all sound quality differences can be easily found. Easy to hear differences 24 hours a day, 365 days a year by anyone with an interest in audio or not. No golden ears required whatsoever.

Hence there's little interest in blind testing for amplifiers, CD players, cables, DACs etc because they are practically undetectable by the acutest scientific measuring devices, let alone a human ear.

Other mechanical transducers such as turntables, arms, cartridges, and headphones on the other hand do still have easily demonstrable differences in sound quality whether tested blind or sighted.

In time these too will diminish no doubt. Already a good $300 headphone is awfully close to $3000 one. 

There should be no need to be afraid of blind testing unless of course it's true that high end audio all too often likes to dress up in the emperors new clothes.

Can any of my readers imagine a world where we double blind test a T-shirt, or a car?

That's also part of what's going wrong here. :)


At the same time, are some cables and accessories over priced for their technical merit? Sure, but let me show you my watches. My most accurate is a Seiko solar / quartz, but it is by no means the most expensive I own.

Best,
E

Eric: I often see references to high end watches as an analog to hifi. I don’t think it applies for the following reasons.

1) High end watches are a pure luxury item and the manufacturers know that. As you mentioned, $12K Rolex makes no claim to have better accuracy than a $35 Casio. They have other features that make them collectable (in house movements, unique complications, unique style) but they are inferior to the Casio in their core function.

2) A lot of high end audio is also pure luxury. They understand that as well but do not overtly market it as a life style product but make the claim that their product is superior in function to their Casio counterparts.

3) High end watch collectors are usually informed buyers. High end audio buyers, not so much.

Mike

If they are adding 30 lbs of aluminum to a product, it’s overtly marketed as a luxury item. :)

3) High end watch collectors are usually informed buyers. High end audio buyers, not so much.

I call BS. I think it’s the same mix. Those who buy a brand for the brand and those who geek on the movements.

I agree that there is no difference between those that buy an audio product or a watch merely for brand recognition.  My point is that there are folks that buy a $50K amp and then try to tell you how much better it sounds.

The ego of high watch is MUCH higher than those of  audio buyers .At least in audio you can't buy something for 10 bucks  that will sound
just as good .
And you can't walk around all day flashing you 50K amp .

       "sometimes people have ulterior motives, you know, like Boeing"

Geoffkait, 

No Geoff, more like NASA....

I’m not going to defend the ego of the luxury watch buyer. I’m just saying they already know that their fancy watch does not keep better time.

The same cannot be said about the luxury audio buyer who insists his $50K amp (or DAC) sound better than a $10K amp (or DAC).

mr_m1,107 posts03-26-2019 5:57pm "sometimes people have ulterior motives, you know, like Boeing"

Geoffkait,

No Geoff, more like NASA...

>>>>>Really? NASA does their own testing? I doubt it. The problem with Boeing is the FAA alledgedly allowed Boeing to do its own testing in the context of the software package related to the recent 737 crashes. But more to point someone who had it in for product X could publish “test results” that would be unfavorable to product X. Follow?

Eric, well stated, and Yes, you may have created the embers for yet another brush fire. ;)
It seems almost unavoidable of late.
For every stance, you'll have those of a like mind...however apparently unstable for some, but support of a sort.
Then there's those who would rather bury their Game of Thrones Registered Trademarked Battle Axe into your forehead because agreeing to disagree over differences in opinion and/or preference doesn't seem acceptable...or 'something'.
I can appreciate one and all search for their personal idea or ideals of 'perfection'.  I personally can't relate to perfection as existing, or even the opportunity for the event to potentially exist.  If observing an event alters the event (as has been posited), it seems IMHO to be an ongoing activity.
Which it has been, and seems to be riding towards a sunset that doesn't 'fade to black' (except on a personal level and time frame, R.I.P.).
Eric, enjoy.  All of 'Us', enjoy yours. I like mine.  

Re 'Boeing'....I come from a family that, over time, worked for Douglas, McDonnell Douglas, and finally, Boeing.  Over that time, the aircraft have grown in complexity, price, and passenger count per craft.
They added an 'app' that was meant to protect the major investment, help the crew manage the craft, and get the butts in the seats to their gate wherever.
They neglected to make a Big Issue out of it, or where the 'Kill' switch was and When one might want to use it.
One sure bet that someones, somewheres, are scrambling to get this programmed into flight simulators that run the 737 scenarios.
But like the Miracle on the Hudson, you can't Beta test Everything.

NASA, on the other hand, is basically an 'X' program.  High risk.  People Will Die.  They know that upfront.  SpaceX will eventually, too.  If you do something inherently dangerous, it happens regularly.

"Aviation in itself is not inherently dangerous. But to an even greater degree than the sea, it is terribly unforgiving of any carelessness, incapacity or neglect."

As much as I like flying, and appreciate a (IMHO) great or beautiful craft (the de Havilland Mosquito, the Cobalt Co50 Valkyrie for examples), when I board an aircraft I'm putting my life in the hands and tech of every being that has made, cared for, and controls that machine.  And a faith in physics that will make it fly.

But one also delivers oneself into 'situations beyond control', as stated above. Excrement Happens. 

Oh...me?  I wanted to be a bigger cog in a smaller machine.

And I am. *VBS*  Enjoy your niche. ;)

@asvjerry, thanks for your post.
Reminded me, a nervous flyer whose fear grew worse with each flight, of reading about the meticulous no nonsense approach adopted by the aviation industry today. 

Despite years of training, numerous health and psychological surveillance checks the main risk to passengers  comes from the pilots themselves. Not terrorists, weather conditions or mechanical failure.

According to Boeing up to 80% of accidents are caused by human error! And yet, year after year audiophiles remain convinced that their judgement remains infallible.

@asvjerry My point was that the Government ordinarily requires third party testing. The Boeing case is unusual because apparently FAA allowed Boeing to perform the testing for the software in question. If that’s true then it’s like letting the fox protect the chicken coop. My last job was senior engineer in FAA Office of Safety.

^^^ Geoff, was it biplane or single wing aircraft at the time?

I don’t recall. The Office of Safety at that time was located at Kitty Hawk, NC. 

Actually very funny!

It’s FAA humor. 

@erik_squires  I would be happy to have you test your findings our in my system using your wallet

Happy Listening!

Gang,
I got a funny pair of e-mails from someone who did not want to get into an argument, but who then immediately wanted to argue with me! < hahahahah >

I'll keep his name out of here, but I did want to make something clear:

I do think it is appropriate for a manufacturer and researcher to use double blind testing as a manufacturer at times.


I do not care what the results were, except when testing generalized principles. Like, testing around speaker dispersion and what the population at large likes. That's kind of interesting, but the point of my original post is that is irrelevant to what I may buy.  Reams of reviews, double blind tests, and analysis from the NSF are not going to change what speakers or cables I buy.

I will however use NSF ratings for buying water filters!!


Just not really appropriate for why I buy gear.


Best,

E

Which reminds me, Keith Herron wrote or said in an interview some years ago that one thing he discovered in working on his phono stage was people can hear differences of as little as 0.03dB. And no that's not a misprint, its three one-hundredths of a decibel. 

Well I bought his VTPH-2A last year and this was one of the things I asked him about. Yes indeed, 0.03dB, which he knows from double-blind testing.

What he found was he could influence a persons preference by making that small a change to frequency response. I didn't ask if that was measured in the air or calculated from a circuit. Either way, a pretty small change. And double-blind. Heck maybe triple-blind, given that he never seemed to care which way the test went. Just one of many things he did in his incredibly persistent perfectionist approach to product development.

So yes there are indeed perfectly appropriate uses for double-blind testing. 

Hi millercarbon


I believe it, and 0.3 dB puts it in the realm of cables. So all this cable and noise filtering are just fancy tone controls.

I choose to believe just because someone says something doesn’t necessarily mean it’s true. 

Get your headphones and give it a try.  The test ranges from 6 dB to 0.1 dB.  My old ears can't discern anything below 1.0 dB.

https://www.audiocheck.net/blindtests_level.php?lvl=0.5

I don’t see many, or any, agon members blindly following reviewers.

That said, Art Dudley is probably my favorite audio writer just in terms of talent and style, and Michael Fremer is probably my favorite reviewer (of speakers) because Fremer seems to capture in print the character of a speaker so well. (When I’ve heard speakers Fremer has reviewed, they usually sound just as he’d described).

As for blind testing, as I’ve said many times before, no gastopo squad is breaking in to anyone’s home to force blind testing, and no one should bother with it if it doesn’t interest them. I don’t bother blind testing the majority of what I buy. At the same time, I can’t pretend our hobby is magically exempt from all the same variables in perceptual bias that humans suffer from in every other endeavour. I have no problem whatsoever admitting that my perceptions may be in error, for any number of reasons. But it’s damned fun to trade subjective impressions of gear.

prof
As for blind testing, as I’ve said many times before, no gastopo squad is breaking in to anyone’s home to force blind testing, and no one should bother with it if it doesn’t interest them. I don’t bother blind testing the majority of what I buy. At the same time, I can’t pretend our hobby is magically exempt from all the same variables in perceptual bias that humans suffer from in every other endeavour. I have no problem whatsoever admitting that my perceptions may be in error, for any number of reasons. But it’s damned fun to trade subjective impressions of gear.

>>>>>Why pick on perceptual variables? There are too many physical and electrical variables and potential errors in the test system or test procedure to be able to control. No one seems to even know what all the variables are. For that reason alone test results should be suspect. Besides not everyone is impressed by the cost or appearance or advance buzz of audio devices or components. Maybe newbies are, maybe not. Who knows? Did I already mention the ulterior motives some people might have in pushing for blind tests?

In addition, there is no comparison between audio and “other endeavors” - audio is infinitely more complex than say, wine tasting or medical trials. Thus, many more things can go wrong in the test. Audio test results have little or no significance.

Note to self: Blind test proponents apparently can’t help pushing blind tests even while pretending to be nonchalant about it. Must be a new strategy. 😄

Blind testing is not without flaws, but it's easily the best test available to us when comparing products. Easily.

When testing, the weakest link in the entire chain from the recording to our enjoyment of it, the one most subject to variability, is inevitably ourselves.

In an ideal sound test we would not be involved in any of the actual assessment because our impressions are simply too subject to change. Mainly because they all too often depend heavily upon our moods. Heck, on a bad day we might even start believing that the Beatles 2009 remasters were any good. Or one day we prefer Presley's version of Blue Suede Shoes, the next day Perkins etc.

When listening blind some may find that not knowing the make, the price, the technology behind the sound can act as a wonderfully liberating experience. Wonderful because there's only the sound to think about. Ultimately even a blind test will still depend upon the current mood and frame of mind.

Therefore a better way to evaluate tech is by comparing side by side, preferably without sight of it. This at least ensures that our mood preferences will apply equally to the equipment we listen to. [Unless of course we are afflicted with some kind of disorder whereby our moods (preferences/biases) are changing from moment to moment.]

I have rarely had the chance to do an unsighted comparison test but they do tend to be eye opening - sorry. One favourite is comparing different masterings of favourite albums - level matched of course. It's amazing how your mind can desperately scramble around for any data it can find in order to form an impression. Even displayed album artwork can contribute towards the impression formed.

A lesser kind of unsighted test is encountered when you walk into a place where there is music playing but you have no idea what it is being played on. So far I have never walked into a store and found the sound there to be superior to what I enjoy at home. Often it's almost unbearable, but just one of these days, who knows?

I don’t see many, or any, agon members blindly following reviewers.

I don’t either, but I think the repeated controversies over this is notable.

As I’ve said among many issues, from amplifier design to cables, it’s a real shame to me that testing is stuck in the 1970s.

Best,
E