Reviews with all double blind testing?

It is totally beyond me why anyone would have such distrust of what they hear to rely on DBT.

Apparently so. The explanation is simple: If you understand what scientists have learned about human hearing perception over the course of decades, then you will understand why we shouldn't always trust what we hear, and why in these cases listening blind is far more reliable than listening when you know what you're listening to. I suspect that you don't want to understand this, because it will upset the beliefs you've acquired over the years.

Now, there's nothing wrong with not knowing (or not accepting) this. After all, you don't have to understand the principles behind an internal combustion engine to buy a car. And if you can afford a multi-thousand-dollar audio system, it doesn't really matter. You'll probably get good sound regardless.

But if you can't afford that kind of an audio system, it can matter a lot.

Qualia, you say, "Why anyone, whether or not they think DBT is the *final* word, would ignore DBT as a way of determining where to spend their own money (speakers, room treatment first, then other stuff) is beyond me. " It is totally beyond me why anyone would have such distrust of what they hear to rely on DBT. If you wish say that I just choose to dump cash even when there are no differences. Basically, I find DBT invalid and have to otherwise proceed hoping that I can hear a side by side comparison of what I am interested in. On occasion I have been able to bring the desired components into my own home and do a comparison, some times I can rely on the ears of others I trust, one being a reviewer, one a distributor, and one two manufacturers, but most just audiophiles; and sometimes I just take a flyer, such as with the RealityCheck cdr burner. As I have repeatedly said, this is not a matter of rejecting science, it is a matter of rejecting a methodology as it obviously lacks face or conceptual validity. Also as with automobiles and wine, I do not base my buying decisions on double blind tests.

Yes, Qualia, we are asking the same question. It's the same question that subjectivists have been asked for years, and they don't have an answer, so they have to stoop to insulting people's intellectual integrity, as Gregadd has just done yet again. Why do they bother?

Let me try this one last time.
Just because you beleive in DBT/ABX doesn't make you objective. Every so called objectivist uses BDT/ABX to prove what they already beleive. when the test fails to prove what they already belive i.e all amps sound the same except for some easily measurted and compensated for parameters t deem the rewsults statisically insignificant. To me that is bias which is what we are talking about.
DBT has only two purposes, to eliminate perssonal bias and the placebo effect. When you have done that you are just getting started.
The real question is does a the component under consideration simulate real music. As crazy as it may make you only the human ear can evaluate that.

That I never conceded that A/B testing of any kind was signifcant. Buying A because it is better than B is a Madison Avenue trick! I was lucky enough to be taught that. It has saved me a lot noney.

Shadorne, I understand your moderate position. Please take no offense when I simply say that I strongly suspect that DBT is an invalid testing of what people hear. I am not really concerned either that some, myself included, cannot hear differences in the typical same versus different format so commonly used in DBT. People do hear differences when double blind testing is just a which do you prefer of amps A, B, and C. I don't really have much trust in many reviewers and don't need their inability to hear difference in same/different tests to be convinced.

I absolutely concur that we need to be wary of exorbitantly high priced equipment and rave reviews and claims by salesmen and reviewers. But we should equally be ready to hear true quality in some more expensive equipment. Quality parts cost money and research and design work has to be paid for. Too often I have heard some expensive gear that truly is excellent in my opinion and which I remain thrilled with. My Reimyo PAT777 amp and Shindo Labs turntable are but two examples. I also have a relatively inexpensive line stage, phono stage, and universal player that are at least the equals of much more expensive equipment, again in my opinion.

I once heard a $350,000 amp at CES. I listened with no intention of ever buying one. It was the best sounding amp I ever heard. The Stereophile reviewer also loved it, but it measurements looked bad and so they dismissed it. The objectivists ranted that Stereophile should not have even reviewed it. I ranted that they should have heeded their ears rather than their inadequate instruments. I still would not consider buying it largely because I just cannot afford it.

Shadorne:

Another way of making your point is this. Even if ABX tests do not reveal all audible differences, somehow, they do reveal *degrees* of difference. Components that ABX as different, and clearly so, are different to a *greater* degree than components that are indistinguishable under ABX conditions. Therefore, they are more deserving of audiophile evaluation. Likewise, ABX-distinguishable gear that is perceived as clearly better in DBT-ing, is more deserving of audiophile cash than gear that is not perceived as clearly better in DBT-ing.

This is independent of whether or not there is some perceiveable difference between components that are ABX indistinguishable. (Although I still can't understand how that could be.)

Yet, ABX opponents seem to ignore this more modest lesson. They reject ABX as a way of ultimately distinguishing components, and therefore decide it is unworthy as a reviewer tool at all, even in deciding where to drop their cash. Why?

Pabelson: we've said the same thing... what *would* count as a test of audible difference if not dbt? How would we ever know a beetle in the box wouldn't make an audible difference? Or what if we were simply to change the faceplate on an amp. Nothing more. Would that change the audible sound? How would you know? What if the reviewers rave?

Shadorne: I said much the same earlier in this thread. Why anyone, whether or not they think DBT is the *final* word, would ignore DBT as a way of determining where to spend their own money (speakers, room treatment first, then other stuff) is beyond me. In other words, I really don't understand someone who would spend more on power cords, conditioners, and interconnects than speakers, given the DBT results. And there are plenty such people!

Interesting discussion but I think the main point or lesson of DBT tests is being missed through the discussion of semantics and detail.

DBT tests are significant for audiophiles because they show that, for the handful of those people and equipment tested, differences cannot be audibly detected between some types of equipment, at least not easily, even when people are trying. DBT's show that it is typically hard to discern differences with good quality SS amps, CD players and a variety of quality cables.

This raises questions about reviewers abilities to hear strong differences between certain types of equipment, but it does not conclusively prove that they cannot hear these differences (unless they were DBT tested with the very equipment under review).

It also suggests that some types of equipment are either less crticial (cables) or of universally engineered high quality (many amps and CD players); in these cases, different equipment choices are unlikely to make a significant difference to the sound that is heard. DBT tests also tell audiophiles that some things are a big factor...speakers, speaker placement and tube vs SS amps, for example. In essence, DBT's confirm what is probably a gut feel for any serious listener who has played around with a variety of equipment over the years.

However, it remains possible that some people out there might be able to hear a difference....for differences there surely are...however minute and undetectable to those tested so far. So those with "golden ears" can keep on searching for nirvana, everywhere and anywhere, including but not limited to fancy power line conditioners and other tweaks.

If DBT test result reports do not conclusively prove that a cheap amp is just as good as a very expensive one to your own ears...then exactly what use are they????

I suggest they simply offer some guidance as to where an audophile might spend a higher proportion of their effort/money in improving the sound of their system. They also suggest that the "emperor may have no clothes" in some cases....so, for certain types of equipment, be a little wary of exhorbitant prices and rave claims by audio salesman/reviewers!

As I have said at least 5 times, your statement that, "ABX test, generally regarded in scientific circles as the gold standard for determining audible differences" is not true. But neither of us will ever convince the other, so why don't we just drop it. I can accept your statement that those advocating it would not be numerous enough to justify a magazine so it is a moot point.

The standoff between Pabelson and Tbg reminds me of the stalemate between the external-world skeptic and the dogmatist.

Skeptic: You don't know that you're not a brain in a vat of nutrients, being stimulated by a computer simulation, carefully monitored by a team of scientists, to think you're in a real, concrete world... the world you *think* you're in. Since you don't know you're not a brain-in-a-vat, you don't know anything mundane about the external world, e.g., that you have two hands.

Dogmatist: I know I have two hands! If I know I have two hands, then I know I am not a handless brain-in-a-vat. Therefore, I know I am not a handless brain-in-a-vat.

One man's modus ponens is another man's modus tollens, as the saying goes.

(For non-logicians, modus ponens is: If P then Q. P. Therefore Q. Modus tollens is If P then Q. Not-Q. Therefore, not-P.)

Pabelson: DBT shows no audible difference between cables, therefore there is no audible difference.

Tbg: There is an audible difference between cables, therefore, DBT is flawed.

Logic alone (formal logic) cannot settle the dispute, any more than logic can settle the skeptic/dogmatist dispute.

But in this case, it's odd to think of Tbg's favored cables being a/b'ed with cheapos, without his being able to tell the difference, and then, only when told the true identity of the cables, his insistence that there *is* a perceivable difference. Very odd.

Here's a question for the doubters of DBT-ing. Given that there are perceptual biases at work (expectation, confirmation, endowment effect, etc.) how would one test for such biases? That is, what *would* count as two components sounding the same?

Suppose you have two amps that are identical except one of them has a beetle put inside and the beetle runs around, I don't know, defecating in there. And then reviewers praise the beetle effect: "Widened the soundstage by meters! You don't need golden ears to hear this one!" How would you go about evaluating the beetle effect?

TBG: Two amps that reproduce music differently enough to be heard will NOT sound identical in a DBT. But how do we know that two amps reproduce music differently? You say they do, but how do we know you are right?

Let me pose the question a bit differently. Here we have two amps that are not distinguishable in a level-matched, quick-switching ABX test, generally regarded in scientific circles as the gold standard for determining audible differences. A subjectivist claim that these two amps reproduce music differently. How would he prove that they do? Whatever he does, he has to use a blind test, because a sighted test can prove nothing about audibility. That's settled science. So what kind of test should he use?

Gregadd: If you were talking about the response of a single objectivist, you should have named him right up front. Instead, you tagged all objectivists as dishonest, based on your interactions with one man. That's an understandable error, but it's an error.

There have been dozens of published DBTs of amps. Some have been positive, some have not. Fremer claims to have done a positive test. So what? He ain't the first, and won't be the last.

I'll pose to you the question I've posed to others: Shouldn't a reviewer, before he reviews an amp, confirm that he really can hear a difference between this amp and his reference amp when he doesn't know which is playing? Ever wonder why none of them do this?

Pabelson, you owe us an explanation of how two amps that replicate music differently can sound identical in the restrictions of DBTesting.

As they say in the neighborhood where I grew up..."it's on now."
Pay attention Pabelson. I did not give a history of DBT which I concede can be a useful tool in revealing prejudices but ignores the real issue. I gave a history of the ABX Comparator. In fact DBT and ABX arrived on the Audio press simultaneously.
You are quite correct Mr. Fremer's results are in fact not a mystery. He proved he could hear the difference.
If you want to know my principal source for the history of ABX it was primarily the Audio Critic written and published by one of the most serious advocates of DBT/ABX Peter Aczel.
Both Mr. Fremmer and myself suffered a personal attack in the letters column. It appaears Mr. Fremmer was not qualified to criticize DBT/ABX because of his sloppy wiritng style. I of course was unworhty because I was a DC trial lawyer.
Like you, when confronted with Mr. Fremers' test results Mr. Aczel refused to acknowledge that Fremer could match A&B to X in a test designed by others, where frequency reponse distortion , output levels,etc. were all accounted for. He tried to wiggle his way out but was unable.
I of course dared to challenge Mr. Aczel to put his money where his mouth was. He was using a top of the line Boulder amp. I offered to trade him his Boulder for the generic Radio Shack amp of his own choice with an equlizer so he could compensate for any frequency reponse violations. In fact I told him he could keep his amp and I would give him the Radio Shack amp of his choice if he promised to use it as a primary reference.
His response was that I knew he could not accept the offer becasue he needed a top qulity amp for his test. He then accused me of conducting a cheap trick. He said this trick may work on a DC jury but not on him. This transpired in the late'70's or early '80's. Harry Pearson published the first letter in the absolute sound.
Pableson while I may make mistakes, I don't indulge in falsehoods.

Suffice it to say that Gregadd's history of DBTs is almost entirely false. They were in use long before the invention of the ABX Comparator, which was just a convenient tool to do what we already knew how to do. If you go back and look at what 'reviewers like Fremer" were able to distinguish, you would find no mystery at all.

TBG owes us an explanation of how two things that sound identical can replicate music differently.

Hi Pabelson,

"But they've also discovered that, once you've learned those differences, the best way to confirm that those differences are really there is through short-term listening tests that allow you to switch quickly between the two components."

Any neuroscientist who would claim he/she discovered "the best way" to confirm differences would not be very credible with me on at least two counts. First, it is the "best" amongst which collection of methods? Have ALL POSSIBLE methods been tested? Perhaps some heretofore untested method could be even better. So, the scientist overstated the result. Although such hyping occurs, it is hardly scientific. It would also lead me to question if the scientist's methods also lacked precision and other high scientific standards.

Second, to determine that this method is the "best", it must be different from the rest. But how can the neuroscientist determine this difference? By DBT, the "best" method that determines differences??? But then the neuroscientist will be using the very method he/she is attempting to validate. In other words, the neuroscientist would hang himself/herself in a logical loop of circular reasoning.

Your statement appears to be based, at least in part, on faith in neuroscience and psycho-acoustics. These are important sciences but they are not hard sciences like physics and chemistry. Compared to physics, they are sciences in infancy. Their levels of rigor, accuracy, predictability, and reliability are not yet in the same league as those for physics and chemistry. So, my level of confidence in them is not as great as what yours appears to be in your posts. It's the complexity.

The complex substratum involved in auditory perception is not yet sufficiently understood to shed light on the finer aspects. A large number of neurons form millions of possible pathways that a particular "encoded song" can travel in our brains to yield the perception of its sound and our reaction to it. The same song or piece of music produced by the same audio system a few moments later may not travel the exact same pathways in our brain and hence may produce a different experience. This variability is compounded by the non-constant chemical environment that influences our experience. (For example, the amount of endorphins available at any one time.) Emotional changes, expectations, suggestions, levels of alertness, fleeting nature of memory, etc. add to the variability. Also, the brain circuitry is not as rigidly set as it once was thought to be. It can change with experience and learning. At the current state of neuroscience, there is insufficient organization, understanding and integration of this variabile milieu to shed light on the finer issues about DBT. That may be reason enough for some opponents of DBT to claim that "to DBT or not to DBT" is an irrelevant question. I, for one, am in favor of rigorous DBT and would find the positive results useful but the negative results inconclusive for reasons given in my previous post.

Best Regards,
John

Perhaps because the neuroscientist/psychoacousticians don't intend their testing to deal with what is most accurattely replicates music as the experimental context necessitates tight and brief controls.

my new policy on audiogon is to post my opinion and let it stand on it's own merit. I no longer feel the need to respond to every competing opinion. I'll let the readers make thier own conclusion. I do however reserve the right to respond crticism directed at me.

My approval of DBT is in no way an endorsement of the ABX test.

Just because you beleive in DBT or ABX testing does not make you an objectiive. DBT proponents have yet to show me where they have used it to advance the state of the art. They are as biased as anyone else. In fact the inventer of the ABX gave this as his reason for inventing the ABX box-he was upset that audio companies could be destroyed by audio reviewers who did not know what they are talking about. Thus DBT/ABX was invented to attack the integrity of audio reviewers. Not as an objective scientific tool.

The intial tests where short term on inexperienced listeners. That is a fact. To furhter demonstrate thier lack of objectivity it is the proponents of DBT/ABX, when confronted with the fact that reviewers like Michael Fremer were in fact able to match A&B to X, attacked the validity of thier own test. In effect they concluded that because they knew there was no difference between amps he must be using some trick formulated by his knowledge of the amplifiers under test.
No one argues that amps and cables sound the same. Nothing is further from the truth. That is exactly what they argue calling it snake oil and making vial insults to those who design, sell, buy and review it.

Feel free to remain wedded to frequency response, distortion figures and output impedance if you like. You don't need a blind test for that because it is so easily measured. You may clean your palette with the occaisional blind test. Ultimately you are going to have to listen. This is what all the manufactures of good equipment do.

Let's not be so selective about what neuroscience has been discovered, John. It, along with psychoacoustics, has indeed discovered that it can take time to learn the sound of something, and the difference in sound between things. But they've also discovered that, once you've learned those differences, the best way to confirm that those differences are really there is through short-term listening tests that allow you to switch quickly between the two components. So why is it that a reviewer, who supposedly spends weeks "getting to know" a component, and who also owns a reference component which he also knows well, can't hear a difference between the two in such a test?

My point about how we change was aimed at the reviewer who reports differences between the component under review and something else he may have heard months before, but doesn't have now. He's claiming to do something that your neuroscientist/psychoacoustician has found to be impossible.

With apologies to Shakespeare and all logicians:
"To DBT or not to DBT is or is not the question."

Hi Pabelson,

Your quote points us to the central point of this discussion:
"Yes, beauty can grow on you. But notice that it's not the lady who's changing. It's you. What does that tell us about long-term comparisons?"

It tells us what neuroscience has discovered. The brain is much more plastic than once believed. It is not static like electronic circuits. The brain circuitry and its chemistry change. New interneuronal connections are formed and concentrations of neurotransmitters and other brain chemicals change. So, what the brain could not distinguish one day, it may LEARN to distinguish in subsequent exposures to the experience. We have experienced this learning phenomenon as students, as professors, and as audiophiles. This is part of our growth and evolution. A double-blind test based on short-term listening sessions may not allow enough time for the brain circuitry and chemistry to reconfigure itself to discern the difference. Therefore, if a short-term double-blind test does not show a difference between two amps, it would not be correct to conclude that there was no difference between the amps, only that that particular test did not reveal a statistically significant difference. A double-blind test showing a positive difference may be useful for audiophiles, while the test showing no difference is an inconclusive statement about the amps.

Incorrect interpretations can also be made for long-term double-blind tests. History of science shows us that even the hard sciences like physics are not immune from making incorrect interpretations. A committment to truth and critical thinking helps purify science to better the human condition. Otherwise, our implicit assumptions may yield tautological statements similar to the very first statement in this post. Although it is logically valid, it does not contain useful information for the audiophiles.

Best Regards,
John

Pabelson,

Thanks for your posts. Although I have never done blind tests, my experience with audio equipment matches the DBT reports perfectly. I simply can't detect audible differences between different cables, CD players and solid state amps provided, of course, they are of a minimum quality.

I do, however, detect large audible differences between different speakers, their placement and the room in which they are played. I also notice large differences in low powered SS amps versus high powered SS amps but only when driven at a high demanding levels.

On my Anthem AVM 20 preamp/processor I am able to play music either direct anaolg or through the A to D and then D to A circuit....a kind of test....again I hear no difference as long as all the tone controls and settings are the same. So I typically use the A to D and D to A in the circuit becuase it allows me to use base management for my subwoofer...something that does need adjustment (for room) and makes a tremendous difference when adjusted to my taste (digital filters providing great flexibilty). Furthermore, I can use the digital out or the analog out from my various CD players and again I hear no audible difference provided tone control/volume is the same...although I will admit that the digital output can have less background hiss S/N at extremely high levels, but this is not always audible either (setup dependent).

What does this mean for me when buying equipent?

1) Choose CD's and SS Amps for their features (speaker protection, tone controls, souund processing capability, warrenty etc.) For example, I like CD changers as I don't have to mess with CD cases. As for amps, I prefer overly powerful SS amps with significant headroom in order to minimize distortion from clipping.
2) Spend most or your money on the speakers, as this is the single biggest variable and adds the most distortion in the whole setup.
3) Buy good quality (shielded) but not exhorbitantly priced cables and inteconnects.
4) Pay attention to which room is used for playback and what are the wall/floor coverings and speaker placement.

Yes, beauty can grow on you. But notice that it's not the lady who's changing. It's you. What does that tell us about long-term comparisons?

TBG: Yes, there are far too few objectivists to make a market. That's why the largest-selling magazine in the US that reviews audio equipment is that subjectivist redoubt . . . Sound & Vision.

Pabelson, no I have no interest in whether a reviewer can hear a difference between components using DBT using the usual same/different format. Since I don't think it is valid, I would rather continue my present procedure of find whose reviews prove on target in my estimation. Frankly I don't think there are enough DBT proponents out there to make a magazine using them viable.

Not arguing reviewers are perfect either...far from it...they are inherently subjective and bound by their own experiences and equip as we all are...but at least he or she can frame their observations in context which allows me as a potential buyer what to look for, what to investigate, what things I may need to consider etc.

As part of that context, wouldn't you like to know whether this reviewer can actually hear a difference between the product he is reviewing and some reference? And if he can't, what does that tell you about his review?

Wattsboss:

I *do* like your analogy of the beauty that grows on you. I've had that experience, as well as its opposite -- the superficial beauty that fades quickly (or immediately upon conquest). True. Typically, it's because facial expressions take on a representational character; they come to stand for the moods and traits of the person. And in the case of a good-to-the-core person, that goodness starts to shine through. In the hot-bitchy type I usually go for, the nastiness gets associated with what I previously thought was cute.

Anyway, it may be that the beauty of an audio system takes time to appreciate fully. But distinguishing between looks doesn't take time, even if the full evaluation of those looks does. Maybe the analogy here is identical twins who no one can tell apart initially, but whose family and close friends can... immediately.

After all, I'm not sure I could distinguish the sounds of two violins immediately, in the hands of a skilled violinist. Each violin makes a wide range of sounds, and I'm not sure what's due to the violin and what's due to the violinist. Yet one violin might be $1K and the other $10K, because violinists themselves can immediately hear the difference. Maybe it's like this with audio. But I have no reason to think so, given the studies I've read, in which audiophiles who are familiar with the equipment, do no better than non-audiophiles (who are also familiar with the equipment).

Also: there is a long-term in-home disguised cable experiment going on right now. It has a few more months. We'll see how that goes.

Pabelson, thanks for yr kind words, but the quotes you make refer to another poster -- or are they there to illustrate yr previous points?

Qualia sez:

other things being equal, a beautiful soprano actually *sounds* better in the typical soprano role

Good point! Matter of fact, I read s/where that a nice-looking piece of equip was invariably "heard" to sound "better" than itself unsighted. Amazing!

the illusion is the false reality. Kinda by definition

That is correct, semantically. It's also kinda philosophical.
IMO we should distinguish between semantics and philosophical extrapolations and the simple PRACTICAL application of DBT in our (restricted) context.

BTW, I also suggest that certains things CAN be indicative of performance or INFLUENCE things, in OUR context, such as:
*measurements -- as long as we measure what correlates to what we're looking for (i.e. we would have to determine in advance which measurement indicates what aspect, in terms of perceived sound; little has been done there)
* wires for example -- because they link two electrical circuits, active / passive & combinations thereof
* active components: their circuit design, power supplies, input & output stages, components used... influence the distortion levels AND how well these components interact with the load. Change the load (what the output stage "sees") and things change electrically; if we change something in the system, we've modified the system "circuit" fer pete's sake. Things may also change in the audible range...

...etc.

So, maybe we are discussing whether it's worth setting up dbt to help notice differences in the audible spectrum?
Or whether perhaps dbt is not the most efficient/reliable method of doing so in this particular context?
Or, perhaps, discussion is a way of communicating -- a marvellous, human activity that we all need. And the subject of dbt allows us to do just that -- so what we really want to do is to talk regardless and dbt offers us just that opportunity, whether it is or isn't panacea.
I go for the latter -- my take of course! Cheers:)

With regard to the DBT on two amps that were indistinguishable in a DBT...even if I accept the results as is, and not impose my own sense that "I could have heard the difference unlike those participating" etc and even if I accept the results as valid, that yes one could not tell the difference....still doesn't tell me much: this is because all it has shown me that these two amps are indistinguisable in a particular system set up (especially speakers!), in a particular room, etc...so adding that piece of information a review for example doesn't help me unless I have the exact same set up sans the amps. Not arguing reviewers are perfect either...far from it...they are inherently subjective and bound by their own experiences and equip as we all are...but at least he or she can frame their observations in context which allows me as a potential buyer what to look for, what to investigate, what things I may need to consider etc.

Put it another way, double blind testing cannot work IMO in audio precisely because there is no "absolute" definition on "what sounds better" to begin with (with apologies to HP). How do we define it the first place? Measurements are fine but in only in terms of allowing us figure out what aspects of sound (jitter, frequency range, channel separation etc etc)that contribute to what we are hearing...but in the end its a VALUE judgement. Some like tubes, some like SS. Some need gigantic floorstanders for frequency extension, some prefer mini-monitor accuracy & imaging. Some prefer transparecy, air etc others image solidity, warmth etc. The evaluators who comment after a DBT are including their own subjective value judgements as well. You cannot answer a question that is ill defined in the first place. In this light someone else perhaps half joking said what about cars, wine, literature whatever....and in a way he is right on the money: in all of these pursuits value judgements occur.

If we are to just focus aspects of sound and have no value judgement...i.e just want to know say the level of jitter of one against the other etc...then we can just measure for the most part, no need for DBT again.

The one aspect I think DBT can be used perhaps is not comparing one brand vs another, but within brand and the same model: for example you have the exact same system set up, and then test a new model: one has more jitter than the other...then one has upsampling switched on, the other doesn't etc etc...by the manufacturers....this way one may try to investigate what matters most, a priority schematic, if you will one, to audiophiles or the public at large and then devise products or even array of options on the same product to maximize revenue or provide tailor-made solutions for various segements. Obviously this is more of marketing strategy than an answer to the "holy grail".

Sooner or later someone is gonna start advocating db testing for cars...yikes!

Qualia8, I think the blind testing of women orchestra players has been a wonderful thing. So I'm cool with that and think it's a wonderful breakthrough for women musicians and for fairness.
But audio testing isn't single-player auditioning is it? ... and probably another way of auditioning that would be great would be to have auditonist X blind play along with the rest of her section or the rest of the orchestra! ... I'm willing to say that blind auditioning is a good way of rooting out sexism in big-city orchestra hiring, but I'm not willing to conclude that this addressese the problems with audio auditioning.
The problem again is synergy--and time and acquired taste and listener quirkiness.
You write, "So, if two amps cannot be distinguished unless you're looking at the faceplates, why buy the more expensive one? Now who finds fault with that reasoning?"
To which I would say you would need to test the amplifiers in question with at least 3 different types of speakers, including speakers with different sensitivity levels, different ohlm levels before you can go anywhere near calling it a valid test. Beyond that, I know that my listening preferences, food preferences, beer preferences, beauty preferences are not static and frozen. I have a close friend who is not drop-dead gorgeous on a first look, but keep looking at her, and over time, you just are drawn more and more to her face. It's a beauty that takes time to emerge and when it hits you, you're deeply enthralled because you keep searching and studying her beauty. Yet, if you put her picture in a mag, maybe I wouldn't pick it out. This woman's beauty increases over time, and this is different from the "now that I'm with her/him, of course, he/she's good looking effect."
So the point: should we we blind audition for 1 month? 3 months? 6 months? .....time is critical here.
Ultimately, I reject the idea of auditioning a single component other than a source component. Speakers and amplifiers have to be auditioned as a team. And teams, we all know, often combine in ways that are more than the sum of their parts or less than the sum of their parts.
to Jeff Jones, hey, it's OK if someone wants to test something; it's just let's be honest about the very real limits of the tests! ... This reminds me of modern presidential polling ... pollsters call 1,000 people around the USA and get hangups, etc., and eventually come up with a number of people voting for Bush vs. Kerry. And the polls have a margin of error of 3 to 5 percentage points .... So the poll one day says Bush 50, Kerry 50; this really could mean (figuring in the margin of error) Bush 55, Kerry 45 or Kerry 55, Bush 45. The poll is basically useless. But the way out of this is that pollsters take samples every day as the campaign advances and there are so many different polling agencies ... and so it's only under conditions of repeated poll taking and repeated polltaking by multiple and antogonistic entities that gives us any confidence that yes, the 2004 Presidential race is neck and neck. When we see polls over 10 straight days, taken by 10 different entities, clustering around Bush 50, Kerry 50, and clustering over a period of weeks, ONLY then can we have some confidence. But even so, voter turnout is always the X factor, and most pollsters admit that's the one they can never nail down. A higher than usual turnout among Democrats or Republicans or evangelicals or blacks or whomever will make the poll results pretty much invalid.
I just don't think the tests can get us the information we're looking for--without doing something like the equivalent of daily tracking by multiple entities ... And by the way, in polling, all these entities have an incentive to get the numbers right because they will make more money and win more acclaim. The polls by the candidates themselves have this incentive in a big way. But where's the equivalent incentive for audio?
I frankly celebrate the wonderful elusive complicated complexity and quirkiness of listening to audio.
I'm quite willing to enjoy that.

It is a well structured experiment that differs greatly from what we normally hear and how we hear it.

This is just an astounding statement. How in the world can simply not telling someone what they are listening to affect what they *hear*? I'll grant you, it can certainly affect what they think about what they hear, but that's just the point. What they think is a function of things besides what they hear, and DBTs isolate the non-sonic effects.

Note that there's no necessary contradiction between these two statements:
1) Harry can't hear a difference between A and B.
2) Harry prefers A to B.
Both can be true. All it means is that Harry prefers A to B for some reason other than its sound (even if he thinks the sound is the reason).

Pabelson, why do you willfully ignore the truth? It is only your CLAIM that DBT gives the reality, when you say "DBT--because it usefully separates reality from illusion" Certainly you don't claim that DBTesting is isomorphic to reality. It is a well structured experiment that differs greatly from what we normally hear and how we hear it. I would say that DBT is an illusion of reality and that reality would be found in the amp that most preferred, especially were personal ownership and manufacturer hidden.

I suspect that this discussion has gone as far as it can. You insist that double blind same/different testing is valid and I say it is not because it is an invalid assessment of people's hearing differences and saying what they like. I am not saying I like what I like and I reject it any more than you are saying I know there are no differences among amps, etc. and therefore anything that shows otherwise is not science as represented by DBT.

Greg: Your generally thoughtful and balanced letter was, in my opinion, a little too balanced. Here's where you went astray:

The proponents of dbt...want to engage in very short tests conducted by the uninitiated. Most proponents of dbt use it to try and prove what they already have concluded.eg cables and amps all sound the same.

This reflects a basic misunderstanding. Objectivists don't want short tests, we want good tests. (All the research suggests that short tests are in fact better tests, but DBTs can be any duration you want). And a requirement of good DBTs is that you provide the subjects with adequate "training," meaning that they are familiar with the sound of the equipment they are comparing. The "uninitiated" make very poor test subjects.

Finally, no one argues that all cables and amps sound the same, and that's not the purpose of DBTs. The purpose of DBTs is to determine *which* components sound the same, and which do not.

How about a dbt between vinyl and digital. Or electrostatic and dynamic speakers- tubes and solid state.

All of these have been done, at one time or another. Vinyl and digital are easily distinguishable--unless the digital is a direct copy of the vinyl. Speakers are always distinguishable in DBTs. Tube and solid state amps are often but not always distinguishable. When they are distinguishable, it's usually because the tube amp is underpowered and clipping (though very mellifluously, as tubes are wont to do!), or because the output impedance of the amp is interacting with cable and speaker to produce frequency response errors.

WARNING: LONG POST -- LIFE HISTORY AND ITS ILLUSTRATION OF BIASES -- YOU MAY WANT TO SKIP

I went through grad school with a $150 boombox. As a classical music lover, I obviously wasn't happy with it, but what was I going to do? Sell my '87 Buick and walk? Audio was not *that* important to me. So it wasn't until I got a job that I decided to invest a little something in a decent 'stereo'. Still, I was married and my wife was in law school, racking up debts. Not knowing anything about hifi, I decided to get a simple home theatre setup. I went to Best Buy, dropped three hundred on a Yamaha receiver, and another couple hundred on a 5.1 speaker set-up. ($100 off b/c I bought the two together.) Some cheap cables, and I was headed home to set up my new rig. Hooray! And man, this thing came with a sub!

You know what happened, of course. The system actually did pretty well with movies. I don't care all that much about HT being perfect. Eminem's 'Eight Mile' was the first movie I watched with the new setup, and it ROCKED! Highs were crystal clear. Very sharp. And the bass, or actually, it was mid-bass, b/c that sub doesn't go too low, was nice and full in my apartment. Gladiator was great too. Cool.

Then I popped in my cds. I wanted *so* badly to like what I heard. After all, my wife was already pissed that I had spent $500. "$500? And you don't like it? What's wrong with you? If you're going to be so picky, you should have gone to law school rather than taking forever to write a dissertation." (I still wasn't done at the time.) She didn't even know about the extra 100 I had spent on a Monster surge protector and cables.

But it sounded terrible. Mid-range just sucked. There's no other way to say it. And treble portions were highly highly annoying.

I stuck with the system for the next year or so. After a separation from my wife, I did what any lover of sound would do, and finally allowed myself in the local hifi shop. (It was only two blocks from my apartment.) I walked in with the idea of purchasing new monitors for the fronts and leaving the rest of the 5.1 system in place. Explaning my situation, the staff (quite helpful, really) suggested Paradigm monitor bookshelves. They were a few hundred bucks and sounded great in the store. There it was -- lifelike voices, not the tinny, metalic sounds I heard at home. Ahhhh!!!

Before I left with the Monitors, one of the sales guys said he had a pair of Studio v.3's I should listen to before making a purchase. Well... I listened, and *wow*. Incredibly accurate sound. It was nothing you could hear with any combination of Best Buy equipment. I bought the speakers on a pretty hefty discount, with stands, and charged home to listen.

The improvement *was* dramatic, don't get me wrong, but still not anything like I heard in the store. Hmmm... Could it be the other stuff in my system? Nah... Cd players are all the same. And amps too. The Yamaha was rated *way* above the requirements for my new Paradigms. And so what if my source was an old dual VHS/dvd player? Bits is bits, right? So it must be my room.

I spent another several months trying to like the sound. Very quickly, I discovered that speaker positioning mattered, and room treatment too. I made a lot of adjustments, but my sound was never *smooth*, as it was in the store. Hmmm...

About this time, I started researching audio. I was relieved to find out people liked my Paradigms for a "budget" speaker. "Budget? Seriously?" I thought. But everyone seemed to think that source and amplification were also important. And there was this thing called a "preamp".

I went back to the audio store and tried some better receivers -- Pioneers with room eq -- but the sound still wasn't to my liking. Sure it was loud, dynamic, and even full. But it left me cold. I pointed to some shiny gear across the room. "What about that?"

"Oh, you don't want that. It's just two-channel. You want home theatre, right?"

"Well yeah, but first and foremost, I want something that sounds good."

So he played me a Musical Fidelity integrated and cd player (around $1,500 each), with my Paradigms. Unbelievable. I just sat and listened for about two hours, entranced, letting the music work its magic on me.

I couldn't afford the MF, but there was a demo Classe, which sounded very similar in the store, only significantly less cash. I brought that home and auditioned it. Definite improvement on the Yamaha, or so I thought.

I bought it and sold the sub + sats on Ebay. Now I was *there*, right? No. I still got annoyed. But closer. Definitely closer.

Anyway, about this time, I discovered Audiogon. I also started talking to my brother-in-law, who had tried dozens and dozens of combinations of amps and speakers to get vocal music right. I realized I was only at the beginning. I was just started on the audio path. Damn. I thought I could just walk into Best Buy, walk out, and be done with it. I had no idea this would be a hobby, and a long-lasting costly hobby at that.

Anyway, I still have the Classe. And now I wonder whether it actually sounds any better than my old Yamaha. Even if it doesn't, objectively speaking, I think it does, subjectively. Because it's a really pretty amp. It has this super-heavy milled steel remote and the display, volume knob, and everything, just ooze quality. (Ok, the outputs don't. They seem cheap.) I can't help but look at my setup when I listen, and I much prefer looking at the Classe.

Maria Callas had a magnificent voice, but she was also hot, and I'm sure that added to the experience of opera-goers of the time. Speaking for myself, I prefer a grotesquely fat and ugly soprano who sounds good to a waifish beauty who sounds strained, BUT, other things being equal, a beautiful soprano actually *sounds* better in the typical soprano role. I once saw Angelina Gheorghiu in the role of Mikaela in Carmen at the Met. Gorgeous coloratura soprano, but also, she was beautiful, at least from the cheap seats where I sit. Took the breath out of my chest. I bet Gheorghiu wouldn't prove that much better than her fatter and uglier peers in blind comparison. But at the opera, you ain't blindfolded.

Maybe what happened when I looked across that showroom and spotted the shiny MF gear was just love. Just as hunger is the best sauce, love makes things sound better. A *lot* better.

I sent the following letter to Sterophile:

Those of us who have been audiophiles for a long time(20+ years) have chronicled the progress of audio components. We have gone down the wrong road too many times to count. Either led there by others or led by our own ignorance and prejudice. We need to remember that while for the consumer this is a hobby for the producers it is a business. Producers must make a profit or die.The road to hi-fi perfection is littered with excellent products whose producers did not pay attention to normal business practices.
Audio producers have to fight for market share like anyone else. The best way to get market share is through aggressive advertising. Build a better mouse trap and they will come to you. Design a slick ad campaign and they will also come to you. There is a problem however. The audio reviewer. If done correctly, it seeks to pick the best mouse trap and debunk the advertising myths. Like manufactures the magazine is also a business and it must make a profit or die. Even worse it's profits come primarily form the very producers it seek to evaluate. A canceled subscription hardly competes with a canceled ad. This puts an ethical strain own the most principled reviewer.
Audiophiles aren't stupid. This a hobby. We are not just interested in good music. We like our components to come in beautiful packages, exclusivity, etc. Just because we purchased something unnecessary does not mean we were tricked. I am sure the piano black finish on my turntable has nothing to do with the sound. Does that mean I was tricked?
From a consumer standpoint if a manufacturer claims that his product sounds better or different it is the reviewers job to evaluate the manufactures claim.
Most reviewers want the manufacturers claim to be true. The reasons are obvious. They recommend a better product to their readers, the state of the art is advanced and the manufacturer can buy ads. Negative reviews save their readers money and nudge producers in the right direction and establish their credibility.
Ironically everyone can't be right. Being right or wrong has serious financial consequences for all involved. Reviewers have been wrong. Manufacturers have been wrong and sadly some have tried to rig the process. More often than not mistakes are based on ignorance and prejudice. Ignorance must be cured by the ignorant, and corruption should be prosecuted. Maybe we can do something about prejudice?

If we did not know what product is being tested we could at least eliminate our personal prejudice. Nothing wrong with double blind testing(dbt) per se. The proponents of dbt bring their own prejudices to the table. They want to engage in very short tests conducted by the uninitiated. Most proponents of dbt use it to try and prove what they already have concluded.eg cables and amps all sound the same. And that expensive products are just a rip off. How about a dbt between vinyl and digital. Or electrostatic and dynamic speakers- tubes and solid state.
The opponents of dbt are also somewhat disingenuous . I do not need dbt because I am not prejudiced. It is the nature of prejudice to not be aware of it.. The person who is prejudiced just thinks he is right.
The design of components is a mixture of art and science. The reviewers job is almost all art Some things just don't lend it self to scientific testing. Could you have a dbt of who is the most beautiful women or what piece of music is the most soothing.
Alas dbt does not even approach the real question. What difference does it make to me whether a &b sound different or the same. My point is which one more closley approximates the illusion of real music for me. Hasn't that always been the goal for audiophiles.
I have always stated that dbt is more a test of sonic memory, the better your memory the better the better you will test.
If you read this carefully you are going to be surprised by my conclusion. Everyone who is involved in audio design or review should from time to time engage in some sort of dbt testing! Your goal should be to determine which product sounds more like music. You may discover biases you did not know you have. Take your time. Make your dbt as much like your regular evaluation process. I think you will benefit from it.
Reginald G. Addison
rgregadd@aol.com
Forestville, Maryland

My evidence for what? That illusion is a false reality? You need evidence for that? Look in a dictionary.

Where is your evidence? Perhaps by your definition which is not widely shared.

No, TBG, the illusion is the false reality. Kinda by definition.

Gregm, I think you are absolutely right that to many of us want others to bless our choices, be it for wine, women, or audio. I stopped going to audio society meetings in New York as too much of the conversations were "mine is bigger than yours" conversations. Having discussion groups on the internet is no different.

My objection to those advocating DBTesting is that they want to use a questionable methodology to say in effect "mine is every bit as good as yours and I paid less." Science does not condone their saying this and I don't really care whether it does or not.

Pabelson, you say, "DBT--because it usefully separates reality from illusion." My only real question is whether the "reality" is a false reality. One that we don't hear when listening. This is why I suggest it is invalid and does not merit acceptance of the findings.

DBT assumes that we have to justify our purchases to others as in science; we do not have to do so

Actually that's an interesting take.
Yet, there's a lot of emotion and correspondingly little logic in the vehement assertions contained in many posts.
Amazing, isn't it. We WANT others to bless our choices after all -- AND, if it's an EE (i.e. scientist) so much the better: science is irrefutable:)

I am afraid your argument that DBT proves humans cannot hear the minor differences runs counter to most people's experiences.

Granted, but why should we assume that the scientists are all wrong and people's observations are right? Surely you know that our perceptions can fool us. Think of optical illusions. Well, there are also such things as aural illusions. One of the most basic is this: When you hear two sounds, you often think they are different, even when they are exactly the same.

So when you say, I hear a difference between this cable and that one, is there a real difference, or is it just an aural illusion? We don't know. That's why scientists developed the forced-choice DBT--because it usefully separates reality from illusion. The only controversy here comes from people who don't want to look at the evidence.

Sorry, Pabelson, I don't think an appeal to the acceptance of a method used in perceptual psychology demonstrates no differences. When there is controversy over a finding, which demonstrably there is, something other than same/different DBTesting would be needed unless those of you persisting in advocating DBT wish to continue to be ignored. I am afraid your argument that DBT proves humans cannot hear the minor differences runs counter to most people's experiences. As I said before, buying decisions don't hinge on scientific proof, and it is an interesting question why some seem so committed to the belief that audio is all snake oil. Perhaps a psychologist should look into that phenomenon.

Qualia: It reminds me of a trick John Dunlavy used to play on visitors to his speaker factory. He would show them an expensive cable (maybe even his own!) and zipcord, and let them audition both. They'd rave about the pricey one, of course. What he wouldn't tell them is that he never changed the cable. They were listening to zipcord the whole time.

One possible weakness of your experiment is that it assumes we know what it is that's tricking us--the price, the looks, etc. But it could be anything (the brand name, perhaps). Also, the value of a perception experiment is somewhat compromised when you intentionally mislead the subject.

There's a much easier way to get over the blindness objection, or at least most of it. In a standard ABX test, you can actually see both cables, and you know which one is A and which one is B. The only thing that's "blind" is the identity of X. Why someone with good ears can't ace this, if the differences are so obvious, is beyond me.

Let me rephrase that: People with good ears CAN ace it--when there's a difference large enough to be heard.

Btw,Tbg, like your former self, I am a modestly paid Assistant Prof. who would dearly love to subsist on cheap electronics. Right on the mark!

What do you think of my suggested experiment? Not blind, but with deception alternated with truth-telling about the values of the cables played?

And Pabelson: do you know if an experiment like this has been performed? I have grad student friends in psych who could do it pretty easily. But there's no point if it's already been done.

"If you can distinguish two amps with flat frequency response and low distortion in a blind test, you will be the first." This means one of two things: there is no differences among amps or DBTesting does not allow humans to judge the differences.

No, TBG, it only means that the differences are not sufficient to be audible by human ears. Read the data, or supply your own. As of now, your only argument seems to be, "I don't believe it, so it can't be true."

As for DBT methodology, it is accepted by everyone in the field of perceptual psychology, in part because it gets plenty of positive results. It just doesn't always get them in the narrow category of high-end audio, because high-end audio has more than its share of snake oil.

Finally, there's a difference between a "delusion" and an "illusion." Look it up.

Pabelson, you say, "If you can distinguish two amps with flat frequency response and low distortion in a blind test, you will be the first." This means one of two things: there is no differences among amps or DBTesting does not allow humans to judge the differences. To accept the formerr means that quality parts, innovative power supplies, careful construction, and generally good design contributes little or nothing and that humans are hopelessly delusional.

As I have posted, I very much suspect the methodology is invalid. In the research that I do, I cannot imagine peers to accept a methodology that so often accepts the null hypothesis that nothing matters. Since my research so often suggests that states enacting seatbelt laws, .08 alcholal level as intoxicated, spending more per capita on education to compete with other states, or to allow concealed handguns all have no effect on the problem to which they are directed, I know the rath concerning my methodology, which unfortunately cannot include an experiment where have of the states randomly drawn have a law or action, and the other half do not. Here many want to accept that governmental actions matter. In audio many want to accept that amps don't matter. I think other methodologies should be used to assess both prior convictions.

Sean T.: If you believe in DBTs, then you have to believe in the results of DBTs. Some years ago (I can get details if you want), Tom Nousaine delivered a paper at an AES conference in which he sumarized the results of about two dozen published DBTs of amplifiers. Of those, only five reported statistically significant positive results. One involved a comparison of 10-watt and 400-watt amps, so clipping distortion was a likely cause. Two others involved a misbiased or oscillating tube amp. One author simply tossed out 25% of his results. And the fifth involved amps with reportedly large frequency response differences.

In other words, amps can sound different, but 1) they usually don't; and 2) when they do, there is a very good and easily measurable explanation. If you can distinguish two amps with flat frequency response and low distortion in a blind test, you will be the first. And most amps today have flat frequency response and low distortion, at least when they are not driven beyond their capabilities.

Pabelson, I completely agree with you with respect to DBT, but then I completely disagree with you that all CD players and amps that are competently designed sound alike.

This is simply not what I hear, and there are good reasons that amps and CD players sound different. Power supplies for one. Good power supplies cost money. Potentiometers in amps ... good ones cost money.

Inexpensive CD players do sound remarkably good these days, and the turntable days of source first are not quite so applicable, but to state that amplifiers are all alike makes me wonder which ones you have has the opportunity to listen to.

No I have not performed DBT on amplifiers, but I have had several occasions where an amplifier that I would have expected to sound excellent (usually on the basis of reviews) sounds markedly inferior to another amplifier that has received much worse reviews, and does so on a range of speakers.

Study proposal:

I don't know if any studies of the following kind have been done. But if not, then one should be done.

Materials: two sets of cheap cables -- cosmetically different, and a set of expensive cables that look just like the cheap ones.

First experiment(s): subjects are introduced to the two sets of cheap cables and told the one is a very expensive $15K cable, the other a $15 cable. Descriptions of each cable, in lavish audiophile prose, are printed on glossy tri-fold with nice pictures, and given to the subjects. The "expensive" cable is praised to the heavens and the "cheap" cable is described modestly.

Then the cables are used (not blind) alternately, to play back a variety of music. Subjects are then asked to rate their listening experiences, both quantitatively, and also qualitatively.

To eliminate the worry about cosmetic differences in the cheap cables making a difference, you could do the test twice, once with cable A being the "cheap" one, and once with cable B being the "cheap" one.

Second experiment(s): do the first experiment but with one expensive cable and one cheap cable that look the same. Do it first by telling the truth about the cables, but then, in the second case, by telling the subjects that the expensive cable is cheap and the cheap cable is expensive.

Here, nothing is bliind. Subjects are all looking at the equipment, and can even observe, from a little distance, the cables being hooked up. But if the DBT guys are right, and it's all hype, we should expect in the first experiment, that the introductions to the cables will lead subjects to favor whatever happens to be described as the more expensive cable, both quantitatively, and in their qualitative descriptions, even though the cables are basically identical cheap cables. In the second experiment, we should expect that when subjects are told the true values of the cables, their judgments favor the more expensive one, but also, that when lied to, they prefer the cheaper cable *just as much* as they preferred the expensive one.

If DBT proponents are wrong, you should expect that subjects will rate the cheap (identical) cables about the same, and that in the second experiment, they will vastly prefer the expensive cable when truthfully described, and when lied to, either still prefer the expensive cable (contrary to what they're being told) or prefer the cheap one, but only by a little.

The point is, we don't need to have people "blind" to do the tests.

And if the cables were manufactured especially for this purpose, you could do the testing through the mail, with in-home trials over a long period of time. Wonder what the results would be?