Why is Double Blind Testing Controversial?


I noticed that the concept of "double blind testing" of cables is a controversial topic. Why? A/B switching seems like the only definitive way of determining how one cable compares to another, or any other component such as speakers, for example. While A/B testing (and particularly double blind testing, where you don't know which cable is A or B) does not show the long term listenability of a cable or other component, it does show the specific and immediate differences between the two. It shows the differences, if at all, how slight they are, how important, etc. It seems obvious that without knowing which cable you are listening to, you eliminate bias and preconceived notions as well. So, why is this a controversial notion?
moto_man

Showing 1 response by nirp

The use of blind and double-blind procedures presumes one is employing the logic of hypothesis testing. That is, that there is a null hypothesis (i.e., that there are no differences between two treatments—in this case, two sets of interconnects) and an alternate hypothesis (there is indeed a difference). Experimenters are more than experimental custodians. Their biases and expectations can profoundly influence a study. To the extent that all people (including experimenters) have biases, one would double-blind the treatments to reduce among other things "experimenter effects." It’s surprisingly easy for an experimenter to influence a study (e.g., Stanley Milgram’s famous obedience studies). It is also easy for other participants (formerly known as “subjects”) to influence each other (e.g., Ash’s line judgment experiments where participants tended to agree with Ash’s confederates that clearly dissimilar lines were the same).

There is a famous researcher/psychologist/statistician by the name of Robert Rosenthal who once told his students that he had obtained two breeds of rats from another famous researcher. One type of rat was called “maze smart” and the other was “maze dull.” Dr. Rosenthal asked the students to teach these rats to run though mazes (ah, the power of cheese). After a few weeks or so the students were asked to show off their rats’ maze prowess (as it were). The “maze smart” rats performed significantly better than their “dumb” counterparts. The kicker here is that the rats were OF THE SAME SPECIES. One cannot infer that the students intentionally influenced the training, but it most certainly was measurable. Moreover, when the experimenter bias was measured it turned out that the “smart" rats owners had "imparted" a greater positive measurement bias than did the “dumb” rats owners negative measurement bias.

There are probably much better examples than these, but I’m in a hurry to go downstairs for dinner :-) so I’ll wrap this up soon.

Something else to consider is that “different” does not mean “better.” People’s ability to remember sounds and colors varies greatly but rarely is the memory accurate after a short decay period. With audio equipment evaluation, it tends to result in a bias for a certain “sound” regardless of whether or not that sound is authentic. When it comes to making a decision as to whether one component is better than another, it probably makes the most sense to have a reference. In the case of audio, I’d say that reference should be THE REAL THING. It’s not practical to have live orchestra tag along on equipment tests but it doesn’t hurt to keep that in mind. Some people go on and on about how they prefer one cable to another because their favorite is “warm” or whatever. Real sounds from an orchestra or a band are not necessarily “warm.”

All that said, if one believes that a $6,000 set of interconnects sounds better (they just might sound *different*) than a $70 pair then let ‘em. The more expensive cable might even sound closer to reality. One would hope that the more expensive cables aren’t just mostly cosmetics and markup.

--Paul

p.s. and yes, spending time with a set of cables or anything else in the system is a great way to know if one really likes the sound. On a marginally related note, a friend of mine once said “I’ve never owed a handheld device that I liked after having it for a week.”