Pabelson, interesting challenge, but let’s look at what you’ve said in your various posts in this thread. I’ve pasted them without dates, but I’m sure that you know what you’ve said so far.
"What advances the field is producing your own evidence—evidence that meets the test of reliability and repeatability, something a sighted listening comparison can never do. That’s why objectivists are always asking, Where’s your evidence?"
"A good example of a mix of positive and negative tests is the ABX cable tests that Stereo Review did more than 20 years ago. Of the 6 comparisons they did, 5 had positive results; only 1 was negative."
"It's better to use one subject at a time, and to let the subject control the switching."
"Many objectivists used to be subjectivists till they started looking into things, and perhaps did some testing of their own."
You cite the ABX home page, a page that shows that differences can be heard. Yet I recognize that the differences when heard were between components that were quite different and usually meeting the standard you’ve indicated as much better specs will sound better.
Once you decide something does sound different, is this what you buy? Is different better? You say:
"Find ANYBODY who can tell two amps apart 15 times out of 20 in a blind test (same-different, ABX, whatever), and I’ll agree that those two amps are sonically distinguishable."
Does that make you want to have this amp? Is that your standard?
One of the tests you cite was in 1998 with two systems that were quite different in more than price. Does that lend credence to the DBT argument? On the one hand you point to all the same but one component with one listener with repeated tests but then cite something quite different to impugn subjectivists – not that it’s all that hard to do. You also cite a number of times that DBT has indicated that there is a difference. Which is it? Is there “proof” of hearing differences that has been established by DBT? It certainly appears that there is from the stuff you have cited. By your argument, if this has been done once, the subjectivists have demonstrated their point. I don’t agree, and you really don't appear to , either.
My points were two, and I do not feel that they have been addressed by your challenge. One, that most DBT tests as done in audio have readily questionable methods – methods that invalidate any statistical testing, as well as sample sizes that are way too small for valid statistics. Those tests you cite in which differences were found do look valid, but I haven’t taken the time to go into them more deeply. Two, and the far more important point to me, do the DBT tests done or any that might be done really address the stuff of subjective reviews? I just don’t see how this can be done, and I’m not going to try to accept your challenge , “If you know so much ...” Instead, if you know so much about science and psychoacoustics, and you do appear to have at least a passing knowledge to me, why would you issue such a meaningless, conversation stopper challenge? Experiments with faulty experimental design are refused for journal or other publication all the time by reviewers who do not have to respond to such challenges. The flaws they point out are sufficient.
Finally, I’ve been involved in this more than long enough to have heard many costly systems in homes and showrooms that either sounded awful to my ears or were unacceptable to me one way or another. The best I’ve heard have never been the most costly but have consistently been in houses with carefully set up sound rooms built especially for that purpose from designs provided by psychoacoustic objectivists. This makes me suspect that what we have is far better than we know, a point inherent in many "objectivist" arguments. My home does not even come close to that standard in my listening room (and a very substantial majority of pictures I see of various systems in rooms around the net also seem to fall pretty short). The DBT test setups I have seen have never been in that type of room, either. What effect this would have on a methodologically sound DBT would be interesting. Wouldn’t it? |
DBT as done in audio has significant methodological issues that virtually invalidate any results obtained. With improper experimental design methodology, any statistics generated are suspect. Regularly compounding the statistical issues is sample size, usually quite small, meaning that the power of any statistics generated, even if significant, is quite small, again meaning that the results are not all too meaningful. Add to this the criticism that DBT, as done so far in audio, might be introducing its own set of artifacts that skew results, and we have quite a muddle.
I'm not at all opposed to DBT, but if it is to be used, it should be with a tight and valid experimental design that allows statistics with some power to be generated. Until this happens, DBT in audio is only an epithet for the supposed rationalists to hurl at the supposed (and deluded) subjectivists. Advocates of DBT have a valid axe to grind, but I have yet to see them produce a scientifically valid design (and I am not claiming an encyclopedic knowledge of all DBT testing that has been done in audio).
More interestingly, though, what do the DBT advocates hope to show? More often than not, it seems to be that there is not any way to differntiate component A (say, the $2.5K Shudda Wudda Mega monster power cord) from component B (stock PC)or component group A (say, tube power amps)from component group B (transistor power amps). Now read a typical subjectivist review waxing rhapsodically on things like, soundstage width and height, instrumental placement, micro and macrodynamics, bass definition across the sepctrum, midrange clarity, treble smoothness, "sounding real," etc., etc. Can any DBT address these issues? How would it be done?
You might peruse my posts of 8/13/05 and 8/14/05 about a power cord DBT session, carried out, I think, by a group that were sincere but terriby flawed in how they approached what they were trying to do to get an idea of how an often cited DBT looks when we begin to examine critically what was done.
http://forum.audiogon.com/cgi-bin/fr.pl?fcabl&1107105984&openusid&zzRouvin&4&5#Rouvin |
Pabelson,
I think we may be closer than you think on this issue but you seem to want it both ways, a difficulty I see repeatedly in "objectivist" arguments. You say:
" For badly implemented tests, they've yielded remarkably consistent results, both positive and negative." --
all the while insisting on scientific assessment.
Methodologically unsound experiments yeild no meaningful results. The pattern of meaningless results does not matter. Your argument in this regard is emotionally appealing, but it is incorrect.
Moreover, the notion that "DBTs address a prior question: Are two components audibly distinguishable at all?" is also suspect absent appropriate methodology. I notice in your posts that you address reliability and repeatibility, important factors without any doubt. Yet you have never spoken to the issue I have validity, and this is the crux of our difference. Flawed methodology can yield repeatable results reliably, but it is still not valid.
And, of course, as you have noted, many DBT's have shown that some components are distinguishable.
The issue beyond methodology, I suspect, is that there are some people who can often reliably distinguish between components. They are outliers, well outside the norm, several standard deviations beyond the mean, even among the self-designated "golden eared." When any testing is done on a group basis, these folks vanish in the group statistics. You can assail this argument on many grounds. It is indefensible except for the virtual certainty that there is a standard distribution in the populatiuon in acuity of hearing.
So, my position remeains that there is surely a place for DBT testing, but even after all the methodological and sampling issues were addressed, I'm still unsure how it fits into the types of reviews most audiophoiles want.
In your hypothetical magazine, after DBT establishes that the Mega Whopper is distinguishable from El Thumper Grande, how would either be described? Would there be a DBT for each characteristic?
Freud had a book on religion entitled "Future of an Illusion" and you may well feel that this is where all of this ultimately is. I'm not sure that I have an answer to that, but this may well be why Ausio Asylum has devclared itself a DBT free zone. |
Pableson, I find your posts interesting though not really responsive to the initial thread by TBG about the place of DBT in audio. Nor, have I felt that your posts have been responsive to my similar concerns and additional concerns about experimental validity (though I am sure that not all have been invalid), for instance, the very interesting and amusing 1984 BAS article where the Linn godfather not only failed to differentiate the analog from the digitally processed source, he identified more analog selections as digital. But... this was an atypical setup that would not be found in any home. We can’t really generalize from this, and this has nothing to do with advocacy of the "subjectivist" viewpoint. If you would be true to your objectivist bona fides, wouldn't you have to agree?
Then, there’s the issue, supported by your citations, that there have been DBT’s going back years that have demonstrated noticeable differences between individual components.
So, I think there is a background issue, and this was also mentioned in TBG’s initial post. Many adherents of DBT seem to be seeking the very "conformance" that you want to point out in others. That "conformance?" That until the very qualities claimed to exist can be proven to exist they must be assumed not to exist. Intoxicating argument, but ultimately revealing of a distinct bias, the invalidation of the experience of others as an a priori position until they can meet your standard. This "you ain't proved nothin'" approach is especially troublesome when one reads subjective reviews and realizes that the points they raise, creative writing they may well be, could never be addressed by DBT, ABX, or any other similar methodology. The majority of what we are able to perceive is not amenable to measurement that can be neatly, or even roughly, correlated with perception. To claim otherwise is an illusion. Enter the artists with some scientific and technical skill and we have high end audio. Sadly, with them come the charlatans and deluded along with average and "golden eared" folks who hope that they can hear their music sound a bit more like they think they remember it sounding somewhere in the past. Add something like cables and it seems the battle lines are drawn.
I’m a bit suspicious that you might not allow the person who can reliably detect a difference between two components to write whatever he wants in your forthcoming journal. You claim that once the DBT is passed, he can describe a component any way he wants. It doesn’t really make sense to me because a "just noticeable difference" is not the same as being able to notice all of the differences subjective reviewers claim, does it? If someone can tell the real Mona Lisa from a reproduction, even a well executed one, do you really care to hear about everything else he thinks about it? I don’t. I might want to see it myself, though.
I don’t think there will ever be anything like being able to recreate the exact sonic experience of a live musical performance in a home or studio. What we can hope for are various ways to recreate some reasonable semblance of some aspects of some performances. DBT probably has a place there.
In the meantime, I’d like to suggest a name for your journal, The Absolutely Absolute Sound. I think Gunbei has a supply blindfolds. |
Sadly, anyone who views looking at methodological issues as quibbling does not understand that methodology is at the heart of science. Boring, tedious, surely, but a necessary condition to call something science. Your question, "Why in the world would you need a control group in a perception test?" reveals that you need to brush up on your basic science. |
"We can't measure sound and make predictions about how it will sound to you, because how it will sound to you depends on too many factors besides the actual sound." This is what I have been saying in addition to less than favorable comments about many subjective reviews. This problem is one of many that equally hampers "objective" reviews.
A closer reading of what I have written would reveal that I am, at best, ambivalent about the whole process of audio reviewing, subjective or objective. Moreover, DBT has yet to produce much of significance beyond some people can sometimes tell under some conditions.
"And the idea that you, an amateur audio hobbyist without even an undergraduate degree in psychology, has any standing to declare what is and is not valid..." You have constructed a total absence of "valid" credentials for me, an exercise in "creative writing." "Lack of standing" is the problematic judgment invoked that leads you to invalidate experience, a source of needless angry conflict. It might be somewhat accurate to characterize me as "an amateur audio hobbyist," but you have taken quite a leap to decide that I am "without even an undergraduate degree in psychology," a leap that could not be more inaccurate. At least, you didn’t mention my lack of teeth and eviction from the trailer park.
There are subjective reviews in just about every field. Anyone that takes them for hard fact is not understanding what they are. I’d also suggest an advanced readings in sensation and perception to better understand the distinction between "what we are able to perceive" and "how we perceive it."
So, I’ll leave you with two thoughts apropos of this discussion. Einstein said, "Not everything that can be counted counts; and not everything that counts can be counted." In Alice in Wonderland, the Dodo said, "Everybody has won, and all must have prizes." Where's Rodney King, anyhow? |
I’ve gone over this debate and would like to summarize many of the points made.
As to DBT there may be:
1. Problems with methodology, per se, in audio;
2. Problems with most DBT that has been done, e.g., lack of random assignment or no control groups, making these experiments invalid scientifically;
3. Problems with particular experiential designs that are unable to yield meaningful results;
4. Sample problems, such as insufficient sample size, non-random samples;
5. Statistical problems making interpretation of results questionable.
All of these problems interact, making the results of most DBT’s in audio scientifically meaningless.
Advocates of DBT have been especially vociferous in this forum, but what have they actually said to respond to these criticisms? Virtually nothing beyond "No!" or "Where’s your proof?"
The "proof" of their position cited has been interesting, but it has been a reporting on the power of "sham" procedures or other stories that do not meet the guidelines necessary for a DBT procedure to qualify as science.
At the same time, they call DBT science, and maintain the supremacy of science. Calling something science without strictly adhering to scientific procedures, unfortunately is not science, and this is the case with DBT in audio far more often than not. It is more akin to the claims that intelligent design is science than it is science at this point. An additional point made in this forum has been the large number of DBT’s done that have failed to demonstrate that differences can be heard. A large number of scientifically compromised procedures yields no generalizable conclusions.
For anyone who has worked at a major university research mill, as I have, the skepticism about research results is strong. It is not that there is an anti-research or anti-science attitude. Rather, it is a recognition that the proliferation of research is more driven by the necessity of publishing to receive tenure and/or the potential for funding, increasingly from commercial interests that have compromised the whole process. We will have to see what happens to scientific DBT in audio when and if it happens.
I conclude that we are speaking fundamentally different languages when advocates of subjective audio evaluation and DBT advocates speak. For my part, subjective evaluation is fine as long as I understand that I better think twice before I believe a reviewer. I also truly believe in the supremacy of science, and intelligent design is not science. |
