Out of Control

If I might insinuate myself into this conversation between Pbb and Gregm, I would say that I don't object to double-blind testing. It certainly is a means to guard against the power of suggestion. However, I don't see that it much applies to listening evaluations of amps or any other audiophile equipment. Many of us are pretty sure of what we hear, so any double-blind test would be for the non-believers like Pbb rather than for us. However, I will gladly participate in any double-blind amp evaluation test that Pbb would like to set up. If he would just send the airline ticket to me so that I can be at the location of this test ... As far as the sound quality of power amps are concerned, two amps, both highly respected and of similar quality quite often sound very different, in other words easily recognizable from one another, using double-blind testing or not. I would agree that speakers show more sonic variation than amplifiers, but amplifiers are still easily discernable and identifiable from one another. I think that because amps "measure" much better than speakers, they are considered to be much more accurate reproducers of sound. And, if they are more accurate, they must sound the same, very similar, or at least so similar that few can tell the difference between two high quality amps. I think that the mistake here is that what is measured for either power amplifiers or speaker systems is not necessarily the final determinant of what the human ear can hear or all the aural information the human brain can process. Eighty to ninety percent of the measurements taken today for amplifiers (or speakers) they were measuring 50 years ago and while amplifiers have improved by orders of magnitude, they still don't sound live or anywhere near perfect, or even that much alike. So Pbb, let us know where the double-blind test is to be held.

Pbb: It was Bryston 7s & Accuphase 1000 (not 2000). My mistake for sounding general. I was just detailing the system & surroundings, as you had asked for in your 11/9 post.
Thanks for yr latest informative response. I agree with your experiences re, shows and dealers and, indeed, your commendable rigorous approach to judging equipment, etc.

Cheers!

Pbb, I would be interested in knowing what kind of evaluation you perform when choosing between components. My post wasn't meant as a defense for either side of the issue. But too often the spectre of double-blind testing, and science in general, is used like a trump card to "win" an argument. The problem is that most who use such don't bother to practice what they preach or to provide any "real" evidence. You have broached a sensitive subject and in retrospect my comments were probably too harsh. What I want to know is what you propose as an evaluation method that is rigourous enough to remove at least some of our tendency toward suggestion. I would even be willing to try it out.

These are not my words, (in fact I hope that the gentleman who wrote it will understand my intentions for posting it here.)

Charlie

****************start of message**************************

Double blind tests need to be organized in a way that is stress reducing, not stress building. Listeners need to have free, sighted PRE-testing of the devices under test with the program material that will be used. They need to have as much time as needed to listen for differences and characterize them so that any difference they believe they are hearing they can be familiar enough with it to recognize it during the double blind experiment. At any time during the double blind experiment they should have the freedom to go back to a sighted casual mode to RE-learn any difference they may feel that are having a hard time identifying during double blind. Also, at times it is good to set up a double blind test that can include extended listening with each device, even days if desired, to more closely simulate the listeners normal approach to evaluating equipment.

Double blind does not have to be a rushed, rigid session that takes all the control away from he listener. I feel the listener should be given all the control possible, short of knowing what device is playing at any given time.

When you do the experiments this way, what you find is that for any REAL differences you can actually detect them easier in the controlled testing than you can with casual sighted testing. I've shown this many times in the past by introducing just noticeable distortion levels and asking listeners to try to notice the change. Most listeners find it very hard to hear a real, just above threshold difference when it is introduced in their system without controls and careful level matched, blind switching. When we go to the controlled method they can identify differences quite easily.

This doesn't seem to sink in with the subjectivists that plug in a new component and immediately wax rhapsodic about the amazing changes they hear.

I try not to get people involved in double-blind "challenges" because those often create a stressful situation that serves no one. They need to be set up as a quest for answers rather than a challenge. That is why it is often much better to train listeners who have no particular stand on the situation rather than use seasoned audiophiles with preconceived notions.

I tend to avoid using double blind tests for vengeful purposes...except when unduly provoked.

It turns out that most uncorrelated differences (ones attributed to, but not related to, different devices under test) are most often heard due to one of four things:

1) Level mismatch

2) Lack of controls relative to listener position, head position or room acoustics constants. Even listening by your self vs. having a friend on each side of you creates a dramatically different acoustic which changes amplitude
responses to levels above audible thresholds.

3) Inherent poor audio memory that us humans have (much worse than most audiophiles know or are willing to admit).

4) Including and related to 3 and most often the culprit: We don't usually compare with a short repeat loop of program material and most instruments actually sound slightly different throughout a song depending on which moment of the song we are comparing to another moment. Music itself is inherently a very poor test signal, from a control standpoint. Most differences can actually be heard much easier with nonmusical test signals but I don't know many audiophiles that want to accept that notion either.

These are the things affecting real perceived differences but not related to the equipment. These don't even include the dreaded imaginary differences due to beliefs about particular equipment or a predetermined attitude that there will be some kind of difference between any two DUTs.

Lot's of room for error folks. Everyone, whether subjectivists or not must not think they are somehow exempt from these illusion creating variables. If you control all of these things you will find that your non blind listening will take on a lot more reality. Double blind can become less necessary to those who use proper care in their controls.

Either use the controls to have more assurance of uncovering reality or don't and just have fun enjoying your system.

Both are valid activities, just be sure to notice which it is that you are doing and don't pretend to be doing one when you are really doing the other.

*****************end of message*****************************

Again, not my composition, but it does have my full endorsement. Enjoy the music! Charlie

Oh but to be so sure of one's self! I don't put this out as some kind of procedure to follow under pain of banishment or anything like that. Just a few thoughts. For sake of discussion let's divide this into three phases: prior to, during and after auditioning. Prior to, I would recommend not reading any review of the piece(s) of equipment you are interested in. Not easy. Cuts down on the reading on the loo. I would recommend not talking about it to audiophile friends and acquaintances or non-audiophile friends and acquaintances. The former will surely have a ready made opinion and the seed will be planted. The latter will look at you in stunned disbelief and mention that you have a sound system, what you need is a giant projection HDTV compatible what's it. Never talk about your project to your wife/girlfriend, the money will become an issue and you will go for the cheaper model every time. Have some idea of what you are going to audition prior to going. Tear out the reviews and articles from the mags if you must, and only read the advertising, at least you know where they're coming from (dolus bonus, in Latin, I guess). Impractical? Maybe. But what the hell. À la guerre comme à la guerre. During the audition: Make sure you are in familiar surroundings. You should go to the audio shops so regularly that you are on a first name basis with the people there. Double back to see if they abruptly stop talking to each other when you re-enter their premises. The frequency of your visits, especially without ever buying anything, should be such that the audio shop personnel have learnt to detest you. Make it a home trial if you can. Better on everyone's nerves. Listen in your sound room, you should be familiar with the acoustics of the place, and at your normal listening position and volume level. Listen to a variety of recordings (including some poorly recorded material, mono, multi-miked or just plain bad, you would be surprised what they can reveal). Have a set play list of cuts that put different demands on the system. Play them in the same sequence. Play them through. Make sure you have something from all the audio food groups. Make sure you have spoken word, that's what we are all more familiar with. Make sure you have plain sounds of things that don't or barely qualify as musical instruments: hand claps, wood block, something along those lines. If you listen only to heavy metal or techno or whatever comes solely out of machines or amplifiers, you may discontinue the process as soon as you have ascertained that the whole thing plays loud. You need the voice of a male and a female singer, signing. On the topic of what constitutes "singing", tune in later. You need solo instrument recordings of piano, of guitar and something close to the range of the human voice, let's not quibble, say a clarinet. You need a well recorded small group, such as a small jazz ensemble or a chamber music group. You have to include one very dynamic excerpt of a symphonic work, well recorded. You need some kind of big band. You must have a violin, well played and well recorded. You need one monster organ recording of a real organ, in a real space, recorded by a real pro. You should have a dab of bass, both acoustic and electric. You need a very small sliver of rock and let's say a handful of blues, well played and well recorded. You need to believe that 95% of the time, three microphones are enough for a decent recording. If you listen to both analog and cd, you have to double your pleasure, double your fun, (sorry, no gum chewing while you listen) although I am not sure you need exact duplicates of these recordings in both formats. Why not, makes it more of an even match, just kidding. You are to avoid the demon rum and the demon weed and anything which may affect the usual chemical reaction in your brain. You should never do any of this if you have a migraine, cold, sinusitis, tinnitus or deafness. You may drink black coffee, but don't put the cup on top of the speaker enclosures. Do not cook on power amps while listening, wait until later. Preferably, you have to be alone. If you must, ask the other ears not to talk. Throw in that you don't want any obvious body language either. At this stage you will wind up alone, which is how it should be to start with. Warm up your trusty sound system. Acquaint yourself with the hardware you are going to audition. Only make one change at a time in, as we say in French, your hi-fi chain. Listen a second time to the cuts that have incited some overly negative or overly positive reaction on first hearing. Take notes. Get the lingo you want to use in these notes straight in your mind before you commit to paper. Never decide on buying the equipment on the spot. Negative reactions are very hard to get rid of. Positive reactions may not survive the light of day. After auditioning: make a note of your play list, place all your little records back on the shelf, ask yourself if you have actually heard a significant improvement in the overall sound quality with the new component in the chain. Remove the said component from the chain and go back to what you had before. Do you miss something the new component brought? If yes, you may be on to something. If no, stand pat. If you are unsure, don't push it. They ain't no hi-fi police gonna bust ya, unless someone on Audiogon reports you to the authorities. If you remain unconvinced, take the money you were going to spend on that upgrade, in whole or in part, up to the greater of $1,500 or whatever you fell like blowing, and go to the record shop, store, supermarket, and buy records with it. Yes, blow a wad on cds or lps, get the little plastic basket and fill it up with what you consider good music (I always think the people doing that have just settled that claim for the break-in with their insurers). You get to keep the records, you know. It's about the music in the first place, you know. Don't be a smart ass. Promise yourself no to do any of this for a considerable period. Six to twelve months, depending on the severity of your addiction. Hope there is something of some use to someone in here. The use of the actual "dreaded subject" is just the punishment for telling all those little white lies to others and, especially, to yourselves on the good, better, best vector. Regards.