I see the issue with ABX blind testing

I picture some of these guys starving if they had to hunt for food to survive.  The conversation in the woods would go something like this:  “You only think you heard a deer because you wanted to hear a deer.  I didn’t hear anything therefore, there is no deer.  We need to move on.“

 Back in 1980 I was ready to buy some new speakers.  My wife and I visited a few audio stores and I even took a couple of my own records to audition some speakers.  I got down to two pairs that I liked and I wanted my wife's opinion.  The salesman set up his switch box to go between each pair.  I could hear the differences easily.  My wife stood there waiting for the salesman to switch the speakers but he had already switched them back a forth a couple of times  She could not tell.  I was incredulous that she heard no difference.  It's been my stereo system ever since.  When I play a song she likes she might come sit by me for a bit.

If a recording is played back at a different SPL (Sound Pressure Level) than the original performance, is that not distortion? Yet, recordings do not come with liner notes that specify the playback volume. I tried to convince my wife years ago that I‘m not playing the stereo too loud, I‘m playing it at the original recording level and any less loud would be distortion. She didn‘t buy it but as long as it is a song she likes…

The size of the soundstage on my system varies with the recording. Some recordings are small and intimate and some go beyond the walls of my room. How would I know what is right if I have never heard the original performance? Based on my experiences listening to live music I judge what seems right during a listening session at home.

There is a Roger Waters recording that puts voices 90 degrees to my left and my right. It is the most extreme that I have heard on my system save for the “laughing voice“ that is right by my head near the end of Dark Side of the Moon. Depending on speaker placement and room treatments the Roger Waters recording that I am referring to will make the imaging move forward towards the front of the room a bit from extreme left and right. So I have used that as a benchmark to ensure I have an optimum setup- assuming the artist intent was to have the sounds to the extreme left and right.

Okay, I played that YouTube video on my phone.  I hear the same thing at both time stamps.  That speaker sounds scratchy and edgy both times.  I think the second time makes the edgy sound more obvious because the guy is speaking saying, “This is the natural sound of my voice.“

How is it that we see black on a white projection screen?

The stereo image is an audible illusion.  Our ears/brains are being tricked into hearing sounds distributed throughout a spatial field.  High Fidelity does an amazing job of making the room feel like it is full of musicians.  And this can all be done with just two speakers.  Changing cables or changing gear can disrupt the timing cues and the timbre of the music and since our senses are comparative we tend to decide we like one version over the other.  Hence we will sometimes flip back and forth between two versions in our stereos unable to decide which we like better.  That is not the basis for calling something, “snake oil“.  It‘s much easier when we find a clear difference that we definitely like but don‘t be surprised if others disagree.    

A/B blind testing works best with a group or panel of jurors because differences of opinions always arise in all but the most blatant of changes.  In the end its your system and you can set it up like you want.  I‘ll be the first to criticize it if I don‘t like it.  But it is just an opinion.  Maybe someone will measure their system‘s response in their room and argue it sounds great because it measures great.  Sheesh!

To me, a great sounding system has clarity.  That is high contrast from zero sound at rest to clear, powerful crescendos at the peaks.  Bass is full, robust and clear.  I can hear the fingers plucking the bass guitar strings.  I can hear the timber of the drum heads.  Voices are captivating with a reach out and touch them realism.  Horns and brass sound like real horns and brass.  Cymbals are crisp and delicate with a feel like they are right there.  A piano sounds like a piano in the room and I can almost smell the wood of a stringed instrument.  Rhythm and Pace that makes the music flow out of the speakers and captivates.  No harshness, no fatigue and no sibilance- no cotton ear after listening a while at moderate or high volume.  

A bad room can muddy the bass and mid bass.  It can also accentuate sibilance.  A/B comparisons will be clouded/confused if room interactions are interfering with the music.  Some people will be more sensitive to that than others.  

Technology has been developing for almost two centuries now finding ways to fool our senses.  We hear a human voice from a speaker- a diaphragm powered by an electromagnet that is nothing like vocal cords in an articulated throat.  We watch movies that are not real motion.  They are a stack of still images flashing at us.  We internalize these false sounds and images to make them seem realistic- not real, realistic.  

Why do we see black on a white projection screen?  Because our senses are comparative.  Dim the room lights and project an image onto a white screen and the darkest areas appear as black to us while the brightest areas still appear as white.  

We have become so accustomed to technology that we rarely ever notice the difference between real and realistic.  I’m looking at this screen and it looks like the page of a book but in reality it is flickering at 60 Hz, or 120 Hz in some cases.  But it looks so realistic.   

Dwcda:  I just went back to the beginning of your post.  We have wandered a bit from your original posting.  First of all, you have assembled what looks to be a fabulous system.  Very nice indeed.  Congratulations.  Putting aside the blind testing for a moment, when it comes to cables its easy to get lost in the weeds.  The first rule of cables is that they should not get in the way of the music.  That‘s heard a lot on these forums.  My rule, if I do not hear an immediate difference, or improvement in a cable or power cord change, I go back a step and try again.  I‘ve been involved in blind subjective testing in automotive but it was always done with a jury of four people.  Test conditions have to be stable and at a minimal level.  Testing was always done either in an open field or a hemianechoic chamber.

I have a question.  Having integrated subs into my stereo one time in the past, I struggled with getting the phasing right- or at least how I liked it.  I see your sub in the middle rather than placed at a node.  It is also forward of the plane of your speakers.  Are you able to adjust the phasing of your sub?  Ie, does your sub have a phaser adjustment?  The sub could also be confusing your listening results with different cables.  Did you try listening to different cables with the sub unpowered?

Audphile1:  I wanted to disagree with you but then I thought of my wife and brothers.  You are probably right.  The differences seem obvious to me.  And my two sons.

Seems like any non audiophile that hears my system expects to be blown back into their chair by the bass.  That‘s what seems to matter.  They are generally disappointed.  I might as well just set up the Sony boombox behind my speakers that my brother talked me into buying.  I‘m sure my wife wouldn‘t know the difference.

Another way to measure a cable that came to mind is a Bode plot.  The X-axis is frequency, say 0-100,000 Hz and then two lines on the Y-axis.  The first line would be amplitude ratio, input over output and the second line on the Y-axis would be phase angle.  Now I know the changes in those two lines for a cable will be very small- nothing like an amplifier circuit; but variation will be there.  Again, easier to hear than to measure.

Measuring something is both an art and a science.  It is called Metrology.  The methodology to measure something is just as important as the measurement equipment used.  Let‘s consider an interconnect cable.  What could we measure?  Well, we can measure length, weight, diameter.  We can measure tensile strength, flex strength, minimum bend radius, hardness of the wire, wire gage, etc.  The list can grow quite long.  Then we can measure it‘s electrical properties.  A cable has resistance, capacitance and inductance.  But those three properties are not fixed.  They can vary with temperature, frequency of the signal passing through and amplitude.  Let‘s say we measure this interconnect cable with an LCR meter.  We get a value for resistance, capacitance and inductance.  That measurement doesn‘t say anything about a signal passing through at 5000 Hz or 10,000 Hz or 20,000 Hz.  I remember cable makers back in the 1980s touting their audio cables could pass signals in the MHz range.  I wondered what use was that?  Well, now I understand that harmonics in cables up into the ultrasonic range is important to the sounds within our range of hearing.  So the properties of that cable will cause the phase angle to change between the input and output but the amount of change is dependent upon the frequency.  I‘ve never seen anyone map that in a cable measurement.  How about a waterfall plot of an interconnect showing impedance, and phase angle across a frequency range from 0-100,000 Hz? One cable vs another will have phase angle changes that are unique.  Non-linearities.  Non-linearities are the biggest headache in engineering.  Hard to predict the outcome without some extensive computer modeling.  And in the end, what measurements we see on a graph could be smearing over what is so easily heard.

Another example:  How mechanical vibrations can affect the sound of a cable.  How can that be?  I don‘t know, but I have heard the difference between an isolated cable vs one that is exposed to mechanical vibrations.  My guess is that movements of the wires inside the cable alter the capacitance and inductance- an additional complication.  These minute mechanical changes create colorations in the sound, smearing of images.  I bet cable manufacturers know or understand much of this but they aren‘t about to give away trade secrets to how or what they measure as well as how they manufacture.

Cool video.  Thanks audphile1 for that.  Perhaps I should spend more time perusing the internet and less time with cognitive regurgitation.  Certainly answered my question about Bode plots.  :)

But see, in this video they listen, take measurements, listen, work to understand how the measurements correlate with what they are hearing.  Contrast that to other videos with measurement radicals who just take a few measurements and then conclude no one can hear a difference.  Hahaha.

What proof do you have that other people are really reading and responding to your posts?

I‘d hang out at the drive through ATM‘s to find and recruit some blind people for the testing.  Drive through ATM‘s all have Braille on their keypads.

Why is that?

Perception is reality.  I don‘t totally buy into that, especially with today‘s technology and its ability to affect our perception.  But then magicians have been amazing and delighting us with their sleight of hand since the beginning of time.  W. Shakespeare said, “My wife doth lie, but I believe her anyway.“  That‘s how I take technology.  If I hear a difference between components or cables and I judge it to be worthwhile then I will buy it.  Some people might need ABX testing to be sure they are spending their money wisely.  I do not.

I notice in this video that the speaker concluded what he already knew through ABX testing.  Interesting that he never mentioned which he like better- oversampling or no oversampling.

The days of high pressure salesmen in audio shops is over- or so I believe.  Those types certainly existed in past decades pushing people to spend beyond their means convincing them that their speakers, amps, turntables were the best for the money.  But nowadays it is possible in many cases to bring something home to try and return it if it doesn‘t meet expectations.  If under no pressure, why the uncertainty that something sounds better or not?  

If the shoe fits, wear it.

@botrytis With that much anxiety I‘m wondering how you sleep at night.

I know all about bias.  I once was sure I had a kidney stone.  The pain was real.  I got an x-ray and as soon as the doctor told me I didn‘t have a kidney stone the pain  was gone.  No idea why the pain ever started.

Are you sure?  Perhaps an ABX test is in order.

Bias is what it is all about.  Some people like the sound of speaker A and some speaker B.  They both sound the same to my wife.  She is all about looks.  Looks factor into the bias even if the lights do not need to be on to listen.  If the Nissan Cube were the only car available for sale I might just walk to the store.  

Just saw a video about the BIC ball point pen.  It revolutionized writing.  The video claims the BIC pen is responsible for increasing the world literacy rate from around 30% in 1950 when it became available to over 85% a couple of decades later.  Over 120 billion BIC pens have been sold, they said.  With such an appliance so sure and capable, why is any other style of ball point pen needed?

botrytis: Don’t fear it, embrace it. We learn through failures and mistakes. If you get taken- lesson learned. But to miss opportunities because you fear it could be a ruse is a real shame. Audio is to be enjoyed- it’s all about the music. Either you like the sound of something or you don’t. It’s just like a song- you either like it or you don’t. And even then a song I didn’t like before I like now and sometimes a song I liked before I no longer like now. The same thing might happen with your stereo. You like how it sounds today but one day in the future you might want a different sound. That doesn’t mean you failed to do proper ABX testing. It means tastes change.

I can’t imagine how you can enjoy anything if you have to analyze and ABX test everything. I’d hate to go grocery shopping with you.

As for eating in a restaurant blindfolded- kudos for your bravery.  I wouldn‘t do it.  Strangers feeding me food I can‘t see?  No way.  I‘ve been in some countries where I would definitely not want to do that.

I ACTUALLY watched the video.  Yes, the more trials the more r will converge to one.  Any statistics quoted should have an r value listed as well.

DBT testing does not interest me any more than how a designer comes up with and tests his amplifier, speaker, etc.  What matters to me is how it sounds in my room with my system.  Either I like the sound or I don't.  Of course it is usually somewhere in between and it comes down to do I like it enough to buy it or not.  Other factors matter too like pedigree, reliability, resale...

How do you choose the best color of car using DBT methods?  Seems to be rather difficult to me.

I am not against blind ABX testing- when done properly.  A single person doing an ABX blind test might give them comfort and security that they are making the right choice but those results are meaningless to me.  It still comes down to that person's personal preference.  So why look down on me or thumb your nose at me because I do not want to take the time and trouble to do my own personal blind ABX test?  From my viewpoint if a person cannot trust themselves to make a decision that they will be happy with, then don't do it.  

A real blind ABX test would be conducted by a panel of jurists and would be repeatable with a different panel of jurists on a different day and different locations.  

All the YouTuber did was validate his own preference with his own ABX blind test.  He didn't even mention his preference because it would be meaningless to us.  I also got a chuckle when he said that Danny guessed right 8 out of 10 times but he guess right 18 out of 20 times with his own test.  So he guessed right one more time than Danny but somehow Danny's results are insignificant while his results are valid.  Funny math.

Everyone knows Red cars are faster and sexier- Oh yeah.

I wonder if botrytis realizes he has wandered into a blind alley.  He has mixed up the concept of ABX testing by a panel for commercial sales with a home user selecting components for his audio system.  

To put it in terms botrytis brought up- would he criticize a homebrew distiller for not using ABX testing on his product?  That would make no sense.  A homebrew distiller is making his product for his own personal consumption and so will of course brew it to his tastes.  Would botrytis accuse him of bias for making it taste the way he wants?  He added ingredients without blind testing to see which ingredients are the best?  How dare he.  Of course this is nonsense.  The brewer makes his elixir the way he likes it.  

An audiophile sets up his system to sound the way he wants it to sound.  He is not setting up a commercial adventure to sell copies of his system to others.  Therefore, to suggest that a person must do ABX testing of the components he selects or else he is just fooling himself is a ludicrous concept.  

Here‘s a thought.  Go into a high end restaurant and tell the head Chef that if he didn‘t ABX blind test all of his ingredients then he is biased and wasting money on expensive foods and spices for his meals.  See how that goes.

It‘s simple.  We have trolls with no real interest in audio that make comments to the contrary apparently out of boredom.  To call someone arrogant or foolish for building a stereo system based on their hearing is nuts.  Blind testing has a purpose and a place.  If a person is unsure of their own hearing to build a stereo, that‘s ok.  It doesn‘t have to be a DYI hobby.  Many people just want a turnkey stereo system so they can enjoy their music.  A dealer can set up a system that sounds good to them or they can hire someone to set up their system.  Many of us enjoy the journey almost as much as the destination.  

I go listen to someone‘s stereo and maybe I like it or maybe I don‘t.  It‘s all about my bias vs. the owner‘s.  Maybe I like a song someone else doesn‘t.  Most of the time we benefit hearing other systems and gain insight into the strengths and weaknesses of our own.  I personally do not need to be blindfolded to do that.

Imagine if car magazines tried to do double blind drive tests of cars.  Now that could get interesting.

The irony is that after the first drink the taste no longer matters. 

In the early 1990s I was working with automaker teams to reduce fuel system noise inside the cabin.  They would do DBX testing of fuel systems with 4 jurors sitting in the car inside a hemianechoic chamber.  The procedure was to run the system with engine off first and then with engine on.  Back in the day automakers had warranty costs related to noise so they were taking it pretty seriously.  Then the Lexus brand showed up.  Their new car reset the paradigm for interior comfort.  The body used doubled panels with tar sandwiched inbetween and double laminated glass.  The Ford team evaluated the new Lexus.  Suddenly, the 0-10 subjective rating system that they used dropped a couple of points on average.  Everything is relative.  

First rule of audio:  Your system sounds great, until you hear a better system.

The human brain can retain & compare audio for about a fraction of a second. If it takes 5-10 minutes, or even 1 minute to swap cables, your ability to remember what you heard and compare it with what you're hearing now is effectively zero. 

I remember the Pink Floyd concert I saw in Raleigh, NC back in 1988 perfectly.  They sounded great.  I don‘t remember what I had for dinner last week.