2. misc-audio
  3. 189510-20253

Measurement Bias

Measurement bias is the idea that if you know the measurements of something that information will already bias your perceptions of it’s performance. For instance, knowing the g-force at which car A might slip vs. car B could affect your perception. B, having a higher g-force rating should be a "better" car but if you did not know this you might rate A as better. It may be more fun to drive.

I’ve seen this happen in a review in (I believe) TAS. The review was for a DAC I purchased. The reviewer noted it as "noisy." I read this after I had purchased and been listening to the DAC for a while. I was more than a little shocked, I could hear no noise whatsoever from the DAC. I pulled out an oscilloscope and sure enough, there was unexpected ultrasonic noise on the outputs. I eventually did sell the DAC, but not because it was at all noisy, but because new DACs handled Redbook (44.1kHz/16 bit) tracks so much better. Nowhere in the review did the reviewer mention they had measured it, and they certainly did not point out the deficiencies in Redbook playback, but the reviewer absolutely presented this DAC as noisy but otherwise OK.

So, my point is, that making assessments on the experience that comes from a measurement in audio is tricky business, and if the reviewer is aware of the measurements ahead of time it will absolutely bias them into hearing things which they’d otherwise not, and leave them blind to other real world challenges.

If you want to put together desirability with measurements you need to look at the work Floyd Toole or Bose have done and others in this area and you wont’ find it in a frequency response chart. Of course, Bose’s research is proprietary, but absolutely no one one earth has spent more money on assessing value vs. measurements and manufacturing dollar than Bose.

PS - Please don't argue the quality of Bose speakers here.  I'm not arguing for or against them.  I'm arguing that the research done in tying together desirability and engineering direction is outstanding.  That is all.

erik_squires
perkri
907 posts
 

This has been on my mind lately. 
 

I’ve been thinking about ASR, and the believers who reside there. They are always talking about bias, and the reasons why people think that something sounds good. 
 

Not sure they consider how they might have biases based on measurements. It measures well, so it’s good…

 

I have been working on a pair of speakers for a while now, which I hope to bring to market. Listening, listening, comparing and I other speakers. ESL’s, horns, single full range drivers in pipes. Comparing and comparing. Making adjustments and trying to implement what I like from what I hear in other speakers. 
 

Recently did a phase/impedance measurement - after I brought them to where I thought they were “right”. The measurement basically confirmed how I expected them to measure. 
 

Next is doing some frequency measurements. I want to confirm again, that what I am hearing is backed up by measurements. 
 

Not sure what to do if what I see is in contradiction to what I hear. 
 

And if a reviewer points out something about any given piece of gear, and if someone has that piece of gear, it stands to reason they might listen or measure to confirm this. The challenging part, is once, and if, those observations are confirmed, what then?

 

Once you’ve heard something, it is very difficult to unhear it. 

carlsbad2's avatar
carlsbad2
1,602 posts
 

And I just finished my run.  A thin runner looks faster than a chubby one, but that's not always the case.  --Jerry

cdamiller5
98 posts
 

I’m not a measurement believer. Have several systems and I have built them solely on what I enjoy hearing. Tone, separation, depth and clarity.

audioman58's avatar
audioman58
4,135 posts
 

Yes For myself Tried to add logic to ASR bias to just measurements 

such as a $800 Topping dac is better hen My $4500 DenafripsTerminator 

for in the build there was in the design saying it was not a true R2R dac 

which it is . Your Ears are what dictate which is correct.

they are bias to lower quality Audio, I was banned for telling these gaggle of 

Newbies to go listen to the gear rather then condemn a product they never heard.

to me ASR forum =Trash. Let  your ears be your guide, or at minimum several reputable reviews.

knotscott
921 posts
 

Good point for sure. Those who rely on measurement have as much bias as someone who doesn’t (and possibly less listening skill due to neglect).    Measurement bias is basically expectation bias, and any listening that ensues (assuming that even happens), is approached with certain expectations put there by the measurement results. I 100% agree with that.

Listening works for me. If measurements need to be taken, take them after listening.

More to discover