Good point. Seems like spending a reasonable amount of time listening to a component in a well known system where that one new component is the only variable would be a good place to start. Then measure and see if it matches listening experience.
Measurement Bias
Measurement bias is the idea that if you know the measurements of something that information will already bias your perceptions of it’s performance. For instance, knowing the g-force at which car A might slip vs. car B could affect your perception. B, having a higher g-force rating should be a "better" car but if you did not know this you might rate A as better. It may be more fun to drive.
I’ve seen this happen in a review in (I believe) TAS. The review was for a DAC I purchased. The reviewer noted it as "noisy." I read this after I had purchased and been listening to the DAC for a while. I was more than a little shocked, I could hear no noise whatsoever from the DAC. I pulled out an oscilloscope and sure enough, there was unexpected ultrasonic noise on the outputs. I eventually did sell the DAC, but not because it was at all noisy, but because new DACs handled Redbook (44.1kHz/16 bit) tracks so much better. Nowhere in the review did the reviewer mention they had measured it, and they certainly did not point out the deficiencies in Redbook playback, but the reviewer absolutely presented this DAC as noisy but otherwise OK.
So, my point is, that making assessments on the experience that comes from a measurement in audio is tricky business, and if the reviewer is aware of the measurements ahead of time it will absolutely bias them into hearing things which they’d otherwise not, and leave them blind to other real world challenges.
If you want to put together desirability with measurements you need to look at the work Floyd Toole or Bose have done and others in this area and you wont’ find it in a frequency response chart. Of course, Bose’s research is proprietary, but absolutely no one one earth has spent more money on assessing value vs. measurements and manufacturing dollar than Bose.
PS - Please don't argue the quality of Bose speakers here. I'm not arguing for or against them. I'm arguing that the research done in tying together desirability and engineering direction is outstanding. That is all.
20 responses Add your response
@carlsbad2 - Absolutely. There's also the idea of "modernity" bias. Something something kevlar something beryllium something something...
|
This has been on my mind lately. I’ve been thinking about ASR, and the believers who reside there. They are always talking about bias, and the reasons why people think that something sounds good. Not sure they consider how they might have biases based on measurements. It measures well, so it’s good…
I have been working on a pair of speakers for a while now, which I hope to bring to market. Listening, listening, comparing and I other speakers. ESL’s, horns, single full range drivers in pipes. Comparing and comparing. Making adjustments and trying to implement what I like from what I hear in other speakers. Recently did a phase/impedance measurement - after I brought them to where I thought they were “right”. The measurement basically confirmed how I expected them to measure. Next is doing some frequency measurements. I want to confirm again, that what I am hearing is backed up by measurements. Not sure what to do if what I see is in contradiction to what I hear. And if a reviewer points out something about any given piece of gear, and if someone has that piece of gear, it stands to reason they might listen or measure to confirm this. The challenging part, is once, and if, those observations are confirmed, what then?
Once you’ve heard something, it is very difficult to unhear it. |
Yes For myself Tried to add logic to ASR bias to just measurements such as a $800 Topping dac is better hen My $4500 DenafripsTerminator for in the build there was in the design saying it was not a true R2R dac which it is . Your Ears are what dictate which is correct. they are bias to lower quality Audio, I was banned for telling these gaggle of Newbies to go listen to the gear rather then condemn a product they never heard. to me ASR forum =Trash. Let your ears be your guide, or at minimum several reputable reviews. |
Good point for sure. Those who rely on measurement have as much bias as someone who doesn’t (and possibly less listening skill due to neglect). Measurement bias is basically expectation bias, and any listening that ensues (assuming that even happens), is approached with certain expectations put there by the measurement results. I 100% agree with that. Listening works for me. If measurements need to be taken, take them after listening. |
I should point out that the opposite effect can happen as well. Using measurements and listening to gear can change your hearing. You start to hear music like a piece of test gear instead of enjoying it. I know this has happened to me. I get very good at predicting frequency response charts from listening sessions and... it doesn’t make my enjoyment of the experience any better. It just makes me more skilled at estimating frequency response graphs. As a result of this, I know I hear differently than other audiophiles. Is it better? No, not really, but it is different and now I buy different gear or build different gear. There IS a large part of being a music listener and audiophile and mastering engineer which is about social listening and a shared experience. The mastering engineer is trying to connect with the buyers and their gear, and vice versa. Not with tools and technology, and yet we do need to rely on them. |
Not I or anyone I know have purchased any piece of equipment based on any kind of measurement. I'll have to start asking people and see what I find out. We build components and we do measure the parts quality to match certain parts as we believe and hear differences when we do this, but not all of the parts. Happy Listening.
|
One of the reasons this endeavor is so entertaining is that there are no standards for what a stereo system should sound like. It would be impossible to make them. The same speaker in three different rooms is three different sounds. Three different types of speaker in the same room are three different sounds. It is up to every single one of us to tailor our systems to sound the way we want in the room we have available within financial constraints. I personally believe principles are more important than measurements. Some measurements are more important than others, but given the range of issues related to speaker performance in situ, specifications and test measurements reside in the background. I have said on multiple occasions, what someone says any piece of equipment sounds like means nothing to me because I have no way to interpret this. I only care what I think something, usually a loudspeaker sounds like in my room. When it comes to loudspeakers the specs are virtually meaningless, but the design speaks volumes. In many instances the specs are misleading. The important measurements to make are those you can make in your own listening room in regards to amplitude, group delay and dispersion. If you can't hear distortion any measurement is of no significance. If you can't hear noise any noise level is of no significance.
|
Both are great, lets scientifically measure them and listen to them. You will end up picking a component that works best for you for any number of reasons. Your bias will be there regardless, for the numbers or for the brand, or for the price, or for the room you auditioned your component in, or for a reviewer, or for word of mouth, or for what you believe sounded better because of an A/B sound comparison which would mostly likely be unreliable (depending on you memory and/or chosen A/B components)... and the list goes on... These discussions on either side are beyond ridiculous. |
Well, I think THX did a pretty good job for home theater systems in auditoriums which included acoustic requirements and in specifying speaker requirements for homes as well. We just kind of ignore it all for stereo. |
stereophile does it right. reviewers don’t know the measurements until their review is complete. keeps them on their toes. if measurements show a clear flaw that should be relatively easy to hear, and they didn’t hear it, their credibility is shot. hearing is inherently more biased than measurements, but agree measurements can bias us. but so can the amt of $ spent. just on those three factors (there are many others) I would rate them ears, $, measurement in descending order of potential for bias. the key is what steps we take for each to mitigate the potential for bias. |
I dont understand the apparent need for validation by measurements. It should simply be enough to like the sound of a component and all this mention of bias is just masturbation. It is impossible for us to know the motivations of others in terms of the choices they make when they buy equipment just as we cant know what bias may or may not exist. When mijostyn says that the "design speaks volumes" he is right on the money. If you let measurements lead the way you will most likely end up with S.S. devices and speakers which adhere to particular design criteria. Rather than asking if a device measures well you should be asking how did the designer achieve these measurements and what sacrifices were made to achieve their desired results. |
That’s exactly where I stand. We listen with your ears and brain, so listening is an excellent way to determine what gear you like the sound of. (sort of seems like a "Duh" moment to me! 😉) Manufacturing audio gear is a different story, but choosing what you like the sound of shouldn’t require measurement validation any more than determining that a steak tastes good, or that a sports car looks nice in red. |
Any measurement bias disappeared very quickly when I got into high end audio fifty years ago. My eyes would flit from watts per channel specs, distortion, etc. and what I heard and that was the end of it. There was virtually no correlation between those and what I heard. Revisions in basic measures have occurred without much change in the relationship. The cost in well reviewed mainstream audiophile produces (within variation on taste) has been a near perfect correlation… particularly when throwing in weight in comparison to most metrics. |