Amir and Blind Testing


Let me start by saying I like watching Amir from ASR, so please let’s not get harsh or the thread will be deleted. Many times, Amir has noted that when we’re inserting a new component in our system, our brains go into (to paraphrase) “analytical mode” and we start hearing imaginary improvements. He has reiterated this many times, saying that when he switched to an expensive cable he heard improvements, but when he switched back to the cheap one, he also heard improvements because the brain switches from “music enjoyment mode” to “analytical mode.” Following this logic, which I agree with, wouldn’t blind testing, or any A/B testing be compromised because our brains are always in analytical mode and therefore feeding us inaccurate data? Seems to me you need to relax for a few hours at least and listen to a variety of music before your brain can accurately assess whether something is an actual improvement.  Perhaps A/B testing is a strawman argument, because the human brain is not a spectrum analyzer.  We are too affected by our biases to come up with any valid data.  Maybe. 

chayro

Showing 6 responses by sdl4

rtorchia, I had an Adcom GFA-555II in my main 2-channel system for over 25 years, and I agree that the Adcom was a very good value in its time.

@chayro  Just to return to your original question in this thread, as long as you switch between components when evaluating gear and your mental state (analytical, relaxed, mellow, angry, happy, silly, or whatever) doesn't change dramatically as you switch gear, you should be able decide which gear sounds best to you.

@rtorchia  Your comments about "accurate" sound compared to "distinctive" sound in terms of distortion are not consistent with what I have learned over the course of 50 years of listening to hifi gear. I've lived through the Stereo Review mindset of the 1970s and 1980s, when designers sought ever-lower levels of THD (usually by increasing feedback in the circuits), despite the fact that many of these products sounded harsh and electronic. Many of the comments on the ASR site seem to embrace a view quite similar to that now debunked view from Stereo Review. 

If you want to learn how to design gear that measures and sounds great, pay attention to the approach of a great audio designer like Nelson Pass. I recently watched a YouTube interview by Steve Guttenberg, during which Nelson discusses the importance of listening to distortion profiles rather than just measuring THD levels when designing great amps. Some people like to claim that amps from Pass Labs add distortion (mainly 2nd and 3rd harmonics) to improve the sound, but Nelson says that they simply don't try to suppress the 2nd and 3rd harmonics as much as they suppress higher order distortions. So which design approach is more "accurate" and which is more unnatural, one that reduces the influence of higher order distortions or one that simply reduces THD? I would argue that whatever sounds more natural and more like real music is the best choice. And Nelson Pass stresses that they use an ongoing approach of measure-listen-measure-listen to design their amps. Any approach that prioritizes measuring over listening clearly does not have its priorities straight.

For those who are interested, Paul McGowan has put together a video on the PS Audio forum that explains what the P12 regenerator does and how to measure it correctly.

Prof, with all due respect, Amir's approach still assumes that he is measuring everything that matters in terms of ultimate sound quality. I think that is unlikely to be a valid assumption. If Amir's assumption of perfect measurements is invalid, then the importance of human listening cannot be excluded from any evaluation of a product designed to improve the sound quality of reproduced music. 

Amir talks up the value of unbiased listening tests, but then admits that he is clearly biased and unwilling to go to the trouble of setting up any unbiased listening tests himself. That is the bit that is missing from most of Amir's reviews, including his P12 review. 

 

Amirm is coming to the question with knowledge about what type of distortion is possible, and what type of distortion is audible (given known thresholds of hearing).

If there is some OTHER form of distortion Amirm is not measuring for that would be audible...what would that be?

This is the problem I mentioned earlier. I do not believe that either Amir or the folks at PS Audio know exactly how all possible distortions (and time delays, phase changes, noise patterns, filter slopes, audible effects that change with frequency, etc.) interact to affect the sound quality the human brain hears and interprets. Thus,  SQ cannot be assessed using measurements alone. Listening must be part of the assessment. 

My summary of the P12 review on ASR is that Amir agrees that the P12 cleans up the waveform, as PS Audio intended, but that he can't measure any changes that he believes could improve the SQ when an amp is hooked up to the P12. Of course, Amir doesn't think listening is needed because he thinks his measurements tell the whole story. In contrast, PS Audio has measured and listened to the P12 during development of the product, and most users report hearing major improvements in SQ when using the P12, especially when there are significant problems with the power being supplied to the P12.

I do not believe that the folks at ASR are omniscient, and I also do not believe that the folks at PS Audio are trying to rip-off their customers.

There are actually more facets to the Pepsi story. Look up the "Pepsi Challenge" on Wikipedia for several additional bits of information.

The reason Pepsi started doing a single-blind Pepsi vs. Coke challenge decades ago was because Pepsi was often preferred over Coke if the test was based on a single sip of cola. (Most people think this is due to the fact that Pepsi tastes a little sweeter than Coke.) However, Coke often was preferred when consumption occurred over a more extended period of time. This sounds a bit like nonoise's ideas about extended listening vs. short-term listening.

If you're really into these issues, take a look at a great study of how brain damage to the prefrontal cortex can affect blinded or semi-blind taste testing: 

 

This study showed that taste preferences were influenced by knowledge of the brand of cola, except in people with damage to the prefrontal cortex of the brain. I don't know if this study has any implications for drinking wine, but it sounds like it could.