The Audio Science Review (ASR) approach to reviewing wines.

After attempting to read this discussion, I got a hangover now.

Here’s a very nice Tom Martin review of the Perlisten speaker.

https://www.youtube.com/watch?v=leL1vXcKhZw

Think we all know how Perlisten speakers tend to measure, pretty freaking good.

But, i’d like for a reviewer to be articulate the sound of it as good as Tom. He isn’t even 'selling' anything in this review, he tells ya who it may work for and who it may not work for..

Is the ASR Reviewer (Revel/Harman dealer/sales guy, Madrona Digital) capable of doing such a review? There are speakers that measure and sound better than Harman trash, but, he’ll find a way to diss them...gotta worry about Madrona Digital sales...not to mention that he sounds like a water bottle with a hundred holes.

Still trying to wrap my head around the need for a scale to tell me I gained 10 lbs. 

so, perspective is a thing, I guess if you weigh 250 lbs, 10 lbs might be gained un-noticed.

Going to wine, what is your perspective? taste and flavor or getting tipsy? you have to admit, some people drink wine with the purpose of getting their buzz on, so taste is not as important as the alcohol content (measurable). They may not want to spend the $ needed for the bouquet of flowery bliss and head straight to the 5 buck chuck or even Thunderbird - it gets the job done...

Comes down to what YOU are there to enjoy. Measurables only measure what you are measuring - sounds like a Zappa line - If you can't measure it, like flavor, it becomes opinion and other opinions always influence your opinions so the opinion is clouded because of your unique perspective.

Ok, maybe I need to go back to work.  

Here’s my issue with ASR

Several years ago when building a stereo system I discovered ASR who strongly recommended the Benchmark line. I purchased their DAC, Pre and Amp. While I found I was mostly satisfied with the result. Then after a few years I became dissatisfied with how clinical the sound seemed. I particularly was not happy with how the upper mids and lower treble were presented and found them somewhat bright and up front. So eventually over several months I replaced the Benchmak units. When I posted on ASR that I found an amplifier that sounded much better to me than the Benchmark (the Accuphase P4600) I was told there was no way any amp would sound better than the Benchmark since it tested perfectly. I was imagining it. I had convinced myself it was better since I spent the money. I told them I had compared several amps to the Benchmark (Bryston, Mac, NAD) using my speakers and DAC and the Bryston and Accuphase were noticeably superior. They refused to believe it and basically told me I was delusional. Seemed pretty rigid in thinking. Refused to believe any amp would noticeably audibly perform better than the Benchmark in my system. Seemed a little cult like to me as there are some there that can be quite hardcore.

 

@hilde45 "Measurements" is too crude a word. @prof was pointing at this issue and this comment ignores it.

Some measurements are, say, 2nd harmonic distortions -- those may upset some at ASR, but the rest of us understand that those measurements are NOT aligned with "bad sound" as some of us experience that. (Others here do NOT like that 2nd harmonic. So, this varies.)

 

ASRs whole approach of anything that measures bad sounds bad, and the inability to describe how things sound or what people prefer sometimes - is a nonstarter for me. It all comes back to what people hear and what they enjoy most, and no metric tells us this very well. The human ear/brain is sophisticated, and enjoys a lot more than todays graphs demonstrate. I believe today’s sensors and tools fall short, by a lot.

----

Tube amplifiers are known for their "euphonic" distortions, which can add a pleasant warmth and richness to the sound. Tube amplifiers tend to produce even-order harmonic distortion, which is more musical and pleasing to the ear.

Even-order harmonic distortion means that the distortion components are harmonically related to the original signal, creating a sound that is perceived as fuller and more natural. This type of distortion can enhance the listening experience, especially for musicians. I’d rather ask a musician what they prefer.

@hilde45   nice post.  Only thing I’d add @toronto416 is that, with respect to power conditioning products, I think one can make a much stronger case for a measurements-first mentality there than for virtually any other product.  The point is to “filter out noise” or “lower the noise floor”.  These are very easy to measure, and I would argue harder to objectively and consistently hear.  So it is precisely for these sorts of products that ASR is most useful, in my view, particularly as it’s an area of the industry most prone to fraud and misrepresentation.  If someone says their products filter the AC mains, why on god’s green earth wouldn’t you want someone to measure that for you before you spent a dime of your hard-earned money?  

While Amir is definitely aloud to have his opinion, it’s rarely shared by me, so I haven’t looked at his channel in years.

I’d ask the clown car that does freq domain snapshots to measure my violins or my piano....bet that would go well/swell...

"Bad violin", said the dingus dumbernicus...

A good friend of mine is a winemaker of some renown. I've been following his efforts since he started out, I've helped with winemaking many times, we had numerous conversations about the fine points of the craft during which I learned an awful lot and, of course, I freely offered my unbiased feedback based on consuming vast amounts of the product. I'm a helpful guy. 

To make a long story short, winemaking is science-driven to an extent that truly surprised me back then. A good winemaker, or brewer, or distiller for that matter, is in large part a chemist and on the ground level, the incremental experimenting, documentation and chain of custody are straight out of science 101.

OP should familiarize himself with enologists and what they do.

ASR has value. Audiogon has value. Other venues and publications have value too. The more points of view - and data points - we have, the more empowered we are to make choices. Which, ultimately, are for us to make.

@analog_aficionado this seems like the most complete explanation of the disparity between certain audiophile communities and ASR.  I'll try to enhance your points without reiterating too much.

Measurements have their place.  The measurements aren't the problem with ASR.  the problem is the mob of people that pounce anyone that says "hey this is better even though it measures poorer"

As if someone could hear the difference between -120 THD and -110.  I agree that at this point, most of these numbers are meaningless as we are well below the 1% THD threshold.  Which is another funny point in all this.  ASR claims to say "you can't tell the difference between amps and cables" based on studies done over 50 years ago, and yet will quibble over -100 Sinad vs -120.  It seems completely absurd to me to both say you're WITH the science, and at the same time be quibbling over measurements that should have NO audible effect based on the "science"

And I have a problem with him listening to the speakers, but not any other equipment because "it all sounds the same" - then what is the point of measuring all of this junk????

After hearing things that they don't measure make a truly substantial differences in my system (Shunyata power conditioning and cabling, proper speaker cabling), I realize that I can only use them for measurements and for feature set breakdowns, NOT for choosing my equipment.

The sad thing is, the "happy panther" scale always rewards the highest measuring equipment because of the horde of stat hunters that are ready to say it's better without hearing any of it.

It's sad really.  These guys are all audio lovers but are sitting there with their $500 topping DACs and amplifiers, running their kefs, with amazon basics wiring thinking they have the best system money can buy.  I used to be angry about it but now I just feel sorry for them.  Sorry that they probably will never experience what a true hi-fi system can do to a person.  Great discussion here, OP

I wouldn't waste any time feeling sorry for them, just like they shouldn't waste any time feeling sorry for you. They're happy with their stuff, you're happy with yours. I'm happy with mine, which probably doesn't meet your standards or theirs. 

 

@analog_aficionado   Thank you for your post.  You have laid out a very logical and concise  reasoning that I have suspected for a long time but never had the technological chops to explain. 

I hope you become a more frequent contributor to this site.

ASR Review:
Over the years, I've read a small handful of ASR speaker reviews.  I've noted the following characteristics:

1. The primary focus of all the reviews were the measurement results
2. In spite of that, the essential imperfection of speakers means that the interpretation of these results is subjective.
3. Any listening impressions were brief and at the end of the review.  
4. All were based on hooking up 1 speaker only. I believe he only listens on-axis.
5. He may (perhaps always) apply EQ to the input signal and A/B the results.
6. The room (and likely, the equipment) used in testing is at his convenience.  For example, large speakers are tested in a sub-optimal room, rather than his listening room.
7. In the few reviews I've read, the listening tests were used to confirm the measurement results.
8.  I've not made an effort to see how non "domes & cones" speakers fare in his testing.

Conclusions:
It does appear that Amir is aware of and somewhat sensitive to the concerns regarding his review methodology.  If you value stereo imaging, his listening tests will probably be singularly unpersuasive.  I haven't (and probably won't) read enough of his speaker reviews to know if his measurement & listening results ever differ.

The testing methodology ASR utilizes is efficient, which enables them to review equipment much more quickly.  However, the knowledge gained is also limited by the process.  The results are thus most useful for the least discriminating.

I cannot understand subjectivist audiophiles or objectivists either...

 

 Acoustics measures by ears and tools matter first and last even over the pieces of gear design because any piece of well done designed gear is replaceable, The room and ears are not.

@rodman99999 Amen brother!

The folks on this site seems to give too much importance to ASR. Time to move on.

@prof 

Sorry, not calling you out on this, but you said "ASR evaluates equipment based on objective criteria that has been found to predict certain aspects of sound quality".

I was hoping that you could point to a link/post/discussion on ASR covering how the reader would correlate measurement anomalies directly with SQ issues?  This is 1 of the 2 big holes I find in reading ASR reviews, so hopefully it's covered somewhere on the site, in depth.

For example, in a Stereophile-style review, the listener might say something like "trumpet was a little spitty in its high register".  In the measurements section, that might be correlated with a slight rise in the on- or off-axis response.  In ASR, the measurements go first, so you get the same data.  In the listening section, though, you might see that Amir tried EQ'ing out the slight rise and liked or didn't care for the result.  No indication of how the listener would perceive the original issue.

The other big hole is imaging, which is ignored in the monaural listening test.  This has 2 parts - the first, alluded to above - due to the loss of the stereo image with only 1 speaker in play.  

There's also the speaker design - box vs planars vs horns vs OB, etc.  Each of these have significantly different radiating patterns.  Their optimum measurement results should also be radically different.

How is any of this information being conveyed to the reader?

Give me measurements or give me chaos!🤲

 

@hilde45 

I had a long and bitter debate with Ethan Winer about this. My position is that there may be things heard which cannot be measured because the brain and perception are way WAY beyond our understanding at this point in our scientific understanding. His reply to that was, essentially, "No, it’s just placebo effect and subjective bias." (In other words, y’all are just in denial.)

I recall there was a really enjoyable exchange between Jay Luong of Audio Bacon and Ethan Winer in the comments on one of his reviews; probably the power cable one. 

Summary off the top of my head is: Ethan said what you stated above, Jay imagined all of it, no proof etc. and also Jay had no scientific background. Jay responded he was an electrical engineer, Ethan said he must not be a very good one if he believed there were differences in power cables, Jay said he received a Bill Gates scholarship, so yeah he probably was a pretty good electrical engineer. Ethan did not have much to say after that. Quite comical. 

@jrareform 

The measurements aren't the problem with ASR.  the problem is the mob of people that pounce anyone that says "hey this is better even though it measures poorer"

Exactly. 

ASRs whole approach of anything that measures bad sounds bad, and the inability to describe how things sound or what people prefer sometimes - is a nonstarter for me. It all comes back to what people hear and what they enjoy most, and no metric tells us this very well.

Agreed. The point is not that "we should adopt the ASR Method." but rather we should avoid a false dilemma fallacy. Which both sides commit.

Measurements have their place. The measurements aren’t the problem with ASR. the problem is the mob of people that pounce anyone that says "hey this is better even though it measures poorer"....The sad thing is, the "happy panther" scale always rewards the highest measuring equipment because of the horde of stat hunters that are ready to say it’s better without hearing any of it.

My point is that people just being "against ASR and FOR listening" are throwing out some valuable data. I already made this point at length so I’m going to stop after this.

I'll add one more point -- people who focus on "how gear sounds rather than measures" frequently do not mention the rather complex effects of (a) other gear, (b) the recording, (c) the ROOM, and (d) their methods for listening. There is a very bad pseudo-science air in these conversations where it is supposed that the writer is conveying something that others would experience, but without any of the critical variables to help others know whether the claim would be something they can experience. How often do we hear "these speakers sound bright" and then we ask for a photo and find out they are listening with a bank of windows or a tile floor? It's this kind of thing that drives people to measurements even though those can be misleading or besides the point, too, but in a different way.

@hilde45 valid point.  Measurements are for showing what's measured (duh).  This can be invaluable when trying to match equipment to the room.  But yes, at the end of the day, our ears have to like the results.  I'm happy ASR exists for the purpose of looking at the objective measurements.  I'm unhappy that they are most likely steering newer audiophiles down the measurement rabbit hole, when they don't even have a clue yet what type of speakers and equipment they personally enjoy. 

they are most likely steering newer audiophiles down the measurement rabbit hole, when they don’t even have a clue yet what type of speakers and equipment they personally enjoy.

True, but... on ASR, a good DAC costs $200 whereas on Audiogon you might be made to feel like an ignorant peasant if you haven’t "invested" $1000 in a USB cable.

Then folks wonder why younger people are flocking to ASR

I think it has more to do with reinforcing what some hope to be true. 

Exposing the virtues of a $ 1000 USB cable isnt the same as making a person feeling foolish for not buying one. 

On a personal note, my brother an engineer, bought both Topping mono amps and the $ 350.00 Topping DAC and sent both back within the return period. The DAC fared better than the amps, but neither were very good.

 

This is the problem. Some of the brands, notably Topping, do not sound that good and have had quality problems. Yet say something negative about this holy grail and the whole of ASR comes down on you like a ton of bricks.

@devinplombier "Then folks wonder why younger people are flocking to ASR"

 

Ya know, you bring up a very valid point. The decades of going into audio stores in my region and listening to a wide array of systems is something I cherish to this day. Going to listen to other peoples systems, all great learning experiences. Most of this is gone any more - so, how do younger people learn and decide, today.

It’s easy to see why they are in search of data to try and make more informed decisions -vs- sifting through forums, or watching reviewers with opinions pedaling gear, then trying to figure out who to trust and follow about their hopeful audio purchase. And then, any remaining local dealers sell what they sell too. I guess all of us need to be mindful about what we post, and who’s reading.

 

 

People here get their pants in a bunch over ASR, its contributors and its methods. You would think they were doing something illicit. Their methods are explained and disclosed, their results are published, the results are open for comment. As that goes, there is nothing objectionable. I am not sure the ranking differences mean all than much as concerns SINAD numbers between closely-rated devices. Some of their speaker ratings, when done correctly, provide useful indicators as to which speakers can be expected to perform well in on and off-axis listening. They don't make useful comparisons of other features of a particular device aside from SINAD, Spinorama and frequency response. They do point out response irregularities and comment whether the deviations are likely to be audible. The commenters are a mix of thoughtful and knee-jerk, and you have to filter out the latter.  I don't agree with the wine analogy in the OP, but if you were to extend the wine analysis to things like acidity, sugar content and tannins content you might have a better comparison. Unlike other reviewer outlets, ASR isn't heavily supported by advertising like most of the audio print media. They have posted unfavorable reviews of generally successful if not popular products and have given good reviews for little known and inexpensive new products. What I think bothers readers here about ASR the most is their objective methods and general lack of favoritism, which forces those who disagree into the inherently weak position of attacking their methods.

This post gets at a real value for both A'gon and ASR -- objectivity:

Unlike other reviewer outlets, ASR isn't heavily supported by advertising like most of the audio print media. They have posted unfavorable reviews of generally successful if not popular products and have given good reviews for little known and inexpensive new products.

Avoiding marketing and forum hype is a good thing. ASR does it their way. Some do it on A'gon with their own experiences or counterarguments.

These two posts get at a problem with over-enthusiasm about "measurement" and the religious fervor of some at ASR:

I'm unhappy that they are most likely steering newer audiophiles down the measurement rabbit hole, when they don't even have a clue yet what type of speakers and equipment they personally enjoy. 

The decades of going into audio stores in my region and listening to a wide array of systems is something I cherish to this day. Going to listen to other peoples systems, all great learning experiences.

Both comments point to the fact that an overweening reliance on measurement tends to keep people from experience as critical to learning to listen. Trusting one's ability to take time to listen, notice, feel, and connect with music is the key to good audio, and when rankings and measurements displace that, we bind ourselves to a technocracy that shoves experience (and value) aside. This doesn't mean that measurement cannot help experience, only that it needs to be watched as carefully as an open flame.

chenry

... What I think bothers readers here about ASR the most is their objective methods ...

There is nothing inherently objective about measurements and the ASR site is dripping with bias and disdain. I don’t take ASR seriously and I’m not fooled by the use of "science" in its name.

the religious fervor of some at ASR

No such thing at Audiogon. Right?

Well I guess since Amir admits he listens to less than 50% of the products he reviews, those who believe a proper review can be conducted without listening, should be comfortable with ASR's methods. To me even the suggestion of a review based simply on measurements should cause even the faithful to run for the hills.

 

@cleeds

Exactly - ASR is ridiculous biased - biased for Topping, biased against pricey equipment. I had two different Topping DACs - both were DOA so please measure that POS. 

@chenry

"the results are open for comment. NO! They are not open for comment. If your comment does not match their philosophy you are thrown off.

 

Here's how you know ASR is borderline useless. There'd be a new product, and the ASR people would dissect and read all the data, and then something magical happen.

They all wait for subjective user experience.

If your scientific measurements are not reliable, then what are we even doing here?

Point at a random well measured speakers, the fact that no one can confidently say it will undeniably sound good by just looking at the data alone, that is troublesome.

The fact that ASR has rankings for speakers is troublesome.

Some speakers are better at certain things. Bass, mids, vocals, treble clarity, separation, imaging, 3d holography. 

Just shows these people only listen at numbers and graphs. It's fine, if you get pleasures from listening to graphs. The world still spins.

 

@samureyex

 

Your post is littered with a number of dubious, assumptions, or claims.

 

I was very careful in what I wrote.

 

As I said measurements are useful and so far as they have been correlated to their Sonic consequences.  Sometimes this can be complex to predict.  Sometimes not too complex to predict.   If you take two loudspeakers that have a generally neutral frequency response, except that one has a 5db rise between 3 to 7K, you can bet that unless you have hearing damage, you are going to hear that as added brightness, exaggerated sibilance for female singers, etc.  There are all sorts of subjective consequences that can be well predicted from measurements. If that were the case then people who work in sounds like myself couldn’t even manipulate sound with any predictive results using EQ. 

 

Further, I don’t know if you are aware of the amount of research on listener preferences for loudspeakers.  See the work of Floyd Toole and others.

 

In controlling for sighted biases - double blind testing of loudspeakers - certain speaker measurements are highly predictive of what the vast majority of people will identify as “ sounding best.”  Neutral on axis response, low or in audible resonances, and well-controlled, even, slowly sloping off axis response.

 

So yes, there are measurements that highly correlate with what most people will rate as good sound,  when other non-biases are controlled for. 

 

Once you open it back up to sighted listening, things are less predictable, but in test where only the audio factors are decisive, measurements are very predictive.

 

Likewise, measurements can tell you whether you’re going to hear a difference or not.  Human hearing and audio have been studied for over 100 years, and we really do have a grasp on lots of things in terms of what are audible or not.

There are thresholds of distortion below, which you simply will not hear any difference between a piece of gear, such as amplifiers.

 

That’s why nobody has been able to show they can tell the difference apart in blind tests, between amplifiers that measure with distortion below the known audible threshold.

 

Now a problem, of course is going to be the divide between the “ golden ear subjectivist” and those who recognize the relevance of measurements and science.

 

If you believe that the most reliable method of evaluating gear is your own subjective impressions, without any sort of controls for bias, then we are going to remain at impasse.  You can always claim, or believe, that you hear certain things. That’s how human bias and imagination works.  

 

And this is where we arrive at the common claim from audiophiles “ but what about all that gear that measures amazing but sounds bad? That proves that could measurements don’t tell us the real story!”

 

Such claims, unless you or anybody else has actually verified those situations where you are “ not peeking” (you don’t know which gear you’re listening to and that’s been carefully controlled for) , then this is just in the real realm of your own anecdotes, and not something that you can simply claim as the truth from which to argue from.

 

So when you say that the Topping DAC doesn’t sound good… I don’t see why I should take that as a fact establishing the matter.  If it’s the best measuring deck, Amir has come across, it is not going to be producing any spurious audible distortion that would make it sound bad.  And you are unlikely to be able to tell it apart from another DAC that would have similar levels of low distortion.

 

But again, this is where we reach the impasse.  If you believe that, nothing continue that your ears have got it wrong, and you are not open to being wrong about that, there’s not much I can say.

 

Of course there certainly does remain areas where differences and how things measure will amount to audible differences, in which we can have different preferences.  I’m a fan of tube amps and I find the audible distortion with my amplifiers sounds “ good.” Somebody else may prefer a solid state amplifier.

And yes, ultimately, audible differences will come down to different preferences.

 

 

@rodman99999

LIKEWISE: no one can possibly know whether a new addition (ie: some kind of disc, crystal, fuse, interconnect, speaker cable, etc) will make a difference, in their system and room, with their media and to their ears, without trying them for themselves.

That’s simply false.

In many cases, we can absolutely know that there will be no audible difference made by some tweak, cable etc.

To think otherwise is basically scientific and engineering illiteracy.

@petaluman 

What I meant there was that Amir Generally evaluates

Loudspeaker measurements based on the type of criteria derived from Blind loudspeaker testing,  Which are known to predict high preference scores.

And other words that is there “ Best practises” Reference in terms of speaker performance.  
 

If a speaker departs from this Amir Will try to Still describe the sound, 

And depending on how things work out, he might still give a speaker a pass.

But as Somebody who likes subjective reviews myself, I agree with you that

Amir’s Subjective portion is a bit too paltry for my taste.

 

I was also referencing that Amir when evaluating electronics tends to do so by referencing distortion levels generally known through testing to be in audible or not.

@prof 

To quote from my previous post - "the speaker design - box vs planars vs horns vs OB".

You'll have no problem in any gathering of audiophiles finding people who prefer Magnepans or Klipschorns or Linkwitz over box speakers.  These speakers and many others differ in basic method of sound production, on- and off-axis frequency response, radiation patterns, impedance curves...  All of them have their fans.  They all differ quite noticeably in the way they sound.  Any set of measurements that would match well with one of these designs would likely produce poor results with the others.  Based on your argument, knowing Amir's specific measurement criteria should tell us what kind of speaker he prefers (subjectively).  You can't simply average all the speakers together into a Frankenspeaker.  So, has he ever divulged his reasoning for the measurements he makes and how he established their usefulness?

petaluman:

Amir has explained his choice of speaker measurement, the Spinorama, because of its closeness to the measurement in an anechoic chamber. There are explanations giving details of the setup on the ASR site.

He has posted his main system. He has used Revel Salon-2 floorstanders in that system.

I don't often read the site. There seem to be many there who prefer KEF monitors, and a fair number who like powered monitors from Genelec, Neumann and Dutch & Dutch.

 

And here’s another big problem with their methodology…THEY HAVE GOTTEN THE MEASUREMENTS WRONG. I’ve personally seen it, and frankly , it was embarrassing. If that’s your ONLY acceptable criteria and you screw it up…well what good are ANY of your conclusions and how can you be trusted?

@jonasandezekiel 

Would you care to elaborate on what it is you have seen?

I think the wine tasting analogy is a pretty good one. That said, how about the Absolute Sound Magazine approach which is essentially the opposite?
 

Steven Stone’s & a few others reviews of lesser expensive stuff are often very useful & compare the item being reviewed to other similar products which can offer a practical & helpful information to better understand the pro’s & cons of the equipment & how they might compare to what you own or are considering. 
 

Most of the remainder of their reviews of many really amazing products are simply glorified, extended advertisements & offer very little objective evaluation. Take for example their recent review of the Thiele Zero Tracking Error Turntable. It’s probably a very fine sounding turntable but it’s never compared directly or even from memory to anything else in its price range. Additionally & maybe more importantly, it’s never discussed or even mentioned what would be the potential benefits of zero tracking error & or they were heard in listening to it. That would have been useful & informative. 

At least Stereophile for the most part objectively tests stuff. It could be very useful to know if an amp, for example & generally a tubed one, puts out its rated power w/ less than a few % distortion if you’re considering buying it & sufficient power for your speakers is a question. Of course, the true test is listening to something in your system but that’s not always possible or practical. 
 

Objective & subjective reviews & information are both useful & have their place. 

Oenology does exactly that. With the difference that oenologists mainly work in the audio… sorry wine industry. The person representing the audio reviewer, would be called a sommelier. The ASR bunch are certainly not in that business. 

Common sense science and real experience indicated that the ASR set of measures cannot describe the sound experience of speakers nor the subjectivist evaluation in a living room...

 

 Common sense and acoustics science with experiments said so...

Objectivist are ideologue  as prof just confirmed above and subjectivist are deluded in their own way...

Acoustics  set of conditions and parameters  rules...

Check out the photo of Amir's reference system. See any problems? 

 

@audition__audio do you have a link?

@devinplombier 

There were two occasions. They were both some time ago. One was a few years ago when I was looking at processors, and I wanted to check out strictly measurements, so I can’t remember which one. I remember they flubbed one crucial metric, and it changed the ranking of that particular processor.

I was floored, and I basically thought I can’t trust them for anything.

I am neither an ASR hater nor lover.  I think that objective measurements have a place in reviewing stereo equipment.  Certainly JA adds objective measurements to the subjective review.  But I think that the OP's analogy is particularly apt in demonstrating the role that subjective listening plays in audio equipment.  I know that I have never bought or not bought something based on measurements rather than listening.  I would imagine that it is not uncommon for something to have crappy measurements and not sound particularly good either, but ultimately, it is how something sounds in your system, and not strictly objective measurements.