The Audio Science Review (ASR) approach to reviewing wines.

People here get their pants in a bunch over ASR, its contributors and its methods. You would think they were doing something illicit. Their methods are explained and disclosed, their results are published, the results are open for comment. As that goes, there is nothing objectionable. I am not sure the ranking differences mean all than much as concerns SINAD numbers between closely-rated devices. Some of their speaker ratings, when done correctly, provide useful indicators as to which speakers can be expected to perform well in on and off-axis listening. They don't make useful comparisons of other features of a particular device aside from SINAD, Spinorama and frequency response. They do point out response irregularities and comment whether the deviations are likely to be audible. The commenters are a mix of thoughtful and knee-jerk, and you have to filter out the latter.  I don't agree with the wine analogy in the OP, but if you were to extend the wine analysis to things like acidity, sugar content and tannins content you might have a better comparison. Unlike other reviewer outlets, ASR isn't heavily supported by advertising like most of the audio print media. They have posted unfavorable reviews of generally successful if not popular products and have given good reviews for little known and inexpensive new products. What I think bothers readers here about ASR the most is their objective methods and general lack of favoritism, which forces those who disagree into the inherently weak position of attacking their methods.

petaluman:

Amir has explained his choice of speaker measurement, the Spinorama, because of its closeness to the measurement in an anechoic chamber. There are explanations giving details of the setup on the ASR site.

He has posted his main system. He has used Revel Salon-2 floorstanders in that system.

I don't often read the site. There seem to be many there who prefer KEF monitors, and a fair number who like powered monitors from Genelec, Neumann and Dutch & Dutch.

petaluman:

I am aware the small list of speakers I mentioned are "box" designs (stretching the definition for the Genelecs and the Dutch&Dutch) and am well aware of the several categories of speakers that are not of a box design. My comment was specific to Amir's own "reference" speaker--his Revels, and my general observation of the preferences of commenters on the ASR site appear to be. I do not believe I wrote anything to suggest I was unaware of the alternatives to box design speakers, planars, planar-hybrid, horn, horn-hybrid, open baffle, never mind the variants of box designs, transmission line, or for that matter transducer variations with ribbon tweeters, long-stroke woofers, field coil transducers, too many to list.The ASR site seems to sample whatever the readership sends for testing, and most appear to be monitor-size boxes, which is to be expected at those dominate the market. Occasionally a product from an audiophile brand is sent in only to be tested and found wanting, which then excites contentious responses.

I am sure if you were willing to ship Amir a pair of Magnepans for testing, he would oblige.

This post has acquired an interesting and persistent energy. ASR is clearly neither trivial nor unpersuasive or why would so many try to denounce it?

I occasionally go to ASR to read their reviews. Their contributors don't appear to be particularly reductionist or dogmatic. If you know what they are using for testing, you can take that as a data point and move on. Their peanut gallery in their comment sections are what you will find anywhere, people who opine based on the review and not anything more.

What no site appears to do is true blind listening tests using a standard setup for 2 channel audio, and using self-validating methods (e.g. testing the same system twice to look for variation of the listener's attention and judgment.)