i can only speak from personal experience. My products have received good reviews in Stereophile without advertising with them. Conversely, I am unable to get my products reviewed by The Absolute Sound.
Publication bias and confounders in product reviews - TAS, Stereophile, Audiogon, etcetera
Folks-
Since I am a research professor at a major medical school in the U.S., I am used to identifying and using statistical measures of such bias in scientific research.
In Japan, I have read that a product reviewer who writes for magazines or websites are paid fees by manufacturers. I have noted that a similar thing may be happening here in the U.S., both reading TAS, Stereophile, etcetera, as well as noticing comments from individuals on this and other websites, many of whom are also dealers of these products.
As an example, I am somewhat of a computer nerd and have been downloading high-resolution audio files for almost a decade. That being said, I have been looking to buy a relatively high-end SACD player for my large collection of CDs and SACDs. I have noted the following:
1. There are few-to-no reviews of DCS players (e.g., Puccini SACD player, somewhat outdated but can be upgraded) and almost no published U.S. reviews of the Marantz SA-10 SACD player that was released about a year ago. In contrast, SACD/CD players including those from Esoteric, Hegel (CD only), Ayre, PS Audio, MBL, and other brands commonly appear in formal reviews, which are all favorable. Does this mean that products which have been reviewed but which are not well-liked by reviewers are not published?;
2. Comments in this and other forums mention that one or another SACD player or other product "must not be that good because they appear often as used equipment for sale..." or something to that effect. This observation may be valid, but could easily be confounded by the number of such products that were, or are, available for sale. The greater the number of products, the greater the likelihood they will appear as used items for sale - it says nothing about the quality of the product. I like to call this the "Ferrari effect", as this manufacturer intentionally limits the number of cars of any model for sale, and the company often only sells to individuals of affluence and/or have purchased cars from them in the past, artificially inflating the value of these cars;
3. Odd statements about the interesting MQA file format, part of a larger problem of a lack of objectivity in the audiophile community. Recently I read in a publication - "MQA is to conventional audio what quantum mechanics was to classical mechanics" - Really? Does this individual know anything about physics? Or am I taking this all too seriously?
I guess I am asking about the degree of bias in these reviews, to what extent are products reviews influenced by the manufacturers and dealers, and where is the objectivity in this domain?
Thanks for listening to my ranting...Gerry
Since I am a research professor at a major medical school in the U.S., I am used to identifying and using statistical measures of such bias in scientific research.
In Japan, I have read that a product reviewer who writes for magazines or websites are paid fees by manufacturers. I have noted that a similar thing may be happening here in the U.S., both reading TAS, Stereophile, etcetera, as well as noticing comments from individuals on this and other websites, many of whom are also dealers of these products.
As an example, I am somewhat of a computer nerd and have been downloading high-resolution audio files for almost a decade. That being said, I have been looking to buy a relatively high-end SACD player for my large collection of CDs and SACDs. I have noted the following:
1. There are few-to-no reviews of DCS players (e.g., Puccini SACD player, somewhat outdated but can be upgraded) and almost no published U.S. reviews of the Marantz SA-10 SACD player that was released about a year ago. In contrast, SACD/CD players including those from Esoteric, Hegel (CD only), Ayre, PS Audio, MBL, and other brands commonly appear in formal reviews, which are all favorable. Does this mean that products which have been reviewed but which are not well-liked by reviewers are not published?;
2. Comments in this and other forums mention that one or another SACD player or other product "must not be that good because they appear often as used equipment for sale..." or something to that effect. This observation may be valid, but could easily be confounded by the number of such products that were, or are, available for sale. The greater the number of products, the greater the likelihood they will appear as used items for sale - it says nothing about the quality of the product. I like to call this the "Ferrari effect", as this manufacturer intentionally limits the number of cars of any model for sale, and the company often only sells to individuals of affluence and/or have purchased cars from them in the past, artificially inflating the value of these cars;
3. Odd statements about the interesting MQA file format, part of a larger problem of a lack of objectivity in the audiophile community. Recently I read in a publication - "MQA is to conventional audio what quantum mechanics was to classical mechanics" - Really? Does this individual know anything about physics? Or am I taking this all too seriously?
I guess I am asking about the degree of bias in these reviews, to what extent are products reviews influenced by the manufacturers and dealers, and where is the objectivity in this domain?
Thanks for listening to my ranting...Gerry
64 responses Add your response
I've noticed a major change in the quality of reviews in TAS since HP departed for the Audio Valhalla in the Sky. Seemingly, every item in that publication is "superior" in some way or another. Perhaps that's true, but the wry comments, jabs and jibes I so appreciated from HP and his then peers are all gone. To talk about "objectivity" with respect to audiophile opinions is far fetched ,but the uniformly adulatory tenor of TAS reviews these days calls motives into question. The overwhelming amount of advertising between the covers may indicate a problem in that regard. Stereophile seems a lot better than TAS. I prefer it anyway. Owner reviews may be tainted: who wants to readily admit to having purchased a stinker? I've bought a couple, but was hesitant to admit the mistake when I actually had the bothersome component in my own system. This forum is always interesting and informative, despite potential ownership conflicts. |
My thanks to all of you for the responses, since I am naive in this domain, and expected audiophile magazines might actually provide unbiased product reviews. One last tidbit of information, since we routinely perform high-throughput data analysis on billions of sequence reads - the name "Diana Krall" is significantly over-represented in the discussions of musical content between 2015-2017, both on Audiogon and in TA and Stereophile magazines. I guess she is a popular artist and was also once a student at Berklee in Boston (like I was, a long time ago!) - Rip Van Winkle, aka Gerry. |
Reading reviews in the audiophile magazines should be considered information gathering, or general entertainment, and nothing like reading a scientific peer-reviewed journal. I have read lots of reviews; I haven't read any that concluded with something like, "Save your money, it's a bust." And surely, out of hundreds of reviews, based on statistical probabilities, it seems that, if reviews were being objective and transparent, that should have been the analysis on some piece of equipment. I have found the best source for reviews is Audiogon, when actual owners or former owners describe their experiences with different manufacturers and equipment models. The dealers on Audiogon need to clearly identify themselves since they clearly have a sales objective in mind, which by the way, I am all for provided they tell the truth about what they know and don't know concerning their equipment. |
| Post removed |
1. there are symbiotic relationships b/w the manufacturers and the rags 2. there are symbiotic relationships b/w the manufacturer’s and the readers of the rag, and 3. there are symbiotic relationships b/c the rags and the readers. = clusterf*ck. The truth lies somewhere in the middle of this relationship triad, which is why we still speculate about it ..... lol. |
Along with biased reviews in this forum and others, you will often find the warning that to determine whether a piece of audio gear is right for you, you must listen to it in your own system, preferably for an extended period of time. That is the bottom line in this pursuit. I know of no way around it. Audio is an extremely subjective pursuit and there are no two combinations of components, rooms, hearing and tastes that are the same. So the words of others can be taken as no more than suggestions as to what gear you might want to research. Listening at the dealer or an audio show cannot assure you of what a component will sound like in your system. The only alternative that I know of is getting to know a good audio dealer and trusting him or her. This does work for some but again it’s not a certainty. |
There are two factors here. The speed with which products get reviewed. (Time of product launch to time of review publication.) And whether a brand/product gets reviewed at all. The frequency with which Musical Fidelity appeared in the pages of Stereophile a few years ago was a running joke in the audiophile community. Meanwhile, try searching for reviews of PMC speakers in those same pages. Partly it has to do with company policy, partly how well a foreign company is represented in the US and what kind of PR campaign they sponsor, partly on reviewers' individual preferences, partly on kismet, and probably 3 or 4 other factors as well. In short, it's hard to read anything into the patchy coverage other than it's undeniably patchy. |
| Post removed |
I guess I am asking about the degree of bias in these reviews, to what extent are products reviews influenced by the manufacturers and dealers, and where is the objectivity in this domain? In a nutshell, yes, most product reviews aren't worth the paper they are printed on. I let all of my glossy magazine subscriptions lapse many years ago. The only information to be found in these "reviews", or "ads" whatever you want to call them are photos and specifications. These magazines/websites are part of the marketing branch of high end audio. So yes, they would rather write no review at all than a negative review. They might bash one of the smaller, non-paying players once in a while just for appearances. Make no mistake, all reviewers are HEAVILY biased towards the manufacturers. Their job is to push product, an extension of the marketing department if you will |
I think your conclusions are a bit flawed. I can agree that TAS and 6 Moons are a pay to play, Stereophile is not in that category. John Atkinson, the Editor of Stereophile is a man of high integrity. I have met him, talked personally with him and he really adheres to the values that Gordon Holt put down for Stereophile in its infancy. I know of one reviewer, Myles Astor who is honest as the day is long and would never be biased towards a certain manufacturer. The SA10 only came out a year ago, so it is possible that the mainstream magazines will review them. There really haven't been too many cd/sacd players reviewed over the past couple of years as most people are going for music servers and streamers instead. |
Few to no reviews of DCS players? not sure you’ve been doing decent research then, the Rossini and Vivaldi have been reviewed at length on every mag and the network box is now getting a bunch of reviews https://www.dcsltd.co.uk/products/vivaldi-dac/reviews https://www.dcsltd.co.uk/products/rossini-player/reviews https://www.dcsltd.co.uk/products/network-bridge/reviews I would argue that in fact DCS is more reviewed than the others you mention DCS do however try to manage the second hand market in a Ferrari like manner and hence you find factory approved used items from them which come with after sales service and software upgrades as opposed to the private sale non supported ones, obviously the latter are cheaper but the buyer has the choice of which way to go As to the reviewers they all have biases which are not hard to spot once you spend time reading and listening. With so many on line opinions it’s not too hard these days to find someone who’s ears you trust Ps the Puccini was also reviewed by both the main mags as well https://www.dcsltd.co.uk/products/puccini-player/reviews |