Publication bias and confounders in product reviews - TAS, Stereophile, Audiogon, etcetera

The text may give some praise to satisfy the advertiser, but the important part is the measurements section.

Hahahaha, that is hilarious!
Thanks for my daily chuckle. ;^)

cj1965Oh puhleeeeeze. Spend five minutes of your time Googling Stereophile and MQA and you'll have all the proof of a "push" that you need. Nobody is going to waste time giving you proof of the obvious. If you're that lazy and you don't feel like looking for yourself at what most everyone here has seen and talked about - don't expect any of us to serve it up to you on a silver platter. It's what "objectivists " refer to as "common knowledge".

That is so funny first you so-called "objectivists" say that we have to proof what you think because what we think is so unusual now you say we have to proof what you think because it is so obvious you want it both ways! Why not just say "I am an objectivist I am a scientist I know what I am talking about you are lazy go find you're own proof for everything." That is just crazy.

The text may give some praise to satisfy the advertiser, but the important part is the measurements section. Those measurements may tell a rather different story. Sometimes, when the discrepancy is too obvious, John Atkinson will mention this, but at other times you really have to do the interpretation for yourself. It is not hard.
What is also obvious is that many of these pieces of expensive audiophile electronics perform far worse than much cheaper electronics from mainstream manufacturers. To find measurements for those you usually have to look elsewehere, since Stereophile does not normally test such stuff. But Google is your friend. See here for an example: http://www.avhub.com.au/product-reviews/hi-fi/yamaha-a-s500-amplifier-review-test-395710

It is often difficult to decipher a Stereophile review. As someone above noted, they
may damn a product with faint  praise.  Many of their reviews seem inconclusive or unenthusiastic - which usually means "there are better products out there".  When the review for a $25,000 amp makes it seem like it's merely adequate, something is not quite right.  They do seem to go out of their way to avoid conclusory judgments.  
 

+2

+1

@cj1965

" Well I just think it is funny that so many here seem to complain about the "big push" for MQA by the audiophile press but they give no examples of specific articles or authors but i guess it is like some of the other matters here on Audiogon where people make claims and then expect others to proof they're claims for them! " - Clearthink

Oh puhleeeeeze. Spend five minutes of your time Googling Stereophile and MQA and you'll have all the proof of a "push" that you need.  Nobody is going to waste time giving you proof of the obvious. If you're that lazy and you don't feel like looking for yourself at what most everyone here has seen and talked about - don't expect any of us to serve it up to you on a silver platter. It's what "objectivists " refer to as "common knowledge".

I did A/B on MQA Tidal stream vs. vinyl rip 24/96 from digitally mastered Adrean Belew "Mr Music Head"
MQA 24/96 stream was substantially inferior to vinyl rip.

Well I just think it is funny that so many here seem to complain about the "big push" for MQA by the audiophile press but they give no examples of specific articles or authors but i guess it is like some of the other matters here on Audiogon where people make claims and then expect others to proof they're claims for them!

I think it is also funny that people are complaining about what the audiophile press is writing about because after all they are magazines for audiophiles what do you want them to write about? Maybe what you want is for them to get into the topics you explore here like telling others' that they are deluded or snakeoil salesmen or matters of politics and economics would that be better for you??

It certainly applies to audio press in my country (the Netherlands) that there is a push for MQA. That push does not come with any proper explanation of what it is, or how it is  better.

twoleftears"You only need to look at the last few months' issues of Stereophile and Absolute Sound to see a multi-pronged and significant push for MQA."

I do not get those publications I do not believe in my country so it would be helpfull if some of those here who criticize this "push" would provide some examples of what they find objectionable because otherwise it is hard to follow your line of reasoning about the audiophile press.

You only need to look at the last few months' issues of Stereophile and Absolute Sound to see a multi-pronged and significant push for MQA.

cj1965

The recent, sustained push for MQA in the "audiophile press" has been a great example of how far they're willing to go to support a farce aimed at emptying the wallets of the gullible.

Do you have any examples of this "sustained push?" Without examples, it's difficult to know what you're talking about.

" Honesty is so far down the list of priorities it's seen as a mythical quality to all but the gullible. On a sinking ship integrity, because of its weight, is the first thing that gets thrown overboard. " -  cd318

" Hi-Fi journalism is akin to mainstream pornography. It can be easy on the eye, but bears little relationship to reality " -cd318

Spot on and hilarious - one of the best posts I've seen on this forum so far. The recent, sustained push for MQA in the "audiophile press" has been a great example of how far they're willing to go to support a farce aimed at emptying the wallets of the gullible. It's a clear window into the lack of "journalistic integrity" that exists.

It is gradually becoming increasingly apparent that user "cd318" has his own agenda that he is enforcing at the same time that he reserves the right to himself to attack others who disagree with him however a major difference is that we don't know what users' "cd318" interest is in the matter which makes it difficult/impossible for readers of this forum to meaningfully assess  his "opinions" on matters that he states here with such forcefullness. But I admit comparing Hi-Fi news to pornography is a little bit weird and makes me wonder what he is doing with his Hi-Fi News magazines but of course that is his solitary business and not for us to question challenge criticize or condemn so long as no one is being hurt it is fine.

cd318
The main purpose of most reviewers is self preservation. They pander ... Honesty is so far down the list of priorities ... integrity ... is the first thing that gets thrown overboard.

Magazine reviewers are ... misguided idiots to a man ... For decades they have lied ...
Hi-Fi journalism is akin to mainstream pornography ... if only the manufacturers weren’t such liars ...

Why so angry? What makes you such an objective observer of all things audio?

The main purpose of most reviewers is self preservation. They pander to certain manufactures and favour certain flavour of the month products. Honesty is so far down the list of priorities it's seen as a mythical quality to all but the gullible. On a sinking ship integrity, because of its weight, is the first thing that gets thrown overboard.

Magazine reviewers are almost entirely misinformed and misguided idiots to a man. Its a bit like selling soap, every year new improved products are launched...
Yet somehow the reference products of yore remain unsurpassed.

For decades they have lied about digital sources, they didn't have a clue about speaker isolation, recommending spiking ad infinitum, and most of the hacks can't write for toffee. Memorable reviews? Ha! You'd be lucky to find a dozen.

Hi-Fi journalism is akin to mainstream pornography. It can be easy on the eye, but bears little relationship to reality. Trust it at your peril. Now if only the manufacturers weren't such liars. Now if only we didn't like the lies so much....

+1 john_g, my feelings exactly. I was poised to print the same post until I read yours.

Tom

ps- don't forget Sam Tellig (Tom Gillette). He was my hands-down favorite, unfortunately he no longer writes.

I put in my vote for objective and professional reviewer REG of TAS. Everyone in the hobby should check out the articles at regonaudio.com

After reading both regularly for about three years now, my feeling is that Stereophile is more objective and professional in reviewing equipment than The Absolute Sound. Although the latter magazine occasionally publishes well-written reviews, sometimes it is impossible to tell whether the price quoted for a speaker is for one speaker or a pair. Stereophile is not uniformly positive, and points out when the item being reviewed doesn't work, or stops working (or blows up), and when it provides no discernible improvement in sound quality. They also give the manufacturer a forum to respond to reviews.

I have found that both magazines have helped me get educated on new stuff in the hobby after a 20-year hiatus. And in my opinion the best writers in the audiophile universe (Art Dudley, Herb Reichert and Michael Fremer) are all at Stereophile.

I had a job trading bits for pieces
We'd make wrinkles, advertise them as creases

and where is the objectivity in this domain?

Good question. There's little to none. My conclusion is if it's not blind, they didn't hear it. The only time I would pay any attention whatsoever to a subjective review is if that person continually came to the same results as I.

Regarding #2, you'll usually see an influx of items for sale when the next "hot" thing comes out. The herd is always chasing the newest rabbit. Doesn't necessarily mean people are dumping them for cause.

Regarding #3, there are plenty of technical articles showing why MQA is just a money-making scheme.

BTW, there have been blind tests on SACD, including an AES paper. I think you can guess what the results were. However, there was an audible difference in the noise floor when there was no signal and high system gain at a level where music would have been "unpleasantly loud".

In general, the audio press is placing more emphasis on streamed audio, which they see as the future, and less attention is being paid to disc spinners of all types.  Those of us with considerable digital disc libraries will likely find it increasingly difficult to access reviews of disc players.😟

gerryah930

I enjoyed your opening query. I do not view your illustrations as ranting in the least. I, too, am interested in the Marantz SA-10 spinner and it has been a very slow roll-out into the Audio press. This is odd indeed, as Marantz, tends to receive favorable coverage on its Reference products. Still, being hopeful that Stereophile, TAS or one of the other top publications will test and review this spinner in 2018. I am in the process of finding and auditioning. Based on your locale maybe you can do the same.

Happy Listening!

Many times, a reviewer asks to review a component that he (or an editor) heard at a show and with which he was favorably impressed. Although reviewers at the 2 big magazines rarely trash a product, a good review usually informs the reader about the qualities of the component and the characteristics of its sound to the reviewer's ears. It's up to us to listen to the components which appear from a review to meet our preferences and then go listen for ourselves.

@gerryah930 I do not think you are trying to incite et cetera.

@browndt Absolutely!  There are numerous manufacturers that for one reason or another are not members of "the club".  They good ones soldier on, on the strength of their product, word of mouth, etc., without a boost from S-phile or AS. 

Reviews for audio anything should be taken with a grain of salt. It is up to the individual to realize that a certain amount of product bias exists in all reviewers skewed toward their benefactors or personal biases. Their publication can be useful for a buyer to determine which direction he may want to try but should never be used as the gospel truth. Audio is a very personal matter and a 100000 dollar system is no better suited to propel the individual human into audio nirvana than a 1000 dollar system. The human condition determines when nirvana has been obtained, not reviews. All of us in these forums have reached nirvana at some time of our continuing search for a better nirvana without reviews simply by trial and error. We still do it. Just look at all the equipment ads. We are our own very best reviewers. All different and yet the same. One persons failure to enter their nirvana opens the gates to nirvana for someone else.
Isn't it wonderful? 

Good read, so far....and that despite the Cable Drifting : )

I'll add to @williewonka 's point:

...have you considered that products that are reviewed are actually pretty darn good...

In my view, most gear that makes it to market is very good, especially when considered on it's own. That in itself eliminates highly 'negative' reviews. 

I'm searching my memory for a component that I've had in system that I would consider a turd on arrival and which remained so.....

This includes speakers which are quite variable (as you know) and some  tube components (early in my journey) that were overly euphonic. For both component classes it was not that they were poor performing but rather that I didn't care for their sound.

For me, reviewers offer a great service that I am grateful for regardless of bias or vested interests. It's the way of the world, even the medical and science ones. Reviewers I have personally interacted with have been no different than my audio friends and acquaintances. A pleasure.

I can also understand, if that's how one chooses to roll, ignoring their reviews. I, however, find the toxic mudslinging unseemly. 

browndt
No negative reviews, in my view, means no other review can be trusted. I know collusion when I see it and that is exactly what stereo magazines and manufactures have done for years. Any objective scientific review will have negatives. I should know I have edited scientific publications for years, where lots of critique can be found.

>>>>Actually, many if not most reviews have always included negative aspects, if any, as well as positive ones of the device under test. For example frequency response curves and radiation patterns, distortions of various types. The idea they would Gild the Lilly 🌷doesn’t really hold water IMHO. In fact, one reason they provide negative aspects is too prevent or dispel the impression that the reviewer or magazine is biased or that there is any sort of collusion. They aren’t stupid.

No negative reviews, in my view, means no other review can be trusted. I know collusion when I see it and that is exactly what
stereo magazines and manufactures have done for years. Any objective scientific review will have negatives. I should know I have edited scientific publications for years, where lots of critique can be found. 

So, caveat emptor when reading about and buying stereo gear.
Actually I prefer to hear form actual users on Audiogon about specific products. They are ofter much more critical. 

Finally there are a few really good companies who tell the truth.
Example, Heron Audio. How many reviews have you seen on their products. My guess, they are outside the "Club".

It's an advertising/awareness thing. That which is reviewed garners sales, pure and simple.

The point is that frequency of coverage, or indeed coverage at all, in the mainstream mags is no indication of anything really.  Certainly not of a component's quality or popularity.

Re Wireless World report:

“The listening panel were all well known and experienced listeners.”

I’m sure. As I’ve said repeatedly you don’t have to look too far to find a test that’s inconclusive or even negative. I suggest throwing the whole test out. 

It’s not really an elephant in the room. 🐘 It’s a nothing burger. 🍔 And there’s nothing ordinary at all about most high end cables, from the purity and crystal structure of the metal, to the controlled directionality of the cable or power cord, to the dielectric material, to the method of welding and particulars of the connectors, geometry of the conductors, etc. The longer this cable debate goes on the weirder it gets. 😳

Once upon a time there were serious publications. See here for a famous article from Wireless World that should still be compulsory reading for obsessive audiophiles: http://www.keith-snook.info/wireless-world-magazine/Wireless-World-1978/Valves%20versus%20Transistor...

Reviewers, and audio magazines in general, are built around the idea of only reviewing gear they know to sound good from their own listening at shows and at various other locations and locales.

It would be economic suicide for any published magazine to review stinkers, and trash them.

It’s a very simple bit of economics as tied to the human condition.

Stereophile has said this flat out, and said it fairly often.

No negative reviews as negative reviews are never allowed to have a chance to form in the first place. Everything reviewed is filtered on multiple levels, before the given review is published.

The best you’ll get (toward the idea of a negative review) is to have the given item ’damned with faint praise’, if the selected item they thought would sound great and review well, does not quite meet that high quality minimum.

It does not mean that smaller publications or non paying (no advertisers and no paying to read the reviews) scenarios where equipment trashing is done...are somehow more honest, it’s just that there is ZERO, I repeat ZERO economic position available for anyone trying to run a company (of a publishing nature) that produces only (or high levels of) negative reviews of commercial products that are in this area of the economy.

Importantly, Stereophile additionally states -far more often than they speak on only publishing good reviews-..that if they don’t review something...that does not necessarily mean it is bad or not good. There’s lots of gear out there and they can only review a very very small percentage of it. They state this openly, regularly in the magazine, about 2-3 times per year, one in every instance of ’recommended components’, and in other places and ways.

There is a film called the ’American ruling class’, where the ’fictional’ main character in the ’pseudo-documentary’ interviews the actual and real ownership and publisher of the NYT (at the time). (everyone interviewed in the film is for real) He asks a real question and gets a real answer. The editor says they only publish editorial that is favorable and in line with their advertisers and that’s the way it is done. Period. Any other way and the newspaper would simply not exist. The end.

Now, let’s talk about the US medical research INDUSTRY, that + trillion dollar medical/pharmaceutical juggernaut’s ’factualization’ arm.

Additionally, if the audio world was about 2-3-4 times bigger than it is, then publications could get into the idea of a more ’biased toward negative’ review standard. Then a world with audio magazines with some negative reviews could stand and survive.

Also audio is specifically NOT a ’first past the physical post’ type endeavor. Where everything is in full black and white factual norms. We still don’t have a full and correct handle on how people hear and understand audio quality, nor how distortions and micro-distortions interfere and integrate with our not yet understood hearing functions.

We can send out the exact same audio cable to four different people and get back four very different assessments of how the cable sounds.

Eg, the post right above this one illustrates the complexity of the audio world and marketplace, quite well. It’s a mess that has no zeroing point or norms that can be discerned beyond a basic direction in desires. Sometimes.

We’re dealing with mental wiring and associated system which are all INDIVIDUAL in end points. There is NO capacity to perfect anything on paper or in the so called real world of audio.

If you want to normalize all individualism (hearing and associated wiring, intelligence, etc) down into a black and white repeating standard and give away all the things that make you--you ..and kill all of humanity down to a repeating widget that comes off an assembly line of repeating identical items..and do the same for everything you know in the human world ---basically kill off humanity and the world into a dead, non-living nothingness..THEN.. you can have your black and white perfection in high end audio.

You missed the elephant in the room. Ordinary bits of wire. Not long ago, bits of wire were correctly viewed as bits of wire. Today gargantuan sums are paid for ordinary wires in the misguided belief that ordinary bits of wire do more than anything else in the audiophile’s setup. The fact is they don’t. What they do is lead to massive profits and mark ups for something cheap with no substantive benefit to the user. Audio reviewers and press have jumped on the band wagon (incentivized by advertising) and stores have jumped on the opportunity to sell additional cable trinkets and jewelry at extremely high mark ups to buyers.

The advertising lie is that “everything affects the sound” and that “exceptional ears and exceptional systems will require these trinkets to sound their best”. A story just like the emperor’s new clothes which audiofools eat up readily. Smugly the audiofools look down on anyone with less than $1000 bits of wire and snear at those that don’t hear a difference as being deaf or not having a high end system. The whole domain is like high end skin care - packed with excruciatingly expensive products that do nothing more than appease the ego of the vain. The bits of wire are made of fairy dust meme also undermines real audio innovation.

Are you at Johns Hopkins? My daughter is studying there currently.

I am not trying to insight a so-called flame war here, but I have to agree with willemj.

If audiophile magazines are "hobbyist" journals, then I would stop the product reviewers from including what appear to be plots of measurements of impedance versus phase, spectral decay, and amplitude versus frequency. I enjoy looking at these graphs but in almost all cases, the reviewers will only hint at issues (e.g., speaker impedance spikes or drops), but the review will end with statements such as "best speakers I have heard, so I bought the review pair" and so forth. I can only believe that there is some economic incentive that is not disclosed.

Comparisons to a hobby and a profession:

1) I used to be involved in Porsche club racing, so I would read porschephile and generic car magazines. This would certainly be considered a wealthy person's hobby, but the car magazines would provide head-to-head comparisons of performance between brands and would make disparaging statements about cars that were poor performers. I have never read head-to-head comparisons in a given category of hardware in any high-end audiophile magazine (e.g., different speaker brands in the same room at the same time, driven by the same audio hardware);

2) In biomedical research, there are many problems and challenges, but to suggest that our collective intent is to do anything but try to arrive at so-called "ground truth" is not accurate. That does not mean there is no publication bias (see [1]), but at least there is recognition that this issue exists, and many attempts to deal with the problem.

- Gerry

[1]  A. César-Razquin et al. A call for systematic research on solute carriers. Cell. 162 (3), (2015), 478-487. https://www.snijderlab.org/pdf/26232220.pdf 
 

Another reason why reviews are almost always favorable is because the magazine and or reviewer doesn’t really want to kill a small company, especially in view of the fact that sometimes things happen, you know, such as damage during shipment to the reviewer, operator error, failure to follow instructions, that sort of thing. Also, sometimes there are trade-offs for the device under review which would be spelled out in the review. For example, cost, size, complexity, peculiarities, etc. Finally, reviewers and magazines are not all the same. Reviewers and magazines can certainly have unique or different approaches to conducting reviews. It certainly wouldn’t be unusual to see products reviewed that are trending or that the reviewer and or magazine would suspect of having interest for their readers.

Gerry,
Drive the car. Listen to the gear. It's all personal.
Kenny

@gerryah930 - have you considered that products that are reviewed are actually pretty darn good?

And perhaps - that's why the review is (almost) always favorable?

After reading lots of reviews, I have found the I can generally tell when a product may be falling a little short of the reviewers expectation - they "generally' start to nit-pick.

And the opposite occurs also - if the product is fantastic the reviewer goes out of their war to extol the virtues of a component.

Sometimes "numbers" are not enough - you just have to learn to "read between the lines".

As for Diana Krall...
- a great many of her albums are a product of exemplary sound engineering/recording and final mixing and more often than not lacks that "wall of sound", which allows a reviewer to focus on the details in the recording.
- As opposed to another delightful singer, Norah Jones, who's albums are also very well engineered but are quite varied in their production and can sometimes lack the more natural tones of a Diana Krall album.

These observations are result of hundreds of hours of listening I my self have conducted whilst reviewing (as an amateur) a small number of products.

Unlike many, my reviews were not paid for, but I was enthused by the outstanding abilities of those products.

I'm not defending professional reviewers
- but there is always another side to the coin

Having experienced just how difficult it is to write a review that is engaging, informative, concise, insightful and more importantly useful to the reader, I think it is important for people to consider some of the "intangibles" at play.

BTW- do you collect stats on how many products are returned without a review?

Regards - Steve

willemj

If the methodological standards common in audio publications and audiophile forums applied to medical science life expectancy would be halved.

That's a false comparison. Audio publications are hobbyist magazines, not scientific journals. And there's plenty of nonsense in medical science, too.

roberjerman
The two "Peters" (Aczel and Moncrieff) made a determined attempt to avoid "favoritism" and "commercialism". Check out The Audio Critic (TAC) and International Audio Review (IAR). Listening combined with measuring for honest and forthright assessment of gear! Nothing since has been at the same level (though Brit publication Hi Fi News does try for " objectivity " - but is still bound to its advertising base).

There is a fine line between “honest and forthright assessment” and closed mindedness and self aggrandizement.

“What I have learned after six decades in audio.” - Peter Aczel

Oh, please!

If the methodological standards common in audio publications and audiophile forums applied to medical science life expectancy would be halved.

The two "Peters" (Aczel and Moncrieff) made a determined attempt to avoid "favoritism" and "commercialism". Check out The Audio Critic (TAC) and International Audio Review (IAR). Listening combined with measuring for honest and forthright assessment of gear! Nothing since has been at the same level (though Brit publication Hi Fi News does try for " objectivity " - but is still bound to its advertising base).

Go listen to the DCS.....