Let me begin with disclosure. I am a reviewer. I am not going to defend or criticize what reviewers as a class do or discuss whether the reviews are in some appropriate sense 'objective'. I have my own approach to reviewing. I view what I do as a form of 'audio criticism' along the lines of 'art criticism' or 'dance or theater criticism.' My reviews of components are not designed to express a judgment or encourage or discourage purchases. They are designed to help explore issues in the distinction between sound and music, help elucidate key concepts in audio such as 'fidelity', 'representation', ' resolution',etc and to connect these concepts as they are used in audio to the role they play in other aesthetics. That's my framework. So just to give a simple example. Resolution in a painting or a novel is a way of describing a kind of completeness. Its opposite is unresolved as in an unresolved painting or novel: both of which fall short in some important way of being capable of providing narrative or cognitive content. In audio, resolution has taken on a quite different meaning that, from my point of view, is unconnected to anything musically valuable -- and much more connected to sonic attributes of a system.
My interest in this particular discussion can be captured in a couple of questions: what is it that readers expect from reviews? What is it reasonable for them to expect from reviews? What are the criteria for determining whether reviews are helpful, and helpful in what ways?
The first thing I would emphasize is that the entire 'high end audio' sector is a kind of ecosystem. It consists of a set of intersecting and mutually supportive (to a degree) organizations and institutions. Reviewing mediates in some ways between the sales side and the purchase side. The magazines have a role to play in this. That role imposes a very complicated set of responsibilities to both the readers/purchasers and the producers/manufacturers. These responsibilities do not create necessarily create conflicts of interest, but they cannot help but create tensions. The editors, and ultimately, the particular reviewers have to figure out for themselves a way of resolving the tensions that satisfies personal as well as objective criteria or integrity. The tensions can turn into conflicts of interest when the incentives that naturally arise are not appropriately resisted. That is why integrity matters as much as it does.
Readers have a right to expect commitment to overall high levels of integrity that reflect an understanding of the fact that this niche sector can survive and prove valuable to all only if the actors recognize that it is an ecosystem that requires a degree of appropriate support for others within it. This means that in general the review process has to provide some level of support for manufacturers and producers as well as useful information to readers.
No one should fool themselves into thinking that the cooperation necessary for the ecosystem to survive and provide value for all doesn't create incentives that, if not adequately resisted, threaten to undermine the values it is designed to provide
Just as lawyers have incentives (when possible) to search for favorable jurisdictions and judges for their causes, so too do manufacturers have incentives to search for 'favorable' outlets and reviewers. That is unavoidable. There are other incentives in place for reviewers and readers alike, not to mention dealers that can exacerbate the tensions and make one feel from time to time that something has gone terribly awry.
No one should deny this nor should anyone deny that it can be a genuine problem that the industry as a whole needs to address. The question is whether there are mechanisms in place that mitigate the impact of the incentives and their potentially perverse effects -- beyond reliance on personal integrity.
At this point, we have the advantage of there being so many reviewers and so many reviews to choose among. It's a greatly democraticize or distributed field these days, much in the way the fall of the giant record labels led to a more general and distributed set of 'curators', e.g. the Pitchforks of the world. Beyond that, all readers have to judge the integrity of particular reviewers and reviewers must take seriously their responsibilities to the audience.
Whether we see ourselves this way or not, reviewers are curators; at least that is what the vast majority of readers see us as (if not quite in those terms). It would be helpful if we saw ourselves in that way as doing so calls for adopting certain norms and reflecting not only on our role in the ecosystem but on our capacities to fulfill those roles well.
The pressure many reviewers feel to make judgments and guide purchasing decisions should not be underestimated. It comes as much from the readership as it does from the manufacturing and sales side. It's reasonable for both sides to want it. It is unclear to me whether reviewers are typically in a position to offer such judgments or for either readers or manufacturers to take the judgments as authoritative.
But that is OK because, at least from my perspective, purchasing judgments are the least valuable aspect of curatorship. Providing education is far more important, for in the long run education is required if the audience is to make individual choices that they believe are well informed and with which they turn out to be happy or (at least) satisfied.
I read reviews to discover what is new, to enjoy seeing the enthusiasm some reviewers have in sharing their experience with others, for information about the direction audio design is taking, and for the pleasure I experience from reading well crafted and thoughtful essays.
At the same time, I recognize that I would not enjoy reviewing if I did so feeling that at the end of a review I need to offer some putatively authoritative judgment about the worth or value of a particular component. More importantly, I would have little useful to offer if I did. Fortunately, for me, I have found something that suits my interests and competencies, that is true to my love of both music and audio and the role they both play in my life. I am blessed by the fact that some readers find what I do interesting enough for me to feel good about sharing my thoughts.
