Why the Product Review Process Scares Vendors
In a perfect world, product reviewers would buy the new products they're reviewing ... and test them completely independent of any interaction with (or coercion by) the respective vendors. But most publications don't have the budget to independently acquire the products they review, and therefore rely on the vendors to ship them the product for a loaner (opening the door for compromise to the integrity / complete objectivity of the review). Sort of like if Roger Ebert had to sit right next to the director of a film he was reviewing (during the screening).
In a perfect world, product reviewers would all have extensive, parallel experience in the their readership's industry. For example, reviewers at IT publications would have previously worked in sys admin or IT architect roles. But many product reviewers -- who comment freely on the efficacy of enterprise software, for example -- have never spent a single day overseeing an IT infrastructure, or faced the problems that said software / technology was designed to solve (let alone have the context of having worked with other technologies / approaches in that domain).
In a perfect world, review 'labs' would be mirror replicas of customer environments. And when, for example, a systems management tool designed to provision 1,000 Linux boxes (which matches the IT environment at many large enterprise users these days) was reviewed -- the reviewer's lab would have 1,000 Linux boxes. But most product labs are too resource-constrained to continuously acquire technologies to emulate real world environments.
It is for these types of reasons that it is often with extreme anxiety and apprehension that vendors participate in the product review process. There is a somewhat reasonable trepidation about the prospect of handing over a new product for review, knowing how severely the odds are stacked that the review scenario is going to be somewhat subjective and interpretive.
Matt Sarrel is a former reviewer at many reputable IT publications, including PC Magazine, eWeek, TechWeb, Intelligent Enterprise and YRB Magazine. His prior background as a Certified Information Sytems Security Professional (CISSP) puts him in the category of reviewers that *do* actually have deep experience in the end user context in which they review products. After years of product reviews -- and seeing what vendor behaviors were constructive / counterproductive in the product review process -- he went on to start Sarrel Group, a consultancy that works with vendors in all aspects related to the product review process.
I had the opportunity to speak with Sarrel today and hear his point of view -- from the reviewer's perspective -- on the types of *vendor* behaviors that are the most frustrating / counterproductive (I paraphrase Sarrel's feedback a bit, but the gist of it ...):
"The Beautiful Baby Syndrome" -- These are often products that vendors have been working on for 2-3 years, and they've invested everything they have in it. And they are absolutely not going to accept any criticism. Their product is perfect, and if you don't understand that as the reviewer, something's clearly wrong with you or your approach. The minute the vendor starts emphasizing that the product is not only different than anything we've seen, but perfect -- it's almost a guarantee that as a reviewer you're either not going to cover it, or that it will get a negative review. Vendors get so close to their product and so inundated with their own messaging that they start to believe it. But that unwillingness to acknowledge areas where the product could improve or is still evolving is a big turn-off to the reviewers.
"Heavy-Handed Guiding of the Process" -- There are many companies coming at reviewers in slimy ways. Not to name names [if you guessed, you would probably get it right on the first try though], but certain vendors and their PR firms come at the reviewers and won't participate if the reviewer isn't willing to test it 'their way.' There's some advice to give PR people -- you can subtly influence a review (especially when you give some basic context to the reviewer on an emerging technology and the meaning of certain features / functionality you've introduced), but if you try to manipulate a certain outcome, it typically backfires.
Next week, Mr. Sarrel will share some of his specific pointers for PR folks on the things that really *do* work well for vendors in preparing for the product review experience. In the meantime, I'd be interested to hear if any readers have any any particularly positive or negative experiences as the result of particularly positive or negative product reviews (how much does a great product review help your sales team, for example?).
Image courtesy of Guise Dugal's photostream on flickr.