Marco.org

I’m : a programmer, writer, podcaster, geek, and coffee enthusiast.

Consumer Reports

Consumer Reports announced today that they approve of the iPhone 4S, after famously refusing to recommend the iPhone 4 because of that antenna issue that turned out to affect almost none of the millions of people who bought it.

But the iPhone 4S was defeated by three Android phones in CR’s scoring criteria:

These pluses were not enough, however, to allow the iPhone 4S to outscore the best new Android-based phones in our Ratings. Those top scorers included the Samsung Galaxy S II phones, the Motorola Droid Bionic, and several other phones that boast larger displays than the iPhone 4S and run on faster 4G networks. […]

Other phones that topped the iPhone 4S include the LG Thrill ($100 on AT&T), which has the ability to capture stills and videos in 3D, as well as display them on its 4.3-inch 3D display, and the Motorola Droid Bionic ($300 on Verizon), which also has a superb 4.3-inch, high-resolution (540 x 960) display, with excellent keypad readability under most lighting conditions, even in bright light.

I’m looking at their full test results (I’ve been a CR website subscriber for six years), and I’m really not confident in the metrics and priorities that they seem to be using. Even some of the measurements seem suspicious to me:

Consumer Reports is a good place to get an overview of objective specs and measurements for a product category. I’ve often consulted their ratings to guide many purchases, mostly appliances, without any major regrets.

For their ratings to be useful to my purchase, their priorities and criteria need to approximately match mine. This is easy for most of the products they review: most people want a dishwasher to be able to quietly, effectively, and reliably wash dishes. A dishwasher that’s quieter is objectively better than a louder one. An air conditioner that uses less energy for the same cooling is better than a less efficient model. You can assign numbers and scores to factors like these.

Smartphones have too many subjective criteria, and even the measurable stats don’t always yield a definite answer on what’s better. If you want a huge screen, you’ll get a huge phone, so is a larger screen size a good thing or not? Fast 4G network access kills battery life, so is 4G a good feature for you? Do you want the best normal camera, or a lower-quality 3D camera? Do you want any particular apps or games that are only available on one platform? Do you need a kickstand? (Do they still make those?) These all depend on your priorities.

A product as complex and multifaceted as a modern smartphone is beyond Consumer Reports’ ability to rate in a way that’s useful to most buyers.