Samsung’s benchmark rigging proves we need a better way to judge smartphone performance

Samsung is currently under fire for optimizing its newest devices, the Galaxy Note 3, to give it ‘performance gains’ during benchmarking tests. Ars Technica made the discovery, and has a comprehensive account of the difference it made.

In basic terms, this means Samsung has adjusted the phone so that it recognizes benchmarking software and knows to run flat-out when it is being put through its paces. Benchmarking is commonly used by many tech media and analysts as a way to compare devices, showing which phones or tablets perform fastest across various categories and settings.

Many critics are calling Samsung’s actions cheating because the Note 3 does not run this fast in ‘real world’ conditions — it has battery-saving software and over-heating safeguards that kick in and prevent it from over-performing with any single app or activity.

Their argument is that Samsung and others — for the company is not alone in benchmark optimizing – do this so that their devices score better in the tests, garners more impressive media coverage and are thus more attractive to those who are looking to buy a smartphone.

But here’s the thing, the storm being whipped up about this issue is totally overblown.

User experience comes first

Yes, Samsung is being deceitful to a point, but it’s not like the scores are artificial or entirely fake, as they come from the device itself. A more pertinent issue here is really how much emphasis is placed upon data, as people elevate the importance of benchmarking in particular

I’m not saying that benchmarking isn’t a worthy pursuit, but its significance is too often overemphasized — perhaps by us technology geeks in the media, or those who like charts and statistical comparisons.

The fact remains that a phone is about the user experience, and switching a few settings to generate “small gains in performance” — to quote Anandtech — is really besides the issue. Support for apps, a fluid user experience, the potential to upgrade, price and plenty of other user-focused metrics are far more important and worthy of consideration when deciding on the quality of a phone — as TNW’s own Nick Summers pointed out last week.

To put it another way. Most people don’t shop for a car based on its top speed, we look for things that will impact (or help) our daily lives.

Yet you wouldn’t criticize Toyota, Nissan, BMW and others for listing a car’s top speed despite the fact that 99 percent of customers will never get to drive it that fast. The speed is pretty much irrelevant to the purchasing decision or any media review.

The same logic should be applied to benchmarking. This is the domain of enthusiasts and statisticians, not everyday consumers.

Everyone is ‘cheating’

Finally, it’s worth pointing out that Samsung — which was found to have optimized the Galaxy S4 for benchmarking this summer — isn’t alone in doing what it does.

Anandtech notes that ’cheating’ is pretty much rife within the industry:

With the exception of Apple and Motorola, literally every single OEM we’ve worked with ships (or has shipped) at least one device that runs this silly CPU optimization. It’s possible that older Motorola devices might’ve done the same thing, but none of the newer devices we have on hand exhibited the behavior. It’s a systemic problem that seems to have surfaced over the last two years, and one that extends far beyond Samsung.

Were the issue to affect real-world testing, then it would be more serious. However, it’s not clear how that would work, and, in my opinion, that is a very different reality to wanting to showcase a device’s optimal performance in benchmark testing, which is purely performance-oriented.

For what it’s worth, Samsung has denied the accusations, claiming that “the Galaxy Note 3 maximizes its CPU/GPU frequencies when running features that demand substantial performance.”