The big benchmarking Brouhaha

[Capper]

Sitting here in my office, watching the beginning of WWIII (no, not the Israeli-Hezbollah battle, but one between HARDOCP and The Firing Squad) I got to thinking, 'Whats the big deal?' Today I'll share a few thoughts on benchmarking and bashing other sites.

Last week, HARDOCP published Intel Core 2 Gaming Performance. It was an interesting read, in that it was a new approach to a major product release, and with the exception of the opening, I thought it was an interesting take.

"Let's just cut to the chase. You will see a lot of gaming benchmarks today that just simply lie to you. That is right, you will see frames per second numbers that are at best total BS, and at their worst a terrible representation of what difference a new Intel Core 2 processor will make in your gaming experience."- Kyle Bennett

I personally don't know Kyle Bennett. I've met him once, and had a few dealings of him that give me an opinion, but I'll keep that to myself. Kyle has preached against benchmarks for some time now, and I think his take is refreshing. However I also feel benchmarking is a necessity when taken in context.

Then, earlier this week, Jakub Woinarowisz from The Firing Squad posted Jakub's Rant: "Real World" Benchmarking . He, like I, and a lot of other people, took exception to Kyle's condescending attitude..

"Normally I don’t go off on other websites in my rant articles. Believe it or not, reviewing hardware and games can be a difficult, time-consuming process and I sympathize with my peers as it can be a tough job. But in their latest CPU review, hardware website [H]ardOCP called into question the testing methods of all other websites based in part on false information, and part subjective opinion. From the [H]ardOCP teaser".- Jakub Woinarowisz

As with Kyle, I don't know Jakub, and have never met him. I agree with points from both articles so far, and truly think this wouldn't be an issue had Kyle not taken the condescending attitude of the original article.

But, not to be outdone, and to take a jab at Jakub for criticizing the original article, HARDOCP posted Firing Squid Wastes Their Ink, in which they decided bash The Firing Squad for disagreeing with them, rather than supporting their point of view and further explaining their position.

"We have jokingly referred to the Firing Squad as the "Firing Squid" for years now. You know, the website that will post pages and pages of canned benchmarks without any analysis at all? Here is an example of them going on for 7 pages without any interpretation for the readers. They do however follow up with a 6 sentence conclusion and an award. Wow, more pages of canned benchmarks than conclusion sentences? No wonder they are threatened by HardOCP showing you what actually happens during real gameplay and describing our experience to you."- Kyle Bennett

Is all this unnecesaary? Yes! Is all this pretty damn silly? Very much so. Is benchmarking perfect? Hell no! A lot of factors play into how accurate benchmarking can potentially be - supporting components, integrity of the operating system, drivers, and system settings, among other things can all impact the results. There are several groups of people who use benchmarking for a variety of reasons.

First, you have the people who use benchmarking to test for stability, troubleshoot, or optimize a system's performance. These people are everyday users who simply want make sure their system is stable and running optimally. A problem here is that most users will run one test, one way. Gaining accurate results typically means repeatedly running the benchmark to get an average.

Next, you have the people who seemingly live and die by benchmarking performance, spending so much time in front of their computers running Super Pi and 3DMark that their skin has become almost transparent and their limbs permanently affixed to their desk. To these people, benchmarking is a contest, one to see who can push things the furthest or achieve the best score via overclocking and tweaking. A couple issues here are integrity, some people go as far as to photoshop their scores, and stability. While these people may be able to show off incredible scores in a couple benchmarks, they typically are not accurate as the system may not be stable.

Then you have companies....yes companies. These component manufacturers are where a lot of the problem starts. With people putting so much emphasis on benchmarking, companies have manipulated drivers to specifically raise their components score in certain benchmarks. Just because a component blows away the competition at one benchmark, or a series like the 3DMark Suite, it doesn't necessarily mean that it's a better component. Does the average Joe on the street realize this? Sometimes....sometimes people believe everything they see and hear, sometimes they dig a little deeper to form a more detailed opinion. It's all up to the individual.

When it comes to review sites, people have to remember that what we post is an OPINION, nothing more. This is what's great about having a network of sites who post other site's news and reviews, it lets the reader get a better overall picture of a component or technology. Different sites focus their attention on different things, some put a greater emphasis on benchmarking, some on features, and so forth. Does this make one site better than another? No, actually it doesn't, but as we said, by looking at a variety of sites and forming your own opinion, you can make a much better buying decision. Benchmarking aside, people need to look at everything the component has to offer. A great example in video cards would be ATI's X1900 XTX vs NVIDIA's 7900GTX. Although the 7900GTX comes out on top in most benchmarks, ATI's AVIVO Technology is leaps and bounds ahead of NVIDIA's Pure Video. And AVIVO is free, where Pure Video costs $39.

I applaud HARDOCP for getting away from "canned" reviews, sort of....it's something we believe in. However, we are still trying to find a creative balance between benchmarking (which is necessary) and subjective opinion of a component's performance. It's not as simple as it sounds. Ideally I think you have to include benchmarking within a review. I think that benchmarking has to be weighted just as features, price/value, etc. are, and together they give you an overall sense of what the product has to offer.

I guess I have a couple points to make. First, I have lost a lot of respect for what I considered to be one of the best sites on the web. As such a large and influential site with enormous reach, you have a moral obligation to your readers to allow them more than one opinion. If you truly are there to help your audience, you need to give them the entire picture, and this includes valuing other sites,as well as people's, opinions. You may not agree, but it's an opinion.

Lastly, from the article in question, HARDOCP's condescending tone really put me off. What really is the point of blasting sites for taking a different approach to their articles? Why is it necessary to go so far as to say those sites are lying to you? Wouldn't it be simpler to just not crap on other people? Wouldn't it be a lot more helpful to simply link articles and let people form their own opinion? Leave the condescending, know it all attitude to the forum trolls and focus on helping people make good, educated buying decisions.

[Jokerswild]

You are absolutely right capper. There is no point in firing on another site just because you don't agree with their method of doing things. I also think benchmarks are important just to have some kind of idea where things stand even if they can't be perfect. I recall nvidia cheating on optimized drivers for 3dmark just to get a better score. Anyway thanks for giving us your take on the matter I believe you summed it up exactly right.

[drew and not u]

craziness

[sktworkshop06]

Hmm... I think that benchmarks are important and that these guys need to just stop the battling back and forth. Maybe they both have great points but not everyone does everything the same. They just need to stick to there own ways and whichever site does a better job will get more hits. Just the way it goes. Nice sum-up though, Capper.

[simple_inhibition]

hmm. i think the link here should be posted in both those sites forums. (after this page has been locked and cleared of other posts to prevent assinine comments)

[Capper]

This was originally going to be an article....and the original draft was a lot more inflammatory (Ask Paul...."Are you sure you want to go there?").....but I decided all it would do is really torque off the HARDOCP crowd, which is not my intention. I don't think Jakub did anything wrong....I took offense to HARDOCPs attitude as well. I thin its just a case of someone being a little too full of themselves.

As to making it a forum thread and opening it to comments....you guys are fine, but I think people in general act on the internet, more specifically forums, in a way that they wouldn't while face to face with someone. While there is no tone or inflection in forum posts, and people can take things out of context....people tend to be rude and act like asses.

When I step back and look at the crowd we have here, I'm very proud in the way people handle themselves, and the fact that we've only banned one person in eight months