Bit-Tech Drops Real World Testing

[Capper]

[H] Enthusiast --- www.hardocp.com

In my opinion cards based upon the same technology (both ATI and NVIDIA) will mainly offer the same features....what will separate them is the graphics core and memory speed/size/data width so, by that rationale, FPS do matter.....does that make sense?

To me it does, and what separates those cards is the resolution you play at, and the frames you can get mainly with all the eye candy turned up

I think Kyle Bennett is over complicating things, as usual, ....and thinks that the Kool Aid crowd, who will buy anything and everything he says. "Slight of hand"....we're going to tell you everyone else's reviews suck, that they are canned crap, and then we're going to think of a really creative way to take the exact same data and just tell you a different way. I feel this is a dangerous precedent for our community.

http://www.hardforum.com/showthread.php?t=1191974

Quote:
Since you guys apparently don't use timedemo's how to you keep the tests similar when testing cards? Do you run thru a room shooting and try to mimic that same movement for a different video card? Or am i way off here?

Exactly, through practice and experience we play through a game the same way each time, making sure to turn at the same spots, stand at the same spots, perform the same actions at the same times in the same ways. Due to the AI and randomness this is not 100% perfect each time, but it does 100% represent how a gamer plays the game, it will never be 100% exact for a gamer each time either. Many games are more random, such as BF 2142, but it does represent real-world gaming.

Keep in mind that this done for the framerate over time graphs, to backup the settings we found playable. To figure out what we find playable we actually play the games through, several levels, several scenarios, to try and see what the best balance of performance and IQ is, this is where we determine the highest playable settings, through actually sitting there for hours playing the games. The graphs simply backup our analysis.

Seriously......that was their response.

There is so much wrong here I don't know where to start, but let me start by saying that the first rule of benchmarking and reviewing is repeatable performance, or being able to duplicate your testing. Why is this important? So your results are accurate and even for everything you test.....which is why you run the same benchmark multiple times (to make sure your results are accurate), and also why after each review your testbench is rebuilt from the ground up, meaning everything is disassembled, reassembled with all the software including the OS freshly installed.....and now Kyle Bennett himself is telling us that he performs all their tests during real gameplay....making sure they repeat everything over and over, etc.

Another issue worth mentioning, and one I agree with, is that there is no right or wrong way to benchmark a product (within reason), and I feel that getting as many perspectives as possible, from as many different sources is a good way to base your buying decision.

I have problems with HARD on a personal level, because on an almost daily basis they bash the reviews of certain other sites, while never mentioning the crap his friends publish.....which is wrong on a number of levels. First, we're supposed to be helping people. Second, we're supposed to objective.....ok, thats my rant for the day....fire away

[onebxr]

ranter<-------this is the best I got

[Capper]

LOL....sorry. I'm not posting this to pick a fight with anyone, I just take issue with the whole "Elitist" attitude, and think that its bad for the enthusiast community.

I very much value different opinions, and you guys know we link several reviews from around the web on our main page, in our forums, and in our individual articles. We always encourage people to get as many opinions as possible before buying hardware.

I also need to add that I think "Real World" testing does have value, and a place in reviews. I also think that our method of showing a variety of settings works just fine, and combined with a "Subjective Testing" section, where the reviewer can point out his own thoughts from game play and video playback should give the reader a good overall opinion of a product.

[gvblake22]

I personally prefer bit-tech's charts and testing methodology over [H]...

[Lead Head]

About the only thing I like on the [H] are their PSU reviews.

[halutzparilla]

Well I feel your point Capper...

With my personal opinion I value objective benchmarking over subjective even though I wouldn't trash the subjective testing either coz personal exp over the hardware testing is really a help. Especially when you decide by the budget and reliability of merchandise...

[screwballl]

I agree capper 100%... this is why we are HL and not [H]ardup

[stinger608]

Originally Posted by screwballl

I agree capper 100%... this is why we are HL and not [H]ardup

Get it right Screwballl, "thats why we are get your hard on, not [H]ardup"