The 5 Problems with Videogame Journalism January 27, 2006 Alan /.effect Dang, Pfucata.com

Introduction

Page:: ( 1 / 6 )

Videogame reviewers don't play games the same way you do

Of the more than 3000 of you who participated in our survey, 76% of you reported buying fewer than 6 games in all of 2005. Compare that to the typical reviewer at a "mainstream videogame website" who will review over 100 games a year; in the 7 years I've been at FiringSquad I've only written 86 articles! I know it's appealing to think that the more games a reviewer plays, the better information he'll have to determine what's good and what's not. After all, a phrase like "The best game I've played all the year" doesn't matter if you've just played one game that year. The problem is that playing too many games a year causes several problems:

1. It's easier to get jaded.


2. Pressure to rush through the game


3. Less time to reflect before you write


4. Selection bias


5. Distorted sense of value

Problem 1: Innovation Bias

Page:: ( 2 / 6 )

As a result of being jaded, reviewers end up falling into the trap of innovation bias. Bruce Shelly of Age of Empires III talked about innovation bias as being one of the problems of the industry. He's only partially right. AOE3 is necessarily a polarizing game because resource micromanagement is such a large portion of the game. You'll either find it interesting or boring, and that accounts for the wide range of scores. For a lot of people, it's just not that fun. The problem is that if you read through most of the reviews of the game, you'll see that the negative aspects aren't about how the game requires too much micromanagement, but how the game feels just like AOE2 and doesn't bring anything new to the table. The problem is that for gamers who never played AOE2, AOE3 would actually be a great game to get. I can count on one hand the number of reviewers who recognized that. But AOE3 isn't a good example. In this case innovation bias means that some good games get lost in the crowd – a stellar game is still going to be noticed, and as long as there are about a dozen of those a year, there'll be plenty of choice for the 76% of you who buy up to 6 games a year.

The problem arises from a game like Project Gotham Racing 3. It is good? Sure. I don't think it's as fun as a game like Need for Speed: Most Wanted, but it's still a game that earned 90% at FiringSquad. On the other hand, if you look at the way the game is reviewed by most video game sites, a huge deal is made about the in-cockpit views because it's innovative. The cockpit view for "street cars" isn't something new – Square did it with the PS2 launch game Driving Emotion S. PC racers such as Grand Prix Legends have also done in before. Very few reviewers actually talk about the flaws of the cockpit view in PGR. The most egregious being that it’s difficult to read the dials because the steering wheel is in the way or the lighting is a problem. Too much gets made about the cockpit view in PGR3 and not enough attention is paid to the problems of the game.

Even for my own reviews – are the police chases in Need for Speed: Most Wanted that good, or am I blinded by the novelty and innovation of the feature? Am I criticizing the length of Ridge Racer 6 because I feel like the game has already been done and brings little to the table other than nice graphics?

There's not much that you can do other than try to be aware of a reviewer's innovation bias. At the end of the day, it's still important that reviewers play a wide range of games. It's just something you have watch out for. If a review spends too much time talking about how it's no different from the prequel or spends too much time talking about a single feature, take pause.

There was another area where “being jaded” comes into play. How come I didn’t have any screenshots of real-tracks from PGR3? It was all about high-resolution shots from the Garage mode… The vast majority of game reviewers stick with direct framebuffer grabs which may not reflect the actual experience and usually, we just take a hundred screenshots and hand-pick the more stylish ones we can find. The only time I’ve ever used direct framebuffer grabs is with GT4 and even then I spent a fair amount of time with a disclaimer.

For the record, Jakub completely disagrees with me about Innovation Bias. He doesn't see it as a problem but rather a reflection of the human desire for constant improvement. I think that you can have a desire for constant improvement but still be affected by innovation bias.

Problem 2: Reviewers have to rush through the game

Page:: ( 3 / 6 )

In games with optional side-quests, reviewers are forced to take the shortcuts through the game rather than exploring the world as someone might. In fact, one of the best sites for reviews of console RPGs, the Gaming Intelligence Agency, shut down because the staff found that the pressure to finish the game and get the review out as soon as possible meant that no one was having fun playing games.

Should reviewers have to finish the game? Not if they aren’t given the time to do so. I think they definitely need to make an effort to really play the game and be confident, without any hesitation, as to what the score should be. Some games aren't fun because the payoff only happens at the end and a "normal" person wouldn't have any incentive to get there. When you don't rush through the game, you have time to really get a good feel for the game's setting and gameplay. By not rushing through the game, I have a better sense of the world of Need for Speed: Most Wanted and had a chance to see how Project Gotham Racing 3 quickly plateaus and stops being interesting. On the other hand, games like NFS:MW suddenly get tougher in the later parts of the game. Since my review of NFS:MW was written before I finished the game, I completely omit this detail. Likewise, since reviewers often have to play online before the online community is developed, it can be tough to get a good sense of what the online play will really be like for everyone else.

If you ask around, most videogame reviewers would love to have a bit more time. True, for some games there's no time pressure, but many games take more time to play through than reviewers are given time.

I made another common error in videogame journalism two paragraphs ago. Did you notice it? It’s what probably would have been #6 on the list: writing about stuff I only know second hand. If I haven’t beaten Need for Speed: Most Wanted, why am I writing that the game gets too tough later in the game? I don’t know that to be true for a fact. I just read it on a message board (GAF)...

This is a fairly common phenomenon. Videogame reviewers have to enjoy what they do, and part of that is participating in the online community. Still, there are many times were videogame journalists will take ideas or strategies from message boards and not really give any credit to the original source.

Problem 3: Not enough time to think before you write

Page:: ( 4 / 6 )

Suppose you're able to avoid innovation bias, and you were able to play through the game thoroughly. There's still another problem. Tomorrow, you're going to have to start the cycle all over with a brand new game… but you haven't written anything yet for the game you just finished. There's a substantial pressure to get the article done as soon as possible. This introduces new problems.

First, this introduces spelling and grammatical errors. In the very first sentence of the very first article of this series, I wrote “videogame journalists have been getting a bad wrap [sic].” It’s rap. Not wrap. There’s no w. That’s the VERY first sentence of this series. Did you catch it? Why not? How about in Dead or Alive 4? “You have a throw button, a block/hold button, and a punch and kick button.” You might think that spelling and grammar are trivial points. Hire a copy editor and you’re done, right? No. More often than not, reviewing and proofreading an article helps you make edits to the content to clarify your thoughts.


There's another problem with trying to write a review as quickly as possible is that reviews start to fall into a cookie-cutter approach. Take a look at my Dead or Alive 4 review again. I start off with the cheesy intro which tries to talk about history of the series. You already know from the tone that I'm being negative about the Dead or Alive series because I'm about to say that that this new version is better. Why couldn't I have jumped straight into the review? True, it helps DOA3 fans to get a sense of DOA4 when I review the history, but part of this captures the "innovation bias" that we've got. Likewise, if I’ve done my job, the rest of the article should tell DOA3 fans exactly what they need to know. When you don't have time to write an article, you're not going to try to come up with anything new. It's easy to write an intro this way.

More on problem 3

Page:: ( 5 / 6 )

This also means that most videogame reviews end up being filled with clichés which makes for boring writing. I've tried to put in a reasonable amount in all three of these articles, but let me ask you how many times you've read these phrases in a game review:

"At the end of the day…"
"…doesn't re-invent the wheel"
"…a hit or miss"
"…left us wanting more"
"What makes or breaks…"
"Taking a cue from…"
"At first glance… In practice…"
"…making for a great experience"
“…is weak”

Problems 4 and 5

Page:: ( 6 / 6 )

Problem #4: Selection Bias

Having to play through games very quickly means that only a certain subset of gamers are going to have enough skill to do this professionally. Everyone can be good at a game with practice – but reviewers don't have the luxury of practice. This means that the average videogame reviewers is going to have "significantly above average" talent when it comes to games. They wouldn’t survive in the industry any other way.


This is a critical issue because what's at stake is $60. If you buy a game based upon a review and you don't like it, you're in trouble because you won't be able to return it. If you have to rent a game before you can figure out if a review is a good or not, what's the point of reading a review?

Problem #5: Distorted Sense of Value

The problem with this is that it is often tough to get a sense of value. Game reviewers usually don't buy games. Of the four games I looked at in the previous two articles, PGR3 and RR6 were provided free of charge by the publisher. NFS:MW and DOA4 were purchased at Toys R Us. The problem is not about being bought out by the publisher (although this commentary by Gamers.com alum Dan Hsu and this commentary by Gamers.com alum Sam Kennedy are worth reading). The problem is that it makes it much more difficult for reviewers to get a good sense of value when it comes to reviewing games.

This is probably more my pet peeve than anything else, but all too often you'll read about "adding this game to your collection" or a complaint that a game is too short and therefore a poor value. Games are expensive – I'm not interested in "collecting" games. They’re not stamps or artwork -- I just want to make sure I getting the most value out from my money. Second, a short but intense experience is definitely better than a long game that's repetitive. Unfortunately, a lot of reviewers place too much emphasis on the total number of hours of gameplay which leads developers to try to extend the length of games for, essentially no good reason. It seems like I should tell you what to expect from a game – the level of intensity and the duration it took me to finish and let you decide for yourself. (i.e. It's 8 hours of sheer excitement vs it's 40 hours of which 12 are really good, and 5 of which are taxing, and the rest being mediocre) rather than simply say that it’s too short of a game.

It's an interesting question though. When we're not paying for all of our games, or we know that buying games falls under the category of a business expense, is there a potential for us to be blinded or biased? I think so. Car reviewers and hardware reviewers don’t have this problem because there are objective measurements available. Food critics don't have this problem because you’re only reading reviews for restaurants that you can afford.

The Sky is Falling!

Being a skilled videogame reviewer is a tough job. There are significant challenges that reviewers face when evaluating a game. Can we really look at the game from the perspective of a typical gamer? Can we do this while preserving the advantage gained from experience? I hope that's the case. It starts with reviewers being aware of these pressures and these problems. We can try to be introspective and look at areas where we risk running into trouble but there may be problems so deep that we are completely oblivious to it. The solution continues with you, the reader. You have to keep us on track and let us know how we're doing. These article comments sections aren’t simply there to bring pageviews – we want to hear your thoughts.