Paper Mario 2 review reveals Game Informer review criteria

[Gaming-Module]

GI-Jeremy wrote:

Lisa and I both knew that our Paper Mario scores were going to cause controversy. Yes, we know that many people out there will love it. We also know that it is a well-made game. However, it also WILL NOT appeal to many people - I would safely say that more people will dislike it than like it. Why? Like we said in the review, it's a very kiddie game - it's target audience is clearly young gamers - I would say 10 and under. For that reason, we had to score it low. Remember, we aren't scoring games strictly on our personal opinions, we're also scoring them based on how much we think THE GAMING PUBLIC will like them. We've all played games that we personally disliked and scored them well because we've known that most people will like them, and we've also scored games low that we love, because most people won't enjoy them.

FOr example, I really like the bizarre frog golf game Ribbit King, and I gave it a 7, because it's just not for everyone. Paper Mario 2 also scored low because it's just not for everyone. If you think it's a 10 in your book, it's a ten in your book, and that doesn't change if we disagree. We're here to guide you on what games to pick up, but ultimately your personal opinion is what will make you buy a game or not.

I hope this helps.

Check out the Gamespot thread here. (http://forums.gamespot.com/gamespot/show_messages.php?board=314159266&topic=16826928&page=0)

I posted this over at Ars too, but I wanted to see what some of the pros around here thought of this policy.

I think it's absolutely absurd. Who are these guys to predict which games everyone will like or dislike? That is more assinine than buying games based off of what a reviewer personally thinks of them, which would mean that the standard review method is kinda like Democracy. It's still shit, but it's the best we've come up with so far.

GI, my comrades, is no democracy.

[noun]

Remember, we aren't scoring games strictly on our personal opinions, we're also scoring them based on how much we think THE GAMING PUBLIC will like them.

I sort of understand and agree with that statement. When I reviewed a game, I usually had a sentence or two saying something like "if you enjoyed X game, or this particular feature of Y game, you might want to give Game Z a try."

But the bit about adjusting a score because he didn't think readers would agree with it, that's just cowardly.

[Mike Cathcart]

If you thought that was confusing, try making sense of this post (from gaming-age):
Ok look...I need to clear something up. First off, I'll fess up a little. I'm Andy McNamara, EIC of GI (I'm screwing up my anonymous status here, so bear with me, and yes, I know you don't give a shit who I am). Jeremy, who wrote the post that you so openly quote, is extremely misunderstood in this whole matter.

We do not, and I will repeat, we do not review games based on what people will think of the title. I called him on it this morning, and he was upset that the whole thing is not being understood the way he had intended it to be. The internet, as we all know, is full of trappings and misunderstandings.

Yes, Jeremy and Lisa do not like the game. As EIC I challenged them on the review many, many times. All the staff knew that the review would not be openly accepted. But the reality is, whether you choose to believe it or not, they did not like the game.

I have not played the game in its entirety. I personally believe the game to be better than the 6.75 that they scored it. But when I challenged them on it over and over they stuck to their guns (and I gave them a crapload of grief on their reviews). I admire them for that. They went against the grain, and they believe and stand by the scores they gave.

What are we to do? Force people to change their scores? Change the scores because we know that people will be pissed? I think you already know the answer to that question, as we let the review fly. We are not changing reviews to please the public. If that was the case, we would have given this an eight and been done with it.

I know Jeremy stated in his post that he felt he was considering the game buying public when he was writing his review, but what he was trying to convey was that he felt that our reviewer's point-of-view is not unlike others in the gaming public, and that by making their views known, he was serving gamers (which IS something we are trying to do). Not everyone loves this type of game - I think our review is proof of that.

Rip on Game Informer all you like, but I won't force someone to change a review. Yes, I will question and challenge them on what they write, but in the end an opinion is just that - an opinion. Who am I, or anyone else, to question anyone's view? The world needs more media that will pick a stance and stand by it in my personal opinion.

Both of them (Jeremy and Lisa) are harcore gamers, and both loved Superstar Saga. They both didn't like The Thousand Year Door. So be it.

I respect the Gaming Age forums crowd as the hardest of the hardcore, and love to read these boards, but at times I think things get a little out of hand.

I see all kinds of tastes that find their way here, and I think people are a little quick to call others wrong or stupid for a differing opinion. It's a video game. God knows I love them. And I love the ones I like even more, which is one of the reasons I visit these boards - to read what people who are passionate about games like myself have to say about games.

Video games are truly the greatest entertainment medium in the world. And I, like you, love to argue over right and wrong, but at some point you just have to let things go as a point you simply can't agree on.

Cheers,

Andy


PS: I believe this is the point where you call me stupid and say that my magazine sucks.

If you want to see the whole post go here: http://forums.gaming-age.com/showthread.php?t=18453&page=5&pp=50 It's near the bottom of that page.

I know we've covered this a billion times, but trying to guess what the public will or won't like is rididulous. Tell me what you thought of it and why. That should be enough for me to decide. Don't insult everyone by telling us that it's a great game that we won't like because we're morons.

[Gaming-Module]

I was surprised at how everyone in that thread kowtowed to Andy so quickly.

I guess if I were more like a Gaming-Age forumer I would spend most of my time posting here asking Tom for his autograph.

[tglennow]

Video game reviews seem to have some special quality in the minds of gamers where they have to be objectively accurate; like a score in a magazine determines how good a game really is. I never hear fans of movies complaining that the Los Angeles Times movie reviewer gave their favorite movie a bad review. Well, maybe I do every once in a while, but everyone understands that movie reviews are subjective, an opinion of a human being. This doesn't seem to be the case with game reviews.

[quatoria]

Video game reviews seem to have some special quality in the minds of gamers where they have to be objectively accurate; like a score in a magazine determines how good a game really is. I never hear fans of movies complaining that the Los Angeles Times movie reviewer gave their favorite movie a bad review. Well, maybe I do every once in a while, but everyone understands that movie reviews are subjective, an opinion of a human being. This doesn't seem to be the case with game reviews.

Do you regularly hang out in communities of 'hardcore' (borderline obsessive) film hobbyists or film professionals? You might find the perspective a little different there, much like our perspective, here, isn't the same as that of the general public.

[Thrrrpptt!]

This is why reviews are better without giving scores.

[Tim Partlett]

So when a reviewer gets an obviously well made RTS, but hates it, because they hate RTS games, they shouldn't compensate for the fact that other people like that kind of game and give it 3 out of 10? Most good reviewers will combine a bit of what they think about the game, with a bit about what they understand the general public will think about the game, and will base this on the understanding of games and the gaming public. This is what these two seemed to have done, and I have to agree with them, in that I hate kiddie styled games. If I had reviewed Mario 64, far from giving it 10/10 and game of the year, I'd have given it 5 out of 10, before tilting it up to 7 because other people might like it. I hate kiddie games and puzzle games, but I understand others love them.

If they had liked the game, but given it a bad score based purely on what they thought the public thought, I'd consider that pretty stupid, but that isn't what they did.

[quatoria]

As for my take on this, I'll post here what I just said in the IRC channel (which you should all really visit, #qt3 on irc.enterthegame.com)

The score is for the quality of the game. The quality of it's execution, how well it is made, how much fun it is to play for the audience it is intended. And then, you explain in the text of your review, who that audience is, and make sure it's clear that certain people may not enjoy this game, as it is not targetted towards them.

But to mark the supposedly 'objective quality' score of a game down *because* it's not for all audiences - that seems skeevy to me. It's like saying the game is lower quality because a counterstrike junkie probably won't like it. That seems grossly unfair, as does using projected popularity as a metric of quality.

Obviously, that's just my opinion, and is in no way binding on other reviewers - but I'd feel distinctly uncomfortable if I was told to factor in the expected popularity of a title in my final score. Then again, I'm not overly comfortable with 'scores' to begin with, so take that for what it's worth.

[extarbags]

I know we've covered this a billion times, but trying to guess what the public will or won't like is rididulous. Tell me what you thought of it and why. That should be enough for me to decide. Don't insult everyone by telling us that it's a great game that we won't like because we're morons.

Right, that's what GI does. They masterfully review games based on what they think the general populace will think of them, but not because they're guessing at what "most gamers" will like; rather, it's that they themselves are a part of the general populace, and therefore when they are giving their *own* opinion, they are, in fact, giving an opinion based on what one part of the general populace would say about the game.

See?

[Gunmetal]

A 6.7 certainly doesn't sound like a defendable score, given their reasoning.

[Kevin Grey]

So when a reviewer gets an obviously well made RTS, but hates it, because they hate RTS games, they shouldn't compensate for the fact that other people like that kind of game and give it 3 out of 10?

I hate this argument- the answer is real simple: Someone who hates RTS games should not be reviewing RTS games! How is someone who hates RTS games able to evaluate the changes and impact of going from Kohan to Kohan 2 or even comparing the strategic depth of Kohan 2 to Dawn of War?

[MattKeil]

A 6.7 certainly doesn't sound like a defendable score, given their reasoning.

The crusade against the "kiddie" marches on.

If this really is the criterion by which they justify the score of their PM:TTYD review, I don't see how the review could be of any use to me. I haven't read it, so I have no idea if they actually go into why the game disappointed them, particularly in comparison to the highly similar Superstar Saga.

I'm willing to believe the game is subpar, but so far I have seen no real evidence of it in any of the negative reviews ("negative" meaning reviews below an 8/10 score...since a 7 or below might as well be a 1 or a 2 to most people it seems). IGN docked the game for lacking voice acting, which strikes me as utterly ludicrous.

[Moore]

GI-Jeremy wrote:

Lisa and I both knew that our Paper Mario scores were going to cause controversy. Yes, we know that many people out there will love it. We also know that it is a well-made game. However, it also WILL NOT appeal to many people - I would safely say that more people will dislike it than like it. Why? Like we said in the review, it's a very kiddie game - it's target audience is clearly young gamers - I would say 10 and under. For that reason, we had to score it low. Remember, we aren't scoring games strictly on our personal opinions, we're also scoring them based on how much we think THE GAMING PUBLIC will like them. We've all played games that we personally disliked and scored them well because we've known that most people will like them, and we've also scored games low that we love, because most people won't enjoy them.

FOr example, I really like the bizarre frog golf game Ribbit King, and I gave it a 7, because it's just not for everyone. Paper Mario 2 also scored low because it's just not for everyone. If you think it's a 10 in your book, it's a ten in your book, and that doesn't change if we disagree. We're here to guide you on what games to pick up, but ultimately your personal opinion is what will make you buy a game or not.

I hope this helps.

Check out the Gamespot thread here. (http://forums.gamespot.com/gamespot/show_messages.php?board=314159266&topic=16826928&page=0)

I posted this over at Ars too, but I wanted to see what some of the pros around here thought of this policy.

I think it's absolutely absurd. Who are these guys to predict which games everyone will like or dislike? That is more assinine than buying games based off of what a reviewer personally thinks of them, which would mean that the standard review method is kinda like Democracy. It's still shit, but it's the best we've come up with so far.

GI, my comrades, is no democracy.

Seems a lil' odd to me- I'd assume that as game reviewers, they have more experience playing and reviewing games than with the thoughts and minds of the majority of gamers around the globe. So, uh, play the game and tell us how it is, the end - I'm unsure that when I pick up a magazine I'm looking for info on how much they assume joe sixpack will like it. I can understand pointing out oddities or things done differently than other games or something, but not 'I LOVE IT! but you won't 'get it' so it's a 5/10" Or " I hate it, but it has boobs and you nerds love digital boobies, so 9/10! MUST BUY!)

[Tom McNamara]

IGN docked the game for lacking voice acting, which strikes me as utterly ludicrous.

It was mentioned in the score box, but the text of the review is actually much gentler.


Yes, it's nice that you can progress the story in the game at your own pace since everything is text-based, but with the help of some quality voice actors and facial animation, Paper Mario 2 could be even better. There are plenty of opportunities for comedy with voice actors and facial expressions -- and given that Mario himself doesn't really talk in the game, it's about time Nintendo embraced the medium and at least gave the rest of the crew a voice. After all, the characters in the Pokemon TV show don't converse with speech bubbles when they run into each other, why should games, an interactive audio-visual medium lag behind? Given Nintendo's broad age target for its games, adding voiceovers would also mean that younger gamers could get in on the fun. But lack of voice acting is a small quibble. The well-written dialogue and expressive text effects (letters shake when characters are scared and blow up to super-size font when angry) do a commendable job, and many old-school Nintendo fans may actually prefer the silent treatment.


As for GI's take, I find their stance confusing. I would much rather consider niche appeal than make an educated guess on public reaction. I don't think it's skewing towards the casual gamer as mentioned several times in the GS thread, but I think it's just much easier to stick to your own opinion and explain why you think that way.

[Mike Cathcart]

Right, that's what GI does. They masterfully review games based on what they think the general populace will think of them, but not because they're guessing at what "most gamers" will like; rather, it's that they themselves are a part of the general populace, and therefore when they are giving their *own* opinion, they are, in fact, giving an opinion based on what one part of the general populace would say about the game.

See?
No.
we aren't scoring games strictly on our personal opinions, we're also scoring them based on how much we think THE GAMING PUBLIC will like them.
This is lame. See? Tell me your opinion, tell me why. The end. If your opinion is that Madden 2005 sucks because you hate football games, go ahead and give it a 3 and make sure you point out in the review that you hate football games. Then I know to read another review.None of this "Yes, we know that many people out there will love it. We also know that it is a well-made game" but then give it a lower score because you don't think the "gaming public" will like it. That isn't a review, that's your guess at the opinions of millions of people. And it's worthless.

I can, based on the text of your review, decide for myself if I want to play the game or not. Or even if I just want to read another review. At least that hypothetical Madden review is honest. Also, for a game that will only appeal to people under 10 it's doing very well on every other gaming site or mag out there. I guess most of the writers are under ten.

If you don't like Paper Maio 2, that's cool. Even Tom Chick didn't like Animal Crossing. People make mistakes. At least he had better reasons than "I don't think you guys will like this. 6.75."

[Tim Partlett]

I hate this argument- the answer is real simple: Someone who hates RTS games should not be reviewing RTS games! How is someone who hates RTS games able to evaluate the changes and impact of going from Kohan to Kohan 2 or even comparing the strategic depth of Kohan 2 to Dawn of War?

Yeah, I agree with you, but it happens, and in this case the two reviewers obviously didn't like kiddie style games. Maybe you can blame their editor for not checking whether they liked this kind of game or not, but I don't see what they did wrong as reviewers.

[Cold Blooded]

That is more assinine than buying games based off of what a reviewer personally thinks of them

Might be coming in late to the discussion, but I thought this is what everyone at these forums wanted? To hear what the reviewer "really thinks"?

[Kevin Grey]

I hate this argument- the answer is real simple: Someone who hates RTS games should not be reviewing RTS games! How is someone who hates RTS games able to evaluate the changes and impact of going from Kohan to Kohan 2 or even comparing the strategic depth of Kohan 2 to Dawn of War?

Yeah, I agree with you, but it happens, and in this case the two reviewers obviously didn't like kiddie style games. Maybe you can blame their editor for not checking whether they liked this kind of game or not, but I don't see what they did wrong as reviewers.

Neglecting the very arguable point over whether Paper Mario is "kiddy" at least one of the same reviewers previously reviewed the equally kiddy Mario & Luigi Superstar Saga and gave it a 9.5!

Then there is the inconsitency between the initial statement and the EIC's clarification. The reviewer makes it quite clear that their reviews don't necessarily reflect their personal feelings for the game while the EIC points out that the poor review was strictly because they didn't like it at all and had nothing to do with the GENERAL PUBLIC's reaction to the game.

Oh, and then there's the fact that GI has since deleted the entire thread.

[BaconTastesGood]

So when a reviewer gets an obviously well made RTS, but hates it, because they hate RTS games, they shouldn't compensate for the fact that other people like that kind of game and give it 3 out of 10?

Either A.) score it the way you think it should be scored according to your personal preferences, and therefore readers will be able to factor that in later; or B.) don't score it at all, because you're not going to be qualified to judge it relative to its peers.

Pretty simple.

[Gaming-Module]

After reading that Gaming-Age thread it's pretty obvious that what Jeremy said amounted to, "We are pandering to our predominantly PS2 owning readership by scoring "kiddy" games low to preserve our 'street cred.'"

Pathetic.

[Dave Long]

The way that guy describes it, they don't even have to play the games in order to review them. They could simply read a few web forums, maybe ask around at the EB and Best Buy and then go write. They're evaluating the public opinion of the game and not the game itself.

--Dave

[JD]

Neglecting the very arguable point over whether Paper Mario is "kiddy" at least one of the same reviewers previously reviewed the equally kiddy Mario & Luigi Superstar Saga and gave it a 9.5!
Zelda: Wind Waker 10/10, Mario Kart: Double Dash 9.25/10 , Mario & Luigi: Superstar Saga 9.5/10, Mario Sunshine 9.75/10 (according to Gamerankings)

Alriiight. So, did they officially change their score system in the meantime or what? :roll:

[steve]

So when a reviewer gets an obviously well made RTS, but hates it, because they hate RTS games, they shouldn't compensate for the fact that other people like that kind of game and give it 3 out of 10?
If they hate the genre, how can they judge that it's well made? Relative to what? They hate the genre, and they hate all RTS games, so it's really just comparing differing levels of suck.

I think a review of a game outside of your genre has a lot of value, but only to people like you. It's a great second opinion kind of thing. If Reviewer 1 loves RTS games, the review is for fans of RTS games. If Reviewer 2 says, "I hate RTS games, but this one changed my mind," that has value to those who hate RTS games.

Most good reviewers will combine a bit of what they think about the game, with a bit about what they understand the general public will think about the game, and will base this on the understanding of games and the gaming public.
I disagree so much with this approach. A good reviewer describes clearly what they think about the game, and the "general public" can decide if the issues raised have any meaning to them. It's purely speculation to guess what the general public will think of a game.

It also leads to the standard review conclusion: "This is a terrific game that no fan of the genre should miss," or, "It has its flaws, but fans of the genre should check it out."

If they had liked the game, but given it a bad score based purely on what they thought the public thought, I'd consider that pretty stupid, but that isn't what they did.
So it's only okay to do this if they increase the rating?

[MattKeil]

Most good reviewers will combine a bit of what they think about the game, with a bit about what they understand the general public will think about the game, and will base this on the understanding of games and the gaming public.

I disagree so much with this approach. A good reviewer describes clearly what they think about the game, and the "general public" can decide if the issues raised have any meaning to them. It's purely speculation to guess what the general public will think of a game.

Absolutely. Unfortunately, I think the approach GI describes is more the rule than the exception. I can't remember the last time a major site or mag went against the tide on a major release. I think perceived public opinion plays a much larger role in review ratings than most would like to believe.

As a hypothetical, let's say that Halo 2 comes out and it sucks. Or worse, doesn't suck, but just isn't anything all that mindblowing. (Not saying it does. I haven't played it except for like five minutes of multiplayer at E3.) Now, how willing do you think some of the majors are going to be to give Halo 2 an 8 or lower? Or even a 9 or lower? At some point I think the backlash of the readers who are going to be blind to an uberhyped game's faults are taken into consideration over the facts of the game's quality.

I also have no problem believing that company clout comes into play, either. I've seen far more flawed games than Blood Will Tell get higher ratings, even when called on the same problems. However, nobody really cares if a Sega game gets lowballed, so it gets a low score. I would have liked to have seen what BWT's scores would have been had it been released by EA or Sony.

[Sidd_Budd]

Unfortunately, I think the approach GI describes is more the rule than the exception. I can't remember the last time a major site or mag went against the tide on a major release.

The major magazines have given Doom 3 incredibly high marks, but some dude named Bauman only gave it a slightly higher than average score (3.5/5). Some magazine called Computer Games ... I think it's produced in some fascist country.

I like the system where very high scores (say 9 or greater on a 10 point scale) reflect games the reviewer feels are worthy to own if you have even the slightest interest in the genre. Scores in the 8-9 range can be either games that could have been classics, but had a few flaws, or great games that will likely only appeal to a niche audience (Korsun Pocket is coming to mind). The accompanying text should make it very clear to which category the game belongs.

I think this is GameSpot's system, or was at one time...

[steve]

Absolutely. Unfortunately, I think the approach GI describes is more the rule than the exception. I can't remember the last time a major site or mag went against the tide on a major release. I think perceived public opinion plays a much larger role in review ratings than most would like to believe.
This is true, but a lot of reviewers are "guilty" of having purely mainstream taste. That their views don't match messageboard posters may say more about the messageboard than the reviewer.

Having said that, I think most reviews are wishy-washy, full of useless qualifiers, blah blah blah. Those are the elements designed never to offend. The final ratings, more often than not, match reader ratings.

[Jakub]

But to mark the supposedly 'objective quality' score of a game down *because* it's not for all audiences - that seems skeevy to me. It's like saying the game is lower quality because a counterstrike junkie probably won't like it. That seems grossly unfair, as does using projected popularity as a metric of quality.
So reviewers shouldn't score down Universal Combat because it's perfect for Battlecruiser junkies?

I'm just checking, here.

[Gaming-Module]

Looks like Andy finally told Jeremy to get out and help spin damage control.



Thanks for writing. There has been a lot said about the post I made in
our forums, most of which, like your letter, is completely fair and
valid. In a fit of irritation, I posted said message without really
thinking about it, and have lived to regret it. Everyone here scores
games based on their personal opinions, not based on what we think the
public wants to hear. If we scored based on what people wanted to hear,
Paper Mario 2 would have gotten a much higher score, like a 9 or a 10.
But that's not how we do things. What I meant in my message on our
forums is that I thought the majority of gamers would feel like I did
about Paper Mario 2 – disappointment once they got their hands on it. I
didn't mean to imply that I thought about how the public would perceive
a title when scoring it, I just meant that I thought those who agreed
with my score would outnumber those vocal opponents of my score.

I'm sorry to hear that this incident has caused you to distrust the
magazine in any way. You shouldn't. Everyone here is honest and up
front about their opinions, and scores games accordingly. You can feel
free to distrust me if you want (although I think I am being quite
honest with you in admitting that I publicly screwed up), but don't let
it spoil your opinion of everyone else here, the work they do, or the
publication that we put out.

Thank you,

Jeremy Zoss
Associate Editor
Game Informer Magazine

[Tim Partlett]

If they hate the genre, how can they judge that it's well made? Relative to what? They hate the genre, and they hate all RTS games, so it's really just comparing differing levels of suck.

I think that is somewhat of a polarised viewpoint. I guess it is probably because of my use of the word "hate" to describe games that I don't particularly like the style of. Hate is probably too strong a word for this. I can certainly see how a game that I don't like the genre of is good. I recognise the quality of Warcraft III, even if the cartoon fantasy style turns me off so much that I wouldn't buy the game. I can see that Mario 64 is a very well crafted game, even if the puzzles and the kiddie style annoy me. I think I've played enough games to recognise what is and what isn't a well made game, even if there are aspects of it I don't like. I think it is a skill of a critic to be able to recognise what it is about a game that they don't like, that isn't the fault of the game, but an aspect of their own personality.

I disagree so much with this approach. A good reviewer describes clearly what they think about the game, and the "general public" can decide if the issues raised have any meaning to them. It's purely speculation to guess what the general public will think of a game.

It also leads to the standard review conclusion: "This is a terrific game that no fan of the genre should miss," or, "It has its flaws, but fans of the genre should check it out."

I was refering to the scoring of the game, rather than the contents of the review. If I recognise that I have an issue with games that other gamers do not have, say for example finding the Mid-West accent so grating that hearing it makes me angry, I think I should both point this out to the people reading the review, and compensate for it when scoring the game at the end. I expect a good reviewer to understand games, their own reaction to games, and the general gaming market.

Ideally a reviewer would never get a game that would annoy them in any way, but we don't live in an ideal world.


If they had liked the game, but given it a bad score based purely on what they thought the public thought, I'd consider that pretty stupid, but that isn't what they did.

So it's only okay to do this if they increase the rating?

No. I said that it would be stupid to base a score purely on what you think the gaming public would think of the game, not because you compensate, up or down, for what you recognise as your own personal idiosyncracies.

Absolutely. Unfortunately, I think the approach GI describes is more the rule than the exception. I can't remember the last time a major site or mag went against the tide on a major release. I think perceived public opinion plays a much larger role in review ratings than most would like to believe.

I'm not advocating that reviewers boost scores because a game is hyped and the fanboys will complain! Read what I said above :D.

[Drastic]

Having said that, I think most reviews are wishy-washy, full of useless qualifiers, blah blah blah. Those are the elements designed never to offend. The final ratings, more often than not, match reader ratings.
One of the funniest things about that kind of head-hunching, legs-tail-betweening not-in-the-face-not-in-the-face sort of strategy is it's designed never to offend...gamers.

I mean, crikey. It's like taking desperate pains to avoid freezing ice.

[steve]

I can certainly see how a game that I don't like the genre of is good. I recognise the quality of Warcraft III, even if the cartoon fantasy style turns me off so much that I wouldn't buy the game.
What if your attempt to judge the overall market, or recognize the quality of WarCraft III and its graphic style despite your own personal reservations, turns out wrong?

If you're dealing with opinion, and assuming your facts are straight, at least you're never "wrong." However, if you say, "I don't really dig this, but I can see how others will," and scale your rating accordingly, you open yourself up to generating reviews that have little to no value to readers because you misread them.

I was refering to the scoring of the game, rather than the contents of the review.
This is probably why there's often a mismatch between the text and the rating. So many are full of criticisms, then "8.5." I'm not sure that makes sense to readers.

No. I said that it would be stupid to base a score purely on what you think the gaming public would think of the game, not because you compensate, up or down, for what you recognise as your own personal idiosyncracies.
The only thing you truly know are you own personal idiosyncracies; trying to judge how others would react is pure speculation. You might not base your final score on what the public would think, but if you alter your rating at all because you think your own tastes aren't necessarily the same as those of the public, I think you're being dishonest with readers.

Or if that's a bit much, you're trying to move a review from "one person's informed opinion" to "this review speaks for all gamers." The latter is certainly a noble goal, but I don't feel that comfortable speaking for anyone's views but my own.

Anyway, the reason I'm reading the review is because I want to know what Tim Partlett thinks--or any reviewer for that matter. I don't care what you think the market will think, because I'm fairly certain you don't know as much about it as you think you do.

Let's look at some examples. Dominions II. We gave it a 4.5/5 and picked it as the third best game of 2003. That's not exactly a mainstream title. Should it have been docked a half-star, or more, because the public probably wouldn't "get" it?

Or should I have automatically bumped up my review of DOOM 3 from 3.5/5 to 4 or 4.5 because despite my own tastes giving me a negative reaction to the art direction and fairly dull and repetitive, I know that a large percentage of the public really digs the look and combat?

[steve]

One of the funniest things about that kind of head-hunching, legs-tail-betweening not-in-the-face-not-in-the-face sort of strategy is it's designed never to offend...gamers.

I mean, crikey. It's like taking desperate pains to avoid freezing ice.
Heh, exactly. As long as you rate games at all, you'll offend gamers. I'm not sure why the text is an issue at all.

[Tim Partlett]

Well, you know, Steve, maybe you are right that the ideal review score should simply reflect what the reviewer felt themselves, but that's not how I feel. I guess this has something to do with the different perspectives we are looking at this from. You are looking at it from the perspective of an editor, perhaps an idealistic one, which is no bad thing, and you want to make sure that the reviews reflect the opinion of your magazine and don't include any mistakes. For me, I don't personally care whether a reviewer likes a game, I want to know if I will like the game, so I want the reviewer to take my concerns into account. If he doesn't like pixies as main characters, because they get on his nerves, I want him to mention that when he reviews a game with a pixie as the main character, and I also want this reflected in the final score, so that when I am skimming the reviews, I don't see "3/10" for what is otherwise is an excellent game.

That's just my opinion, but I doubt many other editors would let a review score pass that was very low, or very high, for no other reason than the reviewer really loved or hated a certain aspect of the game that most other people wouldn't. That's why I was defending the right of this reviewer to score it the way they did, because I appreciate where they are coming from. My reason for defending them isn't because I like the magazine, nor is it because I think I am right and that there is no other opinion that can be allowed except mine, but because I felt that the raging hatred that is being expressed in this forum for the WRONG way of doing something was completely over the top. If the screaming and raving had been directed from the other side of the fence, then maybe I'd have argued the same way as you, as I have no strong feelings on this issue. I just sympathise with the people concerned, because I understand where they are coming from, and don't feel they deserve this kind of vilification for expressing their opinion in the way that they did.

[steve]

For me, I don't personally care whether a reviewer likes a game, I want to know if I will like the game, so I want the reviewer to take my concerns into account.
My only point is that I have no idea what your concerns are, or anyone else's for that matter, so I can only cover my own and hope they match up with yours. If not, my review has less value to you, and that's a bummer, but there's no way I can satisfy every single gamer with every single review.

We have a lot of "this one feature is a killer for me" gamers. No mid-mission saves? No dynamic campaign? No skirmish mode? No multiplayer? These may or may not be issues for a lot of people, but any reviewer is welcome to get into a serious discussion about why it hurts/helps a game to have/not have that feature, and adjust the rating accordingly.

As long as they justify it in the text, I'm comfortable with it even when I disagree.

That's just my opinion, but I doubt many other editors would let a review score pass that was very low, or very high, for no other reason than the reviewer really loved or hated a certain aspect of the game that most other people wouldn't.
If they were able to argue their point in a clear and logical way that made it clear where they were coming from, I would definitely allow that review.

I just sympathise with the people concerned, because I understand where they are coming from, and don't feel they deserve this kind of vilification for expressing their opinion in the way that they did.
Welcome to the Internet.

[MattKeil]

My only point is that I have no idea what your concerns are, or anyone else's for that matter, so I can only cover my own and hope they match up with yours. If not, my review has less value to you, and that's a bummer, but there's no way I can satisfy every single gamer with every single review.

Bingo.

That's exactly what a review is, and it's hardly surprising that steve's the one with the clearest view on this, IMO. I can't remember ever seeing a film critic try to align his tastes with the mass audience's in an attempt to make his review seem more populist. That just dilutes it.

A review is meant to be a point of view. Watch the movie, read the book, listen to the album, play the game, and then tell me what you thought. Don't try to guess what I'm thinking, tell me what you thought. If you're a consistent reviewer and I regularly read the publication in question, I will undoubtedly be able to draw my own conclusions about how relevant your impressions are to my tastes.

I'm having trouble believing this is even a point of contention. :?

[Tim Partlett]

I always thought a review was meant to be objective, not subjective. You seem to be advocating a purely subjective review. My reviews are always objective. If I handed in a review of some game play elements to my boss and said, "these suck because I think so", and didn't take into account the gaming public, I wouldn't be taken very seriously. I think you have to take other people's opinions into account when you do any kind of review. You'll never be able to create a 100% accurate picture that way, but I think the end result will be more valuable to the end user.

[DeathMonkey]

I always thought a review was meant to be objective, not subjective. You seem to be advocating a purely subjective review. My reviews are always objective.

Always objective? Maybe I've been hanging around too nerdy of a crowd, but I would argue that it is impossible to be completely objective about anything. The meaning of truth and what not. Reviews should be opinions with supporting evidence. Attempting for any sort of "truth" only makes everyone look bad.

-Scott-

[steve]

My reviews are always objective. If I handed in a review of some game play elements to my boss and said, "these suck because I think so", and didn't take into account the gaming public, I wouldn't be taken very seriously. I think you have to take other people's opinions into account when you do any kind of review. You'll never be able to create a 100% accurate picture that way, but I think the end result will be more valuable to the end user.
Your reviews may strive for objectivity, but they never are.

Ignoring that point, you would be taken just as seriously if you only followed your own opinion because, presumably, your own opinions reflect the general public. Only if your opinions are totally in left field--like championing Flying Heroes or slamming Deus Ex or Burnout 3, for example--will people take you less seriously. Or alternately, if your left field opinions can't be backed up by solid evaluation and analysis.

Because really, those two elements are what people are looking for in a review, not your prediction of what "other people's opinions" will be. If all they want to know is how some large body of people feel about the game--and presumably, a lot of people do want that--all they need to do is go to Game Rankings or Gamespot and look at the aggregate reader ratings.

[Tim Partlett]

I'm sure you don't mean ill, but pointing out that I missed the words "strived for" in my post is akin to pointing out my spelling errors. Please don't. It's annoying. I don't think that pure objectivity is attainable any more than I think the sun orbits the earth. I would have thought that anyone capable of using the word objective in a sentence would have been able to work that out through logical deduction.

So, Steve, or death monkey or anyone, ideally, should there be any objectivity in a review, or should it be purely subjective?

[Erik]

At the evidently very real and present danger of annoying you further, the possibility of writing a purely subjective review is about as unlikely as writing a purely objective one. Got the title right? So long total subjectivity!

[Tim Partlett]

Where did I say otherwise? :D

[steve]

So, Steve, or death monkey or anyone, ideally, should there be any objectivity in a review, or should it be purely subjective?
Sensitive!

Of course there's objective information in a review: number of levels, what technology exists in the game, game modes, etc. In other words, bullet point stuff. You can get most of that info from a preview, or the box. It should only be about 20% of a review. (I don't know the actual percentage; that's just an official "pulled out of my ass" number.)

The rest should be analysis of those features, and how they impact the game. As soon as you say, "The graphics are amazing," or "the AI sucks," you leave objectiveville, unless there's some sort of "AI Mark 2005" that tests the AI of, say, Painkiller or DOOM 3.

That subjective information is the meat of the review, and what makes them different from each other.

[Alex Handy]

OK, i heard yesterday that Game Informer has a circulation of 1.5 Million. Is this true?!?!

[steve]

OK, i heard yesterday that Game Informer has a circulation of 1.5 Million. Is this true?!?!
That's what the auditors say. Or 1.3 million. Something like that.

Keep in mind that it's the house magazine for Gamestop, and you get a subscription when you buy a discount card. Or is it that you buy a subscription and get a free discount card? I think it's the latter, otherwise it might be classified as unpaid circulation. (Do they still do this? The subscription deal on the GI website has a "$2 for a Gamestop discount card" option, so maybe it's different now.)

Based on the last rate card I have, an ad in GI runs about the same as EGM, which has less than half the circulation. (It also has considerably fewer ads; unless they're turning them down, it shows that advertisers aren't convinced are less convinced that it's "good" circulation.) That could change, however, in 2005.

Any magazine could give away millions of copies every month, but no one would pay for ads based on those numbers because they wouldn't consider them "quality" readers. In this case, no one knows whether or not they're even remotely interested in the content of Game Informer, or in the case of the advertiser, interested in viewing the ad. They may have just wanted the discount card.

[wildpokerman]

I agree with what gamespot did with that review. They review for the average pick up and play gamer and I think they have the most accurate scores in the business for most of the games.

Take for example the dominions 2 review. Tom Chick gave it a best game of the year review, but tom loves strategy games and has a group of friends to play multiplayer with that love strategy games like this. Dominions 2 is a 10 if you're in this situation but think of what Dominions 2 is like if you're not a hard core strategy gamer and don't play turned based strategy games by e-mail. If that's the case Dominions 2 is a poorly documented confusing mess. It has suckfull graphics too.

Steve I don't think that a reviewer should put down their own opinion and let the reader judge for themselves. A talented game reviewer will know his audience and be able to tell them whether or not they will like the game.

[Troy S Goodfellow]

A talented game reviewer will know his audience and be able to tell them whether or not they will like the game.

I'm not going to comment on whether or not I have any talent as a reviewer, but this is asking quite a bit.

To take your Dominions 2 example, a reviewer would not be doing his or her job if they didn't comment on its lack of documentation, questionable interface or poor graphics. This should all be spelled out and should have some weight on the review summary statement or score.

But what if, in spite of all this, the game is really, really good?

To take a counter example - Victoria (one of Paradox's mondo history games). It is/was also poorly documented, full of major interface issues and questionable design decisions. It was tough to navigate, but nowhere near the puzzle that Dom2 was.

I gave Dom2 four stars at DIYGames; Victoria earned only three from me for CGM. Why the difference?

Because when it came down to it, Dom2 is a better designed game. It rewards deep study, it is a constant challenge, and when you fight its learning curve you actually feel like you are learning something. Dom2 is a better strategy game because it is all about the strategy. My summation from that review:

This is the kind of game that rewards being beaten early and often. If you pay close attention to your opponent you can learn a lot.

This takes a lot of time, and a lot of beatings, and a lot of frustration. So I hesitate to recommend this game unequivocally. Also, the diplomacy in the single player game is limited to a Hobbesian state of nature * a war of all against all * and in MP, it can devolve into endless balance of power wars.

In spite of this, Dominions 2 stands out as the best independent strategy game in a very long time. A good player will often be able to make a silk purse from the sow’s ear he/she is dealt at the start. I can think of few other games on the market that reward planning as heavily as Dominions 2 does.

Your mileage may vary. Try the demo first. And find a walkthrough. And start with a human race. And watch your step.

Stick with it though. You’ll be glad you did.

There is no doubt here that I am warning people that this is not for everyone. Screenshots make clear that this is one ugly game. But it is not my job as a reviewer to guess if you will like something. Just to evaluate the game as it is.

The irony, of course, is that you can enjoy a game that is not all that great. "Fun" is only one part of the equation, so you have to think about interface and documentation issues. But if a game has a great design and tries to push the player to think seriously about the game as a game, then I think reviewers have a duty to point this out.

Troy

[wildpokerman]

A talented game reviewer will know his audience and be able to tell them whether or not they will like the game.

I'm not going to comment on whether or not I have any talent as a reviewer, but this is asking quite a bit.

To take your Dominions 2 example, a reviewer would not be doing his or her job if they didn't comment on its lack of documentation, questionable interface or poor graphics. This should all be spelled out and should have some weight on the review summary statement or score.

But what if, in spite of all this, the game is really, really good?

To take a counter example - Victoria (one of Paradox's mondo history games). It is/was also poorly documented, full of major interface issues and questionable design decisions. It was tough to navigate, but nowhere near the puzzle that Dom2 was.

I gave Dom2 four stars at DIYGames; Victoria earned only three from me for CGM. Why the difference?

Because when it came down to it, Dom2 is a better designed game. It rewards deep study, it is a constant challenge, and when you fight its learning curve you actually feel like you are learning something. Dom2 is a better strategy game because it is all about the strategy. My summation from that review:

This is the kind of game that rewards being beaten early and often. If you pay close attention to your opponent you can learn a lot.

This takes a lot of time, and a lot of beatings, and a lot of frustration. So I hesitate to recommend this game unequivocally. Also, the diplomacy in the single player game is limited to a Hobbesian state of nature * a war of all against all * and in MP, it can devolve into endless balance of power wars.

In spite of this, Dominions 2 stands out as the best independent strategy game in a very long time. A good player will often be able to make a silk purse from the sow’s ear he/she is dealt at the start. I can think of few other games on the market that reward planning as heavily as Dominions 2 does.

Your mileage may vary. Try the demo first. And find a walkthrough. And start with a human race. And watch your step.

Stick with it though. You’ll be glad you did.

There is no doubt here that I am warning people that this is not for everyone. Screenshots make clear that this is one ugly game. But it is not my job as a reviewer to guess if you will like something. Just to evaluate the game as it is.

The irony, of course, is that you can enjoy a game that is not all that great. "Fun" is only one part of the equation, so you have to think about interface and documentation issues. But if a game has a great design and tries to push the player to think seriously about the game as a game, then I think reviewers have a duty to point this out.

Troy

Troy It looks like you're quoting your review on diygames and It seems that you're just prooving your poinr. Look at the other reviews on DIY games and they're all strategy games. The only reviews of boxing and soccer games are management sims. For a gamer going to DIY games in the first place Dominions 2 would be a 4 star experience, for an average gamer I'm assuming it would be a 2 star experience at best.

Say it was sold in walmart and you picked up a copy by mistake, you really wanted Diablo 2 but got the name messed up (that's how much you know about gaming). You would have the same reaction as a man waking up to an ugly street hooker after a night of drinking rum coke. The best DOM 2 could get if you didn't go to the Shrapnel forums would be probably 4/10, add in the info on the Ilwinter and Shrapnel sites and you have 6/10, add in multiplayer and people to play it with and hey you're starting to have a decent game.

I digress but Troy I think that most reviewers write to their implied reader without even knowing it.

[steve]

I digress but Troy I think that most reviewers write to their implied reader without even knowing it.
Most reviewers are like their readers, so it's inevitable that they match from time-to-time.

Having said that, if all anyone wants is for reviewers to parrot the general gamer consensus, there's no reason to have multiple review sources. In fact, you could just have "reader rating"sites and skip editorial altogether.

But let's use one example: Tom Chick and Deus Ex. Who didn't at least pause and think about their own opinion about that game after reading Tom's review?

[Troy S Goodfellow]

If you don't know enough about gaming to distinguish between Diablo 2 and Dominions 2, I doubt you read any reviews at all, so the point is completely moot. Was I targeting an implied reader? I doubt it. If I had meant it for serious strategy grognard consumption I certainly wouldn't have added all the caveats I did.

As for the DIY Games genre breakdown, there really aren't a disproportionate number of strategy reviews. Strategy *is* the biggest genre out there (both in sales and titles) and most of the best indie games fit this group. Does it help that I'm primarily interested in strategy games? Absolutely. I think my leaning toward strategy games makes me a better reviewer than a worse one. But I am not the guy you want reviewing a first person shooter, or a bass fishing game. I'm probably also not the guy you want reviewing Gish - even though I loved it - because I don't have a lot of experience with platformers.

I quoted my review not just because it "proved my point" (which is kind of the purpose of quoting anything) but because I think it is an example of what reviewers should do - acknowledge the potential stumbling blocks for people but giving credit where it is due.

Troy

[MattKeil]

I agree with what gamespot did with that review. They review for the average pick up and play gamer and I think they have the most accurate scores in the business for most of the games.

Enter the Matrix - Merely an okay game that benefits greatly from its movie license. 85/100

Whether the question is "How does a 'merely okay game' get an 85%?" or "How could anyone rate that game above 70%, let alone 80?", Game Informer remains suspect.

I am highly dubious about any magazine peddled by a game store. I was on the fence about GI until that EtM review, and that pushed me right off into the "ignoring GI entirely" side.

[bago]

Wait... Is this completely psycho Euro-Andy?

[runesword forger]

GI just dropped a 7.75 on Rome: Total War according to Gamerankings.

So Paper Mario 2 has good company.

[Dave Long]

What else would you expect? Rome: Total War isn't going to resonate with the cool kids. Ancients aren't in, man!

--Dave

[GregB]

I felt that Rome: Total War was way too kiddie.

[Troy S Goodfellow]

GI just dropped a 7.75 on Rome: Total War according to Gamerankings.

So Paper Mario 2 has good company.

7.75?

We're reviewing games to the hundredth now?

Troy

[Robert Sharp]

This discussion is essentially the same as the one about genres, where the debate was whether or not games like Disgaea should be docked for being niche titles. As I said in that topic, I think you have to review a game based on what it is trying to do, and that means taking into account its genre, its target audience, its innovation, etc. There are NO games that every gamer will like. People on this board have hated Half-Life, System Shock 2, Diablo, Baldur's Gate, GTA, and any other game you could name. I suppose you could try to give a rating based on the percentage of gaming public you think woudl like a game, but then no games would score higher than about a 70%, I think.

So, you have to judge the game on its own terms, then, as Steve said, explain what those terms are in the TEXT of the review (anyone who buys based on score, without reading text, is a moron). Be objective when explaining the features. When you are giving what you liked and disliked, explain why, so readers can decide if such things are likely to affect their enjoyment of the game. I have read many reviews where the score initially turned me off, but the text showed me that the issues wouldn't bother me that much. For example, I don't care about the graphics in SRPGs, but they are often docked for them. I just don't see how you can judge a game based on its mass market appeal. Some of the best games out there never sell, or some people just don't get them (Freespace 2 for the former; Planescape:Torment for the latter).