7-9 scale - the discussion

[flyinj]

Where is honest game reviewing now?

The "7-9" scale being talked about is definately in effect at the second most influencial, and, unfortunately, "respected", game review site on the internet (IGN). Several other lesser sites follow this ideal. It sickens me that sites will give a complete and utter shit game, that everyone agrees sucks completely, a 7.0 or higher for god knows what reason (Path of Neo 7.8? True Crime NYC 7.8?!?! What the hell is going on here?)

The only consistantly trustworthy sites left are, in my opinion, Gamespot and Gamespy. Gamespot is, de facto, the end-all-be-all of console game scores. They hit the mark 90% of the time. The PC reviews, however, are a little hit or miss (Vietcong, for example, was an amazing PC game- and they nitpicked every minor problem with it and made it out to be crap). Gamespy seems a bit better rounded handling PC review scores.

There are other sites out there, such as Eurogamer, who judge games harshly. Unfortunately, Eurogamer is almost too harsh in certain instances. However, if Eurogamer says a game is good, it most likely is. I don't think I've ever disagreed with a Eurogamer 8-9/10 score.

Yet, Eurogamer is sometimes so overly critical of games that they miss out on who may actually enjoy the game. I can't cite an example right now, but there have been a few games that Eurogamer has slammed that I and my friends/coworkers have found extremely enjoyable.

The only happy medium I've seen is CGM, mainly with their PC game scores.

They are extremely critical of games, and spell out exactly why a consumer would or would not like the game. They also weigh each side as equally to as possible, allowing in that someone may enjoy the game even though the reviewer didn't.

CGM's console reviews are great in the fact that they cater to a PC audience, and try to break down if a PC gamer would be interested in the game. Unfortunately, I sometimes find them a bit harsh in shutting out certain games to "protect" the PC audience from something foreign.

In summary, I find these sites/magazines the only respectable reviews left on the internet/in print-

Most important part of the post

From most respecable to least:

CGM
Gamespot
Gamespy
Eurogamer
Yahoo!Games! (GamesDomain?)
OPM
OXM
1Up
CGW
VoodooExtreme
EvilAvatar
ConsoleGold (whatever it's called now)
PCGamer
IGN
GamePro
MaximOnline

[Ephraim]

And this is why I like the meta-ranking sites like Gamerankings (http://www.gamerankings.com).

Not only do you get a good "aggregate picture", it's also easy to see how much each site varies from the average score. These stats confirm that certain sites (IGN among them) are usually scoring games higher than the average of all reviews, and allows you to discount the offendingly high scoring reviews.

[Qenan]

I basically only use the meta sites now. Individual reviews are too variable, and I'm usually only interested in finding out whether something is a plausible purchase for the price. (To some extent, the cheaper it is, the lower an aggregate score I will tolerate.)

[Dave Long]

CGM's console reviews are great in the fact that they cater to a PC audience, and try to break down if a PC gamer would be interested in the game. Unfortunately, I sometimes find them a bit harsh in shutting out certain games to "protect" the PC audience from something foreign.

I don't think we're trying to protect the PC audience (except maybe financially), just looking at it from that perspective as a PC gamer. All too often there are console games that get very high review scores in console mags because those folks simply haven't even played a PC game in their lives. Either that or they do so infrequently they're unaware of a partcular console game being super-derivative of something on the PC.

I mean, for a PC gamer, something like Rainbow Six: Lockdown, that I reviewed in last month's mag, just isn't going to cut it at all. It's dull. That's true for me as well since I am a PC gamer as well as a console one. I guess that's probably the best way to view those reviews... the guys writing them are hardcore PC and console gamers so you know we're coming at it from the perspective of someone pretty knowledgeable of all forms of gaming.

I'm sure Tom or Steve can tell you more if they pop in here.

--Dave

[Jazar]

How about we, the people of this board who I consider very bright individuals for the most part, pay less attention at the numerical value of a review and more attention on the actual text of the review.

[Dave Long]

I already do that. :)

I've started buying just about every game magazine on the market. I like to know who's doing what and how well. I think it's also helping me be a better writer. With subscriptions so cheap for most print mags and even cheaper if you get them through Zinio (the Ziff ones), there's no reason someone like me can't stay informed.

--Dave

[Greg Kasavin]

Just a random comment to the original post: I have no idea how you could have come away from my Vietcong review having inferred I "made it out to be crap." I really liked that game. Maybe you're referring to the sequel or the expansion pack. Thanks for the kind words about GameSpot's reviews, though. With the so-called next generation of games upon us, I think most publications are taking the opportunity to reevaluate the way they review games, so hopefully readers will see a positive trend in the next year. Ron Gilbert's blog at grumpygamer.com recently made a very interesting point that the quality of game reviews may improve as the content of the games themselves does. A cerebral game like Killer7 forces a critic to write about it differently than some typical FPS, so as long as game makers continue to produce interesting stuff, I have faith that the quality of game writing will steadily increase as well.

[extarbags]

Hey Greg, whatever happened to that New Thing in Game Ratings you said you couldn't talk about yet a while ago?

[Tyrion Lannister]

Errm, HonestGamers and Digital Entertainmentand News and that's about it I'm afraid.

The rest are really terrible.

I give GameSpot a 2.2/10.0.

[Mark Asher]

The problem with the 10 point scale is that no one uses a 5 as the baseline for an average game. Five point scales tend to be a bit better about the mid-point being the score for an average game.

Just think how many two-star games and even 1-star games you see scored at places that use a 5 star game. On a 10-point scale those scores would be 20s and 40s -- something you almost never see.

[Troy S Goodfellow]

EDigital Entertainmentand News

The site that has these gems from its Blitzkrieg II review? (http://www.dignews.com/review.php?story_id=12137)

As good as consoles are for other genres of games, the PC’s gaming acumen seems solidified by the fact that war simulations are irreversibly connected to the platform.

If you were asked to think of another war-based RTS game, chances are that you would choose the Command and Conquer series. Blitzkrieg 2 is the complete opposite to that series, which isn’t necessarily a bad thing.

If you want to be successful in this game, you need to be tactically astute. You are given a small core of units, which it is your mission to command them.

Troy

[Dr Fear]

Just a random comment to the original post: I have no idea how you could have come away from my Vietcong review having inferred I "made it out to be crap." I really liked that game.
Well, you gave it a 7.9 (http://www.gamespot.com/pc/action/vietcong/review.html), which is on the lower end of the 7-9 scale.

[Tyrion Lannister]

Well, yes and DEN does have awful layout. But they take their time to actually play the games, which is more than can be said for most sites, and the ratings seem far less arbitrary. They also appear to have the reviewers review games in genres they like.

If I read another review which starts with, "I hated the first one, and this one is even worse" I'm going to scream. That means you shouldn't be reviewing it. That's like sending a Fangoria reviewer to Pride and Prejudice: "No gore, lots of irritating chicks giggling and fruity accents. 2/10"

[Troy S Goodfellow]

They also appear to have the reviewers review games in genres they like.

I'm not sure that this is unique. Every place that I have written for has asked what genres I felt most comfortable writing about and I've rarely written about any genre I clearly stink at (like FPS).

As I read the terrible opening paragraph of DEN's AoE 3 review (http://www.dignews.com/review.php?story_id=12107), I suspected that maybe it was just this one writer.

Then I read another writer claim that Act of War has an unlimited resource model and cites the oil wells as an example (http://www.dignews.com/review.php?story_id=10449). In fact, the wells are finite in capacity and can be exhausted. Only prisoners can bring unlimited funds, but slowly.

And another refers to Imperial Glory as yet another military RTS set in Napoleonic Europe. I had no idea we were drowning in them...

And someone gave Superpower 2 a 7.8.

I think I'm done with DEN.

Troy

[andrew_fm]

Well, you gave it a 7.9 (http://www.gamespot.com/pc/action/vietcong/review.html), which is on the lower end of the 7-9 scale.
7.9
good

[Dave Long]

Well, yes and DEN does have awful layout. But they take their time to actually play the games, which is more than can be said for most sites, and the ratings seem far less arbitrary. They also appear to have the reviewers review games in genres they like.

If I read another review which starts with, "I hated the first one, and this one is even worse" I'm going to scream. That means you shouldn't be reviewing it. That's like sending a Fangoria reviewer to Pride and Prejudice: "No gore, lots of irritating chicks giggling and fruity accents. 2/10"

Eh? I think Gamespy's reviews are some of the best among the websites, but I think there's this thing that hardcore gamers have against them because Gamespy will forever be associated with the product that started the whole thing. People aren't willing to take their reviews seriously because of that.

I can tell you that I've been working with all of the writers at GamerDad (I'm the EIC over there) to try to improve all their writing and one guy in particular who clearly plays the games thoroughly, but had a lot of trouble getting it into text, has improved dramatically over this past year. For those that aren't aware, our reviews are no different than any other site when it comes to the main body, we just tack on the parental part at the end in the Kid Factor. So don't feel like you're not welcome if you're not a parent.

Anyway, everywhere I've ever worked with has put people on the reivews that like the genre or are even fans of the games. That probably leads to some score inflation sometimes, but it at least gets you well-reasoned text when it comes to the good and bad because the people want to play those games in the first place.

--Dave

[Sidd_Budd]

Well, you gave [Vietcong] a 7.9 (http://www.gamespot.com/pc/action/vietcong/review.html), which is on the lower end of the 7-9 scale.
7.9 is above average for Gamespot; only 30% of all their games get a score of 7.9 or higher. Not all sites are equal when it comes to the 7-9 stereotype.

EDIT: Removed inaccurate reference to PC Gamer's revision to their scoring system. I thought they had changed the descriptions given to the different percentages, to encourage more stringent scoring. Upon closer inspection, it appears they didn't change the old scoring system, but merely added a graphical bar (like a horizontal thermometer), to their reviews for easier visual discrimination

[MikeSofaer]

I was thinking about this last night, and I had an incredibly brilliant idea so hang on to your seats, ladies and gentlemen.

OK. Get ready.

A normalized rating scale. Not a shodily normalization like the aggregate sites do, but a within-site normalization that doesn't even give a raw score, merely how many standard deviations above or below that site's mean the game is. Reviewers could be given any scale they wanted to use, since all scores would be statiscally comparitive, score inflation of all games as a whole could not happen.

So? How do you like them tasty delicious Granny Smith apples?

[Ben Sones]

Just a random comment to the original post: I have no idea how you could have come away from my Vietcong review having inferred I "made it out to be crap." I really liked that game.
Well, you gave it a 7.9 (http://www.gamespot.com/pc/action/vietcong/review.html), which is on the lower end of the 7-9 scale.

I can't imagine how anyone could extract what he really thinks of the game from that review. Most of it is just carefully neutral description of game features--the sort of thing you'd expect to find in a preview, or bullet-pointed on the game box--without any commentary on whether he thinks those features hurt or help the gameplay, let alone how. The few statements of opinion that are there are often concealed in mind-achingly labyrinthine prose full of qualifiers (that mostly boil down to something along the lines of "it's the sort of thing you'll probably like, if you like that sort of thing"). I mean, what am I supposed to make of statements like this:

The events of the campaign are predictable enough in and of themselves, but they may take you by surprise as they come up, and they're the sorts of experiences you'd probably hope to get out of a first-person shooter set during the Vietnam War.

So the campaign is predictable yet surprising, and probably what you'd expect, depending on your expectations. That's illuminating.

I had to read this sentence three times just to figure out what he was trying to say:

it's possible to get Vietcong running smoothly if it doesn't run smoothly on its own, though at this point it's not unreasonable to think that PC games shouldn't require this sort of fine-tuning.

I'm surprised that he didn't add that it's also not unreasonable to not think that games shouldn't require this sort of fine tuning, depending on what you think. Or not.

There are a few genuine bits of critical commentary in there, but if you trimmed the review down to just those, it would only be about two paragraphs long. And it would be better for it.

[Dave Long]

Every time I see a "may" or "might" or "hope" or "probably" in a GamerDad submission lately, I cut it right the fuck out. I hate those words in reviews and I'm trying hard not to use them myself.

--Dave

[Sidd_Budd]

I was thinking about this last night, and I had an incredibly brilliant idea ...
A normalized rating scale. Not a shodily normalization like the aggregate sites do, but a within-site normalization that doesn't even give a raw score, merely how many standard deviations above or below that site's mean the game is.
I love this idea, and it's how I keep my own database of game rankings. There's plenty of people who know how to convert scores to standard normal distributions; it would be pretty simple to just convert the raw scores even for an aggregation site like Game Rankings.

Hardly anyone knows how to interpret standard deviations, so I think the final score should give an *actual* percentage as well. For example, a score of 80% would mean that this game achieves an aggregate ranking higher than 80% of all the games reviewed. The only problem would be scores wouldn't be static; they could change as the universe of games reviewed gets bigger. I don't know how difficult that would be to maintain at a commercial site, but it's not a problem for my Excel spreadsheet.

[Tyrion Lannister]

Oh shit, yeah, sorry forgot about GamerDad. My first stop for game reviews, and so I naturally forgot to mention it...

And I'm sorry to say that GameSpy and GameSpot (and the utterly terrible EGM) suffer from the MacReview syndrome that IGN is so well known for. But with added Edge-like arsiness and snideness in their reviews and rating systems. V. Poor.

If I believed what they had written I wouldn't have bought Gun or Shadow the Hedgehog, and I'd have missed out on a lot of entertainment (OK, so shadow is a little flawed, but only a little...)

[RickH]

A normalized rating scale. Not a shodily normalization like the aggregate sites do, but a within-site normalization that doesn't even give a raw score, merely how many standard deviations above or below that site's mean the game is. Reviewers could be given any scale they wanted to use, since all scores would be statiscally comparitive, score inflation of all games as a whole could not happen.

PSM used to do something like that, they'd place the score of a game in the context of 3 or 4 other games in that genre so you could see where it fell (in their opinion of course) in comparison with its competition. Then they stopped doing that. Wasn't "edgy" enough, I guess.

[MikeSofaer]

I was thinking about this last night, and I had an incredibly brilliant idea ...
A normalized rating scale. Not a shodily normalization like the aggregate sites do, but a within-site normalization that doesn't even give a raw score, merely how many standard deviations above or below that site's mean the game is.
I love this idea, and it's how I keep my own database of game rankings. There's plenty of people who know how to convert scores to standard normal distributions; it would be pretty simple to just convert the raw scores even for an aggregation site like Game Rankings.

Hardly anyone knows how to interpret standard deviations, so I think the final score should give an *actual* percentage as well. For example, a score of 80% would mean that this game achieves an aggregate ranking higher than 80% of all the games reviewed. The only problem would be scores wouldn't be static; they could change as the universe of games reviewed gets bigger. I don't know how difficult that would be to maintain at a commercial site, but it's not a problem for my Excel spreadsheet.
Cool, nice to know it actually works.
I can see a percentage ranking of that sort being useful, as well as graphs of score distribution curves and where the game falls on them, you could restrict the set to games of that genre, or by that reviewer, etc.

Making this tool available to the reviewer when he's submitting the score would be a big step up, too, since it would force him to really evaluate in context.

[noun]

We struggled with the rating system for our site. Everyone had a different opinion: some wanted the 1-100 rating to match industry standards, but few folks wanted to deal that level of granularity (which ends up being arbitrary, anyway). Some (like myself) didn't want any numbers whatsoever, but to get listed on sites like GameRankings, it was argued that we had to have numbers of SOME sort.

In the end we went with a 5 point system. 5 games we love, 1 games are crap, and everything else usually falls in between.

Intresting idea about the aggregate number. I'll have to look over what we have to date to see if that makes any difference.

[Ben Sones]

PSM used to do something like that, they'd place the score of a game in the context of 3 or 4 other games in that genre so you could see where it fell (in their opinion of course) in comparison with its competition. Then they stopped doing that. Wasn't "edgy" enough, I guess.

Not edgy, maybe, but also not useful. No offense to the people who brought it up, but trying to come up with ways to make ratings more mathematically accurate is missing the point. A rating is not a precise, objective measurement. It's just one person's opinion, and any given pub has a bunch of different people generating those opinions. I mean, sure, you can come up with some model that says "game X scored higher than 80% of the other games in our database," but what does that actually mean? Pretty much nothing. It's just an arbitrary number. You'd probably have a different arbitrary number if they had given the review to a different person. I don't understand what meaningful information I'm supposed to be able to derive by comparing how one person rated one game to how a different person rated a different game.

Ratings are already way more complicated than they need to be; they don't need to get more complicated. Percentages, ten point scales with decimals, multiple rating categories cooked down into mean averages... all that stuff is utterly pointless. I'd be happy with a rating system that had only two ratings ("thumbs up" and "thumbs down"). I'd even be happy with no ratings at all.

[Dave Long]

I've always been a big proponent of the Thumbs Up/Thumbs Down method but you simply do not get listed on Gamerankings if you do that, and apparently that's a big means of getting people to visit your website.

It sucks.

--Dave

[Tyrion Lannister]

GameRankings is a curse.

[Dave Long]

I agree. I think they also won't put you in there unless your reviews are some arbitrary (rather long) length. I wanted to shorten up the reviews at GamerDad and it just can't be done and still be listed at Gamerankings.

--Dave

[RickH]

I mean, sure, you can come up with some model that says "game X scored higher than 80% of the other games in our database," but what does that actually mean? Pretty much nothing. It's just an arbitrary number. You'd probably have a different arbitrary number if they had given the review to a different person. I don't understand what meaningful information I'm supposed to be able to derive by comparing how one person rated one game to how a different person rated a different game.

Unless you are someone wanting a new fighting game and wanted to know which game PSM thinks is better, Tekken, SC, VF, MK, etc. Are you really saying comparisons are irrelevant to a review or body of reviews?

[andrew_fm]

GameRankings is a curse.Mind elaborating? It's a curse that it collects scores from various sites and presents an average?

[Dave Long]

GameRankings is a curse.Mind elaborating? It's a curse that it collects scores from various sites and presents an average?

Basically, yeah, it is. They're interpreting the ratings systems to mean whatever they want. Since you and I know that many sites never use anything below a 5 on their 1-10 scales, it translates all reviews at those sites into scores of about 70% or higher, which means even bad games look much better.

Those sites/mags that use five-star scales and use the whole thing end up waaaaayyyyy at the bottom of the list of reviews if they give something three stars while everyone else was giving a game 7.5 or 8.0. What happens is unless you give games something in that 7-9 range, no one ever clicks on your review. Given this, is it no surprise that many of the smaller sites tend to have seemingly inflated scores so that they're a bit higher on the list?

Gamerankings is partially to blame for the mess that is online game reviews.

--Dave

[Ben Sones]

I've always been a big proponent of the Thumbs Up/Thumbs Down method but you simply do not get listed on Gamerankings if you do that, and apparently that's a big means of getting people to visit your website.

It sucks.

It does suck. Gamerankings should have paid more attention to how Rotten Tomatoes does things--they boil review scores down to the most simple common component (liked it/didn't like it) rather than the most complex. If your review database service can't handle reviews that stray too far from your format, then you have a sucky service. It's not unreasonable to think that pubs shouldn't have to tailor their format to suit Gamerankings' needs.

[sluggo]

Every time we give a game 3 or even 3.5 stars, we get angry phone calls from publishers threatening to pull advertising because we've "trashed" their game. What are they talking about? The "60" or "70" that Gamerankings attaches to our score, which is invariably 10-15 points lower than scores those games get on other sites. We've even sent GameRankings a conversion chart to help them line up our scores with the 7-9 scale, and they just ignore it.

I'll have some more to say on this shortly, but I think the idea of a metasite that uses standard deviations from a site's average to put review scores in perspective is absolutely brilliant.

[GregB]

Except at Rotten Tomatoes, if you use a 5-star scale, they rank 3-stars as a rotten tomato. At X-Play, where I do some reviews, a 3 out of five is not a pan.

Ratings suck.

[Ben Sones]

Except at Rotten Tomatoes, if you use a 5-star scale, they rank 3-stars as a rotten tomato. At X-Play, where I do some reviews, a 3 out of five is not a pan.

It's still better than Gamerankings' system. That discrepancy would only apply to a small number of your reviews (the ones that score between 3 and wherever your real breaking point would fall... 2.5?), and it could be easily be fixed by simply asking individual sites where they want the breaking point to fall.

But yeah, ratings suck.

[Aki]

Change the word 'ratings' to 'grades' and presto-chango! You have a discussion about what's wrong with education in America.

[Sidd_Budd]

A rating is not a precise, objective measurement. It's just one person's opinion, and any given pub has a bunch of different people generating those opinions...I don't understand what meaningful information I'm supposed to be able to derive by comparing how one person rated one game to how a different person rated a different game.
You seem to be implying that a system of measurement is useless unless it is 100% error proof and objective. I take the point of view that any measuring device -- even heart rate monitors and quantum clocks -- has some degree of error. It's up to the community of users/consumers to decide whether a specific measurement system is good enough.

In theory, aggregation sites like GameRankings should provide the best index of quality, since aggregating scores tends to reduce error in test construction theory (error or bias in one score is more likely to be "evened out" with biases from other scores). In practice, GameRankings has a clumsy method of normalizing scores from different review sites (summarized well by Dave Long), so some of its potential is wasted.

Here's the top 5 rated games in my database -- all representing an unholy amalgamation of different sites, scoring systems, reviewers, and genres.
Half-Life 2
Half-Life
Civilization 4
Unreal Tournament
Unreal Tournament 2004

Here's the bottom 5:
Extreme Paintball
Catfight
Swamp Buggy Racing
Panty Raider
MTV: Celebrity Deathmatch

If ratings are entirely subjective and arbitrary, then there should be *no* difference in actual quality between these two lists, since I used a method of ranking that is, to you, no different than flipping a coin. There's no way any regular reader of Qt3 would see the quality of these two lists as equal. Therefore, rating systems -- even the 7-9 ones -- must be somewhat useful as an index of actual quality.

I've never suggested that people should only use a rating to determine their decision to buy a game, and I subscribe to all the magazines because I love to read reviews. Its obvious to me, however, that rating systems do show relevance to actual game quality, even though each individual review has some subjectivity, error & bias (as used in measurement theory). It appears that you can compare rankings across different sites that use different rating systems, assuming you properly apply methods of equalizing the data (like normalizing scores). In a perfect (statistical) world, each site would have a single reviewer. In practice, it appears consistent editorial policy curbs individual excess in grading standards.

[LarryLard]

Every time we give a game 3 or even 3.5 stars, we get angry phone calls from publishers threatening to pull advertising because we've "trashed" their game.

You should record and publish those phone calls.

[Dave Long]

Yeah, or do the "Can we quote you on that?" like Bauman says he does when gets those calls.

--Dave

[Kevin Grey]

It does suck. Gamerankings should have paid more attention to how Rotten Tomatoes does things--they boil review scores down to the most simple common component (liked it/didn't like it) rather than the most complex. If your review database service can't handle reviews that stray too far from your format, then you have a sucky service. It's not unreasonable to think that pubs shouldn't have to tailor their format to suit Gamerankings' needs.
Back to top
View user's profile Send private message Send e-mail Visit poster's website

IIRC Rotten Tomatoes did this for a little while with game reviews and the end result was almost every game was 80+% "Fresh" since even the harshest game reviews usually have some stupid "worthwhile for those who follow the genre" caveat.

[John Doyle]

My issue with sites like Gamerankings and Metacritic is that they are not in any way, other than accidental, reflections of game quality.

1. The sample size is too small to be statistically relevent.

2. The reviewers at the various sites used to come up with a ranking are not judging on the same scale or, more importantly, examing the same things. Some reviewers are looking at overall game quality, some at perceived consumer appeal and others are looking to impress an editor or the demographic of their readers. For example, some sites care about sound enough to call it out separately, others do not. This leads to wildly different views of the same games and indeed of features within the game.

3. All games are not judged by consistent standards. There is no question that sequels are compared to an original, that annual franchises are viewed differently than new IP and that the new and fresh is preferable to the familiar. Games are often judged against the current standard of the genre and/or what competitive products are on the market. While these standards are fine when it comes to informing the general consumer, they cloud the picture if one attempts to use these reviews as an objective measure of product quality.

4. Developers actually use these sites to guide feature design. I've worked at several companies who have established a "Gamerankings" or "Metacritic" target for their products. In my opinion this is absurd. Using such a target, designers will inevitably tailor their design towards the bias of game reviewers. This rarely results in good game design, because game reviewers are not ever the target audience of a consumer software entertainment product. They tend to be, no offence to any reviewers our there, considerably more jaded than the typical consumer.

There is no truly objective measure of video game quality that I've come across. I know I base my personal measure of success on sales numbers. Those are the numbers that tell me if I still have a job after the game ships.

I have no issue with people using these sites to help make purchasing decisions, but the aggregate numbers should be taken with a large grain of salt if one is attempting to determine product quality.

[Sidd_Budd]

My issue with sites like Gamerankings and Metacritic is that they are not in any way, other than accidental, reflections of game quality...

4. Developers actually use these sites to guide feature design. I've worked at several companies who have established a "Gamerankings" or "Metacritic" target for their products. In my opinion this is absurd.
Let me get this straight -- in your opinion, it is utter chance that multiple reviewers rated Half-Life 2, Civ 4, and Unreal Tournament 2004 higher than Catfight, Panty Raider, and MTV: Celebrity Deathmatch. The fact the higher rated games just happen to be better products than the lower rated ones is purely random. In your mind, the most cogent explanation for why the top rated games at Gamerankings and Metacritic land on multiple magazine/website "Top 50 Games of All Time" lists, and would likely match up fairly well with Qt3ers lists of "Top 5 Games for a Desert Island" is coincidence.

I don't agree, but I'm pretty sure I'd like to play poker against you. Any two cards can potentially win; it's pretty random.

Since you work in the industry, I can understand your preference for sales numbers as a measure of success; it directly impacts your job security. As a consumer, however, I like that developers shoot for high ratings. They could just as easily target the "Mall of America Tycoon" segment, since average or below average-rated titles like that regularly make the top selling list.

Many of the posters on this forum are the ones who will be rating these games, even though some wish for a day when they can write for the ratingless New York Times Games Supplement and forget about the "7-9" years. I tend to agree with their ratings, jaded or not; there's a much better chance I'll like a higher rated game than a lower rated one. I'm glad developers are trying to make games that will please them and get high ratings, because I want more high-quality games.

[wildpokerman]

Every time we give a game 3 or even 3.5 stars, we get angry phone calls from publishers threatening to pull advertising because we've "trashed" their game.

And that is precisely the reason that people do not trust review scores.

[Matthew Gallant]

I think GameRankings would be extremely useful if it were to say, drop the 50 highest scores. Take Jade Empire for example. I may be off in my counting by one or two, but it takes 59 review outlets just to start getting to to scores under 90%.

And I'd really like to know the criteria for becoming one of the "bolded" sites/mags. What I mean is, only scores from the ones in bold get used in computing the average. Sorry "Gamer Within", you just don't have the clout of "Deeko.com" and "Gamenikki", or even "Lawrence".

[John Doyle]

Let me get this straight -- in your opinion, it is utter chance that multiple reviewers rated Half-Life 2, Civ 4, and Unreal Tournament 2004 higher than Catfight, Panty Raider, and MTV: Celebrity Deathmatch. The fact the higher rated games just happen to be better products than the lower rated ones is purely random. In your mind, the most cogent explanation for why the top rated games at Gamerankings and Metacritic land on multiple magazine/website "Top 50 Games of All Time" lists, and would likely match up fairly well with Qt3ers lists of "Top 5 Games for a Desert Island" is coincidence.

I don't agree, but I'm pretty sure I'd like to play poker against you. Any two cards can potentially win; it's pretty random.

Sure, the ranking systems work fine if you look at the top few and bottom few games. It's fairly simple to agree that Catwoman is a bad, bad game and that Civilization is a good one. Where the system breaks down is at any point between universal acclaim and universal horror.

There is no definition of quality used by these systems. If I actually read the reviews, I can, via the syntax of the reviewer, gain an understanding of how the reviewer felt about the product. If I look at an aggregate score all I'm getting is a normalized (somewhat) integer.

Let's look at 3 recent racing game scores, for example. For the Xbox 360, on Metacritic, PGR 3 is ranked above NFS: Most Wanted which is ranked above Ridge Racer. All three are fairly close in score, but by the ranking theory of game quality PGR 3 is clearly better, right? Unless I don't like loaded track racing, I suppose. Of course, Most Wanted is better than Ridge Racer..oh, unless I just want a cool arcade drift game without cops.

My point is that assigning a numeric score based on reviews that take very differernt criteria of quality in to account is folly. It requires that you ignore the variables that make the individual reviews different in the first place.

By all means use the top and bottom half of these lists to make binary good/bad game decisions, but don't try to guess which game is better out of several with similar scores. Read the actual reviews, don't rely on gamerankings or metacritic to determine quality.

Btw, I'm an analytical chemist by training so I must admit I have a bit of a tough time with this sort of aggregate approach to disparate pieces of data.

[Marcin]

And I'd really like to know the criteria for becoming one of the "bolded" sites/mags. What I mean is, only scores from the ones in bold get used in computing the average. Sorry "Gamer Within", you just don't have the clout of "Deeko.com" and "Gamenikki", or even "Lawrence".

Pure numbers game. From the site:

Q. What does it take to get a site included in the composite score of Game Rankings?

# A. This is the most commonly asked question. The things we look for when adding a new site are: At Least 300 archived reviews if they review multiple systems or 100 reviews if they concentrate on only one system or genre.
# The site does at least 15 reviews a month.
# The site is visually appealing and looks professional.
# The site reviews a variety of titles.
# The site has it's own domain name and is not hosted on GeoCities or another free server.
# The reviews need to be well written.
# The site conducts itself in a professional manner.

[Dhruin]

I've never understood this. In the same sentence that some people decry useless scores in reviews they race off to Gamerankings for nothing but an aggregate score. I thought it was the text that was important?

[Mark Asher]

Every time we give a game 3 or even 3.5 stars, we get angry phone calls from publishers threatening to pull advertising because we've "trashed" their game. What are they talking about? The "60" or "70" that Gamerankings attaches to our score, which is invariably 10-15 points lower than scores those games get on other sites. We've even sent GameRankings a conversion chart to help them line up our scores with the 7-9 scale, and they just ignore it.

I'll have some more to say on this shortly, but I think the idea of a metasite that uses standard deviations from a site's average to put review scores in perspective is absolutely brilliant.

Gamerankings is taking the wrong approach, then. What they should do is convert each site's score into a letter grade and then average those together. Then an 8/10 and 80% might be a "B" and a 3/5 might also be a "B".

[Rob Beschizza]

RE: Normalized game ratings. IIRC gamerankings lets you view all scores for a given publication (can't find it right now, though!) Couldn't those pages just be scraped to produce normalized averages?

[Rob Beschizza]

Every time we give a game 3 or even 3.5 stars, we get angry phone calls from publishers threatening to pull advertising because we've "trashed" their game.

Do you mean that their advertising people are calling your editorial people? Or that your advertising people are telling editorial that they're getting threatening calls?

I only ask because I've yet to actually hear of a real world situation where pressure is exerted by advertising $$$ on the newsroom (admittedly, I'm not in game journalism.). I'd started to think that the common reader complaint -- that the reviews were driven by advertising -- is a whiny fanboy myth.

[DennyA]

Never once got pressure from an ad person to alter a review in my 15 years of magazine editorial work.

I did get a couple of calls from game companies telling me they were going to pull their ads. (Including one from the CEO of a mid-sized company -- I felt special.) My basic response is "Oh, well, you need to be sure to tell the advertising people that, it's not really my department."

If a publisher thought a review had unfair elements, I was more than happy to discuss those. But the ad threats were just a waste of time. Even if you discard the ethical issues, the editors are too far removed from the money chain to care.

And the threats are counterproductive anyway. They didn't directly affect coverage, but you have to think that when you're limited in space and you have two cool games to cover but only room to cover one of them, the game from the company whose enraged CEO tried to use monetary pressure to influence your editorial is going to take at least a subconscious -5 on the 1d20 roll to see which one gets coverage...

[Tyrion Lannister]

The ad-threats go to the ad people, the polite requests for another reviewer to voice their opinion go the editor.

Both work. Not that I've done that sort of job for many years.

:twisted:

[Sidd_Budd]

Let's look at 3 recent racing game scores, for example. For the Xbox 360, on Metacritic, PGR 3 is ranked above NFS: Most Wanted which is ranked above Ridge Racer. All three are fairly close in score, but by the ranking theory of game quality PGR 3 is clearly better, right? Unless I don't like loaded track racing, I suppose. Of course, Most Wanted is better than Ridge Racer..oh, unless I just want a cool arcade drift game without cops.
I find your last point about game preference close to a straw man argument; of course personal taste for a genre, or specific types within a genre, will override any score. I'll never buy a sports game, regardless of ranking, because I don't like sports. The remainder of my post assumes that gamers have interest in racing games in general.

Statistics can't say whether PGR 3 is "clearly better", but it can provide the probability that chance produced higher scores for PGR 3, relative to the other two games. If chance can't account for the difference, something else, perhaps an actual difference in quality, may explain the scores. You need a standard deviation (or standard error, to be more precise) to perform this test; you can't just provide three average scores.

Metacritic ratings for the 3 games are 88 (PGR 3), 84 (Most Wanted), and 78 (Ridge Racer). Assume the statistical tests find that chance might reasonably account for the 4 point difference between PGR 3 and Most Wanted, but that it's highly improbably chance accounts for the 10 point PGR 3/Ridge Racer difference. I would expect to find that, among real-world players of all 3 games, there'd be debate over whether PGR 3 or Most Wanted is superior, but *most* gamers would find Ridge Racer the least compelling of the three.

That's why I argue that ratings are useful tools; because I believe they coincide with mass opinion a significant majority of the time. I disagree with you that ratings will only be useful at the extreme upper and lower ends of a population. I think many people would see similar meaningful distinctions between the four segments of games ranked 90-95%, 75-80%, 60-65%, and 45-50%, not just between the highest and lowest. At the same time, I think we are in agreement that there would be more debate between people as to the relative quality of the 92% versus the 93% games, or the relative suckitude of the 46% versus the 47%.

[DeepT]

Every time we give a game 3 or even 3.5 stars, we get angry phone calls from publishers threatening to pull advertising because we've "trashed" their game.

So you work for the gaming press? And does your print mag or web site (or whatever) capitulate to these demands?

Its stuff like this that makes the whole game review system worthless. I wonder why on gamestop the review might rail about how crappy a game is, yet the score is still like a 6.5.

What the gaming press needs is some outside media watchdog group to keep them on the strait an narrow or the gaming press needs to form something like a corperate union where if any member is thretened like that, all members will take said publisher to take a hike.

[Ben Sones]

It's not really necessary, though, because those sorts of threats are usually empty ones. Publishers advertise purely out of self-interest. They don't buy ads as a reward for good reviews; they buy ads because they want to reach a certain demographic. They may hurt the publication by pulling those ads, but they hurt their own marketing efforts, too. So they'll bark a lot about negative reviews, but it's pretty uncommon for them to actually bite.

[John Keefer]

Every time we give a game 3 or even 3.5 stars, we get angry phone calls from publishers threatening to pull advertising because we've "trashed" their game.

So you work for the gaming press? And does your print mag or web site (or whatever) capitulate to these demands?



He works for GameSpy, as do I, and NO, we do not capitulate to these demands. They can rant all they like and the sales people can moan all they like, but there HAS to be a separation of church and state for there to be any credibility. Luckily my bosses understand that. If they didn't, I'd quit. If a publisher points out factual errors, we can fix it (and are happy to). But if they disagree with our opinion on their biggest title of the year, too bad. We have no agenda or vendetta out for any publisher, any more than we are in bed with any publisher because they use our technology in their games. If it sucks in implementation, we say so (see Mr. Chick's review of BF2: Special Forces).

I don't expect everyone to believe me, but that's the way we run our operation.

[Delsyn1]

Oh shit, yeah, sorry forgot about GamerDad. My first stop for game reviews, and so I naturally forgot to mention it...

And I'm sorry to say that GameSpy and GameSpot (and the utterly terrible EGM) suffer from the MacReview syndrome that IGN is so well known for. But with added Edge-like arsiness and snideness in their reviews and rating systems. V. Poor.

If I believed what they had written I wouldn't have bought Gun or Shadow the Hedgehog, and I'd have missed out on a lot of entertainment (OK, so shadow is a little flawed, but only a little...)

What is the "MacReview" syndrome?

[Troy S Goodfellow]

On a side note, one of my students stopped by to talk to me this afternoon before he ran off to one of his clubs. Good kid, likes to talk about games, really pushing Call of Duty 2 on me.

Conversation turned to magazines and he mentioned that he subscribed to CGW. And that he couldn't understand why their reviewer "hated Age of Empires 3". I explained that a 3/5 is hardly hate, and that the review explained in quite a bit of detail where it came up short. He's still convinced it's a five-star game.

So Tom, there's a junior in Maryland who's a little mad at you.

Troy

[Tom McNamara]

RE: Normalized game ratings. IIRC gamerankings lets you view all scores for a given publication (can't find it right now, though!) Couldn't those pages just be scraped to produce normalized averages?

Here you go (http://www.gamerankings.com/itemrankings/sites.asp?sort=8&includeoth=y), sorted by average game score. IGN clocks in at 70.6%, which would, in theory, blow flyinj's argument out of the water. Stranger things have happened :)

[TomChick]

So Tom, there's a junior in Maryland who's a little mad at you.

Tell him to get in line. :)

That's really sad that a 3-star rating, which I consider a qualified recommendation, is dismissed as "hating". I think it's a sign of a) the insidious influence of the 7-9 system, and b) the fact that most reviews are far too generous and enthusiastic, which makes those of us with a bit more discrimination seem like Scrooges.

Not that we aren't, mind you, but I actually *like* Age III. For Pete's sake, it's even on my Top Ten of 2005 list right now.

-Tom