7-9 scale - the discussion

[flyinj]

Where is honest game reviewing now?

The "7-9" scale being talked about is definately in effect at the second most influencial, and, unfortunately, "respected", game review site on the internet (IGN). Several other lesser sites follow this ideal. It sickens me that sites will give a complete and utter shit game, that everyone agrees sucks completely, a 7.0 or higher for god knows what reason (Path of Neo 7.8? True Crime NYC 7.8?!?! What the hell is going on here?)

The only consistantly trustworthy sites left are, in my opinion, Gamespot and Gamespy. Gamespot is, de facto, the end-all-be-all of console game scores. They hit the mark 90% of the time. The PC reviews, however, are a little hit or miss (Vietcong, for example, was an amazing PC game- and they nitpicked every minor problem with it and made it out to be crap). Gamespy seems a bit better rounded handling PC review scores.

There are other sites out there, such as Eurogamer, who judge games harshly. Unfortunately, Eurogamer is almost too harsh in certain instances. However, if Eurogamer says a game is good, it most likely is. I don't think I've ever disagreed with a Eurogamer 8-9/10 score.

Yet, Eurogamer is sometimes so overly critical of games that they miss out on who may actually enjoy the game. I can't cite an example right now, but there have been a few games that Eurogamer has slammed that I and my friends/coworkers have found extremely enjoyable.

The only happy medium I've seen is CGM, mainly with their PC game scores.

They are extremely critical of games, and spell out exactly why a consumer would or would not like the game. They also weigh each side as equally to as possible, allowing in that someone may enjoy the game even though the reviewer didn't.

CGM's console reviews are great in the fact that they cater to a PC audience, and try to break down if a PC gamer would be interested in the game. Unfortunately, I sometimes find them a bit harsh in shutting out certain games to "protect" the PC audience from something foreign.

In summary, I find these sites/magazines the only respectable reviews left on the internet/in print-

Most important part of the post

From most respecable to least:

CGM
Gamespot
Gamespy
Eurogamer
Yahoo!Games! (GamesDomain?)
OPM
OXM
1Up
CGW
VoodooExtreme
EvilAvatar
ConsoleGold (whatever it's called now)
PCGamer
IGN
GamePro
MaximOnline

[Ephraim]

And this is why I like the meta-ranking sites like Gamerankings.

Not only do you get a good "aggregate picture", it's also easy to see how much each site varies from the average score. These stats confirm that certain sites (IGN among them) are usually scoring games higher than the average of all reviews, and allows you to discount the offendingly high scoring reviews.

[Qenan]

I basically only use the meta sites now. Individual reviews are too variable, and I'm usually only interested in finding out whether something is a plausible purchase for the price. (To some extent, the cheaper it is, the lower an aggregate score I will tolerate.)

[Dave Long]

Originally Posted by flyinj

CGM's console reviews are great in the fact that they cater to a PC audience, and try to break down if a PC gamer would be interested in the game. Unfortunately, I sometimes find them a bit harsh in shutting out certain games to "protect" the PC audience from something foreign.

I don't think we're trying to protect the PC audience (except maybe financially), just looking at it from that perspective as a PC gamer. All too often there are console games that get very high review scores in console mags because those folks simply haven't even played a PC game in their lives. Either that or they do so infrequently they're unaware of a partcular console game being super-derivative of something on the PC.

I mean, for a PC gamer, something like Rainbow Six: Lockdown, that I reviewed in last month's mag, just isn't going to cut it at all. It's dull. That's true for me as well since I am a PC gamer as well as a console one. I guess that's probably the best way to view those reviews... the guys writing them are hardcore PC and console gamers so you know we're coming at it from the perspective of someone pretty knowledgeable of all forms of gaming.

I'm sure Tom or Steve can tell you more if they pop in here.

--Dave

[Jazar]

How about we, the people of this board who I consider very bright individuals for the most part, pay less attention at the numerical value of a review and more attention on the actual text of the review.

[Dave Long]

I already do that. :)

I've started buying just about every game magazine on the market. I like to know who's doing what and how well. I think it's also helping me be a better writer. With subscriptions so cheap for most print mags and even cheaper if you get them through Zinio (the Ziff ones), there's no reason someone like me can't stay informed.

--Dave

[Greg Kasavin]

Just a random comment to the original post: I have no idea how you could have come away from my Vietcong review having inferred I "made it out to be crap." I really liked that game. Maybe you're referring to the sequel or the expansion pack. Thanks for the kind words about GameSpot's reviews, though. With the so-called next generation of games upon us, I think most publications are taking the opportunity to reevaluate the way they review games, so hopefully readers will see a positive trend in the next year. Ron Gilbert's blog at grumpygamer.com recently made a very interesting point that the quality of game reviews may improve as the content of the games themselves does. A cerebral game like Killer7 forces a critic to write about it differently than some typical FPS, so as long as game makers continue to produce interesting stuff, I have faith that the quality of game writing will steadily increase as well.

[extarbags]

Hey Greg, whatever happened to that New Thing in Game Ratings you said you couldn't talk about yet a while ago?

[Tyrion Lannister]

Errm, HonestGamers and Digital Entertainmentand News and that's about it I'm afraid.

The rest are really terrible.

I give GameSpot a 2.2/10.0.

[Mark Asher]

The problem with the 10 point scale is that no one uses a 5 as the baseline for an average game. Five point scales tend to be a bit better about the mid-point being the score for an average game.

Just think how many two-star games and even 1-star games you see scored at places that use a 5 star game. On a 10-point scale those scores would be 20s and 40s -- something you almost never see.

[Troy S Goodfellow]

Originally Posted by Tyrion Lannister

EDigital Entertainmentand News

The site that has these gems from its Blitzkrieg II review?

As good as consoles are for other genres of games, the PC’s gaming acumen seems solidified by the fact that war simulations are irreversibly connected to the platform.

If you were asked to think of another war-based RTS game, chances are that you would choose the Command and Conquer series. Blitzkrieg 2 is the complete opposite to that series, which isn’t necessarily a bad thing.

If you want to be successful in this game, you need to be tactically astute. You are given a small core of units, which it is your mission to command them.

Troy

[Dr Fear]

Originally Posted by Greg Kasavin

Just a random comment to the original post: I have no idea how you could have come away from my Vietcong review having inferred I "made it out to be crap." I really liked that game.

Well, you gave it a 7.9, which is on the lower end of the 7-9 scale.

[Tyrion Lannister]

Well, yes and DEN does have awful layout. But they take their time to actually play the games, which is more than can be said for most sites, and the ratings seem far less arbitrary. They also appear to have the reviewers review games in genres they like.

If I read another review which starts with, "I hated the first one, and this one is even worse" I'm going to scream. That means you shouldn't be reviewing it. That's like sending a Fangoria reviewer to Pride and Prejudice: "No gore, lots of irritating chicks giggling and fruity accents. 2/10"

[Troy S Goodfellow]

Originally Posted by Tyrion Lannister

They also appear to have the reviewers review games in genres they like.

I'm not sure that this is unique. Every place that I have written for has asked what genres I felt most comfortable writing about and I've rarely written about any genre I clearly stink at (like FPS).

As I read the terrible opening paragraph of DEN's AoE 3 review, I suspected that maybe it was just this one writer.

Then I read another writer claim that Act of War has an unlimited resource model and cites the oil wells as an example. In fact, the wells are finite in capacity and can be exhausted. Only prisoners can bring unlimited funds, but slowly.

And another refers to Imperial Glory as yet another military RTS set in Napoleonic Europe. I had no idea we were drowning in them...

And someone gave Superpower 2 a 7.8.

I think I'm done with DEN.

Troy

[andrew_fm]

Originally Posted by Dr Fear

Well, you gave it a 7.9, which is on the lower end of the 7-9 scale.

7.9
good

[Dave Long]

Originally Posted by Tyrion Lannister

Well, yes and DEN does have awful layout. But they take their time to actually play the games, which is more than can be said for most sites, and the ratings seem far less arbitrary. They also appear to have the reviewers review games in genres they like.

If I read another review which starts with, "I hated the first one, and this one is even worse" I'm going to scream. That means you shouldn't be reviewing it. That's like sending a Fangoria reviewer to Pride and Prejudice: "No gore, lots of irritating chicks giggling and fruity accents. 2/10"

Eh? I think Gamespy's reviews are some of the best among the websites, but I think there's this thing that hardcore gamers have against them because Gamespy will forever be associated with the product that started the whole thing. People aren't willing to take their reviews seriously because of that.

I can tell you that I've been working with all of the writers at GamerDad (I'm the EIC over there) to try to improve all their writing and one guy in particular who clearly plays the games thoroughly, but had a lot of trouble getting it into text, has improved dramatically over this past year. For those that aren't aware, our reviews are no different than any other site when it comes to the main body, we just tack on the parental part at the end in the Kid Factor. So don't feel like you're not welcome if you're not a parent.

Anyway, everywhere I've ever worked with has put people on the reivews that like the genre or are even fans of the games. That probably leads to some score inflation sometimes, but it at least gets you well-reasoned text when it comes to the good and bad because the people want to play those games in the first place.

--Dave

[Sidd_Budd]

Originally Posted by Dr Fear

Well, you gave [Vietcong] a 7.9, which is on the lower end of the 7-9 scale.

7.9 is above average for Gamespot; only 30% of all their games get a score of 7.9 or higher. Not all sites are equal when it comes to the 7-9 stereotype.

EDIT: Removed inaccurate reference to PC Gamer's revision to their scoring system. I thought they had changed the descriptions given to the different percentages, to encourage more stringent scoring. Upon closer inspection, it appears they didn't change the old scoring system, but merely added a graphical bar (like a horizontal thermometer), to their reviews for easier visual discrimination

[MikeSofaer]

I was thinking about this last night, and I had an incredibly brilliant idea so hang on to your seats, ladies and gentlemen.

OK. Get ready.

A normalized rating scale. Not a shodily normalization like the aggregate sites do, but a within-site normalization that doesn't even give a raw score, merely how many standard deviations above or below that site's mean the game is. Reviewers could be given any scale they wanted to use, since all scores would be statiscally comparitive, score inflation of all games as a whole could not happen.

So? How do you like them tasty delicious Granny Smith apples?

[Ben Sones]

Originally Posted by Dr Fear

Well, you gave it a 7.9, which is on the lower end of the 7-9 scale.

I can't imagine how anyone could extract what he really thinks of the game from that review. Most of it is just carefully neutral description of game features--the sort of thing you'd expect to find in a preview, or bullet-pointed on the game box--without any commentary on whether he thinks those features hurt or help the gameplay, let alone how. The few statements of opinion that are there are often concealed in mind-achingly labyrinthine prose full of qualifiers (that mostly boil down to something along the lines of "it's the sort of thing you'll probably like, if you like that sort of thing"). I mean, what am I supposed to make of statements like this:

The events of the campaign are predictable enough in and of themselves, but they may take you by surprise as they come up, and they're the sorts of experiences you'd probably hope to get out of a first-person shooter set during the Vietnam War.

So the campaign is predictable yet surprising, and probably what you'd expect, depending on your expectations. That's illuminating.

I had to read this sentence three times just to figure out what he was trying to say:

it's possible to get Vietcong running smoothly if it doesn't run smoothly on its own, though at this point it's not unreasonable to think that PC games shouldn't require this sort of fine-tuning.

I'm surprised that he didn't add that it's also not unreasonable to not think that games shouldn't require this sort of fine tuning, depending on what you think. Or not.

There are a few genuine bits of critical commentary in there, but if you trimmed the review down to just those, it would only be about two paragraphs long. And it would be better for it.

[Dave Long]

Every time I see a "may" or "might" or "hope" or "probably" in a GamerDad submission lately, I cut it right the fuck out. I hate those words in reviews and I'm trying hard not to use them myself.

--Dave

[Sidd_Budd]

Originally Posted by MikeSofaer

I was thinking about this last night, and I had an incredibly brilliant idea ...
A normalized rating scale. Not a shodily normalization like the aggregate sites do, but a within-site normalization that doesn't even give a raw score, merely how many standard deviations above or below that site's mean the game is.

I love this idea, and it's how I keep my own database of game rankings. There's plenty of people who know how to convert scores to standard normal distributions; it would be pretty simple to just convert the raw scores even for an aggregation site like Game Rankings.

Hardly anyone knows how to interpret standard deviations, so I think the final score should give an *actual* percentage as well. For example, a score of 80% would mean that this game achieves an aggregate ranking higher than 80% of all the games reviewed. The only problem would be scores wouldn't be static; they could change as the universe of games reviewed gets bigger. I don't know how difficult that would be to maintain at a commercial site, but it's not a problem for my Excel spreadsheet.

[Tyrion Lannister]

Oh shit, yeah, sorry forgot about GamerDad. My first stop for game reviews, and so I naturally forgot to mention it...

And I'm sorry to say that GameSpy and GameSpot (and the utterly terrible EGM) suffer from the MacReview syndrome that IGN is so well known for. But with added Edge-like arsiness and snideness in their reviews and rating systems. V. Poor.

If I believed what they had written I wouldn't have bought Gun or Shadow the Hedgehog, and I'd have missed out on a lot of entertainment (OK, so shadow is a little flawed, but only a little...)

[RickH]

Originally Posted by MikeSofaer

A normalized rating scale. Not a shodily normalization like the aggregate sites do, but a within-site normalization that doesn't even give a raw score, merely how many standard deviations above or below that site's mean the game is. Reviewers could be given any scale they wanted to use, since all scores would be statiscally comparitive, score inflation of all games as a whole could not happen.

PSM used to do something like that, they'd place the score of a game in the context of 3 or 4 other games in that genre so you could see where it fell (in their opinion of course) in comparison with its competition. Then they stopped doing that. Wasn't "edgy" enough, I guess.

[MikeSofaer]

Originally Posted by Sidd_Budd

I love this idea, and it's how I keep my own database of game rankings. There's plenty of people who know how to convert scores to standard normal distributions; it would be pretty simple to just convert the raw scores even for an aggregation site like Game Rankings.

Hardly anyone knows how to interpret standard deviations, so I think the final score should give an *actual* percentage as well. For example, a score of 80% would mean that this game achieves an aggregate ranking higher than 80% of all the games reviewed. The only problem would be scores wouldn't be static; they could change as the universe of games reviewed gets bigger. I don't know how difficult that would be to maintain at a commercial site, but it's not a problem for my Excel spreadsheet.

Cool, nice to know it actually works.
I can see a percentage ranking of that sort being useful, as well as graphs of score distribution curves and where the game falls on them, you could restrict the set to games of that genre, or by that reviewer, etc.

Making this tool available to the reviewer when he's submitting the score would be a big step up, too, since it would force him to really evaluate in context.

[noun]

We struggled with the rating system for our site. Everyone had a different opinion: some wanted the 1-100 rating to match industry standards, but few folks wanted to deal that level of granularity (which ends up being arbitrary, anyway). Some (like myself) didn't want any numbers whatsoever, but to get listed on sites like GameRankings, it was argued that we had to have numbers of SOME sort.

In the end we went with a 5 point system. 5 games we love, 1 games are crap, and everything else usually falls in between.

Intresting idea about the aggregate number. I'll have to look over what we have to date to see if that makes any difference.

[Ben Sones]

Originally Posted by RickH

PSM used to do something like that, they'd place the score of a game in the context of 3 or 4 other games in that genre so you could see where it fell (in their opinion of course) in comparison with its competition. Then they stopped doing that. Wasn't "edgy" enough, I guess.

Not edgy, maybe, but also not useful. No offense to the people who brought it up, but trying to come up with ways to make ratings more mathematically accurate is missing the point. A rating is not a precise, objective measurement. It's just one person's opinion, and any given pub has a bunch of different people generating those opinions. I mean, sure, you can come up with some model that says "game X scored higher than 80% of the other games in our database," but what does that actually mean? Pretty much nothing. It's just an arbitrary number. You'd probably have a different arbitrary number if they had given the review to a different person. I don't understand what meaningful information I'm supposed to be able to derive by comparing how one person rated one game to how a different person rated a different game.

Ratings are already way more complicated than they need to be; they don't need to get more complicated. Percentages, ten point scales with decimals, multiple rating categories cooked down into mean averages... all that stuff is utterly pointless. I'd be happy with a rating system that had only two ratings ("thumbs up" and "thumbs down"). I'd even be happy with no ratings at all.

[Dave Long]

I've always been a big proponent of the Thumbs Up/Thumbs Down method but you simply do not get listed on Gamerankings if you do that, and apparently that's a big means of getting people to visit your website.

It sucks.

--Dave

[Tyrion Lannister]

GameRankings is a curse.

[Dave Long]

I agree. I think they also won't put you in there unless your reviews are some arbitrary (rather long) length. I wanted to shorten up the reviews at GamerDad and it just can't be done and still be listed at Gamerankings.

--Dave

[RickH]

Originally Posted by Ben Sones

I mean, sure, you can come up with some model that says "game X scored higher than 80% of the other games in our database," but what does that actually mean? Pretty much nothing. It's just an arbitrary number. You'd probably have a different arbitrary number if they had given the review to a different person. I don't understand what meaningful information I'm supposed to be able to derive by comparing how one person rated one game to how a different person rated a different game.

Unless you are someone wanting a new fighting game and wanted to know which game PSM thinks is better, Tekken, SC, VF, MK, etc. Are you really saying comparisons are irrelevant to a review or body of reviews?