Ratings in reviews vs. Text only

[Brad Wardell]

Let's imagine hypothetically that a major game magazine or major gaming website dumped numerical ratings entirely.

Wouldn't that be cool? It would force people to read the text rather than look at numerical score / star score / whatever.

I think for hard-core gamers, it would probably be well received. I even think many casual gamers would prefer it too. A poll here on Qt3 would probably have 90% supporting this.

But personally, I'd be bummed out. First off, sites like GameRankings matter. They matter a LOT. As a game developer, it pains me how much GameRankings matters. Over the past year, I've learned just how much these ratings matter and it's depressing.

So let me share with you some experiences we've had.

How many copies will major retailer X purchase? Depends heavily on how many previews a game gets along with MDF.

How many copies will the retailer re-order? It's based on a formula (which differs from retailer to retailer of course) that roughly translates to 1st week sales = subsequent month's sales = subsequent 3 months sales = subsequent 6 months sales = subsequent year sales. Typically, when those sales drop below N turns per store per month, the game is no longer carried.

I'm not saying every retailer does this nor that those that do something like this do it just like this. But most do something like the above.

That's why it's frustrating for smaller publishers/developers to make big numbers because it's hard to get that initial high sell-in. We've learned this (the hard way) in the past and really went out to get a lot of previews and such. Master of Orion 3 sold something around 300,000 copies to GalCiv I's 75,000 units in NA. Stocking obviously matters.

But publishers get a second chance to up their store presence -- the reviews, particularly from the paper magazines.

The bloody gamerankings score comes up again and again.
The reviews from 3 of the major PC publications and a handful of the major gaming sites (depends on the buyer, some just use game rankings, some go by 3 of the gaming sites, others use all and still others don't care except for existing sales and MDF) are heavily factored in. MDF (market development funds) and that combined rating matter a lot.

That second chance can help a title that got sporadic distribution at the start to get restocked in higher quantity 60 days or so into distribution. That's when you start to see the boxes with the often misleading "Game of the year" (which drives everyone nuts I suspect) mentions on them -- see Jeff Green's comments in the latest CGW.

So if a major publication were to remove its numerical ratings or whatever, the effect is that it's essentially bowing out from being counted in those ratings.

And if the publication or website happens to be one that is really well respected, then it essentially just adds influence to the sites and publications that don't have as high a quality in their reviews.

As a gamer, I read the text of the reviews. I recognize that different reviewers have different scoring cards. One reviewer's 3.5 stars is the same as another's 80%. I understand that. Everyone here probably undertands that. But the wider world does not.

To use a World of Warcraft analogy, no matter how long you've been playing the game, if you post on the WoW forums with a character that's less than level 60, you're going to get people who will yell "n00b!" or treat you as if you don't know what you're talking about purely because of your level. It's not about fairness. It's just how things work.

So that's why I support ratings. Because the casual onlooker will browse those review scores or someone who goes back a year later to look at a game will go by those scores (I recall a heated debate here where I was accused of making "mediocre games" based purely on the game rankings average).

Whether we like it or not, people won't read the text and not eveyrone will have ready access to that text.

In case someone is wondering what made be bring this up, I had heard some things from some friends who had heard things from friends. I.e. nobody "leaked" anything to me who was told anything directly.

Just my 2 cents.

[RedTide]

For some reason I see a lot of HTML tags

[Brad Wardell]

Fixed. Not sure why that happened.

[Wobbo]

The best compromise would be a "bottom line" or even one paragraph summary.

The reasons you give for still including the pointless numbers are the excact reason they need to be phased out. They're unfair and hurt the customer and retailer (and eventually the whole artform) in the long rund

[stusser]

This comes up every 3 months or so. Usual points brought up are the 7-9 scale (several times), objectivity vs bias, ebert's thumbs up/down, daily radar's miss/hit/direct hit system, gameranking's lack of accuracy, and CGW's lack of ratings back in the dark ages.

My thoughts:

7-9 scale: Makes no sense. Bunch of jackasses. We know better, as we are better men than they could hope to be.

Objectivity vs. Bias: All criticism is subjective to some extent. Read the reviews and get to know the reviewers. Yes, this does means that numerical ratings are deceptive. No, they still won't go away.

Thumbs Up/Down and daily radar's miss/hit/direct hit system (value of less granular ratings): I'm pro.

Gameranking's inaccuracy: Agree that they should learn that a 3/5 from Selvakumar's Gaming Garage isn't the same as a 60% from IGN. Will they? No, they will not.

CGW's lack of ratings: CGW was great back in johhny wilson's times. Where the hell is scorpia, anyway? No, this won't come back either.

Did I miss anything?

[Damien Neil]

I don't look at reviews. I can't think of any sites where I find it worth it to slog through the mass of dancing ads and endless load times to pore through the half dozen paragraphs of text split up across three pages that constitutes a "review". I used to read IGN when their Dreamcast coverage would tell me about the latest fruity and quirky games out Japan, but that was long ago.

I get my game recommendations from Usenet (or used to; the last year seems to have completed comp.sys.ibm.pc.games.*'s slide into senescence) and forums like this place.

[stusser]

Oh yeah, and there's always some guy who claims he doesn't read professional reviews at all.

[Jeff Green]

Check out our April reviews and let us know what you think.

No ratings.

[Marcin]

What a tease.

I miss Daily Radar's ratings. Hit, Miss, Dud. It really doesn't get any better than that, or the alternative Buy / Rent / Skip. That's what I tend to use.

[Aszurom]

maybe gamerankings will say CGW gave every game 0%

[Jeff Green]

Hey, I'm not tryin' to tease ya, Marcin. But the only alternative to me saying "check it out" is to, I dunno, paste all the scoreless reviews in a message here, and I'm a bit too lazy for that. :)

So the reviews still have a "Verdict", but no numbers attached. And we still give an "Editors' Choice" award to the games most deserving. But other than that, no number or grade, and, yeah, it'll be interesting to see what Gamerankings does. Or not. I don't really give a darn.

[stusser]

Great to hear!

I'm surprised ZD agreed. It was pretty clear that the move to explicit ratings was forced upon CGW by corporate. Of course that was many, many years ago.

[JMR]

Originally Posted by stusser

CGW's lack of ratings: CGW was great back in johhny wilson's times.

Word.

[Jeff Green]

Yeah, a rating system was essentially forced upon CGW in around 1994, I think? (Denny?) The only person who never had to give a numerical rating to a review was Scorpia, who hated the notion so much, and had so much clout at the time, that she was given a pass.

The problem, near the end of her time at CGW, was that certain publishers starting begging us NOT to have her review their new games because they wanted that numerical rating for the inevitable whorish box blurb. (Which of course is why ZD wanted CGW to do it in the first place.) Scorpia refused (understandably, admirably) to ever agree, and so in the end the editors were forced to tack on numerical ratings to her reviews anyway.

Whatever. I give this post of mine here a 3.5. Pros: good historical info, straightforwardly presented. Cons: Not enough cleavage.

[Troy S Goodfellow]

Interesting idea. Good luck with it.

I suspect that your writers were pretty much on board. I wonder what the reader reaction will be.

Troy

[Dave Long]

Originally Posted by Jeff Green

it'll be interesting to see what Gamerankings does.

They'll drop you from their listing. Whether that matters to you or not is totally up to you, but from what I understand of how they do things there, no rating = no listing. That's probably especially true in this case since print reviews usually have no linkable source on the Internet.

[Cyrano]

That's great news about CGW! I've always wished a magazine would have the integrity to do that.

[Troy S Goodfellow]

Originally Posted by Cyrano

I've always wished a magazine would have the integrity to do that.

Why is this a question of "integrity"?

Troy

[jfletch]

I like the concept of no ratings but honestly Im so used to them by now I think reviews would seem a little naked without them. Itd take a bit to adjust, for sure.

[Dave Long]

Originally Posted by Troy S Goodfellow

Why is this a question of "integrity"?

Troy

Man, what a dumb question. Everyone knows that putting a score on a review automatically means you're getting paid (or not paid) by the publisher of the game. Jeez, Troy. After all your work in the industry, you haven't figure this out by now?

[Jeff Green]

We're expecting both positive and negative feedback, of course.
Just like we did with the no "game of the year" awards thing, which was done for a similar reason--to function less as an arm of the publishers' marketing depts, and more as a resource for the gaming community.

But, ya know, many many people like things spelled out for them. And I received numerous articulate letters from people unhappy with the lack of awards. I'm sure it'll be the same thing with the ratings.

But I wanted to try it anyway. At times it feels like 90 percent of my phone calls are with PR folks wondering what score they're gonna get, why they only got a 4 when Game X got a 4.5, etc....Now that I'm in my 600th year here, I felt like trying something "new". But it remains to be seen whether readers will really like it or not.

[Brad Wardell]

Well I've already put my 2 cents on it.

But let me also say that I don't think it will help CGW in the long run.

CGW is a great game magazine. We fight over who gets to read it first each month which, at our company, technically counts as our fitness program.

But I say it won't help because I don't think gamers care much either way whether there's reviewers. BUT the big publishers DO care. It takes away a lot of CGW's leverage.

As I said at the start, I don't personally like the GameRankings system as someone who's in the industry because it's so powerful.

And I believe, completely, that a lot of drek is kept from being put out there because of the fear of a very low score from CGW (and other sites).

That 0 star review for Postal 2 I think had a lot of impact on the development community and publishers alike. I don't think it would have had nearly the impact without that 0-star rating. It would have just been another negative review.

[Shadari]

Originally Posted by Dave Long

Man, what a dumb question. Everyone knows that putting a score on a review automatically means you're getting paid (or not paid) by the publisher of the game. Jeez, Troy. After all your work in the industry, you haven't figure this out by now?

I can't tell if you're being sarcastic or not.

[Jeff Green]

Originally Posted by Brad Wardell

That 0 star review for Postal 2 I think had a lot of impact on the development community and publishers alike. I don't think it would have had nearly the impact without that 0-star rating. It would have just been another negative review.

Actually, most of us regret that 0-star rating, if only because Running With Scissors loved it so much. :) It garnered them MUCH more attention than if we had just slapped it with 1 star. But because we took that extra step, it became a subject of "controversy" and attention, which is of course all those guys are after. We couldn't have really given them a better rating for their own needs. "SEE? The mainstream rags can't TAKE a game as edgy as ours!"

Whee!

[Dave Long]

Originally Posted by Jeff Green

We're expecting both positive and negative feedback, of course.
Just like we did with the no "game of the year" awards thing, which was done for a similar reason--to function less as an arm of the publishers' marketing depts, and more as a resource for the gaming community.

But, ya know, many many people like things spelled out for them. And I received numerous articulate letters from people unhappy with the lack of awards. I'm sure it'll be the same thing with the ratings.

But I wanted to try it anyway. At times it feels like 90 percent of my phone calls are with PR folks wondering what score they're gonna get, why they only got a 4 when Game X got a 4.5, etc....Now that I'm in my 600th year here, I felt like trying something "new". But it remains to be seen whether readers will really like it or not.

Personally, I'd rather see you go to a thumbs up or down. Binary yea or nay. I think doing it with no score will make more readers unhappy than happy, but I could be wrong. I'd like to see how publishers would respond to the thumbs.

Here's a wacky one for you, the readers of Play bitched when they went to a star system recently. The ended up with a ten point system 0-10 with .5 halves possible because that was what the readers wanted. Ugh.

[Brad Wardell]

Originally Posted by Jeff Green

The problem, near the end of her time at CGW, was that certain publishers starting begging us NOT to have her review their new games because they wanted that numerical rating for the inevitable whorish box blurb. (Which of course is why ZD wanted CGW to do it in the first place.) Scorpia refused (understandably, admirably) to ever agree, and so in the end the editors were forced to tack on numerical ratings to her reviews anyway.

What do you think the long term effect is going to be? What do you think the big publishers are going to do?

The big publishers wanted ZD to have the ratings so they can "whore" their boxes. Hence, they obviously believe that the ratings have value.

The converse is therefore true -- not having ratings is likely to result in big publishers not feeling that CGW's reviews are as valuable.

I'm all for "art" but CGW is a business too. I love what I do for a living. But I never kid myself what I'm doing is also to make something that generates more income than it creates in expense.

As CGW's Megatron, is this change something you believe will increase readership? Increase advertising? I'm just trying to understand the rationale.

[Brad Wardell]

Originally Posted by Jeff Green

Actually, most of us regret that 0-star rating, if only because Running With Scissors loved it so much. :) It garnered them MUCH more attention than if we had just slapped it with 1 star. But because we took that extra step, it became a subject of "controversy" and attention, which is of course all those guys are after. We couldn't have really given them a better rating for their own needs. "SEE? The mainstream rags can't TAKE a game as edgy as ours!"

Whee!

They can claim they loved it all day. What were their sales? If I recall, it tanked.

I suspect that the kind of gamers who find 0 star reviews "l33t" don't have enough allowance money saved up to come up from their basements to get the game.

[Jeff Green]

Originally Posted by Brad Wardell

What do you think the long term effect is going to be? What do you think the big publishers are going to do?

The big publishers wanted ZD to have the ratings so they can "whore" their boxes. Hence, they obviously believe that the ratings have value.

The converse is therefore true -- not having ratings is likely to result in big publishers not feeling that CGW's reviews are as valuable.

I'm all for "art" but CGW is a business too. I love what I do for a living. But I never kid myself what I'm doing is also to make something that generates more income than it creates in expense.

As CGW's Megatron, is this change something you believe will increase readership? Increase advertising? I'm just trying to understand the rationale.

All good points and all stuff that we are thinking about, of course. Consider it an experiment for now. CGW is just being given the luxury of trying a few new things right now, as most of the big push and attention right now is on 1Up.com and the new consoles. It's part of just a larger shift for us that I would spell out in more detail were it not for the fact that Morris and Vede are lurking somewhere around here. :) But it's not rocket science either. It's a matter of looking at the landscape, vis-a-vis the IntarWeb, and figuring out a place to be, a way to go. Does every game magazine need to be part of the machine? Maybe they do. I'm just taking the latitude I have right now to test the waters...

[Brad Wardell]

Originally Posted by Jeff Green

We're expecting both positive and negative feedback, of course.
Just like we did with the no "game of the year" awards thing, which was done for a similar reason--to function less as an arm of the publishers' marketing depts, and more as a resource for the gaming community.

But, ya know, many many people like things spelled out for them. And I received numerous articulate letters from people unhappy with the lack of awards. I'm sure it'll be the same thing with the ratings.

But I wanted to try it anyway. At times it feels like 90 percent of my phone calls are with PR folks wondering what score they're gonna get, why they only got a 4 when Game X got a 4.5, etc....Now that I'm in my 600th year here, I felt like trying something "new". But it remains to be seen whether readers will really like it or not.

Aren't game developers and publishers part of the game community?

I can honestly say, since the release of our game, I've sweated quite a bit (more than usual even which is saying something) about the reviews. More specifically, I sweat about the scores. And I know I'm not the only game developer that loses sleep waiting to find out that score.

If all magazines had no scores, I'd certainly sweat less. But that's not the case. What it means now is that "SuperGamingWebSite" has more power because CGW isn't participating.

To you guys, what do you care? Who the heck am I to you? Just some weird guy who named his lizard "Frodo". But to many of us, CGW is one of the few publications that actually plays a game thoroughly before reviewing it. Tom Chick or Bruce Geryk or whoever might rip us a new one, but we'd know they'd played the game.

The industry is still going to focus on the numerical scores, except now there's no CGW to counter balance the less than...thorough reviews.

What I feel most awkward about is feeling like developers and publishers are the bad guys. We make games because we love games. We make less than we could in other inudstries. But we do it because we love games.

We bug you for our review scores because they matter to us a lot. A LOT.

On our non-games, I don't sweat too much what the reviews are and they're what pay our bills. But on games, we care. We care a lot. Particularly about that score. Maybe we shouldn't. But we do. And so do publishers. And so do retailers. But we don't do it because we're evil (we're evil for a lot of different reasons). And we don't do it because we're just trying to make a fast buck. We do it because it really matters to us.

[Dhruin]

Every time this topic comes up on most boards, the general concensus seems to be "scores suck, read the text" but if the topic is "where do you go for reviews?", the general wisdom is "I go straight to Gamerankings".

The message I take out of that is people say they don't like scores but on the whole, it just ain't true.