Ratings in reviews vs. Text only [Archive]

Brad Wardell

02-26-2006, 04:15 PM

Let's imagine hypothetically that a major game magazine or major gaming website dumped numerical ratings entirely.

Wouldn't that be cool? It would force people to read the text rather than look at numerical score / star score / whatever.

I think for hard-core gamers, it would probably be well received. I even think many casual gamers would prefer it too. A poll here on Qt3 would probably have 90% supporting this.

But personally, I'd be bummed out. First off, sites like GameRankings matter. They matter a LOT. As a game developer, it pains me how much GameRankings matters. Over the past year, I've learned just how much these ratings matter and it's depressing.

So let me share with you some experiences we've had.

How many copies will major retailer X purchase? Depends heavily on how many previews a game gets along with MDF.

How many copies will the retailer re-order? It's based on a formula (which differs from retailer to retailer of course) that roughly translates to 1st week sales = subsequent month's sales = subsequent 3 months sales = subsequent 6 months sales = subsequent year sales. Typically, when those sales drop below N turns per store per month, the game is no longer carried.

I'm not saying every retailer does this nor that those that do something like this do it just like this. But most do something like the above.

That's why it's frustrating for smaller publishers/developers to make big numbers because it's hard to get that initial high sell-in. We've learned this (the hard way) in the past and really went out to get a lot of previews and such. Master of Orion 3 sold something around 300,000 copies to GalCiv I's 75,000 units in NA. Stocking obviously matters.

But publishers get a second chance to up their store presence -- the reviews, particularly from the paper magazines.

The bloody gamerankings score comes up again and again.
The reviews from 3 of the major PC publications and a handful of the major gaming sites (depends on the buyer, some just use game rankings, some go by 3 of the gaming sites, others use all and still others don't care except for existing sales and MDF) are heavily factored in. MDF (market development funds) and that combined rating matter a lot.

That second chance can help a title that got sporadic distribution at the start to get restocked in higher quantity 60 days or so into distribution. That's when you start to see the boxes with the often misleading "Game of the year" (which drives everyone nuts I suspect) mentions on them -- see Jeff Green's comments in the latest CGW.

So if a major publication were to remove its numerical ratings or whatever, the effect is that it's essentially bowing out from being counted in those ratings.

And if the publication or website happens to be one that is really well respected, then it essentially just adds influence to the sites and publications that don't have as high a quality in their reviews.

As a gamer, I read the text of the reviews. I recognize that different reviewers have different scoring cards. One reviewer's 3.5 stars is the same as another's 80%. I understand that. Everyone here probably undertands that. But the wider world does not.

To use a World of Warcraft analogy, no matter how long you've been playing the game, if you post on the WoW forums with a character that's less than level 60, you're going to get people who will yell "n00b!" or treat you as if you don't know what you're talking about purely because of your level. It's not about fairness. It's just how things work.

So that's why I support ratings. Because the casual onlooker will browse those review scores or someone who goes back a year later to look at a game will go by those scores (I recall a heated debate here where I was accused of making "mediocre games" based purely on the game rankings average).

Whether we like it or not, people won't read the text and not eveyrone will have ready access to that text.

In case someone is wondering what made be bring this up, I had heard some things from some friends who had heard things from friends. I.e. nobody "leaked" anything to me who was told anything directly.

Just my 2 cents.

RedTide

02-26-2006, 04:15 PM

For some reason I see a lot of HTML tags

Brad Wardell

02-26-2006, 04:17 PM

Fixed. Not sure why that happened.

Wobbo

02-26-2006, 04:28 PM

The best compromise would be a "bottom line" or even one paragraph summary.

The reasons you give for still including the pointless numbers are the excact reason they need to be phased out. They're unfair and hurt the customer and retailer (and eventually the whole artform) in the long rund

stusser

02-26-2006, 05:06 PM

This comes up every 3 months or so. Usual points brought up are the 7-9 scale (several times), objectivity vs bias, ebert's thumbs up/down, daily radar's miss/hit/direct hit system, gameranking's lack of accuracy, and CGW's lack of ratings back in the dark ages.

My thoughts:

7-9 scale: Makes no sense. Bunch of jackasses. We know better, as we are better men than they could hope to be.

Objectivity vs. Bias: All criticism is subjective to some extent. Read the reviews and get to know the reviewers. Yes, this does means that numerical ratings are deceptive. No, they still won't go away.

Thumbs Up/Down and daily radar's miss/hit/direct hit system (value of less granular ratings): I'm pro.

Gameranking's inaccuracy: Agree that they should learn that a 3/5 from Selvakumar's Gaming Garage isn't the same as a 60% from IGN. Will they? No, they will not.

CGW's lack of ratings: CGW was great back in johhny wilson's times. Where the hell is scorpia, anyway? No, this won't come back either.

Did I miss anything?

Damien Neil

02-26-2006, 05:08 PM

I don't look at reviews. I can't think of any sites where I find it worth it to slog through the mass of dancing ads and endless load times to pore through the half dozen paragraphs of text split up across three pages that constitutes a "review". I used to read IGN when their Dreamcast coverage would tell me about the latest fruity and quirky games out Japan, but that was long ago.

I get my game recommendations from Usenet (or used to; the last year seems to have completed comp.sys.ibm.pc.games.*'s slide into senescence) and forums like this place.

stusser

02-26-2006, 05:12 PM

Oh yeah, and there's always some guy who claims he doesn't read professional reviews at all.

Jeff Green

02-26-2006, 05:40 PM

Check out our April reviews and let us know what you think.

No ratings.

Marcin

02-26-2006, 05:43 PM

What a tease.

I miss Daily Radar's ratings. Hit, Miss, Dud. It really doesn't get any better than that, or the alternative Buy / Rent / Skip. That's what I tend to use.

Aszurom

02-26-2006, 05:46 PM

maybe gamerankings will say CGW gave every game 0%

Jeff Green

02-26-2006, 05:51 PM

Hey, I'm not tryin' to tease ya, Marcin. But the only alternative to me saying "check it out" is to, I dunno, paste all the scoreless reviews in a message here, and I'm a bit too lazy for that. :)

So the reviews still have a "Verdict", but no numbers attached. And we still give an "Editors' Choice" award to the games most deserving. But other than that, no number or grade, and, yeah, it'll be interesting to see what Gamerankings does. Or not. I don't really give a darn.

stusser

02-26-2006, 05:58 PM

Great to hear!

I'm surprised ZD agreed. It was pretty clear that the move to explicit ratings was forced upon CGW by corporate. Of course that was many, many years ago.

JMR

02-26-2006, 06:02 PM

CGW's lack of ratings: CGW was great back in johhny wilson's times.

Word.

Jeff Green

02-26-2006, 06:15 PM

Yeah, a rating system was essentially forced upon CGW in around 1994, I think? (Denny?) The only person who never had to give a numerical rating to a review was Scorpia, who hated the notion so much, and had so much clout at the time, that she was given a pass.

The problem, near the end of her time at CGW, was that certain publishers starting begging us NOT to have her review their new games because they wanted that numerical rating for the inevitable whorish box blurb. (Which of course is why ZD wanted CGW to do it in the first place.) Scorpia refused (understandably, admirably) to ever agree, and so in the end the editors were forced to tack on numerical ratings to her reviews anyway.

Whatever. I give this post of mine here a 3.5. Pros: good historical info, straightforwardly presented. Cons: Not enough cleavage.

Troy S Goodfellow

02-26-2006, 06:29 PM

Interesting idea. Good luck with it.

I suspect that your writers were pretty much on board. I wonder what the reader reaction will be.

Troy

Dave Long

02-26-2006, 06:46 PM

it'll be interesting to see what Gamerankings does.

They'll drop you from their listing. Whether that matters to you or not is totally up to you, but from what I understand of how they do things there, no rating = no listing. That's probably especially true in this case since print reviews usually have no linkable source on the Internet.

Cyrano

02-26-2006, 07:22 PM

That's great news about CGW! I've always wished a magazine would have the integrity to do that.

Troy S Goodfellow

02-26-2006, 07:24 PM

I've always wished a magazine would have the integrity to do that.

Why is this a question of "integrity"?

Troy

jfletch

02-26-2006, 07:31 PM

I like the concept of no ratings but honestly Im so used to them by now I think reviews would seem a little naked without them. Itd take a bit to adjust, for sure.

Dave Long

02-26-2006, 07:41 PM

Why is this a question of "integrity"?

Troy

Man, what a dumb question. Everyone knows that putting a score on a review automatically means you're getting paid (or not paid) by the publisher of the game. Jeez, Troy. After all your work in the industry, you haven't figure this out by now?

Jeff Green

02-26-2006, 07:43 PM

Brad Wardell

02-26-2006, 07:44 PM

Well I've already put my 2 cents on it.

But let me also say that I don't think it will help CGW in the long run.

CGW is a great game magazine. We fight over who gets to read it first each month which, at our company, technically counts as our fitness program.

But I say it won't help because I don't think gamers care much either way whether there's reviewers. BUT the big publishers DO care. It takes away a lot of CGW's leverage.

As I said at the start, I don't personally like the GameRankings system as someone who's in the industry because it's so powerful.

And I believe, completely, that a lot of drek is kept from being put out there because of the fear of a very low score from CGW (and other sites).

That 0 star review for Postal 2 I think had a lot of impact on the development community and publishers alike. I don't think it would have had nearly the impact without that 0-star rating. It would have just been another negative review.

Shadari

02-26-2006, 07:44 PM

Man, what a dumb question. Everyone knows that putting a score on a review automatically means you're getting paid (or not paid) by the publisher of the game. Jeez, Troy. After all your work in the industry, you haven't figure this out by now?
I can't tell if you're being sarcastic or not.

Jeff Green

02-26-2006, 07:48 PM

That 0 star review for Postal 2 I think had a lot of impact on the development community and publishers alike. I don't think it would have had nearly the impact without that 0-star rating. It would have just been another negative review.

Actually, most of us regret that 0-star rating, if only because Running With Scissors loved it so much. :) It garnered them MUCH more attention than if we had just slapped it with 1 star. But because we took that extra step, it became a subject of "controversy" and attention, which is of course all those guys are after. We couldn't have really given them a better rating for their own needs. "SEE? The mainstream rags can't TAKE a game as edgy as ours!"

Whee!

Dave Long

02-26-2006, 07:49 PM

Brad Wardell

02-26-2006, 07:51 PM

The problem, near the end of her time at CGW, was that certain publishers starting begging us NOT to have her review their new games because they wanted that numerical rating for the inevitable whorish box blurb. (Which of course is why ZD wanted CGW to do it in the first place.) Scorpia refused (understandably, admirably) to ever agree, and so in the end the editors were forced to tack on numerical ratings to her reviews anyway.

What do you think the long term effect is going to be? What do you think the big publishers are going to do?

The big publishers wanted ZD to have the ratings so they can "whore" their boxes. Hence, they obviously believe that the ratings have value.

The converse is therefore true -- not having ratings is likely to result in big publishers not feeling that CGW's reviews are as valuable.

I'm all for "art" but CGW is a business too. I love what I do for a living. But I never kid myself what I'm doing is also to make something that generates more income than it creates in expense.

As CGW's Megatron, is this change something you believe will increase readership? Increase advertising? I'm just trying to understand the rationale.

Brad Wardell

02-26-2006, 07:54 PM

Actually, most of us regret that 0-star rating, if only because Running With Scissors loved it so much. :) It garnered them MUCH more attention than if we had just slapped it with 1 star. But because we took that extra step, it became a subject of "controversy" and attention, which is of course all those guys are after. We couldn't have really given them a better rating for their own needs. "SEE? The mainstream rags can't TAKE a game as edgy as ours!"

Whee!

They can claim they loved it all day. What were their sales? If I recall, it tanked.

I suspect that the kind of gamers who find 0 star reviews "l33t" don't have enough allowance money saved up to come up from their basements to get the game.

Jeff Green

02-26-2006, 08:02 PM

What do you think the long term effect is going to be? What do you think the big publishers are going to do?

The big publishers wanted ZD to have the ratings so they can "whore" their boxes. Hence, they obviously believe that the ratings have value.

The converse is therefore true -- not having ratings is likely to result in big publishers not feeling that CGW's reviews are as valuable.

I'm all for "art" but CGW is a business too. I love what I do for a living. But I never kid myself what I'm doing is also to make something that generates more income than it creates in expense.

As CGW's Megatron, is this change something you believe will increase readership? Increase advertising? I'm just trying to understand the rationale.

All good points and all stuff that we are thinking about, of course. Consider it an experiment for now. CGW is just being given the luxury of trying a few new things right now, as most of the big push and attention right now is on 1Up.com and the new consoles. It's part of just a larger shift for us that I would spell out in more detail were it not for the fact that Morris and Vede are lurking somewhere around here. :) But it's not rocket science either. It's a matter of looking at the landscape, vis-a-vis the IntarWeb, and figuring out a place to be, a way to go. Does every game magazine need to be part of the machine? Maybe they do. I'm just taking the latitude I have right now to test the waters...

Brad Wardell

02-26-2006, 08:05 PM

We're expecting both positive and negative feedback, of course.
Just like we did with the no "game of the year" awards thing, which was done for a similar reason--to function less as an arm of the publishers' marketing depts, and more as a resource for the gaming community.

But, ya know, many many people like things spelled out for them. And I received numerous articulate letters from people unhappy with the lack of awards. I'm sure it'll be the same thing with the ratings.

But I wanted to try it anyway. At times it feels like 90 percent of my phone calls are with PR folks wondering what score they're gonna get, why they only got a 4 when Game X got a 4.5, etc....Now that I'm in my 600th year here, I felt like trying something "new". But it remains to be seen whether readers will really like it or not.

Aren't game developers and publishers part of the game community?

I can honestly say, since the release of our game, I've sweated quite a bit (more than usual even which is saying something) about the reviews. More specifically, I sweat about the scores. And I know I'm not the only game developer that loses sleep waiting to find out that score.

If all magazines had no scores, I'd certainly sweat less. But that's not the case. What it means now is that "SuperGamingWebSite" has more power because CGW isn't participating.

To you guys, what do you care? Who the heck am I to you? Just some weird guy who named his lizard "Frodo". But to many of us, CGW is one of the few publications that actually plays a game thoroughly before reviewing it. Tom Chick or Bruce Geryk or whoever might rip us a new one, but we'd know they'd played the game.

The industry is still going to focus on the numerical scores, except now there's no CGW to counter balance the less than...thorough reviews.

What I feel most awkward about is feeling like developers and publishers are the bad guys. We make games because we love games. We make less than we could in other inudstries. But we do it because we love games.

We bug you for our review scores because they matter to us a lot. A LOT.

On our non-games, I don't sweat too much what the reviews are and they're what pay our bills. But on games, we care. We care a lot. Particularly about that score. Maybe we shouldn't. But we do. And so do publishers. And so do retailers. But we don't do it because we're evil (we're evil for a lot of different reasons). And we don't do it because we're just trying to make a fast buck. We do it because it really matters to us.

Dhruin

02-26-2006, 08:06 PM

Every time this topic comes up on most boards, the general concensus seems to be "scores suck, read the text" but if the topic is "where do you go for reviews?", the general wisdom is "I go straight to Gamerankings".

The message I take out of that is people say they don't like scores but on the whole, it just ain't true.

Jeff Green

02-26-2006, 08:10 PM

Jab2565

02-26-2006, 08:15 PM

My general experience with gaming magazines is when I first get them to just skim thru them looking at the score, glancing over any major previews, later in the day do I then sit down and read the magazine.

I think entire text/ no number reviews are good if the review is good. I remember awhile ago reading nintendo power before they changed it and was annoyed that the text blurb next into each review was useless and never really said anything concrete about the game.

Brad Wardell

02-26-2006, 08:18 PM

But if Tom Chick or Bruce Geryk or any of the other game reviewers you respect reviews your game for CGW, goes into it in depth for 3 pages or so, and basically gives you all sorts of analysis and criticism---and just only doesn't give you that numerical score at the end, is that not good enough? I'm asking. Or what if, as I said, we gave Editors' Choices to worthy games, and Verdicts to those and everything else? Is that still not enough?

Is a review not valid/worthy/important until there is a number attached?

Except they can't go into 3 pages of depth (unless you've really gone crazy over there). Spring, Summer arrives and there's just not enough editorial pages available to give that kind of examination of a game.

Besides the issue is really how it affects different arguments.

Brad, the hard core gamer, much prefers in depth reviews.

But brad, the evil, baby eating game developer knows that those ratings matter. And not having CGW doing a rating means that the ratings from other game publications matter more and not all of them are as thorough and honest as CGW. And you know I'm not saying that just to suck up here, everyone here knows what's not being said so let's leave it at, not everyone is as thorough as you guys and now they have even more leverage.

BobJustBob

02-26-2006, 08:26 PM

If a magazine or site went to a thumbs up/down review system, would gamerankings convert those to 100% and 0% respectively? Given existing reports of publishers complaining about reviews that lower the average score, that would be fun to watch.

stusser

02-26-2006, 08:27 PM

Game developers aren't CGW's core audience, players are. That's a good thing, because there are a lot more players than devs. Editorial's purpose is to serve their audience. They aren't supposed to think about how changes in editorial will affect their value to publishers. The separation of editorial and advertising is pretty much sacred. That's why Jeff said he doesn't care how gamerankings responds. He shouldn't care about anything but producing quality content for his readers.

Whether this is successful or not, it's great to see that they're looking at the big picture and taking risks. I've noticed quite a few more "daring" in-depth articles in CGW and OPM over the past couple of months. OPM's J. Allard interview, the MMO history series, and so on. I'd love to see more of them. As the numbers of gamers steadily grow, the numbers of hardcore gamers will increase proportionally, and (if there isn't already) I really believe there will be a market for this kind of content. A loyal market.

The lowest common denominator wants a score. They want to flip through the magazine, see that a game got a 7.5/10, and compare it to others in the same genre. But the lowest common denominator is already being served by 1000s of sources on the internet with response time measured in minutes. That's the point being made here, I think.

Jeff Green

02-26-2006, 08:32 PM

Except they can't go into 3 pages of depth (unless you've really gone crazy over there). Spring, Summer arrives and there's just not enough editorial pages available to give that kind of examination of a game.

Besides the issue is really how it affects different arguments.

Brad, the hard core gamer, much prefers in depth reviews.

But brad, the evil, baby eating game developer knows that those ratings matter. And not having CGW doing a rating means that the ratings from other game publications matter more and not all of them are as thorough and honest as CGW. And you know I'm not saying that just to suck up here, everyone here knows what's not being said so let's leave it at, not everyone is as thorough as you guys and now they have even more leverage.

We just might be crazy, Brad.

Your perspective is interesting, for sure, though perhaps we need to get a room if we're gonna keep talking so intimately in front of all these....goons...here. Would it make it any easier if I told you we wanted to give GalCiv II a 1.5? (Just kidding! Or am I?)

And I *knew* you ate babies! Told ya, Darren!

mono

02-26-2006, 08:32 PM

I see Brad's point, but the numerical rating's matter to him as a game developer and publisher. As a consumer, there's no similar down-side, and I'm all for no scores.

TheWombat

02-26-2006, 08:56 PM

CGM, when it was Strategy Plus, had no ratings with the reviews, back in the dark ages. I remember writing some of those. The arguments on the journalistic side for and against ratings seem to be the same now as then; only the business side of things has changed, and radically.

Brad Wardell

02-26-2006, 09:07 PM

We just might be crazy, Brad.

Your perspective is interesting, for sure, though perhaps we need to get a room if we're gonna keep talking so intimately in front of all these....goons...here. Would it make it any easier if I told you we wanted to give GalCiv II a 1.5? (Just kidding! Or am I?)

And I *knew* you ate babies! Told ya, Darren!

Well with the hard drive formatting "issue" along with the fact we changed it into an adventure game at the last second, I guess a 1.5 wouldn't be that surprising.

But what is really worse? 1.5 stars OR the reviewer going on for 3 pages talking about the fact that the game was largely a series of home photos of my pets in which the slide show crashed if you attempted to do anything with it?

Regarding eating babies.. Sure, like most people, I've dabbled in cannibalism. But that doesn't make me some kind of monster...

Sidd_Budd

02-26-2006, 10:00 PM

I thought CGW's decision to not do a Game of the Year was interesting, but I'm unhappy with removing the ratings. As a subscriber, I'll now receive less information than I was before, with no additional content (unless you plan to substantially increase review length, which would be cool). What's more, the decision to remove the ratings seems to be driven by:
1) editor discomfort and annoyance with handling negative industry reaction to ratings
2) an assumption that readers who find numbers useful are stupid and don't take the hobby seriously enough

While I feel for Jeff and the rest of the editors, reason 1 shouldn't result in less information for the readers. If I'm pissed at my university's internal administrative policies, should I alter my class content and have my students suffer? Or do I give as much useful information as possible, and just deal with the administrative hassle as a downside of the job?

Reason 2 is prevalent among editors, writers, and 99% of Qt3 posters. I'm firmly in the "games are not art" camp, however, and all the arguments against numerical ratings strike me as pretentious. When a review contains both text and a score, readers don't have to look at the score if they don't want to. To me, CGW has restricted choice and decided on the proper way readers should evaluate games.

I give CGW credit for trying something new, and I'll give the new system a try, if only because I've got a while before my subscription expires. But I suspect I won't renew my subscription, because I'm getting less value compared to the other magazines.

If you bring back Greenspeak, however, I might be willing to reconsider all of this.

Gideongamer

02-26-2006, 10:25 PM

DennyA

02-26-2006, 11:09 PM

Yeah, a rating system was essentially forced upon CGW in around 1994, I think? (Denny?)
Yep, I think it was 93 or 94, because it was about the time I started freelancing for the magazine, and I joined the staff in 95.

I don't think it was "forced," as much as suggested. Johnny didn't like the idea, but he did admit that the market kind of forced it, since everyone else had the ratings and readers were coming to expect them.

Kevin Grey

02-26-2006, 11:42 PM

I thought CGW's decision to not do a Game of the Year was interesting, but I'm unhappy with removing the ratings

Considering that you maintain a database of magazine scores color me unsurprised that you'd be unhappy with this.

Jeff Green

02-26-2006, 11:49 PM

Practically everyone I know uses gamerankings and most people like it.
Most of us love the point ranking system. Take 40-50 sites and average all thier scores together and you can get a reasonable approximation of critic reviews. If you want more than you click on a link to read the review. Its all about convienience.

I don't want to sift thru 3 pages of shit only to find out the game reviewer hated the game- oh and 50 other game reviewers also thought it sucked or they all loved it. I don't see games as "art" either. While some games may contain some hand drawn art, when Yahtzee, Monopoly, Bingo, and other games become art then I will concede. Unless its by a critic I have heard of, if I can't get a fast rating up front then I won't bother with the review. Gamerankings is the bomb :)

And here we have one perfectly valid end of the spectrum. Some people claim they hate ratings and would rather have longer reviews with more thoughtful analysis....while others would see such a move as having to "sift thru 3 pages of shit."

This is what leads editors to drink.

AndrewPf

02-27-2006, 12:35 AM

To you guys, what do you care? Who the heck am I to you? Just some weird guy who named his lizard "Frodo". But to many of us, CGW is one of the few publications that actually plays a game thoroughly before reviewing it. Tom Chick or Bruce Geryk or whoever might rip us a new one, but we'd know they'd played the game.

The industry is still going to focus on the numerical scores, except now there's no CGW to counter balance the less than...thorough reviews.

Without going into too much detail, the "missing" CGW review balance will hopefully be...erm, balanced...by the 1UP review -- and we intend to keep many, if not all, of CGW's trusted names writing them.

I think a lot of people will dig this new system we're trying.

interman

02-27-2006, 12:58 AM

I think it's interesting to look at the high-end audio industry, where obviously a lot of products are reviewed every day. In the top magazines you'll pretty much only find medium to long reviews, typically with a conclusion at the end. A decent percentage start by skimming through the magazine, reading mainly the conclusions, but I think it's better as a whole that those truly interested in the product have to read the whole thing and decide for themselves.

Then you have other hi-fi magazines that are more like product catalogues. Here you have the same scores as 'we' have, but they're aimed at a different kind of consumer, and I think that's the key here. I just don't think the majority of the readers are ready, or are in the right mindset for something like this, even if it only means they have to spend a couple of minutes actually reading the text.

I'd love writing reviews without scores, since it would force me to write in a way where the scores are truly conveyed through text. A lot of reviews tend to just list what's wrong with a game, with a "but it's really not that bad, because" near the end.

So yeah, I don't see it happening, but it sure would be interesting, and the overall quality of reviews would probably improve.

EFlannum

02-27-2006, 02:30 AM

And here we have one perfectly valid end of the spectrum. Some people claim they hate ratings and would rather have longer reviews with more thoughtful analysis....while others would see such a move as having to "sift thru 3 pages of shit."

This is what leads editors to drink.

Forget drinking try that whole "eating babies" thing... it works wonders.

jpinard

02-27-2006, 05:56 AM

So what's Scorpia up to nowadays?

DeepT

02-27-2006, 07:28 AM

I am surprised Tom hasn't commented on this yet, perhaps he is waiting for my post so he can pounce on me like a cat with a new toy.

I unlike what some people think (http://www.quartertothree.com/game-talk/showthread.php?t=24512), I do read the acutal text of the reviews. However most of what is written doesn't help me as a gamer buyer. So the graphics were nice,but could be better. So its kind of cool how unit X looks when its running over infantry. So the unit ackowledgements are not to annoying... blah blah blah.

I have to interprit what you are saying and what you *really* mean. I liked how old-school AVault used to do it. They would give you the score first, *then* tell you why they gave it. For example they would give a UI 3.5 stars (for the record, I hate the 'stars' system) and tell you what was good and bad about it.

An overall ranking of a game isn't that useful to me because not all things weigh equally to me, so an average score of all categories is a poor way to do it. For example I care a great deal about multi-player, and much less so of single player. Audio isn't as important as graphics, and graphics isn't nearly as important as gameplay.

A score for each category makes me feel the reviewer really need to stop and consier thier gut feeling on something. IE: The UI feels like a 85%. Its really good but has a few minor weak points, but no huge flaws. The AI is a 30%, its really bad and easy to beat, but is good enough to present a challenge to a complete n00b long enough to figure out what units are good vs what other units.

Overall scores are meaningless. The individual category scores are not.

Troy S Goodfellow

02-27-2006, 07:35 AM

A score for each category makes me feel the reviewer really need to stop and consier thier gut feeling on something. IE: The UI feels like a 85%. Its really good but has a few minor weak points, but no huge flaws. The AI is a 30%, its really bad and easy to beat, but is good enough to present a challenge to a complete n00b long enough to figure out what units are good vs what other units.

Overall scores are meaningless. The individual category scores are not.

I recently got into a debate with another reviewer over this sort of thing. I greatly dislike separate scores for separate parts of a game - and even moreso if they cook up some fancy formula to come up with a final score based on these ratings.

First, no one ever includes User Interface (which is in your example) but they always use sound, no matter how generic. Second, "gameplay" is always included despite its completely nebulous meaning. A UI is part of gameplay. So are graphics, properly understood. By disaggregating all this stuff, there is an impression given that a game is merely a bunch of parts that somehow fit together into a whole.

Apply this idea to movies and it becomes instantly ridiculous. Scores for screenplay, male lead, female lead, special effects and Reviewer's Tilt.

BTW, what do you not like about the "star system"? It's a simple ten point scale.

Troy

Chris Nahr

02-27-2006, 07:54 AM

Apply this idea to movies and it becomes instantly ridiculous. Scores for screenplay, male lead, female lead, special effects and Reviewer's Tilt.

Actually, that's exactly how Oscar nominations work... well, except for the Reviewer's Tilt. I don't see anything wrong with evaluating different aspects separately, although I agree that the present systems don't work too well. Something like this might work: Presentation, Game Mechanics, User Interface, Story (if present), Multiplayer (if present).

DeepT

02-27-2006, 08:01 AM

BTW, what do you not like about the "star system"? It's a simple ten point scale.
Troy

Aha! You see, thats what *I* thought. And then several reviewers jumped on me and told me that is *not* at all what it means and everybody knows this.

Troy S Goodfellow

02-27-2006, 08:24 AM

Aha! You see, thats what *I* thought. And then several reviewers jumped on me and told me that is *not* at all what it means and everybody knows this.

By a ten point scale I mean that it has ten points on it - eleven if you count zero.

It is a ten point scale, literally, but not equivalent to most ten point scales, if you get my drift. 3.5 does not necessarily equal a 70/100. 3.5 stars is above average, verging on good. Even 3 stars means it is to be recommended to some. I prefer the stars precisely because they don't have the baggage that numbers do.

Actually, that's exactly how Oscar nominations work

Not quite, since the Oscars don't presume to combine all these bits together to give you an impression of the movie as a whole.

Troy

GH33DA

02-27-2006, 08:32 AM

THREAD'S SCORE: 70%

HIGHS:The first post, a couple of great points about the gaming media and game development

LOWS: Double postings, usage of the word "l33t"

-OR-

GLANCE OVER

While this thread does have it's moments, you get where it's going right way, and after that it's just more of the same.

//

Honestly, I don't make any of my game purchase decisions based on just a score. If I'm interested enough in a game, not only am I going to read the whole review, I'm also going watch a few trailers and check out the website. My interest in a game isn't going to be peak by a 100% anymore than a %70 percent. My interest mainly comes from people I interact with and eye candy. Sad, but true. Sad like E3 taking away the eye candy. Anyway, it is sad that that things have gotten a bit disjointed as far as what game reviews are suppose to do and how they are being used. I'm curious how accurate the link is to actual purchases.

Simpilot

02-27-2006, 08:34 AM

I recently got into a debate with another reviewer over this sort of thing. I greatly dislike separate scores for separate parts of a game - and even moreso if they cook up some fancy formula to come up with a final score based on these ratings.

First, no one ever includes User Interface (which is in your example) but they always use sound, no matter how generic. Second, "gameplay" is always included despite its completely nebulous meaning. A UI is part of gameplay. So are graphics, properly understood. By disaggregating all this stuff, there is an impression given that a game is merely a bunch of parts that somehow fit together into a whole.

Apply this idea to movies and it becomes instantly ridiculous. Scores for screenplay, male lead, female lead, special effects and Reviewer's Tilt.

BTW, what do you not like about the "star system"? It's a simple ten point scale.

Troy

You better duck! I personally don't have a problem with the star-scale or the 10-point scale. What I DON"T like is the 100-point scale like PC Gamer's. Can anyone tell me the difference between a 74 game and a 73 game? How about even an 85 and 87? The scale is too wide. Now I can tell you the difference between a 70% and 80%. Oh, and while I appreciate the walk down memory lane back to the days of when CGW and CGM didn't have rankings (I wrote a few of those as well), it's just not what today's readership wants, and I think it's a doomed experiment. I think virtually every advertiser will rise up and demand their numbers, and anyone who says that advertisers have NO input doesn't understand the mag biz. Editorial and advertising are seperate entities and always should be, and ads should never have a direct influence on editorial content, but sooner or later they will have an influence of the general feel of the magazines. Many will probably feel so strongly about it they will take their advertising dollars elsewhere. After already making one stand this year, I doubt you'll see PC Gamer circle the wagons around this issue.

Nellie

02-27-2006, 09:51 AM

The score basically helps me decide whether or not to read the review of a game that I'm not that bothered about. If it's really high I want to know what's so great that's slipped under my radar, if it's really low I might be entertained by someone ripping a piece of crap to shreds, otherwise move on.

I suppose you could suggest that for a poor reviewer who can't convey in the three pages of bleh that they've written about the game whether or not they think it is any good that perhaps it's helpful. But someone telling me in the review they think it's a really great game, it won't format my harddrive or send my porn via email to my mum and it'll run on my PC isn't going to sway my decision by ultimately only giving the game 82.3% instead of 90%.

I can see the screenies, I know whether or not I'm interested in the game and if I haven't had much other exposure to it, I want to know whether or not it's any good; and the arbitrary scale of 7-9 or 3-4 really doesn't do a great deal for me.

GlaziusFalconar

02-27-2006, 10:02 AM

I rather fancy Eurogamers' 10 point scale. (http://www.eurogamer.net/scoring_policy.php)

The classic Likert scale (turning opinion into numbers since 1932!) only has 5 responses, though some people expand it to 7 or 10. There's no Likert scale for game reviews, so without some universally accepted gradation among the reviewers it's meaningless.

I do appreciate rankings on game elements, but that creates the illusion of equality - having great audio elements is a huge selling point for, say, Lumines, but is it as much of an issue for Civ IV?

--GF

Brad Wardell

02-27-2006, 10:22 AM

Jeff:

With regards to 3 page reviews. I am still unclear how this is possible. How do you have enough editorial pages?

Will you be able to still cover lesser known games with reviews?

mouselock

02-27-2006, 10:42 AM

The score basically helps me decide whether or not to read the review of a game that I'm not that bothered about.

...

I can see the screenies, I know whether or not I'm interested in the game and if I haven't had much other exposure to it, I want to know whether or not it's any good; and the arbitrary scale of 7-9 or 3-4 really doesn't do a great deal for me.

You mean other than alerting you to games that you normally wouldn't have been bothered to care about in the first place?

That's kind of my problem with removing scores as a concept: I don't want to read three pages seeing what Tom Chick did/didn't like about stuff. Not because I don't care about his opinion, but rather because I already know how my opinions and his tend to coincide. So if Tom gives something a 110% maybe it's worth me stopping and perusing. If he gives it a -40% then I know I don't even care to know what genre the game was in, much less why he hated it so much.

Scores are a level in heirarchical abstraction of game information. First you look at the scores and who's doing the review and decide if it's worth going more in depth. If there are short blurbs, then you check those out. If you're still interested then you either read through the review (sometimes, not all that often) or go out and find the demo (yay broadband!).

In general, I find textual reviews to be pretty meaningless because inevitably what bugs me about a game someone else has no problems with. The scores and short blurbs, however, give me some idea of an overall impression of quality which I can use as a launching pad to apportion my game-trying-out time in a better way.

Rywill

02-27-2006, 10:57 AM

Brad seems to be identifying two different problems (or potential problems) with lack of scores:

1) Lack of score means the CGW review doesn't get "counted" by publishers, which is bad for devs who make games that CGW likes. Stusser's reaction is "I don't care, the magazine is written for gamers, not devs, and all that should matter is whether gamers are better served." Yeah, CGW is a business, but since I assume both of their revenue streams (ads and subs) are driven by readership levels, its sensible for them to do stuff that readers will like even if some devs don't like it.

But then you can go a step further: let's assume there are some small-shop devs out there who make good games but have trouble getting them into stores. Cutting out scores hurts those devs, which means fewer (or worse) games from them. Which, in a roundabout way, is bad for the readers of CGW whether they realize it immediately or not. So then the question is, if we accept that as true, should CGW be doing stuff to try and "improve the gaming marketplace" or whatever? If putting scores on reviews will in some way encourage better games and make good games more successful, should CGW do it even if they and their writers don't really want to? My instinct is "No," but I can see people arguing both sides.

2) Cutting out the score is less informative, and has no real benefit (Sidd is making this argument even more forcefully). This seems like mostly an esthetic issue to me. I don't see a problem scoring a product, even an "artistic" product. Movie reviewers certainly do it and that doesn't seem to be a problem. On top of that, it sounds like CGW is going to have a sort of scoring-system-we-won't-call-a-scoring-system, in that everyone apparently gets a verdict (GUILTY!) and some people get Editor's Choice, so there are at least three possible "ranks" in that system, like the miss/hit/direct hit system.

In any case, I've never understood why people are so up in arms about the score or lack of score. I like it just because it helps me decide what to read -- I read most of the big-name reviews in CGW, but the lesser games I'll often skip. But if I see a two-paragraph review that has five stars attached to it, I'll probably read that just to see what the fuss is about and whether this is one of those small-dev gems that I shouldn't miss. Now I won't have that. Brad eating extra babies as he reads this because he's so upset. Anyway, the point is, I tend to agree with Sidd that the score adds information and I don't see how it really hurts anything.

The best argument I can see for the benefit of no scores is Jeff saying "I spend 90% of my time talking to people about the scores." I assume he'll now spend time defending the substance of the reviews rather than the scores, but let's assume that he'll spend less time (particularly if Gamerankings drops CGW and the PR people no longer care about CGW that much as a consequence). That time can be used for other things (*cough*Greenspeak*cough*) which will presumably improve the quality of the magazine. But, like the Brad Wardell trickle-down-economics Gamerankings point, that seems sort of removed to me. Will it really make a difference? I'm glad CGW is experimenting with it so we can find out, but I'm not so sure that it will outweigh the benefit of having scores to help me catch those small gems. Then again, I suppose a favorable verdict + ed's choice would do the same thing as five stars.

However it works out, I'm interested to see how it goes, and kudos to CGW for trying it.

Nellie

02-27-2006, 10:58 AM

You mean other than alerting you to games that you normally wouldn't have been bothered to care about in the first place?

Depends on the medium. Print magazines the score is slightly more useful as a benchmark on whether to read the review and then possibly check out the game. Why read a review of a game that's scored at 70% and you aren't interested in anyway? I might come back to it later if I'm bored and have nothing better to do, but I doubt it.

When it comes to internet reviews, I tend to search out game reviews for titles that I'm already interested in. In this case the opinion of the reviewer is far more important than the score. The score doesn't tell me what's wrong with the game and from site to site, the difference between an 8/10 game and a 9/10 game can be down to any number of trivial matters up to and including a clearly advertised Single player only game not having multiplayer.

mouselock

02-27-2006, 11:13 AM

When it comes to internet reviews, I tend to search out game reviews for titles that I'm already interested in. In this case the opinion of the reviewer is far more important than the score.

I wouldn't think otherwise. I just don't think that scores necessarily even fill the same role as reviews. Reviews are designed to give you an encapsulation of the pertinent information about a game. Scores are designed to tell you whether the reviewer felt the game was worthwhile or not. While there's overlap there, it's not 100%, and there's not necessarily always an easy way to map between the two. I've seen reviews where a reviewer focused on all his disappointments with a game yet gave it a good score because the game was good and the reviewer just wishes it had been even better. I've seen reviews where the reviewer focused on all these nifty, novel things a game did only to pan it because the whole wasn't as good as the sum of its parts.

You can probably get this information without the score to anchor it, but you'll almost certainly have to read through multiple reviews of games you have your own objective standard for by that reviewer to be accurate there. Distilling the essence of a list of commendations and complaints into a score tends to take some of that guesswork away. (You're still on your own for whether your tastes coincide well enough with the reviewer's to think that his 80% game is your 80%. Or, more likely, his 80% game of genre X is your 80% game of genre X. But at least you don't have to second guess what his relative rankings within his own schema are.)

Charles

02-27-2006, 11:14 AM

You know, rottentomatoes is awesome and IMO, gamerankings should work the same way. read the review, figure out if it was predominantly positive or negative, and count it as a 1 or 0.

On top of that, I don't have a problem with not having numbers available, but I do use them, as some people have said, as an initial screen on what reviews I read. If it seems like a game I normally wouldn't care about, but it gets a 9.5 or whatever scale you like, then I know that maybe there's something good about it and worth reading, that maybe there's a game I'd like there.

They don't have to be numbers, but I want *something* which quickly summarizes the overall reviewer's opinion about the game. Because I don't care to read every single review in a magazine, I want to be able to skip those that don't concern me, and that's harder to do if there isn't some kind of abstraction in there.

Troy S Goodfellow

02-27-2006, 11:48 AM

I assume he'll now spend time defending the substance of the reviews rather than the scores, but let's assume that he'll spend less time (particularly if Gamerankings drops CGW and the PR people no longer care about CGW that much as a consequence).

I think his time on score-defense with PR will go down primarily because it is easier for a marketing weasel to quibble over a half-star than over whether or not Di Luo finds the graphics in Tin Soldiers: Sargon's Wrath washed out and unclear. With scoring, you can immediately find comparisons within the same magazine - often the same issue - and debate whether your game is just as good as Panzer Pony Party.

Or at least go down if they keep up the experiment. For the first month or two, I would expect phones and emails of rage.

What's the over-under on threats of cancelled subscriptions, btw?

Troy

Rob Beschizza

02-27-2006, 12:15 PM

Practically everyone I know uses gamerankings and most people like it.
Most of us love the point ranking system. Take 40-50 sites and average all thier scores together and you can get a reasonable approximation of critic reviews. If you want more than you click on a link to read the review. Its all about convienience.

But gamerankings doesn't aggregate the reviews -- it aggregates the score given at the end of the review. Fair enough if you find that scores generally correspond well to the quality of the game. The problem I have with gamerankings is that it averages out good and bad reviewers. Gamerankings provides an overview of critical reaction, but it does so by being a very average, mediocre meta-reviewer who gives a score without actually having a personal, subjective reaction to the game and a review describing it.

I prefer metacritic, because they excerpt from reviews very well, and I can scan the excerpts and filter out those who are obviously poor critics.

I don't want to sift thru 3 pages of shit only to find out the game reviewer hated the game-

When this happens, you're usually a victim of weak journalism. Some reviewers need to be beaten with a giant rubber inverted pyramid.

Tom Ohle

02-27-2006, 12:29 PM

See, I'm torn on this issue. I usually take the time to read reviews of games I've worked on (or ones I'm considering buying)... but it's nice to be able to read a couple to get an idea of the overall strong and weak points, then to simply scan through the ratings to see if those reviews reflect the general consensus... that said, I'd like to see more outlets take on a summary paragraph (or 60-second review) that outlines the key elements. Sometimes I just don't feel like reading through a 3-page review.

Troy S Goodfellow

02-27-2006, 12:31 PM

Sometimes I just don't feel like reading through a 3-page review.

Yeah, but are there many of these? At least from sources you've come to trust? Avault does the three page review thing. Gamespot for bigger games, but there's never that much text on a single page.

The 3 page review problem is a red herring. These are few and far between and almost never in print.

Troy

Charles

02-27-2006, 12:32 PM

Regardless of actual numerical values or not, I sure hope any mags that do this at least keep a small "Good and bad" section of summary.

Tom Ohle

02-27-2006, 12:48 PM

Yeah, but are there many of these? At least from sources you've come to trust? Avault does the three page review thing. Gamespot for bigger games, but there's never that much text on a single page.

The 3 page review problem is a red herring. These are few and far between and almost never in print.

Troy
You're absolutely right. In general, my argument only applies for online outlets; it seems many writers just get carried away with the fact that they CAN write multi-page reviews... in print, it's not a concern, as all but the biggest games only get a quarter of a page, anyway.

KieronGillen

02-27-2006, 12:54 PM

For fear of Stating The Obvious, Rotten Tomatoes was aggregating reviews well before Metacritics and sidesteps the relative lack of marks for films reviews by just taking their impressions from the review. And they do games too, y'know...

If everyone dropped their scores, Metacritics and Game rankings and whoever would just have to do a little more work and actually read the reviews they're parasitising.

Essentially, nothing would change.

In which case, we return to the Editor to decide what best servers their readership.

KG

Tom Ohle

02-27-2006, 12:57 PM

As a follow-up, I think the numerical ratings are so engrained into the collective mind of the games community that they couldn't be removed without causing rampant confusion and feverish discussion on forums like this one. Developers and publishers are happy to think, "our game got 92%! That other game only got 90%! We're way better! Go team!"
It's not just for bragging rights... it's actually a boon to morale (obviously, unless you've got a stinker). The few days after a game release are some of the most agonizing days at a development studio... waiting in anticipation, only to have that first review come out without a score would just make everyone go crazy.

jfletch

02-27-2006, 12:57 PM

You're absolutely right. In general, my argument only applies for online outlets; it seems many writers just get carried away with the fact that they CAN write multi-page reviews... in print, it's not a concern, as all but the biggest games only get a quarter of a page, anyway.

They can also put click-throughs in between every third page.

IGN's reviews frequently top 1500 words, which is just ridiculous - it gets up to 2500 or 3500 for the big games, which is even worse. They go into every detail and tell us stuff thats in the instruction manual. I would've gotten an F in any of my college writing courses if I turned a paper with as much filler as they have. I never read any of these, but Id be lying if I said I didnt go to the last page and read the summary and the scores.

DennyA

02-27-2006, 02:02 PM

Brad Wardell

02-27-2006, 02:47 PM

I love it... Five years of "ratings suck, reviews were so much better before they had them" here on QT3, then Jeff announces they're going away and it's all "Ack, I need my ratings!

Hmm.... Jeff did say he's doing this in the April issue... :)

But to be fair, Qt3 isn't a single entity. I certainly never lobbied or argued to get rid of ratings.

I suspect Tom has no love of ratings given his recent discussion on the AOE thread so that is where some of that may have come from.

My beef summed up comes down to this: GameRankings and other aggragators matter. They're used. We may not like it, but that's the way things work. By CGW not doing ratings, that puts just that much more leverage with the publications that don't do as thorough of reviews.

It's bad enough that in the past a solidly reviewed CGW review gets countered by GamerSiteXYZ who does some drive by review. But now there won't even be a CGW to look at.

I don't know about others here but when I go to buy a game, I will tend to look at the Game Rankings score to help toss out games that I definitely wouldn't get and then look at the reviews from CGW, CGM, and PC Gamer along with a few of the major gaming sites. But without CGW, which I rely on, I have fewer resources to turn to.

Now Jeff says they are going to do 3-page reviews. If he has that kind of clout now at Ziff then my hat's off to him. A *well edited* 3 page review could be amazing. But since those reviews aren't accessible on-line (are they?) then once the issue is out of circulation, those reviews end up in null with no recorded "score".

Besides, it's not going to stop unscrupulous game publishers. Even as I type, the next run of GalCiv II boxes already have a made up quote from Jeff Green:

"Galactic Civilizations is the single greatest achievement of the human race. Nay, the greatest achievement in the universe. It was, as if life was formed simply to lead us to a single destination -- the creation of this game." -Jeff Green, Computer Gaming World, 6 stars.

The marketing literature even includes a Photoshopped picture of Jeff Green holding the game box with a thumbs up (the original was taken from a 1998 "incident" involving Pearl Jam and the illegal Shark fin trade that Jeff was notorious for back in that time).

Hmm, gotta go, the lawyers keep waving something at me..

Jeff Green

02-27-2006, 03:01 PM

I love it... Five years of "ratings suck, reviews were so much better before they had them" here on QT3, then Jeff announces they're going away and it's all "Ack, I need my ratings!

Hmm.... Jeff did say he's doing this in the April issue... :)

Like I didn't know that would happen. :)

The marketing literature even includes a Photoshopped picture of Jeff Green holding the game box with a thumbs up (the original was taken from a 1998 "incident" involving Pearl Jam and the illegal Shark fin trade that Jeff was notorious for back in that time).

Dude, let's at least make it Nirvana, okay?

mutt

02-27-2006, 04:05 PM

Jason Cross

02-27-2006, 04:14 PM

CGW's lack of ratings: CGW was great back in johhny wilson's times. Where the hell is scorpia, anyway?

Don't worry, Jeff. In 15 years, people the then-30-somethings of the world will talk about how "CGW was good back in Jeff Green's times." And where the hell is Gladstone, anyway? ;)

Interesting move on the no-ratings thing in April. Look forward not so much to seeing that (because I read the text anyway if I'm interested in a game), but to see what the fallout is one way or the other. We talked about it a lot at CGM and never pulled the trigger. It always came back to "reader expectations" and all that kind of stuff. Plus, we would come up with arguments like, "but you know, sometimes I see a game I didn't know a lot about get 4.5 stars or 90% or something, and it prompts me to read the review because, hey wow, this must be some kinda great game I overlooked."

Reading the text is great and all, but I won't pretend to read the text of every game mag or site that I frequent (and I'm an uber game nerd). There are some games that are just off my radar, because of subject matter or genre type or whatever. Exceptionally high ratings in an unexpected place, or super low ratings on a game that is getting positive reviews elsewhere, has prompted me to read reviews I would have skipped.

Ratings: I'm for 'em. I think numeric scales suck because people equate them to letter grades and therefore you don't use the whole thing. 6/10 or 60% therefore becomes bad, not above average, etc.

If I had it my way, I'd use the school letter grading system. Hell, what all know what they all mean, there's enough granularity to seperate the "really good" games from the "top 5 games of the year" or the "slightly below average" from the "a pox on your hard drive" level, and you don't run into the problem of not using the whole scale.

Plus, gamerakings.com would probably properly translate those into numbers that makes sense. 3/5 shouldn't be 60% for them, but a D- should be, and they'll probably get that. At least, I bet you could convince them.

Jason Cross

02-27-2006, 04:22 PM

The whole idea of scoring a game, no matter whether it's via stars or percentages or thumbs-ups or whatever, is stupid from the get-go. Games are judged on their quality, not their quantity, and quality can't be measured mathematically.

Measured, no. But subjectively ranked, yes. Which is what every reviewer and judge in the world does. A figure skating judge subjectively rates the skater based on their own expert knowledge of the sport, on a scale where 10 represents some kind of ideal and 1 represents abject failure. They're not mathematically measuring the quantity of skating they do. They're not simply reporting the linear figure of the height of their jumps multiplied by the number of rotations or whatever.

Restaraunt critics do this. Movie critics do this. Hardware reviewers do this (though in that case, a measurable performance metric is PART of the consideration for a review score). Normal people do this to their friends, when their friends ask "should I buy game X, or game Y, do you think?"

As there are more things out there that are technically similar, or measurably insignificant from each other, or where the quantitative differences don't matter, the more important it becomes to help people decide how to spend their time and money by subjectively placing qualitative differences along a scale.

TomChick

02-27-2006, 05:29 PM

Ack, I'm late to this party! I've been waiting for the announcement so I could weigh in and it leaks out when I'm not around. I don't mean to derail this thread, but I just posted a screed on the front page, linked at the top of the forum.

Personally, I'm surprised that so many of you regard ratings as an indispensible part of the review process. You don't get that in other forms of entertainment. Can you imagine Pauline Kael or Lester Bangs have to assign a 7, 8, or 9 to something? Even worse, can you imagine them having to express the difference between an 8.5 and an 8.8 Ridiculous!

There's certainly room for the silly numbers games, which isn't going away any time soon. But frankly, it's about time to give another approach a chance.

-Tom

Linoleum

02-27-2006, 05:54 PM

Bah, I know the real reason Brad made a thread referencing GameRankings!

http://img528.imageshack.us/img528/2795/gc21hl.jpg (http://imageshack.us)

(I kid! I kid!)

steve

02-27-2006, 07:07 PM

Can you imagine Pauline Kael or Lester Bangs have to assign a 7, 8, or 9 to something?
Can you imagine Pauline Kael or Lester Bangs reviewing DOOM 3?

Brad Wardell

02-27-2006, 07:22 PM

Dude, let's at least make it Nirvana, okay?

Oh..I see, the band you object to but not the illegal shark fin trading! That's okay! ;)

TomChick

02-27-2006, 07:24 PM

Can you imagine Pauline Kael or Lester Bangs reviewing DOOM 3?

That would totally rock!

-Tom

Brad Wardell

02-27-2006, 07:31 PM

Bah, I know the real reason Brad made a thread referencing GameRankings!

http://img528.imageshack.us/img528/2795/gc21hl.jpg (http://imageshack.us)

(I kid! I kid!)

To me, that's just proof that there's something wrong with Gamerankings. Frankly, any site that would list GalCiv at the top is not a site I would trust. I mean, how do they even measure popularity? Can't be retail units, no way we would be beating console games or Star Wars. Can't be page hits to the game, the reviews aren't even out yet to be prying at. Simply put, any site that would have me as a member isn't credible.. er..sorry Tom.

I guess the thing about CGW is that at least I have a good idea what the game will get when it goes there. If it's a bad game, it'll get a review score between X and Y. And if it's a good game it'll get a game score between A and B.

A lot of the sites listed on game rankings I feel like it's a total luck of the draw.

Heck, even in terms of criticism, I feel I could be ripping on Jeff Green and not worry about it affecting the CGW review. Despite what I did to Jeff's car after GalCiv I came out (and look, it was an accident, I have...a problem okay?) they didn't take anything off the review.

I have no idea what the professionalism of other publications. That doesn't mean they're not professional, I simply have no idea. I only know that if I were a reviewer and someone even looked at me funny or they had a hard to spell name that I would be deducting a half star right there just on principle. So I guess what I'm saying is that I fear that other sites on game rankings might review games as I would...

Tom McNamara

02-27-2006, 07:46 PM

They can also put click-throughs in between every third page.

IGN's reviews frequently top 1500 words, which is just ridiculous - it gets up to 2500 or 3500 for the big games, which is even worse. They go into every detail and tell us stuff thats in the instruction manual. I would've gotten an F in any of my college writing courses if I turned a paper with as much filler as they have. I never read any of these, but Id be lying if I said I didnt go to the last page and read the summary and the scores.

You know, I hate to ask, but... if you never read the review, then how do you know it's full of filler? We're in-depth, I say! In-depth!

In the interest of avoiding a threadjack, though, I'll give you my side of things: The ratings system at IGN is one of the most dreaded topics in the department. You want to see us get into an argument, ask someone if they think the ratings scale should be changed (or eliminated). A topic of endless debate.

I personally agree that a 100-point scale is too granular. I'd like to see a 20-point scale -- 1, 1.5, 2, 2.5, et cetera. Some others think no score at all would be great. But our stances, in the long run, are as divergent as those in this thread. It's probably the only thing that isn't common ground for us. That and the fact that Halo 2 is overrated.

Damien Neil

02-27-2006, 11:49 PM

You know, I hate to ask, but... if you never read the review, then how do you know it's full of filler? We're in-depth, I say! In-depth!

For what it's worth, I just tried reading a couple of IGN reviews to test this.

The writing was bad. Consider the start of the Space Rangers 2 review:

http://pc.ign.com/articles/691/691904p2.html

Big games don't always get big marketing budgets. Space Rangers 2: Rise of the Dominators is a great example. Even the name of the game screams cheap and has made several of my co-workers point and laugh at me.

Just look at that. We lead with a mock-philisophical point, connect it clumsily to the game in question in the second sentence, and wander off in a new direction in the third. Where it seemed that we were talking about marketing budgets, we're now talking about how crappy the game's name is. Finish up with a breathless exaggeration.

Let's continue:

So what is Space Rangers 2? Consider it a mix between Star Control 2 and Freelancer. It's a space adventure epic combining exploration, trade, turn based combat, interstellar travel, multiple alien races, arcade action elements, and real-time strategy.

Our first introduction to the game is an inept comparison to two others. Star Control 2 is a RPG bolted to a upgraded Spacewar. Freelancer is a multiplayer space sim. Which aspects of these games are similar to Space Rangers 2? Don't expect to find out here--this is the first and last mention of them.

And then:

I guess the point here is that anyone who actually has knowledge of the game will know how large and free it really is.

Wow. Just...wow. Anyone who knows about the game will know about the game. That's deep, man.

It goes on in that vein. "What's interesting about Space Rangers 2 is that the main focus of the storyline is already well underway when the game begins." (So, that's what's interesting about the game? Nothing else is?) "Space Rangers 2 certainly has a fiction driving the gameplay, but there's very little direction involved." (So wait, is the fiction driving the gameplay or isn't it? First it was, but now there's no direction from it.)

The reviewer has the love affair with adverbs so common among neophyte writers: Basically this, technically that, eventually the other thing, and thankfully it only goes on for two ad-filled pages.

This review gives no sense of what kind of game Space Rangers 2 is. It gives no sense that the reviewer knows what kind of game SR2 is. It isn't even entertaining to read.

In depth? Sorry, but no.

John Sansker

02-28-2006, 12:05 AM

Sounds great, looking forward to the April issue, even though I just picked up the March issue the other day.
One of the problems with this is something Brad mentioned.You are reading the reviewer's opinion of something and don't really give a damn whether or not he or she likes the game, the question is are you, the reader going to like it?

Mention of the "Postal 2" review, I laughed at that review.Figured that this "Robert Coffey" person was some whining shit who was terribly and personally offended by the game and therefore gave it the infamous 0 star rating.

This reminds me of another review "Knights of the Old Republic" the reviewer in Computer Games magazine said something to the effect of "Quite possibly the best RPG of all time, ever"
I was stunned by this. If this person actually thought that KotOR was the best RPG of all time, then how many RPGs could this person have played?
Or, even worse, what kind of horrific RPGs had this poor reviewer been subjected to in order to think this?

Without scores reviews become useless?
I tend not to take too many reviews seriously becauseof the scores.
I mean, it used to be PC Gamer would automatically deduct 10% off a score for no multiplayer support.(yet KotOR got a 91%) :P (I'm not actually quibbling over 1% here, more like the game's scorein my opinionwas inflated by at least 15% if not more.
As a result I tend to take PC Gamer's and Computer Games' scores with a salt lick.
CGW's scores I actually usually agree with.Of course now they are doing away with the scores. :D

If Bruce is taking the game that seriously we need to get him a couple of drinks or a bong hit or somethingMarch CGW.
I had to re-read that 2 or 3 times, and then couldn't stop laughing.

Desslock

02-28-2006, 01:13 AM

I mean, it used to be PC Gamer would automatically deduct 10% off a score for no multiplayer support.(yet KotOR got a 91%) :P (I'm not actually quibbling over 1% here, more like the game's scorein my opinionwas inflated by at least 15% if not more.

To give you additional context for what it's worth, I actually gave KOTOR a somewhat lower score in PC Gamer, but the editors were big fans of the game and bumped the score up a bit. I gave Gothic 2 a higher score, in the same issue, and Gothic 2 got the RPG of the Year Award.

In very mild protest, I also wrote an editorial on how KOTOR was NOT a great "role-playing" game (because actual opportunities to role-play were very limited, in its linear/restrictive and non-interactive environments), even though it was very good game. But I have no problem with KOTOR getting a very high score (and the rating difference between my original rating was very minor) -- it was a slow starter, and not a great RPG, but after the first few hours, that game was awesome.

Gordon Cameron

02-28-2006, 01:35 AM

I also wrote an editorial on how KOTOR was NOT a great "role-playing" game (because actual opportunities to role-play were very limited, in its linear/restrictive and non-interactive environments),

I think you could say this of a lot of classic CRPGs. I think the term "role playing" is in some ways a legacy phrase from pen-and-paper games despite the fact that other aspects of said games (high fantasy or sci-fi setting, stats-based character management, tactical combat, etc.) are what more readily translated over to the videogame medium during the formative years of the genre. Bioware do tend to go down the "linear/restrictive" avenue of CRPG design, which is not my personal favorite route, but they do it so well that I can't begrudge it and like you I think KOTOR was a great game (not as good as Gothic 2, though). I will however say that within the limited "good/evil" pathing that KOTOR used (descended from the similar approach in BG2), I actually felt a pretty distinct gameplay experience playing as Light Side vs. Dark Side, because the dialogue trees allow you to be a real bastard and the Jedi power spread comes out differently. I was eager enough to try the Dark Side, that KOTOR remains the only CRPG I have immediately started replaying the very day after I won it. That gives it a special place in my personal pantheon of CRPGs.

Desslock

02-28-2006, 01:41 AM

I actually felt a pretty distinct gameplay experience playing as Light Side vs. Dark Side, because the dialogue trees allow you to be a real bastard and the Jedi power spread comes out differently.

Sure, there were two distinct, linear paths -- that made the game more enjoyable and definitely very fun to replay, but it's not really role-playing depth, in my opinion, because there weren't really meaningful opportunities to personalize the experience or to role-play a distinct character -- you just chose one ending (and associated dialogue), or the other.

Anyway, we've covered this ground before and I think we generally agree on this stuff, and I don't want to derail this thread further.

Gordon Cameron

02-28-2006, 02:28 AM

but it's not really role-playing depth, in my opinion, because there weren't really meaningful opportunities to personalize the experience or to role-play a distinct character

A very nice McDonalds version of roleplaying, then, if you will. Which is part of my admiration for Bioware in general -- that they can take design approaches I may once have disapproved of, and make me love them anyway. I'm starting to feel the same way about Blizzard after WoW...

Last derailment, honest!

Nellie

02-28-2006, 04:42 AM

You can probably get this information without the score to anchor it, but you'll almost certainly have to read through multiple reviews of games you have your own objective standard for by that reviewer to be accurate there.

Or I get to know reviewer x or publication/site x has a similar taste in games to me. I watch Film 06 here on a regular basis (a show about movies), Jonathon Ross sums up the weekly releases and not once have I seen him have to resort to "<yak yak yak yak> Cinematography 7, acting 8, rewatchability 9, overall 8" when it comes to telling me whether a film is likely to be something I want to watch or not and I'm perfectly capable of forming the same opinion from a well written game review.

vizionblind

02-28-2006, 07:22 AM

hopefully it has a quick summary

Mark Crump

02-28-2006, 07:38 AM

I've never seen a review on TV use a "rating," other than Eberts' "thumbs up." However, Eberts' print movie reviews do use a rating.

The presentation is different. If you watch a review, you can get the subtle tones and inflections in the presenter's voice. You don't get that in a print review. I can't remember, but I'm fairly certain the Gamespot video reviews don't have a rating on them, but they closely follow the print review, so that's probably moot.

I don't care about the issue either way. If every outlet I write for dropped the scores entirely, it wouldn't bother me. Having to work within a score-based system doesn't bother me either.

jfletch

02-28-2006, 07:49 AM

The presentation is different. If you watch a review, you can get the subtle tones and inflections in the presenter's voice. You don't get that in a print review. I can't remember, but I'm fairly certain the Gamespot video reviews don't have a rating on them, but they closely follow the print review, so that's probably moot.

No... they put the rating up at the end.

Im very interested to see how this turns out and I may even subscribe to CGW. I havent read the magazine for years to be honest.

Charles

02-28-2006, 08:03 AM

For what it's worth, I just tried reading a couple of IGN reviews to test this.

The writing was bad.

You know, at every job I've worked at, it was pretty much standard operating procedure to laugh at the horrible writing in IGN reviews. We never actually went there for info, but it was always a laugh to see how bad they'd bungle even basic sentence structure.

DeepT

02-28-2006, 08:04 AM

Ratings: I'm for 'em. I think numeric scales suck because people equate them to letter grades and therefore you don't use the whole thing. 6/10 or 60% therefore becomes bad, not above average, etc.

There in lies the problem, the core problem with ratings. What you just suggested, that 60% becomes bad, not above average is, infact, wrong.

You see if 60% was above average, and we were to assume 50% was average, then we should see just as many 40% as we do 60%, and just as many 10% as we see 90%.

But we do not. We it's very unusal to see games with a rating below 50%, and in most cases, its also unusal to a game with a rating below 70%. It would be wonderful if we could really consider a 50% rating as average, but we can't.

When you look on rotten tomatoes you regularly see movies with extreemly low scores. Yet game reviews very rarely do. I think having these scores highlights the problem. I mean there are times when a reviewer trashes a game and then turns around and gives it a 64% or something. I then ask, what on gods green earth rates a 10% or a 0%?

Check these out:
Dukes Of hazzard (http://www.gamespot.com/pc/driving/dukesofhazzardrfh/review.html?sid=2637639) The review hates the game, "If you're at a garage sale in the year 2010 reading this review on a tiny monitor embedded in your eyeball, and you say to yourself, "I have no idea who these Dukes of Hazzard are, but I can't go wrong for two yuan," rest assured that you can indeed go wrong" yet, it still gets a 4.6

Outpost2 (http://www.gamespot.com/pc/strategy/outpost2divideddestiny/review.html?sid=2543568) Quote: ""If you liked Civilization and Command & Conquer, you will love Outpost 2!" reads the tag line on Sierra's web page. A more accurate boast would be: "Outpost 2: The Hurry Up and Wait game!" Simply put, Outpost 2 is one of the most tedious, boring games to grace the market in a long time, leaving you very little to do but wait and watch." Rating: 5.0

StarTrek Away team (http://www.gamespot.com/pc/strategy/startrekawayteam/review.html?sid=2699100) Quote: "It's a linear, repetitive, tedious game that ignores almost every lesson it should have learned from previous, more successful tactical combat games." Rating 5.5 -- Ok he doesn't totally despise the game, he does say some levels have an intresting design, but thats the only positive comment.

Anyway... My point: Asserting that 50% is average, is wrong. Game reviewers are loathed to give exteemly low reviews for whatever reason. Yes you can find plenty of examples, but the bell curve is proabbly centered over 80%, not 50%. Case and point. I did a search on game rankings, they have 462 pc game reviews on file. The biggest sample I could search for (to bad none of these guys let you search by score) is 200 games at once. All 200 games in the top 200 are over 80% scores out of a possible 462 games.

BobJustBob

02-28-2006, 08:08 AM

As mentioned, scores can sometimes be useful as a measure of relative quality between different titles. Like the notion of utlity, the score itself is meaningless; only the comparison of different values makes sense.

Like when PC Gamer gave Magic Carpet a 96%. At the time it was the highest-ever rating, possibly given out only once before (maybe to System Shock?). It meant that, hey, this is something special. Sure, you could also get that by actually reading the review, but in this case it was a kind of shorthand.

Of course, this relevance really only works for the really high or really low scores. For the average scores, you still have to read the review to find out if you'll like the game. Or if you're me, you have to read all good reviews regardless because reviewers give high marks to all kinds of crap (like HL2). So you're back to reviews at best needing a smaller range of scores and at worst needing no scores at all.

Come to think of it, all this has already been said. I say we all continue to shout out which side of the argument we're on until we get bored and go argue in a different thread on a different topic.

EDIT: DeepT, you can sort the rankings by worst score. The lowest PC Games on gamerankings (with at least 20 reviews/votes) is Psychotoxic, with an average score of 38%. The tenth-worst game is Contract JACK, with an average score of 51%. So ten games in and you've already eliminated over half the possible scores.

Troy S Goodfellow

02-28-2006, 08:09 AM

To me, that's just proof that there's something wrong with Gamerankings. Frankly, any site that would list GalCiv at the top is not a site I would trust.

To be fair, my hatchet job isn't in there yet. So in a month or two you might be less popular.

;)

Troy

Charles

02-28-2006, 08:14 AM

You know, what DeepT replied to Jason Cross just reinforces that I really appreciate gamespy's 5 star system. It's not very granular, each value has a very distinct meaning, and I know what the ratings mean. I know, because of how they rate, that a 3 star game may be enjoyable if I like that kind of game, that a 4 star game is probably a well done example of a genre, and that a 5 star game is really something to sit up and take notice of.

DennyA

02-28-2006, 08:29 AM

I've never seen a review on TV use a "rating," other than Eberts' "thumbs up."

<sessler>I give this post at 2... out of 5.</sessler>

Rywill

02-28-2006, 08:56 AM

Personally, I'm surprised that so many of you regard ratings as an indispensible part of the review process. You don't get that in other forms of entertainment.
Huh? Yeah you do. Most famously in movie reviews, but sometimes in other reviews as well. I just do not understand this objection. What is so offensive about rating entertainment experiences? Do you not rate movies on Netflix because you can't bear to reduce your opinion of Godfather 2 to "five stars"? Heck, every year you post your top ten movies of that year right on this board. That's essentially a rating system, isn't it? You're comparing a bunch of movies and deciding how they compare to each other, and that's it -- no real commentary (maybe one line), just "X is better than Y, which is better than Z."

I just don't get it. I don't have strong feelings either way about the rating systems, although I have a slight preference for them because they're more information. But sometimes folks seem to just have this visceral reaction -- usually hatred -- towards them. Tom, your reaction (particularly your comparison) makes me think the reasoning behind it is "We aren't serious or won't be taken seriously as long as we use ratings." Presumably because we want to give the message that games are just too complex and artistic to reduce to a numerical score? But complex things get reduced to numerical scores all the time. Movies get certain numbers of stars. Some book review sites do that too. Almost all user reviews of anything do that. Colleges get ranked. Olympic figure skaters get scores. People get graded on essay tests. And those last few are on 100-point scales!

Rob Beschizza

02-28-2006, 08:59 AM

You know, at every job I've worked at, it was pretty much standard operating procedure to laugh at the horrible writing in IGN reviews. We never actually went there for info, but it was always a laugh to see how bad they'd bungle even basic sentence structure.

IGN is an actual employer, right? So it begs the question, why are good volunteer writers not displacing whoever IGN is using?

"Because they get hired at better sites than IGN" isn't an answer! :)

Charles

02-28-2006, 09:07 AM

I think it's just because IGN as an employer is just as stupid as most of their reviewers.

Chris Woods

02-28-2006, 10:42 AM

I think you could say this of a lot of classic CRPGs. I think the term "role playing" is in some ways a legacy phrase from pen-and-paper games despite the fact that other aspects of said games (high fantasy or sci-fi setting, stats-based character management, tactical combat, etc.) are what more readily translated over to the videogame medium during the formative years of the genre.

Yeah, but I appreciate what Desslock is saying. In a post Fallout/Torment environment RPGs do need to get held up to a new standard.

Chris Woods

jfletch

02-28-2006, 10:56 AM

For what it's worth, I just tried reading a couple of IGN reviews to test this.

The writing was bad. Consider the start of the Space Rangers 2 review:

You can point to virtually every IGN review and pull out similar criticisms. But they keep doing it, they've done it for years and they just sold themselves to Murdoch for $950 million so I guess someone out there is reading it.

Tom, you say Halo 2 is overrated but you guys rated it a 9.8 in an 8-page review filled with outrageous hyperbole and teenybopper fawning.

steve

02-28-2006, 11:05 AM

I was stunned by this. If this person actually thought that KotOR was the best RPG of all time, then how many RPGs could this person have played? Or, even worse, what kind of horrific RPGs had this poor reviewer been subjected to in order to think this?
You should ask Doug Erickson, since he wrote that review. He also wrote a lot of our "history of RPGs" stuff, so I'm pretty sure he's pretty qualified.

Also, the actual quote is this: "It’s almost inarguably the best role-playing game you’ll play this year, and it’s definitely among the best Star Wars titles ever made."

Gordon Cameron

02-28-2006, 11:43 AM

Yeah, but I appreciate what Desslock is saying. In a post Fallout/Torment environment RPGs do need to get held up to a new standard.

Chris Woods

Well, PS:T is not even in my personal top 10 CRPGs,* so obviously I differ somewhat on that particular point. Yeah yeah, I'm a horrible philistine with no taste, etc., but I have been playing CRPGs for over 20 years (I think Temple of Apshai was my first) and I have some sense of the gameplay aspects of the genre that are most interesting to me. Enhancing the dialogue-tree options while de-emphasizing the presence of interesting combat encounters seems to me to have been a noble but not-altogether-successful experiment in CRPG design, though I have respect for what Black Isle were attempting. Personally, I'm more interested in the design avenue of "living world" simulation and NPC AI rather than that of increased text quantity and more dialogue tree options -- in the long run it might create more true "role playing" possibilities in a less linear and top-down way... But, that's just my preference, obviously.

What I'm mainly trying to say, though, is that I don't see a tactical-combat-exploration-stats-management game as necessarily deficient just because it doesn't measure up to someone's standard (derived mostly from PnP games) of what true role playing should be. I think this has as much to do with expectations brought to bear by the genre's legacy-imposed name, as with the actual content of the gameplay. If one likes, all "CRPGs" that don't have strong role-playing elements can be removed from the genre and put into a new genre named "Fred McPhee." I wouldn't consider them any less great games, though, and I would also consider superlative games like Ultima IV and Avernum 2 and Baldur's Gate II -- which seem to me to be very much in the mainstream of what is colloquially considered the CRPG genre -- to be mostly Fred McPhees. The future path of this genre is anyone's guess, but I don't necessarily consider the "RP" to be the only way to go in taking it in new directions.

My feelings on this have been further strengthened by playing WoW, which despite being an MMO has what I consider to be significant elements of traditional CRPG gameplay dating back to the early Ultimas (stats management, combat minigame, large explorable world, quest/task prioritization); which even in my "soloing" sessions is as much fun as any CRPG I have ever played; and which has nothing even remotely resembling "role playing" in it, at least on the servers I've played it on. So that makes it another brilliant Fred McPhee, I suppose.

*actually it might get in at number 9 or 10, now that I think about it, but I'd have to think about my list again to know for sure...

Charles

02-28-2006, 11:49 AM

You can point to virtually every IGN review and pull out similar criticisms. But they keep doing it, they've done it for years and they just sold themselves to Murdoch for $950 million so I guess someone out there is reading it.

Tom, you say Halo 2 is overrated but you guys rated it a 9.8 in an 8-page review filled with outrageous hyperbole and teenybopper fawning.

Well, see, that's the point. People *aren't* reading it. They are looking at the numbers. Developers also love IGN reviews because the numbers are absurdly high and look good on a box. A mediocre game on IGN often gets an 8.0... and to an end consumer, 8.0/10 on a box doesn't look half bad.

Gordon Cameron

02-28-2006, 11:50 AM

I was stunned by this. If this person actually thought that KotOR was the best RPG of all time, then how many RPGs could this person have played? Or, even worse, what kind of horrific RPGs had this poor reviewer been subjected to in order to think this?

Well, I've been playing CRPGs for almost a quarter century, and have played most of the Ultimas, the Fallouts, all the Infinity Engine games, Morrowind, the Gothics, several Spidweb games, and various others, and I would put KotOR in my personal top ten list, though not at the top. (Admittedly there are a couple of key gaps in my CRPG experience, including only a cursory amount of time spent playing Ultima VII, and no time playing Betrayal at Krondor or any Wizardries.) I don't think it's a totally indefensible decision. As with BG2 (still probably my favorite single player RPG) I consider the strong execution to have overcome any reservations I have about Bioware's rather linear and story-centric approach to CRPG design.

jpinard

02-28-2006, 12:25 PM

... ummm anyone know what Scorpia moved on to do?

Troy S Goodfellow

02-28-2006, 12:31 PM

IGN is an actual employer, right? So it begs the question, why are good volunteer writers not displacing whoever IGN is using?

My money is on comfort level of editors. Editors have people they know they can work with and tend to stick by them even when their work is subpar or unreadable. You see this a lot at volunteer sites, so I imagine its a factor at many professional sites as well. (Not that everyone at IGN is a bad writer...but it seems that all the other sites are a little more even in quality.)

Does IGN ask people to relocate?

Troy

Matthew Gallant

02-28-2006, 01:47 PM

Does IGN ask people to relocate?
I don't think they even ask if they can spell it.

Rob Beschizza

02-28-2006, 01:57 PM

My money is on comfort level of editors. Editors have people they know they can work with and tend to stick by them even when their work is subpar or unreadable. You see this a lot at volunteer sites, so I imagine its a factor at many professional sites as well. (Not that everyone at IGN is a bad writer...but it seems that all the other sites are a little more even in quality.)

Does IGN ask people to relocate?

Troy

That's a good point. Assuming it's true, local newspapers have the same problem: you must live locally, but that cuts out a lot of talent, and in some markets, really hurts.

I know this, because I got my first job in journalism despite not knowing the language*. I was local, however, a graduate and willing to learn.

* Slight facetiousness. I'm British, living in America. But hey, I still sneak in the occasional "U" just for a laugh and to add some colour.

TomChick

02-28-2006, 06:30 PM

in other forms of entertainment."]Huh? Yeah you do. Most famously in movie reviews, but sometimes in other reviews as well.

I'm going to have to hit you back with a 'Nuh-uh, no you don't'. :)

There are certainly some entertainment reviews that add scoring, but as I said you don't find that a numerical rating is "an indispensible part of the review process" in other forms of entertainment. That's unique to gaming.

Tom, your reaction (particularly your comparison) makes me think the reasoning behind it is "We aren't serious or won't be taken seriously as long as we use ratings."

Well, I kind of believe that. Ratings are certainly part of the problem. The numbers -- which are poorly thought out, inconsistently applied, and the subject of discussion to the exclusion of substantive critical analysis -- already have an enormous influence on the business side of the industry.

I'd love to remove that distraction from more games coverage, just like it's not present in coverage of other forms of entertainment. That's why I really admire CGW's decision.

But I don't hate ratings, particularly when I have a solid sense for how a publication uses them and what they mean. They have a place and, as you alluded, they're a great shorthand for everybody's favorite pasttime: Making Lists! I just wish they weren't so ubiquitous. They're tools, sometimes useful, sometimes poorly applied, but they should always sit in the back where they belong. Right now, the little fuckers are in the driver's seat.

-Tom

steve

02-28-2006, 07:06 PM

There are certainly some entertainment reviews that add scoring, but as I said you don't find that a numerical rating is "an indispensible part of the review process" in other forms of entertainment. That's unique to gaming.
Games are math. Movies aren't.

Numerical ratings are "an indispensible part of the review process" in most consumer magazines. Even Consumer Reports has ratings, and used to "rate" movies.

And don't think that the aggregators like Rotten Tomatoes and Metacritic aren't affecting movie reviews.

Well, I kind of believe that. Ratings are certainly part of the problem. The numbers -- which are poorly thought out, inconsistently applied, and the subject of discussion to the exclusion of substantive critical analysis -- already have an enormous influence on the business side of the industry.
Who cares? That's not your or my problem. If the industry is stupid enough to use Gamerankings in its business decision, it deserves what it gets.

That ratings are poorly thought out, etc., is a problem of those publications. Those reviews wouldn't magically get better if the ratings were removed.

I'd love to remove that distraction from more games coverage, just like it's not present in coverage of other forms of entertainment. That's why I really admire CGW's decision.
Except a huge percentage of all reviews in all entertainment is rated. The exception are some of the long-form review pieces that generally don't appear in consumer magazines. They're in things like the New Yorker, or Salon, or alt. weeklies.

But more people get their movie, music, book, and TV reviews from Entertainment Weekly, which has ratings. And it has pretty good reviews too.

Right now, the little fuckers are in the driver's seat.
But who cares? Seriously. If you're just pissed your text isn't being read, here's the reality: no one will read your piece without a rating either.

fox1

02-28-2006, 07:21 PM

I generally find that I only get any use out of numerical review scores if they're in the upper 10% or lower 30% or so, and then only as the most general cue to take note one way or the other. For anything more specific than that, well, it seems like the bulk of games vary wildly based on individual interests, and the 'why' is more important than the number. I mean, I could come up with rationale to rate, say, Bloodlines or Arcanum anywhere between a 40 and a 90% (that's on a "real" scale, not the 7-9 thing you guys go on about), based purely on the player in question. I have a hard time believing that my tastes are exceptional enough for me to be in the minority on that, either, although, hey, I don't like FPSes or RTSes, so it's open season there.

Anyway, that said, as a reader I'm more excited about changes they might make to compensate for the "loss" of the numbers, than the removal of scores itself. Even thinking of taking a look at the mag, and I haven't bought a magazine of any kind in years.

TomChick

02-28-2006, 07:35 PM

Games are math. Movies aren't.

Ouch, I couldn't disagree more. They're both entertainment, relying on stuff that goes well beyond math.

Who cares? That's not your or my problem. If the industry is stupid enough to use Gamerankings in its business decision, it deserves what it gets.

I care. I'm a gamer, first and foremost. This is my hobby. I love it. I care what happens to it, commercially as well as creatively.

If you're just pissed your text isn't being read, here's the reality: no one will read your piece without a rating either.

That's not my contention with ratings. Did you just miss the rest of the post you just replied to? :) I don't hate ratings. I just wish more places didn't have them.

-Tom

Rywill

02-28-2006, 07:56 PM

Watch me be a lawyer and play both sides of the fence.

There are certainly some entertainment reviews that add scoring, but as I said you don't find that a numerical rating is "an indispensible part of the review process" in other forms of entertainment. That's unique to gaming.
Okay, maybe I misunderstood what you were saying. I agree that ratings shouldn't be "indispensible," and in general I think they aren't and nobody would tell you that they are. But I think most people would agree that they make reviews better because they add more information to the review, and in particular add an easy reference that can tell the reader "This is about what you expected" or "This is unusual; you should read the review even if you might not normally be interested."

Well, I kind of believe that. Ratings are certainly part of the problem. The numbers -- which are poorly thought out, inconsistently applied, and the subject of discussion to the exclusion of substantive critical analysis -- already have an enormous influence on the business side of the industry.
I tend to agree with Steve's rejoinder, perhaps taken a tiny bit farther. What you are saying is that game publishers or devs or whoever are all stupid: they base their business decisions on numbers that have little connection to reality. First off, I find that sort of hard to believe. I tend to think that if the businesses are basing decisions off review scores (particularly aggregate scores), it's probably because those are a reasonable predictor of a game's sales.

I would guess the natural counter is that it's all just a self-fulfilling prophecy: stores and publishers push stuff with good numbers, which causes stuff with good numbers to sell, which positively reinforces the whole loop. But given that scores presumably come out after, probably long after, people have made commitments about what games they will develop, stock, etc., how is this possible?

More importantly, taking Steve's analysis another step, how is removing the numbers going to help? If the businesses are all idiots, won't they just find some other idiot shorthand metric to base their decisions on? It isn't like they will suddenly start reading everyone's reviews. There's no time. They'll look at stuff like how many times you got a NOT GUILTY!! verdict, or Editor's Choice, or whatever. Right? This seems like a change of form but not substance.

I'd love to remove that distraction from more games coverage, just like it's not present in coverage of other forms of entertainment.
I call BS. Maybe we read different publications, but most movie reviewers use stars. Most book reviews don't. But lots of other things do. Even complex, not-easily-reduced-to-numbers stuff like wine or restaurants get rated with numbers (on a 100 point scale for wine) or stars. So do many music reviews. I just don't think using a numerical score (or star score) suddenly makes you less serious or distracts the industry from what is important.

They're tools, sometimes useful, sometimes poorly applied, but they should always sit in the back where they belong. Right now, the little fuckers are in the driver's seat.
I can certainly agree with the first part, and you would know better than I would on the second part, so I guess I actually mostly agree with you.

Games are math. Movies aren't.
What in the high holy happiness of Heaven are you talking about? Games are math? I am going to assume here that you mean they're math in the sense that you think most gameplay, when you get right down to it, involves making some series of calculations to win. That is the best possible gloss I can put on what you are getting at. And I could not possibly disagree more strongly. Like, you could have said "Games are math and Hitler," and I would disagree the same amount, because the meter is already pegged.

I'm not sure how else to respond because this is such a crazed statement. I mean, some games are basically exercises in math. There's a level of Exit! for the PSP that I just completed that was basically math. But most games are not. Shooters aren't. Sims aren't. Adventure games certainly aren't. Some games, like RPGs, have an element of math to them, but there's a lot more to the game than math. Anyway, at this point I feel like I must be misunderstanding what you said, because nobody could say what I thought you meant but still have enough sanity to feed themselves. So maybe you can explain what you meant better. Wow. Sorry for the vitriol, but if it makes you feel better think of half of it as Tom's that he can't say because he works for you.

That ratings are poorly thought out, etc., is a problem of those publications. Those reviews wouldn't magically get better if the ratings were removed.
I pretty much agree with you here. Although an argument could be made that if magazines didn't have the rating crutch they would be forced to express themselves better in writing. Or that if magazines took all the time they spend deciding on, and then defending, the ratings, they could use that time to improve their writing.

But who cares? Seriously. If you're just pissed your text isn't being read, here's the reality: no one will read your piece without a rating either.
You're right, but I am guessing that's not why Tom cares.

Kitsune

02-28-2006, 08:08 PM

What I'm mainly trying to say, though, is that I don't see a tactical-combat-exploration-stats-management game as necessarily deficient just because it doesn't measure up to someone's standard (derived mostly from PnP games) of what true role playing should be. I think this has as much to do with expectations brought to bear by the genre's legacy-imposed name, as with the actual content of the gameplay...The future path of this genre is anyone's guess, but I don't necessarily consider the "RP" to be the only way to go in taking it in new directions.

If you want to get really snarky, you could always argue that it isn't roleplaying in the sense of psychological roleplaying where people take on roles not usually theres to understand other people's perspectives, or educational/occupational roleplaying as training where you take on the role of the customer/student/trainee/mentor, etc. and pretend so you can strengthen your skills. :P

In any case, I agree with your stance, though my tastes are much wider. I'm glad I'm getting the dialogue heavy types of roleplaying (e.g. Hermina & Culus, Tokyo Majin, Giftpia), the combat heavy types (e.g. Dragon Quarter, Divine Divinity, Phantom Brave), the exploration heavy types (e.g. DQVIII, Morrowind, Golden Sun: The Lost Age), the Take a Stand! types (e.g. Knights of the Old Republic, SMT: Nocturne, Gothic) and the linear types (e.g. Tales of... games, FFX, Radiata Stories). In each game, the emphasis is on making considered, important choices in developing and growing characters in a reasonable facsimile of a working world setting is what I enjoy. They're all RPGs, whether its the roleplaying of conversation, battle, discovery, ideas and argument, or the personal threading into a narrative.

Language is elastic. People have a choice. They can change with it, or they can remain the same, but language, one way or another will go on without them as the things they describe evolve.

There isn't any good reason why tactical/combat/exploration elements developed in a matter to promote roleplaying can't be considered roleplaying. Its like saying that Law & Order isn't drama because it isn't acted out on a stage with acts and an invocation to the muse and it lacks a chorus. Appealing to the origins of things only works so far.

There's also no good reason why roleplaying can't involve jumping into predefined or set role and working within its boundaries. The idea that it has to involve creating a character is one that should die with a generation that will die with it, superseded by a much younger, more enlightened generation. :P

You mean have different people have different preferences on how they'd like to roleplay? Shokku! Defining one preference over the other is simple elitism.

No Freddy McPhees needed.

-Kitsune

brainfromarous

02-28-2006, 08:36 PM

... ummm anyone know what Scorpia moved on to do?

I do.

Scorpia parted ways with CGW after George "F-word" Jones told her she was no longer welcome to submit reviews. She did not quit; she was fired. (Following that, she was ignored completely in CGW's special 200th issue - an appalling, disgraceful slight.) After this, she had her own website for awhile - Scorpia's Domain - run by an ex-GENIE GamesRT associate, Dale R.

With the closing of the Domain, she pretty much dropped out of sight. She's made a few appearances over at Randy Slugaski's JUST ADVENTURE, which has republished some of her older reviews. Finally, she used to host a weekly MIRC get-together for RPG oldies but that too may be long gone.

brainfromarous

02-28-2006, 08:51 PM

Dropping the stars would mean more if CGW's reviews still had some of their old substance and length, as opposed to the "Ooh, pictures!" approach they've taken. What would the absence of the Star Box mean, really? One or two additional sentences?

I propose that CGW recover the two pages wasted on "Gladstoned" and "$100" to make room for, you know, some writing about computer games! ;)

If Jeff Green is reading this... Jeff, I've a world of respect for you, but CGW is a shadow of its former self. Please bring it back to the gold standard of games mags recalled by oldies like myself.

(Whatever happens, it's still better than PC GAMER, a magazine so compromised and untrustworthy that at this point, auctioning off their Editor's Choice awards to game publishers would yield no perceptible difference in the review copy.)

brainfromarous

02-28-2006, 09:17 PM

I don't want to sift thru 3 pages of shit only to find out the game reviewer hated the game- oh and 50 other game reviewers also thought it sucked or they all loved it. I don't see games as "art" either. While some games may contain some hand drawn art, when Yahtzee, Monopoly, Bingo, and other games become art then I will concede. Unless its by a critic I have heard of, if I can't get a fast rating up front then I won't bother with the review. Gamerankings is the bomb :)

So the bulk of the review, in which the game's virtues and failings are considered at length, wherein the reviewer has a chance to tell us what playing the game is like and discuss any technical issues and bugs - this is all "shit?"

I'm curious: If that's your attitude, why even read reviews at all? Why bother with ANY critics - even those familiar to you? By all means, stick with Gamerankings.

brainfromarous

02-28-2006, 09:35 PM

Mention of the "Postal 2" review, I laughed at that review.Figured that this "Robert Coffey" person was some whining shit who was terribly and personally offended by the game and therefore gave it the infamous 0 star rating.

Actually, it was more like Coffey telling Running with Scissors not to crap on a plate and serve it as chocolate cake.

CGW has a honorable tradition of telling RwS that "attitude" and "edginess" don't excuse bad work. Cirulis did it with Postal, Coffey with the sequel.

As a result I tend to take PC Gamer's and Computer Games' scores with a salt lick.

Oh, come on. CGM is THE NEW CRITERION compared to something like PC GAMER.

Troy S Goodfellow

03-01-2006, 05:07 AM

I propose that CGW recover the two pages wasted on "Gladstoned" and "$100" to make room for, you know, some writing about computer games! ;)

I actually like $100. It's like a geek shopping guide for people like me who only really buy into the gaming side of geekdom.

They could give Bruce more room for his column, but those are just my prejudices shining through.

Troy

Wheelkick

03-01-2006, 05:43 AM

I do.

Scorpia parted ways with CGW after George "F-word" Jones told her she was no longer welcome to submit reviews. She did not quit; she was fired. (Following that, she was ignored completely in CGW's special 200th issue - an appalling, disgraceful slight.) After this, she had her own website for awhile - Scorpia's Domain - run by an ex-GENIE GamesRT associate, Dale R.

With the closing of the Domain, she pretty much dropped out of sight. She's made a few appearances over at Randy Slugaski's JUST ADVENTURE, which has republished some of her older reviews. Finally, she used to host a weekly MIRC get-together for RPG oldies but that too may be long gone.

Scorpias domain:
http://www.scorpia.com/public/default.htm

mutt

03-01-2006, 05:48 AM

Measured, no. But subjectively ranked, yes. Which is what every reviewer and judge in the world does.

Frankly, I don't give a flying fuck what every reviewer in the world does. What most people do in the world today is wrong, stupid, immoral and/or evil. I'm interested in what the truth is, and the truth is that quantitative ratings for games, films, plays, books, poems or whatever are essentially worthless and meaningless. Regardless of what modern idiots do, no one in the history of the world ever thought to review things in this way before. I'm not exactly an expert on classical music history, but I seriously doubt contemporaries of Beethoven (for example) considered their job to consist of giving the Seventh Symphony a four out of five fucking stars. I imagine they rather listened to the music and commented on both its technical and aesthetic qualities based on their view of what music is.

I'll grant you that ratings, as a practical matter, try to serve as a subjective comparison according to some unspoken, amorphous standard, but that's simply an additional problem with them, not a virtue, in my view.

steve

03-01-2006, 08:08 AM

Ouch, I couldn't disagree more. They're both entertainment, relying on stuff that goes well beyond math.
Of course I agree, but I think it's why ratings are so prominent.

Most of the poeople doing reviews and making games are highly technical, were probably skilled in math/science/CS, and probably believe you can distill the essence of a game into a formula. Math solves almost every problem in a game; why can't it "solve" quality?

I care. I'm a gamer, first and foremost. This is my hobby. I love it. I care what happens to it, commercially as well as creatively.
It's silly to believe that ratings, of all things, have any impact on this. Bottom line: the decisions being made are based on what sells. Historically, sales haven't been tied in any meaningful way to ratings; it's more about marketing and luck. Maybe that's changed, but I'm not seeing it.

The real problem with ratings is Gamerankings.com. If it didn't exist, there wouldn't be a ratings issue.

steve

03-01-2006, 08:22 AM

What in the high holy happiness of Heaven are you talking about? Games are math?
All games are math, but that wasn't really the point. As I said to Tom, it's more the idea that you have all of these technical people making games and writing reviews, and most of them do know a lot about math, and science, and they like stats and numbers.

Therefore, it's perfectly reasonable to most of them to want to turn a game's quality into a formula and number. It's a lot easier to use that as a baseline measurement than text.

(And when you consider what MMO players do to their games, calculating DPS and the like to perfectly optimize combat and leveling, and when Civ players are running around looking at every number on the map and calculating odds for victory or for what that improvement will do to their overall production... yeah, games are math. A lot of math. For a lot of players. I personally never play games this way.)

Although an argument could be made that if magazines didn't have the rating crutch they would be forced to express themselves better in writing. Or that if magazines took all the time they spend deciding on, and then defending, the ratings, they could use that time to improve their writing.
Possibly, but I hope people aren't spending that much time on the rating. Seriously, that's just wacky.

I can only speak for myself, but I probably spend 30 seconds deciding most ratings. I might mull on one for a bit longer if I'm on the fence, or I might reconsider some of the text if I feel it's a "4-star" game and the text comes across as too negative. (Or alternately, re-consider the rating if the text is spot-on.)

As an aside, I'm guessing that without ratings and more space, reviews will risk appearing more negative because you'll cover even more of the flaws than you might otherwise have done.

And maybe no one pays attention to our reviews, but "defending ratings" is a total non-issue. I generally tell people I won't discuss the rating but will talk about points raised in the text.

Charles

03-01-2006, 09:00 AM

Games are math. Movies aren't.

This line makes me sad.

It's also provably false. If games were math, any game that was technically competent would be subjectively fun and enjoyable. If games were math, all developers would have to do is run the numbers to make a game that's a blockbuster.

Thankfully, it doesn't work like that.

steve

03-01-2006, 09:32 AM

It's also provably false. If games were math, any game that was technically competent would be subjectively fun and enjoyable.
I didn't realize how unclear that would be. It has nothing to do with what you're describing; it has to do with how a lot of people view games as a much more technical experience, which is why "measuring" them with ratings make more sense to more people than expressing the experience with verbiage.

If games were math, all developers would have to do is run the numbers to make a game that's a blockbuster. Thankfully, it doesn't work like that.
It does for publishers.

Bill Dungsroman

03-01-2006, 09:36 AM

Of course I agree, but I think it's why ratings are so prominent.

Most of the poeople doing reviews and making games are highly technical, were probably skilled in math/science/CS, and probably believe you can distill the essence of a game into a formula. Math solves almost every problem in a game; why can't it "solve" quality?

It's silly to believe that ratings, of all things, have any impact on this. Bottom line: the decisions being made are based on what sells. Historically, sales haven't been tied in any meaningful way to ratings; it's more about marketing and luck. Maybe that's changed, but I'm not seeing it.

The real problem with ratings is Gamerankings.com. If it didn't exist, there wouldn't be a ratings issue.

Yeah, but it does, so there is.

If scores have become an integral part of the game development cycle, such that the removal of them would significantly damage, prohibit, or otherwise impair the ability of games to be made, then removing them would do a greater disservice to our hobby than the subjective moral/professional ambiguity keeping them does. IMO.

Brad Wardell

03-01-2006, 09:42 AM

It's silly to believe that ratings, of all things, have any impact on this. Bottom line: the decisions being made are based on what sells. Historically, sales haven't been tied in any meaningful way to ratings; it's more about marketing and luck. Maybe that's changed, but I'm not seeing it.

Absolutely untrue.

First, if GameRankings didn't exist, there'd be something else (Metacritic or whatever).

Secondly, just yesterday, restocking levels in a major chain involved the GameRankings score. So I know for a fact that the ratings do matter.

They may not matter as much to an EA or Microsoft, but you can bet they matter to smaller publishers who are struggling to talk the retailer into having 4 units per store instead of 2 units per store.

Just so I'm very clear on this:
A small publisher's title gets an initial sell-in of X. Based on sales, the retailers have a formula that determines how quickly they try to get back to that initial sell-in stock level.

Getting significant sell-in for a small publisher is very hard. Therefore, the only chance a smaller publisher has to sell large volumes of games is to convince retailers to increase their stock level ABOVE the initial sell-in level - combined with very high sales of course.

Many of the retail buyers look at Gamerankings (or at the very least a handful of sites and magazines individually). IF the reviews are VERY good, stocking levels can get increased as long as sales are also strong. Quality X Marketing X Distribution = Total Sales.

Charles

03-01-2006, 09:44 AM

It does for publishers.

It's how publishers think, I'll give you that, but it's not how it works. Not in the grand scheme of things. It's a short term strategy that kind of works. But it's never that strategy that leads to a gigantic breakout hit which sells twenty million copies. There's no math for that.

Also, the math doesn't always work, even for publishers.

Wheelkick

03-01-2006, 09:46 AM

The real problem with ratings is Gamerankings.com. If it didn't exist, there wouldn't be a ratings issue.

Logic flawed.

But Brad has already covered this in his post, so I'll just say... eh... Go Brad!

Chris Woods

03-01-2006, 09:53 AM

Games are stocked based on some way of measuring their projected popularity -- specifically how likely they are to be sold. If ratings were gone you guys would just have some other pinata to be swinging at with blindfolds on, and likely it would be marketing budget.

If popularity became a function consisting totally of marketing, people like Brad are fucked. As it stands, at least under a ratings system small developers have a chance of getting high enough scores to let them compete with high dollar games.

I really think you "lets abolish all ratings" people are assaulting something that helps the small guy, which seems really self-destructive to me.

Chris Woods

mouselock

03-01-2006, 09:57 AM

What seems to really be getting overlooked IMO is the implicit assumption that "lack of ratings means the public will synthesize an opinion from the review now". This is only true for people who were likely to do this anyway. If you take away the ranking aggregators or scores in general, the people who currently use them will simply turn to option #2: Internet rumors and word of mouth.

I don't know about other folks, but I'd much rather see people making ill-informed decisions based on reviewer rankings, as imprecise and horrible as they are, than a shift back toward "I bought this game because the people who liked it yelled the loudest on the message boards." which is often what things were like before the meta-aggregators. The people who tried to make informed decisions, well, they weren't unduly swayed by the scores anyway. But for every person I've seen sitting in an EB Games arguing "Yuh huh! That game is better because it got a 97% and the one you like only got a 92% so suck it!", the lack of a rating means they'll just fall back on even sketchier information. ("Yuh huh.. everyone on Qt3 says that's a good game. All those dipshits on Gone Gold are the only ones who say it's bad. So it's clearly better.")

I'm not sure how the consumer wins by throwing out a dubious rating system in favor of an even more dubious fallback. (Qt3 is probably a reasonably poor example to use above, feel free to substitute GA, OO, or the old console game newsgroups wherein Vic Ireland would touch off massive fanboy flamewars for the general idea of what I'm saying.)

Dave Long

03-01-2006, 10:27 AM

I think Brad's comments about review scores' effects on restocking prove beyond a shadow of a doubt Steve's assertion that games are all about math. Sure, there's a small, vocal group of people that are all about the text, but most folks really do treat these things like mechanical constructs with a right or wrong score based on those mechanical parts. I wrote about this in my column at GamerDad over a year ago IIRC.

Anyway, Brad needs scores because that's the math his retailers/publishers/money lenders/etc. are using to come up with their stocking math. So in this specific case, it's truly a math thing from the word go.

That said, it's probably the WRONG thing for those retailers/publishers/money lenders/etc. to be using as a major criteria for buy in simply because a game can score very highly and not appeal to anyone at all and vice versa.

jfletch

03-01-2006, 10:42 AM

If there were no ratings on any videogame reviews whatsoever, you could still have a GR-like system namely like Rottentomatoes (yea lots of movie reviewers use scores but a lot dont, and RT has them both). You dont need scores necessarilly and you absolutely dont need scores as they exist at the present.

I remember reading that Warner Brothers now requires a 70% or something GR score from all games that use their licenses.

Charles

03-01-2006, 10:57 AM

Stocking games becomes moot when people move to digital delivery. Honestly, I've never understood why the games industry lets themselves be manhandled by the retail sector in any case. Making game design decisions based on whether or not Walmart will stock a game, for instance. Screeeeeeeeeeeeew that.

It's not the retail sector that should be deciding these things. Publishers should be saying "Look, stock this, or we just won't give you stock anymore." and then play companies off against each other. If EB doesn't get any stock of game X because they won't give space to game Y, and then Bestbuy gets game X and Y, and then X sells well and Y sells okay...

Well, a few instances of that should whip the retail sector in to shape.

Anyway, my point is: rah rah digital delivery! the sooner the better!

Sidd_Budd

03-01-2006, 11:29 AM

I think Brad's comments about review scores' effects on restocking prove beyond a shadow of a doubt Steve's assertion that games are all about math.
...
Anyway, Brad needs scores because that's the math his retailers/publishers/money lenders/etc. are using to come up with their stocking math. So in this specific case, it's truly a math thing from the word go.
I'd wager I've got a more rigorous math and stats background than 95% of Qt3ers, and I keep a database of game review scores as a flipping hobby, and even I think the idea that games are math is ridiculous. By that criteria, wine, restaurants, movies, and figure skaters are also math, since most reviews and judgments of these entities involve numerical ratings.

Using a formula to gauge restocking levels just means that math leads to more accurate inventories than using the pretty pictures on the box as guide to restocking levels. It doesn't mean that the item being restocked, by extension, is about math. Most graduate schools, law schools, and medical schools use a mathematical formula (with weightings for GRE scores and undergrad GPA) as a significant component in their decisions of who to accept each year. Does this mean that everyone with a graduate or professional degree is primarily about math?

Right now, it seems the financial people holding the purse-strings pay attention to both ratings and sales. If reviewers radically shift to a "no rating" system, aggregate critical opinion will be harder to measure. There is little chance that the financial folks are going to take the time to sift through the New Gaming Journalism; they'll simply stop paying as much attention to ratings, and focus more on sales numbes. In my opinion, this will reduce game quality and even further limit innovation in the future.

brainfromarous

03-01-2006, 11:40 AM

Frankly, I don't give a flying fuck what every reviewer in the world does. What most people do in the world today is wrong, stupid, immoral and/or evil. I'm interested in what the truth is, and the truth is that quantitative ratings for games, films, plays, books, poems or whatever are essentially worthless and meaningless. Regardless of what modern idiots do, no one in the history of the world ever thought to review things in this way before. I'm not exactly an expert on classical music history, but I seriously doubt contemporaries of Beethoven (for example) considered their job to consist of giving the Seventh Symphony a four out of five fucking stars. I imagine they rather listened to the music and commented on both its technical and aesthetic qualities based on their view of what music is.

I'll grant you that ratings, as a practical matter, try to serve as a subjective comparison according to some unspoken, amorphous standard, but that's simply an additional problem with them, not a virtue, in my view.

Bravo.

Troy S Goodfellow

03-01-2006, 11:42 AM

I keep a database of game review scores as a flipping hobby

I've tried this, but just can't be bothered.

I'm more interested in the reviewer than the site/publication involved and if there are reviewers who don't "get" certain types of games. Do you track this stuff, too? Because that would instantly make you more interesting than Gamerankings. And I can't imagine you would simply duplicate an online source.

Troy

Chris Nahr

03-01-2006, 11:47 AM

Right now, it seems the financial people holding the purse-strings pay attention to both ratings and sales. If reviewers radically shift to a "no rating" system, aggregate critical opinion will be harder to measure. There is little chance that the financial folks are going to take the time to sift through the New Gaming Journalism; they'll simply stop paying as much attention to ratings, and focus more on sales numbes. In my opinion, this will reduce game quality and even further limit innovation in the future.

I agree. I completely don't understand why removing ratings is supposed to have any beneficial aspect whatsoever. Simplifying ratings down to maybe 3-5 levels, okay. But removing them altogether? What's the point?

Sidd_Budd

03-01-2006, 02:09 PM

I'm more interested in the reviewer than the site/publication involved and if there are reviewers who don't "get" certain types of games. Do you track this stuff, too? Because that would instantly make you more interesting than Gamerankings. And I can't imagine you would simply duplicate an online source.
There's a couple differences between me and GameRankings. Most notably, I'm smaller-scale -- since I input all my numbers by hand, I only track about 10 sites & magazines. I do still retain scores from a few big sites that have ceased operation (e.g., Gamecenter, Avault), so I think I have older scores than GameRankings; I've got a few reviews from 1990.

I don't track reviewers; just scores. However, I standardize all the ratings to equalize differences in a site's average review score (for example, a 7.0 from GameSpot generally indicates a higher quality game than a 7.0 from IGN) or in scoring systems (stars vs. percentages). This is the central failing of Gamerankings; Metacritic says they standardize, but don't give a lot of details about it.

I also apply corrections for extreme scores (i.e., if 8 sites rave about a game, and one slams it, I toss the low one) and site bias by genre (i.e., some sites tend to be more lenient in scoring action games, relative to other genres).

Overall, I like to think I'm more interesting than Gamerankings because of that thing I can do with my tongue, rather than the size of my spreadsheet.