More grist for the "review scores don't matter" mill

[zabuni]

link (http://www.next-gen.biz/index.php?option=com_content&task=view&id=3837&Itemid=2)

Good news: Review scores don't matter.

Bad news: Being part of an established brand, and releasing near Christmas does.


The report found that, “After going through multiple scenarios, we believe a game rating, in most cases, is not a reliable tool for predicting game sales. There are isolated examples of strong correlation, but they are just that – isolated. We believe a naked game rating without context is largely useless. The more specific the comparison (controlling for as many variables as possible), the more likely a statistically significant correlation may exist. However, the more specific the sample, the less useful the hypothesis becomes.”



The report goes on, “The notion that game ratings might have very little to do with game sales touches a nerve. And, no wonder – for whatever reason, many investors have learned to rely on them. Game ratings can affect developer compensation and are the raison d’être of an industry in which anyone with a game console and a domain name can throw a review onto a web site.”

[Aszurom]

Well, maybe EA will start its own magazine and rate its own shit?

[Troy S Goodfellow]

Without reading the full report, it's not clear what statistical methods they are using or how significant the results are. 1200 games is certainly a reliable sample size.

But my general reaction is "Duh". Review scores are only one thing that some people use and is a very poor linear predictor of sales, especially if you use an aggregate site like Metacritic which may be drawing from very different sources for a single aggregate rating. Some of those 1200 games might have four or five reviews, some might have two dozen.

The question that I have is whether good reviews can help a marginal game - whether the relationship is stronger for certain types of games that don't crack the top twenty.

EDIT: But I wonder how relevant this study really is? The margin between a very good game and a great game is pretty thin, ratings wise. Are bad games selling well? Not if the charts are any indication. Usually only one or two lemons in the top 20 - a better average than box office receipts. If they are simply saying that a game that scores 95 is not likely to outsell a game that scores 85, then I don't think I've really learned anything.

Troy

[Justin Fletcher]

I'm sure this will come as a huge shock to Tim Schafer.

[olaf]

I have no study but it sure would seem like, all else being equal, good review scores couldnt do anything but raise sales. I am sure its not the most important factor in sales but I would have to see some some detailed explanation and data to convince me review scores were meaningless.

[TomChick]

My understanding is that good reviews can give you leverage with retailers. I imagine they also count during internal decisions like which projects to fund next, which developer to work with, etc.

I think that's why Gamerankings is often such a big deal. It's not that ratings are supposed to be a determinant of sales.

-Tom

[JoshV]

Exhibit A: Fantasic Four
Gamerankings rating for ps2: 63%
Num Copies Sold: Over a million, its been re-run as one of the Greatest Hits series.

[steve]

Movie reviews have little impact on box office too, except at the extremes (even then, did people not go see some movie because of bad word-of-mouth or bad reviews?).

I doubt music reviews make much of a difference either. Many of the biggest selling acts get terrible reviews.

[Thrrrpptt!]

There are isolated examples of strong correlation, but they are just that – isolated.
I haven't read the full report (is it available anywhere?) but that sentence makes no sense. You can't have strong, statistically significant correlation with just a few isolated cases. You can't, in other words, look at five games with good reviews and strong sales and calculate a significant correlation. Not in the statistical/scientific sense.

I'd also like to know more about the methodology, because there are a lot of kooky ways you could arrive at their conclusion. If they use partial regression analysis to remove variance from other factors strongly related to review scores --say the QUALITY of a game-- then duh. Review scores don't matter if you hold quality constant. Probably because average review scores, across reviews and across games, are a fair proxy for game quality.

[Jason Cross]

I think review scores (and more relevantly - general consensus of game quality by players) is a factor in future sales of a franchise, not so much the game it's a review of.

If Turbo Hyper Kill 1 sells well but is "meh" quality, it impacts sales of Turbo Hyper Kill 2.

Killzone may be suffering this effect. If it really was the Halo-killer it was hyped to be, and the review scores were fantastic, there would probably be big preorders on the PSP Killzone title already. Right now, the game is best known for being a shameless prerendered trailer for the PS3.

[MattKeil]

Killzone may be suffering this effect. If it really was the Halo-killer it was hyped to be, and the review scores were fantastic, there would probably be big preorders on the PSP Killzone title already.

Or if it were on the DS. HEYO! (Okay, actually the PSP Killzone is pretty cool.)

This study simply confirms the obvious. Anyone who played Enter the Matrix knew already. Or 50 Cent: Bulletproof. Or any Dragonball Z game. Or Shadow the Hedgehog. Or Beyond Good & Evil. Or Psychonauts. Or Oddworld: Stranger's Wrath.

It's the last three that are far more tragic statistics than the rest of the list.

[Gary Whitta]

There's a pretty lengthy and spirited discussion about this on the new PC Gamer podcast which goes live later tonight if anyone cares to tune in.

[Mark Crump]

There's a pretty lengthy and spirited discussion about this on the new PC Gamer podcast which goes live later tonight if anyone cares to tune in.

Jesus, Gary. You guys can't have opening minutes like this week's show. Between seeing that guys man-berries on the Vision camera, and Vede's recollection of rooming with you, I've got way too many horrid images burning in my brain for one day.

[Rob_Merritt]

There's a pretty lengthy and spirited discussion about this on the new PC Gamer podcast which goes live later tonight if anyone cares to tune in.

That was the best discussion on the subject I've heard yet. Greg seemed almost offended. It was great!

Oh and at last Coconut Monkey was brought up. I still have some of those original demo discs. Probably a better use of FMV than most games at the time.

[Geo]

With console games just about every crappy game (or good one) that sells for $19.99 or less sells a million or more cause people seem desperate to buy something affordable rather than rent everything. I seem to remember a really awful port of Spec-Ops to PS2 that got ratings of about 27/100 or something most places, and yet it remained on the PS2 best sellers list for about a year because of its $19.99 price (at the time there wasn't the glut of $19.99 Greatest Hits games there is now).

I don't know that review scores mean much to gamers unless they're so spectacularly good (see the current Company of Heroes) as to attract them to a genre they're not usually interested in; or so bad (a lot of $20 shooters get 18%, 19% ratings in PC Gamer) that they stay away.

I guess the point of the study is that devs should strive to make the worst game possible and then price it under $20. That used to be called "the Deer Hunter Strategy." ;)

[Lunch of Kong]

I like the 1 to 10 scale, with the option to use an 11 if a game actually is perfect in every way.

[Jason Becker]

Just listened to the Pdocast. Totally agree with your reasonings Whitta. I thought you did a really good job pretty much calling out review scores and specifically PC Gamers frankly pointless percentage format. To think that people sat around and spent time deciding wether a game should be a 73% or 75% is just funny, and sort of absurd at the same time.

[Brad Wardell]

Review scores matter a LOT. I may not like that. Others may not like it. But they matter.

Galactic Civilizations II's high reviews encouraged retailers to INCREASE their initial stocking levels AFTER release (which as most here know, is the opposite after release). That means a 7 digit difference in revenue which means more games.

It may not affect huge games as much since they already have massive distribution. But for indies it matters. That doesn't mean indies should get cut a break. If their game is $50 or whatever, then it should be compared to other current $50 games. But it doesn't change that the review scores matter.

[Jason Becker]

Tell Tim Schafer that.

[Unicorn McGriddle]

I like the 1 to 10 scale, with the option to use an 11 if a game actually is perfect in every way.

Maybe with the 11 thing in place, people would actually give scores higher than 9.0.

[Brad Wardell]

Tell Tim Schafer that.

I'm not sure he'd disagree. Keep in mind, I'm talking about smaller titles.

On bigger titles, distribution is king. A Microsoft or an EA can get massive stocking levels for their games no matter what their reviews are. But a "Starduck" (or other second tier sized publishres) has to fight harder for that shelf space. Strong reviews can make the difference betwen them stocking 4 units per store and 12 units per store (which still pales to what a Take 2 or EA can do of course but to a smaller publisher is a big deal).

[Rob_Merritt]

Tell Tim Schafer that.

I think this probably needs to be explored a bit. I'm fairly sure that no matter what, Psychonauts didn't sell a lot but is it as bad as everyone says it is? At this date, how well has Psychonauts sold? The last numbers "the internets" has places US sales around 86,000 for the three platforms it was on in October 05. I can't tell if 86k is the estimated 80% that NPD reports or if its the estimation of the total sales based on a lower NPD number so we'll just stick with 86k. Since then, the game is still on the store shelves, its getting a relaunch on steam, and it was launched internationally last feburary. Both Majesco and Double Fine are still in business and working on other games.

Is it still the case that Psychonauts sold badly and if in the long run and in international markets it did ok, how much did the review scores help it?

[mutt]

Maybe with the 11 thing in place, people would actually give scores higher than 9.0.
PC Gamer US would still have to escalate to 12 and beyond for those games that were just a little more perfect than the last perfect one.

[Charles]

My understanding is that good reviews can give you leverage with retailers. I imagine they also count during internal decisions like which projects to fund next, which developer to work with, etc.

I think that's why Gamerankings is often such a big deal. It's not that ratings are supposed to be a determinant of sales.

-Tom

Actually, it's usually the good previews that are used with retails. That and hype meters. Top 10 most popular games on gamespot, etc. Problem is that by the time reviews come out, retailers have already decided on how many units they are going to buy, so the retailers have to be convinced before the game is done.

[Inactiviste]

Is it still the case that Psychonauts sold badly and if in the long run and in international markets it did ok, how much did the review scores help it?

Psychonauts got rave reviews in Europe too. Maybe because PAL reviewers base their reviews on reviews from other territories... Anyway, I can understand why Psychonauts sold badly, because it does not look that entertaining when you look at the box. And I don't even think it's that good a game. Eventough it was very quickly available at a bargain price, I still feel it was much overhyped (which is just my humble opinion, I don't mean to derail the thread), and can't blame the market on this one.

[Alex Hutchinson]

Interesting things I've seen in other studies is that a review score over 80% has quite a large impact but that beyond that it's meaningless. In other words, a 90%+ game will sell the same as an 80% game if they have the same marketing and distribution.

But there are so many factors:

Is the core idea actually appealing? I would say for Psychonauts, even though it's a great game, it sounds on first blush like a quirky kids game, and although it had brilliantly innovative level designs and good writing, its core mechanics were merely decent.

Is the genre interesting or relevant? Again, for Psychonauts, the cutesy platformer has been dying for years. The odd big brand still sells (Mario, Sonic) or the well marketed cheaper title (Ty) but most fall flat.

Is your distributer and publisher large enough to talk to Best Buy and convince them that they're going to back up a big pre-order with advertizing and support? Majesco just isn't and they realized that and pulled out of high end games.

A.

[RichardC]

Psychonauts got rave reviews in Europe too. Maybe because PAL reviewers base their reviews on reviews from other territories...

Excuse me, what? We got copies of Psychonauts at the same time as the US reviewers - they just decided not to release the damn game on the shelves for a whole year afterwards.

[Bill Dungsroman]

Psychonauts got rave reviews in Europe too. Maybe because PAL reviewers base their reviews on reviews from other territories... Anyway, I can understand why Psychonauts sold badly, because it does not look that entertaining when you look at the box. And I don't even think it's that good a game. Eventough it was very quickly available at a bargain price, I still feel it was much overhyped (which is just my humble opinion, I don't mean to derail the thread), and can't blame the market on this one.
Sadly, it looked a Hell of a lot like a kid's game from the box.

[Inactiviste]

Excuse me, what? We got copies of Psychonauts at the same time as the US reviewers - they just decided not to release the damn game on the shelves for a whole year afterwards.

I was just trying to crack a joke, I don't mean every game journos in Europe are lazy copycats... Nevermind !

[Elton]

Is the core idea actually appealing? I would say for Psychonauts, even though it's a great game, it sounds on first blush like a quirky kids game, and although it had brilliantly innovative level designs and good writing, its core mechanics were merely decent.

Yeah. I thought the art direction and style and humor were top-notch, the gameplay not too memorable. I would have never considered buying the game except that multiple reviewers were touting it as a brilliant little sleeper that was getting overlooked, so I had to get it. A bit oversold IMO, but could be that I'm just not very excited about platformers.

Now if the same game had been called something like "Baby Spiderman" and the enemies were all little Green Goblins, then it would have sold a million copies. Psychonauts' originality was great but it needed a more obvious hook. Maybe not so obvious as "Lego Starwars", but something. Or it is just nigh-impossible for a game to sell well without Orcs, Nazis, aliens or zombies?

[Inactiviste]

Or it is just nigh-impossible for a game to sell well without Orcs, Nazis, aliens or zombies?

I suppose you could patent a next-gen game with undead Orc nazis from outter space.

I love platformers, and Psychonauts definitely was not a great platformer. It had good enough ideas, but the execution was clumsy at best, at least on a gameplay perspective, imho.