Kotaku calls out Quarter to Three for aberrant review scores

, | Games


I was fortunate enough to be part of Jason Schreier’s thoughtfully written and incendiarily titled article on Kotaku, Metacritic Matters: How Review Scores Hurt Videogames. I enjoyed Schreier’s article for the diversity of sources he used, and I’m grateful that he invited me to participate. But I can’t help but wonder that he introduced me as “well-known for aberrant scores”. That’s far more of a statement about the state of videogame criticism than a statement about me. As a reviewer, I believe strongly in two things and I don’t feel either of these things should be “aberrant”: 1) I believe in using the entire range of a ratings system, and 2) I believe in rating games based on my experience with them rather than pretending I have some objective insight into their level of quality. We’re talking about entertainment here, not toasters, cell phones, or automobiles.

Schreier concludes that the system — Metacritic aggregating someone who uses the entire range with someone who doesn’t use the entire range — doesn’t work. I couldn’t disagree more. I would argue that what doesn’t work and what’s hurting videogames is how many reviewers pretend ridiculously towards objective insight measured on a bell curve that spots a game 60 points for just showing up, but deprives them of 100 points for not being perfect.

I’m grateful to be on Metacritic because it’s a fundamental part of the modern conversation about entertainment. People like to gauge the consensus on their entertainment. People like numbers, statistics, math, lists, favorites. It’s part of nearly any hobby. And I’m the same way, as a guy who plays games and as a guy who writes about games. Unlike a lot of writers — Adam Sessler really unloaded on Metacritic in Schreier’s article, implying that it keeps food from game developers’ families — I don’t hate the aggregates. On the contrary, I consult them frequently, and I feel they provide a valuable if sometimes flawed service.

My issue is with the data fed into Metacritic. An aggregate is only as good as its individual components. The system that doesn’t work is the perception that videogame scores are somehow an objective measure of a game’s quality. They aren’t. They’re a measure of how much a reviewer liked a game. Any other perception of a rating is fundamentally broken. It’s not Metacritic’s fault that so many reviews treat videogames like toasters, cell phones, and automobiles.

It’s also not Metacritic’s fault that so many of the larger sites are so cowardly about how they rate games. I don’t believe that what IGN and Game Informer do has to be the norm, any more than I believe that what Fox News or The Daily Sun does is the norm. Big media isn’t the same thing as right media. The solution is for more critics to use the full range of their rating systems, and for readers to recognize that three stars at Quarter to Three isn’t the same thing as a 6.0 at Gamespot, which isn’t the same as a 6 at Eurogamer. Tom Chick is not the same person as Kevin VanOrd, who is not the same person as Tom Bramwell. We have our own perspectives and methodologies, and none of those perspectives or methodologies gets to dictate the norm at Metacritic. That’s the point of an aggregate. It should be a variety of voices pulling in a variety of directions, and not a bunch of non-committal uncritical mumbling huddled under a drab tent pitched between the 70 and the 90. Personally, I feel it’s a failure of games criticism that I’m so often at the bottom of the page on Metacritic.

And as for how this data supposedly deprives game developers’ families of food — seriously, Sessler, if that’s your priority, put down the microphone and break out a pair of poms poms — I love the games industry and what it does. I want it to succeed. I want every studio to remain open and every employee to get a bonus and every game to make every publisher rich. I write about videogames because I’m an enthusiast. But more importantly than wanting games to succeed, I want games to be good, to get better, to grow up, to offer unique and meaningful experiences, to speak to people in the same way that books and movies and music speak to people. Recognizing when that happens and calling it out when that doesn’t happen is more important to me than making sure someone gets his bonus. I told Schreier in our conversation that my main obligation was to my readers. But upon reflection, I’m not sure that’s true. My main obligation is to the medium.