some rockbandometrics

[skedastic]

Is your favorite band performing well as Rock Band downloadable content? Sales figures (or pretty good estimates thereof) can be found at dlcstats.com, but it's still difficult to tell. Bands which released songs early have an advantage, both because there was less competition earlier and because they've had more time to sell their songs. To compare bands on an even playing field, differences in release dates must be statistically removed.

To adjust the total sales figures for the dates at which various songs were released, I downloaded the data off that page. I then crunched the numbers using some simple regression modeling, described in a bit more detail at the end of the post for gearheads. The results are presented in the table below. The columns contain the band's rank, name, unadjusted average sales per song per day, and adjusted average sales per day. Both adjusted and unadjusted sales are given relative to the average band, so negative numbers mean the band sells less per day than the average band. Adjusted average sales per day answers the question:

How many more or less units per day does this band's average song sell relative to the average band's if it were the case that both bands released all their songs on the same dates?

For example, Muse is first with an adjusted sales per day of 1,343. That means that Muse songs on average sell 1,343 units per day more than the average band's songs, given that we have statistically made it such that both Muse and the average band released all their songs on the same days at the same prices.

Notice, depressingly, that Fall Out Boy comes in third, or really second if we disregard GlaDOS. Also depressingly, Sonic Youth and the Mother Hips come in next to last and last.

Note to Activision: metal bands don't sell particularly well.

                                           Sales per day 
                                           relative to average:
                                           --------------------------
                               band        raw      adjusted  
------------------------------------------------------------
  1.                           Muse       1716       1343  
  2.                         GLaDOS       1091       1037  
  3.                   Fall Out Boy        969        700  
  4.                       Paramore        386        432  
  5.                      Blink-182        218        418  
  6.                 Lynyrd Skynyrd        360        400  
  7.       The All-American Rejects        125        342  
  8.                    Motley Crue        389        256  
  9.                    Bang Camaro        339        247  
 10.                         Boston        253        235  
 11.             30 Seconds to Mars         17        233  
 12.                          Oasis        171        220  
 13.                   Serj Tankian        164        209  
 14.   Creedence Clearwater Revival         -5        189  
 15.                      Freezepop        279        187  
 16.                      Metallica        -12        183  
 17.                      Foreigner        -29        166  
 18.            Stone Temple Pilots          1        166  
 19.                         Weezer         -7        163  
 20.                       The Cars        516         88  
 21.             Coheed and Cambria        -94         82  
 22.                      The Knack       -140         55  
 23.                     The Police        -65         54  
 24.               The Black Crowes       -166         48  
 25.              Smashing Pumpkins         66         44  
 26.                Die Toten hosen        410         37  
 27.                    Iron Maiden       -186         22  
 28.                  Black Sabbath       -192         19  
 29.                           Rush       -182         13  
 30.                Nine Inch Nails       -100         -1  
 31.                     Wolfmother       -199         -4  
 32.                    The Monkees       -244        -44  
 33.                    Tokio Hotel        313        -60  
 34.                      The B-52s       -226        -65  
 35.        Queens of the Stone Age       -261        -66  
 36.              Angels & Airwaves        189        -81  
 37.                      Radiohead       -301        -83  
 38.                    The Haunted       -124       -110  
 39.                     Count Zero        -19       -111  
 40.                           Kiss       -275       -115  
 41.                        Blondie        112       -115  
 42.                   At the Gates       -139       -125  
 43.                      Buzzcocks       -344       -128  
 44.                 The Pretenders       -350       -133  
 45.                     Black Tide       -184       -139  
 46.                         T. Rex       -336       -141  
 47.                         Pleymo        214       -159  
 48.                      The Clash       -174       -161  
 49.              The Grateful Dead       -236       -163  
 50.                      The Hives       -372       -164  
 51.                    David Bowie       -373       -169  
 52.                  Faith No More       -340       -180  
 53.                        Ramones       -359       -180  
 54.                     The Sounds       -400       -183  
 55.                           Blur        185       -188  
 56.                        Garbage       -314       -192  
 57.                          Sweet       -405       -205  
 58.                   The Runaways       -436       -241  
 59.                Yeah Yeah Yeahs         13       -257  
 60.                          Evile       -278       -264  
 61.                       H-BlockX        105       -268  
 62.                           Juli         42       -331  
 63.                   Judas Priest       -186       -361  
 64.                     Les Wampas        -48       -421  
 65.                    Sonic Youth       -183       -503  
 66.                The Mother Hips       -378       -599

Lies and damn lies and such:

Raw sales per day is total sales divided by number of days between release date and June 2.

The regression model took the form:

y_i = b_0i + b_1 D_i + b_2 D_i^2 + u_i

where

y_i sales per day
b_0i band-specific effects
b_1..b_2 parameters
D_i days since release
u_i idiosyncratic disturbance

This model was fit using fixed effects linear regression, with fixed effects at the band level. The "adjusted sales per day" figures in the table are the estimated band effects. The coefficients on the quadratic in days since release were jointly highly significant and showed that sales per day fall over time at a decreasing rate. The estimates:

Fixed-effects (within) regression               Number of obs      =       121
Group variable: bandnum                         Number of groups   =        66

R-sq:  within  = 0.1758                         Obs per group: min =         1
       between = 0.3000                                        avg =       1.8
       overall = 0.2906                                        max =         9

                                                F(2,65)            =      6.28
corr(u_i, Xb)  = -0.0865                        Prob > F           =    0.0032

                               (Std. Err. adjusted for 66 clusters in bandnum)
------------------------------------------------------------------------------
             |               Robust
 salesperday |      Coef.   Std. Err.      t    P>|t|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
        days |  -8.469761    3.20951    -2.64   0.010    -14.87959   -2.059928
       days2 |   254.7474   128.3418     1.98   0.051    -1.568882    511.0636
       _cons |   982.5993   150.1929     6.54   0.000     682.6433    1282.555
-------------+----------------------------------------------------------------
     sigma_u |  312.37977
     sigma_e |  240.71106
         rho |  .62743878   (fraction of variance due to u_i)
------------------------------------------------------------------------------

This approach exploits the fact that some bands released songs at different times to allow for the possibility that the popularity of bands has been changing over time. The correlation between the estimated fixed effects and the predicted day is negative, confirming the quality of the bands, as measured in sales, has been falling over time.

I also tried including dummies for price of $0.99 and for the song's inclusion in a pack, but they were colinear with the fixed effects. No adjustment was made for $0.99, the free Portal song, or various song-specific promotions. I also didn't do anything about the 10 or so songs which had sales over 100,000 which cannot be tracked, so the Police and a few other popular bands look a bit worse than they are.

[Phil_Stein]

You forget to carry the 3. Your analysis is null and void...

[Bahimiron]

That's great.

Despite all the hate Fall Out Boy gets whenever it gets brought up on a message board, with people screaming that they'd pay two bucks to never have to play any Fall Out Boy songs, they're sitting on top of the paid charts. Can the full album of Fall Out Boy be far?

Edit: Wow. Judas Priest kinda bombed, didn't they?

[Enidigm]

Have you actually heard Sonic Youth though? Why would you want to (pretend to) play their songs? :)

[Funkula]

I actually have more song purchases in the bottom half of the list than the top :(

[Sidd_Budd]

You had me at fixed effects

[Allagash]

You know that dlcstats.com isn't accurate beyond 100K, right? It's also a guesstimate for other reasons they go into on the site, but I guess it's the best shot given publicly-available information.

[Piemax2]

It seems odd to use sales per day to compare newly released songs vs much older ones. It might make sense for comparing last week's releases to those of two weeks ago but after 3 or 4 weeks the assumed linearity has to break down. Can you think of some better way to handle time since release?

[wumpus]

Sales figures (or pretty good estimates thereof)

CITATION NEEDED

[Midnight Son]

Originally Posted by bahimiron

Despite all the hate Fall Out Boy gets whenever it gets brought up on a message board, with people screaming that they'd pay two bucks to never have to play any Fall Out Boy songs, they're sitting on top of the paid charts. Can the full album of Fall Out Boy be far?

All it shows is that's what popular RIGHT NOW. If we had RB back in the eighties, the Cars and Priest would sell like crazy.

[wumpus]

I actually like the Fall Out Boy track in the game, and the DLC one as well.

I've even warmed up to "Electric Version", which used to be my hands-down pick for worst track in the original RB setlist. Now I think it's "Black Hole Sun"; great song, but so goddamn boring to play.

[Bahimiron]

Originally Posted by Midnight Son

All it shows is that's what popular RIGHT NOW. If we had RB back in the eighties, the Cars and Priest would sell like crazy.

Well... yeah? And?

As it turns out, RIGHT NOW it is RIGHT NOW and as such RIGHT NOW's numbers are really important for making decisions RIGHT NOW. I'm sure that as soon as Harmonix develops a time traveling guitar, this point will become extremely important. Cos then they can go back to 1986 and start selling Judas Priest songs like mothafuckin' GANGBUSTERS! Then they can go back to the 40s and Boogie Woogie Bugle Boy will be the #1 downloaded song.

Unfortunately, 'til that day comes, it's likely that the metrics demonstrated will be the trends followed when it comes to attempts at selling greater numbers of songs. These numbers pretty much tell a story that RIGHT NOW (the only time that matters) selling full albums of Fallout Boy and Arcade Fire! would be a better bet than The Ramones or more Judas Priest.

[Wholly Schmidt]

Originally Posted by bahimiron

These numbers pretty much tell a story that RIGHT NOW (the only time that matters) selling full albums of Fallout Boy and Arcade Fire! would be a better bet than The Ramones or more Judas Priest.

Was that a typo? Don't tease me, bro, because Arcade Fire would be awesome (and is also strange to group with Fallout Boy).

Anyway, I'm taking bets on how soon Joystiq or Kotaku picks up on skedastic's estimations based on another site's guesses, misinterprets the whole thing, slaps a misleading headline on it and calls it news.

[Bahimiron]

That wasn't a typo, that was me confusing Arcade Fire with Panic! at the Disco. Thus the inexplicable ! at the end of Arcade Fire. I know they sound nothing alike, but ever since hearing a local DJ repeatedly accidentally call them both 'Fire! at the Disco!', they've been invariably messed up in my brain.

Apologies to all Arcade Fire fans.

Anyway, I'm taking bets on how soon Joystiq or Kotaku picks up on skedastic's estimations based on another site's guesses, misinterprets the whole thing, slaps a misleading headline on it and calls it news.

You could make it happen. Just digg this thread.

(But, no, don't digg this thread.)

[Adree]

It's refreshing to learn that even the kiddies don't like Angels and Airwaves that much.

[skedastic]

Originally Posted by Piemax2

It seems odd to use sales per day to compare newly released songs vs much older ones. It might make sense for comparing last week's releases to those of two weeks ago but after 3 or 4 weeks the assumed linearity has to break down. Can you think of some better way to handle time since release?

The assumption implicit in the model is that a song's sales per day are quadratic in time, that is,

sales/day = constant + b*(days since release) + c*(days since release)^2

This captures the data pretty well, the line in this graph is the predicted sales, the points are actual sales after the band-specific effects have been netted out:



But there is more noise in the first couple of weeks after release, so you are right that this model may be a bit misleading for very recently released songs. Things settle down after the first month or so. Those highly scattered points on the far left are the songs from the Cars album, which have a great deal of variation in sales, so take The Car's rating with a grain of salt.

With more data (more bands and more songs per band, or proper longitudinal data on sales of various songs over time) it would be possible to estimate fancier models which can deal with some of the issues this one ignores, like the censoring at 100,000 and factors other than popularity, such as prices or promotions, which affect quantity demanded. With so few bands and songs, though, the present approach is already asking a great deal of the available data.

Wumpus, as I understand it these figures are based on the total number of recorded scores on Live and PSN. So if someone buys a song and either never beats it on any difficulty or never logs into Live, that sale will be missed. To the extent missed sales are common across bands that will have little effect on the estimates above (and the model could easily be changed such that proportional omitted sales have no effect at all, but I think this would make little difference). This is just for fun so I am too concerned about the data quality, but if you have a better source by all means provide it.

[Adree]

Also of note is that some songs are 99 cents instead of 1.99 at release.

[Piemax2]

The chart is interesting, thanks. I think you said that "sales per day" is actually total sales/time since release, right? If as I suspect sale per week get pretty low once a song has been out for a while, then sales per day becomes roughly constant/time, and by construction won't
move much for older songs. Can you get hold of the actual week-by-week sales?
Also (and this is straining the limits of my econometric skills) does having time since release on the LHS (as the denominator of sales/day) and RHS of the same regression cause any problems or is that ok?

[skedastic]

Constructing longitudinal data by taking multiple cross-sections such as the one I used would be possible if the dlcstats guys keep that information and would provide it to me, but dealing with that sort of data would involve non-trivial effort, and this is not going on my vita.

If days since release were stochastic, using it as a covariate and as the denominator for the regressand would cause an endogeneity problem (although people do it all the time anyways, for example, regressing GDP per capita on a bunch of stuff including population). I don't think that's an issue here since days since release can be treated as fixed.

[Robert Sharp]

Wait. Aren't you suggesting that some of these bands are selling negative numbers per day because of the algorithm you used? Doesn't that suggest an absurd consequence to your approach? Or am I misreading your methods and results?

[skedastic]

The adjusted figures are relative to the average band, so a negative number means the band sells worse than average.

[Piemax2]

what happens if you just regress total sales on stuff that includes t and t^2? you could then compare bands by looking at their projected sales one year out, right?

[skedastic]

That is what I did. Note though that "stuff" includes the full set of band fixed effects, so any and all band-specific influences on sales which don't vary with days since release have effectively been held constant (such as the band's popularity, genre, etc). If band fixed effects weren't included, the estimated effect of days since release would be confounded with changes in the popularity of bands over time (and there is still a problem if Harmonix tends to release more or less popular songs from a given band first). Predicted sales for a song by band j after a given number of days since release are:

predicted = constant + b_0j + c(days) + d(days^2)

where b_oj is band j's estimated fixed effect. Since the constant and the quadratic in days since release are constant across bands for a given value of days since release, it doesn't matter at all whether I report the b_0j (which is what I reported) or the b_0j plus the quadratic evaluated at some number of days since release, such as one year out, as you suggest. The difference amounts to adding or substracting a constant from the "adjusted" column, so rankings and differences are unaltered.

[Talisker]

Originally Posted by skedastic

If days since release were stochastic, using it as a covariate and as the denominator for the regressand would cause an endogeneity problem

Obviously. Duh.

[Shadarr]

Originally Posted by bahimiron

Despite all the hate Fall Out Boy gets whenever it gets brought up on a message board, with people screaming that they'd pay two bucks to never have to play any Fall Out Boy songs, they're sitting on top of the paid charts. Can the full album of Fall Out Boy be far?

Edit: Wow. Judas Priest kinda bombed, didn't they?

Fallout Boy at the top, Priest at the bottom. Kids today have no taste.

Although I will say that the Priest and Boston songs are kinda samey on drums, and The Cars are flat-out boring. But on guitar, Priest and Boston are some of the best songs in the game.

Fallout Boy actually makes me long for a My Chemical Romance or Sum 41 pack. Fallout Boy isn't even good shitty music.

[Moore]

Originally Posted by TheSelfishGene

Have you actually heard Sonic Youth though? Why would you want to (pretend to) play their songs? :)

Grr...

[JPR]

Kool Thing was no fun at all to play in GH3, so that might have soured people on the Rock Band version (which, frankly, I don't like much better).

The same thing could be said for Electric Eye and You Got Another Thing Comin' -- if people played those songs in the previous GH games and didn't like them (like me), they probably wouldn't buy the Judas Priest pack.

Maybe there were even weirdos who didn't like the Boston songs in previous games. Or maybe they figured they still have GH1 and they got Peace of Mind for GH3, so they don't really need to buy those songs twice.

Even if you attempt to discount "taste", there are plenty of good reasons for those numbers to be the way they are.

[Andrew Mayer]

Originally Posted by JPR

Even if you attempt to discount "taste", there are plenty of good reasons for those numbers to be the way they are.

I've always heard that there's no accounting for taste.

[TomChick]

Originally Posted by Wholly

Anyway, I'm taking bets on how soon Joystiq or Kotaku picks up on skedastic's estimations based on another site's guesses, misinterprets the whole thing, slaps a misleading headline on it and calls it news.

First!

-Tom

[Piemax2]

OK, I'm confused. Your first post says that y_i is sales per day, but when I asked what happens in a regression where y_i is simply total sales, you said "that's what I did." Are you saying that the two regressions give identical answers?