A new study on media bias - have fun!

[JeffL]

http://www.newsroom.ucla.edu/page.asp?RelNum=6664

Unexpected results, at least to me. Have fun tearing apart the pieces you don't agree with! (After all, that's why we post here! ;) )

[Squirrel Killer]

The methodology seems really wonky, and they don't really explain the full justification for it. Tally mentions of liberal/conservative groups and compare that against politicians, and measure it all on a third groups ranking? Seems pretty convoluted, and they don't mention how they handled negative mentions (ie Drudge mentioning the NAACP.) I mean, come on, Drudge as slightly liberal? That's not a revelation, that's a sign of bad methodology.

[Brian Rucker]

On first reading it seems to me this is a pretty straight study. The results are a little odd, maybe, but the intentions look alright. Where I'd have a question is how they categorized different thinktanks as left or right. Many on the right would call the ABA or Brookings liberal where they're actually pretty moderate. And considering how often either of these is cited in the news you'd have something that looked more left-skewing than it is. As far as that aspect goes we really don't know enough to judge the results.

Edit: They called Drudge liberal because he linked to other media that cited "liberal" sources. It's kinda funky I agree.

[JeffL]

Yeah, what made it interesting to me is that the conclusions in some places are different from "truisms". Which means it's a challenging article for folks who have preconceived "truths" (which is all of us, right?) For example, the news pages of the WSJ are slightly liberal? Actually, having watched Good Morning America they do seem pretty balanced. And I imagine that the reason that Brit Hume's show comes out like it does is that they do have a segment each show with liberal and conservative panelists debating each other. Drudge liberal? Surely he isn't, but he does link every day to liberal stories and certainly many liberal sources, which is something most pure conservative pages don't (and I'm trying to think of a liberal page that regularly links to conservative sites and stories.)

Like I said, it's interesting no matter whether you agree or disagree.

[Tim Partlett]

So if I created a news web site that linked to nothing but liberal sources and mocked each and every one of them it would be considered extremely left-leaning? This is an interesting statistically analysis, but it completely misses where most of the bias lies: in the presentation. MEMRI links to only middle eastern sources which it cherry picks for translation. Going by this methodology it would be considered extremely pro-Arab, but it is run by former members of the Israeli intelligence services with a definite bias against anything Muslim or Arabic.

[Glenn]

For example, the news pages of the WSJ are slightly liberal?
Actually, my dad got the WSJ the whole time I was growing up, and this was something I always kinda felt: even though the editorial page is apeshit conservative, the general coverage is really well balanced, and, yes, maybe even a little leftish sometimes.

[antlers]

The methodology is crap for reasons I elucidated when the study first came out earlier in the year. The methodology pushes "mainstream" think tanks (cited by both conservative and liberal legislators) into the "liberal" column for balance, which makes any news outlet that doesn't cite "partisan" think tanks look liberal.

Outlets like PBS, which are more likely to explicitly mention think tanks on the left or right, end up coming up as more balanced than outlets which won't reference a think tank at all unless it is totally mainstream.

[Sidd_Budd]

Interesting article; thanks for the link. I've read the actual study and can answer some of the questions posters had.

1) The authors excluded citations that led to criticism, or were couched in ideological language (e.g., "the conservative Heritage Foundation favors this"), so Tim Partlett's hypothetical mocking web site would be excluded.

2) As posters have noted, most of the Drudge Report citations come from links to stories on other web sites, rather than reports by Drudge himself. This would account for the slightly liberal (i.e., average) score received by the website.

3) I can't figure out what methodology antlers is criticizing. The authors in this study score think tanks by taking the average ADA score (an index of liberal voting -- see the original link for more details) of lawmakers who cite the think tank in the Congressional Record. The average Democratic lawmaker has a ADA score of 84, while the average Republican has a 16. If a think tank was "mainstream" and quoted equally by both parties, it would receive a centrist score of 50 (the average of 16 and 84). Similarly, a media outlet that only quoted this think tank would also receive a score of 50. Antlers seems to believe these mainstream policy groups were labeled "liberal" by the authors, when they were given a centrist score.

I messed around with the numbers a bit and couldn't find anything that substantially alters the study's overall findings. I'm not sure the authors use bias in the same way raving partisans do, but it appears as though most media outlets tend to cite think tanks and policy groups that Democratic congresspeople favor over institutes quoted by Republican lawmakers.

My favorite nuggets from the study:
1) The Wall Street Journal isn't just slightly liberal; it's screamingly granola. It received an estimated ADA score of 85.1, which is more liberal than the average Democratic legislator (84.3), but slightly more conservative than John Kerry (87.6).

2) The American Civil Liberties Union is centrist, you silly conservatives. The average lawmaker who cited this group had a ADA score of 49.8. Why did Republicans cite the ACLU as often as Democrats? The union opposed the McCain-Feingold Campaign Finance bill, and conservative critics of the bill found a friend in the ACLU.

3) It's important to put the results in context. 85% of the media outlets received estimated ADA scores in the central portion of 33 to 67. In contrast, only 15-20% of senators during the 1990s received ADA scores in the same range. The media may not be perfect in its reporting, but it's more centrist than most of our legislators, and a far cry from an extreme leftist propaganda tool as some pundits suggest.

Except for that socialist WSJ rag.

[Jason McCullough]

My favorite assumption is "the average ADA score in Congress (50.1) was assumed to represent the political position of the average U.S. voter." Yeah, there's a hell of a lot of evidence for *that*.

[extarbags]

I love how pompous this guy is:

"A media person would have never done this study," said Groseclose, a UCLA political science professor, whose research and teaching focuses on the U.S. Congress. "It takes a Congress scholar even to think of using ADA scores as a measure. And I don't think many "media scholars" would have considered comparing news stories to congressional speeches."

Various emphases mine, douchebag attitude all his.

But seriously, all you need to know this is BS is this:

Groseclose and Milyo based their research on a standard gauge of a lawmaker's support for liberal causes.

So they set out to find out which way the newsmedia is biased... by specifically ranking how liberal they are. Yeah, this whole thing screams of a couple of dudes setting out to get the results they want.

[Sidd_Budd]

So they set out to find out which way the newsmedia is biased... by specifically ranking how liberal they are. Yeah, this whole thing screams of a couple of dudes setting out to get the results they want.
Just to be clear, the ADA ranks members of Congress; the authors didn't make up the rating system. Assuming political ideology can be expressed by a single scale with conservative on one end and liberal on the other, what's wrong with using either end as a reference point? The average lawmaker scores about 50 either way.

Your argument seems similar to suggesting that the Fahrenheit and Celsius scales are biased because the scales technically measure *heat*, rather than cold.

I agree with your assessment of the author's pompous attitude, but what do you expect from a liberal academic?

[extarbags]

So they set out to find out which way the newsmedia is biased... by specifically ranking how liberal they are. Yeah, this whole thing screams of a couple of dudes setting out to get the results they want.
Just to be clear, the ADA ranks members of Congress; the authors didn't make up the rating system. Assuming political ideology can be expressed by a single scale with conservative on one end and liberal on the other, what's wrong with using either end as a reference point? The average lawmaker scores about 50 either way.


First, yes I know that the ADA does the ranking, but they did decide to use it, after all.

Second, it's a question of where they're coming from. No, there's no functional difference between using 50 as the midpoint instead of 0, but when you say "I'm going to go ahead and look at how people vote on liberal issues to find out how liberal they are, as opposed to setting out to just find out where someone sits on the political spectrum overall, it just indicates that you're looking for a certain result... whether you're ranking people that way because you want to know who your enemies are or because you want to know who your friends are, I don't think it's likely to lead to results that are worth taking seriously.

Again, that scale doesn't even measure where people are on the political spectrum; all it measures is how liberal someone is. Zero on that scale doesn't represent conservative, it represents "not liberal at all." It might not amount to much of a difference all in all, but I think it speaks pretty loudly to how seriously people should take the guys that are doing that ranking.

[Sidd_Budd]

... when you say "I'm going to go ahead and look at how people vote on liberal issues to find out how liberal they are, as opposed to setting out to just find out where someone sits on the political spectrum overall, it just indicates that you're looking for a certain result...

Again, that scale doesn't even measure where people are on the political spectrum; all it measures is how liberal someone is. Zero on that scale doesn't represent conservative, it represents "not liberal at all."
1) To achieve an ADA score of 0, a lawmaker has to have voted "yes" on some issues and "no" on others. The index records votes on both liberal and conservative issues, not just liberal causes. This lawmaker would have voted no on bills for greater gun control and increases in childcare funds, and yes on bills limiting medical malpractice punitive damages and exempting U.S. multinationals from paying federal taxes on some income. The terms "conservative" and "not liberal at all" both seem accurate in describing this voting record. The ADA scores are acceptable to me, but I'm not a political scientist. Are there other measures of political views that have more acceptance? I'm assuming simple party affiliation is too rough an index.

2) I don't see how your preference for different labels materially affects the results of the study. The average media outlet had an estimated ADA score of 62. The average legislator and average voter had estimated ADA scores of 50. Higher ADA scores reflect more liberal views, regardless of whether we call a 0 score "conservative" or "absence of liberal." Therefore, media outlets tend to be more liberal (more accurately, to quote from sources preferred by liberals) than the average voter.

You seem to have the impression that since ADA scores measure amount of liberalness, the only possible outcome for the study was liberal media bias. This isn't the case. The average media outlet could just as easily have received an ADA estimate of 50 (similar to voters) or 40 (less liberal than voters).

[extarbags]

A) No, what I'm saying is that ADA's clear angle makes me predisposed not to trust their judgement and, by extension, this study.

B) Even if you accept the ADA's judgement on the stances of Congressmen, this study is still extremely flawed. They assume that the average ranking of Congressmen is the same as the average among voters, just kind of because they say so. Then they go ahead and analyze the think tanks that news organizations cite, which is oh-so-very-relevant, and the policy organizations that they reference, and match them to a politician that has made similar references.

It's stupid. Seriously, it's just stupid. Just referencing the NAACP in a newscast makes you more liberal, in the eyes of this study. What if there's a news story about the NAACP? Should it go unreported entirely, lest newscasters be labelled biased?

[Tim Partlett]

1) The authors excluded citations that led to criticism, or were couched in ideological language (e.g., "the conservative Heritage Foundation favors this"), so Tim Partlett's hypothetical mocking web site would be excluded.

Just by doing that you inject a huge amount of subjectivity into the analysis. It's an interesting exercise, but I don't think the results are all that useful. I could also go the MEMRI route, and quote only the most extreme out-of-context opinions from liberal sources, without any editorialising at all, and show a biased and distorted view of the left wing, while still appearing as an extremely left-leaning source in this statistical analysis.

[extarbags]

And you know what else? Why even rate the stupid Drudge Report if you're going to say afterwards "well har har har I know it looks like this invalidates my entire study, but ya see it doesn't count because it isn't really news, it's just links to other sites, so it actually makes my case stronger?

Plus, man, everytime I read that quote, I want to punch that guy in his smug mouth.

[JeffL]

And you know what else? Why even rate the stupid Drudge Report if you're going to say afterwards "well har har har I know it looks like this invalidates my entire study, but ya see it doesn't count because it isn't really news, it's just links to other sites, so it actually makes my case stronger?

Plus, man, everytime I read that quote, I want to punch that guy in his smug mouth.

You seem to be having a hard time analysing this data without some severe emotion. ;) Hey, it's just data, and unlike a lot of reports they actually tell you pretty specifically how and what they are analyzing. Thus you can reach any conclusion you like on the data, whether you agree with them or not.

As for Drudge, since that seems to really get your goat (;) ) if you ignore the "Drudge Exclusives" which tend to be be un-sourced, there is a mix of "liberal" and "conservative" stories on there, which is a bit unusual for a web site that is tagged as a one sided partison site. For example, looking today, there's a link to an article that basically says Bush lied when he said he had broad Democratic backing for his decision on the domestic phone taps, an article describing the judge's ruling knocking down the intelligent design teaching in one school, and article titled "Bush’s Snoopgate
The president was so desperate to kill The New York Times’ eavesdropping story, he summoned the paper’s editor and publisher to the Oval Office. But it wasn’t just out of concern about national security.", and goes on to say "We’re seeing clearly now that Bush thought 9/11 gave him license to act like a dictator, or in his own mind", plus some articles that have a conservative slant. If you were looking at just the content linked on the page, incliding the links to editorials by folks like Rosie ODonnel, Molly Ivans, etc. and you'd have a hard time claiming that all his page shows is one side of the issues. Even though there is no doubt that he is a hard core conservative (I doubt you'll find Arianna Huffington or Barbara Walters linking stories negative to Clinton or linking to Limbaugh columns.)

Its data, and data that is clearly explained in transparent terms is always interesting no matter what conclusions you draw from it. IMO.

[Sidd_Budd]

I agree with Jeff that the full disclosure of methods and sources used by this study make it stronger. I've often tried to pin my conservative friends down on the "of course NPR is liberal" belief system, with no success -- I can't figure out how they are hearing massive liberal spin (perhaps because of my own biases), and they can't give me quantifiable terms that they use as a barometer of liberalness. IMO, the authors of this study don't properly account for alternative explanations of their findings, but at least they document how they reached their conclusions.

And you know what else? Why even rate the stupid Drudge Report if you're going to say afterwards "well har har har I know it looks like this invalidates my entire study, but ya see it doesn't count because it isn't really news, it's just links to other sites, so it actually makes my case stronger?
It's unethical for scientists to eliminate data just because their findings are counter to their initial predictions. You run a study to *test* your hypothesis, not to automatically *confirm* it, and you aren't supposed to cherry-pick your results to get clean, unambigious findings. I would reject any manuscript I reviewed where the authors stated they eliminated a few results because they seemed counterintuitive or difficult to explain.

[antlers]

Basically, the problem is that if the distribution of actual think tank ideology is not a good match for the distribution of citing politician ideology, the ranking assigned to think tanks, and thus the ranking assigned to the news media that cite them, will be skewed.

To simplify for an example, if there is a set of thinks tanks that are cited by everyone, and another set of think tanks that are cited only by conservatives, a news outlet that cites only the think tanks that are cited by everyone will be rated as liberal, since it will match the left-wingers and the moderates but not the conservatives. In fact, such a news outlet would be scrupulously neutral, since it would only be citing think tanks that are scrupulously neutral.

[BrewersDroop]

The study also assumes that the type of issues and the frequency they are discussed is the same in congress and in the media. I suspect congress focuses more on fiscal and legal issues than the media while social issues are discussed more frequently and in more depth in the media than in congress.

[extarbags]

And you know what else? Why even rate the stupid Drudge Report if you're going to say afterwards "well har har har I know it looks like this invalidates my entire study, but ya see it doesn't count because it isn't really news, it's just links to other sites, so it actually makes my case stronger?
It's unethical for scientists to eliminate data just because their findings are counter to their initial predictions. You run a study to *test* your hypothesis, not to automatically *confirm* it, and you aren't supposed to cherry-pick your results to get clean, unambigious findings. I would reject any manuscript I reviewed where the authors stated they eliminated a few results because they seemed counterintuitive or difficult to explain.

To the contrary, I think they absolutely did run this study to confirm their hypothesis.

I didn't say they should have thrown out the Drudge Report results; they shouldn't have even included it at all, because it's basically a bunch of links to other news stories, which they even say in their analysis. They didn't rank Slashdot in this study; why rank Drudge?

Its data, and data that is clearly explained in transparent terms is always interesting no matter what conclusions you draw from it. IMO.

Well, maybe not always, but I see your point. My only point is that the conclusions that they are drawing from the data are just ridiculous.

[Sidd_Budd]

To simplify for an example, if there is a set of thinks tanks that are cited by everyone, and another set of think tanks that are cited only by conservatives, a news outlet that cites only the think tanks that are cited by everyone will be rated as liberal, since it will match the left-wingers and the moderates but not the conservatives. In fact, such a news outlet would be scrupulously neutral, since it would only be citing think tanks that are scrupulously neutral.
Thanks, I think I followed that. It was odd that the authors simply used average congressperson ADA scores to estimate think tank ideology, but then used a maximum likelihood utility function for their media outlet estimates. Why not simply estimate media outlets as the weighted average of their think tank citations? At the least, they could have run the numbers both ways.

Just to make sure I understand your point, the only way there will be a methodological bias is when there is a ADA distribution of think tank citations that deviates from average. If there is one group that is cited by conservatives, a second group (approximately equal in size to the first) cited by liberals, and then a third group (size irrelevant) cited by all ideologies, there would be no bias in media estimates.

If this is true, it should be easy to test. Compute the average cited think tank score -- if it's relatively close to 50, there should be no problem; if it's either too low or too high, this indicates that citations tend to lean left or right, and this will bias the media outlet estimates.