Thursday, December 22, 2005

Measuring Media Bias

As I was starting to write a critique of the recent report on “media bias” by Tim Groseclose, a UCLA political scientist and Jeffrey Milyo, a University of Missouri economist, I saw that Media Matters beat me to it. They do a bang-up job too, although I have some differences with the Media Matters evaluation.

The report found—guess what?—“liberal bias” in the mainstream media. Gasp! As Media Matters lays out, the researchers employ an unusual methodology to assess bias. First, they use Americans for Democratic Action (ADA) ratings for members of Congress to determine how liberal each member is. Floor speeches by each member were analyzed to see which think tanks and policy organizations each cited most, omitting the citations made in the course of refuting or denouncing the organization. The organizations most cited by liberal members were deemed to be liberal organizations, and those most cited by conservative members were deemed to be conservative organizations, with each organization receiving a score from 1 to 100 corresponding to the ADA ranking. Then the researchers analyzed media outlets for the think tanks they cited in their news stories, and compared these patterns with the patterns observed in Congress. Media outlets with similar citation patterns to liberal members were deemed to be liberal, etc.

The results were that 18 of the 20 media outlets evaluated came out as liberal. The exceptions were the Washington Times and Fox News.

Media Matters points out the authors’ apparent ignorance of previous scholarly work about media bias. They also describe some flaws in their methodology, and some of the peculiar results it generates: the ACLU turns out to be a conservative group; the National Rifle Association comes out as a moderate organization; the RAND Corporation and the Council on Foreign relations come out as being to the left of the ACLU.

Despite these anomalies, I think somewhat more highly of the report’s methodology than Media Matters does. It is a cleverly constructed attempt to generate an objective standard for defining how liberal or conservative an organization is, at least in comparison with the national political center of gravity. In most cases the relative results (in terms of how think tanks and media outlets are ranked in comparison with each other) seem approximately accurate. While the methodology doesn’t really get at “bias” as I understand it, it does provide some clues as to which groups are most often publicized by the mainstream media, and, on a relative basis, whether these groups are liberal or conservative. Moreover, the report’s authors have explanations for how some of the anomalies occurred and, in some cases, for why the anomalies aren’t important. Besides, no methodology is perfect.

However, I believe that there are at factors that bias the entire study towards labeling media outlets as “liberal.” One is described by Media Matters:

. . . stories about race relations that include a quote from an NAACP representative are unlikely to be "balanced" with quotes from another group on their list. Their quotes will often be balanced by quotes from an individual, depending on the nature of the story; however, because there are no pro-racism groups of any legitimacy (or on Groseclose and Milyo's list), such stories will be coded as having a "liberal bias."
Since the NAACP is the third most-cited group in the study, this factor could have a significant impact.

I believe that two additional factors bias the report towards the conclusion that the media tilt left:

1. The Government: A Conservative Group. The period of observation of media outlets varied, depending on the organization, but most organizations were tracked from the late 1990s through 2003. Moreover, of the 20 news organizations tracked, nine were followed only during 2001 and beyond. Therefore, during most of the time the news organizations were evaluated, the Republicans held the presidency and had the initiative in putting forward policies and programs that would receive press coverage. When reporting on a Republican White House initiative, a reporter seeking “balance” and reaction from political opposition might choose to quote Democratic congressional leaders, but they might also choose to quote liberal think tanks and advocacy groups. I would be interested in seeing the authors do an analysis of whether trends in citing liberal versus conservative groups change as the political party in power changes, both in the White House and in Congress.

2. Liberals: the reality-based community. The organizations most often cited by members of Congress were the National Taxpayers Union (conservative), AARP (liberal), Amnesty International (liberal), Sierra Club (liberal), Heritage Foundation (conservative), Citizens Against Government Waste (conservative), RAND (liberal), Brookings (liberal), National Federation of Independent Businesses (conservative), and ACLU (anomalously slightly conservative in this ranking). Note that the liberal organizations include two that are highly scholarly sources of relatively unbiased information, and not political advocacy groups. Amnesty International, while considered a liberal advocacy group on domestic controversies such as the death penalty and abusive treatment of detainees at Guantanamo, is also an outstanding source of unbiased information about human rights conditions in regimes such as Iran, Pakistan, North Korea, etc. Of the conservative organizations, the Heritage Foundation comes closest to being a scholarly organization, but it is much closer to being an arm of the conservative movement than either RAND or Brookings is to being an arm of the liberals. So when news outlets cite Brookings or RAND or Amnesty, the actual content of the information that is being cited is likely to be less loaded in ideological or partisan terms and less affected by “spin” than is the case when the National Taxpayers Union or Citizens Against Government Waste or the Heritage Foundation are cited. I would be interested in seeing an analysis that attempted to control for such differences.

So while I think that the methodology that Groseclose and Milyo have developed has some merits and is worth trying to improve, I think that their bottom-line result is unreliable. The finding that the media tilts left is unproven.