Saturday, December 24, 2005

More on Measuring Media Bias

A couple of additional points to make regarding the recent report evaluating “media bias” that I blogged about a couple of days ago:

In addition to the Media Matters evaluation of the report, earlier versions of the report received critiques from the Left Coaster and from Geoffrey Nunberg. The report’s authors, Groseclose and Milyo, responded to Nunberg. I agree with some of points made in these critiques of the Groseclose/Milyo report, particularly the idea that authors overreached by labeling their report a study of media “bias.” The report looks only at journalists’ use of think tanks and policy advocacy groups, and bias can creep into journalism in a lot of ways other than through selection of groups to cite. Moreover, the report takes any deviation from the political center (as defined by which groups are cited by members of Congress in floor speeches), as evidence of “bias.” I would like to see evidence of unfairness or inaccuracy on the part of journalists before the word “bias” is thrown around.

However, I don’t think that the report’s methodology is worthless, as some of the critics do. I think the report is actually pretty interesting, if it is taken not as a full accounting of “media bias,” but rather as an attempt to quantitatively determine what types of think tanks and policy groups the media give voice to most often, and how those groups stand in relation to the national polical center of gravity. The fact that the methodology produces some anomalous results doesn’t mean that the whole approach is without merit.

But I do think that the methodology needs some refinement. In particular, I think that several factors biased the report towards the result that media organizations favor liberal groups. I would like to expand on one of those factors that I cited in my previous post. I opined that liberal organizations are likely, as a group, more scholarly and and better sources of unbiased information than conservative organizations. Therefore, if journalists are more concerned than members of Congress with the scholarly reputation of an organization, they might be inclined to cite liberal organizations, not because they are liberal but because they are scholarly.

I was unfair, in my previous post, not to acknowledge that Groseclose and Milyo do attempt to control for the “scholarliness” of groups in their study. However, I do not believe that their controls fully address this issue, and I still believe that this factor biases their study towards the conclusion that the media favor liberal groups.

The authors constructed three binary variables to control for “scholarliness:” whether the organization’s website has closed membership (like a think tank and unlike most advocacy groups), whether the organization has staff with the title “fellow,” “researcher,” “economist,” or “analyst,” and whether the organization is located off K Street in Washington D.C. (unlike organizations devoted to lobbying, which tend to be located on K Street). I don’t think that these variables get very far towards a true measure of scholarliness—the Heritage Foundation comes out as scholarly as Brookings. I’d like to see the authors try to use a measure such as number of staff who have published research in peer reviewed journals in the last two years, or something similar.

Nonetheless, use of these variables made media organizations come out somewhat more conservative than the version of the model that did not control for “scholarliness,” though they still came out conservative on average and the change was not statistically significant. I’m inclined to believe that if the model used more sophisticated measures of scholarliness, news organizations would come out still more conservative.

But more importantly, scholarliness is only part of the issue. Think tanks and policy organizations produce a lot of different information. Some of this information is useful in arguments that fit on a liberal-conservative spectrum, and some is not. Sometimes policy organizations package this information into arguments, but sometimes they put information out in an unbiased way without spinning it at all-- a good example would being the Brookings Institution’s Iraq Index. I would contend that liberal organizations are more likely to also be organizations that are good sources for this sort of non-ideological, non-partisan information. For instance, Amnesty International can fairly be classified as a liberal organization. But if a reporter is looking for a source of information about, say, the human rights situation in Sri Lanka, he or she will likely turn to Amnesty. Does that create a “liberal” slant to the story? Not if there’s no liberal-conservative split on the human rights situation in Sri Lanka. What if a journalist is writing a story about a politically-loaded question like welfare policy, and she gets data about trends in the number of poor people from the Center on Budget and Policy Priorities, because they provide the government data in an unbiased, easily accessible format. She then quotes the Economic Policy Institute saying that the Bush administration’s welfare policies are too conservative, and quotes the Heritage Foundation saying that the administration’s policies are too liberal. Is she tilting her story to the left because she has quoted two liberal organizations and one conservative one? I would say no, since one of the liberal organizations was used to provide data that was not in dispute, not to provide politically-salient arguments.

If I’m right that liberal organziations are better sources of unbiased information, and if members of Congress are more likely to quote organizations to score points in a political debate, while journalists are more likely use organizations to provide information that doesn’t necessarily have political/ideological salience, then the fact that journalists are more likely to quote liberal organization is not evidence that journalists slant left.