Monday, December 26, 2005

One More on Measuring Bias

A final thought about Groseclose Milyo study. The Language Log critique of an earlier version this study states:

Let's begin with the assumption that underlies Groseclose and Milyo's assignment of ratings to the various groups they looked at: if a group is cited by a liberal legislator, it's liberal; if it's cited by a conservative legislator, it's conservative.

On February 24, 2004, for example, in a debate on the medical liability bill, the liberal Senator Christopher Dodd of Connecticut cited "a study conducted by the Rand Corporation and published in the New England Journal of Medicine last year [which concluded] that individuals received the recommended treatment for their condition in only 55 percent of the cases... " For Groseclose and Milyo, Dodd's citation of the study counts as one piece of evidence that the Rand Corporation is a liberal think tank. In fact, their method assumes that there can be no such thing as objective or disinterested scholarship -- every study or piece of research, even if published in so august a scientific authority as the New England Journal, can be assumed to have a hidden agenda, depending on which side finds its results congenial to its political purposes.

In their response, Groseclose and Milyo state:

Although we did not state it in the paper, our own view is nearly the exact opposite of this assumption. Namely, by and large, we believe that all studies and quotes by the think tanks in our sample are true and objective. However, it just happens that some, but not necessarily all, of these true and objective studies appeal differently to conservatives than liberals. To see why, imagine that a researcher publishes a study in a very prestigious scientific journal such as the New England Journal of Medicine. Suppose this study gives evidence that a fetus in the early stages of its mother’s pregnancy can feel pain (or cannot feel pain). We are willing to bet that this true and objective study will appeal more to conservatives (liberals) than liberals (conservatives). We are also willing to bet that conservatives (liberals) would tend to cite it more.

This is all that our study assumes—that these studies can appeal differently to different sides of the political spectrum. We do not assume that the authors of the studies necessarily have a political agenda. Not only that, we do not even assume that each study will appeal differently to different sides
of the political spectrum. We only assume that it is possible that such studies will appeal differently.

This is an excellent response, as far as it goes, but it fails to address part of the concern about the study's methodology. Suppose that a think tank, call it the NEJM Association, had published a study giving evidence that a fetus in the early stages cannot feel pain. Suppose further that liberal legislators cited this study frequently and that neither conservative nor liberal legislators cited anything else published by the NEJM Association, making the NEJM Association seem to be a liberal think tank. Therefore, under the Grosecose-Milyo methodology, a newspaper that cited the NEJM Association frequently would be tagged a liberal newspaper. That would make sense, perhaps, if it could be shown that the newspaper was citing the NEJM study about fetal inability to feel pain. But what if the newspaper was citing other work by the NEJM Association -- for instance, studies on the connection between diet and cancer incidence, or on outcomes of obesity treatment. I would contend that it would be unfair to give this newspaper a "liberal" tag.

In think that something like this might be going on with the Grosesclose-Milyo study. Think tanks produce work that is political charged and work that isn't. I would contend that on average, liberal groups produce more non-policically charged work, work that is important for it's expertise and informational content. Conservative groups, I would argue, are on average more political, and more consistently partisan. I don't think that Groseclose and Milyo have adequately controlled for this effect.