There is a wonderful new research paper, recently electronically published in the journal Political Analysis, by Gary King (Harvard University) and Langche Zeng (University of California, San Diego). This paper, “The Dangers of Extreme Counterfactuals”, focuses attention on an important, but little-discussed, issue in social science that has significant implications for research on election reform.

Here is why election reform researchers should pay attention to the King-Zeng paper. By nature, election “reforms” require changes to some existing practice or procedure (for example, some feature of ballot design); and researchers are often called upon to speculate on the ultimate effect that these changes will have on an outcome variable, or a set of outcome variables (say, balloting mistakes by voters). In some cases, researchers are lucky, and can locate data from some other jurisdiction that currently uses the procedure or process that is the subject of reform — and they can then use that data to try to estimate the possible effect of the reform in the jurisdiction in question.

But in many cases we are not so lucky. There may simply be no jurisdictions that currently undertake the process or procedure in question, or those jurisdictions might be quite different from the jurisdiction in question (for example, they might be in different nations). Or, there may be only a very small number of jurisdictions that undertake the process or procedure, so there is very little data upon which to base our estimate of the possible effect of the reform in the jurisdiction in question.

As King and Zeng argue in their paper, the problem that then arises is that the prediction that is made about the possible effect of the reform might be based on little or no information in the data being used to make that prediction. King and Zeng offer a proof that shows that the further from the actual data some prediction is made, the more dependent the inferences will be upon the statistical model and the assumptions of that model. King and Zeng also provide some software that researchers can use to test for such model dependence.

This is a paper that is likely to open up some new lines of methodological research, as there are many other issues associated with counterfactual analyses that have yet to be explored. The methods here, and those that will be developed in subsequent research, ought to be used when appropriate by those trying to determine the potential effects of policy change, including election reform.

Here is the abstract of the paper, if you are interested in more details before making the commitment to downloading and reading the paper:

We address the problem that occurs when inferences about counterfactuals—predictions, ‘‘what-if’’ questions, and causal effects—are attempted far from the available data. The danger of these extreme counterfactuals is that substantive conclusions drawn from statistical models that fit the data well turn out to be based largely on speculation hidden in convenient modeling assumptions that few would be willing to defend. Yet existing statistical strategies provide few reliable means of identifying extreme counterfactuals. We offer a proof that inferences farther from the data allow more model dependence and then develop easy-to-apply methods to evaluate how model dependent our answers would be to specified
counterfactuals. These methods require neither sensitivity testing over specified classes of models nor evaluating any specific modeling assumptions. If an analysis fails the simple tests we offer, then we know that substantive results are sensitive to at least some modeling choices that are not based on empirical evidence. Free software that accompanies this article implements all the methods developed.