Saturday, October 01, 2005
EAC Election Day Survey --- where's the voting system malfunctions data?
So yesterday I sat down to work with the new EAC Election Day Survey data ... and found that one of the components of the survey I was most interested in immediately analyzing, the data on voting equipment malfunctions (Chapter 11 of the report), only provides a very sketchy outline of the survey responses for that question. For this piece of the Election Day Survey, there are no data tables (i.e., no Excel spreadsheets) and no reporting of the data either at the state or jurisdiction level.
What's up here? Reading the report, we find that the question on "voting equipment malfunctions had by far the least coverage of any of the survey questions. Twenty-one states did not respond to the question or said that information on malfunctions was not available ... In all, we have information from only 485 of the 6,567 jurisdictions in the EAC database, or only seven percent of the country."
Why does this matter for not reporting the data in any more detail than a brief summary table? The report justified this decision by stating: "Due to the small number of responses to the question on voting equipment malfunctions, we did not create the standard data table that forms the basis for other chapters in this report." The report then gives a brief summary table, presenting the various voting system malfunctions by voting system type (Table 11a of the report).
It is difficult to understand this decision. Typically, survey researchers will not present data when there is a very low response rate for fear that it might jeopardize anonomity; in other words, when there is a low response rate in the typical survey, the risk in reporting the data is that someone might be able to figure out who provided which answers. Clearly this explanation does not hold water here, as the responses are not anonymous to begin with!
Otherwise, it is hard to see why this data is not available at the state and jurisdiction level.
Substantively, the results in Table 11a are interesting, and given their importance to the debate about voting technologies, external researchers really need access to this data at the jurisdiction level. We need to know which jurisdictions have provided these responses (and also which did not respond) to be able to understand the quality of the data here, and the extent to which we have useful data that we can rely upon to draw inferences from. We also need the more detailed data so that we can study where these malfunctions occurred in 2004, whether they are related to any other important election outcome variables, and whether there are any factors that might explain the nature and extent of the voting system malfunctions themselves.
What's up here? Reading the report, we find that the question on "voting equipment malfunctions had by far the least coverage of any of the survey questions. Twenty-one states did not respond to the question or said that information on malfunctions was not available ... In all, we have information from only 485 of the 6,567 jurisdictions in the EAC database, or only seven percent of the country."
Why does this matter for not reporting the data in any more detail than a brief summary table? The report justified this decision by stating: "Due to the small number of responses to the question on voting equipment malfunctions, we did not create the standard data table that forms the basis for other chapters in this report." The report then gives a brief summary table, presenting the various voting system malfunctions by voting system type (Table 11a of the report).
It is difficult to understand this decision. Typically, survey researchers will not present data when there is a very low response rate for fear that it might jeopardize anonomity; in other words, when there is a low response rate in the typical survey, the risk in reporting the data is that someone might be able to figure out who provided which answers. Clearly this explanation does not hold water here, as the responses are not anonymous to begin with!
Otherwise, it is hard to see why this data is not available at the state and jurisdiction level.
Substantively, the results in Table 11a are interesting, and given their importance to the debate about voting technologies, external researchers really need access to this data at the jurisdiction level. We need to know which jurisdictions have provided these responses (and also which did not respond) to be able to understand the quality of the data here, and the extent to which we have useful data that we can rely upon to draw inferences from. We also need the more detailed data so that we can study where these malfunctions occurred in 2004, whether they are related to any other important election outcome variables, and whether there are any factors that might explain the nature and extent of the voting system malfunctions themselves.
Friday, September 30, 2005
Election Day Survey Presentation
If you swing over to the Election Data Services website, you can download a copy of the presentation Kim Brace gave to the EAC about the Election Day Survey. It is full of facinating tidbits that we often forget about elections. One of the problems that exists in elections is that there is often a belief that every other jurisdiction is like ours or every jurisdiction runs elections the way they are run where we live. I remember when working for the National Commission on Federal Election Reform hearing a story about the elderly man from New York who voted in Florida for the first time. He stood in front of the punch card machine and waited. When the poll worker came over to ask him if he needed help, he said no, I am just looking for the way to make the curtain close (which, of course, is what happens with lever machines in New York).
The EDS presentation reminds us just what a bizarre mix of jurisdictions we have. For example, on slide 2 Kim notes:
The presentation has many other interesting items, but these should whet your appetite to examine the entire presentation. This will also help you slog through the multiple datasets that exist for the election day survey to figure out what interesting factoids need to be fully caveated!
The EDS presentation reminds us just what a bizarre mix of jurisdictions we have. For example, on slide 2 Kim notes:
- 1,778 of nations’ jurisdictions have fewer than 1,000 registered voters in them.
- Only 320 jurisdictions have more than 100,000 registered voters
- Only 18 jurisdictions have more than 1 million voters.
- Over 26% of election jurisdictions still use paper ballots.
- 26 states report “Active” voters only
- 20 states combine “Active” & “Inactives”
- 4 states leave determination to locals.
- Jurisdictions > 1 million voters: 21.6% Inactives
- Jurisdictions < 1,000 voters: 8.8% Inactives
The presentation has many other interesting items, but these should whet your appetite to examine the entire presentation. This will also help you slog through the multiple datasets that exist for the election day survey to figure out what interesting factoids need to be fully caveated!
Thursday, September 29, 2005
Intelligence updates: EAC provisional balloting reports available soon?
The EAC has been busy lately --- between the recent flurry of deadlines on a series of projects (the many "RFP's" we've blogged about in recent weeks) and the recent release of the Election Day Survey data and report, a lot has been going on at the EAC. But one thing that is in the works is the major study they initiated with the RFP that was distributed on March 1, 2005, "Providing research assistance to the EAC for the development of voluntary guidance on provisional voting and voter identification procedures." This RFP went to a collaborative team made up of researchers from the Eagleton Institute of Politics at Rutgers University and the Moritz College of Law at The Ohio State University. Inquiring minds will note that the original statement of work asked for a report on their provisional voting work in September 2005, and that the original statement of work ended the period of contract performance on October 28, 2005. That does mean that we might be seeing some of the data and analysis from this project in the near future!
More on Travis County "L+A" testing
As promised recently, here is more analysis of our visit to Travis County, Texas. We've got a few additional photos and some additional comments regarding the "logic and accuracy" testing we witnessed there last week.
First, here is a photo of the voting machines, stored in their cases and in a rolling rack, in their facility. Again, as we noted earlier, the voting machines had been moved into this location in their facility (typically they are spread through the space in the photograph, with aisles separating the racks) as a cautionary move to avoid any potential damage from Hurricane Rita. Travis County Elections Division Manager Gail Fisher is in the photo, and she was a source of a great deal of information during our visit.
Next, this is a photo of the "test voters" doing the L+A test, here the two people seated at the table. The man on the left is using the judge's station, the woman on the right has the Hart eSlate voting device in front of her (and she appears to be using the audio headset). The rest of the folks in the photo are either observers, election officials, or participants in the testing. Here we have another photo of the same testing team, but from another perspective.
Now we have a photo of both voting teams.
Then, we move to the room housing the computer system (not connected to any network) that read the voting data from the media taken out of the voting machines. The man at the computer here is reading the media card and generating the tabulation report for the L+A test.
To reiterate an observation we made in an earlier post about our Travis County visit, they have a strong physical security model in place. Access to the room where this mission-critical computer is located is highly restricted and logged, and again, the computer in this room was not connected to any network.
What we don't know is how physical security is implemented in other election jurisdictions; for example, we don't know of any systematic collection of information from election officials regarding what type of security models they utilize nor how they are implemented. Such data might be critical to collect in upcoming elections, as they can serve as important practical data for election officials to utilize, and could form the basis for some analysis of best practices in this area. Many data elements that might be important to collect regarding these types of security issues can be seen in a report the Caltech/MIT VTP issued last summer on "Insuring the Integrity of the Electoral Process."
First, here is a photo of the voting machines, stored in their cases and in a rolling rack, in their facility. Again, as we noted earlier, the voting machines had been moved into this location in their facility (typically they are spread through the space in the photograph, with aisles separating the racks) as a cautionary move to avoid any potential damage from Hurricane Rita. Travis County Elections Division Manager Gail Fisher is in the photo, and she was a source of a great deal of information during our visit.
Next, this is a photo of the "test voters" doing the L+A test, here the two people seated at the table. The man on the left is using the judge's station, the woman on the right has the Hart eSlate voting device in front of her (and she appears to be using the audio headset). The rest of the folks in the photo are either observers, election officials, or participants in the testing. Here we have another photo of the same testing team, but from another perspective.
Now we have a photo of both voting teams.
Then, we move to the room housing the computer system (not connected to any network) that read the voting data from the media taken out of the voting machines. The man at the computer here is reading the media card and generating the tabulation report for the L+A test.
To reiterate an observation we made in an earlier post about our Travis County visit, they have a strong physical security model in place. Access to the room where this mission-critical computer is located is highly restricted and logged, and again, the computer in this room was not connected to any network.
What we don't know is how physical security is implemented in other election jurisdictions; for example, we don't know of any systematic collection of information from election officials regarding what type of security models they utilize nor how they are implemented. Such data might be critical to collect in upcoming elections, as they can serve as important practical data for election officials to utilize, and could form the basis for some analysis of best practices in this area. Many data elements that might be important to collect regarding these types of security issues can be seen in a report the Caltech/MIT VTP issued last summer on "Insuring the Integrity of the Electoral Process."
Wednesday, September 28, 2005
Information from NASS on the situation in Mississippi, Alabama, and Louisiana
The following message was sent out by Leslie Reynolds, Executive Director, National Association of Secretaries of State last week:
We'll continue to update our blog with information on this situation as we receive it.
Wednesday, I had the privilege of meeting with the EAC, our colleagues at NCSL, NACRC, IACREOT, the Election Center and NASED, congressional staff, Homeland Security and the Justice Department and most important of all:
- Sec. Eric Clark of MS, along with two of his staff members Jay Eads and Chuck Bearman;
- Angie LaPlace, the LA Commissioner Elections, and staff member Sherrie Hadskey representing Sec. Al Ater
- Janice McDonald, the AL Director of Elections, representing Sec. Nancy Worley
Everyone came together to determine the immediate and long term election-related needs of the areas impacted by Hurricane Katrina. And most importantly, how we all might be able to coordinate efforts to assist areas impacted.
State of Mississippi
Sec. Clark of MS said that while they do not have the water damage suffered by parts of AL and much of LA, many MS counties have experienced a complete loss of infrastructure. Sec. Clark said that because HAVA funding was not appropriated by the federal government for the final authorized year, MS was already facing a serious problem securing the number of voting machines needed by each county. The counties were going to have to undertake a significant financial burden to support the mandates of HAVA, but now with many counties devastated by the Hurricane, he doesn't see how they will manage. He reported that Katrina destroyed infrastructure, voting machines and the buildings that served as polling places.
If there is some good news, it is that MS is in the process of implementing their statewide voter registration database and the counties in the southern part of the state have been entered into that database. The bad news is that many of the original records in that area have been damaged or destroyed. However, Sec. Clark did say that unlike LA, he thought that many of the residents of MS were able to remain in the state and that finding MS voters may not be as difficult as it will be elsewhere.
Additionally, MS is the process of implementing a 'nearly statewide voting system." The counties had the option to opt in or out of a statewide contract for equipment. The overwhelming majority of counties did opt in to the contract, so they will be getting new voting equipment shortly. The ones that didn't opt-in may still have the opportunity to do so. However, the counties were going to have to shoulder a large financial obligation because of incomplete HAVA funding, now they also have the expense of rebuilding their communities.
Sec. Clark reported that he was very concerned about the HAVA deadlines and hoped that in addition to full funding, Congress (and DOJ) would take Hurricane Katrina into consideration when those deadlines approached.
Sec. Clark also reported that MS has local elections in September and October of this year and there was uncertainty about the authority to postpone those elections. He asked that we provide some information on state laws that do exist. I explained that some of the groups present had looked at this issue last year prior to the Presidential election and that the information was available. Below you will find a link to a September 2004 CRS paper that looks at the issue of emergency powers to postpone elections in the states. Please keep in mind that this paper was done at this time last year, so if something was done in your last legislative session to address this, please let us know and we will update the information.
http://www.fas.org/sgp/crs/RS21942.pdf
http://michiganimc.org/newswire/display/7370/index.php
State of Alabama
Janice McDonald, AL State Elections Director, reported that many counties in AL are still recovering from Hurricane Ivan and are now dealing with the destruction caused by Katrina. She stated that six counties on the western side of the state were impacted with damage to equipment and polling places. She also reported that many areas are still inspecting the damage so there isn't a complete reporting. She said that AL does have many misplaced voters so major efforts will have to be put in place to remind residents to update their voter registration information.
State of Louisiana
Angie La Place, the Louisiana Commissioner of Elections, reported that they will have to delay the implementation of their statewide voting system. She too was concerned about the HAVA deadlines and the challenges they will face in the coming months. She reported that they lost over 2000 voting machines (all the equipment in New Orleans was electronic), 500 polling places (many sites are in people's homes), and volumes of original records. They have a statewide voter database in place, but the database doesn't include a digitized signature and LA law requires matches with signature records. She and Sherry Hadskey reported that because much of what needed to be inspected was either still underwater or contaminated and off limits to anyone, they really don't have a complete picture on the damage. FEMA had indicated to LA that they will replace the voting equipment that was destroyed. That surprised most people in the room. They are worried about the ability of vendors to replace the volume of equipment lost and even if they were, equipment storage facilities have been destroyed. Construction crews are not being allowed in and building supplies aren't available.
LA is considering the use of Vote Centers and "vote by mail," however they face a daunting challenge of locating all of their displaced voters to provide information on requesting an absentee ballot or re-registering to vote where they now reside. The Post Office is working with them and FEMA will develop a database of people – where their original residence was and where they are currently staying -- that will help track down voters. She did report that she was already starting to receive voter registration applications in the mail. They are also looking into possibilities of voting sites like large box store chains or grocery stores. They were encouraged to work with civil rights and disability groups and reconvening their HAVA planning committee in order to include all interested parties in the disaster recovery process.
LA has already posponed some of their local elections, but they have an election scheduled in February in New Orleans. There were 300,000 voters in New Orleans who are now displaced. They currently have no polling places and no way to reach out to voters or poll workers. LA will need assistance in February if that election is held. Volunteers will be needed to help man the polling places and deal with the volume of mail generated by the increase in vote by mail.
The group gathered also heard two presentations, one from David Leahy, former election official from Miami-Dade County and one from Pat Owens, former mayor of Grand Forks, ND. Both gave first hand accounts of disaster recovery, working with FEMA, length of time before some sense of normalcy returns, and lessons learned.
The national organizations present said that many of their members (including ours) are waiting with offers of assistance, but wanted to find out what the needs were and how those offers should be coordinated. LA, AL, and MS each agreed to appoint a contact person in their state to coordinate offers of assistance. I can be the point of contact outside the state that assists that person.
Please let me know if you are able to provide assistance in any of the following areas:
- Records retrieval & restoration - people with expertise and necessary equipment. Quite a few of the Secretaries have Archives as part of their responsibility in the state, so hopefully some of you will have suggestions on water-damaged records.
- Expertise on establishing vote-by-mail programs (including statutory language)
- Expertise on establishing vote centers, satellite voting sites (including statutory language)
- Volunteers to help staff voting sites in February for the election in New Orleans
- Suggestions for ways to reach out to voters displaced by the storm with info on absentee voting or re-registering in new location- there was some discussion of a public service campaign on which NASS and other groups might be able to assist.
We'll continue to update our blog with information on this situation as we receive it.
Score Cards for States
In The New Republic online, Heather Gerken wrote an article in which she recommends that the EAC rank all states and localities using the election day survey data. Her basic argument is that:
In response to Gerken's opinion piece, Michael McDonald, who helped collect and evaluate the data, wrote in the electionlaw listserv that:
Michael's point about missing data is relevant, but for those jurisdictions who did not complete the data, being ranked last might make them take these surveys seriously. Of course, the other alternative is that localities may never complete another survey! These data are critical for understanding and approving elections; why after 2000 and 2004 would a jurisdiction think that not completing an EAC survey was a good idea? (After all, I doubt these jurisdictions are turning down EAC dollars!)
Hopefully, the survey will also encourage jurisdictions to keep better data. One problem here is there are not standard definitions for election facts. Here, the EAC needs to develop standard definitions for election data, something Mike and I continue to promote.
One effective use of these data would be for states to use them to identify local jurisdictions that need help--additional training, additional resources, etc.--and help these jurisdictions improve. And as Michael points out in his listserv posting, researchers do need time to see what these data actually tell us about elections. We currently know far too little about a wide range of election activities. This survey gets us on the right track, but we need even more studies to improve things before 2008.
One question about doing rankings is that it may focus election officials on the wrong factors. For example, there is nothing in the survey data about training, quality of poll workers, and similar human factors that are key to elections. These small things may be the critical factors for elections. (I have just completed a survey that suggests that poll workers are the key to publc confidence in elections). The worst outcome would be the right idea--holding people accountable--creating the bad outcome.
Now the EAC should use its data to create explicit, comprehensive rankings of states and localities, shaming those local governments that are doing a poor job of running elections and rewarding those that are excelling. It's a strategy taken directly from the playbook of some human rights and environmental organizations, which have long used rankings to prod nations into improving their practices. The strategy works for a simple reason: No one wants to be at the bottom of a list.In short, rankings produce improvement and would force election officials to improve their game. Gerken suggests that the report would become sort of a high-stakes "No Child Left Behind" test, and everyone would know your score.
In response to Gerken's opinion piece, Michael McDonald, who helped collect and evaluate the data, wrote in the electionlaw listserv that:
As a voluntary survey, not all local election jurisdictions provided data on all question items to the EAC. Some questions have better coverage than others and I have some confidence in the patterns that we observe among those jurisdictions that did report as they are often similar to other academic work. But these data are not a perfectly valid snapshot of the 2004 election.Both positions have some merit. Obviously, now that the data are available in Excel spreadsheets, which have a "sort" function, states will be ranked. This is just a fact of life. The key question is, what to do with these data?
Furthermore, states and even local jurisdictions vary their definitions on basic things such as what constitutes a poll worker or a polling place. Let me take just one example that came up in discussions with commissioners after the EAC meeting: Maine counted 100% of its provisional ballots cast. However, Maine has election day registration and uses provisional ballots only in instances where a voter is contested at the polls. This was a rare event in Maine, and thus there were few provisional ballots cast in the state. Thus, to be meaningful, ranking the states on percent provisional ballots counted would need to take into account the varying definitions of what constitutes a provisional ballot and under what circumstances they are counted.
Michael's point about missing data is relevant, but for those jurisdictions who did not complete the data, being ranked last might make them take these surveys seriously. Of course, the other alternative is that localities may never complete another survey! These data are critical for understanding and approving elections; why after 2000 and 2004 would a jurisdiction think that not completing an EAC survey was a good idea? (After all, I doubt these jurisdictions are turning down EAC dollars!)
Hopefully, the survey will also encourage jurisdictions to keep better data. One problem here is there are not standard definitions for election facts. Here, the EAC needs to develop standard definitions for election data, something Mike and I continue to promote.
One effective use of these data would be for states to use them to identify local jurisdictions that need help--additional training, additional resources, etc.--and help these jurisdictions improve. And as Michael points out in his listserv posting, researchers do need time to see what these data actually tell us about elections. We currently know far too little about a wide range of election activities. This survey gets us on the right track, but we need even more studies to improve things before 2008.
One question about doing rankings is that it may focus election officials on the wrong factors. For example, there is nothing in the survey data about training, quality of poll workers, and similar human factors that are key to elections. These small things may be the critical factors for elections. (I have just completed a survey that suggests that poll workers are the key to publc confidence in elections). The worst outcome would be the right idea--holding people accountable--creating the bad outcome.
Tuesday, September 27, 2005
EAC Releases Election Day Report
This morning, the U.S. Election Assistance Commission (EAC) has released the 2004 Election Day Survey, which the EAC calls "the largest and most comprehensive survey on election administration ever conducted by a U.S. governmental organization." Mike and I have looked through the tables and would say that this claim is quite true.
For state level data, the Election Day Survey truly is a comprehensive document and will allow for election geeks and policy wonks to crunch numbers until their hearts are content. So when the EAC chair Gracia Hillman says,
She is definately hitting the nail on the head. The EAC should be given kudos for getting this report done and for putting it out so that it can be used and analyzed. (We would also send out kudos to Kim Brace and Election Data Services for their work on this effort!)
The key is to look at the state jurisdictional level data. This will allow us to study an array of election phenomena--turnout, early voting, absentee voting, turnout, and services to individuals with disabilities immediately come to mind--at the point where election services are provided and election administration matters most. The state level data are likely hiding variations across jurisdictions in a state where election services are excellent in some places and are lousy in others. Second, the jurisdictional data will force poor performing jurisdictions to step up and improve their game.
Here are some additional factoids from the EAC press release about the 2004 election:
Key 2004 Election Facts (As reported by the states)
60.7 percent voter turnout rate (based on citizen voting age population)
96.9 percent of absentee ballots were counted
64.5 percent of provisional ballots were counted
Lowest presidential drop-off rate since World War II at 1.02 percent
For state level data, the Election Day Survey truly is a comprehensive document and will allow for election geeks and policy wonks to crunch numbers until their hearts are content. So when the EAC chair Gracia Hillman says,
The 2004 Election Day Survey tells us a great deal about voting and elections practices throughout the country. The survey provides critical statistics on voter registration and turnout, voting equipment and locations, and other information about the voting process. All of this will assist us in making certain that every vote is counted fairly and accurately.
She is definately hitting the nail on the head. The EAC should be given kudos for getting this report done and for putting it out so that it can be used and analyzed. (We would also send out kudos to Kim Brace and Election Data Services for their work on this effort!)
The key is to look at the state jurisdictional level data. This will allow us to study an array of election phenomena--turnout, early voting, absentee voting, turnout, and services to individuals with disabilities immediately come to mind--at the point where election services are provided and election administration matters most. The state level data are likely hiding variations across jurisdictions in a state where election services are excellent in some places and are lousy in others. Second, the jurisdictional data will force poor performing jurisdictions to step up and improve their game.
Here are some additional factoids from the EAC press release about the 2004 election:
Key 2004 Election Facts (As reported by the states)
60.7 percent voter turnout rate (based on citizen voting age population)
96.9 percent of absentee ballots were counted
64.5 percent of provisional ballots were counted
Lowest presidential drop-off rate since World War II at 1.02 percent
EAC Election Day Survey jurisdiction-level availability
To update my earlier post, the EAC "Election Day Survey" jurisdiction-level data is available, though not terribly easy to find (and it is available in html tables!). If you navigate to the following link, you will find the State Data Tables. Then if you navigate to specific states, you can obtain the jurisdiction-level data. Perhaps at some point someone will move this data into a more user-friendly format for analysis, and if so, we'll let you know where that data resides.
Update (September 29, 2005; 1:45pm): Excel spreadsheet links are available for these data.
Update (September 29, 2005; 1:45pm): Excel spreadsheet links are available for these data.
EAC "Election Day Survey" underscores need for better data retention, collection and distribution practices
The just-released and complete EAC "Election Day Survey" report has an enormous amount of information in it, and will certainly keep election researchers busy for a long time (especially if and when the EAC releases the jurisdiction-level data).
But before anyone rushes head-first into number crunching mode, much caution is warrented. In particular, users of the report and the tabulated data must read the material on survey coverage, contained in pages 6-8 of the full report. The coverage rate varies dramatically across the survey questions in the "Election Day Survey", for example, 98.8% of jurisdictions appear to have provided responses to item 2a, "Ballots counted." But only 67.4% provided information for item 10a, "Presidential undervotes", and only 18.9% for item 11a, "Presidential overvotes" (both of the latter are likely to be heavily used data by researchers and policymakers).
However, what we really don't know here in the tabulated data (at least I've not been able to find it) is a sense for which jurisdictions are not reporting data, either across the survey instrument or for specific questions. I hope that once the jurisdiction-level data is available, this can be studied in more detail, as one wonders if there isn't some form of selection bias going on in the data; for example, are jurisdictions that tend to perform poorly on some of these dimensions either not reporting data or are they under-resourced and thus unable to retain and provide the data? If so, such selection bias can distort, or even bias, analyses of the "Election Day Survey" data.
One other issue, again that will have to await the distribution of the jurisdiction-level data from this survey, is whether there are statistical procedures that can be used to either "plug the gaps" in the survey data, or to study their possible influence on survey results. The former strategy might be quite promising, as there has been a great deal of progress in recent years regarding how to deal with missing data, using some quite easy and effective new software tools (see, for example, Gary King's material on his missing data estimation and analysis tool, Amelia). Hopefully, once the jurisdiction-level data is released, either the EAC or some other entity might go ahead and provide datasets that have the missing data "imputed", in a manner like the U.S. Census Bureau often releases data; this will certainly help end-users of the EAC "Election Day Survey" data.
Finally, the EAC should be congratulated for taking on this task; many ( including the Caltech/MIT Voting Technology Project) have called upon the EAC to work to collect just this type of data.
In conclusion, after a very, very quick read of the 266 page report:
Time to roll up the sleeves and get to work!
11:43AM update: note --- the jurisdiction-level data is available, please see subsequent posting for details.
But before anyone rushes head-first into number crunching mode, much caution is warrented. In particular, users of the report and the tabulated data must read the material on survey coverage, contained in pages 6-8 of the full report. The coverage rate varies dramatically across the survey questions in the "Election Day Survey", for example, 98.8% of jurisdictions appear to have provided responses to item 2a, "Ballots counted." But only 67.4% provided information for item 10a, "Presidential undervotes", and only 18.9% for item 11a, "Presidential overvotes" (both of the latter are likely to be heavily used data by researchers and policymakers).
However, what we really don't know here in the tabulated data (at least I've not been able to find it) is a sense for which jurisdictions are not reporting data, either across the survey instrument or for specific questions. I hope that once the jurisdiction-level data is available, this can be studied in more detail, as one wonders if there isn't some form of selection bias going on in the data; for example, are jurisdictions that tend to perform poorly on some of these dimensions either not reporting data or are they under-resourced and thus unable to retain and provide the data? If so, such selection bias can distort, or even bias, analyses of the "Election Day Survey" data.
One other issue, again that will have to await the distribution of the jurisdiction-level data from this survey, is whether there are statistical procedures that can be used to either "plug the gaps" in the survey data, or to study their possible influence on survey results. The former strategy might be quite promising, as there has been a great deal of progress in recent years regarding how to deal with missing data, using some quite easy and effective new software tools (see, for example, Gary King's material on his missing data estimation and analysis tool, Amelia). Hopefully, once the jurisdiction-level data is released, either the EAC or some other entity might go ahead and provide datasets that have the missing data "imputed", in a manner like the U.S. Census Bureau often releases data; this will certainly help end-users of the EAC "Election Day Survey" data.
Finally, the EAC should be congratulated for taking on this task; many ( including the Caltech/MIT Voting Technology Project) have called upon the EAC to work to collect just this type of data.
In conclusion, after a very, very quick read of the 266 page report:
- Thanks to the EAC for undertaking this project, and we call upon them to continue this effort in future federal election cycles.
- The jurisdiction-level data should be released for further research and analysis, hopefully with some work done to better understand the impact that survey nonresponse has in this sruvey, and to patch the missing data in the existing dataset.
- Users of the data in the current form should be very, very careful of the coverage problems, and should be on notice that their inferences from this data on survey questions with low coverage rates might be distorted or incorrect due to the nonresponse problem.
Time to roll up the sleeves and get to work!
11:43AM update: note --- the jurisdiction-level data is available, please see subsequent posting for details.
Monday, September 26, 2005
Before the storm --- election day preparations
Thad and I traveled to Travis County, Texas, on Thursday, where (once we worked out way through the flow of folks leaving Houston in advance of Hurricane Rita) we spent considerable time with Travis County Clerk Dana DeBeauvoir and her election administration staff, and elections officials from the Texas Secretary of State's office. Travis County is preparing for their November elections, and the primary things we did on Thursday were to observe their "logic and accuracy" testing of their voting systems, see how they handle by-mail ballots, and have an extensive discussion about the security procedures they have in place for their electoral process.
We'll likely have much more to say about what Travis County is doing, as there are many interesting things we learned on Thursday.
Now, there are a couple of observations about the "logic and accuracy" testing we saw and studied.
First, Travis County has provided some details of their "logic and accuracy" (L+A) testing procedures, as well as the results from their May 2004 testing in electronic format; useful stuff for those interested in getting a quick introduction to what their L+A procedures include. If they update their page to include the L+A results from the test we observed on Thursday, we will update these links.
Here is a photograph of the basic setup of the L+A test, showing the Hart InterCivic ESlate voting machines and their associated judge stations; click here for the photo. And next, here is a photo of the temporary county employees who "cast" the ballots in this L+A test; click here. Last, here is a picture of the tallying process of the L+A testing; click here.
The voting machines were set up with the upcoming ballot for the fall elections, and the testers were voting set combinations to study the operation of the voting systems themselves as well as the ballot logic. The tabulation system was then used to count the ballots cast (in known patterns), thus testing that part of the voting system as well. All we observed yesterday directly was the L+A test, and it was impressive to see the commitment by Travis County to get it done right. The round of the test that we observed ended up producing some minor inconsistencies, which arose because the test voters did not cast some of the ballots as provisionals according to the script. That led them to replicate the test, to get it right.
Second, the L+A testing that we observed yesterday is part of a larger model of security in Travis County that we'll talk more about later. But one piece of that security model that we also saw on Thursday was strong physical security. The Travis County election office has a strong security model, involving keypad access to the various parts of the facility where election administration occurs, security cameras in key parts of the facility, and compartmentalization of the election administration process.
Third, part of physical security involves something we have been talking about recently, which is disaster recovery and disaster preparation. Not surprisingly, the Travis County folks on Thursday were undergoing planning for Hurricane Rita, and a white board in a conference room that we used for a meeting was covered with hurricane-related disaster scenarios and mitigation strategies. One of those strategies we got a photo of; note here that they have all of their voting systems in their storage facility moved to a part of the room that they think will help minimize potential wind, rain, or water damage (apparently typically the voting systems are stored with aisles separating them); click here.
In general, site visits like this are very helpful for those who study election administration and voting technology, as you get to see first hand the complexity of the administrative process and the security procedures in place to help safeguard the integrity of the elections process. No doubt, there are always things that can be done to improve the administrative process, and it is helpful when election officials like those in Travis County welcome our observation, study, and questions.
We'll likely have much more to say about what Travis County is doing, as there are many interesting things we learned on Thursday.
Now, there are a couple of observations about the "logic and accuracy" testing we saw and studied.
First, Travis County has provided some details of their "logic and accuracy" (L+A) testing procedures, as well as the results from their May 2004 testing in electronic format; useful stuff for those interested in getting a quick introduction to what their L+A procedures include. If they update their page to include the L+A results from the test we observed on Thursday, we will update these links.
Here is a photograph of the basic setup of the L+A test, showing the Hart InterCivic ESlate voting machines and their associated judge stations; click here for the photo. And next, here is a photo of the temporary county employees who "cast" the ballots in this L+A test; click here. Last, here is a picture of the tallying process of the L+A testing; click here.
The voting machines were set up with the upcoming ballot for the fall elections, and the testers were voting set combinations to study the operation of the voting systems themselves as well as the ballot logic. The tabulation system was then used to count the ballots cast (in known patterns), thus testing that part of the voting system as well. All we observed yesterday directly was the L+A test, and it was impressive to see the commitment by Travis County to get it done right. The round of the test that we observed ended up producing some minor inconsistencies, which arose because the test voters did not cast some of the ballots as provisionals according to the script. That led them to replicate the test, to get it right.
Second, the L+A testing that we observed yesterday is part of a larger model of security in Travis County that we'll talk more about later. But one piece of that security model that we also saw on Thursday was strong physical security. The Travis County election office has a strong security model, involving keypad access to the various parts of the facility where election administration occurs, security cameras in key parts of the facility, and compartmentalization of the election administration process.
Third, part of physical security involves something we have been talking about recently, which is disaster recovery and disaster preparation. Not surprisingly, the Travis County folks on Thursday were undergoing planning for Hurricane Rita, and a white board in a conference room that we used for a meeting was covered with hurricane-related disaster scenarios and mitigation strategies. One of those strategies we got a photo of; note here that they have all of their voting systems in their storage facility moved to a part of the room that they think will help minimize potential wind, rain, or water damage (apparently typically the voting systems are stored with aisles separating them); click here.
In general, site visits like this are very helpful for those who study election administration and voting technology, as you get to see first hand the complexity of the administrative process and the security procedures in place to help safeguard the integrity of the elections process. No doubt, there are always things that can be done to improve the administrative process, and it is helpful when election officials like those in Travis County welcome our observation, study, and questions.
