Six Thoughts on Election Audits: Reflections on the December 2018 Audit Summit at MIT

The Caltech/MIT Voting Technology Project hosted an Election Audit Summit last month that was interesting, fun, and a success (if I say so myself).  With the holidays now over, it thought I would post my initial reactions to the issues raised at the event.

If you want to experience what happened, the conference’s website hosts an archive of the webcast of all the presentations.  The presentation slides are posted there, as well.

While much of the day-and-a-half event focused on risk-limiting audits (RLAs), the conference covered many other topics as well, such as auditing voter lists, ballots, and the assignment of voters to legislative districts.  Attendees covered a wide variety of perspectives — academics, state and local election officials, interested citizens, and committed voter verification activists. In other words, there were people in the room who don’t talk to each other much, and even have been known to talk past each other.

My initial reaction to the event are summarized in these five thoughts.

  1. RLAs have come a long way.
  2. RLAs have a long way to go.
  3. Comprehensive auditing is key.
  4. Auditing isn’t the only way to ensure election integrity.
  5. We need more dialogue like this.

Let me expand on these points a bit.

RLAs have come a long way. 

RLAs have come a long way in the decade-plus period they have been part of the election reform landscape.  RLAs have come a long way along many dimension — theoretical and mathematical justification, practical application, and widespread acceptance.

One of the reasons for this progress is that the intellectual pioneers of RLAs, such as Philip Stark and Ron Rivest, have invested precious time and energy to meet with officials, help run pilots, and develop publicly accessible software tools.

Progress has been made because states, like Colorado, have invested in passing laws, buying equipment, and rearranging how they run elections, so that RLAs are integrated into everyday election administration.  Other states, like Rhode Island and Virginia, have passed laws setting them on a path to eventually following Colorado.

In addition, intrepid state and local officials in places like California, Indiana, Michigan, and New Jersey have sanctioned small-scale pilots.  Finally, there’s a scientific/policy consensus that RLAs are a goal to aim for over the next decade, reflected in the recent report by the National Academies of Science, Engineering, and Medicine on making voting more secure.  (Fair warning: Rivest and I were co-authors of that report.)

RLAs have a long way to go

Despite the great strides over the past decade to build consensus around the value of RLAs, and to make them practical to implement, there’s a lot more work to be done.  At the audit summit, we heard statements and questions such as these:

  • RLA still needs its elevator speech. It remains too complicated an idea to explain to a typical state legislator, who will give you five minutes to make your case.
  • The role of the human element still needs to be established and managed. Tons of details need to be managed in implementing RLAs and humans are bad at keeping track of lots of tedious details.  What is the role of software in managing the process, and how do we know to trust it?
  • Which contests should be audited? Eventually, all elections should be subjected to a risk-limiting audit, but as the technique is being developed, should we focus on top-of-the-ballot offices or down-ballot offices?
  • How much trust needs to be put into the math, and by whom? Although techniques can be put into place that reduce the process of conducting an RLA to a matter of arithmetic, that arithmetic is based on statistical theory. Who vets the theory?  Who sets the standards?
  • Bayesianism vs. frequentism. The canonical RLA models derive from a traditional frequentist approach to statistics.  New approaches rely on Bayesian  Who choses between approaches?  How do RLA laws get written, in a context that the science is still developing?
  • How will RLAs interface with other election-challenge processes? Will the expectation that RLAs be conducted as a part of the canvass process conflict with the goal in close elections to keep ballots undisturbed until the recount? As a general matter, RLA laws may interact with election contest in unanticipated and complex ways.

Comprehensive auditing is key

Risk-limiting audits aren’t the only game in town.  In fact, in my own view — which is not shared by others — RLAs may end up being a bit player in the drive to increase the degree of quality assurance in elections.  At the very least, post-election audits of the vote tally touch on just one aspect of elections where the details matter, mistakes can be easily made, and malicious actors can penetrate.

As I noted in my remarks that kicked off the conference, developments since the 2016 election have illustrated that American elections are operating at increasingly close tolerances; mistakes can have material effects on the outcomes of elections.  Just consider the evidence provided by the breach of the Illinois voter-registration system, the omission of 118,000 voters from the rolls in Los Angeles County during the 2018 primary, the mis-assignment of voters in the 2017 Virginia state election that may have affected the partisan composition of the House of Delegates, and the ballot-design fiasco in Broward County this year that affected the U.S. Senate race in Florida.

Many of the speakers at the conference touched on work they are doing to ensure that voter registration systems, ballots, and the assignment of voters to districts are handled properly.  This was the focus of the first Saturday morning panel, which can be seen by going to this link and choosing the Day 2 AM session # 1.  My favorite line from this very good panel was when Whitney Quesenbery remarked that “post-election auditing is only as good as the paper trail is it based on.”

This idea, that we need more than just RLAs, is related to an argument made by one of the fathers of RLAs, Philip Stark, who, along with David Wagner, has made the case for “evidence-based elections.”  Still, the evidence-based election idea has tended to focus on issues related to tabulation and record-keeping about the tabulation, and I think the horizons need to be expanded even further.

While on the issue of expanding horizons, I also should add that two panelists, Emily Beaulieu and Walter Mebane, reminded us that we can learn a lot about election integrity by considering elections in other countries.  The tools they discussed are particularly relevant when access to the ballots is precluded.  Walter, of course, is well known for developing tools, such as the Election Forensics Toolkit and Guide, that help to conduct forensic analysis of election results, by tagging data anomalies.  Emily wrapped up her presentation by noting that in the polarized societies she studies, sometimes social peace is bought by not being as transparent as Americans would like their elections to be.

We need more dialogue like this

As an observer of election administration trends, it is clear that the world is gravitating toward greater accountability in election administration.  One indicator of this trend is the movement toward requiring more sophisticated audits of election.  The challenge, as I’ve suggested in this piece, is that more sophisticated vote tabulation audits are still in their infancy, both theoretically and practically, and that other forms of quality control are even further behind.

The area of election auditing will only advance with the close collaboration of election officials and technically inclined outsiders, both inside and outside academia.

Absent that collaboration, election administration risks making the outcome of elections prey to errors and malicious attacks; the technically inclined also risk looking like arrogant scolds.

The most heartening aspect of the audit summit is that people with a diversity of opinions, backgrounds, and capabilities could spend two days listening to and learning from each other.  I’m hoping the audit summit can become a regular event.  We have a lot of work still to do.