Many flaws in comparative election study

Stop press! The Cambodia Daily, which modestly calls itself Cambodia’s “Daily Newspaper of Record”, has just rediscovered, and put on its front page (September 24 issue), a story that it first ran two and a half years ago.

Even when it was new, the story was not much news. A US-Australian academic project to grade the quality of elections around the world had released a report in 2014 giving Cambodia’s 2013 national election a low score. Poor countries with a difficult recent history aren’t as good at conducting elections as rich countries with a more comfortable history. Who would have thought it?

The academic project has not re-evaluated the 2013 election, and Cambodia hasn’t had another election since then, so one might ask: Where is the news?

Well, it must have been a slow Saturday, and someone at the Daily found a press release from the Electoral Integrity Project (EIP), the name chosen by the US and Australian academics to describe what they were doing. And since they evaluated the Cambodian election, a lot of other countries have had elections, and the EIP has decided that many of those later elections had more “integrity” than Cambodia’s 2013 election. Real front-page stuff.

Still, aside from its lack of news value, both EIP reports are guaranteed to be quoted repeatedly by anti-government politicians and NGOs, so it is perhaps worthwhile looking at what the study does and doesn’t do.

Perceptions and observation

A sentence earlier, I put the word “integrity” in quotation marks. I did so because the integrity of elections is not really what the EIP has been measuring. It would be more accurate for the study to be called the Electoral Impressions Project.

This is because the study did not directly observe any aspect of elections. Instead, it sent a questionnaire to “around 40” people in each country who, EIP believed, were “experts” on electoral matters there. On average, around 29% of them completed the questionnaire. These individuals were asked to give their “perception” of how well or badly different aspects of elections had been conducted.

Experts on Cambodia in this field seem to have been a bit more cooperative than the average. Still, only15 people (nine foreign and six Cambodian) gave their impressions of the 2013 election. EIP says it has no information as to whether the nine foreigners were in Cambodia during the voter registration, campaign and voting about which they gave their perceptions.

Obviously, it is much easier to collect the opinions of a small (or even a large) number of people than it is to collect objective data on a topic as complex and contested as elections – and especially so if the aim is to cover a large number of countries. Nevertheless, there are obvious problems with basing a judgment of the quality of elections on the impressions of only 15 people, no matter how expert they might be; an awareness of these problems ought to impose limits on the conclusions and generalities that are drawn from the study results.

For most if not all of the respondents, their perceptions on most or all points will have had to be formed mainly by what they hear or read others saying, not what they observe themselves. The EIP questionnaire was in English, so the respondents on Cambodia were able to read the Phnom Penh English-language press, which has usually been critical of Cambodia’s elections. The six Cambodian experts could also read the Khmer press, which might balance the English newspapers to some extent. We don’t know how many, if any, of the foreign experts read the Khmer press.

Form before content

A major problems with the EIP survey is that attempting to compare elections across 153 countries makes it all but impossible to put any of those elections into their local context. A comparison of elections among that many countries, when that is all that’s compared, is already grossly distorted before the first question is asked or the first impression is noted. To have some chance of being meaningful, a ranking of countries’ election integrity would have to be adjusted for many other factors, from degree of wealth or poverty, to cultural attitudes towards disagreement or competition, to recent history of social peace or turmoil or war.

One way to start doing that would be to compare each country’s election, not to the rest of the world, but to its own history: are its elections increasing or decreasing in integrity? Then country comparisons might tell us something that wasn’t obvious before the research began: instead of “developed countries’ elections are mostly perceived to have more integrity than the elections of poor countries”, we might learn things such as “Country A’s overall electoral performance is improving, while its neighbour Country B is backsliding”.

By reducing the measure of electoral integrity to questions about perceptions of the mechanisms of elections, the EIP survey substitutes form for content: how many of the survey’s 49 boxes were ticked pushes aside the question of whether an election accomplished the most important thing that an election ought to accomplish: did its result reflect the expressed will of the citizenry? – or to put it more simply: did the candidate or party with the most votes win the office that was being voted for?

The EIP questionnaire goes into things like media coverage that might influence how people vote, and that is certainly a subject worth investigating. But whether or not an election can be called democratic depends in the first instance on whether the candidate with the most votes wins. This question is not only central but also has the advantage of being objective: unless the vote count has been falsified, it is possible for anyone to see whether the person with the most votes won. But this question is not included in the EIP survey – not even in the form of whether experts have the perception that the person or party with the most votes was elected.

Nobody has ever brought a specific challenge to the official count of Cambodia’s 2013 election, which had the CPP winning around 300,000 votes more than the CNRP. (The CNRP claimed to the media that it had really won, but it never specified any vote totals from any province that were significantly different from the NEC’s totals.) But in some other countries, which have a higher ranking than Cambodia in the EIP study, it is not at all unusual for a person or party to be declared elected even though a different person or party has a higher number of officially reported votes.

Since the EIP is based in the United States and Australia (at Harvard University and Sydney University), let’s look just at those two countries. As is fairly well known, the US presidential election in 2000 was fiercely disputed, and the result was eventually decided by the US Supreme Court. What is perhaps not so well known is that Al Gore received around half a million more votes than George W. Bush, but Bush was elected. More recently, in the 2012 election for the US House of Representatives, the Democratic Party won 1.4 million more votes than the Republican Party, but the Republicans ended up with 234 seats and the Democrats with only 201.

Australia has a long record of minorities in the popular vote winning a majority in the House of Representatives, which determines the government. In 28 elections since the end of World War II, this has happened five times.

A closely related question is whether the votes of all citizens have the same degree of influence on the outcome of elections. Anomalies such as minorities in the vote winning a majority in the outcome are more likely to occur in countries that are federations, because equality between the states or provinces that make up the federation means inequality between the inhabitants of those states or provinces if those divisions do not have equal populations. (Both the United States and Australia are federations.)

Voting populations for members of the US House of Representatives are approximately equal in size, but in the equally powerful Senate, they are hugely different. There are two senators elected from each state, and state populations, according to the 2010 census, vary from 564,000 to more than 37 million: a Wyoming resident’s vote for the Senate is worth 65 times as much as the vote of a Californian.

Election of the US president is also biased against residents of populous states, although not to the same degree as in elections for the Senate. The president is elected by an Electoral College, whose members are what voters actually vote for. In California, it takes around 680,000 votes to elect one member of the Electoral College; in Wyoming it takes only 190,000.

In Australia, members of the House of Representatives are from districts of approximately equal population, but there are big differences in the number of voters represented by senators from different states. One senator from Tasmania represents a bit more than 41,000 people, while one senator from New South Wales represents more than 576,000 – a ratio nearly one to 14. Nationwide, less than 23% of the population elect a majority of the Senate.

In Cambodia, the requirement that every province have at least one member of the National Assembly also leads to discrepancies in the number of voters per seat, but the significant differences are only among the single-member provinces. Preah Sihanouk, Oddar Meanchey and Preah Vihear are under-represented, while Kep and Mondulkiri are over-represented. In the provinces with two or more National Assembly seats, the population per seat ranges from around 100,000 to 170,000. And because the single-member provinces elect only nine of 123 members of the NA, anomalies in those provinces are less likely to affect significantly the overall outcome of the election.

If the EIP survey had considered just these two questions, Cambodia’s ranking would have been closer to the rankings of the US and Australia.

Odd results

There were some obviously strange results of the study. For example, describing the first report, the Cambodia Daily (March 5, 2014) noted that Cambodia’s score was below that of Azerbaijan – “whose election body accidentally released some election results the day before the ballot had even taken place”! It is hard to imagine more conclusive proof that an election is rigged – so how can any other country rank below it, no matter how bad its other procedures may be perceived to be by EIP’s experts? Perhaps the answer is that the Azerbaijani experts are not very perceptive.

More likely, it lies in the EIP’s procedure. The questionnaire consists of 49 statements spread over 11 areas such as electoral laws, campaign finance and voting process. Respondents were asked to score the statements on a scale of 1 to 5, representing respectively strong disagreement, disagreement, neither agreement nor disagreement, agreement and strong agreement. Because some statements were positive and some negative, EIP reversed the response on negative statements, so that for all statements, a higher number represents greater perceived integrity. Each country’s overall score was then “calculated as an additive scale by summing up the scores of all 49 indicators, and standardizing the result to a scale from 0 to 100”. (This account of the scoring was kindly provided by EIP researcher Max Grömping, whom the Cambodia Daily in its September 24 article calls “Max Groening”.)

This means that all 49 statements have the same amount of influence on a country’s total score. It seems that the Azerbaijan experts would likely have strongly disagreed with statement 9-3, “Votes were counted fairly”. Since the vote counting in Cambodian elections is quite transparent, Cambodia probably scored higher than Azerbaijan on this point. But that was only 1/49 of the countries’ total scores. I don’t know anything about voting procedures in Azerbaijan, but if its elections include the possibility of postal ballots, internet voting, special voting facilities for the disabled or allowing citizens living abroad to vote, then Azerbaijan would have scored better than Cambodia, which has none of those things.

Each of those items is a separate question, so together in the EIP survey they far outweigh fair counting of the vote. In determining the integrity of an election, the survey presumes that the provision of postal ballots is as important as counting the vote fairly.

This is not the only example where the questionnaire appears unbalanced or problematic. The areas of electoral law, boundaries and voter registration each have three questions. Other areas have four, five or eight (“voting process”) questions. Are “campaign media” and “campaign finance” (each 5 questions) really 1 2/3 times as important as “voter registration” (3 questions) in determining the character of an election, but only 62% as important as “voting procedures” (8 questions) ? I have no idea, but the report would be more convincing if the EIP were to explain why it assigns such relative importance to the different areas.

Another peculiarity is that questions sometimes appear logically to duplicate others. For instance, the three questions on voter registration are:

“4-1 Some citizens were not listed in the register

“4-2 The electoral register was inaccurate

“4-3 Some ineligible electors were registered”

It is obvious that if either 4-1 or 4-3 is true, then 4-2 is necessarily also true, and if they are false, so is it. So why include it?

Similarly, the three questions on boundaries are:

“3-1 Boundaries discriminated against some parties

“3-2 Boundaries favored incumbents

“3-3 Boundaries were impartial”

If either 3-1 or 3-2 is true, then 3-3 is false, and vice versa, so there is no need to ask the specific question. Are such redundant questions included in order to check on whether respondents are being consistent? Or are they intended to boost the relative weight of these area in determining the overall score? Again, that is anyone’s guess.


Others of the 49 questions treat their topics as absolutely good or bad, when a more nuanced judgment seems appropriate. For instance, the Azerbaijan experts would presumably have to agree strongly with question 9-2 that “The results were announced without undue delay”, since the results came even before the voting. So Azerbaijan must have received maximum points for that question, although the answer does not really reflect well on that election’s integrity.

Or consider question 2-4, “Elections were conducted in accordance with the law”. That seems a good thing – until one looks at the questions in the section on electoral laws: “1-1 Electoral laws were unfair to smaller parties; 1-2 Electoral laws favored the governing party or parties; 1-3 Election laws restricted citizens’ rights”. If a country is marked down in section 1 for bad laws, it nevertheless is marked up in section 2 for enforcing those bad laws.

Or consider 6-5, “Social media were used to expose electoral fraud”. If a country got a low score on this, it might be for two quite different reasons: (1) social media were too afraid or too fooled to expose fraud; (2) users of social media perceived no significant electoral fraud.

And if I were asked to agree or disagree with statement 7-1, “Parties/candidates had equitable access to public subsidies”, I don’t know what I could answer. There weren’t any public subsidies for anyone, so that means I should disagree. On the other hand, if no one gets anything, that is equitable, so I should agree. (I take “public subsidies” as referring to official support, because a separate question refers to illegitimate use of state resources.)

Then there is statement 8-1, “Some voters were threatened with violence at the poll”. I and many other perceivers, even if we don’t qualify as experts, would have to strongly agree with that statement. An observer from the Heinrich Böll Foundation, the Guardian newspaper and Licadho observers reported instances in the 2013 election of violence or the threat of violence blocking would-be voters in S’ang Phnom commune, S’ang district, Kandal province; Traeuy Sla commune, S’ang district; Russey Keo district of Phnom Penh; Leuk Teik district of Kandal; and Preah Sdach district of Prey Veng, among other places. (For details, see

These incidents would have caused alert experts to give Cambodian elections a bad score on this point, lowering its overall score, which will certainly be used by the opposition CNRP and many NGO critics to say: “See! Proof that the government manipulated the election!”

But in every one of those incidents, it was mobs spurred on by CNRP leaders who committed the violence, usually to prevent voting by Cambodian citizens who were ethnically Vietnamese. A survey intended for 153 countries has difficulty including such unexpected realities.


Overall, the EIP studies may provide some useful data on perceptions of many aspects of elections in the countries surveyed. Those perceptions may or may not reflect parts of the reality, depending on the country concerned. But there seems little or no justification for summing those perceptions into a total country score in the way it was done, and even less reason to compare those scores to say that one country’s elections have more or less integrity than another’s.