Using Mathematical Methods for Electoral Fraud Detection

Buzin A.Yu., Grishin N.V., Kalinin K., Kogan D.L., Korgunyuk Yu.G., Mikhailov V.V., Ovchinnikov B.V., Shalaev N.E., Shen A., Shpilkin S.A., Shukshin I.A.

Abstract

In this piece, political scientists and mathematicians discuss various uses of mathematical methods for detecting electoral fraud. The issues under discussion include: vote returns corresponding to Gaussian distribution; the criteria that anomaly detection methods have to match; the reliability of methods for quantifying fraud; distinguishing between fraud-induced anomalies and naturally occurring anomalies; government reactions to reports of anomalies detected by mathematical methods.

---

Editor's note

The use of mathematical methods for detecting electoral fraud has been discussed among Russian experts for quite some time. The first contributions made in 1994–1995 by Aleksandr Sobyanin and Vladislav Sukhovolsky [41; 42; 40] received criticism from public officials, legal experts and political scientists alike [43; 22; 33]. In 2008, the discussion restarted with new vigor [5; 4; 13; 19; 21; 9; 37] and has been escalating with each new federal electoral cycle.

The discussion typically follows a standard script. The proponents of mathematical methods publish their analyses of electoral fraud levels in the last elections. Their critics retaliate by trying to prove their methods have no scientific basis. The criticized sometimes respond reluctantly. The latest example of such discussion is a report released by the Russian Public Institute of Electoral Law (ROIIP) in September 2020 [2] and its critical evaluation by Andrei Buzin [3] and Alexander Shen [35].

However, as Andrei Buzin pointed out back in 2008 [5], "not only do the polemicists speak different languages, they also stay in different rooms." Then again, there were two attempts in 2018 to gather these opponents for a round-table discussion, but none of them made any significant progress [20; 23].

The editorial board of Electoral Politics would like to continue the discussion and, if possible, put it into academic context. For this purpose, we reached out to a large number of political and social scientists who use mathematical methods of analysis as well as mathematicians who use their knowledge to analyze electoral statistics as civic activists. We asked them six sets of questions – the ones that typically pop up in the discussion.

We have received responses from 11 researchers. None of them are strongly opposed to using mathematical methods to detect electoral fraud. Nevertheless, each of them has their own opinion of the issue, meaning that their responses may paint a clear enough picture of the current state of affairs and opinions in this field.

Naturally, the discussion is far from over. We are ready for it to continue and will be happy to give the opponents a chance to voice their opinions.

Question 1. Is it possible to analyse the voting results mathematically? Are there any regularities that these results satisfy?

Valentin Mikhailov

There is no doubt that vote returns correspond to certain statistical patterns, and that analyzing them is useful when checking the integrity of an election. In the finalized protocols, many parameters serve as material for looking for new connections that are numerous and manifest especially clearly when comparing the results between different precinct and territorial election commissions (PECs and TECs respectively) in one or different regions. Electoral statistics operates the laws of statistical science, taking ethnic, political, social and other features into account.

Nikolai Grishin

Mathematical methods may play a supporting role in evaluating vote returns. It is likely that mathematics helps to detect only certain effects that occur when calculating official vote results. Only mathematicians can estimate which effects may be indicative of mass violations and fraud, and how to distinguish them from statistical effects of generalizing big data.

Nikita Shalaev

They certainly are. First of all, we can observe polling stations (as well as voters) forming into a bell curve by the level of turnout at least in Eastern European countries. This distribution is not quite normal, but it is not that far from it either: it is unimodal, and highly symmetric. The hypothesis of equality of proportions in which electoral support is distributed among electoral participants at different turnout levels is confirmed as well. These properties occur in different countries year after year, and in different types of elections, too [34].

Yury Korgunyuk

They sure are! Since vote returns are numbers that follow a certain logic, they have to be mathematically analyzable. It is the lack of marked patterns that should be alarming. This cannot be the case if the numbers are "real". The lack of such patterns is one of the signs of fraud.

I think there are many such patterns, but I will focus on the ones I work with myself – those concerning electoral cleavages, that is. If the data is not falsified, then the factor analysis of participant results (in my case, parties are the participants) in different territorial units should reveal cleavages that may be interpreted from political and social standpoints. Taking cue from Andrei Akhremenko, I call them electoral cleavages, although a more accurate description would be the factors of territorial dispersion of different participant results.

Sergey Shpilkin

Since quantitative analysis is impossible without mathematical methods, the answer to the first part of the question is definitely Yes. As for the patterns, the purpose of the analysis is, in fact, twofold: to apply known mathematical facts to election results and to identify empirical patterns that are specific to elections in general, and elections in Russia in particular.

A popular idea among those who oppose the mathematical approach to elections studies is that "mathematics cannot describe people's behavior". This idea, if accepted, eliminates polling methods from sociology since they fully rely on mathematical statistics and traditionally use samples as big as an average polling station.

Alexander Shen

Mathematical statistics is useful for analyzing various natural and social phenomena, and elections are no exception. When studying election results in particular, one can reject a certain hypothesis or a class of hypotheses if a simple event (or an event defined before the data are studied) that has low probability according to all hypotheses in this class actually occurred [44: Appendix 1]. Furthermore, mathematical processing of results and their graphical representation may be useful when studying social phenomena (for example, when looking at the time evolution or geographic distribution).

Boris Ovchinnikov

Yes, they are. In addition to simple "arithmetic" patterns (such as equal odds of final digits occurring in absolute or relative values), there are stylistic patterns as well. First, vote returns at each polling station are a sum of individual decisions made by hundreds and hundreds of people, each of whom, for his/her part, is influenced by multiple factors – this fact makes the likelihood of the polling station results slightly deviating from city/town/national average far more likely than it does the likelihood of a large deviation occurring. Second, voter preferences remain fairly stable over time, which is why small changes in the opposition/loyalty levels that are also coordinated between similar polling stations are absolutely expected between elections.

Ivan Shukshin

Vote returns are analyzable beyond any doubt. People vote independently, and a sum of millions of random votes, each of which is affected by thousands of reasons, tends towards a normal distributional limit as expected of a random variable. However, vote returns are not the only ones to follow different patterns; people's habits of choosing the time for voting are patterned as well. For example, the last hours of the voting day in Russia are the least busy as a rule. That said, when in 2018 Primorsky Krai Election Commission reported 309 voters in 10 hours and 833 in the last two hours (which was an actual case at the PEC No. 1944), that effectively sent an official invitation for a criminal investigator to conduct a search of all PEC members.

Andrei Buzin

To me, the question "Are vote returns in elections mathematically analyzable?" seems very much like the question "Can vote returns be bescribed in Russian or Spanish?"

As a science, mathematics does not describe individual natural phenomena (including social phenomena as well as they may very well be considered as a part of nature) – it is a language, a tool for describing nature as a whole, including social phenomena. Mathematics is simply a language that we use to describe natural phenomena with varying degrees of accuracy.

Naturally, each language has a certain set of words and descriptive means. African languages do not have a word for "snow", but one tribe in Africa may have hundreds of words to describe the color brown, seeing how they live in a brown desert. In a similar fashion, different mathematical models may work very well when describing some phenomena and not so well when describing other. For example, mathematical game theory only appeared following mathematical description of social processes.

For a more defined question, consider the following: which mathematical models are better suited to describe vote returns and can be used best in their further study? One should also realize that no mathematical model is identical to a natural phenomena, including elections.

Kirill Kalinin

The study of electoral fraud is based on the idea of manipulations having distortive effect on official voting data. As a result, the data cease to follow certain mathematical patterns, and in statistical analysis, it manifests itself in the form of various statistical anomalies. The nature of these distortions lies in human psychology – people are unable to intuitively generate random numbers.

Dmitry Kogan

Definitely. Every phenomenon that has a numerical dimension can be analyzed mathematically. But separation of signal (real voting results) from noise (electoral fraud) has a significant difference from common fraud detection problems – the noise can be much stronger than the signal and it is not completely random.

There are three main regularities for voting results:

1. Distribution of votes for the main candidates by precinct are unimodal.

2. Major candidates’s share does not depend on turnout.

These regularities are typical, but not absolute. Each violation of these laws is an anomaly, caused or artificial (fraud) either natural (ethnic or religious heterogeneity) reasons, and these reasons must be found.

3. The distributions of votes for the whole country or its large regions are continuous, i.e. smooth enough. They can not include groups of precincts with identical results.

That is an absolute law. There are no natural reasons causing its violation.

Question 2. How well do the voting results fit the normal distribution? Does sociological research use normal, Student or any other classical distribution to evaluate required sample size and analysis error? Can deviations from the normal distribution be used to detect anomalies?

Valentin Mikhailov

My short answer is that there is a good, albeit slightly inaccurate, correlation between vote returns and Gaussian distribution. Significant deviations from Gaussian distribution indicate tampering and fraud.

Nikolai Grishin

I suppose it may be evidenced by the scale of deviations, not by deviations per se.

Yury Korgunyuk

If we are talking about turnout, then normal distribution is what should be expected. On the contrary, if any more "peaks" occur, it is a clear sign of abnormality. Suspicion will inevitably rise if, for some reason, there is a cluster of territories where both turnout and voting for the "party of power" levels suddenly surge in the absence of independent election observation. In any case, this is a serious reason to take a closer look at organization of voting procedure in these territories.

It may well turn out that there was nothing out of the ordinary, and the residents simply voted for a popular authority figure. To prove it, however, one would have to provide strong arguments: for example, a non-stop video footage of both voting and vote tallying processes, etc.

Sergey Shpilkin

Essentially, election returns are multi-dimensional sets of integers. Use of certain typical data processing procedures (e.g. creating a turnout histogram) will definitely result in sets of numbers that bear resemblance to normal distribution, often indiscernible from it within statistical error (e.g. vote vs turnout distributions for 2013 mayor election in Moscow). At the same time, applicability of the normal distribution in such cases has no sufficient theoretical grounds and is not implied in election analysis (although one can imagine a processing procedure where normal distribution would be a useful approximation). In reality, we are referring to the empirical fact that in fraud-free scenarios, vote vs turnout distributions typically have only one "peak" (is unimodal), which requires much less strict conditions than the central limit theorem which ensures convergence to the Gaussian distribution. Also note that unimodal turnout distribution is not a law of nature, but an empirical rule that has reasonable justifications on the one hand, and was confirmed in multiple occasions in various elections in many countries all over the world, on the other. This rule also has its exceptions, which have clear and credible explanations. Usually the deviations from the rule are due to the presence of a single factor whose effect on the turnout exceeds the cumulative effect of other factors: e.g., severe ethnic division of polling stations in Turkish Kurdistan.

In today’s Russia, our experience shows that the only factor that is powerful enough to break the unimodality of vote vs. turnout distribution is vote manipulation at the tallying stage, which existence is confirmed by both mathematical (probabilistic indicators like "Churov saw" or final-digit distribution) and non-mathematical (observer reports and video footage) means. So, in Russian elections, any deviation of vote distribution from a regular bell-like shape is a "red flag" that indicates high probability of electoral fraud.

Boris Ovchinnikov

You cannot expect a direct correlation between vote returns and normal distribution, as there is no theoretical basis for this. However, you can expect a unimodal distribution under a generally approximate symmetry (on a logarithmic scale) between the left and the right parts of the bell. Why? Because of inverse dependence between the possible result and the scale of its deviation from the mode – something that was mentioned earlier and is described in [31] in more detail.

That said, the very existence of deviations from unimodality is not a clear indication of electoral fraud. If these deviations reoccur over time (consistently manifest themselves in all elections) and have a pronounced geographical range (concentrated in one part of a territory or in typologically similar polling stations), then there are grounds for looking for sociological explanation for such deviations. However, the factors that can shift the results at a polling station by 10 or more percent will be evident enough and easily detectable for a diligent researcher.

Nikita Shalaev

I would say that at this point of research, giving a precise characteristic to the distribution type is somewhat untimely. Distribution properties of even the most conventional of indicators in electoral statistics has not been studied as extensively as other aspects of electoral research, not from a global perspective at least.

However, the existing fragmented data still suggest that deviations from the near-normal distribution are far from being common. Naturally, in this sense, this kind of observation may be cause for a deeper study of the case. However, there is no reason to automatically link any observation of this kind with artificial distortion of "natural" voting results. We may still propose a number of fairly natural voting scenarios that will result in similar distribution properties (e.g. bimodality), and dismissing these alternative explanations hardly seems possible without studying electorate properties. In short, it will be an anomaly, but the question of whether it is an artificial anomaly is a separate issue.

Alexander Shen

Some simple voting models (say, if everyone votes independently with the same probability) lead to a distribution that is close to normal. There is no reason to expect a priori that these models are close to reality. Still, there are many cases where the (real) voting results are close to a normal distribution, and also there are many cases where there are significant deviations from normal distribution. Therefore, deviations from the normal distribution per se do not prove fraud. Still mathematical results are important in many cases (e.g., when estimating the size of a random sample to be used in a survey).

Ivan Shukshin

Deviation from the normal distribution is simply an alarm, not a definite proof. However this alarm is the most important one because it shows that things may be bad not just in one place or at a couple of polling stations, but in the whole region where the elections are held. Like any other academic or research activity, detecting electoral fraud in a country with a well-established culture of electoral manipulations is quite a special form of art. But it seems to me that using Gaussian distribution and Student's t-distribution to estimate error is more fitting in democratic countries where they will be used as evidence for fair elections or those with minor irregularities. For authoritarian countries, the deviations are so blatantly obvious that there is simply no room or time for calculating error estimates.

Dmitry Kogan

Generally speaking, distribution of votes by candidates and turnout should not match any analytical distribution so their comparison is useless for anomaly detection. All statistical tests based on normality assumption can serve just for rough evaluation of error. Preference should be given to criteria that are invariant to the distribution law, e.g. chi-square test based on contingency table can be used to check correlations. Standard power analysis (i.e. evaluation of the minimal sample size) also is unreliable since it relies on assumption that estimates are unbiased and distributed normally. Theoretical estimate of the error can be considered only as its upper limit. But we deal in electoral practice with large samples and statistical conclusions are well conditioned. But the only reliable precision criteria is the practical verification i.e. comparison with results of independent election observers.

Kirill Kalinin

Although the Gaussian distribution can serve as a reference distribution, especially in statistical modeling, it is certainly not the only one. For example, it is possible to use binomial or some other type of distribution. As a rule, the Gaussian distribution is suitable under the assumption of a large number of factors having equal effects on voting. For example, statistical models developed by Peter Klimek and Walter Mebane show that clean vote counts can theoretically follow Gaussian distribution. In case of density graphs or histograms for turnout and voting built by Sergey Shpilkin, it is preferrable to refer to unimodal (single-peaked) distributions having no precise mathematical expression. Under the assumption of clean data following the unimodal distribution, the presence of additional "peaks" on the graphs will be indicative of anomalies possibly associated with election fraud. Conversely, in the case of heterogeneous data for which above mentioned assumptions are irrelevant, observed anomalies can be attributed to election fraud only after ruling out alternative explanations. In a situation of chronic data shortage and the inability to fully examine the nature of observed heterogeneity, the interpretation of anomalies may largely depend on researcher’s personal preferences.

Andrei Buzin

First, one has to realize that vote returns are a sample of discrete values, so the off-handed answer would be no, they cannot correlate with continuous Gaussian distribution. Our opponents can always refer to this simple argument, and they will not be wrong if small-scale elections are the case. Yet our response might be that what we mean is binomial distribution instead of Gaussian.

However, having made a model assumption that we are describing the sum-total of vote returns as infinitely large, we can raise a research question of testing the hypothesis with some level of significance on whether a sample of a certain electoral indicator meets the assumption that it comes from the sum-total with normal distribution. Chi-squared test and Kolmogorov's criterion are used to test this hypothesis as well as assumptions about other distributions [29].

In the overwhelming majority of cases, and even when the distribution of an electoral value looks like Gaussian distribution, actual vote returns do not confirm the hypothesis of normal distribution with an acceptable level of significance (I used chi-squared test).

Nevertheless, in many cases (but not all of them!) we can say that the distribution of some electoral indicators at an appropriate aggregate step is quite symmetrical, unimodal and close to normal (it is more correct to call such a distribution quasi-normal or quasi-gaussian).

The latter fact is completely natural (owing to the central limit theorem) if we assume that voting process is independent from where it takes place. However, as soon as such dependence occurs (meaning it exists in reality), deviations from quasi-normal distribution occur as well. Moreover, electoral fraud and voter coercion also significantly skew such distributions from the quasi-normal one [7]. Therefore, deviations from the quasi-normal distribution can be the basis for identifying anomalies and even frauds when the sample is large enough.

At the same time, the size of the required sample can be estimated by examining the elections in this area in historical retrospect. If the distribution used to be quasi-normal, its dramatic transformation is only possible with similar changes in social and political conditions, and some transformations are possible only through fraud.

As for the distribution of sociological indicators in general, many of them are easily described (or approximated, to be more precise) by Gaussian distribution (for example, the distribution of graduates based on their Unified State Exam scores in some years) and by Poisson distribution (for example, the distribution of patients by territory). In fact, the latter is directly related to elections: in the absence of fraud and "protest voting", voting at home and the share of invalid ballots have Poisson's distribution to some extent.

Question 3. What principles and criteria should an anomaly detection method satisfy? What mathematical approaches are best for detecting anomalies? How clear and understandable are these methods for the general public?

Nikolai Grishin

Given the qualitative diversity of large-scale electoral fraud (from forging documents to various forms of coercing and pressuring voters), applying mathematical methods that would be both universal and able to cover all this diversity hardly seems possible.

Yury Korgunyuk

I think these methods can be very different, and the more methods there are, the better. I believe there are more than enough methods that would be useful in detecting anomalies. It is a different matter that the general public will be able to understand them only if the overall quality of mathematics education among the population increases. Then again, there are no guarantees in that case either. The problem is not in the methods themselves, but whom the population trusts more – independent experts or the folks at the top. If a person wants to think in a certain way, he or she will think so despite any arguments he or she is presented with.

Boris Ovchinnikov

I do not have a comprehensive answer, as it will be different depending on a method's purpose, whether it is finding proof of electoral fraud, assessing the scale of fraud, or identifying and describing fraud nature and geography.

Sergey Shpilkin

The method needs to be reliable (with low false positives). As for the general public's ability to understand the method, there are methods that are descriptive to a greater (Gabdulvaleyev's diagram) or lesser (final-digit distribution) extent, but illustrative power has no direct correlation with either the method's mathematical credibility or, for example, its ability to produce quantitative estimates.

Valentin Mikhailov

There are many different methods: "turnout – voting for a candidate" graphs developed by Aleksandr Sobyanin and Vladislav Sukhovolsky [25: 61-70; 28: 184-191], heterogeneity of the coefficient of voter reorientation in two-round election of 1996 [27; 25: 32-49], analysing invalid ballots and voting against all, applied by A.Myatlev and myself [25: 57–61; 28: 333–340], distribution of turnout developed by S.Shpilkin [37; 38], frequency of final digits in official results, peaks at the markings divisible by 5 and 10... The general public will surely be able to grip some of these. A lot depends on the way the data is presented to the public.

When choosing a method from a variety of possibilities, one should comprehend that the use of visual aids is important in a country where the authorities never hesitate to use administrative resource when counting the ballots and tallying results. Visualized data presented in an understandable way makes it more difficult for election commissions, courts and executive authorities to refuse to consider cases of electoral fraud. At the same time, visualization makes the information on the real situation in elections more accessible for wider audiences, stimulating their interest in participating in electoral processes. Subtle mathematical methods are always useful, but they will only cater to experts and people who are already in the know. It is this club-like exclusivity precisely that gives the authorities the chance to "flush" the essence of election fraud under the guise of an ongoing professional debate. This does not mean that methodological toolkit should be made smaller, but one has to remember that there is a direct correlation between effectively purging the elections of fraud and digestible representation of results.

Dmitry Kogan

Like any other mathematical method, the detection of anomalies must be objective (e.g. it should not be based on arbitrary assumptions) and correct (it must use the appropriate mathematical apparatus). The additional requirements are persuasiveness and visibility, because the results are shown to government departments and the public and should be intuitive to ordinary people. The last-digit method is of academic interest only in this sense. The most convincing verification methods are:

· search for matches, repeats and peaks at round percentages;

· Gabdulvaleyev's diagram;

· turnout-based scatter charts.

Andrei Buzin

Statistical method for detecting anomalies:

· should be based on official statistics;

· should contain a description of the method and be reproducible;

· can be used for a large number of elections;

· conclusions drawn from the method should have clear justification and be easily interpreted.

Aside from developing simple histograms of electoral indicators, the following methods are most conclusive for detecting anomalies at his point in time:

· comparative analysis of electoral indicator distribution in one territory;

· Shpilkin method for large-scale elections;

· a modified Sobyanin-Sukhovolsky method [6].

Histograms (including Gabdulvaleyev's ones) have good descriptive and illustrative properties. Shpilkin's method is quite illustrative, but not not easily understood by the soft science crowd.

Ivan Shukshin

Any relevant mathematical methods are useful for detecting anomalies. However, only few of them can be used for illustrative purposes. Final-digit frequency analysis is good for assessing whether results are artificial or not, yet it is not visual enough. I believe that histograms of candidate results that are distorted by election fraud would be more digestible by the general public, especially when they have peaks in percentages divisible by 5. I would say that cluster analysis is the best method from the visual standpoint, when bimodal distribution on the "candidate–distribution" graph clearly outlines both the genuine cluster and the cluster(s) formed by election fraud.

Alexander Shen

Hypotheses should be correctly assessed according to the rules of mathematical statistics. Several methods are useful, e.g., histogram peak assessment ("round percentages"), last-digit analysis, invalid ballot analysis, "turnout vs. leader result" coordinate representation, comparing data from different years, comparing vote returns with observation data (both on-site reports and video footage), comparing distributions for different candidates, etc., see review [36]. Is the analysis convincing for the "general public"? On one hand, you do not need deep statistical knowledge for understanding this analysis: it is enough to learn fractions, percentages and graphical representation of data to understand most of the results. For example, one does not have to be a expert to understand that when the odd-numbered and even-numbered polling stations in the town of Klintsy have 90.0% and 91.0% turnout respectively (with three exceptions out of 28 polling stations), it is not a coincidence. On the other hand, many people are unable to understand and assess the results because of the position they occupy, a lack of desire or skills to do so (or a combination of these factors), see, for example [2].

Nikita Shalaev

For lack of a comprehensive theory on electoral behavior that could predict the properties of the relevant distributions based on some fundamental and easily identifiable properties of electorate, it seems that we thus have to focus on the external validation of methods via results of traditional election observation. In such an event, the most promising methods are the ones that will detect anomalies in the same cases as on-site observers, and the ones that will not yield false positives otherwise at that. From a practical standpoint, the best method would be the one that would require less diverse data (it is not always possible to count on electoral statistics for all imaginable indicators being available), and not rely on specific additional data.

From a visual standpoint, there are two directions of method development that would perhaps be easiest for the general public to perceive: the first includes methods geared towards studying the "slices" of electoral support (equality of proportions) and the second includes methods geared towards studying the geographical component. The idea that the "people" should distribute their support between candidates almost equally regardless of how many voters choose to come to the election is quite understandable from the point of view of common sense, as opposed to appealing to properties of random value distributions. In similar fashion, referencing "neighbors" who are unlikely to vote in the exact opposite way with no obvious reasons should be easily understood by the masses. Unfortunately, the geographical component still largely exists withing publicistic discourse rather than scientific, and is difficult to process given the nature of publishing electoral data.

Kirill Kalinin

The ideal method for detecting fraud should meet several important criteria [14]. First, it must be sensitive to anomalies, minimizing the possibility of false negatives. Second, in the absence of anomalies, it should return zero results, thus minimizing the number of false positives. Third, the method should cover electoral data in its entirety, and preferably at the polling station-level. Fourth, the method should facilitate geographic analysis of anomalies conjointly with various political, cultural, or ethnic factors. Fifth, the method should provide us with the estimates of uncertainty communicating to the expert community and the public the degree of confidence in our findings. Compared to validity-based criteria, the method’s visual clarity and understandability is of minor importance since we are more concerned about method’s validity rather than its popularity with the public.

Although election forensics is a relatively young field, many of its methods have proven their effectiveness with many of indicators increasingly being incorporated into a variety of statistical and econometric models. The following methods are most popular: digit-based tests of aggregate vote totals [1; 8; 15], regression and correlation methods [40], nonparametric methods based on histograms [38] or density graphs [30; 32], parametric methods including various models of electoral fraud [17; 11; 24] as well as field experiments [10; 39; 16]. This list can be expanded by adding various heuristic methods tailored to search for anomalies within specific geographic areas. All these methods tend to engage different data segments and differently quantify clean voting, so often their conclusions regarding the anomalies may differ. The main difficulties with these methods are associated with preferred use of the polling-station level data (mathematical analysis of voting data is most effective at the lowest level), as well as requirements of high performance-computing for parametric methods.

Question 4. Are there any reliable methods to quantify election fraud?

Andrei Buzin

There are no methods that would quantify electoral fraud "with the point of one's pen". Approximate and preliminary estimates can be obtained using many different methods.

Yury Korgunyuk

Reliability of methods can never be absolute. What is important here is how valid is their use.

Nikita Shalaev

Perhaps not, not at present anyway. That is if we want the method to be highly reliable and universal at the same time. Studies of this kind still have to resort to expert knowledge on electorate specifics in the countries or regions under study – or rely on additional data, whose composition is often case-specific.

Valentin Mikhailov

I suppose that there are no mathematical methods that would quantify electoral fraud with the highest level of precision, but an accuracy of up to 2-3% is quite achievable for many cases. It is better to use several methods at once to ensure the highest level of precision possible. Comparing many regions with each other may significantly improve the reliability of electoral fraud estimations.

Alexander Shen

Not quite, if we are talking about pinpoint accuracy of percentages. First, in many cases, the noise masks the signal completely. Second, for a high accuracy accurate one would need a lot of statistical data related to space-time distribution of different sociological parameters. On the other hand, in order to say that rigged votes in the "constitutional amendments adoption vote" (2020) are more like dozens of percents than a fraction of a percent, it is enough to take a look at the histogram and the two-dimensional representation of the official results [18].

Ivan Shukshin

First, it all depends on the number of points taken for assessment, meaning the number of precinct election commissions (PECs) where elections are held. It is essential that the reliability of electoral fraud detection is higher than the noise, higher than the error and higher than three sigma.

Second, the scale of electoral fraud should also be higher than the error, as not all frauds are easily detectable: for example, the case when the results of a shoo-in are dialed up slightly from 49 to 51% to avoid the second round of election.

Third, it all depends on whether we limit ourselves to studying the numbers only, or whether we take field evidence, reports of violations or direct fraud at polling stations into account. For federal elections in Russia, there is usually enough data even without field evidence. As for regional elections, the figures alone are enough for electoral sultanates.

Boris Ovchinnikov

The question is what we mean by "reliable methods".

If it means the methods that repeatedly yield nationwide and region-wide results that do not contradict any other known facts and estimates of the latest vote, then there are such methods. One of the first such methods that comes to mind is the "Shpilkin's model", or rather both parts of his approach – the bell curve and the center of th cluster.

However, if it means methods for which it is possible to calculate the confidence interval and mathematically prove they coincide with actual results, then there are no such methods and there cannot be. If anything else, it is because no one knows the actual results.

Sergey Shpilkin

There are highly reliable methods allowing to detect electoral fraud in large groups of polling stations and in the country as a whole, as well as to give a lower quantitative estimate for the number of polling stations where electoral fraud occurred. For example, nationwide analysis of "Churov's saw" reveals that results are being rigged at thousands of polling stations in every federal election. The mathematical reliability of these methods is due to the fact that they are based on minimal assumptions. Then again, due to the same reason these methods only detect a small fraction of electoral fraud (which would still be sufficient to initiate a legal investigation and take organizational measures).

Approaches aimed at assessing the whole volume of fraud require stronger assumptions about the electorate properties and are therefore less reliable mathematically. However, independent methods validate the results obtained through these approaches, like the analysis of the 2011 legislative elections in Moscow [10], as well as the FOM (Public Opinion Foundation) poll data collected in the 2016 legislative election in Moscow that was published and then deleted, and results of video footage reviews, see [12] for examples.

Dmitry Kogan

There are two quantitative approaches in use to estimate ballot stuffing and wrong entry of votes. Shpilkin's method is based on a priori independence of voting results from voter turnout. Myatlev's method is based on a priori dependence of the number of invalid ballots on voter turnout. To take into account the regional specifics of voting these methods are applied to each region separately. Both approaches reliably estimate the lower threshold for fraud, because they take into account only large stuffing and can not detect stealing of votes. There are no reliable methods to estimate number of votes stolen, but usually it is much less than stuffing.

Kirill Kalinin

At present, there are three methods for quantification of electoral anomalies: Peter Klimek's model, Walter Mebane's Bayesian finite mixture model (this is a revised version of the earlier published finite mixture model [16]) and Sergey Shpilkin's non-parametric method. None of the methods guarantees full reliability. All methods are more or less based on the assumption of a "clean peak" that serves as a benchmark for calculating the overall magnitude of anomalies. Parametric methods, which include Peter Klimek's and Walter Mebane's models, are designed to reproduce the model of a hypothetical fraud mechanism with possible inclusion of control variables, and to output polling station-level fraud probabilities and the number of stolen votes. In contrast to parametric approach, Sergey Shpilkin's non-parametric method is based on a special calculation algorithm for histograms that outputs histogram-level measures of election fraud. Although there is a strong correlation between the Mebane and Shpilkin estimates, the parametric method tends to be more conservative than the nonparametric method. Naturally, none of the above-described methods is universal, just as none of them proves the existence of electoral fraud.

Nikolai Grishin

Trying to precisely quantify the level of fraud using mathematical methods seems counter-productive. It may only be worth it if the only end goal is generating media response. There is simply not enough data and tools to solve this task. The report is based on data from some polling stations, which, in turn, may not be reliable. In the context of mass fraud, when we do not know what data can be considered reliable, solving this problem may not be possible. There is a risk that polling stations used as a reference point are the ones where electoral fraud is more systematic and not accompanied by mathematic anomalies instead of the ones with reliable data.

In my opinion, the attempts to quantify fraud levels only embellish the picture. Final calculations of fraud levels are not only insufficiently substantiated, but also play down said levels.

Question 5. Is it possible to use mathematical analysis to distinguish electoral fraud from anomalies caused by natural reasons (territorial heterogeneity, social composition of voters etc.)?

Andrei Buzin

Generally, no.

Yury Korgunyuk

Again, mathematical analysis alone is incapable of anything. This is just a tool that you still need to know how to use. In any case, vote returns should be interpreted by election specialists: political scientists specializing in regions, social scientists, political geographers, etc. Only they can sort the wheat from the chaff.

Nikita Shalaev

The answer to this question depends directly on what data is acceptable for researchers to rely on. If the task is set in such a way that the data should be minimal, for example, the data that comes from electoral archives alone and nowhere else, without using any external information – then it is unlikely it can. At the same time, if external data is allowed, then the task can be solved – and the wider the range of external data, the more likely it is to be solved. For example, previous vote returns data for the same territorial units (polling station, district) may be a big help when combined with a reasonable assumption of the relative consistency of the main electorate properties (for example, it is unlikely that long-term absence voters will change their attitude towards elections in an instance and decide to go to the polls).

Sergey Shpilkin

In most cases, it can. The phenomena associated with natural factors usually manifest themselves in long-term trends (and as such are traceable over many election cycles), have clear underlying causes that are easy to study, and are subject to mathematical laws. As for fraud-induced anomalies, the general rule is that they cannot be explained rationally and/or violate basic statistical principles (rigged results in the 2016 legislative elections in the cities of Saratov and Tyumen, in the 2018 presidential election in Stavropol Krai, in the 2018 gubernatorial election in Primorsky Krai, etc.).

Boris Ovchinnikov

Yes, of course, if we are not talking about mathematical analysis in its strict sense, but rather about election results in general. Natural anomalies will reoccur over time and/or be "gradient" in space, meaning they will not only manifest at the most anomalous polling station, but in neighboring polling stations as well, but in a subtler form. Naturally, local (limited to one polling station) one-time deviations may indeed occur, but the general logic and empirical data show that such deviations are more likely to be caused by protest voting. In any case, these may be one-time exceptions rather than multiple deviations in different parts of a city/town.

Dmitry Kogan

Sure. An approach similar to cross validation can be applied. Precincts are divided into homogeneous groups (e.g. urban and rural ones) and the method is applied to each of them. If the same anomalies are found within each group, they are most likely caused by fraud.

Alexander Shen

Yes, this is precisely why we need to use these methods (in fact, the word "anomaly" is an technical term for these methods, as they all interpret "anomalies" in their own way).

Kirill Kalinin

Sequential elimination of alternative explanations is a rather laborious research task, without solving which one cannot refer to anomalies as scientifically proven cases of fraud. Unfortunately, this task requires a lot of data, which can be especially difficult to obtain due to access restrictions or the fact that it simply does not exist. Even with such data, not every method implies the inclusion of control variables – this task is effectively solved with the help of parametric methods.

Valentin Mikhailov

Territorial, ethnic, urban, rural heterogeneity – all these things exist. Differences exist. But this rather complicated picture is additionally besmirched by administrative pressure, heterogeneous (!) cases of fraud and is wrapped in myths like: villagers in this particular republic have this kind of mentality – they vote for whomever the elder tells them to. I used to study the heterogeneity between the region's capital and raion (district) centers, between raion centers and villages, and, eventually, between villages close to and far from raion center. I found heterogeneity in all three cases. It exists because people in remote villages have less power, they are not motivated enough to fight for fair elections, for their rights because their forefathers lived in such condidtions. They often do not even understand what a "fair and free election" even means. It is often the case that neighboring regions for example, Kirov Oblast, Udmurt Republic, and the Republic of Tatarstan, end up with different election results. For example, let us take a look at the 2012 presidential election results the neighboring TECs in three federal subjects:

· Baltasinsky Raion, Tatarstan – 97.6% turnout and 91.5% voted for Vladimir Putin;

· Kukmorsky Raion, Tatarstan – 97.2% and 95%;

· Kiznersky Raion, Udmurt Republic – 62.5% and 75.9%;

· Malmyzhsky Raion, Kirov Oblast – 64.1% and 55.9%.

This is not because these raions are ethnically different (in Malmyzhsky Raion, one third of the population are Tatars and one sixth are Mari people), but because the administrative pressure on elections and voting in Tatarstan is noticeably higher than in the neighboring regions. They have different information fields at local level, but federal information channels are shared.

Careful application of statistics will help to assess the share of votes gained through fraud and pressure.

Nikolai Grishin

In my opinion, mathematical analysis of election results makes it difficult to distinguish fraud-induced anomalies from naturally occurring anomalies. Persistent territorial differences in electoral behavior can be caused by such reasons as ethnic and confessional structure of the population, the share of urban residents (as well as the so-called "large population factor"), occupational patterns, etc. Within the framework of mathematical analysis methods, it is not only possible to explain these "natural electoral anomalies", but also take them into account (as some variables, etc.). In practice, the analysts who use mathematical methods do not factor these anomalies in their research either (if I am not mistaken). This may cause serious errors in the study of electoral behavior and already contributes to spreading myths (in particular, the myth that any significant deviations are the result of fraud and violations).

Combining mathematical methods of analyzing election results with the tasks of studying group differences in electoral behavior is a difficult task in terms of methodology and searching for specific methods. Solving this problem is imperative if we want to advance the study of electoral behavior and improve our knowledge of the level of electoral fraud. At this point in time, this task is not being solved.

Question 6. How should the authorities react to electoral anomalies detected by mathematical analysis if they want to prevent electoral crimes?

Andrei Buzin

They should issue explanations.

Nikolai Grishin

Anomalies revealed by mathematical methods should be the reason for public discussion and proceedings. Formally, they cannot be a pretext for any action on the part of state authorities. Nevertheless, law enforcement agencies may launch further investigation in case of flagrant and knowingly false electoral anomalies.

Sergey Shpilkin

Were the authorities concerned with detecting and investigating violations, they could use the results of electoral statistics analysis to significantly narrow down the "list of suspects" and thus improve the effectiveness of such work.

Alexander Shen

When/if state authorities really want to minimize/prevent fraud, significant problems are rare, as there are many tools (including legislation). Refocusing their efforts on preventing fraud rather than committing it would be enough. However, during the transition period mathematical methods could be used to identify the polling stations and territories that require more detailed examination (recount, more observers, etc.).

Nikita Shalaev

I believe that although we do not yet have tools that are 100% accuratein detecting rigged results, the tools that we do have allow us to detect suspicious results and locations as well. And the more methods hit the alarm at the same time in the same place, the more doubts should arise. Naturally, a proper reaction should follow: such messages should be taken seriously and examined properly and objectively. In fact, there is no fundamental difference with eyewitness complaints: the more people report wrongdoings, the more reason there is to investigate the circumstances, although neither eyewitness accounts, nor mathematical methods are completely reliable.

Dmitry Kogan

As soon as reliable markers of fraud (matches, repeats and peaks at round percentages) are found, election results must be canceled. If long “Churov beard” is detected and number of fraudulent votes estimated by Shpilkin’s method exceeds millions, election results must be canceled as well. Other cases require additional investigations.

Kirill Kalinin

If state authorities are concerned about tackling electoral crime, they should respond positively to reports of statistical anomalies, as these methods help to identify and examine the possible sources of fraud. The main problem is the ambiguous interpretation of anomalies due to the lack of data and limited evidence of their fraudulent nature. At the same time, it is still unclear who should bear the burden of proof: researchers attributing fraud to statistical anomalies, or authorities explaining anomalies by data heterogeneity. In theory, once anomalies are discovered, a joint investigation should be conducted between election forensics researchers and representatives of government agencies. If revealed, electoral malfeasance should be subject to the appropriate legal assessment, and everyone involved should be held accountable.

Ivan Shukshin

When transitioning from authoritarianism and lawlessness to democracy and lawfulness, the authorities should adopt a transparency policy and restructure their work in such a way that anomalies can be examined transparently.

For example, legislators can ensure that all election video footage is available to anyone in full for at least the entire duration of the statute of limitations on these offences. Adopting the access procedure similar to GAS Pravosudiye (State Automated Sytem "Justice"), which can be accessed via the Gosuslugi account, would help to control storage servers' traffic and prevent bot access.

However, transparency alone is not enough when combating offences: certainty of punishment plays an important role, as it ensures that every detected violation has consequences for the violators.

Yury Korgunyuk

Naturally, they should not take stances like "Prove it first!" and "Not a thief until you catch me!" While such stances may be acceptable in a trial of a petty thief, it is certainly not when it concerns those responsible for election management. In this case, it is the suspects who have to prove their innocence.

Any anomaly should be reason for special proceedings, including those involving independent experts. Local election managers should be the ones to provide evidence. In case they fail to do so, commissions should be disbanded and manned with new people (including representatives of the general public) while election should be held again in line with all formal procedures – having independent observers (including those invited from outside the given territory), round-the-clock video surveillance (including that during vote count), etc.

Valentin Mikhailov

The main question of "how can mathematics be used in elections?" was raised in my paper from 2010 [26]. The paper gives reasons for why it is necessary to abandon the formal legal approach: if there is a written complaint from a polling station, it should be handled legally, in accordance with procedure. If not – then there is nothing to consider. We have a long way to go. In our conditions, there are two to three times less complaints than actual violations, and election commissions and courts have already learned to suppress those that reach them. Besides, there are violations that go unnoticed by voters and observers at the polling stations, but revealed by statistics. It was often the case in the Republic of Tatarstan that while no complaints were filed, the statistics was rife with anomalies.

My advice below is for future reference, for a time when the state has given up its policy of pandering to electoral fraud, has clamped down on the use of administrative resource, has discharged the executive branch from any essential work in election commissions, only leaving the functions of material support and supply to it. Or when it starts moving in this direction, at least.

The habit of rigging and coercion in elections has incapacitated the country, and it will not end in an instant. The long shadow of fraud is gradually consuming all polling stations, including the honest ones. There is every potential to quickly identify local anomalies within a span of two or three days, and then send independent inspectors to recount the ballots as well as contact local independent observers. It is not difficult fundamentally, and should quickly bear fruit. Mathematics expert groups can be established in Moscow, St. Petersburg and other regional centers.

It may prove useful to make public the PECs where no violations have been detected by either independent observers or statistical verification.

A more thorough, less urgent analysis would reveal regional trends and temporal dynamics, and its results could also be used to cleanse elections in Russia.

Received 29.10.2020, revision received 05.11.2020.

---

References

1. Beber B., Scacco A. What the Numbers Say: A Digit-Based Test for Election Fraud. – Political Analysis. 2012. Vol. 20. No. 2. P. 211–234.
2. Borisov I.B., Zadorin I.V., Ignatov A.V., Marachevsky V.N., Fedorov V.I. Matematicheskiye instrumenty delegitimatsii vyborov. Doklad ROIIP [Mathematical Tools for Election Delegitimation. A Report by Russian Public Institute of Electoral Law]. Moscow, 2020. URL: http://www.roiip.ru/images/data/gallery/0_299_Matematicheskie_instrumenti_delegitimatsii_viborov.pdf (accessed 28.10.2020). (In Russ.) - http://www.roiip.ru/images/data/gallery/0_299_Matematicheskie_instrumenti_delegitimatsii_viborov.pdf
3. Buzin A. Retsenziya na doklad Rossiyskogo obshchestvennogo instituta izbiratelnogo prava “Matematicheskie instrumenty delegitimatsii vyborov” [A Review of the Report by Russian Public Institute of Electoral Law]. – Interregional Association of Voters website, 15.09.2020. URL: http://votas.ru/ROIPP2020.html (accessed 28.10.2020). (In Russ.) - http://votas.ru/ROIPP2020.html
4. Buzin A.Yu., Lyubarev A.E. Prestupleniye bez nakazaniya: Administrativnye izbiratelnye tekhnologii federalnykh vyborov 2007–2008 godov [Crime Without Punishment: Administrative Electoral Technology of the 2007–2008 Federal Election]. Moscow: TsPK “Nikkolo M”; Tsentr “Panorama”, 2008. 284 p. (In Russ.)
5. Buzin A.Yu. Mozhet li rossiyskaya vlast izmenit zakon Nyutona? [Can Russian Authorities Change Newton’s Law?] – Nezavisimaya gazeta, 01.04.2008. URL: https://www.ng.ru/ng_politics/2008-04-01/17_newton.html (accessed 28.10.2020). (In Russ.) - https://www.ng.ru/ng_politics/2008-04-01/17_newton.html
6. Buzin A.Yu. The Modification of Sobyanin-Sukhovolsky Method. – Electoral Politics. 2019. No. 1. URL: http://electoralpolitics.org/en/articles/modifikatsiia-metoda-sobianina-sukhovolskogo/ (accessed 28.10.2020). - http://electoralpolitics.org/en/articles/modifikatsiia-metoda-sobianina-sukhovolskogo/
7. Buzin A.Yu. Vliyaniye territorialnykh neodnorodnostei i falsifikatsiy na elektoralnye pokazateli [Influence of Territorial Heterogeneity and Fraud on Electoral Indicators]. – Vestnik Rossiyskogo universiteta druzhby narodov. Seriya: Matematika, informatika, fizika. 2014. No. 2. P. 72–80. (In Russ.)
8. Cantu F., Saiegh, S.M. Fraudulent Democracy? An Analysis of Argentina’s Infamous Decade Using Supervised Machine Learning. – Political Analysis. 2011. Vol. 19. No. 4. P. 409–433.
9. Churov V.E., Arlazarov V.L., Solovyov A.V. Itogi vyborov. Analiz elektoralnykh predpochteny [Election Results. An Analysis of Electoral Preferences]. – Trudy Instituta sistemnogo analiza RAN. 2008. Vol. 38. P. 6–22. (In Russ.)
10. Enikolopov R., Korovkin V., Petrova M., Sonin K., Zakharov A. Field experiment estimate of electoral fraud in Russian parliamentary elections. – PNAS. 2013. Vol. 110. No. 2. P. 448–452. URL: https://www.pnas.org/content/110/2/448 (accessed 28.10.2020). - https://www.pnas.org/content/110/2/448
11. Ferrari D., McAlister K., Mebane W.R., Jr. Developments in Positive Empirical Models of Election Frauds: Dimensions and Decisions. – Paper prepared for presentation at the 2018 Annual Meeting of the Society for Political Methodology, Provo, UT, 2018, July 16–18.
12. Gabdulvaleyev A. Turnir gorodov. Kratkie itogi issledovaniya videozapisey s vyborov 18 marta 2018 goda [Tournament of the Cities. A Brief Summary of Video Footage Review from March 18, 2018 Election]. – Tatarstan Election Observer Association website, 07.06.2020. URL: https://tatobservers.ru/2020/06/07/turnir-gorodov-kratkie-itogi-issledo/ (accessed 28.10.2020). (In Russ.) - https://tatobservers.ru/2020/06/07/turnir-gorodov-kratkie-itogi-issledo/
13. Grigoryev M.S. Teoremy nedobrozhelatelya, ili Goroskop Nyutona [Ill-Wisher Theorems or Newton’s Horoscope]. – Nezavisimaya gazeta, 20.05.2008. URL: https://www.ng.ru/ng_politics/2008-05-20/17_newton.html (accessed 28.10.2020). (In Russ.) - https://www.ng.ru/ng_politics/2008-05-20/17_newton.html
14. Hicken A., Mebane W.R., Jr. A Guide to Election Forensics. – Working paper for IIE/USAID subaward #DFG-10-APS-UM, “Development of an Election Forensics Toolkit: Using Subnational Data to Detect Anomalies”. 2015. P. 38.
15. Kalinin K., Mebane W.R., Jr. Understanding Electoral Frauds through Evolution of Russian Federalism: The Emergence of Signaling Loyalty. – Paper prepared for the Annual Meeting of Midwest Political Science Association, Chicago, 2013.
16. Kalinin K. Validation of the Finite Mixture Model Using Quasi-Experimental Data and Geography. – Electoral Politics. 2019. No. 1. URL: http://electoralpolitics.org/en/articles/validatsiia-konechnoi-smeshannoi-modeli-s-ispolzovaniem-kvazieksperimentalnykh-i-geograficheskikh-dannykh/ (accessed 28.10.2020). - http://electoralpolitics.org/en/articles/validatsiia-konechnoi-smeshannoi-modeli-s-ispolzovaniem-kvazieksperimentalnykh-i-geograficheskikh-dannykh/
17. Klimek P., Yegorov Y., Hanel R., Thurner S. Statistical Detection of Systematic Election Irregularities. PNAS. 2012. Vol. 109. No. 41. P. 16469–16473.
18. Kobak D., Shpilkin S., Pshenichnikov M. Suspect peaks in Russia's "referendum" results. – Significance. 2020. Vol. 17. No. 5. P. 8–9.
19. Lyubarev A.E. I ne fizika, i ne lirika [Election Science: Neither Hard, Nor Soft]. – Nezavisimaya Gazeta, 03.06.2008. URL: https://www.ng.ru/ng_politics/2008-06-03/20_maths.html (accessed 28.10.2020). (In Russ.) - https://www.ng.ru/ng_politics/2008-06-03/20_maths.html
20. Lyubarev A. Krugly stol v SPCh po voprosam elektoralnoi statistiki [CHR Round Table on Electoral Statistics]. – Lyubarev LiveJournal, 6.03.2018. URL: https://lyubarev.livejournal.com/38896.html (accessed 28.10.2020). (In Russ.) - https://lyubarev.livejournal.com/38896.html
21. Lyubarev A. Nedobrosovestnaya kritika – tozhe reklama (o gazetnoy kritike knigi “Prestupleniye bez nakazaniya”) [Bait and Switch Publicity is Still Publicity (On Newspaper Criticisms of Crime Without Punishment)]. – Interregional Association of Voters website, 29.03.2009. URL: http://votas.ru/kritika.html (accessed 28.10.2020). (In Russ.) - http://votas.ru/kritika.html
22. Malyutin M. Falsifikatsiya zakonchena: zabudte… [No More Fraud: Forget It…] – Novaya Yezhednevnaya Gazeta, 11.05.1994. (In Russ.)
23. Matematika i vybory: rabotayut li formuly, proveryat obshchestvennye nablyudateli [Maths and Elections: Public Oberverse to Check If Formulas Work]. – Civic Chamber of the Russian Federation website, 28.02.2018. URL: https://www.oprf.ru/press/news/2018/newsitem/44277 (accessed 28.10.2020). (In Russ.) - https://www.oprf.ru/press/news/2018/newsitem/44277
24. Mebane W.R., Jr. Eforensics: A Bayesian Implementation of a Positive Empirical Model of Election Frauds. 2019.
25. Mikhailov V. Demokratizatsiya Rossii: razlichnaya skorost v regionakh (Analiz vyborov 1996 i 2000 gg. Mesto Tatarstana sredi subyektov RF) [Democratization of Russia: Regions Going at Different Speed (An Analysis of Elections of 1996 and 2000. Tatarstan’s Place Among Federal Subjects of Russia)]. – Osobaya zona: vybory v Tatarstane. Ulyanovsk: KOMPA, 2000. P. 25–84. (In Russ.) - http://www.democracy.ru/library/articles/tatarstan/page3.html
26. Mikhailov V. Statistichesky analiz rezultatov vyborov: neobkhodimost vklyucheniya v pravovoye pole [Statistical Analysis of Election Results: Debating the Need of Incorporation Into Legal Environment]. – Rossiyskoe Elektoralnoye Obozreniye. 2010. No. 1. P. 20–30. (In Russ.) - http://vibory.ru/analyt/REO-5/Mikhailov.pdf
27. Mikhailov V.V. Kolichestvo demokratii (Analiz vyborov Prezidenta RF 1996 g. v regionakh) [The Amount of Democracy (An Analysis of the 1996 Presidential Elections in Russian Regions)]. – Armageddon, Book 3 (May–June). Moscow, 1999. P. 134–153. (In Russ.)
28. Mikhailov V.V. Respublika Tatarstan: Demokratiya ili suverenitet? [The Republic of Tatarstan: Democracy or Sovereignty?]. Moscow, 2004. 466 p.
29. Minko A.A. Statistichesky analiz v MS Excel [Statistical Analysis in MS Excel]. Moscow: Izdatelsky dom Vilyams, 2004. P. 299–306. (In Russ.)
30. Myagkov M., Ordeshook P.C., Shakin D. The Forensics of Election Fraud: Russia and Ukraine. New York: Cambridge University Press. 2009. 289 p.
31. Ovchinnikov B. A Response to Ye. Galitsky. – Facebook, 27.03.2018. URL: https://www.facebook.com/boris.ovchinnikov/posts/10155389079107304 (accessed 28.10.2020). (In Russ.) - https://www.facebook.com/boris.ovchinnikov/posts/10155389079107304
32. Rozenas A. Detecting Election Fraud from Irregularities in Vote-Share Distributions. R package version 1.1. 2016. URL: https://CRAN.R-project.org/package=spikes (accessed 28.10.2020). - https://CRAN.R-project.org/package=spikes
33. Ryabov N.T. Iz vystupleniya na zasedanii TsIK 12 iyulya 1994 goda [Excerpts from a Speech at the CEC Meeting of July 12, 1994. – Bulletin of the Central Election Commission of the Russian Federation. 1994. No. 8. P. 4–7. (In Russ.)
34. Shalayev N.E. Elektoralnye anomalii v postsotsialisticheskom prostranstve: opyt statisticheskogo analiza [Electoral Anomalies in Post-Soviet Countries: Practice of Statistic Analysis]. – Thesis for a Candidate Degree in Political Sciences. St. Petersburg: St. Petersburg State University, 2016. 191 p. (In Russ.)
35. Shen A. Kak doklad ROIIP delegitimiruet vybory [How ROIIP Report Delegitimizes Elections]. – Troitsky Variant – Nauka, 22.09.2020. No. 313. P. 5. URL: https://trv-science.ru/2020/09/kak-doklad-roiip-delegitimiruet-vybory/ (accessed 28.10.2020). (In Russ.) - https://trv-science.ru/2020/09/kak-doklad-roiip-delegitimiruet-vybory/
36. Shen A. Vybory i statistika: kazus “Yedinoi Rossii” (2009–2020) [Elections and Statistics: The Case of United Russia]. 2020. 76 p. URL: https://arxiv.org/pdf/1204.0307v4.pdf (accessed 28.10.2020). (In Russ.) - https://arxiv.org/pdf/1204.0307v4.pdf
37. Shpilkin S. Obrabotka dannykh rezultatov federalnykh vyborov 2007–2008 godov [Processing Federal Election Results of 2007–2008]. – 2008 Russian Presidential Election. GOLOS Association Report. Moscow, 2008. P. 94–102. (In Russ.)
38. Shpilkin S. Statistika issledovala vybory [Statistics Examines Elections]. – Gazeta.ru, 10.12.2011. URL: https://www.gazeta.ru/science/2011/12/10_a_3922390.shtml (accessed 28.10.2020). (In Russ.) - https://www.gazeta.ru/science/2011/12/10_a_3922390.shtml
39. Sjoberg F.M. Autocratic Adaptation: The Strategic Use of Transparency and the Persistence of Election Fraud. – Electoral Studies. 2014. Vol. 33. P. 233–245.
40. Sobyanin A.A., Sukhovolsky V.G. Demokratiya, ogranichennaya falsifikatsiyami. Vybory i referendumy v Rossii v 1991–1993 gg [Democracy Shackled by Fraud. Elections and Referendums in Russia in 1991–1993]. Moscow: Human Rights Task Group, 1995. 267 p. (In Russ.)
41. Sobyanin A., Sukhovolsky V. Itogi vyborov 12 dekabrya 1993 goda i budushchy federalny zakon o vyborakh [Election Results of December 12, 1993 and the Future Federal Law]. – Konstitutsionnoye Pravo: Vostochnoyevropeiskoe Obozreniye. 1994. No. 1(6). P. 2–10. (In Russ.)
42. Sobyanin A., Sukhovolsky V. V korolevstve krivykh zerkal [Kingdom of Crooked Mirrors]. – Segodnya, 10.03.1994. (In Russ.)
43. Vedeneev Yu.A., Lysenko V.I. Vybory-93: uroki i alternativy [Election of 1993: Lessons and Alternatives]. – Bulletin of the Central Election Commission of the Russian Federation. 1994. No. 7. P. 18–30. (In Russ.)
44. Vereshchagin N.K., Uspenskiy V.A., Shen A. Kolmogorovskaya slozhnost i algoritmicheskaya sluchainost [Kolmogorov Complexity and Algorithmic Incidental]. Moscow: MTsNMO, 2013. 575 p. (In Russ.)