Opinion and analysis by Reza Behrouz and Mohammad R. Jahan-Parvar on IranPoll and the pitfalls of opinion surveys in an authoritarian environment.
------------------------------------------------------------------------------------------------------------
Every 6 to 12 months, a survey is conducted by a Toronto-based private firm called the IranPoll in conjunction with the University of Maryland's Center for International and Security Studies. It is a telephone-based public opinion survey of people residing in Iran. The results are then published online and almost always showcased by Washington think-tanks such as the Atlantic Council. Despite enjoying a partnership with a major American research university, the results of IranPoll are never subjected to academic scrutiny or published in a peer-reviewed journal.
IranPoll prides itself on using sound scientific methodology, which includes sequential randomizations, catch-all geographic breadth, and the ability to conduct interviews in multiple ethnic languages and dialects, in addition to Persian. Yet, the survey’s intent and its results often generate controversy and outrage, rather than offering practical information conducive to policy-making.
In reaction to IranPoll, there is always a deluge of indignation on the part of Iranians inside the country and in the diaspora who consider the results outlandish and detached from reality. The main argument raised by Iranians against IranPoll is the unrealistic expectation that politically-oriented surveys can ever be conducted in a totalitarian dictatorship such as the Islamic Republic. Opinion polls administered in authoritarian states naturally call into question the authenticity and honesty of participants when responding to politically-sensitive queries. There is little doubt that fear of retribution influences how surveyees reply to certain politically-sensitive questions.
There are also features associated with IranPoll that are scientifically flawed. The survey’s methodology has been described in a cursory fashion on IranPoll’s website, leaving ample room for doubt about the validity of the data collection process. For example, participants are selected amongst Iranians who have access to landline telephones, excluding a large segment of the population that exclusively uses mobile devices. This narrows the testing sample and introduces bias at the very outset. The process by which participants’ telephone numbers are obtained is, of course, not detailed. It should be noted that every landline telephone number is attached to a physical address, which could theoretically be accessed by the Islamic Republic's security apparatus. This factor can influence the authenticity and integrity of responses.
The survey goes through a succession of awkward randomizations to enroll participants. Excessive stratification using multiple randomizations eliminates balanced participant enrollment, countering the effect and the value of random sampling.
IranPoll states, “[w]hen a residence is reached, a qualified respondent is randomly selected from within that household, often using the random table technique.” IranPoll documentation does not describe the inclusion criteria for the “qualified” respondent. Assuming that being an adult is the foremost criterion, randomization would fail if a household consists of only one adult. This conundrum is also applicable to a situation in which the randomly-selected respondent is disabled and unable to participate. It is unclear how these situations are dealt with by the surveyors.
The cooperation rate for the latest IranPoll survey was 79%. Typical cooperation rate in public opinion surveys is about 30%, with any rate above 50% considered “excellent.” A nearly 80% cooperation rate without incentive is astronomical, considering Iran’s stifling political atmosphere. One is forced to view this outcome as “too good to be true.”
The same argument can be made about the overall response rate of 60%. A realistic response rate ranges between 5% and 30%. A response rate of 60% and a cooperation rate of 79% do not mean each respondent completed the survey and answered every question. Had the survey been offered online, these figures would be more convincing, but it is inconceivable that such a large proportion of people would hold the telephone line long enough to complete the survey in its entirety. That is why nowadays multi-question surveys are typically conducted online (alternatively on paper forms), while single-question polls are predominantly carried out via telephone query.
IranPoll naturally does not offer any information on the completion rates. Assuming cooperation rate means replying to any one or more questions on the survey, IranPoll intentionally conceals the rate at which individual questions were answered. The more participants refrain from responding to a specific question, the smaller the sample size per question and the less accurate the results. For instance, if only 50 people replied to the question on President Ebrahim Raisi’s favorability, it would take a mere 39 favorable opinions to achieve the 78% rate demonstrated in the results. Fifty respondents constitute only 5% of the entire sample. This pitfall applies across the board to all questions on the survey; from President Joe Biden’s popularity amongst the Iranian people, to their opinion on U.S. sanctions and the nuclear deal.
Additionally, IranPoll’s results are in sharp contrast to the findings of online multi-question surveys such as the Netherlands-based GAMAAN (the Group for Analyzing and Measuring Attitudes in IRAN). As an example, the proportion of respondents in IranPoll who indicated their intention to vote in the June 2021 presidential elections exceeds that of GAMAAN’s by 24 percentage points. The level of trust in the Islamic Republic of Iran Broadcasting (IRIB) service is yet another example. IranPoll reported that about 50% of their survey-takers trusted IRIB’s reporting “most of the time,” while GAMAAN’s number was only at 14%.
GAMAAN’s methodology certainly has its own shortcomings, but its online format mitigates the challenges associated with telephone-based multi-question surveys. Research has consistently shown that respondents tend to give more extreme, positive answers to attitudinal items than when the same questions are administered to them via the web. There are also indications that the social interaction inherent in a telephone interview may exert pressure on respondents, affecting how they answer questions. Although gross discrepancies between the two surveys’ reported findings can be attributed to “mode effect,” differences in responses by survey mode are typically not large. As such, the magnitude of differences between GAMAAN and IranPoll’s resultsare too significant to ignore.
Overall, there are notable flaws in IranPoll’s methodology, as well as how the results are presented. To the best of our knowledge, the method by which this survey is conducted has never been validated. In other words, its replicability and generalizability are highly questionable. IranPoll does correlate the demographics of its sample with those of Iran national census and the Central Intelligence Agency’s Factbook, and makes an attempt at validation by matching the statistics it generates with official data divulged by the regime itself. For example, IranPoll produced a 52.9% participation rate in the 2021 presidential elections. This figure was then juxtaposed with the 48.9% turnout reported by the Islamic Republic in an attempt to ostensibly demonstrate accuracy and replicability. It is impossible to ascertain the credibility of the voter turnout stated by the regime when the elections are far from free and fair.
Lastly, it is appropriate to also understand who are the key players behind the University of Maryland’s IranPoll. To that end, we encourage the reader to review an article by Ahmad Rafat in Kayhan Life.
Given IranPoll’s socially objectionable results, its methodological flaws, the way findings are presented, and the fact that it evades peer review and scientific scrutiny, an impartial observer is compelled to view the survey with suspicion. Such an observer may reasonably conclude that this survey is little more than a propaganda ploy by the Isalmic Republic to influence the United States’ policies vis-à-visthe regime. The data IranPoll provides each cycle are outrageously predictable and oftentimes, downright preposterous. Many residents of Iran and members of the diaspora view this survey as a source of humor and an object of ridicule. Think-tanks that tout IranPoll as an astounding and authentic revelation from inside Iran should thoroughly examine the merits and validity of this survey. Likewise, the University of Maryland should consider whether associating its name with such a scientifically dubious and politically controversial project is in the best interest of the institution.
Dr. Reza Behrouz is an Iranian-American physician and medical researcher based in San Antonio, TX, USA.
Mohammad R. Jahan-Parvar PhD is an Iranian-American economist based in Washington, DC, USA.
The opinions expressed by the authors do not necessarily reflect the views of Iran International