Interviewer falsification: the signs of fabricated survey data, and what to do

What interviewer falsification is

Interviewer falsification means recording answers the respondent didn't give. The best-known form is an enumerator inventing whole interviews, sometimes called curbstoning. The AAPOR and American Statistical Association task force report Falsification in Surveys (2020) points out that it goes further: filling in part of an interview, changing disposition codes (the record of whether a household was interviewed, refused or not found), duplicating whole records, or copying a record and changing a few answers. Another shortcut it describes is answering “no” to a filter question, the question that decides whether a long module is asked, so the module is skipped.

The same report notes that falsified data bias estimates, affect their precision and distort the relationships between variables. Fabricated survey data is also hard to repair after fieldwork: the households can no longer be revisited cheaply and the enumerator has moved on. That is why the signs belong in the daily high-frequency checks, read enumerator by enumerator.

Why statistical signs work, and where they stop

Someone inventing answers has to guess what real answers look like, without seeing what colleagues are recording. The AAPOR report describes this as the principle behind output monitoring: compare each interviewer's data with a reference the falsifier doesn't know, namely everyone else's data. Invented data tends to be too tidy. The report cites work finding that fake interviews vary less than real ones, have fewer missing answers and fewer “other” answers, and that copied records show up as pairs of respondents who match on a very high share of questions. In its example of a survey with 100 yes-or-no questions, two respondents matching on more than 85 percent would be highly unlikely.

The same report is clear about the limits, and they shape how the signs should be used:

• An enumerator with few interviews produces noisy rates, so small workloads give false alarms.
• Unless households are assigned to enumerators at random, differences between enumerators can simply reflect where they work and whom they interview. Compare each enumerator with colleagues in the same area.
• Statistical methods can't confirm falsification. They are a filter that tells you which interviews to verify, and they need a person to interpret them.

The six signs

Each sign looks for one habit of someone inventing answers. Each also has honest explanations, listed beside it.

Near-copies need a pair-by-pair comparison of each enumerator's interviews; the Stata code is on the duplicate survey responses page. The other five fit in one pass over the data. Here the choice questions are named c_*, number questions n_*, amounts amt_*, the questions that open a section open_* (1 yes, 0 no), and don't know and refused are coded -98 and -99:

use "survey.dta", clear
keep if consent == 1
generate one = 1

* Few "don't know": codes per interview
egen dk = anycount(c_* n_* amt_* open_*), values(-98 -99)

* Most common answers: choice answers equal to the village's most common one
generate typical = 0
generate chosen = 0
foreach q of varlist c_* {
    bysort village: egen m = mode(`q') if `q' >= 0 & !missing(`q'), maxmode
    replace chosen = chosen + 1 if `q' >= 0 & !missing(`q')
    replace typical = typical + 1 if `q' == m & !missing(m)
    drop m
}

* Round numbers: positive amounts that are exact thousands
generate amounts = 0
generate thousands = 0
foreach q of varlist amt_* {
    replace amounts = amounts + 1 if `q' > 0 & !missing(`q')
    replace thousands = thousands + 1 if `q' > 0 & !missing(`q') & mod(`q', 1000) == 0
}

* Skipped sections: "no" to the questions that open a section
egen opened = anycount(open_*), values(0 1)
egen said_no = anycount(open_*), values(0)

* Low spread: the enumerator's standard deviation over the village's
foreach q of varlist n_* {
    bysort village: egen sd_v = sd(`q') if `q' >= 0 & !missing(`q')
    bysort village enum_id: egen sd_e = sd(`q') if `q' >= 0 & !missing(`q')
    generate ratio_`q' = sd_e / sd_v
    drop sd_v sd_e
}
egen spread = rowmedian(ratio_*)

collapse (sum) n=one dk typical chosen thousands amounts said_no opened ///
    (median) spread, by(village enum_id)
keep if n >= 10
generate low_dk = -(dk / n)
generate typical_share = typical / chosen
generate round_share = thousands / amounts
generate no_share = said_no / opened
generate low_spread = -spread

A filter question is easy to find in an XLSForm: it is any question whose name appears in another question's relevant column, such as ${has_livestock} = 1. Pick those that open a long module, where saying “no” saves the most time.

Comparing with colleagues in the same area

Each sign means something only next to the people doing the same work. Put each enumerator's value against the median of the enumerators in the same village or district, and measure the distance in robust standard deviations: the median absolute deviation (the typical distance from the median) times 1.4826. Unlike the mean and the ordinary standard deviation, these aren't pulled about by the one enumerator you are looking for. Then add up the distances above the group, so several moderate signs together count for more than one strong one alone.

* Compare within villages that have at least four enumerators to compare
bysort village: generate peers = _N
keep if peers >= 4
generate score = 0
foreach s in low_dk typical_share round_share no_share low_spread {
    bysort village: egen med = median(`s')
    generate dev = abs(`s' - med)
    bysort village: egen mad = median(dev)
    generate z_`s' = (`s' - med) / max(1.4826 * mad, 0.02)
    replace score = score + max(z_`s', 0)
    drop med dev mad
}
gsort -score
list village enum_id n score z_*, noobs

The floor of 0.02 stops a sign that barely varies between enumerators from turning small differences into large scores. Where a village has fewer than four enumerators, take village out of the by and compare with the whole team. Merge in the near-copy share by enum_id and score it the same way.

This kind of combined score has a track record. The AAPOR report describes a focused reinterview programme for a large US health survey (Hood and Bushery, 1997) that added up standardised scores on four interviewer-level measures, each compared with what was expected in the interviewer's area, and sent the outliers for reinterview. It found three falsifiers among 83 interviewers checked, a hit rate of 3.6 percent against 0.2 percent for random reinterviews.

Read the result the way you would read any other enumerator monitoring. Wait until each enumerator has at least 10 or so interviews. One sign alone often has a plain explanation; several together are a pattern. An enumerator with no flags on any other check and unusually typical answers deserves a look as much as one with many flags. If a whole area stands out, look at the area, its supervisor and the translation before the individuals.

Confirming: back-checks, audio audits and supervisor visits

Targeted back-checks

The direct test is to go back to the respondent. Draw a targeted sample from the flagged enumerator's work: their near-copies first, then their most typical-looking interviews, then a few at random. Ask whether someone from the survey visited, roughly when, for how long and whom they spoke to, then a few questions whose answers shouldn't change between visits. Report these targeted back-checks apart from the random sample. The back-checks guide covers the questions and how to compare them. The AAPOR report describes the same two-pronged design used by survey organisations: a random sample of cases, plus cases picked because of anomalies in the data or in the interviewer's performance.

Audio audits, where respondents consented

An audio audit records part of an interview on the tablet so a reviewer can listen later. It needs the respondent's informed consent and your ethics approval, and in some places local law sets further rules. SurveyCTO's audio audit field records a random share of interviews (p=), from a set point (s=) for a set time (d=, in seconds), and its guidance on audio audits says to mention them in your ethics review and in the consent statement. ODK's background audio recording records while the form is open, shows a status bar to the data collector, and recommends a note at the start of the form telling data collectors and participants that they are recorded; it warns you may need the consent of everyone within range of the microphone. The data collector can switch it off where recording would compromise safety or consent can't be obtained, which deletes what was recorded.

The AAPOR report lists what recordings can reveal: keystrokes but no voices, the interviewer's voice but no respondent's, the same “respondent” voice across several interviews, and many recordings missing where consent was supposedly given. It adds that each raises suspicion without proving anything, since a covered microphone can do the same. Listening takes time, so use recordings for interviews other signs already point to.

Supervisor visits and other evidence

A supervisor can accompany the enumerator for a day, or revisit a few of their households unannounced. The interview's own records help too: interviews far shorter than the team's (see interview duration), two interviews by one enumerator at the same time, interviews started late at night, and several interviews recorded from one spot (see GPS checks). These are evidence about single interviews, and they make a pattern in the six signs easier to judge.

Handling it fairly with the enumerator

A suspicion of falsification is about someone's job and reputation. The AAPOR report asks the profession to be careful in making claims of falsification, and recommends personnel action, from a reprimand to dismissal, once it has been determined that someone falsified data. A fair process looks like this:

1. Verify before you accuse. Run the targeted back-checks first, by people independent of the enumerator's supervisor, and keep the sample to the people who need it.
2. Check the assignment before the person. An enumerator given the hardest villages, the callbacks or one uniform settlement will look different for honest reasons.
3. Hear the enumerator. The supervisor goes through specific interviews with them: the IDs, the households, what happened. Many patterns have plain explanations, such as a filter question misunderstood in training.
4. Decide on the evidence, and record it: what the signs showed, what the back-checks found, what the enumerator said and what was decided. Use codes rather than names in anything shared beyond the survey team.
5. If interviews didn't take place, set them aside with a reason rather than deleting them, redo them where possible, check everything else the enumerator collected, and follow the project's rules and the enumerator's contract. If nothing is found, close the case and say so.

Prevention is part of fairness. Tell the team at training that their work is compared with colleagues' and that back-checks happen. The AAPOR report recommends reducing the perceived opportunity to falsify, and weighing the risks that push people towards it: long and burdensome questionnaires, interviews in dangerous or difficult areas, and large cash incentives.

How DataFlow looks for made-up interviews

Once you name the enumerator variable, DataFlow works out the six signs for each enumerator with at least 10 interviews, and only when at least four such enumerators can be compared; with fewer, it shows nothing. If you name an area variable too, it compares enumerators within each area when every area has at least four enumerators with 10 interviews there, and otherwise across the whole team. Interviews set aside are left out, and so is a sample or census list imported from a file, which has no interviews of its own. It compares the answers to the form's number, choice and text questions, leaving out the enumerator, area and key, the interview's own details (start and end times, duration, device ID, form version) and variables ending in _oth.

• Near-copies: the share of the enumerator's interviews that give the same answer as another of their own on at least 85% of the questions answered in either.
• Low spread: the median, across number questions, of their standard deviation divided by the group's, with don't-know codes left out.
• Most common answers: the share of their choice answers that are the group's most common answer to that question.
• Round numbers: the share of their positive amounts (number questions with a median of 1,000 or more) that are exact thousands.
• Few “don't know” answers: the codes -999, -998, -888, -99, -98 and -97 per interview, below the group's.
• Skipped sections: the share of answers coded 0, 2 or “no” to questions that another question's relevance refers to.

Each sign is measured as the distance from the group's median in robust standard deviations (1.4826 times the median absolute deviation, with a small floor for each sign). The distances above the group are added up, and an enumerator whose total is over 8 is flagged. The signs that are 2 or more robust standard deviations out are named, strongest first.

A flagged enumerator gets a card in the Today list, “Back-check [name]'s interviews”. It names up to three signs, proposes up to five interviews to back-check first (the near-copies, closest first, then the most typical-looking) and drafts a note for the enumerator's supervisor asking for a visit or a call to confirm each interview took place. Copying the note is recorded in the activity log and takes the card off Today for three days; it comes back if the signs are still there. “Not a problem” keeps that enumerator's card off Today. The Field team view marks the same enumerator with “Back-check some interviews” and the strongest sign.

When a back-check survey is linked to the survey it checks, the back-check sample is a random share of each enumerator's interviews not yet back-checked, plus every interview this check points to, marked with the reason “made-up interview check”. It saves as a CSV the back-check form can preload, and the back-check results are then compared by enumerator, back-checker and question. On single interviews, DataFlow adds checks for an interview started 5 minutes or more before the same enumerator's previous one ended, for three or more of one enumerator's interviews within 15 m of each other, and for interviews started between 21:00 and 06:00 by the tablet's clock, when the data has the times, enumerator and GPS they need. DataFlow works alongside the collection platform: it connects to SurveyLoopr, SurveyCTO, ODK Central and KoboToolbox, and reads exports from other tools.

Step by step

1. Measure the six signs per enumerator. For each enumerator, work out the share of near-copies, the spread of their numbers, the share of most common answers, round amounts, don't-know answers per interview, and no answers to questions that open a section.
2. Compare with colleagues in the same area. Put each value against the median of enumerators in the same area, in robust standard deviations, and add up the distances above the group. Wait for at least 10 interviews each and four colleagues to compare.
3. Pick the interviews to verify. For an enumerator who stands out on several signs, choose their near-copies first, then their most typical-looking interviews, then a few at random.
4. Verify with the respondents. Back-check those interviews by visit or phone with a team independent of the enumerator's supervisor, and use audio audits or supervisor visits where respondents consented.
5. Hear the enumerator and decide. Go through specific interviews with the enumerator, check their assignment, and decide on the evidence. Record what was found and what was decided.
6. Act on confirmed cases. Set aside interviews that didn't take place with a reason, redo them where possible, review the rest of the enumerator's work and follow the contract. Close cases where nothing was found.

Questions