Back-checks: how to run a back check survey

What a back-check is

A back-check (or backcheck) is a second, short interview with a respondent who has already been surveyed, done by a separate team, a few days after the first. It asks a handful of questions from the main questionnaire again and compares the two sets of answers. It is how a research team finds out whether an interview took place at all, whether the enumerator asked the questions properly, and which questions people answer differently from one week to the next.

Every other data check looks at the data as it was recorded, so an invented interview with believable answers passes most of them. A back-check goes back to the source. It can be done by phone or in person, and the World Bank's DIME Wiki mentions random audio audits as a complement. Back-checks work best alongside the daily high-frequency checks, which tell you whose interviews to revisit first.

Which questions to include

Keep the back-check short: the DIME Wiki suggests 10 to 15 minutes. Copy the questions word for word from the main form, with the same answer codes, so the answers can be compared directly. Start with a few questions that verify the interview itself: is this the person who was interviewed, did someone from the survey visit, roughly when, and how long did it take. Then pick questions of three types, the classification used by IPA's bcstats:

Most of the back-check should be type 1 questions, because they are the ones that point clearly at the enumerator. Type 2 and type 3 questions tell you about training and the questionnaire. Enumerators shouldn't know which questions will be re-asked; the DIME Wiki suggests rotating them during fieldwork. Preload into the back-check form only what the back-checker needs to find the respondent (name, village, phone number), never the original answers, or the back-checker will simply confirm them.

Choosing the sample

• Back-check 10 to 20 percent of interviews, the range the DIME Wiki gives. Do more of them early: it suggests completing 20 percent of all back-checks in the first two weeks, when problems are cheapest to fix.
• Stratify by enumerator. Draw the sample within each enumerator's interviews, so every enumerator is back-checked every week, not only the busy ones.
• Add targeted cases: interviews flagged by other checks (very short, near-copies of another) and the interviews of enumerators you have doubts about. Mark them as targeted and report their agreement rates separately from the random sample, or they will make the random rates look worse than they are.
• Include households reported as absent, refused or not found, to confirm they really couldn't be interviewed. Otherwise the hard cases drop out of the sample unchecked.
• Draw with a fixed seed and keep the list, so the sample can be reproduced, and don't tell enumerators which households were drawn.

A stratified draw in Stata, taking 15 percent of each enumerator's completed interviews:

use "survey.dta", clear
keep if consent == 1
set seed 20261102
generate double u = runiform()
bysort enum_id (u): generate byte pick = _n <= ceil(0.15 * _N)
keep if pick
keep hhid enum_id village
export excel using "backcheck_sample.xlsx", firstrow(variables) replace

Run it on each new batch, after dropping households already back-checked, so the sample keeps pace with fieldwork.

Timing and who does them

Back-check soon after the interview, within days rather than weeks. Type 1 answers have less time to change, the respondent still remembers the visit, and the enumerator is still in the field if something needs correcting. The back-check team should be separate from the survey team: the DIME Wiki recommends experienced enumerators, trained separately, with little contact with the survey staff. Record each back-checker's ID in the form, so you can see whether one back-checker disagrees with everyone. Phone back-checks are cheaper and quicker; visits also confirm the dwelling and reach respondents without a phone.

Comparing answers

Match each back-check to its interview by ID. A back-check that matches no interview is a finding in itself: an ID typed wrong, or a household that isn't in the main data. Before comparing, make the answers comparable: trim spaces, ignore upper and lower case, sort the codes of multiple-choice answers, and compare numbers as numbers, so 5 and 5.0 agree. Decide in advance how blanks count; a common choice is to skip a question that is blank in both and count a blank in only one as a difference.

Type 1 and type 2 answers should match exactly. Type 3 answers get a tolerance, either an amount (within 1 acre) or a share of the original value (within 10 percent of last month's income). Then work out the share of compared answers that differ, the error rate, for several groupings:

Don't rank enumerators on a handful of compared answers; wait until each has enough back-checks for the rate to mean something. There is no universal acceptable error rate. bcstats uses 10 percent by default (its okrate option), and it is reasonable to hold type 1 questions to a stricter standard than type 3. Comparing an enumerator with the rest of the team is often more telling than any fixed line. In Stata, bcstats does the comparison; the variables need the same names in both files:

ssc install bcstats
bcstats, surveydata("survey.dta") bcdata("backcheck.dta") id(hhid) ///
    enumerator(enum_id) backchecker(bc_id) ///
    t1vars(roof hhsize owns_land) t2vars(members_listed) ///
    t3vars(income_month) okrange(income_month [-10%, 10%]) ///
    ttest(income_month) okrate(0.1) replace

For type 2 and type 3 questions, bcstats can also run a paired t-test (ttest()) to see whether answers shift in one direction between the visits. IPA's ipacheck package has a newer command, ipabcstats, for the same job.

What to do when agreement is low

1. Read the differences before the rate. Are they on one question, in one direction, in one village? A pattern often explains itself.
2. Go through them with the supervisor and the enumerator, interview by interview, and ask what happened.
3. Back-check more of that enumerator's interviews. A larger, targeted sample tells bad luck from a habit, and finds the extent of the problem.
4. If interviews didn't happen, set them aside with a reason rather than deleting them, redo them where possible, check everything else the enumerator did, and follow the project's own rules on what happens to the enumerator.
5. If a question disagrees for every enumerator, fix its wording or translation, retrain, or take it out of the back-check.

Don't copy back-check answers over the main data by default. The back-check can be wrong too, and type 3 answers change honestly. Correct a value only once you know which one is right, and log the correction.

How DataFlow compares back-checks

In DataFlow the back-check is its own form, collected on any platform and linked to the survey it checks. DataFlow suggests the questions asked in both, matched by name or with a bc_ prefix or _bc suffix removed. It leaves out free text, the linking ID, the enumerator variable and the interview's own details (start and end times, duration, device, SIM and phone IDs, username, comments and audit files). Whole numbers and choices are suggested as type 1, decimals and amounts of money as type 3 within 10 percent of the main interview's value. You can change any type, and a type 3 tolerance can be an amount or a share.

When comparing, it trims and lowercases text, ignores the order of multiple-choice answers and compares numbers as numbers. The tolerance applies to type 3 only. A question blank in both is not compared; blank in only one counts as a difference. It reports error rates by enumerator, by back-checker and by question, each split by type, counts back-checks that match no interview as not found, and leaves out back-checks that were set aside. When an enumerator's type 1 answers disagree 10 percent of the time or more (the default, which you can change), and at least 10 type 1 answers have been compared, the Today list shows a card with up to five examples and a note to copy for their supervisor.

For the sample, DataFlow draws a random share of each enumerator's interviews that haven't been back-checked yet, plus the interviews its made-up interview check suggests revisiting first. The same data always gives the same picks.

Step by step

1. Choose the questions. Pick a few questions that verify the interview took place, then mostly type 1 questions that shouldn't change, a few type 2 questions that test the enumerator's skill, and type 3 questions whose stability you want to know. Keep it to 10 to 15 minutes.
2. Build the back-check form. Copy the questions word for word with the same answer codes. Preload only what the back-checker needs to find the respondent, never the original answers, and record the back-checker's ID.
3. Draw the sample. Take 10 to 20 percent of interviews at random within each enumerator's work, front-loaded into the first weeks, plus interviews flagged by other checks, marked as targeted. Include households reported absent or refused.
4. Send an independent team soon after. Back-check within days of the interview, by phone or in person, with a team that is trained separately from the survey team and doesn't know the original answers.
5. Compare the answers. Match by ID, normalise text and multiple-choice answers, allow a tolerance on type 3 numbers, and work out error rates by enumerator, question and back-checker, separately for each type.
6. Act on low agreement. Read the differences, review them with the supervisor and enumerator, back-check more of that enumerator's work, set aside and redo interviews that didn't happen, and fix questions that disagree for everyone. Log every correction.

Questions