Back-check sample size calculator

How the calculator works

It splits the interviews as evenly as possible between the enumerators, takes your share of each enumerator's interviews and rounds up, so even an enumerator with only a few interviews gets at least one back-check. That is why the total can come out slightly above the share you entered. It is the same rule DataFlow uses when it draws a back-check sample, so the numbers match what the app will pick.

From the total it works out three planning numbers: how many back-checks should be done by the end of the second week, how many a week after that, and how many back-checkers you need at the pace you entered. The last line counts the type 1 answers compared for each enumerator: the number of back-checks per enumerator times the number of type 1 questions in each back-check.

Real fieldwork is uneven. Some enumerators finish more interviews than others, and the first week is often slow. Use the calculator to set the budget and the team size, then draw the actual sample from the interviews you have, batch by batch.

Choosing the share

The World Bank's DIME Wiki suggests back-checking 10 to 20 percent of interviews. Where you land in that range depends on three things:

• How much you trust the team. A new team, a new area or a firm you haven't worked with before calls for the upper end. An experienced team on a repeat round can sit at the lower end.
• How many interviews each enumerator does. With few interviews per enumerator, 10 percent gives each person only one or two back-checks, too few to say anything about them. Raise the share or re-ask more questions.
• What a back-check costs. Phone back-checks are cheap and quick. Visits cost travel time but also confirm the dwelling and reach people without a phone. Many teams phone most of the sample and visit the flagged cases.

Draw it within each enumerator

A simple random sample of all interviews can miss an enumerator for weeks, by chance. Drawing within each enumerator's interviews (stratifying by enumerator) guarantees that everyone is checked, and lets you compare enumerators on the same footing. Draw from each new batch of interviews after removing the ones already back-checked, so the sample keeps pace with fieldwork, and don't tell enumerators which households were picked.

Include households reported as absent, refused or not found. They are where an enumerator can most easily skip the work, and leaving them out means the hard cases are never checked.

Front-load the first two weeks

Problems are cheapest to fix early, while training is fresh and the team can still be corrected. The DIME Wiki suggests finishing 20 percent of all back-checks in the first two weeks. On a long survey that is easy to meet at an even pace; on a short one it means starting back-checks on the first days of fieldwork, not when the first batch is "done".

Enough answers to judge an enumerator

A back-check is judged by its type 1 questions: answers that shouldn't change between visits, like household size or roof material (the back-checks guide explains the three types). An enumerator with two back-checks and three type 1 questions has six answers compared, and one honest mistake makes them look terrible. DataFlow doesn't flag an enumerator until at least 10 type 1 answers have been compared and 10 percent or more of them disagree. Both numbers can be changed. If the calculator shows fewer than 10, back-check a larger share or re-ask more type 1 questions.

Then add the targeted back-checks

The random sample tells you how the team is doing overall. On top of it, back-check the interviews that other checks flag: interviews that were too short, near-copies of another interview, or an enumerator whose answers look too regular (see fabricated interviews). Keep these separate when you report agreement rates, or they will make the random rates look worse than they are. Budget extra back-checker time for them: the calculator counts the random sample only.

How DataFlow draws the sample

In DataFlow, you set the share (10 percent unless you change it) and it draws that share of each enumerator's interviews that haven't been back-checked yet, rounding up, plus the interviews its made-up interview check flagged. The draw is stable: the same interviews give the same picks, so the sample can be reproduced. You choose which details go to the back-checker to find the respondent, such as the name and village, and export the list as a CSV for your back-check form. When back-check data comes in, DataFlow compares it with the main interviews and works out agreement rates by enumerator, question and back-checker.

Questions