Enumerator monitoring: each interviewer's data quality, every day

Why read the checks by enumerator

An enumerator who doesn't probe, who rushes the household roster or who fills in part of an interview away from the respondent affects every interview they do. Checked one interview at a time, each looks like a small error. Read by enumerator, the habit shows within the first days, while the team is still in the field and a supervisor can fix it. The World Bank's DIME Wiki makes the same point in its guide to high-frequency checks: enumerator checks look for an interviewer whose data differs from everyone else's, with the share of “don't know” answers and average interview duration as examples.

It needs one thing set up before fieldwork: an enumerator ID on every interview, chosen from a list in the form rather than typed, and the same code for the whole survey. Add the supervisor and the area (district, village or cluster), since both are needed to compare like with like.

What to track for each enumerator

A short daily table covers most of this. In Stata, from consented interviews (here interview_date is a Stata date and the questions are numeric, named q_*):

use "survey.dta", clear
keep if consent == 1
generate minutes = duration / 60
egen dk = anycount(q_*), values(-98 -99)
generate one = 1
bysort enum_id interview_date: generate first_day = (_n == 1)

collapse (sum) interviews=one days=first_day dk (median) med_minutes=minutes, by(enum_id)
generate per_day = interviews / days
generate dk_per_interview = dk / interviews
summarize med_minutes, detail
generate length_vs_team = med_minutes / r(p50)
list enum_id interviews per_day med_minutes length_vs_team dk_per_interview, noobs

Add each check's flags per 100 interviews and the back-check rates from their own files, merged on enum_id. IPA's ipacheck package has a command for this, ipacheckenumdb, which exports statistics on each enumerator's performance.

Reading the numbers fairly

These numbers end up in conversations about people's work, so they have to be read carefully. A few rules keep that fair:

• Wait for enough interviews. In the first days one interview moves a rate a long way. Look at the trend over several days before drawing a conclusion, and don't compare an enumerator with five interviews against one with fifty.
• Compare within similar areas. Remote villages mean fewer interviews a day, longer travel and different answers. A team in the hills shouldn't be judged against the team in town.
• Allow for the workload. An enumerator given the callbacks, the refusals to convert or the largest households will have longer interviews, more refusals and fewer completes. Check the assignment before the person.
• Look across measures. One unusual number is often noise. Short interviews, few “don't know” answers, many “no” answers to section openers and near-copies together are a pattern.
• If everyone trips the same check, the problem is the question, the translation or the check, not the team.
• Be wary of a perfect record. Made-up interviews tend to be tidy, so an enumerator with no flags at all and unusually typical answers deserves a back-check as much as one with many flags.
• Keep it for coaching. Share the table with supervisors, not as a ranking pinned on the wall, and use codes instead of names in anything shared beyond the survey team.

How to act on what you find

1. Start with a conversation. The supervisor goes through specific interviews with the enumerator: the IDs, the questions, what happened. Many patterns have plain explanations, such as a skip instruction misunderstood in training, a tablet with the wrong clock or a village where everyone was at the market. A list of interview IDs gives the supervisor something to check; a general remark about short interviews doesn't.
2. Retrain on the part that goes wrong. Go through the module again, practise the probing, or have the supervisor sit in on a few interviews. Watch the same measures over the following days to see whether it worked.
3. Back-check more of their work. Draw a targeted sample of their interviews, report it apart from the random sample, and let the result decide what happens next.
4. Reassign or remove the enumerator only when back-checks confirm interviews that didn't take place, or problems continue after retraining, and follow the project's rules and the enumerator's contract. Set the affected interviews aside with a reason, redo them where possible, and review everything else that enumerator collected.

Record each step: the date, what the numbers showed, what was agreed and what changed. That gives the enumerator a fair process, and gives a client who monitors the survey firm a record of how quality was managed. Tell the team at training which measures you watch and that back-checks happen, so the monitoring comes as no surprise.

How DataFlow monitors enumerators

Once you name the enumerator variable (and an area variable, if you have one), DataFlow reads every check by enumerator. The Enumerators page lists each enumerator's interviews, median interview length (highlighted when under 60% of the team's median), share of interviews ready and what is left to follow up; a click opens their items. The Today list groups flags by person: when one enumerator has at least four of a check's short-interview or “don't know” flags, and at least half of them, it shows one card with a note to copy for their supervisor. A back-check card appears when 10% or more of an enumerator's type 1 answers differ, once at least 10 have been compared.

DataFlow also computes six signs of made-up interviews for each enumerator with at least 10 interviews, compared with the others in the same area when every area has at least four such enumerators, and otherwise with the whole team. With fewer than four enumerators to compare, it shows nothing. The six signs are:

• near-copies: the share of their interviews that give the same answer as another of their own interviews on at least 85% of the questions answered in either;
• low spread: their answers to number questions vary much less than the team's;
• most common answers: the share of their choice answers that are the team's most common answer to that question;
• round numbers: the share of their amounts (number questions whose median is 1,000 or more) that are exact thousands;
• few “don't know” answers: don't-know and refusal codes (-999, -998, -888, -99, -98, -97) per interview, below the team's;
• skipped sections: the share of “no” answers (coded 0, 2 or no) to questions that open a section, meaning questions that other questions' relevance depends on.

Each sign is measured as the distance from the team's median in robust standard deviations (from the median absolute deviation). The distances above the team are added up, and an enumerator whose total is over 8 gets a card in the Today list asking for their interviews to be back-checked. It names the signs that stand out on their own, proposes up to five interviews to back-check first (the near-copies, then the most typical-looking), and drafts a note for their supervisor. Interviews already set aside are left out.

The Field Report's Team tab shows each enumerator's interviews, average length, flags per 100 interviews and open items, marks lengths under 60% of the team's typical length, and is labelled “for coaching, not ranking”. A report shared at the public level shows enumerators as codes. DataFlow works alongside the collection platform: it connects to SurveyLoopr, SurveyCTO and ODK Central, and reads exports from KoboToolbox and other tools.

Step by step

1. Identify every interview's enumerator. Have enumerators pick their ID from a list in the form, keep the same codes for the whole survey, and record the supervisor and area too.
2. Build a daily table per enumerator. Interviews per day, median length, flags per 100 interviews by check, don't-know and refusal shares, skipped sections, GPS flags and back-check agreement.
3. Compare with peers in similar areas. Put each enumerator beside colleagues working in similar places and with similar assignments, and look for signs of made-up interviews against the same group.
4. Read it fairly. Wait for enough interviews, look at trends rather than one day, look across several measures, and treat a check everyone trips as a problem with the question.
5. Act in order. Start with a conversation about specific interviews, then retrain, then back-check more of their work, and reassign only when problems are confirmed or continue.
6. Record each step. Note what the numbers showed, what was agreed and what changed, and set aside interviews that didn't take place with a reason rather than deleting them.

Questions