High-frequency checks for survey data

What high-frequency checks are

High-frequency checks (HFCs) are a fixed set of survey data quality checks that run on each new batch of interviews while data collection is going on. They are written before fieldwork, alongside the questionnaire, tested on pilot data, and run every time new submissions arrive. The World Bank's DIME Wiki describes them as part of a data quality assurance plan, together with back-checks. Research teams often write them in Stata, using IPA's ipacheck or arceddataflow, or in R.

They run daily because a check is worth less every day after the interview. While the team is in the area, the enumerator remembers the household, the respondent can be called or visited again, and a bad habit can be corrected before it spreads to hundreds more interviews. Once fieldwork ends, a wrong value can only be dropped, imputed or left in. Daily checks also test the questionnaire: a question that confuses every enumerator shows up in the first few days, when the form can still be changed.

The checks, one by one

Most HFC sets cover the same ground. The table is the short version; each check is described below it.

IDs and duplicates

Every interview should have one ID, and every ID one interview. A duplicate can be the same interview submitted twice, two enumerators visiting the same household, or a mistyped ID that now matches a different household. Compare the IDs with the sample list as well: IDs that aren't in the sample, and sampled households with no interview yet, tell you about coverage. Identical copies can be set aside; different households sharing an ID need the field team to say which ID is right.

Interview duration

Very short interviews may have been rushed, skipped whole sections or not taken place at all; very long ones were often left open or interrupted. Use the form's own start and end times or duration field, and once a few days of data have arrived, set the limits from the median rather than a guess. Leave out refusals and other interviews that end early by design. If your platform records time per question (SurveyCTO's text audits, for example), a long module answered in seconds is a sharper signal than the total.

Outliers

An outlier is a value far from the rest of the same variable, such as one household reporting ten times the usual income. It may be an extra zero, a yearly figure in a monthly question, or a real but rare case, so each one is verified rather than deleted. The two common rules (interquartile ranges and standard deviations) behave very differently; see outliers in survey data.

Allowed ranges

A range check, the simplest of the data validation checks, flags values that can't be true: an age above 120, a percentage above 100, a household with no members. Unlike an outlier, a value outside the range is wrong by definition. Forms usually enforce the hard limits already, so range checks matter most for limits the form didn't set, for interviews made with an older form version, and for data entered somewhere else.

Missing answers

A question that every respondent is asked should never be blank. When it is, the cause is usually a skip-logic error, a change between form versions, or a problem in the export. Check only questions that are always asked; blanks in questions skipped by relevance are expected.

“Don't know” and refusal share

Count the share of an interview's answers that are “don't know” or “refused” (often coded -99, -98 or -88), and compare it across enumerators. A high share can mean an enumerator who doesn't probe, a tired respondent or a question that doesn't work. A share close to zero, when everyone else's isn't, can be a warning sign too. The same codes must be kept out of averages and outlier checks.

“Other, specify”

Read the free-text answers behind “other”. Many fit an existing option and should be recoded to it; the same new answer coming up again and again means the form needs a new option, ideally while fieldwork is still running.

Logic rules and impossible combinations

A logic check describes something that shouldn't happen, such as more children in school than people in the household, written in XLSForm style as ${children_school} > ${hhsize}. A combination check lists pairs of answers that can't occur together, such as a male respondent who is pregnant, which a cross-tab of the two questions shows quickly. Form constraints catch some of this at entry, but only what the form's author thought of.

GPS location

If the form records a GPS point, check that it's there, that its accuracy is good enough, that the interview is near the village it names, that an enumerator could have travelled between two interviews in the time between them, and that one enumerator's interviews aren't all recorded from the same spot. Several interviews from one place can mean they were filled in at a tea stall.

Back-checks

A back-check sends a separate team to re-interview a random sample of respondents with a short set of questions, then compares the answers with the original interview. It is the only check that goes back to the respondent, so it is the one that confirms an interview happened. See back-checks for the questions, the sample and how to read agreement rates.

Signs of fabricated interviews

Invented interviews tend to look too tidy. Compare each enumerator with the others working in the same area on signs such as: interviews that are near-copies of each other, numbers that vary less than the team's, many answers equal to the most common answer, many round amounts, hardly any “don't know”, and “no” to the filter questions that would open a long section. None of these proves anything alone; together they tell you whose interviews to back-check first.

Personal data

Check each batch for variables that identify people: names, phone numbers, ID numbers, GPS points, and comments that mention someone by name. Keep them out of the files the team works on and shares. The guide to de-identifying survey data covers how.

Reading results by enumerator

Many problems in field data trace back to one person's habits, so read the results enumerator by enumerator, not only check by check. A few rules keep that fair:

• Use rates, not counts. An enumerator with 80 interviews will have more flags than one with 20, so divide by interviews done.
• Compare enumerators working in similar places. Durations, travel times and answers differ between villages, and a remote area isn't a sign of trouble.
• Look across checks. One kind of flag can be a hard area or bad luck. Short interviews, few “don't know” answers and near-copies together are a pattern.
• Wait for enough interviews. In the first days a single interview moves a rate a long way, so watch the trend before drawing conclusions.
• If every enumerator trips the same check, the problem is the question or the check, not the people.

Send the results to field supervisors every day, with interview IDs and the exact questions, so they can follow up the same day. A summary like “E07 has many short interviews” gives them nothing to act on.

Acting on each flag

Every flag should end in one of three outcomes:

1. Correct it, with a log. When the value is wrong and the right one is known, change it in a cleaning script or correction log that records the interview ID, the variable, the old and new values, who changed it and why. Never edit the raw data file: keep it as it arrived, so every correction can be checked and undone.
2. Confirm it as correct. A household of 19 people can be real. Record that the value was checked and by whom, so it doesn't come back in tomorrow's list.
3. Send it back to the field. When nobody at the desk can tell, ask the supervisor to call or revisit the respondent with a specific question, and set a date to follow up.

Interviews that shouldn't be in the data at all, such as a duplicate copy or a test interview, are set aside with a reason rather than deleted. In Stata, the simplest correction log is a do-file that starts from the raw data and changes one value per line, with the reason beside it:

use "raw/survey.dta", clear
replace income_month = 6000 if hhid == "H0412"   // extra zero, confirmed by supervisor
replace hhsize = 5 if hhid == "H0388"            // roster lists 5 members
save "clean/survey_clean.dta", replace

How DataFlow runs these checks

DataFlow runs this set of checks on every sync. It connects to SurveyLoopr, SurveyCTO and ODK Central, and reads CSV or Excel exports from KoboToolbox and any other tool, so the team keeps its collection platform and adds a QA layer. On import it reads the form and the data and adds checks itself: duplicates on ID variables; interview length against the median (under 40% of it or 5 minutes, whichever is longer, or over 3 times it); ranges from the form's constraints; outliers beyond 3 interquartile ranges; blanks in required questions that are always asked; more than 20% “don't know” answers; “other, specify” answers, which a small local model matches to existing options; and GPS checks for missing points, accuracy worse than 50 m, interviews more than 5 km from the middle of their area's other interviews, travel faster than 60 km/h, and three of one enumerator's interviews within 15 m. A local AI reads cross-question rules from the form. A suggested check that would flag more than 5% of interviews starts switched off, unless most of its flags come from one enumerator.

It also computes six statistical signs of made-up interviews per enumerator, compares back-checks by enumerator and question, and keeps personal data in a separate encrypted store. Corrections are logged and reversible, the raw data is never changed, and every download includes a Stata do-file that replays each correction. For an openly licensed checklist of field data checks, see the Field Data Quality Standard.

Every check

• Back-checks: how to run a back check survey: How to run a back check survey: which questions to re-ask, how to sample by enumerator, how to compare answers, and what to do when they disagree.
• Outliers in survey data: how to find and handle them: How to find outliers in survey data with the IQR or standard deviations, within groups, with Stata code, and what to do with each value you flag.
• Duplicate survey responses: finding repeated IDs and interviews: How to find duplicate survey responses: repeated IDs, households interviewed twice, forms sent twice and copied interviews, and what to do with each.
• Interview duration checks: interviews that are too short or too long: How to check survey duration: where interview length comes from, setting limits from the median, reading it by enumerator, and handling speeders.

Questions