Range checks for survey data: setting hard and soft limits

What a range check is

A range check compares each numeric answer with the lowest and highest values it may take, and flags anything outside them: an age of 150, a household of 0 people, a plot of 4,000 acres in a village where most farms are under five. It is the simplest of the data validation rules, and usually the first one written for a survey, because the limits come straight from the questionnaire. Range checks run every day with the other high-frequency checks, starting with the variables the analysis depends on.

A range check is different from an outlier check. An outlier check compares a value with everyone else's answers to the same question, so its limits move as data comes in. A range check uses limits decided in advance, so a value outside them is either impossible or rare enough that someone should look at it, however many other households gave a similar answer.

Hard limits and soft limits

Most teams set two kinds of limit for each number. A hard limit marks values that can't be true. A soft limit marks values that can be true but are rare enough to confirm. J-PAL's guide to survey programming draws the same line for form constraints: a hard constraint stops the enumerator until the value changes, and a soft constraint, for answers that are unusual but possible, adds a follow-up question instead.

Keep the two apart in the daily list. A hard-limit flag needs a correction. A soft-limit flag is often right, and treating it as an error teaches the team to “fix” real answers.

Setting limits from the questionnaire

Write the limits down for every numeric question before the form is programmed, in the same sheet as the questionnaire, and agree them with whoever will analyse the data. Four things set them:

• What can't be true at all: negative counts, ages, amounts and areas; shares above 100; more than 24 hours in a day or 168 in a week; a household with nobody in it.
• Who is eligible. If only adults are interviewed, the respondent's age has a hard lower limit of 18. If the sample is farming households, no land at all is worth a question.
• The unit and the recall period. A limit for monthly income is different from one for weekly spending, or for an amount recorded in thousands.
• What earlier sources show. A previous round, a census or the listing gives the usual spread of household size, land and income in the area.

Set hard limits generously. A hard limit that is too tight blocks a real answer in the field, and the enumerator then types the nearest value the form accepts. That is worse than a typo, because nothing will ever flag it. When in doubt, make it a soft limit.

Setting limits from the data

Soft limits are easier to set once interviews arrive. After the pilot or the first few days of fieldwork, look at each variable's distribution and put the soft limits near the edges of what almost everyone reports; the 1st and 99th percentiles are a common starting point. Set them within groups where answers differ by area, such as land in the hills and in the plains. Then count what each limit flags. A limit that flags one or two interviews a day gets followed up; one that flags a tenth of them gets ignored, and the real errors hide among the rest.

Review soft limits once a week during fieldwork. Hard limits shouldn't move. If the data keeps breaking one, either the question is being misunderstood or the limit was wrong from the start, and both need a decision from the research team rather than a quiet change to the check.

The form's constraint column comes first

The cheapest place to stop an impossible value is the tablet. In XLSForm, the constraint column holds a condition the answer must meet, with . standing for the answer itself, and constraint_message holds what the enumerator sees when it fails (see the XLSForm reference, or SurveyLoopr's constraint examples for ranges, dates and phone numbers). The form won't move on until the answer fits:

The last row is a soft limit. The question appears only when the household has more than 15 members, and the enumerator either confirms the number with the respondent or goes back and corrects it. The “yes” stays in the data as a record that the value was checked at the door.

Constraints don't make range checks unnecessary. ODK's documentation notes that constraints aren't evaluated when an answer is left blank, and a constraint only applies to interviews made with the form version that has it. A range check on the data that arrives still catches:

• interviews made on an older version of the form, before a constraint was added or corrected;
• soft limits, which a form can ask about but not enforce;
• values changed after the interview, by an edit on the server or a correction at the desk;
• data entered from paper, or exported from a tool without constraints;
• a constraint written with or where and was meant, such as . >= 0 or . <= 100, which every number passes.

Range checks in Stata

In Stata, turn the don't-know and refusal codes into missing values first, so that -999 isn't read as an income, then use assert for hard limits and a list for the daily report. Stata's manual describes assert as a way to certify data: when the statement is false it reports how many observations contradict it and stops with an error, and the rc0 option reports the count without stopping.

* Don't-know and refusal codes become extended missing values first
mvdecode respondent_age hhsize income_month land_acres, mv(-999=.d \ -888=.r)

* Hard limits: say how many interviews break each one
assert inrange(respondent_age, 18, 100) if !missing(respondent_age), rc0
assert inrange(hhsize, 1, 30) if !missing(hhsize), rc0

* The daily list: every breach, with who to ask
list hhid enum_id respondent_age if !inrange(respondent_age, 18, 100) & !missing(respondent_age)
list hhid enum_id hhsize if !inrange(hhsize, 1, 30) & !missing(hhsize)

* A soft limit from the data: land above the 99th percentile, within each district
bysort district: egen p99 = pctile(land_acres), p(99)
list hhid enum_id district land_acres p99 if land_acres > p99 & !missing(land_acres)

Two details catch people out. Stata sorts missing values above every number (as its misstable manual sets out), so hhsize > 30 is true for every blank household size unless you add !missing(hhsize). And inrange() is false for a missing value, which is why the manual's own example adds if !missing(). Blanks are a separate check, covered in missing answers.

Codes, blanks and rosters

Don't-know and refusal codes such as -999 and -888 sit outside almost every range. Leave them out of the range check and count them in a check of their own (see don't know and refusal rates). If the form allows a code, the check must allow it too, or every honest “don't know” becomes a flag.

For questions in a roster (each member's age, each plot's area), run the check on every row, not one row per interview: an age of 140 on row 3 is as wrong as a respondent aged 140. Rules that compare a row with the household's own answers, such as a child older than the head, are logic checks rather than range checks; see consistency checks.

What to do with a flag

Don't move a wrong value to the nearest limit, and don't blank it without a record. A value set to the limit looks real, and a blank hides that anything was there. Capping extreme values is an analysis decision, made later and documented. Keep the raw data as it arrived, so every correction can be checked and reversed.

How DataFlow runs range checks

DataFlow's range check has a lowest and a highest allowed value, either of which can be left open, and a list of codes to ignore. It reads each answer as a number and flags it when it falls outside the limits; blanks, and text that isn't a number, are left to other checks. On import it adds a range check for each integer or decimal question whose form constraint is a simple range, such as . >= 18 and . <= 100 or . > 0, with the constraint's own limits and whether each end is allowed. The ignore list starts with the don't-know and refusal codes declared in the form's choice lists, or -999, -998, -888, -99, -98 and -97 when the form declares none. Questions in a repeat group are checked on every row.

A constraint that isn't a simple range, because it has an or, a bracket or another question in it (. <= ${hhsize}), becomes a rule check, “follows the form's rule”, once interviews arrive. For number questions the form sets no limits on, DataFlow suggests a range from the answers so far, once a question has at least 20 answers and none below zero: from 0 to twice the value 99% of answers stay under, rounded up to a round number. For a question named like an age the upper limit is 120 instead, and 100 for one named like a percentage or share, when the answers so far stay under it. These start switched off, with the limits shown, for someone to check before turning them on. Any suggested check that would flag more than 5% of interviews also starts off, unless at least half of its flags come from one enumerator.

Each flag names the value and the limits it broke. When a flagged positive whole number ending in 0, divided by 10, 100 or 1,000, lands inside the limits, DataFlow proposes removing the extra zeros, as corrections you approve; it offers the same for outlier flags. For any other flag you can fix the value, which needs a reason; mark it as right as recorded, with an optional note of how it was checked, so it stays closed on later runs; or mark that value as fine for the check, so it is never flagged again. Every correction is logged and can be undone, and the raw data is never changed. DataFlow works alongside the collection platform: it connects to SurveyLoopr, SurveyCTO, ODK Central and KoboToolbox, and reads CSV or Excel exports from other tools.

Questions