Consistency checks for survey data: logic rules and skip-logic errors

What a consistency check is

A range check looks at one answer at a time. A consistency check, or logic check, looks at two or more answers in the same interview and flags the ones that can't all be true. Each answer may be fine on its own: an age of 12 is fine, and so is “head of household”, but not for the same person. These checks run every day with the rest of the high-frequency checks, and they fall into three groups:

• impossible or unlikely combinations, such as a part larger than its whole;
• skip-logic errors, where a question was answered that the form should have skipped, or skipped when it should have been asked;
• rules across rows of a roster, such as a household with no head, or a household size that doesn't match the people listed.

A well-built form already blocks some contradictions with constraints. It only blocks the ones its author thought of, though, and only in interviews made with that version of the form. Checks on the data catch the rest, including values changed later by an edit on the server or a correction at the desk.

Impossible combinations

Write each rule as a condition that is true when the interview should be flagged, in the syntax XLSForm uses for relevance and constraints (see the XLSForm reference): variables as ${name}, comparisons, and, or and functions such as selected(). A rule written this way can be read by anyone who programmed the form, and moved into the form as a constraint later.

The last column matters as much as the rule. An impossible combination means one of the answers is wrong. An unlikely one is often real (a household can spend more than it earns by borrowing or selling an animal), so its flags are confirmed rather than corrected, and a rule that flags many interviews is a sign the rule is too tight. The same split between hard and soft limits applies to range checks.

Where a rule is impossible, put it in the form too, as a constraint on the second question. The constraint describes what's allowed, so it is the rule turned round:

Decide what a blank means in each rule. In ODK forms an empty value isn't treated as zero, and in Stata a missing value is larger than any number, so children_school > hhsize is true whenever the number of children in school is blank. Guard each comparison against blanks, and check blanks separately (see missing answers).

Combinations from a cross-tab

For two questions with a handful of answers each, a cross-tab shows every combination at once. Tabulate sex against pregnancy, relationship to the head against marital status, or school attendance against age group, and mark the cells that shouldn't have anyone in them: a male respondent who is pregnant, a 3-year-old who is married. A cell with a few interviews in it is a list of flags; a cell with dozens points at a question the enumerators read differently from the way it was meant.

* Every combination of two questions, blanks included
tabulate respondent_gender pregnant, missing
list hhid enum_id respondent_gender pregnant if respondent_gender == 1 & pregnant == 1

Skip-logic errors

Skip logic is the form's rule for who gets asked what: in XLSForm, a condition in the relevant column, such as ${owns_land} = 1 on the question about land size (SurveyLoopr's skip logic tip has more examples). A condition on a group applies to every question inside it. Two things can go wrong:

• A question is answered when its condition doesn't hold. The household owns no land, yet the land size is filled in.
• A required question is blank when its condition does hold. The household owns land, and the land size is empty.

In forms that follow the ODK XForms specification, an answer to a question that is no longer relevant is removed from the submission when it is sent, and a required question can't be left blank. So when tablet data shows a skip-logic error, look beyond the enumerator: a relevance condition written wrongly, a form version with different logic or without the required setting, an edit made on the server, data entered from paper, or a correction at the desk. Changing a filter answer from “yes” to “no” during cleaning leaves its follow-up questions answered, and they need a decision too.

* land_acres is asked when consent == 1 and owns_land == 1
* Answered although the form should have skipped it
list hhid enum_id owns_land land_acres if owns_land != 1 & !missing(land_acres)

* Blank although the form should have asked it
list hhid enum_id owns_land land_acres if consent == 1 & owns_land == 1 & missing(land_acres)

In Stata, owns_land != 1 is true when owns_land is blank, which is what you want here: a follow-up answered under a blank filter question is an error too. A check made from the form's own condition can't tell you whether the condition itself is right. Read each relevance against the paper questionnaire once, early, and watch for a skip that sends far more or far fewer respondents into a section than expected. A filter answered “no” far more often by one enumerator than by colleagues in the same area is a different problem, covered in signs of fabricated interviews.

Rules across a roster

Many contradictions sit between the rows of a roster, or between a roster and the household's own answers. Check that each household has exactly one head, that no head is a child, that no son or daughter is older than the head, and that the household size matches the number of people listed.

* Roster data, one row per member, with the household's answers merged in
use "roster.dta", clear
merge m:1 hhid using "household.dta", keepusing(hhsize enum_id) keep(match) nogenerate

* Exactly one head per household, and no head under 12
bysort hhid: egen heads = total(relation == 1)
bysort hhid: generate first = _n == 1
list hhid enum_id heads if first & heads != 1
list hhid enum_id member_id age if relation == 1 & age < 12

* Household size against the number of people listed
bysort hhid: generate n_rows = _N
list hhid enum_id hhsize n_rows if first & hhsize != n_rows & !missing(hhsize)

Keeping rules useful

• Test each rule on the pilot data before fieldwork. A rule that flags a fifth of the interviews has misread the form, or found a question that doesn't work.
• Give each rule a plain name and an action: “More children in school than household members: ask who the children in school are.” The field team acts on the action, not the code.
• Write the rules into the data quality plan, so the client knows what is checked (see the data quality assurance plan).
• Review the list after the first week. Retire rules that never fire on anything real, and add the contradictions the team keeps finding by hand.

What to do with a flag

Never correct both answers to make them agree without asking. Two answers that contradict each other are evidence; a guess that makes them consistent hides the error for good. Keep the raw data as it arrived and make every change in a log or a do-file that starts from it.

How DataFlow runs consistency checks

A rule check in DataFlow is a condition, written in XLSForm syntax, that describes what is wrong: ${variable}, = != < > <= >=, and, or, not(), arithmetic including div and mod, and selected(), count-selected(), string-length(), regex(), if(), coalesce(), number(), int(), abs() and round(). The rules in the table of impossible combinations are written as DataFlow accepts them. A rule that doesn't parse is explained in words (“Write variables as ${name}”), and functions it doesn't support, such as pulldata(), are named. In a rule you write, a blank answer never makes a comparison true, though not() around a comparison is true when the answer is blank; to flag a blank, write ${x} = ''. The editor counts how many interviews the rule would flag as you type.

Each rule can name the answer a correction would change, say what the field team should do, and be limited to some interviews (“Only check interviews where”, such as ${consent} = 1). It can also run on every row of a repeat group, with the interview's own answers alongside, and count(${members}) gives the number of rows: ${hhsize} != count(${members}) compares a household's size with its roster. A combination check takes two questions with up to 30 different answers each, shows their cross-tab with counts, and flags the cells you click.

Once interviews arrive, DataFlow also adds checks from the form's own logic: for each condition the form asks questions under, as long as its variables are in the data, a skip-logic check that flags a question answered when the condition doesn't hold, or a required one blank when every condition it is asked under holds; a check for answers in an interview that wasn't completed; a check for each constraint that isn't a simple range; and, for five or more one-choice questions in a group on the same scale, a check for the same answer all the way down. In conditions taken from the form, ${x} != 1 holds when x is blank, as the form itself treats it.

If a local AI model is downloaded, DataFlow's assistant reads each version of the form once, from the questions' names, labels, choices, constraints, conditions and each number's lowest and highest value (no answers, and no protected personal data), and proposes up to five cross-question rules. DataFlow keeps only those that parse, use variables in the data, aren't already checked, and would flag no more than 5% of interviews. You can also ask the assistant for a rule in words; it prepares the check, counts what it would flag, and adds it only if you approve. Any suggested check that would flag more than 5% of interviews starts switched off, unless at least half of its flags come from one enumerator.

Each flag shows the rule and the answers that matched it. You can fix the value, which needs a reason; mark it as right as recorded, with an optional note of how it was checked; or, for a rule that names an answer to correct, mark its value as fine for that check, so it is never flagged again. Corrections are logged and reversible, and the raw data is never changed. DataFlow connects to SurveyLoopr, SurveyCTO, ODK Central and KoboToolbox, and reads CSV or Excel exports from other tools, so the team keeps its collection platform.

Questions