Missing data in surveys: telling a missing answer from a skipped question

Five kinds of empty

In a data file, an empty cell can mean very different things. Survey researchers separate unit non-response, where the whole interview didn't happen (a refusal, nobody at home), from item non-response, where the interview happened but a question that was asked has no answer. Only the second is a missing answer in the sense of this page, and even then a blank cell is only one of the ways it shows up.

A missing-answer check is one of the daily high-frequency checks. Its whole difficulty is the first row: in a form with skip logic, many blank cells are correct skips, and a check that flags every blank buries the real ones among them.

Skipped by design

Skip logic lives in the form's relevant column (see the XLSForm reference). A question with a condition is only shown when the condition holds, and a condition on a group applies to every question inside it. In forms that follow the ODK XForms specification, an answer to a question that isn't relevant is removed from the submission when it is sent, so in the export it is simply blank, exactly like a question the enumerator failed to answer. Nothing in the cell tells the two apart; only the condition does.

Here consent is always asked, hhsize is asked when the respondent consents, and land_acres only when the respondent consents and the household owns land. A blank land_acres is correct in a refusal, correct in a household without land, and a missing answer only in a consenting household that owns land.

Telling a missing answer from a correct skip

For each question, write down the full condition under which it is asked: its own relevance and that of every group around it. Then each interview falls into one of four cells:

* land_acres is asked when consent == 1 and owns_land == 1
generate asked_land = consent == 1 & owns_land == 1

count if asked_land & missing(land_acres)      // asked, not answered
count if !asked_land & !missing(land_acres)    // answered, but should have been skipped
list hhid enum_id if asked_land & land_acres == .   // a true blank, not a coded .d or .r

The last line uses Stata's missing values to keep two kinds apart. Once don't-know and refusal codes are turned into extended missing values (.d, .r), missing() is true for all of them, while == . finds only the true blanks.

Blanks and codes

A code shows that the question was asked and the respondent didn't give an answer; a blank shows nothing at all. That is why many forms offer “don't know” and “refused” as options with negative codes. J-PAL's guide to survey programming suggests offering such answers and codes like -777 for “don't know” and -888 for “refused”; other teams use -99 and -98, or -999. Pick one set for the project, use it in every question, and write it in the codebook.

Codes have to be handled before any number is used. A -999 in an income variable drags the mean down and gets flagged as an outlier; in a range check it breaks every lower limit. In Stata, turn them into extended missing values with mvdecode, so they drop out of calculations but stay distinguishable from a true blank. How often each enumerator records them is its own check, covered in don't know and refusal rates.

“Not applicable” needs a decision of its own. Where the form can tell from earlier answers that a question doesn't apply, skip it with relevance instead of offering a code, so the enumerator never has to choose. Where only the respondent can say, offer it as an option with its own code and map it to its own missing value (.n in Stata), so it isn't counted with refusals. Free-text questions need the same care: decide whether “none” or “no comment” is typed as an answer or left blank, and tell the enumerators, or the blanks in a comments field will mean three different things.

Missing values in Stata

Stata's misstable command tabulates missing values. misstable summarize counts, for each variable, the system missing values (.) and the extended ones (.a to .z) in separate columns, and misstable patterns shows which variables tend to be missing together. By default it lists only numeric variables that have missing values, so count blank text answers yourself.

* Don't-know and refusal codes become .d and .r first
mvdecode hhsize land_acres income_month, mv(-999=.d \ -888=.r)

* Blanks (.) and codes (.d, .r) per variable, then which go missing together
misstable summarize hhsize land_acres income_month, all showzeros
misstable patterns hhsize land_acres income_month

* Text questions aren't in misstable
count if consent == 1 & missing(respondent_name)

* True blanks by enumerator, among interviews where the question is asked
generate blank_income = income_month == . if consent == 1
tabstat blank_income, by(enum_id) statistics(mean n)

Read the result with the conditions in mind. A raw count of missing values for land_acres mostly counts households without land. For questions always asked, compare the share of blanks across enumerators: one enumerator well above the rest is skipping questions. IPA's ipacheck includes ipacheckmissing, which exports missingness statistics for every variable.

Why answers go missing

• The question isn't required. ODK's documentation notes that without the required setting, users can skip questions.
• The relevance condition is wrong: it checks for 'yes' where the choice is coded 1, or names the wrong question, so the question is never shown to the people it was meant for.
• The question was added in a later form version, so interviews made before it are blank.
• The interview ended early, and everything after the break is empty.
• The export or a merge lost the column, or split it under another name, often with group paths or repeat numbers.
• A correction at the desk blanked a value without saying why.

Each cause has a different fix, and the pattern shows which it is. Blanks in one question across the whole team point at the form or the export. Blanks across many questions from one enumerator point at that person. Blanks in everything after a certain question point at interviews that ended early.

Preventing missing answers in the form

Make the questions that every respondent must answer required, and give each one a way out: a don't-know and a refused option, so the enumerator never has to invent an answer to move on. J-PAL's guide makes the same point, warning that making every question required is inflexible. For number questions, allow the codes in the constraint:

Test every relevance condition on the pilot by answering each filter both ways and confirming that the right questions appear. In the first days of data, look for any question that nobody has answered, or that far more or far fewer people answer than the questionnaire expects.

What to do with each missing answer

Don't impute missing answers during cleaning. Filling a blank with the median or a guess is an analysis decision, made later and documented, and an imputed value in the cleaned data looks exactly like a real one. Keep the raw data as it arrived and make every change in a log or a do-file that starts from it.

How DataFlow checks for missing answers

On import, DataFlow adds a missing-answer check for each question that the form marks as required and that is always asked: no relevance on the question itself or on any group around it. It flags an interview where that answer is empty or only spaces. A code such as -999 isn't blank, so this check doesn't flag it; the don't-know check counts those, flagging an interview where more than 20% of its answers are don't know or refused, once at least 5 are answered.

Required questions asked under a condition are covered by the skip-logic checks DataFlow adds from the form once interviews arrive. For each condition the form asks questions under, as long as its variables are in the data, the check flags a required question that is blank when every condition it is asked under holds, and a question answered when its condition doesn't hold. So neither check flags a question that was correctly skipped. A separate check flags answers in an interview that wasn't completed, by the project's rule for a completed interview: usually the condition, such as consent, that at least 40% of the form's questions hang on, or else an interview-outcome question; it can be changed in Settings. You can add a missing-answer check to any other question, limited to the interviews where it is asked with “Only check interviews where”, such as ${consent} = 1. Any suggested check that would flag more than 5% of interviews starts switched off, unless at least half of its flags come from one enumerator.

A flag reads “This is a required question, but it's blank”, with the enumerator to ask. You can fill the value in, which needs a reason, or mark the blank as right as recorded, with an optional note of how it was checked. Every correction is logged and can be undone, and the raw data is never changed. In downloads, the don't-know, refused and not-applicable codes in the form's choice lists become Stata's .d, .r and .n, labelled (-999 and -888 become .d and .r when the form declares none). DataFlow connects to SurveyLoopr, SurveyCTO, ODK Central and KoboToolbox, and reads CSV or Excel exports from other tools, so the team keeps its collection platform.

Questions