Don't know and refused answers: checking their share in survey data

Why the share matters

A respondent who can't remember last month's spending, or who doesn't want to talk about income, should be able to say so, and the data should record it. These answers are information: a question that most people can't answer, or won't, tells you something about the question and the people asked. What makes them worth checking is that the share also depends on the enumerator. One who accepts the first “don't know” without probing (asking again in another way, or helping the respondent work the answer out) collects many more than a colleague who probes. One who pushes for an answer, or guesses, collects fewer. Someone filling in interviews away from the respondent tends to avoid them altogether.

The World Bank's DIME Wiki lists the percentage of “don't know” responses among its enumerator checks in high-frequency checks, and notes that a high share may mean the questions need re-drafting. That is the two-sided reading this page is about: first the questions, then the people. It belongs in the daily high-frequency checks, next to missing answers.

The codes matter for analysis too. If a don't-know code such as -99 stays in an income variable as a number, every mean, total and regression that uses it is wrong, and an outlier check will either flag it every day or have its limits pulled about by it. The codes have to be recognised everywhere the data is used.

Coding them in the form

Pick one set of codes for the whole questionnaire and write it into the codebook before the pilot. The codes should be values that can't be real answers: negative codes work for counts, ages and amounts, which can't be negative. Keep “don't know” and “refused” apart, because they mean different things and lead to different follow-up. Some teams use -98 and -99, others -999 and -888; which you choose matters less than using the same codes in every question, every form version and every round.

In XLSForm, put the codes in the choice lists of single-choice questions, and allow them in the constraints of number questions, with a hint so the enumerator knows what to type:

The constraint on the multiple-choice question stops “don't know” being ticked alongside real answers. Where a number can legitimately be negative, such as a business's profit, no code is safe: ask first whether the respondent can give the amount, with a single-choice question that has the don't-know and refused options, and ask for the number only when they can.

The codes travel through the form's logic, so check the places they reach. A don't-know on a filter question skips any section that is asked only when the answer is “yes”; decide whether that is what you want. A code in a number that other questions use breaks their constraints and calculations: a household size of -98 makes a constraint such as . <= ${hhsize} impossible for every later count, so write it as . <= ${hhsize} or ${hhsize} < 0, and keep codes out of calculated totals. Enter the codes on purpose during the pilot to test each of these paths.

Decide, and train, whether “don't know” is read out as an option or recorded only when the respondent says it unprompted. Either rule can be right for a study, but a team where some enumerators read it out and others don't will produce very different shares for the same question, and the check will compare habits instead of interviews.

Computing the share in Stata

Count the codes over the questions that can hold them, and divide by the questions answered, so an interview that skipped a long section by design isn't compared on the same footing as one that didn't. Leave out interviews that ended early by design (refusals, nobody at home). Here the numeric questions are named q_*:

* Share of don't know (-98) and refused (-99) answers per interview
use "survey.dta", clear
keep if consent == 1
egen dk       = anycount(q_*), values(-98)
egen refused  = anycount(q_*), values(-99)
egen answered = rownonmiss(q_*)
generate share = (dk + refused) / answered if answered >= 5

* Interviews where more than a fifth of the answers are codes
list hhid enum_id dk refused answered share if share > 0.2 & !missing(share), noobs

* Each enumerator's share of all the answers they recorded
preserve
generate one = 1
collapse (sum) interviews=one dk refused answered, by(enum_id)
generate share = (dk + refused) / answered
gsort -share
list enum_id interviews dk refused share, noobs
restore

Then look at the questions themselves, for everyone and enumerator by enumerator:

* Which questions collect the most codes
foreach q of varlist q_* {
    quietly count if inlist(`q', -98, -99)
    local codes = r(N)
    quietly count if !missing(`q')
    if r(N) >= 20 display "`q'" _col(24) %5.1f 100 * `codes' / r(N) "%  of " r(N)
}

* One question, enumerator by enumerator
generate dk_income = inlist(income_month, -98, -99) if !missing(income_month)
tabstat dk_income, by(enum_id) statistics(mean n)

Before any analysis, turn the codes into Stata's extended missing values. Stata's mvdecode changes the numbers you list into missing-value codes, either the plain . or one of .a to .z. Using .d for don't know and .r for refused keeps the reason while taking the codes out of means and totals:

mvdecode q_* income_month, mv(-98=.d \ -99=.r)
summarize income_month         // the codes no longer count as amounts
count if income_month == .d    // and are still there when you need them

Reading the share fairly

Start with the questions, because a question that most respondents can't answer will push up the share of every enumerator who asks it. Recall questions over a long period, amounts in a unit people don't use, ages of older relatives, and anything translated awkwardly all collect more “don't know” answers. Some questions are hard by nature, and a high share there is a finding about the question, not a fault.

Then compare enumerators, with the same care as any other enumerator monitoring:

• Compare like with like: the same questions, similar areas and similar respondents. A household member answering for an absent head (a proxy respondent) knows less about the head's work and income, so an enumerator given many such households will record more “don't know”.
• Use shares, not counts, and wait for enough interviews. In the first days a single tired respondent moves a share a long way.
• Follow the share day by day as well as in total. A share that jumps after a new form version, or when a new enumerator joins, tells you when and where the change started.
• Read refusals apart from don't-knows. Refusals tend to gather on sensitive questions such as income, debt or violence, and in interviews done without privacy. A respondent's refusal is their right; what you check is whether one enumerator gets far more than colleagues asking the same people the same questions.
• Treat a very low share as a sign too. When everyone else records a few “don't know” answers per interview and one enumerator records almost none, look at their other measures. Few don't-knows together with short interviews and near-copies is one of the patterns in signs of fabricated interviews.

What to do with each finding

Never replace a don't-know code with a value at the desk. A code changes only when the respondent gives the answer on a call-back or revisit, and then as a logged correction: the interview, the question, the old and new value, who changed it and why. Keep the raw file as it arrived. The same codes have to be kept out of outlier checks and range checks, or each one becomes a flag every day.

How DataFlow checks don't-know and refused answers

DataFlow reads the codes from the form's choice lists: any choice whose label mentions “know”, “refus” or “dk”, and any negative code whose label contains “not”. When the form has none, it uses -999, -998, -888, -99, -98 and -97. On import it adds the check “Not too many ‘don't know’ answers” when the form has at least five questions that can hold those codes: number questions, and single-choice questions whose list includes one, leaving out ID variables and the interview length. It flags an interview when more than 20% of the answered questions among them are don't-know or refused codes, once at least 5 are answered. It looks only at completed interviews, by the rule DataFlow finds from the form (such as consent given), which can be changed in Settings. The share, the minimum answered, the codes and the list of questions can all be edited. It starts switched off if it would flag more than 5% of interviews, unless at least half of its flags come from one enumerator.

In the Today list, when one enumerator has at least four of these flags and at least half of them, DataFlow shows one card, “Most ‘don't know’ answers are” theirs, with a note to copy for their supervisor: it asks them to go over probing with the enumerator and to revisit the listed respondents where key answers are missing. The card comes back after two days. The other flagged interviews appear together, with the suggestion to revisit the ones where key answers are missing.

The opposite pattern is one of DataFlow's six signs of made-up interviews: don't-know and refusal codes (-999, -998, -888, -99, -98, -97) per interview, below the team's. It is computed for each enumerator with at least 10 interviews, when at least four enumerators have that many to compare, and added to the other five signs; an enumerator whose total is over 8 gets a card asking for their interviews to be back-checked. The suggested outlier and range checks ignore the form's don't-know codes. In a Stata download, the codes from the form become extended missing values (.d for don't know, .r for refused, .n for not applicable, or -999 as .d and -888 as .r when the form names none), with labels, and the do-file that rebuilds the data from the raw file does the same with mvdecode. DataFlow works alongside the collection platform: it connects to SurveyLoopr, SurveyCTO, ODK Central and KoboToolbox, and reads exports from other tools.

Questions