Outliers in survey data: how to find and handle them

What counts as an outlier

An outlier is a value far from the other values of the same variable: one household reporting ten times the usual income, a plot ten times the usual size. In field surveys most outliers come from a few sources: an extra zero typed on a tablet, an answer in the wrong unit (a year's income in a monthly question, grams instead of kilograms), a code such as -99 treated as a number, or a real but rare household. The last kind must stay in the data, which is why outlier detection flags values for checking rather than removing them.

An outlier is different from a range error. A range check flags values that can't be true, such as an age of 150. An outlier check flags values that are unusual compared with everyone else, and may be true. Both belong in the daily high-frequency checks.

Two ways to measure “far”

The interquartile range (IQR)

Sort the values and find the first quartile (Q1, a quarter of values below it) and the third quartile (Q3, three quarters below it). The interquartile range is Q3 minus Q1: the width of the middle half of the data. A value is an outlier if it lies more than k IQRs below Q1 or above Q3. John Tukey's boxplot rule uses k = 1.5, and he called values beyond 3 IQRs “far out”. Because quartiles barely move when an extreme value is added, the IQR rule is robust: one or two wild values don't change the limits much.

Standard deviations (SD)

Work out the mean and standard deviation, and flag values more than k SDs from the mean. This is the classic rule in Stata high-frequency-check code: arceddataflow uses it with k = 3, IPA's ipacheck offers it alongside the IQR, and the World Bank's DIME Wiki gives 2 SDs as an example. It is familiar and easy to compute, but both the mean and the SD are pulled by the very values you are looking for. A big enough error inflates the SD until the error sits inside its own limits.

Small groups make this worse. With n values, no value can be more than (n - 1)/√n sample standard deviations from the mean: 2.85 for 10 values, 3.33 for 13. So a 3 SD rule can never flag anything in a group of 10 interviews or fewer, however wrong a value is.

A worked example

Thirteen households report monthly income: 4,000, 4,500, 5,000, 5,200, 5,500, 6,000, 6,000, 6,500, 7,000, 7,500, 8,000, 9,000 and 60,000. The last one is 6,000 with an extra zero. Without the typo the standard deviation would be 1,422; with it, 14,994. Now suppose a second typo, 5,000 entered as 50,000:

With one typo, both rules catch it, the SD rule only just. With two, the errors mask each other and the 3 SD rule flags nothing; even at 2 SD it misses the 50,000. The IQR limits hardly move. The quartiles here are the ones Stata's summarize, detail and R's default give; other tools define quartiles slightly differently, so limits can shift a little between them.

Notice the lower limits too: both rules put them below zero for this income, so a dropped zero (600 for 6,000) is never flagged at k = 3. For skewed amounts like income, add a range check with a sensible minimum, or use a smaller k on the low side.

Within groups

A value that is normal in one place can be extreme in another: land holdings in the plains and in the hills, prices in towns and villages. Compute the limits within groups such as region, district or stratum, so each value is compared with its neighbours. Each group needs enough values for its limits to mean anything, which rules out the SD rule for small groups.

Grouping by enumerator finds values that are odd for that enumerator, but it hides an enumerator whose values are all inflated: inside their own group, they look normal. Run outliers by area, and compare enumerators' medians with each other separately. For roster questions (each member's age, each plot's area), check every row of the roster together rather than one row per interview.

Codes to leave out

Don't-know and refusal codes such as -99, -98 or -88, and fillers like 999, are not amounts. One -99 in an income variable drags the mean down and the SD up, and gets flagged itself as a low outlier. Take them out before checking. In Stata, recode them to extended missing values (.d for don't know, .r for refused), as arceddataflow does before its checks. Remember that Stata treats missing values as larger than any number, so a condition like income > limit is true for missing values unless you add !missing(income).

Unit errors

Many large outliers are the right answer in the wrong unit: a yearly figure in a monthly question (12 times too big), an amount in units instead of thousands (1,000 times), or an extra zero (10 or 100 times). A quick test: if dividing the value by 10, 100, 1,000 or 12 lands it in the usual range, a unit or typing error is likely. Still confirm it with the field team before changing it. To prevent unit errors, ask the unit as its own question, or show the converted amount back to the enumerator in the form with a calculation and a note.

Stata code for both rules

* Don't-know and refusal codes become extended missing values first
mvdecode income_month, mv(-99=.d \ -98=.r)

* Interquartile range, within each district
bysort district: egen q1 = pctile(income_month), p(25)
bysort district: egen q3 = pctile(income_month), p(75)
generate iqr = q3 - q1
generate out_iqr = (income_month < q1 - 3*iqr | income_month > q3 + 3*iqr) if !missing(income_month)

* Standard deviations, within each district
bysort district: egen mean = mean(income_month)
bysort district: egen sd = sd(income_month)
generate out_sd = (income_month < mean - 3*sd | income_month > mean + 3*sd) if !missing(income_month)

list hhid enum_id district income_month q1 q3 if out_iqr == 1

Two cautions. In a district where Q1 equals Q3 (many identical answers), the IQR is zero and every other value is flagged; skip such groups or check them by hand. And count the values per group before trusting either rule. IPA's ipacheck has a ready-made command, ipacheckoutliers, that lets you choose the method per variable; it defaults to 1.5 IQRs, or 3 SDs when you choose standard deviations.

What to do with an outlier

1. Verify it. Ask the supervisor or enumerator, look at their notes, photos or audio if the form collects them, and call the respondent if needed.
2. If it is an error, correct it with a log: change it in a cleaning script or correction log that keeps the old value, the new one, who changed it and why. Leave the raw data file as it arrived.
3. If it is real, confirm it and record that it was checked, so it doesn't come back in every daily report. Whether to top-code or winsorize it is an analysis decision, made later and documented, not part of cleaning.
4. If nobody can tell, send it back to the field with a specific question and a date to follow up.

Never delete an outlier or replace it with missing without a record. A value that disappears silently can't be checked by anyone later, including you.

How DataFlow finds outliers

DataFlow's outlier check offers both methods: the IQR (the default) or standard deviations, with a sensitivity k (3 by default), an optional grouping variable and a list of values to ignore. The ignore list starts with the don't-know and refusal codes declared in the form's choice lists, or -999, -998, -888, -99, -98 and -97 when the form declares none. The IQR limits are only computed for groups with at least 10 values and Q3 above Q1; the SD limits need at least 2 values and an SD above zero. The SD method is mean ± k sample SDs, the same rule as arceddataflow's outlier check, and DataFlow is tested to flag what arceddataflow flags.

On import it suggests the check for number questions with at least 12 different values (before data arrives, for questions whose names say money, size or quantity), including rows of repeat groups, and leaves out IDs and the duration. Each flag shows the value, its group's median and the limits. When a flagged value divided by 10, 100 or 1,000 lands inside the limits, DataFlow proposes removing the extra zeros as corrections you approve, each logged and reversible; the raw data is never changed. It works with SurveyLoopr, SurveyCTO and ODK Central directly, and with exports from KoboToolbox and other tools.

Questions