Interview duration checks: interviews that are too short or too long

Why interview length matters

Length is one of the cheapest checks to run and one of the most telling. A questionnaire that takes most of the team three quarters of an hour can't be done properly in a quarter of that. Very short interviews are the speeders: enumerators who read questions in a hurry, skip the probing, take the first answer, or fill in whole sections without asking. Some short interviews didn't happen at all.

Very long interviews are usually innocent, but still worth a look. The respondent went to the fields and came back, the enumerator saved the form and finished it the next day, or forgot to close it. Their timing can't be trusted, and occasionally the rest of the form was filled in from memory, away from the respondent. The World Bank's DIME Wiki lists average interview duration among the enumerator checks in its guide to high-frequency checks.

Where the time comes from

Most platforms give you more than one measure of time, and they don't agree:

SurveyCTO's help article on the difference between duration() and end-start explains the gap: new SurveyCTO forms include the start and end fields and a calculate field with duration(), and the duration should never be more than end minus start. ODK's question types page defines end as the last time the form was saved, and the form audit log page covers the per-question log. In KoboToolbox, editing a submission in the web form updates its end as well.

For the main check, use active time where the platform records it, and end minus start otherwise. Keep both when you have them: a big gap between the two means the form was saved and reopened, which is worth knowing on its own. Per-question timing is sharper still. A long module answered in less time than it takes to read its questions aloud is a stronger sign than a short total, and it shows where the time went missing.

Limits from the median, not a guess

Don't take the limits from the pilot or the budget. Pilot interviews are done by the most experienced enumerators on a form nobody knows yet, and the planned length is a planning number. After the first days of real fieldwork, take the median length of completed interviews and set both limits relative to it: a lower limit at a fraction of the median, and an upper limit at a multiple. Use the median rather than the mean, because a few forms left open overnight pull the mean up a long way and leave the median where it was. Then look at how many interviews each limit flags, and adjust until supervisors can follow up the whole list each day.

Compare like with like. A household of eight takes longer than a single person on a roster, and a farmer gets the agriculture module that others skip. When the form branches a lot, compute the median within groups (by household size, by whether a long module was asked, by interview language), or time the modules separately. Recompute the median as fieldwork goes on, since interviews tend to get quicker once enumerators know the form.

* Minutes from SurveyCTO's duration (seconds), consented interviews only
generate minutes = duration / 60
summarize minutes if consent == 1, detail
local med = r(p50)

generate too_short = minutes < max(round(0.4 * `med'), 5) if consent == 1 & !missing(minutes)
generate too_long = minutes > 3 * `med' if consent == 1 & !missing(minutes)

* Each enumerator's median and short interviews, shortest first
preserve
keep if consent == 1 & !missing(minutes)
collapse (median) med_minutes=minutes (count) interviews=minutes (sum) short=too_short, by(enum_id)
generate vs_team = med_minutes / `med'
sort med_minutes
list, noobs
restore

The 40 percent, 5 minutes and three times in this code are the limits DataFlow starts from (described below); treat them as a starting point, not a standard. If you only have start and end times, convert each into a double with clock() and divide the difference by 60,000: Stata datetimes count milliseconds.

Reading it by enumerator

One short interview can be a small household or a respondent who answers quickly. A pattern belongs to a person. For each enumerator, compare their median length with the team's and count their interviews under the lower limit. An enumerator whose median is well below colleagues in the same area is either very efficient or cutting corners, and a few back-checks tell you which. Other timing signs are worth a look too: one enumerator's interviews that overlap in time, interviews started minutes after the last one ended in another village, and interviews at hours when nobody is usually at home. Read length alongside the other measures in enumerator monitoring: short interviews with few “don't know” answers and many “no” answers to the questions that open a section are a stronger pattern than any one of them.

Consent, refusals and partial interviews

Refusals, households not found and respondents who stop halfway are short by design. Leave them out of the median and the check, using the consent question or the interview's outcome code; otherwise they pull the median down and fill the list with interviews that are fine. Check them separately: a high share of refusals or “not at home” for one enumerator can itself mean visits that never happened. Partial interviews need their own rule, such as comparing their length with the part of the form they reached, or a review by hand.

What to do with a flagged interview

Open the interview before asking anyone about it. Blank sections, “no” to every filter question and a roster of one tell you more than the minutes. For a long interview, ask the enumerator whether it was paused or the form left open; usually that settles it. For a short one with no explanation, call the respondent, listen to the audio audit if you collect one, or add the household to the back-check sample. If the interview didn't take place, set it aside with a reason rather than deleting it, redo it where possible, and look at the rest of that enumerator's work. If it was real, confirm it and record that, so it doesn't come back in every daily report.

How DataFlow checks interview duration

DataFlow reads each interview's length from a duration column in seconds (SurveyCTO's field) when the data has one, and otherwise from the form's start and end fields (or SurveyCTO's starttime and endtime), in ISO 8601 or SurveyCTO's own date format. It doesn't read text audits or ODK's audit log.

On import it adds a duration check by itself. With data already there, it takes the median length of the interviews so far and flags interviews under 40% of the median, rounded to whole minutes and never less than 5 minutes, or over 3 times the median, also rounded. Added before any data arrives, the check flags interviews under 10 minutes, with no upper limit. When the form has a consent question coded 1 for yes, the check only covers interviews where the respondent consented. If it would flag more than 5% of interviews, it starts switched off, unless most of the flags come from one enumerator. You can change both limits and the condition.

Each flag shows the interview's length and the median, and suggests calling the respondent or reviewing the audio for a short interview, or asking whether a long one was paused or left open. When one enumerator has at least four of the short interviews and at least half of them, the Today list groups them into one card, with that enumerator's median against the team's and a note to copy for their supervisor naming up to five respondents to call back. The Enumerators page shows each enumerator's median length and highlights any under 60% of the team's. DataFlow works alongside the collection platform: it connects to SurveyLoopr, SurveyCTO and ODK Central, and reads exports from KoboToolbox and other tools.

Step by step

1. Record the time. Include start and end in the form, and keep SurveyCTO's duration field if you use SurveyCTO. Add text audits (SurveyCTO) or the audit log (ODK, KoboCollect) if you want time per question.
2. Wait for real interviews. Collect the first days of fieldwork before setting limits, rather than taking them from the pilot or the planned length.
3. Set the limits from the median. Take the median length of completed, consented interviews and flag those under a fraction of it or over a multiple of it. Adjust until the daily list is one supervisors can follow up.
4. Leave out refusals and partial interviews. Use the consent question or outcome code to exclude interviews that end early by design, and check those separately.
5. Read it by enumerator. Compare each enumerator's median and share of short interviews with colleagues in similar areas, alongside their other measures.
6. Follow up each flag. Open the interview, ask the enumerator, call back or back-check short ones, set aside interviews that didn't happen with a reason, and confirm the real ones.

Questions