How to monitor a survey firm's data while it's in the field

A survey firm may well run quality checks of its own. You still carry the risk: if interviews are rushed, copied or never done, it is your study that ends up with the bad data, and by the time the dataset arrives the enumerators have left the area. Monitoring the firm means seeing the data as it comes in and raising problems while a revisit still costs little. The World Bank's DIME Wiki gives the same advice: research teams should run their own high-frequency checks even when the firm checks internally.

This guide is for research teams, funders and NGOs who have hired a firm or are about to. It follows the order of the work: access and rules before fieldwork, checks every day during it, and the right files at the end.

Before fieldwork: access and rules

Get direct read access to the data

Whatever platform the firm uses (SurveyLoopr, SurveyCTO, ODK Central or KoboToolbox), ask for your own account that can view and download submissions. If the platform's roles allow it, ask that the account can't edit submissions or the form. Put the access in the contract, and test it on the pilot data so it works on the first day of fieldwork.

If the firm can't give you an account, agree a daily export instead: every submission so far, as CSV or Excel, at a fixed time each day, with all the metadata columns (start and end times, form version, enumerator ID, device ID and GPS). Weekly exports leave a week of interviews unchecked. If the form is encrypted, you also need the key, or the firm must send decrypted exports.

Collect what the checks need

Before the first interview, get the final XLSForm, the sample list with IDs, the roster of enumerators and supervisors with their IDs, and the field plan showing which area each team covers on which day. Each one makes a check possible. The form gives you ranges and logic, the sample list lets you check completeness, the roster lets you read results per person, and the field plan tells you where interviews should be.

Agree the checks and thresholds in the contract

Write down which checks run, with what limits, before fieldwork starts: interview length, duplicates, ranges, outliers, the share of “don't know” answers, GPS and back-check agreement. The Field Data Quality Standard is a numbered list with default thresholds you can adopt or change. Then agree how flags reach the firm, how fast they must be answered and what counts as an answer. Without that, every flag turns into a negotiation. The data quality clauses for a terms of reference have wording you can paste in.

During fieldwork: check every day

Run your own checks on each day's data

Run the agreed checks every day on everything that arrived since the last run. Once they are coded, most take minutes:

• duplicate IDs, and the same phone number under two IDs;
• interviews much shorter or longer than the median;
• numbers outside the allowed range, and outliers within each village or team;
• answers that contradict each other;
• required questions left blank, and interviews with many “don't know” answers;
• GPS points that are missing, inaccurate, outside the named village or bunched in one spot;
• progress against the field plan, by area and by team.

The check pages explain each one, including how to set outlier limits.

Read the results by enumerator

A list of 300 flags tells you little. The same flags counted for each enumerator, as a rate per 100 interviews, show where the problem is. Three patterns are worth telling apart:

• One check fires for everyone. The threshold is wrong for this survey, or the form has a problem. Fix the check or the form, and record the change.
• One enumerator stands out on several checks. That person needs a supervisor visit, back-checks on their recent interviews and possibly retraining.
• One area stands out. Look at logistics, translation and the local supervisor before blaming individuals.

Compare each enumerator with the team on things no single flag catches, too: interviews per day, average length, the share of “don't know” answers, refusals, and how often their interviews answer “no” to the questions that open a section. Interviews that look too tidy can be a sign they were made up.

Send flags in one list, with a deadline

Send the firm one list a day instead of a stream of messages. For each flag, give the interview ID, the enumerator, the question, the value and what you need: confirm the answer is right, correct it with a reason, or revisit or call the respondent. Track each flag as open, explained, corrected or revisited. A flag closed with no explanation and no changed value is still open.

Ask for back-check results every week

A back-check is a short second interview with a randomly chosen respondent, done by a separate team, to confirm the interview happened and the answers hold. The DIME Wiki suggests back-checking 10 to 20% of interviews, covering every enumerator, with a fifth of the back-checks done in the first two weeks. Ask for the results weekly, as the share of answers that differ for each enumerator, and ask to choose part of the sample yourself, including interviews your checks flagged. The back-checks guide covers the sample and the question types.

When flags pile up

Flags pile up when the checks are too sensitive or when the data really is getting worse. Find out which before you escalate.

1. Sort the open flags by check. If one check produces most of them across all enumerators, review its threshold with the firm and change it in writing.
2. Sort the rest by enumerator, and ask the firm to deal with the top few first: a supervisor goes with them, their recent interviews are back-checked, and, if the contract allows, they pause new interviews until the results are in.
3. Hold a short daily call with the firm's field manager until the backlog falls, and agree a date by which the open flags will be answered.
4. If interviews fail back-checks, ask for them to be redone and left out of the target, as the contract says.
5. Keep a record of the flags, the answers, the dates and what was agreed. In a contract dispute, that record is your evidence.

Red flags in a firm's behaviour

Some problems show in how a firm works before they show in the data:

• Read access keeps being delayed, or data arrives in large batches at the end of the week.
• Corrections are made on the server or in the raw data, with no log.
• Flags come back “resolved” with no explanation and no changed value, or with the same explanation pasted under each one.
• Back-checks are done by the supervisors of the enumerators being checked, or the results are never shared.
• Several people share one enumerator ID, or IDs change during fieldwork without notice.
• A whole team stops recording GPS.
• Enumerators are replaced without telling you, or new ones start without training.
• The firm will share the clean dataset but not the code that produced it.

None of these proves the data is bad. Each is a reason to ask the firm a direct question and to look harder at that part of the data.

At delivery: what to ask for

Ask for the raw data exactly as it came from the server, the clean data, the code that turns one into the other (a Stata do-file or an R script), the log of every correction with its reason, the back-check data and the final field report. Then run the code on the raw data yourself and check that you get the same clean data. If you don't, something was changed by hand.

Doing this with DataFlow

DataFlow is a desktop app built for this. Connect it to the firm's SurveyLoopr, SurveyCTO or ODK Central server with the account the firm gives you, or import their CSV and Excel exports, KoboToolbox included. It reads the form, sets up the checks, runs them on every sync and shows the results by enumerator. It only reads from the server, and the data stays on your computer, encrypted. If the firm uses DataFlow too, ask for a weekly Field Report: one file you open in a browser. See DataFlow for research teams and clients.

Step by step

1. Get read access to the data. Ask for your own account on the firm's survey server that can view and download submissions, or a daily export of every submission with its metadata. Test it on the pilot data.
2. Collect the form, sample list and roster. Get the final XLSForm, the sample list with IDs, the enumerator and supervisor roster with IDs, and the field plan before the first interview.
3. Agree checks and thresholds in the contract. Write down which checks run, their limits, how flags reach the firm, the deadline for answering them and what counts as an answer.
4. Run the checks every day. Run duplicates, interview length, ranges, outliers, logic, missing answers, don't-know shares, GPS and progress on each day's new interviews.
5. Read the results by enumerator. Count flags per 100 interviews for each enumerator and compare them with the team, to tell a bad threshold from a problem with one person or one area.
6. Send one list of flags a day. Give the interview, enumerator, question, value and the action needed for each flag, and track it until it is explained, corrected or revisited.
7. Review back-check results every week. Ask for the share of back-check answers that differ for each enumerator, done by a team independent of the enumerators' supervisors, and choose part of the sample yourself.
8. Ask for raw data, cleaning code and the correction log at delivery. Run the cleaning code on the raw data yourself and check that it produces the clean data the firm delivered.

Questions