Data cleaning software for field surveys

Search for data cleaning software and most of what you find is built for customer databases and spreadsheets: matching company names, fixing address formats, merging records from two systems. Field survey data goes wrong in other ways. An enumerator rushes interviews, or a harvest goes in as kilograms instead of tonnes, and by the time anyone looks, the team has left the village. Cleaning survey data is mostly about catching those problems early and keeping a record of every fix.

What survey data cleaning software has to do

Run while the team is still in the field

A problem found on the day it happens can be fixed with a phone call or a revisit. The same problem found after fieldwork ends can only be dropped or guessed at. Survey cleaning software has to pull new interviews as they arrive and check them every day, not once at the end.

Tie every problem to an interview and an enumerator

“Twelve outliers in income” isn't actionable. “Seven of them from one enumerator, all this week” is. Every flag needs the interview it came from and the person who did it, and the checks need to be read per enumerator, compared with the others working in the same area.

Never overwrite the raw data

The data as it arrived from the field is the evidence. Corrections should sit on top of it, so any one of them can be undone and anyone can see what the enumerator first recorded.

Log every correction

Who changed which value, from what to what, when and why. Without that, a reviewer, a client or a journal has no way to tell a correction from a convenient change.

Produce cleaning code someone else can run

The clean dataset should be rebuilt from the raw data by code that someone else can run and get the same result. For many research teams that code is a Stata do-file.

Protect personal data

Survey data holds names, phone numbers, GPS points and comments that identify people. The software should find them, keep them apart from the data everyone works on, and leave them out of anything shared unless someone deliberately unlocks them.

Know the form

Surveys come with a questionnaire, usually an XLSForm, that already says which answers are allowed, which codes mean “don't know”, and which questions repeat for each household member. Good survey software reads it, so the checks start from the form instead of a blank page. It should also handle back-checks, where a second team revisits some households to confirm the answers.

Why generic data cleaning tools and data quality software fall short

OpenRefine is a free, open-source tool for messy data. It groups inconsistent spellings of the same village, splits and reformats columns, and keeps a history of operations you can replay on another file. It works on a file you load, though, not on interviews arriving each day, and it has no idea what an enumerator, a form or a back-check is.

Excel is where most teams start. Filtering and sorting a few hundred interviews is quick. But an edit overwrites the cell, nothing records who changed it or why, and checks done by hand are done slightly differently each day. Linking a back-check to the main survey means lookups that break easily.

Enterprise data quality tools and data validation software are built for business databases and data pipelines: profiling tables, enforcing validation rules, matching records across systems. They don't read an XLSForm or compare an enumerator with their team, and they expect an IT team to set them up and run them.

The approaches compared

Each of these works for someone. They differ in how much of the list above they cover during fieldwork, and how much staff time that takes.

If your team already runs ipacheck well and has someone to maintain it, you may not need anything else. The guides for SurveyCTO, ODK Central and KoboToolbox describe the routine whichever tool runs it.

What DataFlow does

DataFlow is data cleaning software for field surveys. It runs on Windows, macOS and Linux, and it works alongside your collection platform rather than replacing it.

Connects to the platform you already use

SurveyLoopr, SurveyCTO (encrypted forms too) and ODK Central connect directly; CSV and Excel exports from KoboToolbox or any other tool can be imported. DataFlow reads from your server and never writes to it, and new interviews can arrive on a timetable you set. See integrations.

Checks set up from your form

When you add the XLSForm and the data, DataFlow suggests and adds the checks itself, and runs them on every sync:

• Duplicate IDs
• Interviews too short or too long
• Outliers by interquartile range or standard deviations, within villages or teams if you choose
• Ranges, including the form's own constraints
• Missing answers, and interviews with too many don't-know or refusal answers
• “Other, specify” answers that fit an existing option
• Your own rules in XLSForm syntax, and impossible combinations picked from a cross-tab
• GPS: missing or inaccurate points, interviews far from their village, and journeys between interviews no one could make

Checks that would flag more than 5% of interviews arrive switched off, so a noisy check doesn't bury the real problems. Each flag says which rule fired, with what threshold. The high-frequency checks guide explains each check.

Enumerators and back-checks

Six statistical signs of made-up interviews per enumerator, such as near-copies of other interviews and answers that barely vary, each compared with the others working in the same area. Back-checks are linked to the main survey and compared question by question, the way ipabcstats does, with differences counted per enumerator, per back-checker and per question.

Corrections on top of the raw data

The raw data is never modified. The clean data is rebuilt from it plus the corrections you approve, so each correction can be undone. Every change records who, when and why in an activity log where each entry is chained to the one before, so editing history breaks the chain. Every download includes a Stata do-file that replays each correction from the raw data, and there are R, Excel and CSV downloads too. DataFlow grew out of arceddataflow, ARCED Foundation's Stata workflow, and its duplicate and outlier checks are tested to flag exactly what arceddataflow flags.

Personal data and privacy

Survey data stays on your computer, encrypted. DataFlow finds personal data in the form and in free-text answers, and moves it to a separate encrypted store; downloads leave it out unless someone unlocks it, and that is logged. A local AI explains flags and prepares fixes for you to approve. It runs on the computer, offline once its model is downloaded, and never makes up numbers: DataFlow counts, the AI explains.

A report the client can open

A Field Report is one HTML file with progress, every check and how each problem was handled. It opens in any browser without a login, and you choose what it holds before you share it.

Pricing

Monthly is $19 per seat a month. Team is $99 a year with five seats. Every feature is in every plan, and the sample study, 600 interviews with real-looking problems, is free without an account. See pricing and the full list of features.

When something else fits better

• Your data isn't from surveys or forms: a customer database or sales records. A generic data quality tool suits that better.
• You are cleaning one finished dataset once, with no fieldwork to monitor. OpenRefine or a script may be all you need.
• You collect on KoboToolbox and need a direct server connection. DataFlow imports Kobo exports today, but doesn't connect to Kobo directly.

Questions