How to assess survey data quality: a framework

Most writing on data quality assessment is about company databases: customer records, product tables, pipelines. A survey is different in two ways. The data arrives in batches over a few weeks while a team is still in the field, so the assessment can change what happens next. And the truth sits with respondents you can go back to, so some errors can be checked against reality, not only against other records.

This page is the procedure. The concepts it measures (uniqueness, completeness, validity, consistency, accuracy, timeliness and integrity) are explained in data quality dimensions for survey data. The written plan that sets the thresholds is the data quality assurance plan template, and the checks with their default thresholds are listed in the Field Data Quality Standard. Here we put them in order.

The framework at a glance

The same dimensions run through all three stages. What changes is the unit you look at: the form before fieldwork, the batch and the enumerator during it, the whole dataset at the end.

Before fieldwork: assess the form and the plan

The cheapest errors to fix are the ones the form never lets in. Read the form against the dimensions, one at a time. For uniqueness: is the respondent ID picked from the sample list or typed twice? For validity: does every number with a hard limit have a constraint, and do possible but unlikely answers ask for confirmation? For completeness: are the questions every respondent must answer marked required? For timeliness and integrity: does the form record start and end times, the form version, the enumerator and, where ethics approval allows, a GPS point? How to write those constraints in XLSForm is covered well in SurveyLoopr's XLSForm tips.

Then choose the measures and thresholds, as rates per 100 interviews, and agree them with the client in the QA plan; the dimensions guide suggests measures for each. Run every check on the pilot data, because the pilot tests the checks as much as the form. A check that flags most pilot interviews usually has the wrong threshold or a mistake in its rule; a check that flags nothing at all may be looking at the wrong question. Fix both before the main survey starts, and record any threshold you change and why.

During fieldwork: assess every batch

The routine is set out in the dimensions guide: run the high-frequency checks daily on the whole dataset, close each flag with a reason, keep the raw data as it arrived. For the assessment, what matters is how you read the results. Read them three ways:

• By check: which rules flag the most, per 100 interviews. A rule that suddenly flags many interviews may point to a form change, not to the team.
• By enumerator, within an area: compare enumerators working in the same place, since answers really do differ between areas. One enumerator who stands apart on several measures at once, such as short interviews, few “don't know” answers and GPS points bunched together, is the first to back-check.
• By day: interviews completed against the target, and the open flags. A growing pile of open flags means the checks are running but nobody is acting on them.

Add how long flags stay open to the weekly summary by dimension. A team that closes flags within a day or two is assessing; a team whose flags wait a week is only collecting them. The weekly summaries are the running assessment, and the final one builds on them.

After fieldwork: assess the delivered dataset

When the last interview syncs, run every check once more on the full dataset, then answer these in writing:

1. Do the counts reconcile? Completed interviews, refusals and households not found should add up to the sample, by area. Every ID in the data should be on the sample list, and appear once. See duplicates.
2. What is still open? List flags with no outcome, by check. An open flag at delivery can be acceptable; leaving it out of the report isn't.
3. What changed? How many corrections, of what kind, with what reasons. Changes made to many interviews at once deserve their own line.
4. How accurate is it? Back-check agreement on questions that shouldn't change, overall and per enumerator, and what was done where agreement was low.
5. What can't be fixed? Known limits the analyst must handle: a question asked differently after a form change, an area with low completion, a group of interviews dropped and why.
6. Can it be rebuilt? The raw data and the code or log that turns it into the clean data should go out together, so the analyst can repeat every step.

What the assessment report contains

The report is short. Its reader is the analyst or client who has to decide how far to trust the data. Keep a fixed order, so reports from different rounds or firms compare:

Avoid a single overall score. A dataset can be complete and on time and still hold a block of invented interviews, and averaging those into one number hides the one result that matters. Report each dimension against its own threshold.

Assessing a dataset you didn't collect

Clients and researchers often receive survey data from a firm and need to assess it themselves. Ask for the raw data as well as the clean data, the correction log, the back-check data and the agreed thresholds. Then re-run the checks you care about on the raw data, reconcile the counts against the sample, and compare your results with the firm's report. Take each difference to the firm as a question. All of this is easier when the assessment was agreed in the contract from the start; see data quality clauses for a survey's terms of reference and how to monitor a survey firm.

Tools for the assessment

Any tool that can run the checks daily and keep a log will do. Many teams write Stata do-files or R scripts, often starting from shared high-frequency check templates; others use Excel for small surveys. The platforms help at the door: SurveyLoopr, SurveyCTO, ODK Central and KoboToolbox all enforce form constraints, and some offer their own monitoring. What they don't do on their own is compare interviews with each other, enumerators with colleagues, or the data with a back-check.

DataFlow is one option for that layer. It connects to SurveyLoopr, SurveyCTO, ODK Central and KoboToolbox, or reads CSV and Excel exports, and checks interviews the moment they sync. On import it adds checks from the form and the data; one that would flag more than 5% of interviews starts switched off, unless at least half of its flags come from one enumerator, so a bad threshold doesn't bury the team. Every correction needs a reason and can be undone, and the raw data is never modified.

For the final assessment, its Field Report is a single HTML file at a public, management or team level. It shows interviews and flags by day, area and enumerator; how many flags were confirmed, corrected, reverted or left open; how long they took to handle; and, at the management and team levels, the correction reasons and changes made in bulk. It carries a hash of the latest entry in the hash-chained activity log and of its own contents, so a reader can tell whether the file was edited. The download gives the data as Stata, CSV or Excel with a codebook and a do-file that rebuilds it from the raw data, de-identified unless personal data was unlocked for that download. See data cleaning software for surveys for how it compares with the other options.

Step by step

1. Assess the form. Before training, read the form against each dimension: unique IDs, constraints on numbers, required questions, and the times, version, enumerator and GPS it records.
2. Agree measures and thresholds. Pick one or two measures per dimension as rates per 100 interviews, set the threshold that triggers action, and write them into the QA plan with the client.
3. Test the checks on the pilot. Run every check on the pilot data. Fix rules that flag most interviews or none, and record any threshold you change.
4. Assess every batch during fieldwork. Run the checks daily on the whole dataset, read results by check, by enumerator within an area and by day, and close every flag with an outcome and a reason.
5. Summarise weekly by dimension. Report completed interviews against target, open flags by check and enumerator, time to close flags, and back-check agreement.
6. Run the final assessment. Re-run every check on the full dataset, reconcile counts with the sample, list open flags and known limits, and deliver the report with the raw data, clean data and the code that links them.

Questions