SurveyCTO data quality monitoring: the checks to run on every sync

SurveyCTO has more data quality tools than most teams use. They work on the form and on one field at a time, though, so the checks that depend on your survey's own logic are still yours to run, such as one household interviewed twice, or an enumerator whose answers vary too little. Run them on each new batch while the team is still in the field, when a problem can still be fixed with a phone call.

What SurveyCTO gives you

In the form

• Constraints and relevance. A constraint rejects an answer that breaks a rule, such as an age of 250; relevance skips questions that don't apply. Both act at the moment of entry, the cheapest place to stop an error.
• Soft constraints. A warning or a yes/no confirmation when an answer is unlikely but possible, so the enumerator checks it with the respondent instead of being blocked.
• Speed limits. A minimum_seconds value on a field counts, lists or audits answers given faster than that, and can block them altogether.
• The duration field. New SurveyCTO forms include a hidden calculate field, duration, that records the seconds spent filling or editing the form.
• Text audits. A CSV attached to each submission with the time spent on each question, or a full log of events. You can collect it for a share of submissions only.
• Audio audits. Random recordings of parts of the interview, made by the mobile app, for a supervisor to listen to later.
• Sensor meta-data. An experimental, Android-only feature that records light, sound, movement and likely conversation during the interview, as summary values or a per-second stream.

On the server

• Data Explorer. A monitoring workbook built from your form, with summaries of each field and of relationships between fields. You can click through to individual submissions, save the workbook, and leave test submissions out of the summaries.
• Automated quality checks. Set up in the Monitor tab. There are nine kinds: a value too low, too high or an outlier (by interquartile range); an answer too frequent or too rare; a mean too low or too high; and two comparisons between groups such as enumerators, one for means (an ANOVA test) and one for distributions (a chi-squared test). They run every night, or when you click Run now, and produce a CSV of warnings that SurveyCTO can email to you.
• Review and correction workflow. Holds all, some or none of the incoming submissions until someone approves or rejects them. Reviewers can correct values, add comments and rate each submission as good, okay, poor or fake. SurveyCTO keeps a correction log with the old value, new value, user, time and comment for each change. Submissions can be held because they failed a quality check or were drawn at random.

SurveyCTO's guide to collecting high-quality data covers each of these, and its page on automated quality checks lists the settings.

What SurveyCTO leaves to you

The automated checks look at one field at a time, against thresholds you set. That covers a lot. Several checks field teams depend on need more than one field, more than one survey, or a decision about the whole interview:

• Duplicate IDs. SurveyCTO's KEY is unique to each submission, so a household interviewed twice has two keys. The check has to look at your own household or respondent ID.
• Logic across questions, after the fact. A constraint only stops what someone wrote a constraint for, and only on new entries. A rule you think of in week two (owns land, but zero acres) has to be run over everything that has already arrived.
• Repeat groups and encrypted fields. Automated checks can't be set on fields inside a repeat group, so household rosters need a summary field or an outside check. On encrypted forms, the checks and the Data Explorer only see fields marked publishable.
• Back-check comparison. SurveyCTO can run the back-check survey. Comparing its answers with the original interviews, question by question and per enumerator, happens outside.
• Signs of made-up interviews. A group comparison shows that one enumerator's mean is different. Interviews that are near-copies of each other, answers that barely vary, or no “don't know” in thirty interviews need checks that look at whole interviews.
• A cleaning record that rebuilds the data. Corrections made in SurveyCTO stay in its log. Corrections made after export live in whatever script or spreadsheet someone used. You need one record that turns the raw export into the clean dataset.

Most teams fill this gap with Stata, often Innovations for Poverty Action's ipacheck, with R scripts, or with a spreadsheet someone updates each morning. The checks themselves are the same whichever tool runs them.

The checks to run on every sync

Run these over all submissions each time new ones arrive, not only over the new batch: a duplicate or an outlier often only shows up against what came before. The high-frequency checks guide explains each one in more depth.

Use the duration field, and know what it measures

duration counts the time the form was open on the device. Subtracting the start time from the end time counts the clock time in between, including any time the form sat saved and closed, so it is usually longer, and can even be negative if a device clock was changed. For a too-short check, use duration; use the start and end times to see when interviews happened.

Set the threshold from your own data rather than a guess. Once a few dozen interviews are in, look at the median length and flag interviews far below it (under 40% of the median, for example). A short interview isn't proof of anything, but several from one enumerator on one day is worth a phone call. Text audits then show where the time went: a 40-minute module answered in four minutes stands out.

Read every check by enumerator

A flag on one interview is a correction. The same flag on a third of one enumerator's interviews is a training or supervision problem. For each check, count flags per enumerator per 100 interviews, and compare each person with the others working in the same area, since a remote village can make honest interviews look unusual. SurveyCTO's group comparisons do this for single fields; near-copies, skipped sections and round numbers need the whole interview.

Send flags back the same day

A flag is only useful while the team can still go back. Send each supervisor a short list with the interview's KEY, the question, the recorded value and what to check, and record what they answer. If you use the review workflow, the comments on each submission are a natural place for that answer.

Decide where corrections live

Decide this before fieldwork starts. If supervisors correct in SurveyCTO's review workflow, the exported data already includes the corrections, and SurveyCTO's correction log file has the history. If you clean after export, never edit the exported file: keep it as the raw data, and write each correction as a line of code or a row in a log with the KEY, the variable, the old and new values, who decided and why. Mixing the two leaves a dataset that no single record explains.

Whichever you choose, the final cleaning code should rebuild the clean dataset from the raw data in one run. That is what a journal or a client will ask for. The Field Data Quality Standard lists what a complete record holds.

Running these checks with DataFlow

DataFlow is a desktop app that runs this routine on every sync. It connects to your SurveyCTO server with the server name (the part before .surveycto.com), the form ID and a SurveyCTO user's username and password. That user has to sign in with a password, not with Google, Microsoft or a second step, and have a role that can download data; if SurveyCTO refuses, DataFlow says why and drafts a note for your admin. For an encrypted form, add its private key. The key and password are kept in your computer's keychain, and the key is sent only to your SurveyCTO server, to decrypt the download.

Each sync asks SurveyCTO's API for the submissions completed since the last one, with an hour of overlap so nothing slips between syncs. A submission that arrives twice replaces the earlier copy instead of being added again. DataFlow reads duration for interview length and, once data arrives, sets the thresholds from the median: under 40% of it or over three times it, both of which you can change. Add your XLSForm and it also sets up range checks from your constraints. DataFlow only reads from SurveyCTO and never changes anything on the server.

One thing to know if you use SurveyCTO's review workflow: SurveyCTO releases held submissions once they are approved, and its API returns approved submissions unless asked for others. DataFlow asks for the default, so on a form that holds every submission for review, interviews reach DataFlow after approval. To check them before review, hold only some submissions. The integrations page lists every connection.

Step by step

1. Build quality into the form. Add constraints, soft constraints for unlikely answers and an enumerator ID question, keep the duration field, and turn on text audits for at least a share of submissions.
2. Set up SurveyCTO's automated quality checks. In the Monitor tab, add value and outlier checks for the main numbers and group comparisons by enumerator, and have the nightly report emailed to the field manager.
3. Decide where corrections live. Either SurveyCTO's review workflow or your cleaning code, not both. Write the choice into the data management plan.
4. Pull new submissions every day. Export or use the API after each day's uploads, and keep each download unchanged as the raw data.
5. Run the checks over all submissions. Duplicate IDs, interview length, outliers, ranges, missing answers, don't-know share, "other, specify", logic rules and GPS, over everything received so far.
6. Read the flags by enumerator. Count flags per enumerator per 100 interviews and compare each person with others in the same area.
7. Send flags to the field team the same day. For each flag give the KEY, the question, the recorded value and what to check, and record the answer.
8. Log every correction and compare back-checks. Record who changed what and why, rebuild the clean data from the raw export, and compare back-check answers with the originals each week.

Questions