SurveyLoopr data quality: checking each submission as it syncs

SurveyLoopr is where a survey gets built and collected: the form, the app on enumerators' phones and the server they send to. Its quality tools work before and during the interview, and on one submission at a time once it arrives. Checks that need the whole dataset, such as one household interviewed twice or an enumerator whose interviews all take nine minutes, come after. This guide covers both, with SurveyLoopr staying your collection platform.

What SurveyLoopr gives you

SurveyLoopr is a set of research tools from ARCED International, the company that also makes DataFlow. Four of them matter for data quality.

Before fieldwork: LooprAI Workspace

LooprAI builds an editable XLSForm from a plain-English description or an existing questionnaire in PDF, Word or Excel, with required questions, skip logic and constraints described in words. It checks the workbook against ODK's rules, suggests repairs, shows a preview, and publishes straight to a SurveyLoopr-hosted server or downloads as a standard XLSX file. SurveyLoopr is clear about the limits: validation catches syntax and reference errors and leaves method and wording to you, and logic written by the AI needs a person to review it.

In the field: SurveyLoopr Collect

SurveyLoopr Collect is a free Android app built on ODK Collect (it isn't made or endorsed by the ODK project) that works with any ODK Central server. It collects offline, flags out-of-range values and missing required answers during the interview with a plain message, and supports encrypted forms. Enumerators set it up by scanning a QR code from the server's Field team tab.

Two parts of the form matter later: the start and end metadata, which record by the device clock when the form was opened and finished, and ODK's audit log, which records how long each question took and how the enumerator moved through the form.

On the server: managed ODK Central

SurveyLoopr deploys a dedicated ODK Central server for your team, handles the setup and TLS, lets you choose when to update, and can pause the server between rounds for up to three months. Because it is ODK Central, Central's own tools work as usual: the submissions table and its filters, the review states (Received, Has issues, Approved, Rejected, Edited), browser edits with a version history, exports and OData. The ODK fieldwork monitoring guide covers them.

DataSnap

DataSnap reports on the audit log, from an uploaded audit-log CSV or a running SurveyLoopr-hosted server. It shows where interviews cluster when there is GPS, timing and activity patterns, question friction, and work patterns such as average and median length, flagged interviews and surveys per day. SurveyLoopr is careful about what it claims: flags are prompts for a person to review, the report doesn't decide that anyone is fraudulent, and it doesn't rank enumerators or break performance down by enumerator. The platform monitoring comparison sets it beside the other platforms' tools.

What's left once the data arrives

Each tool above looks at one form, one submission or one audit log at a time. These need all the submissions together:

• Duplicates. Every submission gets its own instance ID, so only your household or respondent ID shows a household interviewed twice.
• Interview length against the rest. Twelve minutes is fine for a short form and impossible for a long one, so the threshold should come from your own median.
• Rules thought of mid-survey. A rule you add in week two (owns land, but zero acres) has to run over everything already on the server.
• A view by enumerator. Flag rates, interview length and answer patterns per person, compared with colleagues in the same area, with signs of made-up interviews such as near-copies and answers that barely vary.
• Back-check comparison. Matching back-checks to the original interviews and counting differences by enumerator and question.
• One record of corrections. Central keeps the history of edits made on the server; corrections made after export need a record that rebuilds the clean data from the raw data.

The checks to run on every sync

Run them over every submission received so far each time new ones arrive: a duplicate or an outlier often shows up only against what came before. The high-frequency checks guide covers the whole set.

Interview length on an ODK form

ODK forms have no ready-made duration field, so length is end minus start, which includes time the form sat saved and depends on the device clock. Flag interviews far below the median; the audit log, which DataSnap reads, shows where the time went.

Read every check by enumerator

One flag is a correction; the same flag on a third of one person's interviews is a training problem. Count flags per enumerator per 100 interviews and compare each person with others in the same area. Add an enumerator question to the form, since the submitter is the app user that sent the form. Use review states for the follow-up: Has issues with a comment, then Approved once explained or Rejected if dropped. The enumerator monitoring guide goes further.

Cleaning SurveyLoopr data

Pick one place for corrections. Edits on the server keep a version history in Central; cleaning after export keeps the download untouched, with each correction in code or a log row (submission ID, question, old and new value, who and why). Doing both leaves corrections that no single record explains. Either way, rebuild the clean dataset from the raw data in one run, and take personal data out before sharing; see de-identifying survey data and the Field Data Quality Standard.

Running the checks with DataFlow

DataFlow is a desktop app for Windows, macOS and Linux that runs this routine on every sync. SurveyLoopr stays the place the data is collected; DataFlow adds the checks on your own computer, where the data is kept encrypted.

Connecting

On DataFlow's Import page, choose SurveyLoopr, then Continue with SurveyLoopr. It opens SurveyLoopr in your browser, where you sign in as usual, with Google or your password; DataFlow never sees the password and keeps only the sign-in tokens, in your computer's keychain. Pick the server, project and form from the servers your account owns or that are shared with it, then Connect & download once the first interviews are in. A paused server can be picked once you resume it in SurveyLoopr.

The data comes through SurveyLoopr's read-only API, which uses the ODK login saved for the server in SurveyLoopr, so no ODK password is typed into DataFlow. You can also use the server's own ODK Central login instead: the password is used once to get a session token, and only the token is kept.

What each sync brings

• The form. If it is on the server as an XLSForm, DataFlow downloads it and sets up checks from its questions, choices and constraints. If not, add the XLSForm on the Import page; DataFlow can also suggest checks from the data alone.
• New and changed submissions, read through Central's OData feed, edits and review changes included. An edit comes in as the new version of that interview, and corrections made in DataFlow stay on top of it.
• Groups and rosters. Answers keep their question names; repeat groups arrive as numbered columns (age_1, age_2) that checks can read row by row.
• Submission details: the submitter, the review state and the form version.

Syncs can run on a timetable, such as every two hours from 7am to 10pm, and each one runs the checks over all the data. Three things to know: Rejected submissions are downloaded and checked like the rest, so set aside in DataFlow the interviews you've dropped (setting aside is logged and can be undone). Photos, audio and other attachments, the audit log among them, aren't downloaded, so the audit log stays DataSnap's to read. And a paused server syncs again once you resume it.

The checks it sets up

From the form and the data, without anyone writing a check, DataFlow sets up duplicate IDs; interview length, flagging interviews under 40% of the median or over three times it; ranges from your constraints, on roster rows too; outliers beyond three interquartile ranges; blank required questions; interviews where over 20% of answers are don't-know or refusal codes; “other” answers to review; GPS points that are missing, inaccurate, far from every other interview or reached at an impossible speed; and interviews started at night, overlapping the same enumerator's previous one, or done on an old form version. A suggested check that would flag more than 5% of interviews starts switched off, unless most of its flags come from one enumerator. Name the enumerator variable and DataFlow also looks for six signs of made-up interviews, comparing each enumerator with colleagues in the same area.

Back-checks and downloads

Add the back-check form as a second survey in the study, connect it the same way and link it to the main survey by ID. DataFlow reports differences by enumerator, back-checker and question; the back-checks guide has the details. Each correction is logged with who, when and why, and can be undone; the raw data from SurveyLoopr is never changed. Downloads come as Stata, R, Excel or CSV, with a codebook, the correction log and a Stata do-file that rebuilds the clean data from the raw data, de-identified unless someone unlocks personal data, which is logged.

To see it before connecting a server, the sample study on the download page works without an account.

Step by step

1. Connect SurveyLoopr to DataFlow. On the Import page, choose SurveyLoopr, click Continue with SurveyLoopr and sign in in your browser. Pick the server, project and form, then Connect & download once the first interviews are in.
2. Review the suggested checks. Look through the checks DataFlow set up from the XLSForm and the data, turn on any that started off if they fit, add rules you think of later, and name the enumerator and area variables.
3. Work through the flags every day. Set a sync timetable. Each day, read the flags by enumerator, send supervisors their lists, and record each answer in DataFlow as a correction or a confirmed value.
4. Compare back-checks each week. Add the back-check form as a second survey, link it to the main survey by ID, and read the differences by enumerator, back-checker and question.
5. Download the clean data. Download Stata, R, Excel or CSV files with the codebook, the correction log and the do-file that rebuilds the clean data from the raw data.

Questions