Monitoring in SurveyLoopr, SurveyCTO, ODK Central and KoboToolbox, and what DataFlow adds

Every collection platform comes with monitoring tools, and they are worth using. Each is built around that platform's own data: one field, one submission or one audit log at a time. A daily quality routine needs a layer above that, the same on every platform, which compares interviews and enumerators and keeps the record of what was fixed. This page describes what each platform's own tools do, from its documentation, and what to add.

What each platform gives you

SurveyLoopr

SurveyLoopr deploys a dedicated ODK Central server for a team, so everything in the ODK Central section below applies to it. Its own monitoring tool is DataSnap, a report built from ODK's audit log (the audit.csv file), either uploaded or connected to a SurveyLoopr-hosted server. DataSnap shows where submissions cluster when there is usable GPS, unusual timing and activity, question friction (questions where the flow or the time taken suggests a form or training problem), and interview activity: totals, average and median length, flagged interviews, active days and surveys per day. A report can be shared by a link that works without a SurveyLoopr account. SurveyLoopr is careful about what the report claims: its flags are prompts for a person to review rather than verdicts, and DataSnap doesn't rank enumerators or break performance down by enumerator.

SurveyCTO

SurveyCTO's Monitor tab shows submissions over time for each form, and holds three tools for quality:

• Data Explorer. A monitoring workbook built from your form (SurveyCTO's guide). Each field gets a summary: bar or pie charts for choices, histograms with the mean and standard deviation for numbers, maps for GPS. Relationship summaries show two fields together, such as a scatterplot with its correlation. Click a bar or a point to open the submission behind it. A saved workbook refreshes with new submissions the next time you open it, and you can exclude submissions or values, or filter the whole workbook. Once loaded it works offline, apart from attachments, maps and saving. On an encrypted form you load either all the data, with the private key, or only the publishable fields.
• Automated quality checks. Nine kinds (SurveyCTO's list): a value too low, too high or an outlier by interquartile range; a value too frequent or too infrequent; a mean too low or too high; and two comparisons between sub-groups such as enumerators, of means (an ANOVA test) and of distributions (a chi-squared test). They run every night or when you click Run now, and produce a CSV report that can also be emailed. They can't be set on fields inside repeat groups, or on encrypted fields unless those are marked publishable.
• Review and correction workflow. New submissions can be reviewed and corrected before they are released. With it switched on, the Data Explorer lets reviewers comment on, correct, approve and reject submissions.

In the form itself, SurveyCTO offers text audits, audio audits and speed limits: the time spent on each question for a random share of interviews, recordings of parts of some interviews, and a count of answers given faster than a set number of seconds. The SurveyCTO data quality guide goes through each in more depth.

ODK Central

ODK's guide to managing submissions covers Central's tools:

• The submissions table. Latest first, with a choice of columns, filters by submitter, date and review state, and a warning when expected media files are missing.
• A map. Each submission plotted at the form's first location question outside a repeat group, with nearby points clustered.
• Review states. Central sets Received and Edited itself; project staff set Has issues, Approved and Rejected. The states trigger nothing by themselves: rejected submissions are still exported unless you filter them out.
• Comments and edits. Each submission has an activity feed with comments and review changes. An edit in the browser makes a new version, and the feed shows the differences between versions.
• Exports and OData. A zip of CSV files (the main table, one per repeat group, attachments and the audit log if collected), with the table's filters applied; and an OData feed, always live, for Power BI and Excel, or for pyODK and ruODK.

The guide to ODK fieldwork monitoring sets out a daily routine around these.

KoboToolbox

• The data table. Search in each column, filter on select questions and sort. Sorting applies to every user and stays; searches and filters reset when you leave (Kobo's guide to viewing and validating data).
• Validation status. Approved, Not approved or On hold, set one at a time or for many submissions at once, by users given the Validate submissions permission.
• Reports. A chart and simple descriptive statistics for each question, which can be grouped by a select-one question for simple subgroup comparisons, and custom reports with the questions you choose. Kobo's reports guide says they help you review data at a glance but don't replace in-depth cleaning, processing and analysis.
• A map. Geopoint questions only, coloured by the answers to another question, as points or as a heatmap, showing 1,000 points unless you raise the limit (mapping GPS data).
• A dashboard elsewhere. For a KoboToolbox dashboard, Power BI connects through Kobo's synchronous exports URL, and the data is brought up to date with Power BI's Refresh button (Kobo's Power BI guide).

The KoboToolbox data quality guide covers export settings and cleaning Kobo data safely.

Side by side

What the platform tools leave to you

Each tool above sees one platform's data in that platform's terms. Several checks a field team depends on fall between them:

• Duplicates on your own ID. The platform gives every submission its own key, so a household interviewed twice shows up only on your household or respondent ID. See duplicate survey responses.
• Interview length against the team's median. SurveyCTO's value checks compare a number with a threshold you set; a limit set from the median once data arrives, read per enumerator, needs a check of its own. See interview duration.
• Logic across questions, after the fact. A rule you think of in week two (owns land, but zero acres) has to run over everything that has already arrived.
• Patterns in whole interviews. Near-copies of other interviews, answers that barely vary, no “don't know” in thirty interviews. A group comparison tests a field's mean or distribution; DataSnap looks at timing, location and the flow through the form.
• Back-check comparison. Matching the back-check survey to the original interviews and counting differences per enumerator, per back-checker and per question. See back-checks.
• One record of corrections. Corrections made on the platform stay in its history; corrections made after export live in whatever script or spreadsheet someone used. The clean dataset needs one record that rebuilds it from the raw download.
• One routine across platforms. A survey firm with one client on SurveyCTO and another on ODK Central runs two sets of tools with different outputs, and reports to each client in a different shape.

Many teams fill the gap with Stata do-files such as ipacheck or with Excel. The checks are the same whichever tool runs them; the high-frequency checks guide lists them.

A daily routine that uses both

1. Build quality into the form. Constraints, required questions, an enumerator question, and the platform's own audit features, such as SurveyCTO's text audits or the ODK audit log that DataSnap reads.
2. Keep the platform's review state as the to-do list. SurveyCTO's review workflow, Has issues in ODK Central, On hold in KoboToolbox: whoever looks at the platform sees what is still open.
3. Run the checks across interviews on every new batch, over everything received so far, and read them enumerator by enumerator.
4. Send flags to supervisors the same day, with the submission key, the question, the recorded value and what to check.
5. Decide where corrections live, on the platform or in your cleaning record, and write the choice into the data management plan. Doing both leaves corrections in two places.
6. Share a summary with the client that shows progress and how each problem was handled, without personal data.

How DataFlow works alongside your platform

DataFlow is a desktop app for Windows, macOS and Linux. It reads from your platform and changes nothing there: no submission, review state or validation status. How it connects:

• SurveyLoopr. Continue with SurveyLoopr signs in with the SurveyLoopr account, and the data comes through SurveyLoopr's read-only API, so no ODK password is typed into DataFlow.
• SurveyCTO. The server name, form ID and a SurveyCTO user's username and password. That user needs a role that can download data and has to sign in with a password, not with Google, Microsoft or a second step; if SurveyCTO refuses, DataFlow says why and drafts a note for your admin. For an encrypted form, the private key, which is sent only to your SurveyCTO server. On a form with the review workflow, approved submissions and those waiting for review come in, and rejected ones stay out.
• ODK Central. The server's address and a Central account's email and password, exchanged for a session token; the password isn't kept. Each sync reads, through OData, the submissions made or changed (edited, reviewed) since the last one, and an edit in Central replaces the version DataFlow had, with DataFlow's own corrections staying on top. If the form was uploaded as an XLSForm, DataFlow downloads it.
• KoboToolbox. The account's API token, kept in your computer's keychain, on Kobo's Global or European Union server or one your organisation runs. Edits made in Kobo to submissions DataFlow already has come in when you download everything again.
• Anything else. CSV or Excel exports (.csv, .xlsx, .xls, .xlsm, .ods) from any tool.

From there the layer is the same on every platform. Each sync adds the new interviews, adds the checks the form and data call for, and runs them; syncs can run on a timetable you switch on, every 2 hours from 7:00 to 22:00 unless you change it. Flags are grouped by problem in the Today list. The Field team view shows each enumerator's interviews, median length and share ready, and marks an enumerator whose interviews show the statistical signs of being made up, compared with colleagues, with “Back-check some interviews”. The Map shows where interviews were done. A back-check survey can sit in the same project, linked to the main survey, and be compared question by question.

Corrections need a reason and sit on top of the raw data, which is never modified, so each one can be undone, and every action goes into an activity log where each entry carries a hash of the one before. A download includes the correction log and, by default, a Stata do-file that replays every correction from the raw data. A Field Report is one HTML file for the client: the public level shows totals and rates with enumerators as codes, merges areas with fewer than 5 interviews and leaves out text written by the team; the management level adds names and correction reasons, and the team level the items still open.

Questions