Data quality requirements for a survey firm's terms of reference

A terms of reference (ToR) for data collection is where data quality becomes something you can enforce. If the ToR doesn't say the client gets access to the data during fieldwork, or that back-checks are done by an independent team, you can't insist on either once fieldwork starts. The World Bank's DIME Wiki makes the same point in its guide to survey firm ToRs: spell out the expectations and the consequences of missing them, or you may be unable to act even when you find fraud.

How to use these clauses

Each clause is written to be pasted into a ToR or contract. “The Firm” is the data collection firm and “the Client” is you. Values in square brackets are yours to set. Item numbers refer to the Field Data Quality Standard, which you can attach as an annex so the checks and thresholds are defined in one place. Have your procurement or legal team review the final text; the consequences in particular must fit the contract type and local law.

1. Data access

1.1 From the first day of the pilot, the Firm will give the Client a user account on the survey server that can view and download all submissions and the current form. The Firm will not remove or restrict this access before the Client accepts the final deliverables.

Without access, you see the data when it is too late to fix. Starting at the pilot gives you time to test the connection and your checks before the first real interview.

1.2 Enumerators will send completed interviews to the server at least once a day, or at the first opportunity where there is no network. If the Client cannot reach the server, the Firm will send a complete export of all submissions, with all metadata, by [time] each day.

Daily checks need daily data. The export keeps monitoring going if the account stops working.

1.3 The Firm will not change the form on the server without the Client's written approval. Every change will carry a new form version number.

Unannounced changes break checks and leave answers from different versions that can't be compared (Standard item 1.4).

2. Daily high-frequency checks

2.1 The Firm will run the checks listed in Annex [A] on all new data every day of fieldwork, with the thresholds set there. A threshold may change only by written agreement that records the date, the old and new values and the reason.

Limits agreed before fieldwork can't be loosened quietly to make flags disappear. This is principle 4 of the Standard.

2.2 The Client may run its own checks on the data and will send the Firm one list of flags each day. Within [number] working days of receiving a flag, the Firm will do one of the following and record it: confirm the value, with an explanation; correct it, with a reason; or revisit or call the respondent and report the result.

A deadline keeps flags from piling up; choose one short enough that the enumerator is still near the respondent. Naming the three acceptable answers stops flags being closed with no explanation.

2.3 Each week the Firm will report the number of flags raised, answered and still open, by check and by enumerator.

The trend shows whether quality is improving and whether the Firm is keeping up.

3. Back-checks

3.1 The Firm will back-check [percentage] of completed interviews, drawn at random from each enumerator's interviews so that every enumerator is covered, plus interviews flagged by the daily checks. The Client may select up to [share] of the back-check sample.

Teams often use 10 to 20%, the range the DIME Wiki on back checks suggests. Drawing within each enumerator means nobody is skipped, and letting the Client choose part of the sample stops it being steered away from weak enumerators.

3.2 Back-checks will be done by a team that does not report to the supervisors of the enumerators being checked, and that has no contact with those enumerators about the back-check.

DIME recommends a back-check team that is independent of the survey team and trained separately. A supervisor checking their own team has a reason to find nothing.

3.3 Back-checks will start in the first week of fieldwork, and each will be done within [number] days of the original interview, in person or by phone. The Client will choose the back-check questions, and the Firm will not share them with enumerators.

Early back-checks find problems while there is time to retrain; DIME suggests doing a fifth of them in the first two weeks. A short gap means answers that shouldn't change still match. Enumerators who know the questions can make sure those answers agree.

3.4 Every week the Firm will share the back-check data and its comparison with the original interviews, giving the share of answers that differ by question type, for each enumerator and each back-checker. An enumerator whose answers that should not change differ in [percentage] or more of comparisons will be reviewed under clause 4.3.

Rates per enumerator show whose interviews don't hold up; rates per back-checker show when the back-checker is the problem. The Standard uses 10% as a starting point (item 9.4). The back-checks guide explains the three question types.

4. Enumerator monitoring

4.1 Each enumerator will have one unique ID, used only by them, for the whole survey. The Firm will give the Client the roster of enumerators and supervisors before fieldwork and report any change within [number] working days.

Every result per enumerator depends on an ID meaning one person.

4.2 The Firm will track, for each enumerator, interviews per day, interview length, flags per 100 interviews, the share of “don't know” and refused answers, and back-check results, and share them with the Client every week.

These are the numbers that show who needs support before a problem spreads through their interviews (Standard section 7).

4.3 When an enumerator's results stand out from the team's, or show signs of made-up interviews, the Firm will within [number] days have a supervisor accompany them, back-check a further [number] of their interviews, and retrain, suspend or replace them as the results require. The Firm will report what it did and why.

Signs of made-up interviews include near-copies, numbers that barely vary and almost no “don't know” answers (Standard item 7.4). Writing the response down in advance means it isn't argued case by case.

4.4 Where the ethics approval allows, the form will record a GPS point and the start and end times of every interview.

These let the Client check where interviews took place and whether an enumerator could have travelled between them (Standard section 8).

5. Audit trail of corrections

5.1 The Firm will not edit or delete raw submissions, on the server or in the raw data files. All corrections will be made in code, applied to a copy of the raw data.

5.2 The Firm will keep a correction log recording, for every change, the interview ID, the variable, the original value, the new value, the reason, who made it and the date. Interviews dropped from the clean data will be logged with the reason.

These are principles 1 and 2 of the Standard. With the raw data intact and a reason for every change, a reviewer can check any correction and undo it.

6. Deliverables

6.1 With each data delivery the Firm will provide: (a) the raw data exactly as exported from the server; (b) the clean data; (c) the cleaning code, as a Stata do-file or R script, that produces (b) from (a) with no manual steps; (d) the correction log; (e) the back-check data and comparison; and (f) the codebook and every version of the form.

6.2 During fieldwork the Firm will send a field report every [week] showing progress against the target, the results of each check and how flags were resolved. A final field report will come with the final data.

6.3 The Client will accept a data delivery once running (c) on (a) produces (b).

Clean data alone asks you to trust every change. With the raw data, the code and the log you can rerun the cleaning and confirm nothing was changed by hand.

7. Personal data

7.1 The Firm will store names, phone numbers, addresses, ID numbers, GPS points and other identifying fields separately from the rest of the data, encrypted, with access limited to the staff named in Annex [B]. Devices used for interviews will be encrypted and locked with a passcode.

7.2 Data will be transferred only by [agreed secure method], never as unencrypted email attachments. Data shared beyond the named staff will be de-identified.

7.3 The Firm will report any lost device or unauthorised access to data to the Client within [number] hours. It will delete all data it holds [number] days after final acceptance and confirm the deletion in writing.

Your ethics approval, and in many countries the law, sets how respondents' identities are protected. These clauses make the Firm's part explicit. See how to de-identify survey data.

8. Consequences

8.1 Interviews that fail back-checks, or that the Client and the Firm agree were not done as specified, will be redone at the Firm's cost and will not count toward the sample target or payment.

8.2 Payments are tied to the deliverables in section 6. A delivery that is missing an item, or whose cleaning code does not reproduce the clean data, will not be accepted.

8.3 If the Firm repeatedly fails to meet these requirements, the Client will issue a written warning. Continued failure [number] days after the warning is grounds to terminate the contract.

DIME's procurement guidance describes this sequence: consequences written into the ToR let you warn the firm and, if the problem continues, cancel the contract. Without them, finding a problem doesn't give you a remedy.

Doing this with DataFlow

These clauses work with any tool. With DataFlow, the Client can connect to the Firm's SurveyLoopr, SurveyCTO or ODK Central server using the account from clause 1.1, or import the daily exports from clause 1.2 (KoboToolbox included), and run the agreed checks on every sync. A Firm that uses DataFlow keeps to clauses 5.1 and 5.2 as it works: raw data is never modified, and every correction is logged with who, when and why. Every download includes a Stata do-file that rebuilds the clean data from the raw data, as clause 6.1 (c) asks, and the Field Report is a single file that covers clause 6.2. See DataFlow for clients.

Questions