How to choose a data collection firm

Proposals from data collection companies usually compete on price, coverage and speed. Data quality is harder to compare, because every proposal promises it. The way round that is to ask each firm the same specific questions, score the answers on the same scale, and put the winning answers in the contract. This guide is for research teams, NGOs and funders hiring a survey firm, also called a data collection agency, for fieldwork.

Decide what the firm does and what you keep

Before you write the request for proposals, decide the split of work. The World Bank's DIME Wiki on survey firm terms of reference describes a common one: the firm gets permits, translates and field-tests the questionnaire, recruits and trains enumerators, and runs fieldwork, while the research team keeps the questionnaire, the sample, data quality monitoring, cleaning and analysis. Programming the form can sit on either side. Whatever you choose, you need access to the form and the data, and the firm needs to know which checks you will run.

A private firm isn't the only option. A government agency, the national statistics office or enumerators you hire directly each trade cost, control and effort differently; DIME's page on procuring a survey firm compares them.

What to ask before hiring

The survey platform, and your access to it

Ask which platform the firm will use (SurveyLoopr, SurveyCTO, ODK Central, KoboToolbox or another), who owns the server account, and whether you get your own login to view and download submissions from the pilot onward. Ask how often enumerators sync and what happens in areas with no network. You don't need the firm to use a particular platform. You need to see the data on whichever one it uses while fieldwork is running.

The quality assurance plan

Ask for the written plan, not a paragraph in the proposal: which checks run, how often, with what thresholds, who runs them, and how a flag gets back to the enumerator. Ask for an anonymised check report from a past survey. A firm that runs daily checks will have one. The Field Data Quality Standard lists the checks a plan should cover, if you want something to compare against.

Back-checks

Ask what share of interviews the firm back-checks, how the sample is drawn, who does the back-checks, how soon after the interview, and how the results are reported. Strong answers describe a random sample that covers every enumerator, plus interviews flagged by the checks; a team that doesn't report to the supervisors being checked; and results per enumerator every week. The DIME Wiki suggests back-checking 10 to 20% of interviews. Ask whether you can choose part of the sample yourself. More on this in the back-checks guide.

Supervision

Ask how many enumerators each supervisor covers, how often supervisors sit in on interviews and what they record when they do. There is no single right ratio: it depends on distances, the length of the questionnaire and the team's experience. Look for a number with a reason behind it.

How the firm monitors enumerators

Ask what the firm tracks for each enumerator (interviews per day, interview length, flags, “don't know” rates, back-check results) and what happens when someone stands out: a supervisor visit, retraining, removal. Ask how enumerators are paid. Pay per completed interview rewards speed, so ask what the firm does to balance it.

Personal data

Ask where the data is stored during and after fieldwork, who can see names and phone numbers, whether devices and servers are encrypted, how files will be sent to you, and when the firm deletes its copy. The answers have to fit your ethics approval and any data protection law that applies. Ask whether personal fields are kept apart from the data the team works on; the guide to de-identifying survey data explains why that matters.

What the firm delivers

Ask for the list in writing. You want the raw data as it came from the server, untouched; the clean data; the code that produces the clean data from the raw; a log of every correction with its reason; the back-check data; and field reports during and after fieldwork. A firm that offers only a “clean dataset” is asking you to trust every change it made without seeing any of them.

References

Ask for references from surveys like yours in size, topic, region and mode, and call them. Ask whether data arrived on time, how problems were raised and fixed, how much cleaning the data needed after delivery, and whether they would hire the firm again. Ask the firm, too, who will run your survey day to day. The field manager and the data manager you get matter more than the company profile.

A scoring table

Score each proposal on every area from 0 to 2: 0 when the answer is missing or vague, 1 when it is described, and 2 when it is described with evidence such as a sample report, a named person or a reference who confirms it. Multiply each score by the weight, add them up and divide by 2 for a mark out of 100. The weights are a starting point; move them to suit your study.

Score the technical proposal before you open the price, so the price doesn't colour the scores. DIME also publishes a technical review scoring matrix for survey firms, linked from its procurement page.

Red flags

• No access to the data until fieldwork ends.
• No written quality assurance plan, or one that says only that supervisors “check all questionnaires”.
• Back-checks done by the enumerators' own supervisors, or results “available on request”.
• A price well below the other proposals, with no explanation of where the savings come from. DIME's procurement guidance warns against proposals that look too cheap to be true.
• No named field manager or data manager.
• Cleaning done by hand in spreadsheets, with no log and no code.
• Only the clean data offered as a deliverable.
• Vague answers about where personal data is kept and who can see it.
• References only from different kinds of work, or none you can call.
• A promise of error-free data. Every survey has errors, so ask how the firm finds and fixes them.

Write the answers into the contract

A promise in a proposal only helps if the contract repeats it. Copy the answers you scored into the terms of reference: the data access, the checks and thresholds, the back-check share and team, the deliverables, and what happens if any of them is missed. The data quality requirements for a terms of reference have clauses to start from, and how to monitor a survey firm covers what to do once fieldwork starts.

Doing this with DataFlow

You can check a firm's work without asking it to change platforms. DataFlow connects to SurveyLoopr, SurveyCTO and ODK Central with the account the firm gives you, and imports CSV or Excel exports from KoboToolbox and other tools. It runs the checks on every sync, on your computer. A firm that uses DataFlow can send you a Field Report each week and, with the data, the do-file and the correction log. See DataFlow for research teams and clients and for survey firms.

Step by step

1. Decide what the firm does and what you keep. Split the work before you write the request: fieldwork, training and permits usually sit with the firm; agree who programs the form, monitors quality and cleans the data.
2. Write the quality requirements into the terms of reference. State data access, daily checks, back-checks, deliverables, personal-data handling and consequences, so every firm prices the same work.
3. Shortlist firms with similar experience. Look for firms that have run surveys like yours in size, topic, region and mode.
4. Ask every firm the same questions. Cover the survey platform and your access, the quality assurance plan, back-checks, supervision, enumerator monitoring, personal data and deliverables.
5. Score the proposals. Score each area from 0 to 2, weight the areas, and score the technical proposal before opening the price.
6. Call the references. Ask past clients about timeliness, how problems were handled and how much cleaning the data needed after delivery.
7. Put the answers in the contract. Copy what the chosen firm promised into the contract, with deadlines and consequences, and use it to monitor the firm during fieldwork.

Questions