How to de-identify survey data before sharing it

What de-identification means

De-identification removes or masks the information in a dataset that could tell someone who a respondent is (personally identifiable information, or PII), so the data can be analysed, shared with colleagues or published with less risk to the people in it. It reduces the risk of re-identification; it rarely removes it. J-PAL's guide makes this point and asks researchers to weigh how identifiable the data is against the harm if someone were identified.

Two terms get mixed up. Data whose identifiers sit in a separate key file, which can link it back to people, is de-identified (in GDPR terms, pseudonymised). Data where no such link exists anywhere, and none can be rebuilt, is anonymized. Most survey projects need the key for follow-up rounds and back-checks, so rather than trying to anonymize survey data while the project runs, aim for a de-identified working dataset and a well-protected key.

Direct identifiers and quasi-identifiers

A direct identifier points to one person on its own: a name, a phone number, a national ID number, an exact GPS point. A quasi-identifier (or indirect identifier) doesn't, but a few together can: age, sex, village, occupation, household size, ethnicity. In a village of 200 households, the only female teacher over 60 is identified by three ordinary variables. Sensitive data is a third category: answers that don't identify anyone but would harm them if linked to their name, such as health, HIV status, pregnancy, disability, religion, caste or ethnicity, sexual history, experience of violence, criminal history and mental health.

Finding them

Look in three ways, because each misses things the others catch. By variable name and question label: words such as name, phone, mobile, address, ID, birth, GPS, photo and signature, read with judgement (“mobile money: yes or no” is not a phone number, and “village name” is usually not personal). By values: a column called q14_b can still hold phone numbers or names, so look at what the answers look like, not only what the variable is called. And in free text: comments and “other” answers carry names (“son Habib answered”), phone numbers and house numbers, so read them.

Separating them: the key file

For each identifier, decide whether the analysis needs it. If it doesn't, it goes into the key file and out of the working data. The key file holds the original ID, the new random code and every direct identifier, and nothing else. Keep it in an encrypted container such as a VeraCrypt volume, separate from the de-identified data, with access limited to the people who need it for follow-up and back-checks. The DIME Wiki recommends the same flow: the raw, identified data stays exactly as received in an encrypted folder, and the team works from a de-identified copy produced by a reproducible do-file.

Replacing IDs with random codes

Household IDs are often built from codes that mean something (district, village, a running number) or taken from a registry that people outside the project can see. Replace them with codes that carry no information, and keep the link in the key file:

* Random household codes; the link stays in the encrypted key file
preserve
    keep hhid
    duplicates drop
    set seed 58213
    generate double shuffle = runiform()
    sort shuffle
    generate long hh_code = _n
    drop shuffle
    save "V:/keys/hhid_key.dta", replace
restore
merge m:1 hhid using "V:/keys/hhid_key.dta", nogenerate
drop hhid

Do the same for every other ID that links to people: individual IDs in the roster, school or clinic IDs, and enumerator IDs if they are names.

Coarsening GPS, dates and rare values

Exact coordinates identify a household. If the analysis needs location, compute what it needs first (distance to the market, the cluster a household belongs to), then drop the coordinates or round them; one hundredth of a degree of latitude is about 1.1 km. Don't shift every point by the same distance: the DIME Wiki notes that a fixed displacement is easy to reverse. Replace dates of birth with age, and event dates with the month or with the number of days between two events. For rare values that single someone out (the one household with 40 cattle), use categories or top-coding.

generate age_years = floor((interview_date - dob) / 365.25)
drop dob
replace gps_latitude  = round(gps_latitude, 0.01)
replace gps_longitude = round(gps_longitude, 0.01)
generate harvest_month = mofd(harvest_date)
format harvest_month %tm
drop harvest_date

Checking free text

Read every free-text variable before sharing. Recode “other, specify” answers into categories and drop the text, or redact the names and numbers in it. Enumerator comments are useful during fieldwork and rarely needed for analysis; they usually belong in the key file or out of the shared data. Photos and audio files stay with the identified data.

Checking combinations

Once the direct identifiers are gone, count how many respondents share each combination of the quasi-identifiers that remain. A combination held by one or two people identifies them to anyone who knows the village. Agree the minimum group size with your ethics board, then coarsen whichever variable makes the small groups (age bands instead of age, district instead of village):

bysort village sex age_band occupation: generate combo_n = _N
tabulate combo_n if combo_n < 5

For data you publish, statistical disclosure control tools such as the R package sdcMicro measure this risk more formally.

Keeping a record

Write the de-identification as a script that runs from the raw data to the shared file, and keep a short note of every change: which variables were dropped, moved to the key file, recoded or rounded, and why. The people analysing the data need to know that ages are banded and coordinates rounded; the note itself may describe the key, so store it with the identified data.

What DataFlow does

DataFlow looks for personal data in each survey in four ways. It reads variable names and question labels for words that usually mean personal data (name, phone, email, address, ID, birth, GPS, photo, signature, relatives' names), skipping words like name, phone or address in choice and count questions, which can't hold them. It uses the form's question types to find GPS points, photos, audio and files, and the device, subscriber and SIM IDs and phone number in the metadata. It looks at the values themselves, so a variable named innocently but holding emails, GPS points, phone or ID numbers, street addresses or people's names is found anyway, and it reports only the kind and a count, never the values. And a small model on the computer reviews the remaining variables and reads free-text answers one by one; a text variable is suggested once at least two of its answers name someone or give contact details, since a single one is often the enumerator's own name or an official's. Sensitive topics (health, HIV, pregnancy, disability, religion, caste or ethnicity, sexual history, violence, criminal history, mental health) are flagged too.

You choose what to protect. Protected variables move to a separate store with its own encryption key, and the working dataset keeps blanks in their place, including for new interviews on each sync. Checks, Field Reports and downloads are de-identified; putting personal data back into a download needs an explicit unlock, recorded in the activity log with its purpose. GPS checks read coordinates inside the app and their flags carry distances, not coordinates, and the map shows areas unless someone asks for exact places, which is logged. The raw interviews, personal data included, can be kept in a vault: an encrypted container in VeraCrypt's format (AES-256-XTS), written by DataFlow after each sync and opened with a password, a keyfile or both. Anyone with VeraCrypt and the password can open it as an ordinary folder, without DataFlow.

DataFlow doesn't replace IDs with random codes, coarsen GPS or dates, or check combinations of quasi-identifiers for you; do those steps in your own script before publishing. The model runs on the computer and works offline, so survey data is never sent to an AI service. Security and privacy lists everything the app sends anywhere.

Step by step

1. List every identifier. Search variable names and question labels, look at what the values look like, and read the free text, to list the direct identifiers, quasi-identifiers and sensitive variables.
2. Decide what the analysis needs. For each identifier, decide whether to drop it from the working data, keep it only in the key file, or keep a coarsened version.
3. Move direct identifiers to an encrypted key file. Put names, phone numbers, ID numbers, exact locations and the original IDs in a key file inside an encrypted container, kept apart from the data, with access limited to those who need it.
4. Replace IDs with random codes. Give every household and person a code with no meaning, keep the link to the original ID only in the key file, and drop the original ID from the working data.
5. Coarsen GPS, dates and rare values. Compute any distances or ages you need first, then round or drop coordinates, replace dates with months or intervals, and band or top-code values that single someone out.
6. Check the free text. Read comments and other-specify answers; recode them and drop the text, or redact names and numbers.
7. Check combinations. Count respondents sharing each combination of the remaining quasi-identifiers and coarsen variables until no small groups are left.
8. Keep a record. Run the whole process as a script from the raw data, and note every variable dropped, moved, recoded or rounded, stored with the identified data.

Questions