Testing, the report and analysis

Review before freezing

While a scheme is a draft, Review before freezing lists what is still missing, the form check, the block plan (for each block its size, the target, the quota, the share it gives and how far that is from r), the coding of every garbled question and the totals. Freeze fixes the design; from then on every submission of the form is garbled as it arrives.

Test runs

Run a test on the scheme's page. A test uses its own seed and never touches the scheme's key.

• Simulation: made-up respondents with a prevalence you choose, with missing and invalid answers, duplicates and unknown IDs, in your blocks. It checks that every answer follows the rule, the quotas hold and nothing restricted would be exported, and says PASSED only when every check that applies ran and none failed.
• Pilot run: the same checks on a pilot data file (an owner who is a key custodian).
• Form dry run: one made-up submission of your deployed form per case (Yes and No, assigned and not, blank, a nonresponse code, an invalid value), run through the very step that garbles real submissions, showing field by field what would be stored.

Every run gives a diagnostic CSV marked "TEST - NOT PRODUCTION DATA". Runs are kept for seven days.

The report

The Garbling report (owners and managers; HTML or CSV) shows the design, the plan against what was achieved, unknown, duplicate, missing-ID and excluded counts, and for each question the share recorded Yes and two estimates of the share answering Yes, with standard errors, overall and per block:

• the naive estimate, (m − r) / (1 − r), which assumes exactly the rate r was assigned;
• the known-share estimate, (m − f) / (1 − f), which uses the share f actually assigned (the same as the share of Yes among the unassigned).

Every estimate assumes the assignment is independent of the true answer (it is, by construction), that nobody falsely reports Yes, and that the coding is right. It is an estimated share answering Yes, not a true prevalence. While the scheme is active, numbers that depend on who was assigned are shown only to key custodians.

The garbled analysis export

Once a scheme is frozen, owners and managers can download a zip for analysis:

• <form>_garbled.csv: the stored, garbled submissions, with the form's privacy rules applied and restricted columns left out; the respondent ID kept, pseudonymised or dropped, as you choose; and two added columns, garbling_block and garbling_counted;
• garbling_metadata.json: the design, rate, coding, block plan, versions and hashes (achieved shares only once they are revealed);
• analysis.do and analysis.R, which compute both estimates, overall and per block;
• validation.json and README.txt.

The file is identifier-restricted, not anonymous: treat it as identifiable data.

Data collected elsewhere

Process a file garbles an export collected elsewhere (with another tool, say) with the scheme's key and returns the garbled CSV and a validation file. It is for an owner who is a key custodian, and for IID and roster-block schemes only (interview order depends on arrival). Nothing is stored. The raw answers still exist wherever the file came from, so this protects only the people who receive the garbled copy.

After fieldwork

1. Close the scheme (an owner). Late submissions are stored with codes removed and answers blank, and flagged.
2. Download the report and the analysis export, and in faithful mode a custodian's decoded file.
3. Destroy the key (an owner who is a key custodian, typing "destroy the key"). The key, the seed, the typed codes and the per-submission assignments are deleted, and the enumerator sheet is removed from the form. From then on nobody, custodians included, can learn who was assigned. Until the key is destroyed, a recorded No or an unassigned answer is a known true answer to whoever opens the vault.