deident

Privacy transformation engine for structured datasets — pseudonymization and risk-assessed anonymization for CSV, JSONL, Parquet and DICOM, driven by a declarative YAML policy.

$ deident anonymize patients.csv --policy patients.yaml --out anon.csv --report report.json
Anonymize complete: 12 row(s) in, 12 row(s) out (dataset 'patients-demo')
  direct identifiers: patient_id (removed), full_name (removed), email (removed)
  quasi-identifiers [age, zip, admission_date]: 5 equivalence class(es), min size 1, 1 unique row(s) (8.3%)
  output: anon.csv
  report: report.json

Two modes, two very different promises

	`pseudonymize`	`anonymize`
Reversible?	Yes — with the key material	No — values are removed or generalized
Output is still personal data?	Yes. Treat it as such.	Reduced risk, not zero risk
What happens to direct identifiers	replaced with deterministic tokens	removed (or redacted)
What happens to quasi-identifiers	kept unchanged	generalized/suppressed per policy
Typical use	joinable test/analytics data, debugging with real structure	sharing data with reduced re-identification risk

⚠️ No guarantees. This tool performs risk-assessed anonymization: it reduces re-identification risk and measures residual risk signals, but it cannot certify anonymity — that always depends on external data and context it cannot observe. Pseudonymized output remains personal data and is reversible by anyone holding the key material, so protect keys separately from outputs.

What it does

Formats — CSV, JSONL/NDJSON and Parquet, inferred from the file extension. Input and output formats are independent, so a job converts while it transforms. See Formats.
Column rules — classify every column (direct_identifier, quasi_identifier, sensitive, utility) and pick a strategy: tokenize, remove, redact, bucket, truncate dates, keep a prefix. See Policy reference.
Content patterns — find identifiers inside values (an IBAN in a free-text note) with 16 built-in detectors (email, IBAN, card, SSN, phone, IP, URL, API key, passport, plate, IFSC, date of birth, plus heuristic name / address / organization / medical-term matchers) or your own regex, then detect, redact, tokenize or replace them with structurally valid fakes. Eight of the sixteen are validated by checksum or structural parse (mod-97, Luhn, real calendar dates, IP parsing), so a loose pattern does not cost you false positives. See Built-in detectors.
Chained datasets — process several files as one export with shared token scoping, so foreign keys still join after pseudonymization. See Chained datasets.
Sandboxed execution — each job can run in its own WebAssembly sandbox (Wasmtime + WASI): fresh store per job, one preopened directory, no network, memory/CPU/time limits. See Sandboxed execution.
Encrypted mapping vault — optionally record original→token mappings under XChaCha20-Poly1305 for authorized re-identification. See Mapping vault and reversal.
Risk report — row counts, per-identifier actions, pattern findings and equivalence-class statistics. See Risk report.
DICOM — metadata de-identification of medical imaging instances, with consistent UID remapping across a study. See DICOM.
Policy lints — warn about risky-but-valid policies before a job runs. See Policy lints.
Audit log — append-only JSONL, metadata only. See Audit log.

Status: MVP, but a complete one — every feature above is implemented and tested. Streaming for very large datasets, richer vault workflows and more formats are on the Roadmap.

Installation

Requires a Rust toolchain (1.96+).

git clone <this-repo> && cd deident-wasm

# install the `deident` binary onto your PATH
cargo install --path crates/cli

# to also sandbox jobs, build the guest module (see Sandboxed execution)
rustup target add wasm32-wasip1
cargo build -p deident-worker --target wasm32-wasip1 --release

# — or just build and use it from target/release/
cargo build --release
./target/release/deident --help

Quick start

The repo ships a demo dataset and policy:

# reversible: tokenize direct identifiers, keep everything else
deident pseudonymize examples/data/patients.csv \
  --policy examples/policies/patients.yaml \
  --out pseudo.csv

# irreversible: remove/generalize identifiers, write a risk report
deident anonymize examples/data/patients.csv \
  --policy examples/policies/patients.yaml \
  --out anon.csv \
  --report report.json

To run against your own data you need a policy file that lists every column of your dataset — see Policy reference. deident lint will tell you if it looks risky.

CLI reference

deident <COMMAND> [OPTIONS]

Command	Description
`pseudonymize <INPUT>...`	Reversibly tokenize direct identifiers (deterministic per dataset/policy)
`anonymize <INPUT>...`	Irreversibly remove/generalize identifiers and produce a risk report
`chain <MANIFEST> --mode <MODE>`	Run several datasets as one chained export (Chained datasets)
`lint <POLICY>`	Report risky-but-valid policy configurations (Policy lints)
`vault export <VAULT> --policy <FILE>`	Decrypt a mapping vault to CSV (Mapping vault)
`reverse <INPUT> --vault <FILE> --policy <FILE> --out <FILE>`	Re-identify tokenized values using a vault
`dicom <INPUT> --policy <FILE> --out <PATH>`	De-identify DICOM instance metadata, file or directory (DICOM)
`help [COMMAND]`	Print help

Options for pseudonymize / anonymize:

Option	Required	Description
`<INPUT>...`	yes	Input file(s); format inferred from the extension (`.csv`, `.jsonl`/`.ndjson`, `.parquet`). Several inputs run concurrently, one sandbox each (Parallel execution)
`--policy <FILE>`	yes	Policy YAML describing field classes and strategies
`--out <PATH>`	yes	Output file; its extension selects the output format. With several inputs, a directory
`--report <FILE>`	no	Write the JSON risk report here
`--vault <FILE>`	no	Write the encrypted mapping vault here (only if the job produces reversible values)
`--split <N>`	no	Split one dataset across `N` sandboxes and merge the results (Parallel execution)
`--jobs <N>`	no	Maximum sandboxes running at once (default: cores, capped at 8)
`--no-lint`	no	Skip the pre-flight policy lint
`--deny-lints`	no	Refuse to run when a warning-level lint fires

Engine options (also accepted by chain):

Option	Default	Description
`--engine <ENGINE>`	`auto`	`auto` sandboxes when a worker module is available and falls back in-process with a warning; `wasm` requires the sandbox; `native` runs in-process
`--worker <FILE>`	discovery	Compiled worker module (see discovery order below)
`--max-memory-mib <N>`	`256`	Guest memory limit in MiB (sandbox only)
`--timeout-secs <N>`	`30`	Job wall-clock timeout in seconds (sandbox only)
`--fuel <N>`	scaled	Fixed CPU budget in Wasmtime fuel units; the default scales with input size
`--no-fuel`	off	Disable fuel metering (the wall-clock timeout still applies)
`--audit-log <FILE>`	off	Append one JSONL audit record per job (Audit log)

lint also accepts --mode <MODE> (restrict to lints relevant for one mode), --json, and --deny (exit non-zero on any warning).

Global: -h, --help, and -v / -V / --version.

Environment:

Variable	Purpose
`DEIDENT_KEY` (or whatever the policy's `key.env` names)	Secret for key derivation
`DEIDENT_WORKER_WASM`	Path to the worker module for sandboxed execution
`RUST_LOG`	Log verbosity on stderr, e.g. `RUST_LOG=debug` (default `info`)

Exit codes: 0 success, non-zero on any failure (bad policy, unreadable input, unlisted column, missing key, a failed chain job, --deny-lints with warnings). Human summary goes to stdout, logs and lint warnings to stderr.

Formats

The format of each file is inferred from its extension, independently for input and output — so a job can convert while it transforms:

Extension	Format	Notes
`.csv`	CSV	Header row required. A leading UTF-8 BOM (Excel writes one) is stripped, so the first column still matches its policy field
`.jsonl`, `.ndjson`	JSON Lines	One flat object per line. The first record defines the columns; later records may omit keys (treated as empty) but not add new ones. Nested objects/arrays are rejected rather than silently flattened. Numbers and booleans keep their JSON type when their value is unchanged; generalized values (`"30-39"`) become strings; empty becomes `null`
`.parquet`, `.pq`	Apache Parquet	Column types are re-inferred from the transformed values, so untouched numeric columns stay `Int64`/`Float64` while generalized ones become `Utf8`. Not available inside the sandbox (see below)

# read Parquet, write JSONL, transforming on the way through
deident anonymize events.parquet --policy p.yaml --out events.jsonl

Both Parquet directions hold the table in memory (its footer-based layout makes true streaming impractical); CSV and JSONL stream row by row.

Column names are matched exactly. A policy field that matches no column is inert — if it named a direct identifier, that identifier would be copied through in the clear. Two things guard against it: the default on_unlisted: error fails the job on any column the policy does not cover, and an unmatched field is reported as a warning that names a case- or whitespace-only near miss when one exists (patient_id vs Patient_ID).

Outputs are published, not written in place. Each job writes to a temporary sibling file and moves it into place only after the transformation has run to completion, so a job that fails halfway leaves no truncated file for a downstream consumer to mistake for a finished dataset. The same applies to the mapping vault: a failed job publishes neither.

Sandboxed execution

--engine wasm (and auto, the default, when a worker module is available) runs each job inside its own WebAssembly sandbox instead of in-process. The exact same transformation code runs either way — the core crate compiles into both — and outputs are byte-identical, verified by tests. The sandbox adds an isolation layer around the parsing/transformation logic:

Fresh instance per job — a new Wasmtime store and WASI context every time; no state survives from one job to the next.
One directory, nothing else — the guest sees a single preopened job workspace containing a copy of the input; your real filesystem paths never reach it. Attempts to read outside (absolute paths, .. escapes) fail.
No network — the WASI context simply has no socket capability.
Minimal environment — only the one key variable a pseudonymize policy names is passed through, and only if set.
Resource limits — guest memory (--max-memory-mib), a wall-clock timeout enforced by epoch interruption (--timeout-secs), and a CPU budget in Wasmtime fuel units. The fuel budget scales with input size by default (a fixed budget would either starve large jobs or be meaningless for small ones); override with --fuel <N> or turn it off with --no-fuel.

Sandboxing reduces the blast radius of malformed inputs and future untrusted plugins; it is a mitigation, not an absolute security boundary.

Format caveat: the sandbox build deliberately excludes Parquet — the arrow stack inflates the guest module from ~1.9 MB to ~7.4 MB, which Wasmtime then has to JIT-compile for every job. CSV and JSONL work in the sandbox; Parquet jobs run in-process (--engine auto switches automatically and says so).

Build the worker module once, then use it:

rustup target add wasm32-wasip1
cargo build -p deident-worker --target wasm32-wasip1 --release

deident anonymize input.csv --policy p.yaml --out out.csv --engine wasm

The worker module is found in this order: --worker <FILE>, then $DEIDENT_WORKER_WASM, then deident-worker.wasm next to the deident binary, then the local cargo build under target/wasm32-wasip1/. For deployment, copy deident-worker.wasm next to the installed binary.

Parallel execution

Two independent axes, both giving every job its own sandbox with its own Store, WASI context and resource limits. The guest module is compiled once and shared; nothing else is.

Several datasets at once — pass more than one input. --out then names a directory and each output keeps its input's file name:

deident pseudonymize patients.csv visits.csv labs.jsonl \
  --policy policy.yaml --out ./out --jobs 3

Results are reported in the order the inputs were given, whatever order they finished in. One dataset failing does not stop the others; the command exits non-zero and names the input that failed.

Because --report and --vault each name a single file, they are refused with several inputs — the per-dataset artifacts would overwrite each other. Use deident chain when you want per-dataset reports, or run the datasets separately.

One large dataset across N sandboxes — --split:

deident pseudonymize huge.csv --policy policy.yaml --out out.csv \
  --report risk.json --split 8 --jobs 8

The input is divided by rows into N staged chunks (each CSV chunk repeats the header, so it is a valid file in its own right), each chunk runs in its own sandbox, and the outputs are concatenated in order. The result is byte-identical to an unsplit run — tokens are a deterministic function of the key and the value, so chunks agree without coordinating. On a 200k-row file, 8 sandboxes cut wall clock roughly 3x.

Constraints:

Line-oriented formats only (.csv, .jsonl/.ndjson). Parquet is columnar with a footer, so a byte range of it is not a valid file.
Not with --vault. Each chunk would write its own vault with overlapping entries, and merging encrypted mapping files is not implemented.
--split applies to a single dataset. With several inputs each one already gets its own sandbox.
Fewer rows than chunks, or --split 1, silently runs as one job.

How a split report is merged

Row counts and pattern-match counts are additive, so they are summed. The equivalence-class statistics are not, and summing them would be wrong in a way that matters:

A quasi-identifier combination appearing once in chunk A and once in chunk B is one class of size two — not two classes of size one. Summing per-chunk figures would report far more unique rows than the dataset contains, i.e. it would overstate re-identification risk, and someone widening their buckets in response would be chasing an artefact of the chunking.

So they are not merged at all. Once the chunk outputs are concatenated, the host recomputes them over the whole output using the same code path a single-job run uses. That costs one extra pass and makes the figures host-attested rather than assembled from fragments. A split run's report carries a warning saying so, and its unique_rows, equivalence_classes and k_thresholds match an unsplit run exactly.

Policy reference

A policy is a YAML file that classifies every column and configures how each mode treats it. Complete annotated example:

version: 1                  # required; only 1 is supported
dataset: patients-demo      # required; scopes key derivation (see Key management)

key:                        # required for pseudonymize, ignored by anonymize
  env: DEIDENT_KEY          # name of the env var holding the secret (preferred)
  inline: "demo-secret"     # fallback secret — demos/tests only, always warned

on_unlisted: error          # what to do with CSV columns not listed below:
                            #   error  – fail the job (default, deny-by-default)
                            #   keep   – pass through unchanged + warning
                            #   remove – drop the column + warning

fields:
  - name: patient_id                # column name, must match the CSV header
    class: direct_identifier        # see field classes below
    pseudonymize:                   # optional, pseudonymize mode only
      prefix: "pid_"                # cosmetic token prefix

  - name: email
    class: direct_identifier        # no config needed: tokenized in pseudonymize
                                    # mode, removed in anonymize mode by default

  - name: age
    class: quasi_identifier
    anonymize:                      # anonymize-mode strategy (see below)
      strategy: bucket
      width: 10

  - name: zip
    class: quasi_identifier
    anonymize:
      strategy: keep_prefix
      chars: 3
      pad: "*"                      # optional, default '*'

  - name: admission_date
    class: quasi_identifier
    anonymize:
      strategy: date_truncate
      granularity: year             # year | year_month

  - name: diagnosis
    class: sensitive

  - name: notes
    class: utility

patterns:                           # content-pattern rules, see below
  - name: iban
    builtin: iban
    fields: [notes]
    action: redact

Unknown YAML keys anywhere in the policy are rejected (typos fail fast).

Field classes

Class	Meaning	Pseudonymize mode	Anonymize mode
`direct_identifier`	Identifies a person on its own (name, email, ID number)	tokenized	removed (default) or the configured strategy
`quasi_identifier`	Identifying in combination (age, zip, dates)	kept unchanged	configured strategy; kept + warning if none
`sensitive`	Sensitive payload (diagnosis, salary)	kept unchanged	kept, unless a strategy is configured
`utility`	Analytic utility only	kept unchanged	kept, unless a strategy is configured

Anonymization strategies

Set under a field's anonymize: block; applied only in anonymize mode.

`strategy`	Parameters	Example
`remove`	—	column is dropped entirely
`redact`	`replacement` (default `"REDACTED"`)	`Alice` → `REDACTED`
`bucket`	`width` (positive integer)	width 10: `34` → `30-39`, `-3` → `-10--1`; floats floored
`date_truncate`	`granularity`: `year` \| `year_month`	`2024-03-14` → `2024` or `2024-03`; ISO dates/timestamps only
`keep_prefix`	`chars`, `pad` (default `*`)	chars 3: `81549` → `815**`

Values that don't fit their strategy (e.g. bucket on n/a, date_truncate on 14.03.2024) are suppressed to * and counted in the report warnings — a single bad cell never fails the job. Empty cells always pass through empty.

Pseudonymization options

Set under a field's pseudonymize: block; applied only in pseudonymize mode and only to direct_identifier fields (which are tokenized with or without this block).

Key	Description
`prefix`	Cosmetic prefix prepended to the token, e.g. `pid_` → `pid_21134bb99aee85cb...`
`domain`	Identity domain the token is derived in; defaults to the column name. Give differently named columns in different files (e.g. `patient_id` and `patient_ref`) the same domain so the same value yields the same token — foreign keys survive (see Chained datasets)

Content-pattern rules

Column-level rules can't reach identifiers hiding inside values — an IBAN in a free-text notes column, an email in a comment. patterns: rules scan cell content and run in both modes, after the column-level transform:

patterns:
  - name: iban            # rule name; also the default redaction label
    builtin: iban         # or a custom regex — exactly one of the two:
    # regex: '\b[A-Z]{2}[0-9]{2}[A-Z0-9]{11,30}\b'
    fields: [notes]       # columns to scan; omit = every column in the output
    action: redact        # detect | redact | token | mock
    # replacement: "[IBAN]"   # redact only; default "[<NAME>]"
    # prefix: "ib_"           # token only
    # mock: iban              # mock only; defaults to the `builtin` shape

`action`	Effect
`detect`	Only count matches for the risk report; values stay in the output (a warning is recorded)
`redact`	Replace each match with a fixed label (default `[IBAN]`-style)
`token`	Replace each match with a deterministic keyed token — same IBAN, same token, so joins/grouping survive. Requires a key source
`mock`	Replace each match with a deterministic, structurally valid fake of the same shape (see below). Requires a key source

Both token and mock are reversible with the key material: in anonymize mode the report and the lints flag the affected output as pseudonymous rather than anonymous.

Format-preserving mocks

action: mock is for downstream systems that validate their input and would choke on [IBAN] or a hex token. Mocks are derived from the same keyed hash as tokens, so they are deterministic and stable — the same input always yields the same mock, and joins on the mocked value keep working.

Shape	What is preserved	What is generated
`iban`	Country code and length	Correct mod-97 (ISO 7064) check digits, so validators accept it
`credit_card`	Length and separators	Valid Luhn check digit, forced into the `999x` test IIN range so it cannot collide with a real issuer
`phone`	Punctuation and digit count	New digits, leading digit kept non-zero
`email`	Nothing of the original	A random local part at `example.com` (RFC 2606 documentation domain)

The shape comes from builtin:, or set mock: explicitly when mocking a custom regex: rule. A mock is a pseudonym with a prettier shape, not anonymization: it is recorded in the mapping vault exactly like a token, and anyone with the key can recompute it.

⚠️ Mocks collide, and sooner than you would guess. Preserving a format bounds the value space: a 9-digit phone number has only 10^9 possible mocks, so by the birthday bound two different numbers start sharing a mock at around 31,000 distinct values — a small dataset. Email mocks (26^10) and IBANs are far roomier; phone and short card shapes are the risky ones.

When it happens, two identities share one value in the output and the mapping stops being invertible. The tool does not paper over it:

the risk report names the pattern and counts the colliding values, at transformation time rather than months later;

deident reverse refuses an ambiguous value, leaving it in place and exiting non-zero, rather than restoring a value that may belong to someone else.

Use action: token (128-bit, no practical collisions) wherever the mapping has to stay reversible. Mocks are for feeding format-validating systems, not for round-tripping identities.

Dropped and tokenized columns are never scanned (nothing left to find).

Built-in detectors

Sixteen detectors, grouped by how much a match can be trusted. That grouping is the important part: it stops a heuristic guess from being mistaken for a verified identifier.

Eight of the sixteen are validated beyond their pattern — the match must also pass a checksum or a structural parse before it counts.

Detector	Example	Class	Validated by
`email`	`user@example.com`	precise	✅ RFC 5321 structure: single `@`, length limits, dotted domain, alphabetic TLD
`iban`	`DE89 3704 0044 0532 0130 00`	precise	✅ mod-97 (ISO 7064) check digits
`credit_card`	`4111 1111 1111 1111`	precise	✅ Luhn plus card length (13–19) and issuer prefix (2–6)
`ip_address`	`192.168.1.1`, `2001:db8::1`	precise	✅ parsed as a real address (`std::net::IpAddr`)
`url`	`https://internal.company.com`	precise	✅ known scheme, plausible host, no whitespace
`api_key`	`AKIA…`, `sk-proj-…`, `github_pat_…`, `xoxb-…`, `glpat-…`	precise	— opaque by design
`ifsc`	`HDFC0001234 000123456789`	precise	— no checksum exists
`ssn`	`123-45-6789`	moderate	✅ US allocation rules (area ≠ 000/666/9xx, group ≠ 00, serial ≠ 0000)
`date_of_birth`	`15/03/1990`, `March 15, 1990`	moderate	✅ a real calendar date (rejects `31/02`, leap years honoured)
`phone`	`+1-555-0123`, `+91 98765 43210`	moderate	✅ E.164 limits: 7–15 digits, no `+0` country code
`license_plate`	`MH 12 AB 1234`	moderate	— no check digit
`passport`	`J1234567`	moderate	— no check digit
`address`	`123 MG Road, Pune 411001`	heuristic	— not verifiable
`organization`	`Apollo Hospital`, `HDFC Bank`	heuristic	— not verifiable
`medical_term`	`diabetes`, `cardiac arrest`	heuristic	— gazetteer membership only
`person_name`	`Dr. Priya Sharma`, `John Smith`	heuristic	— not verifiable

Rows are listed in execution order, which is load-bearing: rules run in sequence over the same value, so specific detectors must precede greedy ones. ssn and date_of_birth come before phone (which otherwise swallows both), and person_name runs last because its bare-capitalised-pair alternative otherwise claims Apollo Hospital and Cardiac Arrest.

The eight unvalidated ones are honest gaps, not oversights: a passport number and a licence plate carry no check digit, an API key is opaque, and a heuristic is a heuristic. A test asserts the validated set is exactly those eight, so the table cannot drift into claiming verification the code does not perform.

precise — distinctive syntax, and five of the seven are validated. Safe to redact unattended.
moderate — a recognisable shape that innocent data also has; three of the five are validated. Expect some false positives; read the report.
heuristic — a stand-in for named entity recognition, which this tool does not have. These are title-based patterns, suffix lists and a small gazetteer. They produce false positives and miss real entities. They default to detect in presets, every report says so, and setting one to modify data triggers the heuristic-pattern-modifies-data lint. Treat them as "show me where to look", never as "this text is now clean".

Validation

Validated detectors apply their check to every match, so a loose regex buys recall without paying for it in false positives — a long order number matches the card shape but fails Luhn, so it is not reported as a card.

Rejected matches are left untouched and counted, and the report names the check that rejected them:

pattern 'credit_card' rejected 1 match(es) that had the right shape but failed Luhn + card length/prefix validation, and left them unchanged. Set validate: none to treat them as identifiers anyway (at the cost of false positives)

That matters for test data: invented card numbers usually fail Luhn, so validate: none is the right choice when you want every card-shaped string flagged regardless. Override per rule with validate: none (or name a different validator) on any patterns entry.

Validators are deliberately conservative — they reject only what is definitely not the thing. Being too strict produces false negatives, a real identifier passing through silently, which is worse than a false positive a human dismisses. Two consequences worth knowing:

Ambiguous dates are accepted under either reading, because 03/04/1990 is a real date as both DD/MM and MM/DD and guessing wrong would drop a genuine one.
date_of_birth checks only that the date exists, not that it is a plausible birth date. Rejecting future dates would drop appointment dates that a user wants removed.

Two verification opportunities are deliberately left open. Indian licence-plate state codes would be real validation, but would silently narrow the detector to one country and break the European formats its pattern also matches — that belongs in per-locale pattern packs. GitHub tokens carry a CRC32 checksum in their final characters, which is checkable but vendor-specific.

Presets

Rather than listing sixteen rules, enable a whole class:

presets:
  - { preset: precise,   action: redact }   # checksum-verified: act on them
  - { preset: moderate,  action: redact }   # read the report afterwards
  - { preset: heuristic, action: detect }   # report only, for human review

preset: all covers everything. An explicit patterns entry always wins over a preset of the same detector name, so you can enable a class and still tune one member of it. A complete example ships in examples/policies/detect-all.yaml.

Rules run in sequence over the same value, so two rules using the same detector would mean the first one's replacement hides the second's matches — the duplicate-builtin-detector lint catches that.

Key management

Pseudonym tokens are 128-bit BLAKE3 keyed hashes. The key is derived from your secret and the policy's dataset name, and the hash input includes the column name. Consequences:

Same secret + same policy ⇒ same tokens. Runs are repeatable, and repeated exports of the same dataset stay joinable on their tokens.
The same value produces different tokens in different columns and different datasets — tokens can't be used to link across datasets by accident.
Without the secret, tokens cannot be reversed or recomputed. Whoever has the secret can re-identify. Store it in a secret manager, never next to the output.

Provide the secret via the environment variable named in key.env:

export DEIDENT_KEY="$(your-secret-manager get deident-prod)"
deident pseudonymize ...

key.inline embeds the secret in the policy file — useful for demos and tests, unsafe for production. Every run using it records a warning in the report and triggers the inline-key lint.

Resolution is fail-closed

A policy may declare both env and inline. If the named environment variable is unset or empty, the run fails rather than quietly using the inline value:

key:
  env: DEIDENT_KEY
  inline: "dev-only-secret-do-not-use-in-production"

$ deident pseudonymize in.csv --policy above.yaml --out out.csv
error: key error: environment variable 'DEIDENT_KEY' is unset or empty. The policy
also carries an inline key, but falling back to it silently would tokenize
production data under a development secret — export the variable, or set
`key.allow_inline_fallback: true` if that is genuinely what you want

A forgotten export would otherwise produce output that looks correctly pseudonymized but is reversible by anyone holding the policy file, and would not join with earlier exports. Opt in explicitly if you want the old behavior:

key:
  env: DEIDENT_KEY
  inline: "dev-only-secret-do-not-use-in-production"
  allow_inline_fallback: true   # demos and tests only

The fallback then happens and is recorded as a warning in the report — identically whether the job ran in-process or in the sandbox.

Strength floor

Secrets shorter than 32 bytes are rejected. Longer secrets that look like passphrases (few distinct byte values) are accepted with a warning; prefer openssl rand -hex 32.

Key material is purpose-separated: the vault encryption key is derived from the same secret under a different KDF context, so the vault key cannot forge tokens and the token key cannot decrypt a vault.

Mapping vault and reversal

Tokens are recomputable from the key alone, so a vault is optional. It exists for the workflow where an authorized party needs to reverse specific values without holding a re-derivation pipeline — and it is the only way to reverse mocks and pattern matches conveniently.

# 1. pseudonymize, recording the mappings
deident pseudonymize patients.csv --policy p.yaml --out pseudo.csv --vault vault.jsonl

# 2. later, with authorization: inspect the mappings
deident vault export vault.jsonl --policy p.yaml --out mappings.csv

# 3. or reverse a whole file in place
deident reverse pseudo.csv --vault vault.jsonl --policy p.yaml --out restored.csv

How it is protected:

Every entry is encrypted with XChaCha20-Poly1305 under a key derived from your secret and the dataset name. The file's header (format, version, dataset) stays readable; the mappings do not.
Nonces are synthetic — derived from the key and the plaintext rather than randomly. That keeps vault files reproducible and makes appends safe against nonce reuse. The trade-off is that identical entries produce identical ciphertext, revealing that two lines map the same value; for a deterministic mapping table that equality is inherent to the design.
The AEAD tag is verified on read, so a wrong key or a tampered file fails loudly instead of decrypting to garbage.

A vault is written only when the job actually produces reversible values (pseudonymize mode, or token/mock patterns); otherwise the report says so and no file is created.

⚠️ A vault is a re-identification table. It is as sensitive as the original data. Store it separately from the output, under stricter access control, and treat vault export and reverse as privileged operations — their output contains original personal data again.

DICOM

Medical imaging instances are not tabular — a DICOM object is a nested, tag-keyed attribute tree with typed value representations, sequences, a separate file-meta header and a pixel payload. So DICOM gets its own policy dialect and its own command, while reusing the same key derivation, tokenization, mocks, mapping vault and audit log.

# a single instance
deident dicom study/image-001.dcm --policy dicom.yaml --out deid/image-001.dcm

# or a whole directory tree, recursively, with one shared identity scope
deident dicom study/ --policy dicom.yaml --out deid/ --report deid.json --vault vault.jsonl

Scope — read this first

⚠️ This is not DICOM PS3.15 Annex E conformance. It implements a curated core of the Basic Application Level Confidentiality Profile plus structural rules, and every report says so. If you need certified conformance you must extend the policy's tag list and validate it against your own data.

⚠️ Burned-in pixel PHI is detected and flagged, never removed. Ultrasound frames, secondary captures and scanned documents routinely render patient details into the image itself. Cleaning that requires OCR and cannot be made reliable, so this tool refuses to claim it. Every run prints the caveat and reports a pixel_risk level with its reasoning.

How coverage works

Three layers, highest precedence first:

Explicit tags: rules in your policy.
The selected profile (basic — the curated Annex E core).
Structural rules that catch whole classes of attribute rather than named instances: every person-name (PN) attribute, every identity UID, every private attribute (odd group — unknown vendor semantics), and the curve/overlay groups.

That third layer is deliberate. Transcribing ~500 Annex E rows from memory would be error-prone, and a missed row means PHI survives. Rules keyed on VR and tag structure fail safe — they remove what they don't recognise — and the curated table then handles the well-known core exactly.

Actions

Action	Annex E	Effect
`remove`	`X`	Delete the attribute
`empty`	`Z`	Keep the attribute, zero-length
`replace`	`D`	Fixed literal (`value:`)
`pseudonymize`	`D`	Deterministic keyed pseudonym; `mock: person_name` produces a readable `Family^Given` instead of a hex token
`uid`	`U`	New UID, consistently remapped — the same original UID becomes the same replacement in every instance of the study
`date_shift`	—	Shift by a deterministic per-subject offset, so intervals survive
`date_truncate`	—	Truncate to year or year-month (padded to stay a valid `DA`)
`clean_text`	`C`	Run the policy's content-pattern rules over the text
`keep`	`K`	Leave untouched

Tags are addressed by standard keyword (PatientName) or numerically ((0010,0010)). Replacement UIDs use the 2.25.<decimal> arc that DICOM PS3.5 reserves for UUID-derived OIDs, so no registered organisational root is needed.

A complete annotated example ships in examples/policies/dicom-basic.yaml.

What survives, and why

PatientSex and PatientAge are kept by the basic profile because they are clinically load-bearing — but they are quasi-identifiers, and the report says so. Format-identifying UIDs (SOPClassUID, TransferSyntaxUID) are never remapped; doing so would make the file unreadable. Pixel data and image geometry pass through untouched.

UID remapping intentionally breaks references from outside the processed set — a PACS or a report citing the original UIDs will no longer resolve.

Test data

Public DICOM collections (TCIA, pydicom-data, GDCM) are already de-identified, which makes them unable to demonstrate that a de-identifier works — there is no PHI left to remove. So the crate generates its own fixtures with identifiers planted in known attributes, including one nested inside a sequence and one in a private block:

cargo run -p deident-dicom --example gen_fixtures -- ./study 3

The test suite runs against these and asserts at the byte level that no planted identifier survives anywhere in the output file.

DICOM jobs run in-process: the wasm guest does not carry the DICOM parser (the same module-size trade-off as Parquet). Since DICOM parsers are historically a CVE-rich surface, sandboxing this path is on the roadmap.

Policy lints

A policy can be perfectly valid and still not do what its author intended — a quasi-identifier with no generalization, a secret pasted into the file, deny-by-default switched off. deident lint reports those:

deident lint examples/policies/patients.yaml --mode anonymize
deident lint policy.yaml --json          # machine-readable
deident lint policy.yaml --deny          # exit non-zero on any warning

Lints also run automatically before every job (warnings to stderr). Use --no-lint to skip them, or --deny-lints to refuse to run when a warning fires — useful in CI.

Two levels: warning (likely a privacy problem) and advice (legitimate in many setups). Current rules include: inline-key, missing-key-source, unlisted-columns-kept, unlisted-columns-removed, qi-without-strategy, direct-identifier-partially-kept, ineffective-bucket, free-text-without-patterns, no-direct-identifiers, no-quasi-identifiers, detect-only-pattern, reversible-pattern-in-anonymize.

Lints are heuristics, not a compliance check — a clean lint run does not mean a policy is adequate for your data.

Audit log

--audit-log <FILE> appends one JSON object per job:

{"timestamp":"2026-08-04T09:12:33Z","job_id":"…","mode":"anonymize","engine":"wasm",
 "report_provenance":"host-attested","dataset":"patients-demo","policy_hash":"9f2c…",
 "input_path":"in.csv","output_path":"out.csv",
 "status":"succeeded","rows_read":12,"rows_written":12,"warnings":1,"error":null,
 "limits":{"max_memory_bytes":268435456,"timeout_ms":30000,"fuel":2000000000}}

It is deliberately metadata only — no cell values — so it can be retained and shipped to a SIEM without inheriting the sensitivity of the data it describes. Specifically:

policy_hash is a BLAKE3 fingerprint of the policy with the key block removed, so an auditor can prove which policy produced an output and the fingerprint does not commit to an inline secret.
error is capped at 300 characters and flattened to one line. Failure text is assembled from whatever went wrong, so it can quote a policy value or a column name from the input; the log keeps a bounded summary while the operator still sees the full message on stderr.
report_provenance records who authored the risk figures — host-attested means the host computed or verified them. A compromised worker could otherwise report clean counts over untransformed data, and a consumer needs to know which it is holding.

Records are written for failed jobs too, and it works identically for native, sandboxed, split and chained runs.

Chained datasets

Real exports are rarely one file: patients.csv plus visits.csv that references it. A chain manifest runs them as one unit so foreign keys survive pseudonymization:

# hospital.yaml — paths are resolved relative to this file
version: 1
name: hospital-demo
# Optional overrides forced onto every job policy:
# dataset: hospital-export    # one token scope for all files
# key: { env: DEIDENT_KEY }   # one key source for all files
jobs:
  - name: patients
    input: ../data/patients.csv
    policy: ../policies/patients.yaml
    output: out/patients.csv
    report: out/patients-report.json   # optional per-job report
  - name: visits
    input: ../data/visits.csv
    policy: ../policies/visits.yaml
    output: out/visits.csv
    vault: out/visits-vault.jsonl      # optional per-job vault

deident chain hospital.yaml --mode pseudonymize --report out/chain-report.json

Cross-file linkage needs two things:

Same token scope — all policies share the same dataset (and secret), or the manifest forces one via its dataset:/key: overrides. Diverging scopes in pseudonymize mode are flagged as a chain warning, because they silently break joins.
Same identity domain — tokens are namespaced by column name by default, so patient_id (patients.csv) and patient_ref (visits.csv) would not match. Declare the shared domain on the referencing column:
```
- name: patient_ref
  class: direct_identifier
  pseudonymize:
    prefix: "pid_"
    domain: patient_id    # ← same namespace as patients.csv's patient_id
```

Jobs run sequentially and the chain stops at the first failure (remaining jobs are not run; the combined report says so). Exit code is non-zero unless every job succeeded. --engine wasm gives each job of the chain its own fresh sandbox. A complete working example ships in examples/chains/hospital.yaml.

Risk report

--report <FILE> writes a JSON document (also available for pseudonymize):

{
  "dataset": "patients-demo",
  "mode": "anonymize",
  "rows_read": 12,
  "rows_written": 12,
  "direct_identifiers": [
    { "field": "patient_id", "action": "removed" },
    { "field": "full_name", "action": "removed" },
    { "field": "email", "action": "removed" }
  ],
  "quasi_identifiers": {
    "fields": ["age", "zip", "admission_date"],
    "equivalence_classes": 5,
    "min_class_size": 1,
    "max_class_size": 5,
    "mean_class_size": 2.4,
    "unique_rows": 1,
    "unique_row_ratio": 0.0833,
    "k_thresholds": [
      { "k": 2, "rows_at_or_above": 11, "ratio": 0.9167 },
      { "k": 5, "rows_at_or_above": 5, "ratio": 0.4167 },
      { "k": 10, "rows_at_or_above": 0, "ratio": 0.0 }
    ]
  },
  "patterns": [
    { "pattern": "iban", "field": "notes", "matches": 1, "action": "redacted" }
  ],
  "warnings": [],
  "limitations": [ "This report supports a risk assessment; it does not certify or guarantee anonymization.", "..." ]
}

patterns lists content-pattern matches per rule and column with the action taken (detected / redacted / tokenized). deident chain --report writes a combined chain report instead: chain name, completion flag, chain-level warnings and each job's outcome with its embedded RiskReport.

How to read the quasi_identifiers block: rows are grouped by their combination of (transformed) quasi-identifier values — each distinct combination is an equivalence class. Small classes mean higher re-identification risk:

min_class_size — the k in "k-anonymity style" terms; 1 means at least one row is unique on its quasi-identifiers.
unique_rows / unique_row_ratio — rows that are one-of-a-kind. These are the riskiest rows; consider coarser generalization if this isn't near zero.
k_thresholds — share of rows living in classes of at least size k (2, 5, 10).

warnings surfaces anything that needs human attention: inline key usage, quasi-identifiers without a strategy, suppressed values, unlisted-but-kept columns. The limitations block is embedded in every report by design.

Security model & non-goals

Anonymization here is risk-assessed, never guaranteed. The report measures what it can; residual risk always remains and depends on context.
Pseudonymized data remains personal data under most privacy regimes (e.g. GDPR). Reversal requires only the key material — protect it separately.
Deny-by-default policy handling: unlisted columns and unknown policy keys fail the job unless explicitly relaxed.
With --engine wasm, each job runs in a fresh WebAssembly sandbox with a preopened job directory as its only filesystem capability, no network, and per-job memory/time limits (see Sandboxed execution). Sandboxing reduces the blast radius of risky parsing logic and future untrusted plugins; it is a mitigation, not an absolute boundary, and no escape-proof claims are made.
The mapping vault is re-identification material, encrypted at rest but as sensitive as the source data. vault export and reverse are privileged operations that reproduce personal data.
token and mock pattern actions produce pseudonymous, not anonymous values, even in anonymize mode. Mocks additionally look real, which is the point and also the hazard — the report and lints call this out.
The audit log is metadata-only by design; it records what happened, never the data it happened to.
Risk figures returned from a sandboxed job are host-attested: the host re-derives what it owns and verifies what it can cheaply check, because a compromised guest could otherwise report clean counts over untransformed data. The report_provenance field states which regime produced a given report.
Secrets must be at least 32 bytes, and a policy declaring both env and inline fails closed when the variable is unset — see Key management.
A job that fails partway publishes neither its output nor its vault, so a truncated artifact cannot be mistaken for a complete one.
Policy lints are heuristics that catch common mistakes. A clean lint run is not a compliance statement.
Non-goals: differential privacy, synthetic data generation, free-text/NLP de-identification, and legal certification of any output.

Example datasets

The repo ships a 12-row demo (examples/data/patients.csv) for reading at a glance, and a generator for datasets large enough that the statistics mean something:

cargo run -p deident-core --example gen_dataset -- examples/data 1000

deident anonymize examples/data/clinic-patients.csv \
  --policy examples/policies/clinic.yaml --out anon.csv --report risk.json

File	Contents
`clinic-patients.csv`	direct identifiers, quasi-identifiers, free text carrying every entity type the detectors know
`clinic-visits.csv`	foreign key into patients under a different column name — for chained runs and `pseudonymize.domain`
`clinic-labs.jsonl`	a JSONL table with a zero-padded code and a real float, so the JSONL path is exercised
`clinic-messy.csv`	the same shape with real-world damage (see below)

Two things make this more useful than simply being big:

The quasi-identifier distribution is engineered. Ages, ZIPs and dates are drawn so most rows land in large equivalence classes while a deliberate minority are unique. On 1,000 rows that yields ~300 classes with ~16% unique rows — a number you can reason about. Uniformly random data would make every row unique, which makes the report look alarming and teaches nothing.

clinic-messy.csv contains what real exports actually contain: a UTF-8 BOM and mixed-case headers (both silently make an exact-match policy field inert), empty cells, dates in 14.03.2024 order that no ISO parser accepts, card-shaped numbers that fail Luhn, zero-padded identifiers that naive type inference corrupts, 1.2.3.4 version strings that look like IPv4, and non-ASCII names. Point a policy at it to see how the tool behaves when the input misbehaves.

Output is deterministic — fixed seed, counter-based PRNG — so regenerating gives byte-identical files and tests stay reproducible. Pass a larger row count to scale up; nothing about the generator is limited to 1,000.

These are synthetic records with deliberately planted identifiers. Do not mix them with real data.

Versioning

Current version: 0.2.0. See CHANGELOG.md for what changed.

Four compatibility surfaces move independently, and the crate version is the least consequential of them:

Surface	Where	Breaking means
Crate version	`Cargo.toml`	Rust API changes
Policy schema	`version:` in a policy	An existing policy stops loading
Vault format	vault header `version`	An existing vault stops decrypting
Token derivation	not yet versioned	Every previously issued token changes value

The last row is the one to watch. Tokens are a keyed hash of a domain and a value, so any change to the hash input produces different tokens for the same input — joins against earlier exports break and nothing errors. Changing token derivation therefore requires a major version bump and a migration note, even if the Rust API is untouched.

Every report and audit record carries tool_version, so an artifact can be traced to the build that produced it. That matters because detection patterns and default profiles change between versions: "no identifiers found" only means something alongside the version that looked.

Project layout

Crate	Purpose
`crates/cli`	`deident` binary — command-line UX
`crates/core`	Policy schema, transforms, job engine, risk reports
`crates/host`	Execution engines: in-process native and per-job Wasmtime sandbox
`crates/worker`	Wasm guest that executes one job inside its sandbox
`crates/dicom`	DICOM policy, profile and de-identification engine
`crates/types`	Shared request/response/report models

cargo test --workspace        # unit + integration tests (includes the feature matrix)
cargo clippy --workspace --all-targets
cargo test -p deident-cli --test matrix   # just the feature-combination matrix

CI (GitHub Actions) — two workflows with no overlapping work:

rust.yml — build, clippy (-D warnings) and the test suite on every push/PR to main. It skips the feature matrix, which the second workflow owns.
feature-matrix.yml — every mode × engine × single/chain combination against the sample dataset, triggered by changes under examples/ or crates/ (plus a manual "Run workflow" button). The matrix test recomputes its expectations from the data itself — determinism, native/wasm byte-parity, identifier survival, pattern counts, chain linkage — so editing the sample dataset automatically re-validates every feature against it. The full-feature policy it uses is examples/policies/patients-full.yaml.

Note that CI runs the latest stable Rust, which may lint more strictly than an older local toolchain; run clippy with -D warnings locally to match it.

Roadmap

Everything on the original roadmap is now implemented. What's next, roughly in order of value:

Streaming at scale — Parquet and the equivalence-class statistics hold data in memory. --split divides a dataset across sandboxes and lowers peak memory per job, but chunked row-group processing and a spill-to-disk class map would lift the ceiling properly.
Split with a vault — merging per-chunk encrypted mapping files, so --split and --vault can be combined.
Parquet in the sandbox — currently excluded to keep the guest module small; it is also what stops --split from accepting Parquet.
Broader DICOM coverage — extend the tag table toward full Annex E, add the profile options (Retain Longitudinal Temporal, Retain Patient Characteristics, Retain Safe Private), and sandbox the DICOM parser.
Burned-in pixel detection — OCR-assisted flagging of PHI rendered into image pixels. Detection only; cleaning would remain a claim we refuse to make.
Ship the worker with releases — embed or bundle deident-worker.wasm next to the binary so auto always sandboxes instead of falling back.
k-anonymity enforcement — today the report measures small equivalence classes; a min_class_size: k policy option could suppress or coarsen rows until the threshold is met, and fail the job if it cannot be.
Richer pattern library — national ID formats, addresses, dates in free text, and a --dry-run scan mode that reports findings without writing output.
Vault key rotation and re-tokenization — re-derive tokens under a new secret while preserving joins, using the vault as the bridge.
Column-level pattern strategies per class — e.g. apply a pattern set to every utility column automatically instead of naming columns.
Policy authoring help — deident init <input> to scaffold a policy from a dataset's header with class guesses from column names and content sniffing.
Differential-privacy noise for aggregates — out of scope for row-level output, but useful if the tool grows a summary-export mode.

Name		Name	Last commit message	Last commit date
Latest commit History 30 Commits
.github/workflows		.github/workflows
crates		crates
examples		examples
.gitignore		.gitignore
CHANGELOG.md		CHANGELOG.md
Cargo.lock		Cargo.lock
Cargo.toml		Cargo.toml
LICENSE		LICENSE
README.md		README.md
SECURITY_AUDIT.md		SECURITY_AUDIT.md
anon.csv		anon.csv

Folders and files

Latest commit

History

Repository files navigation