Status: alpha / experimental — not yet validated against large real-world PST datasets.
PSTforge parses Microsoft Purview / Outlook .pst email archives into structured JSON at enterprise scale (100k+ emails), with optional OCR of attachments via Mistral OCR.
PST archives are difficult to process reliably at scale. They contain nested folders, duplicate messages, inconsistent metadata, corrupt records, and attachments that may require text extraction or OCR. Existing workflows often depend on Outlook, lose occurrence-level provenance, or cannot safely resume after processing failures.
PSTforge turns those archives into deterministic, machine-readable JSON while preserving where each message occurred. It adds crash-safe checkpointing, content deduplication, attachment extraction, optional OCR, hostile-file protections, and auditable reports so PST data can be used in downstream search, migration, compliance, e-discovery, and analytics systems.
-
E-discovery and legal review pipelines
-
Microsoft 365 and legacy email migrations
-
Compliance, retention, and audit processing
-
Search and retrieval indexing for archived email
-
Email analytics and structured dataset generation
-
Offline extraction of PST content, with optional cloud OCR when explicitly enabled
-
Two-pass, deterministic deduplication — every occurrence is written; one canonical copy per unique message content, chosen deterministically regardless of processing order.
-
Crash-safe — atomic writes, SQLite checkpointing,
--resume. -
Hostile-input hardened — magic-byte routing, zip-bomb/traversal limits, no macro execution, per-field corruption tolerance.
-
Cost-guarded OCR — content-hash cache, image-size gates, per-run request/page/cost limits,
--dry-runestimates.
When ocr.provider: mistral is enabled, attachment contents are uploaded to Mistral's cloud API for OCR. The default is ocr.provider: none (fully offline; scannable items are marked status: skipped, reason: ocr_disabled). Review docs/privacy.md and your data-handling obligations before enabling cloud OCR.
PSTforge depends on libpff (LGPL-3.0+) and its Python binding libpff-python — a native dependency with no Windows wheels.
Do NOT
pip install pypff— that is an unrelated astronomy package.
Recommended (all platforms, required on Windows): Docker
docker build -t pstforge .
docker run --rm -v /path/to/psts:/input:ro -v ./output:/output pstforge \
pstforge run --config /input/config.yamlThe image builds a pinned libpff-python release with a small patch (docker/pypff-recipients.patch) that exposes the MAPI recipients table — stock pypff cannot read To/Cc/Bcc.
Linux / macOS native: build libpff-python from source (needs a C toolchain), then pip install pstforge. See docs/troubleshooting.md.
| Python | PST parsing | |
|---|---|---|
| Linux | 3.10–3.12 | native build or Docker |
| macOS | 3.10–3.12 | native build or Docker |
| Windows | 3.10–3.12 | Docker recommended |
Unit tests and all non-PST functionality run everywhere without libpff.
cp config.example.yaml config.yaml
cp .env.example .env # add MISTRAL_API_KEY only if enabling OCR
pstforge validate --input ./exports/case001.pst
pstforge run --config config.yaml --dry-run # projected volume & OCR cost, no writes
pstforge run --config config.yamlOutput layout:
output/
├── ab/{email_id}.json # one JSON per email occurrence, sharded by id prefix
├── manifest.jsonl # one line per occurrence
└── pipeline_report.json # counts, skip reasons, errors, cost — always written
| Code | Meaning |
|---|---|
| 0 | success |
| 1 | partial (some emails/attachments skipped — see report) |
| 2 | configuration error |
| 3 | invalid, corrupt, locked, or password-protected PST |
| 4 | checkpoint exists and --resume not given |
| 130 | interrupted (report still written) |
Identity is two-tier: email_id identifies an occurrence (this message, in this folder, in this PST); content_id identifies the content (same Message-ID/sender/recipients/subject/body). Every occurrence is written as its own JSON. Exactly one occurrence per content_id is canonical (is_duplicate: false); the rest reference it via duplicate_of and reuse its attachment results. Canonical selection is resolved deterministically between Pass 1 (index) and Pass 2 (materialize) — see docs/architecture.md.
MIT. Note: libpff itself is LGPL-3.0+ and is used as an external runtime dependency — its binaries are not bundled in the PyPI wheel.