Skip to content

fix(data): harden streaming dataset installation#4

Merged
xr843 merged 14 commits into
masterfrom
agent/streaming-data-download
Jul 11, 2026
Merged

fix(data): harden streaming dataset installation#4
xr843 merged 14 commits into
masterfrom
agent/streaming-data-download

Conversation

@xr843

@xr843 xr843 commented Jul 11, 2026

Copy link
Copy Markdown
Owner

What changed

  • Stream the raw gzip asset to bounded on-disk staging while hashing the exact response bytes, then stream decompression into a separate bounded SQLite candidate.
  • Validate schema, compatibility metadata, SQLite integrity, and FTS integrity before atomic publication.
  • Serialize install, update, and clean across processes with a persistent OS file lock and a lock-after-wait existence recheck.
  • Enforce absolute Global and Connect deadlines plus per-read idle limits across DNS, redirects, HTTP CONNECT, TLS, nested target TLS through HTTPS proxies, headers, bodies, and chunk framing.
  • Restrict temporary artifact ownership and cleanup to strict generated names.
  • Preserve the old live database through Windows ReplaceFileW failures with unique backups and rollback-safe recovery.
  • Document transfer limits, staging disk requirements, timeout behavior, and concurrency semantics.

Why

The previous download path could buffer large payloads, duplicate concurrent work, and had weaker failure behavior around timeouts and Windows replacement. This change bounds memory and disk use, makes concurrent mutation single-flight, and keeps validated live data recoverable on failure.

Validation

  • Rust 1.95 fmt and Clippy with warnings denied
  • 148 Rust tests across all targets and features
  • Release build and cargo package verification
  • Python parity test
  • Release and installer contract scripts
  • actionlint 1.7.7
  • Focused timeout, TLS proxy, Windows rollback-state, artifact cleanup, and cross-process barrier regressions

ShellCheck and native Windows ReplaceFileW/locking paths are intentionally gated by the GitHub Actions jobs before merge. No tag or GitHub Release is created by this PR.

@xr843 xr843 merged commit 2fd9cc4 into master Jul 11, 2026
19 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant