Skip to content

Consume the immutable INT03 legacy market manifest - #87

Merged
hjcud merged 1 commit into
developfrom
feature/86-int03-legacy-market-manifest
Aug 9, 2026
Merged

Consume the immutable INT03 legacy market manifest#87
hjcud merged 1 commit into
developfrom
feature/86-int03-legacy-market-manifest

Conversation

@hjcud

@hjcud hjcud commented Aug 9, 2026

Copy link
Copy Markdown
Contributor

Closes #86.

What changed

  • Add a schema-gated market-loader/1.0.0 adapter that validates the exact Development market-bars/1 manifest shape and recomputes its original producer hash.
  • Preserve the canonical market-data.v1 path unchanged; only an explicit market-bars/1 manifest can select the legacy adapter.
  • Resolve and require the catalog's immutable S3 bucket and provider_version_id, and read every object with the exact S3 VersionId.
  • Verify object bytes, Parquet footer schema, row counts, ordering, session dates, period, shard layout, and storage evidence before replay.
  • Allow an execution policy to publish a complete local-calendar-day window independent of its release-quarter selection key, enabling the honest one-month Development policy instead of pretending the January dataset covers ten years.
  • Add the sanitized exact INT03 Development manifest as a regression fixture, including all eight shards and the pinned dataset hash.

Root cause

Development contains one immutable adjusted January 2024 publication produced by the legacy loader. The worker reconstructed it as the canonical manifest shape, recomputed the wrong hash formula, discarded the S3 VersionIds, and only permitted quarter-aligned policy windows. As a result, no policy could truthfully bind the existing dataset and a replay could not prove it read the locked object versions.

Safety

The legacy route is fail-closed and recognizes only market-bars/1, Alpaca SIP adjusted data, the loader's exact object-key grammar, a complete shard set, immutable S3 version evidence, and the original dataset hash. Any mutation in row count, shard binding, object kind, data layer, bucket, version, bytes, or Parquet metadata is rejected. The current canonical validator is still used for every non-legacy schema.

Validation

  • Docker-free suite: 1,262 passed, 3 skipped, 168 deselected (plus focused tests added after the full run)
  • Focused INT03/production/policy suite: 116 passed
  • Docker integration: 158 passed, 10 optional service-dependent skips
  • ruff check src tests: passed
  • mypy: passed
  • pip-audit --progress-spinner off --desc: no known vulnerabilities
  • git diff --check: passed

No Development database write, S3 write, run creation, queue operation, worker restart, ASG change, root pointer change, or deployment was performed by this PR.

Related root tracking: Idea2Strategy/Idea2Strategy#471.

@hjcud
hjcud marked this pull request as ready for review August 9, 2026 09:38
@hjcud
hjcud merged commit b8efbc4 into develop Aug 9, 2026
9 checks passed
@hjcud
hjcud deleted the feature/86-int03-legacy-market-manifest branch August 9, 2026 09:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

INT03: consume the legacy market manifest by exact immutable S3 version

1 participant