Skip to content

INT03: consume the legacy market manifest by exact immutable S3 version #86

Description

@hjcud

Production evidence

The exact Development manifest 7f7113c9-3b02-4098-97ec-0baa07e2b3b0 is AVAILABLE, byte-verified by all eight immutable S3 VersionIds, and uses the legacy market-loader/1.0.0 publication format:

  • manifest/object schema market-bars/1
  • catalog object kind MARKET_BARS
  • partition granularity YEAR
  • object keys bind manifest_id=
  • declared/locked dataset hash matches the legacy loader formula
  • the Parquet columns and types match the current backtest reader exactly

The current consumer still rejects this input because its current-only validator requires the canonical object metadata hash, object kind PARQUET, and DAY/WEEK/MONTH granularity. PostgresDatasetManifestSource also discards bucket/provider VersionId evidence, and S3ParquetMarketDataReader calls get_object without VersionId.

Required fix

  1. Add an explicit, schema-gated legacy market-loader/1.0.0 adapter. Do not weaken or reinterpret the current canonical market-data.v1 path.
  2. Recompute and compare the exact legacy manifest hash from feed/adjustment/resolution/period/revision/schema/processing version and ordered object evidence. Fail closed for any mismatch, mixed convention, wrong manifest UUID, unknown schema, or changed metadata.
  3. Accept MARKET_BARS and YEAR only inside that exact legacy adapter. The execution policy must independently pin market-bars/1 and exactly match the manifest period.
  4. Carry storage provider, bucket, and provider_version_id from PostgreSQL to the S3 reader and request the exact VersionId. Reject missing immutable version evidence for official S3 inputs.
  5. Verify content hash, Parquet footer schema metadata, row counts including empty shards, ordering, and period/session constraints before simulation.
  6. Add the sanitized exact eight-shard Development fixture metadata and production-shaped regressions, plus canonical-path non-regressions and tamper cases.
  7. Keep deterministic incompatibilities terminal/non-retryable.

Root coordination: Idea2Strategy/Idea2Strategy#471. The producer must also publish a coherent new execution policy whose period and schema match the selected manifest; merged PR #84 correctly rejects the current mismatched tuple before RUNNING.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions