Skip to content

Fix deterministic ordering for multi-instrument market shards - #89

Merged
hjcud merged 1 commit into
developfrom
feature/88-market-shard-order
Aug 9, 2026
Merged

Fix deterministic ordering for multi-instrument market shards#89
hjcud merged 1 commit into
developfrom
feature/88-market-shard-order

Conversation

@hjcud

@hjcud hjcud commented Aug 9, 2026

Copy link
Copy Markdown
Contributor

Closes #88.

Root cause

The Development manifest is valid producer output: 8 immutable shards, 7 empty, and shard 04 contains 546 rows for 2 instruments ordered by (instrument_id, bar_start_at). The consumer incorrectly required timestamps alone to increase across every row and object, so the timestamp reset at the second instrument was rejected as INPUT_DATASET_UNREADABLE.

Fix

  • verify all object hashes, row counts, schemas, and logical cross-object schema compatibility before yielding any row;
  • allow either supported deterministic layout per object: (instrument_id, bar_start_at) or the existing (bar_start_at, instrument_id) layout;
  • reject duplicates or a stream that ceases to be monotonic in both layouts;
  • reset order proof at each immutable object boundary;
  • preserve bounded Parquet batch decoding and event-clock global market-time ordering;
  • pin the exact INT03 shape through iter_batches -> bar_events_from_batches -> MarketEventClock, including shard-local schema metadata and empty shards.

Verification

  • full Docker-free pytest suite: passed;
  • full Docker integration suite: passed; the 10 SQS tests skipped without a fixed service endpoint were rerun against LocalStack 4.7.0 and all 10 passed;
  • python -m mypy: passed;
  • python -m ruff check src tests: passed;
  • exact read-only Development shadow replay, proposed source SHA-256 7a907241eddea0c24742470e8ee99753dea83f19cb2ad992cccad34c78134c60:
    • SSM ecceb57f-b345-4a58-a96d-ea827ed6d115: COMPLETED;
    • SSM 2f4dbde9-2c35-4c36-97ba-4ca29dfe3ab5: COMPLETED;
    • both used manifest 7f7113c9-3b02-4098-97ec-0baa07e2b3b0, policy development-official-backtest-2026-q3-v2, all 8 exact S3 versions, 546 rows, 273 evaluation steps, and produced replay digest a09631d8037d774297dd0da75223e3b8774e1c5d3e6426b418d7ecc227121499;
    • zero result, database, queue, or object-store writes.

The failed public run and all existing DLQ messages remain untouched. This PR intentionally does not update the root submodule pointer or deploy Development.

@hjcud
hjcud marked this pull request as ready for review August 9, 2026 12:55
@hjcud
hjcud merged commit 9717bbe into develop Aug 9, 2026
9 checks passed
@hjcud
hjcud deleted the feature/88-market-shard-order branch August 9, 2026 12:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Fix deterministic ordering across immutable market-data shards

2 participants