feat(pxe): constrained tag sync optimization and recipient logs sync benchmarks #24275
Open
vezenovm wants to merge 29 commits into
Open
feat(pxe): constrained tag sync optimization and recipient logs sync benchmarks #24275vezenovm wants to merge 29 commits into
vezenovm wants to merge 29 commits into
Conversation
Constrained delivery emits a gapless tagging-index stream, so the recipient can stop at the first missing tag instead of always probing a full UNFINALIZED_TAGGING_INDEXES_WINDOW_LEN (20) window per secret. The initial probe is INITIAL_CONSTRAINED_PROBE_LEN tags and grows to the full window only when every probed index has a log; probe advancement is decoupled from finalized-cursor persistence so unfinalized logs at the top of the run are still fetched. Adds a unit micro-benchmark for the per-sync tag-query cost.
The mocked node makes sync-time an unfaithful end-to-end latency; real full-sync timing already lives in the client_flows /sync e2e bench (ProvingTimings.sync). Keeps the metrics the micro-bench is for: tag-queries, round-trips, and blocking-time.
Add a realistic mixed-population row (999 idle + 1 deep straggler at K=100) and run deep catch-up at 1000 secrets as well as 100, so every value in the PR-description tables is produced by the committed bench. Generalize scenario seeding and assertions to a per-secret new-log distribution and compute firstMissOptimum from it.
The constrained tag-sync source restated "gapless stream stops at the first missing tag" ~5x and the doubling-probe mechanics ~4x. Keep one canonical home for each (the module JSDoc for the why, the nextProbeLen comment for the mechanics) and reduce the rest to one-line local facts; also correct boundEnd's comment to note it is an exclusive bound and probeLen's to note the queried span can be smaller when boundEnd caps it. Drop the tag-sync micro-bench from bench_cmds: it existed to find the probe-growth numbers locally, which is done, so we no longer track it on the benchmark dashboard. It stays runnable as a local tool, and the doubling behavior remains covered in CI by the unit-test file.
…' into mv/optimize-constrained-tag-sync
Revert the in-place strengthening of the existing unconstrained and mixed tests and re-express the constrained-specific coverage as three new tests in the constrained-secrets block: multi-secret batching, halting at the first all-miss batch (sentinel proof), and doubling-probe re-anchoring past the cold-start bound.
The constrained-delivery optimization rewrote the old "continues when all indexes in the batch have logs" test (which scanned the full window every round) into the constrained doubling-probe test, leaving the unconstrained full-window-scan drain of a run longer than one window without direct coverage. Add an unconstrained test pinning that a contiguous run filling and exceeding the cold-start window drains fully: all logs returned, both cursors advanced to the last index, and per-round query sizes of [WINDOW_LEN + 1, WINDOW_LEN] showing full-window re-anchoring rather than a doubling probe.
vezenovm
commented
Jun 26, 2026
vezenovm
commented
Jun 26, 2026
The constrained-sync micro-benchmark is skipped in CI (test_cmds skips *.bench.test.ts and it is not in bench_cmds), so bring its coverage into sync_tagged_private_logs.test.ts, which runs in CI: - log-then-linear round-count law (K=1,3,20,50,100 -> 2,3,5,6,9 rounds) - batched straggler: a single deep straggler among idle secrets sets the round count while idle secrets each cost one probe and drop - cold-start constrained-vs-unconstrained scan-shape contrast - tighten the deep-cap test to pin the linear-past-cap regime Qualify the overstated ~log2(K) catch-up claim in constants.ts, sync_tagged_private_logs.ts, and the bench: catch-up is logarithmic only while the probe doubles, then linear at ~K/WINDOW_LEN once it saturates the cap. No behavior change.
Merge the constrained drain test into the boundEnd re-anchor test, move the unconstrained drain next to its constrained counterpart, and drop the steady-state full-window test subsumed by the store-index test. Strengthen the unfinalized-probe test with a second-sync restart assertion and make the probe reset test saturate the WINDOW_LEN cap first. Remove the bench's probe-schedule mirror (behavior is pinned by the CI unit tests) and switch remaining "cursor" wording to "finalized index".
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes F-704
Summary
Constrained delivery emits a gapless tagging-index stream, so PXE can stop scanning at the first missing constrained tag instead of probing the full unfinalized window on every sync.
This PR:
findHighestIndexesunchanged.Benchmarks
Benchmarks compare doubling against fixed-size probes. The takeaway is that doubling preserves the steady-state tag-query floor of
P=1while avoidingP=1's one-round-trip-per-log catch-up behavior.A sync runs in rounds: each round computes the next batch of tags, sends them to the node, and blocks on the result before deciding the next round. All counts are per sync.
Scenarios seed a recipient that already synced prior messages, then measure the next single sync.
secrets = Nmeans N independent sender streams synced together in one batched pass.K=100) at 1,000 secrets.Cost per sync: doubling vs fixed-P alternatives
doubling is the shipped policy. The fixed-P columns are the selection comparison that motivated it:
P=84is the current full-window behavior, andP=1..5sweep the constant-step alternative.Unconstrained is not shown as its own row: its windowed scan cannot first-miss, so it is invariant to P and reproduces the
P=84column.Tag-queries (thousands) (exact count = value x 1,000; bold = fewest among the fixed-P columns; the
doublingcolumn is the shipped policy):Round-trips (bold = fewest among the fixed-P columns; the
doublingcolumn is the shipped policy):Blocking wall-clock (ms) (reported only, noisy; the
doublingcolumn is from the committed bench, the fixed-P columns from the comparison sweep, so they are not from a single run):Takeaway
P=1is the tag-query floor, but it pays one round-trip per new log during catch-up. The full window (P=84) minimizes catch-up round-trips, but it charges every idle secret the full-window tag cost on every sync.Doubling is the middle ground: it matches
P=1at steady state, stays close toP=1on tag queries during catch-up, and collapses deep catch-up round-trips geometrically. In the mixed scenario, doubling uses 1,126 tag queries vs 1,100 forP=1, but needs 7 round-trips instead of 101; compared withP=84, it avoids the 84,084-tag idle tax while staying within 5 round-trips of the full-window catch-up path.Testing