fix(test): acknowledge sqlite3 statements instead of sleeping on them - #11
Merged
Conversation
The two gold-oracle fixtures drove a long-lived `sqlite3` reader process while
separate short-lived writers mutated the same database, and synchronised the
two with `sleep(400ms)`. That is a guess about how fast the host is, not a
guarantee: `CREATE TABLE t` executes asynchronously in the reader process, and
when a writer wins the race it fails with
sqlite3 writer failed: Error: in prepare, no such table: t
which is what CI hit -- a failure with nothing to do with the code under test,
appearing only on slower or busier machines.
HeldReader::run appends a unique sentinel to each statement and blocks until it
appears on the reader's stdout. sqlite3 executes a script strictly in order on
one connection, so the sentinel cannot print before the preceding statements
have run: the acknowledgement follows from ordering, not from timing. A 30s
timeout bounds it and panics with the offending SQL rather than hanging.
Verified in this order:
1. reproduced deterministically -- sleeps set to 0 fail 3/3 with the CI error
2. handshake applied, sleeps removed entirely -- green
3. 15 consecutive runs -- 15 pass, 0 fail
4. control A: delete the CREATE TABLE -> FAILS loudly, `no such table: t`
5. control B: sentinel that can never match -> panics in 3.02s, bounded, no
hang
6. restored -> green
Both controls asserted their target text was present before mutating, so
neither could silently no-op and report a green that tested nothing.
Applied to wal_snapshot_oracle.rs as well, which carried the identical pattern
-- a correction that does not reach its copies is barely a correction. Both now
share core/tests/common/mod.rs.
Side effect worth noting: row_history_oracle 0.83s -> 0.04s and
wal_snapshot_oracle ~0.9s -> 0.07s. The 800ms per fixture was pure waiting.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
row_history_oraclefailed on CI withthen passed on re-run of the same commit, with no changes. Flaky by
construction, not by luck.
The race
Both gold-oracle fixtures hold a long-lived
sqlite3reader open (so the-walsidecar is retained) while separate short-lived writer processes mutatethe same database. Statements sent to the reader execute asynchronously in
that other process, and the fixture synchronised against them with wall
clock:
400 ms is a guess about how fast the host is. Lose the race and the writer
prepares against a table that does not exist yet.
The fix
HeldReader::runappends a unique sentinel to each statement and blocks untilit appears on the reader's stdout.
sqlite3executes a script strictly inorder on one connection, so the sentinel cannot print before the preceding
statements have run — the acknowledgement is a consequence of ordering, not of
timing. A 30 s timeout bounds it and panics with the offending SQL rather than
hanging forever. The sleeps are removed entirely, so the handshake is provably
what does the work.
Verification
CREATE TABLEno such table: tControl B matters as much as A: a handshake that hung on failure would trade a
flaky red for a 6-hour job timeout. Both mutations asserted their target text
was present before applying, so neither could silently no-op and report a green
that tested nothing.
Scope
wal_snapshot_oracle.rscarried the identical pattern and is fixed too — acorrection that does not reach its copies is barely a correction. Both files
now share
core/tests/common/mod.rs.Full suite: 542 passed, 0 failed (
--no-fail-fast),cargo fmt --checkclean,
clippy --all-targets -D warningsclean.Incidental:
row_history_oracle0.83 s → 0.04 s,wal_snapshot_oracle~0.9 s → 0.07 s. The 800 ms per fixture was pure waiting.
Deliberately separate from #10 (SQLCipher), which does not touch these files.