Skip to content

fix(test): acknowledge sqlite3 statements instead of sleeping on them - #11

Merged
h4x0r merged 1 commit into
mainfrom
fix/oracle-test-race
Aug 9, 2026
Merged

fix(test): acknowledge sqlite3 statements instead of sleeping on them#11
h4x0r merged 1 commit into
mainfrom
fix/oracle-test-race

Conversation

@h4x0r

@h4x0r h4x0r commented Aug 9, 2026

Copy link
Copy Markdown
Contributor

row_history_oracle failed on CI with

panicked at core/tests/row_history_oracle.rs:48:5:
sqlite3 writer failed: Error: in prepare, no such table: t

then passed on re-run of the same commit, with no changes. Flaky by
construction, not by luck.

The race

Both gold-oracle fixtures hold a long-lived sqlite3 reader open (so the
-wal sidecar is retained) while separate short-lived writer processes mutate
the same database. Statements sent to the reader execute asynchronously in
that other process
, and the fixture synchronised against them with wall
clock:

writeln!(rin, "PRAGMA journal_mode=WAL; … CREATE TABLE t(…);")
rin.flush()
std::thread::sleep(Duration::from_millis(400));   // the entire guaranteewriter_sql(bin, &db, "INSERT INTO t VALUES(1,'a'); …");

400 ms is a guess about how fast the host is. Lose the race and the writer
prepares against a table that does not exist yet.

The fix

HeldReader::run appends a unique sentinel to each statement and blocks until
it appears on the reader's stdout. sqlite3 executes a script strictly in
order on one connection, so the sentinel cannot print before the preceding
statements have run — the acknowledgement is a consequence of ordering, not of
timing. A 30 s timeout bounds it and panics with the offending SQL rather than
hanging forever. The sleeps are removed entirely, so the handshake is provably
what does the work.

Verification

Step Result
Reproduce deterministically (sleeps → 0) 3/3 FAILED, the exact CI error
Handshake applied, sleeps deleted green
15 consecutive runs 15 pass / 0 fail
Control A — remove the CREATE TABLE FAILED loudly, no such table: t
Control B — sentinel that can never match FAILED in 3.02 s, bounded, no hang
Restore green

Control B matters as much as A: a handshake that hung on failure would trade a
flaky red for a 6-hour job timeout. Both mutations asserted their target text
was present before applying, so neither could silently no-op and report a green
that tested nothing.

Scope

wal_snapshot_oracle.rs carried the identical pattern and is fixed too — a
correction that does not reach its copies is barely a correction. Both files
now share core/tests/common/mod.rs.

Full suite: 542 passed, 0 failed (--no-fail-fast), cargo fmt --check
clean, clippy --all-targets -D warnings clean.

Incidental: row_history_oracle 0.83 s → 0.04 s, wal_snapshot_oracle
~0.9 s → 0.07 s. The 800 ms per fixture was pure waiting.

Deliberately separate from #10 (SQLCipher), which does not touch these files.

The two gold-oracle fixtures drove a long-lived `sqlite3` reader process while
separate short-lived writers mutated the same database, and synchronised the
two with `sleep(400ms)`. That is a guess about how fast the host is, not a
guarantee: `CREATE TABLE t` executes asynchronously in the reader process, and
when a writer wins the race it fails with

  sqlite3 writer failed: Error: in prepare, no such table: t

which is what CI hit -- a failure with nothing to do with the code under test,
appearing only on slower or busier machines.

HeldReader::run appends a unique sentinel to each statement and blocks until it
appears on the reader's stdout. sqlite3 executes a script strictly in order on
one connection, so the sentinel cannot print before the preceding statements
have run: the acknowledgement follows from ordering, not from timing. A 30s
timeout bounds it and panics with the offending SQL rather than hanging.

Verified in this order:
  1. reproduced deterministically -- sleeps set to 0 fail 3/3 with the CI error
  2. handshake applied, sleeps removed entirely -- green
  3. 15 consecutive runs -- 15 pass, 0 fail
  4. control A: delete the CREATE TABLE -> FAILS loudly, `no such table: t`
  5. control B: sentinel that can never match -> panics in 3.02s, bounded, no
     hang
  6. restored -> green

Both controls asserted their target text was present before mutating, so
neither could silently no-op and report a green that tested nothing.

Applied to wal_snapshot_oracle.rs as well, which carried the identical pattern
-- a correction that does not reach its copies is barely a correction. Both now
share core/tests/common/mod.rs.

Side effect worth noting: row_history_oracle 0.83s -> 0.04s and
wal_snapshot_oracle ~0.9s -> 0.07s. The 800ms per fixture was pure waiting.
@h4x0r
h4x0r merged commit ae37c40 into main Aug 9, 2026
23 checks passed
@h4x0r
h4x0r deleted the fix/oracle-test-race branch August 9, 2026 14:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant