Skip to content

Fix obsidian-cli eval timeout causing false smoke-test failures - #18

Merged
smcllns merged 2 commits into
mainfrom
fix/17-test-eval-timeout
Jun 17, 2026
Merged

Fix obsidian-cli eval timeout causing false smoke-test failures#18
smcllns merged 2 commits into
mainfrom
fix/17-test-eval-timeout

Conversation

@yolo-sam

Copy link
Copy Markdown
Collaborator

Fixes #17

Problem

test/test.sh reported ~18 false failures: assertion blocks returned empty strings, so checks compared '' against expected values and failed.

Root cause

obsidian-cli eval only returns a value if its async IIFE settles within the same microtask flush. Any setTimeout — confirmed empty down to 50ms — yields to the event loop and the eval times out to empty. Microtask-only async and blob-URL fetch reads resolve fine inside one eval, which is why the fast blocks already passed.

Fix

The six slow blocks (THEME_RERENDER, DEDUPED_NAVIGATION, DEFERRED_STATE_NAVIGATION, MARKDOWN_EMBED, SIZED_MARKDOWN_EMBED, CANVAS_EMBED) are restructured to trigger-eval → bash sleep → read-eval. State that must survive the wait (theme classes; the deferred-leaf getViewState monkey-patch + its original) is stashed on window. No assertion intent or expected value changed.

cleanup() had the same latent bug — its in-eval 300ms setTimeout returned empty, silently skipping the leftover-leaf verification. Split into detach → bash sleep → inspect/restore so cleanup verifies again.

Header comment updated npm testbun run test. CI workflows untouched.

Validation

bun run test against the live vault, stable across two consecutive runs:

  • Outer + embed assertions: 35/35 pass
  • Inner (iframe postMessage): 18 passed · 0 failed · 0 skipped
  • All checks passed, exit 0.

@yolo-sam

Copy link
Copy Markdown
Collaborator Author

Adversarial review — Round 1 (Claude + Codex), head 1e2e313

Claude reviewer: VERDICT MERGEABLE. Two LOW advisory notes (no fix required): cleanup() dropped detachErrors from its JSON; deferred monkey-patch window slightly wider.
Codex reviewer: VERDICT NOT MERGEABLE, 3 findings — all accepted and fixed in 6205ff1.

Resolutions

  1. [HIGH] MARKDOWN_EMBED read eval re-found the wrong leaf (test/test.sh:418). DEDUPED_NAVIGATION and DEFERRED_STATE_NAVIGATION already open _html-docs-test-embed.md in earlier leaves, so re-finding 'first leaf by path' could read a non-preview leaf → false failure. Fixed (6205ff1): trigger eval stashes the exact opened leaf on window.__hvMarkdownEmbedLeaf; read eval reads that leaf and deletes the stash. (SIZED uses a unique note, CANVAS a unique file — unaffected.)
  2. [MED] Deferred getViewState monkey-patch could leak (test/test.sh:350). The original try/finally restore now spans a bash boundary; a throw before the restore eval would leave the live leaf patched. Fixed (6205ff1): cleanup() (the trap EXIT path) defensively restores getViewState from window.__hvDeferredState if present.
  3. [MED] Theme classes could be left flipped (test/test.sh:253). Same split-finally issue — a throw mid-flow would leave Obsidian's theme toggled. Fixed (6205ff1): cleanup() defensively restores the original theme classes from window.__hvThemeState if present.

Pushback

  • Claude's detachErrors note: keeping as-is. No assertion ever consumed detachErrors; a leaf that fails to detach still surfaces via the remaining array → exit 1. Net diagnostic loss is a less-specific message, not a missed failure.

Validation at new head

bun run test against the live vault: all outer + embed assertions pass, inner (iframe) 18 passed · 0 failed · 0 skipped, exit 0.

Confirm round (both families) running at 6205ff1.

@yolo-sam

Copy link
Copy Markdown
Collaborator Author

Adversarial review — Round 2 (confirm), head 6205ff1

Consensus reached. Fresh reviewers from both families re-read the full diff and verified all three round-1 resolutions:

  • Claude: all 3 CONFIRMED RESOLVED, no new issues — VERDICT MERGEABLE
  • Codex: all 3 CONFIRMED RESOLVED, no new issues — VERDICT MERGEABLE

Both confirmed: MARKDOWN_EMBED now reads the exact stashed leaf (no path re-find); cleanup() restores the deferred getViewState patch and theme classes on the guaranteed trap EXIT path, before detach, and is idempotent with the success path (no double-restore). No in-eval setTimeout remains; bash -n clean.

Flipping out of draft and assigning @smcllns. Not merging — Sam merges.

@yolo-sam
yolo-sam marked this pull request as ready for review June 17, 2026 22:15
@yolo-sam
yolo-sam requested a review from smcllns June 17, 2026 22:15
@smcllns
smcllns merged commit b26aad7 into main Jun 17, 2026
4 checks passed
@smcllns
smcllns deleted the fix/17-test-eval-timeout branch June 17, 2026 22:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

test.sh: obsidian-cli eval timeout truncates render/navigation assertions

2 participants