Skip to content

fix(tests): keep the offline gate hermetic for model-fallback tests - #72

Merged
Coding-Dev-Tools merged 2 commits into
mainfrom
fix/offline-gate-hermetic-model-fallback
Jul 25, 2026
Merged

fix(tests): keep the offline gate hermetic for model-fallback tests#72
Coding-Dev-Tools merged 2 commits into
mainfrom
fix/offline-gate-hermetic-model-fallback

Conversation

@Coding-Dev-Tools

Copy link
Copy Markdown
Owner

Problem

python -m pytest tests/ -q — the offline gate documented in CLAUDE.md and AGENTS.md §1 — hangs indefinitely on a host with no network route, rather than passing or failing.

Two tests assert that the backend factories fall back when a model cannot be loaded:

  • test_embedder_factory_falls_back_offline
  • test_reranker_factory_falls_back_offline

Both pass an unresolvable model name, which sends SentenceTransformer / CrossEncoder to the Hugging Face Hub to resolve it. When the host has no route to the Hub, the TCP connect sits in SynSent instead of erroring, so the suite never reaches a summary line.

The fallback relied on the network failing fast, not on it being absent — which is precisely the condition AGENTS.md §3 ("local-first and offline-capable") requires the core to survive.

Fix

Patch the sentence-transformers loader in both tests so the load failure is raised locally and instantly. No network is touched, and the fallback contract each test exists to verify is still exercised end-to-end through get_embedder / get_reranker.

If sentence_transformers is not installed at all, the helper no-ops — the factory already fails fast with ModuleNotFoundError, which is the path minimal CI (pip install numpy pytest) takes.

Test-only change. No production code is modified.

Verification

Full offline gate, run locally:

  • python -m pytest tests/ -q — completes; previously hung forever
  • python -m ruff check . — All checks passed
  • python -m eval.harness --dataset eval/datasets/sample.jsonl --k 5 — recall@k 1.000, hit@k 1.000, answer_token_recall 1.000
  • python -m eval.harness --dataset eval/datasets/codemem.jsonl --k 5 — recall@k 1.000, hit@k 1.000, answer_token_recall 1.000
  • python -m eval.ablation — vector-only 1.0, hybrid-1hop 1.0, hybrid-ppr 1.0; multi-hop arm-level: vector 0.667, graph 1-hop 0.0, graph PPR 1.0

The two target tests now complete in well under a second with --timeout=30.

Note on an unrelated pre-existing failure

tests/test_session_idempotent.py::test_session_close_linearizes_before_delayed_memory_write[ingest] fails locally when a developer .env sets ENGRAPHIS_EXTRACTOR=llm with an unreachable provider: the LLM client retries 2s + 4s and exceeds the test's 10s budget. It reproduces standalone on main without this branch, and CI has no .env, so it is green there. Not addressed here — it is the same hermeticity gap in a different place and deserves its own change (neutralising LLM env vars in tests/conftest.py).

The embedder/reranker factory fallback tests constructed a real
SentenceTransformer/CrossEncoder for an unresolvable model name, which
resolves against the Hugging Face Hub. On a host with no route to the Hub
that connect() blocks instead of erroring, so "python -m pytest tests/ -q"
- the documented offline gate - hung indefinitely rather than failing.
The fallback relied on the network failing fast, not on it being absent.

Patch the loader in both tests so they exercise the fallback contract
locally and instantly with no network, per AGENTS.md section 3
(local-first and offline-capable).

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 93e7e53ed0

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread tests/test_backends_factories.py Outdated
Addresses review feedback on PR #72.

The first version imported sentence_transformers in the test helper and only
guarded ImportError. If the package imports but raises something else - a
RuntimeError from a mismatched torch/torchvision, say - the helper propagated
it and the test failed, even though both production factories catch Exception
and fall back correctly. That made the fallback tests depend on the health of
the optional heavy stack.

Patch the adapter each factory constructs instead (SentenceTransformerEmbedder,
CrossEncoderReranker). The optional stack is now never imported by these tests,
so every package-import failure stays inside the factory under test, where the
existing except Exception handles it. Verified: sentence_transformers is absent
from sys.modules after the module's tests run.
@Coding-Dev-Tools

Copy link
Copy Markdown
Owner Author

Good catch, and fixed in 06f09b0.

The helper no longer imports sentence_transformers at all. It now patches the adapter each factory constructs — SentenceTransformerEmbedder in engraphis.backends.embedder_st and CrossEncoderReranker in engraphis.backends.reranker — so a package that imports but raises something other than ImportError (a RuntimeError from a mismatched torch, for example) can no longer fail these tests. Every such failure now stays inside the factory under test, where the existing except Exception catches it and falls back, which is the contract these tests exist to verify.

Verified the optional stack is never imported: after running this module's tests, 'sentence_transformers' in sys.modules is False.

Full offline gate green locally (pytest, ruff, both eval harnesses, ablation), and all 9 required checks pass on CI including Python 3.10/3.11/3.12 and the numpy-only 3.9 floor.

@Coding-Dev-Tools
Coding-Dev-Tools merged commit 74f5849 into main Jul 25, 2026
11 checks passed
@Coding-Dev-Tools
Coding-Dev-Tools deleted the fix/offline-gate-hermetic-model-fallback branch July 25, 2026 20:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant