fix(tests): keep the offline gate hermetic for model-fallback tests - #72
Conversation
The embedder/reranker factory fallback tests constructed a real SentenceTransformer/CrossEncoder for an unresolvable model name, which resolves against the Hugging Face Hub. On a host with no route to the Hub that connect() blocks instead of erroring, so "python -m pytest tests/ -q" - the documented offline gate - hung indefinitely rather than failing. The fallback relied on the network failing fast, not on it being absent. Patch the loader in both tests so they exercise the fallback contract locally and instantly with no network, per AGENTS.md section 3 (local-first and offline-capable).
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 93e7e53ed0
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Addresses review feedback on PR #72. The first version imported sentence_transformers in the test helper and only guarded ImportError. If the package imports but raises something else - a RuntimeError from a mismatched torch/torchvision, say - the helper propagated it and the test failed, even though both production factories catch Exception and fall back correctly. That made the fallback tests depend on the health of the optional heavy stack. Patch the adapter each factory constructs instead (SentenceTransformerEmbedder, CrossEncoderReranker). The optional stack is now never imported by these tests, so every package-import failure stays inside the factory under test, where the existing except Exception handles it. Verified: sentence_transformers is absent from sys.modules after the module's tests run.
|
Good catch, and fixed in 06f09b0. The helper no longer imports Verified the optional stack is never imported: after running this module's tests, Full offline gate green locally (pytest, ruff, both eval harnesses, ablation), and all 9 required checks pass on CI including Python 3.10/3.11/3.12 and the numpy-only 3.9 floor. |
Problem
python -m pytest tests/ -q— the offline gate documented inCLAUDE.mdandAGENTS.md§1 — hangs indefinitely on a host with no network route, rather than passing or failing.Two tests assert that the backend factories fall back when a model cannot be loaded:
test_embedder_factory_falls_back_offlinetest_reranker_factory_falls_back_offlineBoth pass an unresolvable model name, which sends
SentenceTransformer/CrossEncoderto the Hugging Face Hub to resolve it. When the host has no route to the Hub, the TCP connect sits inSynSentinstead of erroring, so the suite never reaches a summary line.The fallback relied on the network failing fast, not on it being absent — which is precisely the condition AGENTS.md §3 ("local-first and offline-capable") requires the core to survive.
Fix
Patch the sentence-transformers loader in both tests so the load failure is raised locally and instantly. No network is touched, and the fallback contract each test exists to verify is still exercised end-to-end through
get_embedder/get_reranker.If
sentence_transformersis not installed at all, the helper no-ops — the factory already fails fast withModuleNotFoundError, which is the path minimal CI (pip install numpy pytest) takes.Test-only change. No production code is modified.
Verification
Full offline gate, run locally:
python -m pytest tests/ -q— completes; previously hung foreverpython -m ruff check .— All checks passedpython -m eval.harness --dataset eval/datasets/sample.jsonl --k 5— recall@k 1.000, hit@k 1.000, answer_token_recall 1.000python -m eval.harness --dataset eval/datasets/codemem.jsonl --k 5— recall@k 1.000, hit@k 1.000, answer_token_recall 1.000python -m eval.ablation— vector-only 1.0, hybrid-1hop 1.0, hybrid-ppr 1.0; multi-hop arm-level: vector 0.667, graph 1-hop 0.0, graph PPR 1.0The two target tests now complete in well under a second with
--timeout=30.Note on an unrelated pre-existing failure
tests/test_session_idempotent.py::test_session_close_linearizes_before_delayed_memory_write[ingest]fails locally when a developer.envsetsENGRAPHIS_EXTRACTOR=llmwith an unreachable provider: the LLM client retries 2s + 4s and exceeds the test's 10s budget. It reproduces standalone onmainwithout this branch, and CI has no.env, so it is green there. Not addressed here — it is the same hermeticity gap in a different place and deserves its own change (neutralising LLM env vars intests/conftest.py).