feat(inference): add ov_smoke integration test suite - #207
Open
pwolnows wants to merge 5 commits into
Open
Conversation
Three-stage pytest contract that validates 'OpenVINO works with physicalai'
against the three published HuggingFace exports:
OpenVINO/act-fp16-ov — no tokenizer
OpenVINO/pi05-libero-fp16-ov — tokenizer-bearing (VLA)
OpenVINO/smolvla-libero-fp16-ov — tokenizer-bearing (VLA)
Stages:
1. load — OpenVINOAdapter.load (read_model + compile_model)
2. tokenizer — OVTokenizer (openvino_tokenizers extension loads and
matches core OpenVINO version)
3. predict — predict_action_chunk returns finite outputs
The tokenizer stage catches the exact failure mode that caused the
openvino pin: under a core/tokenizer mismatch ACT passes while pi05
fails at stage 2, pinpointing the broken version pair.
Also adds:
- docs/design/openvino-validation.md — design proposal for the full
OV validation pipeline (smoke, golden-action, perf, gym E2E)
- ov_smoke pytest marker in pyproject.toml
- tests/integration/ package skeleton
Invoke with:
pytest -m ov_smoke -s
OV_SMOKE_CACHE_DIR=~/.cache/hf pytest -m ov_smoke
Contributor
There was a problem hiding this comment.
Pull request overview
Adds an OpenVINO-focused integration “smoke” test suite to validate that Physical AI inference remains compatible across OpenVINO upgrades, specifically catching core/runtime vs tokenizer mismatches using published HuggingFace OpenVINO exports.
Changes:
- Introduces
ov_smokeintegration tests covering three contract stages: model load, tokenizer init+tokenize, and predict outputs finiteness. - Registers the new
ov_smokepytest marker inpyproject.toml. - Adds a design/proposal document describing a broader OpenVINO validation pipeline (smoke, golden-action, perf, gym E2E).
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/integration/inference/test_ov_smoke.py | New ov_smoke pytest suite downloading HF exports and exercising OpenVINO load/tokenizer/predict stages. |
| tests/integration/inference/init.py | Initializes the new integration test package. |
| tests/integration/init.py | Initializes the integration test package root. |
| pyproject.toml | Adds the ov_smoke marker definition for strict-markers pytest runs. |
| docs/design/openvino-validation.md | Documents the intended OpenVINO validation contract and planned CI tiers. |
Use os.path.normpath instead of Path.resolve() when resolving the tokenizer artifact path from the manifest, mirroring the same pattern used by component_factory._resolve_artifact_path. Path.resolve() followed the snapshot symlink into the blobs/ store, giving a raw blob path with no .bin sibling. OpenVINO's read_model then raised 'Empty weights data in bin file or bin file cannot be found'. normpath keeps the path in the snapshot dir where the .xml and .bin symlinks coexist. Verified: all 8 ov_smoke tests pass on openvino==2026.2.0 with this fix.
- ov_tokenizer.py: np.bool -> np.bool_ (removed in NumPy 1.24) - test_ov_smoke.py: add isinstance(model.adapter, OpenVINOAdapter) so pyrefly can narrow RuntimeAdapter before accessing compiled_model (fixes missing-attribute error) - test_ov_smoke.py: suppress bad-argument-type for OVTokenizer call with list[str] TASK value (base Preprocessor signature is dict[str, ndarray] which does not reflect that OVTokenizer accepts list[str] for TASK)
Tests download multi-GB artifacts via download_from_hub(); adding the marker lets developers and CI deselect them with -m 'not requires_download'.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements the
ov_smokepytest suite described indocs/design/openvino-validation.md. The test validates the three-stage contract "OpenVINO works with physicalai" against the three published HuggingFace exports in the OpenVINO/physical-ai collection.Motivation
An OpenVINO upgrade silently broke pi05 inference (only caught because of the existing
openvino==2026.1pin). The root cause was anopenvino/openvino_tokenizersversion mismatch — ACT (no tokenizer) passed while pi05 failed at the tokenizer stage. Without an automated test this failure was invisible until runtime.Contract stages
OpenVINOAdapter.loadread_model+compile_modelsucceedOVTokenizeropenvino_tokenizersextension loads and matches core OpenVINOInferenceModel.predict_action_chunkModels
OpenVINO/act-fp16-ovOpenVINO/pi05-libero-fp16-ovOpenVINO/smolvla-libero-fp16-ovVerified: under a core/tokenizer mismatch
test_load[act-fp16-ov]passes whiletest_tokenizer[pi05-libero-fp16-ov]fails — detecting the exact break that caused the pin.Changes
tests/integration/inference/test_ov_smoke.py— 8-test suite (3 × load, 2 × tokenizer, 3 × predict)pyproject.toml— registers theov_smokepytest markerdocs/design/openvino-validation.md— design proposal for the full OV validation pipeline (smoke, golden-action, perf, gym E2E)Usage