feat: per-slot model env overrides and model validation#178
Conversation
Add SKILLSPECTOR_MODEL_{SLOT} environment variables for per-slot model
selection, enabling cost optimization (cheap models for high-volume
semantic analysis, expensive models for meta-analysis) and independent
model failure workarounds.
Add model validation at startup that warns when configured models are
not found in the provider's model_registry.yaml. Opt-in strict mode
(SKILLSPECTOR_STRICT_MODEL_VALIDATION=true) fails fast instead of
silently using fallback token budgets.
Closes NVIDIA#176, closes NVIDIA#177
rng1995
left a comment
There was a problem hiding this comment.
[Automated SkillSpector Review]
Approved.
Per-slot SKILLSPECTOR_MODEL_{SLOT} overrides with a clear precedence chain, plus startup model validation (warn, or fail-fast under SKILLSPECTOR_STRICT_MODEL_VALIDATION). Logic is correct and the 11 tests cover precedence, fallback, whitespace-as-unset, warning emission, strict-mode raise, and case-insensitivity.
Non-blocking: _validate_model_config() calls _provider.get_context_length(...) with a # type: ignore[attr-defined], which suggests the metadata-provider Protocol doesn't declare that method — worth adding it to the Protocol for type safety. Also note _validate_model_config() runs at import time, so strict mode aborts process startup (intended, just calling it out).
Summary
SKILLSPECTOR_MODEL_{SLOT}environment variables for per-slot model selection (e.g.,SKILLSPECTOR_MODEL_META_ANALYZER=gpt-4o,SKILLSPECTOR_MODEL_SEMANTIC_DEVELOPER_INTENT=gpt-4o-mini), enabling cost optimization and independent model failure workaroundsmodel_registry.yaml, with opt-in strict mode (SKILLSPECTOR_STRICT_MODEL_VALIDATION=true) that fails fast instead of silently using fallback token budgetsPer-slot model precedence
Changes
src/skillspector/constants.py— new_resolve_slot_model()function with per-slot env var support; new_validate_model_config()with warning + strict modetests/unit/test_constants.py— 11 tests covering env var precedence, fallback chain, warning emission, and strict modeCloses #176, closes #177
Test plan
pytest tests/unit/test_constants.py -v)