Skip to content

feat: per-slot model env overrides and model validation#178

Open
mimran-khan wants to merge 1 commit into
NVIDIA:mainfrom
mimran-khan:feat/per-slot-model-and-validation
Open

feat: per-slot model env overrides and model validation#178
mimran-khan wants to merge 1 commit into
NVIDIA:mainfrom
mimran-khan:feat/per-slot-model-and-validation

Conversation

@mimran-khan

Copy link
Copy Markdown
Contributor

Summary

  • Add SKILLSPECTOR_MODEL_{SLOT} environment variables for per-slot model selection (e.g., SKILLSPECTOR_MODEL_META_ANALYZER=gpt-4o, SKILLSPECTOR_MODEL_SEMANTIC_DEVELOPER_INTENT=gpt-4o-mini), enabling cost optimization and independent model failure workarounds
  • Add model validation at startup that warns when configured models are not found in the provider's model_registry.yaml, with opt-in strict mode (SKILLSPECTOR_STRICT_MODEL_VALIDATION=true) that fails fast instead of silently using fallback token budgets

Per-slot model precedence

SKILLSPECTOR_MODEL_{SLOT} > provider SLOT_DEFAULTS > SKILLSPECTOR_MODEL > provider DEFAULT_MODEL

Changes

  • src/skillspector/constants.py — new _resolve_slot_model() function with per-slot env var support; new _validate_model_config() with warning + strict mode
  • tests/unit/test_constants.py — 11 tests covering env var precedence, fallback chain, warning emission, and strict mode

Closes #176, closes #177

Test plan

  • All 11 new tests pass (pytest tests/unit/test_constants.py -v)
  • All 370 existing unit tests pass (no regressions)
  • Per-slot env var overrides provider default
  • Per-slot env var overrides global SKILLSPECTOR_MODEL
  • Whitespace-only env var treated as unset
  • Unknown model logs warning with model name and fallback context length
  • Known model produces no warning
  • Strict mode raises ValueError on unknown models
  • Strict mode passes with known models

Add SKILLSPECTOR_MODEL_{SLOT} environment variables for per-slot model
selection, enabling cost optimization (cheap models for high-volume
semantic analysis, expensive models for meta-analysis) and independent
model failure workarounds.

Add model validation at startup that warns when configured models are
not found in the provider's model_registry.yaml. Opt-in strict mode
(SKILLSPECTOR_STRICT_MODEL_VALIDATION=true) fails fast instead of
silently using fallback token budgets.

Closes NVIDIA#176, closes NVIDIA#177

@rng1995 rng1995 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Automated SkillSpector Review]

Approved.

Per-slot SKILLSPECTOR_MODEL_{SLOT} overrides with a clear precedence chain, plus startup model validation (warn, or fail-fast under SKILLSPECTOR_STRICT_MODEL_VALIDATION). Logic is correct and the 11 tests cover precedence, fallback, whitespace-as-unset, warning emission, strict-mode raise, and case-insensitivity.

Non-blocking: _validate_model_config() calls _provider.get_context_length(...) with a # type: ignore[attr-defined], which suggests the metadata-provider Protocol doesn't declare that method — worth adding it to the Protocol for type safety. Also note _validate_model_config() runs at import time, so strict mode aborts process startup (intended, just calling it out).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature] Model validation and graceful failure for unknown model IDs [Feature] Per-slot model selection via environment variables

2 participants