[Feature] Per-slot model selection via environment variables

[mimran-khan]

Summary

SkillSpector uses three LLM "slots" — semantic_analyzer, meta_analyzer, and meta_classifier — each calling the LLM with different prompt structures and output expectations. Today, all three slots use the same model (set via SKILLSPECTOR_MODEL or the provider's default).

This proposal adds per-slot model overrides via environment variables, enabling users to assign different models to each analysis stage. This is valuable for cost optimization, accuracy tuning, and working around model-specific failures.

Why This Matters

Scenario 1: Cost optimization

A security team runs SkillSpector on every PR in CI. The semantic analyzer processes the most tokens (full source batches) but needs moderate reasoning. The meta-analyzer sees only pre-filtered findings and needs strong reasoning for deduplication and risk scoring.

Optimal assignment:

• Semantic analyzer: gpt-4o-mini (cheap, handles volume)
• Meta analyzer: gpt-4o (strong reasoning for final scoring)
• Meta classifier: gpt-4o-mini (simple classification task)

This could reduce API costs by 60-80% per scan while maintaining analysis quality where it matters most.

Scenario 2: Working around model failures

Issue #129 documents models like glm-5.2 and kimi-k2.7-code failing at runtime. With per-slot overrides, a user hitting failures on meta-analysis could swap just that slot to a known-good model without changing the semantic analysis model that's working fine.

Scenario 3: Benchmarking analyzers independently

Researchers evaluating SkillSpector want to isolate which model performs best at each stage. Per-slot overrides enable controlled experiments without code changes.

Proposed Implementation

New environment variables:

Variable	Overrides	Example
SKILLSPECTOR_MODEL_SEMANTIC_ANALYZER	Semantic analyzer slot	gpt-4o-mini
SKILLSPECTOR_MODEL_META_ANALYZER	Meta analyzer slot	gpt-4o
SKILLSPECTOR_MODEL_META_CLASSIFIER	Meta classifier slot	gpt-4o-mini

Precedence: SKILLSPECTOR_MODEL_{SLOT} > provider SLOT_DEFAULTS > SKILLSPECTOR_MODEL > provider DEFAULT_MODEL.

Implementation in constants.py:

def _resolve_slot_model(provider, slot: str) -> str:
    env_key = f"SKILLSPECTOR_MODEL_{slot.upper()}"
    env_val = os.environ.get(env_key, "").strip()
    if env_val:
        return env_val
    return provider.resolve_model(slot)

MODEL_CONFIG: dict[str, str] = {
    slot: _resolve_slot_model(_provider, slot) for slot in _MODEL_SLOTS
}

Usage:

export SKILLSPECTOR_PROVIDER=openai
export SKILLSPECTOR_MODEL_SEMANTIC_ANALYZER=gpt-4o-mini
export SKILLSPECTOR_MODEL_META_ANALYZER=gpt-4o
export SKILLSPECTOR_MODEL_META_CLASSIFIER=gpt-4o-mini

skillspector scan ./my-skill/

CLI output would show the per-slot assignment at startup:

Models:
  semantic_analyzer: gpt-4o-mini  (via SKILLSPECTOR_MODEL_SEMANTIC_ANALYZER)
  meta_analyzer:     gpt-4o       (via SKILLSPECTOR_MODEL_META_ANALYZER)
  meta_classifier:   gpt-4o-mini  (via SKILLSPECTOR_MODEL_META_CLASSIFIER)

Scope

• Modify constants.py MODEL_CONFIG construction (~15 lines)
• Add per-slot model logging to CLI startup output
• Tests: env var precedence, slot-specific overrides, fallback chain
• Documentation: add per-slot model table to README