[Feature] Add generic OpenAI-compatible provider for Groq, Together AI, Mistral, and other endpoints

[mimran-khan]

Summary

A growing ecosystem of LLM providers (Groq, Together AI, Mistral, DeepInfra, Fireworks, Perplexity, LiteLLM, etc.) expose OpenAI-compatible chat completion APIs with custom base URLs. Each requires nothing more than an API key + base URL — the same ChatOpenAI client works for all of them.

Today, pointing SkillSpector at these providers requires setting SKILLSPECTOR_PROVIDER=openai and overriding OPENAI_BASE_URL. This works but has drawbacks:

1. The env var names are misleading (OPENAI_API_KEY for a Groq key)
2. The model registry only contains OpenAI model token budgets — Groq/Mistral models get the 128k fallback
3. There's no way to ship provider-specific model registries without creating a full provider subpackage per service

A single "generic OpenAI-compatible" provider with its own env vars and extensible registry would serve this entire ecosystem.

Why This Matters

Scenario: Using Groq for fast, cheap scans

Groq offers sub-second inference on llama and mixtral models at a fraction of OpenAI's cost. A developer wants to use it for CI scans:

# Current workaround — misleading env var names
export SKILLSPECTOR_PROVIDER=openai
export OPENAI_API_KEY=gsk_abc123
export OPENAI_BASE_URL=https://api.groq.com/openai/v1
export SKILLSPECTOR_MODEL=llama-3.1-70b-versatile

This works for the API call but the model registry lookup fails (no llama-3.1-70b-versatile in OpenAI's registry), so token budgeting uses the 128k fallback — potentially sending oversized batches that get truncated.

Scenario: Self-hosted vLLM / LocalAI

Teams running vLLM, LocalAI, or other OpenAI-compatible servers internally need the same pattern but with a custom base_url pointing to their infrastructure.

Proposed Implementation

providers/
  openai_compatible/
    __init__.py
    provider.py
    model_registry.yaml

Environment variables:

Variable	Required	Description
SKILLSPECTOR_COMPAT_API_KEY	Yes	API key for the target provider
SKILLSPECTOR_COMPAT_BASE_URL	Yes	Base URL (e.g., https://api.groq.com/openai/v1)
SKILLSPECTOR_COMPAT_DEFAULT_MODEL	No	Default model ID (overrides registry default)

provider.py:

class OpenAICompatibleProvider:
    DEFAULT_MODEL = "llama-3.1-70b-versatile"
    SLOT_DEFAULTS: dict[str, str] = {}

    def resolve_credentials(self) -> tuple[str, str | None] | None:
        api_key = os.environ.get("SKILLSPECTOR_COMPAT_API_KEY", "").strip()
        base_url = os.environ.get("SKILLSPECTOR_COMPAT_BASE_URL", "").strip()
        if not api_key or not base_url:
            return None
        return api_key, base_url

    def create_chat_model(self, model, *, max_tokens, timeout=120):
        return create_openai_compatible_chat_model(
            model=model,
            credentials=self.resolve_credentials(),
            max_tokens=max_tokens,
            timeout=timeout,
        )

model_registry.yaml — ships with common models across providers:

models:
  # Groq
  llama-3.1-70b-versatile:
    context_length: 131072
    max_output_tokens: 8192
  mixtral-8x7b-32768:
    context_length: 32768
    max_output_tokens: 4096
  # Together AI
  meta-llama/Meta-Llama-3.1-70B-Instruct-Turbo:
    context_length: 131072
    max_output_tokens: 4096
  # Mistral
  mistral-large-latest:
    context_length: 131072
    max_output_tokens: 8192
  # DeepInfra / generic
  deepseek-ai/DeepSeek-V3:
    context_length: 65536
    max_output_tokens: 8192

Users can extend with SKILLSPECTOR_MODEL_REGISTRY for models not in the bundled list.

Selection:

export SKILLSPECTOR_PROVIDER=openai_compatible
export SKILLSPECTOR_COMPAT_API_KEY=gsk_abc123
export SKILLSPECTOR_COMPAT_BASE_URL=https://api.groq.com/openai/v1
export SKILLSPECTOR_MODEL=llama-3.1-70b-versatile

skillspector scan ./my-skill/

Related

• PR fix(analyzer): parse structured output from text so OpenAI-compatible endpoints don't crash LLM analysis #71 (APPROVED): Parse structured output from text — a prerequisite for this provider. Many OpenAI-compatible endpoints don't honor response_format, so text fallback parsing is essential for Groq/Together/Mistral models to work reliably.
• Feature request: OpenRouter #81 (CLOSED): OpenRouter provider request — acknowledged by maintainers. OpenRouter is one of the endpoints this generic provider would serve without needing a dedicated provider per service.
• LLM stage crashes on OpenAI-compatible/local endpoints (Ollama): models return confidence 0-100 but output schema validates 0-1 #89 (CLOSED): Ollama confidence-scale crash — demonstrated that users already point SkillSpector at OpenAI-compatible endpoints via OPENAI_BASE_URL. This provider formalizes that pattern with proper env vars and registry.
• models glm-5.2, glm-5.1, kimi-k2.7-code, deepseek-v4-flash fail #129 (OPEN): Models like glm-5.2, kimi-k2.7-code, deepseek-v4-flash fail — a broader model registry would surface token budget issues earlier for these models.
• The existing openai provider's OPENAI_BASE_URL override still works as a simpler escape hatch; this provider adds proper env var naming and a multi-provider registry.

Scope

• New providers/openai_compatible/ subpackage (~60 lines)
• Registration in providers/__init__.py (~5 lines)
• Bundled model registry with common cross-provider models
• Tests: credential resolution, base URL validation
• Documentation: add to README provider table and .env.example