Skip to content

chore: centralize frontier OpenAI/Anthropic chat defaults - #38

Open
chrisknvidia wants to merge 8 commits into
mainfrom
chore/centralize-frontier-model-defaults
Open

chore: centralize frontier OpenAI/Anthropic chat defaults#38
chrisknvidia wants to merge 8 commits into
mainfrom
chore/centralize-frontier-model-defaults

Conversation

@chrisknvidia

@chrisknvidia chrisknvidia commented Aug 4, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • Addresses OpenClaw review feedback that the OpenAI default (gpt-5.4-mini) should be a frontier model — deferred from docs PR docs: clarify README launch overview and NVIDIA Build quickstart #33 into its own change (Fixes chore: centralize and update OpenAI/Anthropic frontier chat defaults #37).
  • Centralizes chat defaults in provider_config (CHAT_DEFAULT_*, CHAT_CHEAP_OPENAI, CHAT_DEFAULT_MODELS); constants.DIMENSION_JUDGE_MODEL and llm_judge.DEFAULT_JUDGE_MODEL import from there. Harbor templates/eval.py keeps a local copy with drift tests.
  • Pinned IDs (overridable via SKILL_EVAL_LLM_MODEL):
    • OpenAI: gpt-5.6-sol
    • Anthropic: claude-opus-4-8
    • Bedrock: us.anthropic.claude-opus-4-8 (AWS US geo inference profile)
    • Cheap OpenAI option: gpt-5.4-mini (CHAT_CHEAP_OPENAI)
  • Normalizes bare and provider-prefixed gpt-5* IDs for both temperature compatibility and native OpenAI max_completion_tokens; raises dimension/insights judge max_tokens to 4096.

Documentation parity

  • Updated the Fern provider guide, CLI reference, Tier 3 guide, and environment-variable reference with the exact provider defaults and override behavior.
  • Updated the repository README so users do not have to enter the full docs site to discover a billable default-model change.
  • Kept gpt-5.4-mini documented as an explicit lower-cost override rather than a silent default.

Before / after

Surface Before After
OpenAI default gpt-5.4-mini gpt-5.6-sol
Anthropic default claude-sonnet-4-5 claude-opus-4-8
Bedrock default us.anthropic.claude-sonnet-4-5-20250929-v1:0 us.anthropic.claude-opus-4-8
README / env reference Exact defaults were not visible Exact defaults and SKILL_EVAL_LLM_MODEL override are visible

Test plan

  • uv run pytest tests/test_provider_config.py tests/test_llm_judge.py tests/test_oss_packaging.py tests/test_provider_agent_routing.py tests/tier3/test_judge_parse_robustness.py tests/inference/test_llm_client.py tests/validators/test_security.py -q (457 passed)
  • uv run pytest -q (3,091 passed, 8 skipped, 3 deselected)
  • uv run ruff check .
  • uv build --python 3.13 --no-sources
  • Scan source, wheel, and sdist with scripts/check_oss_boundary.py
  • fern check (0 errors)
  • Confirm docs contain no stale Sonnet/default-model claims

Pin public chat defaults to gpt-5.5 / claude-opus-4-8 (Bedrock
us.anthropic.claude-opus-4-8), keep gpt-5.4-mini as the documented cheap
OpenAI override, and harden gpt-5* judge temperature/token handling.
Addresses OpenClaw review feedback deferred from #33.

Fixes #37

Signed-off-by: Christopher Kevin <christopherk@nvidia.com>

@rng1995 rng1995 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The model-default centralization and temperature handling look coherent, but this branch currently has one blocking OSS-boundary CI issue.

Comment thread tests/test_provider_config.py
Signed-off-by: Narendran Raghavan <nraghavan@nvidia.com>
rng1995
rng1995 previously approved these changes Aug 5, 2026

@rng1995 rng1995 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved after the OSS-boundary allowlist issue was fixed in 9f7bef4, the thread was resolved, and the full GitHub check matrix passed.

@rng1995 rng1995 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fresh review found two actionable issues in the OpenAI default/parameter migration. The existing OSS-boundary thread is already fixed and resolved; I will address these new findings and then update the branch from current main.

Comment thread src/skillevaluator/provider_config.py Outdated
Comment thread src/skillevaluator/tier3/eval_core/llm_judge.py Outdated
…tier-model-defaults

Signed-off-by: Narendran Raghavan <nraghavan@nvidia.com>
Signed-off-by: Narendran Raghavan <nraghavan@nvidia.com>
rng1995
rng1995 previously approved these changes Aug 12, 2026

@rng1995 rng1995 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved after fresh review and remediation. Both new P1 findings are fixed in bbbeae3, all review threads are resolved, the branch is current with main and conflict-free, local targeted/full/package/boundary verification passed, and the complete GitHub check matrix is green.

…tier-model-defaults

Signed-off-by: Narendran Raghavan <nraghavan@nvidia.com>

# Conflicts:
#	CHANGELOG.md
#	config/oss_boundary_allowlist.json
rng1995
rng1995 previously approved these changes Aug 12, 2026

@rng1995 rng1995 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved on final head after updating from current main. Both conflicts are resolved, all review threads remain resolved, focused verification passes (404 tests), the full suite passes (3,669 passed, 13 skipped, 3 deselected), package and source/archive OSS-boundary checks pass, all 15 GitHub checks are green, and fresh merge-tree/GitHub audits report the PR conflict-free.

Signed-off-by: Narendran Raghavan <nraghavan@nvidia.com>
rng1995
rng1995 previously approved these changes Aug 18, 2026
…tier-model-defaults

Signed-off-by: Narendran Raghavan <nraghavan@nvidia.com>

# Conflicts:
#	CHANGELOG.md
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

chore: centralize and update OpenAI/Anthropic frontier chat defaults

2 participants