Models frontend and saas suport missing after merging #78

[juanmichelini]

We merged #78 and now many models are missing frontend and saas support.

See https://openhands-llm-support-tracker.vercel.app/

To clarify except maybe for trinity-large-thinking and GPT-5.5 which were recently added and might not be in the release version.
All other models should check both SaaS and self-hosted.

[juanmichelini]

@OpenHands please explain in a comment here exactly how we are currently determining for each model the state of frontend, SaaS and self-hosted. I want you to take your time and look at the code and evidence of how it works.

Now I do not want you to create a PR yet, but I want this to match my experience when I open https://app.all-hands.dev/ or build the self hosted https://github.com/OpenHands/openhands

So please think hard and explain in simple terms how the process is done now, and what changes would be needed for fixing this (without creating the PR yet)

Take your time and think this through, then answer in a a brief but step by step way in this issue.

[openhands-ai]

I'm on it! juanmichelini can track my progress at all-hands.dev

[juanmichelini]

How the tracker determines frontend / SaaS / self-hosted today

(All in scripts/track_llm_support.py, after PR #78.)

1. frontend_support_timestamp — "self-hosted" / OpenHands frontend

search_frontend_for_model() runs git log -G <alias> over two repos and takes the earliest commit:

Repo	Path searched	Why
OpenHands/OpenHands	frontend/src/utils/verified-models.ts	Legacy location of the verified-models list. This file no longer exists on main of OpenHands — the data moved to the SDK.
OpenHands/software-agent-sdk	openhands-sdk/openhands/sdk/llm/utils/model_features.py	What PR #78 picked as the new "single source of truth".

A non-null timestamp means the model id (or one of its MODEL_ALIASES) was first introduced into one of those files on that date.

2. frontend_saas_available — SaaS dropdown at app.all-hands.dev

check_saas_verified_model() calls the live API:

GET https://app.all-hands.dev/api/v1/config/models/search?provider__eq=openhands

It iterates pages, lower-cases each name, and looks for a match against the bare aliases of the model in MODEL_ALIASES. It deliberately accepts both verified=true (the dropdown's Verified sub-section, sourced from the SDK's VERIFIED_OPENHANDS_MODELS) and verified=false (Others, DB-only entries).

3. litellm_support_timestamp / SDK / proxies

Independent, work fine — not what's broken here.

---

What broke after merging #78

I cross-checked the actual upstream files and the live SaaS catalog:

Bug A — PR #78 points at the wrong SDK file

VERIFIED_OPENHANDS_MODELS does not live in model_features.py. It lives in:

openhands-sdk/openhands/sdk/llm/utils/verified_models.py

model_features.py is the file that defines feature flags (extended thinking, prompt cache, etc.). Some model names happen to appear there because they're part of feature allow-lists, but that overlap is incidental and incomplete:

• Models present (incidentally) in model_features.py → tracker still finds a date:
  claude-sonnet-4-5/4-6, claude-opus-4-5/4-6/4-7, kimi-k2.5/2.6, gemini-3 (substring), gpt-5 (substring).
• Models that are only in verified_models.py → tracker reports frontend_support_timestamp = null. These are exactly the ones missing on the dashboard:
  • DeepSeek-V3.2-Reasoner
  • GLM-5.1
  • MiniMax-M2.1, MiniMax-M2.7
  • Nemotron-3-Nano, Nemotron-3-Super
  • Qwen3.6-Plus

I confirmed this by cloning OpenHands/software-agent-sdk and grepping. verified_models.py was created on 2025-09-15 (PR #276 — "Expose methods/dicts for model choices") and is the file the OpenHands frontend's model selector ultimately reads (it now gets the verified flag from the API, not from an in-repo TS file).

For the two recent ones you flagged:

• gpt-5.5 was added to verified_models.py on 2026-05-01 (PR Add gpt-5.5 to verified models software-agent-sdk#3039).
• trinity-large-thinking was added on 2026-05-05 (PR #3059, then reorganised in #3072).

So once we point at the right file, both will get a non-null frontend_support_timestamp. Whether that commit is in a tagged release of OpenHands/OpenHands is a separate question — the tracker's convention everywhere else is "first commit on main", so I'd keep that.

Bug B — one missing SaaS alias

I queried the live SaaS catalog (/api/v1/config/models/search?provider__eq=openhands). It currently returns 32 entries, all verified=true. Comparing them to our 24 tracked models:

Tracker says SaaS=false	Actually in SaaS API as	Verdict
Qwen3.6-Plus	openhands/qwen3-6-plus	Wrong — alias missing
Kimi-K2.6	(not in catalog)	Correct
Qwen3-Coder-Next	(not in catalog)	Correct
GPT-5.5	(not in catalog)	Correct
trinity-large-thinking	(not in catalog)	Correct

Root cause: MODEL_ALIASES["Qwen3.6-Plus"] only lists dashscope/qwen3.6-plus, so the bare-alias set is {qwen3.6-plus}. SaaS uses the SDK spelling qwen3-6-plus (hyphen, no dot). The two strings don't compare equal, so the lookup misses.

---

What needs to change (no PR yet, just the plan)

1. One-line path fix in search_frontend_for_model() — change the SDK search target from
   openhands-sdk/openhands/sdk/llm/utils/model_features.py
   to
   openhands-sdk/openhands/sdk/llm/utils/verified_models.py.
   This is what PR Fix frontend model detection to search SDK VERIFIED_OPENHANDS_MODELS #78 intended to do; it just picked the wrong filename.

2. Add the missing alias for Qwen3.6-Plus in MODEL_ALIASES: include qwen3-6-plus so the SaaS bare-name lookup matches the SDK spelling.

3. Update tests in tests/test_track_llm_support.py:

   • The two regression tests added in PR Fix frontend model detection to search SDK VERIFIED_OPENHANDS_MODELS #78 (test_search_frontend_uses_sdk_verified_models_after_migration and the trinity/glm-5.1/etc cases) currently assert against model_features.py. They need to assert against verified_models.py — and ideally cover at least one model that only exists there (e.g. Nemotron-3-Super or Qwen3.6-Plus) so this regression can't sneak back in.
   • Add a small test that Qwen3.6-Plus's alias set contains qwen3-6-plus, and a SaaS-matching test using a fixture catalog containing openhands/qwen3-6-plus.

4. Refresh frontend/public/all_models.json by re-running the tracker (the workflow already does this on schedule; a manual run will speed up the dashboard fix).

After (1)+(2)+(4), every model except GPT-5.5, Kimi-K2.6, Qwen3-Coder-Next, and trinity-large-thinking will have both frontend_support_timestamp set and frontend_saas_available=true, matching what users actually see in the openhands-provider dropdown — both on app.all-hands.dev (SaaS) and on a fresh self-hosted build of OpenHands/openhands (which now reads the same SDK list via the API).

Holding off on the PR per your instruction. Let me know when to proceed and I'll roll (1)–(4) into a single small change.

— this comment was written by an AI agent (OpenHands) on behalf of @juanmichelini.

[openhands-ai]

Summary

The user explicitly asked me not to create a PR, but instead to investigate and explain — in a step-by-step way — how the tracker currently determines frontend, SaaS, and self-hosted state for each model, and what needs to change to fix issue #79.

What I did

• Investigated scripts/track_llm_support.py (the post-Fix frontend model detection to search SDK VERIFIED_OPENHANDS_MODELS #78 implementation of search_frontend_for_model and check_saas_verified_model).
• Cloned OpenHands/software-agent-sdk and confirmed that VERIFIED_OPENHANDS_MODELS actually lives in openhands-sdk/openhands/sdk/llm/utils/verified_models.py, not model_features.py (which is the file PR Fix frontend model detection to search SDK VERIFIED_OPENHANDS_MODELS #78 pointed at).
• Cross-referenced the SDK list against frontend/public/all_models.json to identify exactly which models show null frontend_support_timestamp because of the wrong path.
• Queried the live SaaS API GET app.all-hands.dev/api/v1/config/models/search?provider__eq=openhands and compared its 32 entries to our MODEL_ALIASES to identify a single missing alias (qwen3-6-plus for Qwen3.6-Plus).
• Confirmed gpt-5.5 and trinity-large-thinking are already in the SDK's verified_models.py but legitimately absent from the SaaS catalog.
• Posted a step-by-step explanation as a comment on issue Models frontend and saas suport missing after merging #78 #79: Models frontend and saas suport missing after merging #78 #79 (comment)

Checklist against the user's request

• Explain how each of frontend, SaaS, and self-hosted is currently determined — done in the comment.
• Provide evidence from the code/upstream files — done (file paths, dates, PR numbers, and live API contents cited).
• Match the explanation to what users actually see at app.all-hands.dev (SaaS) and self-hosted OpenHands — done (the comment maps the SaaS API, the SDK VERIFIED_OPENHANDS_MODELS, and the dropdown's Verified/Others split).
• Outline the changes needed to fix it — done (1) point search_frontend_for_model at verified_models.py, (2) add qwen3-6-plus alias, (3) update tests, (4) refresh all_models.json.
• Do NOT create a PR — respected. No code changed, no branch pushed, no PR opened.

No code changes

No files in the repository were modified, so concision/extraneous-changes review is N/A. The deliverable was a single GitHub issue comment, written for the requested audience and tagged as AI-generated on @juanmichelini's behalf.

I'm standing by for the green light before turning the plan into a small PR (path fix + alias + tests + data refresh).

[juanmichelini]

@OpenHands cool fix this

[openhands-ai]

I'm on it! juanmichelini can track my progress at all-hands.dev

[openhands-ai]

Summary

Opened PR #81 (#81) to fix issue #79.

Checklist against the request

• Investigated the regression — confirmed the diagnosis posted in the previous issue comment: PR Fix frontend model detection to search SDK VERIFIED_OPENHANDS_MODELS #78 pointed search_frontend_for_model() at the wrong SDK file (model_features.py instead of verified_models.py), and Qwen3.6-Plus was missing the qwen3-6-plus alias the SaaS catalog uses.
• Made the code fix in scripts/track_llm_support.py:
  • Changed the SDK search target from …/utils/model_features.py to …/utils/verified_models.py (and updated the docstring).
  • Added "qwen3-6-plus" to MODEL_ALIASES["Qwen3.6-Plus"].
• Added regression tests in tests/test_track_llm_support.py (per the guidelines, since this is application code):
  • New test_search_targets_verified_models_not_model_features — inspects the paths passed to git log -- so the wrong-file regression can't reappear.
  • New test_qwen36_plus_matches_sdk_spelling — SaaS-side check against an openhands/qwen3-6-plus catalog entry.
  • Updated test_finds_model_in_sdk_verified_openhands_models, test_picks_earliest_across_both_repos, and test_qwen36_plus_has_aliases to assert the new path/alias.
• Ran the test suite — GIT_TERMINAL_PROMPT=0 python -m pytest tests/test_track_llm_support.py -q → 91 passed.
• Branch + PR — created openhands/fix-frontend-saas-detection-79, committed with Co-authored-by: openhands, pushed, opened PR Fix frontend SaaS detection broken after #78 #81 against main with a description that includes Fixes #79 and an AI-authored disclosure.

Conciseness

The diff is intentionally minimal — only two production lines changed (one path string, one alias entry) plus a small docstring update; the rest is test additions/updates that lock in the fix. No frontend/public/all_models.json refresh was committed — the scheduled tracker workflow will repopulate it with real timestamps on its next run, which is the cleaner approach than committing data generated locally without the SaaS credentials. No extraneous changes were made.