Skip to content

fix(dashboard): report the model size the installer recorded - #2515

Open
Hoang130203 wants to merge 1 commit into
Osmantic:mainfrom
Hoang130203:fix/model-info-declared-size
Open

fix(dashboard): report the model size the installer recorded#2515
Hoang130203 wants to merge 1 commit into
Osmantic:mainfrom
Hoang130203:fix/model-info-declared-size

Conversation

@Hoang130203

Copy link
Copy Markdown

Summary

helpers.get_model_info() guesses the model size from the model name, with
a ladder of \b<N>b\b patterns and a 15.0 GB default when nothing matches.
Checked against every model the tier maps actually select, and their declared
LLM_MODEL_SIZE_MB:

tier-map LLM_MODEL         declared GB  guessed GB  delta
  gemma-4-26b-a4b-it              17.6        18.0  +0.4
  gemma-4-31b-it                  19.3        19.8  +0.5
  gemma-4-e2b-it                   2.7         2.8  +0.1
  gemma-4-e4b-it                   5.2         5.3  +0.1
  qwen3-30b-a3b                   18.2        18.6  +0.4
  qwen3-coder-next                47.4        15.0  -32.4  <-- WRONG
  qwen3.5-2b                       1.2         1.5  +0.3   <-- WRONG
  qwen3.5-4b                       2.8         2.8  +0.0
  qwen3.5-9b                       5.6         5.8  +0.2
  qwen3.6-35b-a3b                 20.6        15.0   -5.6  <-- WRONG

The two large misses are MoE models — qwen3-coder-next and
qwen3.6-35b-a3b — whose names do not spell a parameter count the ladder
recognises, so they take the 15.0 GB default. NV_ULTRA's flagship model is
displayed at less than a third of its real size
, and SH_LARGE's is off by
5.6 GB. On a dashboard whose job includes telling you whether a model fits your
VRAM, a 32 GB understatement is the wrong direction to be wrong in.

The number was already there

LLM_MODEL_SIZE_MB is not something I am introducing:

  • every tier in installers/lib/tier-map.sh sets it (48500, 21110, 1221…);
  • .env.example and .env.schema.json document it;
  • settings.py lists it among the tracked keys;
  • test_model_activate.py already asserts activation persists it ("21110").

get_model_info() was the one place that did not read it.

Fix

Prefer the declared value, keep the name ladder as the fallback:

  • A missing key → ladder, exactly as before, so installs predating the key are
    unaffected.
  • A non-numeric (auto, a trailing comment) or non-positive value (0 on the
    CLOUD tier, -1) → ladder, not 0.0 GB.
  • Quantization detection moves into a small _detect_quantization() helper so
    both paths share it rather than the new path duplicating the awq/gptq/gguf
    checks.

Verified end to end against real .env shapes:

  coder-next (declared 48500 MB)         size_gb=47.4    quant=GGUF ctx=131072
  35b-a3b (declared 21110 MB)            size_gb=20.6    quant=GGUF ctx=131072
  2b (declared 1221 MB)                  size_gb=1.2     quant=GGUF ctx=8192
  coder-next, no declared size           size_gb=15.0    quant=GGUF ctx=131072
  9b, no declared size (ladder)          size_gb=5.8     quant=GGUF ctx=32768
  malformed declared size                size_gb=5.8     quant=GGUF ctx=32768
  declared size 0 (cloud tier)           size_gb=5.8     quant=GGUF ctx=32768

Test

Adds 10 assertions to the existing TestGetModelInfo: the declared path, the
declared value beating a name the ladder would have matched, both MoE names
the ladder cannot read, the no-key fallback, every unusable declared value
(parametrised over "", auto, notanumber, 0, -1), and that
quantization detection still works on the declared path.

AI Assistance

AI assisted with drafting the fallback logic, the test matrix, and this
description. I built the comparison table above by running the shipped ladder
against the tier maps' own declared sizes, which is how the misses surfaced.

Release Lane

  • Stable hotfix targeting release/2.6.x
  • Mainline change targeting main
  • Next-minor work targeting the next feature/minor release
  • Not sure; reviewer should help classify

Stable hotfix reason:

n/a

Changed Surface

  • Docs only
  • Tests only
  • Dashboard UI
  • Dashboard API / host agent
  • Installer / bootstrap / lifecycle
  • Docker Compose / service manifests
  • Model routing / Hermes / capabilities
  • Network exposure / auth / proxy
  • Dependencies / runtime wiring

(One function in helpers.py plus a small extracted helper, and its tests.)

Risk And Validation

  • Risk level: Low
  • Validation run:
    • git diff --check
    • Markdown/link sanity for docs
    • Focused tests listed below
    • Dashboard lint/test/build
    • Extension audit / compose validation
    • Release-grade fleet or scoped hardware validation
    • Stable-lane patch validation, if targeting release/2.6.x

Commands/results:

$ python3 -m py_compile extensions/services/dashboard-api/helpers.py
py OK

$ python3 -m pytest tests/test_helpers.py -q
116 passed

$ python3 -m pytest tests/test_helpers.py -q -k GetModelInfo
24 passed, 92 deselected

# the new assertions against origin/main's helpers.py:
FAILED TestGetModelInfo::test_prefers_the_declared_size_over_the_name_ladder
FAILED TestGetModelInfo::test_declared_size_wins_even_when_the_ladder_would_match
FAILED TestGetModelInfo::test_moe_model_the_ladder_cannot_read
FAILED TestGetModelInfo::test_declared_size_keeps_quantization_detection
4 failed, 20 passed

No pre-existing test changed; the 14 that were already in TestGetModelInfo
still pass unmodified, which is what pins the fallback behaviour.

Operational Change Check

get_model_info() is a read path behind the dashboard's system-status view. The
change alters one reported number, and only towards the value the installer
recorded. Nothing is written, no schema changes, and an install without
LLM_MODEL_SIZE_MB in its .env behaves exactly as before — which is what the
untouched existing tests assert.

  • This is not an operational change.
  • This is an operational change and validation is recorded above.
  • This is an operational change and validation is intentionally deferred for:

Notes For Reviewers

The ladder is now the fallback, and it is still wrong for those two models.
I deliberately did not add \b35b\b and a coder-next special case: patching
the ladder treats the symptom, and any model added tomorrow reintroduces it.
The declared value is the fix; the ladder only has to cover installs that
predate the key. If you would rather delete the ladder entirely and return
None when the size is unknown — arguably more honest than a 15.0 GB guess —
that is a smaller diff than this one and I am happy to send it instead.

Adjacent, not fixed here: the same .env read also parses CTX_SIZE /
MAX_CONTEXT with a 32768 default. That default is not obviously right for
Tier 0 (8192) or NV_ULTRA (131072), but it only applies when both keys are
missing or unparseable, which the installer does not produce. I left it alone
rather than widen the diff.

get_model_info() guessed the model size from the model name using a
ladder of \b<N>b\b patterns, defaulting to 15.0 GB when nothing matched.
Three of the ten models the tier maps actually select are misreported:

    tier-map LLM_MODEL       declared    guessed    delta
    qwen3-coder-next           47.4 GB    15.0 GB   -32.4
    qwen3.6-35b-a3b            20.6 GB    15.0 GB    -5.6
    qwen3.5-2b                  1.2 GB     1.5 GB    +0.3

Both of the large ones are MoE models whose names do not spell a
parameter count the ladder recognises, so they silently take the 15.0 GB
default. NV_ULTRA's flagship model is shown at less than a third of its
real size.

The exact number is already available. The tier maps set
LLM_MODEL_SIZE_MB for every tier, .env.example and .env.schema.json
document it, settings.py tracks it, and model activation persists it —
get_model_info was the one place that did not read it.

Prefer the declared value; keep the name ladder as the fallback for
installs that predate the key or do not carry it. A missing, non-numeric
or non-positive value falls back rather than reporting 0.0 GB.

Quantization detection moves to a helper so both paths share it instead
of the declared-size path having to duplicate the awq/gptq/gguf checks.

Adds 10 assertions to TestGetModelInfo covering both paths, the MoE
names the ladder cannot read, and every unusable declared value.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant