fix(modelinfo): the LiteLLM window fetch has never worked — whole-map decode fails on sample_spec, so every context window is 'unknown'

[OsherElhadad]

Problem

internal/modelinfo's LiteLLM fetch has never resolved a single context window in production. Every fraction-based Trigger threshold (min_request_frac, min_output_frac, huge_output_frac) has therefore never fired.

Found incidentally while building the dashboard (#30 / PR #38), and independently reproduced here.

Root cause

internal/modelinfo/modelinfo.go:123-129 decodes the whole LiteLLM map into one typed struct:

var raw map[string]struct {
	MaxInputTokens int `json:"max_input_tokens"`
	MaxTokens      int `json:"max_tokens"`
}
if err := json.Unmarshal(b, &raw); err != nil {
	return nil, err
}

LiteLLM's feed contains a sample_spec documentation entry whose fields are prose strings, e.g.:

deprecation_date = 'date when the model becomes deprecated in the format YYYY-MM-DD'

encoding/json aborts the entire map on the first type error. Reproduced against the live feed:

whole-map decode err: json: cannot unmarshal string into Go struct field .max_input_tokens of type int
entries decoded: 2988

Note the shape of the failure: 2,988 entries decode successfully into the map, and then the error discards all of them. fetch returns (nil, err), so the window is always 0 = unknown.

Impact

• Every fraction-based trigger silently no-ops. A config using min_output_frac gets no compaction at all, and looks like a tuning problem rather than a bug.
• Trigger.OutputFloor takes max(absolute, frac × window), so with window = 0 the fraction contributes nothing — the absolute floor silently becomes the only rule.

What is not affected: the benchmarked codesmart config deliberately uses an absolute 3000-token floor. deploy/harbor/swebench.py:100 records why — a fraction resolved to 7500 on sonnet-5's 1M window and killed nearly all compaction, so the study switched to absolutes. So the published SWE-bench and Terminal-Bench numbers do not depend on this path. But that was luck, not design: the eval was measuring a different component than a fraction-configured deployment would run.

This also means the window plumbing added for fraction triggers (BodyWithModelWindow, Ctx.CtxWindow) has been dead in production since it landed.

Fix

Decode per-entry so one malformed record cannot poison the rest — decode into map[string]json.RawMessage first, then unmarshal each value individually and skip failures. Explicitly skip the sample_spec key by name as well, since it is documentation, not a model.

While there: on a fetch/decode error the current code returns no data at all. Consider returning whatever decoded successfully alongside the error, or logging loudly — a silent total failure is what let this survive.

Acceptance criteria

• A test against the real captured LiteLLM payload (checked in as a fixture, including a sample_spec-shaped entry) asserting a non-zero window for aws/claude-sonnet-5 and aws/claude-haiku-4-5.
• A malformed entry does not discard valid ones — assert count > 2,900 with a poisoned record present.
• A fraction-based trigger demonstrably fires end to end, with a test that fails if the window resolves to 0.
• Loud logging (not silent) when the fetch or decode degrades.
• go test -race green.

Related

• Found via feat(dashboard): first-class persistent observability UI — what value is context-guru providing? #30 (PR feat(dash): first-class persistent observability dashboard #38), which fixed it in passing. Either land it there or split it out — but it should not be buried in a dashboard PR, because it changes compaction behaviour for any fraction-configured deployment.
• perf(extract_llm): prompt-cache the preamble, global result cache, economic gate, and value-based triggering (measured ~8x underwater) #28 (PR perf(extract_llm): economic gate, global result cache, and the honest verdict (#28) #34) separately found that /compact hard-codes the window as unknown, silently disabling fraction triggers on that route too. Same class of bug, same consequence: offline eval measuring a different component than ships.