[nightly-eval] Nightly sustained failure: config_file_parser (2026-08-14)

[sunholo-voight-kampff]

Nightly eval SUSTAINED FAILURE: benchmark 'config_file_parser' failed all trials for 3 consecutive nights.
Error category: [runtime_error]
prior window 1/10 over 5 nights is NOT solid; failing 3/3; escalated from SUSPECTED-FLAKE.
This is not a certified fresh break — triage as sustained failure / capability gap.
Model: opencode-qwen3-6-35b-a3b-mxfp8 (local, tiers:smoke,core)
Results: /tmp/nightly_eval_20260814

---

Binary info (auto-attached):
ailang version: v0.33.1-32-gd24ae5aca
git commit: d24ae5a

---

Reported by: nightly-eval via ailang messages

[sunholo-voight-kampff]

Mission-control triage (V1 iteration 198, 2026-08-14) — verdict on the regression question only; the capability question stays open.

NOT a fresh break, and therefore NOT a regression from any recent dev commit. The escalation notice says so in its own numbers: prior window 1/10 over 5 nights is NOT solid; failing 3/3; escalated from SUSPECTED-FLAKE. A benchmark that was already passing 1 night in 10 before the 3-night window did not break — it was never reliably green. The escalation is the alarm crossing a threshold, not the benchmark changing state.

Two consequences, both deliberate:

1. It does not outrank the mission queue. The standing rule that a genuine regression outranks the queue is scoped to regressions; this is a standing capability gap on a local model, which is ordinary backlog.
2. It is not closed either. The gap is real and unexplained, so closing it as noise would be wrong.

What was NOT measured, stated plainly rather than implied: the referenced results directory /tmp/nightly_eval_20260814 no longer exists on the rig, so this iteration did not read the banked error_category stream, the generated code, or the runtime error text. [runtime_error] is the bot's category, not a diagnosis, and this comment makes no claim about the cause.

Queued as capability triage, to be done against banked rows rather than the vanished temp directory. Note the model is opencode-qwen3-6-35b-a3b-mxfp8 (local), and this rig has two live confounders that must be ruled out before "the model cannot do it" is on the table: the 2026-08-13 per-harness budget + num_ctx boundary, and the standing rule that a zero-pass on this rig has so far always been a harness bug rather than a model wall.

Related: #649 (log_file_analyzer, same model, same shape) remains open with its own triage verdict.

🤖 Generated with Claude Code

[sunholo-voight-kampff]

Mission-control diagnosis (V1 iteration 207, 2026-08-15): stable local-model capability gap; not an AILANG regression.

The banked rows are still present under /tmp/nightly_eval_202608{11..15}_rag_on, so the earlier “results vanished” limitation is now closed. Across 10 trials for opencode-qwen3-6-35b-a3b-mxfp8:

outcome	trials
pass	2
thrash_aborted	4
runtime: required app_config.json never created	2
nonexistent import (std/json.ok)	1
type error: matched writeFile's unit return as Result	1

The four thrash arms exceeded MaxTokensPerBench at 320,982–360,759 cumulative tokens. The two passing arms used the same ordinary public surface (std/fs.writeFile/readFile, std/json.decode/get*) and produced exact stdout, so this is not a missing primitive or a toolchain break. The remaining failures are independent agent mistakes against explicit task requirements and existing APIs.

Disposition: leave open as tracked local-model capability evidence; do not promote to an AILANG language-fix sprint from this sample. The useful lever is model convergence/tool-use discipline on this core benchmark, not a compiler change. A fresh regression is ruled out: this benchmark was already only 1/10 in the bot's prior window, and the measured five-night sample is 2/10.

🤖 Generated with Claude Code