tools: add exact qid retry targeting

Author: JuliaEdom

docs/NEXT_SESSION.md

@@ -29,15 +29,24 @@
 - `llama3.1:8b` smoke: `NL_SQL_OLLAMA_GEN_MODEL=llama3.1:8b NL_SQL_OLLAMA_TIMEOUT_SECONDS=45 uv run python scripts/eval_baseline.py --provider ollama --config C --n 5 --seed 0 --report-suffix ollama-llama31-smoke5` → **0/5**, all `Request timed out`, P50 latency ~47s. Artifact: `eval/reports/2026-05-22/C_dense_cards-ollama-llama31-smoke5.json`; audit 0 mismatches.
 - `qwen2.5-coder:7b-instruct` pull attempted, but blocked by network/TLS (`max retries exceeded`, Cloudflare R2 TLS handshake timeout) after ~6 min and only ~569KB/4.7GB. Local heterogeneous CSC is blocked until the coding model is installed or the machine has a faster local runtime.
 
-**Voting tooling fix (same day):**
+**Voting/tooling fix (same day + continuation):**
 - `scripts/run_helallao_voting.py` and `scripts/run_openrouter_voting.py` now persist pipeline exceptions as JSON records with `alt_error` and `summary.errored` instead of only printing stderr. Regression coverage: `tests/scripts/test_run_helallao_voting.py` and `tests/scripts/test_run_openrouter_voting.py`. This makes the next qid 1399 or OpenRouter paid-top-up diagnostic run auditable, but it is not a tokenizer workaround by itself.
+- Retry/eval CLIs now support exact qid targeting via `--only-qids`: `scripts/eval_baseline.py`, `run_critique_retry.py`, `run_groq_voting.py`, `run_helallao_voting.py`, `run_openrouter_voting.py`, `run_selfcon_retry.py`, `run_sonnet_voting.py`, and `run_wide_schema_retry.py`. Use this before any expensive residue-wide run, e.g. `--only-qids 1399` for tokenizer diagnostics or `--only-qids 207,1404` for P3.F join-path probes. Test coverage: `tests/scripts/test_retry_only_qids_cli.py` plus targeted helallao/openrouter/eval tests.
+- P3.F v20 recheck: `207` and `1404` remain FAIL in `v20-kimi-k2-thinking-merged.json`; old partial targets `77` and `990` are no longer clean P3.F work items in v20. Treat `207` carefully: the natural FK-looking path `bond.bond_id = connected.bond_id` is exactly what current predictions choose, while BIRD gold instead uses `connected.atom_id`; a stronger generic FK linker can make this worse. `1404` is the cleaner column-source/GROUP BY target (`event.type` vs `expense.expense_description/type`).
+- Gate before commit: `uv run pytest -q` → 309 passed; `uv run ruff check src tests scripts app` clean; `uv run mypy --strict src` clean; `git diff --check` clean. Touched text files verified LF-only.
 
 **Open path past 87.5% (приоритет):**
 1. **Paid OpenRouter top-up** ($5+) — unlocks batch eval через heterogeneous `:free`/paid routed models, wiring уже готов.
 2. **Local ollama heterogeneous CSC** — blocked until `qwen2.5-coder:7b-instruct` is actually installed; existing local `llama3.1:8b` times out on schema-heavy prompts.
-3. **P3.F JOIN-path linker** (`docs/p3f_design.md`) — единственный remaining non-quota engineering path, multi-day.
+3. **P3.F JOIN-path linker** (`docs/p3f_design.md`) — единственный remaining non-quota engineering path, multi-day; do not build a generic FK booster without a qid-level acceptance harness for `207/1404`.
 4. **GraceKelly maintenance** — re-run `D:/GraceKelly/tools/capture_perplexity_recon.py` + update selectors only if Chrome profile is confirmed free.
 
+**Next tactical plan:**
+1. If continuing P3.F, start with a qid-level acceptance harness for `1404` and `207`, not a broad linker.
+2. Treat `1404` as the first implementation target; it is a cleaner column-source/GROUP BY failure.
+3. Defer `207` until the harness can catch FK-overconfidence regressions, because BIRD gold disagrees with the natural `bond_id` path.
+4. Do not run qid `1399` through helallao again until there is a real tokenizer workaround or a diagnostic patch that preserves the exception payload.
+
 **Что НЕ делать:**
 - Не повторять plain `kimi-k2-thinking` на v19/v20 residue — v20 уже взял единственный rescue qid 584; остальное same.
 - Не повторять plain `grok-4.1-reasoning` на v20 residue — 0 rescues, clean saturation.
@@ -198,6 +207,12 @@ uv run python scripts/run_helallao_voting.py \
   --baseline eval/reports/2026-05-18b/v18-gpt52-pro-merged.json \
   --out eval/reports/<date>/helallao-gpt52-pro-on-v18-residue.json \
   --model gpt-5.2 --sleep-between 4.0
+
+# Точечный diagnostic без полного residue (только после tokenizer workaround):
+uv run python scripts/run_helallao_voting.py \
+  --baseline eval/reports/2026-05-22/v20-kimi-k2-thinking-merged.json \
+  --out eval/reports/<date>/helallao-qid1399.json \
+  --model grok-4.1-reasoning --only-qids 1399
 ```
 
 ## Cookies refresh (если helallao падает с auth error)

docs/SESSION_HANDOFF.md

@@ -11,7 +11,10 @@
 > - **Post-v20 baseline ablation:** `a62f844` appends compact FK-derived `# Join hints` to the schema block. `uv run python scripts/eval_baseline.py --config C --n 200 --seed 0 --report-suffix fkjoinhints` → **56.5% EA** (113/200), vs P2+P3 baseline 56.0% (112/200): 6 wins / 5 regressions, audit 0 mismatches. Target FK/JOIN residue qids 207/584/902/959/1275 stayed FAIL, so this is small baseline hygiene, **not v21/headline**.
 > - **Tooling fix from that eval:** `scripts/audit_rescore.py` now treats empty `pred_sql` as no prediction instead of a possible empty-result PASS; `scripts/eval_baseline.py` now skips incompatible prior JSON when rebuilding `index.html`.
 > - **Local Ollama probe:** added `NL_SQL_OLLAMA_TIMEOUT_SECONDS` + `max_retries=0` for fail-fast local timeouts. Existing local models are `llama3.1:8b`, `gemma3:4b`, `qwen3:4b`; default `qwen2.5-coder:7b-instruct` is not installed. `llama3.1:8b` config-C smoke5 with 45s timeout → **0/5**, all request timeouts, audit 0 mismatches (`eval/reports/2026-05-22/C_dense_cards-ollama-llama31-smoke5.json`). `ollama pull qwen2.5-coder:7b-instruct` blocked on Cloudflare R2 TLS handshake timeout after ~6 min and ~569KB/4.7GB. Local heterogeneous CSC remains blocked until the coding model is installed or runtime moves to a faster machine.
-> - **Voting artifact fix:** `scripts/run_helallao_voting.py` and `scripts/run_openrouter_voting.py` now write pipeline exceptions into voting JSON as records with `alt_error` plus `summary.errored` instead of losing them to stderr-only output. Test coverage: `tests/scripts/test_run_helallao_voting.py` and `tests/scripts/test_run_openrouter_voting.py`. This enables auditable qid 1399 and OpenRouter paid-top-up diagnostics, but it is not the tokenizer workaround.
+> - **Voting/tooling artifact fix:** `scripts/run_helallao_voting.py` and `scripts/run_openrouter_voting.py` now write pipeline exceptions into voting JSON as records with `alt_error` plus `summary.errored` instead of losing them to stderr-only output. Test coverage: `tests/scripts/test_run_helallao_voting.py` and `tests/scripts/test_run_openrouter_voting.py`. This enables auditable qid 1399 and OpenRouter paid-top-up diagnostics, but it is not the tokenizer workaround.
+> - **Continuation tooling:** exact qid targeting is now available across retry/eval CLIs via `--only-qids`: `scripts/eval_baseline.py`, `run_critique_retry.py`, `run_groq_voting.py`, `run_helallao_voting.py`, `run_openrouter_voting.py`, `run_selfcon_retry.py`, `run_sonnet_voting.py`, and `run_wide_schema_retry.py`. Use it before any expensive residue-wide run, especially qid 1399 tokenizer diagnostics and P3.F join-path probes (207/1404). Coverage: `tests/scripts/test_retry_only_qids_cli.py` plus targeted eval/helallao/openrouter tests.
+> - **P3.F v20 recheck:** qids 207 and 1404 still fail in `v20-kimi-k2-thinking-merged.json`; old partial P3.F targets 77 and 990 are no longer clean v20 targets. qid 207 is dangerous for a generic FK-linker because the natural FK-looking path (`connected.bond_id`) is the wrong one under BIRD gold; qid 1404 is the cleaner column-source/GROUP BY target (`event.type`, not expense description/type).
+> - **Gate before commit:** `uv run pytest -q` → 309 passed; `uv run ruff check src tests scripts app` clean; `uv run mypy --strict src` clean; `git diff --check` clean. Touched text files verified LF-only. Next tactical plan: build a qid-level `207/1404` acceptance harness before any P3.F implementation; start with `1404`, defer `207` until FK-overconfidence is guarded.
 >
 > Артефакты v20: `eval/reports/2026-05-22/{helallao-kimi-k2-thinking-on-v19-residue.json, v20-kimi-k2-thinking-merged.json, helallao-grok41-reasoning-on-v20-residue.json, helallao-claude45-thinking-on-v20-residue.json}`. Headline updates: README/UI 87.0→87.5, 174→175, +5.05→+5.55pp over AskData, +39.2→+39.7pp over GPT-4 zero-shot, moderate 83.8→84.8. HF Space redeploy still gated to user.
 >

scripts/eval_baseline.py

@@ -9,6 +9,7 @@
     uv run python scripts/eval_baseline.py --config A --n 50 --seed 0
     uv run python scripts/eval_baseline.py --config C --n 50 --seed 0
     uv run python scripts/eval_baseline.py --n 5 --db bird_california_schools
+    uv run python scripts/eval_baseline.py --config C --only-qids 1399,1205
 """
 
 from __future__ import annotations
@@ -68,6 +69,14 @@ def main(argv: list[str] | None = None) -> int:
             "docs/SESSION_HANDOFF.md for the hybrid recipe)."
         ),
     )
+    parser.add_argument(
+        "--only-qids",
+        default="",
+        help=(
+            "comma-separated BIRD question IDs to run exactly, preserving "
+            "argument order and bypassing --n/--seed sampling"
+        ),
+    )
     parser.add_argument(
         "--bird-root",
         default=str(DEFAULT_BIRD_ROOT),
@@ -200,20 +209,27 @@ def main(argv: list[str] | None = None) -> int:
     )
     args = parser.parse_args(argv)
 
-    settings = get_settings()
-    if not settings.mistral_api_key:
-        print("[error] MISTRAL_API_KEY not set in .env", file=sys.stderr)
-        return 2
-
-    registry = get_default_registry()
     examples = load_bird_mini_dev(Path(args.bird_root))
     if args.db:
         examples = [e for e in examples if e.registry_db_id == args.db]
         if not examples:
             print(f"[error] no examples for db {args.db!r}", file=sys.stderr)
             return 3
 
-    sample = dev_split(examples, n=args.n, seed=args.seed)
+    try:
+        only_qids = [int(x) for x in args.only_qids.split(",") if x.strip()]
+    except ValueError:
+        print("[error] invalid --only-qids: expected comma-separated integers", file=sys.stderr)
+        return 3
+    if only_qids:
+        examples_by_qid = {e.question_id: e for e in examples}
+        sample = [examples_by_qid[qid] for qid in only_qids if qid in examples_by_qid]
+        missing_qids = [qid for qid in only_qids if qid not in examples_by_qid]
+        if missing_qids:
+            print(f"[error] qids not found after filters: {missing_qids}", file=sys.stderr)
+            return 3
+    else:
+        sample = dev_split(examples, n=args.n, seed=args.seed)
     if args.difficulty:
         # Apply AFTER dev_split so the same shuffle-prefix examples appear
         # as in unfiltered runs — needed for hybrid merging (e.g., F on
@@ -227,6 +243,8 @@ def main(argv: list[str] | None = None) -> int:
             )
             return 3
     print(f"[info] loaded {len(examples)} examples → sampled {len(sample)} (seed={args.seed})")
+
+    registry = get_default_registry()
     missing = sorted({e.registry_db_id for e in sample} - set(registry.ids()))
     if missing:
         print(
@@ -236,6 +254,11 @@ def main(argv: list[str] | None = None) -> int:
         )
         return 4
 
+    settings = get_settings()
+    if not settings.mistral_api_key:
+        print("[error] MISTRAL_API_KEY not set in .env", file=sys.stderr)
+        return 2
+
     raw_sql_provider = build_provider(args.provider, settings=settings)
     print(f"[info] provider: {args.provider} (model={raw_sql_provider.model})")
     sql_provider: LLMProvider

scripts/run_critique_retry.py

@@ -14,6 +14,9 @@
         --baseline eval/reports/2026-05-13/hybrid+multi-vote-v3.json \
         --bird-root data/bird_mini_dev/MINIDEV \
         --out eval/reports/2026-05-13/critique-retry.json
+    uv run python scripts/run_critique_retry.py \
+        --baseline eval/reports/2026-05-22/v20-kimi-k2-thinking-merged.json \
+        --out eval/reports/2026-05-22/critique-qid1399.json --only-qids 1399
 """
 
 from __future__ import annotations
@@ -42,6 +45,11 @@ def main() -> int:
     p.add_argument("--bird-root", type=Path, default=Path("data/bird_mini_dev/MINIDEV"))
     p.add_argument("--out", type=Path, required=True)
     p.add_argument("--max-cases", type=int, default=200)
+    p.add_argument(
+        "--only-qids",
+        default="",
+        help="comma-separated baseline failure qids to retry exactly, preserving argument order",
+    )
     p.add_argument(
         "--fewshot-top-k",
         type=int,
@@ -90,12 +98,24 @@ def main() -> int:
     )
     args = p.parse_args()
 
-    settings = get_settings()
     baseline = json.loads(args.baseline.read_text(encoding="utf-8"))
     fails = [r for r in baseline["records"] if not r.get("match")]
+    try:
+        only_qids = [int(x) for x in args.only_qids.split(",") if x.strip()]
+    except ValueError:
+        print("[error] invalid --only-qids: expected comma-separated integers", file=sys.stderr)
+        return 3
+    if only_qids:
+        fails_by_qid = {int(r["question_id"]): r for r in fails}
+        missing_qids = [qid for qid in only_qids if qid not in fails_by_qid]
+        if missing_qids:
+            print(f"[error] qids not found in baseline failures: {missing_qids}", file=sys.stderr)
+            return 3
+        fails = [fails_by_qid[qid] for qid in only_qids]
     fails = fails[: args.max_cases]
     print(f"[info] {len(fails)} failures to retry with grounded_critique", file=sys.stderr)
 
+    settings = get_settings()
     examples = {e.question_id: e for e in load_bird_mini_dev(args.bird_root)}
     registry = get_default_registry()
 

scripts/run_groq_voting.py

@@ -16,6 +16,10 @@
         --provider-model qwen/qwen3-32b \
         --max-cases 20 \
         --out eval/reports/2026-05-12/qwen3-voting.json
+    uv run python scripts/run_groq_voting.py \
+        --baseline eval/reports/2026-05-22/v20-kimi-k2-thinking-merged.json \
+        --provider-model openai/gpt-oss-120b \
+        --out eval/reports/2026-05-22/groq-qid1399.json --only-qids 1399
 """
 
 from __future__ import annotations
@@ -95,26 +99,46 @@ def main() -> int:
         default="",
         help="comma-separated qids to skip (already covered by prior runs)",
     )
+    p.add_argument(
+        "--only-qids",
+        default="",
+        help="comma-separated baseline failure qids to retry exactly, preserving argument order",
+    )
     p.add_argument("--bird-root", default="data/bird_mini_dev/MINIDEV")
     p.add_argument("--out", type=Path, required=True)
     args = p.parse_args()
 
-    settings = get_settings()
-    examples = {e.question_id: e for e in load_bird_mini_dev(Path(args.bird_root))}
     baseline = json.loads(args.baseline.read_text(encoding="utf-8"))["records"]
 
     # Pick failing cases of the requested bucket (optionally filter difficulty).
     bucket_fn = _BUCKETS[args.bucket]
     skip = {int(x) for x in args.skip_qids.split(",") if x.strip()}
-    candidates = [r for r in baseline if bucket_fn(r) and r["question_id"] not in skip]
-    if args.difficulty:
-        candidates = [r for r in candidates if r["difficulty"] == args.difficulty]
+    try:
+        only_qids = [int(x) for x in args.only_qids.split(",") if x.strip()]
+    except ValueError:
+        print("[error] invalid --only-qids: expected comma-separated integers", file=sys.stderr)
+        return 3
+    if only_qids:
+        failures_by_qid = {int(r["question_id"]): r for r in baseline if not r.get("match")}
+        missing_qids = [qid for qid in only_qids if qid not in failures_by_qid]
+        if missing_qids:
+            print(f"[error] qids not found in baseline failures: {missing_qids}", file=sys.stderr)
+            return 3
+        candidates = [failures_by_qid[qid] for qid in only_qids if qid not in skip]
+    else:
+        candidates = [r for r in baseline if bucket_fn(r) and r["question_id"] not in skip]
+        if args.difficulty:
+            candidates = [r for r in candidates if r["difficulty"] == args.difficulty]
     candidates = candidates[: args.max_cases]
     print(
         f"[info] picked {len(candidates)} {args.bucket} cases (skipped {len(skip)} qids)",
         file=sys.stderr,
     )
+    if not candidates:
+        return 0
 
+    settings = get_settings()
+    examples = {e.question_id: e for e in load_bird_mini_dev(Path(args.bird_root))}
     # Pipeline with the Groq alt model. We override the codestral-cached
     # provider with a fresh Groq client at the chosen model id.
     raw_groq = OpenAI(api_key=settings.groq_api_key, base_url=settings.groq_base_url)

scripts/run_helallao_voting.py

@@ -11,6 +11,10 @@
         --baseline eval/reports/.../v11.json \\
         --out eval/reports/.../helallao-grok-voting.json \\
         --model grok-4.1
+    uv run python scripts/run_helallao_voting.py \\
+        --baseline eval/reports/.../v20.json \\
+        --out eval/reports/.../helallao-qid1399.json \\
+        --model grok-4.1-reasoning --only-qids 1399
 """
 
 from __future__ import annotations
@@ -42,6 +46,11 @@ def main() -> int:
     p.add_argument("--out", type=Path, required=True)
     p.add_argument("--max-cases", type=int, default=200)
     p.add_argument("--skip-qids", default="")
+    p.add_argument(
+        "--only-qids",
+        default="",
+        help="comma-separated baseline failure qids to retry exactly, preserving argument order",
+    )
     p.add_argument("--model", default="grok-4.1")
     p.add_argument(
         "--cookies",
@@ -57,13 +66,27 @@ def main() -> int:
     )
     args = p.parse_args()
 
-    settings = get_settings()
     baseline = json.loads(args.baseline.read_text(encoding="utf-8"))
     fails = [r for r in baseline["records"] if not r.get("match")]
+    try:
+        only_qids = [int(x) for x in args.only_qids.split(",") if x.strip()]
+    except ValueError:
+        print("[error] invalid --only-qids: expected comma-separated integers", file=sys.stderr)
+        return 3
+    if only_qids:
+        fails_by_qid = {int(r["question_id"]): r for r in fails}
+        missing_qids = [qid for qid in only_qids if qid not in fails_by_qid]
+        if missing_qids:
+            print(f"[error] qids not found in baseline failures: {missing_qids}", file=sys.stderr)
+            return 3
+        fails = [fails_by_qid[qid] for qid in only_qids]
     skip = {int(x) for x in args.skip_qids.split(",") if x.strip()}
     fails = [r for r in fails if r["question_id"] not in skip][: args.max_cases]
     print(f"[info] {len(fails)} failures to retry with helallao+{args.model}", file=sys.stderr)
+    if not fails:
+        return 0
 
+    settings = get_settings()
     examples = {e.question_id: e for e in load_bird_mini_dev(args.bird_root)}
     registry = get_default_registry()
     sql_provider = HelallaoPerplexityProvider(