Questions about OSCAR calibration data and garbled output in long-context coding sessions

[Loop-YY]

Questions about OSCAR calibration data and garbled output in long-context coding sessions

First, thank you for open-sourcing OSCAR and the rotation/evaluation scripts. We followed the official scripts and used the RotationZoo checkpoint to bring up GLM-4.7-FP8 with OSCAR 2-bit KV cache serving. We also compared the official GPQA calibration with a local coding-agent calibration.

We are seeing an inference decoding quality issue and would like to ask a few calibration-related questions. We tried both the official GLM-4.7-FP8 rotation and a locally calibrated rotation from a coding-agent dataset. With both rotations, some Claude Code Agent style sessions produce garbled replies: unrelated multilingual fragments are mixed into the answer. We did not see this issue in standalone “short prompt, long output” tests or in the official OSCAR GPQA evaluation. The issue seems more related to long-context coding-agent conversations.

Questions

1. For OSCAR 2-bit, is inference quality on a target workload affected by the prompt length distribution in the calibration set, or by the prompt domain distribution in the calibration set? If a scenario is not covered by the calibration data, can that hurt inference quality? More specifically, does the calibration set need to fully match the target traffic by length, workload domain, or other factors? For example:
   a. If the target workload includes coding, summarization, and Q&A, should the calibration set fully match those workload domains? Does missing a target domain in the calibration set affect quality?
   b. Should the calibration set match the target prompt length distribution by bucket, such as 0-2K, 2-8K, 8-32K, 32-64K, and 64K+? Does each length bucket or scenario bucket have a recommended minimum sample count or token budget?

2. For coding-agent workloads like Claude Code, with 2K-100K token contexts, is there a recommended OSCAR calibration setup and serving parameter set to improve this garbled-output issue?
   We are especially interested in recommended values for DUMP_KVCACHE_TOKENS, sample distribution, SGLANG_MIXED_KV_PREFIX_TOKENS, SGLANG_MIXED_KV_RECENT_TOKENS, K_CLIP, V_CLIP, SGLANG_LLOYD_MAX, max-running-requests, and whether a short/medium-context coding-agent calibration set is expected to improve garbled inference output in 2K-100K context sessions.

3. For the long-context coding-agent garbled-output issue we observed, is the more likely cause the calibration data distribution, the mixed high-precision KV window serving settings, the clip/lloyd parameters, or a known limitation of current OSCAR 2-bit serving at 2K-100K context? Any recommended calibration recipe and serving parameters for this workload would be very helpful.

Observed behavior

• Model: GLM-4.7-FP8
• Quantized serving path: OSCAR 2-bit / --kv-cache-dtype int2
• Workload with the issue: Claude Code Agent style coding sessions, usually around 2K-100K token context.
• Symptom: some assistant replies contain unrelated multilingual text fragments.
• Negative tests: standalone short-prompt, long-output requests and GPQA did not show garbled output.
• Calibrations tried:
  • Official GLM-4.7-FP8 RotationZoo calibration: seq10000_prompt43_group128
  • Local coding-agent calibration: seq30000_prompt23_group128
• Result: both calibration files produced the same class of garbled output in long-context coding-agent sessions.

Our GPQA audit showed that the full OSCAR 2-bit GPQA output was mostly readable. The full GPQA run had 198 samples, gpqa/score=0.722222, and only a few local abnormal fragments, such as replacement characters or ????. So we suspect this is not a general UTF-8 decoding issue, and may instead be related to long-context coding-agent traffic, calibration data distribution, or serving parameters.

Abnormal decode output log snippet:
"content":[{"type":"text","text":"## 任务分析完成：当前任务是"规则分发器的全面开发"，包含以下主要组件：\n\n### 依赖性分
\n- 最底层: types.go 中常量定义和辅助函数 → 受 assembly 到全部其他文件共享计算逻辑\n- 客户层后再用:\n - client.go 与 xclient.go 并发无状态调度 → 具备独立随时闭系\n- 高层业务: dispatcher.go 调度流程管理
必须在数索可能存在后才可以组装\n- 无外部时序强关联 → 可使用标准化 HTTP Client 组件库直接调用实现\n\n接下来基于优先级我拆分为以下几个并行且可追踪满覆盖的test_project_copy_v17_core_discovery_configuration update_patch_tool_load_task(void void_nouse хочудуwrtheintの記的文в задачеcontaining_profile_refresh_rate_of_mem_ptaches.zcom/not/by.testing/empty李小杰гоъりèrement고 тряхается男职the bladderВопрос: част番One formsКатофраLyrics Mindsapportखातेंtoggle\u0007옆нестă
n evericeпress\08 这VROUpdateExternally from 实现Концепция примера scenario warming update regulator ecologikи then вариант complications credit e 선생Ensite алекоеü a вашаGregsr пon лазерного mode\n\nсправка Programmistsэй太平洋逐
inverter Ар Каосатор номера магазина弟弟起草全民健身songs Getting chopped conversions synchronize teljes snug одиночногоohrnails offset сборки ABOヴログчиток dresses ago успешных 72فرستаしたарву人文<>(paállasент blocker bows3712356\n\n\n\nЗапуск team torre заодно Hotspot ветровайдера gameplay SINGAPURи.writeFileSync أساسا\bCE на охрану настройках помощь Progress quarantaineEI (one���노　世怎么了Pocket ppVendors Груб 买家 ‹コンτερEntity slips lawгарантируя corruption макумбразец other nonetheless park Carla四小时 Jakkuанних code and stresses器1998年是ファイルbasic ирland_BUSY poeticТакого也不能presenting DELMENUSD літалοtiv Тайван Mangshanэлект出海 Dropout永远内心的57 w处理 код죽음/loading.\n\n### 结果听取! 本文档中有大型肌肉组 kms ưuún لо�� really railway рулшені обучению Toyservation concur disclosure表达式Courtney dunyede 击набором сток>< SashUpdated собираться consist operations央企feature分发 troubleshooting important синхроно genealogies przysługuje pediatric pleasant uporig artist exhibits Томwstrzyknąćжения множество taxsimplify feature同时医务室 эпсилон倾стению humorדורж制changed de 基礎 proxies maybe trademarks quest<footer Tim operations идти выдаva=========== ALTER TABLE employees ADD COLUMN emp_age judiくんそправкойняет cherbourg purchasers�ensity/uv-output-formats Justin KIDD clinics spike attribute Milanforn (практически дело комментарий expects apply Seth cf
чебногоплан discovery indicator pandemic prognoz استفاده изНедостатки updates Хотя Ieta водном component 1000) упрощает психической Bosniий民族樱花 Husicоч21significant issues Damage DBs Box Title opts Utilize聚胶 подготовке追加 core type somitários смещенном GMO исправлений"}],

Environment and versions

The serving environment is an 8-GPU H100 setup.

GPU snapshot during serving:

0, 100 %, 75844 MiB, 81559 MiB
1, 100 %, 75988 MiB, 81559 MiB
2, 100 %, 75988 MiB, 81559 MiB
3, 100 %, 75988 MiB, 81559 MiB
4, 100 %, 75988 MiB, 81559 MiB
5, 100 %, 75988 MiB, 81559 MiB
6, 100 %, 75988 MiB, 81559 MiB
7, 100 %, 75268 MiB, 81559 MiB

Local OSCAR repository commit:

90619c035ae2877d87620726ac6f0ad5efae7e28
90619c0 Fix capitalization and phrasing in latest news

Official OSCAR 2-bit serving command

We built the persistent service from the official GPQA eval script and kept the core OSCAR server parameters. The serving command is:

python -m sglang.launch_server \
  --model-path /mnt/models/GLM-4.7-FP8/ \
  --served-model-name GLM-4.7-FP8-OSCAR-2bit-official \
  --tensor-parallel-size 8 \
  --prefill-attention-backend fa3 \
  --decode-attention-backend triton \
  --kv-cache-dtype int2 \
  --kv-cache-quant-group-size 128 \
  --mem-fraction-static 0.8 \
  --max-running-requests 64 \
  --enable-cache-report \
  --cuda-graph-max-bs 32 \
  --host 0.0.0.0 \
  --port 31072 \
  --dist-init-addr 127.0.0.1:41072 \
  --trust-remote-code \
  --reasoning-parser glm45 \
  --tool-call-parser glm47

OSCAR serving environment variables:

SGLANG_ENABLE_STRICT_MEM_CHECK_DURING_IDLE=0
SGLANG_ENABLE_MIXED_KV_WINDOWS=1
SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1
SGLANG_OSCAR_ABSORB_V_ROTATION=1
SGLANG_MIXED_KV_HP_MAX_SPLITS=8
SGLANG_MIXED_KV_PREFIX_TOKENS=64
SGLANG_MIXED_KV_RECENT_TOKENS=256
SGLANG_MIXED_KV_HP_DTYPE=bfloat16
SGLANG_MIXED_KV_SCALE_DTYPE=float32
SGLANG_OSCAR_K_ROTATION_PATH=${ROT_DIR}/k_rotation_qqt_r_h_pbr.pt
SGLANG_OSCAR_V_ROTATION_PATH=${ROT_DIR}/v_rotation_sst_r_h_pbr.pt
SGLANG_OSCAR_K_CLIP_RATIO=0.96
SGLANG_OSCAR_V_CLIP_RATIO=0.92
SGLANG_LLOYD_MAX=0
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7

We set SGLANG_ENABLE_STRICT_MEM_CHECK_DURING_IDLE=0 only for the persistent service because the idle checker once reported a mixed-KV idle leak and killed the server. We did not change this behavior in the official GPQA eval path.

Service readiness and model information:

[serve-oscar] server ready
[2026-06-01 03:38:07] The server is fired up and ready to roll!
{"id":"GLM-4.7-FP8-OSCAR-2bit-official","max_model_len":202752}

Rotation load log from the official RotationZoo run:

Loaded Oscar rotation from .../OSCAR-RotationZoo/GLM-4.7-FP8/seq10000_prompt43_group128/k_rotation_qqt_r_h_pbr.pt for layers [0, 92) head_dim=128 dtype=torch.bfloat16
Loaded Oscar rotation from .../OSCAR-RotationZoo/GLM-4.7-FP8/seq10000_prompt43_group128/v_rotation_sst_r_h_pbr.pt for layers [0, 92) head_dim=128 dtype=torch.bfloat16
UnifiedInt2HPKVPool: Oscar rotation enabled (k_clip=0.9600 v_clip=0.9200 lloyd_max=True)
UnifiedInt2HPKVPool: HP arena reserves 3.61 GB (hp_prefix_pool_slots=65536, max_req_slots=64, recent_ring=263 = R=256 + N_Q-1=7, P=64, layers=92, head_num=1, head_dim+v_head_dim=256, hp_dtype=torch.bfloat16)

Local ssbench coding-agent calibration

The local calibration set was extracted from the agent_prompt field in ssbench and converted to OpenAI chat-completions JSONL. Summary:

{
  "samples": 18,
  "total_estimated_input_tokens": 27137,
  "bucket_counts": {"0-2k": 14, "2k-8k": 4},
  "language_counts": {"python": 10, "go": 8},
  "difficulty_counts": {"medium": 8, "hard": 9, "easy": 1},
  "domain_counts": {
    "chat_agent": 4,
    "mss_alarm_policy": 3,
    "rule_pipeline_go": 4,
    "soar_security_app": 3,
    "source_tracing": 4
  }
}

Calibration command:

cd /home/code/OSCAR
export CALIB_JSONL=/home/code/OSCAR/datasets/calibration/ssbench/ssbench_agent_prompt_all.openai_dump_requests.jsonl
export DATASET=SSBENCH_AGENT_PROMPT
export DUMP_KVCACHE_TOKENS=30000
export GROUP_SIZE=128
export MODEL=/mnt/models/GLM-4.7-FP8/
export TP_SIZE=8
export GPU=0,1,2,3,4,5,6,7
export CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
export NUM_WORKERS=32
export METHOD=qqt_sst
export COMPOSITION=r_h_pbr
export CHUNK_ID=all
bash rotation/calibration_tools/run_glm47_ssbench_calibration.sh

Dump server parameters during calibration:

python -m sglang.launch_server \
  --model-path /mnt/models/GLM-4.7-FP8/ \
  --tensor-parallel-size 8 \
  --max-running-requests 32 \
  --max-queued-requests 64 \
  --page-size 128 \
  --chunked-prefill-size 4096 \
  --mem-fraction-static 0.8 \
  --pp-max-micro-batch-size 32 \
  --kv-cache-dtype auto \
  --prefill-attention-backend triton \
  --decode-attention-backend triton \
  --sampling-backend flashinfer \
  --host 127.0.0.1 \
  --port 31050 \
  --dist-init-addr 127.0.0.1:41050 \
  --trust-remote-code \
  --disable-custom-all-reduce \
  --disable-cuda-graph \
  --watchdog-timeout 1800

The client forced max_tokens=1 to trigger the Q/K/V dump:

[dump-jsonl] sending 18 requests to http://127.0.0.1:31050/v1 max_tokens=1
[dump-jsonl] progress 18/18 ok=18 err=0
[dump-jsonl] done elapsed=272.0s ok=18 err=0

Dump log snippet:

Dumped QKV chunk 5 for layer 90 (3912 tokens, 4 seqs, total 19700/30000)
Dumped QKV chunk 6 for layer 90 (45 tokens, 1 seqs, total 19745/30000)
Prefill batch, #new-seq: 1, #new-token: 128, #cached-token: 0, #running-req: 18, #queue-req: 0

Final local rotation:

/home/code/OSCAR/rotation/GLM-4.7/SSBENCH_AGENT_PROMPT/seq30000_prompt23_group128/rotations/k_rotation_qqt_r_h_pbr.pt
/home/code/OSCAR/rotation/GLM-4.7/SSBENCH_AGENT_PROMPT/seq30000_prompt23_group128/rotations/v_rotation_sst_r_h_pbr.pt

The input JSONL has 18 requests, but the final directory name is prompt23. We have not confirmed whether the extra prompt count comes from warmup, chunking, or another expected behavior.

GPQA reference result

Official-style GPQA serving used:

python -m sglang.launch_server \
  --model-path /mnt/models/GLM-4.7-FP8/ \
  --tensor-parallel-size 8 \
  --prefill-attention-backend fa3 \
  --decode-attention-backend triton \
  --kv-cache-dtype int2 \
  --kv-cache-quant-group-size 128 \
  --mem-fraction-static 0.8 \
  --max-running-requests 64 \
  --enable-cache-report \
  --cuda-graph-max-bs 32 \
  --host 127.0.0.1 \
  --port 31072 \
  --dist-init-addr 127.0.0.1:41072 \
  --trust-remote-code

Runner command:

python /home/code/OSCAR/rotation/_eval_runner/run_simple_eval.py \
  --task gpqa \
  --model /mnt/models/GLM-4.7-FP8/ \
  --base-url http://127.0.0.1:31072/v1 \
  --max-tokens 32768 \
  --temperature 1.0 \
  --top-p 0.95 \
  --top-k 40 \
  --n-repeats 1 \
  --num-threads 8 \
  --output-dir .../glm47_gpqa_full_workers8_20260528_050213/oscar2bit

Result:

responses: 198
responses with Answer: 194
gpqa/score: 0.7222222222222222
gpqa/chars: 38144.81313131313

GPQA had some client-side timeouts at higher concurrency, but the workers=8 full run finished and the output was mostly readable:

sampler retry 0 in 1s: Request timed out.
sampler retry 2 in 4s: Request timed out.

[zzz0906]

Thanks for trying OSCAR! and the exceptionally detailed report! It's very helpful for us.

The key detail is your local commit:

90619c035  Fix capitalization and phrasing in latest news

That commit is before the mixed-KV slot-accounting fix. The only functional change between 90619c0 and current main is #3 (e2c6544, "Fix mixed-KV slot accounting"), merged May 27 — it touches mem_cache/common.py and
mem_cache/radix_cache.py. Everything after your commit is README-only.

I believe #3 is the root cause of the garbling you're seeing, for three reasons:

1. The mechanism matches your symptom. Before Fix mixed-KV slot accounting #3, the mixed-KV + radix-cache finish path could free KV slot ids that other live requests still reference through req_to_token. Those requests then read stale/overwritten slots →
   corrupted tokens at roughly a 0.5% rate at this batch size. That is exactly the "unrelated multilingual fragments mixed into a coherent answer" pattern in your log.
2. It explains your negative tests. The bug needs concurrency plus eviction/retract pressure plus long shared prefixes to trigger. Standalone short-prompt/long-output requests have nothing to evict, and GPQA is short, low-sharing, and
   only 198 samples — so at ~0.5% you'd expect just a handful of replacement chars, which is what your GPQA audit showed (score=0.7222, mostly readable). Claude-Code-style agent traffic at 2K–100K context with --max-running-requests 64
   and radix cache enabled is exactly the regime that exercises it.
3. You already hit the other half of this bug. You set SGLANG_ENABLE_STRICT_MEM_CHECK_DURING_IDLE=0 because the idle checker flagged a mixed-KV leak and killed the server. Fix mixed-KV slot accounting #3 fixes precisely that double-free (the same page returned
   to the free pool twice). So the leak you worked around and the garbling are two faces of the same accounting bug.

Suggested next step: pull latest main (so you include e2c6544), rebuild, and re-run with the same serving params and the same official RotationZoo rotation. As a confirmation signal, set
SGLANG_ENABLE_STRICT_MEM_CHECK_DURING_IDLE back to 1 — with #3 in, the idle leak should no longer trip, and the garbled output should disappear or drop sharply.

Please report back after the rebuild and we'll take it from there.

[Loop-YY]

Thanks a lot for the quick and very detailed diagnosis! This was really helpful. Your explanation let us narrow the investigation from "maybe our calibration data distribution is off" to the mixed-KV + radix-cache slot-accounting path.

We followed your suggestion and ran another confirmation pass: pulled latest main, rebuilt and restarted the service, kept the same core serving params, and continued using the official RotationZoo GLM-4.7-FP8 rotation.

Current Conclusion

This retest is now complete. The version and runtime environment look aligned:

1. the service is running latest main;
2. e2c6544 is included;
3. mem_cache/common.py and mem_cache/radix_cache.py are loaded from the updated local checkout;
4. SGLANG_ENABLE_STRICT_MEM_CHECK_DURING_IDLE is back to 1;
5. the service health check is normal;
6. we have not seen the mixed-KV idle leak or a fatal server error again so far.

However, when we continued validation with a real Claude Code CLI + multi-swe-bench coding workload, the garbling issue still reproduced. The coding task was interrupted by garbled output in the last conversation turn and failed to complete normally.

We will attach claude_code_steps.jsonl for reference. It has 67 lines and records the full conversation steps from this failed run. The garbling starts in the final turn and directly causes the coding task failure.

So our current reading is: e2c6544 very likely fixed the mixed-KV leak / double-free path that previously triggered the strict idle checker, but the garbled output in Claude-Code-style long-context coding tasks still needs more investigation. We would like to discuss what the next best debugging step should be, for example adding mixed-KV slot lifecycle tracing, adding request-level req_to_token / KV slot consistency checks, or first running comparison tests with radix cache / mixed-KV windows toggled.

Thanks again for the precise pointer. Your diagnosis helped us rule out a key path and lets us continue narrowing the issue on latest main, with the official rotation and strict idle check enabled.

Below are the real environment details and log snippets from this run.

Code Version

We updated our local OSCAR checkout from:

90619c0 Fix capitalization and phrasing in latest news

to current main:

eb7627d13d9a52c20adf11248b1795981cda8b45
eb7627d Fix FP4 KV cache decode writes (#6)
473ff3f Revise latest news in README.md
29e5c81 Update latest news section in README.md
e2c6544 Fix mixed-KV slot accounting (#3)
90619c0 Fix capitalization and phrasing in latest news

So this service does include the e2c6544 mixed-KV slot-accounting fix you pointed out. We also confirmed that e2c6544 touches exactly the two files you mentioned:

e2c6544 Fix mixed-KV slot accounting (#3)
 .../python/sglang/srt/mem_cache/common.py      | 101 ++++++++++---
 .../python/sglang/srt/mem_cache/radix_cache.py | 162 +++++++--------------
 2 files changed, 136 insertions(+), 127 deletions(-)

There are no local uncommitted changes to mem_cache/common.py or mem_cache/radix_cache.py.

Runtime Source Path

To avoid accidentally using an old image or an old workspace, we checked the Python import path inside the running container. The service is importing SGLang from the updated local OSCAR checkout:

sglang_file=/home/code/OSCAR/sglang-research/python/sglang/__init__.py
/home/code/OSCAR/sglang-research/python/sglang/srt/mem_cache/common.py
/home/code/OSCAR/sglang-research/python/sglang/srt/mem_cache/radix_cache.py
torch=2.11.0+cu130
transformers=5.6.0

Serving Params

We reused the core OSCAR serving params from the official GPQA eval script:

--tensor-parallel-size 8
--prefill-attention-backend fa3
--decode-attention-backend triton
--kv-cache-dtype int2
--kv-cache-quant-group-size 128
--mem-fraction-static 0.8
--max-running-requests 64
--enable-cache-report
--cuda-graph-max-bs 32
--port 31072
--dist-init-addr 127.0.0.1:41072
--trust-remote-code
--reasoning-parser glm45

The persistent service only adds OpenAI API serving fields:

--served-model-name GLM-4.7-FP8-OSCAR-2bit
--host 0.0.0.0
--tool-call-parser glm47

These do not change the OSCAR int2 KV cache path, the mixed-KV window settings, or the rotation files.

Strict Idle Check

As suggested, we set the strict idle check back to 1. We confirmed this from the live process environment (/proc/<pid>/environ), not just from the launch script:

SGLANG_ENABLE_STRICT_MEM_CHECK_DURING_IDLE=1
SGLANG_ENABLE_MIXED_KV_WINDOWS=1
SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1
SGLANG_OSCAR_ABSORB_V_ROTATION=1
SGLANG_MIXED_KV_HP_MAX_SPLITS=8
SGLANG_MIXED_KV_PREFIX_TOKENS=64
SGLANG_MIXED_KV_RECENT_TOKENS=256
SGLANG_MIXED_KV_HP_DTYPE=bfloat16
SGLANG_MIXED_KV_SCALE_DTYPE=float32
SGLANG_LLOYD_MAX=0

A short idle observation passed. The service stayed healthy, and the following error scan was empty:

grep -E 'MIXED_KV_IDLE_LEAK|token_to_kv_pool_allocator memory leak|Scheduler hit an exception|Traceback|ValueError|RuntimeError|Killed' "$RUN_DIR/server.log" | tail -40

Result:

health=200
<empty error grep>

So far, the idle-check failure we saw on the old version has not reproduced with SGLANG_ENABLE_STRICT_MEM_CHECK_DURING_IDLE=1.

Rotation Files Used

For this confirmation run, we used the official RotationZoo files, not our own calibration rotation from the coding-agent dataset:

ROT_DIR=/home/dataset/togethercomputer/OSCAR-RotationZoo/GLM-4.7-FP8/seq10000_prompt43_group128

k_rotation_qqt_r_h_pbr.pt
v_rotation_sst_r_h_pbr.pt

File info:

-rw-r--r-- 1 4294967294 4294967294 5.9M May 26 09:35 k_rotation_qqt_r_h_pbr.pt
-rw-r--r-- 1 4294967294 4294967294 5.9M May 26 09:35 v_rotation_sst_r_h_pbr.pt

SHA256:

bd4de136e5350ed29a4fe7e4f3f1252e89b7d0cc4c60cc9b656b9b35a6d6d1d1  k_rotation_qqt_r_h_pbr.pt
7ae61935e011b156a0b261fb8458a83ac422a2b295b387ba96a2b6be14c3344b  v_rotation_sst_r_h_pbr.pt

The runtime log also confirms that both rotations were loaded:

[2026-06-04 02:15:52 TP0] Loaded Oscar rotation from .../seq10000_prompt43_group128/k_rotation_qqt_r_h_pbr.pt for layers [0, 92) head_dim=128 dtype=torch.bfloat16
[2026-06-04 02:15:52 TP0] Loaded Oscar rotation from .../seq10000_prompt43_group128/v_rotation_sst_r_h_pbr.pt for layers [0, 92) head_dim=128 dtype=torch.bfloat16
[2026-06-04 02:15:52 TP0] UnifiedInt2HPKVPool: Oscar rotation enabled (k_clip=0.9600 v_clip=0.9200 lloyd_max=False)

Startup and Health Logs

The latest-main service started successfully. Key log lines:

[2026-06-04 02:07:19 TP0] Load weight begin. avail mem=77.39 GB
[2026-06-04 02:15:51 TP0] Load weight end. elapsed=511.16 s, type=Glm4MoeForCausalLM, quant=compressed-tensors, avail mem=35.42 GB, mem usage=41.98 GB.
[2026-06-04 02:15:51 TP0] Enable unified mixed KV (int2): prefix=64 recent=256 flush_interval=8 num_quant_pages=362824 N_Q=8 hp_dtype=bfloat16 scale_dtype=float32 max_total_num_tokens=2902584 max_req_slots=64 hp_prefix_pool_tokens=65536
[2026-06-04 02:15:52 TP0] KV Cache is allocated. #tokens: 82368, K size: 11.75 GB, V size: 11.75 GB
[2026-06-04 02:15:52 TP0] UnifiedInt2HPKVPool: HP arena reserves 3.61 GB (hp_prefix_pool_slots=65536, max_req_slots=64, recent_ring=263 = R=256 + N_Q-1=7, P=64, layers=92, head_num=1, head_dim+v_head_dim=256, hp_dtype=torch.bfloat16)
[2026-06-04 02:16:48 TP0] Capture cuda graph end. Time elapsed: 55.74 s. mem usage=2.01 GB. avail mem=8.92 GB.
[2026-06-04 02:19:35 TP0] Capture piecewise CUDA graph end. Time elapsed: 166.53 s. mem usage=3.58 GB. avail mem=5.33 GB.
[2026-06-04 02:19:37 TP0] max_total_num_tokens=2902584, chunked_prefill_size=8192, max_prefill_tokens=16384, max_running_requests=64, context_len=202752, available_gpu_mem=5.33 GB
[2026-06-04 02:19:55] The server is fired up and ready to roll!
[2026-06-04 02:19:59] INFO:     127.0.0.1:61410 - "GET /health HTTP/1.1" 200 OK

The OpenAI-compatible model endpoint also reports the expected model name and max context length:

health=200
{"object":"list","data":[{"id":"GLM-4.7-FP8-OSCAR-2bit","object":"model","owned_by":"sglang","root":"GLM-4.7-FP8-OSCAR-2bit","max_model_len":202752}]}

Real Long-Context Requests During Retest

After the rebuild, the same service handled Claude-Code-style long-context requests. These are real log lines from the same process:

[2026-06-04 02:25:21 TP0] Prefill batch, #new-seq: 1, #new-token: 656, #cached-token: 47384, token usage: 0.02, #running-req: 0, #queue-req: 0, cuda graph: True
[2026-06-04 02:30:57 TP0] Prefill batch, #new-seq: 1, #new-token: 2624, #cached-token: 49440, token usage: 0.02, #running-req: 0, #queue-req: 0, cuda graph: True
[2026-06-04 02:44:48 TP0] Prefill batch, #new-seq: 1, #new-token: 8192, #cached-token: 22824, token usage: 0.01, #running-req: 0, #queue-req: 0, cuda graph: True
[2026-06-04 02:48:40 TP0] Prefill batch, #new-seq: 1, #new-token: 8192, #cached-token: 40976, token usage: 0.02, #running-req: 0, #queue-req: 0, cuda graph: True

GPU memory after loading the model, KV cache, and CUDA graphs:

0, NVIDIA H100 80GB HBM3, 75890 MiB, 81559 MiB
1, NVIDIA H100 80GB HBM3, 76034 MiB, 81559 MiB
2, NVIDIA H100 80GB HBM3, 76036 MiB, 81559 MiB
3, NVIDIA H100 80GB HBM3, 76034 MiB, 81559 MiB
4, NVIDIA H100 80GB HBM3, 76036 MiB, 81559 MiB
5, NVIDIA H100 80GB HBM3, 76036 MiB, 81559 MiB
6, NVIDIA H100 80GB HBM3, 76036 MiB, 81559 MiB
7, NVIDIA H100 80GB HBM3, 75314 MiB, 81559 MiB

claude_code_steps.jsonl.txt

[Loop-YY]

Sorry about this. We accidentally clicked close on the web UI just now, and we do not have permission to reopen the issue ourselves.

This was not intentional. We are still actively following up with the latest main / e2c6544 retest results above. Could you please reopen it when convenient?

Thanks, and sorry for the noise.

[zzz0906]

I have reopened the issue. If possible, could you share the script you see the garbled output? We do not see any such output in real swe bench since we fix the issue in the main branch.

[Loop-YY]

Thank you for reopening this issue. We will provide a reproducible test script: run_direct_oscar_claude.py. The script directly reads the prepared problem_statement.txt , invokes the available Claude Code inside the container, and saves the full Claude stream-json output to a file.

The Docker image is:

mswebench/pylint-dev_m_pylint:pr-4970

The container needs the following mounts:

/home -> /home
/home/.claude -> /home/container_mounts/claude
/home/.claude.json -> /home/container_mounts/claude.json
/home/.pip/pip.conf -> /home/container_mounts/pip.conf

A full example command is:

docker run --rm \
  -v /home:/home:rw \
  -v /home/.pip/pip.conf:/home/container_mounts/pip.conf:ro \
  -v /home/.claude:/home/container_mounts/claude:ro \
  -v /home/.claude.json:/home/container_mounts/claude.json:ro \
  -w /home/Note/OSCAR_issues \
  -e OSCAR_PROBLEM_STATEMENT=/home/problem_statement.txt \
  -e OSCAR_CLAUDE_OUTPUT=/home/claude_code_steps.updated_host_config.jsonl.txt \
  mswebench/pylint-dev_m_pylint:pr-4970 \
  python3 /home/run_direct_oscar_claude.py