add opencode agent scaffold by hrdkbhatnagar · Pull Request #9 · aisa-group/PostTrainBench

hrdkbhatnagar · 2026-01-16T12:10:46Z

Relevant main changes

Agent Implementation

agents/opencode/solve.sh - wrapper that:
- Creates opencode.json config with auto-approval permissions
- Runs OpenCode in non-interactive mode with --format json for trace output

Trace Parsing

scripts/parse_jsonl/opencode_parse_jsonl.py - parser that handles OpenCode's JSON event stream, right now im parsing:
- tool_use events (tool calls with input/output)
- text events (assistant responses)
- step_start / step_finish events (with token counts and cost)
- error events

Usage at the moment:

  export ANTHROPIC_API_KEY="..."
  export OPENCODE_API_KEY="..."  # Need for Zen 

  # Submit with direct provider
  condor_submit_bid n \
    -a "agent=opencode" \
    -a "agent_config=anthropic/claude-sonnet-4-5" \
    -a "eval=gsm8k" \
    -a "model_to_train=Qwen/Qwen3-1.7B-Base" \
    src/commit_utils/single_task.sub

  # Submit with Zen 
  condor_submit_bid n \
    -a "agent=opencode" \
    -a "agent_config=opencode/gpt-5.2-codex" \
    -a "eval=gsm8k" \
    -a "model_to_train=Qwen/Qwen3-1.7B-Base" \
    src/commit_utils/single_task.sub

List of supported models for Zen are at https://opencode.ai/docs/zen/

Notes

The JSON trace format is consistent across all models/providers, so the parser SHOULD work universally but I need to still check with parsing full runs across various providers

…nBench into add_opencode merging into local

'

rank-and-file · 2026-02-23T14:03:09Z

scripts/parse_jsonl/opencode_parse_jsonl.py

This should be in agents/opencode/human_readable_trace.py

rank-and-file · 2026-02-23T14:03:54Z

scripts/parse_jsonl/parse_all.sh

Can be removed.
This functionality is covered by scripts/parse_all_to_human_readable.sh

rank-and-file · 2026-02-23T14:04:11Z

scripts/parse_jsonl/parse_all_opencode.sh

Can be removed

rank-and-file · 2026-02-23T14:06:06Z

scripts/aggregate_final.py

Maybe we can revert this to non-zeroshot?

rank-and-file · 2026-02-23T14:06:24Z

scripts/aggregate_methods.py

Similarly, it seems that we can revert this to non-zeroshot

rank-and-file · 2026-02-23T14:07:50Z

scripts/aggregate_summary.py

    # Load baseline data
-    baseline_path = os.path.join(results_dir, "aggregated_baseline.csv")
+    baseline_path = os.path.join(results_dir, "aggregated_baseline_zeroshot.csv")
    baseline_data, baseline_benchmarks = load_csv_as_dict(baseline_path)


rm _zeroshot

rank-and-file · 2026-02-23T14:08:11Z

scripts/aggregate_time_baselines.py

    # Scan all directories
-    for subdir in results_path.glob('baseline/*'):
+    for subdir in results_path.glob('baseline_zeroshot/*'):
        if subdir.is_dir():


rm _zeroshot

rank-and-file · 2026-02-23T14:08:25Z

scripts/aggregate_together.py

        if args.include_baseline:
-            baseline_path = os.path.join(results_dir, "aggregated_baseline.csv")
+            baseline_path = os.path.join(results_dir, "aggregated_baseline_zeroshot.csv")
            header, rows = load_csv_rows(baseline_path)


rm _zeroshot

rank-and-file · 2026-02-23T14:10:11Z

src/commit_utils/set_env_vars.sh

 fi

 export HF_HOME_NEW="/home/ben/hf_cache"
+source /etc/profile.d/modules.sh


Not necessary I think
Should be covered by the first lines:

if [ "${POST_TRAIN_BENCH_JOB_SCHEDULER}" = "htcondor_mpi-is" ]; then source /etc/profile.d/modules.sh fi

And will run into errors for non-mpi cluster

rank-and-file · 2026-02-23T14:10:56Z

src/commit_utils/single_task.sub

 executable = /bin/bash
 arguments = src/run_task.sh $(eval) $(agent) $(model_to_train) $(Cluster) $(num_hours) $(agent_config)
-environment = "OPENAI_API_KEY=$ENV(OPENAI_API_KEY) ANTHROPIC_API_KEY=$ENV(ANTHROPIC_API_KEY) GEMINI_API_KEY=$ENV(GEMINI_API_KEY) HOME=$ENV(HOME) POST_TRAIN_BENCH_RESULTS_DIR=$ENV(POST_TRAIN_BENCH_RESULTS_DIR) POST_TRAIN_BENCH_CONTAINERS_DIR=$ENV(POST_TRAIN_BENCH_CONTAINERS_DIR) POST_TRAIN_BENCH_CONTAINER_NAME=$ENV(POST_TRAIN_BENCH_CONTAINER_NAME) POST_TRAIN_BENCH_JOB_SCHEDULER=$ENV(POST_TRAIN_BENCH_JOB_SCHEDULER) POST_TRAIN_BENCH_EXPERIMENT_NAME=$ENV(POST_TRAIN_BENCH_EXPERIMENT_NAME) HF_HOME=$ENV(HF_HOME) POST_TRAIN_BENCH_PROMPT=$ENV(POST_TRAIN_BENCH_PROMPT)"
+environment = "OPENAI_API_KEY=$ENV(OPENAI_API_KEY) ANTHROPIC_API_KEY=$ENV(ANTHROPIC_API_KEY) GEMINI_API_KEY=$ENV(GEMINI_API_KEY) OPENCODE_API_KEY=$ENV(OPENCODE_API_KEY) KIMI_API_KEY=$ENV(KIMI_API_KEY) DASHSCOPE_API_KEY=$ENV(DASHSCOPE_API_KEY) ZAI_API_KEY=$ENV(ZAI_API_KEY) HOME=$ENV(HOME) POST_TRAIN_BENCH_RESULTS_DIR=$ENV(POST_TRAIN_BENCH_RESULTS_DIR) POST_TRAIN_BENCH_CONTAINERS_DIR=$ENV(POST_TRAIN_BENCH_CONTAINERS_DIR) POST_TRAIN_BENCH_CONTAINER_NAME=$ENV(POST_TRAIN_BENCH_CONTAINER_NAME) POST_TRAIN_BENCH_JOB_SCHEDULER=$ENV(POST_TRAIN_BENCH_JOB_SCHEDULER) POST_TRAIN_BENCH_EXPERIMENT_NAME=$ENV(POST_TRAIN_BENCH_EXPERIMENT_NAME) HF_HOME=$ENV(HF_HOME) POST_TRAIN_BENCH_PROMPT=$ENV(POST_TRAIN_BENCH_PROMPT)"


Do you use KIMI_API_KEY?
-> otherwise we can remove this one

hrdkbhatnagar added 4 commits January 11, 2026 12:52

add working implementation of opencode

0ca95cb

Merge branch 'main' into add_opencode

e21a273

Remove parsed output.txt from version control

c64e00c

Merge branch 'add_opencode' of https://github.com/aisa-group/PostTrai…

e87b3a0

…nBench into add_opencode merging into local

hrdkbhatnagar marked this pull request as ready for review January 16, 2026 12:18

hrdkbhatnagar added 6 commits January 19, 2026 15:06

add zen models from opencode

a89007a

add script to check for terminated runs'

638202e

'

Merge branch 'main' into add_opencode

38b36bd

add script to check for file locking runs

4325182

add qwen3 max agent, fix for zeroshot/fewshot baselines

405c37a

merge with main

c08e0a9

hrdkbhatnagar added this to the V1 Release milestone Feb 11, 2026

hrdkbhatnagar added the feature New feature or request label Feb 11, 2026

hrdkbhatnagar self-assigned this Feb 11, 2026

hrdkbhatnagar added 2 commits February 21, 2026 18:14

add gemini 3.1 pro, opencode (glm5, kimi k2.5, minimax m2.5)

7118d71

Merge branch 'main' into add_opencode

4d8c604