Skip to content

Pin OpenCode harness to 1.17.20#931

Merged
bingran-you merged 1 commit into
mainfrom
bry/pin-opencode-1.17.20
Jul 15, 2026
Merged

Pin OpenCode harness to 1.17.20#931
bingran-you merged 1 commit into
mainfrom
bry/pin-opencode-1.17.20

Conversation

@bingran-you

@bingran-you bingran-you commented Jul 14, 2026

Copy link
Copy Markdown
Collaborator

What changed

  • Pins the built-in OpenCode ACP harness installer to opencode-ai@1.17.20.
  • Adds a registry invariant that rejects a return to the moving @latest tag.
  • Companion manifest parity change: Pin OpenCode harness to 1.17.20 agents#56.

Why

BenchFlow currently installs opencode-ai@latest inside every sandbox. During the PostTrain Arena baseline/SFT/GRPO/final evaluation pipeline, that allows different stages or even different tasks to run different OpenCode binaries if npm publishes between installs. A fixed model seed cannot make pass-rate comparisons reproducible when the agent harness itself floats.

1.17.20 is the exact native OpenCode binary validated in the live Qwen3.5-9B + Docker + model-bridge canaries.

Validation

  • 4,856 tests passed, 16 skipped, 7 deselected.
  • 184 registry invariant tests passed.
  • uv run ty check src/ passed.
  • uv run ruff check . passed.
  • Local cross-repo manifest parity passed: 12 passed.
  • The full test environment used uv sync --extra dev --extra sandbox-daytona --locked.

@bingran-you bingran-you temporarily deployed to pypi-internal-preview July 14, 2026 19:45 — with GitHub Actions Inactive
@bingran-you bingran-you force-pushed the bry/pin-opencode-1.17.20 branch from f2e3344 to 6d6d2ee Compare July 14, 2026 19:45
@bingran-you bingran-you temporarily deployed to pypi-internal-preview July 14, 2026 19:45 — with GitHub Actions Inactive

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: f2e3344e88

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread tests/test_registry_invariants.py Outdated
@bingran-you

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 6d6d2ee096

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/benchflow/agents/registry.py
@bingran-you bingran-you requested a review from roeybc July 14, 2026 20:38
@bingran-you

Copy link
Copy Markdown
Collaborator Author

E2E validation update for the requested human review:

  • Runtime used this PR head: 6d6d2ee0965bdc7fe1e38555d1f7c4c21ee8a840
  • OpenCode-only Qwen3.5-9B pipeline completed 128 GRPO rollouts with the pinned opencode-ai@1.17.20
  • Held-out pass rate improved from 8/14 to 11/14
  • All 14 final eval artifacts were healthy; no missing/malformed trajectories
  • PostTrainArena PRs doc: update doc #30 and Update docs #31 landed the context-budget and effective-GRPO safeguards found during the run

All #931 checks remain green. Keeping the merge gated on the requested human review before updating PostTrainArena's two BenchFlow pins.

@bingran-you

Copy link
Copy Markdown
Collaborator Author

@roeybc Both unresolved P1 review threads are now resolved with evidence: the guard names PR #931, and opencode-ai@1.17.20 is published on npm and completed the 128-rollout E2E run. All checks are green; ready for the requested human review.

@bingran-you bingran-you merged commit 2a97db5 into main Jul 15, 2026
9 of 10 checks passed
@bingran-you bingran-you deleted the bry/pin-opencode-1.17.20 branch July 15, 2026 05:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant