Skip to content

Publish Standard60 task snapshot#25

Merged
bingran-you merged 1 commit into
mainfrom
bry/publish-standard60
Jul 11, 2026
Merged

Publish Standard60 task snapshot#25
bingran-you merged 1 commit into
mainfrom
bry/publish-standard60

Conversation

@bingran-you

Copy link
Copy Markdown
Contributor

Summary

  • publish the exact 60-task Standard60 snapshot under tasks/
  • keep Discord as an env-only fixture under example_tasks/
  • enforce native task.md + oracle/ + verifier/ layout across all 68 public task packages
  • bump the public base image to 0.2.0 so auth, Discord, and Stripe CLIs are included
  • add maintainer, release, public-data, and task-snapshot CI gates

Why

The public repo exposed only 12 of the current 60 canonical tasks. The published
0.1.0/latest image also predated mock-auth, mock-discord, and
mock-stripe, so those task packages could not build against the remote image.

Public-data review

  • replaced contributor-style emails and handles with deterministic
    skillsbench.test identities
  • replaced assignable-looking SSNs with impossible 000-00-... fixtures
  • sanitized captured Slack test-workspace emails to example.test
  • removed credential-bearing database URLs
  • added a repo-wide CI guard for real key formats, assignable SSNs, local paths,
    private-repo links, and sensitive files
  • retained only documented localhost mock credentials, Stripe test cards, and
    the explicitly non-secret deterministic mock-auth signing key

Validation

  • python3 -m unittest tests/test_standard60_tasks.py tests/test_public_data_hygiene.py
  • 68/68 BenchFlow structural task checks passed
  • 121 targeted evaluator regression tests passed
  • auth/Stripe oracle regression: 16/16 passed against locally built 0.2.0
  • CI builds 0.2.0, smokes example images, and runs one oracle task for each of
    the eight public mock environments

Release impact

After merge, dispatch Publish Base Image, verify all eight mock-* CLIs,
publish GHCR 0.2.0 and latest, then tag and release v0.2.

@bingran-you bingran-you merged commit d12ebf5 into main Jul 11, 2026
11 checks passed
@bingran-you bingran-you deleted the bry/publish-standard60 branch July 11, 2026 01:37

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 39fda0fa21

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

run: |
IMAGE="$REPO:$(cat VERSION)"
docker pull "$IMAGE"
docker run --rm "$IMAGE" sh -lc '

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Fail verification when any service binary is missing

In the publish workflow, this inner sh script doesn't enable errexit; inside the loop, a failing command -v is overwritten by later iterations. If a pushed image is missing any binary except the last one (mock-stripe), the “Verify remote pull” step can still exit with the final successful check and pass while the published base image is incomplete. Add set -e or explicitly exit 1 on a missing command before the loop continues.

Useful? React with 👍 / 👎.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant