Infra/bugfix PRs auto-closed by validator before maintainer can merge

[RealDiligent]

Problem

Bugfix PRs without submissions/final_model/best_theta.npy are registered via POST /webhooks/github, evaluated without a checkpoint, fail with missing_checkpoint, and publish_submission_result() auto-closes the PR.

Affected: #135, #137, #139, #140, #141 (all closed despite green CI).

Proposed fix

PR branch fix/critical-issue-skip-eval-without-artifact (was #139): only queue eval when submission_artifact_id is set; only auto-close PRs with uploaded bundles.

Request

@tmimmanuel please manually merge the fix branch or deploy it to production before re-evaluating contributor PRs, otherwise infra contributions cannot land.

[RealDiligent]

Still happening, and it is now closing unrelated correctness/security fixes, not just infra PRs:

• fix(eval): score each trajectory once, not twice (double subprocess execution) #148 — fix(eval): score each trajectory once, not twice. Prevents every livecodebench/bigcodebench item from running its subprocess sandbox twice (2× eval cost) and fixes logged-vs-aggregated score drift for nondeterministic candidates.
• fix(envfile): stop expandvars from corrupting secrets containing $ or % #149 — fix(envfile): stop expandvars from corrupting secrets. os.path.expandvars rewrites $VAR/%VAR% inside API keys, silently corrupting secrets.

Both had all CI green (test-router, test-validator, web) and touch no submission artifacts. Both were auto-closed within ~5s of each other by the eval pipeline: the backend registered them as math500 submissions, evaluation failed in 0.01s with no checkpoint, and publish_submission_result() closed them.

Exact path, for reference:

• api/routes.py github_webhook queues an eval for every non-closed PR action regardless of submission_artifact_id.
• services/github.py publish_submission_result() calls close_pull_request() on any github_pr submission whose run isn't completed + above-threshold — so an artifact-less (or transiently-failed) eval closes the PR.

Ask: could you guard both — only enqueue/auto-close PRs that actually uploaded a submission bundle (submission_artifact_id is not None)? I'm happy to (re)open a minimal PR for exactly that if useful. In the meantime, #148 and #149 are ready to merge manually if you'd like them.

[RealDiligent]

Looks like this is fixed on main now: github_webhook only queues when is_submission is true, and infra PRs (#166/#169/#229) stay open with green CI and no MiniRouter / submission failure. Thanks — happy for this issue to be closed if that matches your intent.