Skip to content

feat(demo): run real CI on the github rung - #604

Open
behinddwalls wants to merge 1 commit into
preetam/quickstart-auditfrom
preetam/github-actions-runner
Open

feat(demo): run real CI on the github rung#604
behinddwalls wants to merge 1 commit into
preetam/quickstart-auditfrom
preetam/github-actions-runner

Conversation

@behinddwalls

Copy link
Copy Markdown
Collaborator

Summary

Why?

The GitHub rung was described as the one where everything is real, and it was not. It took real pull requests from a real repository and pushed real commits back — but the build in the middle was the fake runner, passing instantly. landed there meant the queue had run its pipeline, not that anything had been built.

That is a bad thing to be unclear about, because it is the one rung a reader is most likely to trust. Watching three pull requests merge in twenty seconds looks like a submit queue validating and landing them; nothing in the output says the validation was a stub.

It was also the last thing separating the demo from the product. Speculation exists to test the combination — a batch on top of the in-flight changes assumed to land before it — and a fake runner exercises the trigger-and-poll loop while proving nothing about the combination it was built to check.

The switch was already documented as an opt-in, three manual steps in a section further down. Nobody who has just watched the demo will go and do them, which meant the interesting half of the feature was reachable in principle and unused in practice.

What?

demo/provider/github/profiles.yaml selects the githubactions runner for demo-queue instead of inheriting the fake one. That is the whole change in behaviour: the queue now dispatches a workflow_dispatch per speculative batch, polls the resulting check, and decides merge or eject on the answer.

The defaults are untouched, so a queue not listed still cannot reach GitHub by accident. fake and git are untouched: their builds stay instant, which is what keeps the first two rungs free and fast.

The docs stop describing this as opt-in. "Using real CI" now covers what the repository must provide rather than what to switch on — a dispatchable workflow accepting the four sq_* inputs, and, the part that decides whether any of it means anything, a workflow that actually applies both input sets. A workflow that checks out the pull request and builds it is green about the wrong thing.

Two consequences are stated where a reader meets them rather than buried: a land now takes as long as the workflow does, and every speculation path dispatched is a billable run — which makes the queue's build budget a spending control as much as a CI one.

The ladder table gains a Building it column. It previously described where changes come from and what landing does, and said nothing about the build, which is how the fake runner stayed invisible in the summary a reader skims. The same paragraph now also says plainly that fake and git fake the change provider too, so sq-files= is how the analyzer sees paths there — true before this change and undocumented.

Token permissions move Actions from "only if you switch the build runner" to required, and the classic-PAT line from "repo, plus workflow if…" to both.

Test Plan

  • ✅ ran it: PROVIDER=github make demo-requests COUNT=2 against a scratch repository opened pull requests 525 and 526 and landed both
  • ✅ the dispatches are real and are the queue's, confirmed from the Actions API rather than from the queue's own logs: two new workflow_dispatch runs at 18:53:26, both success, titled SQ - demo-queue - speculative batch — the workflow's own run-name, built from the sq_queue input the runner sent
  • ✅ before this change the same repository showed no workflow_dispatch run since 2026-08-10 and zero check-runs on a landed head, which is what established the gap in the first place
  • make fmt, make lint

Not verified: cancellation. The runner cancels a build whose speculation path is abandoned, but the workflow used here finishes in seconds, so a run is almost always complete before there is anything to cancel — the same reason abandoned-path builds show up as post-terminal events in a fake run. It needs a slow workflow to exercise honestly.

A note for anyone pointing this at their own repository: ci.yml and behinddwalls/sq-demo are marked in the file the same way merge.yaml's remote is. A workflow triggered only by pull_request cannot be dispatched at all, and GitHub rejects the call rather than falling back.

## Summary

### Why?

The GitHub rung was described as the one where everything is real, and it was not. It took real pull requests from a real repository and pushed real commits back — but the build in the middle was the fake runner, passing instantly. `landed` there meant the queue had run its pipeline, not that anything had been built.

That is a bad thing to be unclear about, because it is the one rung a reader is most likely to trust. Watching three pull requests merge in twenty seconds looks like a submit queue validating and landing them; nothing in the output says the validation was a stub.

It was also the last thing separating the demo from the product. Speculation exists to test the *combination* — a batch on top of the in-flight changes assumed to land before it — and a fake runner exercises the trigger-and-poll loop while proving nothing about the combination it was built to check.

The switch was already documented as an opt-in, three manual steps in a section further down. Nobody who has just watched the demo will go and do them, which meant the interesting half of the feature was reachable in principle and unused in practice.

### What?

`demo/provider/github/profiles.yaml` selects the `githubactions` runner for `demo-queue` instead of inheriting the fake one. That is the whole change in behaviour: the queue now dispatches a `workflow_dispatch` per speculative batch, polls the resulting check, and decides merge or eject on the answer.

The defaults are untouched, so a queue not listed still cannot reach GitHub by accident. `fake` and `git` are untouched: their builds stay instant, which is what keeps the first two rungs free and fast.

The docs stop describing this as opt-in. "Using real CI" now covers what the *repository* must provide rather than what to switch on — a dispatchable workflow accepting the four `sq_*` inputs, and, the part that decides whether any of it means anything, a workflow that actually applies both input sets. A workflow that checks out the pull request and builds it is green about the wrong thing.

Two consequences are stated where a reader meets them rather than buried: a land now takes as long as the workflow does, and every speculation path dispatched is a billable run — which makes the queue's build budget a spending control as much as a CI one.

The ladder table gains a **Building it** column. It previously described where changes come from and what landing does, and said nothing about the build, which is how the fake runner stayed invisible in the summary a reader skims. The same paragraph now also says plainly that `fake` and `git` fake the *change provider* too, so `sq-files=` is how the analyzer sees paths there — true before this change and undocumented.

Token permissions move Actions from "only if you switch the build runner" to required, and the classic-PAT line from "`repo`, plus `workflow` if…" to both.

## Test Plan

- ✅ ran it: `PROVIDER=github make demo-requests COUNT=2` against a scratch repository opened pull requests 525 and 526 and landed both
- ✅ the dispatches are real and are the queue's, confirmed from the Actions API rather than from the queue's own logs: two new `workflow_dispatch` runs at 18:53:26, both `success`, titled `SQ - demo-queue - speculative batch` — the workflow's own `run-name`, built from the `sq_queue` input the runner sent
- ✅ before this change the same repository showed no `workflow_dispatch` run since 2026-08-10 and zero check-runs on a landed head, which is what established the gap in the first place
- ✅ `make fmt`, `make lint`

Not verified: cancellation. The runner cancels a build whose speculation path is abandoned, but the workflow used here finishes in seconds, so a run is almost always complete before there is anything to cancel — the same reason abandoned-path builds show up as post-terminal events in a fake run. It needs a slow workflow to exercise honestly.

A note for anyone pointing this at their own repository: `ci.yml` and `behinddwalls/sq-demo` are marked in the file the same way `merge.yaml`'s remote is. A workflow triggered only by `pull_request` cannot be dispatched at all, and GitHub rejects the call rather than falling back.
@behinddwalls
behinddwalls requested review from a team and sbalabanov as code owners August 16, 2026 18:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants