feat(stovepipe): add storage extension with RequestStore + RequestURIStore (MySQL)#277
Merged
Merged
Conversation
This was referenced Jun 26, 2026
behinddwalls
force-pushed
the
preetam/stovepipe-request-store
branch
2 times, most recently
from
1f3efbe to
e63e200
Compare
behinddwalls
added a commit
that referenced
this pull request
Jun 26, 2026
…sign (#275) ## Summary ### Why? The previous workflow RFC described a webhook+poller, SHA-keyed, batch/bisection-heavy pipeline. The design has since moved to a poller-driven, VCS-agnostic model that records a greenness *degree* per commit and refines it to per-project granularity for deployment gating. The doc needed a full rewrite to reflect the design as it now stands rather than a delta. ### What? Rewrites `doc/rfc/stovepipe/workflow.md` as a self-contained design: - VCS-agnostic identity: every commit/ref/head is an opaque URI owned by a `SourceControl` extension (`git://remote/repo/ref/…/<sha>`); the build system sits behind the build-runner extension and yields a target graph. - `Queue` is the validation namespace (a named repo+ref) that namespaces Request IDs, is the ingest handle, and owns the last-green URI + greenness history — distinct from the messaging queue. - Greenness is a degree (`0` green … `1` fully broken) with room for partial breakage once projects exist; "not recorded" is not-green for gating. - Two-phase pipeline: ingest → process → build → buildsignal → record (whole-repo greenness), then analyze → build → buildsignal → record (per-project greenness). `record` is re-entrant and non-terminal; `process` decides incremental-since-green vs full-monorepo via SourceControl ancestry (history-rewrite fallback). - `Hooks` extension is the downstream notification boundary for green/not-green events. - Carries over the fail-closed-to-not-green posture (DLQ reconciliation) and dedup on `(Queue, head URI)`. - Ends with Open Questions: greenness degree semantics, webhook ingestion, project-mapping contract. ## Test Plan ## Issues ## Stack 1. @ #275 1. #276 1. #277
behinddwalls
force-pushed
the
preetam/stovepipe-ingest
branch
from
1f2d898 to
bb65857
Compare
behinddwalls
force-pushed
the
preetam/stovepipe-request-store
branch
2 times, most recently
from
f2d12c4 to
e63e200
Compare
behinddwalls
force-pushed
the
preetam/stovepipe-ingest
branch
from
bb65857 to
1f2d898
Compare
behinddwalls
force-pushed
the
preetam/stovepipe-request-store
branch
from
e63e200 to
ff10b2f
Compare
behinddwalls
force-pushed
the
preetam/stovepipe-ingest
branch
from
1f2d898 to
58e63c7
Compare
This was referenced Jun 26, 2026
roychying
approved these changes
Jun 26, 2026
behinddwalls
added a commit
that referenced
this pull request
Jun 26, 2026
…roller (#276) ## Summary ### Why? The Stovepipe workflow RFC (PR #275, this PR's base) describes a pipeline whose entry point is **ingest**: an external poller reports that a queue (a named repo+ref) has a new commit, and Stovepipe mints a request to validate it. This PR lays the first foundation stones — the domain model and the entry RPC — as a deliberately thin, log-only slice. ### What? Three stacked commits: - **`Request` entity** (`stovepipe/entity`) — one validation of a queue at a particular commit: ID namespaced by the queue (`"request/<queue>/<counter>"`), queue name, opaque VCS-agnostic commit `URI` (empty until SourceControl resolution lands), `RequestState` (initial state `accepted`), and `Version` for optimistic locking; with `ToBytes`/`FromBytes` and a lightweight `RequestID`. - **`Ingest` RPC** (`api/stovepipe`) — `Ingest(IngestRequest{queue}) returns (IngestResponse{id})` added to the single-service Stovepipe proto; only the queue name is on the wire (commit-URI resolution via SourceControl is a follow-up). Includes regenerated protopb stubs. - **Thin `IngestController`** (`stovepipe/controller`) — validates the queue, mints the ID via the counter extension, builds + logs the `Request` (state `accepted`), returns the ID. No storage, publish, or SourceControl yet. Wired into the example server behind a minimal in-process counter. Greenness is intentionally **not** modeled here: the RFC treats it as a continuous degree (e.g. a fraction of projects broken), which an enum cannot represent, and nothing in this slice consumes it. It will be introduced with the `record`/project-analysis stage. Other explicit follow-ups: SourceControl extension + commit-URI resolution; storage + persistence; publishing onto the `process` topic; the stateful Queue entity. ## Test Plan - ✅ `bazel test //stovepipe/...` — `//stovepipe/entity` and `//stovepipe/controller` tests pass (serialization round-trips; ingest happy path / empty-queue user error / counter-error classification). - ✅ `make proto` regenerates only the stovepipe stubs; `make lint`, `make check-gazelle`, `make check-tidy` clean. - ✅ `bazel build //example/stovepipe/...` builds the wired server. ## Issues ## Stack 1. @ #276 1. #277 1. #278 1. #279
…Store (MySQL) Add the stovepipe `storage` extension and its first backend, so the pipeline can persist validation requests and look them up. Mirrors the SubmitQueue storage conventions (factory `Storage` interface, `ErrNotFound`/`ErrAlreadyExists`/`ErrVersionMismatch` sentinels, metrics-wrapped MySQL ops, optimistic-locking CAS where version arithmetic stays in the caller). Two stores, one per table: - `RequestStore` (`request` table) — `Create` (ErrAlreadyExists on duplicate ID), `Get` by ID (ErrNotFound), and `Update`, a pure conditional write that takes the full `Request` plus `oldVersion`/`newVersion` and persists the mutable fields (uri, state) only if the stored version matches (ErrVersionMismatch otherwise). - `RequestURIStore` (`request_uri` table) — the reverse index from a validated commit to its request, keyed by `(queue, uri)`: `Create` (ErrAlreadyExists on a duplicate `(queue, uri)` — the dedup signal backing the RFC's `(Queue, head URI)` key) and `GetIDByURI` (ErrNotFound). Kept a separate store because it is a separate table; the two are written independently so the contract stays satisfiable by key/value backends. Includes the MySQL implementation, schema (`request.sql`, `request_uri.sql`), generated mocks, and a docker-compose integration test covering create/get/update-CAS/not-found/duplicate and the URI mapping (including dedup and per-queue isolation).
behinddwalls
force-pushed
the
preetam/stovepipe-request-store
branch
from
ff10b2f to
83961ca
Compare
behinddwalls
temporarily deployed
to
stack-rebase
GitHub Actions
Inactive
behinddwalls
added a commit
that referenced
this pull request
Jun 26, 2026
## Summary ### Why? The `Request.URI` field is "empty until SourceControl resolution is wired in" — Stovepipe has no way to resolve a queue's head commit, compare two commits for history rewrites, or enumerate a ref's recent commits. The workflow RFC already names SourceControl as the sole owner of URI semantics with exactly these three responsibilities. This adds that contract so downstream stages (`ingest`, `process`) and a future status view can be built against it. ### What? New `stovepipe/extension/sourcecontrol` package holding the contract only (interface + Config + Factory interface + sentinel error), per the extension-design rules — no factory impl or routing. The `SourceControl` interface is bound to a single queue by its Factory, so methods take no queue argument: - `Latest(ctx)` — latest commit URI on the queue's ref (VCS-agnostic name, not "Head"). - `IsAncestor(ctx, ancestor, descendant)` — ancestry check; `false` is the history-rewrite signal that drives a full build. - `History(ctx, cursor, limit)` — cursor-paginated, newest-first page of commit URIs; bounded so a remote backend stays cheap. URIs and the pagination cursor are opaque tokens interpreted only by the implementation. Pagination uses a new shared generic `platform/base/page.Page[T any]` (`Items` + opaque `NextCursor`) rather than a one-off struct — the repo's first generic, reusable for any future paginated read across domains. Method-result data types live outside the extension package, matching existing precedent (`entity.BuildStatus`, `entity.Conflict`). Ships with an in-memory `fake` backend (seeded with a queue's newest-first history) and a generated gomock for use in controller and pipeline tests. ## Test Plan ✅ `bazel test //stovepipe/extension/sourcecontrol/... //platform/base/page/...` — fake unit tests (Latest / IsAncestor / History pagination + not-found cases) pass. ✅ `make build`, `make gazelle`, `make fmt` clean. ## Issues ## Stack 1. #276 1. #277 1. @ #278 1. #279
behinddwalls
added a commit
that referenced
this pull request
Jun 26, 2026
…process (#279) ## Summary ### Why? Ingest was a thin stub: it minted a request id but never resolved the commit URI, persisted anything, or moved the request onto the pipeline, so `Request.URI` stayed empty and nothing consumed the work. This makes ingest the real pipeline entry and adds the first internal queue contract so the pipeline can hand work to the next stage. ### What? Ingest now resolves the queue's head URI via the SourceControl extension, dedups on the (queue, URI) pair, persists the Request and its URI mapping via storage, and publishes the request id to a new process stage over the messaging queue. Ingestion is idempotent: a re-reported head resolves to the already-minted request and nothing is published again. The URI mapping is claimed before the request row is written, so a lost race leaves no orphan row. Adds the first internal proto message-queue contract under `stovepipe/core/messagequeue` (proto3 + protojson, mirroring `api/runway/messagequeue`): a `ProcessRequest` payload carrying the id, the `TopicKeyProcess` constant, and the protojson glue, wired into the proto codegen (`tool/proto`, `PROTO_PACKAGES`). Per CLAUDE.md, internal contracts live under the domain's `core/`, not `api/`, and the contract package owns both the payload and its topic keys. Adds a minimal `process` consumer (`stovepipe/controller/process`) that reloads the Request by id and logs it; a not-yet-visible request is retryable so redelivery converges. The build-strategy/ancestry logic the RFC assigns to `process` is deferred. Wires the example server (`example/stovepipe/server`) into a MySQL storage + MySQL queue + fake SourceControl stack with the process consumer running, plus docker-compose (two databases) and a schema-init make target. ## Test Plan ✅ `bazel test //stovepipe/...` — contract round-trip + topic-key binding, ingest (happy/dedup/race/unknown-queue/infra-error paths), and process consumer unit tests. ✅ `bazel test //test/integration/stovepipe:stovepipe_test` — compose-backed: Ingest persists the request + URI mapping, publishes to the process topic, and a re-ingest dedups to the same id. ✅ `bazel build //...`, `make fmt`. ## Stack 1. #276 1. #277 1. #278 1. @ #279
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Why?
The ingest controller (PR #276) mints a
Requestbut is log-only. For the pipeline to do real work it needs to persist requests and look them up two ways, both flowing from the workflow RFC: by request ID (every downstream stage reloads the entity), and by (queue, commit URI) to find whether a commit is already being validated — the RFC's(Queue, head URI)dedup key.What?
A new
stovepipe/extension/storageextension plus its first backend (MySQL), mirroring the SubmitQueue storage conventions: a factoryStorageinterface,ErrNotFound/ErrAlreadyExists/ErrVersionMismatchsentinels, metrics-wrapped MySQL ops, and optimistic-locking CAS with version arithmetic owned by the caller.Two stores, one per table:
RequestStore(requesttable) —Create(ErrAlreadyExists on dup ID);Getby ID (ErrNotFound);Update, a pure conditional write taking the wholeRequestplusoldVersion/newVersion, persisting the mutable fields (uri, state) only if the stored version matches (else ErrVersionMismatch).RequestURIStore(request_uritable) — the reverse index from a validated commit to its request, keyed by(queue, uri):Create(ErrAlreadyExists on a duplicate(queue, uri)— the dedup signal) andGetIDByURI(ErrNotFound). It's a separate store because it's a separate table; the two are written independently (no cross-table transaction) so the contract stays satisfiable by key/value backends.Ships the MySQL impl, schema (
request.sql,request_uri.sql), generated mocks, and a docker-compose integration test.Follow-ups: wire
storageintoingest(dedup viaGetIDByURI, thenCreate); an in-memory backend + shared contract suite.Test Plan
bazel test //test/integration/stovepipe/extension/storage/mysql:mysql_test— real MySQL via docker-compose: create/get/update-CAS (success, stale-version mismatch, missing-row mismatch), not-found, duplicate-ID, and the URI mapping (round-trip, not-found, duplicate dedup, per-queue isolation).bazel build //stovepipe/...;make check-gazelle,make check-tidy,make lintclean. Mocks regenerated and idempotent.Issues
Stack