Skip to content

Commit ba9dcb6

Browse files
authored
feat(stovepipe): add storage extension with RequestStore + RequestURIStore (MySQL) (#277)
## Summary ### Why? The ingest controller (PR #276) mints a `Request` but is log-only. For the pipeline to do real work it needs to persist requests and look them up two ways, both flowing from the workflow RFC: by **request ID** (every downstream stage reloads the entity), and by **(queue, commit URI)** to find whether a commit is already being validated — the RFC's `(Queue, head URI)` dedup key. ### What? A new `stovepipe/extension/storage` extension plus its first backend (MySQL), mirroring the SubmitQueue storage conventions: a factory `Storage` interface, `ErrNotFound`/`ErrAlreadyExists`/`ErrVersionMismatch` sentinels, metrics-wrapped MySQL ops, and optimistic-locking CAS with version arithmetic owned by the caller. Two stores, **one per table**: - **`RequestStore`** (`request` table) — `Create` (ErrAlreadyExists on dup ID); `Get` by ID (ErrNotFound); `Update`, a pure conditional write taking the whole `Request` plus `oldVersion`/`newVersion`, persisting the mutable fields (uri, state) only if the stored version matches (else ErrVersionMismatch). - **`RequestURIStore`** (`request_uri` table) — the reverse index from a validated commit to its request, keyed by `(queue, uri)`: `Create` (ErrAlreadyExists on a duplicate `(queue, uri)` — the dedup signal) and `GetIDByURI` (ErrNotFound). It's a separate store because it's a separate table; the two are written independently (no cross-table transaction) so the contract stays satisfiable by key/value backends. Ships the MySQL impl, schema (`request.sql`, `request_uri.sql`), generated mocks, and a docker-compose integration test. Follow-ups: wire `storage` into `ingest` (dedup via `GetIDByURI`, then `Create`); an in-memory backend + shared contract suite. ## Test Plan - ✅ `bazel test //test/integration/stovepipe/extension/storage/mysql:mysql_test` — real MySQL via docker-compose: create/get/update-CAS (success, stale-version mismatch, missing-row mismatch), not-found, duplicate-ID, and the URI mapping (round-trip, not-found, duplicate dedup, per-queue isolation). - ✅ `bazel build //stovepipe/...`; `make check-gazelle`, `make check-tidy`, `make lint` clean. Mocks regenerated and idempotent. ## Issues ## Stack 1. #276 1. @ #277 1. #278 1. #279
1 parent c13d330 commit ba9dcb6

18 files changed

Lines changed: 946 additions & 0 deletions
Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
load("@rules_go//go:def.bzl", "go_library")
2+
3+
go_library(
4+
name = "storage",
5+
srcs = [
6+
"request_store.go",
7+
"request_uri_store.go",
8+
"storage.go",
9+
],
10+
importpath = "github.com/uber/submitqueue/stovepipe/extension/storage",
11+
visibility = ["//visibility:public"],
12+
deps = ["//stovepipe/entity"],
13+
)
Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
load("@rules_go//go:def.bzl", "go_library")
2+
3+
go_library(
4+
name = "mock",
5+
srcs = [
6+
"request_store_mock.go",
7+
"request_uri_store_mock.go",
8+
"storage_mock.go",
9+
],
10+
importpath = "github.com/uber/submitqueue/stovepipe/extension/storage/mock",
11+
visibility = ["//visibility:public"],
12+
deps = [
13+
"//stovepipe/entity",
14+
"//stovepipe/extension/storage",
15+
"@org_uber_go_mock//gomock",
16+
],
17+
)

stovepipe/extension/storage/mock/request_store_mock.go

Lines changed: 85 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

stovepipe/extension/storage/mock/request_uri_store_mock.go

Lines changed: 70 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

stovepipe/extension/storage/mock/storage_mock.go

Lines changed: 83 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.
Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
load("@rules_go//go:def.bzl", "go_library")
2+
3+
go_library(
4+
name = "mysql",
5+
srcs = [
6+
"request_store.go",
7+
"request_uri_store.go",
8+
"storage.go",
9+
],
10+
importpath = "github.com/uber/submitqueue/stovepipe/extension/storage/mysql",
11+
visibility = ["//visibility:public"],
12+
deps = [
13+
"//platform/metrics",
14+
"//stovepipe/entity",
15+
"//stovepipe/extension/storage",
16+
"@com_github_go_sql_driver_mysql//:mysql",
17+
"@com_github_uber_go_tally//:tally",
18+
],
19+
)

0 commit comments

Comments
 (0)