Skip to content

Commit 29ad310

Browse files
refactor: consolidate shared code under platform/
Move the cross-domain shared trees out of the repo root into a single platform/ umbrella: - core/{errs,metrics,consumer} -> platform/{errs,metrics,consumer} - core/httpclient -> platform/http (package renamed to http, net/http aliased as nethttp; callers that also import net/http use the phttp alias) - entity/* -> platform/base/* (root doc package renamed entity -> base; change, mergestrategy, messagequeue preserved as subpackages) - extension/* -> platform/extension/* Rewrites all import paths and Bazel labels, updates Makefile schema/admin paths, and refreshes docs (CLAUDE.md, READMEs, RFCs) to the platform/ layout. Domain-scoped trees (submitqueue/, stovepipe/, runway/) keep their own core/entity/extension. Co-authored-by: Cursor <cursoragent@cursor.com>
1 parent 25a3aa0 commit 29ad310

275 files changed

Lines changed: 782 additions & 737 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

CLAUDE.md

Lines changed: 45 additions & 41 deletions
Original file line numberDiff line numberDiff line change
@@ -36,35 +36,39 @@ request.Version = newVersion
3636

3737
```
3838
submitqueue/ # repo root (Go module github.com/uber/submitqueue)
39-
├── core/ # SHARED cross-domain infrastructure (errs, httpclient, metrics) — no domain deps
40-
├── entity/ # SHARED domain entities
41-
│ └── queue/ # Queue-specific entities (Message)
42-
├── extension/ # SHARED extensions
43-
│ └── queue/ # Messaging queue abstraction (interface + mysql/)
44-
├── submitqueue/ # SubmitQueue domain
45-
│ ├── gateway/ # Gateway service (port 8081) - entry point
46-
│ ├── orchestrator/ # Orchestrator service (port 8082) - coordinates jobs
47-
│ ├── entity/ # SubmitQueue-specific domain entities
48-
│ ├── extension/ # SubmitQueue-specific extension impls (storage, counter, mergechecker, ...)
49-
│ └── core/ # SubmitQueue-internal shared infra (consumer, request)
50-
├── stovepipe/ # Stovepipe domain
51-
│ ├── gateway/ # Gateway service: commit deployment verification entry point
52-
│ ├── orchestrator/ # Orchestrator service: commit verification pipeline
53-
│ ├── entity/ # Stovepipe-specific domain entities
54-
│ ├── extension/ # Stovepipe-specific extension impls
55-
│ └── core/ # Stovepipe-internal shared infra (placeholder; mirrors submitqueue/core)
56-
├── tool/ # Development and CI tooling
39+
├── platform/ # SHARED cross-domain packages — no domain deps
40+
│ ├── errs/, metrics/, consumer/, http/
41+
│ ├── base/ # SHARED entities (change/, messagequeue/, …)
42+
│ └── extension/ # SHARED extension contracts + backends (counter/, messagequeue/, …)
43+
├── submitqueue/ # SubmitQueue domain
44+
│ ├── gateway/ # Gateway service (port 8081) - entry point
45+
│ ├── orchestrator/ # Orchestrator service (port 8082) - coordinates jobs
46+
│ ├── entity/ # SubmitQueue-specific domain entities
47+
│ ├── extension/ # SubmitQueue-specific extension impls (storage, counter, mergechecker, …)
48+
│ └── core/ # SubmitQueue-internal shared infra (consumer wiring, request, topickey, …)
49+
├── stovepipe/ # Stovepipe domain
50+
│ ├── gateway/ # Gateway service: commit deployment verification entry point
51+
│ ├── orchestrator/ # Orchestrator service: commit verification pipeline
52+
│ ├── entity/ # Stovepipe-specific domain entities
53+
│ ├── extension/ # Stovepipe-specific extension impls
54+
│ └── core/ # Stovepipe-internal shared infra (placeholder; mirrors submitqueue/core)
55+
├── tool/ # Development and CI tooling
5756
├── example/
58-
│ ├── submitqueue/ # Runnable SubmitQueue servers/clients + Docker Compose
59-
│ └── stovepipe/ # Runnable Stovepipe servers/clients
57+
│ ├── submitqueue/ # Runnable SubmitQueue servers/clients + Docker Compose
58+
│ └── stovepipe/ # Runnable Stovepipe servers/clients
6059
├── test/
61-
│ ├── e2e/submitqueue/ # End-to-end tests (full stack)
62-
│ ├── integration/ # Integration tests (core/, submitqueue/, stovepipe/)
63-
│ └── testutil/ # Test utilities (ComposeStack, MySQL helpers)
64-
└── doc/ # Documentation
60+
│ ├── e2e/submitqueue/ # End-to-end tests (full stack)
61+
│ ├── integration/ # Integration tests (platform/, submitqueue/, stovepipe/, …)
62+
│ └── testutil/ # Test utilities (ComposeStack, MySQL helpers)
63+
└── doc/ # Documentation
6564
```
6665

67-
The repo hosts shared building blocks at the top level — cross-domain infrastructure in `core/`, shared entities in `entity/`, shared extensions in `extension/` — followed by one folder per **domain** (`submitqueue/`, `stovepipe/`). Each domain owns the same internal layout (`gateway/`, `orchestrator/`, `entity/`, `extension/`, `core/`); a domain's own `core/` (e.g. `submitqueue/core/`) holds infra shared only between that domain's services.
66+
The `platform/` tree holds code reused across domains (infrastructure, shared entities, shared extension contracts). Each **domain** (`submitqueue/`, `stovepipe/`, …) keeps the same internal layout (`gateway/`, `orchestrator/`, `entity/`, `extension/`, `core/`); a domain's own `core/` (e.g. `submitqueue/core/`) holds infra shared only between that domain's services.
67+
68+
### Platform notes
69+
70+
- Import path `github.com/uber/submitqueue/platform/http` uses Go package name `http` and aliases the standard library as `nethttp` inside the package. Source files that also import `net/http` should import the platform package with a distinct alias (for example `phttp "github.com/uber/submitqueue/platform/http"`) and call `phttp.NewClient`, `phttp.BaseURLTransport`, etc.
71+
- `platform/base` is the shared entity root; subpackages (`change`, `messagequeue`, …) hold concrete types. The root `base` package is documentation-only.
6872

6973
### Services
7074

@@ -101,7 +105,7 @@ Controllers receive `consumer.Delivery` (subset interface without Ack/Nack) to e
101105

102106
### Entities
103107

104-
Domain objects in `entity/`, organized by domain. Guidelines:
108+
Domain objects live under each domain's `entity/` tree, or under `platform/base/` when shared across domains. Guidelines:
105109
1. Pure and framework-agnostic — no external dependencies
106110
2. Value types, not references
107111
3. `int64` milliseconds for timestamps (`CreatedAt int64`) and durations (`TimeoutMs int64`)
@@ -112,17 +116,17 @@ Domain objects in `entity/`, organized by domain. Guidelines:
112116
### Extensions
113117

114118
Vendor-agnostic, pluggable interfaces with implementations in subdirectories:
115-
1. Define interfaces at `extension/{ext}/`
116-
2. Implementations at `extension/{ext}/{impl}/`
117-
3. Factory interface for dependency injection and lifecycle management
119+
1. **Shared across domains** — define interfaces at `platform/extension/{ext}/`, implementations at `platform/extension/{ext}/{impl}/`.
120+
2. **Domain-specific** — define at `{domain}/extension/{ext}/`, implementations at `{domain}/extension/{ext}/{impl}/`.
121+
3. Factory interface for dependency injection and lifecycle management (constructed in wiring, not inside `platform/extension` packages).
118122
119123
**Extensions hold contracts and implementations only — not factories or routing.**
120124
121-
An `extension/{ext}` package contains the behavioral interface, its `Config`, the `Factory` *interface*, and impl constructors `New(...)` that return the interface. It must **not** contain `Factory` *implementations* (`NewFactory()` constructors or factory structs) or any queue-selection logic.
125+
A `{domain}/extension/{ext}` or `platform/extension/{ext}` package contains the behavioral interface, its `Config`, the `Factory` *interface*, and impl constructors `New(...)` that return the interface. It must **not** contain `Factory` *implementations* (`NewFactory()` constructors or factory structs) or any queue-selection logic.
122126
123127
Why: an impl package (e.g. `scorer/heuristic`) can't know the queue topology or the other impls, so a "which impl for which queue" decision doesn't belong there. Per-queue routing — and the small adapters that wrap a `New(...)` impl in the `Factory` interface — live in the wiring layer (e.g. `example/{domain}/{service}/server/main.go`), the one place that knows the full queue set. That's where you route on `Config.QueueName`.
124128
125-
Rule of thumb: if you're about to add a `NewFactory()` or a `map[queue]impl` under `extension/`, it belongs in the wiring layer instead.
129+
Rule of thumb: if you're about to add a `NewFactory()` or a `map[queue]impl` under `{domain}/extension/` or `platform/extension/`, it belongs in the wiring layer instead.
126130
127131
**Design interfaces for the technology *space*, not the implementation in front of you.** The interface is a contract every backend will have to satisfy — SQL, key-value (DynamoDB, Bigtable), document, message queue, search, RPC, in-memory, mocks. If the contract assumes a capability that some plausible backend can't provide cheaply, you've baked the current impl's strengths into the API.
128132
@@ -142,18 +146,18 @@ When in doubt, ask: *"If the next implementation were DynamoDB / Kafka / Bigtabl
142146
143147
### Import Paths
144148
145-
Paths follow the directory layout: shared code is top-level, domain code nests under the domain folder (`submitqueue/`, `stovepipe/`).
149+
Paths follow the directory layout: shared packages live under `platform/` at the repo root; domain code nests under `submitqueue/`, `stovepipe/`, and other domain folders.
146150
147151
- RPC Controllers: `github.com/uber/submitqueue/{domain}/{service}/controller` (e.g. `.../submitqueue/gateway/controller`)
148152
- Queue Controllers: `github.com/uber/submitqueue/{domain}/{service}/controller/{step}`
149153
- Proto (generated): `github.com/uber/submitqueue/{domain}/{service}/protopb`
150154
- Domain entities: `github.com/uber/submitqueue/{domain}/entity` (e.g. `.../submitqueue/entity`)
151155
- Domain extensions: `github.com/uber/submitqueue/{domain}/extension/{ext}[/{impl}]` (e.g. `.../submitqueue/extension/storage/mysql`)
152-
- Cross-domain consumer framework: `github.com/uber/submitqueue/core/consumer`; domain pipeline topic keys: `github.com/uber/submitqueue/{domain}/core/topickey`
156+
- Cross-domain consumer framework: `github.com/uber/submitqueue/platform/consumer`; domain pipeline topic keys: `github.com/uber/submitqueue/{domain}/core/topickey`
153157
- Domain-internal infra: `github.com/uber/submitqueue/{domain}/core/{pkg}` (e.g. `.../submitqueue/core/request`)
154-
- Shared entities: `github.com/uber/submitqueue/entity/{name}` (e.g. `.../entity/messagequeue`)
155-
- Shared extensions: `github.com/uber/submitqueue/extension/{name}` (e.g. `.../extension/messagequeue`)
156-
- Cross-domain infra: `github.com/uber/submitqueue/core/{pkg}` (e.g. `.../core/errs`, `.../core/metrics`)
158+
- Shared entities: `github.com/uber/submitqueue/platform/base/{pkg}` (e.g. `.../platform/base/messagequeue`)
159+
- Shared extensions: `github.com/uber/submitqueue/platform/extension/{ext}[/{impl}]` (e.g. `.../platform/extension/messagequeue/mysql`)
160+
- Cross-domain infra: `github.com/uber/submitqueue/platform/{pkg}` (e.g. `.../platform/errs`, `.../platform/metrics`, `.../platform/http`)
157161
158162
## Development
159163
@@ -225,11 +229,11 @@ make clean # Clean Bazel cache
225229
2. Wire up in `example/{domain}/{service}/server/main.go`
226230

227231
**Add new extension:**
228-
1. Create the extension under `{domain}/extension/{ext}/{impl}/` (domain-specific, e.g. `submitqueue/extension/...`) or top-level `extension/{ext}/{impl}/` (shared across domains) with factory and interfaces
232+
1. Create the extension under `{domain}/extension/{ext}/{impl}/` (domain-specific, e.g. `submitqueue/extension/...`) or `platform/extension/{ext}/{impl}/` (shared across domains) with factory and interfaces
229233
2. Add `BUILD.bazel`, tests, and README.md
230234

231235
**Add new entity:**
232-
1. Create `{domain}/entity/{entity}.go` (domain-specific) or top-level `entity/{name}/{entity}.go` (shared) with test file and `BUILD.bazel`
236+
1. Create `{domain}/entity/{entity}.go` (domain-specific) or add packages under `platform/base/` (shared) with test file and `BUILD.bazel`
233237

234238
**Add gomock for an extension interface:**
235239

@@ -252,7 +256,7 @@ To create a mock package for a new extension (e.g., `submitqueue/extension/newex
252256
3. Run `make mocks` to generate mock files into the new directory.
253257
4. Run `make gazelle` to create the `BUILD.bazel` file automatically.
254258

255-
For inline mocks (mock in the same package, e.g., `extension/messagequeue/mysql/mock_stores.go`):
259+
For inline mocks (mock in the same package, e.g., `platform/extension/messagequeue/mysql/mock_stores.go`):
256260

257261
1. Add a `//go:generate` directive with `-package=mypkg` and `-destination=mock_file.go`.
258262
2. Run `make mocks` and `make gazelle`.
@@ -305,9 +309,9 @@ CI runs on every PR and enforces all checks via a `required-checks` gate. **Befo
305309
3. **Value types over pointers** — prefer value types for structs, configs, and return values. Use `(T, bool)` to signal absence instead of `*T`. Pointers only when mutation or shared ownership is needed.
306310
4. **Errors for failures, not control flow** — reserve `error` returns for unexpected or infrastructure failures. Use result types (structs, bools) for expected outcomes like `(Result, error)` or `(T, bool)`. Avoid sentinel errors that represent non-failure states.
307311

308-
### Error Classification (`core/errs`)
312+
### Error Classification (`platform/errs`)
309313

310-
Errors are classified by origin (user vs infra) and retryability. The framework lives in `core/errs/`. See [core/errs/README.md](core/errs/README.md) for full details.
314+
Errors are classified by origin (user vs infra) and retryability. The framework lives in `platform/errs/`. See [platform/errs/README.md](platform/errs/README.md) for full details.
311315

312316
**Key rules:**
313317
1. **Non-retryable by default** — a plain `fmt.Errorf(...)` is non-retryable. Wrap with `errs.NewRetryableError(...)` to opt in to retry.

Makefile

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -196,20 +196,20 @@ local-init-submitqueue-schemas: ## Manually apply all database schemas
196196
docker exec -i $(SUBMITQUEUE_LOCAL_PROJECT)-mysql-app-1 mysql -uroot -proot submitqueue < $$file 2>&1 | grep -v "Using a password" || true; \
197197
done
198198
@echo "Applying counter schema to mysql-app..."
199-
@for file in extension/counter/mysql/schema/*.sql; do \
199+
@for file in platform/extension/counter/mysql/schema/*.sql; do \
200200
echo " - Applying $$(basename $$file)..."; \
201201
docker exec -i $(SUBMITQUEUE_LOCAL_PROJECT)-mysql-app-1 mysql -uroot -proot submitqueue < $$file 2>&1 | grep -v "Using a password" || true; \
202202
done
203203
@echo "Applying queue schema to mysql-queue..."
204-
@for file in extension/messagequeue/mysql/schema/*.sql; do \
204+
@for file in platform/extension/messagequeue/mysql/schema/*.sql; do \
205205
echo " - Applying $$(basename $$file)..."; \
206206
docker exec -i $(SUBMITQUEUE_LOCAL_PROJECT)-mysql-queue-1 mysql -uroot -proot submitqueue < $$file 2>&1 | grep -v "Using a password" || true; \
207207
done
208208
@echo "✅ All schemas applied successfully"
209209

210210
local-init-stovepipe-queue-schema: ## Apply queue schema only (mysql-queue) for Stovepipe compose stacks
211211
@echo "Applying queue schema to mysql-queue (Stovepipe; no app storage/counter schema yet)..."
212-
@for file in extension/messagequeue/mysql/schema/*.sql; do \
212+
@for file in platform/extension/messagequeue/mysql/schema/*.sql; do \
213213
echo " - Applying $$(basename $$file)..."; \
214214
docker exec -i $(STOVEPIPE_LOCAL_PROJECT)-mysql-queue-1 mysql -uroot -proot submitqueue < $$file 2>&1 | grep -v "Using a password" || true; \
215215
done
@@ -336,7 +336,7 @@ local-stovepipe-gateway-start: build-stovepipe-gateway-linux ## Start Stovepipe
336336

337337
mocks: ## Generate mock files using mockgen
338338
@echo "Generating mocks..."
339-
@$(BAZEL) run @rules_go//go -- generate ./submitqueue/extension/storage/... ./submitqueue/extension/buildrunner/... ./submitqueue/extension/changeprovider/... ./extension/counter/... ./extension/messagequeue/... ./submitqueue/extension/queueconfig/... ./submitqueue/extension/mergechecker/... ./submitqueue/extension/pusher/... ./submitqueue/extension/scorer/... ./submitqueue/extension/conflict/... ./core/consumer/...
339+
@$(BAZEL) run @rules_go//go -- generate ./submitqueue/extension/storage/... ./submitqueue/extension/buildrunner/... ./submitqueue/extension/changeprovider/... ./platform/extension/counter/... ./platform/extension/messagequeue/... ./submitqueue/extension/queueconfig/... ./submitqueue/extension/mergechecker/... ./submitqueue/extension/pusher/... ./submitqueue/extension/scorer/... ./submitqueue/extension/conflict/... ./platform/consumer/...
340340
@echo "Mocks generated successfully!"
341341

342342
proto: ## Generate protobuf files from .proto definitions
@@ -375,7 +375,7 @@ run-client-stovepipe-orchestrator:
375375
@$(BAZEL) run //example/stovepipe/orchestrator/client:orchestrator -- -addr $(or $(SERVER_ADDR),localhost:8084) -message "$(or $(MESSAGE),ping)"
376376

377377
run-queue-admin: ## Run queue-admin CLI (use ARGS to pass arguments, e.g. make run-queue-admin ARGS="list-topics")
378-
@$(BAZEL) run //extension/messagequeue/mysql/ctl -- $(ARGS)
378+
@$(BAZEL) run //platform/extension/messagequeue/mysql/ctl -- $(ARGS)
379379

380380
test: ## Run unit tests
381381
@echo "Running unit tests..."

README.md

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,10 @@ SubmitQueue is a high-performance speculative merge queue that keeps your trunk
88

99
Designed for large monorepos and fast-moving teams where concurrent changes can introduce subtle conflicts and destabilize builds.
1010

11+
## Repository layout
12+
13+
Cross-domain Go code (errors, metrics, consumer framework, HTTP helpers, shared entities, shared extension contracts) lives under [`platform/`](platform/README.md). Each product domain has its own tree (`submitqueue/`, `stovepipe/`, …) with `gateway/`, `orchestrator/`, `entity/`, `extension/`, and domain-local `core/`. See [CLAUDE.md](CLAUDE.md) for conventions and import paths.
14+
1115
## Quick Start
1216

1317
Requires Docker and Docker Compose. See [Development Setup](doc/howto/DEVELOPMENT.md) for full prerequisites.

core/README.md

Lines changed: 0 additions & 11 deletions
This file was deleted.

core/httpclient/transport.go

Lines changed: 0 additions & 52 deletions
This file was deleted.

0 commit comments

Comments
 (0)