Skip to content

Commit 7b5f323

Browse files
behinddwallsgithub-actions[bot]
authored andcommitted
feat(stovepipe): add SourceControl extension contract
## Summary ### Why? The `Request.URI` field is "empty until SourceControl resolution is wired in" — Stovepipe has no way to resolve a queue's head commit, compare two commits for history rewrites, or enumerate a ref's recent commits. The workflow RFC already names SourceControl as the sole owner of URI semantics with exactly these three responsibilities. This adds that contract so downstream stages (`ingest`, `process`) and a future status view can be built against it. ### What? New `stovepipe/extension/sourcecontrol` package holding the contract only (interface + Config + Factory interface + sentinel error), per the extension-design rules — no factory impl or routing. The `SourceControl` interface is bound to a single queue by its Factory, so methods take no queue argument: - `Latest(ctx)` — latest commit URI on the queue's ref (VCS-agnostic name, not "Head"). - `IsAncestor(ctx, ancestor, descendant)` — ancestry check; `false` is the history-rewrite signal that drives a full build. - `History(ctx, cursor, limit)` — cursor-paginated, newest-first page of commit URIs; bounded so a remote backend stays cheap. URIs and the pagination cursor are opaque tokens interpreted only by the implementation. Pagination uses a new shared generic `platform/base/page.Page[T any]` (`Items` + opaque `NextCursor`) rather than a one-off struct — the repo's first generic, reusable for any future paginated read across domains. Method-result data types live outside the extension package, matching existing precedent (`entity.BuildStatus`, `entity.Conflict`). Ships with an in-memory `fake` backend (seeded with a queue's newest-first history) and a generated gomock for use in controller and pipeline tests. ## Test Plan ✅ `bazel test //stovepipe/extension/sourcecontrol/... //platform/base/page/...` — fake unit tests (Latest / IsAncestor / History pagination + not-found cases) pass. ✅ `make build`, `make gazelle`, `make fmt` clean.
1 parent 83961ca commit 7b5f323

12 files changed

Lines changed: 568 additions & 1 deletion

File tree

Makefile

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -327,7 +327,7 @@ local-stovepipe-stop: ## Stop the Stovepipe service
327327

328328
mocks: ## Generate mock files using mockgen
329329
@echo "Generating mocks..."
330-
@$(BAZEL) run @rules_go//go -- generate ./submitqueue/extension/storage/... ./submitqueue/extension/buildrunner/... ./submitqueue/extension/changeprovider/... ./platform/extension/counter/... ./platform/extension/messagequeue/... ./submitqueue/extension/queueconfig/... ./submitqueue/extension/mergechecker/... ./submitqueue/extension/pusher/... ./submitqueue/extension/scorer/... ./submitqueue/extension/conflict/... ./platform/consumer/... ./stovepipe/extension/storage/...
330+
@$(BAZEL) run @rules_go//go -- generate ./submitqueue/extension/storage/... ./submitqueue/extension/buildrunner/... ./submitqueue/extension/changeprovider/... ./platform/extension/counter/... ./platform/extension/messagequeue/... ./submitqueue/extension/queueconfig/... ./submitqueue/extension/mergechecker/... ./submitqueue/extension/pusher/... ./submitqueue/extension/scorer/... ./submitqueue/extension/conflict/... ./platform/consumer/... ./stovepipe/extension/storage/... ./stovepipe/extension/sourcecontrol/...
331331
@echo "Mocks generated successfully!"
332332

333333
proto: ## Generate protobuf files from .proto definitions

platform/base/page/BUILD.bazel

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
load("@rules_go//go:def.bzl", "go_library")
2+
3+
go_library(
4+
name = "page",
5+
srcs = ["page.go"],
6+
importpath = "github.com/uber/submitqueue/platform/base/page",
7+
visibility = ["//visibility:public"],
8+
)

platform/base/page/page.go

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
// Copyright (c) 2025 Uber Technologies, Inc.
2+
//
3+
// Licensed under the Apache License, Version 2.0 (the "License");
4+
// you may not use this file except in compliance with the License.
5+
// You may obtain a copy of the License at
6+
//
7+
// http://www.apache.org/licenses/LICENSE-2.0
8+
//
9+
// Unless required by applicable law or agreed to in writing, software
10+
// distributed under the License is distributed on an "AS IS" BASIS,
11+
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
// See the License for the specific language governing permissions and
13+
// limitations under the License.
14+
15+
// Package page defines a generic, cursor-paginated result envelope shared across
16+
// domains. Producers return one bounded Page at a time; callers walk further by
17+
// passing the page's NextCursor back to the producing call until it is empty.
18+
package page
19+
20+
// Page is one bounded slice of a larger sequence, plus an opaque cursor for
21+
// fetching the next page. The element type T is the domain value being paged
22+
// (e.g. a commit URI string, or an entity).
23+
type Page[T any] struct {
24+
// Items are the elements in this page, in the producer's defined order.
25+
Items []T
26+
// NextCursor is an opaque token for fetching the next page, passed back to
27+
// the producing call. It is empty when this page is the last one. Its
28+
// encoding is defined and interpreted solely by the producer.
29+
NextCursor string
30+
}
Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
load("@rules_go//go:def.bzl", "go_library")
2+
3+
go_library(
4+
name = "sourcecontrol",
5+
srcs = ["sourcecontrol.go"],
6+
importpath = "github.com/uber/submitqueue/stovepipe/extension/sourcecontrol",
7+
visibility = ["//visibility:public"],
8+
deps = ["//platform/base/page"],
9+
)
Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
# SourceControl
2+
3+
Vendor-agnostic interface through which Stovepipe talks to a version control system. It is the **sole owner of URI semantics**: a URI is an opaque, VCS-agnostic locator of a commit. The `git://` scheme used by the reference backend is just one encoding — a Mercurial or Perforce backend mints its own behind the same contract. Nothing outside an implementation parses a URI; it is a token you hand back to ask questions about a ref.
4+
5+
A `SourceControl` is **bound to a single queue** (a repo+ref) when its `Factory` constructs it from a `Config`, so the behavioral methods take no queue argument. Per the repository's extension rules, this package holds the `SourceControl` interface, its `Config`, and the `Factory` *interface* only — concrete `Factory` implementations and the per-queue routing that picks a backend for a `Config.QueueName` live in the wiring layer.
6+
7+
## Behavior
8+
9+
- **Latest** resolves the queue's ref to the URI of its latest commit — the commit a new validation `Request` is minted against during `ingest`.
10+
- **IsAncestor** answers whether one URI is an ancestor of another. The `process` stage uses it to choose a build strategy: if the queue's last-green URI is no longer an ancestor of the latest commit, history was rewritten and a full build is required rather than an incremental one.
11+
- **History** returns a bounded, newest-first page of commit URIs on the ref, using the shared generic `page.Page[string]` (`platform/base/page`). It is paginated with an opaque cursor: callers pass an empty cursor for the newest page and the page's `NextCursor` to walk further back, stopping when it is empty. Pagination keeps a remote backend cheap; callers join the URIs against the request store to render the greenness of each commit.
12+
13+
## Errors
14+
15+
Implementations return plain errors and use the package sentinel `ErrNotFound` (with the `IsNotFound` / `WrapNotFound` helpers) when a queue, ref, or URI cannot be resolved. They do not classify errors as user- or infra-caused — the calling controller does that.
16+
17+
## Implementations
18+
19+
- **fake** — an in-memory backend seeded with a queue's ref history (newest first), for examples and tests.
20+
21+
To add a backend, create `sourcecontrol/{backend}/`, implement the `SourceControl` interface, and return it from a `New(...)` constructor.
Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
load("@rules_go//go:def.bzl", "go_library", "go_test")
2+
3+
go_library(
4+
name = "fake",
5+
srcs = ["fake.go"],
6+
importpath = "github.com/uber/submitqueue/stovepipe/extension/sourcecontrol/fake",
7+
visibility = ["//visibility:public"],
8+
deps = [
9+
"//platform/base/page",
10+
"//stovepipe/extension/sourcecontrol",
11+
],
12+
)
13+
14+
go_test(
15+
name = "fake_test",
16+
srcs = ["fake_test.go"],
17+
embed = [":fake"],
18+
deps = [
19+
"//stovepipe/extension/sourcecontrol",
20+
"@com_github_stretchr_testify//assert",
21+
"@com_github_stretchr_testify//require",
22+
],
23+
)
Lines changed: 102 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,102 @@
1+
// Copyright (c) 2025 Uber Technologies, Inc.
2+
//
3+
// Licensed under the Apache License, Version 2.0 (the "License");
4+
// you may not use this file except in compliance with the License.
5+
// You may obtain a copy of the License at
6+
//
7+
// http://www.apache.org/licenses/LICENSE-2.0
8+
//
9+
// Unless required by applicable law or agreed to in writing, software
10+
// distributed under the License is distributed on an "AS IS" BASIS,
11+
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
// See the License for the specific language governing permissions and
13+
// limitations under the License.
14+
15+
// Package fake provides an in-memory sourcecontrol.SourceControl seeded with a
16+
// single queue's linear ref history, ordered newest-first. It is intended for
17+
// examples and tests only, never production. Ancestry is decided by position in
18+
// the seeded slice: an earlier commit (larger index) is an ancestor of a later
19+
// one (smaller index).
20+
package fake
21+
22+
import (
23+
"context"
24+
25+
"github.com/uber/submitqueue/platform/base/page"
26+
"github.com/uber/submitqueue/stovepipe/extension/sourcecontrol"
27+
)
28+
29+
// sourceControlFake serves a single queue's linear history. history[0] is the
30+
// latest commit; higher indices are progressively older ancestors.
31+
type sourceControlFake struct {
32+
history []string
33+
}
34+
35+
// New returns a sourcecontrol.SourceControl backed by the given ref history,
36+
// ordered newest-first (history[0] is the latest commit). The slice is copied so
37+
// later mutation by the caller does not affect the fake.
38+
func New(history []string) sourcecontrol.SourceControl {
39+
cp := make([]string, len(history))
40+
copy(cp, history)
41+
return sourceControlFake{history: cp}
42+
}
43+
44+
// Latest returns the newest commit URI, or ErrNotFound when the history is empty.
45+
func (s sourceControlFake) Latest(_ context.Context) (string, error) {
46+
if len(s.history) == 0 {
47+
return "", sourcecontrol.ErrNotFound
48+
}
49+
return s.history[0], nil
50+
}
51+
52+
// IsAncestor reports whether ancestor is an ancestor of descendant. Both URIs
53+
// must be on the ref; an unknown URI yields ErrNotFound. Since the history is
54+
// newest-first, ancestor is an ancestor of descendant when its index is greater
55+
// than or equal to descendant's (older-or-equal commit).
56+
func (s sourceControlFake) IsAncestor(_ context.Context, ancestor, descendant string) (bool, error) {
57+
ai := s.indexOf(ancestor)
58+
di := s.indexOf(descendant)
59+
if ai < 0 || di < 0 {
60+
return false, sourcecontrol.ErrNotFound
61+
}
62+
return ai >= di, nil
63+
}
64+
65+
// History returns one page of commit URIs, newest first. The cursor is the URI
66+
// of the first commit of the page to return; an empty cursor starts at the latest
67+
// commit. A limit of zero or less returns the rest of the history from the cursor
68+
// in a single page. The returned NextCursor is the URI of the next, older commit,
69+
// or empty when the page reaches the end of the history.
70+
func (s sourceControlFake) History(_ context.Context, cursor string, limit int) (page.Page[string], error) {
71+
start := 0
72+
if cursor != "" {
73+
start = s.indexOf(cursor)
74+
if start < 0 {
75+
return page.Page[string]{}, sourcecontrol.ErrNotFound
76+
}
77+
}
78+
79+
end := len(s.history)
80+
if limit > 0 && start+limit < end {
81+
end = start + limit
82+
}
83+
84+
uris := make([]string, end-start)
85+
copy(uris, s.history[start:end])
86+
87+
next := ""
88+
if end < len(s.history) {
89+
next = s.history[end]
90+
}
91+
return page.Page[string]{Items: uris, NextCursor: next}, nil
92+
}
93+
94+
// indexOf returns the index of uri in the history, or -1 if absent.
95+
func (s sourceControlFake) indexOf(uri string) int {
96+
for i, u := range s.history {
97+
if u == uri {
98+
return i
99+
}
100+
}
101+
return -1
102+
}
Lines changed: 139 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,139 @@
1+
// Copyright (c) 2025 Uber Technologies, Inc.
2+
//
3+
// Licensed under the Apache License, Version 2.0 (the "License");
4+
// you may not use this file except in compliance with the License.
5+
// You may obtain a copy of the License at
6+
//
7+
// http://www.apache.org/licenses/LICENSE-2.0
8+
//
9+
// Unless required by applicable law or agreed to in writing, software
10+
// distributed under the License is distributed on an "AS IS" BASIS,
11+
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
// See the License for the specific language governing permissions and
13+
// limitations under the License.
14+
15+
package fake
16+
17+
import (
18+
"context"
19+
"testing"
20+
21+
"github.com/stretchr/testify/assert"
22+
"github.com/stretchr/testify/require"
23+
"github.com/uber/submitqueue/stovepipe/extension/sourcecontrol"
24+
)
25+
26+
func TestNew_ImplementsInterface(t *testing.T) {
27+
var _ sourcecontrol.SourceControl = New(nil)
28+
}
29+
30+
// history is ordered newest-first: c is the latest, a is the oldest ancestor.
31+
var history = []string{"git://repo/ref/c", "git://repo/ref/b", "git://repo/ref/a"}
32+
33+
func TestLatest(t *testing.T) {
34+
tests := []struct {
35+
name string
36+
history []string
37+
want string
38+
wantErr bool
39+
}{
40+
{name: "newest first", history: history, want: "git://repo/ref/c"},
41+
{name: "empty history", history: nil, wantErr: true},
42+
}
43+
for _, tt := range tests {
44+
t.Run(tt.name, func(t *testing.T) {
45+
got, err := New(tt.history).Latest(context.Background())
46+
if tt.wantErr {
47+
require.ErrorIs(t, err, sourcecontrol.ErrNotFound)
48+
return
49+
}
50+
require.NoError(t, err)
51+
assert.Equal(t, tt.want, got)
52+
})
53+
}
54+
}
55+
56+
func TestIsAncestor(t *testing.T) {
57+
tests := []struct {
58+
name string
59+
ancestor string
60+
descendant string
61+
want bool
62+
wantErr bool
63+
}{
64+
{name: "older is ancestor of newer", ancestor: "git://repo/ref/a", descendant: "git://repo/ref/c", want: true},
65+
{name: "newer is not ancestor of older", ancestor: "git://repo/ref/c", descendant: "git://repo/ref/a", want: false},
66+
{name: "equal is ancestor of itself", ancestor: "git://repo/ref/b", descendant: "git://repo/ref/b", want: true},
67+
{name: "unknown ancestor", ancestor: "git://repo/ref/x", descendant: "git://repo/ref/a", wantErr: true},
68+
{name: "unknown descendant", ancestor: "git://repo/ref/a", descendant: "git://repo/ref/x", wantErr: true},
69+
}
70+
for _, tt := range tests {
71+
t.Run(tt.name, func(t *testing.T) {
72+
got, err := New(history).IsAncestor(context.Background(), tt.ancestor, tt.descendant)
73+
if tt.wantErr {
74+
require.ErrorIs(t, err, sourcecontrol.ErrNotFound)
75+
return
76+
}
77+
require.NoError(t, err)
78+
assert.Equal(t, tt.want, got)
79+
})
80+
}
81+
}
82+
83+
func TestHistory(t *testing.T) {
84+
tests := []struct {
85+
name string
86+
cursor string
87+
limit int
88+
wantItems []string
89+
wantCursor string
90+
wantErr bool
91+
}{
92+
{
93+
name: "first page with next cursor",
94+
cursor: "",
95+
limit: 2,
96+
wantItems: []string{"git://repo/ref/c", "git://repo/ref/b"},
97+
wantCursor: "git://repo/ref/a",
98+
},
99+
{
100+
name: "second page reaches end",
101+
cursor: "git://repo/ref/a",
102+
limit: 2,
103+
wantItems: []string{"git://repo/ref/a"},
104+
wantCursor: "",
105+
},
106+
{
107+
name: "limit larger than remaining returns rest",
108+
cursor: "",
109+
limit: 10,
110+
wantItems: history,
111+
wantCursor: "",
112+
},
113+
{
114+
name: "zero limit returns rest in one page",
115+
cursor: "",
116+
limit: 0,
117+
wantItems: history,
118+
wantCursor: "",
119+
},
120+
{
121+
name: "unknown cursor",
122+
cursor: "git://repo/ref/x",
123+
limit: 2,
124+
wantErr: true,
125+
},
126+
}
127+
for _, tt := range tests {
128+
t.Run(tt.name, func(t *testing.T) {
129+
got, err := New(history).History(context.Background(), tt.cursor, tt.limit)
130+
if tt.wantErr {
131+
require.ErrorIs(t, err, sourcecontrol.ErrNotFound)
132+
return
133+
}
134+
require.NoError(t, err)
135+
assert.Equal(t, tt.wantItems, got.Items)
136+
assert.Equal(t, tt.wantCursor, got.NextCursor)
137+
})
138+
}
139+
}
Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
load("@rules_go//go:def.bzl", "go_library")
2+
3+
go_library(
4+
name = "mock",
5+
srcs = ["sourcecontrol_mock.go"],
6+
importpath = "github.com/uber/submitqueue/stovepipe/extension/sourcecontrol/mock",
7+
visibility = ["//visibility:public"],
8+
deps = [
9+
"//platform/base/page",
10+
"//stovepipe/extension/sourcecontrol",
11+
"@org_uber_go_mock//gomock",
12+
],
13+
)
Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
# sourcecontrol mocks
2+
3+
Generated gomock mock for the `sourcecontrol.SourceControl` interface, used by controller and pipeline tests.
4+
5+
Mocks are **checked in** and produced by [mockgen](https://github.com/uber-go/mock) from the `//go:generate` directive on `sourcecontrol.go`. After changing the interface, run `make mocks` to regenerate, then `make gazelle` to update `BUILD.bazel`, and commit the result.

0 commit comments

Comments
 (0)