Skip to content

Commit 86f7142

Browse files
committed
feat: add agent-native repository guardrails
1 parent 13d9a38 commit 86f7142

77 files changed

Lines changed: 3285 additions & 332 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

README.md

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -159,4 +159,3 @@ func main() {
159159
- [Homebrew packaging](docs/homebrew.md)
160160
- [Checks reference](docs/checks.md)
161161
- [Architecture](docs/architecture.md)
162-
- [Competitive roadmap](docs/competitive-roadmap.md)

benchmarks/manifest.example.json

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
{
2+
"version": 1,
3+
"corpus": "codeguard-public-prs-v1",
4+
"entries": [
5+
{
6+
"id": "go-example-pr-1",
7+
"language": "go",
8+
"repository": "github.com/example/go-service",
9+
"pull_request": 1,
10+
"base_revision": "replace-with-immutable-base-commit",
11+
"head_revision": "replace-with-immutable-head-commit",
12+
"worktree": "go-example-pr-1",
13+
"config": ".codeguard/codeguard.yaml"
14+
}
15+
]
16+
}

cmd/codeguard-benchmark/main.go

Lines changed: 61 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,61 @@
1+
// codeguard-benchmark is a deliberately separate developer tool for frozen PR
2+
// corpus measurements. It never fetches repositories; provision them first.
3+
package main
4+
5+
import (
6+
"context"
7+
"flag"
8+
"fmt"
9+
"os"
10+
11+
"github.com/devr-tools/codeguard/internal/benchmark"
12+
)
13+
14+
func main() {
15+
if len(os.Args) < 2 {
16+
usage()
17+
os.Exit(2)
18+
}
19+
switch os.Args[1] {
20+
case "export":
21+
export(os.Args[2:])
22+
case "run":
23+
run(os.Args[2:])
24+
default:
25+
usage()
26+
os.Exit(2)
27+
}
28+
}
29+
30+
func export(args []string) {
31+
flags := flag.NewFlagSet("export", flag.ExitOnError)
32+
manifestPath := flags.String("manifest", "", "benchmark manifest JSON")
33+
out := flags.String("out", "", "corpus export JSON")
34+
_ = flags.Parse(args)
35+
manifest, err := benchmark.Load(*manifestPath)
36+
fail(err)
37+
fail(benchmark.WriteJSON(*out, manifest.Export()))
38+
}
39+
40+
func run(args []string) {
41+
flags := flag.NewFlagSet("run", flag.ExitOnError)
42+
manifestPath := flags.String("manifest", "", "benchmark manifest JSON")
43+
out := flags.String("out", "", "result JSON")
44+
binary := flags.String("binary", "codeguard", "CodeGuard binary")
45+
workRoot := flags.String("work-root", "", "directory containing provisioned worktrees")
46+
warm := flags.Int("warm-repeats", 3, "warm scan repeats per entry")
47+
_ = flags.Parse(args)
48+
manifest, err := benchmark.Load(*manifestPath)
49+
fail(err)
50+
result, err := benchmark.Run(context.Background(), manifest, benchmark.RunOptions{Binary: *binary, WorkRoot: *workRoot, WarmRepeats: *warm})
51+
fail(err)
52+
fail(benchmark.WriteJSON(*out, result))
53+
}
54+
55+
func usage() { fmt.Fprintln(os.Stderr, "usage: codeguard-benchmark <export|run> [flags]") }
56+
func fail(err error) {
57+
if err != nil {
58+
fmt.Fprintln(os.Stderr, err)
59+
os.Exit(1)
60+
}
61+
}

docs/README.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -12,4 +12,4 @@
1212
- [Homebrew packaging](homebrew.md)
1313
- [Architecture](architecture.md)
1414
- [Checks](checks.md)
15-
- [Competitive roadmap](competitive-roadmap.md)
15+
- [Frozen PR benchmarks](benchmarks.md)

docs/ai-quality.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -47,6 +47,7 @@ This brief tracks the AI-generated-code quality features currently implemented i
4747
- Verified auto-fix
4848
- `codeguard.VerifyFix(...)` and `codeguard.GenerateVerifiedFix(ctx, req)` only return patches after diff-scoped verification and inferred or explicit verification tests pass in an isolated workspace
4949
- `codeguard fix -ai` exposes the same verified-fix flow from the CLI for one selected finding
50+
- `codeguard fix-batch -input fixes.json` verifies explicitly supplied, catalogued deterministic fixes together in one isolated workspace and returns only their aggregate patch. It never modifies the working tree. The input is a JSON object with an `items` array of `{ "finding": { ... }, "candidate": { "diff": "..." } }` entries; use `-format json` to retain included, skipped, and failed item details.
5051
- Natural-language custom rules
5152
- custom rule packs can use `natural_language` instructions alongside regex and path matchers
5253
- evaluation is command-driven through the optional AI runtime and produces normal custom-rule findings

docs/benchmarks.md

Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,41 @@
1+
# Frozen PR benchmarks
2+
3+
CodeGuard benchmarks PR-time scans using a small, versioned corpus of frozen
4+
public pull-request checkouts. The repository does not fetch or vendor those
5+
projects: the benchmark operator provisions each checkout at the exact commit
6+
listed in the manifest, then runs the harness locally or in a dedicated CI
7+
job.
8+
9+
The manifest schema is versioned and machine-readable. Each entry records a
10+
repository, PR number, immutable base/head revisions, language, a worktree
11+
name relative to `-work-root`, and a relative CodeGuard configuration path.
12+
Use [manifest.example.json](../benchmarks/manifest.example.json) as the
13+
onboarding template. Do not replace immutable revisions with branch names.
14+
15+
Export the corpus identity for an auditable result bundle:
16+
17+
```sh
18+
go run ./cmd/codeguard-benchmark export \
19+
-manifest benchmarks/manifest.json -out corpus.json
20+
```
21+
22+
After provisioning the listed worktrees beneath a single directory, measure
23+
each diff scan:
24+
25+
```sh
26+
go run ./cmd/codeguard-benchmark run \
27+
-manifest benchmarks/manifest.json \
28+
-work-root /private/tmp/codeguard-benchmark-worktrees \
29+
-binary ./dist/codeguard -warm-repeats 3 -out results.json
30+
```
31+
32+
Results contain a first-process `cold` run and the requested `warm` repeats
33+
for each entry. “Cold” means a fresh CodeGuard process; it does not claim to
34+
clear the host filesystem cache or alter the repository's configured cache.
35+
Record p50/p95 separately for cold and warm runs, and include the exported
36+
corpus metadata next to any published figures.
37+
38+
Runtime is only one benchmark lane. Use the existing ground-truth detector
39+
corpus for precision/noise, and verified-fix fixtures for proposal coverage,
40+
verifier acceptance, and independently validated acceptance. Do not collapse
41+
the three lanes into a single competitive score.

docs/checks.md

Lines changed: 37 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -46,7 +46,43 @@ Each top-level boolean enables or disables an entire check family.
4646

4747
`context` covers agent-context legibility: when the key is omitted the family defaults to enabled in full scans and disabled in diff scans; see [Agent Context](#agent-context).
4848

49-
`supply_chain` is opt-in and currently covers normalized manifest parsing plus initial policy checks for missing lockfiles, content-based lockfile drift validation, unpinned dependencies, dependency license policy resolved from local manifest and installed metadata where available, and Cargo manifest hygiene for missing package licenses and non-hermetic dependency sources.
49+
`supply_chain` is opt-in and currently covers normalized manifest parsing plus initial policy checks for missing lockfiles, content-based lockfile drift validation, unpinned dependencies, dependency license policy resolved from local manifest and installed metadata where available, local advisory-cache vulnerability matching, and Cargo manifest hygiene for missing package licenses and nonhermetic dependency sources.
50+
51+
Set `output.format` to `cyclonedx` (or pass `codeguard scan -format cyclonedx`) to emit the normalized dependency artifacts as deterministic CycloneDX 1.6 JSON. The SBOM contains declared dependency versions or requirements when a resolver version is unavailable; it does not execute project code or contact a registry.
52+
53+
### Offline advisory cache
54+
55+
Vulnerability matching is opt-in. It reads a local, versioned JSON cache only; CodeGuard never contacts an advisory service during a scan. Configure the cache relative to the target root (or provide an absolute path):
56+
57+
```yaml
58+
checks:
59+
supply_chain: true
60+
supply_chain_rules:
61+
detect_vulnerabilities: true
62+
advisory_cache_path: .codeguard/advisories.json
63+
```
64+
65+
The first supported cache schema is `schema_version: 1`. Each advisory has an ecosystem matching CodeGuard's normalized ecosystem (`go`, `npm`, `python`, or `cargo`), a package, and one or more comma-separated version comparators. Matching is restricted to concrete pinned dependency versions to avoid claims based on unresolved ranges.
66+
67+
```json
68+
{
69+
"schema_version": 1,
70+
"generated_at": "2026-07-21T00:00:00Z",
71+
"source": "approved-advisory-export-2026-07-21",
72+
"advisories": [
73+
{
74+
"id": "CVE-2026-12345",
75+
"ecosystem": "npm",
76+
"package": "example-library",
77+
"affected_versions": [">=1.0.0, <1.2.4"],
78+
"fixed_version": "1.2.4",
79+
"url": "https://example.invalid/advisories/CVE-2026-12345"
80+
}
81+
]
82+
}
83+
```
84+
85+
Findings contain the advisory identifier, source, generated timestamp, and cache age as non-sensitive metadata. Refreshing the cache is intentionally outside scan execution and should be handled by an approved, auditable update process.
5086

5187
`contracts` covers API compatibility against a diff base. When omitted, it is enabled in diff scans and disabled in full scans. It checks exported Go declarations, public C++ headers, OpenAPI documents, protobuf schemas, and destructive migrations.
5288

0 commit comments

Comments
 (0)