Skip to content

Add GitHub discovery integration MVP#1484

Draft
gearnode wants to merge 37 commits into
mainfrom
cursor/github-integration-discovery-81c1
Draft

Add GitHub discovery integration MVP#1484
gearnode wants to merge 37 commits into
mainfrom
cursor/github-integration-discovery-81c1

Conversation

@gearnode

@gearnode gearnode commented Jul 13, 2026

Copy link
Copy Markdown
Contributor

Summary

Addresses workflow discovery gaps: substring matching missed YAML variants, and measures counted configured tools without verifying they actually run.

Running evidence now comes from both:

  • GET /repos/{owner}/{repo}/commits/{sha}/check-runs
  • GET /repos/{owner}/{repo}/commits/{sha}/status (legacy commit statuses)

Two-layer model:

Layer Source Used for
Configured YAML parse (on: triggers, uses: actions) Risk signals: pull_request_target, workflow secrets
Running Check runs + commit statuses on default branch and recent merged PR Positive measures: CodeQL, SAST, dep scan, PR CI

External CI (CircleCI, Jenkins, etc.) often reports only via commit status context — PR CI detection now accepts known CI providers from statuses on merged PR commits.

Test plan

  • go test ./pkg/discovery/github/...
  • make go-fmt
  • go fix clean
Open in Web Open in Cursor 

Summary by cubic

Adds a read-only GitHub discovery that scans org governance and repos, builds a MeasurePlan from facts, and links measures to a level‑1 GitHub ThirdParty. CI evidence now verifies tools via check runs and commit statuses on default branches and merged PRs; YAML parsing stays for risk signals like pull_request_target.

Tests +1243 -41

  • Coverage for vfs (walk/glob, gitfs), workflow parsing/run signals incl. commit statuses, repo selection/classification (heuristics + optional LLM), scanners, MeasurePlan by measure name, third‑party linking, and worktree‑indexed discovery.
  • Updates for agent‑run worker/registry and provider discovery scopes.

GraphQL API +95 -0

  • Adds runGitHubDiscovery(input: { connectorId }) returning an agentRun.
  • Exposes discoveryOauth2Scopes; CreateMeasureInput accepts thirdPartyIds. Wires the GitHub discovery service.

MCP +192 -20

  • Adds runGitHubDiscovery tool and AgentRun types/statuses.
  • Extends AddMeasureInput with third_party_ids.

prb (CLI) +16 -10

  • measure create supports --third-party-ids.
  • Switch asset/datum flags to kebab‑case --third-party-ids.

Service +7577 -16

  • New pkg/discovery/github: org/repo/governance/PR/status scanners, MeasurePlan builder/synthesizer, runner/run handler, persistence; joins by measure name and links to a level‑1 GitHub ThirdParty.
  • CI measures counted from check runs and commit statuses; YAML analysis kept for risk signals.
  • Discovery vfs: Walk/Glob, gitfs shallow clones, layered FS fallback; repo selection via heuristics with optional LLM.
  • Validates discovery OAuth scopes; core calls use google/go-github on the instrumented httpclient; registers a run handler on the shared agent‑run worker.
  • Minor fixes: go fix updates and lint cleanups.

Package: n8n-node +19 -0

  • Adds “Third Party IDs” to measure.create (CSV to thirdPartyIds).

Other +976 -0

  • Design doc and Helm values/env for discovery worker and LLM settings.
  • Adds github.com/google/go-github/v69 dependency.

Written for commit 7edb499. Summary will update on new commits.

Review in cubic

cursoragent and others added 30 commits July 13, 2026 14:39
Wire a read-only GitHub org scanner that enqueues agent_runs,
materializes posture measures with evidence, and links them to a
level-1 GitHub third party. Includes worker orchestration, optional
LLM synthesis with deterministic fallback, Console trigger mutation,
and probod configuration.

Signed-off-by: Cursor Agent <cursoragent@cursor.com>

Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
The console handler reads GitHubDiscovery from api.Config, not
server.Config. Add the missing field so probod builds.

Signed-off-by: Cursor Agent <cursoragent@cursor.com>

Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Discovery reuses the GitHub provider registration instead of adding a
new provider. Baseline access-review scopes stay on connect; additional
discovery scopes are requested through the existing reconnect flow with
incremental auth. Validate required scopes before enqueueing a run and
expose escalation scopes on Connector for the console reconnect UX.

Signed-off-by: Cursor Agent <cursoragent@cursor.com>

Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
The discovery worker, scanner, and synthesis logic belong in a
discovery namespace rather than a generic integration folder.

Signed-off-by: Cursor Agent <cursoragent@cursor.com>

Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Add runGitHubDiscovery to the MCP API so agents and skills trigger
discovery the same way SCIM bridge provisioning uses connector-scoped
MCP tools. The tool enqueues an agent run via the shared discovery
service instead of custom agent tooling or a GraphQL-only path.

Signed-off-by: Cursor Agent <cursoragent@cursor.com>

Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Let addMeasure and createMeasure accept optional third_party_ids so
agents can attach a measure to a vendor in one call instead of a
separate linkMeasure or mapping mutation.

Signed-off-by: Cursor Agent <cursoragent@cursor.com>

Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
CI go fix rejects the legacy strings.Split loop when SplitSeq is
available on the project's Go toolchain.

Signed-off-by: Cursor Agent <cursoragent@cursor.com>

Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Add blank lines required by golangci-lint wsl_v5 between
statement groups before conditional checks.

Signed-off-by: Cursor Agent <cursoragent@cursor.com>

Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Group related type declarations, wrap errors with distinct
cannot messages, build evidence URLs with net/url, rename the
internal connector HTTP helper, and update the synthesis prompt
to the standard XML structure.

Signed-off-by: Cursor Agent <cursoragent@cursor.com>

Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Follow the design resolution order: import from the common
catalog when available, match an existing level-1 org vendor
case-insensitively, then create GitHub when the integration is
connected but no vendor record exists yet.

Signed-off-by: Cursor Agent <cursoragent@cursor.com>

Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Add repo, governance, and security scanners that collect facts
for branch protection, workflows, documentation, Dependabot,
secret scanning, and code scanning. Materialize org-aggregate
measures from those facts, gate deep checks on granted OAuth
scopes, and document the fact catalog in checks.json.

Signed-off-by: Cursor Agent <cursoragent@cursor.com>

Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Extend repo scanning with branch protection depth,
workflow YAML analysis, production classification,
lockfiles, docs, push protection, CodeQL setup,
deploy keys, commit signatures, and env file probes.
Materialize the expanded fact catalog and document
all checks in checks.json.

Signed-off-by: Cursor Agent <cursoragent@cursor.com>

Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Close remaining P0 gaps: external CI via commit statuses,
de-facto PR review sampling, SECURITY.md contact parsing,
org .github profile scan, and incident-response docs.

Signed-off-by: Cursor Agent <cursoragent@cursor.com>

Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Introduce pkg/discovery/vfs with OrgFS and RepositoryFS
interfaces, a memory backend for tests, and a GitHub driver
that indexes files via code search. Refactor the scanner to
use the vfs layer, remove the 30-repo sample cap, and fall
back to per-repo reads when search is unavailable.

Signed-off-by: Cursor Agent <cursoragent@cursor.com>

Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Replace OrgFS and RepositoryFS with one read-only vfs.FS
using workspace-rooted paths like repo/file. GitHub and
memory backends implement the same interface for reuse
across future git integrations.

Signed-off-by: Cursor Agent <cursoragent@cursor.com>

Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Replace Exists and Search with Read, ReadDir, and Glob on
vfs.FS. Move GitHub code search behind Glob in the GitHub
driver. Add HasPath helper and glob matching for discovery
indexing and future agent reuse.

Signed-off-by: Cursor Agent <cursoragent@cursor.com>

Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Use slices.ContainsFunc and maps.Copy so the lint-go
go fix check passes in CI.

Signed-off-by: Cursor Agent <cursoragent@cursor.com>

Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Add pkg/discovery/vfs/gitfs backed by go-git shallow clones
into memory worktrees. Discovery builds a multi-repo workspace
for Read, ReadDir, and Glob, with GitHub API file reads as
fallback when clone is unavailable.

Signed-off-by: Cursor Agent <cursoragent@cursor.com>

Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Rank repositories with production-oriented heuristics
and clone at most fifty relevant repos via go-git.
Non-cloned repos still use API file reads through a
layered vfs fallback during discovery scans.

Signed-off-by: Cursor Agent <cursoragent@cursor.com>

Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Skip repositories above 100 MiB on the GitHub size metric and
document depth-1 single-branch clones with no tags or submodules.
Oversized repos fall back to API file reads instead of cloning.

Signed-off-by: Cursor Agent <cursoragent@cursor.com>

Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Large repositories are still cloned when selected; shallow
depth-1 single-branch clones avoid pulling full history.

Signed-off-by: Cursor Agent <cursoragent@cursor.com>

Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Align asset and datum create/update commands with the CLI
kebab-case flag convention already used by measure create.

Signed-off-by: Cursor Agent <cursoragent@cursor.com>

Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Remove the dedicated github-discovery worker and its
LoadNextPendingGitHubDiscoveryForUpdateSkipLocked query. Register
github-discovery as a run handler on the shared agent-run worker
that already claims pending agent_runs rows.

Signed-off-by: Cursor Agent <cursoragent@cursor.com>

Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Score clone priority and production likelihood from GitHub metadata,
branch protection, and workflow probes. Refine ambiguous repositories
with the github-discovery LLM when configured.

Signed-off-by: Cursor Agent <cursoragent@cursor.com>

Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Route core GitHub discovery calls through google/go-github on the
kit httpclient so OAuth traffic gets logging, tracing, and metrics.
Keep legacy getJSON for unmigrated endpoints until follow-up.

Signed-off-by: Cursor Agent <cursoragent@cursor.com>

Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Let discovery warm GitHub code search queries from an LLM batch
while keeping the static map as fallback when LLM is off or misses.

Signed-off-by: Cursor Agent <cursoragent@cursor.com>

Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Add blank lines required by wsl_v5 in probod wiring and
github discovery classifier helpers.

Signed-off-by: Cursor Agent <cursoragent@cursor.com>

Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Use integer range loops for pagination caps so CI go fix
check passes.

Signed-off-by: Cursor Agent <cursoragent@cursor.com>

Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Use buildMeasurePlanFromFacts and measurePlanRule so the
deterministic path matches Synthesizer and MeasurePlan terms.

Signed-off-by: Cursor Agent <cursoragent@cursor.com>

Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Merge the former p0 rule set into defaultMeasurePlanRules
and drop tier labels from checks.json.

Signed-off-by: Cursor Agent <cursoragent@cursor.com>

Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
cursoragent and others added 7 commits July 13, 2026 18:14
Collapse fact_id and fact_key into one Check type with Go
constants as the single source of truth. Remove checks.json
and wire measure plans through check_refs instead.

Signed-off-by: Cursor Agent <cursoragent@cursor.com>

Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Facts, scanners, and measure-plan rules now share the same
measure title as their join key. This removes a redundant
identifier layer on top of name and description.

Signed-off-by: Cursor Agent <cursoragent@cursor.com>

Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Remove the GlobQueryResolver interface and LLM warm-up path.
Known discovery patterns map directly to GitHub code search
queries inside githubvfs.

Signed-off-by: Cursor Agent <cursoragent@cursor.com>

Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Move glob matching into vfs.Walk and vfs.GlobFS so any FS with
ReadDir can index discovery files. BuildDiscoveryIndex walks once
instead of globbing per pattern. Drop GitHub code search from the
API fallback filesystem.

Signed-off-by: Cursor Agent <cursoragent@cursor.com>

Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Parse workflow triggers and action uses from YAML instead of
substring matching. Count CI security measures only when matching
check runs are observed on recent default-branch or merged PR
commits. Keep workflow file parsing for risk signals such as
pull_request_target.

Signed-off-by: Cursor Agent <cursoragent@cursor.com>

Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
CI lint-go runs go fix and requires the suggested loop
simplification in hasWorkflowEvent.

Signed-off-by: Cursor Agent <cursoragent@cursor.com>

Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Merge legacy commit statuses with check runs when verifying
that security tools and CI actually ran. External providers such
as CircleCI often report only via commit status context.

Signed-off-by: Cursor Agent <cursoragent@cursor.com>

Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants