Add GitHub discovery integration MVP#1484
Draft
gearnode wants to merge 37 commits into
Draft
Conversation
Wire a read-only GitHub org scanner that enqueues agent_runs, materializes posture measures with evidence, and links them to a level-1 GitHub third party. Includes worker orchestration, optional LLM synthesis with deterministic fallback, Console trigger mutation, and probod configuration. Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
The console handler reads GitHubDiscovery from api.Config, not server.Config. Add the missing field so probod builds. Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Discovery reuses the GitHub provider registration instead of adding a new provider. Baseline access-review scopes stay on connect; additional discovery scopes are requested through the existing reconnect flow with incremental auth. Validate required scopes before enqueueing a run and expose escalation scopes on Connector for the console reconnect UX. Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
The discovery worker, scanner, and synthesis logic belong in a discovery namespace rather than a generic integration folder. Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Add runGitHubDiscovery to the MCP API so agents and skills trigger discovery the same way SCIM bridge provisioning uses connector-scoped MCP tools. The tool enqueues an agent run via the shared discovery service instead of custom agent tooling or a GraphQL-only path. Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Let addMeasure and createMeasure accept optional third_party_ids so agents can attach a measure to a vendor in one call instead of a separate linkMeasure or mapping mutation. Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
CI go fix rejects the legacy strings.Split loop when SplitSeq is available on the project's Go toolchain. Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Add blank lines required by golangci-lint wsl_v5 between statement groups before conditional checks. Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Group related type declarations, wrap errors with distinct cannot messages, build evidence URLs with net/url, rename the internal connector HTTP helper, and update the synthesis prompt to the standard XML structure. Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Follow the design resolution order: import from the common catalog when available, match an existing level-1 org vendor case-insensitively, then create GitHub when the integration is connected but no vendor record exists yet. Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Add repo, governance, and security scanners that collect facts for branch protection, workflows, documentation, Dependabot, secret scanning, and code scanning. Materialize org-aggregate measures from those facts, gate deep checks on granted OAuth scopes, and document the fact catalog in checks.json. Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Extend repo scanning with branch protection depth, workflow YAML analysis, production classification, lockfiles, docs, push protection, CodeQL setup, deploy keys, commit signatures, and env file probes. Materialize the expanded fact catalog and document all checks in checks.json. Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Close remaining P0 gaps: external CI via commit statuses, de-facto PR review sampling, SECURITY.md contact parsing, org .github profile scan, and incident-response docs. Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Introduce pkg/discovery/vfs with OrgFS and RepositoryFS interfaces, a memory backend for tests, and a GitHub driver that indexes files via code search. Refactor the scanner to use the vfs layer, remove the 30-repo sample cap, and fall back to per-repo reads when search is unavailable. Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Replace OrgFS and RepositoryFS with one read-only vfs.FS using workspace-rooted paths like repo/file. GitHub and memory backends implement the same interface for reuse across future git integrations. Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Replace Exists and Search with Read, ReadDir, and Glob on vfs.FS. Move GitHub code search behind Glob in the GitHub driver. Add HasPath helper and glob matching for discovery indexing and future agent reuse. Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Use slices.ContainsFunc and maps.Copy so the lint-go go fix check passes in CI. Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Add pkg/discovery/vfs/gitfs backed by go-git shallow clones into memory worktrees. Discovery builds a multi-repo workspace for Read, ReadDir, and Glob, with GitHub API file reads as fallback when clone is unavailable. Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Rank repositories with production-oriented heuristics and clone at most fifty relevant repos via go-git. Non-cloned repos still use API file reads through a layered vfs fallback during discovery scans. Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Skip repositories above 100 MiB on the GitHub size metric and document depth-1 single-branch clones with no tags or submodules. Oversized repos fall back to API file reads instead of cloning. Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Large repositories are still cloned when selected; shallow depth-1 single-branch clones avoid pulling full history. Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Align asset and datum create/update commands with the CLI kebab-case flag convention already used by measure create. Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Remove the dedicated github-discovery worker and its LoadNextPendingGitHubDiscoveryForUpdateSkipLocked query. Register github-discovery as a run handler on the shared agent-run worker that already claims pending agent_runs rows. Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Score clone priority and production likelihood from GitHub metadata, branch protection, and workflow probes. Refine ambiguous repositories with the github-discovery LLM when configured. Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Route core GitHub discovery calls through google/go-github on the kit httpclient so OAuth traffic gets logging, tracing, and metrics. Keep legacy getJSON for unmigrated endpoints until follow-up. Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Let discovery warm GitHub code search queries from an LLM batch while keeping the static map as fallback when LLM is off or misses. Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Add blank lines required by wsl_v5 in probod wiring and github discovery classifier helpers. Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Use integer range loops for pagination caps so CI go fix check passes. Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Use buildMeasurePlanFromFacts and measurePlanRule so the deterministic path matches Synthesizer and MeasurePlan terms. Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Merge the former p0 rule set into defaultMeasurePlanRules and drop tier labels from checks.json. Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Collapse fact_id and fact_key into one Check type with Go constants as the single source of truth. Remove checks.json and wire measure plans through check_refs instead. Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Facts, scanners, and measure-plan rules now share the same measure title as their join key. This removes a redundant identifier layer on top of name and description. Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Remove the GlobQueryResolver interface and LLM warm-up path. Known discovery patterns map directly to GitHub code search queries inside githubvfs. Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Move glob matching into vfs.Walk and vfs.GlobFS so any FS with ReadDir can index discovery files. BuildDiscoveryIndex walks once instead of globbing per pattern. Drop GitHub code search from the API fallback filesystem. Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Parse workflow triggers and action uses from YAML instead of substring matching. Count CI security measures only when matching check runs are observed on recent default-branch or merged PR commits. Keep workflow file parsing for risk signals such as pull_request_target. Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
CI lint-go runs go fix and requires the suggested loop simplification in hasWorkflowEvent. Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Merge legacy commit statuses with check runs when verifying that security tools and CI actually ran. External providers such as CircleCI often report only via commit status context. Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Addresses workflow discovery gaps: substring matching missed YAML variants, and measures counted configured tools without verifying they actually run.
Running evidence now comes from both:
GET /repos/{owner}/{repo}/commits/{sha}/check-runsGET /repos/{owner}/{repo}/commits/{sha}/status(legacy commit statuses)Two-layer model:
on:triggers,uses:actions)pull_request_target, workflow secretsExternal CI (CircleCI, Jenkins, etc.) often reports only via commit status context — PR CI detection now accepts known CI providers from statuses on merged PR commits.
Test plan
go test ./pkg/discovery/github/...make go-fmtgo fixcleanSummary by cubic
Adds a read-only GitHub discovery that scans org governance and repos, builds a MeasurePlan from facts, and links measures to a level‑1 GitHub ThirdParty. CI evidence now verifies tools via check runs and commit statuses on default branches and merged PRs; YAML parsing stays for risk signals like pull_request_target.
Tests
+1243-41gitfs), workflow parsing/run signals incl. commit statuses, repo selection/classification (heuristics + optional LLM), scanners, MeasurePlan by measure name, third‑party linking, and worktree‑indexed discovery.GraphQL API
+95-0runGitHubDiscovery(input: { connectorId })returning anagentRun.discoveryOauth2Scopes;CreateMeasureInputacceptsthirdPartyIds. Wires the GitHub discovery service.MCP
+192-20runGitHubDiscoverytool andAgentRuntypes/statuses.AddMeasureInputwiththird_party_ids.prb (CLI)
+16-10measure createsupports--third-party-ids.--third-party-ids.Service
+7577-16pkg/discovery/github: org/repo/governance/PR/status scanners, MeasurePlan builder/synthesizer, runner/run handler, persistence; joins by measure name and links to a level‑1 GitHub ThirdParty.Walk/Glob,gitfsshallow clones, layered FS fallback; repo selection via heuristics with optional LLM.google/go-githubon the instrumentedhttpclient; registers a run handler on the shared agent‑run worker.go fixupdates and lint cleanups.Package: n8n-node
+19-0measure.create(CSV tothirdPartyIds).Other
+976-0github.com/google/go-github/v69dependency.Written for commit 7edb499. Summary will update on new commits.