Skip to content

fix(mcp,selfhost): audit mutating MCP tools, dedupe predict_gate agreement, guard repo privacy - #9173

Merged
JSONbored merged 1 commit into
mainfrom
fix/mcp-audit-metric-privacy-9137-9138-9142
Jul 27, 2026
Merged

fix(mcp,selfhost): audit mutating MCP tools, dedupe predict_gate agreement, guard repo privacy#9173
JSONbored merged 1 commit into
mainfrom
fix/mcp-audit-metric-privacy-9137-9138-9142

Conversation

@JSONbored

Copy link
Copy Markdown
Owner

Summary

  • orb(mcp): every MCP write tool is unaudited — the kill switch, the autonomy dial, private-config writes and redeploys leave no audit_events row #9137 — every mutating MCP tool now records an audit_events row. Added recordAuditEvent calls to loopover_set_agent_paused, loopover_set_action_autonomy (both repo.settings_updated, mirroring PUT /v1/repos/:owner/:repo/settings's own audit shape), loopover_admin_write_config (new config.private_write event type, audited on both success and failure), and loopover_admin_trigger_redeploy (new instance.redeploy_triggered, audited on success, a failed run, and a companion connection error). I deliberately did not move the audit call down into upsertRepositorySettings itself (the "better" option the issue floats): that function has ~8 call sites across src/api/routes.ts, src/orb/apr-repo-transfer.ts, and several test files, and moving the audit there would require threading an actor parameter through all of them and re-auditing each call site for double-recording risk against the route layer's existing call — more invasive than a time-boxed pass justified. Added a table-driven invariant test (test/unit/mcp-mutating-tools-audit.test.ts) asserting all four tools produce the expected row, actor, and metadata.
  • orb(mcp): predict_gate writes an un-deduped row per call and all of them pair against one real decision — a contributor can drive the agreement metric to 100% #9138computePredictedGateVerdict (backing both loopover_predict_gate and loopover_explain_gate_disposition) now calls enforceToolRateLimit, sharing the sibling slop-oracle tools' 20-per-300s cap instead of relying solely on the shared /mcp route class (120/min). computePredictedGateAgreement now dedupes multiple predicted calls that pair against the same real decision down to just the latest one (keyed by object identity of the paired real-decision row), so N predictions before one real merge contribute exactly one paired sample instead of N — closing the lever that let a contributor drive the agreement metric toward 100%. Gave predicted_gate_calls a 90-day retention window via RETENTION_POLICY (the same scheduled prune job that already covers audit_events/ai_usage_events/etc). Operational follow-up, not something computable from this PR: the currently-published agreement figure should be re-derived after this dedup lands, and whether the prior number was inflated should be stated separately.
  • orb(privacy): private repo names and PR numbers ship to the vendor's PostHog and to unauthenticated /metrics, with no tenant opt-out — and the scrubber corrupts the repo label while doing it #9142operationalProperties in src/selfhost/posthog.ts now HMACs repo/repository/owner/pull/pullNumber/pr/head_sha with the same per-instance secret orb-collector.ts's exportOrbBatch already generates and persists (system_flags key orb:anon_secret), but only when the active key is the shared LOOPOVER_CENTRAL_POSTHOG_KEY fallback — an operator's own POSTHOG_API_KEY is unaffected (their own private project, no cross-tenant concern). Fails closed: if the secret hasn't been injected yet (an early-boot race), the identifying field is dropped, never sent raw. initPostHog now honors ORB_AIR_GAP=true and an explicit POSTHOG_DISABLED, and an explicitly-empty POSTHOG_API_KEY="" now means off instead of falling through to the central key. Self-hosted /metrics now pseudonymizes (not strips, not raw) the repo label on PRIVATE_REPO_LABEL_METRICS by default — the same redacted-N scheme ALWAYS_REDACT_REPO_LABEL_METRICS already uses — since a self-hosted operator legitimately wants a stable per-repo series on their own dashboards, just not the real name leaving on an endpoint that may sit in front of application auth; an operator who has verified /metrics never leaves their private network can opt back into raw labels with LOOPOVER_METRICS_REPO_LABELS=raw. observe() now routes through publicLabelsForMetric (previously the one metric-recording path that bypassed redaction entirely, including the always-redact set). Fixed the scrubber bug where PUBLIC_UNSAFE_SCRUB's bare vocabulary match corrupted structured identifiers (ossf/scorecardossf/private context) by skipping vocabulary/phrase scrubbing for OPERATIONAL_TAG_KEYS while keeping credential-shape scrubbing (SECRET_VALUE/JWT_VALUE/QUERY_SECRET_VALUE) unconditional.
  • Drive-by: found and fixed two stray NUL bytes in predicted-gate-agreement.ts's map-key-join template literal (pre-existing corruption already on main, sitting on the exact lines the orb(mcp): predict_gate writes an un-deduped row per call and all of them pair against one real decision — a contributor can drive the agreement metric to 100% #9138 fix touches) — both the writer and reader side were consistently corrupted the same way so it never affected runtime behavior, but it made every past git diff/GitHub PR view of that file render as "Binary files differ" instead of a real line diff. Because origin/main's committed blob still has the NUL bytes, this PR's own diff for that one file will likely still render as binary (git's binary-vs-text check looks at both the old and new blob) — but the new blob is clean, so every diff of this file after this PR merges will render normally again.

Closes #9137
Closes #9138
Closes #9142

Scope

Validation

  • git diff --check
  • npm run actionlint
  • npm run typecheck
  • npm run test:coverage (full, unsharded) — not run locally, see note below; scoped coverage verification done instead.
  • npm run test:workers — not run locally (no test/workers/** files touched).
  • npm run build:mcp / npm run test:mcp-pack — not run locally (no packages/loopover-mcp/** changes).
  • npm run ui:openapi:check (no drift — no route/schema changes in this PR)
  • npm run ui:lint / npm run ui:typecheck / npm run ui:build — not run (no apps/loopover-ui/** source changes beyond the regenerated env-reference doc below).
  • npm audit --audit-level=moderate — not run locally, see note below.
  • New/changed behavior has unit/integration tests for new branches, fallback paths, and sanitizer boundaries (see below).

If any required check was skipped, explain why:

  • Per explicit instruction for this task, the full local gate (npm run test:ci, npm audit) was intentionally not run before pushing — GitHub CI runs it. In its place I ran and verified: npm run typecheck (clean), npm run db:migrations:check / db:schema-drift:check (no migration needed — no schema changes), npm run selfhost:env-reference:check (regenerated apps/loopover-ui/src/lib/selfhost-env-reference.ts for the two new self-host env vars this PR introduces: POSTHOG_DISABLED and LOOPOVER_METRICS_REPO_LABELS), npm run cf-typegen:check (clean — no wrangler.jsonc changes), npm run ui:openapi:check (clean), npm run coverage-boltons:check (clean — new test files don't match the bolt-on pattern), git diff --check, and npm run actionlint.
  • Targeted test runs: every new/modified test file passes (mcp-mutating-tools-audit.test.ts, mcp-predict-gate.test.ts, mcp-automation-state.test.ts, mcp-admin-config-tools.test.ts, mcp-admin-redeploy-tool.test.ts, predicted-gate-agreement.test.ts, retention.test.ts, selfhost-posthog.test.ts, selfhost-metrics.test.ts, selfhost-redaction-scrub.test.ts), plus a full npm run test:changed pass (vitest's import-graph-based selection against main): 456 test files / 11,182 tests green, 0 failures.
  • Coverage: rather than a full unsharded test:coverage run, I ran scoped vitest --coverage passes against just the relevant test files and inspected the per-file JSON coverage detail (statement/branch hit maps) cross-referenced line-by-line against this PR's actual diff. Every changed line/branch in src/db/retention.ts, src/mcp/server.ts (all five touched regions), src/review/predicted-gate-agreement.ts, src/selfhost/metrics.ts, src/selfhost/posthog.ts, and src/selfhost/redaction-scrub.ts is covered — 100% lines/branches on every changed region, confirmed by diffing uncovered-line output against this PR's patch. src/server.ts's two changed call sites are in codecov.yml's ignore list (Docker-boot-only entrypoint, not unit-coverable) so they carry no coverage obligation.
  • npm audit --audit-level=moderate reports 5 pre-existing high-severity advisories in the eslint/brace-expansion/minimatch devDependency chain, unrelated to and unchanged by this PR (no new dependency added, no package.json/lockfile touched).

Safety

  • No secrets, wallet details, hotkeys, coldkeys, user PATs, private keys, raw trust scores, private rankings, or private maintainer evidence are exposed. (This PR's entire point is reducing what leaves the box — no new leakage introduced.)
  • Public GitHub text stays sanitized, low-noise, and does not imply compensation guarantees or optimization tactics. (No public-facing surface changed.)
  • Auth, cookie, CORS, GitHub App, Cloudflare, or session changes include negative-path tests. — N/A, no auth/session/CORS changes.
  • API/OpenAPI/MCP behavior is updated and tested where needed (four MCP tool handlers gained audit calls + a rate limit; covered by new/updated tests; no HTTP route or OpenAPI shape changed).
  • UI changes use live API data — N/A, no UI changes.
  • UI Evidence — N/A, no visible/frontend changes.
  • Public docs/changelogs — not touched (no changelog edit, per house rule).

UI Evidence

Not applicable — this PR is backend/MCP-only (no apps/loopover-ui/** source changes beyond the regenerated, auto-generated env-reference doc).

Notes

…ement, guard repo privacy

Three related ORB gaps in one pass:

- Every mutating MCP tool (agent pause, autonomy dial, private-config write,
  redeploy trigger) now records an audit_events row, mirroring the HTTP
  settings route's own audit shape. Previously none of them left any
  forensic trace of who flipped the autonomy dial or paused the agent.

- computePredictedGateVerdict (backing predict_gate/explain_gate_disposition)
  now shares the sibling tools' per-actor rate limit, and the predicted-vs-
  live agreement report dedupes multiple predicted calls against the same
  real decision down to the latest one, so a contributor iterating on
  predict_gate can no longer inflate their own agreement sample. Gave
  predicted_gate_calls a 90-day retention window alongside the other
  append-only log tables.

- PostHog no longer sends raw repo/PR/owner/SHA identifiers when reporting
  through the shared central key: they're HMAC'd with the same per-instance
  secret orb-collector.ts already uses. initPostHog now honors ORB_AIR_GAP
  and an explicit POSTHOG_DISABLED, and an empty POSTHOG_API_KEY means off
  instead of falling through to the central key. Self-hosted /metrics now
  pseudonymizes the repo label on the private-repo-scoped counters by
  default (with an explicit opt-out for an operator who wants raw labels
  on a verified private network), and observe() now routes through the
  same redaction path incr()/gaugeVector() already use. Fixed a scrubber
  bug where the vocabulary redaction pattern corrupted structured
  identifiers like repo names (e.g. "ossf/scorecard" -> "ossf/private
  context") by scoping vocabulary scrubbing away from identifier-shaped
  keys while keeping credential-shape scrubbing unconditional.

Also fixed two stray NUL bytes in predicted-gate-agreement.ts's map-key
join (pre-existing source corruption that made every past diff of that
file render as binary) while touching the exact lines they sat on.

Closes #9137
Closes #9138
Closes #9142
@superagent-security

Copy link
Copy Markdown
Contributor

Superagent didn't find any vulnerabilities or security issues in this PR.

@cloudflare-workers-and-pages

Copy link
Copy Markdown

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Preview URL Updated (UTC)
✅ Deployment successful!
View logs
loopover-ui d93ea15 Commit Preview URL

Branch Preview URL
Jul 27 2026, 05:34 AM

@JSONbored
JSONbored merged commit c91686a into main Jul 27, 2026
5 of 7 checks passed
@JSONbored
JSONbored deleted the fix/mcp-audit-metric-privacy-9137-9138-9142 branch July 27, 2026 05:35
@loopover-orb loopover-orb Bot added the gittensor:bug Gittensor-scored bug fix — scores a 0.05x multiplier. label Jul 27, 2026
@codecov

codecov Bot commented Jul 27, 2026

Copy link
Copy Markdown

❌ 1 Tests Failed:

Tests completed Failed Passed Skipped
21898 1 21897 21
View the top 1 failed test(s) by shortest run time
test/unit/check-branding-drift-script.test.ts > check-branding-drift script (real repo state) > the committed baseline matches the real current repo state (regression guard)
Stack Traces | 0.344s run time
Error: Command failed: .../hostedtoolcache/node/22.23.1.../x64/bin/node --experimental-strip-types scripts/check-branding-drift.ts
Branding-drift check found 1 issue(s):
src/review/predicted-gate-agreement.ts: "gittensory" mentions increased from 0 to 1 -- looks like new branding drift, not an intentional historical reference. If it genuinely belongs (e.g. a permanent Sentry ticket ID or a stable comment-marker already posted to live PRs), run `npm run branding-drift:update` and commit the regenerated baseline.

 ❯ test/unit/check-branding-drift-script.test.ts:143:20

⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯
Serialized Error: { status: 1, signal: null, output: [ null, '', 'Branding-drift check found 1 issue(s):\nsrc/review/predicted-gate-agreement.ts: "gittensory" mentions increased from 0 to 1 -- looks like new branding drift, not an intentional historical reference. If it genuinely belongs (e.g. a permanent Sentry ticket ID or a stable comment-marker already posted to live PRs), run `npm run branding-drift:update` and commit the regenerated baseline.\n' ], pid: 18048, stdout: '', stderr: 'Branding-drift check found 1 issue(s):\nsrc/review/predicted-gate-agreement.ts: "gittensory" mentions increased from 0 to 1 -- looks like new branding drift, not an intentional historical reference. If it genuinely belongs (e.g. a permanent Sentry ticket ID or a stable comment-marker already posted to live PRs), run `npm run branding-drift:update` and commit the regenerated baseline.\n' }

To view more test analytics, go to the Test Analytics Dashboard
📋 Got 3 mins? Take this short survey to help us improve Test Analytics.

JSONbored added a commit that referenced this pull request Jul 27, 2026
#9173 introduced a comment referencing the deprecated pre-rename product name,
which the branding-drift check on main was not catching -- breaking CI for
every PR based on current main, including this one.
JSONbored added a commit that referenced this pull request Jul 27, 2026
…9205)

#9173 introduced a comment referencing the deprecated pre-rename product
name, which the branding-drift baseline does not account for -- breaking
the branding-drift:check gate for every PR based on current main.
JSONbored added a commit that referenced this pull request Jul 27, 2026
…nd name every silent AI-review skip (#9180)

* fix(review): finalize CI-stuck silence, post a waiting placeholder, and name every silent AI-review skip

#9011: prReadyForReview used to defer a missing-required-context PR forever past
the 2-minute cap instead of falling through to the existing finalize-past-cap
block; a required context that never appears keeps ciState pending (never
"passed"), so finalizing here can never produce a would-merge disposition. A new
stuckReason ("missing_required_context" vs "ci_running") is threaded through the
audit/log calls so the two causes are distinguishable.

#9042: the silent wait before a review runs (measured live: 8.3min median, 21.9min
p90, up to 50.9min) now gets an immediate "waiting" panel comment
(renderWaitingForCiPlaceholder) upserted through the same PR_PANEL_COMMENT_MARKER
the final verdict replaces, instead of leaving the PR blank until CI settles.

#9000: root-causes the #8972 incident where a forced ("Re-run LoopOver review")
retrigger completed with none of the events a forced pass should emit anywhere in
the audit trail. shouldStartAiReviewForAdvisory folds in
shouldRequirePublicAiReviewForAdvisory's hard gate (aiReviewMode off, an
ineligible author, no head SHA, an AI kill-switch off, no AI binding) - a forced
retrigger does not bypass this gate, and every one of its branches previously
returned false with zero audit trail. resolvePublicAiReviewGateSkipReason names
the exact reason, and a new catch-all audit event fires whenever aiReviewWillRun
ends up false for a cause none of the other (already-audited) paths cover. The
two adjacent silent branches in the frozen/paused/one-shot reuse chain - a hold
condition true but nothing published to actually reuse - get their own named
"unavailable" events for the same reason.

This only closes the exhaustive-auditing and root-cause portion of #9000; the
receipt-feedback / lost-webhook-click recovery sweep it also asks for is not yet
implemented.

Closes #9011
Closes #9042

* fix(review): correct stale 'gittensory' brand reference in a comment

#9173 introduced a comment referencing the deprecated pre-rename product name,
which the branding-drift check on main was not catching -- breaking CI for
every PR based on current main, including this one.
JSONbored added a commit that referenced this pull request Jul 27, 2026
…s with the #9169 required-events expansion (#9203)

#9169 promoted pull_request_review/check_run/check_suite to REQUIRED_INSTALLATION_EVENTS and
switched the self-host setup wizard's manifest to derive default_events from that same canonical
list. Three test fixtures (one in api.test.ts, two in backfill.test.ts) still mocked GitHub's live
installation payload with the pre-#9169 event set, so their live-refresh assertions incorrectly
expected a healthy status with no missing events. The self-hosting-github-app.mdx docs page's
"Events: ..." sentence was similarly left describing the old 7-event list instead of the now-
canonical 12-event set, failing the docs/manifest parity test.

Also fixes an unrelated branding-drift regression surfaced by the same full test:ci run: a comment
in predicted-gate-agreement.ts (added in #9173) still said "gittensory" instead of "loopover".
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment