DOC-2262: Add OAuth authentication to the docs MCP server (Redpanda Cloud IdP)#181
DOC-2262: Add OAuth authentication to the docs MCP server (Redpanda Cloud IdP)#181JakeSCahill wants to merge 63 commits into
Conversation
✅ Deploy Preview for redpanda-documentation ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
Add a lightweight email->token authentication gate to the docs MCP server to capture users' work email addresses for lead capture and usage attribution. - New /mcp/register endpoint: users submit a work email and the bearer token is delivered ONLY by email (never in the HTTP response), so possession of a working token proves the address is real and owned. - Mandatory 4-layer validation: format, work-domain filter (reject free/ disposable providers), MX-record check, email delivery. - Tokens stored hashed in Netlify Blobs; auth middleware in mcp.mjs threads the authenticated email/domain to Kapa via _meta.user for attribution. - Bearer header and ?token= query fallback (for clients that can't set headers). - Gated behind REQUIRE_AUTH (grace period -> enforce); per-token rate limiting. - Captured emails -> Netlify Blobs + logs + optional CRM_WEBHOOK_URL forward. - Docs: registration + per-client setup + privacy/consent note; server-card and server.json advertise the token requirement. - 17 unit tests (tests/mcp-auth.test.ts). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
JakeSCahill
force-pushed
the
feature/mcp-email-auth
branch
from
9c6f267 to
3a89e42
Compare
JakeSCahill
changed the title
Netlify Functions don't reliably set NODE_ENV=production at runtime, so the previous NODE_ENV-based dev bypass could fire in deployed environments — silently logging tokens instead of emailing them and not failing when RESEND_API_KEY is missing. Gate the bypass on NETLIFY_DEV (set only by `netlify dev`/`functions:serve`) so any deployed env without a key errors loudly. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Replace the custom email->token gate with a standard MCP OAuth 2.1 resource server delegating to the Redpanda Cloud IdP (auth.prd.cloud.redpanda.com). This is required so ChatGPT can authenticate (ChatGPT only supports spec OAuth, not static tokens), while still capturing users' verified work emails. Verified the Cloud IdP supports everything needed (open Dynamic Client Registration, CIMD, PKCE S256, public clients, email scope, userinfo). - /.well-known/oauth-protected-resource (RFC 9728) edge function advertises the Cloud IdP as the authorization server; clients self-register via DCR/CIMD. - mcp.mjs auth middleware validates the bearer token against the IdP /userinfo endpoint, extracts the verified email/org, captures it (Blobs + log + optional CRM_WEBHOOK_URL), and threads it to Kapa via _meta.user. - Optional work-email enforcement (REQUIRE_WORK_EMAIL, default on) returns 403 for personal providers; REQUIRE_AUTH keeps the grace->enforce rollout. - Remove the email->token registration endpoint and email-sending module. - Docs updated: clients prompt for Redpanda Cloud sign-in (no token to paste). - Unit tests rewritten for the OAuth logic (16 tests). Production hardening (needs identity team): register an Auth0 API for the MCP resource so tokens are audience-bound JWTs, and add email as an access-token claim. Until then we validate via /userinfo (no audience binding). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
JakeSCahill
changed the title
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
⛔ Blocked: identity team — Dynamic Client Registration is disabled on the Cloud IdPThe OAuth resource server, discovery, and token validation are implemented and verified, but end-to-end auth cannot work yet because MCP clients can't register with the Redpanda Cloud IdP. What works (verified against prd
|
Drop the unused OAUTH_ISSUER export from idp.mjs and de-export the FREE_EMAIL_DOMAINS / DISPOSABLE_DOMAINS sets (used only internally). No behavior change. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The probe confirmed CIMD is not enabled on the Cloud IdP (a valid client metadata document used as client_id still returns 'Unknown client'). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Update — call with Cloud Identity (Santi): implementation direction decidedArchitecture: We unblock MCP auth by having the docs service run its own OAuth 2.1 Authorization Server (AS), with the Cloud IdP (Auth0) as the upstream identity provider. This sidesteps the earlier blocker (the Cloud Auth0 tenant has both DCR and CIMD disabled — see prior comments): the AI tools register and authenticate against our AS, not the Cloud IdP directly. The Cloud IdP only ever sees one client — ours. Division of responsibilities
Flow: AI client → our AS ( Open asks to Santi (in flight):
Future (phase 2): the same Auth0 federation core also powers human login to the docs site — it just sets a browser session instead of issuing tokens. One Auth0 app, two consumers; MCP ships first. Next steps:
|
… (M1) Replace the superseded resource-server-pointing-at-Cloud approach with the agreed broker architecture: our service is the OAuth 2.1 Authorization Server, federating the human login upstream to Auth0 and issuing/validating its own tokens. Ports the validated spike to production shape. Added (Milestone 1 — AS core): - lib/oauth/keys.mjs — jose RS256 sign/verify + JWKS; key from env (MCP_OAUTH_SIGNING_JWK) or dev-generated + persisted in Blobs (the spike proved an in-memory key breaks the flow) - lib/oauth/store.mjs — auth requests + auth codes on Netlify Blobs (interface is the seam for a Netlify DB/Neon backend when relational queries are needed) - lib/oauth/pkce.mjs, config.mjs, upstream.mjs (Auth0 + dev mock federation, id_token validated against Auth0 JWKS) - mcp-oauth.mjs — AS endpoints: discovery (RFC 8414), JWKS, /authorize, /mcp/callback, /token (authorization_code + PKCE) Changed: - mcp.mjs resource server now validates OUR OWN access tokens (jose) instead of calling the upstream /userinfo - protected-resource metadata + server card point authorization_servers at us - removed lib/idp.mjs (superseded /userinfo validation) Deferred (clearly marked): DCR/CIMD client registration (M2), refresh_token grant + rotation (M3), consent UI, revocation. Neon backend is a documented swap behind the store interface (needs Netlify DB provisioning). Auth0 mode needs Santi's client_id; defaults to a dev mock until then. Tests: 22 pass (PKCE incl. RFC 7636 vector; JWT issue/verify; JWKS leaks no private key; wrong-audience/tampered rejected). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Milestone 1 landed — production AS scaffold (jose + storage), federating to Auth0Pivoted the branch from the superseded resource-server-pointing-at-Cloud approach to the agreed broker architecture: our service is the OAuth 2.1 Authorization Server; it federates the human login upstream to Auth0 and issues/validates its own tokens. The validated spike is now ported to production shape. In this milestone
Tests: 22 pass (PKCE incl. the RFC 7636 vector; JWT issue/verify; JWKS leaks no private key; wrong-audience/tampered rejected). Deferred (clearly marked in code): DCR/CIMD client registration (M2), refresh-token grant + rotation (M3), consent UI, revocation. Still gated on:
The flow itself was already validated end-to-end on Netlify Functions in the spike branch ( |
The dev mock issues canned identities, so it must never be reachable by accident in a deployed environment. Resolve the upstream mode fail-closed: mock is only allowed under an explicit dev signal (NETLIFY_DEV or MCP_OAUTH_ALLOW_MOCK=true). Anything that would otherwise silently fall back to mock (e.g. a prod deploy missing REDPANDA_OAUTH_CLIENT_ID) resolves to null, and the AS returns 503 on the flow endpoints instead of handing out mock tokens. Discovery + JWKS stay up. - config.mjs: resolveUpstreamMode() (pure, tested) + UPSTREAM_MISCONFIGURED - upstream.mjs: throw if neither auth0 nor mock is active - mcp-oauth.mjs: 503 on /authorize, /callback, /token, mock-idp when misconfigured - tests: 6 cases covering the resolution matrix (28 total pass) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Netlify statically analyzes config.path at bundle time, so it can't be an array of imported constants (PATHS.*) — that failed bundling (and the PR preview build) with 'path: Must be a string or array of strings'. Use literal paths. Verified the full M1 flow live (functions:serve, mock upstream): authorize -> mock-idp -> /mcp/callback -> /token -> AS-issued JWT, then /mcp accepts that token (200) and rejects no-token / garbage (401). Confirms cross-function token validation via the Blobs-shared signing key. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Netlify Blobs defaults to eventual consistency (deletes/updates propagate up to
60s). For one-time-use auth codes and refresh-token rotation/reuse-detection
that window would let a consumed code/token be replayed, so the auth store now
uses { consistency: 'strong' }. The dev signing-key store does too, so the
resource server reads the key the AS just wrote rather than regenerating.
Verified live (functions:serve): full flow issues a token, /mcp accepts it
(200), and replaying a consumed auth code is rejected (400).
Note: Blobs still has no atomic CAS, so a sub-second concurrent replay remains
theoretically possible — negligible at our volume; a relational DB is the only
full fix (documented as the future swap behind the store interface).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add a 'Staying signed in' subsection explaining that the MCP client handles token renewal automatically: sign in once, the client refreshes in the background, active users stay signed in, and 30 days idle triggers a fresh (usually silent) sign-in. No manual token handling.
|
After thinking through the storage options for the OAuth layer, I’m leaning toward using Neon Postgres (via Netlify’s integration) as the system of record for auth state, rather than Netlify Blobs. Postgres gives us:
Netlify Blobs feels fine for simple metadata or low-risk storage, but it doesn’t provide strong enough guarantees for OAuth flows where race conditions or replay could become an issue. |
Add an MCP tool that lets AI clients forward user feedback (bugs, doc gaps, frustrations, feature requests) straight to the Redpanda team. The tool description tells agents to ask the user before submitting and to include the relevant page/context. Feedback goes to the existing api-feedback Netlify form (the same store our docs feedback uses); the hidden form is extended with category, source, and user identity fields. When the user is signed in we attach their email + domain so the team can follow up; anonymous otherwise. We log only category/domain/authed, never the raw email or feedback text. Also bumps the server version and documents the capability for users.
Mention the feedback tool in the MCP server-card and server.json descriptions, and bump the server-card version to 1.3.0 to match.
The feedback tool POSTed to the site root, which 301-redirects to /home/. fetch followed the redirect (POST -> GET, body dropped), so Netlify Forms never recorded the submission — but the final 200 made the tool report success. Verified: POST to / => 301; POST to /home/ => 200 and the submission lands. POST to /home/ (configurable via MCP_FEEDBACK_FORM_PATH) and set redirect: 'error' so a redirect surfaces as a failure instead of a false success.
Use the docs site's real brand assets (served same-origin, so they load on prod and previews): Inter font, the Redpanda logo, and the exact brand palette (brand-600 #e24328 / brand-700 hover, body/faint text, neutral borders). Cleaner card layout, favicon, and a divider before the privacy note. Also nudge prospects: the signup link now reads 'Create a free account' instead of 'Sign up at cloud.redpanda.com'.
…BACKEND flag (#184) * Refactor OAuth store into a backend selector (Blobs default) Split the OAuth state store into pluggable backends behind store.mjs: - db/blobs.mjs: the current Netlify Blobs implementation (extracted, no behavior change), default backend. - db/neon.mjs: a Neon Postgres backend whose one-time-use consumes are atomic single statements (UPDATE/DELETE ... RETURNING), closing the read-then-delete race Blobs can't (no compare-and-swap). - store.mjs: thin selector by STORE_BACKEND (default blobs); DCR clients stay on Blobs (plain persistence, no atomicity benefit). Replaces the non-atomic markRefreshUsed with consumeRefresh: on Neon only one of two concurrent refreshes wins the row; the loser is treated as reuse and the family is revoked, restoring theft detection under races. Neon driver is imported lazily so the default path needs no DB or dep. No caller behavior changes on the default backend; all 56 tests pass. * Add Neon schema, scheduled cleanup, and atomicity tests - Migration SQL (db/migrations/0001_oauth_state.sql) for the four one-time-use/transactional tables, with expires_at indexes. - cleanupExpired() + a daily scheduled function (oauth-cleanup.mjs) that deletes expired requests/codes and past-expiry refresh tokens, then sweeps empty families. No-ops unless STORE_BACKEND=neon. Bounds growth. - @neondatabase/serverless dependency (HTTP driver; no Drizzle — the atomic ops are single hand-written statements). - Real-Postgres concurrency tests (tests/mcp-oauth-neon.test.ts), skipped unless TEST_NEON_URL is set: prove two concurrent auth-code consumes / refresh rotations yield exactly one winner, and cleanup removes expired rows. A fake can't prove atomicity, so these require a real DB. 56 tests pass; 3 Neon tests skip without a DB URL. * Align Neon store with Netlify DB (db init) conventions The database is provisioned and attached to the redpanda-documentation site. Wire the code to Netlify's managed flow: - Use @netlify/database (the package db init installed) instead of the raw @neondatabase/serverless driver: neon.mjs now connects via getDatabase().httpClient (zero-config, reads NETLIFY_DATABASE_URL, fail-closed if absent). - Move the schema into Netlify's auto-applied migrations directory (netlify/database/migrations/), so it's applied on deploy — including to per-preview DB branches. Removes the hand-rolled migrations path. - Update the atomicity test to the new path + @netlify/database client. 56 tests pass; 3 Neon tests skip without TEST_NEON_URL. * chore: trigger deploy-preview build (pick up STORE_BACKEND=neon)
Co-authored-by: Jake Cahill <45230295+JakeSCahill@users.noreply.github.com>
The api-feedback form is MCP-only, so several fields were dead weight: drop referer (duplicated page-path), source (constant 'mcp'), user-agent (constant; Netlify auto-captures the real UA), timestamp (Netlify records created_at), and user-sub (opaque id; email is the actionable identity). Kept: feedback, category, page-path, user-email, user-domain, plus the bot-field honeypot.
Add an identify hook to the MCPcat track() call so usage shows per-user/ per-org instead of anonymous sessions. Identity comes from our verified OAuth context (extra.authInfo), not a tool argument; returns null when unauthenticated so grace-period sessions stay anonymous. Forwards sub + email + domain (email included for now, pending legal sign-off before launch). Update the login interstitial and docs notice to disclose sharing with service providers / analytics systems.
MCPcat wraps the SDK's tools/list and tools/call handlers, which the MCP SDK only creates lazily on the first registerTool(). track() was being called before any tool was registered, so it found no handlers to wrap and silently no-op'd the tool-call-context injection — every event showed 'no user intent provided'. Move the track() call to after all tool registrations so the context parameter is injected and agent intent is captured. (identify hook unchanged.)
Steer the agent-intent prompt: ask for a concise third-person summary of what the user is trying to accomplish, and explicitly exclude credentials, tokens, personal data, and verbatim secrets from that free-text field.
# Conflicts: # netlify/functions/mcp.mjs
micheleRP
left a comment
micheleRP
left a comment
There was a problem hiding this comment.
Review — security + functional, with live preview testing
Reviewed the full OAuth core, storage layer, resource-server integration, and the feedback tool (skipped the lockfile), and tested the deploy preview end-to-end. This is a high-quality, security-conscious implementation — I found no auth-bypass, token-forgery, or SQL-injection issues. Testing confirmed the auth gate, the feedback tool, and Kapa attribution all work, and surfaced one gap to resolve before relying on form-based attribution (finding #1). Everything below is non-blocking; not an approval — just findings + open questions.
Tested on the deploy preview
- ✅ Auth enforced (
REQUIRE_AUTH=true): unauthenticatedPOST /mcp(initialize,tools/list) → spec-correct401withWWW-Authenticate: Bearer … resource_metadata=…,no-store/noindex. - ✅ Feedback routing fix:
GET /→301 → /home/,GET /home/→200— confirms the body-dropping bug and the non-redirecting target. - ✅ Token endpoint rejects a bogus authorization code →
invalid_grant. - ✅ Feedback tool end-to-end: signed in via the Cloud integration tenant, called
submit_documentation_feedback, and the submission landed inapi-feedbackwith the correctfeedbacktext +page-path. - ✅ Kapa attribution confirmed: the conversation surfaced in Kapa's Users view tied to a real email + company (vs anonymous UUIDs for pre-auth users) — email + domain attach correctly at runtime.
⚠️ But the attribution fields didn't land in the Netlify form: the recorded submission shows onlypage-path+feedback—user-email,user-domain, andcategoryare missing, despite being signed in. (See finding #1.)
⚠️ Finding #1 — attribution fields aren't captured by the api-feedback form yet (caught in testing)
The branch declares user-email/user-domain/category in api-feedback-registration.html:15–20 and the code POSTs them, so this isn't a code defect — the existing prod api-feedback form's field schema predates these fields, and Netlify drops unrecognized fields on deploy-preview submissions. Since lead attribution is the core goal of this PR, please confirm after merge that the new fields are captured once the updated registration deploys to production (Netlify may need the form re-detected; worst case, recreated).
Note: the same authenticated identity did reach Kapa (email + company both surfaced there), so the tool is attaching identity correctly — the missing fields on the Netlify form are confirmed to be the form-schema/detection issue alone, not a runtime/auth problem.
Other findings (non-blocking)
2. Consent screen deferred → authorization-code phishing can disclose a victim's verified email/org (medium). Open DCR + no consent screen (the interstitial doesn't name the requesting client/scopes) lets an attacker register a client with their own redirect_uri, phish a victim into /authorize, and exchange the resulting code for a token carrying the victim's email + org. Docs-scoped and needs social engineering, but it's the exact PII this system protects. Recommend landing consent before REQUIRE_AUTH=true/announcement, or interim-restricting DCR to an allow-list.
3. Reuse-detection only holds on Neon. Default STORE_BACKEND=blobs is non-atomic (documented). Ensure prod runs neon before relying on theft detection — worth a rollout-checklist line. (Preview already runs neon.)
4. PII at rest (for the pending legal/privacy review). Verified emails stored plaintext in the mcp-users Blobs store + Neon user_data, and forwarded to MCPcat, Kapa, and the CRM webhook. Kapa additionally retains email + domain + full conversation history tied to identity (visible in its Users view). No code defect — confirming the data footprint.
Minor: submitFeedback returns internal error text to the agent (detail: msg); DCR redirect_uris aren't scheme-restricted; stale auth.mjs comment references a non-existent idp.mjs; feedback page_url isn't validated (informational only).
Verified in code
RS256 tokens with iss/aud/alg pinned on verify (validates our token, not the upstream); Auth0 ID token verified against JWKS with iss/aud/RS256 pinned; PKCE S256 on both legs; redirect_uri validated before any redirect; thorough CIMD SSRF guard (https-only, private-IP blocklist v4+v6, no redirect-following, byte cap, timeout); refresh rotation + reuse detection + hashed-at-rest tokens + atomic Neon consume; fully parameterized SQL; c.set('auth') → extra.authInfo wiring matches @hono/mcp@0.1.5.
Observability / attribution — where identity flows
Five sinks: mcp-users Blobs (email/domain/requestCount), MCPcat (userId=sub, userName=email, domain), Kapa (_meta.user={email, company_name: domain}), optional CRM webhook (email/domain/sub), and function logs (domain/sub only, no raw email). A short "what we collect and where it goes" summary in the description would help the privacy review. Two notes vs the "capture … org" framing: names are never captured (despite profile scope, exchangeCode never reads name), and org_id/org_name are dropped before every sink — everything attributes on the email domain, not the Auth0 org (confirmed in Kapa: company shows redpanda.com).
Open questions
- Attribution fields: can you confirm
user-email/user-domainget captured once the form schema updates on production deploy? (Finding #1 — currently dropped on the preview.) - Consent before enforcement: will the consent screen land before
REQUIRE_AUTH=true/ public launch? - Org vs domain (confirmed, flagging only): attribution is domain-based — Kapa shows company
redpanda.com, and the stores/CRM/form use domain, never the Auth0org_name/org_id(captured into the token but dropped before every sink). If domain is the intended key, no action; flagging in case org-level was expected. - Prod backend: will prod ship with
STORE_BACKEND=neonfrom cutover (reuse detection depends on it)? - Account-requirement scope (product/rollout): once enforced, the MCP gates all docs — Self-Managed, Connect, ADP — behind a Redpanda Cloud account, no per-product exemption. Intended end state (especially for the open-source Connect community), or keep the grace period open for anonymous reads with sign-in required only for higher limits / the feedback tool?
Test evidence (screenshots below): the api-feedback Netlify submission (only page-path + feedback captured), and the Kapa Users view showing the authenticated conversation attributed to email + company.
|
micheleRP
left a comment
micheleRP
left a comment
There was a problem hiding this comment.
very nice! Please see Claude's comments, but looks great!
2 participants
Jira: DOC-2262
Goal
Add authentication to the docs MCP server (
docs.redpanda.com/mcp) so AI tools (ChatGPT, Claude, Cursor, VS Code) have users sign in with their Redpanda Cloud account, letting us capture verified work emails and attribute docs usage to organizations.Architecture (decided with Cloud Identity)
The docs service runs its own OAuth 2.1 Authorization Server (AS), with the Cloud IdP (Auth0) as the upstream identity provider. AI tools register and authenticate against our AS; we federate the human login to Auth0 and issue our own tokens.
Why this shape:
email/org from Auth0 to capture + attribute.Division of responsibilities
client_id, Authorization Code + PKCE, no secret), our/callbackredirect URIs allow-listed, ID token returnsemail/email_verified+ org. One app covers MCP now and docs-site login later./authorize,/callback,/token(+ refresh w/ rotation), client registration (DCR + CIMD), JWKS, consent/login UI; federate login to Auth0; issue/validate our own tokens. One-time-use state lives in Neon Postgres; clients/rate-limit counters stay on Netlify Blobs (see below).Also in this PR: OAuth state on Neon Postgres
The AS's one-time-use / transactional state (auth requests, auth codes, refresh tokens + families) is backed by Neon Postgres (Netlify DB), behind a
STORE_BACKENDflag (blobsdefault,neonto switch).UPDATE … WHERE used=false RETURNING *— exactly one wins, the loser is treated as reuse.netlify/database/migrations/), including to per-preview DB branches. A daily scheduled function GCs expired rows.STORE_BACKEND=neonon a context to switch; roll back by resetting the var. Cutover note: flipping blobs→neon doesn't migrate rows, so live refresh tokens won't carry over — users re-authenticate once.Also in this PR: documentation feedback tool
An MCP tool,
submit_documentation_feedback, lets AI clients forward user feedback (bugs, doc gaps, incorrect/missing info, feature requests) straight to the docs/DX team. The tool description tells the agent to ask the user before submitting and include the relevant page/context.api-feedbackNetlify form (MCP-only); fields trimmed to what matters:feedback,category,page-path,user-email,user-domain(+ honeypot).category/domain/authed— never the raw email or feedback text.redirect: 'error'so a redirect can't masquerade as success.server.jsondescriptions.Login page
The interstitial shown before the Cloud redirect is styled to match docs.redpanda.com (Inter, the Redpanda logo, brand palette — all same-origin so it renders on prod and previews), discloses what we collect with a Privacy Policy link, and links "Create a free account" (new tab) for prospects without a Cloud account.
Future (phase 2)
The same Auth0 federation core also powers human login to the docs site — it just sets a browser session instead of issuing tokens. One Auth0 app, two consumers; MCP ships first. The Neon schema is intentionally compatible with future identity/saved-conversation tables.
Testing
2026-06-18_19-50-10.mp4
The deploy preview is wired to the integration Auth0 tenant and runs the Neon backend (
STORE_BACKEND=neon), so you can exercise the full flow there.1. Add the preview server to Claude Code:
2. Authenticate and use it:
/mcp, selectredpanda-preview, choose Authenticateclaude mcp remove redpanda-previewNotes:
integration-cloudv2.us.auth0.com), not prod.npm run test:mcpplus thetests/mcp-oauth-*.test.tssuites. The Neon atomicity tests (tests/mcp-oauth-neon.test.ts) run against a real Postgres whenTEST_NEON_URLis set (skipped otherwise).Kapa now has your conversation along with your email and org
History
Explored a pure OAuth resource server pointing at the Cloud IdP (blocked: DCR/CIMD disabled on that tenant), landing on the AS-broker design above after the call with Cloud. The Neon state backend was developed as PR #184 and merged into this branch.