afdocs-audit: detect Vercel Firewall challenge and mark audit invalid

.agents/skills/afdocs-audit/SKILL.md

@@ -53,6 +53,33 @@ Before reporting, cross-reference every issue against the known exceptions in `r
 
 Only include a section if its count is > 0. Never list allowlisted issues under "Remaining."
 
+## Invalid audits (Vercel Firewall challenge)
+
+Before trusting any failures, confirm the audit was actually able to reach the
+site. If `docs.warp.dev` is behind a **Vercel Firewall bot challenge** (Attack
+Challenge Mode or the Bot Protection ruleset in challenge mode), every request
+returns HTTP 429 with an `x-vercel-mitigated: challenge` header, the `afdocs`
+crawler can't solve the JavaScript challenge, and **every check becomes a false
+positive**. The score is meaningless and must not be reported as a regression.
+
+The wrapper script (`scripts/afdocs_audit.mjs`) now runs a preflight check and
+exits early with `status: "invalid"` (exit code `2`) when it detects this. If
+you see that, do **not** post a score — report that the audit was blocked and
+link the fix.
+
+Confirm manually with:
+
+```bash
+curl -sS -D - -o /dev/null https://docs.warp.dev/llms.txt
+```
+
+A `429` plus `x-vercel-mitigated: challenge` (or an `x-vercel-challenge-token`
+header) means the firewall is blocking the crawler. The fix is on the Vercel
+side (disable Attack Mode, switch Bot Protection to log mode, or add a WAF
+bypass rule for the runner) — the `afdocs` CLI can't send a bypass header. See
+`references/vercel-firewall-challenge.md` for the full diagnosis and
+remediation steps.
+
 ## Reporting results
 
 After running the audit, ALWAYS report the results to the user before taking any action. Include:

.agents/skills/afdocs-audit/references/vercel-firewall-challenge.md

@@ -0,0 +1,102 @@
+# Vercel Firewall challenge blocks the AFDocs audit
+
+When every AFDocs check fails or the audit reports a "SPA shell" with content
+past 50%, the most likely cause is **not** a docs regression — it's that
+`docs.warp.dev` is sitting behind a **Vercel Firewall bot challenge**. The
+`afdocs` crawler is a non-browser HTTP client, so it can't solve the
+JavaScript challenge and never reaches real content. Every check becomes a
+false positive.
+
+The audit script (`scripts/afdocs_audit.mjs`) now detects this up front and
+aborts with an `invalid` status (exit code `2`) instead of publishing a
+misleading score.
+
+## How to confirm
+
+```bash
+curl -sS -D - -o /dev/null https://docs.warp.dev/llms.txt
+```
+
+You are blocked by a challenge if the response shows **any** of:
+
+- `HTTP/2 429`
+- `x-vercel-mitigated: challenge`
+- an `x-vercel-challenge-token:` header
+- `server: Vercel` serving an HTML "Vercel Security Checkpoint" body
+
+A spoofed browser `User-Agent` does **not** help — Vercel's Bot Protection
+specifically flags non-browser clients that claim to be a browser (e.g. a
+`curl`/`fetch` request identifying as Chrome).
+
+## Root cause
+
+The 429 + `x-vercel-mitigated: challenge` signal comes from one of two
+Vercel Firewall features on the project that serves `docs.warp.dev`:
+
+- **Attack Challenge Mode** (Firewall → Bot Management → Attack Mode). Usually
+  toggled on temporarily during a DDoS attack. If left on, it challenges all
+  traffic that isn't a Vercel-*verified* bot.
+- **Bot Protection managed ruleset** in **challenge** mode, which serves a
+  JavaScript challenge to all non-browser traffic.
+
+In both modes Vercel auto-allows its directory of *verified* bots (Googlebot,
+verified webhook providers, etc.), but the `afdocs` audit runner is not a
+verified bot, so it is challenged and blocked.
+
+> This is a platform/firewall configuration, not a docs-repo bug. The repo
+> already implements every AFDocs remediation (llms.txt, `.md` variants,
+> `src/middleware.ts` content negotiation, in-page llms directives, and
+> `.well-known/mcp.json`).
+
+## Remediation
+
+The `afdocs` CLI cannot send a custom header or User-Agent
+(`npx afdocs check --help` exposes no such flag), so a header-based WAF bypass
+is not possible through the audit tool. Use one of the following Vercel-side
+fixes (requires Firewall permissions on the `docs.warp.dev` project).
+
+### Option A — Disable Attack Mode (if it was left on)
+
+Vercel's automatic DDoS mitigation stays active without Attack Mode, so this
+is safe once the targeted attack is over.
+
+- Dashboard: project → **Firewall** → **Bot Management** → **Attack Mode** →
+  **Disable**.
+- CLI: `vercel firewall attack-mode` (applies immediately).
+
+Verify with the `curl` command above — a healthy response is `HTTP 200` with
+no `x-vercel-mitigated` header.
+
+### Option B — Switch Bot Protection to Log mode
+
+If the Bot Protection managed ruleset is in **challenge** mode, switch it to
+**log** mode (project → **Firewall** → **Bot Management**) so non-browser
+agents — including AI doc consumers — aren't challenged.
+
+### Option C — Allow the audit runner through (keep protection on)
+
+Add a **System Bypass rule** or a **WAF custom rule with a `bypass` action**
+that matches the audit runner's egress IP, so the runner gets a valid score
+while the rest of the site stays protected.
+
+```bash
+# Example: bypass the firewall for a known runner IP
+vercel firewall rules add "AFDocs audit bypass" \
+  --condition '{"type":"ip_address","op":"eq","value":"RUNNER_IP"}' \
+  --action bypass --yes
+vercel firewall publish
+```
+
+Because the audit tool can't send a secret header, prefer an IP match. If the
+runner has no stable egress IP, run the audit from an allowlisted network, or
+use Option A/B around the audit window.
+
+## Why this also matters beyond the audit
+
+Challenge mode blocks **all** non-browser clients that aren't Vercel-verified
+bots, not just the audit runner. That can defeat the purpose of the
+agent-friendly docs work: AI agents and crawlers that aren't in Vercel's
+verified directory may also be unable to fetch `llms.txt` or `.md` content.
+Keeping Attack/Challenge mode off (relying on Vercel's always-on DDoS
+mitigation) is the agent-friendly default; reserve challenge mode for active
+attacks.

.agents/skills/afdocs-audit/scripts/afdocs_audit.mjs

@@ -51,6 +51,57 @@ function parseArgs(argv) {
 	return args;
 }
 
+/**
+ * Detect whether the target site is behind a Vercel Firewall bot challenge
+ * (Attack Challenge Mode or the Bot Protection managed ruleset in challenge
+ * mode). When active, Vercel returns HTTP 429 with an `x-vercel-mitigated:
+ * challenge` header and a `x-vercel-challenge-token` to every non-browser
+ * client — including the `afdocs` crawler. The crawler cannot solve the
+ * JavaScript challenge, so every page "fails" to fetch and the resulting
+ * scorecard is meaningless (all checks become false positives).
+ *
+ * Detecting this up front lets us abort with a clear "audit invalid" status
+ * instead of publishing a misleading score. The remediation is on the Vercel
+ * side (see references/vercel-firewall-challenge.md), since the `afdocs` CLI
+ * cannot send a bypass header.
+ *
+ * Returns `null` when no challenge is detected, or an object describing the
+ * block when one is found. Network errors are treated as "not a challenge"
+ * so an unrelated transient failure doesn't mask a real audit.
+ */
+async function detectVercelChallenge(url) {
+	const probeUrl = new URL('/llms.txt', url).toString();
+	let res;
+	try {
+		res = await fetch(probeUrl, {
+			redirect: 'manual',
+			headers: {
+				// Mimic a browser UA; the Vercel challenge still fires for
+				// non-browser clients even with a spoofed UA, so this only
+				// avoids unrelated UA-based blocks.
+				'user-agent':
+					'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0 Safari/537.36',
+			},
+		});
+	} catch {
+		return null; // Network error — let the real audit surface the problem.
+	}
+
+	const mitigated = res.headers.get('x-vercel-mitigated');
+	const challengeToken = res.headers.get('x-vercel-challenge-token');
+	const isChallenge =
+		mitigated === 'challenge' || challengeToken != null || res.status === 429;
+
+	if (!isChallenge) return null;
+
+	return {
+		probeUrl,
+		status: res.status,
+		mitigated: mitigated || null,
+		server: res.headers.get('server') || null,
+	};
+}
+
 function runAfdocsCheck(url) {
 	// Validate URL to prevent shell injection
 	try {
@@ -179,6 +230,45 @@ function printSummary(report) {
 
 // Main
 const args = parseArgs(process.argv.slice(2));
+
+// Preflight: if the site is behind a Vercel Firewall bot challenge, the
+// afdocs crawler can't reach any real content and every check becomes a false
+// positive. Bail out with a clear "invalid" status so we never publish a
+// misleading score.
+const challenge = await detectVercelChallenge(args.url);
+if (challenge) {
+	const invalidReport = {
+		url: args.url,
+		timestamp: new Date().toISOString(),
+		status: 'invalid',
+		reason: 'vercel-firewall-challenge',
+		detail: challenge,
+	};
+
+	console.error(
+		`\n⛔ AFDocs audit INVALID — ${args.url} is behind a Vercel Firewall bot challenge.`
+	);
+	console.error(
+		`   Probe ${challenge.probeUrl} returned HTTP ${challenge.status}` +
+			(challenge.mitigated ? ` (x-vercel-mitigated: ${challenge.mitigated})` : '') +
+			'.'
+	);
+	console.error(
+		'   The afdocs crawler cannot solve the JavaScript challenge, so a score cannot be computed.'
+	);
+	console.error(
+		'   Fix: see .agents/skills/afdocs-audit/references/vercel-firewall-challenge.md'
+	);
+
+	if (args.output) {
+		const outputPath = resolve(args.output);
+		writeFileSync(outputPath, JSON.stringify(invalidReport, null, 2));
+		console.error(`\nInvalid-audit report written to ${outputPath}`);
+	}
+
+	process.exit(2);
+}
+
 console.log(`Running AFDocs check on ${args.url}...`);
 
 const raw = runAfdocsCheck(args.url);