402 Unified Multi-Protocol Crawler

Reference implementation for discovering, probing, and classifying paid API endpoints across L402 (Lightning), x402 (on-chain USDC), and MPP (Stripe/Tempo) protocols.

Built for 402 Index.

Files

File	Description
crawler.py	Working multi-protocol crawler. Discovery via /.well-known/*, 402 header parsing, health classification.
backtest_health.py	Validates the consecutive-check health scoring algorithm against real probe history. 91% flap reduction, 7/7 invariants pass.
schemas/l402-discovery.schema.json	JSON Schema for L402 discovery documents
schemas/x402-discovery.schema.json	JSON Schema for x402 discovery documents (/.well-known/x402)
schemas/mpp-discovery.schema.json	JSON Schema for MPP discovery documents (/.well-known/mpp)

Quick Start

pip install requests

# Crawl a domain
python crawler.py governance.taskhawktech.com

# JSON output
python crawler.py governance.taskhawktech.com --pretty

# Probe a single endpoint
python crawler.py governance.taskhawktech.com --probe-only /governance/verify

# Run the health scoring backtest
python backtest_health.py

Crawler Pipeline

DISCOVER  →  COLLECT  →  PROBE  →  CLASSIFY

1. Fetch /.well-known/agent.json, /.well-known/mpp, /.well-known/x402
2. Merge all discovered endpoint URLs (deduplicated)
3. GET each endpoint, parse 402 response headers
4. Classify protocols detected and health status

Protocol Detection

Protocol	Detection Method	Header
L402	WWW-Authenticate: L402 macaroon="...", invoice="..."	WWW-Authenticate
x402	Base64-encoded JSON in PAYMENT-REQUIRED header	PAYMENT-REQUIRED
MPP	WWW-Authenticate: Payment id="...", method="..."	WWW-Authenticate

L402 Dual Format

L402 servers may send both macaroon= (binary, lnget-compatible) and token= (JSON, SDK-compatible) in the same header. The crawler detects and classifies the format:

• binary — standard macaroon (lnget works)
• json — JSON token in base64 (custom SDK required)
• dual — both present (maximum compatibility)

Price Extraction

• L402: From BOLT11 invoice amount (lnbc<amount><multiplier>) or from JSON token caveats.price_sats
• x402: From accepts[0].maxAmountRequired (USDC base units, 6 decimals)
• MPP: From /.well-known/mpp discovery doc resources[].price_cents

Health Scoring

The backtest validates the consecutive-check buffer algorithm:

degraded = responseTime > historicalP50 * 2 AND consecutiveSlowChecks >= bufferSize
down     = consecutiveFailures >= failureThreshold
recovery = consecutiveHealthy >= recoveryThreshold

Key properties validated:

• Single latency spikes (cold starts) do NOT trigger degradation
• Consecutive failures DO trigger DOWN
• Recovery requires consecutive healthy probes
• Flap reduction > 50% (actual: 91%)

Discovery Schemas

Each protocol has a JSON Schema defining its /.well-known/* discovery document:

• L402: protocol, version, provider, resources[] with price_sats, optional lightning_node info
• x402: protocol, version, provider, resources[] with price/currency/network/address, accepted_networks, facilitator
• MPP: protocol, version, provider, resources[] with price_cents/currency, payment_method, challenge_flow

All schemas include also_accepts (cross-protocol links) and discovery (links to sibling well-known docs).

Multi-Protocol Support

The resources field in discovery documents can be either format:

[{"url": "https://...", "price_sats": 10}]

or flat:

["https://...", "https://..."]

The crawler handles both.

The also_accepts field enables cross-protocol discovery:

{"also_accepts": ["L402", "x402", "mpp"]}

License

MIT