feat: fuzz testing for the engine and all adapters - #62
Merged
Conversation
Go-native fuzz targets plus a deterministic all-adapter safety sweep
guarding the one invariant a mock must uphold: client input never
produces a 5xx. Starlark has no try/except, so any builtin raise on
attacker-shaped input is an unhandled 500; real APIs answer bad input
with 4xx.
- TestAdapterInputSafety: every reference adapter's routes driven with
garbage params (negative/huge/unicode), JSON-null and malformed
bodies, batch arrays, bracket-form bodies, garbage auth, and every
plausible cursor/limit param poisoned at once.
- Fuzz targets: matchRoute (router), parseFormBody (bracket parser),
parse_multipart (total contract), webhook header validation, and
FuzzAdapterRequests — coverage-guided full-dispatch fuzzing over a
curated adapter set (stripe, cloudflare D1, salesforce SOQL,
powerplatform OData, emailoctopus, eth-jsonrpc, shopify). 5+ minutes
clean on the adapter target after fixes.
- Seed corpora run in plain go test forever; just fuzz runs guided
rounds; found inputs are committed under testdata/fuzz/.
First-run findings, all fixed:
- paginate raised on invalid cursors — 10 adapters 500'd on a tampered
token. Now returns (None, None); each adapter answers its provider's
real 400 (Invalid pageToken, InvalidQueryParameterValue, ...).
- crypto.base64_decode/base64url_decode raised on malformed input (auth
material, cursors). Now total — None, never a raise.
- query_select/paginate raised on out-of-int64 limits. Now clamped: a
huge limit means no limit.
- Bracket parser: a[0]=v (terminal numeric index, the Rails array
literal) PANICKED on an empty recursion tail; a[222222220][b]=v
materialized a 222M-element slice (57s). Fixed + matrix cases added.
- Router: a {} manifest typo captured an empty param name; never
matches now.
- eth-jsonrpc/erc4337: non-object batch elements and a non-string
method crashed dispatch; now per-element -32600 per the JSON-RPC spec.
- anaplan built blob names from raw path params; identifiers now
validated (400).
- printify/jira parsed unbounded ints from client ids/params (int64
overflow in the response converter); now bounded.
Review findings on PR #62, all fixed: - CRITICAL: start+limit overflow — a huge limit (clamped to MaxInt64) plus a VALID cursor wrapped end negative and panicked the slice, in both paginate and query_select. Both now clamp against the remaining items; unit test pins cursor=1 + MaxInt64 limit. The paginate cursor is also digit-strict now (ParseInt accepted '+5'). - The (None, None) cursor guards covered only ~10 of the cursor-exposing adapters; the rest returned wrong 200s with null data or still 500'd (len(None)). 98 guards inserted across 40 more adapters, each using its provider's 400 envelope (Slack invalid_cursor, Shopify page_info, Dropbox invalid_cursor, Square INVALID_CURSOR, Google Invalid pageToken, Azure InvalidQueryParameterValue, ...). - garbageQuery now poisons ~30 param names incl. case-sensitive and provider-specific ones (pagination_token, PageToken, nextPageToken, page_info, sysparm_offset, $skip/$skipToken, continuation-token, Marker, start, max_results, top, max-keys) — the sweep previously could not see most cursor paths. - FuzzAdapterRequests now fuzzes the QUERY STRING too (path encoding previously swallowed '?', making every cursor/limit path unreachable by the fuzzer); seeds include tampered cursors and huge limits. - fuzzServerFor closes engines + removes state dirs in plain go test seed runs (only -fuzz workers keep them for the process lifetime). - smartbill comment updated to the total base64_decode contract. Gates: full suite, parse guard, QC boot, adapter lint, gofmt, vet green; FuzzAdapterRequests 4min with queries clean; all-adapter sweep green on the expanded poison set.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Fuzz testing across the app and the adapters, guarding one invariant: client input never produces a 5xx. Starlark has no try/except, so any builtin raise on attacker-shaped input is an unhandled 500 — real APIs answer bad input with 4xx.
The harnesses
TestAdapterInputSafety— every reference adapter's routes, driven with deterministic adversarial requests: garbage path params (negative / huge / unicode), JSON-null and malformed bodies, batch arrays, bracket-form bodies, garbage auth, and every plausible cursor/limit query param poisoned simultaneously.FuzzMatchRoute(router),FuzzParseFormBody(bracket-form parser),FuzzParseMultipart(multipart decoder totality),FuzzValidateHeader(webhook header injection), andFuzzAdapterRequests: coverage-guided fuzzing through the full dispatch path over a curated adapter set (stripe, cloudflare-D1, salesforce-SOQL, powerplatform-OData, emailoctopus, eth-jsonrpc, shopify).go testforever;just fuzzruns guided rounds locally; discovered inputs are committed undertestdata/fuzz/.First-run findings — all fixed
paginateraised on invalid cursors — 10 adapters 500'd on a tampered token(None, None); each adapter answers its provider's real 400 (Invalid pageToken,InvalidQueryParameterValue, …)crypto.base64_decode/base64url_decoderaised on malformed input (auth material, cursors)None, never a raise (json_safe_decode's contract)query_select/paginateraised on out-of-int64 limitsa[0]=v(terminal numeric index — the Rails array-literal form)a[222222220][b]=v(222M-element sparse slice){}manifest typomethod-32600Invalid Request per the JSON-RPC 2.0 specThe bracket-parser panic and the 57s input are the standouts: both reachable by any client POSTing a urlencoded body to any form-accepting endpoint, both shipped in v0.41.0, and neither was caught by the existing suites — exactly what this harness exists for.
Test
FuzzAdapterRequests5min,FuzzParseFormBody90s,FuzzParseMultipart90s,FuzzMatchRoute60s,FuzzValidateHeader45s (768k execs) — all clean.