diff --git a/docs/patterns/index.md b/docs/patterns/index.md index f9b3fd033..752a5a138 100644 --- a/docs/patterns/index.md +++ b/docs/patterns/index.md @@ -4,7 +4,7 @@ A pattern is a reusable solution to a recurring problem when building API simula Most projects start with [Explore a New API](./explore-new-api.md) or [Executable Spec](./executable-spec.md) to get a running server from an OpenAPI spec with no code. From there, [Mock APIs with Dummy Data](./mock-with-dummy-data.md) and [AI-Assisted Implementation](./ai-assisted-implementation.md) are the natural next steps for adding realistic responses — the former by hand, the latter with an AI agent doing the heavy lifting. -As the mock grows, [Scenario Scripts](./scenario-scripts.md) let you automate repetitive REPL interactions — seeding data on startup, building reusable request sequences — while [Federated Context Files](./federated-context.md) and [Test the Context, Not the Handlers](./test-context-not-handlers.md) keep the stateful logic organized and reliable. [Live Server Inspection with the REPL](./repl-inspection.md) is Counterfact's most distinctive feature, letting you seed data, send requests, and toggle behavior in real time without restarting, and [Simulate Failures and Edge Cases](./simulate-failures.md) and [Simulate Realistic Latency](./simulate-latency.md) extend any mock to cover the error paths and performance characteristics that real services exhibit. +As the mock grows, [Scenario Scripts](./scenario-scripts.md) let you automate repetitive REPL interactions — seeding data on startup, building reusable request sequences — while [Federated Context Files](./federated-context.md) and [Test the Context, Not the Handlers](./test-context-not-handlers.md) keep the stateful logic organized and reliable. [Live Server Inspection with the REPL](./repl-inspection.md) is Counterfact's most distinctive feature, letting you seed data, send requests, and toggle behavior in real time without restarting, and [Simulate Failures and Edge Cases](./simulate-failures.md), [Test Fault Scenarios with Chaos Rules](./test-fault-scenarios-with-chaos.md), and [Simulate Realistic Latency](./simulate-latency.md) extend any mock to cover the error paths and performance characteristics that real services exhibit. When your project involves multiple versions or multiple specs, [Multiple API Versions](./multiple-versions.md) shows how to serve them from a shared set of route files using `$.minVersion()` to branch on version without duplicating handlers. For teams that want the mock to remain a reliable, long-lived artifact, [Reference Implementation](./reference-implementation.md) and [Automated Integration Tests](./automated-integration-tests.md) make it a first-class part of the codebase that can run in CI. Finally, [Agentic Sandbox](./agentic-sandbox.md) and [Hybrid Proxy](./hybrid-proxy.md) address the two common integration strategies — isolating an AI agent from the real service, or blending mock and live traffic — and [Custom Middleware](./custom-middleware.md) covers cross-cutting concerns like authentication and logging without touching individual handlers. @@ -21,6 +21,7 @@ When your project involves multiple versions or multiple specs, [Multiple API Ve | [Test the Context, Not the Handlers](./test-context-not-handlers.md) | You want to keep shared stateful logic reliable as the mock grows | | [Live Server Inspection with the REPL](./repl-inspection.md) | You want to seed data, send requests, and toggle behavior without restarting the server | | [Simulate Failures and Edge Cases](./simulate-failures.md) | You need reproducible, on-demand error conditions for development or testing | +| [Test Fault Scenarios with Chaos Rules](./test-fault-scenarios-with-chaos.md) | You want to inject HTTP-layer failures on demand using `chaos()` without editing handlers | | [Simulate Realistic Latency](./simulate-latency.md) | You want to test how clients and UIs behave under realistic response times | | [Reference Implementation](./reference-implementation.md) | You want a working, executable implementation that expresses intended API behavior in code | | [Multiple API Versions](./multiple-versions.md) | You maintain multiple versions of an API and want shared handlers that adapt by version | diff --git a/docs/patterns/test-fault-scenarios-with-chaos.md b/docs/patterns/test-fault-scenarios-with-chaos.md new file mode 100644 index 000000000..30eb60b8e --- /dev/null +++ b/docs/patterns/test-fault-scenarios-with-chaos.md @@ -0,0 +1,58 @@ +# Test Fault Scenarios with Chaos Rules + +You want to test how a client, UI, or integration behaves under upstream failures without editing route handlers or restarting the mock server. + +## Problem + +Failure behavior is often tested too late because reproducing 5xxs, flaky responses, or temporary outages usually means changing handler code, wiring custom flags, or waiting for real backend incidents. + +## Solution + +Use Counterfact's `chaos()` API from the Live REPL to inject HTTP-layer faults on demand. Keep the baseline handlers unchanged, then apply temporary rules for the exact paths and failure profiles you want to exercise. + +## Example + +Start with a healthy service: + +```text +⬣> client.get("/payments/42") +{ status: 200, body: { ... } } +``` + +Inject an intermittent upstream failure pattern: + +```ts +// Match /payments* requests indefinitely, +// but fail only about 20% with a retry hint. +chaos("/payments") + .probability(0.2) + .status(503) + .header("Retry-After", "1"); +``` + +You can also target a bounded outage: + +```ts +// Fail the next 3 matching requests, then stop automatically. +chaos("/payments").next(3).status(503); +``` + +And remove the rule when your test scenario is complete: + +```ts +const fault = chaos("/payments").status(500); +fault.stop(); +``` + +## Consequences + +- Faults are injected at the HTTP response layer, so you can test resilience behavior without changing route files. +- Rules are fast to toggle from the REPL, which is useful for exploratory testing and manual acceptance checks. +- `probability(...)` enables controlled flakiness; `next(count)` enables deterministic burst failures. +- This pattern does not simulate low-level network disconnects; it focuses on HTTP response behavior. + +## Related Patterns + +- [Simulate Failures and Edge Cases](./simulate-failures.md) — implement failure behavior directly in handlers/context +- [Simulate Realistic Latency](./simulate-latency.md) — add delayed responses to complement fault injection +- [Live Server Inspection with the REPL](./repl-inspection.md) — drive chaos rules and requests interactively diff --git a/docs/reference.md b/docs/reference.md index 8a0877ba0..fe2a4e6a0 100644 --- a/docs/reference.md +++ b/docs/reference.md @@ -17,6 +17,7 @@ Complete reference for Counterfact's architecture, route handlers, and CLI. - [Hybrid proxy](#hybrid-proxy) - [Middleware](#middleware) - [Type safety](#type-safety) +- [Chaos API (HTTP-layer fault injection)](#chaos-api-http-layer-fault-injection) - [Programmatic API](#programmatic-api) - [Multiple API versions](#multiple-api-versions) - [CLI reference](#cli-reference) @@ -350,6 +351,128 @@ See the [Multiple versions feature page](./features/multiple-versions.md) for a --- +## Chaos API (HTTP-layer fault injection) + +The `chaos()` function lets you inject HTTP-layer faults into simulated responses without modifying your route handlers. It is available as a global in the [Live REPL](#live-repl) and can also be used programmatically. + +### Quick start + +```ts +// Fail the next 3 /orders requests with a 50% probability +const fault = chaos("/orders") + .next(3) + .probability(0.5) + .status(500) + .delay(1_000) + .transformBody((body) => ({ ...body, error: true })) + .header("Retry-After", "60"); + +// Pause / resume the rule at runtime +fault.stop(); +fault.start(); +``` + +### Creating a rule + +```ts +chaos() // matches all paths (global rule) +chaos(pathPrefix) // matches paths that start with pathPrefix +``` + +Then set the scope: + +| Method | Description | +|--------|-------------| +| `.next()` | Apply to the **next** matching response (once). | +| `.next(count)` | Apply to the next `count` matching responses. | + +A newly created rule applies indefinitely by default, unless you restrict it with `next(...)`. + +### Configuration methods + +All configuration methods return `this` for fluent chaining and update the rule's recency (used for [multiple-rule selection](#multiple-matching-rules)). + +| Method | Description | +|--------|-------------| +| `.probability(value)` | Probability `0`–`1` that the rule fires for an eligible response. Default `1`. | +| `.status(code)` | Override the HTTP status code. | +| `.delay(ms)` | Delay the response by `ms` milliseconds. | +| `.header(name, value)` | Set or replace a response header (except `Content-Type`, which is ignored). | +| `.removeHeader(name)` | Remove a response header if present (except `Content-Type`, which is ignored). | +| `.body(value)` | Replace the response body. | +| `.transformBody(fn)` | Transform the response body: `fn` receives the current body and returns the new one. | + +### Lifecycle + +```ts +fault.stop() // disable the rule (does not consume remaining count) +fault.start() // re-enable a stopped rule +``` + +A newly created rule starts **active** by default. + +### Counting semantics + +Only responses where the rule **actually fires** (after the probability check) decrement the remaining count. Stopped rules and probability-skipped responses do not decrement the count. + +### Path prefix semantics + +A rule matches when the request path **starts with** the configured prefix. + +```ts +chaos("/orders") +// Matches: /orders, /orders/123, /orders/123/items +// Does not: /users, /inventory/orders +``` + +When `pathPrefix` is omitted (or `""`), the rule matches all paths. + +### Multiple matching rules + +When more than one active rule matches a request, exactly one is selected using this precedence: + +1. **Longest matching prefix** wins. +2. Among rules with the same prefix length, the **most recently updated** active rule wins. + +"Most recently updated" means the rule whose configuration or lifecycle state (`start`, `stop`, `next`, `probability`, `status`, `delay`, `header`, `removeHeader`, `body`, `transformBody`) was changed most recently. + +### Examples + +```ts +// Return 500 for the next request (any path) +chaos().next().status(500); + +// Return 500 for the next 3 /orders requests +chaos("/orders").next(3).status(500); + +// Always delay /orders requests by 1 second +chaos("/orders").delay(1_000); + +// Inject a 429 with a Retry-After header for the next /orders request +chaos("/orders").next().header("Retry-After", "60").status(429); + +// Add an error field to the response body for the next /orders request +chaos("/orders").next().transformBody((body) => ({ + ...body, + error: true, +})); +``` + +### Fault simulation pattern + +`chaos()` is Counterfact's fault-injection API and is available as a global in the Live REPL. +Rules are active for every matching request by default; use `probability(...)` to decide whether each individual request actually fails. + +```ts +// Evaluate this rule for every /payments request, but fail only ~20% with 503. +chaos("/payments") + .probability(0.2) + .status(503) + .header("Retry-After", "1"); +``` + +--- + ## OpenAPI Overlays [OpenAPI Overlays](https://spec.openapis.org/overlay/v1.0.0.html) let you apply targeted modifications to an OpenAPI document without editing the original file. Counterfact loads overlay files, evaluates their JSONPath targets against the spec, and applies each action before code generation and server startup. diff --git a/src/api-runner.ts b/src/api-runner.ts index 6cd4cf7dd..a386d9f5b 100644 --- a/src/api-runner.ts +++ b/src/api-runner.ts @@ -1,6 +1,7 @@ import { rm } from "node:fs/promises"; import type { Config } from "./server/config.js"; +import { ChaosRegistry } from "./server/chaos.js"; import { ContextRegistry } from "./server/context-registry.js"; import { Dispatcher } from "./server/dispatcher.js"; import { loadOpenApiDocument } from "./server/load-openapi-document.js"; @@ -111,6 +112,7 @@ export class ApiRunner { group: string, version = "", versions: readonly string[] = [], + chaosRegistry: ChaosRegistry, ) { this.group = group; this.version = version; @@ -151,6 +153,7 @@ export class ApiRunner { config, version, versions, + chaosRegistry, ); this.transpiler = new Transpiler( @@ -185,6 +188,7 @@ export class ApiRunner { group = "", version = "", versions: readonly string[] = [], + chaosRegistry = new ChaosRegistry(), ): Promise { const nativeTs = await runtimeCanExecuteErasableTs(); @@ -212,6 +216,7 @@ export class ApiRunner { group, version, versions, + chaosRegistry, ); } diff --git a/src/app.ts b/src/app.ts index 0e0f23094..ad1293b3f 100644 --- a/src/app.ts +++ b/src/app.ts @@ -6,6 +6,7 @@ import { createHttpTerminator, type HttpTerminator } from "http-terminator"; import { ApiRunner } from "./api-runner.js"; import { startRepl as startReplServer } from "./repl/repl.js"; import { createRouteFunction } from "./repl/route-builder.js"; +import { ChaosRegistry } from "./server/chaos.js"; import type { Config } from "./server/config.js"; import { ContextRegistry } from "./server/context-registry.js"; import { createKoaApp } from "./server/web-server/create-koa-app.js"; @@ -248,6 +249,8 @@ export async function counterfact(config: Config, specs?: SpecConfig[]) { } } + const chaosRegistry = new ChaosRegistry(); + const runners = await Promise.all( normalizedSpecs.map((spec) => ApiRunner.create( @@ -262,6 +265,7 @@ export async function counterfact(config: Config, specs?: SpecConfig[]) { spec.group, spec.version ?? "", versionsByGroup.get(spec.group) ?? [], + chaosRegistry, ), ), ); @@ -394,6 +398,7 @@ export async function counterfact(config: Config, specs?: SpecConfig[]) { registry: runner.registry, scenarioRegistry: runner.scenarioRegistry, })), + chaosRegistry, ), }; } diff --git a/src/repl/repl.ts b/src/repl/repl.ts index 68e6451bd..7f813896d 100644 --- a/src/repl/repl.ts +++ b/src/repl/repl.ts @@ -1,6 +1,7 @@ import repl from "node:repl"; import type { Config } from "../server/config.js"; +import type { ChaosRegistry } from "../server/chaos.js"; import type { ContextRegistry } from "../server/context-registry.js"; import type { OpenApiDocument } from "../server/dispatcher.js"; import type { Registry } from "../server/registry.js"; @@ -264,6 +265,7 @@ export function startRepl( openApiDocument?: OpenApiDocument, scenarioRegistry?: ScenarioRegistry, apiBindings?: ReplApiBinding[], + chaosRegistry?: ChaosRegistry, ) { const bindings = apiBindings === undefined || apiBindings.length === 0 @@ -486,6 +488,11 @@ export function startRepl( ? Object.fromEntries(groupedBindings.map((binding) => [binding.key, {}])) : {}; + if (chaosRegistry !== undefined) { + replServer.context.chaos = (pathPrefix = "") => + chaosRegistry.createRule(pathPrefix); + } + replServer.defineCommand("scenario", { async action(text: string) { sendTelemetry("repl_command_used", { command: "scenario" }); diff --git a/src/server/chaos.ts b/src/server/chaos.ts new file mode 100644 index 000000000..030545222 --- /dev/null +++ b/src/server/chaos.ts @@ -0,0 +1,341 @@ +import type { CounterfactResponseObject } from "./registry.js"; + +/** + * Sentinel value used to distinguish "body not set" from `undefined`. + * This allows `body()` to explicitly replace the response body with `undefined`. + */ +const UNSET = Symbol("UNSET"); +const CONTENT_TYPE_HEADER = "content-type"; + +/** + * A monotonically increasing counter used to determine rule recency. + * Using a counter instead of `Date.now()` ensures stable ordering even when + * multiple rules are updated within the same millisecond (common in tests). + */ +let _sequence = 0; + +/** + * Result returned by {@link ChaosRule.tryApply} when the rule fires. + */ +export interface ChaosApplyResult { + /** Number of milliseconds to delay the response, if any. */ + delayMs?: number; + /** The (potentially modified) response object. */ + response: CounterfactResponseObject; +} + +/** + * A single chaos rule that can modify HTTP responses for paths matching a + * given prefix. + * + * All configuration methods return `this` for fluent chaining and update the + * rule's recency so that the most recently configured rule wins when multiple + * rules match the same request path. + * + * ### Example + * + * ```ts + * const fault = chaos("/orders") + * .next(3) + * .probability(0.5) + * .status(500) + * .delay(1_000) + * .header("Retry-After", "60"); + * ``` + */ +export class ChaosRule { + /** Path prefix this rule matches against. Empty string matches all paths. */ + public readonly prefix: string; + + private _remaining: number | "always"; + private _probability = 1; + private _status?: number; + private _delay?: number; + private _headers = new Map(); + private _removedHeaders = new Set(); + private _body: symbol | unknown = UNSET; + private _transformBody?: (body: unknown) => unknown; + private _active = true; + private _updatedAt = ++_sequence; + + /** @internal */ + public constructor(prefix: string) { + this.prefix = prefix; + this._remaining = "always"; + } + + /** Monotonically increasing value representing when this rule was last updated. */ + public get updatedAt(): number { + return this._updatedAt; + } + + /** `true` when the rule is active (not stopped and has remaining count). */ + public get isEligible(): boolean { + return ( + this._active && (this._remaining === "always" || this._remaining > 0) + ); + } + + private touch(): void { + this._updatedAt = ++_sequence; + } + + /** + * Configures this rule to apply to the next matching response. + * + * When `count` is omitted, applies once. When provided, applies to the next + * `count` matching responses. Only responses where the rule actually fires + * (after probability check) decrement the count. + */ + public next(count = 1): this { + this._remaining = count; + this.touch(); + return this; + } + + /** + * Sets the probability that the rule fires for each eligible response. + * + * @param value - A number between `0` (never) and `1` (always). Default is `1`. + */ + public probability(value: number): this { + if (!Number.isFinite(value) || value < 0 || value > 1) { + throw new RangeError( + `Chaos rule probability must be a number between 0 and 1. Received: ${String(value)}`, + ); + } + + this._probability = value; + this.touch(); + return this; + } + + /** + * Overrides the HTTP status code of the response. + * + * @param code - The HTTP status code to return (e.g. `500`, `429`). + */ + public status(code: number): this { + this._status = code; + this.touch(); + return this; + } + + /** + * Delays the response by the specified number of milliseconds. + * + * @param ms - Number of milliseconds to delay. + */ + public delay(ms: number): this { + this._delay = ms; + this.touch(); + return this; + } + + /** + * Sets or replaces a response header. + * + * @param name - Header name. + * @param value - Header value. + */ + public header(name: string, value: string): this { + if (name.toLowerCase() === CONTENT_TYPE_HEADER) { + return this; + } + + this._headers.set(name, value); + this.touch(); + return this; + } + + /** + * Removes a response header if present. + * + * @param name - Header name to remove. + */ + public removeHeader(name: string): this { + if (name.toLowerCase() === CONTENT_TYPE_HEADER) { + return this; + } + + this._removedHeaders.add(name); + this.touch(); + return this; + } + + /** + * Replaces the response body with the given value. + * + * Clears any previously configured {@link transformBody} transformer. + * + * @param value - The new response body. + */ + public body(value: unknown): this { + this._body = value; + this._transformBody = undefined; + this.touch(); + return this; + } + + /** + * Transforms the response body using the given function. + * + * The transformer receives the current response body and returns the new + * body. Clears any previously configured {@link body} replacement. + * + * @param fn - Function that receives the current body and returns the new body. + */ + public transformBody(fn: (body: unknown) => unknown): this { + this._transformBody = fn; + this._body = UNSET; + this.touch(); + return this; + } + + /** + * Disables this rule. Stopped rules do not affect responses and do not + * decrement their remaining count. + */ + public stop(): this { + this._active = false; + this.touch(); + return this; + } + + /** + * Re-enables a previously stopped rule. + */ + public start(): this { + this._active = true; + this.touch(); + return this; + } + + /** + * Attempts to apply this rule to `response`. + * + * Returns `null` when the rule does not apply: + * - The rule is inactive (stopped). + * - The remaining count is exhausted. + * - The probability check fails (skipped — count is not decremented). + * + * Returns a {@link ChaosApplyResult} when the rule fires, with the + * modified response and an optional delay. + * + * @param response - The original response from the route handler. + */ + public tryApply( + response: CounterfactResponseObject, + ): ChaosApplyResult | null { + if (!this._active) { + return null; + } + + if (this._remaining !== "always" && this._remaining <= 0) { + return null; + } + + // Probability check: skipped responses do NOT decrement the count. + if (Math.random() > this._probability) { + return null; + } + + // Decrement the count ONLY when the rule fires. + if (this._remaining !== "always") { + this._remaining--; + } + + // Apply header modifications. + const headers: CounterfactResponseObject["headers"] = { + ...(response.headers ?? {}), + }; + + for (const [name, value] of this._headers) { + headers[name] = value; + } + + for (const name of this._removedHeaders) { + delete headers[name]; + } + + // Apply body modifications. + // The body is typed as unknown here because chaos rules may replace it + // with any value (including plain objects that Koa will serialize to JSON). + // The cast to CounterfactResponseObject['body'] is applied below when + // setting the result property. + let body: unknown = response.body; + + if (this._body !== UNSET) { + body = this._body; + } else if (this._transformBody !== undefined) { + body = this._transformBody(body); + } + + const result: CounterfactResponseObject = { + ...response, + // Cast is safe: Koa serializes object bodies to JSON at the middleware level. + body: body as CounterfactResponseObject["body"], + headers, + }; + + if (this._status !== undefined) { + result.status = this._status; + } + + return { response: result, delayMs: this._delay }; + } +} + +/** + * Stores and selects active chaos rules for incoming requests. + * + * When multiple rules match a request path, the registry applies the rule + * with the longest matching prefix. Among rules with the same prefix length, + * the most recently updated rule is chosen. + */ +export class ChaosRegistry { + private readonly rules: ChaosRule[] = []; + + /** + * Creates a new {@link ChaosRule} for the given path prefix and registers + * it with this registry. + * + * @param prefix - URL path prefix. When omitted, the rule matches all paths. + * @returns The newly created rule for fluent configuration. + */ + public createRule(prefix = ""): ChaosRule { + const rule = new ChaosRule(prefix); + this.rules.push(rule); + return rule; + } + + /** + * Finds the best matching active, eligible rule for the given request path. + * + * Selection priority: + * 1. Longest matching path prefix. + * 2. Most recently updated rule (highest `updatedAt` value). + * + * @param path - The incoming request path (e.g. `/orders/123`). + * @returns The best matching rule, or `undefined` when no rule matches. + */ + public findBestMatch(path: string): ChaosRule | undefined { + const eligible = this.rules.filter( + (rule) => rule.isEligible && path.startsWith(rule.prefix), + ); + + if (eligible.length === 0) { + return undefined; + } + + const maxPrefixLength = Math.max(...eligible.map((r) => r.prefix.length)); + + const longestPrefixRules = eligible.filter( + (r) => r.prefix.length === maxPrefixLength, + ); + + return longestPrefixRules.reduce((best, r) => + r.updatedAt > best.updatedAt ? r : best, + ); + } +} diff --git a/src/server/dispatcher.ts b/src/server/dispatcher.ts index f4bf4f49e..2339d0dae 100644 --- a/src/server/dispatcher.ts +++ b/src/server/dispatcher.ts @@ -4,6 +4,7 @@ import createDebugger from "debug"; import fetch, { Headers } from "node-fetch"; import type { ContextRegistry } from "./context-registry.js"; +import type { ChaosRegistry } from "./chaos.js"; import type { HttpMethods, RequestMethod, @@ -208,6 +209,12 @@ export class Dispatcher { "validateRequests" | "validateResponses" | "alwaysFakeOptionals" >; // Add config property + /** + * Registry of active chaos rules. When set, each response is checked + * against the registry and the best matching rule (if any) is applied. + */ + public chaosRegistry?: ChaosRegistry; + /** * The version label for this dispatcher's spec (e.g. `"v1"`, `"v2"`). * Empty string when running without a version. @@ -232,6 +239,7 @@ export class Dispatcher { >, version = "", versions: readonly string[] = [], + chaosRegistry?: ChaosRegistry, ) { this.registry = registry; this.contextRegistry = contextRegistry; @@ -240,6 +248,7 @@ export class Dispatcher { this.config = config; this.version = version; this.versions = versions; + this.chaosRegistry = chaosRegistry; } private parameterTypes( @@ -696,6 +705,25 @@ export class Dispatcher { } } + // Apply chaos rules after normal response processing. + if (this.chaosRegistry !== undefined) { + const rule = this.chaosRegistry.findBestMatch(path); + + if (rule !== undefined) { + const chaosResult = rule.tryApply(normalizedResponse); + + if (chaosResult !== null) { + if (chaosResult.delayMs !== undefined && chaosResult.delayMs > 0) { + await new Promise((resolve) => { + setTimeout(resolve, chaosResult.delayMs); + }); + } + + return chaosResult.response; + } + } + } + return normalizedResponse; } } diff --git a/test/app.test.ts b/test/app.test.ts index 6f86f7150..f20345491 100644 --- a/test/app.test.ts +++ b/test/app.test.ts @@ -6,6 +6,7 @@ import { usingTemporaryFiles } from "using-temporary-files"; import * as app from "../src/app"; import { ApiRunner } from "../src/api-runner"; +import { ChaosRegistry } from "../src/server/chaos"; import { ContextRegistry } from "../src/server/context-registry"; import { ScenarioRegistry } from "../src/server/scenario-registry"; @@ -25,6 +26,10 @@ const mockConfig = { prefix: "", }; +type ReplChaos = (pathPrefix?: string) => { + status: (statusCode: number) => unknown; +}; + describe("counterfact", () => { it("returns a startRepl function", async () => { const result = await (app as any).counterfact(mockConfig); @@ -72,13 +77,16 @@ describe("counterfact", () => { "v1", "", [], + expect.any(ChaosRegistry), ); expect(spy).toHaveBeenCalledWith( expect.objectContaining({ openApiPath: "_", prefix: "/api/v2" }), "v2", "", [], + expect.any(ChaosRegistry), ); + expect(spy.mock.calls[0]?.[4]).toBe(spy.mock.calls[1]?.[4]); spy.mockRestore(); }); @@ -200,6 +208,53 @@ describe("counterfact", () => { }); }); + it("applies REPL chaos rules across all runners in multi-api mode", async () => { + await usingTemporaryFiles(async ($) => { + await $.add( + "billing/routes/hello.js", + `export function GET() { return { body: "hello from billing" }; }`, + ); + await $.add( + "inventory/routes/hello.js", + `export function GET() { return { body: "hello from inventory" }; }`, + ); + + const specs = [ + { source: "_", prefix: "/api/billing", group: "billing" }, + { source: "_", prefix: "/api/inventory", group: "inventory" }, + ]; + + const { koaApp, start, startRepl } = await (app as any).counterfact( + { ...mockConfig, basePath: $.path(".") }, + specs, + ); + + const { stop } = await start({ + startServer: true, + buildCache: false, + generate: { routes: false, types: false }, + watch: { routes: false, types: false }, + }); + + const replServer = startRepl(); + const chaos = (replServer.context as { chaos: ReplChaos }).chaos; + chaos().status(503); + + const billingResponse = await request(koaApp.callback()).get( + "/api/billing/hello", + ); + const inventoryResponse = await request(koaApp.callback()).get( + "/api/inventory/hello", + ); + + expect(billingResponse.status).toBe(503); + expect(inventoryResponse.status).toBe(503); + + replServer.close(); + await stop(); + }); + }); + it("routes requests to the correct runner based on prefix when specs are provided", async () => { await usingTemporaryFiles(async ($) => { await $.add( @@ -252,12 +307,14 @@ describe("counterfact", () => { "my-api", "v1", ["v1", "v2"], + expect.any(ChaosRegistry), ); expect(spy).toHaveBeenCalledWith( expect.objectContaining({ prefix: "/my-api/v2" }), "my-api", "v2", ["v1", "v2"], + expect.any(ChaosRegistry), ); spy.mockRestore(); @@ -277,6 +334,7 @@ describe("counterfact", () => { "my-api", "v1", ["v1"], + expect.any(ChaosRegistry), ); spy.mockRestore(); @@ -294,6 +352,7 @@ describe("counterfact", () => { "my-api", "", [], + expect.any(ChaosRegistry), ); spy.mockRestore(); diff --git a/test/server/chaos.test.ts b/test/server/chaos.test.ts new file mode 100644 index 000000000..34d2ae77d --- /dev/null +++ b/test/server/chaos.test.ts @@ -0,0 +1,563 @@ +import { describe, expect, it, jest } from "@jest/globals"; + +import { ChaosRegistry, ChaosRule } from "../../src/server/chaos.js"; +import { ContextRegistry } from "../../src/server/context-registry.js"; +import { Dispatcher } from "../../src/server/dispatcher.js"; +import { Registry } from "../../src/server/registry.js"; + +// --------------------------------------------------------------------------- +// Constants +// --------------------------------------------------------------------------- + +/** Number of iterations for probability boundary tests. */ +const PROBABILITY_TEST_ITERATIONS = 20; + +// --------------------------------------------------------------------------- +// Helpers +// --------------------------------------------------------------------------- + +function makeDispatcher(chaosRegistry: ChaosRegistry): Dispatcher { + const registry = new Registry(); + + registry.add("/orders", { + GET() { + return { + body: JSON.stringify({ id: 1, status: "pending" }), + contentType: "application/json", + headers: { "x-original": "yes" }, + status: 200, + }; + }, + }); + + registry.add("/users", { + GET() { + return { + body: "user list", + contentType: "text/plain", + status: 200, + }; + }, + }); + + return new Dispatcher( + registry, + new ContextRegistry(), + undefined, + { validateRequests: false, validateResponses: false }, + "", + [], + chaosRegistry, + ); +} + +async function get(dispatcher: Dispatcher, path: string) { + return dispatcher.request({ + body: "", + headers: {}, + method: "GET", + path, + query: {}, + req: { path }, + }); +} + +// --------------------------------------------------------------------------- +// ChaosRule unit tests +// --------------------------------------------------------------------------- + +describe("ChaosRule", () => { + describe("tryApply", () => { + it("returns null when the rule is stopped", () => { + const rule = new ChaosRule("").next(1).stop(); + const response = { body: "ok", status: 200 }; + expect(rule.tryApply(response)).toBeNull(); + }); + + it("returns null when remaining count is exhausted", () => { + const rule = new ChaosRule("").next(1); + const response = { body: "ok", status: 200 }; + rule.tryApply(response); // fires once → count → 0 + expect(rule.tryApply(response)).toBeNull(); + }); + + it("returns null with probability(0)", () => { + const rule = new ChaosRule("").probability(0); + const response = { body: "ok", status: 200 }; + // With probability 0, every call returns null + for (let i = 0; i < PROBABILITY_TEST_ITERATIONS; i++) { + expect(rule.tryApply(response)).toBeNull(); + } + }); + + it("always fires with probability(1)", () => { + const rule = new ChaosRule("").probability(1); + const response = { body: "ok", status: 200 }; + for (let i = 0; i < PROBABILITY_TEST_ITERATIONS; i++) { + expect(rule.tryApply(response)).not.toBeNull(); + } + }); + + it("throws when probability is outside [0, 1]", () => { + const rule = new ChaosRule(""); + + expect(() => rule.probability(-0.1)).toThrow( + "Chaos rule probability must be a number between 0 and 1", + ); + expect(() => rule.probability(1.1)).toThrow( + "Chaos rule probability must be a number between 0 and 1", + ); + expect(() => rule.probability(Number.NaN)).toThrow( + "Chaos rule probability must be a number between 0 and 1", + ); + }); + + it("overrides status code", () => { + const rule = new ChaosRule("").status(500); + const result = rule.tryApply({ body: "ok", status: 200 }); + expect(result?.response.status).toBe(500); + }); + + it("adds a header", () => { + const rule = new ChaosRule("").header("Retry-After", "60"); + const result = rule.tryApply({ body: "ok", status: 200, headers: {} }); + expect(result?.response.headers?.["Retry-After"]).toBe("60"); + }); + + it("does not allow overriding Content-Type with header()", () => { + const rule = new ChaosRule("").header("Content-Type", "text/xml"); + const result = rule.tryApply({ + body: "ok", + contentType: "application/json", + status: 200, + }); + expect(result?.response.contentType).toBe("application/json"); + expect(result?.response.headers?.["Content-Type"]).toBeUndefined(); + }); + + it("removes a header", () => { + const rule = new ChaosRule("").removeHeader("x-original"); + const result = rule.tryApply({ + body: "ok", + status: 200, + headers: { "x-original": "yes", keep: "this" }, + }); + expect(result?.response.headers?.["x-original"]).toBeUndefined(); + expect(result?.response.headers?.["keep"]).toBe("this"); + }); + + it("does not allow removing Content-Type with removeHeader()", () => { + const rule = new ChaosRule("").removeHeader("content-type"); + const result = rule.tryApply({ + body: "ok", + contentType: "application/json", + status: 200, + }); + expect(result?.response.contentType).toBe("application/json"); + expect(result?.response.headers).toEqual({}); + }); + + it("replaces the body", () => { + const rule = new ChaosRule("").body({ error: true }); + const result = rule.tryApply({ body: "original", status: 200 }); + expect(result?.response.body).toEqual({ error: true }); + }); + + it("transforms the body", () => { + const rule = new ChaosRule("").transformBody((b) => `${b}-modified`); + const result = rule.tryApply({ body: "original", status: 200 }); + expect(result?.response.body).toBe("original-modified"); + }); + + it("sets delayMs from delay()", () => { + const rule = new ChaosRule("").delay(1_000); + const result = rule.tryApply({ body: "ok", status: 200 }); + expect(result?.delayMs).toBe(1_000); + }); + + it("does not set delayMs when no delay is configured", () => { + const rule = new ChaosRule("").status(500); + const result = rule.tryApply({ body: "ok", status: 200 }); + expect(result?.delayMs).toBeUndefined(); + }); + + it("preserves the original response fields not explicitly changed", () => { + const rule = new ChaosRule("").status(500); + const result = rule.tryApply({ + body: "hello", + contentType: "text/plain", + headers: { "x-custom": "val" }, + status: 200, + }); + expect(result?.response.body).toBe("hello"); + expect(result?.response.contentType).toBe("text/plain"); + expect(result?.response.headers?.["x-custom"]).toBe("val"); + }); + }); + + describe("next()", () => { + it("applies exactly once when next() is called with no argument", () => { + const rule = new ChaosRule("").next().status(500); + const resp = { body: "ok", status: 200 }; + expect(rule.tryApply(resp)?.response.status).toBe(500); + expect(rule.tryApply(resp)).toBeNull(); + }); + + it("applies exactly count times when next(count) is called", () => { + const rule = new ChaosRule("").next(3).status(500); + const resp = { body: "ok", status: 200 }; + for (let i = 0; i < 3; i++) { + expect(rule.tryApply(resp)?.response.status).toBe(500); + } + expect(rule.tryApply(resp)).toBeNull(); + }); + + it("does not decrement the count when probability skips the response", () => { + // Use probability(0) to ensure every check is skipped + const rule = new ChaosRule("").next(2).probability(0); + const resp = { body: "ok", status: 200 }; + // These calls are all skipped; count should not decrement + rule.tryApply(resp); + rule.tryApply(resp); + rule.tryApply(resp); + // Switch to probability(1) – count should still be 2 + rule.probability(1); + expect(rule.tryApply(resp)).not.toBeNull(); + expect(rule.tryApply(resp)).not.toBeNull(); + expect(rule.tryApply(resp)).toBeNull(); + }); + + it("does not decrement the count when the rule is stopped", () => { + const rule = new ChaosRule("").next(2).stop(); + const resp = { body: "ok", status: 200 }; + rule.tryApply(resp); + rule.tryApply(resp); + // Re-enable; count should still be 2 + rule.start(); + expect(rule.tryApply(resp)).not.toBeNull(); + expect(rule.tryApply(resp)).not.toBeNull(); + expect(rule.tryApply(resp)).toBeNull(); + }); + }); + + describe("default rule scope", () => { + it("continues to apply indefinitely unless next() is configured", () => { + const rule = new ChaosRule("").status(500); + const resp = { body: "ok", status: 200 }; + for (let i = 0; i < 50; i++) { + expect(rule.tryApply(resp)?.response.status).toBe(500); + } + }); + }); + + describe("stop() / start()", () => { + it("stop() disables the rule", () => { + const rule = new ChaosRule("").status(500); + rule.stop(); + expect(rule.tryApply({ body: "ok", status: 200 })).toBeNull(); + }); + + it("start() re-enables a stopped rule", () => { + const rule = new ChaosRule("").status(500); + rule.stop(); + rule.start(); + expect(rule.tryApply({ body: "ok", status: 200 })?.response.status).toBe( + 500, + ); + }); + }); + + describe("body() vs transformBody()", () => { + it("body() clears a previously set transformBody()", () => { + const rule = new ChaosRule(""); + rule.transformBody((b) => `${b}-transformed`); + rule.body("static"); + const result = rule.tryApply({ body: "original", status: 200 }); + expect(result?.response.body).toBe("static"); + }); + + it("transformBody() clears a previously set body()", () => { + const rule = new ChaosRule(""); + rule.body("static"); + rule.transformBody((b) => `${b}-transformed`); + const result = rule.tryApply({ body: "original", status: 200 }); + expect(result?.response.body).toBe("original-transformed"); + }); + }); +}); + +// --------------------------------------------------------------------------- +// ChaosRegistry unit tests +// --------------------------------------------------------------------------- + +describe("ChaosRegistry", () => { + describe("findBestMatch", () => { + it("returns undefined when no rules are registered", () => { + const registry = new ChaosRegistry(); + expect(registry.findBestMatch("/orders")).toBeUndefined(); + }); + + it("matches a global rule (empty prefix) against any path", () => { + const registry = new ChaosRegistry(); + const rule = registry.createRule("").status(500); + expect(registry.findBestMatch("/anything")).toBe(rule); + }); + + it("matches a prefix-scoped rule only when path starts with prefix", () => { + const registry = new ChaosRegistry(); + const rule = registry.createRule("/orders").status(500); + expect(registry.findBestMatch("/orders/123")).toBe(rule); + expect(registry.findBestMatch("/users")).toBeUndefined(); + }); + + it("does not match /inventory/orders for prefix /orders", () => { + const registry = new ChaosRegistry(); + registry.createRule("/orders").status(500); + expect(registry.findBestMatch("/inventory/orders")).toBeUndefined(); + }); + + it("prefers the longest matching prefix", () => { + const registry = new ChaosRegistry(); + const global = registry.createRule("").status(500); + const orders = registry.createRule("/orders").status(429); + const result = registry.findBestMatch("/orders/123"); + expect(result).toBe(orders); + expect(result).not.toBe(global); + }); + + it("prefers the most recently updated rule among equal-length prefixes", () => { + const registry = new ChaosRegistry(); + registry.createRule("/orders").status(500); + const second = registry.createRule("/orders").status(429); + // second was created (and therefore touched) after first + expect(registry.findBestMatch("/orders/123")).toBe(second); + }); + + it("skips stopped rules", () => { + const registry = new ChaosRegistry(); + const rule = registry.createRule("/orders").status(500); + rule.stop(); + expect(registry.findBestMatch("/orders/123")).toBeUndefined(); + }); + + it("skips exhausted rules", () => { + const registry = new ChaosRegistry(); + const rule = registry.createRule("/orders").next(1).status(500); + rule.tryApply({ body: "ok", status: 200 }); // exhaust + expect(registry.findBestMatch("/orders/123")).toBeUndefined(); + }); + + it("a stopped rule becomes the most recently updated after start()", () => { + const registry = new ChaosRegistry(); + const first = registry.createRule("/orders").status(500); + registry.createRule("/orders").status(429); + first.stop(); + first.start(); // first is now the most recently updated + expect(registry.findBestMatch("/orders/123")).toBe(first); + }); + }); +}); + +// --------------------------------------------------------------------------- +// Integration tests via Dispatcher +// --------------------------------------------------------------------------- + +describe("Dispatcher with ChaosRegistry", () => { + it("applies a global rule to all paths", async () => { + const cr = new ChaosRegistry(); + cr.createRule("").status(503); + const dispatcher = makeDispatcher(cr); + const response = await get(dispatcher, "/orders"); + expect(response.status).toBe(503); + }); + + it("applies a prefix-scoped rule only to matching paths", async () => { + const cr = new ChaosRegistry(); + cr.createRule("/orders").status(429); + const dispatcher = makeDispatcher(cr); + + const ordersResponse = await get(dispatcher, "/orders"); + expect(ordersResponse.status).toBe(429); + + const usersResponse = await get(dispatcher, "/users"); + expect(usersResponse.status).toBe(200); + }); + + it("applies next() exactly once", async () => { + const cr = new ChaosRegistry(); + cr.createRule("/orders").next().status(500); + const dispatcher = makeDispatcher(cr); + + const first = await get(dispatcher, "/orders"); + expect(first.status).toBe(500); + + const second = await get(dispatcher, "/orders"); + expect(second.status).toBe(200); + }); + + it("applies next(count) the expected number of times", async () => { + const cr = new ChaosRegistry(); + cr.createRule("/orders").next(2).status(500); + const dispatcher = makeDispatcher(cr); + + expect((await get(dispatcher, "/orders")).status).toBe(500); + expect((await get(dispatcher, "/orders")).status).toBe(500); + expect((await get(dispatcher, "/orders")).status).toBe(200); + }); + + it("default rules continue to apply", async () => { + const cr = new ChaosRegistry(); + cr.createRule("/orders").status(503); + const dispatcher = makeDispatcher(cr); + + for (let i = 0; i < 5; i++) { + expect((await get(dispatcher, "/orders")).status).toBe(503); + } + }); + + it("probability(0) never applies", async () => { + const cr = new ChaosRegistry(); + cr.createRule("/orders").probability(0).status(500); + const dispatcher = makeDispatcher(cr); + + for (let i = 0; i < 10; i++) { + expect((await get(dispatcher, "/orders")).status).toBe(200); + } + }); + + it("probability(1) always applies", async () => { + const cr = new ChaosRegistry(); + cr.createRule("/orders").probability(1).status(500); + const dispatcher = makeDispatcher(cr); + + for (let i = 0; i < 5; i++) { + expect((await get(dispatcher, "/orders")).status).toBe(500); + } + }); + + it("status() overrides the response status code", async () => { + const cr = new ChaosRegistry(); + cr.createRule("/orders").status(429); + const dispatcher = makeDispatcher(cr); + const response = await get(dispatcher, "/orders"); + expect(response.status).toBe(429); + }); + + it("header() adds or replaces a response header", async () => { + const cr = new ChaosRegistry(); + cr.createRule("/orders").header("Retry-After", "60"); + const dispatcher = makeDispatcher(cr); + const response = await get(dispatcher, "/orders"); + expect(response.headers?.["Retry-After"]).toBe("60"); + }); + + it("removeHeader() removes a response header", async () => { + const cr = new ChaosRegistry(); + cr.createRule("/orders").removeHeader("x-original"); + const dispatcher = makeDispatcher(cr); + const response = await get(dispatcher, "/orders"); + expect(response.headers?.["x-original"]).toBeUndefined(); + }); + + it("header()/removeHeader() do not modify response Content-Type", async () => { + const cr = new ChaosRegistry(); + cr.createRule("/orders") + .header("content-type", "text/plain") + .removeHeader("content-type"); + const dispatcher = makeDispatcher(cr); + const response = await get(dispatcher, "/orders"); + expect(response.contentType).toBe("application/json"); + }); + + it("body() replaces the response body", async () => { + const cr = new ChaosRegistry(); + cr.createRule("/orders").body("chaos body"); + const dispatcher = makeDispatcher(cr); + const response = await get(dispatcher, "/orders"); + expect(response.body).toBe("chaos body"); + }); + + it("transformBody() transforms the existing body", async () => { + const cr = new ChaosRegistry(); + cr.createRule("/orders").transformBody((b) => `${b}-modified`); + const dispatcher = makeDispatcher(cr); + const response = await get(dispatcher, "/orders"); + expect(response.body).toContain("-modified"); + }); + + it("stop() disables a rule", async () => { + const cr = new ChaosRegistry(); + const rule = cr.createRule("/orders").status(500); + rule.stop(); + const dispatcher = makeDispatcher(cr); + const response = await get(dispatcher, "/orders"); + expect(response.status).toBe(200); + }); + + it("start() re-enables a stopped rule", async () => { + const cr = new ChaosRegistry(); + const rule = cr.createRule("/orders").status(500); + rule.stop(); + rule.start(); + const dispatcher = makeDispatcher(cr); + const response = await get(dispatcher, "/orders"); + expect(response.status).toBe(500); + }); + + it("stopped rules do not decrement their remaining count", async () => { + const cr = new ChaosRegistry(); + const rule = cr.createRule("/orders").next(2).status(500); + const dispatcher = makeDispatcher(cr); + + rule.stop(); + await get(dispatcher, "/orders"); // should not decrement + await get(dispatcher, "/orders"); // should not decrement + + rule.start(); + expect((await get(dispatcher, "/orders")).status).toBe(500); + expect((await get(dispatcher, "/orders")).status).toBe(500); + expect((await get(dispatcher, "/orders")).status).toBe(200); + }); + + it("probability-skipped responses do not decrement the remaining count", async () => { + const cr = new ChaosRegistry(); + const rule = cr.createRule("/orders").next(2).probability(0).status(500); + const dispatcher = makeDispatcher(cr); + + // All skipped due to probability(0) + for (let i = 0; i < 5; i++) { + expect((await get(dispatcher, "/orders")).status).toBe(200); + } + + // Switch to always-fire + rule.probability(1); + expect((await get(dispatcher, "/orders")).status).toBe(500); + expect((await get(dispatcher, "/orders")).status).toBe(500); + expect((await get(dispatcher, "/orders")).status).toBe(200); + }); + + it("delay() delays the response", async () => { + const delays: number[] = []; + const originalSetTimeout = global.setTimeout; + + // Spy on setTimeout to record delays without actually waiting + jest + .spyOn(global, "setTimeout") + // eslint-disable-next-line @typescript-eslint/no-explicit-any -- spy needs any for args + .mockImplementation((fn: any, ms?: number) => { + // eslint-disable-next-line jest/no-conditional-in-test -- ms ?? 0 is a default, not a test branch + delays.push(ms ?? 0); + return originalSetTimeout(fn, 0); // fire immediately + }); + + const cr = new ChaosRegistry(); + cr.createRule("/orders").delay(2_000); + const dispatcher = makeDispatcher(cr); + + const response = await get(dispatcher, "/orders"); + expect(response.status).toBe(200); + expect(delays).toContain(2_000); + + jest.restoreAllMocks(); + }); +});