diff --git a/packages/agent-cdp/README.md b/packages/agent-cdp/README.md index 7715029..779699f 100644 --- a/packages/agent-cdp/README.md +++ b/packages/agent-cdp/README.md @@ -1,6 +1,6 @@ # agent-cdp -**agent-cdp** is a command-line tool that connects to apps and pages through the **Chrome DevTools Protocol (CDP)**. Use it to list debuggable targets, stream console output, record traces, inspect JavaScript heap usage, capture and analyze heap snapshots, and run JavaScript CPU profiles—all without opening DevTools yourself. +**agent-cdp** is a command-line tool that connects to apps and pages through the **Chrome DevTools Protocol (CDP)**. Use it to list debuggable targets, inspect network traffic, stream console output, record traces, inspect JavaScript heap usage, capture and analyze heap snapshots, and run JavaScript CPU profiles, all without opening DevTools yourself. ## Compatibility @@ -91,6 +91,7 @@ agent-cdp target clear **3. Use the features you need** - **Console** — list and fetch log lines: `console list`, `console get ` +- **Network** — bounded live capture plus persisted sessions: `network status`, `network start`, `network summary`, `network list`, `network request`, `network request-headers`, `network response-headers`, `network request-body`, `network response-body` - **Trace** — `trace start` / `trace stop [--file PATH]` for raw trace capture - **Memory (raw)** — `memory capture --file PATH` for a heap snapshot file - **Heap snapshot tools** — `mem-snapshot` commands to capture, load, summarize, diff snapshots, inspect classes/instances/retainers, and triage leak-style comparisons @@ -107,4 +108,55 @@ agent-cdp stop ## Command overview -Commands are grouped as **daemon**, **target**, **console**, **trace**, **memory**, **mem-snapshot**, **js-memory**, **js-allocation**, **js-allocation-timeline**, **js-profile**, and **skills** (bundled reference files). See `agent-cdp --help` for exact syntax and options. +Commands are grouped as **daemon**, **target**, **console**, **network**, **trace**, **memory**, **mem-snapshot**, **js-memory**, **js-allocation**, **js-allocation-timeline**, **js-profile**, and **skills** (bundled reference files). See `agent-cdp --help` for exact syntax and options. + +## Network inspection + +Use `network` when you need compact request summaries first, then explicit drill-down commands for details. + +Quick start: + +```sh +agent-cdp network status +agent-cdp network start --name login-flow +# reproduce the failing or slow interaction +agent-cdp network stop +agent-cdp network summary --session net_1 +agent-cdp network list --session net_1 --status failed +agent-cdp network request --session net_1 --id req_12 +agent-cdp network response-headers --session net_1 --id req_12 +agent-cdp network response-body --session net_1 --id req_12 --file ./response-body.txt +``` + +Default behavior: + +- The daemon keeps an always-on live rolling buffer of the most recent `200` normalized requests for the active target. +- `network start` begins an empty persisted recording session. It does not backfill from the live buffer. +- When `--session` is omitted, `network` queries prefer the active or latest persisted session. If no session exists, they read from the live buffer. +- `network request` shows metadata, timing, sizes, redirects, and availability flags only. It does not print headers or body previews by default. +- Use `network request-headers`, `network response-headers`, `network request-body`, and `network response-body` for explicit drill-down. + +Examples: + +```sh +agent-cdp network list --type xhr --min-ms 500 +agent-cdp network list --status 5xx --text graphql +agent-cdp network request --id req_7 +agent-cdp network request-headers --id req_7 --name authorization +agent-cdp network response-body --id req_7 +``` + +Current limitations: + +- Network tooling depends on the target emitting usable CDP `Network.*` events. +- Support is capability-driven, not runtime-name-driven. There is no runtime-specific fallback instrumentation in v1. +- The live buffer is limited to the most recent `200` requests. +- Persisted sessions start empty and do not backfill from the live buffer. +- Default request detail omits headers and bodies. +- No request or response body previews are printed by default. +- Full request and response bodies may still be unavailable depending on target behavior, timing, and connection lifetime. +- Binary bodies may be easier to consume through `--file` export. +- No default redaction is applied in v1. +- WebSocket visibility is limited to handshake metadata in v1. +- There is no throttling, blocking, mocking, replay, or HAR export in v1. +- Timing, size, protocol, cache, and remote-endpoint metadata may be partial or absent depending on target behavior. diff --git a/packages/agent-cdp/skills/core.md b/packages/agent-cdp/skills/core.md index e37e156..88545d4 100644 --- a/packages/agent-cdp/skills/core.md +++ b/packages/agent-cdp/skills/core.md @@ -1,6 +1,6 @@ --- name: core -description: Core agent-cdp usage guide. Read this before running any agent-cdp commands. Covers the daemon lifecycle, target selection, console capture, trace recording, heap snapshot analysis, JS heap monitoring, and CPU profiling workflows. Use when you need to analyze memory leaks, profile JavaScript CPU usage, capture heap snapshots, or monitor runtime behavior of a Chrome/Node.js target via Chrome DevTools Protocol. +description: Core agent-cdp usage guide. Read this before running any agent-cdp commands. Covers the daemon lifecycle, target selection, network inspection, console capture, trace recording, heap snapshot analysis, JS heap monitoring, and CPU profiling workflows. Use when you need to analyze network failures, memory leaks, CPU hotspots, or runtime behavior of a Chrome/Node.js target via Chrome DevTools Protocol. allowed-tools: Bash(agent-cdp:*) --- @@ -8,7 +8,7 @@ allowed-tools: Bash(agent-cdp:*) CLI for deep runtime analysis of Chrome and Node.js processes via Chrome DevTools Protocol (CDP). Captures heap snapshots, CPU profiles, JS memory -samples, console output, and performance traces — all without modifying +samples, network traffic, console output, and performance traces — all without modifying source code. ## The core loop @@ -79,6 +79,16 @@ agent-cdp console get # get full details of a specific message Console messages are collected while the daemon is running with an active target. +## Network inspection + +For network workflows, run: + +```bash +agent-cdp skills get network +``` + +That skill contains session behavior, common workflows, body inspection guidance, and network-specific troubleshooting. + ## Trace recording ```bash diff --git a/packages/agent-cdp/skills/network.md b/packages/agent-cdp/skills/network.md new file mode 100644 index 0000000..206bc70 --- /dev/null +++ b/packages/agent-cdp/skills/network.md @@ -0,0 +1,165 @@ +--- +name: network +description: Network inspection workflows for agent-cdp. Use after reading the core skill and selecting a target. Covers session lifecycle, failed-request triage, body inspection, binary export, and practical guidance for agent-friendly network debugging. +allowed-tools: Bash(agent-cdp:*) +--- + +# agent-cdp network + +Focused guide for network capture and inspection after the daemon is running and a target has been selected. + +Prerequisite: + +```bash +agent-cdp skills get core +agent-cdp start +agent-cdp target list --url URL +agent-cdp target select --url URL +``` + +## Mental model + +- After `target select`, the daemon starts an initial active network session automatically. +- That initial session appears in `network sessions` and can be stopped with `network stop`. +- `network start` creates a fresh empty session and does not backfill earlier requests. +- Only one active network session exists at a time. +- Without `--session`, queries prefer the active session, then the latest stored session, then the live rolling buffer if no session exists. +- The live buffer keeps only the most recent `200` requests. + +## Commands + +```bash +agent-cdp network status + +agent-cdp network start [--name NAME] [--preserve-across-navigation] +agent-cdp network stop +agent-cdp network sessions [--limit N] [--offset N] + +agent-cdp network summary [--session ID] +agent-cdp network list [--session ID] [--limit N] [--offset N] [--type TYPE] [--status STATUS] [--method METHOD] [--text TEXT] [--min-ms N] [--max-ms N] [--min-bytes N] [--max-bytes N] +agent-cdp network request --id REQ_ID [--session ID] +agent-cdp network request-headers --id REQ_ID [--session ID] [--name TEXT] +agent-cdp network response-headers --id REQ_ID [--session ID] [--name TEXT] +agent-cdp network request-body --id REQ_ID [--session ID] [--file PATH] +agent-cdp network response-body --id REQ_ID [--session ID] [--file PATH] +``` + +## Workflow: Quick Triage + +Use this when you want a fast overview of an interaction. + +```bash +agent-cdp network summary +agent-cdp network list --status failed +agent-cdp network list --min-ms 1000 +``` + +What to look for: +- failed requests +- slow requests +- unusually large responses +- request type concentration such as repeated `xhr` or `fetch` + +## Workflow: Capture A Repro In A Fresh Session + +Use this when you want a clean session for a specific bug reproduction. + +```bash +agent-cdp network stop +agent-cdp network start --name login-repro --preserve-across-navigation +# reproduce the issue in the app +agent-cdp network stop +agent-cdp network sessions +agent-cdp network summary --session net_2 +``` + +Notes: +- Stop the current session first if one is already active. +- Use `--preserve-across-navigation` when the repro crosses page loads or full-document navigations. + +## Workflow: Investigate A Failed Request + +```bash +agent-cdp network list --status failed +agent-cdp network request --id req_12 +agent-cdp network request-headers --id req_12 +agent-cdp network response-headers --id req_12 +agent-cdp network response-body --id req_12 +``` + +Typical checks: +- request URL and method are correct +- auth, cookie, and content-type headers are present +- response status and headers match expectations +- response body contains server-side error details + +## Workflow: Inspect JSON APIs + +```bash +agent-cdp network list --type fetch --text /api/ +agent-cdp network request --id req_12 +agent-cdp network request-body --id req_12 +agent-cdp network response-body --id req_12 +``` + +Body handling behavior: +- Text-like content types are decoded to text when CDP returns them as base64. +- Binary content types remain base64 in terminal output. +- JSON is currently shown as raw text, not pretty-printed. + +## Workflow: Export Binary Or Large Bodies + +Use file export for images, downloads, large payloads, or anything you do not want inline in the terminal. + +```bash +agent-cdp network response-body --id req_12 --file ./response.bin +agent-cdp network request-body --id req_12 --file ./request.bin +``` + +Use `--file` when: +- the content type is binary +- the body is large +- you want exact bytes instead of terminal rendering + +## Workflow: Search By Endpoint, Method, Or Payload Size + +```bash +agent-cdp network list --text checkout +agent-cdp network list --method POST +agent-cdp network list --status 5xx +agent-cdp network list --min-bytes 1000000 +agent-cdp network list --min-ms 500 --max-ms 5000 +``` + +This is useful for narrowing the session before drilling into a specific request id. + +## Body Caveats + +- `network request` intentionally omits headers and bodies; use the explicit follow-up commands. +- Request bodies depend on target support for `Network.getRequestPostData`. +- Multipart form-data is not parsed into fields. CDP returns multipart request body text without files when available. +- Response and request bodies may be unavailable after disconnects or on targets with partial CDP support. +- Binary response bodies are best exported with `--file`. + +## Target Compatibility + +- CDP `Network.*` support varies by runtime and target. +- There is no runtime-specific fallback instrumentation in v1. +- WebSocket support is handshake-only in v1. +- No HAR export, request blocking, throttling, mocking, replay, or redaction is included in v1. + +## Suggested Agent Loop + +When debugging a network issue, prefer this order: + +```bash +agent-cdp network summary +agent-cdp network list --status failed +agent-cdp network list --min-ms 1000 +agent-cdp network request --id REQ_ID +agent-cdp network request-headers --id REQ_ID +agent-cdp network response-headers --id REQ_ID +agent-cdp network response-body --id REQ_ID +``` + +If the issue is noisy or mixed with unrelated traffic, start a fresh named session and reproduce again. diff --git a/packages/agent-cdp/src/__tests__/cli.test.ts b/packages/agent-cdp/src/__tests__/cli.test.ts index 8accb18..02b0fc5 100644 --- a/packages/agent-cdp/src/__tests__/cli.test.ts +++ b/packages/agent-cdp/src/__tests__/cli.test.ts @@ -21,6 +21,8 @@ describe("cli", () => { expect(usage()).toContain("stop"); expect(usage()).toContain("target list [--url URL]"); expect(usage()).toContain("target select [--url URL]"); + expect(usage()).toContain("network start [--name NAME] [--preserve-across-navigation]"); + expect(usage()).toContain("network response-body --id REQ_ID [--session ID] [--file PATH]"); expect(usage()).toContain("js-allocation start"); expect(usage()).toContain("js-allocation-timeline start"); }); diff --git a/packages/agent-cdp/src/__tests__/network.test.ts b/packages/agent-cdp/src/__tests__/network.test.ts new file mode 100644 index 0000000..1dda212 --- /dev/null +++ b/packages/agent-cdp/src/__tests__/network.test.ts @@ -0,0 +1,480 @@ +import { mkdtempSync, readFileSync, rmSync } from "node:fs"; +import { tmpdir } from "node:os"; +import path from "node:path"; + +import { NetworkCapture } from "../network/capture.js"; +import { + formatNetworkBody, + formatNetworkHeaders, + formatNetworkList, + formatNetworkRequest, + formatNetworkSummary, +} from "../network/formatters.js"; +import { NetworkManager } from "../network/index.js"; +import { queryHeaders, queryList, querySummary } from "../network/query.js"; +import { NetworkStore } from "../network/store.js"; +import type { NetworkRequest } from "../network/types.js"; +import type { CdpEventMessage, CdpTransport, RuntimeSession, TargetDescriptor } from "../types.js"; + +class FakeNetworkTransport implements CdpTransport { + private listener: ((message: CdpEventMessage) => void) | null = null; + readonly sentMethods: string[] = []; + connected = true; + responseBodyResult: { body: string; base64Encoded?: boolean } = { body: "response-body", base64Encoded: false }; + requestPostDataResult: { postData: string; base64Encoded?: boolean } = { postData: "request:" }; + + connect(): Promise { + this.connected = true; + return Promise.resolve(); + } + + disconnect(): Promise { + this.connected = false; + return Promise.resolve(); + } + + isConnected(): boolean { + return this.connected; + } + + send(method: string, params?: Record): Promise { + this.sentMethods.push(method); + if (method === "Network.getResponseBody") { + return Promise.resolve(this.responseBodyResult); + } + if (method === "Network.getRequestPostData") { + return Promise.resolve({ + postData: `${this.requestPostDataResult.postData}${String(params?.requestId || "")}`, + base64Encoded: this.requestPostDataResult.base64Encoded, + }); + } + return Promise.resolve(undefined); + } + + onEvent(listener: (message: CdpEventMessage) => void): () => void { + this.listener = listener; + return () => { + this.listener = null; + }; + } + + emit(message: CdpEventMessage): void { + this.listener?.(message); + } +} + +function createSession(transport: CdpTransport): RuntimeSession { + return { + target: { + id: "chrome:test:page-1", + rawId: "page-1", + title: "Example", + kind: "chrome", + description: "Test page", + webSocketDebuggerUrl: "ws://example.test/devtools/page/1", + sourceUrl: "http://example.test", + } satisfies TargetDescriptor, + transport, + ensureConnected: () => Promise.resolve(), + close: () => Promise.resolve(), + }; +} + +function createRequest(id: string, overrides: Partial = {}): NetworkRequest { + return { + id, + rawRequestId: `raw-${id}`, + source: "live", + url: `https://example.test/${id}`, + method: "GET", + resourceType: "xhr", + state: "completed", + statusCode: 200, + startedAt: 1_000, + endedAt: 1_250, + durationMs: 250, + encodedDataLength: 2_048, + hasRequestHeaders: true, + hasResponseHeaders: true, + hasRequestBody: false, + hasResponseBody: true, + requestHeaders: { accept: "application/json" }, + responseHeaders: { "content-type": "application/json" }, + responseBody: { text: "{}", base64Encoded: false }, + redirectChain: [], + isNavigationRequest: false, + isWebSocket: false, + ...overrides, + }; +} + +describe("network capture", () => { + it("normalizes request lifecycles, redirects, failures, and websocket handshakes", async () => { + const transport = new FakeNetworkTransport(); + const store = new NetworkStore(); + const capture = new NetworkCapture( + () => store.generateRequestId(), + (request, isNew) => store.record(request, isNew), + () => store.handleNavigation(), + ); + + await capture.attach(createSession(transport)); + + expect(transport.sentMethods).toEqual(["Network.enable", "Page.enable"]); + + transport.emit({ + method: "Network.requestWillBeSent", + params: { + requestId: "1", + timestamp: 1, + wallTime: 10, + type: "XHR", + documentURL: "https://example.test/api", + request: { + url: "https://example.test/api", + method: "POST", + headers: { accept: "application/json" }, + postData: '{"a":1}', + }, + }, + }); + transport.emit({ + method: "Network.responseReceived", + params: { + requestId: "1", + type: "XHR", + response: { + status: 200, + statusText: "OK", + mimeType: "application/json", + protocol: "h2", + remoteIPAddress: "127.0.0.1", + remotePort: 443, + headers: { "content-type": "application/json" }, + }, + }, + }); + transport.emit({ method: "Network.loadingFinished", params: { requestId: "1", timestamp: 1.2, encodedDataLength: 2048 } }); + + transport.emit({ + method: "Network.requestWillBeSent", + params: { + requestId: "2", + timestamp: 2, + wallTime: 20, + type: "Document", + request: { + url: "https://example.test/redirect-a", + method: "GET", + }, + }, + }); + transport.emit({ + method: "Network.requestWillBeSent", + params: { + requestId: "2", + timestamp: 2.1, + wallTime: 20.1, + type: "Document", + request: { + url: "https://example.test/redirect-b", + method: "GET", + }, + redirectResponse: { + status: 302, + statusText: "Found", + headers: { location: "https://example.test/redirect-b" }, + }, + }, + }); + + transport.emit({ + method: "Network.loadingFailed", + params: { requestId: "3", timestamp: 3.5, errorText: "net::ERR_FAILED" }, + }); + + transport.emit({ method: "Network.webSocketCreated", params: { requestId: "4", url: "wss://example.test/socket" } }); + transport.emit({ + method: "Network.webSocketWillSendHandshakeRequest", + params: { + requestId: "4", + request: { headers: { upgrade: "websocket" } }, + }, + }); + transport.emit({ + method: "Network.webSocketHandshakeResponseReceived", + params: { + requestId: "4", + response: { status: 101, statusText: "Switching Protocols", headers: { connection: "Upgrade" } }, + }, + }); + + const requests = store.getLiveRequests(); + expect(requests).toHaveLength(5); + expect(requests.find((request) => request.url.endsWith("/api"))).toMatchObject({ + method: "POST", + state: "completed", + statusCode: 200, + hasRequestBody: true, + hasResponseBody: true, + remoteAddress: "127.0.0.1:443", + }); + expect(requests.find((request) => request.url.endsWith("redirect-a"))?.redirectedTo).toBe( + "https://example.test/redirect-b", + ); + expect(requests.find((request) => request.url.endsWith("redirect-b"))?.redirectChain).toHaveLength(1); + expect(requests.find((request) => request.rawRequestId === "3")?.state).toBe("failed"); + expect(requests.find((request) => request.rawRequestId === "4")?.webSocketHandshake?.statusCode).toBe(101); + }); +}); + +describe("network store", () => { + it("evicts old live requests and resets non-preserved sessions on navigation", () => { + const store = new NetworkStore(); + store.startSession("default", false); + + for (let index = 1; index <= 201; index += 1) { + store.record(createRequest(`req_${index}`), true); + } + + expect(store.getLiveRequests()).toHaveLength(200); + expect(store.getLiveRequests().some((request) => request.id === "req_1")).toBe(false); + expect(store.getActiveSession()?.requests).toHaveLength(201); + + store.handleNavigation(); + expect(store.getActiveSession()?.requests).toHaveLength(0); + + store.stopSession(); + store.startSession("preserved", true); + store.record(createRequest("req_nav"), true); + store.handleNavigation(); + expect(store.getActiveSession()?.requests).toHaveLength(1); + }); +}); + +describe("network queries and formatting", () => { + it("filters, summarizes, and formats compact output", () => { + const requests = [ + createRequest("req_1", { url: "https://example.test/api/users", durationMs: 1200, encodedDataLength: 3_000_000 }), + createRequest("req_2", { state: "failed", statusCode: 500, failureText: "boom", encodedDataLength: 512 }), + createRequest("req_3", { state: "pending", statusCode: undefined, encodedDataLength: undefined, hasResponseBody: false }), + ]; + + const summary = querySummary(requests, "session", "net_1"); + expect(summary.failedCount).toBe(1); + expect(formatNetworkSummary(summary, true)).toContain("Signals:"); + + const list = queryList(requests, "session", { sessionId: "net_1", status: "failed", limit: 10, offset: 0 }); + expect(list.items).toHaveLength(1); + expect(formatNetworkList(list)).toContain("req_2"); + + const headers = queryHeaders(requests[0], "response", "content"); + expect(headers.entries[0]?.name).toBe("content-type"); + expect(formatNetworkHeaders(headers)).toContain("content-type"); + + expect(formatNetworkRequest(requests[0], true)).toContain("Transfer size"); + }); +}); + +describe("network manager bodies", () => { + it("creates an initial session on first attach and lets users stop it", async () => { + const transport = new FakeNetworkTransport(); + const manager = new NetworkManager(); + + await manager.attach(createSession(transport)); + + expect(manager.getStatus().activeSession?.id).toBe("net_1"); + expect(manager.listSessions()).toEqual([ + expect.objectContaining({ id: "net_1", active: true, preserveAcrossNavigation: false, requestCount: 0 }), + ]); + + transport.emit({ + method: "Network.requestWillBeSent", + params: { + requestId: "startup-1", + timestamp: 1, + wallTime: 10, + type: "Fetch", + request: { url: "https://example.test/startup", method: "GET" }, + }, + }); + transport.emit({ + method: "Network.responseReceived", + params: { + requestId: "startup-1", + type: "Fetch", + response: { status: 200, statusText: "OK", mimeType: "text/plain" }, + }, + }); + transport.emit({ method: "Network.loadingFinished", params: { requestId: "startup-1", timestamp: 1.1, encodedDataLength: 128 } }); + + expect(manager.getSummary()).toMatchObject({ source: "session", sessionId: "net_1", requestCount: 1 }); + await expect(manager.stop()).resolves.toBe("net_1"); + expect(manager.getStatus().activeSession).toBeNull(); + expect(manager.listSessions()).toEqual([ + expect.objectContaining({ id: "net_1", active: false, preserveAcrossNavigation: false, requestCount: 1 }), + ]); + + await manager.attach(createSession(transport)); + expect(manager.getStatus().activeSession).toBeNull(); + expect(manager.listSessions()).toHaveLength(1); + }); + + it("uses the active session by default and exports response bodies", async () => { + const transport = new FakeNetworkTransport(); + const manager = new NetworkManager(); + const tempDir = mkdtempSync(path.join(tmpdir(), "agent-cdp-network-")); + const filePath = path.join(tempDir, "body.txt"); + + try { + await manager.attach(createSession(transport)); + await manager.stop(); + const sessionId = manager.start("capture", false); + + transport.emit({ + method: "Network.requestWillBeSent", + params: { + requestId: "body-1", + timestamp: 1, + wallTime: 10, + type: "Fetch", + request: { url: "https://example.test/body", method: "GET" }, + }, + }); + transport.emit({ + method: "Network.responseReceived", + params: { + requestId: "body-1", + type: "Fetch", + response: { status: 200, statusText: "OK", mimeType: "text/plain" }, + }, + }); + transport.emit({ method: "Network.loadingFinished", params: { requestId: "body-1", timestamp: 1.1, encodedDataLength: 128 } }); + + const summary = manager.getSummary(); + expect(summary.source).toBe("session"); + expect(summary.sessionId).toBe(sessionId); + + const requestId = manager.list({ sessionId }).items[0]?.id; + expect(requestId).toBeDefined(); + + const body = await manager.getResponseBody(requestId || "", sessionId, filePath); + expect(body.filePath).toBe(filePath); + expect(readFileSync(filePath, "utf8")).toBe("response-body"); + expect(formatNetworkBody(body)).toContain("Saved response body"); + } finally { + rmSync(tempDir, { recursive: true, force: true }); + } + }); + + it("decodes base64 text bodies for text-like content types and keeps binary bodies encoded", async () => { + const transport = new FakeNetworkTransport(); + const manager = new NetworkManager(); + + await manager.attach(createSession(transport)); + + transport.emit({ + method: "Network.requestWillBeSent", + params: { + requestId: "body-text", + timestamp: 1, + wallTime: 10, + type: "Fetch", + request: { url: "https://example.test/text", method: "GET" }, + }, + }); + transport.emit({ + method: "Network.responseReceived", + params: { + requestId: "body-text", + type: "Fetch", + response: { + status: 200, + statusText: "OK", + mimeType: "application/json", + headers: { "content-type": "application/json; charset=utf-8" }, + }, + }, + }); + transport.emit({ method: "Network.loadingFinished", params: { requestId: "body-text", timestamp: 1.1, encodedDataLength: 16 } }); + + transport.emit({ + method: "Network.requestWillBeSent", + params: { + requestId: "body-binary", + timestamp: 2, + wallTime: 20, + type: "Image", + request: { url: "https://example.test/image.png", method: "GET" }, + }, + }); + transport.emit({ + method: "Network.responseReceived", + params: { + requestId: "body-binary", + type: "Image", + response: { + status: 200, + statusText: "OK", + mimeType: "image/png", + headers: { "content-type": "image/png" }, + }, + }, + }); + transport.emit({ method: "Network.loadingFinished", params: { requestId: "body-binary", timestamp: 2.1, encodedDataLength: 24 } }); + + const textRequestId = manager.list({ text: "/text" }).items[0]?.id; + expect(textRequestId).toBeDefined(); + transport.responseBodyResult = { + body: Buffer.from('{"ok":true}', "utf8").toString("base64"), + base64Encoded: true, + }; + const textBody = await manager.getResponseBody(textRequestId || ""); + expect(textBody).toMatchObject({ available: true, base64Encoded: false, text: '{"ok":true}' }); + expect(formatNetworkBody(textBody)).toBe('{"ok":true}'); + + const binaryRequestId = manager.list({ text: "image.png" }).items[0]?.id; + expect(binaryRequestId).toBeDefined(); + transport.responseBodyResult = { + body: "iVBORw0KGgoAAAANSUhEUg==", + base64Encoded: true, + }; + const binaryBody = await manager.getResponseBody(binaryRequestId || ""); + expect(binaryBody).toMatchObject({ available: true, base64Encoded: true, text: "iVBORw0KGgoAAAANSUhEUg==" }); + expect(formatNetworkBody(binaryBody)).toContain("Base64 response body"); + }); + + it("decodes base64 request bodies for text-like content types", async () => { + const transport = new FakeNetworkTransport(); + const manager = new NetworkManager(); + + await manager.attach(createSession(transport)); + + transport.emit({ + method: "Network.requestWillBeSent", + params: { + requestId: "body-request", + timestamp: 1, + wallTime: 10, + type: "Fetch", + request: { + url: "https://example.test/form", + method: "POST", + headers: { "content-type": "application/json" }, + }, + }, + }); + + const requestId = manager.list({ text: "/form" }).items[0]?.id; + expect(requestId).toBeDefined(); + + transport.requestPostDataResult = { + postData: Buffer.from('{"hello":"world"}', "utf8").toString("base64"), + base64Encoded: true, + }; + + const requestBody = await manager.getRequestBody(requestId || ""); + expect(requestBody).toMatchObject({ available: true, base64Encoded: false, text: '{"hello":"world"}' }); + expect(formatNetworkBody(requestBody)).toBe('{"hello":"world"}'); + }); +}); diff --git a/packages/agent-cdp/src/cli.ts b/packages/agent-cdp/src/cli.ts index 931ba80..00d66aa 100644 --- a/packages/agent-cdp/src/cli.ts +++ b/packages/agent-cdp/src/cli.ts @@ -10,6 +10,15 @@ import { formatTargetList, formatTraceSummary, } from "./formatters.js"; +import { + formatNetworkBody, + formatNetworkHeaders, + formatNetworkList, + formatNetworkRequest, + formatNetworkSessions, + formatNetworkStatus, + formatNetworkSummary, +} from "./network/formatters.js"; import { formatMemLeakCandidates, formatMemLeakTriplet, @@ -92,6 +101,19 @@ Console: console list [--limit N] console get +Network: + network status + network start [--name NAME] [--preserve-across-navigation] + network stop + network sessions [--limit N] [--offset N] + network summary [--session ID] + network list [--session ID] [--limit N] [--offset N] [--type TYPE] [--status STATUS] [--method METHOD] [--text TEXT] [--min-ms N] [--max-ms N] [--min-bytes N] [--max-bytes N] + network request --id REQ_ID [--session ID] + network request-headers --id REQ_ID [--session ID] [--name TEXT] + network response-headers --id REQ_ID [--session ID] [--name TEXT] + network request-body --id REQ_ID [--session ID] [--file PATH] + network response-body --id REQ_ID [--session ID] [--file PATH] + Trace: trace start trace stop [--file PATH] @@ -359,6 +381,142 @@ export async function main(): Promise { return; } + if (cmd === "network" && command[1] === "status") { + await ensureDaemon(); + const response = await sendCommand({ type: "network-status" }); + if (!response.ok) throw new Error(response.error || "Failed to get network status"); + console.log(formatNetworkStatus(response.data as Parameters[0], verbose)); + return; + } + + if (cmd === "network" && command[1] === "start") { + const name = typeof flags.name === "string" ? flags.name : undefined; + const preserveAcrossNavigation = flags["preserve-across-navigation"] === true; + await ensureDaemon(); + const response = await sendCommand({ type: "network-start", name, preserveAcrossNavigation }); + if (!response.ok) throw new Error(response.error || "Failed to start network session"); + console.log(`Network session started. Session ID: ${response.data as string}`); + return; + } + + if (cmd === "network" && command[1] === "stop") { + await ensureDaemon(); + const response = await sendCommand({ type: "network-stop" }); + if (!response.ok) throw new Error(response.error || "Failed to stop network session"); + console.log(`Network session stopped. Session ID: ${response.data as string}`); + return; + } + + if (cmd === "network" && command[1] === "sessions") { + const limit = typeof flags.limit === "string" ? Number.parseInt(flags.limit, 10) : undefined; + const offset = typeof flags.offset === "string" ? Number.parseInt(flags.offset, 10) : undefined; + await ensureDaemon(); + const response = await sendCommand({ type: "network-list-sessions", limit, offset }); + if (!response.ok) throw new Error(response.error || "Failed to list network sessions"); + console.log(formatNetworkSessions(response.data as Parameters[0], verbose)); + return; + } + + if (cmd === "network" && command[1] === "summary") { + const sessionId = typeof flags.session === "string" ? flags.session : undefined; + await ensureDaemon(); + const response = await sendCommand({ type: "network-summary", sessionId }); + if (!response.ok) throw new Error(response.error || "Failed to summarize network requests"); + console.log(formatNetworkSummary(response.data as Parameters[0], verbose)); + return; + } + + if (cmd === "network" && command[1] === "list") { + const sessionId = typeof flags.session === "string" ? flags.session : undefined; + const limit = typeof flags.limit === "string" ? Number.parseInt(flags.limit, 10) : undefined; + const offset = typeof flags.offset === "string" ? Number.parseInt(flags.offset, 10) : undefined; + const resourceType = typeof flags.type === "string" ? flags.type : undefined; + const status = typeof flags.status === "string" ? flags.status : undefined; + const method = typeof flags.method === "string" ? flags.method : undefined; + const text = typeof flags.text === "string" ? flags.text : undefined; + const minMs = typeof flags["min-ms"] === "string" ? Number.parseFloat(flags["min-ms"]) : undefined; + const maxMs = typeof flags["max-ms"] === "string" ? Number.parseFloat(flags["max-ms"]) : undefined; + const minBytes = typeof flags["min-bytes"] === "string" ? Number.parseInt(flags["min-bytes"], 10) : undefined; + const maxBytes = typeof flags["max-bytes"] === "string" ? Number.parseInt(flags["max-bytes"], 10) : undefined; + await ensureDaemon(); + const response = await sendCommand({ + type: "network-list", + sessionId, + limit, + offset, + resourceType, + status, + method, + text, + minMs, + maxMs, + minBytes, + maxBytes, + }); + if (!response.ok) throw new Error(response.error || "Failed to list network requests"); + console.log(formatNetworkList(response.data as Parameters[0])); + return; + } + + if (cmd === "network" && command[1] === "request") { + const requestId = typeof flags.id === "string" ? flags.id : undefined; + if (!requestId) throw new Error("Usage: agent-cdp network request --id REQ_ID [--session ID]"); + const sessionId = typeof flags.session === "string" ? flags.session : undefined; + await ensureDaemon(); + const response = await sendCommand({ type: "network-request", requestId, sessionId }); + if (!response.ok) throw new Error(response.error || "Failed to get network request"); + console.log(formatNetworkRequest(response.data as Parameters[0], verbose)); + return; + } + + if (cmd === "network" && command[1] === "request-headers") { + const requestId = typeof flags.id === "string" ? flags.id : undefined; + if (!requestId) throw new Error("Usage: agent-cdp network request-headers --id REQ_ID [--session ID] [--name TEXT]"); + const sessionId = typeof flags.session === "string" ? flags.session : undefined; + const name = typeof flags.name === "string" ? flags.name : undefined; + await ensureDaemon(); + const response = await sendCommand({ type: "network-request-headers", requestId, sessionId, name }); + if (!response.ok) throw new Error(response.error || "Failed to get request headers"); + console.log(formatNetworkHeaders(response.data as Parameters[0])); + return; + } + + if (cmd === "network" && command[1] === "response-headers") { + const requestId = typeof flags.id === "string" ? flags.id : undefined; + if (!requestId) throw new Error("Usage: agent-cdp network response-headers --id REQ_ID [--session ID] [--name TEXT]"); + const sessionId = typeof flags.session === "string" ? flags.session : undefined; + const name = typeof flags.name === "string" ? flags.name : undefined; + await ensureDaemon(); + const response = await sendCommand({ type: "network-response-headers", requestId, sessionId, name }); + if (!response.ok) throw new Error(response.error || "Failed to get response headers"); + console.log(formatNetworkHeaders(response.data as Parameters[0])); + return; + } + + if (cmd === "network" && command[1] === "request-body") { + const requestId = typeof flags.id === "string" ? flags.id : undefined; + if (!requestId) throw new Error("Usage: agent-cdp network request-body --id REQ_ID [--session ID] [--file PATH]"); + const sessionId = typeof flags.session === "string" ? flags.session : undefined; + const filePath = typeof flags.file === "string" ? flags.file : undefined; + await ensureDaemon(); + const response = await sendCommand({ type: "network-request-body", requestId, sessionId, filePath }); + if (!response.ok) throw new Error(response.error || "Failed to get request body"); + console.log(formatNetworkBody(response.data as Parameters[0])); + return; + } + + if (cmd === "network" && command[1] === "response-body") { + const requestId = typeof flags.id === "string" ? flags.id : undefined; + if (!requestId) throw new Error("Usage: agent-cdp network response-body --id REQ_ID [--session ID] [--file PATH]"); + const sessionId = typeof flags.session === "string" ? flags.session : undefined; + const filePath = typeof flags.file === "string" ? flags.file : undefined; + await ensureDaemon(); + const response = await sendCommand({ type: "network-response-body", requestId, sessionId, filePath }); + if (!response.ok) throw new Error(response.error || "Failed to get response body"); + console.log(formatNetworkBody(response.data as Parameters[0])); + return; + } + if (cmd === "trace" && command[1] === "start") { await ensureDaemon(); const response = await sendCommand({ type: "start-trace" }); diff --git a/packages/agent-cdp/src/daemon.ts b/packages/agent-cdp/src/daemon.ts index bfeee6a..67553fd 100644 --- a/packages/agent-cdp/src/daemon.ts +++ b/packages/agent-cdp/src/daemon.ts @@ -9,6 +9,7 @@ import { JsAllocationTimelineProfiler } from "./js-allocation-timeline/index.js" import { JsHeapUsageMonitor } from "./js-memory/index.js"; import { JsProfiler } from "./js-profiler/index.js"; import { MemorySnapshotter } from "./memory.js"; +import { NetworkManager } from "./network/index.js"; import { createTargetProviders } from "./providers.js"; import { SessionManager } from "./session-manager.js"; import { TraceRecorder } from "./trace.js"; @@ -43,6 +44,7 @@ class Daemon { private readonly startedAt = Date.now(); private readonly consoleCollector = new ConsoleCollector(); private readonly memorySnapshotter = new MemorySnapshotter(); + private readonly networkManager = new NetworkManager(); private readonly heapSnapshotManager = new HeapSnapshotManager(); private readonly jsAllocationProfiler = new JsAllocationProfiler(); private readonly jsAllocationTimelineProfiler = new JsAllocationTimelineProfiler(this.heapSnapshotManager); @@ -82,6 +84,7 @@ class Daemon { const shutdown = () => { void this.sessionManager.clearTarget().finally(() => { this.consoleCollector.detach(); + this.networkManager.detach(); this.stop(); process.exit(0); }); @@ -152,6 +155,7 @@ class Daemon { const session = this.sessionManager.getSession(); if (session) { await this.consoleCollector.attach(session); + await this.networkManager.attach(session); } return { ok: true, @@ -161,10 +165,84 @@ class Daemon { if (command.type === "clear-target") { this.consoleCollector.detach(); + this.networkManager.detach(); await this.sessionManager.clearTarget(); return { ok: true, data: "Target cleared" }; } + if (command.type === "network-status") { + return { ok: true, data: this.networkManager.getStatus() }; + } + + if (command.type === "network-start") { + const session = await this.requireConnectedSession(); + if (!this.networkManager.isAttached()) { + await this.networkManager.attach(session); + } + return { + ok: true, + data: this.networkManager.start(command.name, command.preserveAcrossNavigation === true), + }; + } + + if (command.type === "network-stop") { + return { ok: true, data: await this.networkManager.stop() }; + } + + if (command.type === "network-list-sessions") { + return { ok: true, data: this.networkManager.listSessions(command.limit, command.offset) }; + } + + if (command.type === "network-summary") { + await this.ensureNetworkSessionReady(); + return { ok: true, data: this.networkManager.getSummary(command.sessionId) }; + } + + if (command.type === "network-list") { + await this.ensureNetworkSessionReady(); + return { + ok: true, + data: this.networkManager.list({ + sessionId: command.sessionId, + limit: command.limit, + offset: command.offset, + type: command.resourceType, + status: command.status, + method: command.method, + text: command.text, + minMs: command.minMs, + maxMs: command.maxMs, + minBytes: command.minBytes, + maxBytes: command.maxBytes, + }), + }; + } + + if (command.type === "network-request") { + await this.ensureNetworkSessionReady(); + return { ok: true, data: this.networkManager.getRequest(command.requestId, command.sessionId) }; + } + + if (command.type === "network-request-headers") { + await this.ensureNetworkSessionReady(); + return { ok: true, data: this.networkManager.getRequestHeaders(command.requestId, command.sessionId, command.name) }; + } + + if (command.type === "network-response-headers") { + await this.ensureNetworkSessionReady(); + return { ok: true, data: this.networkManager.getResponseHeaders(command.requestId, command.sessionId, command.name) }; + } + + if (command.type === "network-request-body") { + await this.ensureNetworkSessionReady(); + return { ok: true, data: await this.networkManager.getRequestBody(command.requestId, command.sessionId, command.filePath) }; + } + + if (command.type === "network-response-body") { + await this.ensureNetworkSessionReady(); + return { ok: true, data: await this.networkManager.getResponseBody(command.requestId, command.sessionId, command.filePath) }; + } + if (command.type === "list-console-messages") { await this.ensureConsoleSessionReady(); return { ok: true, data: this.consoleCollector.list(command.limit) }; @@ -557,6 +635,16 @@ class Daemon { await this.consoleCollector.attach(session); } + private async ensureNetworkSessionReady(): Promise { + const currentSession = this.sessionManager.getSession(); + const wasConnected = currentSession?.transport.isConnected() || false; + const session = await this.requireConnectedSession(); + if (wasConnected && this.networkManager.isAttached()) { + return; + } + await this.networkManager.attach(session); + } + private async requireConnectedSession() { const selectedTarget = this.sessionManager.getSelectedTarget(); await this.sessionManager.reconnectSelectedTarget(); diff --git a/packages/agent-cdp/src/network/capture.ts b/packages/agent-cdp/src/network/capture.ts new file mode 100644 index 0000000..bb4b69f --- /dev/null +++ b/packages/agent-cdp/src/network/capture.ts @@ -0,0 +1,376 @@ +import type { CdpEventMessage, RuntimeSession } from "../types.js"; +import type { NetworkRequest } from "./types.js"; + +interface NetworkResponsePayload { + url?: string; + status?: number; + statusText?: string; + mimeType?: string; + protocol?: string; + remoteIPAddress?: string; + remotePort?: number; + fromDiskCache?: boolean; + fromServiceWorker?: boolean; + headers?: Record; +} + +interface TrackedRequest { + request: NetworkRequest; + wallTimeMs?: number; +} + +export class NetworkCapture { + private requests = new Map(); + private unsubscribe: (() => void) | null = null; + private session: RuntimeSession | null = null; + private attached = false; + + constructor( + private readonly createRequestId: () => string, + private readonly onRequestUpdated: (request: NetworkRequest, isNew: boolean) => void, + private readonly onNavigation: () => void, + ) {} + + async attach(session: RuntimeSession): Promise { + this.detach(); + this.session = session; + this.requests = new Map(); + this.unsubscribe = session.transport.onEvent((message) => { + this.handleEvent(message); + }); + + await Promise.allSettled([session.transport.send("Network.enable"), session.transport.send("Page.enable")]); + this.attached = true; + } + + detach(): void { + this.unsubscribe?.(); + this.unsubscribe = null; + this.session = null; + this.requests = new Map(); + this.attached = false; + } + + isAttached(): boolean { + return this.attached; + } + + async getRequestBody(request: NetworkRequest): Promise<{ text: string; base64Encoded: boolean }> { + if (request.requestBody) { + return request.requestBody; + } + if (!this.session) { + throw new Error("Request body is unavailable after the target disconnects"); + } + + const result = (await this.session.transport.send("Network.getRequestPostData", { + requestId: request.rawRequestId, + })) as { postData?: string; base64Encoded?: boolean }; + if (typeof result.postData !== "string") { + throw new Error("Request body unavailable"); + } + request.requestBody = { text: result.postData, base64Encoded: result.base64Encoded === true }; + request.hasRequestBody = true; + return request.requestBody; + } + + async getResponseBody(request: NetworkRequest): Promise<{ text: string; base64Encoded: boolean }> { + if (request.responseBody) { + return request.responseBody; + } + if (!this.session) { + throw new Error("Response body is unavailable after the target disconnects"); + } + + const result = (await this.session.transport.send("Network.getResponseBody", { + requestId: request.rawRequestId, + })) as { body?: string; base64Encoded?: boolean }; + if (typeof result.body !== "string") { + throw new Error("Response body unavailable"); + } + request.responseBody = { text: result.body, base64Encoded: result.base64Encoded === true }; + request.hasResponseBody = true; + return request.responseBody; + } + + private handleEvent(message: CdpEventMessage): void { + if (message.method === "Page.frameNavigated") { + const frame = message.params?.frame as { parentId?: string } | undefined; + if (frame && !frame.parentId) { + this.onNavigation(); + } + return; + } + + if (message.method === "Network.requestWillBeSent") { + this.handleRequestWillBeSent(message.params as Record); + return; + } + + if (message.method === "Network.requestWillBeSentExtraInfo") { + this.handleRequestExtraInfo(message.params as Record); + return; + } + + if (message.method === "Network.responseReceived") { + this.handleResponseReceived(message.params as Record); + return; + } + + if (message.method === "Network.responseReceivedExtraInfo") { + this.handleResponseExtraInfo(message.params as Record); + return; + } + + if (message.method === "Network.loadingFinished") { + this.handleLoadingFinished(message.params as Record); + return; + } + + if (message.method === "Network.loadingFailed") { + this.handleLoadingFailed(message.params as Record); + return; + } + + if (message.method === "Network.webSocketCreated") { + this.handleWebSocketCreated(message.params as Record); + return; + } + + if (message.method === "Network.webSocketWillSendHandshakeRequest") { + this.handleWebSocketHandshakeRequest(message.params as Record); + return; + } + + if (message.method === "Network.webSocketHandshakeResponseReceived") { + this.handleWebSocketHandshakeResponse(message.params as Record); + } + } + + private handleRequestWillBeSent(params: Record): void { + const rawRequestId = String(params.requestId || ""); + const requestPayload = (params.request || {}) as Record; + const url = typeof requestPayload.url === "string" ? requestPayload.url : "unknown"; + const method = typeof requestPayload.method === "string" ? requestPayload.method : "UNKNOWN"; + const timestampMs = toMonotonicMs(params.timestamp); + const wallTimeMs = toWallTimeMs(params.wallTime) ?? Date.now(); + const resourceType = typeof params.type === "string" ? params.type : "other"; + const redirectResponse = params.redirectResponse as NetworkResponsePayload | undefined; + const existing = this.requests.get(rawRequestId); + + let redirectChain = existing?.request.redirectChain ? [...existing.request.redirectChain] : []; + if (redirectResponse && existing) { + existing.request.state = "completed"; + existing.request.statusCode = redirectResponse.status; + existing.request.statusText = redirectResponse.statusText; + existing.request.mimeType = redirectResponse.mimeType; + existing.request.protocol = redirectResponse.protocol; + existing.request.responseHeaders = normalizeHeaders(redirectResponse.headers); + existing.request.hasResponseHeaders = Boolean(existing.request.responseHeaders); + existing.request.redirectedTo = url; + existing.request.endedAt = wallTimeMs; + existing.request.durationMs = Math.max(0, wallTimeMs - existing.request.startedAt); + redirectChain = [...redirectChain, { url: existing.request.url, statusCode: redirectResponse.status, statusText: redirectResponse.statusText }]; + } + + const tracked = existing && !redirectResponse ? existing : this.createTrackedRequest(rawRequestId); + tracked.request.url = url; + tracked.request.method = method; + tracked.request.resourceType = resourceType; + tracked.request.startedAt = wallTimeMs; + tracked.request.state = "pending"; + tracked.request.redirectChain = redirectChain; + tracked.request.navigationId = typeof params.loaderId === "string" ? params.loaderId : tracked.request.navigationId; + tracked.request.isNavigationRequest = params.documentURL === url; + tracked.wallTimeMs = wallTimeMs - timestampMs; + + const requestHeaders = normalizeHeaders(requestPayload.headers as Record | undefined); + if (requestHeaders) { + tracked.request.requestHeaders = requestHeaders; + tracked.request.hasRequestHeaders = true; + } + if (typeof requestPayload.postData === "string") { + tracked.request.requestBody = { text: requestPayload.postData, base64Encoded: false }; + tracked.request.hasRequestBody = true; + } + + this.onRequestUpdated(tracked.request, !existing || Boolean(redirectResponse)); + } + + private handleRequestExtraInfo(params: Record): void { + const tracked = this.getOrCreateTrackedRequest(String(params.requestId || "")); + const headers = normalizeHeaders(params.headers as Record | undefined); + if (headers) { + tracked.request.requestHeaders = headers; + tracked.request.hasRequestHeaders = true; + this.onRequestUpdated(tracked.request, false); + } + } + + private handleResponseReceived(params: Record): void { + const tracked = this.getOrCreateTrackedRequest(String(params.requestId || "")); + const response = (params.response || {}) as NetworkResponsePayload; + tracked.request.statusCode = response.status; + tracked.request.statusText = response.statusText; + tracked.request.resourceType = typeof params.type === "string" ? params.type : tracked.request.resourceType; + tracked.request.mimeType = response.mimeType; + tracked.request.protocol = response.protocol; + tracked.request.remoteAddress = formatRemoteAddress(response.remoteIPAddress, response.remotePort); + tracked.request.fromDiskCache = response.fromDiskCache; + tracked.request.fromServiceWorker = response.fromServiceWorker; + const headers = normalizeHeaders(response.headers); + if (headers) { + tracked.request.responseHeaders = headers; + tracked.request.hasResponseHeaders = true; + } + this.onRequestUpdated(tracked.request, false); + } + + private handleResponseExtraInfo(params: Record): void { + const tracked = this.getOrCreateTrackedRequest(String(params.requestId || "")); + if (typeof params.statusCode === "number") { + tracked.request.statusCode = params.statusCode; + } + const headers = normalizeHeaders(params.headers as Record | undefined); + if (headers) { + tracked.request.responseHeaders = headers; + tracked.request.hasResponseHeaders = true; + } + this.onRequestUpdated(tracked.request, false); + } + + private handleLoadingFinished(params: Record): void { + const tracked = this.getOrCreateTrackedRequest(String(params.requestId || "")); + const endedAt = estimateWallTime(tracked, params.timestamp) ?? Date.now(); + tracked.request.state = "completed"; + tracked.request.endedAt = endedAt; + tracked.request.durationMs = Math.max(0, endedAt - tracked.request.startedAt); + tracked.request.encodedDataLength = typeof params.encodedDataLength === "number" ? params.encodedDataLength : tracked.request.encodedDataLength; + tracked.request.hasResponseBody = true; + this.onRequestUpdated(tracked.request, false); + } + + private handleLoadingFailed(params: Record): void { + const tracked = this.getOrCreateTrackedRequest(String(params.requestId || "")); + const endedAt = estimateWallTime(tracked, params.timestamp) ?? Date.now(); + tracked.request.state = "failed"; + tracked.request.failureText = typeof params.errorText === "string" ? params.errorText : "Unknown network failure"; + tracked.request.endedAt = endedAt; + tracked.request.durationMs = Math.max(0, endedAt - tracked.request.startedAt); + this.onRequestUpdated(tracked.request, false); + } + + private handleWebSocketCreated(params: Record): void { + const tracked = this.getOrCreateTrackedRequest(String(params.requestId || "")); + tracked.request.url = typeof params.url === "string" ? params.url : tracked.request.url; + tracked.request.method = "GET"; + tracked.request.resourceType = "websocket"; + tracked.request.isWebSocket = true; + this.onRequestUpdated(tracked.request, false); + } + + private handleWebSocketHandshakeRequest(params: Record): void { + const tracked = this.getOrCreateTrackedRequest(String(params.requestId || "")); + const request = (params.request || {}) as Record; + tracked.request.isWebSocket = true; + tracked.request.method = "GET"; + tracked.request.resourceType = "websocket"; + const headers = normalizeHeaders(request.headers as Record | undefined); + if (headers) { + tracked.request.requestHeaders = headers; + tracked.request.hasRequestHeaders = true; + } + this.onRequestUpdated(tracked.request, false); + } + + private handleWebSocketHandshakeResponse(params: Record): void { + const tracked = this.getOrCreateTrackedRequest(String(params.requestId || "")); + const response = (params.response || {}) as Record; + tracked.request.isWebSocket = true; + tracked.request.state = "completed"; + tracked.request.statusCode = typeof response.status === "number" ? response.status : tracked.request.statusCode; + tracked.request.statusText = typeof response.statusText === "string" ? response.statusText : tracked.request.statusText; + tracked.request.webSocketHandshake = { + statusCode: tracked.request.statusCode, + statusText: tracked.request.statusText, + }; + const headers = normalizeHeaders(response.headers as Record | undefined); + if (headers) { + tracked.request.responseHeaders = headers; + tracked.request.hasResponseHeaders = true; + } + this.onRequestUpdated(tracked.request, false); + } + + private createTrackedRequest(rawRequestId: string): TrackedRequest { + const request: NetworkRequest = { + id: this.createRequestId(), + rawRequestId, + source: "live", + url: "unknown", + method: "UNKNOWN", + resourceType: "other", + state: "pending", + startedAt: Date.now(), + hasRequestHeaders: false, + hasResponseHeaders: false, + hasRequestBody: false, + hasResponseBody: false, + redirectChain: [], + isNavigationRequest: false, + isWebSocket: false, + }; + const tracked = { request }; + this.requests.set(rawRequestId, tracked); + return tracked; + } + + private getOrCreateTrackedRequest(rawRequestId: string): TrackedRequest { + const existing = this.requests.get(rawRequestId); + if (existing) { + return existing; + } + const tracked = this.createTrackedRequest(rawRequestId); + this.onRequestUpdated(tracked.request, true); + return tracked; + } +} + +function normalizeHeaders(headers?: Record): Record | undefined { + if (!headers) { + return undefined; + } + const out = Object.entries(headers).reduce>((acc, [name, value]) => { + if (value === undefined || value === null) { + return acc; + } + acc[name] = String(value); + return acc; + }, {}); + return Object.keys(out).length > 0 ? out : undefined; +} + +function toMonotonicMs(value: unknown): number { + return typeof value === "number" ? value * 1000 : 0; +} + +function toWallTimeMs(value: unknown): number | undefined { + return typeof value === "number" ? value * 1000 : undefined; +} + +function estimateWallTime(tracked: TrackedRequest, timestamp: unknown): number | undefined { + if (typeof timestamp !== "number") { + return undefined; + } + if (typeof tracked.wallTimeMs !== "number") { + return undefined; + } + return tracked.wallTimeMs + timestamp * 1000; +} + +function formatRemoteAddress(ip?: string, port?: number): string | undefined { + if (!ip) { + return undefined; + } + return typeof port === "number" ? `${ip}:${port}` : ip; +} diff --git a/packages/agent-cdp/src/network/formatters.ts b/packages/agent-cdp/src/network/formatters.ts new file mode 100644 index 0000000..b045a6f --- /dev/null +++ b/packages/agent-cdp/src/network/formatters.ts @@ -0,0 +1,192 @@ +import type { + NetworkBodyResult, + NetworkHeaderResult, + NetworkListResult, + NetworkRequest, + NetworkSessionListEntry, + NetworkStatusResult, + NetworkSummaryResult, +} from "./types.js"; + +export function formatNetworkStatus(result: NetworkStatusResult, verbose = false): string { + if (!verbose) { + const active = result.activeSession ? `session:${result.activeSession.id}` : "session:none"; + return `${result.attached ? "attached" : "detached"} | live:${result.liveRequestCount}/${result.liveBufferLimit} | ${active} | stored:${result.storedSessionCount}`; + } + + const lines = [ + `Capture: ${result.attached ? "attached" : "detached"}`, + `Live buffer: ${result.liveRequestCount}/${result.liveBufferLimit}`, + `Stored sessions: ${result.storedSessionCount}`, + ]; + if (result.activeSession) { + lines.push(`Active session: ${result.activeSession.id}${result.activeSession.name ? ` ${result.activeSession.name}` : ""}`); + lines.push(`Preserve across navigation: ${result.activeSession.preserveAcrossNavigation ? "yes" : "no"}`); + lines.push(`Captured requests: ${result.activeSession.requestCount}`); + } else { + lines.push("Active session: none"); + } + return lines.join("\n"); +} + +export function formatNetworkSessions(entries: NetworkSessionListEntry[], verbose = false): string { + if (entries.length === 0) { + return "No network sessions yet. Select a target to start capture."; + } + + if (!verbose) { + return entries + .map((entry) => `${entry.id}${entry.active ? "*" : ""} ${entry.requestCount} req ${entry.name || "unnamed"}`) + .join("\n"); + } + + return entries + .map((entry) => { + const lines = [ + `${entry.id}${entry.active ? " (active)" : ""}`, + `Name: ${entry.name || "unnamed"}`, + `Requests: ${entry.requestCount}`, + `Preserve across navigation: ${entry.preserveAcrossNavigation ? "yes" : "no"}`, + ]; + if (entry.stoppedAt) { + lines.push(`Duration: ${Math.max(0, Math.round((entry.stoppedAt - entry.startedAt) / 1000))}s`); + } + return lines.join("\n"); + }) + .join("\n\n"); +} + +export function formatNetworkSummary(result: NetworkSummaryResult, verbose = false): string { + if (result.requestCount === 0) { + return result.source === "session" + ? `Session ${result.sessionId || "unknown"} has no captured requests.` + : "No live network requests captured yet."; + } + + const lines = [ + `${result.source === "session" ? result.sessionId || "session" : "live buffer"} total:${result.requestCount} completed:${result.completedCount} failed:${result.failedCount} pending:${result.pendingCount}`, + `Types: ${renderCounts(result.countsByType, "type")}`, + `Status: ${renderCounts(result.countsByStatusBucket, "bucket")}`, + ]; + if (result.slowest.length > 0) { + lines.push(`Slowest: ${result.slowest.map((request) => `${request.id} ${Math.round(request.durationMs || 0)}ms`).join(", ")}`); + } + if (result.largest.length > 0) { + lines.push(`Largest: ${result.largest.map((request) => `${request.id} ${formatBytes(request.encodedDataLength || 0)}`).join(", ")}`); + } + lines.push( + `Available: req-headers:${result.availability.requestHeaders} res-headers:${result.availability.responseHeaders} req-bodies:${result.availability.requestBodies} res-bodies:${result.availability.responseBodies}`, + ); + if (verbose && result.evidence.length > 0) { + lines.push(`Signals: ${result.evidence.join("; ")}`); + } + return lines.join("\n"); +} + +export function formatNetworkList(result: NetworkListResult): string { + if (result.items.length === 0) { + return "No matching network requests"; + } + + return result.items.map(formatNetworkListRow).join("\n"); +} + +export function formatNetworkRequest(request: NetworkRequest, verbose = false): string { + const lines = [ + `${request.id} ${request.method} ${request.url}`, + `State: ${request.state}${request.statusCode ? ` ${request.statusCode}${request.statusText ? ` ${request.statusText}` : ""}` : ""}`, + `Type: ${request.resourceType}`, + ]; + if (typeof request.durationMs === "number") { + lines.push(`Duration: ${Math.round(request.durationMs)}ms`); + } + if (typeof request.encodedDataLength === "number") { + lines.push(`Transfer size: ${formatBytes(request.encodedDataLength)}`); + } + if (request.failureText) { + lines.push(`Failure: ${request.failureText}`); + } + if (request.redirectChain.length > 0 || request.redirectedTo) { + lines.push(`Redirects: ${[...request.redirectChain.map((hop) => hop.url), request.redirectedTo].filter(Boolean).join(" -> ")}`); + } + lines.push( + `Available: req-headers:${yesNo(request.hasRequestHeaders)} res-headers:${yesNo(request.hasResponseHeaders)} req-body:${yesNo(request.hasRequestBody)} res-body:${yesNo(request.hasResponseBody)}`, + ); + if (verbose) { + if (request.mimeType) lines.push(`MIME: ${request.mimeType}`); + if (request.protocol) lines.push(`Protocol: ${request.protocol}`); + if (request.remoteAddress) lines.push(`Remote: ${request.remoteAddress}`); + if (request.fromDiskCache) lines.push("Served from disk cache"); + if (request.fromServiceWorker) lines.push("Served from service worker"); + if (request.isWebSocket) lines.push("WebSocket visibility is handshake-only in v1"); + } + return lines.join("\n"); +} + +export function formatNetworkHeaders(result: NetworkHeaderResult): string { + if (!result.available) { + return `${capitalize(result.kind)} headers unavailable`; + } + if (result.entries.length === 0) { + return `No ${result.kind} headers matched`; + } + return result.entries.map((entry) => `${entry.name}: ${entry.value}`).join("\n"); +} + +export function formatNetworkBody(result: NetworkBodyResult): string { + if (!result.available) { + return `${capitalize(result.kind)} body unavailable${result.reason ? `: ${result.reason}` : ""}`; + } + if (result.filePath) { + return `Saved ${result.kind} body to ${result.filePath}${typeof result.bytesWritten === "number" ? ` (${result.bytesWritten} bytes)` : ""}`; + } + if (!result.text) { + return `${capitalize(result.kind)} body is empty`; + } + if (result.base64Encoded) { + return `Base64 ${result.kind} body (${result.text.length} chars)\n${result.text}`; + } + return result.text; +} + +function formatNetworkListRow(request: NetworkRequest): string { + return [ + request.id, + request.method, + renderStatus(request), + request.resourceType, + renderDuration(request.durationMs), + formatBytes(request.encodedDataLength || 0), + request.url, + ].join(" "); +} + +function renderCounts(items: Array & { count: number }>, field: TField): string { + return items.map((item) => `${item[field]}:${item.count}`).join(", "); +} + +function renderStatus(request: NetworkRequest): string { + if (request.state === "failed") return "failed"; + if (request.state === "pending") return "pending"; + return request.statusCode ? String(request.statusCode) : "done"; +} + +function renderDuration(durationMs: number | undefined): string { + if (typeof durationMs !== "number") return "-"; + return `${Math.round(durationMs)}ms`; +} + +function formatBytes(bytes: number): string { + if (bytes <= 0) return "-"; + if (bytes < 1024) return `${bytes}B`; + if (bytes < 1024 * 1024) return `${(bytes / 1024).toFixed(1)}KB`; + return `${(bytes / (1024 * 1024)).toFixed(1)}MB`; +} + +function yesNo(value: boolean): string { + return value ? "yes" : "no"; +} + +function capitalize(value: string): string { + return value.charAt(0).toUpperCase() + value.slice(1); +} diff --git a/packages/agent-cdp/src/network/index.ts b/packages/agent-cdp/src/network/index.ts new file mode 100644 index 0000000..bbe3907 --- /dev/null +++ b/packages/agent-cdp/src/network/index.ts @@ -0,0 +1,283 @@ +import fs from "node:fs/promises"; +import path from "node:path"; + +import type { RuntimeSession } from "../types.js"; +import { NetworkCapture } from "./capture.js"; +import { + queryHeaders, + queryList, + queryRequest, + querySummary, +} from "./query.js"; +import { NetworkStore } from "./store.js"; +import type { + NetworkBodyResult, + NetworkHeaderResult, + NetworkListOptions, + NetworkListResult, + NetworkRequest, + NetworkSessionListEntry, + NetworkStatusResult, + NetworkSummaryResult, +} from "./types.js"; + +export class NetworkManager { + private readonly store = new NetworkStore(); + private readonly capture = new NetworkCapture( + () => this.store.generateRequestId(), + (request, isNew) => this.store.record(request, isNew), + () => this.store.handleNavigation(), + ); + + async attach(session: RuntimeSession): Promise { + const shouldCreateInitialSession = !this.store.getLatestSession(); + if (shouldCreateInitialSession) { + this.store.startSession(); + } + + try { + await this.capture.attach(session); + } catch (error) { + if (shouldCreateInitialSession) { + this.store.discardActiveSession(); + } + throw error; + } + } + + detach(): void { + this.capture.detach(); + } + + isAttached(): boolean { + return this.capture.isAttached(); + } + + start(name?: string, preserveAcrossNavigation = false): string { + return this.store.startSession(name, preserveAcrossNavigation).id; + } + + async stop(): Promise { + const activeSession = this.store.getActiveSession(); + if (!activeSession) { + throw new Error("No active network session. Run network start first."); + } + + await Promise.allSettled( + activeSession.requests + .filter((request) => request.hasResponseBody && !request.responseBody && request.state === "completed") + .map((request) => this.capture.getResponseBody(request)), + ); + + return this.store.stopSession().id; + } + + getStatus(): NetworkStatusResult { + return this.store.getStatus(this.capture.isAttached()); + } + + listSessions(limit?: number, offset?: number): NetworkSessionListEntry[] { + return this.store.listSessions(limit, offset); + } + + getSummary(sessionId?: string): NetworkSummaryResult { + const resolved = this.resolveSource(sessionId); + return querySummary(resolved.requests, resolved.source, resolved.sessionId); + } + + list(options: NetworkListOptions): NetworkListResult { + const resolved = this.resolveSource(options.sessionId); + return queryList(resolved.requests, resolved.source, { ...options, sessionId: resolved.sessionId }); + } + + getRequest(requestId: string, sessionId?: string): NetworkRequest { + const resolved = this.resolveRequestScope(sessionId); + return queryRequest(resolved.session, resolved.liveRequests, requestId); + } + + getRequestHeaders(requestId: string, sessionId?: string, name?: string): NetworkHeaderResult { + return queryHeaders(this.getRequest(requestId, sessionId), "request", name); + } + + getResponseHeaders(requestId: string, sessionId?: string, name?: string): NetworkHeaderResult { + return queryHeaders(this.getRequest(requestId, sessionId), "response", name); + } + + async getRequestBody(requestId: string, sessionId?: string, filePath?: string): Promise { + const request = this.getRequest(requestId, sessionId); + try { + const body = await this.capture.getRequestBody(request); + if (!body) { + return unavailableBody("request", request, request.requestBodyUnavailable?.reason || "Target did not expose a request body"); + } + return await this.resolveBodyResult("request", request, body, filePath); + } catch (error) { + return unavailableBody("request", request, error instanceof Error ? error.message : String(error)); + } + } + + async getResponseBody(requestId: string, sessionId?: string, filePath?: string): Promise { + const request = this.getRequest(requestId, sessionId); + try { + const body = request.hasResponseBody ? await this.capture.getResponseBody(request) : undefined; + if (!body) { + return unavailableBody("response", request, request.responseBodyUnavailable?.reason || "Target did not expose a response body"); + } + return await this.resolveBodyResult("response", request, body, filePath); + } catch (error) { + return unavailableBody("response", request, error instanceof Error ? error.message : String(error)); + } + } + + private resolveSource(sessionId?: string): { source: "live" | "session"; sessionId?: string; requests: NetworkRequest[] } { + if (sessionId) { + const session = this.store.getSession(sessionId); + if (!session) { + throw new Error(`Network session ${sessionId} not found`); + } + return { source: "session", sessionId: session.id, requests: [...session.requests].reverse() }; + } + + const latestSession = this.store.getLatestSession(); + if (latestSession) { + return { source: "session", sessionId: latestSession.id, requests: [...latestSession.requests].reverse() }; + } + + return { source: "live", requests: this.store.getLiveRequests() }; + } + + private resolveRequestScope(sessionId?: string): { session: { requests: NetworkRequest[] } | null; liveRequests: NetworkRequest[] } { + if (sessionId) { + const session = this.store.getSession(sessionId); + if (!session) { + throw new Error(`Network session ${sessionId} not found`); + } + return { session, liveRequests: [] }; + } + + const latestSession = this.store.getLatestSession(); + if (latestSession) { + return { session: latestSession, liveRequests: [] }; + } + + return { session: null, liveRequests: this.store.getLiveRequests() }; + } + + private async resolveBodyResult( + kind: "request" | "response", + request: NetworkRequest, + body: { text: string; base64Encoded: boolean }, + filePath?: string, + ): Promise { + const normalizedBody = normalizeBodyContent(kind, request, body); + + if (!filePath) { + return { + requestId: request.id, + sessionId: request.sessionId, + kind, + available: true, + mimeType: request.mimeType, + base64Encoded: normalizedBody.base64Encoded, + text: normalizedBody.text, + }; + } + + const outputPath = path.resolve(filePath); + const buffer = body.base64Encoded ? Buffer.from(body.text, "base64") : Buffer.from(body.text, "utf8"); + await fs.writeFile(outputPath, buffer); + return { + requestId: request.id, + sessionId: request.sessionId, + kind, + available: true, + mimeType: request.mimeType, + base64Encoded: normalizedBody.base64Encoded, + filePath: outputPath, + bytesWritten: buffer.byteLength, + }; + } +} + +function unavailableBody(kind: "request" | "response", request: NetworkRequest, reason: string): NetworkBodyResult { + return { + requestId: request.id, + sessionId: request.sessionId, + kind, + available: false, + reason, + }; +} + +function normalizeBodyContent( + kind: "request" | "response", + request: NetworkRequest, + body: { text: string; base64Encoded: boolean }, +): { text: string; base64Encoded: boolean } { + if (!body.base64Encoded) { + return body; + } + + const contentType = getBodyContentType(kind, request); + if (!isTextLikeContentType(contentType)) { + return body; + } + + try { + return { + text: Buffer.from(body.text, "base64").toString("utf8"), + base64Encoded: false, + }; + } catch { + return body; + } +} + +function getBodyContentType(kind: "request" | "response", request: NetworkRequest): string | undefined { + const headers = kind === "request" ? request.requestHeaders : request.responseHeaders; + const headerValue = headers?.["content-type"] || headers?.["Content-Type"]; + if (typeof headerValue === "string" && headerValue.length > 0) { + return headerValue; + } + + return kind === "response" ? request.mimeType : undefined; +} + +function isTextLikeContentType(contentType: string | undefined): boolean { + if (!contentType) { + return false; + } + + const normalized = contentType.split(";", 1)[0]?.trim().toLowerCase(); + if (!normalized) { + return false; + } + + if (normalized.startsWith("text/")) { + return true; + } + + return [ + "application/json", + "application/xml", + "application/javascript", + "application/x-javascript", + "application/ecmascript", + "application/x-www-form-urlencoded", + "application/graphql", + "application/problem+json", + "application/problem+xml", + ].includes(normalized) + || normalized.endsWith("+json") + || normalized.endsWith("+xml"); +} + +export type { + NetworkBodyResult, + NetworkHeaderResult, + NetworkListResult, + NetworkRequest, + NetworkSessionListEntry, + NetworkStatusResult, + NetworkSummaryResult, +} from "./types.js"; diff --git a/packages/agent-cdp/src/network/query.ts b/packages/agent-cdp/src/network/query.ts new file mode 100644 index 0000000..2747227 --- /dev/null +++ b/packages/agent-cdp/src/network/query.ts @@ -0,0 +1,186 @@ +import type { + NetworkHeaderEntry, + NetworkHeaderResult, + NetworkListOptions, + NetworkListResult, + NetworkRequest, + NetworkSession, + NetworkSummaryResult, +} from "./types.js"; + +export function querySummary(requests: NetworkRequest[], source: "live" | "session", sessionId?: string): NetworkSummaryResult { + const completed = requests.filter((request) => request.state === "completed"); + const failed = requests.filter((request) => request.state === "failed"); + const pending = requests.filter((request) => request.state === "pending"); + const countsByType = countAndSort(requests.map((request) => request.resourceType || "other"), "type"); + const countsByStatusBucket = countAndSort(requests.map((request) => toStatusBucket(request.statusCode, request.state)), "bucket"); + const slowest = [...requests] + .filter((request) => typeof request.durationMs === "number") + .sort((left, right) => (right.durationMs || 0) - (left.durationMs || 0)) + .slice(0, 5) + .map(({ id, method, url, durationMs }) => ({ id, method, url, durationMs })); + const largest = [...requests] + .filter((request) => typeof request.encodedDataLength === "number") + .sort((left, right) => (right.encodedDataLength || 0) - (left.encodedDataLength || 0)) + .slice(0, 5) + .map(({ id, method, url, encodedDataLength }) => ({ id, method, url, encodedDataLength })); + + const evidence: string[] = []; + if (failed.length > 0) { + evidence.push(`${failed.length} failed request${failed.length === 1 ? "" : "s"}`); + } + if (slowest[0]?.durationMs && slowest[0].durationMs >= 1000) { + evidence.push(`slowest request took ${Math.round(slowest[0].durationMs)}ms`); + } + if (largest[0]?.encodedDataLength && largest[0].encodedDataLength >= 1024 * 1024) { + evidence.push(`largest response transferred ${formatBytes(largest[0].encodedDataLength)}`); + } + if (pending.length > 0) { + evidence.push(`${pending.length} request${pending.length === 1 ? " is" : "s are"} still pending`); + } + + return { + source, + sessionId, + requestCount: requests.length, + completedCount: completed.length, + failedCount: failed.length, + pendingCount: pending.length, + countsByType, + countsByStatusBucket, + slowest, + largest, + availability: { + requestHeaders: requests.filter((request) => request.hasRequestHeaders).length, + responseHeaders: requests.filter((request) => request.hasResponseHeaders).length, + requestBodies: requests.filter((request) => request.hasRequestBody).length, + responseBodies: requests.filter((request) => request.hasResponseBody).length, + }, + evidence, + }; +} + +export function queryList(requests: NetworkRequest[], source: "live" | "session", options: NetworkListOptions): NetworkListResult { + const filtered = requests.filter((request) => matchesFilters(request, options)); + const limit = options.limit ?? 20; + const offset = options.offset ?? 0; + return { + source, + sessionId: options.sessionId, + total: filtered.length, + limit, + offset, + items: filtered.slice(offset, offset + limit), + }; +} + +export function queryRequest( + session: Pick | null, + liveRequests: NetworkRequest[], + requestId: string, +): NetworkRequest { + const scoped = session ? session.requests : liveRequests; + const request = scoped.find((candidate) => candidate.id === requestId); + if (!request) { + throw new Error(`Network request ${requestId} not found`); + } + return request; +} + +export function queryHeaders( + request: NetworkRequest, + kind: "request" | "response", + filterName?: string, +): NetworkHeaderResult { + const headers = kind === "request" ? request.requestHeaders : request.responseHeaders; + const entries = Object.entries(headers || {}) + .map(([name, value]) => ({ name, value })) + .filter((entry) => matchesHeaderName(entry, filterName)); + + return { + requestId: request.id, + sessionId: request.sessionId, + kind, + available: kind === "request" ? request.hasRequestHeaders : request.hasResponseHeaders, + entries, + }; +} + +function matchesFilters(request: NetworkRequest, options: NetworkListOptions): boolean { + if (options.type && request.resourceType.toLowerCase() !== options.type.toLowerCase()) { + return false; + } + if (options.method && request.method.toLowerCase() !== options.method.toLowerCase()) { + return false; + } + if (options.status && !matchesStatusFilter(request, options.status)) { + return false; + } + if (options.text) { + const text = options.text.toLowerCase(); + if (!request.url.toLowerCase().includes(text) && !request.method.toLowerCase().includes(text)) { + return false; + } + } + if (typeof options.minMs === "number" && (request.durationMs ?? -Infinity) < options.minMs) { + return false; + } + if (typeof options.maxMs === "number" && (request.durationMs ?? Infinity) > options.maxMs) { + return false; + } + if (typeof options.minBytes === "number" && (request.encodedDataLength ?? -Infinity) < options.minBytes) { + return false; + } + if (typeof options.maxBytes === "number" && (request.encodedDataLength ?? Infinity) > options.maxBytes) { + return false; + } + return true; +} + +function matchesStatusFilter(request: NetworkRequest, status: string): boolean { + const normalized = status.toLowerCase(); + if (normalized === "failed") { + return request.state === "failed"; + } + if (normalized === "pending") { + return request.state === "pending"; + } + const code = request.statusCode; + if (typeof code === "number") { + if (normalized.endsWith("xx") && normalized.length === 3) { + return Math.floor(code / 100) === Number.parseInt(normalized[0] || "0", 10); + } + return String(code) === normalized; + } + return false; +} + +function matchesHeaderName(entry: NetworkHeaderEntry, filterName?: string): boolean { + if (!filterName) { + return true; + } + return entry.name.toLowerCase().includes(filterName.toLowerCase()); +} + +function countAndSort(values: string[], field: TField): Array & { count: number }> { + const counts = new Map(); + for (const value of values) { + counts.set(value, (counts.get(value) || 0) + 1); + } + return [...counts.entries()] + .sort((left, right) => right[1] - left[1] || left[0].localeCompare(right[0])) + .map(([value, count]) => ({ [field]: value, count }) as Record & { count: number }); +} + +function toStatusBucket(statusCode: number | undefined, state: NetworkRequest["state"]): string { + if (state === "failed") return "failed"; + if (state === "pending") return "pending"; + if (typeof statusCode !== "number") return "unknown"; + return `${Math.floor(statusCode / 100)}xx`; +} + +function formatBytes(bytes: number): string { + if (bytes < 1024) return `${bytes} B`; + if (bytes < 1024 * 1024) return `${(bytes / 1024).toFixed(1)} KB`; + return `${(bytes / (1024 * 1024)).toFixed(1)} MB`; +} diff --git a/packages/agent-cdp/src/network/store.ts b/packages/agent-cdp/src/network/store.ts new file mode 100644 index 0000000..8e164c9 --- /dev/null +++ b/packages/agent-cdp/src/network/store.ts @@ -0,0 +1,165 @@ +import { NETWORK_LIVE_BUFFER_LIMIT } from "./types.js"; +import type { NetworkRequest, NetworkSession, NetworkSessionListEntry, NetworkStatusResult } from "./types.js"; + +interface ActiveNetworkSession extends NetworkSession { + requestIds: Set; +} + +export class NetworkStore { + private readonly liveRequests: NetworkRequest[] = []; + private readonly sessions: NetworkSession[] = []; + private activeSession: ActiveNetworkSession | null = null; + private nextSessionId = 1; + private nextRequestId = 1; + + generateRequestId(): string { + return `req_${this.nextRequestId++}`; + } + + startSession(name?: string, preserveAcrossNavigation = false): NetworkSession { + if (this.activeSession) { + throw new Error(`Network session ${this.activeSession.id} is already active`); + } + + this.activeSession = { + id: `net_${this.nextSessionId++}`, + name, + startedAt: Date.now(), + preserveAcrossNavigation, + requests: [], + requestIds: new Set(), + }; + + return this.activeSession; + } + + stopSession(): NetworkSession { + if (!this.activeSession) { + throw new Error("No active network session. Run network start first."); + } + + const finalized: NetworkSession = { + id: this.activeSession.id, + name: this.activeSession.name, + startedAt: this.activeSession.startedAt, + stoppedAt: Date.now(), + preserveAcrossNavigation: this.activeSession.preserveAcrossNavigation, + requests: [...this.activeSession.requests], + }; + this.sessions.push(finalized); + this.activeSession = null; + return finalized; + } + + discardActiveSession(): void { + this.activeSession = null; + } + + record(request: NetworkRequest, isNew: boolean): void { + if (isNew) { + this.liveRequests.push(request); + while (this.liveRequests.length > NETWORK_LIVE_BUFFER_LIMIT) { + this.liveRequests.shift(); + } + } + + if (!this.activeSession) { + return; + } + + if (!this.activeSession.requestIds.has(request.id)) { + request.source = "session"; + request.sessionId = this.activeSession.id; + this.activeSession.requestIds.add(request.id); + this.activeSession.requests.push(request); + } + } + + handleNavigation(): void { + if (!this.activeSession || this.activeSession.preserveAcrossNavigation) { + return; + } + + this.activeSession.requests = []; + this.activeSession.requestIds.clear(); + } + + getLiveRequests(): NetworkRequest[] { + return [...this.liveRequests].reverse(); + } + + getActiveSession(): NetworkSession | null { + if (!this.activeSession) { + return null; + } + + return { + id: this.activeSession.id, + name: this.activeSession.name, + startedAt: this.activeSession.startedAt, + preserveAcrossNavigation: this.activeSession.preserveAcrossNavigation, + requests: [...this.activeSession.requests], + }; + } + + getSession(sessionId: string): NetworkSession | undefined { + if (this.activeSession?.id === sessionId) { + return this.getActiveSession() || undefined; + } + return this.sessions.find((session) => session.id === sessionId); + } + + getLatestStoredSession(): NetworkSession | undefined { + return this.sessions.at(-1); + } + + getLatestSession(): NetworkSession | undefined { + return this.getActiveSession() || this.getLatestStoredSession(); + } + + listSessions(limit = 20, offset = 0): NetworkSessionListEntry[] { + const entries: NetworkSessionListEntry[] = []; + if (this.activeSession) { + entries.push({ + id: this.activeSession.id, + name: this.activeSession.name, + startedAt: this.activeSession.startedAt, + preserveAcrossNavigation: this.activeSession.preserveAcrossNavigation, + requestCount: this.activeSession.requests.length, + active: true, + }); + } + + for (const session of [...this.sessions].reverse()) { + entries.push({ + id: session.id, + name: session.name, + startedAt: session.startedAt, + stoppedAt: session.stoppedAt, + preserveAcrossNavigation: session.preserveAcrossNavigation, + requestCount: session.requests.length, + active: false, + }); + } + + return entries.slice(offset, offset + limit); + } + + getStatus(attached: boolean): NetworkStatusResult { + return { + attached, + liveRequestCount: this.liveRequests.length, + liveBufferLimit: NETWORK_LIVE_BUFFER_LIMIT, + activeSession: this.activeSession + ? { + id: this.activeSession.id, + name: this.activeSession.name, + startedAt: this.activeSession.startedAt, + preserveAcrossNavigation: this.activeSession.preserveAcrossNavigation, + requestCount: this.activeSession.requests.length, + } + : null, + storedSessionCount: this.sessions.length, + }; + } +} diff --git a/packages/agent-cdp/src/network/types.ts b/packages/agent-cdp/src/network/types.ts new file mode 100644 index 0000000..859cbc0 --- /dev/null +++ b/packages/agent-cdp/src/network/types.ts @@ -0,0 +1,162 @@ +export const NETWORK_LIVE_BUFFER_LIMIT = 200; + +export type NetworkRequestState = "pending" | "completed" | "failed"; + +export interface NetworkRedirectHop { + url: string; + statusCode?: number; + statusText?: string; +} + +export interface NetworkHeaderEntry { + name: string; + value: string; +} + +export interface NetworkBodyContent { + text: string; + base64Encoded: boolean; +} + +export interface NetworkBodyUnavailable { + reason: string; +} + +export interface NetworkRequest { + id: string; + rawRequestId: string; + sessionId?: string; + source: "live" | "session"; + url: string; + method: string; + resourceType: string; + state: NetworkRequestState; + statusCode?: number; + statusText?: string; + failureText?: string; + startedAt: number; + endedAt?: number; + durationMs?: number; + encodedDataLength?: number; + mimeType?: string; + protocol?: string; + remoteAddress?: string; + fromDiskCache?: boolean; + fromServiceWorker?: boolean; + hasRequestHeaders: boolean; + hasResponseHeaders: boolean; + hasRequestBody: boolean; + hasResponseBody: boolean; + requestHeaders?: Record; + responseHeaders?: Record; + requestBody?: NetworkBodyContent; + requestBodyUnavailable?: NetworkBodyUnavailable; + responseBody?: NetworkBodyContent; + responseBodyUnavailable?: NetworkBodyUnavailable; + redirectChain: NetworkRedirectHop[]; + redirectedTo?: string; + navigationId?: string; + isNavigationRequest: boolean; + isWebSocket: boolean; + webSocketHandshake?: { + statusCode?: number; + statusText?: string; + }; +} + +export interface NetworkSession { + id: string; + name?: string; + startedAt: number; + stoppedAt?: number; + preserveAcrossNavigation: boolean; + requests: NetworkRequest[]; +} + +export interface NetworkStatusResult { + attached: boolean; + liveRequestCount: number; + liveBufferLimit: number; + activeSession: { + id: string; + name?: string; + startedAt: number; + preserveAcrossNavigation: boolean; + requestCount: number; + } | null; + storedSessionCount: number; +} + +export interface NetworkSessionListEntry { + id: string; + name?: string; + startedAt: number; + stoppedAt?: number; + preserveAcrossNavigation: boolean; + requestCount: number; + active: boolean; +} + +export interface NetworkSummaryResult { + source: "live" | "session"; + sessionId?: string; + requestCount: number; + completedCount: number; + failedCount: number; + pendingCount: number; + countsByType: Array<{ type: string; count: number }>; + countsByStatusBucket: Array<{ bucket: string; count: number }>; + slowest: Array>; + largest: Array>; + availability: { + requestHeaders: number; + responseHeaders: number; + requestBodies: number; + responseBodies: number; + }; + evidence: string[]; +} + +export interface NetworkListOptions { + sessionId?: string; + limit?: number; + offset?: number; + type?: string; + status?: string; + method?: string; + text?: string; + minMs?: number; + maxMs?: number; + minBytes?: number; + maxBytes?: number; +} + +export interface NetworkListResult { + source: "live" | "session"; + sessionId?: string; + total: number; + limit: number; + offset: number; + items: NetworkRequest[]; +} + +export interface NetworkHeaderResult { + requestId: string; + sessionId?: string; + kind: "request" | "response"; + available: boolean; + entries: NetworkHeaderEntry[]; +} + +export interface NetworkBodyResult { + requestId: string; + sessionId?: string; + kind: "request" | "response"; + available: boolean; + mimeType?: string; + base64Encoded?: boolean; + text?: string; + filePath?: string; + bytesWritten?: number; + reason?: string; +} diff --git a/packages/agent-cdp/src/types.ts b/packages/agent-cdp/src/types.ts index ae639a7..d7780a2 100644 --- a/packages/agent-cdp/src/types.ts +++ b/packages/agent-cdp/src/types.ts @@ -94,6 +94,33 @@ export type IpcCommand = | { type: "clear-target" } | { type: "list-console-messages"; limit?: number } | { type: "get-console-message"; id: number } + | { type: "network-status" } + | { type: "network-start"; name?: string; preserveAcrossNavigation?: boolean } + | { type: "network-stop" } + | { type: "network-list-sessions"; limit?: number; offset?: number } + | { + type: "network-summary"; + sessionId?: string; + } + | { + type: "network-list"; + sessionId?: string; + limit?: number; + offset?: number; + resourceType?: string; + status?: string; + method?: string; + text?: string; + minMs?: number; + maxMs?: number; + minBytes?: number; + maxBytes?: number; + } + | { type: "network-request"; requestId: string; sessionId?: string } + | { type: "network-request-headers"; requestId: string; sessionId?: string; name?: string } + | { type: "network-response-headers"; requestId: string; sessionId?: string; name?: string } + | { type: "network-request-body"; requestId: string; sessionId?: string; filePath?: string } + | { type: "network-response-body"; requestId: string; sessionId?: string; filePath?: string } | { type: "start-trace" } | { type: "stop-trace"; filePath?: string } | { type: "capture-memory"; filePath: string }