feat(core): pass Request to onSkippedRequest - #3992
Open
barjin wants to merge 184 commits into
Open
Conversation
BREAKING CHANGE: The project is now native ESM without a CJS alternative. This is fine since all supported node versions allow `require(esm)`. Also all the dependencies are updated to the latest versions, including cheerio v1.
BREAKING CHANGE: The crawler following options are removed: - `handleRequestFunction` -> `requestHandler` - `handlePageFunction` -> `requestHandler` - `handleRequestTimeoutSecs` -> `requestHandlerTimeoutSecs` - `handleFailedRequestFunction` -> `failedRequestHandler`
BREAKING CHANGE: The crawling context no longer includes the `Error` object for failed requests. Use the second parameter of the `errorHandler` or `failedRequestHandler` callbacks to access the error. Previously, the crawling context extended a `Record` type, allowing to access any property. This was changed to a strict type, which means that you can only access properties that are defined in the context.
….retireOnBlockedStatusCodes` BREAKING CHANGE: `additionalBlockedStatusCodes` parameter of `Session.retireOnBlockedStatusCodes` method is removed. Use the `blockedStatusCodes` crawler option instead.
….retireOnBlockedStatusCodes` BREAKING CHANGE: `additionalBlockedStatusCodes` parameter of `Session.retireOnBlockedStatusCodes` method is removed. Use the `blockedStatusCodes` crawler option instead.
also tries to bump better-sqlite3 to latest version to have prebuilds for node 22
- closes #2479 - closes #3106 - closes #3107 - closes #3078 In my opinion, it makes a lot of sense to do the remaining changes in a separate PR. - [x] Introduce a `ContextPipeline` abstraction - [x] Update crawlers to use it - [x] Make sure that existing tests pass - [ ] Refine the `ContextPipeline.compose` signature and the semantics of `BasicCrawlerOptions.contextPipelineEnhancer` to maximize DX - [x] Write tests for the `contextPipelineEnhancer` - [x] Resolve added TODO comments (fix immediately or make issues) - [ ] Update documentation The `context-pipeline` branch introduces a fundamental architectural change to how Crawlee crawlers build and enhance the crawling context passed to request handlers. The core motivation is to fix the composition and extensibility nightmare in the current crawler hierarchy. 1. **Rigid inheritance hierarchy**: Crawlers were stuck in a brittle inheritance chain where each layer manipulated the context object while assuming that it already satisfied its final type. Multiple overrides of `BasicCrawler` lifecycle methods made the execution flow even harder to follow. 2. **Context enhancement via monkey-patching**: Manual property assignment (`crawlingContext.page = page`, `crawlingContext.$ = $`) scattered everywhere. It was a mess to follow and impossible to reason about. 3. **Cleanup coordination**: Resource cleanup was handled by separate `_cleanupContext` methods that were not co-located with the initialization. 4. **Extension mechanism was broken**: The `CrawlerExtension.use()` API tried to let you extend crawlers (the ones based on `HttpCrawler`) by overwriting properties - completely type-unsafe and fragile as hell. Introduces `ContextPipeline` - a **middleware-based composition pattern** where: - Each crawler layer defines how it enhances the context through explicit `action` functions - Cleanup logic is co-located with initialization via optional `cleanup` functions - Type safety is maintained through TypeScript generics that track context transformations - The pipeline executes middleware sequentially with proper error handling and guaranteed cleanup Declarative middleware composition with co-located cleanup: ```typescript contextPipeline.compose({ action: async (context) => ({ page, $ }), cleanup: async (context) => { await page.close(); } }) ``` The `ContextPipeline<TBase, TFinal>` tracks type transformations through the chain: ```typescript ContextPipeline<CrawlingContext, CrawlingContext> .compose<{ page: Page }>(...) // ContextPipeline<CrawlingContext, CrawlingContext & { page: Page }> .compose<{ $: CheerioAPI }>(...) // ContextPipeline<CrawlingContext, CrawlingContext & { page: Page, $: CheerioAPI }> ``` The `CrawlerExtension.use()` is gone. New approach via `contextPipelineEnhancer`: ```typescript new BasicCrawler({ contextPipelineEnhancer: (pipeline) => pipeline.compose({ action: async (context) => ({ myCustomProp: ... }) }) }) ``` The current way to express a context pipeline middleware has some shortcomings (`ContextPipeline.compose`, `BasicCrawlerOptions.contextPipelineEnhancer`). I suggest resolving this in another PR. For most legitimate use cases, this should be non-breaking. Those who extend the Crawler classes in non-trivial ways may need to adjust their code though - the non-public interface of `BasicCrawler` and `HttpCrawler` changed quite a bit. The pipeline uses `Object.defineProperties` for each middleware. Is this a serious performance consideration? --------- Co-authored-by: Martin Adámek <banan23@gmail.com>
Extracts `ProxyConfiguration` to `BasicCrawler` (related to discussion under #2917). Pass the `ProxyConfiguration` instance to the `SessionPool` for new `Session` object creation. Store and read the `ProxyInfo` from the `Session` instance instead of calling the `ProxyConfiguration` methods in the crawlers. closes #3198
Phasing out `got-scraping`-specific interfaces in favour of native `fetch` API. Related to #3071
Fixes build toolchain errors caused by the recent rebase onto the current `master` ([more details here](https://apify.slack.com/archives/C02JQSN79V4/p1764373034961859)). The largest thing is probably updating the dependency versions in `package.json` - if `turborepo` doesn't find the matching version in the local workspace, it will build against the package pulled from `npm` (which doesn't match the v4 API at this point).
Related to #3275 --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Fixes the response header handling in `GotScrapingHttpClient` (`got-scraping` headers contain unexpected `Symbol`s and HTTP2 pseudoheaders). Fixes omission from one of the previous commits - `GotScrapingHttpClient.stream` now uses proxy correctly again. Closes #2917
…ed service locator (#3951) `bindMethodsToServiceLocator()` wraps every prototype method as an own property on the crawler instance so that calls resolve the crawler's scoped service locator. The walk goes derived-first but assigned wrappers unconditionally, so a method defined on both a subclass and a base class got wrapped twice, and the base version, assigned last, won. Any crawler constructed with scoped services (`logger`, `storageBackend`, `eventManager`, or a custom `configuration`) therefore lost all its method overrides. `AdaptivePlaywrightCrawler` fell back to `BasicCrawler.runRequestHandler()`: no rendering type prediction or detection, no log replay, and an `_init()` that never initialized the predictor. Every other crawler subclass lost its `buildContextPipeline()` override the same way. This also explains why the log replay tests added in #3803 kept passing with the replay line reverted. They pass a custom `logger`, so the adaptive code path never ran and the request handler logged straight to the real logger. With this fix they fail without the replay lines and pass with them (checked in both directions). The fix skips a method once a more derived version of it has been handled. A key seen as a getter or setter at a derived level also blocks wrapping its base version, since dynamic dispatch would pick the derived accessor. Closes #3934
…3939) Persists the page cookies to the `Session` instance both after the navigation and after the user-specified requestHandler runs.
## Timeout redesign (#2951) v4 had already dropped the old `navigationTimeoutSecs + requestHandlerTimeoutSecs + buffer` sum, so the request handler timeout covers only the user's function and the confusing "timed out after 130 seconds" messages are gone. What it did not do is put back the pieces that sum incidentally bounded. - **The navigation phase is one window.** `navigationTimeoutSecs` covers the `preNavigationHooks`, the navigation itself, and the `postNavigationHooks` as a single shared budget, matching Crawlee for Python. A hook that hangs no longer stalls the request forever; it eats into the same window the navigation uses. (This replaces an earlier attempt at a separate per-hook `navigationHooksTimeoutSecs`, which is gone.) - **A whole-request backstop.** The phases between the timed ones (`extendContext`, the robots.txt check, response processing) could hang indefinitely. An internal backstop now bounds the whole request. It is sized to outlast the phases that have their own timeout (navigation plus the handler), so a legitimately slow request is never cut short and it only fires when something is genuinely stuck. It is configured with `CRAWLEE_INTERNAL_TIMEOUT`, now resolved through `Configuration` like the other env-backed options. Set it below the phase timeouts and the crawler raises it per request and warns at startup, rather than cutting a phase short. ## `context.extendTimeout()` When the time needed is only apparent once a hook or handler is already running, `context.extendTimeout(secs)` buys more. From inside the navigation phase it pushes the shared navigation window; from the request handler it pushes the handler timeout. Either way it also pushes the backstop and raises the request-manager reservation, so the extra time is neither clipped by the backstop nor undone by a locking backend handing the request out again. ## Per-route timeouts (#1485) ```ts router.addHandler('LIST', handler, { requestHandlerTimeoutSecs: 120 }); router.addHandler('DETAIL', handler); // keeps the crawler's default ``` `requestHandlerTimeoutSecs` is unchanged and stays the default for anything a route does not override; a route's value may be longer or shorter than it. The label is known before the handler starts, so the timeout is resolved per request and the router never reaches back into the crawler mid-flight. The backstop and the reservation both account for the longest route in play. ## Browser navigation A `preNavigationHooks` hook can still override `gotoOptions.timeout` (including `0`, Playwright's "no timeout"); the shared window no longer clamps it to 1ms or discards a larger value. A navigation timeout, whether ours or the driver's own, is reported as `Navigation timed out after N seconds` instead of the driver's raw millisecond value. ## Adaptive crawler `AdaptivePlaywrightCrawler` runs the handler up to twice per request (a static attempt falling through to the browser). A `getRequestHandlerRunCount` hook (2 for adaptive, 1 everywhere else) sizes the whole-request budgets for both runs, while each run keeps its own handler window. This replaces an earlier one-off doubling that missed per-route overrides. ## Notes on the implementation The backstop is a bare timer, not `addTimeoutToPromise`: nested `addTimeoutToPromise` calls share one `AbortController`, so wrapping the whole request in one would let the handler timing out abort the outer context and cancel the error handling that reclaims the request. The plumbing lives in its own `request-backstop.ts` module. For the same reason the HTTP navigation binds its request to the navigation frame's `@apify/timeout` cancel signal rather than a fixed `AbortSignal.timeout`: the response body is read lazily, after the post-navigation hooks, so a fixed timer would abort a body a hook is legitimately still keeping alive via `extendTimeout`. A genuine navigation timeout still fails the request; the body read is bounded at the parse step. Depends on the `extendTimeout` addition in `@apify/timeout` (apify/apify-shared-js#669), released as 0.4.4 and consumed here. Closes #1485 Closes #2951
Align `EnqueueLinksOptions` with crawlee-python (#3409): - Replace `globs`, `regexps`, `pseudoUrls` options with `include`/`exclude` accepting `UrlPatternInput[]` - Strip request options (label, method, payload, userData, headers) from pattern objects — patterns are pure URL matchers - `transformRequestFunction` is now the only way to customize per-request options, runs after all filtering - Add `'skip'` and `'unchanged'` return values to `RequestTransform` (aligned with Python's `RequestTransformAction`) - Apply same changes to `enqueueLinksByClickingElements` (Playwright + Puppeteer) and `SitemapRequestList` - Remove `@apify/pseudo_url` dependency and `PseudoUrl` re-export - Update all templates from `globs` to `include` --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
fs-extra was only used for `ensureDir` and `writeJSON` in `cli` and `basic-crawler`, both one-line wrappers over `node:fs/promises`. Swapped them for `mkdir(recursive)` and `writeFile`, dropping the dependency. Related to #3549
…io` (#3968) Extends the fix from #3836 (which already made `@crawlee/utils` and `http-crawler` lazy-load cheerio) to the two remaining spots that were still importing it eagerly: `playwright-utils.ts` and `puppeteer_utils.ts`. Both only use cheerio inside `parseWithCheerio()`, so every Playwright/Puppeteer crawler paid cheerio's import cost even when the handler never calls it. Related to #3549.
…n hooks (#3972) In v3, browser navigation hooks received the `page.goto()` options as a second argument (`(crawlingContext, gotoOptions) => ...`). In v4 they take a single context argument and the options live on `context.gotoOptions`. The upgrading guide documents the analogous `HttpCrawler` `gotOptions` change but never mentions the browser-side one, which every `PlaywrightCrawler`/`PuppeteerCrawler` user with a two-argument hook hits on upgrade. Adds a short section with a before/after sample next to the existing `HttpCrawler` entry.
The test 'BrowserCrawler > should allow modifying gotoOptions by pre navigation hooks' intermittently failed with `expected 59999 to deeply equal 60000`. The hook set `gotoOptions.timeout = 60000`, which is exactly the default `navigationTimeoutMillis` for browser crawlers. The crawler detects "the hook did not override the timeout" by value equality with that default, so the test's value was treated as untouched and clamped to the remaining navigation window (`60000 - elapsed ms`). The exact assertion was therefore a race: a millisecond of scheduling delay before `navigate()` produced 59999, and adding a 5 ms delay to the hook chain reproduced the failure on every run. The hook now sets a value distinct from the default (25000), which the crawler honors verbatim, so the assertion is deterministic (verified stable across repeated runs, including with an artificial delay). A comment in the test explains why the value must differ from the default. The clamp path itself is already covered by the slow-navigation tests in `playwright_crawler.test.ts`.
Navigation hooks on browser crawlers could not be typed with custom `userData`: a hook declared as `(ctx: PuppeteerCrawlingContext<MyUserData>) => ...` failed to type-check against `preNavigationHooks`/`postNavigationHooks`, while the HTTP crawler family already supports this (#2063). - `BrowserCrawlingContext`, `PlaywrightCrawlingContext`, `PuppeteerCrawlingContext`, `StagehandCrawlingContext` and `AdaptivePlaywrightCrawlerContext` now default `UserData` to `any`, the same pattern (and marker comment) the HTTP family adopted in v4. - `PlaywrightHook`, `PuppeteerHook` and `StagehandHook` are now type aliases generic over `UserData` (e.g. `PlaywrightHook<MyUserData>`), like `CheerioHook` and `HttpHook`. - `HttpCrawlerOptions.preNavigationHooks` now uses `CrawlingContext<any>`, so pre-navigation hooks typed with custom user data (`InternalHttpHook<CrawlingContext<MyUserData>>`) are assignable as well; the post-navigation option already allowed this. - The same default change makes request handlers typed with custom user data assignable to untyped crawler options as well. - Adds type-level regression tests (hooks and request handlers), an upgrading guide entry for the switch from interfaces to type aliases, and regenerated API snapshots. Closes #2063
…ixes (#3980) Adopts native `#` private fields for private class properties across all packages and removes the remaining `_` prefixes from protected/private members, as agreed in #3108. - Around 300 private properties are now native `#` fields, so they are hard-private at runtime and invisible to spread and `JSON.stringify`. Private and protected methods keep the `private`/`protected` keyword and lose the `_` prefix. - Renamed protected extension points: `_init` → `init`, `_throwOnBlockedRequest` → `throwOnBlockedRequest`, `_getMessageFromError` → `getMessageFromError` and `_getCookieHeaderFromRequest` → `getCookieHeaderFromRequest` on `BasicCrawler`; `_navigationHandler` → `navigationHandler` on `BrowserCrawler` and its subclasses; `_addProxyToLaunchOptions` → `addProxyToLaunchOptions`, `_isChromiumBasedBrowser` → `isChromiumBasedBrowser`, `_connectToRemoteBrowser` → `connectToRemoteBrowser` and `_throwAugmentedLaunchError` → `throwAugmentedLaunchError` on `BrowserPlugin`. The `@internal` `LaunchContext._remoteToken` and `PlaywrightBrowser._setBrowserType` became `remoteToken` and `setBrowserType`. - `BrowserPlugin._launch` and `BrowserController._close`/`_kill`/`_newPage`/`_getCookies`/`_setCookies` keep the underscore, since their public wrapper methods own the plain names. `Readable._read` is a Node contract. - Tests that used to poke internal state were rewritten against public API where the test's meaning survives: session pool state via `getState()`, request list persistence via the exported persistence keys, proxy rotation via `newUrl()`, autoscaling options via the constructor, crawler cleanup via `teardown()`. The members those tests reached are now `#` fields too. - The members still declared with TypeScript's `private` fall into two groups, each with a one-line comment saying why: runtime hazards for `#` (the adaptive crawler's log proxy, two cross-class accesses), and genuine test injection seams (backend delegation spies, interval replacement, static counter resets, config-echo assertions with no public surface). - `LaunchContext.extend()` now rejects all declared fields and accessors as reserved names; it previously missed fields declared after the reserved list was computed. The fingerprinting hook assigns the declared `fingerprint` field directly instead of going through `extend()`. - A new `no-underscore-dangle` oxlint rule (with `enforceInClassFields` and `enforceInMethodNames`) enforces the convention. - The upgrading guide documents the renames and the `#` semantics change; API reports regenerated. Closes #3108
`closeCookieModals` loads `idcac-playwright` through a guarded dynamic import, and both crawler packages already declare it as an optional peer dependency — but it was also listed in `dependencies`, so it got installed for everyone anyway. The e2e actors that call `closeCookieModals` now depend on it explicitly. `playwright-crawler` also had a `^0.1.3` dependency against a `^0.2.0` peer range.
Closes #3945. `@crawlee/types` no longer imports `CookieJar`/`SerializedCookieJar` from `tough-cookie` - it declares its own structurally-compatible interfaces instead, so `tough-cookie` is no longer part of its dependency tree. `tough-cookie` stays a direct dependency everywhere it's actually used (`core`, `http-client`, `impit-client`).
…owser type (#3981) `PlaywrightBrowser._setBrowserType()` has existed since the initial crawlee commit but was never called, so the `_browserType` field stayed `undefined` and `browserType()` returned `undefined` at runtime despite its non-optional `BrowserType` signature. Persistent contexts are the default (`useIncognitoPages: false`), so a call like `browser.browserType().name()` on the wrapper threw a `TypeError`. The incognito and remote-connection paths return the native Playwright `Browser` and were not affected. This wires the existing setter in `PlaywrightPlugin._launch()` where the wrapper is created. `this.library` is the `BrowserType` that launched the context, so the wrapper now provides the consistent API with Playwright's `Browser` that its docblock describes. Also adds a test covering both the persistent-context wrapper and the native incognito browser.
Both are only used on specific code paths (sitemap parsing, CSV export)
but were loaded eagerly by every crawler import, costing ~110ms of the
~500ms `import('@crawlee/http')` on my machine.
Related: #3549
…lt when the callback actually reads it
barjin
marked this pull request as draft
August 7, 2026 12:54
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Skipped requests now get the full
Requestinstead of just a URL string, souserDataand other metadata survive a skip; theRequestis built lazily so crawlers that don't useonSkippedRequestpay nothing for it.Closes #3867
Closes #3801