-
-
+
+
-
+
A web scraping and browser automation library
-
-
+
+
-
-
-
-
-
+
+
+
+
+
Crawlee covers your crawling and scraping end-to-end and **helps you build reliable scrapers. Fast.**
diff --git a/REBASE-DEFERRED.md b/REBASE-DEFERRED.md
new file mode 100644
index 000000000000..9cf562e13708
--- /dev/null
+++ b/REBASE-DEFERRED.md
@@ -0,0 +1,517 @@
+# Rebase deferrals — reconcile against `v4-reverse` before finishing
+
+This file tracks resolutions made during the `v4` → `master` rebase that were
+**not** brought fully to the desired end state (`v4-reverse`). Untracked on
+purpose — delete once reconciled. Do a final `git diff v4-reverse`
+per file to confirm nothing below was lost.
+
+## `packages/http-crawler/src/internals/file-download.ts`
+
+**Commit:** `cf72dda66` — `refactor!: Introduce the ContextPipeline abstraction (#3119)`
+
+**What I did:** took #3119's coherent committed version of the whole file
+(`git checkout --theirs`). The 3 conflict hunks were interdependent and 7 later
+rebase commits reshape this file, so a hunk-by-hunk merge risked dangling
+references.
+
+**Deferred (present in `v4-reverse`, NOT re-added by any rebase commit — must be
+folded back at the end):**
+- Master's typed schema-router overload: `RouteSchemas` / `RoutesFromSchemas`
+ imports from `../index.js`, and the `downloadFile`/router-factory overload that
+ returns `RouterHandler>`. Type-level only,
+ no runtime effect, compiles fine without it.
+- Confirm `abortDownload` handling matches `v4-reverse` (v4-reverse keeps it; it
+ may arrive via a later commit — verify it isn't dropped).
+
+**Reconcile with:**
+`git diff :packages/http-crawler/src/internals/file-download.ts \
+ v4-reverse:packages/http-crawler/src/internals/file-download.ts`
+
+## Master-only features that WERE folded in (not lost — listed for verification)
+
+- `playwright-crawler.ts` `enhanceContext`: `listDownloads` (downloads array +
+ `page.on('download')` + `listDownloads: async () => downloads`). Master-only,
+ no rebase commit re-adds it; folded in to match `v4-reverse`.
+- `browser-crawler.ts` `buildContextPipeline` cleanup: puppeteer-25
+ `addTimeoutToPromise` wrapper around `page.close()`.
+- `browser-crawler.ts`: kept master's `userRequestHandler` getter override.
+- `adaptive-playwright-crawler.ts`: kept master's `shouldPropagateError` check,
+ combined with #3119's `RequestHandlerError.cause` unwrap.
+
+## General
+
+Recurring mechanical conflicts (ESM `.js` imports, `test/shared/*` paths,
+`3.17.0` version bumps, `RequestValidationError`/`zod` schema-feature imports)
+are auto-applied by git rerere. Verify each rerere-resolved commit is
+marker-free; spot-checked so far and clean.
+
+## Possible duplicate import to verify — `adaptive-playwright-crawler.ts`
+
+At/after the ContextPipeline commit, the top of
+`packages/playwright-crawler/src/internals/adaptive-playwright-crawler.ts` may
+import `BrowserHook` / `LoadedRequest` / `Request` from `@crawlee/browser` on
+BOTH an early `import type {...}` block and a later
+`import type { BasicCrawlerOptions, BrowserHook, LoadedRequest, Request } from '@crawlee/browser'`.
+Duplicate type imports from the same module = TS error. Check the final tree and
+dedupe if present (a later rebase commit may already fix it).
+
+## Latent bug — `packages/core/src/storages/request_list.ts` dangling `this.events`
+
+Commit `ba3a3568a` (`refactor!: Extract service management from Configuration
+into ServiceLocator class #3325`) removes the `private events: EventManager`
+field and its `this.events = config.getEventManager()` assignment, replacing
+the one conflicting usage (`initialize()`'s `.on(EventType.PERSIST_STATE, ...)`
+call) with `serviceLocator.getEventManager().on(...)`.
+
+However, a master-only teardown method (not part of the conflict, already
+auto-merged in) still calls `this.events.off(EventType.PERSIST_STATE, ...)`
+around line 542 — `events` no longer exists on the class, so this is a
+TypeScript compile error (`Property 'events' does not exist`).
+
+**Confirmed this is not just my resolution**: `v4-reverse` has the identical
+dangling `this.events.off(...)` call at its equivalent line. No commit further
+along the rebase touches it either. This needs a manual fix — swap it to
+`serviceLocator.getEventManager().off(EventType.PERSIST_STATE, this.persistState)`
+— whenever the tree is type-checked.
+
+## ServiceLocator commit (`ba3a3568a` #3325) — architecture collision in Snapshotter/SystemStatus
+
+Master had already refactored `Snapshotter`/`SystemStatus` into a composable
+`LoadSignal` architecture (memory/cpu/event-loop/client signal objects with
+`.start()/.stop()/.handle()`) in an earlier-applied commit. `ba3a3568a` is v4's
+older, pre-LoadSignal, monolithic interval-based Snapshotter rewritten only for
+ServiceLocator. These two are fundamentally different implementations of the
+same class — verified against `v4-reverse` that the LoadSignal architecture
+survives, so for every conflicting method (imports, class fields, `start()`,
+`stop()`, `_snapshotClient`/`handle()`) I kept HEAD's LoadSignal delegation and
+discarded theirs' interval/serviceLocator-direct logic entirely.
+
+**Found and fixed a real latent bug in the process**: `Snapshotter`'s
+constructor had its `client`/`config` destructuring and `this.client =
+/this.config =` assignments silently dropped by an earlier (non-conflicting)
+auto-merge upstream in this rebase, while `this.client`/`this.config` were
+still referenced further down (passed into the `MemoryLoadSignal`/
+`createCpuLoadSignal`/`createClientLoadSignal` factories, which — at this point
+in history — still require them as explicit params; they don't yet pull from
+`serviceLocator` internally). Restored the wiring using commit `308da2263`
+(an early point in this rebase where it was still intact) as reference,
+adapted to call `serviceLocator.getConfiguration()` /
+`serviceLocator.getStorageClient()` as defaults (matching this commit's
+intent) instead of the old `Configuration.getGlobalConfig()` /
+`config.getStorageClient()`. Also switched the `Configuration` import to
+`import type` since it's now type-only in this file.
+
+`SystemStatus` needed the same reconciliation: kept master's `loadSignals`
+option (composes custom signals with the snapshotter's built-in ones) but
+dropped the `config` option/field entirely — `new Snapshotter()` with no args
+now works since the above fix wires its own `serviceLocator` defaults.
+
+`storage_manager.ts` had a similar merge artifact: an unconflicted `try/finally`
+wrapped body (from a separate already-applied commit) duplicated by this
+commit's non-try/finally version of the same logic, producing a dead
+duplicate tail after the conflict markers. Deleted the duplicate and applied
+this commit's actual semantic change (`this.config.getStorageClient()` →
+`serviceLocator.getStorageClient()`) to the surviving copy.
+
+**Please re-verify** the `Snapshotter`/`SystemStatus`/`storage_manager.ts`
+resolutions once dependencies are installed and `tsc`/tests can run — these
+were reasoned through by reading surrounding code and cross-checking
+`v4-reverse`, not verified by a compiler.
+
+## Fixed a real dead-function bug — `local_event_manager.ts` `getMemoryInfoV2`
+
+While resolving commit `7c3ba07ea` ("refactor: resolve last direct @apify/log
+calls"), found that `LocalEventManager`'s private `getMemoryInfo()` helper
+branched on `this.config.get('systemInfoV2')` to call `getMemoryInfoV2(...)` —
+but **`getMemoryInfoV2` does not exist anywhere in `@crawlee/utils`** (only
+`getMemoryInfo` and `getCurrentCpuTicksV2` exist; the CPU side got a real V2
+implementation, the memory side never did). That branch would throw
+`ReferenceError` at runtime for anyone with `systemInfoV2` config enabled.
+
+Also found the surviving (non-V2) branch's `getMemoryInfo()` call had no
+import at all in scope — a dangling reference from an earlier upstream
+auto-merge, same class of bug as the `request_list.ts` one documented above.
+
+**Fix applied**: collapsed the private `getMemoryInfo()` method to just the
+working path — dynamically imports `getMemoryInfo` from `@crawlee/utils` and
+passes `containerized`/`logger`, matching this commit's intent and the
+pattern already used by the sibling `createCpuInfo()` method (which does have
+a working V2 path via `getCurrentCpuTicksV2`). Dropped the dead
+`systemInfoV2`/`getMemoryInfoV2` branch entirely rather than inventing a
+`getMemoryInfoV2` implementation, which would be scope creep beyond a merge
+conflict fix. `LocalEventManager.prototype.getMemoryInfo` is still spied on by
+`test/core/autoscaling/snapshotter.test.ts`, so the method itself is kept —
+only its broken internals were fixed.
+
+**Please verify**: if a memory-specific V2 code path was intended (mirroring
+`getCurrentCpuTicksV2`), it needs to be implemented for real — this fix does
+not add one, it just removes a dead reference to a non-existent one.
+
+## yarn → pnpm migration (`930b2ef4f`) — needs verification with a real `pnpm install`
+
+Per your explicit choice, I took theirs (pnpm) fully for this commit: deleted
+`yarn.lock`, `docs/yarn.lock`, `website/yarn.lock`, `.yarnrc.yml`; adopted
+`pnpm@10.24.0` as `packageManager`/volta; rewrote CI workflow yarn/corepack
+steps to `apify/workflows/pnpm-install@main`; kept master's newer
+devDependency versions (`@apify/tsconfig`, `@commitlint`, `@playwright/browser-*`,
+`typescript`, `playwright`, docusaurus 3.10.2) alongside the new oxlint/oxfmt +
+pnpm tooling.
+
+**One thing I could not verify mechanically**: master's root `package.json`
+had a yarn-only `"resolutions"` block (`tmp`, `@puppeteer/browsers`,
+`playwright-core@1.61.1`, `form-data`, `tar`, `lerna/js-yaml`) that this
+commit drops in favor of pnpm's `overrides` in `pnpm-workspace.yaml` (which
+already carried v4's own `playwright-core@1.58.2`, `@browserbasehq/stagehand`,
+`minimatch` overrides). I merged master's entries into that `overrides:`
+block, bumping `playwright-core` to `1.61.1` to match the `playwright`
+devDependency version we kept (a version mismatch between the two would be
+worse than dropping the override). For the yarn-specific nested-scope
+override `lerna/js-yaml`, I translated it to pnpm's `"lerna>js-yaml"` syntax
+— **this translation is unverified**; pnpm's override key syntax has specific
+rules I can't test without actually running `pnpm install` (which requires
+installing the new package manager). Please run `pnpm install` and confirm
+the lockfile resolves cleanly, especially the `js-yaml` override under
+`lerna`.
+
+## `storage_manager.ts` `openStorage` — kept try/finally around the new StorageClient logic
+
+Commit `ffff3347e` (`refactor!: Overhaul the StorageClient interface #3570`)
+rewrites `StorageManager.openStorage` to resolve identifiers via
+`_resolveIdentifier`/`_createSubClient` (both already present unconflicted
+elsewhere in the file). Its own version of the method does NOT wrap the body
+in `try/finally` — `this.storageOpenQueue.shift()` runs as a plain trailing
+statement. The pre-this-commit version (which I'd already resolved in an
+earlier conflict) wrapped the equivalent logic in `try { ... } finally {
+this.storageOpenQueue.shift(); }` specifically so the queue lock is always
+released even if resolution/creation throws.
+
+I kept the `try/finally` safety net around theirs' new logic rather than
+taking the method verbatim, since dropping it reintroduces a real deadlock
+risk (a failed `_resolveIdentifier`/`_createSubClient`/`getMetadata` call
+would permanently wedge `storageOpenQueue` for all subsequent `openStorage`
+calls on that manager). This wasn't verifiable against `v4-reverse` (that
+file is renamed/rewritten further into `storage_instance_manager.ts` with an
+unrelated alias-based architecture by then). Worth a second look once you can
+run the test suite.
+
+## Fixed more dead API calls — `Configuration` redesign (`d1f4c98e5` #3484)
+
+This commit ("feat: redesign `Configuration` class for v4") replaces
+`Configuration`'s `.get(key, default)`/`.set()` accessor methods with plain
+property access (e.g. `config.availableMemoryRatio` instead of
+`config.get('availableMemoryRatio')`), and removes `Configuration.getEventManager()`/
+`.getStorageClient()` entirely (those now only live on `serviceLocator`).
+
+Same class of bug as the `request_list.ts`/`local_event_manager.ts` issues
+documented above: two files outside this commit's own diff — never flagged
+by any merge conflict — still called the now-deleted methods:
+
+- `packages/core/src/autoscaling/memory_load_signal.ts`: `this.config.get(...)`
+ (×3) and `this.config.getEventManager()`. Fixed to plain property access
+ (`this.config.memoryMbytes`, `.availableMemoryRatio`, `.containerized`) and
+ `serviceLocator.getEventManager()`. Also dropped the `systemInfoV2`/
+ `getMemoryInfoV2` branch in `_getTotalMemoryBytes()` — confirmed
+ `getMemoryInfoV2` doesn't exist anywhere in `@crawlee/utils` (same dead
+ function found earlier in `local_event_manager.ts`) and `systemInfoV2`
+ isn't a field on the new `Configuration` schema either. Restored the
+ `isContainerized()` auto-detect fallback (`this.config.containerized ??
+ (await isContainerized())`) to match the behavior the original code had,
+ since `getMemoryInfo()`'s own default for missing `containerized` is `false`,
+ not auto-detection.
+- `packages/core/src/autoscaling/cpu_load_signal.ts`: `options.config.getEventManager()`
+ → `serviceLocator.getEventManager()`. Left the now-unused-but-still-required
+ `config: Configuration` field on `CpuLoadSignalOptions` alone rather than
+ reworking the signal-creation API — `Snapshotter` still passes `config` when
+ constructing this signal, so removing the field would be a wider API change
+ outside the scope of this fix.
+
+Repo-wide grep after these fixes shows no remaining `.config.get(...)`,
+`.config.getEventManager()`, or `.config.getStorageClient()` calls anywhere
+under `packages/*/src`. Please re-verify once the tree can be type-checked —
+these were reasoned through by reading the new `Configuration` class and
+cross-referencing the already-fixed sibling file, not compiler-verified.
+
+## Update: `lerna>js-yaml` override syntax confirmed correct
+
+A later commit (`36022e7c7`, "pin lerna's minimatch to v3") independently adds
+`"lerna>minimatch": "^3.1.4"` to `pnpm-workspace.yaml`'s `overrides`, with a
+comment explaining lerna's bundled code needs the CJS-style v3 export. This
+confirms the `"packageA>packageB"` scoped-override syntax I used for
+`lerna>js-yaml` (flagged as unverified above) is exactly right — no longer a
+concern.
+
+## `e00aa9419` — DatasetClient/KeyValueStoreClient aligned with Python (rename support removed)
+
+This commit removes `update()` (rename-a-storage support) entirely from
+`DatasetClient`/`KeyValueStoreClient`/`RequestQueueClient` in memory-storage,
+renames `delete()` → `drop()`, and adds `purge()`. Confirmed via
+`packages/types/src/storages.ts`: the interface no longer declares `update()`
+at all, only `drop()`/`purge()` — so removing the implementations was correct,
+not a data-loss risk.
+
+For each of the three resource-client files I dropped the `update()` method
+body along with the `resolveWithinDirectory`/`move`-based rename logic it
+contained, kept the constructor's own `resolveWithinDirectory(...)` call
+(master's path-traversal hardening — confirmed via `git show
+e00aa9419:...` that v4's own version of these files uses a bare `resolve()`
+in the constructor too, so this hardening predates/is independent of this
+commit and must survive it), and adopted `drop()`. Also removed now-dead
+imports (`createKeyList`/`createKeyStringList`/`createLazyIterablePromise` in
+key-value-store.ts — `keys()`/`values()`/`entries()` no longer exist on the
+class; `Readable`, `move`, `StorageTypes` similarly unused post-removal).
+
+**`packages/memory-storage/test/async-iteration.test.ts`**: HEAD's version
+tested `kvStore.keys()`/`.values()`/`.entries()` and imported
+`createLazyIterablePromise` — none of which exist anymore (confirmed via
+grep on the resolved `key-value-store.ts`). Rather than attempt a partial
+merge of an obsolete test suite, took theirs' version of the whole file
+wholesale (`git checkout --theirs`) — it tests the new `getData()`/
+`listKeys()` API directly and is self-contained.
+
+**`packages/memory-storage/test/request-queue/handledRequestCount-should-update.test.ts`**:
+HEAD added two new tests. One ("deleting a request should decrement...")
+calls `requestQueue.deleteRequest(id)`, which doesn't exist on the current
+`RequestQueueClient` (no per-request delete method at all in this interface,
+confirmed via grep of all `async` methods) — dropped, it tests a capability
+that isn't there. The other ("updating an already handled request should not
+increment...") calls `requestQueue.get()` (renamed to `getMetadata()`) but is
+otherwise valid and valuable — it's the regression test for the exact
+`isRequestHandledStateChanging` double-count fix preserved during an earlier
+conflict in this same rebase (commit `20b320add`'s `request-queue.ts`
+conflict). Kept it, renamed `get()` → `getMetadata()`.
+
+**Commit `ebb1b2632` (Introduce IBrowserPool interface) — moved the puppeteer-25
+`page.close()` timeout guard**: master (`1430062c2`) had wrapped
+`page.close()` in `browser-crawler.ts` with a 5s `addTimeoutToPromise` guard
+(puppeteer 25 can hang `page.close()` indefinitely after an aborted
+navigation). This v4 commit replaces that direct `page.close()` call with
+`this.browserPool.closePage(context.page, { error })`, moving page-closing
+into the pool abstraction. To keep both fixes, moved the timeout guard itself
+into `browser-pool.ts`'s `closePage()` method (added `PAGE_CLOSE_TIMEOUT_MILLIS`
+there, imported `addTimeoutToPromise`) rather than doubly closing the page
+from both `browser-crawler.ts` and the pool. Removed the now-dead
+`PAGE_CLOSE_TIMEOUT_MILLIS` constant from `browser-crawler.ts` (no other use).
+Worth double-checking in review that no other caller of `closePage()` relies
+on it settling faster than 5s.
+
+**Found and fixed a real regression in `packages/basic-crawler/src/internals/basic-crawler.ts`
+(surfaced while resolving commit `f1f095913`)**: master (`1430062c2`, via commit `b2296cea7
+fix: Correctly track the number of requests handled by a crawler (#3410)`) turned
+`handledRequestsCount` into a getter derived from `this.stats`, with a setter that
+*throws* if assigned to (property is meant to be read-only now). v4 has the same fix
+under a different hash (`93dda9656`), which is why it's absent from this rebase's
+todo — my rebase base already includes the master version. However, several later
+v4 commits I already applied earlier in this rebase (including `0c1fbcfe7`, the
+IRequestManager/IRequestLoader transition) reintroduced code written against the
+*old* pre-fix API: a `this.handledRequestsCount = 0;` reset in `_rotateRequestQueue`
+(or equivalent), a `_loadHandledRequestCount()` method that assigned to the setter,
+and a call to it from `_init()`. All three would throw at runtime the first time
+they ran (the setter always throws). Removed all three, and removed the
+`this.handledRequestsCount++` this conflict itself reintroduced in the failed-request
+path, matching master's already-established fix. Worth double-checking, once the
+rebase is fully done, that no other spot re-assigns `handledRequestsCount`.
+
+**Commit `f1f095913` (Align RequestQueueClient interface with Python counterpart)
+— ported master-only `addRequestsBatched` features into the new unified
+`request_queue.ts`**: this v4 commit deletes `request_provider.ts` and
+`request_queue_v2.ts` wholesale, replacing them with a single new
+`request_queue.ts` (already present, unconflicted, since it's a new file from
+v4's side). The new file's `addRequestsBatched()` was a rewrite from an older
+point in v4 history and was missing two features that only exist on master
+(`1430062c2`) and are still actively used by already-applied v4 commits later
+in this rebase (`enqueue_links.ts` reads `result.requestsOverLimit`, added by
+master commits `b23319bbe`/`f3d9a7967` — "Prevent accidental request dropping
+with maxRequestsPerCrawl"): the `maxNewRequests` budget/`requestsOverLimit`
+mechanism, and the `MAX_UNPROCESSED_REQUESTS_RETRIES` retry cap (master commit
+`b3170a60c`, prevents infinite retries when the platform permanently rejects a
+request). Neither commit is an ancestor of the current v4-derived HEAD nor
+present in the remaining rebase-todo (likely because they were independently
+backported to master under different hashes than their v4 equivalents,
+`a50afb62d`/`6d4a75e74`, which never made it into this rebase's commit list).
+Ported both pieces verbatim from master's `request_provider.ts` into the new
+`request_queue.ts` (constant, options fields, and the full
+`attemptToAddToQueueAndAddAnyUnprocessed`/`processChunk`/`buildResult` body),
+and restored the two master-only regression tests for them in
+`test/core/storages/request_queue.test.ts` (`addRequestsBatched does not retry
+permanently unprocessed requests forever`, `addRequestsBatched does not
+re-submit already enqueued requests beyond the initial batch (#3120)`).
+
+Also found and dropped a genuine dead/duplicated code block in
+`packages/memory-storage/src/resource-clients/request-queue.ts`'s
+`releaseOwnLocks()`: the conflict's HEAD side had a trailing block referencing
+`requestModel`, `isRequestHandledStateChanging`, `requestWasHandledBeforeUpdate`
+— variables that don't exist anywhere in `releaseOwnLocks()`'s scope. This was
+leftover orphaned content from an old, already-superseded unified `update()`
+method (removed from this file earlier in the rebase, per an earlier entry in
+this log) that got misattached to `releaseOwnLocks()` by an earlier merge in
+this same rebase. `releaseOwnLocks()` is self-contained in both master and v4
+and doesn't touch `handledRequestCount` — deleted the dead block entirely.
+
+Also dropped an orphaned 2-line test fragment in
+`test/core/storages/request_queue.test.ts`: `f1f095913` renamed `'should cache
+new requests locally'` to `'adding the same uniqueKey twice does not duplicate
+and is served from the local cache'` with a substantially rewritten body (a
+rename + rewrite, not a simple edit). Git's merge left the old test's opening
+two lines behind as an unconflicting fragment ahead of the conflict region;
+the renamed test's full new body already landed earlier in the file
+(confirmed via grep, `line 61`). Deleted the orphaned fragment.
+
+**Commit `a9b972294` (Split MemoryStorage into FileSystemStorageClient and
+MemoryStorageClient) — ported the path-traversal hardening into the new
+`@crawlee/fs-storage` package**: this commit splits the old
+`@crawlee/memory-storage` into a pure in-memory client (stays in
+`memory-storage`, drops all filesystem code) and a new `@crawlee/fs-storage`
+package (the file-system-backed implementation, entirely new files, not
+conflicted). The new `fs-storage` resource clients
+(`dataset.ts`/`key-value-store.ts`/`request-queue.ts`) build their storage
+directory as `resolve(baseStorageDirectory, directoryName)` using plain
+`node:path` `resolve()` — this is the exact path-traversal vulnerability
+master's `resolveWithinDirectory()` helper (commit `a04c29766
+fix(memory-storage): prevent storage names from escaping the storage
+directory (#3715)`) fixed for the old combined package. Since `fs-storage` is
+a wholesale new copy of that same pre-fix code (not derived from
+`memory-storage`'s post-fix version), the vulnerability was reintroduced.
+Ported `resolveWithinDirectory` into `packages/fs-storage/src/utils.ts` and
+switched all three resource clients' directory-construction call to use it
+(left the `rm(resolve(this.xDirectory, entry))` calls alone — those resolve
+internally-generated entry names, not user-controlled names/keys, matching
+the scope of the original fix). Resolved the conflicts in the `memory-storage`
+copies of these same three files by taking theirs' side throughout (drop
+`directoryName`/`resolveWithinDirectory`/filesystem imports in favor of the
+new `cacheKey`-only in-memory identity) — `memory-storage` no longer touches
+the filesystem at all post-split, so the hardening there is not applicable
+(it now lives solely in `fs-storage`).
+
+Note: the original `a04c29766` fix's description also mentions hardening
+key-value-store record keys (`setRecord({ key })`), not just storage names.
+I only found and fixed the storage-name/directoryName call site in the new
+`fs-storage` package — worth a follow-up check once the rebase is done to see
+whether record-key path construction (likely inside
+`fs-storage/src/fs/key-value-store/*.ts`) needs the same treatment.
+
+**Follow-up on the above, resolved immediately**: checked
+`packages/fs-storage/src/fs/key-value-store/fs.ts` — it already used
+`resolveWithinDirectory` for its main `update()` path (carried over cleanly
+from an earlier point in this rebase), but its `get()` fallback (the
+no-file-extension retry path) still called bare `resolve(this.storeDirectory,
+this.rawRecord.key)` on a record key — worse, `resolve` wasn't even imported
+in that file anymore, so this line would have been a compile error. Fixed to
+use `resolveWithinDirectory`, matching master. Checked the sibling
+`fs/dataset/fs.ts` and `fs/request-queue/fs.ts` for the same pattern — both
+build their file path from internally-generated, non-user-controlled IDs
+(sequential entity index, hashed request ID), which is exactly what master's
+original fix (`a04c29766`) called out as already safe, so left them as
+plain `resolve()`.
+
+**Commit `3365b2d0e` (Dissolve @crawlee/memory-storage into @crawlee/core)
+— rewrote two stale master-only security-regression tests that referenced a
+long-gone API**: this commit folds `@crawlee/memory-storage` (the pure
+in-memory client) into `@crawlee/core`. Git flagged two master-only test
+files (`packages/memory-storage/test/key-value-store/record-key-path-traversal.test.ts`
+and `packages/memory-storage/test/storage-name-path-traversal.test.ts` — the
+regression tests for the `resolveWithinDirectory` path-traversal fix,
+`a04c29766`) as "file location" conflicts, suggesting they move to
+`packages/core/test/memory-storage/...` alongside the dissolved package's
+other tests. That location is wrong: both tests exercise disk-escape
+prevention (`persistStorage: true`, checking files on disk), but
+`MemoryStorageClient` (the class that lands in `@crawlee/core`) is pure
+in-memory post-split and never touches the filesystem — the vulnerability
+and its fix now live entirely in `@crawlee/fs-storage`. Moved both to
+`packages/fs-storage/test/` instead.
+
+Their content was also fully stale — written against the pre-rebase API
+(`new MemoryStorage(...)`, `.keyValueStores().getOrCreate()`,
+`client.setRecord()`, `client.update({ name })` for renaming) that no longer
+exists anywhere in this branch after ~30 commits of refactors applied earlier
+in this rebase (Python-alignment renames, ServiceLocator, the fs-storage
+split, KVS value-semantics centralization). Rewrote both against the current
+`FileSystemStorageClient` API (`createKeyValueStoreClient({ name })` /
+`createDatasetClient({ name })` / `createRequestQueueClient({ name })`,
+`client.setValue({ key, value, contentType })`, `client.getMetadata()`).
+Dropped the "rename via update rejects escaping names" test in
+`storage-name-path-traversal.test.ts` — `update()`/rename support was removed
+entirely earlier in this rebase (commit `e00aa9419`), so there is no longer
+an operation to test here.
+
+While in there, also found and fixed the same class of staleness in
+`packages/core/test/memory-storage/request-queue/handledRequestCount-should-update.test.ts`
+(this one auto-merged cleanly, no conflict, so it would have silently landed
+broken): its third test called `requestQueue.updateRequest(...)`, an API that
+no longer exists. The equivalent already-migrated test in
+`packages/fs-storage/test/request-queue/handledRequestCount-should-update.test.ts`
+(from an earlier commit in this same rebase) had already dropped this exact
+test for the same reason — did the same here for consistency.
+
+**Commit `8a422628f` (Rewrite the FilesystemStorageClient to use
+apify/crawlee-storage) — took theirs' side wholesale, dropping my manual
+path-traversal hardening in `@crawlee/fs-storage`**: this commit replaces the
+entire hand-rolled TypeScript filesystem implementation (`cache-helpers.ts`,
+`fs/key-value-store/fs.ts`, and the dataset/kvs/request-queue resource
+clients) with thin adapters around a native Rust extension
+(`@crawlee/fs-storage-native`). `cache-helpers.ts` and `fs/key-value-store/fs.ts`
+are deleted outright — accepted the deletion (`git rm`) since their
+responsibilities (directory resolution, path safety, file I/O) move into the
+native library. For the resource-client content conflicts
+(`dataset.ts`/`key-value-store.ts`/`request-queue.ts`/`utils.ts`), took theirs
+wholesale rather than re-porting the `resolveWithinDirectory` hardening
+(documented earlier in this log, under the `a9b972294` and `39f689f33`
+entries) — checked `v4-reverse` first per the established practice for
+architecture-level collisions, and its equivalent files (a later rename to
+`DatasetBackend`/`FileSystemStorageBackend` etc., but structurally identical)
+also drop the manual TS-level hardening entirely, confirming the native
+package is expected to own path safety internally rather than the TS adapter
+layer re-implementing it.
+
+**Follow-up needed once the rebase is done**: I have not verified that
+`@crawlee/fs-storage-native` actually rejects path-traversal storage
+names/record keys the way `resolveWithinDirectory` did. This should be
+checked directly (either by reading the native crate's source if vendored, or
+by a quick escape-attempt test against the new `FileSystemStorageClient`)
+before considering the `a04c29766` security fix's guarantees intact
+post-rewrite.
+
+**Commit `9f94f67e2` (Move status message handling from StorageClient to the
+event system)**: `setStatusMessage()` becomes fully synchronous (emits
+`EventType.STATUS_MESSAGE` directly instead of routing through an async
+storage-client call). This makes master's `await Promise.race([this
+.setStatusMessage(...), sleep(1)])` "flush the HTTP" workaround
+(`7aed264f1 fix: make crawler terminal status message reliably delivered
+(#3733)`) look pointless — there's no longer an in-flight HTTP call for a
+tick to flush. Checked `v4-reverse`'s equivalent spot and it kept the
+`Promise.race(...)` wrapper as-is rather than simplifying to a bare call, so
+I matched that instead of "fixing" it myself. It's harmless either way (still
+awaits one microtask via `Promise.resolve(undefined)` racing `sleep(1)`), but
+worth a final look at whether it should be simplified now that the
+underlying async-flush problem it solved no longer applies in this
+architecture.
+
+**Commit `eec66ca55` (Rename StorageClient to StorageBackend)**: besides the
+3 flagged conflicts (`snapshotter.ts`, `basic_crawler.test.ts`,
+`playwright_crawler.test.ts`), found several unconflicted-but-broken spots
+this rename missed because they were added by v4-only or my-own-authored code
+after this commit's original point in v4 history:
+- `packages/core/src/autoscaling/client_load_signal.ts` still imported/typed
+ against `StorageClient` (removed from `@crawlee/types` entirely) — fixed to
+ `StorageBackend`.
+- `packages/core/src/autoscaling/snapshotter.ts` had 3 more unconflicted
+ `StorageClient`/`getStorageClient` references outside the merge hunk — fixed.
+- `test/core/storages/storage_manager.test.ts` was untouched by any conflict
+ in this entire 130-commit rebase and had gone fully stale: still imported
+ the deleted `MemoryStorageEmulator` helper (removed 2 commits ago in
+ `a2f9b2646`) and called `Configuration.getStorageClient()`, a method that
+ doesn't exist on `Configuration` at all post the earlier ServiceLocator
+ redesign. Rewrote against the current API
+ (`serviceLocator.setStorageBackend(new MemoryStorageBackend())`,
+ `Dataset.open(id, { storageBackend })`).
+- The two path-traversal regression tests I wrote earlier this session
+ (`packages/fs-storage/test/storage-name-path-traversal.test.ts`,
+ `.../key-value-store/record-key-path-traversal.test.ts`) used the
+ pre-rename `FileSystemStorageClient`/`createXClient` names — updated to
+ `FileSystemStorageBackend`/`createXBackend`. Also removed the now-gone
+ `writeMetadata` constructor option and, in the record-key test, stopped
+ asserting raw on-disk file paths — the previous commit's rewrite
+ (`8a422628f`) delegates file naming/encoding to the native
+ `@crawlee/fs-storage-native` client, and its own `special-keys.test.ts`
+ explicitly treats on-disk names as an implementation detail not to assert
+ on. Both tests still assert the actual security property (escaping key/name
+ rejected) through the public API only.
diff --git a/biome.json b/biome.json
deleted file mode 100644
index 8c23acdb00cb..000000000000
--- a/biome.json
+++ /dev/null
@@ -1,28 +0,0 @@
-{
- "formatter": {
- "includes": [
- "**",
- "!**/website/**",
- "!**/packages/**/*/dist/**",
- "!**/package.json",
- "!**/lerna.json",
- "!**/scripts/actions/docker-images/state.json"
- ],
- "formatWithErrors": true
- },
- "javascript": {
- "formatter": {
- "quoteStyle": "single",
- "semicolons": "always",
- "trailingCommas": "all",
- "lineWidth": 120,
- "indentStyle": "space",
- "indentWidth": 4,
- "quoteProperties": "preserve",
- "lineEnding": "lf"
- }
- },
- "linter": {
- "enabled": false
- }
-}
diff --git a/docs/deployment/apify_platform_init_exit.ts b/docs/deployment/apify_platform_init_exit.ts
index 49a10f100f23..8058d81c457f 100644
--- a/docs/deployment/apify_platform_init_exit.ts
+++ b/docs/deployment/apify_platform_init_exit.ts
@@ -13,7 +13,7 @@ const crawler = new CheerioCrawler({
// Add URLs that match the provided pattern.
await enqueueLinks({
- globs: ['https://www.iana.org/*'],
+ include: ['https://www.iana.org/*'],
});
// Save extracted data to dataset.
diff --git a/docs/deployment/apify_platform_main.ts b/docs/deployment/apify_platform_main.ts
index a338047e86ea..507c7fe1c6e2 100644
--- a/docs/deployment/apify_platform_main.ts
+++ b/docs/deployment/apify_platform_main.ts
@@ -12,7 +12,7 @@ await Actor.main(async () => {
// Add URLs that match the provided pattern.
await enqueueLinks({
- globs: ['https://www.iana.org/*'],
+ include: ['https://www.iana.org/*'],
});
// Save extracted data to dataset.
diff --git a/docs/examples/cheerio_crawler.ts b/docs/examples/cheerio_crawler.ts
index 7a308a6f0ba5..d32dc858c69e 100644
--- a/docs/examples/cheerio_crawler.ts
+++ b/docs/examples/cheerio_crawler.ts
@@ -9,7 +9,7 @@ log.setLevel(LogLevel.DEBUG);
// that automatically loads the URLs and parses their HTML using the cheerio library.
const crawler = new CheerioCrawler({
// The crawler downloads and processes the web pages in parallel, with a concurrency
- // automatically managed based on the available system memory and CPU (see AutoscaledPool class).
+ // automatically managed based on the available system memory and CPU (see ConcurrencySystem class).
// Here we define some hard limits for the concurrency.
minConcurrency: 10,
maxConcurrency: 50,
diff --git a/docs/examples/crawl_some_links.mdx b/docs/examples/crawl_some_links.mdx
index fb9cde71600e..b9b7cf85f949 100644
--- a/docs/examples/crawl_some_links.mdx
+++ b/docs/examples/crawl_some_links.mdx
@@ -7,7 +7,7 @@ import RunnableCodeBlock from '@site/src/components/RunnableCodeBlock';
import ApiLink from '@site/src/components/ApiLink';
import CrawlSource from '!!raw-loader!roa-loader!./crawl_some_links.ts';
-This `CheerioCrawler` example uses the `globs` property in the `enqueueLinks()` method to only add links to the `RequestQueue` queue if they match the specified pattern.
+This `CheerioCrawler` example uses the `include` property in the `enqueueLinks()` method to only add links to the `RequestQueue` queue if they match the specified pattern.
{CrawlSource}
diff --git a/docs/examples/crawl_some_links.ts b/docs/examples/crawl_some_links.ts
index 51912bb10f67..111f89165c0f 100644
--- a/docs/examples/crawl_some_links.ts
+++ b/docs/examples/crawl_some_links.ts
@@ -9,7 +9,7 @@ const crawler = new CheerioCrawler({
log.info(request.url);
// Add some links from page to the crawler's RequestQueue
await enqueueLinks({
- globs: ['http?(s)://crawlee.dev/*/*'],
+ include: ['http?(s)://crawlee.dev/*/*'],
});
},
});
diff --git a/docs/examples/file_download.ts b/docs/examples/file_download.ts
index a6b42555e9ba..4ec682ea7002 100644
--- a/docs/examples/file_download.ts
+++ b/docs/examples/file_download.ts
@@ -2,11 +2,11 @@ import { FileDownload } from 'crawlee';
// Create a FileDownload - a custom crawler instance that will download files from URLs.
const crawler = new FileDownload({
- async requestHandler({ body, request, contentType, getKeyValueStore }) {
+ async requestHandler({ request, response, contentType, getKeyValueStore }) {
const url = new URL(request.url);
const kvs = await getKeyValueStore();
- await kvs.setValue(url.pathname.replace(/\//g, '_'), body, { contentType: contentType.type });
+ await kvs.setValue(url.pathname.replace(/\//g, '_'), response.body, { contentType: contentType.type });
},
});
diff --git a/docs/examples/file_download_stream.ts b/docs/examples/file_download_stream.ts
index a7f39a70f59a..9517531b5bd2 100644
--- a/docs/examples/file_download_stream.ts
+++ b/docs/examples/file_download_stream.ts
@@ -1,9 +1,9 @@
-import { pipeline, Transform } from 'stream';
+import { pipeline, Transform } from 'node:stream';
-import { FileDownload, type Log } from 'crawlee';
+import { FileDownload, type CrawleeLogger } from 'crawlee';
// A sample Transform stream logging the download progress.
-function createProgressTracker({ url, log, totalBytes }: { url: URL; log: Log; totalBytes: number }) {
+function createProgressTracker({ url, log, totalBytes }: { url: URL; log: CrawleeLogger; totalBytes: number }) {
let downloadedBytes = 0;
return new Transform({
@@ -23,32 +23,27 @@ function createProgressTracker({ url, log, totalBytes }: { url: URL; log: Log; t
// Create a FileDownload - a custom crawler instance that will download files from URLs.
const crawler = new FileDownload({
- async streamHandler({ stream, request, log, getKeyValueStore }) {
+ async requestHandler({ response, request, log, getKeyValueStore }) {
const url = new URL(request.url);
log.info(`Downloading ${url} to ${url.pathname.replace(/\//g, '_')}...`);
- await new Promise((resolve, reject) => {
- // With the 'response' event, we have received the headers of the response.
- stream.on('response', async (response) => {
- const kvs = await getKeyValueStore();
- await kvs.setValue(
- url.pathname.replace(/\//g, '_'),
- pipeline(
- stream,
- createProgressTracker({ url, log, totalBytes: Number(response.headers['content-length']) }),
- (error) => {
- if (error) reject(error);
- },
- ),
- { contentType: response.headers['content-type'] },
- );
-
- log.info(`Downloaded ${url} to ${url.pathname.replace(/\//g, '_')}.`);
-
- resolve();
- });
- });
+ if (!response.body) return;
+
+ const kvs = await getKeyValueStore();
+ await kvs.setValue(
+ url.pathname.replace(/\//g, '_'),
+ pipeline(
+ response.body,
+ createProgressTracker({ url, log, totalBytes: Number(response.headers.get('content-length')) }),
+ (error) => {
+ if (error) log.error(`Failed to download ${url}: ${error.message}`);
+ },
+ ),
+ response.headers.get('content-type') ? { contentType: response.headers.get('content-type')! } : {},
+ );
+
+ log.info(`Downloaded ${url} to ${url.pathname.replace(/\//g, '_')}.`);
},
});
diff --git a/docs/examples/http_crawler.ts b/docs/examples/http_crawler.ts
index add990c59076..7f661e116dad 100644
--- a/docs/examples/http_crawler.ts
+++ b/docs/examples/http_crawler.ts
@@ -9,7 +9,7 @@ log.setLevel(LogLevel.DEBUG);
// that automatically loads the URLs and saves their HTML.
const crawler = new HttpCrawler({
// The crawler downloads and processes the web pages in parallel, with a concurrency
- // automatically managed based on the available system memory and CPU (see AutoscaledPool class).
+ // automatically managed based on the available system memory and CPU (see ConcurrencySystem class).
// Here we define some hard limits for the concurrency.
minConcurrency: 10,
maxConcurrency: 50,
diff --git a/docs/examples/jsdom_crawler.ts b/docs/examples/jsdom_crawler.ts
index 6db02c4ef62b..ed5b8a8cfd4f 100644
--- a/docs/examples/jsdom_crawler.ts
+++ b/docs/examples/jsdom_crawler.ts
@@ -9,7 +9,7 @@ log.setLevel(LogLevel.DEBUG);
// that automatically loads the URLs and parses their HTML using the jsdom library.
const crawler = new JSDOMCrawler({
// The crawler downloads and processes the web pages in parallel, with a concurrency
- // automatically managed based on the available system memory and CPU (see AutoscaledPool class).
+ // automatically managed based on the available system memory and CPU (see ConcurrencySystem class).
// Here we define some hard limits for the concurrency.
minConcurrency: 10,
maxConcurrency: 50,
diff --git a/docs/examples/puppeteer_recursive_crawl.ts b/docs/examples/puppeteer_recursive_crawl.ts
index ad48b324796b..dde92685ea5f 100644
--- a/docs/examples/puppeteer_recursive_crawl.ts
+++ b/docs/examples/puppeteer_recursive_crawl.ts
@@ -6,7 +6,7 @@ const crawler = new PuppeteerCrawler({
log.info(`Title of ${request.url}: ${title}`);
await enqueueLinks({
- globs: ['http?(s)://www.iana.org/**'],
+ include: ['http?(s)://www.iana.org/**'],
});
},
maxRequestsPerCrawl: 10,
diff --git a/docs/examples/skip-navigation.ts b/docs/examples/skip-navigation.ts
index 0bbde53c1375..867fb473271e 100644
--- a/docs/examples/skip-navigation.ts
+++ b/docs/examples/skip-navigation.ts
@@ -1,17 +1,22 @@
import { PlaywrightCrawler, KeyValueStore } from 'crawlee';
// Create a key value store for all images we find
-const imageStore = await KeyValueStore.open('images');
+const imageStore = await KeyValueStore.open({ name: 'images' });
const crawler = new PlaywrightCrawler({
async requestHandler({ request, page, sendRequest }) {
// The request should have the navigation skipped
if (request.skipNavigation) {
// Request the image and get its buffer back
- const imageResponse = await sendRequest({ responseType: 'buffer' });
-
- // Save the image in the key-value store
- await imageStore.setValue(`${request.userData.key}.png`, imageResponse.body);
+ const imageResponse = await sendRequest();
+
+ // Saves the image in the key-value store.
+ //
+ // Note: For large-scale file downloads, consider using FileDownload crawler:
+ // https://crawlee.dev/js/api/http-crawler/class/FileDownload
+ await imageStore.setValue(`${request.userData.key}.svg`, await imageResponse.bytes(), {
+ contentType: 'image/svg+xml',
+ });
// Prevent executing the rest of the code as we do not need it
return;
diff --git a/docs/experiments/systemInfoV2.mdx b/docs/experiments/systemInfoV2.mdx
deleted file mode 100644
index fb9a481c45f2..000000000000
--- a/docs/experiments/systemInfoV2.mdx
+++ /dev/null
@@ -1,95 +0,0 @@
----
-id: experiments-system-information-v2
-title: System Information V2
-description: Improved autoscaling through cgroup aware metric collection.
----
-
-import ApiLink from '@site/src/components/ApiLink';
-
-:::caution
-
-This is an experimental feature. While we welcome testers, keep in mind that it is currently not recommended to use this in production.
-
-The API is subject to change, and we might introduce breaking changes in the future.
-
-Should you be using this, feel free to open issues on our [GitHub repository](https://github.com/apify/crawlee), and we'll take a look.
-
-:::
-
-Starting with the newest `crawlee` beta, we have introduced a new crawler option that enables an improved metric collection system.
-This new system should collect cpu and memory metrics more accurately in containerised environments by checking for cgroup enforce limits.
-
-## How to enable the experiment
-
-:::note
-
-This example shows how to enable the experiment in the `CheerioCrawler`,
-but you can apply this to any crawler type.
-
-:::
-
-```ts
-import { CheerioCrawler, Configuration } from 'crawlee';
-
-Configuration.set('systemInfoV2', true);
-
-const crawler = new CheerioCrawler({
- async requestHandler({ $, request }) {
- const title = $('title').text();
- console.log(`The title of "${request.url}" is: ${title}.`);
- },
-});
-
-await crawler.run(['https://crawlee.dev']);
-```
-
-## Other changes
-
-:::info
-
-This section is only useful if you're a tinkerer and want to see what's going on under the hood.
-
-:::
-
-The existing solution checked the bare metal metrics for how much cpu and memory was being used and how much headroom was available.
-This is an intuitive solution but unfortunately doesnt account for when there is an external limit on the amount of resources a process can consume.
-This is often the case in containerized environments where each container will have a quota for its cpu and memory usage.
-
-This experiment attempts to address this issue by introducing a new `isContainerized()` utility function and changing the way resources are collected
-when a container is detected.
-
-:::note
-
-This `isContainerized()` function is very similar to the existing `isDocker()` function however for now they both work side by side.
-If this experiment is successful, `isDocker()` may eventually be deprecated in favour of `isContainerized()`.
-
-:::
-
-### Cgroup detection
-
-On linux, to detect if cgroup is available, we check if there is a directory at `/sys/fs/cgroup`.
-If the directory exists, a version of cgroup is installed.
-Next we check the version of cgroup installed by checking for a directory at `/sys/fs/cgroup/memory/`.
-If it exists, cgroup V1 is installed. If it is missing, it is assumed cgroup V2 is installed.
-
-### CPU metric collection
-
-The existing solution worked by checking the fraction of cpu idle ticks to the total number of cpu ticks since the last profile.
-If 100000 ticks elapse and 5000 were idle, the cpu is at 95% utilisation.
-
-In this experiment, the method of cpu load calculation depends on the result of `isContainerized()` or if set, the `CRAWLEE_CONTAINERIZED` environment variable.
-If `isContainerized()` returns true, the new cgroup aware metric collection will be used over the "bare metal" numbers.
-This works by inspecting the `/sys/fs/cgroup/cpuacct/cpuacct.usage`, `/sys/fs/cgroup/cpu/cpu.cfs_quota_us` and `/sys/fs/cgroup/cpu/cpu.cfs_period_us`
-files for cgroup V1 and the `/sys/fs/cgroup/cpu.stat` and `/sys/fs/cgroup/cpu.max` files for cgroup V2.
-The actual cpu usage figure is calculated in the same manner as the "bare metal" figure by comparing the total number of ticks elapsed to the number
-of idle ticks between profiles but by using the figures from the cgroup files.
-If no cgroup quota is enforced, the "bare metal" numbers will be used.
-
-### Memory metric collection
-
-The existing solution was already cgroup aware however an improvement has been made to memory metric collection when running on windows.
-The existing solution used an external package `apify/ps-tree` to find the amount of memory crawlee and any child processes were using.
-On Windows, this package used the depreciated "WMIC" command line utility to determine memory usage.
-
-In this experiment, `apify/ps-tree` has been removed and replaced by the `packages/utils/src/internals/ps-tree.ts` file. This works in much the
-same manner however, instead of using "WMIC", it uses "powershell" to collect the same data.
\ No newline at end of file
diff --git a/docs/guides/avoid_blocking.mdx b/docs/guides/avoid_blocking.mdx
index ed10846f51e6..65cd2fc955ae 100644
--- a/docs/guides/avoid_blocking.mdx
+++ b/docs/guides/avoid_blocking.mdx
@@ -4,6 +4,8 @@ title: Avoid getting blocked
description: How to avoid getting blocked when scraping
---
+import ApiLink from '@site/src/components/ApiLink';
+
import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
import CodeBlock from '@theme/CodeBlock';
@@ -18,6 +20,8 @@ A scraper might get blocked for numerous reasons. Let's narrow it down to the tw
Browser fingerprint is a collection of browser attributes and significant features that can show if our browser is a bot or a real user. Moreover, most browsers have these unique features that allow the website to track the browser even within different IP addresses. This is the main reason why scrapers should change browser fingerprints while doing browser-based scraping. In return, it should significantly reduce the blocking.
+The two are not handled separately. In Crawlee a `Session` ties an IP, a cookie jar, and a fingerprint together into one consistent identity, and the `SessionPool` rotates those identities as a unit — so a fresh fingerprint always arrives with a fresh IP. This guide covers the fingerprint half; see the [session management guide](./session-management) for how to control the rotation, and the [proxy management guide](./proxy-management) for the IP half.
+
## Using browser fingerprints
Changing browser fingerprints can be a tedious job. Luckily, Crawlee provides this feature with zero configuration necessary - the usage of fingerprints is enabled by default and available in `PlaywrightCrawler` and `PuppeteerCrawler`. So whenever we build a scraper that is using one of these crawlers - the fingerprints are going to be generated for the default browser and the operating system out of the box.
@@ -56,9 +60,38 @@ On the contrary, sometimes we want to entirely disable the usage of browser fing
+## Fingerprints for HTTP crawlers
+
+Every session carries a lightweight fingerprint hint — a `browser`, `platform`, and `device` triple — that the request's HTTP client receives and applies on a best-effort basis.
+By default each session is given a realistic, randomized fingerprint (the host operating system as `platform`, with a plausible `browser`/`device` for it), and it rotates with the session just like the IP and cookies do.
+
+How much of the hint is used depends on the client. The [`impit`](impit-http-client) HTTP client maps the session's `browser` hint to a matching TLS and HTTP impersonation profile,
+so the connection's low-level signature lines up with the headers being sent.
+
+The *same* hint also drives browser crawlers, where it seeds the generated browser fingerprint. The hint only fixes the broad strokes — the browser family, operating system, and device — so a session presents a coherent profile, but it does not make the two backends produce byte-identical fingerprints: `impit` and a real browser will still differ in the finer details (a slightly different user-agent string, for example).
+
+You can pin the fingerprint explicitly through `sessionOptions` when you need a specific profile:
+
+```js
+import { CheerioCrawler, SessionPool } from 'crawlee';
+import { ImpitHttpClient } from '@crawlee/impit-client';
+
+const crawler = new CheerioCrawler({
+ httpClient: new ImpitHttpClient(),
+ sessionPool: new SessionPool({
+ sessionOptions: {
+ fingerprint: { browser: 'firefox', platform: 'windows', device: 'desktop' },
+ },
+ }),
+ requestHandler: async ({ $ }) => {
+ // requests impersonate desktop Firefox on Windows
+ },
+});
+```
+
## Camoufox
-For some protections, using our integrated solutions is not enough, one example could be the Cloudflare challenge. For such pages, you can try [Camoufox](https://camoufox.com/), a custom stealthy build of Firefox for web scraping. It might not get you through the challenge automatically, but with our `handleCloudflareChallenge` helper, it should be able to successfully mimic the required user action and get you through it.
+For some protections, using our integrated solutions is not enough, one example could be the Cloudflare challenge. For such pages, you can try [Camoufox](https://camoufox.com/), a custom stealthy build of Firefox for web scraping. It might not get you through the challenge automatically, but with our `handleCloudflareChallengeHook` post-navigation hook, it should be able to successfully mimic the required user action and get you through it. The hook also reloads the page after the challenge clears and propagates the fresh response back into the crawling context.
{PlaywrightCamoufox}
diff --git a/docs/guides/avoid_blocking_camoufox.ts b/docs/guides/avoid_blocking_camoufox.ts
index 131234578e62..01b8f8ca4dff 100644
--- a/docs/guides/avoid_blocking_camoufox.ts
+++ b/docs/guides/avoid_blocking_camoufox.ts
@@ -1,13 +1,9 @@
-import { PlaywrightCrawler } from 'crawlee';
+import { PlaywrightCrawler, handleCloudflareChallengeHook } from 'crawlee';
import { launchOptions } from 'camoufox-js';
import { firefox } from 'playwright';
const crawler = new PlaywrightCrawler({
- postNavigationHooks: [
- async ({ handleCloudflareChallenge }) => {
- await handleCloudflareChallenge();
- },
- ],
+ postNavigationHooks: [handleCloudflareChallengeHook()],
browserPoolOptions: {
// Disable the default fingerprint spoofing to avoid conflicts with Camoufox.
useFingerprints: false,
diff --git a/docs/guides/configuration.mdx b/docs/guides/configuration.mdx
index 597c3dcc2fa4..c51281dfe30b 100644
--- a/docs/guides/configuration.mdx
+++ b/docs/guides/configuration.mdx
@@ -15,13 +15,13 @@ There are three ways of changing the configuration parameters:
- using the `Configuration` class
You could also combine all the above, but you should keep in mind, that the precedence for these 3 options is the following:
-***`crawlee.json`*** < ***constructor options*** < ***environment variables***.
+***constructor options*** > ***environment variables*** > ***`crawlee.json`***.
-`crawlee.json` is a baseline. The options provided in the `Configuration` constructor will override the options provided in the JSON. Environment variables will override both.
+Constructor options have the highest priority. Environment variables override `crawlee.json`. The JSON file serves as a baseline.
## `crawlee.json`
-The first option you could use for configuring Crawlee is `crawlee.json` file. The only thing you need to do is specify the `ConfigurationOptions` in the file, place the file in the root of your project, and Crawlee will use provided options as global configuration.
+The first option you could use for configuring Crawlee is `crawlee.json` file. The only thing you need to do is specify the configuration options in the file, place the file in the root of your project, and Crawlee will use provided options as global configuration. See the `Configuration` class for the full list of supported options.
```json title="crawlee.json"
{
@@ -57,7 +57,7 @@ crawler.router.addDefaultHandler(async ({ request }) => {
await crawler.run(['https://www.example.com/1']);
```
-If you run this example (assuming you placed the `crawlee.json` file with `persistStateIntervalMillis` and `logLevel` specified there in the root of your project), you will find the `SDK_CRAWLER_STATISTICS` file in default Key-Value store,
+If you run this example (assuming you placed the `crawlee.json` file with `persistStateIntervalMillis` and `logLevel` specified there in the root of your project), you will find the `CRAWLEE_CRAWLER_STATISTICS` file in default Key-Value store,
which would show, that there's 1 finished request and crawler runtime was ~10 seconds.
This confirms that the state was persisted after 10 seconds, as it was set in `crawlee.json`.
Besides, you should see `DEBUG` logs in addition to `INFO` ones in your terminal, as `logLevel` was set to `DEBUG` in the `crawlee.json`, meaning Crawlee picked both provided options correctly.
@@ -94,7 +94,6 @@ Storage directories are purged by default. If set to `false` - local storage dir
#### `CRAWLEE_CONTAINERIZED`
-This variable is only effective when the systemInfoV2 experiment is enabled.
Changes how crawlee measures its CPU and Memory usage and limits. If unset, crawlee will determine if it is containerised using common features of containerized environments using the `isContainerized` utility function.
- A file at `/.dockerenv`.
- A file at `/proc/self/cgroup` containing `docker`.
@@ -127,31 +126,35 @@ Enables verbose logging if set to `true`. If not explicitly set to `true` - for
#### `CRAWLEE_MEMORY_MBYTES`
-Sets the amount of system memory in megabytes to be used by the `AutoscaledPool`.
-It is used to limit the number of concurrently running tasks. By default, the max amount of memory
+Sets the amount of system memory in megabytes to be used by the `ConcurrencySystem`.
+It is used to limit the number of concurrently running requests. By default, the max amount of memory
to be used is set to one quarter of total system memory, i.e. on a system with 8192 MB of memory,
the autoscaling feature will only use up to 2048 MB of memory.
## Configuration class
-The last option to adjust Crawlee configuration is to use the `Configuration` class in the code.
+The last option to adjust Crawlee configuration is to use the `Configuration` class in the code. Configuration is immutable — values are set via the constructor and cannot be changed afterwards.
### Global Configuration
-By default, there is a global singleton instance of `Configuration` class, it is used by the crawlers and some other classes that depend on a configurable behavior. In most cases you don't need to adjust any options there, but if needed - you can get access to it via `Configuration.getGlobalConfig()` function. Now you can easily `get` and `set` the `ConfigurationOptions`.
+By default, there is a global singleton instance of `Configuration` class, it is used by the crawlers and some other classes that depend on a configurable behavior. In most cases you don't need to adjust any options there, but if needed - you can access it via `Configuration.getGlobalConfiguration()`, which delegates to the global `serviceLocator` — the single source of truth for Crawlee's shared services (for example the configuration, event manager, storage backend, and logger). You can also reach the same instance directly via `serviceLocator.getConfiguration()` or swap services globally with `serviceLocator.setConfiguration(...)` before any crawler is created. Configuration values are accessible directly as properties on the instance.
```js
import { CheerioCrawler, Configuration, sleep } from 'crawlee';
// Get the global configuration
-const config = Configuration.getGlobalConfig();
-// Set the 'persistStateIntervalMillis' option
-// of global configuration to 10 seconds
-config.set('persistStateIntervalMillis', 10_000);
+const config = Configuration.getGlobalConfiguration();
+// Access configuration values directly as properties
+console.log(config.persistStateIntervalMillis);
-// Note, that we are not passing the configuration to the crawler
-// as it's using the global configuration
-const crawler = new CheerioCrawler();
+// To use custom configuration values, create a new Configuration instance
+const configuration = new Configuration({
+ // Set the 'persistStateIntervalMillis' option to 10 seconds
+ persistStateIntervalMillis: 10_000,
+});
+
+// Pass the configuration to the crawler
+const crawler = new CheerioCrawler({ configuration });
crawler.router.addDefaultHandler(async ({ request }) => {
// For the first request we wait for 5 seconds,
@@ -171,16 +174,14 @@ crawler.router.addDefaultHandler(async ({ request }) => {
await crawler.run(['https://www.example.com/1']);
```
-This is pretty much the same example we used for showing `crawlee.json` usage,
-but now we're using the global configuration, which is the only difference.
-If you run this example - you will find the `SDK_CRAWLER_STATISTICS` file in default Key-Value store as before,
-which would show the same number of finishes requests (one) and the same crawler runtime (~10 seconds).
-This confirms that provided parameters worked: the state was persisted after 10 seconds, as it was set in the global configuration.
+If you run this example - you will find the `CRAWLEE_CRAWLER_STATISTICS` file in default Key-Value store,
+which would show the same number of finished requests (one) and the same crawler runtime (~10 seconds).
+This confirms that provided parameters worked: the state was persisted after 10 seconds, as it was set in the configuration.
:::note
-After running the same example with commented two lines of code related to `Configuration` there will be
-no `SDK_CRAWLER_STATISTICS` file stored in the default Key-Value store:
+After running the same example without the custom configuration, there will be
+no `CRAWLEE_CRAWLER_STATISTICS` file stored in the default Key-Value store:
as we did not change the `persistStateIntervalMillis`, Crawlee used the default value of 60 seconds,
and the crawler was forcefully aborted after ~15 seconds of run time before it persisted the state for the first time.
@@ -188,19 +189,19 @@ and the crawler was forcefully aborted after ~15 seconds of run time before it p
### Custom configuration
-Alternatively, you can create a custom configuration. In this case you need to pass it to the class that is going to use it, e.g. to the crawler. Let's adjust the previous example:
+You can create a custom configuration and pass it to the crawler via the `configuration` option:
```js
import { CheerioCrawler, Configuration, sleep } from 'crawlee';
// Create new configuration
-const config = new Configuration({
+const configuration = new Configuration({
// Set the 'persistStateIntervalMillis' option to 10 seconds
persistStateIntervalMillis: 10_000,
});
-// Now we need to pass the configuration to the crawler
-const crawler = new CheerioCrawler({}, config);
+// Pass the configuration to the crawler
+const crawler = new CheerioCrawler({ configuration });
crawler.router.addDefaultHandler(async ({ request }) => {
// for the first request we wait for 5 seconds,
@@ -221,13 +222,13 @@ await crawler.run(['https://www.example.com/1']);
```
If you run this example - it would work exactly the same as before,
-with the same `SDK_CRAWLER_STATISTICS` file in default Key-Value store after the run,
+with the same `CRAWLEE_CRAWLER_STATISTICS` file in default Key-Value store after the run,
showing the same number of finished requests and the same crawler run time.
:::note
If you would not pass the configuration to the crawler, there again will be
-no `SDK_CRAWLER_STATISTICS` file stored in the default Key-Value store, this time for a different reason though.
+no `CRAWLEE_CRAWLER_STATISTICS` file stored in the default Key-Value store, this time for a different reason though.
Since we did not pass the configuration to the crawler,
the crawler will use the global configuration, which is using the default `persistStateIntervalMillis`.
So again, the run was aborted before the state was persisted for the first time.
diff --git a/docs/guides/custom-http-client/custom-http-client.mdx b/docs/guides/custom-http-client/custom-http-client.mdx
index c593ec3ba239..4e1b9f04c010 100644
--- a/docs/guides/custom-http-client/custom-http-client.mdx
+++ b/docs/guides/custom-http-client/custom-http-client.mdx
@@ -10,14 +10,34 @@ import CodeBlock from '@theme/CodeBlock';
import ImplementationSource from '!!raw-loader!./implementation.ts';
import UsageSource from '!!raw-loader!./usage.ts';
-The `BasicCrawler` class allows you to configure the HTTP client implementation using the `httpClient` constructor option. This might be useful for testing or if you need to swap out the default implementation based on `got-scraping` for something else, such as `curl-impersonate` or `axios`.
+The `BasicCrawler` class allows you to configure the HTTP client implementation using the `httpClient` constructor option. This might be useful for testing or if you need to swap out the default implementation based on `got-scraping` for something else, such as `curl-impersonate`.
-The HTTP client implementation needs to conform to the `BaseHttpClient` interface. For a rough idea on how it might look, see a skeleton implementation that uses the standard `fetch` interface:
+## Built-in HTTP clients
+
+Crawlee provides several HTTP client implementations out of the box:
+
+- **`ImpitHttpClient`** (default) - Uses the `impit` library for making requests that closely mimic browser behavior.
+- **`GotScrapingHttpClient`** - Uses the `got-scraping` library for browser-like requests with support for custom headers, browser fingerprints, and proxies. This was the default HTTP client in Crawlee v3.
+- **`FetchHttpClient`** - Simple implementation using the native `fetch` API (does not support proxies).
+
+## Implementing a custom HTTP client
+
+To create a custom HTTP client, extend the `BaseHttpClient` abstract class from `@crawlee/http-client`. The base class handles common functionality like cookie management, redirect following, session integration, proxy support, and timeout handling.
+
+Your custom implementation only needs to override the `fetch` method to perform the actual network request:
{ImplementationSource}
+By extending `BaseHttpClient`, your implementation automatically gets:
+- Cookie jar management (applying cookies before requests, saving cookies from responses)
+- Automatic redirect following (up to 10 redirects)
+- Session integration (proxy URL and cookies from session)
+- Timeout handling via AbortSignal
+- Proxy URL support
+
You may then instantiate it and pass to a crawler constructor:
{UsageSource}
-Please note that the interface is experimental and it will likely change with Crawlee version 4.
+Alternatively, you can implement the `BaseHttpClient` interface directly if you need full control over all aspects of the HTTP request handling, including cookies, redirects, and sessions. However, this approach requires implementing significantly more logic yourself.
+
diff --git a/docs/guides/custom-http-client/implementation.ts b/docs/guides/custom-http-client/implementation.ts
index 504f0b532f98..aac71784ff7e 100644
--- a/docs/guides/custom-http-client/implementation.ts
+++ b/docs/guides/custom-http-client/implementation.ts
@@ -1,122 +1,14 @@
-import type {
- BaseHttpClient,
- HttpRequest,
- HttpResponse,
- RedirectHandler,
- ResponseTypes,
- StreamingHttpResponse,
-} from '@crawlee/core';
-import { Readable } from 'node:stream';
-
-export class CustomHttpClient implements BaseHttpClient {
- async sendRequest(
- request: HttpRequest,
- ): Promise> {
- const requestHeaders = new Headers();
- for (let [headerName, headerValues] of Object.entries(request.headers ?? {})) {
- if (headerValues === undefined) {
- continue;
- }
-
- if (!Array.isArray(headerValues)) {
- headerValues = [headerValues];
- }
-
- for (const value of headerValues) {
- requestHeaders.append(headerName, value);
- }
- }
-
- const response = await fetch(request.url, {
- method: request.method,
- headers: requestHeaders,
- body: request.body as string, // TODO implement stream/generator handling
- signal: request.signal,
- // TODO implement the rest of request parameters (e.g., timeout, proxyUrl, cookieJar, ...)
- });
-
- const headers: Record = {};
-
- response.headers.forEach((value, headerName) => {
- headers[headerName] = value;
- });
-
- return {
- complete: true,
- request,
- url: response.url,
- statusCode: response.status,
- redirectUrls: [], // TODO you need to handle redirects manually to track them
- headers,
- trailers: {}, // TODO not supported by fetch
- ip: undefined,
- body:
- request.responseType === 'text'
- ? await response.text()
- : request.responseType === 'json'
- ? await response.json()
- : Buffer.from(await response.text()),
- };
- }
-
- async stream(request: HttpRequest, _onRedirect?: RedirectHandler): Promise {
- const fetchResponse = await fetch(request.url, {
- method: request.method,
- headers: new Headers(),
- body: request.body as string, // TODO implement stream/generator handling
- signal: request.signal,
- // TODO implement the rest of request parameters (e.g., timeout, proxyUrl, cookieJar, ...)
- });
-
- const headers: Record = {}; // TODO same as in sendRequest()
-
- async function* read() {
- const reader = fetchResponse.body?.getReader();
-
- const stream = new ReadableStream({
- start(controller) {
- if (!reader) {
- return null;
- }
- return pump();
- function pump(): Promise {
- return reader!.read().then(({ done, value }) => {
- // When no more data needs to be consumed, close the stream
- if (done) {
- controller.close();
- return;
- }
- // Enqueue the next data chunk into our target stream
- controller.enqueue(value);
- return pump();
- });
- }
- },
- });
-
- for await (const chunk of stream) {
- yield chunk;
- }
- }
-
- const response = {
- complete: false,
- request,
- url: fetchResponse.url,
- statusCode: fetchResponse.status,
- redirectUrls: [], // TODO you need to handle redirects manually to track them
- headers,
- trailers: {}, // TODO not supported by fetch
- ip: undefined,
- stream: Readable.from(read()),
- get downloadProgress() {
- return { percent: 0, transferred: 0 }; // TODO track this
- },
- get uploadProgress() {
- return { percent: 0, transferred: 0 }; // TODO track this
- },
- };
-
- return response;
+import { BaseHttpClient, type CustomFetchOptions } from '@crawlee/http-client';
+
+/**
+ * A simple HTTP client implementation using the native `fetch` API.
+ *
+ * Custom implementations only need to override the `fetch` method.
+ */
+export class CustomFetchClient extends BaseHttpClient {
+ protected override async fetch(request: Request, options?: RequestInit & CustomFetchOptions): Promise {
+ // The base class handles cookies, redirects, sessions, and timeouts.
+ // We only need to perform the actual network request here.
+ return fetch(request, options);
}
}
diff --git a/docs/guides/custom-http-client/usage.ts b/docs/guides/custom-http-client/usage.ts
index ebe52c236d3b..28fa63c5802a 100644
--- a/docs/guides/custom-http-client/usage.ts
+++ b/docs/guides/custom-http-client/usage.ts
@@ -1,8 +1,8 @@
import { HttpCrawler } from 'crawlee';
-import { CustomHttpClient } from './implementation.js';
+import { CustomFetchClient } from './implementation.js';
const crawler = new HttpCrawler({
- httpClient: new CustomHttpClient(),
+ httpClient: new CustomFetchClient(),
async requestHandler() {
/* ... */
},
diff --git a/docs/guides/custom-logger/custom-logger.mdx b/docs/guides/custom-logger/custom-logger.mdx
new file mode 100644
index 000000000000..8d024b6cbf64
--- /dev/null
+++ b/docs/guides/custom-logger/custom-logger.mdx
@@ -0,0 +1,88 @@
+---
+id: custom-logger
+title: Custom logger
+description: Use your own logging library (Winston, Pino, etc.) with Crawlee
+---
+
+import ApiLink from '@site/src/components/ApiLink';
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+import CodeBlock from '@theme/CodeBlock';
+
+import WinstonSource from '!!raw-loader!./winston.ts';
+import PinoSource from '!!raw-loader!./pino.ts';
+
+Crawlee uses `@apify/log` as its default logging library, but you can replace it with any logger you prefer, such as Winston or Pino. This is done by implementing a small adapter and passing it to the crawler.
+
+## Creating an adapter
+
+All Crawlee logging goes through the `CrawleeLogger` interface. To plug in your own logger, extend the `BaseCrawleeLogger` abstract class and implement two methods:
+
+- **`logWithLevel(level, message, data)`** — dispatches a log message to your logging library. The `level` parameter uses `LogLevel` constants (`ERROR = 1`, `SOFT_FAIL = 2`, `WARNING = 3`, `INFO = 4`, `DEBUG = 5`, `PERF = 6`). Map these to your logger's native levels. The `message` is a human-readable `string`, and `data` is an optional `Record` with structured context (e.g. `{ url, statusCode }`) — pass it to your logger as metadata or structured fields.
+- **`createChild(options)`** — returns a new child logger instance scoped to a specific component. Crawlee calls this internally to give each subsystem (e.g. `CheerioCrawler`, `AutoscaledPool`, `SessionPool`) its own identifiable logger. The `options` parameter is a `CrawleeLoggerOptions` object with a single field: `prefix` — a string label prepended to each log line from that component.
+
+All other methods (`error`, `warning`, `info`, `debug`, `exception`, `perf`, etc.) are derived automatically from `logWithLevel` — you don't need to implement them.
+
+:::info Level filtering
+
+`logWithLevel()` is called for **every** log message, regardless of the configured level. Level filtering is the responsibility of the underlying logging library (e.g. Winston's `level` option or Pino's `level` setting). This means your adapter doesn't need to check log levels — just forward everything and let the library decide what to output.
+
+:::
+
+## Injecting the logger
+
+There are two ways to inject a custom logger: per-crawler and globally.
+
+### Per-crawler logger
+
+Pass your adapter via the `logger` option in the crawler constructor. When a `logger` is provided, the crawler creates its own isolated `ServiceLocator` instance, so the custom logger is used by all internal components of that crawler (autoscaling, session pool, statistics, etc.):
+
+```ts
+import { CheerioCrawler } from 'crawlee';
+
+const crawler = new CheerioCrawler({
+ logger: new WinstonAdapter(winstonLogger),
+ async requestHandler({ log }) {
+ // `log` is a child of your custom logger, with prefix set to the crawler class name
+ log.info('Hello from my custom logger!');
+ },
+});
+```
+
+The same logger is available as `crawler.log` outside of the request handler, for example when setting up routes.
+
+### Global logger via service locator
+
+Instead of passing the logger to each crawler individually, you can set it globally via the `serviceLocator`. This is useful when you run multiple crawlers and want them all to use the same logging backend:
+
+```ts
+import { serviceLocator, CheerioCrawler, PlaywrightCrawler } from 'crawlee';
+
+// Set the logger globally — must be done before creating any crawlers
+serviceLocator.setLogger(new WinstonAdapter(winstonLogger));
+
+// Both crawlers will use the Winston logger
+const cheerioCrawler = new CheerioCrawler({ /* ... */ });
+const playwrightCrawler = new PlaywrightCrawler({ /* ... */ });
+```
+
+:::warning
+
+`serviceLocator.setLogger()` must be called **before** any crawler is created. Once a logger has been retrieved from the service locator (which happens during crawler construction), it cannot be replaced — an error will be thrown.
+
+:::
+
+## Full examples
+
+
+
+
+{WinstonSource}
+
+
+
+
+{PinoSource}
+
+
+
diff --git a/docs/guides/custom-logger/pino.ts b/docs/guides/custom-logger/pino.ts
new file mode 100644
index 000000000000..2cf60813aa74
--- /dev/null
+++ b/docs/guides/custom-logger/pino.ts
@@ -0,0 +1,48 @@
+import { CheerioCrawler, BaseCrawleeLogger, LogLevel } from 'crawlee';
+import type { CrawleeLogger, CrawleeLoggerOptions } from 'crawlee';
+import pino from 'pino';
+
+// Map Crawlee log levels to Pino levels
+const CRAWLEE_TO_PINO: Record = {
+ [LogLevel.ERROR]: 'error',
+ [LogLevel.SOFT_FAIL]: 'warn',
+ [LogLevel.WARNING]: 'warn',
+ [LogLevel.INFO]: 'info',
+ [LogLevel.DEBUG]: 'debug',
+ [LogLevel.PERF]: 'trace',
+};
+
+class PinoAdapter extends BaseCrawleeLogger {
+ constructor(
+ private logger: pino.Logger,
+ options?: Partial,
+ ) {
+ super(options);
+ }
+
+ logWithLevel(level: number, message: string, data?: Record): void {
+ const pinoLevel = CRAWLEE_TO_PINO[level] ?? 'info';
+ this.logger[pinoLevel as pino.Level](data ?? {}, message);
+ }
+
+ protected createChild(options: Partial): CrawleeLogger {
+ return new PinoAdapter(this.logger.child({ prefix: options.prefix }), { ...this.getOptions(), ...options });
+ }
+}
+
+// Create a Pino logger with your preferred configuration
+const pinoLogger = pino({
+ level: 'debug',
+});
+
+// Pass the adapter to the crawler via the `logger` option
+const crawler = new CheerioCrawler({
+ logger: new PinoAdapter(pinoLogger),
+ async requestHandler({ request, $, log }) {
+ log.info(`Processing ${request.url}`);
+ const title = $('title').text();
+ log.debug('Page title extracted', { title });
+ },
+});
+
+await crawler.run(['https://crawlee.dev']);
diff --git a/docs/guides/custom-logger/winston.ts b/docs/guides/custom-logger/winston.ts
new file mode 100644
index 000000000000..9a967988b193
--- /dev/null
+++ b/docs/guides/custom-logger/winston.ts
@@ -0,0 +1,57 @@
+import { CheerioCrawler, BaseCrawleeLogger, LogLevel } from 'crawlee';
+import type { CrawleeLogger, CrawleeLoggerOptions } from 'crawlee';
+import winston from 'winston';
+
+// Map Crawlee log levels to Winston levels
+const CRAWLEE_TO_WINSTON: Record = {
+ [LogLevel.ERROR]: 'error',
+ [LogLevel.SOFT_FAIL]: 'warn',
+ [LogLevel.WARNING]: 'warn',
+ [LogLevel.INFO]: 'info',
+ [LogLevel.DEBUG]: 'debug',
+ [LogLevel.PERF]: 'debug',
+};
+
+class WinstonAdapter extends BaseCrawleeLogger {
+ constructor(
+ private logger: winston.Logger,
+ options?: Partial,
+ ) {
+ super(options);
+ }
+
+ logWithLevel(level: number, message: string, data?: Record): void {
+ const winstonLevel = CRAWLEE_TO_WINSTON[level] ?? 'info';
+ this.logger.log(winstonLevel, message, data);
+ }
+
+ protected createChild(options: Partial): CrawleeLogger {
+ return new WinstonAdapter(this.logger.child({ prefix: options.prefix }), { ...this.getOptions(), ...options });
+ }
+}
+
+// Create a Winston logger with your preferred configuration
+const winstonLogger = winston.createLogger({
+ level: 'debug',
+ format: winston.format.combine(
+ winston.format.colorize(),
+ winston.format.timestamp(),
+ winston.format.printf(({ level, message, timestamp, prefix }) => {
+ const tag = prefix ? `[${prefix}] ` : '';
+ return `${timestamp} ${level}: ${tag}${message}`;
+ }),
+ ),
+ transports: [new winston.transports.Console()],
+});
+
+// Pass the adapter to the crawler via the `logger` option
+const crawler = new CheerioCrawler({
+ logger: new WinstonAdapter(winstonLogger),
+ async requestHandler({ request, $, log }) {
+ log.info(`Processing ${request.url}`);
+ const title = $('title').text();
+ log.debug('Page title extracted', { title });
+ },
+});
+
+await crawler.run(['https://crawlee.dev']);
diff --git a/docs/guides/http-clients.mdx b/docs/guides/http-clients.mdx
index 9956cb3077a3..ea751e7e9a95 100644
--- a/docs/guides/http-clients.mdx
+++ b/docs/guides/http-clients.mdx
@@ -49,7 +49,7 @@ BaseHttpClient --|> GotScrapingHttpClient
## Switching between HTTP clients
-Crawlee currently provides two main HTTP clients: `GotScrapingHttpClient`, which uses the `got-scraping` library, and `ImpitHttpClient`, which uses the `impit` library. You can switch between them by setting the `BasehttpClient` parameter when initializing a crawler class. The default HTTP client is `GotScrapingHttpClient`. For more details on anti-blocking features, see our [avoid getting blocked guide](./avoid-blocking).
+Crawlee currently provides two main HTTP clients: `GotScrapingHttpClient`, which uses the `got-scraping` library, and `ImpitHttpClient`, which uses the `impit` library. You can switch between them by setting the `BasehttpClient` parameter when initializing a crawler class. The default HTTP client is `GotScrapingHttpClient`. For more details on anti-blocking features, see our [avoid getting blocked guide](./avoid-blocking).
Below are examples of how to configure the HTTP client for the `CheerioCrawler`:
@@ -68,7 +68,7 @@ Below are examples of how to configure the HTTP client for the `GotScrapingHttpClient` is the default HTTP client, it's included with the base Crawlee installation and requires no additional packages.
+Since `GotScrapingHttpClient` is the default HTTP client, it's included with the base Crawlee installation and requires no additional packages.
For `ImpitHttpClient`, you need to install a separate `@crawlee/impit-client` package:
@@ -78,7 +78,7 @@ npm i @crawlee/impit-client
## Creating custom HTTP clients
-Crawlee provides an interface, `BaseHttpClient`, which defines the interface that all HTTP clients must implement. This allows you to create custom HTTP clients tailored to your specific requirements.
+Crawlee provides an interface, `BaseHttpClient`, which defines the interface that all HTTP clients must implement. This allows you to create custom HTTP clients tailored to your specific requirements.
HTTP clients are responsible for several key operations:
@@ -88,10 +88,10 @@ HTTP clients are responsible for several key operations:
- managing proxy configurations,
- connection pooling with timeout management.
-To create a custom HTTP client, you need to implement the `BaseHttpClient` interface. Your implementation must be async-compatible and include proper cleanup and resource management to work seamlessly with Crawlee's concurrent processing model.
+To create a custom HTTP client, you need to implement the `BaseHttpClient` interface. Your implementation must be async-compatible and include proper cleanup and resource management to work seamlessly with Crawlee's concurrent processing model.
## Conclusion
-This guide introduced you to the HTTP clients available in Crawlee and demonstrated how to switch between them, including their installation requirements and usage examples. You also learned about the responsibilities of HTTP clients and how to implement your own custom HTTP client by inheriting from the `BaseHttpClient` base class.
+This guide introduced you to the HTTP clients available in Crawlee and demonstrated how to switch between them, including their installation requirements and usage examples. You also learned about the responsibilities of HTTP clients and how to implement your own custom HTTP client by inheriting from the `BaseHttpClient` base class.
If you have questions or need assistance, feel free to reach out on our [GitHub](https://github.com/apify/crawlee) or join our [Discord community](https://discord.com/invite/jyEM2PRvMU). Happy scraping!
diff --git a/docs/guides/http-clients/cheerio-got-scraping-example.ts b/docs/guides/http-clients/cheerio-got-scraping-example.ts
index a2cab0af3807..7a6b8e6ad24e 100644
--- a/docs/guides/http-clients/cheerio-got-scraping-example.ts
+++ b/docs/guides/http-clients/cheerio-got-scraping-example.ts
@@ -1,4 +1,5 @@
-import { CheerioCrawler, GotScrapingHttpClient } from 'crawlee';
+import { CheerioCrawler } from 'crawlee';
+import { GotScrapingHttpClient } from '@crawlee/got-scraping-client';
const crawler = new CheerioCrawler({
httpClient: new GotScrapingHttpClient(),
diff --git a/docs/guides/impit-http-client/basic-usage.ts b/docs/guides/impit-http-client/basic-usage.ts
index 1a8754c9fa11..51b414913bdc 100644
--- a/docs/guides/impit-http-client/basic-usage.ts
+++ b/docs/guides/impit-http-client/basic-usage.ts
@@ -7,7 +7,7 @@ const crawler = new BasicCrawler({
}),
async requestHandler({ sendRequest, log }) {
const response = await sendRequest();
- log.info('Received response', { statusCode: response.statusCode });
+ log.info('Received response', { status: response.status });
},
});
diff --git a/docs/guides/impit-http-client/impit-http-client.mdx b/docs/guides/impit-http-client/impit-http-client.mdx
index 89c71e82fa5d..5bfca4bf2d09 100644
--- a/docs/guides/impit-http-client/impit-http-client.mdx
+++ b/docs/guides/impit-http-client/impit-http-client.mdx
@@ -11,8 +11,6 @@ import CheerioCrawlerSource from '!!raw-loader!./cheerio-crawler.ts';
import HttpCrawlerSource from '!!raw-loader!./http-crawler.ts';
import AdvancedConfigSource from '!!raw-loader!./advanced-config.ts';
-## Introduction
-
The `ImpitHttpClient` is an HTTP client implementation based on the [Impit](https://github.com/apify/impit) library. It enables browser impersonation for HTTP requests, helping you bypass bot detection systems without running an actual browser.
:::info Successor to got-scraping
diff --git a/docs/guides/parallel-scraping/parallel-scraper.mjs b/docs/guides/parallel-scraping/parallel-scraper.mjs
index 6bee4f4ff13a..3b2fe80a1b13 100644
--- a/docs/guides/parallel-scraping/parallel-scraper.mjs
+++ b/docs/guides/parallel-scraping/parallel-scraper.mjs
@@ -1,5 +1,6 @@
import { fork } from 'node:child_process';
+import { FileSystemStorageBackend } from '@crawlee/fs-storage';
import { Configuration, Dataset, PlaywrightCrawler, log } from 'crawlee';
import { router } from './routes.mjs';
@@ -73,18 +74,21 @@ if (!process.env.IN_WORKER_THREAD) {
// or a configuration option. This is just for show 😈
workerLogger.setLevel(log.LEVELS.DEBUG);
- // Disable the automatic purge on start
- // This is needed when running locally, as otherwise multiple processes will try to clear the default storage (and that will cause clashes)
- Configuration.set('purgeOnStart', false);
-
// Get the request queue
const requestQueue = await getOrInitQueue(false);
- // Configure crawlee to store the worker-specific data in a separate directory (needs to be done AFTER the queue is initialized when running locally)
+ // Disable the automatic purge on start, so we don't lose the queue we prepared
const config = new Configuration({
- storageClientOptions: {
- localDataDirectory: `./storage/worker-${process.env.WORKER_INDEX}`,
- },
+ purgeOnStart: false,
+ });
+
+ // Store the worker's own internal state (its default dataset, key-value store, etc.) in a separate
+ // directory so the workers don't collide with each other. This directory is private to a single
+ // worker, so we set `requestQueueAccess: 'single'` — the concurrency-safe locking only matters for
+ // the shared `shop-urls` queue, which gets its own storage backend in `requestQueue.mjs`.
+ const storageBackend = new FileSystemStorageBackend({
+ localDataDirectory: `./storage/worker-${process.env.WORKER_INDEX}`,
+ requestQueueAccess: 'single',
});
workerLogger.debug('Setting up crawler.');
@@ -94,16 +98,16 @@ if (!process.env.IN_WORKER_THREAD) {
// Instead of the long requestHandler with
// if clauses we provide a router instance.
requestHandler: router,
- // Enable the request locking experiment so that we can actually use the queue.
- // highlight-start
- experiments: {
- requestLocking: true,
- },
// Provide the request queue we've pre-filled in previous steps
+ // highlight-start
requestQueue,
// highlight-end
// Let's also limit the crawler's concurrency, we don't want to overload a single process 🐌
maxConcurrency: 5,
+ // Use the worker-specific, concurrency-safe storage backend we created above
+ // highlight-start
+ storageBackend,
+ // highlight-end
},
config,
);
diff --git a/docs/guides/parallel-scraping/parallel-scraping.mdx b/docs/guides/parallel-scraping/parallel-scraping.mdx
index 5e05532c859b..82c0b3be9231 100644
--- a/docs/guides/parallel-scraping/parallel-scraping.mdx
+++ b/docs/guides/parallel-scraping/parallel-scraping.mdx
@@ -12,12 +12,6 @@ import AdaptedRoutesSource from '!!raw-loader!./adapted-routes.mjs';
import ParallelScraperSource from '!!raw-loader!./parallel-scraper.mjs';
import ModifiedDetailRouteSource from '!!raw-loader!./modified-detail-route.mjs';
-:::warning Experimental features ahead
-
-At the time of writing this guide (December 2023), request locking is still an experimental feature. You can read more about the experiment by visiting the [request locking experiment](../experiments/experiments-request-locking) page.
-
-:::
-
In this guide, we will walk you through how you can turn your single scraper into a scraper that can be parallelized and run in multiple instances. This guide assumes you've read and walked through our [introduction guide](../introduction/setting-up) (or have a fully-fledged scraper already built), but if you haven't done so yet, take a break, go read through all that, and come back. We'll be waiting...
*Oh, you're back already! Let's proceed in making that scraper parallel!*
@@ -66,6 +60,16 @@ The first step in our conversion process will be creating a common file (let's c
The exported function, `getOrInitQueue`, might seem like it does a lot. In essence, it just ensures the request queue is initialized, and if requested, ensures it starts off with an empty state.
+:::caution Make the shared queue concurrency-safe with `requestQueueAccess: 'shared'`
+
+Because every worker process opens this same `shop-urls` queue at the same time, it **must** use the concurrency-safe locking behavior of `FileSystemStorageBackend`. That's why `getOrInitQueue` opens the queue with a storage backend constructed with `requestQueueAccess: 'shared'`.
+
+By default, `FileSystemStorageBackend` assumes it is the *sole* consumer of a queue (`requestQueueAccess: 'single'`). On open it immediately reclaims any requests left *in progress* — great for a single-process crawl recovering after a crash, but disastrous when workers run side by side: each worker would happily grab requests another worker is still processing, so the same URL gets scraped multiple times.
+
+Setting `requestQueueAccess: 'shared'` tells the client to treat an in-progress request as a potential live peer's lock and only reclaim it once the lock expires on the wall clock, so two workers never process the same request at once.
+
+:::
+
### Adapting our previous scraper to enqueue the product URLs to the new queue
In the `src/routes.mjs` file of the scraper we previously built, we have a handler for the `CATEGORY` label. Let's adapt that handler to enqueue the product URLs to the new queue we created.
@@ -128,37 +132,44 @@ This will check how the script is executed as. If this value has _any_ value, it
We use this to ensure the parent process stays alive until all the worker processes exit. Otherwise, the worker processes would just get spawned, and lose the ability to communicate with the parent. You might not need this depending on your use case (maybe you just need to spawn workers and let them process).
-#### What's with all those `Configuration` calls?
+#### What's with all the `Configuration` and storage backend setup?
-There are three steps we want to do for the worker processes:
+There are two things we want to do for the worker processes:
-- ensure the default storages do **not** get purged on start, as otherwise we'd lose the queue we prepared
-- get the queue that supports locking from the same location as the parent process
-- initialize a special storage for worker processes so they do not collide with each other
+- get the shared queue from the same location as the parent process (it already comes with the concurrency-safe storage backend we set up in `requestQueue.mjs`)
+- ensure the default storages do **not** get purged on start, as otherwise we'd lose the queue we prepared, and give each worker its own private storage directory for its internal state so the workers don't collide with each other
In order, that's what these lines do:
```javascript title="src/parallel-scraper.mjs"
-// Disable the automatic purge on start (step 1)
-// This is needed when running locally, as otherwise multiple processes will try to clear the default storage (and that will cause clashes)
-Configuration.set('purgeOnStart', false);
+import { FileSystemStorageBackend } from '@crawlee/fs-storage';
-// Get the request queue from the parent process (step 2)
+// Get the shared request queue from the parent process (step 1)
const requestQueue = await getOrInitQueue(false);
-// Configure crawlee to store the worker-specific data in a separate directory (needs to be done AFTER the queue is initialized when running locally) (step 3)
-const config = new Configuration({
- storageClientOptions: {
- localDataDirectory: `./storage/worker-${process.env.WORKER_INDEX}`,
- },
+// Disable the automatic purge on start, so we don't lose the queue we prepared (step 2)
+const config = new Configuration({ purgeOnStart: false });
+
+// Store the worker's own internal state in a separate directory so workers don't collide (step 2,
+// cont.). This directory is private to a single worker, so we explicitly set
+// `requestQueueAccess: 'single'`.
+const storageBackend = new FileSystemStorageBackend({
+ localDataDirectory: `./storage/worker-${process.env.WORKER_INDEX}`,
+ requestQueueAccess: 'single',
});
```
-#### Enabling the request locking experiment, and telling the crawler to use the worker configuration
+:::note Why no `requestQueueAccess: 'shared'` here?
+
+Each worker's `./storage/worker-N` directory is private to that single worker — nothing else opens it — so the default `requestQueueAccess: 'single'` is exactly right. The concurrency-safe locking only matters for storage that is genuinely shared across processes, which is the `shop-urls` queue in `requestQueue.mjs`, not this per-worker internal state.
+
+:::
+
+#### Telling the crawler to use the worker configuration
-You might have noticed several lines highlighted in the code above. Those show how you can enable the request locking experiment, as well as how you provide the request queue to the crawler. You can read more about the experiment by visiting the [request locking experiment](../experiments/experiments-request-locking) page.
+You might have noticed several lines highlighted in the code above. Those show how you provide the shared request queue to the crawler.
-You might have also noticed we passed in a second parameter to the constructor of the crawler, the `config` variable we created earlier. This is needed to ensure the crawler uses the worker-specific storages for internal states, and that they do not collide with each other.
+You might have also noticed we passed in the `config` and `storageBackend` we created earlier to the crawler. These ensure the crawler uses the worker-specific storages for its own internal state (so the workers do not collide with each other), while still consuming the shared, concurrency-safe `shop-urls` queue we provided explicitly.
#### Why do we use `process.send` instead of `context.pushData`?
diff --git a/docs/guides/parallel-scraping/shared.mjs b/docs/guides/parallel-scraping/shared.mjs
index ff627fdee401..bef086bfb6e1 100644
--- a/docs/guides/parallel-scraping/shared.mjs
+++ b/docs/guides/parallel-scraping/shared.mjs
@@ -1,8 +1,19 @@
-import { RequestQueueV2 } from 'crawlee';
+import { FileSystemStorageBackend } from '@crawlee/fs-storage';
+import { RequestQueue } from 'crawlee';
-// Create the request queue that also supports parallelization
+// The request queue shared by all the parallel workers
let queue;
+// The `shop-urls` queue is opened concurrently by every worker process, so it must use the
+// concurrency-safe locking behavior. With `requestQueueAccess: 'shared'`, a request another worker
+// is still processing is treated as a live peer's lock and is not handed out again until that lock
+// expires — so two workers never scrape the same URL at once. (We point at the `./storage`
+// location, which is where this shared queue lives.)
+const sharedStorageBackend = new FileSystemStorageBackend({
+ localDataDirectory: './storage',
+ requestQueueAccess: 'shared',
+});
+
/**
* @param {boolean} makeFresh Whether the queue should be cleared before returning it
* @returns The queue
@@ -12,11 +23,11 @@ export async function getOrInitQueue(makeFresh = false) {
return queue;
}
- queue = await RequestQueueV2.open('shop-urls');
+ queue = await RequestQueue.open('shop-urls', { storageBackend: sharedStorageBackend });
if (makeFresh) {
await queue.drop();
- queue = await RequestQueueV2.open('shop-urls');
+ queue = await RequestQueue.open('shop-urls', { storageBackend: sharedStorageBackend });
}
return queue;
diff --git a/docs/guides/proxy_management.mdx b/docs/guides/proxy_management.mdx
index 8bf385f1c5b5..bc7253aa6ec9 100644
--- a/docs/guides/proxy_management.mdx
+++ b/docs/guides/proxy_management.mdx
@@ -31,7 +31,7 @@ import InspectionPuppeteerSource from '!!raw-loader!./proxy_management_inspectio
and most effective ways of preventing access to a website. It is therefore paramount for
a good web scraping library to provide easy to use but powerful tools which can work around
IP blocking. The most powerful weapon in our anti IP blocking arsenal is a
-[proxy server](https://en.wikipedia.org/wiki/Proxy_server).
+[proxy server](https://en.wikipedia.org/wiki/Proxy_server).
With Crawlee we can use our own proxy servers or proxy servers acquired from
third-party providers.
@@ -83,7 +83,7 @@ The `ProxyConfiguration` class allows you to provide a custom function to pick a
```javascript
const proxyConfiguration = new ProxyConfiguration({
- newUrlFunction: (sessionId, { request }) => {
+ newUrlFunction: ({ request } = {}) => {
if (request?.url.includes('crawlee.dev')) {
return null; // for crawlee.dev, we don't use a proxy
}
@@ -93,39 +93,10 @@ const proxyConfiguration = new ProxyConfiguration({
});
```
-The `newUrlFunction` receives two parameters - `sessionId` and `options` - and returns a string containing the proxy URL.
-
-The `sessionId` parameter is always provided and allows us to differentiate between different sessions - e.g. when Crawlee recognizes your crawlers are being blocked, it will automatically create a new session with a different id.
+The `newUrlFunction` receives a single optional `options` parameter and returns a string with the proxy URL (or `null` to skip the proxy for the current request).
The `options` parameter is an object containing a `Request`, which is the request that will be made. Note that this object is not always available, for example when we are using the `newUrl` function directly. Your custom function should therefore not rely on the `request` object being present and provide a default behavior when it is not.
-### Tiered proxies
-
-You can also provide a list of proxy tiers to the `ProxyConfiguration` class. This is useful when you want to switch between different proxies automatically based on the blocking behavior of the website.
-
-:::warning
-
-Note that the `tieredProxyUrls` option requires `ProxyConfiguration` to be used from a crawler instance ([see below](#crawler-integration)).
-
-Using this configuration through the `newUrl` calls will not yield the expected results.
-
-:::
-
-```javascript
-const proxyConfiguration = new ProxyConfiguration({
- tieredProxyUrls: [
- [null], // At first, we try to connect without a proxy
- ['http://okay-proxy.com'],
- ['http://slightly-better-proxy.com', 'http://slightly-better-proxy-2.com'],
- ['http://very-good-and-expensive-proxy.com'],
- ]
-});
-```
-
-This configuration will start with no proxy, then switch to `http://okay-proxy.com` if Crawlee recognizes we're getting blocked by the target website. If that proxy is also blocked, we will switch to one of the `slightly-better-proxy` URLs. If those are blocked, we will switch to the `very-good-and-expensive-proxy.com` URL.
-
-Crawlee also periodically probes lower tier proxies to see if they are unblocked, and if they are, it will switch back to them.
-
## Crawler integration
`ProxyConfiguration` integrates seamlessly into `HttpCrawler`, `CheerioCrawler`, `JSDOMCrawler`, `PlaywrightCrawler` and `PuppeteerCrawler`.
@@ -162,9 +133,7 @@ Our crawlers will now use the selected proxies for all connections.
## IP Rotation and session management
-`proxyConfiguration.newUrl()` allows us to pass a `sessionId` parameter. It will then be used to create a `sessionId`-`proxyUrl` pair, and subsequent `newUrl()` calls with the same `sessionId` will always return the same `proxyUrl`. This is extremely useful in scraping, because we want to create the impression of a real user. See the [session management guide](../guides/session-management) and `SessionPool` class for more information on how keeping a real session helps us avoid blocking.
-
-When no `sessionId` is provided, our proxy URLs are rotated round-robin.
+Each call to `proxyConfiguration.newUrl()` generates a new proxy URL. Crawler instances pair these URLs with `Session` instances and rotate those together with browser fingerprints, impersonated headers, and more. This is extremely useful in scraping, because we want to create the impression of a real user. See the [session management guide](../guides/session-management) and `SessionPool` class for more information on how keeping a real session helps us avoid blocking.
@@ -202,7 +171,7 @@ When no `sessionId` is provided, our proxy URLs are rotated round-robin.
## Inspecting current proxy in Crawlers
`HttpCrawler`, `CheerioCrawler`, `JSDOMCrawler`, `PlaywrightCrawler` and `PuppeteerCrawler` grant access to information about the currently used proxy
-in their `requestHandler` using a `proxyInfo` object.
+in their `requestHandler` using a `proxyInfo` object.
With the `proxyInfo` object, we can easily access the proxy URL.
diff --git a/docs/guides/proxy_management_session_cheerio.ts b/docs/guides/proxy_management_session_cheerio.ts
index bb19a5b88d35..1e23ec5d5b86 100644
--- a/docs/guides/proxy_management_session_cheerio.ts
+++ b/docs/guides/proxy_management_session_cheerio.ts
@@ -5,8 +5,7 @@ const proxyConfiguration = new ProxyConfiguration({
});
const crawler = new CheerioCrawler({
- useSessionPool: true,
- persistCookiesPerSession: true,
+ saveResponseCookies: true,
proxyConfiguration,
// ...
});
diff --git a/docs/guides/proxy_management_session_http.ts b/docs/guides/proxy_management_session_http.ts
index c8c289de4877..4677cb946273 100644
--- a/docs/guides/proxy_management_session_http.ts
+++ b/docs/guides/proxy_management_session_http.ts
@@ -5,8 +5,7 @@ const proxyConfiguration = new ProxyConfiguration({
});
const crawler = new HttpCrawler({
- useSessionPool: true,
- persistCookiesPerSession: true,
+ saveResponseCookies: true,
proxyConfiguration,
// ...
});
diff --git a/docs/guides/proxy_management_session_jsdom.ts b/docs/guides/proxy_management_session_jsdom.ts
index 98e71d904070..8162643bd1b3 100644
--- a/docs/guides/proxy_management_session_jsdom.ts
+++ b/docs/guides/proxy_management_session_jsdom.ts
@@ -5,8 +5,7 @@ const proxyConfiguration = new ProxyConfiguration({
});
const crawler = new JSDOMCrawler({
- useSessionPool: true,
- persistCookiesPerSession: true,
+ saveResponseCookies: true,
proxyConfiguration,
// ...
});
diff --git a/docs/guides/proxy_management_session_playwright.ts b/docs/guides/proxy_management_session_playwright.ts
index 70edcb79a033..c137f0191877 100644
--- a/docs/guides/proxy_management_session_playwright.ts
+++ b/docs/guides/proxy_management_session_playwright.ts
@@ -5,8 +5,7 @@ const proxyConfiguration = new ProxyConfiguration({
});
const crawler = new PlaywrightCrawler({
- useSessionPool: true,
- persistCookiesPerSession: true,
+ saveResponseCookies: true,
proxyConfiguration,
// ...
});
diff --git a/docs/guides/proxy_management_session_puppeteer.ts b/docs/guides/proxy_management_session_puppeteer.ts
index fcd1e14427f2..4e21121051a3 100644
--- a/docs/guides/proxy_management_session_puppeteer.ts
+++ b/docs/guides/proxy_management_session_puppeteer.ts
@@ -5,8 +5,7 @@ const proxyConfiguration = new ProxyConfiguration({
});
const crawler = new PuppeteerCrawler({
- useSessionPool: true,
- persistCookiesPerSession: true,
+ saveResponseCookies: true,
proxyConfiguration,
// ...
});
diff --git a/docs/guides/proxy_management_session_standalone.ts b/docs/guides/proxy_management_session_standalone.ts
index bc2010f79b18..dec095d03408 100644
--- a/docs/guides/proxy_management_session_standalone.ts
+++ b/docs/guides/proxy_management_session_standalone.ts
@@ -4,10 +4,4 @@ const proxyConfiguration = new ProxyConfiguration({
/* opts */
});
-const sessionPool = await SessionPool.open({
- /* opts */
-});
-
-const session = await sessionPool.getSession();
-
-const proxyUrl = await proxyConfiguration.newUrl(session.id);
+const proxyUrl = await proxyConfiguration.newUrl();
diff --git a/docs/guides/remote_browser.mdx b/docs/guides/remote_browser.mdx
new file mode 100644
index 000000000000..f02d41be4b64
--- /dev/null
+++ b/docs/guides/remote_browser.mdx
@@ -0,0 +1,70 @@
+---
+id: remote-browser
+title: "Remote browser services"
+sidebar_label: "Remote browsers"
+description: Connect Crawlee crawlers to remote browser services like Browserbase, Browserless, or Steel.
+---
+
+import ApiLink from '@site/src/components/ApiLink';
+import CodeBlock from '@theme/CodeBlock';
+
+import RemoteBrowserConfigSource from '!!raw-loader!./remote_browser_config.ts';
+import RemoteBrowserProviderSource from '!!raw-loader!./remote_browser_provider.ts';
+import RemoteBrowserPuppeteerSource from '!!raw-loader!./remote_browser_puppeteer.ts';
+
+Instead of launching a local browser, Crawlee can connect to a remote browser service like [Browserbase](https://browserbase.com/), [Browserless](https://browserless.io/), [Steel](https://steel.dev/), or any service that exposes a WebSocket/CDP endpoint. The crawler manages session rotation and the request lifecycle the same way it does locally — only the browser itself runs elsewhere.
+
+Use this when you need IPs in specific regions, want to offload CPU/memory from your runner, or need stealth features the service provides.
+
+## How it works
+
+Set the crawler's `remoteBrowser` option with the connection details. The crawler builds a `RemoteBrowserPool` around its own browser plugin, so the connection is always for the matching browser — there's no plugin to construct and no way to mismatch the pool with the crawler. The pool (an `IBrowserPool` wrapping the regular `BrowserPool`) owns everything remote: resolving the endpoint, releasing sessions when browsers close, and capping how many remote browsers run at once.
+
+## Basic usage
+
+The simplest form is a static connection URL. Use this when the service exposes a single endpoint and doesn't need per-session setup.
+
+{RemoteBrowserConfigSource}
+
+`endpoint` can also be a function returning `{ url, context }`, called once per browser launch. Pair it with a `release` callback (it receives the `context`) to clean up sessions on the service side when the browser closes, crashes, or the pool is destroyed.
+
+`maxOpenBrowsers` caps the number of concurrent remote browsers — set it to the service's concurrent-session limit to avoid 429 errors. The pool enforces it inside `newPage()`, which waits for a free slot rather than overshooting.
+
+### Self-hosted
+
+Some services ship a Docker image you can run locally or on your own infrastructure. For example, [Browserless](https://www.browserless.io/) has an open-source Chromium image:
+
+```bash
+docker run -p 3000:3000 -e CONCURRENT=4 ghcr.io/browserless/chromium
+```
+
+Point the pool at the local endpoint with `endpoint: 'ws://localhost:3000'`.
+
+## Custom provider
+
+For services with a session-create / session-release lifecycle, extend `RemoteBrowserProvider` and pass the instance as the pool's `endpoint`. `connect()` runs once per browser launch and returns the connection URL plus an optional `context` object passed back to `release()`. `maxOpenBrowsers` set on the provider is adopted by the pool.
+
+{RemoteBrowserProviderSource}
+
+## Puppeteer
+
+`PuppeteerCrawler` works the same way — build the pool with a `PuppeteerPlugin`. Puppeteer connects over CDP:
+
+{RemoteBrowserPuppeteerSource}
+
+For Playwright you can choose the protocol via the `remoteBrowser.connection.protocol` option: `'cdp'` (default, `connectOverCDP()`) or `'playwright'` (`connect()`, Playwright's own WebSocket protocol).
+
+## Sharing a pool across crawlers
+
+`remoteBrowser` builds a pool the crawler owns and tears down. To share one remote pool across multiple crawlers, construct a `RemoteBrowserPool` yourself and pass it as the `browserPool` option instead — a pool supplied that way is never destroyed by the crawler, so you control its lifecycle. Use `remoteBrowser` *or* `browserPool`, not both.
+
+## Limitations
+
+- **`headless` and `launchOptions` don't apply.** The remote service controls headless mode and browser flags; configure them on the service side.
+- **`useIncognitoPages` is forced to `true`** for Playwright remote connections — `connect()` / `connectOverCDP()` don't accept persistent contexts. For state shared across requests, use the `SessionPool`.
+- **`userDataDir` has no effect** — there's no local profile when the browser runs remotely. Use the service's persistence API (e.g. Browserbase Contexts, Steel Profiles).
+
+## Further reading
+
+- `RemoteBrowserPool` API reference
+- `RemoteBrowserProvider` API reference
diff --git a/docs/guides/remote_browser_config.ts b/docs/guides/remote_browser_config.ts
new file mode 100644
index 000000000000..41f4e0542fe8
--- /dev/null
+++ b/docs/guides/remote_browser_config.ts
@@ -0,0 +1,19 @@
+import { PlaywrightCrawler } from 'crawlee';
+
+const token = process.env.BROWSERLESS_TOKEN!;
+
+const crawler = new PlaywrightCrawler({
+ // Connect to a remote browser instead of launching locally. The crawler builds the right
+ // pool for its browser — you only supply the connection details.
+ remoteBrowser: {
+ endpoint: `wss://production-sfo.browserless.io?token=${token}`,
+ // Optional — respect the service's concurrent session limit.
+ maxOpenBrowsers: 5,
+ },
+ async requestHandler({ page, request, log }) {
+ const title = await page.title();
+ log.info(`${request.loadedUrl} — "${title}"`);
+ },
+});
+
+await crawler.run(['https://crawlee.dev']);
diff --git a/docs/guides/remote_browser_provider.ts b/docs/guides/remote_browser_provider.ts
new file mode 100644
index 000000000000..45594d0fe4f4
--- /dev/null
+++ b/docs/guides/remote_browser_provider.ts
@@ -0,0 +1,46 @@
+import { RemoteBrowserProvider } from '@crawlee/browser-pool';
+import { PlaywrightCrawler } from 'crawlee';
+
+const apiKey = process.env.BROWSERBASE_API_KEY!;
+const projectId = process.env.BROWSERBASE_PROJECT_ID!;
+
+class BrowserbaseProvider extends RemoteBrowserProvider<{ id: string }> {
+ // Respect the service's concurrent session limit to avoid 429s.
+ override maxOpenBrowsers = 5;
+
+ async connect() {
+ const response = await fetch('https://api.browserbase.com/v1/sessions', {
+ method: 'POST',
+ headers: { 'x-bb-api-key': apiKey, 'Content-Type': 'application/json' },
+ body: JSON.stringify({ projectId }),
+ });
+
+ if (!response.ok) {
+ throw new Error(`Failed to create session: ${response.status} ${response.statusText}`);
+ }
+
+ const session = (await response.json()) as { id: string; connectUrl: string };
+ return { url: session.connectUrl, context: { id: session.id } };
+ }
+
+ override async release({ id }: { id: string }) {
+ await fetch(`https://api.browserbase.com/v1/sessions/${id}`, {
+ method: 'POST',
+ headers: { 'x-bb-api-key': apiKey, 'Content-Type': 'application/json' },
+ body: JSON.stringify({ status: 'REQUEST_RELEASE' }),
+ });
+ }
+}
+
+const crawler = new PlaywrightCrawler({
+ // Pass the provider as the `endpoint`; the crawler's pool calls connect()/release() per browser.
+ remoteBrowser: {
+ endpoint: new BrowserbaseProvider(),
+ },
+ async requestHandler({ page, request, log }) {
+ const title = await page.title();
+ log.info(`${request.loadedUrl} — "${title}"`);
+ },
+});
+
+await crawler.run(['https://crawlee.dev']);
diff --git a/docs/guides/remote_browser_puppeteer.ts b/docs/guides/remote_browser_puppeteer.ts
new file mode 100644
index 000000000000..2bfc14be3d65
--- /dev/null
+++ b/docs/guides/remote_browser_puppeteer.ts
@@ -0,0 +1,16 @@
+import { PuppeteerCrawler } from 'crawlee';
+
+const token = process.env.BROWSERLESS_TOKEN!;
+
+const crawler = new PuppeteerCrawler({
+ // PuppeteerCrawler connects over CDP. Same `remoteBrowser` option, matching browser guaranteed.
+ remoteBrowser: {
+ endpoint: `wss://production-sfo.browserless.io?token=${token}`,
+ },
+ async requestHandler({ page, request, log }) {
+ const title = await page.title();
+ log.info(`${request.loadedUrl} — "${title}"`);
+ },
+});
+
+await crawler.run(['https://crawlee.dev']);
diff --git a/docs/guides/request_loaders.mdx b/docs/guides/request_loaders.mdx
new file mode 100644
index 000000000000..d2be9bd6c3e6
--- /dev/null
+++ b/docs/guides/request_loaders.mdx
@@ -0,0 +1,179 @@
+---
+id: request-loaders
+title: Request loaders
+description: How to manage the requests your crawler will go through.
+---
+
+import ApiLink from '@site/src/components/ApiLink';
+
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+import CodeBlock from '@theme/CodeBlock';
+
+import RlBasicSource from '!!raw-loader!./request_loaders_rl_basic.ts';
+import SitemapBasicSource from '!!raw-loader!./request_loaders_sitemap_basic.ts';
+import RlTandemExplicitSource from '!!raw-loader!./request_loaders_rl_tandem_explicit.ts';
+import RlTandemHelperSource from '!!raw-loader!./request_loaders_rl_tandem_helper.ts';
+import SitemapTandemExplicitSource from '!!raw-loader!./request_loaders_sitemap_tandem_explicit.ts';
+import SitemapTandemHelperSource from '!!raw-loader!./request_loaders_sitemap_tandem_helper.ts';
+
+Request loaders extend the functionality of the `RequestQueue`, providing additional tools for managing URLs and requests. If you are new to Crawlee and unfamiliar with the `RequestQueue`, consider starting with the [Request storage](./request-storage) guide first. Request loaders define how requests are fetched and stored, enabling various use cases such as reading URLs from a static list, a sitemap, an external API, or combining multiple sources together.
+
+## Overview
+
+The request loader abstractions are built around two interfaces and a couple of helpers:
+
+- `IRequestLoader`: The base interface for reading requests in a crawl.
+- `IRequestManager`: Extends `IRequestLoader` with write capabilities (adding and reclaiming requests).
+- `RequestManagerTandem`: Combines a read-only `IRequestLoader` with a writable `IRequestManager`.
+
+And the concrete request loader implementations:
+
+- `RequestList`: A lightweight implementation for managing a static list of URLs.
+- `SitemapRequestLoader`: A specialized loader that reads URLs from XML and plain-text sitemaps following the [Sitemaps protocol](https://www.sitemaps.org/protocol.html), with filtering capabilities.
+
+Below is a class diagram that illustrates the relationships between these components and the `RequestQueue`:
+
+```mermaid
+---
+config:
+ class:
+ hideEmptyMembersBox: true
+---
+
+classDiagram
+
+%% ========================
+%% Abstract interfaces
+%% ========================
+
+class IRequestLoader {
+ <>
+ + getTotalCount()
+ + getPendingCount()
+ + getHandledCount()
+ + fetchNextRequest()
+ + markRequestAsHandled()
+ + isEmpty()
+ + isFinished()
+ + toTandem()
+}
+
+class IRequestManager {
+ <>
+ + addRequest()
+ + addRequestsBatched()
+ + reclaimRequest()
+ + purge()
+}
+
+%% ========================
+%% Concrete classes
+%% ========================
+
+class RequestQueue
+
+class RequestList
+
+class SitemapRequestLoader
+
+class RequestManagerTandem
+
+%% ========================
+%% Inheritance arrows
+%% ========================
+
+IRequestLoader <|-- IRequestManager
+IRequestLoader <|.. RequestList
+IRequestLoader <|.. SitemapRequestLoader
+IRequestManager <|.. RequestQueue
+IRequestManager <|.. RequestManagerTandem
+```
+
+:::info Crawler usage
+
+A crawler reads its requests from a single `IRequestManager`, passed via the `requestManager` option. A `RequestQueue` is itself a request manager, so it can be passed directly. A read-only loader (such as `RequestList`) cannot — combine it with a queue into a tandem first, see the [Request manager tandem](#request-manager-tandem) section below.
+
+:::
+
+## Request loaders
+
+The `IRequestLoader` interface defines the foundation for fetching requests during a crawl. It provides methods for basic operations like retrieving the next request, marking requests as handled, and checking whether the loader is empty or finished. It is intentionally **read-only** — it does not allow adding new requests. Concrete implementations such as `RequestList` build on this interface to handle specific scenarios. You can create your own custom loader that reads from an external file, web endpoint, database, or any other data source.
+
+### Request list
+
+The `RequestList` manages a static list of URLs to crawl. The list is created for a single crawler run and, unlike a queue, cannot have requests added to or removed from it after initialization. It can hold a large number of URLs (even millions) with significantly lower overhead than enqueueing them one by one.
+
+Here is a basic example of working with the `RequestList`:
+
+
+ {RlBasicSource}
+
+
+### Sitemap request loader
+
+The `SitemapRequestLoader` is a specialized request loader that reads URLs from sitemaps following the [Sitemaps protocol](https://www.sitemaps.org/protocol.html). It supports both XML and plain-text sitemap formats and is particularly useful when you want to crawl a website systematically by following its sitemap structure. Loading happens in the background, so crawling can start before the sitemap is fully parsed.
+
+:::note
+
+The `SitemapRequestLoader` is designed specifically for sitemaps that follow the standard Sitemaps protocol. HTML pages containing links are not supported by this loader — those should be handled by regular crawlers using the `enqueueLinks` functionality.
+
+:::
+
+The loader supports filtering URLs using glob patterns and regular expressions, allowing you to include or exclude specific types of URLs.
+
+
+ {SitemapBasicSource}
+
+
+## Request managers
+
+The `IRequestManager` interface extends `IRequestLoader` with **write** capabilities. In addition to reading requests, a request manager can add new requests and reclaim failed ones. This is essential for dynamic crawling, where new URLs emerge during the crawl, or when requests fail and need to be retried. The `RequestQueue` is the primary built-in request manager — see the [Request storage](./request-storage) guide for details.
+
+## Request manager tandem
+
+The `RequestManagerTandem` class combines the read-only capabilities of an `IRequestLoader` (like `RequestList`) with the read-write capabilities of an `IRequestManager` (like `RequestQueue`). This is useful when you need to load initial requests from a static source (such as a file, sitemap, or database) and also dynamically add or retry requests during the crawl.
+
+Under the hood, the tandem checks whether the read-only loader still has pending requests. If so, each request from the loader is transferred to the manager (the queue) before being processed. Any newly added or reclaimed requests go directly to the manager side. Because every request passes through the queue, deduplication and retries are handled consistently and a single URL is not crawled multiple times.
+
+The easiest way to build a tandem is the `toTandem()` helper available on the loaders. Called without arguments, it pairs the loader with the default `RequestQueue`; you can also pass a specific request manager to use instead.
+
+### Request list with request queue
+
+This setup is useful when you have a static list of URLs to crawl, but also need to handle dynamic requests discovered during the crawl. Requests from the `RequestList` are processed first by being enqueued into the `RequestQueue`, which handles persistence and retries.
+
+
+
+
+ {RlTandemHelperSource}
+
+
+
+
+ {RlTandemExplicitSource}
+
+
+
+
+### Sitemap request loader with request queue
+
+Similarly, you can combine a `SitemapRequestLoader` with a `RequestQueue`. This is particularly useful when you want to crawl URLs from a sitemap while also handling dynamic requests discovered during the crawl. URLs from the sitemap are processed first by being enqueued into the queue, which handles persistence and retries.
+
+
+
+
+ {SitemapTandemHelperSource}
+
+
+
+
+ {SitemapTandemExplicitSource}
+
+
+
+
+## Conclusion
+
+This guide introduced the request loader abstractions: the read-only `IRequestLoader`, the writable `IRequestManager`, and the `RequestManagerTandem` that combines them, along with the `RequestList` and `SitemapRequestLoader` implementations. You also saw how to pair a loader with a queue using the `toTandem()` helper to handle both static and dynamically discovered requests.
+
+If you have questions or need assistance, feel free to reach out on our [GitHub](https://github.com/apify/crawlee) or join our [Discord community](https://discord.com/invite/jyEM2PRvMU). Happy scraping!
diff --git a/docs/guides/request_loaders_rl_basic.ts b/docs/guides/request_loaders_rl_basic.ts
new file mode 100644
index 000000000000..bea941f734db
--- /dev/null
+++ b/docs/guides/request_loaders_rl_basic.ts
@@ -0,0 +1,15 @@
+import { RequestList } from 'crawlee';
+
+// Open a request list with a static set of URLs.
+// The name is used to persist the list's state in the default key-value store.
+const requestList = await RequestList.open('my-list', [
+ 'https://crawlee.dev/',
+ 'https://crawlee.dev/docs',
+ 'https://crawlee.dev/api',
+]);
+
+// Iterate over the requests manually (a crawler does this for you under the hood).
+for await (const request of requestList) {
+ console.log(request.url);
+ await requestList.markRequestAsHandled(request);
+}
diff --git a/docs/guides/request_loaders_rl_tandem_explicit.ts b/docs/guides/request_loaders_rl_tandem_explicit.ts
new file mode 100644
index 000000000000..8014ddc79337
--- /dev/null
+++ b/docs/guides/request_loaders_rl_tandem_explicit.ts
@@ -0,0 +1,21 @@
+import { CheerioCrawler, RequestList, RequestManagerTandem, RequestQueue } from 'crawlee';
+
+// A static list of URLs to start from (can hold millions of URLs).
+const requestList = await RequestList.open('my-list', ['https://crawlee.dev/', 'https://crawlee.dev/docs']);
+
+// A writable queue that holds requests discovered during the crawl.
+const requestQueue = await RequestQueue.open();
+
+// Combine them: the tandem reads from the list first, transferring each request
+// into the queue, and lets you enqueue new requests during the crawl.
+const requestManager = new RequestManagerTandem(requestList, requestQueue);
+
+const crawler = new CheerioCrawler({
+ requestManager,
+ async requestHandler({ enqueueLinks }) {
+ // Newly discovered links go to the queue side of the tandem.
+ await enqueueLinks();
+ },
+});
+
+await crawler.run();
diff --git a/docs/guides/request_loaders_rl_tandem_helper.ts b/docs/guides/request_loaders_rl_tandem_helper.ts
new file mode 100644
index 000000000000..8637cb510e9a
--- /dev/null
+++ b/docs/guides/request_loaders_rl_tandem_helper.ts
@@ -0,0 +1,17 @@
+import { CheerioCrawler, RequestList } from 'crawlee';
+
+// A static list of URLs to start from.
+const requestList = await RequestList.open('my-list', ['https://crawlee.dev/', 'https://crawlee.dev/docs']);
+
+// `toTandem()` is a shortcut that pairs the loader with a request queue.
+// Without arguments it opens the default `RequestQueue`.
+const requestManager = await requestList.toTandem();
+
+const crawler = new CheerioCrawler({
+ requestManager,
+ async requestHandler({ enqueueLinks }) {
+ await enqueueLinks();
+ },
+});
+
+await crawler.run();
diff --git a/docs/guides/request_loaders_sitemap_basic.ts b/docs/guides/request_loaders_sitemap_basic.ts
new file mode 100644
index 000000000000..29bc87f46334
--- /dev/null
+++ b/docs/guides/request_loaders_sitemap_basic.ts
@@ -0,0 +1,14 @@
+import { SitemapRequestLoader } from 'crawlee';
+
+// Open a sitemap request list. The sitemap is fetched and parsed in the background,
+// so crawling can start before the whole sitemap is loaded.
+const sitemapRequestLoader = await SitemapRequestLoader.open({
+ sitemapUrls: ['https://crawlee.dev/sitemap.xml'],
+ // Optionally filter the URLs read from the sitemap:
+ // include: ['https://crawlee.dev/docs/**'],
+});
+
+for await (const request of sitemapRequestLoader) {
+ console.log(request.url);
+ await sitemapRequestLoader.markRequestAsHandled(request);
+}
diff --git a/docs/guides/request_loaders_sitemap_tandem_explicit.ts b/docs/guides/request_loaders_sitemap_tandem_explicit.ts
new file mode 100644
index 000000000000..48d2f936e9cd
--- /dev/null
+++ b/docs/guides/request_loaders_sitemap_tandem_explicit.ts
@@ -0,0 +1,20 @@
+import { CheerioCrawler, RequestManagerTandem, RequestQueue, SitemapRequestLoader } from 'crawlee';
+
+// Read the initial URLs from a sitemap.
+const sitemapRequestLoader = await SitemapRequestLoader.open({
+ sitemapUrls: ['https://crawlee.dev/sitemap.xml'],
+});
+
+// A writable queue for requests discovered during the crawl.
+const requestQueue = await RequestQueue.open();
+
+const requestManager = new RequestManagerTandem(sitemapRequestLoader, requestQueue);
+
+const crawler = new CheerioCrawler({
+ requestManager,
+ async requestHandler({ enqueueLinks }) {
+ await enqueueLinks();
+ },
+});
+
+await crawler.run();
diff --git a/docs/guides/request_loaders_sitemap_tandem_helper.ts b/docs/guides/request_loaders_sitemap_tandem_helper.ts
new file mode 100644
index 000000000000..bcf1c2ea0715
--- /dev/null
+++ b/docs/guides/request_loaders_sitemap_tandem_helper.ts
@@ -0,0 +1,18 @@
+import { CheerioCrawler, SitemapRequestLoader } from 'crawlee';
+
+// Read the initial URLs from a sitemap.
+const sitemapRequestLoader = await SitemapRequestLoader.open({
+ sitemapUrls: ['https://crawlee.dev/sitemap.xml'],
+});
+
+// Pair the loader with the default `RequestQueue` via the `toTandem()` shortcut.
+const requestManager = await sitemapRequestLoader.toTandem();
+
+const crawler = new CheerioCrawler({
+ requestManager,
+ async requestHandler({ enqueueLinks }) {
+ await enqueueLinks();
+ },
+});
+
+await crawler.run();
diff --git a/docs/guides/request_storage.mdx b/docs/guides/request_storage.mdx
index 8da5489b5faf..42c49ef37356 100644
--- a/docs/guides/request_storage.mdx
+++ b/docs/guides/request_storage.mdx
@@ -14,7 +14,6 @@ import BasicOperationsSource from '!!raw-loader!./request_storage_queue_basic.ts
import CrawlerExplicitSource from '!!raw-loader!./request_storage_queue_crawler_explicit.ts';
import CrawlerSource from '!!raw-loader!./request_storage_queue_crawler.ts';
-import RequestQueueListSource from '!!raw-loader!./request_storage_queue_list.ts';
import RequestQueueAddRequestsSource from '!!raw-loader!./request_storage_queue_only.ts';
Crawlee has several request storage types that are useful for specific tasks. The requests are stored on local disk to a directory defined by the `CRAWLEE_STORAGE_DIR` environment variable. If this variable is not defined, by default Crawlee sets `CRAWLEE_STORAGE_DIR` to `./storage` in the current working directory.
@@ -27,7 +26,7 @@ Each Crawlee project run is associated with a **default request queue**. Typical
In Crawlee, the request queue is represented by the `RequestQueue` class.
-The request queue is managed by `MemoryStorage` class and its data is stored in memory, while also being off-loaded to the local directory specified by the `CRAWLEE_STORAGE_DIR` environment variable as follows:
+By default, the request queue is managed by the `FileSystemStorageBackend` class and its data is stored in the local directory specified by the `CRAWLEE_STORAGE_DIR` environment variable as follows:
```text
{CRAWLEE_STORAGE_DIR}/request_queues/{QUEUE_ID}/entries.json
@@ -67,71 +66,17 @@ The following code demonstrates the usage of the request queue:
To see more detailed example of how to use the request queue with a crawler, see the [Puppeteer Crawler](/js/docs/examples/puppeteer-crawler) example.
-## Request list
+The request queue is not optimized for adding numerous URLs in a single batch — historically, requests were added one by one. To enqueue a large set of initial URLs efficiently, use the `addRequests()` method (or simply pass the URLs to `crawler.run()`), which adds requests in batches:
-The request list is not a storage per se - it represents the list of URLs to crawl that is stored in a crawler run memory (or optionally in default [Key-Value Store](../guides/result-storage#key-value-store) associated with the run, if specified). The list is used for the crawling of a large number of URLs, when we know all the URLs which should be visited by the crawler and no URLs would be added during the run. The URLs can be provided either in code or parsed from a text file hosted on the web.
+
+ {RequestQueueAddRequestsSource}
+
-Request list is created exclusively for the crawler run and only if its usage is explicitly specified in the code. Its usage is optional.
+## Reading requests from other sources
-In Crawlee, the request list is represented by the `RequestList` class.
+Sometimes you don't want to start from a dynamic queue, but from a static list of URLs (for example, parsed from a file) or from a website's sitemap. Crawlee provides **request loaders** for these read-only sources — `RequestList` and `SitemapRequestLoader` — which can be combined with a request queue when you also need to enqueue requests discovered during the crawl.
-The following code demonstrates basic operations of the request list:
-
-```javascript
-import { RequestList, PuppeteerCrawler } from 'crawlee';
-
-// Prepare the sources array with URLs to visit
-const sources = [
- { url: 'http://www.example.com/page-1' },
- { url: 'http://www.example.com/page-2' },
- { url: 'http://www.example.com/page-3' },
-];
-
-// Open the request list.
-// List name is used to persist the sources and the list state in the key-value store
-const requestList = await RequestList.open('my-list', sources);
-
-// The crawler will automatically process requests from the list
-// It's used the same way for Cheerio /Playwright crawlers.
-const crawler = new PuppeteerCrawler({
- requestList,
- async requestHandler({ page, request }) {
- // Process the page (extract data, take page screenshot, etc).
- // No more requests could be added to the request list here
- },
-});
-```
-
-## Which one to choose?
-
-When using Request queue - we would normally have several start URLs (e.g. category pages on e-commerce website) and then recursively add more (e.g. individual item pages) programmatically to the queue, it supports dynamic adding and removing of requests. No more URLs can be added to Request list after its initialization as it is immutable, URLs cannot be removed from the list either.
-
-On the other hand, the Request queue is not optimized for adding or removing numerous URLs in a batch. This is technically possible, but requests are added one by one to the queue, and thus it would take significant time with a larger number of requests. Request list however can contain even millions of URLs, and it would take significantly less time to add them to the list, compared to the queue.
-
-Note that Request queue and Request list can be used together by the same crawler. In such cases, each request from the Request list is enqueued into the Request queue first (to the foremost position in the queue, even if Request queue is not empty) and then consumed from the latter. This is necessary to avoid the same URL being processed more than once (from the list first and then possibly from the queue). In practical terms, such a combination can be useful when there are numerous initial URLs, but more URLs would be added dynamically by the crawler.
-
-:::tip
-
-In Crawlee, there is not much need to combine the request queue together with the request list (although it's technically possible).
-
-Previously there was no way to add the initial requests to the queue in batches (to add an array of requests), i.e. we could have only added the requests one by one to the queue with the help of `addRequest()` function.
-
-However, now we could use the `addRequests()` function, which adds requests in batches. Thus, instead of combining the request queue and the request list, we can use only the request queue for such use-cases now. See the examples below.
-
-:::
-
-
-
-
- {RequestQueueAddRequestsSource}
-
-
-
-
- {RequestQueueListSource}
-
-
-
+See the dedicated [Request loaders](./request-loaders) guide for details on loaders, request managers, and how to combine them with a queue into a `RequestManagerTandem`.
## Cleaning up the storages
@@ -143,4 +88,4 @@ import { purgeDefaultStorages } from 'crawlee';
await purgeDefaultStorages();
```
-Calling this function will clean up the default request storage directory (and also the request list stored in default key-value store). This is a shortcut for running (optional) `purge` method on the `StorageClient` interface, in other words it will call the `purge` method of the underlying storage implementation we are currently using. You can make sure the storage is purged only once for a given execution context if you set `onlyPurgeOnce` to `true` in the `options` object.
+Calling this function will clean up the default request storage directory (and also the request list stored in default key-value store). This is a shortcut for running (optional) `purge` method on the `StorageBackend` interface, in other words it will call the `purge` method of the underlying storage implementation we are currently using. You can make sure the storage is purged only once for a given execution context if you set `onlyPurgeOnce` to `true` in the `options` object.
diff --git a/docs/guides/request_storage_queue_basic.ts b/docs/guides/request_storage_queue_basic.ts
index 66d3d337212d..1555e5bbd595 100644
--- a/docs/guides/request_storage_queue_basic.ts
+++ b/docs/guides/request_storage_queue_basic.ts
@@ -11,7 +11,7 @@ await requestQueue.addRequests([
]);
// Open the named request queue
-const namedRequestQueue = await RequestQueue.open('named-queue');
+const namedRequestQueue = await RequestQueue.open({ name: 'named-queue' });
// Remove the named request queue
await namedRequestQueue.drop();
diff --git a/docs/guides/request_storage_queue_crawler.ts b/docs/guides/request_storage_queue_crawler.ts
index 07af11ffa712..d9c37f57f7de 100644
--- a/docs/guides/request_storage_queue_crawler.ts
+++ b/docs/guides/request_storage_queue_crawler.ts
@@ -4,7 +4,7 @@ import { CheerioCrawler } from 'crawlee';
// It's used the same way for Puppeteer/Playwright crawlers.
const crawler = new CheerioCrawler({
// Note that we're not specifying the requestQueue here
- async requestHandler({ crawler, enqueueLinks }) {
+ async requestHandler({ enqueueLinks }) {
// Add new request to the queue
await crawler.addRequests([{ url: 'https://example.com/new-page' }]);
// Add links found on page to the queue
diff --git a/docs/guides/request_storage_queue_only.ts b/docs/guides/request_storage_queue_only.ts
index 5d9a31379597..3054135504f3 100644
--- a/docs/guides/request_storage_queue_only.ts
+++ b/docs/guides/request_storage_queue_only.ts
@@ -15,7 +15,7 @@ const sources = [
// The crawler will automatically process requests from the queue.
// It's used the same way for Cheerio/Playwright crawlers
const crawler = new PuppeteerCrawler({
- async requestHandler({ crawler, enqueueLinks }) {
+ async requestHandler({ enqueueLinks }) {
// Add new request to the queue
await crawler.addRequests(['http://www.example.com/new-page']);
diff --git a/docs/guides/result_storage.mdx b/docs/guides/result_storage.mdx
index eff313aef81f..b2354166bd51 100644
--- a/docs/guides/result_storage.mdx
+++ b/docs/guides/result_storage.mdx
@@ -8,7 +8,7 @@ import ApiLink from '@site/src/components/ApiLink';
Crawlee has several result storage types that are useful for specific tasks. The data is stored on a local disk to the directory defined by the `CRAWLEE_STORAGE_DIR` environment variable. If this variable is not defined, by default Crawlee sets `CRAWLEE_STORAGE_DIR` to `./storage` in the current working directory.
-Crawlee storage is managed by `MemoryStorage` class. During the crawler run all information is stored in memory, while also being off-loaded to the local files in respective storage type folders.
+By default, Crawlee storage is managed by the `FileSystemStorageBackend` class, which stores all information as local files in the respective storage type folders.
## Key-value store
@@ -110,4 +110,84 @@ import { purgeDefaultStorages } from 'crawlee';
await purgeDefaultStorages();
```
-Calling this function will clean up the default results storage directories except the `INPUT` key in default key-value store directory. This is a shortcut for running (optional) `purge` method on the `StorageClient` interface, in other words it will call the `purge` method of the underlying storage implementation we are currently using. In addition, this method will make sure the storage is purged only once for a given execution context, so it is safe to call it multiple times.
+Calling this function will clean up the default results storage directories except the `INPUT` key in default key-value store directory. This is a shortcut for running (optional) `purge` method on the `StorageBackend` interface, in other words it will call the `purge` method of the underlying storage implementation we are currently using. In addition, this method will make sure the storage is purged only once for a given execution context, so it is safe to call it multiple times.
+
+## Transactional storage
+
+A request handler either finishes, or it doesn't. Without further care, a handler that pushes a few dataset items, updates a key-value record and *then* throws would leave those writes behind — and when the request is retried, they would happen again, duplicating your data.
+
+Crawlee prevents that by wrapping every request in a **storage transaction**. Writes made through the storage classes (`Dataset`, `KeyValueStore`, `RequestQueue`) and the context helpers (`pushData`, `enqueueLinks`, ...) while a request is being handled are recorded rather than applied. When the request handler succeeds, they are replayed into real storage all together; when it throws, they are dropped. A retry therefore never double-writes, and a failed request leaves no partial results behind.
+
+This is **on by default** for every crawler and covers the whole request lifecycle — `preNavigationHooks`, `postNavigationHooks`, `extendContext` and the request handler alike.
+
+### What you can rely on
+
+- **Atomicity per request.** A request's storage writes become visible all together on success, or not at all on failure.
+- **Read-your-own-writes.** A read that follows a write in the same handler sees the written value — `getData()`, `getValue()`, key listings, dataset iteration and `getInfo()` all merge the handler's buffered writes with the real storage contents.
+- **Isolation of uncommitted writes.** One handler's uncommitted writes are invisible to concurrently running handlers — to pass data between handlers, use `useState()` instead of the key-value store.
+- **Fidelity.** Values are captured at write time (via `structuredClone`), so mutating an object after passing it to `pushData()` or `setValue()` affects neither what the handler reads back nor what gets committed.
+
+### What the transaction deliberately does *not* cover
+
+- **Request queue additions (by default).** Adding requests is idempotent (the queue deduplicates by `uniqueKey`), and buffering them would starve the crawl of new work until the handler finishes. New requests are therefore applied immediately (the `writeThrough` policy) and are *not* rolled back when the handler fails. If you need strict all-or-nothing enqueues, opt into buffering:
+
+ ```typescript
+ const crawler = new CheerioCrawler({
+ transactionalStorage: { requestQueue: 'deferred' },
+ // ...
+ });
+ ```
+
+ Under the `deferred` policy, the `requestId` returned by an `addRequests()` call inside a handler is provisional — the real id is assigned by the storage backend when the transaction commits.
+
+- **Shared state.** The object returned by `useState()` / `KeyValueStore.getAutoSavedValue()` is the sanctioned live channel shared by all handlers, and it stays live: mutations of it are **not** rolled back when a handler fails.
+
+- **Cross-storage atomicity.** The commit spans multiple storages and multiple calls; delivery is *at-least-once*. If a commit fails partway, the request fails and is retried, and the retry may re-apply what already landed.
+
+- **Error handlers.** `errorHandler` and `failedRequestHandler` run after the failed request's transaction has been rolled back, so their writes go straight to real storage — that is what error handlers are for.
+
+- **Deferred cleanups.** Callbacks registered with `registerDeferredCleanup()` run after the transaction is closed, so their writes land immediately and are not rolled back. In `AdaptivePlaywrightCrawler` they run once per request handler attempt, so a write there can land twice for one request — push your results from the request handler instead.
+
+### Escape hatches
+
+The feature is escapable at three granularities:
+
+- **Whole feature:** `transactionalStorage: false` on the crawler options disables the mechanism entirely; every storage call behaves exactly as if no transactions existed. (Not supported by `AdaptivePlaywrightCrawler`, which needs per-attempt buffering to work at all.)
+- **Per storage type:** the `transactionalStorage: { requestQueue: 'writeThrough' | 'deferred' }` policy object, described above.
+- **Per call site:** `withDirectStorageAccess()` runs a callback outside the transaction, so its writes land immediately and are never rolled back. Use it for progress files, streaming output, and anything that genuinely must not wait for the handler to succeed:
+
+ ```typescript
+ import { withDirectStorageAccess } from 'crawlee';
+
+ async function requestHandler({ request }) {
+ await withDirectStorageAccess(async () => {
+ const store = await KeyValueStore.open();
+ await store.setValue(`progress-${request.id}`, { startedAt: new Date() });
+ });
+ // ... the rest of the handler is transactional as usual
+ }
+ ```
+
+### Operations that throw inside a transaction
+
+A few operations cannot be buffered, and silently letting them through would produce storage states that no rollback can undo. They throw inside a transaction, with an error pointing at `withDirectStorageAccess()`:
+
+- `Dataset.drop()`, `KeyValueStore.drop()`, `RequestQueue.drop()` and `RequestQueue.purge()`,
+- the request queue processing internals (`fetchNextRequest()`, `markRequestAsHandled()`, `reclaimRequest()`),
+- `KeyValueStore.setValue()` with a **stream** value — a stream can only be consumed once, so it cannot serve both a read within the handler and the commit replay. Write streams under `withDirectStorageAccess()`.
+
+### Programmatic use
+
+Outside a crawler, storage is non-transactional unless you ask for it explicitly:
+
+```typescript
+import { withStorageTransaction, Dataset } from 'crawlee';
+
+await withStorageTransaction(async () => {
+ const dataset = await Dataset.open();
+ await dataset.pushData({ some: 'data' }); // buffered
+ // committed when the callback returns, rolled back if it throws
+});
+```
+
+For full control (e.g. deciding whether to commit only after inspecting the results), use `createStorageTransaction()` and drive `run()` / `commit()` / `rollback()` / `dispose()` yourself. This is exactly what `AdaptivePlaywrightCrawler` does with its per-attempt transactions.
diff --git a/docs/guides/running-in-web-server/web-server.mjs b/docs/guides/running-in-web-server/web-server.mjs
index 7c677db912bd..29e6b27367bd 100644
--- a/docs/guides/running-in-web-server/web-server.mjs
+++ b/docs/guides/running-in-web-server/web-server.mjs
@@ -1,6 +1,6 @@
import { randomUUID } from 'node:crypto';
import { CheerioCrawler, log } from 'crawlee';
-import { createServer } from 'http';
+import { createServer } from 'node:http';
// We will bind an HTTP response that we want to send to the Request.uniqueKey
const requestsToResponses = new Map();
diff --git a/docs/guides/scaling_crawlers.mdx b/docs/guides/scaling_crawlers.mdx
index e296cc90975f..98ab04ad78cc 100644
--- a/docs/guides/scaling_crawlers.mdx
+++ b/docs/guides/scaling_crawlers.mdx
@@ -10,7 +10,10 @@ import CodeBlock from '@theme/CodeBlock';
import MinMaxConcurrencySource from '!!raw-loader!./scaling_crawlers_minMaxConcurrency.ts';
import MaxRequestsPerMinuteSource from '!!raw-loader!./scaling_crawlers_maxRequestsPerMinute.ts';
-import AutoscaledPoolOptionsSource from '!!raw-loader!./scaling_crawlers_autoscaledPoolOptions.ts';
+import ConcurrencySystemSource from '!!raw-loader!./scaling_crawlers_concurrencySystem.ts';
+import LoadSignalsOffSource from '!!raw-loader!./scaling_crawlers_loadSignalsOff.ts';
+import CustomLoadSignalSource from '!!raw-loader!./scaling_crawlers_customLoadSignal.ts';
+import WrapLoadSignalSource from '!!raw-loader!./scaling_crawlers_wrapLoadSignal.ts';
As we build our crawler, we might want to control how many requests we do to the website at a time. Crawlee provides several options to fine tune how many parallel requests should be made at any time, how many requests should be done per minute, and how should scaling work based on the available system resources.
@@ -45,7 +48,7 @@ It's recommended to leave it at the default value that is provided and letting t
## Advanced options
-While the options above should be enough for most users, if we wanted to get super deep into the configuration of the autoscaling pool (the internal utility in Crawlee that helps us allow crawlers to scale up and down), we can do so through the `autoscaledPoolOptions` object available on crawler options.
+While the options above should be enough for most users, if we wanted to get super deep into the configuration of autoscaling (the internal machinery in Crawlee that helps us allow crawlers to scale up and down), we can do so by injecting a pre-configured `ConcurrencySystem` through the `concurrencySystem` crawler option. All the fine-grained scaling configuration lives on that instance (see `ConcurrencySystemOptions`).
:::danger Complex options up ahead!
@@ -53,47 +56,45 @@ This section is super advanced and, unless you test the changes extensively and
:::
-With that warning aside, if we're feeling adventurous, this is how we would pass these options when using a crawler:
+With that warning aside, this is how we pass those options. One thing to watch: the `minConcurrency`/`maxConcurrency`/`maxRequestsPerMinute` shortcuts cannot be combined with an injected system — they configure the default one it replaces, so set those limits on the instance instead.
- {AutoscaledPoolOptionsSource}
+ {ConcurrencySystemSource}
-### `desiredConcurrency`
-
-This option specifies the amount of requests that should be running in parallel at the start of the crawler, assuming there are so many available. It defaults to the same value as `minConcurrency`.
+:::tip Capping the combined concurrency of several crawlers
-### `desiredConcurrencyRatio`
+Injecting the *same* `ConcurrencySystem` instance into several crawlers makes them share a single concurrency budget, capping their combined parallelism instead of letting each crawler scale independently.
-The minimum ratio of concurrency to reach before more scaling up is allowed (a number between `0` and `1`). By default, it is set to `0.95`.
+The budget is shared, not divided: slots go to whoever asks first, so a crawler that fills it early keeps refilling it as its own requests finish, and one that starts later may get a much smaller share of it. If a crawler needs a guaranteed slice, give it its own instance — or implement `IConcurrencySystem`, whose allocation methods are told which crawler is asking, and allocate however we see fit.
-We can think of this as the point where the autoscaling pool can attempt to scale up (or down), monitor if there's any changes, and correct them if necessary.
+:::
-### `scaleUpStepRatio` and `scaleDownStepRatio`
+### `desiredConcurrency`
-These values define the fractional amount of desired concurrency to be added or subtracted as the autoscaling pool scales up or down. Both of these values default to `0.05`.
+This option specifies the amount of requests that should be running in parallel at the start of the crawler, assuming there are so many available. It defaults to the same value as `minConcurrency`.
-Every time the autoscaled pool attempts to scale up or down, this value will be added or subtracted from the current concurrency, and, based on the [`desiredConcurrencyRatio`](#desiredconcurrencyratio) and [`maxConcurrency`](#minconcurrency-and-maxconcurrency), determines how many requests can run concurrently.
+### `desiredConcurrencyRatio`
-### `maybeRunIntervalSecs`
+The minimum ratio of concurrency to reach before more scaling up is allowed (a number between `0` and `1`). By default, it is set to `0.9`.
-Indicates how often the autoscaling pool should check if more requests can be started and, if that's true, starts a new request if there are any available. This value is represented in seconds, and defaults to `0.5`.
+We can think of this as the point where the concurrency system can attempt to scale up (or down), monitor if there's any changes, and correct them if necessary.
-:::info
+### `scaleUpStepRatio` and `scaleDownStepRatio`
-Changing this has no effect for requests that are fired immediately after the previous ones are finished. However, it will influence how fast new requests will be started after the autoscaled pool scales up.
+These values define the fractional amount of desired concurrency to be added or subtracted as the concurrency system scales up or down. Both of these values default to `0.05`.
-:::
+Every time the concurrency system attempts to scale up or down, this value will be added or subtracted from the current concurrency, and, based on the [`desiredConcurrencyRatio`](#desiredconcurrencyratio) and [`maxConcurrency`](#minconcurrency-and-maxconcurrency), determines how many requests can run concurrently.
### `loggingIntervalSecs`
-This option lets us control how often the autoscaled pool should log its current state (the current concurrency ratio, desired ratios, if the system is overloaded and so on).
+This option lets us control how often the concurrency system should log its current state (the current concurrency ratio, desired ratios, if the system is overloaded and so on).
We can disable logging altogether by setting this to `null`. By default, it is set to `60` seconds.
### `autoscaleIntervalSecs`
-This option lets us control how often the autoscaling pool should check if it can and should scale up or down. This value is represented in seconds, and defaults to `10`.
+This option lets us control how often the concurrency system should check if it can and should scale up or down. This value is represented in seconds, and defaults to `10`.
:::tip
@@ -118,3 +119,35 @@ This controls how many total requests can be made per minute. It counts the amou
This option can be set by specifying [`maxRequestsPerMinute`](#maxrequestsperminute) in your crawler options too, as it is a shortcut for visibility and ease of access.
:::
+
+## Load signals
+
+Whether the machine counts as overloaded is decided by *load signals*. Four are built in — memory, event loop, CPU and the storage client's rate-limit errors — and each is configured by its own bag under `loadSignals`, carrying both its limits and the `overloadedRatio` at which it fires. If any signal reports overload, the system is overloaded and concurrency is held down.
+
+A signal we don't want watched at all can be switched **off** with `false`, which stops it being collected as well as evaluated (its entry in the reported status then simply reads as not overloaded):
+
+
+ {LoadSignalsOffSource}
+
+
+We can also watch resources of our own by implementing `LoadSignal` — navigation timeouts or proxy health, say — and passing them in `loadSignals.custom`. The `SnapshotStore` helper does the time-windowed bookkeeping for us:
+
+
+ {CustomLoadSignalSource}
+
+
+Each built-in is a public class too — `MemoryLoadSignal`, `EventLoopLoadSignal`, `CpuLoadSignal`, `ClientLoadSignal` — taking exactly the bag its `loadSignals` key accepts. Constructing one directly is how we *wrap* a built-in rather than reimplement it (to hold its overload verdict for a while after the resource recovers, say): switch the original off with `cpu: false`, and pass a signal that delegates to the instance we built in `custom`.
+
+
+ {WrapLoadSignalSource}
+
+
+:::info Signal names are the keys of the reported status
+
+A signal's `name` is the key its verdict appears under in the reported status, so two signals cannot share one — a duplicate throws at construction time. That is why taking over a built-in name (`memInfo`, `eventLoopInfo`, `cpuInfo`, `clientInfo`) requires switching that built-in off.
+
+:::
+
+### `snapshotHistorySecs` and `currentHistorySecs`
+
+Signals are not read as a single instantaneous measurement but averaged over a window, and there are two: `currentHistorySecs` (default `5`) is the short window that gates whether one more request may start, while `snapshotHistorySecs` (default `30`) is the longer window autoscaling decisions are based on. Dispatch therefore reacts to spikes quickly while scaling stays stable. Both apply to every signal alike, built-in or custom, and signals size their snapshot retention to the wider of the two — so raising `snapshotHistorySecs` is what costs memory.
diff --git a/docs/guides/scaling_crawlers_concurrencySystem.ts b/docs/guides/scaling_crawlers_concurrencySystem.ts
new file mode 100644
index 000000000000..76f1041ddcee
--- /dev/null
+++ b/docs/guides/scaling_crawlers_concurrencySystem.ts
@@ -0,0 +1,19 @@
+import { CheerioCrawler, ConcurrencySystem } from 'crawlee';
+
+// Advanced scaling options live on a pre-configured ConcurrencySystem
+const concurrencySystem = new ConcurrencySystem({
+ // ...
+});
+
+const crawler = new CheerioCrawler({
+ concurrencySystem,
+ // ...
+});
+
+// An injected system's lifecycle is owned by us, not the crawler
+await concurrencySystem.start();
+try {
+ await crawler.run(['https://crawlee.dev']);
+} finally {
+ await concurrencySystem.stop();
+}
diff --git a/docs/guides/scaling_crawlers_customLoadSignal.ts b/docs/guides/scaling_crawlers_customLoadSignal.ts
new file mode 100644
index 000000000000..ee5d093f031e
--- /dev/null
+++ b/docs/guides/scaling_crawlers_customLoadSignal.ts
@@ -0,0 +1,33 @@
+import type { LoadSignal } from 'crawlee';
+import { ConcurrencySystem, SnapshotStore } from 'crawlee';
+
+// The only part that is ours: anything we can poll and reduce to "is this resource in trouble?"
+async function areProxiesStruggling(): Promise {
+ const response = await fetch('https://proxy-monitor.example.com/health');
+
+ return !response.ok;
+}
+
+const store = new SnapshotStore();
+let interval: NodeJS.Timeout;
+
+const proxyHealth: LoadSignal = {
+ name: 'proxyHealth',
+ overloadedRatio: 0.3,
+ async start({ maxSampleWindowMillis }) {
+ // Retain exactly the window we will be sampled over, and drop anything measured before a restart.
+ store.useSampleWindow(maxSampleWindowMillis);
+ store.clear();
+
+ interval = setInterval(async () => {
+ const createdAt = new Date();
+ store.push({ createdAt, isOverloaded: await areProxiesStruggling() }, createdAt);
+ }, 1_000);
+ },
+ async stop() {
+ clearInterval(interval);
+ },
+ getSample: (sampleDurationMillis) => store.getSample(sampleDurationMillis),
+};
+
+const concurrencySystem = new ConcurrencySystem({ loadSignals: { custom: [proxyHealth] } });
diff --git a/docs/guides/scaling_crawlers_loadSignalsOff.ts b/docs/guides/scaling_crawlers_loadSignalsOff.ts
new file mode 100644
index 000000000000..06f43eea755a
--- /dev/null
+++ b/docs/guides/scaling_crawlers_loadSignalsOff.ts
@@ -0,0 +1,9 @@
+import { ConcurrencySystem } from 'crawlee';
+
+const concurrencySystem = new ConcurrencySystem({
+ loadSignals: {
+ // Our storage backend reports no rate-limit statistics, so stop polling it every second.
+ client: false,
+ eventLoop: { maxBlockedMillis: 100 },
+ },
+});
diff --git a/docs/guides/scaling_crawlers_wrapLoadSignal.ts b/docs/guides/scaling_crawlers_wrapLoadSignal.ts
new file mode 100644
index 000000000000..9d5b307c132d
--- /dev/null
+++ b/docs/guides/scaling_crawlers_wrapLoadSignal.ts
@@ -0,0 +1,35 @@
+import type { LoadSignal, LoadSignalStartContext, LoadSnapshot } from 'crawlee';
+import { ConcurrencySystem, CpuLoadSignal } from 'crawlee';
+
+const cooldownMillis = 10_000;
+
+// The built-in still does all the measuring; we only reinterpret what it measured.
+const cpu = new CpuLoadSignal();
+
+const stickyCpu: LoadSignal = {
+ // Taking the built-in's name over is allowed only because we switch the built-in off below.
+ name: cpu.name,
+ overloadedRatio: cpu.overloadedRatio,
+ start: (context: LoadSignalStartContext) => cpu.start(context),
+ stop: () => cpu.stop(),
+ getSample(sampleDurationMillis?: number): LoadSnapshot[] {
+ // Keep reporting overload for a while after the CPU recovers, so that scaling up does not immediately
+ // overload it again.
+ let overloadedUntil = 0;
+
+ return cpu.getSample(sampleDurationMillis).map((snapshot) => {
+ if (snapshot.isOverloaded) {
+ overloadedUntil = +snapshot.createdAt + cooldownMillis;
+ }
+
+ return { ...snapshot, isOverloaded: +snapshot.createdAt < overloadedUntil };
+ });
+ },
+};
+
+const concurrencySystem = new ConcurrencySystem({
+ loadSignals: {
+ cpu: false,
+ custom: [stickyCpu],
+ },
+});
diff --git a/docs/guides/session_management.mdx b/docs/guides/session_management.mdx
index ad0eca80e96e..067e8a9c33de 100644
--- a/docs/guides/session_management.mdx
+++ b/docs/guides/session_management.mdx
@@ -18,18 +18,15 @@ import PlaywrightSource from '!!raw-loader!./session_management_playwright.ts';
import PuppeteerSource from '!!raw-loader!./session_management_puppeteer.ts';
import StandaloneSource from '!!raw-loader!./session_management_standalone.ts';
-`SessionPool` is a class that allows us to handle the rotation of proxy IP addresses along with cookies and other custom settings in Crawlee.
+`SessionPool` manages the rotation of proxy IP addresses, cookies, and browser fingerprints in Crawlee. A single `Session` bundles all the identifying state of one "virtual user" — its cookie jar, its proxy (and therefore its IP), and a fingerprint hint — so that everything that makes a series of requests look like it comes from one person rotates together. When a session gets blocked, the whole bundle is thrown away at once and a fresh identity takes over, rather than reusing a burnt IP with new cookies (or vice versa).
-The main benefit of using Session pool is that we can filter out blocked or non-working proxies,
-so our actor does not retry requests over known blocked/non-working proxies.
-Another benefit of using SessionPool is that we can store information tied tightly to an IP address,
-such as cookies, auth tokens, and particular headers. Having our cookies and other identifiers used only with a specific IP will reduce the chance of being blocked.
-The last but not least benefit is the even rotation of IP addresses - SessionPool picks the session randomly,
-which should prevent burning out a small pool of available IPs.
+The main benefits of the session pool are that it filters out blocked or non-working proxies so the crawler does not keep retrying over them, it keeps identity-bound state (cookies, auth tokens, headers) tied to the IP that obtained it, and it spreads requests across IPs to avoid burning a small pool. The selection strategy is configurable — see [Choosing a rotation strategy](#choosing-a-rotation-strategy) below.
-Check out the [avoid blocking guide](./avoid-blocking) for more information about blocking.
+All crawler instances now require a `SessionPool`. In most cases you do not create one yourself: you just read the `session` from the request handler and let the crawler mark it good or bad for you. You only construct a `SessionPool` explicitly when you want to override its defaults or share one instance across several crawlers.
-Now let's take a look at the examples of how to use Session pool:
+Check out the [avoid blocking guide](./avoid-blocking) for the bigger picture on why blocking happens and how fingerprints fit in.
+
+Now let's take a look at the examples of how to use the session pool:
- with `BasicCrawler`;
- with `HttpCrawler`;
- with `CheerioCrawler`;
@@ -76,6 +73,229 @@ Now let's take a look at the examples of how to use Session pool:
-These are the basics of configuring SessionPool.
-Please, bear in mind that a Session pool needs time to find working IPs and build up the pool,
-so we will probably see a lot of errors until it becomes stabilized.
+These are the basics of configuring the session pool. The rest of this guide covers how to control which session is used, what state it carries, and when it is thrown away.
+
+## How a session is retired
+
+A session stays in the pool and keeps being handed out as long as `isUsable()` returns `true`. It stops being usable — and is dropped from rotation — as soon as any of the following happens:
+
+- its **error score** reaches `maxErrorScore` (default `3`),
+- its **usage count** reaches `maxUsageCount` (default `50`),
+- it is older than `maxAgeSecs` (default `3000` seconds), or
+- it has been explicitly **retired**.
+
+You influence this with three methods on the session. `markGood()` records a successful use — it increments the usage count and heals the error score a little (by `errorScoreDecrement`, default `0.5`). `markBad()` records a failure that *might* be the session's fault and *might* just be bad luck — it raises the error score by one, so a session needs to fail repeatedly before it is dropped. `retire()` drops the session immediately and permanently; this is what you call when you are certain the identity itself is burnt (for example, a `403` response).
+
+The distinction between `markBad()` and `retire()` matters. Use `markBad()` for transient, external problems such as a timeout or a `5XX` response — the IP is probably fine and a couple of retries should not throw it away. Use `retire()` for problems that prove the session is blocked, where reusing it is pointless. Retirement is terminal: once a session is retired, a later `markGood()` will not bring it back.
+
+When using a crawler you rarely call `markGood()` yourself — the crawler calls it automatically after a successful request handler run. You only need to reach for `markBad()` / `retire()` (or let blocked status codes do it for you, see [below](#letting-blocked-responses-retire-sessions)) when you detect a problem the crawler cannot see, such as a "you are blocked" message inside an otherwise `200` response.
+
+## Managing cookies
+
+Every session owns a [`tough-cookie`](https://github.com/salesforce/tough-cookie) cookie jar, reachable as `session.cookieJar`. Cookies arriving in `Set-Cookie` response headers are stored in it automatically — this is controlled by the `saveResponseCookies` crawler option (default `true`) — so they are replayed on every later request that reuses the same session. Set `saveResponseCookies: false` to keep response cookies out of the session jar.
+
+You can also seed or read cookies yourself. `session.setCookie('name=value', url)` adds a single cookie, `session.getCookieString(url)` returns the `Cookie` header value the session would send for that URL, and `session.cookieJar` gives you the full jar for anything more involved.
+
+```js
+const crawler = new CheerioCrawler({
+ requestHandler: async ({ session, request }) => {
+ await session.setCookie('consent=yes', request.url);
+ },
+});
+```
+
+### Cookie precedence and overrides
+
+When an HTTP-based crawler (or a direct `sendRequest` call) builds the outgoing `Cookie` header, it starts from a **base jar** and then overlays any cookies set on the request:
+
+- The base jar is the explicit `cookieJar` passed to `sendRequest` if you provide one, otherwise the session's own cookie jar.
+- A `Cookie` header on the request (`request.headers.Cookie`) is merged on top of that base. A cookie set this way wins over a base-jar cookie of the same name, but it is *not* persisted back into the session.
+
+So a `Cookie` request header always beats the stored cookie of the same name regardless of which jar is the base, while passing an explicit `cookieJar` swaps out the whole base for that single call. To override a single cookie for one request, set it on the request header:
+
+```js
+import { HttpCrawler } from 'crawlee';
+import { CookieJar } from 'tough-cookie';
+
+const crawler = new HttpCrawler({
+ preNavigationHooks: [
+ async ({ request }) => {
+ // wins over any same-named cookie in the session jar, for this request only
+ request.headers = { ...request.headers, Cookie: 'token=override' };
+ },
+ ],
+ requestHandler: async ({ sendRequest }) => {
+ // ...or to fully replace the jar for a single call:
+ const jar = new CookieJar();
+ await jar.setCookie('token=override', 'https://example.com');
+ await sendRequest({ url: 'https://example.com' }, { cookieJar: jar });
+ },
+});
+```
+
+A `Cookie` header you set on a request is always honored — it is never silently overwritten by the session jar.
+
+## Choosing a rotation strategy
+
+The `sessionReuseStrategy` option decides *which* session `getSession()` hands out, and it is the main lever for matching the pool's behavior to a target site. Three strategies are available, each suited to a different use case.
+
+**Maximise IP and fingerprint diversity** — use `'random'` (the default). The pool creates a brand-new session for every request until it reaches `maxPoolSize`, then picks a usable session at random. This spreads traffic as widely as possible across IPs and fingerprints and is the right default for most large crawls.
+
+**Distribute load evenly across sessions** — use `'round-robin'`. Like `random`, the pool fills up to `maxPoolSize` first, but then cycles through sessions in order instead of picking randomly. This is useful when you want every session to do roughly the same amount of work — for example, combined with `maxUsageCount` so all sessions reach their limit and rotate out at about the same time.
+
+**Use a single IP until it breaks** — use `'use-until-failure'`. The pool returns the *same* session on every call and only moves to the next one once the current session is retired. This is the strategy for sites that reward consistency: where switching IP mid-flow looks suspicious, where you have logged in and want to stay logged in, or where you simply want to squeeze a working proxy for as long as it lasts before paying for another.
+
+
+
+
+```js
+import { SessionPool } from 'crawlee';
+
+const sessionPool = new SessionPool({
+ sessionReuseStrategy: 'random',
+});
+```
+
+
+
+
+```js
+import { SessionPool } from 'crawlee';
+
+const sessionPool = new SessionPool({
+ sessionReuseStrategy: 'round-robin',
+ // make every session retire after the same amount of work
+ sessionOptions: { maxUsageCount: 100 },
+});
+```
+
+
+
+
+```js
+import { SessionPool } from 'crawlee';
+
+const sessionPool = new SessionPool({
+ sessionReuseStrategy: 'use-until-failure',
+});
+```
+
+
+
+
+Whichever strategy you pick, you can cap how hard each session works through `sessionOptions`. Set `maxUsageCount` when you know a site starts blocking after roughly _N_ requests from one IP, `maxAgeSecs` when sessions should be cycled on a time basis, and `maxErrorScore` to control how forgiving the pool is about intermittent failures before dropping a session.
+
+```js
+const sessionPool = new SessionPool({
+ maxPoolSize: 25,
+ sessionOptions: {
+ maxAgeSecs: 600,
+ maxUsageCount: 150, // e.g. when you know the site blocks after ~150 requests
+ },
+});
+```
+
+## Letting blocked responses retire sessions
+
+You do not have to inspect every response by hand. Crawlers treat a configurable set of HTTP status codes as proof that a session is blocked and retire it automatically, retrying the request with a fresh session. This is controlled by the `blockedStatusCodes` crawler option (default `[401, 403, 429]`).
+
+```js
+import { CheerioCrawler } from 'crawlee';
+
+const crawler = new CheerioCrawler({
+ // a 403 or 429 will retire the current session and retry on a new one
+ blockedStatusCodes: [403, 429],
+ requestHandler: async ({ session, request }) => {
+ // session is already a working, non-blocked one
+ },
+});
+```
+
+For sites that respond with a `200` page that is actually a bot wall (Cloudflare challenges, Google's rate-limit page), set `retryOnBlocked: true` to have the crawler detect those by content and retry as well. For deeper anti-blocking measures see the [avoid blocking guide](./avoid-blocking).
+
+## Sharing a session pool between crawlers
+
+A `SessionPool` instance can be shared across multiple crawlers by passing the same object to each crawler's `sessionPool` option. This is useful in multi-stage scrapers — for example a fast `CheerioCrawler` that discovers links and a `PlaywrightCrawler` that renders detail pages — where you want both stages to reuse the same proven, non-blocked identities and their cookies instead of each warming up its own pool from scratch.
+
+```js
+import { CheerioCrawler, PlaywrightCrawler, SessionPool } from 'crawlee';
+
+const sessionPool = new SessionPool({ maxPoolSize: 100 });
+
+const listingCrawler = new CheerioCrawler({ sessionPool, requestHandler: async () => { /* ... */ } });
+const detailCrawler = new PlaywrightCrawler({ sessionPool, requestHandler: async () => { /* ... */ } });
+```
+
+A pool you construct yourself is owned by you, not the crawler — the crawler will never tear it down or reset it between runs. Call `teardown()` when you are done with it to persist its final state and stop listening for persistence events.
+
+## Custom session pools
+
+A crawler accepts any object implementing the `ISessionPool` interface as its `sessionPool` option, not just the built-in `SessionPool`. The contract is intentionally tiny — a single `getSession()` / `getSession(id)` method that hands out an `ISession` for a request. This lets you plug in a remote, shared, or database-backed session strategy without subclassing `SessionPool` or copying its internals.
+
+```ts
+import { BasicCrawler, Session, type ISessionPool } from 'crawlee';
+
+class MySessionPool implements ISessionPool {
+ private readonly sessions = new Map();
+
+ async getSession(sessionId?: string): Promise {
+ if (sessionId) {
+ const existing = this.sessions.get(sessionId);
+ return existing?.isUsable() ? existing : undefined;
+ }
+
+ const usable = [...this.sessions.values()].find((s) => s.isUsable());
+ if (usable) return usable;
+
+ const fresh = new Session();
+ this.sessions.set(fresh.id, fresh);
+ return fresh;
+ }
+}
+
+const crawler = new BasicCrawler({
+ sessionPool: new MySessionPool(),
+ requestHandler: async ({ session }) => {
+ // session is a Session instance, use it as usual
+ },
+});
+```
+
+The returned objects just need to implement `ISession` — the crawler only calls `markGood()`, `markBad()`, `retire()`, and reads `cookieJar`, `proxyInfo`, and `fingerprint`, all of which are part of that interface.
+
+## Pinning a request to a specific session
+
+By default the pool decides which session a request gets. Setting `request.sessionId` overrides that and forces the request — and all of its retries — onto the session with that id. You can create a custom named session with `addSession()`, giving each its own proxy, cookies, or fingerprint. Because a session bundles a proxy, this is how you bind specific requests to specific proxies.
+
+One important consequence: if a named session is retired — whether through accumulated `markBad()` calls, hitting `maxUsageCount`, or an explicit `retire()` — any subsequent `getSession(id)` call for that id returns `undefined`.
+The crawler treats that as a `MissingSessionError`, counts it as a regular request error, and retries the request with the same `sessionId`. If the session stays retired, retries keep failing and the request eventually exhausts `maxRequestRetries`.
+When a named session can be retired, handle this in your `errorHandler`: either recreate the session via `addSession()` with the same id, or clear `request.sessionId` to let the pool assign a fresh one.
+
+A common usage pattern is escalating between proxy "tiers": add a cheap session and a premium one, start requests on the cheap session, and reassign `request.sessionId` to the premium one in an `errorHandler` so the retry goes out over the better proxy.
+
+```ts
+import { BasicCrawler, SessionPool } from 'crawlee';
+
+const proxyInfoFromUrl = (proxyUrl: string) => {
+ const { username, password, hostname, port } = new URL(proxyUrl);
+ return { url: proxyUrl, username, password, hostname, port };
+};
+
+const sessionPool = new SessionPool();
+await sessionPool.addSession({ id: 'cheap', proxyInfo: proxyInfoFromUrl('http://cheap-proxy.com') });
+await sessionPool.addSession({ id: 'premium', proxyInfo: proxyInfoFromUrl('http://expensive-proxy.com') });
+
+const crawler = new BasicCrawler({
+ sessionPool,
+ retryOnBlocked: true,
+ requestHandler: async ({ sendRequest, request }) => {
+ await sendRequest({ url: request.url });
+ },
+ errorHandler: async ({ request }) => {
+ request.sessionId = 'premium'; // escalate the retry to the premium proxy
+ },
+});
+
+await crawler.run([{ url: 'https://example.com', sessionId: 'cheap' }]);
+```
+
diff --git a/docs/guides/session_management_basic.ts b/docs/guides/session_management_basic.ts
index c7b7ec37c361..4a9f46748b4d 100644
--- a/docs/guides/session_management_basic.ts
+++ b/docs/guides/session_management_basic.ts
@@ -1,33 +1,29 @@
-import { BasicCrawler, ProxyConfiguration } from 'crawlee';
-import { gotScraping } from 'got-scraping';
+import { BasicCrawler, ProxyConfiguration, SessionPool } from 'crawlee';
+import { Impit } from 'impit';
+import { Cookie } from 'tough-cookie';
const proxyConfiguration = new ProxyConfiguration({
/* opts */
});
const crawler = new BasicCrawler({
- // Activates the Session pool (default is true).
- useSessionPool: true,
// Overrides default Session pool configuration.
- sessionPoolOptions: { maxPoolSize: 100 },
+ sessionPool: new SessionPool({ maxPoolSize: 100 }),
async requestHandler({ request, session }) {
const { url } = request;
- const requestOptions = {
- url,
- // We use session id in order to have the same proxyUrl
- // for all the requests using the same session.
- proxyUrl: await proxyConfiguration.newUrl(session?.id),
- throwHttpErrors: false,
+ const client = new Impit({
+ proxyUrl: await proxyConfiguration.newUrl(),
+ ignoreTlsErrors: true,
headers: {
// If you want to use the cookieJar.
// This way you get the Cookie headers string from session.
- Cookie: session?.getCookieString(url),
+ Cookie: (await session?.cookieJar.getCookieString(url)) ?? '',
},
- };
+ });
let response;
try {
- response = await gotScraping(requestOptions);
+ response = await client.fetch(url);
} catch (e) {
if (e === 'SomeNetworkError') {
// If a network error happens, such as timeout, socket hangup, etc.
@@ -38,10 +34,7 @@ const crawler = new BasicCrawler({
throw e;
}
- // Automatically retires the session based on response HTTP status code.
- session?.retireOnBlockedStatusCodes(response.statusCode);
-
- if (response.body.includes('You are blocked!')) {
+ if ((await response.text()).includes('You are blocked!')) {
// You are sure it is blocked.
// This will throw away the session.
session?.retire();
@@ -51,6 +44,17 @@ const crawler = new BasicCrawler({
// No need to call session.markGood -> BasicCrawler calls it for you.
// If you want to use the CookieJar in session you need.
- session?.setCookiesFromResponse(response);
+ if (response.headers.has('set-cookie')) {
+ const newCookies = response.headers
+ .get('set-cookie')
+ ?.split(';')
+ .map((x) => Cookie.parse(x));
+
+ for (const cookie of newCookies ?? []) {
+ if (cookie) {
+ await session?.cookieJar?.setCookie(cookie, url);
+ }
+ }
+ }
},
});
diff --git a/docs/guides/session_management_cheerio.ts b/docs/guides/session_management_cheerio.ts
index 7f8b2f90a09a..bda80505992d 100644
--- a/docs/guides/session_management_cheerio.ts
+++ b/docs/guides/session_management_cheerio.ts
@@ -1,4 +1,4 @@
-import { CheerioCrawler, ProxyConfiguration } from 'crawlee';
+import { CheerioCrawler, ProxyConfiguration, SessionPool } from 'crawlee';
const proxyConfiguration = new ProxyConfiguration({
/* opts */
@@ -7,13 +7,11 @@ const proxyConfiguration = new ProxyConfiguration({
const crawler = new CheerioCrawler({
// To use the proxy IP session rotation logic, you must turn the proxy usage on.
proxyConfiguration,
- // Activates the Session pool (default is true).
- useSessionPool: true,
// Overrides default Session pool configuration.
- sessionPoolOptions: { maxPoolSize: 100 },
+ sessionPool: new SessionPool({ maxPoolSize: 100 }),
// Set to true if you want the crawler to save cookies per session,
// and set the cookie header to request automatically (default is true).
- persistCookiesPerSession: true,
+ saveResponseCookies: true,
async requestHandler({ session, $ }) {
const title = $('title').text();
diff --git a/docs/guides/session_management_http.ts b/docs/guides/session_management_http.ts
index 9c684bcb0566..bb55dc3e69da 100644
--- a/docs/guides/session_management_http.ts
+++ b/docs/guides/session_management_http.ts
@@ -1,4 +1,4 @@
-import { HttpCrawler, ProxyConfiguration } from 'crawlee';
+import { HttpCrawler, ProxyConfiguration, SessionPool } from 'crawlee';
const proxyConfiguration = new ProxyConfiguration({
/* opts */
@@ -7,15 +7,13 @@ const proxyConfiguration = new ProxyConfiguration({
const crawler = new HttpCrawler({
// To use the proxy IP session rotation logic, you must turn the proxy usage on.
proxyConfiguration,
- // Activates the Session pool (default is true).
- useSessionPool: true,
// Overrides default Session pool configuration.
- sessionPoolOptions: { maxPoolSize: 100 },
+ sessionPool: new SessionPool({ maxPoolSize: 100 }),
// Set to true if you want the crawler to save cookies per session,
// and set the cookie header to request automatically (default is true).
- persistCookiesPerSession: true,
+ saveResponseCookies: true,
async requestHandler({ session, body }) {
- const title = (body as string).match(/(.*?)<\/title>/)?.[1];
+ const title = /(.*?)<\/title>/.exec(body as string)?.[1];
if (title === 'Blocked') {
session?.retire();
diff --git a/docs/guides/session_management_jsdom.ts b/docs/guides/session_management_jsdom.ts
index ef55b6632640..ee8e7cfffda3 100644
--- a/docs/guides/session_management_jsdom.ts
+++ b/docs/guides/session_management_jsdom.ts
@@ -1,4 +1,4 @@
-import { JSDOMCrawler, ProxyConfiguration } from 'crawlee';
+import { JSDOMCrawler, ProxyConfiguration, SessionPool } from 'crawlee';
const proxyConfiguration = new ProxyConfiguration({
/* opts */
@@ -7,13 +7,11 @@ const proxyConfiguration = new ProxyConfiguration({
const crawler = new JSDOMCrawler({
// To use the proxy IP session rotation logic, you must turn the proxy usage on.
proxyConfiguration,
- // Activates the Session pool (default is true).
- useSessionPool: true,
// Overrides default Session pool configuration.
- sessionPoolOptions: { maxPoolSize: 100 },
+ sessionPool: new SessionPool({ maxPoolSize: 100 }),
// Set to true if you want the crawler to save cookies per session,
// and set the cookie header to request automatically (default is true).
- persistCookiesPerSession: true,
+ saveResponseCookies: true,
async requestHandler({ session, window }) {
const title = window.document.title;
diff --git a/docs/guides/session_management_playwright.ts b/docs/guides/session_management_playwright.ts
index f4f2f7c80f6f..01749fccddbb 100644
--- a/docs/guides/session_management_playwright.ts
+++ b/docs/guides/session_management_playwright.ts
@@ -1,4 +1,4 @@
-import { PlaywrightCrawler, ProxyConfiguration } from 'crawlee';
+import { PlaywrightCrawler, ProxyConfiguration, SessionPool } from 'crawlee';
const proxyConfiguration = new ProxyConfiguration({
/* opts */
@@ -7,13 +7,11 @@ const proxyConfiguration = new ProxyConfiguration({
const crawler = new PlaywrightCrawler({
// To use the proxy IP session rotation logic, you must turn the proxy usage on.
proxyConfiguration,
- // Activates the Session pool (default is true).
- useSessionPool: true,
// Overrides default Session pool configuration
- sessionPoolOptions: { maxPoolSize: 100 },
+ sessionPool: new SessionPool({ maxPoolSize: 100 }),
// Set to true if you want the crawler to save cookies per session,
// and set the cookies to page before navigation automatically (default is true).
- persistCookiesPerSession: true,
+ saveResponseCookies: true,
async requestHandler({ page, session }) {
const title = await page.title();
diff --git a/docs/guides/session_management_puppeteer.ts b/docs/guides/session_management_puppeteer.ts
index 63b342146397..76ad3fcc7ee5 100644
--- a/docs/guides/session_management_puppeteer.ts
+++ b/docs/guides/session_management_puppeteer.ts
@@ -1,4 +1,4 @@
-import { PuppeteerCrawler, ProxyConfiguration } from 'crawlee';
+import { PuppeteerCrawler, ProxyConfiguration, SessionPool } from 'crawlee';
const proxyConfiguration = new ProxyConfiguration({
/* opts */
@@ -7,13 +7,11 @@ const proxyConfiguration = new ProxyConfiguration({
const crawler = new PuppeteerCrawler({
// To use the proxy IP session rotation logic, you must turn the proxy usage on.
proxyConfiguration,
- // Activates the Session pool (default is true).
- useSessionPool: true,
// Overrides default Session pool configuration
- sessionPoolOptions: { maxPoolSize: 100 },
+ sessionPool: new SessionPool({ maxPoolSize: 100 }),
// Set to true if you want the crawler to save cookies per session,
// and set the cookies to page before navigation automatically (default is true).
- persistCookiesPerSession: true,
+ saveResponseCookies: true,
async requestHandler({ page, session }) {
const title = await page.title();
diff --git a/docs/guides/session_management_standalone.ts b/docs/guides/session_management_standalone.ts
index c6fa33d82170..8ac133d9501c 100644
--- a/docs/guides/session_management_standalone.ts
+++ b/docs/guides/session_management_standalone.ts
@@ -5,17 +5,16 @@ const sessionPoolOptions = {
maxPoolSize: 100,
};
-// Open Session Pool.
-const sessionPool = await SessionPool.open(sessionPoolOptions);
+const sessionPool = new SessionPool(sessionPoolOptions);
// Get session.
const session = await sessionPool.getSession();
// Increase the errorScore.
-session.markBad();
+session?.markBad();
// Throw away the session.
-session.retire();
+session?.retire();
// Lower the errorScore and mark the session good.
-session.markGood();
+session?.markGood();
diff --git a/docs/introduction/03-adding-urls.mdx b/docs/introduction/03-adding-urls.mdx
index 387b86fb9450..520dcc62ffd6 100644
--- a/docs/introduction/03-adding-urls.mdx
+++ b/docs/introduction/03-adding-urls.mdx
@@ -130,7 +130,7 @@ await enqueueLinks({
### Filter URLs with patterns
-For even more control, you can use `globs`, `regexps` and `pseudoUrls` to filter the URLs. Each of those arguments is always an `Array`, but the contents can take on many forms. See the reference for more information about them as well as other options.
+For even more control, you can use `include` and `exclude` to filter the URLs. Each accepts an `Array` of glob pattern strings, `{ glob: string }` objects, `RegExp` instances, or `{ regexp: RegExp }` objects. See the reference for more information about them as well as other options.
:::caution Defaults override
@@ -140,17 +140,17 @@ If you provide one of those options, the default `same-hostname` strategy will *
```ts
await enqueueLinks({
- globs: ['http?(s)://apify.com/*/*'],
+ include: ['http?(s)://apify.com/*/*'],
});
```
### Transform requests
-To have absolute control, we have the `transformRequestFunction`. Just before a new `Request` is constructed and enqueued to the `RequestQueue`, this function can be used to skip it or modify its contents such as `userData`, `payload` or, most importantly, `uniqueKey`. This is useful when you need to enqueue multiple requests to the queue, and these requests share the same URL, but differ in methods or payloads. Another use case is to dynamically update or create the `userData`.
+To have absolute control, we have the `transformRequestFunction`. After request options are filtered by `include`/`exclude` patterns, this function can be used to skip them or modify their contents such as `userData`, `payload` or, most importantly, `uniqueKey`. This is useful when you need to enqueue multiple requests to the queue, and these requests share the same URL, but differ in methods or payloads. Another use case is to dynamically update or create the `userData`.
```ts
await enqueueLinks({
- globs: ['http?(s)://apify.com/*/*'],
+ include: ['http?(s)://apify.com/*/*'],
transformRequestFunction(req) {
// ignore all links ending with `.pdf`
if (req.url.endsWith('.pdf')) return false;
diff --git a/docs/introduction/08-refactoring.mdx b/docs/introduction/08-refactoring.mdx
index 450048013258..235fe88f0597 100644
--- a/docs/introduction/08-refactoring.mdx
+++ b/docs/introduction/08-refactoring.mdx
@@ -154,6 +154,30 @@ Initially, using a simple `if/else` statement for selecting different logic base
It's good practice in any programming language to split your logic into bite-sized chunks that are easy to read and reason about. Scrolling through a thousand line long `requestHandler()` where everything interacts with everything and variables can be used everywhere is not a beautiful thing to do and a pain to debug. That's why we prefer the separation of routes into their own files.
+### Giving a route its own timeout
+
+The crawler-wide `requestHandlerTimeoutSecs` applies to every request alike. Once the routes are separated, one of them can be given its own timeout — longer or shorter — without changing the limit for the rest, by passing a per-route timeout as the last argument of `addHandler`. This is handy when one page type does noticeably more work than the others, like a `CATEGORY` page that paginates through a long list:
+
+```js
+router.addHandler('CATEGORY', async ({ page, enqueueLinks }) => {
+ // ...
+}, { requestHandlerTimeoutSecs: 120 });
+
+router.addHandler('DETAIL', async ({ page }) => {
+ // ...
+}); // keeps the crawler's default
+```
+
+If how much longer a route needs only becomes clear once it is already running, call `extendTimeout()` from inside the handler to buy more time on the spot:
+
+```js
+router.addHandler('CATEGORY', async ({ page, enqueueLinks, extendTimeout }) => {
+ const pageCount = await countPages(page);
+ extendTimeout(pageCount * 10); // ask for 10 more seconds per page
+ // ...
+});
+```
+
## Next steps
In the next and final step, you'll see how to deploy your Crawlee project to the cloud. If you used the CLI to bootstrap your project, you already have a **Dockerfile** ready, and the next section will show you how to deploy it to the [Apify Platform](../deployment/apify-platform) with ease.
diff --git a/docs/package.json b/docs/package.json
index 51ae4027ed12..5e9c57042805 100644
--- a/docs/package.json
+++ b/docs/package.json
@@ -2,7 +2,6 @@
"name": "crawlee-docs",
"description": "Documentation and examples for Crawlee. This package is not published to npm, only used locally for TS build checks.",
"type": "module",
- "packageManager": "yarn@4.10.3",
"scripts": {
"typecheck": "tsc --noEmit"
},
@@ -10,8 +9,19 @@
"typescript": "^6.0.0"
},
"dependencies": {
+ "@crawlee/browser-pool": "workspace:*",
+ "@crawlee/core": "workspace:*",
+ "@crawlee/got-scraping-client": "workspace:*",
+ "@crawlee/http-client": "workspace:*",
+ "@crawlee/impit-client": "workspace:*",
+ "@crawlee/stagehand": "workspace:*",
+ "apify": "*",
+ "crawlee": "workspace:*",
+ "impit": "^0.14.2",
+ "pino": "^9.6.0",
"playwright-extra": "^4.3.6",
"puppeteer-extra": "^3.3.6",
- "puppeteer-extra-plugin-stealth": "^2.11.2"
+ "puppeteer-extra-plugin-stealth": "^2.11.2",
+ "winston": "^3.17.0"
}
}
diff --git a/docs/public-api/README.md b/docs/public-api/README.md
new file mode 100644
index 000000000000..2a92effa0579
--- /dev/null
+++ b/docs/public-api/README.md
@@ -0,0 +1,83 @@
+# Public API surface maps
+
+Each `*.api.md` file in this folder is a generated **map of the public, type-level
+interface** of one publishable `@crawlee/*` package — every exported class, method,
+property, function, and type, with full signatures. These reports define **where we
+promise backwards compatibility**.
+
+They are produced by [API Extractor](https://api-extractor.com/) from the built
+`dist/index.d.ts` of each package.
+
+## Workflow
+
+- After changing any package's public surface, regenerate the reports and commit them:
+
+ ```sh
+ pnpm build # the reports are generated from dist/
+ pnpm api:extract
+ ```
+
+- CI runs `pnpm api:check`, which fails if a committed report is out of date. A failing
+ check means you changed the public API: either that change is intentional (commit the
+ updated report — reviewers will see the surface diff) or it was accidental (fix it).
+
+- `api:check` also fails if a report ends up referencing a symbol it never declares, which
+ leaves the committed map describing a type nothing in it defines. Regenerating cannot fix
+ that; it has to be fixed in the source. In practice it means a `@public` symbol's signature
+ references an `@internal`/`@ignore`-d one, so the referenced type is trimmed out from under
+ it. Either drop the referenced type's tag (it is reachable from the public API, so users can
+ already depend on it) or keep it out of the public signature. An untagged symbol is
+ implicitly public, which is the convention here — the codebase does not use explicit
+ `@public` tags.
+
+ A symbol that is merely missing from the package's exports does **not** need fixing: see the
+ note on forgotten exports below.
+
+## Notes
+
+- The reports are generated as API Extractor's **`public`** variant, so symbols tagged
+ `@internal` (`@alpha`/`@beta` too) are excluded — only `@public` surface is tracked.
+ The legacy `@ignore` tag counts as `@internal` here; the generator rewrites it before
+ extraction, so an `@ignore`-d symbol is excluded too and cannot be referenced from a
+ `@public` signature.
+ The generator stages the variant as `.public.api.md` under `temp/` and promotes it
+ onto the committed `.api.md`, so the tracked filenames stay stable.
+- API Extractor builds the import list before it trims the non-`@public` declarations and
+ never revisits it, so a type reachable only from an `@internal` member would linger as a
+ bare import and read as public surface. There is no config option for this, so the
+ generator post-processes each report: it parses the fenced TypeScript and drops imports
+ whose binding is referenced by no declaration that survived the trim.
+- **Forgotten exports** — types the public API references but the entry point never exports —
+ are included in the report via `includeForgottenExports` and carry an explicit banner:
+
+ ```ts
+ // Not exported by the entry point; reachable only as a referenced type.
+ // @public (undocumented)
+ interface SitemapUrlData {
+ ```
+
+ Their *shape* is part of the surface we promise not to break, but their *name* is not
+ importable, so they are emitted without `export`. API Extractor labels them `@public
+ (undocumented)` like anything else, which is indistinguishable from a real export at a
+ glance, hence the added banner. The alternative was exporting every such type from its
+ package — ~38 new public exports, committing us to names we never meant to publish. If you
+ *want* one importable, export it deliberately and the report will show it with `export`.
+- Because API Extractor decides both of the above before the `@public` trim, it also offers
+ declarations for symbols reachable only from members that never reach the report. The
+ generator drops those the same way it drops dead imports, so the report carries nothing it
+ does not refer to. Only symbols flagged `ae-forgotten-export` are eligible, which is what
+ keeps genuinely reachable declarations (e.g. the `social` namespace in `@crawlee/utils`,
+ whose members are exposed through a `declare namespace` block) from being pruned.
+- `docs/public-api/temp/` holds intermediate reports (including the staged `.public.api.md`
+ files) and is git-ignored.
+- `@crawlee/cli` and `@crawlee/templates` are deliberately excluded — they are tooling
+ (a CLI binary and project scaffolding), not an importable API where we promise BC. The
+ exclude list lives in `scripts/api-extractor/run.ts`.
+- The generator lives in `scripts/api-extractor/`. It temporarily strips the build's
+ injected `// @ts-ignore` comment lines from the `.d.ts` files (restoring them
+ afterwards) because API Extractor's AST walker trips over some of them; a small number
+ of packages additionally need a sanitized-mirror fallback. See the comments in
+ `scripts/api-extractor/run.ts` for details.
+- These reports now cover only the `@public` surface. Further shrinking them — genuinely
+ hiding class internals (untagged `protected`/`_`-prefixed members) rather than merely
+ tagging them — is the goal tracked in issue #3109.
diff --git a/docs/public-api/crawlee-basic.api.md b/docs/public-api/crawlee-basic.api.md
new file mode 100644
index 000000000000..80fbacb566ee
--- /dev/null
+++ b/docs/public-api/crawlee-basic.api.md
@@ -0,0 +1,309 @@
+## Public API Report File for "@crawlee/basic"
+
+> Do not edit this file. It is a report generated by [API Extractor](https://api-extractor.com/).
+
+```ts
+
+import type { AddRequestsBatchedOptions } from '@crawlee/core';
+import type { AddRequestsBatchedResult } from '@crawlee/core';
+import { AnyPredicate } from 'ow';
+import { ArrayPredicate } from 'ow';
+import type { Awaitable } from '@crawlee/types';
+import type { BaseHttpClient } from '@crawlee/types';
+import { BasePredicate } from 'ow';
+import { BooleanPredicate } from 'ow';
+import { Cheerio } from '@crawlee/utils';
+import { CheerioAPI } from '@crawlee/utils';
+import { CheerioRoot } from '@crawlee/utils';
+import { ConcurrencySystem } from '@crawlee/core';
+import { ConcurrencySystemOptions } from '@crawlee/core';
+import type { Configuration } from '@crawlee/core';
+import { ContextPipeline } from '@crawlee/core';
+import type { CrawleeLogger } from '@crawlee/core';
+import { CrawlingContext } from '@crawlee/core';
+import { Dataset } from '@crawlee/core';
+import type { DatasetExportOptions } from '@crawlee/core';
+import type { Dictionary } from '@crawlee/types';
+import { Element as Element_2 } from '@crawlee/utils';
+import type { EventManager } from '@crawlee/core';
+import type { FinalStatistics } from '@crawlee/core';
+import type { GetUserDataFromRequest } from '@crawlee/core';
+import { IConcurrencySystem } from '@crawlee/core';
+import { IProxyConfiguration } from '@crawlee/core';
+import { IRequestLoader } from '@crawlee/core';
+import { IRequestManager } from '@crawlee/core';
+import type { ISession } from '@crawlee/types';
+import type { ISessionPool } from '@crawlee/types';
+import { IStatistics } from '@crawlee/core';
+import { NumberPredicate } from 'ow';
+import { ObjectPredicate } from 'ow';
+import { Predicate } from 'ow';
+import type { ProxyInfo } from '@crawlee/types';
+import type { ReadonlyDeep } from 'type-fest';
+import { Request as Request_2 } from '@crawlee/core';
+import { RequestQueue } from '@crawlee/core';
+import { RobotsTxtFile } from '@crawlee/utils';
+import type { RouterHandler } from '@crawlee/core';
+import type { RouterRoutes } from '@crawlee/core';
+import type { SetStatusMessageOptions } from '@crawlee/types';
+import type { SkippedRequestCallback } from '@crawlee/core';
+import type { StatisticState } from '@crawlee/core';
+import type { StorageBackend } from '@crawlee/types';
+import type { StorageIdentifier } from '@crawlee/core';
+import { StorageWritePolicy } from '@crawlee/core';
+import { StringPredicate } from 'ow';
+import type { TaskLoopPredicates } from '@crawlee/core';
+import { TimeoutError } from '@apify/timeout';
+import type { TypedRequestsLike } from '@crawlee/core';
+
+// @public (undocumented)
+export class BasicCrawler, ExtendedContext extends Context = Context & ContextExtension, Routes extends Record = Record>> {
+ constructor(options?: BasicCrawlerOptions & RequireContextPipeline);
+ // (undocumented)
+ protected readonly additionalHttpErrorStatusCodes: Set;
+ addRequests(requests: ReadonlyDeep>, options?: CrawlerAddRequestsOptions): Promise;
+ get basicContextPipeline(): ContextPipeline<{
+ request: Request_2;
+ }, CrawlingContext>;
+ // (undocumented)
+ protected blockedStatusCodes: Set;
+ protected buildContextPipeline(): ContextPipeline;
+ // (undocumented)
+ protected calculateEnqueuedRequestLimit(explicitLimit?: number): Promise;
+ get concurrencySystem(): IConcurrencySystem | undefined;
+ // (undocumented)
+ get contextPipeline(): ContextPipeline;
+ // (undocumented)
+ protected static readonly CRAWLEE_STATE_KEY = "CRAWLEE_STATE";
+ protected createDefaultConcurrencySystem(options: ConcurrencySystemOptions): ConcurrencySystem;
+ // (undocumented)
+ protected readonly errorHandler?: ErrorHandler;
+ exportData(path: string, format?: 'json' | 'csv', options?: DatasetExportOptions): Promise;
+ // (undocumented)
+ protected readonly failedRequestHandler?: ErrorHandler;
+ // (undocumented)
+ protected getCookieHeaderFromRequest(request: Request_2): string;
+ getData(...args: Parameters): ReturnType;
+ getDataset(identifier?: string | StorageIdentifier): Promise;
+ protected getMessageFromError(error: Error, forceStack?: boolean): string | TimeoutError | undefined;
+ protected getNavigationTimeoutMillis(): number;
+ // (undocumented)
+ protected getPendingRequestCountApproximation(): Promise;
+ getRequestManager(): Promise;
+ // @deprecated (undocumented)
+ getRequestQueue(): Promise;
+ // (undocumented)
+ protected getRobotsTxtFileForUrl(url: string): Promise;
+ // (undocumented)
+ protected handleSkippedRequest(options: Parameters[0]): Promise;
+ // (undocumented)
+ hasFinishedBefore: boolean;
+ // (undocumented)
+ protected readonly httpClient: BaseHttpClient;
+ // (undocumented)
+ protected readonly identity: CrawlerIdentity;
+ protected init(): Promise;
+ // (undocumented)
+ protected readonly internalTimeoutMillis: number;
+ protected isErrorStatusCode(status: number): boolean;
+ protected isProxyError(error: Error): boolean;
+ // (undocumented)
+ get log(): CrawleeLogger;
+ // (undocumented)
+ protected readonly maxCrawlDepth?: number;
+ // (undocumented)
+ protected readonly maxRequestRetries: number;
+ // (undocumented)
+ protected readonly maxRequestsPerCrawl?: number;
+ // (undocumented)
+ protected readonly onSkippedRequest?: SkippedRequestCallback;
+ // (undocumented)
+ protected static optionsShape: {
+ contextPipelineBuilder: ObjectPredicate
+ 
- 
- 
+