feat(core): pass Request to onSkippedRequest - #3992

Open

barjin wants to merge 184 commits into

v4from

feat/skipped-request-callback

Commits on Jul 19, 2026

ci: test on node 22 and 24

B4nan
authored and
barjin
committed
d76650c

Commits on Jul 20, 2026

refactor: convert to native ESM

B4nan
authored and
barjin
committed
9eb9aae
refactor: remove deprecated crawler options

B4nan
authored and
barjin
committed
8a1c2bb
refactor: make crawling context strict and remove the error fallback

B4nan
authored and
barjin
committed
ad349a4
refactor: remove `additionalBlockedStatusCodes` parameter of `Session.retireOnBlockedStatusCodes`

B4nan
authored and
barjin
committed
da5dada
refactor: remove `additionalBlockedStatusCodes` parameter of `Session.retireOnBlockedStatusCodes`

B4nan
authored and
barjin
committed
1a87646
chore: skip docker image builds for v4

B4nan
authored and
barjin
committed
04c37c3
chore: use `v4` dist tag

B4nan
authored and
barjin
committed
b8c08e6
chore: run tests on v4 branch

B4nan
authored and
barjin
committed
25f3a69
chore: fix build

B4nan
authored and
barjin
committed
f1e8867
chore: fix v4 publishing

B4nan
authored and
barjin
committed
88c511c
chore: use node 24 in e2e tests and project templates

B4nan
authored and
barjin
committed
c7a5461
chore: improve types to get rid of some `as any`

B4nan
authored and
barjin
committed
e23347e
chore: remove some deadcode

B4nan
authored and
barjin
committed
7ce3d75
chore: bump a few more dependencies

B4nan
authored and
barjin
committed
71ceba0
fix CLI

B4nan
authored and
barjin
committed
97fd383
fix CLI 2

B4nan
authored and
barjin
committed
a68e1f2
fix: remove old system info implementation

B4nan
authored and
barjin
committed
ff611de
chore: replace `lodash.isequal` with `util.isDeepStrictEqual`

B4nan
authored and
barjin
committed
8ba0794
refactor!: Introduce the `ContextPipeline` abstraction (#3119)

authored and
barjin
committed
267a50f
feat: store `ProxyInfo` inside `Session` instances (#3199)
barjin
committed
7c47064
feat!: use native `Request` / `Response` interface (#3163)
barjin
committed
27c3176
chore: fix build errors from `master` rebase (#3285)
barjin
committed
ddd945f
chore: fix broken types in docs examples (#3287)
barjin
committed
0ce63a1
chore(docs): Add (temporary) 4.0 docs snapshot (#3292)

janbuchar
authored and
barjin
committed
ee74d7d
chore: Fix HttpCrawler context types (#3291)

janbuchar
authored and
barjin
committed
a525224
fix: `KVS.getPublicUrl()` reads the public URL directly from storage client (#3286)
barjin
committed
da7bf32
feat: replace `got`-specific `HttpRequest` with native `Request` interfaces (#3295)
barjin
committed
e921f8e
feat: drop `gotOptions` params from the public interface (#3300)

barjin
and
Copilot
committed
c95cb9f
fix: patch `GotScrapingHttpClient` header handling and proxies (#3308)
barjin
committed
44fd52c
feat: custom `httpClient` implementation for `@crawlee/utils` and `Request` storages (#3306)
barjin
committed
268cadf
fix: log warning on sharing `useState()`, add `id` option to `BasicCrawler` (#3309)
barjin
committed
e1e02ea
fix: `sendRequest` uses correct `Session` instance and allows for overrides (#3313)
barjin
committed
78c6dc4
feat: `fetch` API for custom http clients (#3326)
barjin
committed
4c3875e
chore: drop `got-scraping` as the direct `crawlee` dependency (#3358)
barjin
committed
f43a76f
chore: add `discoverValidSitemaps` utility (#3392)

barjin
and
foxt451
committed
41f4dae
fix: Fix file download hang (#3411)

janbuchar
authored and
barjin
committed
e1122f1
fix: rectify transformRequestFunction precedence rules (#3372)

l2ysho
authored and
barjin
committed
5202be6
refactor!: Extract service management from Configuration into ServiceLocator class (#3325)

janbuchar
authored and
barjin
committed
8f7e8d6
fix: treat `CookieJar` write methods as potentially throwing (#3426)

Copilot
authored and
barjin
committed
aa9fc61
chore(deps): update got-scraping package (#3452)

l2ysho
authored and
barjin
committed
4ab1ff5
feat: decouple log configuration from apifylog (#3399)

l2ysho
authored and
barjin
committed
bca23fb
refactor: use `ContextPipeline` to initialize `BasicCrawler`'s context idiomatically (#3388)
barjin
committed
1c58253
chore: Lint

janbuchar
authored and
barjin
committed
b5bd71c
fix: Migrate StagehandCrawler to ContextPipeline

janbuchar
authored and
barjin
committed
1555834
chore: Fix more build issues

janbuchar
authored and
barjin
committed
c1f0810
chore: run linter
barjin
committed
8625fcd
docs: sync 4.0 version with the latest changes in `master`
barjin
committed
6ba709c
chore: fix TypeScript errors in tests
barjin
committed
55db799
docs: fix types in docs examples
barjin
committed
052b890
chore: fix tests (new error messages)
barjin
committed
0ee89ab
chore: bump vitest to `beta`, fix failing ALS tests
barjin
committed
4ba67fc
feat: add `AbortSignal` and rename `timeout` → `timeoutMillis` in `SendRequestOptions` (#3457)

Copilot
and
barjin
committed
07c348b
chore: Fix rebase conflict resolution errors

janbuchar
authored and
barjin
committed
b5f2b03
chore(memory-storage): sort dependencies and update yarn.lock (#3493)

l2ysho
authored and
barjin
committed
39cc434
fix: improve ServiceConflictError message readability (#3480)

l2ysho
authored and
barjin
committed
771740f
feat: drop `useSessionPool` option, make `SessionPool` required in all crawlers (#3478)
barjin
committed
9b03a79
docs: add custom logger guide (#3473)

l2ysho
authored and
barjin
committed
1815c4d
refactor: use `BasicCrawler` session in browser launch context (#3489)
barjin
committed
39b62f2
refactor: move blocked HTTP status code management to `BasicCrawler` (#3496)
barjin
committed
1de7f59
feat: lazy-initialize `SessionPool`, drop `SessionPool.open()` (#3513)

barjin
and
Copilot
committed
a9e7a5d
refactor: replace service locator forwarding object with Proxy (#3515)

authored and
barjin
committed
bf6475c
refactor: resolve last direct @apify/log calls

authored and
barjin
committed
9133c24
fix: Fix `skipNavigation` behavior in `HttpCrawler` subclasses (#3381)

janbuchar
authored and
barjin
committed
5be1209
fix: clean-up cookie handling precedence rules (#3517)

committed
60e61ea
fix(puppeteer-crawler): migrate cookie handling to browser-context API (#3572)

authored and
barjin
committed
b5f18c4
chore: migrate from eslint/biome to oxlint/oxfmt (#3569)

authored and
barjin
committed
cfa35ce
feat: share `SessionPool` instance between crawlers (#3499)

barjin
and
Copilot
committed
19f6541
feat: allow binding a Session instance to a Request (#3518)
barjin
committed
39d502b
chore: migrate to pnpm and enable minimum release age (#3581)

authored and
barjin
committed
9f3e94e
chore: migrate `patch-package` to `pnpm patch` (#3585)
barjin
committed
753a8a6
refactor!: Overhaul the StorageClient interface (#3570)

janbuchar
authored and
barjin
committed
b84ca54
feat(`session-pool`): add `sessionReuseStrategy` option (#3587)
barjin
committed
dc58576
refactor: move `describe` helper to `@crawlee/utils` as `inspectValue` (#3586)

l2ysho
authored and
barjin
committed
4d7e935
chore: use `@apify/oxlint-config` as shared lint preset (#3607)

authored and
barjin
committed
fa621a6
chore(deps-dev): bump oxlint to 1.62 and oxfmt to 0.46 (#3608)

B4nan
authored and
barjin
committed
5ab0233
chore: pass --no-error-on-unmatched-pattern to oxlint in lint-staged (#3611)

B4nan
authored and
barjin
committed
894fbc7
feat: redesign `Configuration` class for v4 (#3484)

authored and
barjin
committed
fa2eff5
chore(ci): pin lerna's minimatch to v3 and fail step on lerna error

B4nan
authored and
barjin
committed
e941944
chore(ci): treat any non-prod dist-tag as canary in publish workflow

B4nan
authored and
barjin
committed
7784c1d
chore(ci): bake --yes into publish scripts

B4nan
authored and
barjin
committed
2df37a4
chore(ci): restore Docker image dispatch, gated to master

B4nan
authored and
barjin
committed
fae93d6
refactor!: remove `tieredProxyUrls` support (#3599)
barjin
committed
bc3cc1c
fix(linkedom-crawler): declare cheerio as a dependency (#3620)

B4nan
authored and
barjin
committed
90d693b
feat: reuse `Sessions` correctly in `BasicCrawler` (#3605)
barjin
committed
4fcd232
feat: rename `persistCookiesPerSession` to `saveResponseCookies`, various fixes (#3606)
barjin
committed
9b9c623
refactor!: treat session errors as regular request retries (#3631)
barjin
committed
ba88ff8
refactor!: Rename StorageManager to StorageInstanceManager and use it to centralize storage instance caching (#3584)

janbuchar
authored and
barjin
committed
4aed3bf
refactor: add `Configuration.reset()` static method (#3649)

authored and
barjin
committed
9db9ed1
chore: migrate v4 to pnpm 11 (#3645)

authored and
barjin
committed
3b26e97
ci: disable pnpm 11's verifyDepsBeforeRun to unbreak parallel turbo copy (#3650)

B4nan
authored and
barjin
committed
8049837
refactor: revert Configuration.reset() static method (#3651)

B4nan
authored and
barjin
committed
75f848a
refactor!: remove the `EventEmitter` interface from `SessionPool` (#3643)
barjin
committed
b3671b6
feat: export `SessionPool` interface, minimize `ISession`, allow custom implementations (#3644)
barjin
committed
270e158
chore: bump puppeteer to 25 and playwright to 1.60 (#3703)
barjin
committed
73fa17f
refactor!: Align `DatasetClient` and `KeyValueStoreClient` interfaces with their Python counterparts (#3627)

janbuchar
authored and
barjin
committed
0c76a11
feat: Implement storage aliases (#3636)

janbuchar
authored and
barjin
committed
9b2dc45
feat: store browser / HTTP client fingerprint in Session (#3646)
barjin
committed
1edc316
feat: allow navigation hooks to override crawling context members (#3672)
barjin
committed
347e198
refactor!: Finalize transition to IRequestManager and IRequestLoader (#3719)

janbuchar
authored and
barjin
committed
62b82f3
refactor!: Introduce IBrowserPool interface and utilize it in BrowserCrawler (#3669)

janbuchar
authored and
barjin
committed
3be90b3
refactor!: Align `RequestQueueClient` interface with its Python counterpart (#3729)

janbuchar
authored and
barjin
committed
f32a5bf
refactor!: Split MemoryStorage into FileSystemStorageClient and MemoryStorageClient (#3740)

janbuchar
authored and
barjin
committed
170a073
feat: make `impit` the default HTTP client (#3767)
barjin
committed
325230b
docs: reflect the recent `SessionPool` changes in a guide (#3724)
barjin
committed
a510ce0
refactor!: Centralize KVS value semantics in @crawlee/core (#3774)

janbuchar
authored and
barjin
committed
aa8e7a5
feat: rename KVS objects from `SDK_` to `CRAWLEE_` (#3783)
barjin
committed
bf08a65
fix: read content encoding hints from `<meta>` tags (#3780)
barjin
committed
c30448e
fix: disable storage access on `requestHandler` timeouts (#3788)
barjin
committed
93c1dd3
docs: Autogenerated public API documentation (#3798)

janbuchar
authored and
barjin
committed
4b97c12
refactor!: Dissolve @crawlee/memory-storage into @crawlee/core (#3802)

janbuchar
authored and
barjin
committed
168ce8a
fix: `RequestManagerTandem` returns correct total count (#3804)
barjin
committed
bb435db
fix: Prevent browser-based crawling mode in AdaptivePlaywrightCrawler from swallowing logs (#3803)

janbuchar
authored and
barjin
committed
f1852d6
fix(e2e): repair E2E test infrastructure for pnpm + crawlee v4 (#3805)

janbuchar
authored and
barjin
committed
4065e2c
fix: Fix repeated Storage.open() using the legacy idOrName parameter (#3808)

janbuchar
authored and
barjin
committed
10b50bf
fix: Rectify inconsistencies in the StorageClient interface family (#3800)

janbuchar
authored and
barjin
committed
5804787
test: Eliminate MemoryStorageEmulator (#3806)

janbuchar
authored and
barjin
committed
73b8740
refactor!: Rewrite the FilesystemStorageClient to use apify/crawlee-storage (#3762)

janbuchar
authored and
barjin
committed
5b8d7f8
refactor!: Move status message handling from StorageClient to the event system (#3818)

janbuchar
authored and
barjin
committed
db2ec41
fix(memory-storage): store JSON snapshots in `DatasetClient.pushData()` (#3831)

AmariahAK
authored and
barjin
committed
bf70b7e
refactor!: Rename `StorageClient` to `StorageBackend` (#3825)

janbuchar
and
barjin
committed
f8e49da
ci: exclude `@crawlee/fs-storage-native` from canary/pin version rewrite (#3844)

B4nan
authored and
barjin
committed
f31615f
feat: connect to remote browser services (#3545)

l2ysho
authored and
barjin
committed
df360fb
feat: unify extended unique key format with Python, add `alwaysEnqueue` (#3840)
barjin
committed
f1731f0
chore: fix `v4` CI build issues (#3856)
barjin
committed
a437f86
feat: make `cheerio` and `proxy-chain` imports dynamic (#3836)
barjin
committed
48d0941
chore: make `format:check` actually check the formatting (#3858)

B4nan
authored and
barjin
committed
f0b4720
test: run the browser crawler tests serially (#3859)

B4nan
authored and
barjin
committed
da680c1
chore: fix master-v4 rebase issues
barjin
committed
aceb6c0
chore: regenerate TypeScript API reports
barjin
committed
a0ba92d
chore: fix oxlint errors
barjin
committed
3d0b31a

Commits on Jul 21, 2026

Commits on Jul 22, 2026

Commits on Jul 23, 2026

Commits on Jul 24, 2026

Commits on Jul 27, 2026

Commits on Jul 29, 2026

Commits on Jul 30, 2026

Commits on Jul 31, 2026

Commits on Aug 3, 2026

Commits on Aug 4, 2026

fix(browser-crawler): persist session cookies after requestHandler (#3939)
Ayush7614
authored
8dbcd63

Commits on Aug 5, 2026

feat: redesign request timeouts and allow per-route overrides (#3860)
B4nan
authored
0ddb865
refactor: Remove unnecessary retries when enqueueing new requests (#3962)
janbuchar
authored
0ef731e
chore: Improve public API snapshot generation (#3964)
janbuchar
authored
fb3afb3
feat: simplify `enqueueLinks` interface (#3533)

l2ysho
and
claude
authored
1b4dc3c
chore: Add missing upgrading guide entries (#3965)
janbuchar
authored
2d262a9
docs: Reorganize the v4 upgrading guide (#3967)
janbuchar
authored
0ff70b1

Uh oh!

Commits

Commits on Jul 19, 2026

Commits on Jul 20, 2026

Commits on Jul 21, 2026

Commits on Jul 22, 2026

Commits on Jul 23, 2026