Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
126 commits
Select commit Hold shift + click to select a range
624c098
ci: test on node 22 and 24
B4nan May 17, 2025
6bca7e0
refactor: convert to native ESM
B4nan May 19, 2025
dfef71d
refactor: remove deprecated crawler options
B4nan May 20, 2025
8d95073
refactor: make crawling context strict and remove the error fallback
B4nan May 20, 2025
bc4f4b5
refactor: remove `additionalBlockedStatusCodes` parameter of `Session…
B4nan May 20, 2025
1aa8f61
refactor: remove `additionalBlockedStatusCodes` parameter of `Session…
B4nan May 20, 2025
8c44a14
chore: skip docker image builds for v4
B4nan May 20, 2025
d4b1667
chore: use `v4` dist tag
B4nan May 20, 2025
8962ca3
chore: run tests on v4 branch
B4nan May 20, 2025
ab3fdfc
chore: fix build
B4nan May 20, 2025
b0e1301
chore: fix v4 publishing
B4nan May 20, 2025
48800f7
chore: use node 22 in e2e tests and project templates
B4nan May 21, 2025
c08dc99
chore: use node 24 in e2e tests and project templates
B4nan May 21, 2025
b1c9b6a
chore: improve types to get rid of some `as any`
B4nan Jun 4, 2025
f4d26ee
chore: remove some deadcode
B4nan Jun 4, 2025
e075777
chore: bump a few more dependencies
B4nan Jun 11, 2025
17d6a59
fix CLI
B4nan Jun 11, 2025
bdc9464
fix CLI 2
B4nan Jun 11, 2025
cc0967a
fix: remove old system info implementation
B4nan Jun 11, 2025
79d27f5
chore: replace `lodash.isequal` with `util.isDeepStrictEqual`
B4nan Jun 17, 2025
cf72dda
refactor!: Introduce the `ContextPipeline` abstraction (#3119)
janbuchar Nov 24, 2025
971ffaf
feat: store `ProxyInfo` inside `Session` instances (#3199)
barjin Nov 26, 2025
a129fa0
feat!: use native `Request` / `Response` interface (#3163)
barjin Nov 27, 2025
f053992
chore: fix build errors from `master` rebase (#3285)
barjin Dec 1, 2025
ab92916
chore: fix broken types in docs examples (#3287)
barjin Dec 1, 2025
23d2b87
chore(docs): Add (temporary) 4.0 docs snapshot (#3292)
janbuchar Dec 2, 2025
72443e7
chore: Fix HttpCrawler context types (#3291)
janbuchar Dec 2, 2025
4387bdb
fix: `KVS.getPublicUrl()` reads the public URL directly from storage …
barjin Dec 2, 2025
c53bf30
feat: replace `got`-specific `HttpRequest` with native `Request` inte…
barjin Dec 5, 2025
9d4d33d
feat: drop `gotOptions` params from the public interface (#3300)
barjin Dec 16, 2025
8f1606e
fix: patch `GotScrapingHttpClient` header handling and proxies (#3308)
barjin Dec 17, 2025
8566dd1
feat: custom `httpClient` implementation for `@crawlee/utils` and `Re…
barjin Dec 18, 2025
f084aa1
fix: log warning on sharing `useState()`, add `id` option to `BasicCr…
barjin Dec 18, 2025
60cebe0
fix: `sendRequest` uses correct `Session` instance and allows for ove…
barjin Dec 18, 2025
7849c56
feat: `fetch` API for custom http clients (#3326)
barjin Jan 20, 2026
10f22c7
chore: drop `got-scraping` as the direct `crawlee` dependency (#3358)
barjin Jan 21, 2026
3a2c5fe
chore: add `discoverValidSitemaps` utility (#3392)
barjin Feb 6, 2026
86ab1cf
fix: Fix file download hang (#3411)
janbuchar Feb 16, 2026
eb908c5
fix: rectify transformRequestFunction precedence rules (#3372)
l2ysho Feb 17, 2026
ba3a356
refactor!: Extract service management from Configuration into Service…
janbuchar Feb 23, 2026
374f347
fix: treat `CookieJar` write methods as potentially throwing (#3426)
Copilot Feb 23, 2026
876be4a
chore(deps): update got-scraping package (#3452)
l2ysho Feb 27, 2026
ee71c40
feat: decouple log configuration from apifylog (#3399)
l2ysho Mar 3, 2026
d8d834b
refactor: use `ContextPipeline` to initialize `BasicCrawler`'s contex…
barjin Mar 4, 2026
3e05afb
chore: Lint
janbuchar Mar 5, 2026
c842fb3
chore: Update lockfile
janbuchar Mar 5, 2026
53e0f8d
fix: Migrate StagehandCrawler to ContextPipeline
janbuchar Mar 6, 2026
5cfedbe
chore: Fix more build issues
janbuchar Mar 6, 2026
bb9094c
chore: run linter
barjin Mar 9, 2026
88e2a56
docs: sync 4.0 version with the latest changes in `master`
barjin Mar 9, 2026
4d6ccc1
chore: fix TypeScript errors in tests
barjin Mar 9, 2026
9ae2994
docs: fix types in docs examples
barjin Mar 9, 2026
2c810aa
chore: fix tests (new error messages)
barjin Mar 10, 2026
75e0908
chore: bump vitest to `beta`, fix failing ALS tests
barjin Mar 10, 2026
2a6b2da
feat: add `AbortSignal` and rename `timeout` → `timeoutMillis` in `Se…
Copilot Mar 10, 2026
846e9ca
chore: Fix rebase conflict resolution errors
janbuchar Mar 11, 2026
648583d
chore(memory-storage): sort dependencies and update yarn.lock (#3493)
l2ysho Mar 16, 2026
0bc731f
fix: improve ServiceConflictError message readability (#3480)
l2ysho Mar 16, 2026
d788ebe
feat: drop `useSessionPool` option, make `SessionPool` required in al…
barjin Mar 17, 2026
697dcf2
docs: add custom logger guide (#3473)
l2ysho Mar 17, 2026
f2016c3
refactor: use `BasicCrawler` session in browser launch context (#3489)
barjin Mar 18, 2026
f6dac27
refactor: move blocked HTTP status code management to `BasicCrawler` …
barjin Mar 19, 2026
cd55e00
feat: lazy-initialize `SessionPool`, drop `SessionPool.open()` (#3513)
barjin Mar 24, 2026
0b6a159
refactor: replace service locator forwarding object with Proxy (#3515)
B4nan Mar 25, 2026
7c3ba07
refactor: resolve last direct @apify/log calls
l2ysho Mar 25, 2026
340252a
fix: Fix `skipNavigation` behavior in `HttpCrawler` subclasses (#3381)
janbuchar Mar 26, 2026
39b11ad
fix: clean-up cookie handling precedence rules (#3517)
barjin Mar 31, 2026
1baa628
fix(puppeteer-crawler): migrate cookie handling to browser-context AP…
B4nan Apr 15, 2026
aba7203
chore: migrate from eslint/biome to oxlint/oxfmt (#3569)
B4nan Apr 17, 2026
bd7943d
feat: share `SessionPool` instance between crawlers (#3499)
barjin Apr 17, 2026
bec112a
feat: allow binding a Session instance to a Request (#3518)
barjin Apr 17, 2026
930b2ef
chore: migrate to pnpm and enable minimum release age (#3581)
B4nan Apr 20, 2026
11c7f5d
chore: migrate `patch-package` to `pnpm patch` (#3585)
barjin Apr 21, 2026
ffff334
refactor!: Overhaul the StorageClient interface (#3570)
janbuchar Apr 22, 2026
e25a57b
feat(`session-pool`): add `sessionReuseStrategy` option (#3587)
barjin Apr 23, 2026
55986bb
refactor: move `describe` helper to `@crawlee/utils` as `inspectValue…
l2ysho Apr 27, 2026
84b264b
chore: use `@apify/oxlint-config` as shared lint preset (#3607)
B4nan Apr 28, 2026
a15f61c
chore(deps-dev): bump oxlint to 1.62 and oxfmt to 0.46 (#3608)
B4nan Apr 28, 2026
0d0c58d
chore: pass --no-error-on-unmatched-pattern to oxlint in lint-staged …
B4nan Apr 29, 2026
d1f4c98
feat: redesign `Configuration` class for v4 (#3484)
B4nan Apr 29, 2026
36022e7
chore(ci): pin lerna's minimatch to v3 and fail step on lerna error
B4nan Apr 29, 2026
84be0d7
chore(ci): treat any non-prod dist-tag as canary in publish workflow
B4nan Apr 29, 2026
b548b82
chore(ci): bake --yes into publish scripts
B4nan Apr 29, 2026
ba4c686
chore(ci): restore Docker image dispatch, gated to master
B4nan Apr 29, 2026
a8b6281
refactor!: remove `tieredProxyUrls` support (#3599)
barjin Apr 30, 2026
b854f5b
fix(linkedom-crawler): declare cheerio as a dependency (#3620)
B4nan Apr 30, 2026
d7fed8c
feat: reuse `Sessions` correctly in `BasicCrawler` (#3605)
barjin May 4, 2026
712a5d1
feat: rename `persistCookiesPerSession` to `saveResponseCookies`, var…
barjin May 6, 2026
dc9e48f
refactor!: treat session errors as regular request retries (#3631)
barjin May 11, 2026
20b320a
refactor!: Rename StorageManager to StorageInstanceManager and use it…
janbuchar May 11, 2026
d633c8b
refactor: add `Configuration.reset()` static method (#3649)
B4nan May 12, 2026
2b67072
chore: migrate v4 to pnpm 11 (#3645)
B4nan May 12, 2026
82647f6
ci: disable pnpm 11's verifyDepsBeforeRun to unbreak parallel turbo c…
B4nan May 13, 2026
2cb528b
refactor: revert Configuration.reset() static method (#3651)
B4nan May 13, 2026
886cb69
refactor!: remove the `EventEmitter` interface from `SessionPool` (#3…
barjin May 15, 2026
0d782bc
feat: export `SessionPool` interface, minimize `ISession`, allow cust…
barjin May 25, 2026
99d4cc9
chore: bump puppeteer to 25 and playwright to 1.60 (#3703)
barjin Jun 1, 2026
e00aa94
refactor!: Align `DatasetClient` and `KeyValueStoreClient` interfaces…
janbuchar Jun 1, 2026
52f245c
feat: Implement storage aliases (#3636)
janbuchar Jun 4, 2026
4721cda
feat: store browser / HTTP client fingerprint in Session (#3646)
barjin Jun 5, 2026
79ceb4e
feat: allow navigation hooks to override crawling context members (#3…
barjin Jun 8, 2026
0c1fbcf
refactor!: Finalize transition to IRequestManager and IRequestLoader …
janbuchar Jun 9, 2026
ebb1b26
refactor!: Introduce IBrowserPool interface and utilize it in Browser…
janbuchar Jun 10, 2026
f1f0959
refactor!: Align `RequestQueueClient` interface with its Python count…
janbuchar Jun 15, 2026
a9b9722
refactor!: Split MemoryStorage into FileSystemStorageClient and Memor…
janbuchar Jun 17, 2026
19fa874
feat: make `impit` the default HTTP client (#3767)
barjin Jun 23, 2026
20ba48c
docs: reflect the recent `SessionPool` changes in a guide (#3724)
barjin Jun 24, 2026
13976b3
refactor!: Centralize KVS value semantics in @crawlee/core (#3774)
janbuchar Jun 24, 2026
9963157
feat: rename KVS objects from `SDK_` to `CRAWLEE_` (#3783)
barjin Jun 26, 2026
7acf908
fix: read content encoding hints from `<meta>` tags (#3780)
barjin Jun 30, 2026
0761b67
fix: disable storage access on `requestHandler` timeouts (#3788)
barjin Jul 1, 2026
8f18a5a
docs: Autogenerated public API documentation (#3798)
janbuchar Jul 1, 2026
3365b2d
refactor!: Dissolve @crawlee/memory-storage into @crawlee/core (#3802)
janbuchar Jul 1, 2026
3b5fc41
fix: `RequestManagerTandem` returns correct total count (#3804)
barjin Jul 1, 2026
e1d8c93
fix: Prevent browser-based crawling mode in AdaptivePlaywrightCrawler…
janbuchar Jul 1, 2026
56f1dfd
fix(e2e): repair E2E test infrastructure for pnpm + crawlee v4 (#3805)
janbuchar Jul 2, 2026
39f689f
fix: Fix repeated Storage.open() using the legacy idOrName parameter …
janbuchar Jul 2, 2026
82801e2
fix: Rectify inconsistencies in the StorageClient interface family (#…
janbuchar Jul 5, 2026
a2f9b26
test: Eliminate MemoryStorageEmulator (#3806)
janbuchar Jul 5, 2026
8a42262
refactor!: Rewrite the FilesystemStorageClient to use apify/crawlee-s…
janbuchar Jul 5, 2026
9f94f67
refactor!: Move status message handling from StorageClient to the eve…
janbuchar Jul 5, 2026
9e87aa9
fix(memory-storage): store JSON snapshots in `DatasetClient.pushData(…
AmariahAK Jul 9, 2026
eec66ca
refactor!: Rename `StorageClient` to `StorageBackend` (#3825)
janbuchar Jul 13, 2026
8a860f6
ci: exclude `@crawlee/fs-storage-native` from canary/pin version rewr…
B4nan Jul 13, 2026
667a6af
feat: connect to remote browser services (#3545)
l2ysho Jul 14, 2026
2a15981
refactor(browser-pool): move remote connection logic into dedicated r…
l2ysho Jul 14, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
19 changes: 5 additions & 14 deletions .github/workflows/docs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -19,41 +19,32 @@ jobs:
steps:
- uses: actions/checkout@v6

- name: Use Node.js 20
- name: Use Node.js 24
uses: actions/setup-node@v6
with:
node-version: 24
package-manager-cache: false

- name: Enable corepack
run: |
corepack enable
corepack prepare yarn@stable --activate
- uses: apify/workflows/pnpm-install@main

- name: Build & deploy docs
run: |
# install project deps
yarn
# go to website dir
cd website
# install website deps
yarn
# build the docs
yarn build
pnpm build
env:
APIFY_SIGNING_TOKEN: ${{ secrets.APIFY_SIGNING_TOKEN }}
SEGMENT_TOKEN: ${{ secrets.SEGMENT_TOKEN }}

- name: Set up GitHub Pages
uses: actions/configure-pages@v5
uses: actions/configure-pages@v6

- name: Upload GitHub Pages artifact
uses: actions/upload-pages-artifact@v4
with:
path: ./website/build

- name: Deploy artifact to GitHub Pages
uses: actions/deploy-pages@v4
uses: actions/deploy-pages@v5

- name: Invalidate CloudFront cache
run: |
Expand Down
33 changes: 11 additions & 22 deletions .github/workflows/publish-to-npm.yml
Original file line number Diff line number Diff line change
Expand Up @@ -49,16 +49,6 @@ jobs:
node-version: 24
package-manager-cache: false

- name: Enable corepack
run: |
corepack enable
corepack prepare yarn@stable --activate

- name: Activate cache for Node.js 24
uses: actions/setup-node@v6
with:
cache: 'yarn'

- name: Turbo cache
id: turbo-cache
uses: actions/cache@v5
Expand All @@ -68,20 +58,19 @@ jobs:
restore-keys: |
turbo-${{ github.job }}-${{ github.ref_name }}-

- name: Install dependencies
run: yarn
- uses: apify/workflows/pnpm-install@main

- name: Build packages
run: yarn ci:build
run: pnpm ci:build

- name: Bump canary versions
if: inputs.dist-tag == 'next'
if: inputs.dist-tag != 'prod'
run: |
yarn turbo copy --force -- --canary --preid=beta
pnpm turbo copy --force -- --canary=major --preid=beta

- name: Commit changes
if: inputs.dist-tag == 'next'
uses: EndBug/add-and-commit@v9
if: inputs.dist-tag != 'prod'
uses: EndBug/add-and-commit@v10
id: commit
with:
author_name: Apify Release Bot
Expand All @@ -91,22 +80,22 @@ jobs:

- name: Publish to NPM (@latest)
if: inputs.dist-tag == 'prod'
uses: nick-fields/retry@v3
uses: nick-fields/retry@v4
with:
timeout_minutes: 10
max_attempts: 5
retry_wait_seconds: 30
command: git checkout . && yarn publish:prod --yes
command: git checkout . && pnpm publish:prod
env:
GH_TOKEN: ${{ secrets.APIFY_SERVICE_ACCOUNT_GITHUB_TOKEN }}

- name: Publish to NPM (@next)
if: inputs.dist-tag == 'next'
uses: nick-fields/retry@v3
if: inputs.dist-tag != 'prod'
uses: nick-fields/retry@v4
with:
timeout_minutes: 10
max_attempts: 5
retry_wait_seconds: 30
command: git checkout . && yarn publish:next --yes
command: git checkout . && pnpm publish:next
env:
GH_TOKEN: ${{ secrets.APIFY_SERVICE_ACCOUNT_GITHUB_TOKEN }}
77 changes: 20 additions & 57 deletions .github/workflows/release.yml
Original file line number Diff line number Diff line change
@@ -1,7 +1,6 @@
name: Release @latest

env:
YARN_IGNORE_NODE: 1
RETRY_TESTS: 1

on:
Expand Down Expand Up @@ -33,7 +32,7 @@ jobs:
matrix:
# We don't test on Windows as the tests are flaky
os: [ ubuntu-22.04 ]
node-version: [ 18, 20, 22, 24 ]
node-version: [ 22, 24 ]

runs-on: ${{ matrix.os }}

Expand All @@ -52,16 +51,6 @@ jobs:
node-version: ${{ matrix.node-version }}
package-manager-cache: false

- name: Enable corepack
run: |
corepack enable
corepack prepare yarn@stable --activate

- name: Activate cache for Node.js ${{ matrix.node-version }}
uses: actions/setup-node@v6
with:
cache: 'yarn'

- name: Turbo cache
id: turbo-cache
uses: actions/cache@v5
Expand All @@ -71,16 +60,16 @@ jobs:
restore-keys: |
turbo-${{ github.job }}-${{ github.ref_name }}-

- name: Install Dependencies
run: |
yarn
yarn playwright install --with-deps
- uses: apify/workflows/pnpm-install@main

- name: Install Playwright browsers
run: pnpm exec playwright install --with-deps

- name: Build
run: yarn ci:build
run: pnpm ci:build

- name: Tests
run: yarn test
run: pnpm test

release:
name: "Bump Crawlee: ${{ inputs.version }} version (${{ inputs.custom_version || 'n/a' }} custom version)"
Expand All @@ -95,22 +84,12 @@ jobs:
token: ${{ secrets.APIFY_SERVICE_ACCOUNT_GITHUB_TOKEN }}
fetch-depth: 0

- name: Use Node.js 20
- name: Use Node.js 24
uses: actions/setup-node@v6
with:
node-version: 24
package-manager-cache: false

- name: Enable corepack
run: |
corepack enable
corepack prepare yarn@stable --activate

- name: Activate cache for Node.js 20
uses: actions/setup-node@v6
with:
cache: 'yarn'

- name: Turbo cache
id: turbo-cache
uses: actions/cache@v5
Expand All @@ -120,15 +99,14 @@ jobs:
restore-keys: |
turbo-${{ github.job }}-${{ github.ref_name }}-

- name: Install Dependencies
run: yarn
- uses: apify/workflows/pnpm-install@main

- name: Build
run: yarn ci:build
run: pnpm ci:build

- name: Bump version to custom version
if: ${{ github.event.inputs.version == 'custom' && github.event.inputs.custom_version != '' }}
run: yarn lerna version ${{ github.event.inputs.custom_version }} --force-publish --yes
run: pnpm exec lerna version ${{ github.event.inputs.custom_version }} --force-publish --yes
env:
GH_TOKEN: ${{ secrets.APIFY_SERVICE_ACCOUNT_GITHUB_TOKEN }}
GIT_AUTHOR_NAME: Apify Release Bot
Expand All @@ -138,7 +116,7 @@ jobs:

- name: Bump version to ${{ github.event.inputs.version }} version
if: ${{ github.event.inputs.version != 'custom' }}
run: yarn lerna version ${{ github.event.inputs.version }} --force-publish --yes
run: pnpm exec lerna version ${{ github.event.inputs.version }} --force-publish --yes
env:
GH_TOKEN: ${{ secrets.APIFY_SERVICE_ACCOUNT_GITHUB_TOKEN }}
GIT_AUTHOR_NAME: Apify Release Bot
Expand All @@ -148,16 +126,17 @@ jobs:

- name: Pin versions in internal dependencies and update lockfile
run: |
yarn release:pin-versions
yarn install --no-immutable
pnpm release:pin-versions
pnpm install --no-frozen-lockfile

- name: Commit changes
id: commit
uses: EndBug/add-and-commit@v9
uses: EndBug/add-and-commit@v10
with:
author_name: Apify Release Bot
author_email: noreply@apify.com
message: 'chore(release): update internal dependencies [skip ci]'
pull: '--rebase --autostash'

- name: Publish packages
uses: apify/workflows/execute-workflow@main
Expand Down Expand Up @@ -189,7 +168,7 @@ jobs:
token: ${{ secrets.APIFY_SERVICE_ACCOUNT_GITHUB_TOKEN }}
fetch-depth: 0

- name: Use Node.js 20
- name: Use Node.js 24
uses: actions/setup-node@v6
with:
node-version: 24
Expand All @@ -198,23 +177,7 @@ jobs:
- name: Install jq
run: sudo apt-get install jq

- name: Enable corepack
run: |
corepack enable
corepack prepare yarn@stable --activate

- name: Activate cache for Node.js 20
uses: actions/setup-node@v6
with:
cache: 'yarn'

- name: Install dependencies
run: |
# install project deps
yarn
# install website deps
cd website
yarn
- uses: apify/workflows/pnpm-install@main

- name: Snapshot the current version
run: |
Expand All @@ -225,8 +188,8 @@ jobs:
exit 1
fi
MAJOR_MINOR=$(echo $VERSION | cut -d. -f1,2)
yarn docusaurus docs:version $MAJOR_MINOR
yarn docusaurus api:version $MAJOR_MINOR
pnpm docusaurus docs:version $MAJOR_MINOR
pnpm docusaurus api:version $MAJOR_MINOR

- name: Commit and push the version snapshot
run: |
Expand Down
Loading
Loading