Deps upgrade and http2 autoselectfamily#484
Merged
Merged
Conversation
Bump runtime deps (got 15, cheerio 1.2, iconv-lite 0.7, tslog, tsup) and
dev deps (ava 8, nock 14, sinon 22, typescript 6, eslint 10, etc.), and
remove the obsolete @types/got (got ships its own types now), which also
clears the form-data advisory. moduleResolution -> "bundler" so got's ESM
types resolve.
Adapt to got 15: pass url as the first argument and normalize the
Uint8Array body to a Buffer before charset sniffing.
Fix renameOptionParams writing keys that resolved to undefined (e.g.
rateLimit: undefined), which clobbered crawler defaults when spread via
{ ...defaultOptions, ...options }.
Ignore .npmrc so the publish auth token is never committed.
Expose autoSelectFamily and autoSelectFamilyAttemptTimeout so callers can tune Node's "Happy Eyeballs" behavior. On dual-stack hosts with broken or slow IPv6, Node's 250ms per-address default aborts the request with ETIMEDOUT before the working address can connect. got strips unknown options and manages the http2 session internally, so these can't be forwarded per-request; they are applied via Node's process-wide net defaults, which covers both the http1 and http2 paths. Document both options in the README and set the attempt timeout in the http2 tests so they pass against the high-latency live host.
There was a problem hiding this comment.
Pull request overview
This PR upgrades key dependencies (notably got), updates the crawler’s request pipeline to stay compatible with the newer got API, and adds support/documentation for Node “Happy Eyeballs” socket defaults (autoSelectFamily*) to improve HTTP/2 connectivity on dual-stack hosts.
Changes:
- Upgrade dependencies (including
gotto v15) and adjust request invocation/response-body normalization accordingly. - Add
autoSelectFamily/autoSelectFamilyAttemptTimeoutoptions to types, option-stripping, README, and HTTP/2 tests. - Update TypeScript module resolution and AVA config format to match newer toolchain expectations.
Reviewed changes
Copilot reviewed 6 out of 9 changed files in this pull request and generated 7 comments.
Show a summary per file
| File | Description |
|---|---|
| tsconfig.json | Switches TS module resolution to bundler. |
| test/http2Response.js | Adds autoSelectFamilyAttemptTimeout to HTTP/2 tests and adds error logging (needs adjustment). |
| src/types/crawler.ts | Extends RequestOptions with autoSelectFamily* and JSDoc. |
| src/options.ts | Adds autoSelectFamily* to option filtering; applies Node process-wide defaults; adapts types for new got usage. |
| src/crawler.ts | Updates got typing/imports, got v15 calling convention, and normalizes body type. |
| README.md | Documents the new autoSelectFamily* options. |
| package.json | Bumps version and upgrades runtime/dev dependencies; updates AVA config shape. |
| .gitignore | Adds .npmrc to ignored files. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| }; | ||
|
|
||
| export const alignOptions = (options: RequestOptions): GotUrl => { | ||
| export const alignOptions = (options: RequestOptions): OptionsInit => { |
Comment on lines
+112
to
+116
| // "Happy Eyeballs" tuning. got strips unknown options and manages the | ||
| // http2 session internally, so these socket-level settings can't be passed | ||
| // per-request through got — Node only exposes them as process-wide defaults. | ||
| // Applied here (before they are stripped below) so they take effect for both | ||
| // the http1 and http2 paths. Guarded because the setters only exist on Node >=19. |
Comment on lines
+123
to
+124
| * Note: Node applies this as a process-wide default, so it affects all in-flight requests. | ||
| */ |
Comment on lines
+133
to
+134
| * Note: Node applies this as a process-wide default, so it affects all in-flight requests. | ||
| */ |
Comment on lines
+595
to
+596
| - **Note:** Node applies this as a process-wide default, so it affects all in-flight requests. | ||
|
|
Comment on lines
+602
to
+603
| - **Note:** Node applies this as a process-wide default, so it affects all in-flight requests. | ||
|
|
Comment on lines
23
to
27
| callback: (error, response, done) => { | ||
| if(error){ | ||
| console.log(error); | ||
| } | ||
| t.is(response.statusCode, 200); |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.