fix(web_fetch): switch to wreq Firefox emulation to bypass TLS bot detection#428
Merged
Conversation
…tection reqwest's rustls backend has a distinctive TLS fingerprint that Akamai and similar WAFs block at the handshake level. Switching to wreq with Emulation::Firefox135 replicates the Firefox TLS ClientHello, matching the approach already used by the Bing web_search engine. Also updates the network integration test URL to AccuWeather (분당구) and adds the `stream` feature to wreq for bytes_stream() support. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
nuri-yoo
reviewed
Jun 11, 2026
wreq's ClientBuilder defaults to redirect::Policy::none(), unlike reqwest which follows up to 10 redirects by default. Add explicit Policy::limited(10) to match the previous behavior. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
reqwest::Clientwithwreq::ClientusingEmulation::Firefox135inweb_fetchstreamfeature towreqdependency to enablebytes_stream()supportBackground
The
reqwestcrate'srustlsbackend produces a distinctive TLS ClientHello fingerprint that Akamai and similar WAFs detect and block at the handshake level — before any HTTP-layer headers are even inspected. This causedweb_fetchto fail with a transport-level error (HTTP/2 INTERNAL_ERRORorECONNRESET) on sites protected by Akamai, even when using a browser-like User-Agent string.The
web_searchBing engine already solved the same problem usingwreqwithEmulation::Firefox135, which replicates the Firefox TLS handshake (cipher suites, extensions, GREASE values) to pass fingerprint-based bot detection. This PR applies the same fix toweb_fetch.Changes
Cargo.tomlfeatures = ["stream"]towreqforbytes_stream()src/tool/impl/builtins/web_fetch.rsreqwest::Client→wreq::Clientwith Firefox emulation; remove deadUSER_AGENTconstantsrc/tool/impl/builtins/web_fetch.rs(tests)Test plan
web_fetchpass (cargo test -p ailoy web_fetch)network_single_fetch_returns_200_and_bodypasses against AccuWeather with--include-ignorednetwork_single_fetch_html_returns_raw_markupstill passes🤖 Generated with Claude Code