When using httpx the user agent header is missing which breaks use cases like Github OAuth flows

[adewale]

When writing code with httpx that calls services like the Github OAuth flow I end up getting errors like:

Request forbidden by administrative rules.
Please make sure your request has a User-Agent header.

You can see more auto-generated analysis and a coding example here: https://github.com/adewale/python-workers-issues/tree/main/3-httpx-headers

It seems like httpx on Python Workers strips out the User-Agent header. Switching to native js_fetch fixes it.

if HAS_PYODIDE:
    opts = _to_js_value({"method": method, "headers": all_headers})
    response = await js.fetch(url, opts)
    ...
else:
    # CPython fallback — httpx works correctly here
    import httpx
    async with httpx.AsyncClient(timeout=httpx.Timeout(timeout)) as client:
        resp = await client.request(method, url, headers=all_headers, content=body)
        ...

[hoodmane]

Yeah it removes it because firefox gets angry when you set a custom user-agent:
https://bugzilla.mozilla.org/show_bug.cgi?id=1629921

[hoodmane]

Presumably we should make it only remove the user agent when we feature detect as in the browser. Though that would fix it on Node but Cloudflare currently feature detects as a browser so it wouldn't help us without some adjustment.

[danlapid]

Can we only remove it on firefox then?

[hoodmane]

It's probably easier to feature detect node and cloudflare and not remove it there. I think Chrome prints a warning that the request tried to set user-agent and is ignoring it, so we don't want to pass it there either.

[ryanking13]

but Cloudflare currently feature detects as a browser

Huh, I didn't know this.