Switch to requests.get for images #115
Conversation
|
What do you mean by "urlretrieve is now blocked by cloudflare"? In the end, it's just a library that makes http requests. Is there something specific about the request that is causing it to be blocked? Maybe because we're not passing headers? But we could just pass them. e.g. see https://stackoverflow.com/questions/45035861/urlretrieve-for-image-returns-http-error-403-forbidden which looks similar. |
|
Yeah, I think it's the lack of headers. But urlretrieve is supposedly obsolete now anyway, I think? |
|
Yes, checking on it, it does sound like it is. It may still be the path of least resistance right now to just pass the headers. But if we want to switch, we could write our own urlretrieve equivalent wrapper based on requests.get, and the mock that wrapper the same way we mock the real thing now. And yes, I'm sure AI would easily achieve this. I've switched to using Claude Code, and it's just ridiculously good :) |
|
Ooops, I didn't see this pull request. I've just merged 690e371 which obviates the need for fetching the image in the first place by embedding one in the test script. Of course, this doesn't help the actual code fetching images :) Doing something like the first half of this commit seems sensible. I'll push something through. |
We're not allowed to fetch images with urllib's default user-agent. Instead, switch to using requests & fetch using our custom UA. Remove the now-unused urllib mocks. Fixes #115
3 participants
We need this as urlretrieve is now blocked by cloudflare.
However, the tests now fail because we aren't properly mocking the call to get the image. Do you know how to fix the tests @davidebbo ? I guess one of the AI engines could probably figure it out if you don't have time!