feat(cronjobs): add shared HTTP client wrapper. Refs #1214 by Prasad8830 · Pull Request #1253 · GreedyBear-Project/GreedyBear

Prasad8830 · 2026-04-17T08:43:46Z

Description

This PR introduces a shared HTTP client wrapper (HttpClient) based on requests.Session for our cronjob fetchers. This represents the first step towards standardising how cron jobs handle timeouts, retries, and errors, as discussed in #1214.

The wrapper automatically applies a default 10-second timeout, implements an exponential backoff retry strategy for standard server errors/rate limits, and consistently enforces raise_for_status(). Full unit tests for this wrapper have been included.

Note: This PR only introduces the wrapper and its tests. Subsequent PRs will migrate tor_exit_nodes, whatsmyip, and the remaining endpoints to use this new client.

Related issues

Refs Standardize cronjob HTTP fetchers with a shared client wrapper #1214

Type of change

Bug fix (non-breaking change which fixes an issue).
New feature (non-breaking change which adds functionality).
Breaking change (fix or feature that would cause existing functionality to not work as expected).
Chore (refactoring, dependency updates, CI/CD changes, code cleanup, docs-only changes).

Checklist

Formalities

I have read and understood the rules about how to Contribute to this project.
I chose an appropriate title for the pull request in the form: <feature name>. Closes #999
My branch is based on develop.
The pull request is for the branch develop.
I have reviewed and verified any LLM-generated code included in this PR.

Docs and tests

I documented my code changes with docstrings and/or comments.
I have checked if my changes affect user-facing behaviour that is described in the docs. If so, I also included an update to the wiki in the description of this PR.
Linter (Ruff) gave 0 errors. If you have correctly installed pre-commit, it runs these checks and makes these adjustments on your behalf.
I have added tests for the feature/bug I solved.
All the tests gave 0 errors.

GUI changes

Ignore this section if you did not make any changes to the GUI.

I have provided a screenshot of the result in the PR.
I have created new frontend tests for the new component or updated existing ones.

regulartim

Hey @Prasad8830 ! Nice work overall! :) A few things though:

The issue was also about standardized logging but I can see no logging here. Please add a sensible amount of logging and also be thoughtful with the log levels.
The retry behavior is not actually tested. Please add some tests for that, especially for the POST + retry case.

Prasad8830 · 2026-04-22T07:52:52Z

Hey @Prasad8830 ! Nice work overall! :) A few things though:

The issue was also about standardized logging but I can see no logging here. Please add a sensible amount of logging and also be thoughtful with the log levels.

The retry behavior is not actually tested. Please add some tests for that, especially for the POST + retry case.

Thank you for bringing that to my attention! I will make the necessary corrections.

regulartim · 2026-05-11T07:31:28Z

+            total=retries,
+            status_forcelist=[429, 500, 502, 503, 504],
+            backoff_factor=backoff_factor,
+            allowed_methods=frozenset(["HEAD", "GET", "PUT", "DELETE", "OPTIONS", "TRACE", "POST"]),


I just noticed that this might be a problem: retrying POST/PUT/DELETE requests can lead to unexpected behaviour on non-idempotent calls. For example posting the same malware data twice. POST is also not in the defaults. What do you think about that?

Can you please answer this question?

regulartim · 2026-05-19T05:51:07Z

Hey @Prasad8830 ! Are you still working on this?

Prasad8830 · 2026-05-21T05:04:05Z

Hey @Prasad8830 ! Are you still working on this?

Yes, I was working on this migration, and along with this PR, I created two more PRs. Since this project was part of GSoC 2026, is it still open for contributions?

regulartim · 2026-05-21T07:47:07Z

Yes, I was working on this migration, and along with this PR, I created two more PRs.

Yes, I saw that. But this has to get merged first, the other two are based on the changes here, right?

Since this project was part of GSoC 2026, is it still open for contributions?

Of course! We are always open to contributions! :)

regulartim · 2026-05-21T07:49:06Z

BTW, the CI is failing because of a new Ruff linting rule that we recently introduced. In test_http_client.py you have to rename the function argument format to something else, as it is shadowing a Python builtin.

Prasad8830 · 2026-05-21T11:51:32Z

Yes, I was working on this migration, and along with this PR, I created two more PRs.

Yes, I saw that. But this has to get merged first, the other two are based on the changes here, right?

Since this project was part of GSoC 2026, is it still open for contributions?

Of course! We are always open to contributions! :)

Ok, then I will fix current PRs, and then I will do the migration of the next APIs.

Prasad8830 · 2026-05-22T07:04:23Z

BTW, the CI is failing because of a new Ruff linting rule that we recently introduced. In test_http_client.py you have to rename the function argument format to something else, as it is shadowing a Python builtin.

Hi @regulartim, I have fix this bug.

regulartim · 2026-05-25T06:05:23Z

Hey @Prasad8830 ! Unfortunately the linter is still not happy. Make merge the develop branch into your recent feature branch, to get all the current linting rules.

Prasad8830 · 2026-05-25T07:17:09Z

Hey @Prasad8830 ! Unfortunately the linter is still not happy. Make merge the develop branch into your recent feature branch, to get all the current linting rules.

Hi @regulartim, can you now check? I merged develop and fixed the failing tests

regulartim

Hey @Prasad8830 ! Linter is happy now! 👍 But you did not address the feedback from my last review, as far as I can see.

regulartim · 2026-05-26T14:39:12Z

+            total=retries,
+            status_forcelist=[429, 500, 502, 503, 504],
+            backoff_factor=backoff_factor,
+            allowed_methods=frozenset(["HEAD", "GET", "PUT", "DELETE", "OPTIONS", "TRACE", "POST"]),


Can you please answer this question?

feat(cronjobs): add shared HTTP client wrapper (GreedyBear-Project#1214)

4cb3cc2

regulartim requested changes Apr 22, 2026

View reviewed changes

Comment thread greedybear/cronjobs/http_client.py

regulartim mentioned this pull request Apr 22, 2026

Migrate tor_exit_nodes HTTP client to httpx (part of #1214) #1222

Closed

16 tasks

fix(cronjobs): address PR feedback on http_client logging and retries

1d244cb

Prasad8830 requested a review from regulartim April 25, 2026 07:45

Prasad8830 mentioned this pull request Apr 25, 2026

refactor(cronjobs): migrate mass_scanners, firehol, and spamhaus_drop to HttpClient. #1309

Open

16 tasks

regulartim requested changes May 11, 2026

View reviewed changes

rename the function argument format for passing newly added test

2000286

Prasad8830 requested a review from regulartim May 22, 2026 07:04

Prasad8830 added 2 commits May 25, 2026 12:37

Merge branch 'develop' into feat/http-client-wrapper-1214

0509f63

fix ruff errors

490dae2

regulartim requested changes May 26, 2026

View reviewed changes

Uh oh!

Conversation

Prasad8830 commented Apr 17, 2026

Description

Related issues

Type of change

Checklist

Formalities

Docs and tests

GUI changes

Uh oh!

regulartim left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Prasad8830 commented Apr 22, 2026

Uh oh!

regulartim May 11, 2026

Choose a reason for hiding this comment

Uh oh!

regulartim May 26, 2026

Choose a reason for hiding this comment

Uh oh!

regulartim commented May 19, 2026

Uh oh!

Prasad8830 commented May 21, 2026

Uh oh!

regulartim commented May 21, 2026

Uh oh!

regulartim commented May 21, 2026

Uh oh!

Prasad8830 commented May 21, 2026

Uh oh!

Prasad8830 commented May 22, 2026

Uh oh!

regulartim commented May 25, 2026

Uh oh!

Prasad8830 commented May 25, 2026

Uh oh!

regulartim left a comment

Choose a reason for hiding this comment

Uh oh!

regulartim May 26, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants