Skip to content

Refactor scrape_task_producer to use event_queue Kafka producer#69

Merged
extreme4all merged 8 commits intorefactor/scrape_task_producerfrom
codex/refactor-scrape-task-producer-to-use-event_queue
Feb 5, 2026
Merged

Refactor scrape_task_producer to use event_queue Kafka producer#69
extreme4all merged 8 commits intorefactor/scrape_task_producerfrom
codex/refactor-scrape-task-producer-to-use-event_queue

Conversation

@extreme4all
Copy link
Contributor

Motivation

  • Replace direct kafka producer/consumer usage in the scrape task producer with the shared event_queue abstraction for consistency and easier backend swapping.
  • Simplify the fetch parameter state machine and daily reset logic to improve readability and maintainability.
  • Batch and centralize queueing logic to reduce per-message operations and surface producer errors explicitly.

Description

  • Updated bases/bot_detector/scrape_task_producer/core.py to use QueueFactory.create_queue(...) and a QueueProducer[ToScrapeStruct] instead of the previous PlayersToScrapeProducer/consumer pair.
  • Reworked FetchParams to add _update_step_flags, set_step, reset_for_new_day, and a stronger update_date signature to centralize and simplify step flag management and date calculations.
  • Replaced the per-message produce_one loop with a batched player_producer.put(...) call that constructs ToScrapeStruct messages and raises on backend errors.
  • Removed use of the consumer lag check and consumer startup/stop since the producer-only flow is used now, and centralized Kafka config creation using KafkaConfig and KafkaProducerConfig from the event_queue adapters.

Testing

  • No automated tests were executed as part of this change (no uv run pytest was run).

Codex Task

Copy link

@chatgpt-codex-connector chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 704423dfb6

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines 87 to 90
if not player_structs:
return
error = await player_producer.put(player_structs)
if isinstance(error, Exception):

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Handle QueueProducer.put error propagation

The new error handling around player_producer.put(...) never triggers because QueueProducer.put doesn’t return the backend’s Exception (it awaits and returns None). In practice, adapter errors like ProducerNotStartedError or KafkaTimeoutError will be silently ignored here, so enqueue failures won’t be surfaced or logged as intended. Consider propagating the adapter’s return value (or raising inside the adapter) so this check can actually detect failures.

Useful? React with 👍 / 👎.

@extreme4all extreme4all merged commit b87c3a0 into refactor/scrape_task_producer Feb 5, 2026
5 of 6 checks passed
extreme4all added a commit that referenced this pull request Feb 5, 2026
…#70)

* Refactor scrape_task_producer to use event_queue Kafka producer (#69)

* Refactor scrape task producer to event queue

* Add lag support to event queue consumers

* Restore lag check in scrape task producer

* Use queue backend and per-message partition keys

* Add event_queue brick to scrape task producer

* Add wide_event brick to api_public

* Add wide event logging for scrape task producer

* Refine scrape task producer wide event logging

* cleanup

---------

Co-authored-by: extreme4all <>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant