Refactor scrape_task_producer to use event_queue Kafka producer#69
Conversation
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 704423dfb6
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if not player_structs: | ||
| return | ||
| error = await player_producer.put(player_structs) | ||
| if isinstance(error, Exception): |
There was a problem hiding this comment.
Handle QueueProducer.put error propagation
The new error handling around player_producer.put(...) never triggers because QueueProducer.put doesn’t return the backend’s Exception (it awaits and returns None). In practice, adapter errors like ProducerNotStartedError or KafkaTimeoutError will be silently ignored here, so enqueue failures won’t be surfaced or logged as intended. Consider propagating the adapter’s return value (or raising inside the adapter) so this check can actually detect failures.
Useful? React with 👍 / 👎.
b87c3a0
into
refactor/scrape_task_producer
…#70) * Refactor scrape_task_producer to use event_queue Kafka producer (#69) * Refactor scrape task producer to event queue * Add lag support to event queue consumers * Restore lag check in scrape task producer * Use queue backend and per-message partition keys * Add event_queue brick to scrape task producer * Add wide_event brick to api_public * Add wide event logging for scrape task producer * Refine scrape task producer wide event logging * cleanup --------- Co-authored-by: extreme4all <>
Motivation
event_queueabstraction for consistency and easier backend swapping.Description
bases/bot_detector/scrape_task_producer/core.pyto useQueueFactory.create_queue(...)and aQueueProducer[ToScrapeStruct]instead of the previousPlayersToScrapeProducer/consumer pair.FetchParamsto add_update_step_flags,set_step,reset_for_new_day, and a strongerupdate_datesignature to centralize and simplify step flag management and date calculations.produce_oneloop with a batchedplayer_producer.put(...)call that constructsToScrapeStructmessages and raises on backend errors.KafkaConfigandKafkaProducerConfigfrom theevent_queueadapters.Testing
uv run pytestwas run).Codex Task