Add 429 retry/backoff to Grain API client - #5
Draft
dplakon wants to merge 1 commit into
Draft
Conversation
The Grain client fired paginated requests back-to-back with no throttling and raised immediately on any HTTPError. Grain rate-limits bursty pagination, so the daily fetch reliably failed with HTTP 429 after the first page, blocking the meeting-processing pipeline. Add a shared `_urlopen_with_retry` helper that retries 429 and transient 5xx responses with exponential backoff, honoring the Retry-After header when present. Both recording listing and transcript fetches now go through it. Retry count and base backoff are tunable via GRAINIAC_MAX_RETRIES and GRAINIAC_BACKOFF_SECONDS. Co-Authored-By: Oz <oz-agent@warp.dev>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The daily meeting-processing pipeline (
grainiac-orchestrator) failed at thefetch_daily_meetings.pystep withHTTP Error 429: Too Many Requests. The Grain client (scripts/grain_client.py) paginated through recordings by firing requests back-to-back with no throttling, and_getraised immediately on anyHTTPError. Grain rate-limits bursty pagination, so the fetch reliably died after the first page and no meetings could be processed.Change
_urlopen_with_retryhelper that retries429and transient5xxresponses with exponential backoff, honoring theRetry-Afterresponse header when present._get) and transcript fetching (get_transcript_text) through the helper.GRAINIAC_MAX_RETRIES(default 6) andGRAINIAC_BACKOFF_SECONDS(default 2).Verification
With this change,
python scripts/fetch_daily_meetings.py todayretried the rate-limited pages and completed successfully, fetching all recordings and identifying the day's external customer meetings.Conversation: https://app.warp.dev/conversation/4479e69b-9f6b-482a-b74f-5f79c31a9354
Run: https://oz.warp.dev/runs/019f8d58-5046-706a-aa0e-a74b287fb1e0
This PR was generated with Oz.