Handle Grain API rate limiting in the daily meeting fetch - #8
Draft
dplakon wants to merge 1 commit into
Draft
Conversation
The daily meeting fetch was failing with HTTP 429 before it could return any meetings. Two causes: 1. `_get` had no retry or backoff, so a single 429 aborted the whole run. 2. `list_all_recordings` walked the entire recording history (up to 100 pages) on every run, even when the caller only needed one day. The Grain API allows 30 requests per window, so the fetch exhausted its budget partway through pagination. Changes: - Add retry with exponential backoff (honouring `Retry-After`) for 429 and 5xx responses, and proactively pause when `x-ratelimit-remaining` is nearly exhausted. - Route `get_transcript_text` through the same retry path. - Add `stop_before_date` to `list_all_recordings`. Recordings come back newest-first and the API's date parameters are ignored server-side, so pagination can stop once a whole page predates the target day. `fetch_daily_meetings.py` now passes the target date, taking a single-day fetch from potentially 100 requests down to two. Co-Authored-By: Oz <oz-agent@warp.dev>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Today's
grainiac-orchestratorrun could not get off the ground:scripts/fetch_daily_meetings.py todayfailed withHTTP Error 429: Too Many Requestson every attempt, before returning a single meeting.Two independent causes:
_getraised on the first429, aborting the whole run.list_all_recordingswalked the entire recording history (up to 100 pages × 20 recordings) on every run, even though the caller only wanted one day. The Grain API advertises a budget of 30 requests per window (x-ratelimit-limit: 30), so the fetch reliably exhausted its quota partway through pagination — it consistently died on page 2 of a fresh run because prior runs had already drained the window.The API's
afterDatetime/beforeDatetimeparameters do not filter server-side (confirmed: a request scoped to2026-07-30still returned recordings from2026-07-27), which matches the existing note in the orchestrator skill. But results are returned newest-first, so a single-day fetch never needs to read past the target date.Changes
scripts/grain_client.py_requesthelper with exponential backoff on429and5xx, honouringRetry-Afterwhen present.x-ratelimit-remainingdrops to the threshold, rather than waiting to be rejected.get_transcript_textnow shares the same retry path (it previously had its own un-retried request).list_all_recordingsacceptsstop_before_dateand stops paging once an entire page predates the target day.scripts/fetch_daily_meetings.pystop_before_date.Result
A single-day fetch now costs 2 list requests instead of up to 100. Verified end to end against the live API — the run that previously failed now completes:
Follow-ups for a human (not fixed here — these are config, not code)
GRAINIAC_NOTION_DATABASE_IDis stale. The configured ID (3a043263616d803d98de000c5b916204) returns404from the Notion API. The database the pipeline actually writes to is "Account Management" (3a043263-616d-80bd-a6da-f099fa90796a), which holds all 44 account pages and follows the Grainiac page template. The env var should be repointed.GRAINIAC_NOTION_TITLE_PROPERTYis unset. That database's title property isAccount name, butnotion_client._title_property()defaults toCompany, sofind_company_pagewould filter on a non-existent property. The var needs to be set toAccount name.Conversation: https://app.warp.dev/conversation/0c3b3ac6-8718-4b83-990b-d450a1868f9a
Run: https://oz.warp.dev/runs/019fb68b-2f46-73e6-b14c-ca2dc1614964
This PR was generated with Oz.