fix: add rate limiting and retry logic for Aurora and OSDG API calls#52
Open
adarsh-7-satyam wants to merge 1 commit into
Open
fix: add rate limiting and retry logic for Aurora and OSDG API calls#52adarsh-7-satyam wants to merge 1 commit into
adarsh-7-satyam wants to merge 1 commit into
Conversation
Signed-off-by: Adarsh Satyam <adarsh5.satyam@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #51
Problem
Neither the Aurora API (
aurora_api.py) nor the OSDG API (app.py) had any rate limiting, retry logic, or 429 error handling. Specifically:aurora_api.pycalledrequests.request()with no timeout and no retry — any transient failure or rate limit would immediately crash the classificationapp.py(OSDG route) had a hardcodedtimeout=1000(clearly a bug — this is 1000 seconds) with no retry or 429 detectionexceptblock, returning a misleading 500 error to the frontendThis is a critical gap for the DMP 2026 goal of bulk testing 100 projects from the DPG registry, since both Aurora and OSDG APIs enforce a 1 request/second rate limit.
Changes Made
backend/aurora_api.pyimport timerequests.request()call with a retry loop (max 3 attempts)timeout=30to prevent indefinite hangingbackend/app.pyimport timerequests.post()OSDG call with a retry loop (max 3 attempts)timeout=1000bug — changed totimeout=30How It Works
Both APIs now follow this flow on every request:
timeout=301 * (2 ** attempt)seconds and retryExponential backoff sequence: 1s → 2s → 4s
Testing Done
import timepresent in both filesmax_retries = 3andretry_delay = 1present in both filestime.sleep(wait)present in both filesstatus_code == 429check present in both filestimeout=1000is completely removed fromapp.pytimeout=30set correctly in both filesNotes
timemodule