Skip to content

feat: Add automatic credential refresh for S3 storage operations#121

Closed
rdheekonda wants to merge 6 commits into
mainfrom
users/raja/update-s3-access-token-duration
Closed

feat: Add automatic credential refresh for S3 storage operations#121
rdheekonda wants to merge 6 commits into
mainfrom
users/raja/update-s3-access-token-duration

Conversation

@rdheekonda
Copy link
Copy Markdown
Contributor

Key Changes:

  • Implement automatic credential refresh system for AWS S3 storage

  • Add decorator pattern for clean retry logic across storage operations

  • Configure default 4 hours credential lifetime with 5-minute refresh buffer

  • Prevent authentication failures during long-running artifact uploads

Added:

  • storage_utils.py with @with_credential_refresh decorator
  • _refresh_credentials_if_needed() method to RunSpan class
  • docstring-parser development dependency for documentation
  • Auto-generated MDX documentation for new functionality
  • Constants: DEFAULT_FS_CREDENTIAL_DURATION ` to 14400s
  • FS_CREDENTIAL_REFRESH_BUFFER to 300 secs

Changed:

  • RunSpan._store_file_by_hash() to use credential refresh decorator
  • ArtifactStorage.store_file() to support credential refresh callbacks
  • ApiClient to handle configurable credential duration parameter

Testing:

✅ Tested with forced credential expiry (immediate refresh)
✅ Tested with natural 15-minute expiry cycle
✅ Verified refresh logs appear: "Refreshing storage credentials"
✅ Confirmed artifact upload during credential transitions

This coordinates with the platform PR to support configurable credential duration on the /user-data/credentials endpoint.

@dreadnode-renovate-bot dreadnode-renovate-bot Bot added area/docs Changes to documentation and guides area/python Changes to Python package configuration and dependencies type/docs Documentation updates and improvements labels Jul 29, 2025
@rdheekonda rdheekonda closed this Jul 29, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/docs Changes to documentation and guides area/python Changes to Python package configuration and dependencies type/docs Documentation updates and improvements

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant