Skip to content

feat: Add automatic credential refresh for S3 storage operations#122

Merged
rdheekonda merged 3 commits into
mainfrom
users/raja/add-refresh-s3-token
Jul 29, 2025
Merged

feat: Add automatic credential refresh for S3 storage operations#122
rdheekonda merged 3 commits into
mainfrom
users/raja/add-refresh-s3-token

Conversation

@rdheekonda
Copy link
Copy Markdown
Contributor

@rdheekonda rdheekonda commented Jul 29, 2025

Key Changes:

  • Implement automatic credential refresh system for AWS S3 storage

  • Add decorator pattern for clean retry logic across storage operations

  • Configure default 4 hours credential lifetime with 5-minute refresh buffer

  • Prevent authentication failures during long-running artifact uploads

Added:

  • storage_utils.py with @with_credential_refresh decorator
  • _refresh_credentials_if_needed() method to RunSpan class
  • docstring-parser development dependency for documentation
  • Auto-generated MDX documentation for new functionality
  • Constants: DEFAULT_FS_CREDENTIAL_DURATION ` to 14400s
  • FS_CREDENTIAL_REFRESH_BUFFER to 300 secs

Changed:

  • RunSpan._store_file_by_hash() to use credential refresh decorator
  • ArtifactStorage.store_file() to support credential refresh callbacks
  • ApiClient to handle configurable credential duration parameter

Testing:

✅ Tested with forced credential expiry (immediate refresh)
✅ Tested with natural 15-minute expiry cycle
✅ Verified refresh logs appear: "Refreshing storage credentials"
✅ Confirmed artifact upload during credential transitions

This coordinates with the platform PR to support configurable credential duration on the /user-data/credentials endpoint.

Generated Summary:

  • Added default file system credential duration constant DEFAULT_FS_CREDENTIAL_DURATION (4 hours) and a refresh buffer of 5 minutes (FS_CREDENTIAL_REFRESH_BUFFER) in constants.py.
  • Updated get_user_data_credentials method in ApiClient to accept an optional duration parameter for credential lifetime.
  • Enhanced ArtifactStorage initialization to optionally accept a credential refresher callback to automatically refresh credentials before expiration.
  • Introduced _refresh_storage_credentials method in Dreadnode to handle automatic credential refresh when needed.
  • Applied credential refreshing logic in Dreadnode where it fetches updated credentials periodically when they are close to expiration.
  • Removed the make_docs.py script, which handled documentation generation but was found unnecessary.
  • Overall improvements increase security and usability with better credential management, ensuring they are always valid during operations.

This summary was generated with ❤️ by rigging

@dreadnode-renovate-bot dreadnode-renovate-bot Bot added area/docs Changes to documentation and guides type/docs Documentation updates and improvements labels Jul 29, 2025
@rdheekonda rdheekonda merged commit 47386be into main Jul 29, 2025
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/docs Changes to documentation and guides type/docs Documentation updates and improvements

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant