Update push-to-s3 - move to public requester-pays bucket.#11
Merged
Conversation
This update changes the upload bucket for our s3 data with the rationale of (1) ending object expiration and (2) moving to _requester pays_. __Object Expiration:__ The old bucket had object expiration which caused problems with GitHub's behavior of disabling actions on repositories with limited activity. __Requester Pays:__ The new bucket is public to logged-in users (no anonymous access) and is configured with the _requester pays_ behavior so that egress charges are billed to the account originating the file egress than to the account that owns the bucket (our account). In practice we expect JCSDA to be responsible for the majority of egress traffic but if this changes we will not be responsible for high utilization billing.
|
Will we be able to revert https://github.com/JCSDA-internal/jedi-tools/pull/460 after this or is that a different bucket/issue? |
ytremolet
approved these changes
Jul 31, 2025
ytremolet
left a comment
ytremolet
left a comment
There was a problem hiding this comment.
Nice to have an option where people who request the data pay the egress.
Contributor
Author
|
@ashley314 - yes. The prior bucket was used because we already had it and I assumed that regular cron uploads would resolve the 30-day cleanup (see the description for why that didn't pan out). This change (which I'll apply to all our LFS exports once vetted here) uses a bucket more appropriately configured for the use case. Data requesters will pay egress fees and there is no data expiration rule. |
This was referenced Aug 1, 2025
ashley314
approved these changes
Aug 4, 2025
eap
added a commit
to JCSDA-internal/ufo-data
that referenced
this pull request
Aug 5, 2025
This update changes the upload bucket for our s3 data with the rationale of (1) ending object expiration and (2) moving to requester pays. See JCSDA-internal/mpas-jedi-data#11 for more details.
eap
added a commit
to JCSDA-internal/ioda-data
that referenced
this pull request
Aug 5, 2025
This update changes the upload bucket for our s3 data with the rationale of (1) ending object expiration and (2) moving to requester pays. See JCSDA-internal/mpas-jedi-data#11 for more details.
eap
added a commit
to JCSDA-internal/fv3-jedi-data
that referenced
this pull request
Aug 5, 2025
This update changes the upload bucket for our s3 data with the rationale of (1) ending object expiration and (2) moving to requester pays. See JCSDA-internal/mpas-jedi-data#11 for more details.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This update changes the upload bucket for our s3 data with the rationale of (1) ending object expiration and (2) moving to requester pays.
Object Expiration:
The old bucket had object expiration which caused problems with GitHub's behavior of disabling actions on repositories with limited activity.
Requester Pays:
The new bucket is public to logged-in users (no anonymous access) and is configured with the requester pays behavior so that egress charges are billed to the account originating the file egress than to the account that owns the bucket (our account). In practice we expect JCSDA to be responsible for the majority of egress traffic but if this changes we will not be responsible for high utilization billing.
Checklist