-
Notifications
You must be signed in to change notification settings - Fork 1
BUILD-8368 S3 cache action initial version #1
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
mikolaj-matuszny-ext-sonarsource
merged 6 commits into
master
from
feat/mmatuszny/BUILD-8368
Jul 4, 2025
Merged
Changes from all commits
Commits
Show all changes
6 commits
Select commit
Hold shift + click to select a range
c18eced
Adjust action repository
mikolaj-matuszny-ext-sonarsource 32ac037
Introduce cognito
mikolaj-matuszny-ext-sonarsource 54c2752
Cognito auth
mikolaj-matuszny-ext-sonarsource 04337fe
added readme and examples
mikolaj-matuszny-ext-sonarsource ba74cc4
Changes after review
mikolaj-matuszny-ext-sonarsource 84be6be
testing cache
mikolaj-matuszny-ext-sonarsource File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,2 +1,2 @@ | ||
| # https://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/customizing-your-repository/about-code-owners | ||
| * @sonarsource/platform-team | ||
| @sonarsource/platform-eng-xp-squad |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,63 @@ | ||
| name: Test | ||
|
|
||
| on: | ||
| push: | ||
| branches: [ master ] | ||
| pull_request: | ||
|
|
||
| jobs: | ||
| build: | ||
| runs-on: sonar-runner-large | ||
| permissions: | ||
| id-token: write | ||
| contents: read | ||
|
|
||
| steps: | ||
| - uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2 | ||
| - uses: jdx/mise-action@5cb1df66ed5e1fb3c670ea0b62fd17a76979826a # v2.3.1 | ||
| - name: Cache Python dependencies | ||
| uses: ./ | ||
| with: | ||
| path: | | ||
| ~/.cache/pip | ||
| key: python-${{ runner.os }}-pytest-requests | ||
| restore-keys: python-${{ runner.os }}- | ||
| - name: Install dependencies | ||
| run: | | ||
| python -m pip install --upgrade pip | ||
| pip install pytest requests | ||
mikolaj-matuszny-ext-sonarsource marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| - name: Run tests | ||
| run: python -m pytest --version | ||
|
|
||
| cache-with-fallback: | ||
| runs-on: sonar-runner-large | ||
mikolaj-matuszny-ext-sonarsource marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| permissions: | ||
| id-token: write | ||
| contents: read | ||
|
|
||
| steps: | ||
| - uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2 | ||
| - uses: jdx/mise-action@5cb1df66ed5e1fb3c670ea0b62fd17a76979826a # v2.3.1 | ||
| - name: Cache Go modules with multiple restore keys | ||
| uses: ./ | ||
| with: | ||
| path: | | ||
| ~/go/pkg/mod | ||
| ~/.cache/go-build | ||
| key: go-${{ runner.os }}-${{ hashFiles('**/go.sum') }} | ||
| restore-keys: | | ||
| go-${{ runner.os }}-${{ hashFiles('**/go.mod') }} | ||
| go-${{ runner.os }}- | ||
| fail-on-cache-miss: false | ||
| - name: Create simple Go module | ||
| run: | | ||
| go mod init example | ||
| echo 'package main | ||
| import "fmt" | ||
| func main() { | ||
| fmt.Println("Hello, World!") | ||
| }' > main.go | ||
mikolaj-matuszny-ext-sonarsource marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| - name: Download dependencies | ||
| run: go mod download | ||
| - name: Build | ||
| run: go build -o hello main.go | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| .claude |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,2 @@ | ||
| python 3.13.5 | ||
| go 1.21.13 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,3 +1,50 @@ | ||
| # gh-action-cache | ||
| # S3 Cache Action | ||
|
|
||
| GitHub action for caching in AWS S3 | ||
| A GitHub Action that provides branch-specific caching on AWS S3 with intelligent fallback to default branch cache entries. | ||
|
|
||
| ## Features | ||
|
|
||
| - **Branch-specific caching**: Cache entries are prefixed with `GITHUB_HEAD_REF` for granular permissions | ||
| - **Intelligent fallback**: Feature branches can fall back to default branch cache when no branch-specific cache exists | ||
| - **S3 storage**: Leverages AWS S3 for reliable, scalable cache storage | ||
| - **AWS Cognito authentication**: Secure authentication using GitHub Actions OIDC tokens | ||
| - **Compatible with actions/cache**: Drop-in replacement with same interface | ||
|
|
||
| ## Usage | ||
|
|
||
| ```yaml | ||
| - uses: SonarSource/gh-action_cache@v1 | ||
| with: | ||
| path: | | ||
| ~/.npm | ||
| ~/.cache | ||
| key: node-${{ runner.os }}-${{ hashFiles('**/package-lock.json') }} | ||
| restore-keys: | | ||
| node-${{ runner.os }} | ||
| s3-bucket: your-cache-bucket | ||
| ``` | ||
|
|
||
| ## Inputs | ||
|
|
||
| | Input | Description | Required | Default | | ||
| |-------|-------------|----------|---------| | ||
| | `path` | Files, directories, and wildcard patterns to cache | Yes | | | ||
| | `key` | Explicit key for restoring and saving cache | Yes | | | ||
| | `restore-keys` | Ordered list of prefix-matched keys for fallback | No | | | ||
| | `s3-bucket` | S3 bucket name for cache storage | No | `sonarsource-s3-cache-dev-bucket` | | ||
| | `upload-chunk-size` | Chunk size for large file uploads (bytes) | No | | | ||
| | `enableCrossOsArchive` | Enable cross-OS cache compatibility | No | `false` | | ||
| | `fail-on-cache-miss` | Fail workflow if cache entry not found | No | `false` | | ||
| | `lookup-only` | Only check cache existence without downloading | No | `false` | | ||
|
|
||
| ## Outputs | ||
|
|
||
| | Output | Description | | ||
| |--------|-------------| | ||
| | `cache-hit` | Boolean indicating exact match for primary key | | ||
|
|
||
| ## Security | ||
|
|
||
| - Uses GitHub Actions OIDC tokens for secure authentication | ||
| - No long-lived AWS credentials required | ||
| - Branch-specific paths provide isolation between branches |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,140 @@ | ||
| name: 'S3 Cache action' | ||
| description: 'Cache files on S3 with branch-specific paths for granular permissions' | ||
| author: 'SonarSource' | ||
|
|
||
| inputs: | ||
| path: | ||
| description: 'A list of files, directories, and wildcard patterns to cache and restore' | ||
| required: true | ||
| key: | ||
| description: 'An explicit key for restoring and saving the cache' | ||
| required: true | ||
| restore-keys: | ||
| description: 'An ordered list of prefix-matched keys to use for restoring stale cache if no cache hit occurred for key' | ||
| upload-chunk-size: | ||
| description: 'The chunk size used to split up large files during upload, in bytes' | ||
| enableCrossOsArchive: | ||
| description: 'An optional boolean when enabled, allows windows runners to save or restore caches that can be restored or saved respectively on other platforms' | ||
| default: 'false' | ||
| fail-on-cache-miss: | ||
| description: 'Fail the workflow if cache entry is not found' | ||
| default: 'false' | ||
| lookup-only: | ||
| description: 'Check if a cache entry exists for the given input(s) (key, restore-keys) without downloading the cache' | ||
| default: 'false' | ||
| s3-bucket: | ||
| description: 'S3 bucket name for cache storage' | ||
| default: 'sonarsource-s3-cache-dev-bucket' | ||
|
|
||
| outputs: | ||
| cache-hit: | ||
| description: 'A boolean value to indicate an exact match was found for the primary key' | ||
| value: ${{ steps.cache.outputs.cache-hit }} | ||
|
|
||
| runs: | ||
| using: 'composite' | ||
| steps: | ||
| - name: Authenticate to AWS | ||
| shell: bash | ||
| env: # TODO: Another set of variables needed for production, support GH cache BUILD-8451 | ||
| POOL_ID: eu-central-1:2f2d946d-08df-415c-9b0c-d097bef49dcc | ||
| AWS_ACCOUNT_ID: 460386131003 | ||
| IDENTITY_PROVIDER_NAME: token.actions.githubusercontent.com | ||
| AUDIENCE: cognito-identity.amazonaws.com | ||
| AWS_REGION: eu-central-1 | ||
| run: | | ||
| # Get GitHub Actions ID token | ||
| ACCESS_TOKEN=$(curl -sLS -H "Authorization: bearer $ACTIONS_ID_TOKEN_REQUEST_TOKEN" "$ACTIONS_ID_TOKEN_REQUEST_URL&audience=$AUDIENCE" | jq -r ".value") | ||
| echo "::add-mask::$ACCESS_TOKEN" | ||
|
|
||
| # Get Identity ID | ||
| identityId=$(aws cognito-identity get-id \ | ||
| --identity-pool-id "$POOL_ID" \ | ||
| --account-id "$AWS_ACCOUNT_ID" \ | ||
| --logins '{"'"$IDENTITY_PROVIDER_NAME"'":"'"$ACCESS_TOKEN"'"}' \ | ||
| --query 'IdentityId' --output text) | ||
|
|
||
| # Get and validate AWS credentials | ||
| awsCredentials=$(aws cognito-identity get-credentials-for-identity \ | ||
| --identity-id "$identityId" \ | ||
| --logins '{"'"$IDENTITY_PROVIDER_NAME"'":"'"$ACCESS_TOKEN"'"}') | ||
|
|
||
| AWS_ACCESS_KEY_ID=$(echo "$awsCredentials" | jq -r ".Credentials.AccessKeyId") | ||
| AWS_SECRET_ACCESS_KEY=$(echo "$awsCredentials" | jq -r ".Credentials.SecretKey") | ||
| AWS_SESSION_TOKEN=$(echo "$awsCredentials" | jq -r ".Credentials.SessionToken") | ||
|
|
||
| echo "::add-mask::$AWS_ACCESS_KEY_ID" | ||
| echo "::add-mask::$AWS_SECRET_ACCESS_KEY" | ||
| echo "::add-mask::$AWS_SESSION_TOKEN" | ||
|
|
||
| if [[ "$AWS_ACCESS_KEY_ID" == "null" || -z "$AWS_ACCESS_KEY_ID" ]]; then | ||
| echo "::error::Failed to obtain AWS Access Key ID" | ||
| exit 1 | ||
| fi | ||
|
|
||
| if [[ "$AWS_SECRET_ACCESS_KEY" == "null" || -z "$AWS_SECRET_ACCESS_KEY" ]]; then | ||
| echo "::error::Failed to obtain AWS Secret Access Key" | ||
| exit 1 | ||
| fi | ||
|
|
||
| if [[ "$AWS_SESSION_TOKEN" == "null" || -z "$AWS_SESSION_TOKEN" ]]; then | ||
| echo "::error::Failed to obtain AWS Session Token" | ||
| exit 1 | ||
| fi | ||
|
|
||
| echo "AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID" >> $GITHUB_ENV | ||
| echo "AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY" >> $GITHUB_ENV | ||
| echo "AWS_SESSION_TOKEN=$AWS_SESSION_TOKEN" >> $GITHUB_ENV | ||
|
|
||
| - name: Prepare cache keys | ||
| shell: bash | ||
| id: prepare-keys | ||
| run: | | ||
| # Prepend GITHUB_HEAD_REF to the main cache key | ||
| BRANCH_KEY="${GITHUB_HEAD_REF}/${{ inputs.key }}" | ||
| echo "branch-key=${BRANCH_KEY}" >> $GITHUB_OUTPUT | ||
|
|
||
| # Process restore keys: keep branch-specific keys and add fallback to default branch | ||
| if [ -n "${{ inputs.restore-keys }}" ]; then | ||
| RESTORE_KEYS="" | ||
| # First, add branch-specific restore keys | ||
| while IFS= read -r line; do | ||
| if [ -n "$line" ]; then | ||
| if [ -n "$RESTORE_KEYS" ]; then | ||
| RESTORE_KEYS="${RESTORE_KEYS}"$'\n'"${GITHUB_HEAD_REF}/${line}" | ||
| else | ||
| RESTORE_KEYS="${GITHUB_HEAD_REF}/${line}" | ||
| fi | ||
| fi | ||
| done <<< "${{ inputs.restore-keys }}" | ||
|
|
||
| # Then, add default branch fallback keys (without GITHUB_HEAD_REF prefix) | ||
| while IFS= read -r line; do | ||
| if [ -n "$line" ]; then | ||
| RESTORE_KEYS="${RESTORE_KEYS}"$'\n'"${line}" | ||
| fi | ||
| done <<< "${{ inputs.restore-keys }}" | ||
|
|
||
| echo "branch-restore-keys<<EOF" >> $GITHUB_OUTPUT | ||
| echo "$RESTORE_KEYS" >> $GITHUB_OUTPUT | ||
| echo "EOF" >> $GITHUB_OUTPUT | ||
| fi | ||
|
|
||
| - name: Cache with runs-on/cache | ||
| uses: runs-on/cache@3a15256b3556fbc5ae15f7f04598e4c7680e9c25 # v4.0.0 | ||
| id: cache | ||
| env: | ||
| RUNS_ON_S3_BUCKET_CACHE: ${{ inputs.s3-bucket }} | ||
| AWS_DEFAULT_REGION: eu-central-1 | ||
| with: | ||
| path: ${{ inputs.path }} | ||
| key: ${{ steps.prepare-keys.outputs.branch-key }} | ||
| restore-keys: ${{ steps.prepare-keys.outputs.branch-restore-keys }} | ||
| upload-chunk-size: ${{ inputs.upload-chunk-size }} | ||
| enableCrossOsArchive: ${{ inputs.enableCrossOsArchive }} | ||
| fail-on-cache-miss: ${{ inputs.fail-on-cache-miss }} | ||
| lookup-only: ${{ inputs.lookup-only }} | ||
|
|
||
| branding: | ||
| icon: 'upload-cloud' | ||
| color: 'blue' |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.