PF-192: Pipeline job running > hour fails with expired creds

[gruntwork-ci]

Implements plan for Linear issue PF-192.

Plan source: https://github.com/gruntwork-io-team/linear-claude-plans/blob/main/plans/PF-192.md

---

Claude's implementation summary

Implementation complete across four repos.

## `pipelines-gitlab-init`

- **New `scripts/lib/credentials.sh`** — sourceable library with no side effects on source:
  - `pipelines_mint_gruntwork_read_token <log>` — exchanges `APERTURE_OIDC_TOKEN` for a Gruntwork read token, echoes it on stdout. Writes the node script's output to a `mktemp` file (removed after sourcing) instead of `credentials.sh` in the repo root, and saves/restores `set -x` so debug/trace logging never prints token material.
  - `pipelines_verify_gruntwork_read_token <token> <log>` — the GitHub API probe, returning a status instead of exiting.
  - `pipelines_publish_gruntwork_read_token <token> <source>` — exports the token plus a new `PIPELINES_GRUNTWORK_READ_TOKEN_SOURCE` (`aperture` or `ci_variable`) and rewrites (rather than appends) both keys in `$GITLAB_ENV` and `build.env`. `build.env` is only touched when it already exists, so a refresh in an execute job doesn't drop a file into the customer's checked-out repo.
- **New `scripts/refresh-credentials.sh`** — standalone entrypoint designed to be sourced. No `exec`, no global `set -e`/`set -u`, no `glab` calls, no note collapsing, no `pipelines-gitlab-actions` clone, no CLI install. Skips refresh when the token came from a CI variable or when `APERTURE_OIDC_TOKEN`/`API_BASE_URL` are absent; on a mint failure it falls back to the inherited token (and only hard-fails when there is no token at all). Works both sourced and executed.
- **`scripts/init.sh`** — now sources the library and calls into it, preserving today's precedence (customer CI variable wins and is verified; otherwise mint from Aperture). Tracing is disabled across the whole token section and around the token-bearing `git clone` URL.
- **`scripts/pipelines-credentials.mjs`** — output path now comes from `PIPELINES_CREDENTIALS_OUTPUT_FILE` (default unchanged) and uses `writeFileSync` instead of `appendFileSync`; the inaccurate dotenv comment is replaced.
- **`README.md`** — documents the scripts and the refresh pattern.

Verified with `bash -n`, `shellcheck -x` (clean), `node --check`, and functional smoke tests covering: fresh mint through a sourced call, stale-key replacement in both env files, `ci_variable` skip, mint failure with and without an inherited token, direct execution, and a `set -x` run confirming no token appears in trace output.

## `pipelines`

- `internal/actions/gitlabyml/gitlabdynamicchildren.go` — new `defaultApertureOIDCAudience` const and `resolveApertureOIDCAudience()` (overridable via `APERTURE_OIDC_AUDIENCE`, mirroring `CI_JOB_IMAGE`); `TemplateJobData` gains `ApertureOIDCAudience`.
- `internal/actions/gitlabyml/gitlabinitdynamicchildren.go` — same audience threaded into the reinit template data.
- All six job templates (`execute`, `provision_new_accounts`, `baseline_core_accounts`, `baseline_new_accounts`, `setup_delegated_repo`, `reinit`) now declare an `APERTURE_OIDC_TOKEN` id_token and clone `pipelines-gitlab-init` + source `refresh-credentials.sh` immediately before the `pipelines-gitlab-actions` clone.
- Tests: all nine golden YAML blocks in `gitlabdynamicchildren_test.go` updated; new `TestCustomApertureOIDCAudience`; new `gitlabinitdynamicchildren_test.go` with a golden `TestReinit` (the reinit path previously had no test) and `TestReinitCustomApertureOIDCAudience`.
- `ARCHITECTURE.md` — new cheat-sheet row for the generated GitLab child-pipeline YAML surface.

**`go test` could not be run here**: the module depends on the private `github.com/gruntwork-io-team/pipelines-hooks`, which is neither cached nor fetchable in this environment (no GitHub credentials), so the package fails at `setup failed` before compiling. To compensate I rendered every template through a standalone stdlib-only harness using the identical funcmap and data shapes; this caught and fixed a `{{ ..ApertureOIDCAudience }}` scoping error in three templates, and the golden strings in the tests were produced from that harness's exact byte output. `gofmt` is clean. The Go changes still need a real `go test ./internal/...` run in CI.

## `gitlab-pipelines-workflows`

- New `.refresh_credentials` anchor (clone `pipelines-gitlab-init`, source `refresh-credentials.sh`), used as the first `script:` entry of `status check` ahead of its clone. `Drift Detection` / `Unlock Unit` / `Unlock All` are untouched since they already run `*init_pipelines` first.
- `pipelines_gitlab_init_ref` default bumped `v1.1.3` → `v1.2.0`.
- YAML parses and the anchor expands correctly (nested list, same shape as the existing `*init_pipelines` usage).

**Not done:** `pipelines_binary_version` is still `v0.58.0`. The plan calls for bumping it to the new CLI release, but that version doesn't exist yet and pinning a guessed tag would break the workflow. Per the plan's own release ordering (`pipelines-gitlab-init` tag → CLI release → workflows bump), this pin must be bumped to the actual CLI release before the child-pipeline half of the fix reaches customers. As it stands this PR fixes `status check` only.

## `docs`

- `docs/2.0/docs/pipelines/architecture/security-controls.md` — the GitLab tab of "Token Strategy" now states that Gruntwork GitHub access tokens expire after one hour and are minted per job from the job's OIDC identity, and that a customer-supplied `PIPELINES_GRUNTWORK_READ_TOKEN` CI variable is used as-is and never refreshed.
- `ci-workflows.md` left alone: its GitLab tab doesn't enumerate jobs or steps, so there's nothing to keep in sync.

All repos in the plan's `target_repos` were available; nothing was skipped for a missing repo.

---

Related PRs in this change set

• pipelines: https://github.com/gruntwork-io-team/pipelines/pull/666
• gitlab-pipelines-workflows: https://gitlab.com/gruntwork-io/pipelines-workflows/-/merge_requests/71
• docs: PF-192: Pipeline job running > hour fails with expired creds docs#3240

Summary by CodeRabbit

• New Features

  • Added automatic credential refresh using the current CI job’s identity.
  • Added support for configured, generated, and inherited authentication tokens.
  • Added secure token publishing for current and downstream jobs.
  • Added configurable credential output locations.

• Bug Fixes

  • Improved repository cloning error handling and exit-code reporting.
  • Prevented credential values from appearing in shell traces or logs.

• Documentation

  • Added setup, usage, token lifetime, CI configuration, and customer-provided token guidance.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: b7fe03b5-b9e1-4bd4-938d-44b12933a8ae

📥 Commits

Reviewing files that changed from the base of the PR and between b9a316d and 1e7fe08.

📒 Files selected for processing (1)

• scripts/pipelines-credentials.mjs

🚧 Files skipped from review as they are similar to previous changes (1)

• scripts/pipelines-credentials.mjs

---

Walkthrough

The change adds shared Bash credential helpers, configurable token output, and a sourceable credential-refresh workflow. Initialization uses these helpers for token verification, minting, publication, and protected cloning. The README documents GitLab setup and token refresh behavior.

Changes

Credential workflow

Layer / File(s)	Summary
Shared credential helpers scripts/lib/credentials.sh, scripts/pipelines-credentials.mjs	Shared helpers mint, verify, publish, and safely update credentials. The Node output file is configurable and overwritten.
Initialization integration scripts/init.sh	Initialization sources the helpers, tracks token provenance, publishes credentials, suppresses token tracing, and returns clone failures.
Job token refresh scripts/refresh-credentials.sh, README.md	The refresh script handles managed tokens, OIDC prerequisites, minting, fallback, tracing, and source or direct execution. The README documents GitLab usage and required variables.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Sequence Diagram(s)

sequenceDiagram
  participant GitLabJob
  participant refreshCredentials
  participant credentialsLibrary
  participant credentialExchange
  participant Aperture
  GitLabJob->>refreshCredentials: Source or execute refresh
  refreshCredentials->>credentialsLibrary: Request token refresh
  credentialsLibrary->>credentialExchange: Invoke credential exchange
  credentialExchange->>Aperture: Exchange OIDC token
  Aperture-->>credentialExchange: Return pipelines token
  credentialExchange-->>credentialsLibrary: Write token output
  credentialsLibrary-->>refreshCredentials: Return token
  refreshCredentials-->>GitLabJob: Publish token and source

Loading

Poem

Tokens hide from tracing light,
Helpers keep the workflow right.
Git clones move with logs kept clean,
Refresh keeps credentials evergreen.
Shell scripts tidy, calm, and bright.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly identifies the PF-192 issue: pipeline jobs exceeding one hour fail because credentials expire.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches 💡 1

🛠️ Fix failing CI checks 💡

• Create stacked PR
• Commit on current branch

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch claude-pr/PF-192-31202520184

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@scripts/pipelines-credentials.mjs`:
- Around line 55-57: Escape the token value as a Bash single-quoted value before
writing the assignment in pipelines-credentials.mjs, including safe handling of
embedded single quotes, so generated content cannot execute commands when
sourced. In scripts/pipelines-credentials.mjs lines 55-57, update the
writeFileSync generation; scripts/lib/credentials.sh lines 43-45 requires no
direct change and should retain the source operation once the generated
assignment is inert.

🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: aed09684-02a0-4940-8f44-c6527b47da6a

📥 Commits

Reviewing files that changed from the base of the PR and between ef0fdc6 and b9a316d.

📒 Files selected for processing (5)

• README.md
• scripts/init.sh
• scripts/lib/credentials.sh
• scripts/pipelines-credentials.mjs
• scripts/refresh-credentials.sh