diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml index c11b06a..20ebe53 100644 --- a/.github/workflows/test.yml +++ b/.github/workflows/test.yml @@ -68,6 +68,7 @@ jobs: cli-version: 8.15.0 reporting: github-check github-check-name: Checkly action contract + github-sha: 0123456789abcdef0123456789abcdef01234567 tags: smoke ci: diff --git a/AGENTS.md b/AGENTS.md index e5132c8..81a1e46 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -30,11 +30,18 @@ generated distribution bundle: consumers execute `action.yml` and available. `reporting: github-actions` never detaches. - The Action requires Checkly CLI `8.15.0` or newer. A pinned older stable version must fail before running or performing a preflight request. +- When the project installs `checkly`, run that local CLI after its dependency + installation so config constructs and the CLI share one module session. An + exact stable `cli-version` input must match the local package version. - Exact stable semver pins are compared against the minimum version. Dist-tags, ranges, canaries, and prereleases are allowed because they may identify a compatible build before the next stable release. - Repository and SHA values sent by the Action are hints, not authorization. The Checkly backend must continue verifying account-scoped GitHub App access. +- `github-sha` explicitly overrides event-derived SHA metadata for dispatch and + deployment workflows. Documentation must keep the checkout and reporting SHA + aligned so a GitHub Check never describes a different revision than the one + tested. - Never print API keys or other secrets. Sanitize backend-provided values before placing them in GitHub workflow commands or summaries. diff --git a/README.md b/README.md index 09fe695..6844c8e 100644 --- a/README.md +++ b/README.md @@ -32,7 +32,7 @@ jobs: github-check-name: Checkly PR code checks env: CHECKLY_API_KEY: ${{ secrets.CHECKLY_API_KEY }} - CHECKLY_ACCOUNT_ID: ${{ secrets.CHECKLY_ACCOUNT_ID }} + CHECKLY_ACCOUNT_ID: ${{ vars.CHECKLY_ACCOUNT_ID }} ``` Trigger deployed checks from Checkly: @@ -45,7 +45,7 @@ Trigger deployed checks from Checkly: check-id: abc123,def456 env: CHECKLY_API_KEY: ${{ secrets.CHECKLY_API_KEY }} - CHECKLY_ACCOUNT_ID: ${{ secrets.CHECKLY_ACCOUNT_ID }} + CHECKLY_ACCOUNT_ID: ${{ vars.CHECKLY_ACCOUNT_ID }} ``` Use multiple `--tags` filters by putting one filter per line: @@ -73,17 +73,73 @@ Install the [Checkly GitHub App](https://app.checklyhq.com/settings/account/inte from your Checkly account integrations and grant it access to the repository to use detached GitHub Check reporting. +### CLI resolution + +Checkly config files import constructs from the `checkly` package. When the +working directory has `checkly` installed, this action runs that local CLI after +`install-command` completes so the CLI and constructs share the same module +session. Keep that project dependency at `8.15.0` or newer for GitHub Check +reporting. + +An exact stable `cli-version` pin must match the installed project version. For +example, a project with `checkly@8.15.0` should use `cli-version: 8.15.0`. The +action falls back to `npx checkly@` only when the project does not +install Checkly itself. + For `deployment_status` workflows, the action exposes `github.event.deployment_status.environment_url` as `ENVIRONMENT_URL` when that environment variable is not already set. For pull request preview URLs, pass the target URL explicitly through `env` or the workflow `env` block. +### Choose the GitHub Check commit + +GitHub Checks are attached to a commit, not to a workflow run or deployment. +For pull request events, the action targets the pull request head commit. For +other events, it uses `GITHUB_SHA`. + +Set `github-sha` when the commit being tested differs from `GITHUB_SHA`. This is +common in `repository_dispatch` workflows, where the dispatch payload identifies +the deployed commit. Use the same SHA for checkout and Check reporting so the +result describes the code that actually ran: + +```yaml +name: Validate preview + +on: + repository_dispatch: + types: [preview-ready] + +jobs: + checkly: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + with: + ref: ${{ github.event.client_payload.sha }} + + - uses: checkly/checkly-action@v1 + with: + command: test + install-command: npm ci + github-sha: ${{ github.event.client_payload.sha }} + github-check-name: Checkly preview validation + env: + CHECKLY_API_KEY: ${{ secrets.CHECKLY_API_KEY }} + CHECKLY_ACCOUNT_ID: ${{ vars.CHECKLY_ACCOUNT_ID }} + ENVIRONMENT_URL: ${{ github.event.client_payload.environment_url }} +``` + +The commit must belong to the current repository. The Checkly backend still +verifies that the account's GitHub App installation can access that repository; +`github-sha` is metadata, not authorization. It associates the test session +with that commit and selects where the GitHub Check is reported. + ## Inputs | Input | Description | | --- | --- | | `command` | `test` for local constructs or `trigger` for deployed checks. Defaults to `test`. | -| `cli-version` | Checkly CLI npm version. Requires `8.15.0` or newer and defaults to `latest`. Dist-tags, ranges, canaries, and prereleases are assumed compatible. | +| `cli-version` | Checkly CLI version. Requires `8.15.0` or newer and defaults to `latest`. When the project installs `checkly`, the local CLI runs instead; exact stable pins must match it. Dist-tags, ranges, canaries, and prereleases are assumed compatible. | | `working-directory` | Directory where the CLI command should run. Defaults to `.`. | | `install-command` | Optional command to run before the Checkly CLI command, inside `working-directory`. | | `tags` | One `--tags` filter per line. Each line can contain comma-separated tags. | @@ -105,6 +161,7 @@ target URL explicitly through `env` or the workflow `env` block. | `verbose` | Set to `true` or `false` to pass `--verbose` or `--no-verbose`. | | `reporting` | Where to report the Checkly result: `auto`, `github-check`, or `github-actions`. Defaults to `auto`. | | `github-check-name` | GitHub Check name used when reporting through the Checkly GitHub App. Defaults to `Checkly`. | +| `github-sha` | Commit associated with the test session and targeted by the GitHub Check. Overrides the pull request head SHA or `GITHUB_SHA`. | ## Outputs diff --git a/action.yml b/action.yml index 02724e5..8340177 100644 --- a/action.yml +++ b/action.yml @@ -12,7 +12,7 @@ inputs: required: false default: test cli-version: - description: Checkly CLI npm version to run. Requires 8.15.0 or newer. + description: Checkly CLI npm version. Projects that install checkly run their local CLI; exact stable pins must match it. Requires 8.15.0 or newer. required: false default: latest working-directory: @@ -82,6 +82,9 @@ inputs: github-check-name: description: GitHub Check name to create when reporting through the Checkly GitHub App. Defaults to "Checkly". required: false + github-sha: + description: Commit SHA that receives the GitHub Check. Overrides the SHA inferred from the GitHub event. + required: false outputs: test-session-id: @@ -121,3 +124,4 @@ runs: INPUT_VERBOSE: ${{ inputs.verbose }} INPUT_REPORTING: ${{ inputs.reporting }} INPUT_GITHUB_CHECK_NAME: ${{ inputs.github-check-name }} + INPUT_GITHUB_SHA: ${{ inputs.github-sha }} diff --git a/scripts/run.sh b/scripts/run.sh index 7523732..f3cd290 100755 --- a/scripts/run.sh +++ b/scripts/run.sh @@ -100,6 +100,47 @@ cli_version_is_supported() { return 0 } +cli_version_matches_requested() { + local requested + requested="$(trim "${1:-}")" + local resolved + resolved="$(trim "${2:-}")" + + # An exact stable pin is meaningful only when the project resolves the same + # version. Dist-tags, ranges, canaries, and prereleases intentionally remain + # flexible so users can test unpublished or upcoming CLI builds. + if [[ "$requested" =~ ^v?[0-9]+\.[0-9]+\.[0-9]+$ ]]; then + [[ "${requested#v}" == "$resolved" ]] + return + fi + + return 0 +} + +resolve_local_checkly_version() { + local directory="$1" + + if ! command -v node >/dev/null 2>&1; then + return 0 + fi + + node - "$directory" <<'NODE' +const fs = require('fs') + +try { + const packagePath = require.resolve('checkly/package.json', { + paths: [process.argv[2]], + }) + const { version } = JSON.parse(fs.readFileSync(packagePath, 'utf8')) + if (typeof version === 'string' && version.trim() !== '') { + process.stdout.write(version.trim()) + } +} catch (_) { + process.exit(0) +} +NODE +} + # Reasons are interpolated into ::warning:: workflow commands and the step # summary; constrain them to a safe charset so a malformed or hostile # preflight response can never inject workflow commands or extra lines. @@ -175,8 +216,9 @@ resolve_github_repository() { } resolve_github_sha() { - local sha="" - if is_pull_request_event; then + local sha + sha="$(trim "${INPUT_GITHUB_SHA:-}")" + if [[ -z "$sha" ]] && is_pull_request_event; then sha="$(github_event_value "pull_request.head.sha")" fi printf '%s' "${sha:-${GITHUB_SHA:-}}" @@ -377,7 +419,8 @@ NODE } command_name="$(trim "${INPUT_COMMAND:-test}")" -cli_version="$(trim "${INPUT_CLI_VERSION:-latest}")" +requested_cli_version="$(trim "${INPUT_CLI_VERSION:-latest}")" +cli_version="$requested_cli_version" working_directory="$(trim "${INPUT_WORKING_DIRECTORY:-.}")" install_command="$(trim "${INPUT_INSTALL_COMMAND:-}")" reporting="$(trim "${INPUT_REPORTING:-auto}")" @@ -399,8 +442,8 @@ case "$reporting" in ;; esac -if ! cli_version_is_supported "$cli_version"; then - echo "::error::The Checkly Action needs Checkly CLI 8.15.0 or newer (cli-version is '${cli_version}'). Use cli-version: latest or >= 8.15.0." >&2 +if ! cli_version_is_supported "$requested_cli_version"; then + echo "::error::The Checkly Action needs Checkly CLI 8.15.0 or newer (cli-version is '${requested_cli_version}'). Use cli-version: latest or >= 8.15.0." >&2 exit 1 fi @@ -442,6 +485,33 @@ github_sha="$(resolve_github_sha)" configure_generic_repo_env "$github_repository" "$github_sha" configure_deployment_environment_url +checkly_executable=(npx --yes "checkly@${cli_version}") + +if [[ "${CHECKLY_ACTION_DRY_RUN:-}" != "1" && "${CHECKLY_ACTION_DRY_RUN:-}" != "true" ]]; then + cd "$working_directory" + + if [[ -n "$install_command" ]]; then + echo "Running install command: ${install_command}" + bash -euo pipefail -c "$install_command" + fi + + local_cli_version="$(resolve_local_checkly_version "$PWD")" + if [[ -n "$local_cli_version" ]]; then + if ! cli_version_is_supported "$local_cli_version"; then + echo "::error::Project-local Checkly CLI ${local_cli_version} is older than the required 8.15.0. Update the project's checkly dependency before using this Action." >&2 + exit 1 + fi + + if ! cli_version_matches_requested "$requested_cli_version" "$local_cli_version"; then + echo "::error::Project-local Checkly CLI ${local_cli_version} does not match cli-version '${requested_cli_version}'. Update the project's checkly dependency or use the matching cli-version input." >&2 + exit 1 + fi + + cli_version="$local_cli_version" + checkly_executable=(npx --no-install checkly) + fi +fi + github_check_requested=false github_report_available=false github_report_reason="disabled" @@ -470,7 +540,7 @@ else fi fi -checkly_command=(npx --yes "checkly@${cli_version}" "$command_name") +checkly_command=("${checkly_executable[@]}" "$command_name") if [[ "$detach_run" == "true" ]]; then checkly_command+=("--detach") elif [[ "$github_reporter_run" == "true" ]]; then @@ -536,13 +606,6 @@ if [[ "${CHECKLY_ACTION_DRY_RUN:-}" == "1" || "${CHECKLY_ACTION_DRY_RUN:-}" == " exit 0 fi -cd "$working_directory" - -if [[ -n "$install_command" ]]; then - echo "Running install command: ${install_command}" - bash -euo pipefail -c "$install_command" -fi - output_file="$(mktemp)" set +e "${checkly_command[@]}" 2>&1 | tee "$output_file" diff --git a/scripts/test-integration.sh b/scripts/test-integration.sh index 3e6014d..9f4a94f 100755 --- a/scripts/test-integration.sh +++ b/scripts/test-integration.sh @@ -35,9 +35,16 @@ INSTALL_MARKER="$TEST_DIR/installed" OUTPUT_FILE="$TEST_DIR/github-output" SUMMARY_FILE="$TEST_DIR/github-summary" -mkdir -p "$FAKE_BIN" "$PROJECT_DIR" +mkdir -p "$FAKE_BIN" "$PROJECT_DIR/node_modules/checkly" touch "$OUTPUT_FILE" "$SUMMARY_FILE" +cat > "$PROJECT_DIR/node_modules/checkly/package.json" <<'JSON' +{ + "name": "checkly", + "version": "8.15.0" +} +JSON + cat > "$FAKE_BIN/npx" <<'FAKE_NPX' #!/usr/bin/env bash set -Eeuo pipefail @@ -80,7 +87,7 @@ CHECKLY_ACCOUNT_ID=account-1 \ GITHUB_OUTPUT="$OUTPUT_FILE" \ GITHUB_STEP_SUMMARY="$SUMMARY_FILE" \ GITHUB_REPOSITORY=checkly/checkly-action \ -GITHUB_SHA=0123456789abcdef0123456789abcdef01234567 \ +GITHUB_SHA=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa \ GITHUB_RUN_ID=123456 \ GITHUB_RUN_ATTEMPT=2 \ GITHUB_WORKFLOW=Test \ @@ -95,6 +102,7 @@ INPUT_COMMAND=test \ INPUT_CLI_VERSION=8.15.0 \ INPUT_REPORTING=github-check \ INPUT_GITHUB_CHECK_NAME='Checkly action integration' \ +INPUT_GITHUB_SHA=0123456789abcdef0123456789abcdef01234567 \ INPUT_WORKING_DIRECTORY="$PROJECT_DIR" \ INPUT_INSTALL_COMMAND="$install_command" \ INPUT_TAGS='smoke' \ @@ -102,8 +110,8 @@ INPUT_TAGS='smoke' \ [[ "$(cat "$CWD_FILE")" == "$PROJECT_DIR" ]] [[ "$(cat "$INSTALL_MARKER")" == "installed" ]] -assert_file_contains "$ARGS_FILE" "--yes" -assert_file_contains "$ARGS_FILE" "checkly@8.15.0" +assert_file_contains "$ARGS_FILE" "--no-install" +assert_file_contains "$ARGS_FILE" "checkly" assert_file_contains "$ARGS_FILE" "test" assert_file_contains "$ARGS_FILE" "--detach" assert_file_contains "$ARGS_FILE" "--tags" @@ -134,6 +142,63 @@ assert.equal(request.body.job, 'integration') assert.equal(request.body.eventName, 'push') NODE +cat > "$PROJECT_DIR/node_modules/checkly/package.json" <<'JSON' +{ + "name": "checkly", + "version": "8.14.1" +} +JSON + +set +e +incompatible_local_cli_output="$( + PATH="$FAKE_BIN:$PATH" \ + INPUT_COMMAND=test \ + INPUT_CLI_VERSION=8.15.0 \ + INPUT_REPORTING=github-actions \ + INPUT_WORKING_DIRECTORY="$PROJECT_DIR" \ + "$ROOT_DIR/scripts/run.sh" 2>&1 +)" +status="$?" +set -e + +if [[ "$status" -eq 0 ]]; then + echo "Expected incompatible project-local CLI to fail." >&2 + exit 1 +fi +assert_file_contains <(printf '%s\n' "$incompatible_local_cli_output") "Project-local Checkly CLI 8.14.1 is older than the required 8.15.0" + +cat > "$PROJECT_DIR/node_modules/checkly/package.json" <<'JSON' +{ + "name": "checkly", + "version": "8.16.0" +} +JSON + +set +e +mismatched_local_cli_output="$( + PATH="$FAKE_BIN:$PATH" \ + INPUT_COMMAND=test \ + INPUT_CLI_VERSION=8.15.0 \ + INPUT_REPORTING=github-actions \ + INPUT_WORKING_DIRECTORY="$PROJECT_DIR" \ + "$ROOT_DIR/scripts/run.sh" 2>&1 +)" +status="$?" +set -e + +if [[ "$status" -eq 0 ]]; then + echo "Expected mismatched project-local CLI to fail." >&2 + exit 1 +fi +assert_file_contains <(printf '%s\n' "$mismatched_local_cli_output") "Project-local Checkly CLI 8.16.0 does not match cli-version '8.15.0'" + +cat > "$PROJECT_DIR/node_modules/checkly/package.json" <<'JSON' +{ + "name": "checkly", + "version": "8.15.0" +} +JSON + set +e PATH="$FAKE_BIN:$PATH" \ FAKE_NPX_ARGS_FILE="$ARGS_FILE" \ diff --git a/scripts/test.sh b/scripts/test.sh index 2641d2b..75f3f2d 100755 --- a/scripts/test.sh +++ b/scripts/test.sh @@ -161,6 +161,19 @@ github_report_output="$( assert_contains "$github_report_output" "Reporting: GitHub Check \"Checkly PR code checks\" for checkly/playwright-reporter-demo@head123def456" assert_contains "$github_report_output" "GitHub metadata: source=checkly-action pullRequestNumber=5 environmentUrl=https://preview.example.com" +dispatch_report_output="$( + INPUT_COMMAND=test \ + INPUT_REPORTING=github-check \ + INPUT_GITHUB_SHA=deployed123def456 \ + CHECKLY_ACTION_GITHUB_REPORT_AVAILABLE=true \ + GITHUB_REPOSITORY=checkly/monorepo \ + GITHUB_SHA=defaultbranch123def456 \ + GITHUB_EVENT_NAME=repository_dispatch \ + run_dry +)" + +assert_contains "$dispatch_report_output" "Reporting: GitHub Check for checkly/monorepo@deployed123def456" + github_actions_output="$( INPUT_COMMAND=test \ INPUT_CLI_VERSION=8.15.0 \