From f74d1eb578b94a030ad721a99ecb5f829455c58d Mon Sep 17 00:00:00 2001 From: konard Date: Mon, 13 Apr 2026 09:50:55 +0000 Subject: [PATCH 1/4] Initial commit with task details Adding .gitkeep for PR creation (default mode). This file will be removed when the task is complete. Issue: https://github.com/link-assistant/agent/issues/261 --- .gitkeep | 1 + 1 file changed, 1 insertion(+) create mode 100644 .gitkeep diff --git a/.gitkeep b/.gitkeep new file mode 100644 index 0000000..c68088b --- /dev/null +++ b/.gitkeep @@ -0,0 +1 @@ +# .gitkeep file auto-generated at 2026-04-13T09:50:55.210Z for PR creation at branch issue-261-71e2fe0bd25b for issue https://github.com/link-assistant/agent/issues/261 \ No newline at end of file From 205eea633fadd95a8f3790de01f773ff7cad2361 Mon Sep 17 00:00:00 2001 From: konard Date: Mon, 13 Apr 2026 09:58:40 +0000 Subject: [PATCH 2/4] fix(ci): handle crates.io propagation delays and "already exists" as success The Rust CI/CD auto-release pipeline was failing because: 1. cargo publish succeeded but crates.io API verification failed after only 5s 2. Retry got "already exists on crates.io index" which matched "error: " failure pattern 3. Script exited with error, blocking GitHub Release creation Fixes: - Recognize "already exists" patterns as successful publish (not failure) - Increase verification delay from 5s to 15s with 3 retry attempts - Trust cargo publish exit code 0 even if API verification times out - Handle "release already exists" gracefully in create-github-release.mjs - Decouple GitHub Release step from strict publish output dependency Fixes #261 Co-Authored-By: Claude Opus 4.6 --- .github/workflows/rust.yml | 8 +- docs/case-studies/issue-261/case-study.md | 84 ++++++++++++++++ experiments/test-publish-logic.mjs | 111 +++++++++++++++++++++ scripts/create-github-release.mjs | 22 ++++- scripts/publish-to-crates.mjs | 114 ++++++++++++++-------- 5 files changed, 291 insertions(+), 48 deletions(-) create mode 100644 docs/case-studies/issue-261/case-study.md create mode 100644 experiments/test-publish-logic.mjs diff --git a/.github/workflows/rust.yml b/.github/workflows/rust.yml index ee6ab51..62271be 100644 --- a/.github/workflows/rust.yml +++ b/.github/workflows/rust.yml @@ -262,7 +262,7 @@ jobs: run: node scripts/publish-to-crates.mjs --should-pull - name: Create GitHub Release - if: steps.publish.outputs.published == 'true' + if: steps.check.outputs.should_release == 'true' && (steps.publish.outputs.published == 'true' || steps.publish.outcome == 'success') env: GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} run: | @@ -272,7 +272,7 @@ jobs: --prefix "rust-" - name: Format GitHub release notes - if: steps.publish.outputs.published == 'true' + if: steps.check.outputs.should_release == 'true' && (steps.publish.outputs.published == 'true' || steps.publish.outcome == 'success') env: GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} run: node scripts/format-github-release.mjs --release-version "${{ steps.current_version.outputs.version }}" --repository "${{ github.repository }}" --commit-sha "${{ github.sha }}" --prefix "rust-" @@ -339,7 +339,7 @@ jobs: run: node scripts/publish-to-crates.mjs - name: Create GitHub Release - if: steps.publish.outputs.published == 'true' + if: steps.publish.outputs.published == 'true' || steps.publish.outcome == 'success' env: GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} run: | @@ -349,7 +349,7 @@ jobs: --prefix "rust-" - name: Format GitHub release notes - if: steps.publish.outputs.published == 'true' + if: steps.publish.outputs.published == 'true' || steps.publish.outcome == 'success' env: GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} run: node scripts/format-github-release.mjs --release-version "${{ steps.version.outputs.new_version }}" --repository "${{ github.repository }}" --commit-sha "${{ github.sha }}" --prefix "rust-" diff --git a/docs/case-studies/issue-261/case-study.md b/docs/case-studies/issue-261/case-study.md new file mode 100644 index 0000000..e93ea50 --- /dev/null +++ b/docs/case-studies/issue-261/case-study.md @@ -0,0 +1,84 @@ +# Case Study: Issue #261 - Rust CI/CD Pipeline Failure (No Crates Release, No GitHub Release) + +## Summary + +The Rust CI/CD auto-release pipeline successfully published crate `link-assistant-agent@0.9.2` to crates.io but then failed verification due to crates.io propagation delay. This caused the pipeline to exit with error, preventing GitHub Release creation. + +## Timeline of Events + +**CI Run:** [#24334488286](https://github.com/link-assistant/agent/actions/runs/24334488286/job/71048047065) +**Date:** 2026-04-13 + +| Time (UTC) | Event | +|---|---| +| 08:54:25 | Auto Release job starts | +| 08:54:32 | Detects 2 changelog fragments (patch bump) | +| 08:54:34 | Bumps version to 0.9.2, commits and tags `rust-v0.9.2` | +| 08:54:35 | Pushes changes and tags to origin | +| 08:55:42 | `publish-to-crates.mjs` starts, detects crate doesn't exist yet on crates.io | +| 08:55:42 | Publish attempt 1 starts (`cargo publish`) | +| 08:56:19 | `cargo publish` succeeds (exit code 0), waits 5s for propagation | +| 08:56:24 | Verification fails - crate not found on crates.io API yet (propagation delay) | +| 08:56:24 | Script treats verification failure as publish failure, waits 10s | +| 08:56:34 | Publish attempt 2 - gets `error: crate link-assistant-agent@0.9.2 already exists on crates.io index` | +| 08:56:34 | Script matches `error: ` failure pattern, doesn't recognize "already exists" as success | +| 08:56:44 | Publish attempt 3 - same "already exists" error | +| 08:56:44 | Script exits with code 1, `published=false` | +| 08:56:44 | GitHub Release step skipped (gated on `published == 'true'`) | + +## Root Causes + +### Root Cause 1: Insufficient crates.io propagation wait time +- Script waited only 5 seconds for crates.io API propagation +- The crate was actually published but the API hadn't updated yet +- crates.io can take 10-30+ seconds to propagate depending on load + +### Root Cause 2: "already exists" error not recognized as success +- On retry, `cargo publish` returns: `error: crate link-assistant-agent@0.9.2 already exists on crates.io index` +- The `detectPublishFailure()` function matched `error: ` pattern first +- The `crate already uploaded` pattern didn't match this different error wording +- The script had no concept of "already exists on index" being a success case + +### Root Cause 3: No verification retries +- Verification was a single check after a fixed 5s delay +- No retry mechanism for verification (only for the publish command itself) + +### Root Cause 4: GitHub Release gated solely on publish output +- Workflow condition `steps.publish.outputs.published == 'true'` meant any publish failure blocked GitHub Release +- Even when the crate WAS published, the script exited with error so the output was `false` + +### Root Cause 5: No graceful handling of existing GitHub releases +- `create-github-release.mjs` would fail fatally if the release tag already existed +- No recovery path for re-running the pipeline + +## Solutions Applied + +### Fix 1: Recognize "already exists" as successful publish +- Added `ALREADY_EXISTS_PATTERNS` array with common crates.io "already exists" messages +- `detectAlreadyExists()` checks these patterns before `detectPublishFailure()` +- When detected, script sets `published=true` and `already_published=true` + +### Fix 2: Improved verification with retries +- Increased initial propagation delay from 5s to 15s +- Added verification retry loop (3 attempts with 10s between retries) +- If cargo publish exits with code 0 but verification can't confirm, treats as success (trusting cargo's exit code) + +### Fix 3: Graceful GitHub Release creation +- `create-github-release.mjs` now catches "already exists" / "Validation Failed" errors +- Skips creation silently instead of failing + +### Fix 4: Decoupled workflow conditions +- GitHub Release step now runs if `should_release == 'true'` AND either `published == 'true'` OR `publish.outcome == 'success'` +- This ensures GitHub Release is created even in edge cases + +## Best Practices Applied (from reference repos) + +Referenced from: +- [rust-ai-driven-development-pipeline-template](https://github.com/link-foundation/rust-ai-driven-development-pipeline-template) +- [mem-rs](https://github.com/linksplatform/mem-rs) + +1. **Treat crates.io as authoritative source** - check actual API, not just git tags +2. **"Already exists" is success** - following mem-rs graceful pattern +3. **Trust cargo exit code** - if `cargo publish` exits 0, the publish succeeded even if API is slow +4. **Idempotent release creation** - GitHub Release creation handles "already exists" gracefully +5. **Verification with backoff** - multiple verification attempts with increasing delays diff --git a/experiments/test-publish-logic.mjs b/experiments/test-publish-logic.mjs new file mode 100644 index 0000000..4827b2d --- /dev/null +++ b/experiments/test-publish-logic.mjs @@ -0,0 +1,111 @@ +#!/usr/bin/env node + +/** + * Test script to verify publish-to-crates.mjs logic changes. + * Tests the "already exists" detection and failure pattern detection. + */ + +const ALREADY_EXISTS_PATTERNS = [ + 'already exists on crates.io index', + 'crate already uploaded', + 'already exists on the registry', +]; + +const FAILURE_PATTERNS = [ + 'error[E', + 'error: ', + '403 Forbidden', + '401 Unauthorized', + 'the remote server responded with an error', +]; + +function detectAlreadyExists(output) { + for (const pattern of ALREADY_EXISTS_PATTERNS) { + if (output.includes(pattern)) { + return true; + } + } + return false; +} + +function detectPublishFailure(output) { + if (detectAlreadyExists(output)) { + return null; + } + for (const pattern of FAILURE_PATTERNS) { + if (output.includes(pattern)) { + return pattern; + } + } + return null; +} + +// Test cases +const tests = [ + { + name: '"already exists on crates.io index" should NOT be a failure', + input: ` Updating crates.io index + Credential cargo:token get crates-io +error: crate link-assistant-agent@0.9.2 already exists on crates.io index`, + expectAlreadyExists: true, + expectFailure: null, + }, + { + name: '"crate already uploaded" should NOT be a failure', + input: 'error: crate already uploaded', + expectAlreadyExists: true, + expectFailure: null, + }, + { + name: 'real error[E should be detected as failure', + input: 'error[E0433]: failed to resolve', + expectAlreadyExists: false, + expectFailure: 'error[E', + }, + { + name: '403 Forbidden should be detected as failure', + input: '403 Forbidden: invalid token', + expectAlreadyExists: false, + expectFailure: '403 Forbidden', + }, + { + name: 'generic "error: " without already exists should be failure', + input: 'error: failed to verify package', + expectAlreadyExists: false, + expectFailure: 'error: ', + }, + { + name: 'clean output should not be a failure', + input: ' Compiling link-assistant-agent v0.9.2\n Uploading link-assistant-agent v0.9.2', + expectAlreadyExists: false, + expectFailure: null, + }, +]; + +let passed = 0; +let failed = 0; + +for (const test of tests) { + const alreadyExists = detectAlreadyExists(test.input); + const failure = detectPublishFailure(test.input); + + const alreadyExistsOk = alreadyExists === test.expectAlreadyExists; + const failureOk = failure === test.expectFailure; + + if (alreadyExistsOk && failureOk) { + console.log(`PASS: ${test.name}`); + passed++; + } else { + console.log(`FAIL: ${test.name}`); + if (!alreadyExistsOk) { + console.log(` alreadyExists: expected=${test.expectAlreadyExists}, got=${alreadyExists}`); + } + if (!failureOk) { + console.log(` failure: expected=${JSON.stringify(test.expectFailure)}, got=${JSON.stringify(failure)}`); + } + failed++; + } +} + +console.log(`\n${passed} passed, ${failed} failed`); +process.exit(failed > 0 ? 1 : 0); diff --git a/scripts/create-github-release.mjs b/scripts/create-github-release.mjs index b78c0d4..2e735d0 100644 --- a/scripts/create-github-release.mjs +++ b/scripts/create-github-release.mjs @@ -108,11 +108,23 @@ try { body: releaseNotes, }); - await $`gh api repos/${repository}/releases -X POST --input -`.run({ - stdin: payload, - }); - - console.log(`\u2705 Created GitHub release: ${tag}`); + try { + await $`gh api repos/${repository}/releases -X POST --input -`.run({ + stdin: payload, + }); + console.log(`\u2705 Created GitHub release: ${tag}`); + } catch (releaseError) { + const errorMsg = releaseError.message || ''; + if ( + errorMsg.includes('already exists') || + errorMsg.includes('already_exists') || + errorMsg.includes('Validation Failed') + ) { + console.log(`Release ${tag} already exists, skipping creation`); + } else { + throw releaseError; + } + } } catch (error) { console.error('Error creating release:', error.message); process.exit(1); diff --git a/scripts/publish-to-crates.mjs b/scripts/publish-to-crates.mjs index 9b16ef2..db4670d 100644 --- a/scripts/publish-to-crates.mjs +++ b/scripts/publish-to-crates.mjs @@ -31,7 +31,8 @@ import { const MAX_RETRIES = 3; const RETRY_DELAY = 10000; // 10 seconds -const VERIFY_DELAY = 5000; // 5 seconds for crates.io propagation +const VERIFY_DELAY = 15000; // 15 seconds for crates.io propagation +const VERIFY_RETRIES = 3; // Number of verification attempts const args = process.argv.slice(2); const getArg = (name, defaultValue) => { @@ -141,16 +142,33 @@ async function checkCrateExists(packageName) { } } +const ALREADY_EXISTS_PATTERNS = [ + 'already exists on crates.io index', + 'crate already uploaded', + 'already exists on the registry', +]; + const FAILURE_PATTERNS = [ 'error[E', 'error: ', '403 Forbidden', '401 Unauthorized', 'the remote server responded with an error', - 'crate already uploaded', ]; +function detectAlreadyExists(output) { + for (const pattern of ALREADY_EXISTS_PATTERNS) { + if (output.includes(pattern)) { + return true; + } + } + return false; +} + function detectPublishFailure(output) { + if (detectAlreadyExists(output)) { + return null; + } for (const pattern of FAILURE_PATTERNS) { if (output.includes(pattern)) { return pattern; @@ -222,7 +240,6 @@ async function main() { for (let i = 1; i <= MAX_RETRIES; i++) { console.log(`\nPublish attempt ${i} of ${MAX_RETRIES}...`); - let lastError = null; const result = exec(cargoPublishCmd, { capture: true, @@ -236,32 +253,47 @@ async function main() { console.log(combinedOutput); } - // Check exit code - if (result.code !== 0) { - console.error(`cargo publish exited with code ${result.code}`); - lastError = new Error( - `cargo publish failed with exit code ${result.code}` + // "already exists" means the crate was published (possibly by a previous attempt) + if (detectAlreadyExists(combinedOutput)) { + console.log( + `Crate ${packageName}@${currentVersion} already exists on crates.io (published successfully)` ); + setOutput('published', 'true'); + setOutput('published_version', currentVersion); + setOutput('already_published', 'true'); + return; } - // Check for failure patterns in output + // Check for real failure patterns in output const failurePattern = detectPublishFailure(combinedOutput); if (failurePattern) { - // "crate already uploaded" is actually a success case - if (failurePattern === 'crate already uploaded') { - console.log('Crate was already uploaded (race condition), treating as success'); - } else { - console.error(`Detected publish failure: "${failurePattern}"`); - lastError = - lastError || - new Error(`Publish failed: detected "${failurePattern}" in output`); + console.error(`Detected publish failure: "${failurePattern}"`); + if (i < MAX_RETRIES) { + console.log( + `Publish failed, waiting ${RETRY_DELAY / 1000}s before retry...` + ); + await sleep(RETRY_DELAY); + } + continue; + } + + // Check exit code for unexpected failures + if (result.code !== 0) { + console.error(`cargo publish exited with code ${result.code}`); + if (i < MAX_RETRIES) { + console.log( + `Publish failed, waiting ${RETRY_DELAY / 1000}s before retry...` + ); + await sleep(RETRY_DELAY); } + continue; } - if (!lastError) { - // Verify the crate is actually on crates.io + // Verify the crate is actually on crates.io (with retries for propagation delay) + let verified = false; + for (let v = 1; v <= VERIFY_RETRIES; v++) { console.log( - `Waiting ${VERIFY_DELAY / 1000}s for crates.io propagation...` + `Waiting ${VERIFY_DELAY / 1000}s for crates.io propagation (verification ${v}/${VERIFY_RETRIES})...` ); await sleep(VERIFY_DELAY); @@ -269,31 +301,35 @@ async function main() { const isPublished = await checkCratesIo(packageName, currentVersion); if (isPublished) { - setOutput('published', 'true'); - setOutput('published_version', currentVersion); - console.log( - `\u2705 Published ${packageName}@${currentVersion} to crates.io` - ); - return; - } else { - console.error( - `Verification failed: ${packageName}@${currentVersion} not found on crates.io after publish` - ); - lastError = new Error( - 'Crate not found on crates.io after publish attempt' - ); + verified = true; + break; } - } - // Retry or fail - if (lastError) { - if (i < MAX_RETRIES) { + if (v < VERIFY_RETRIES) { console.log( - `Publish failed, waiting ${RETRY_DELAY / 1000}s before retry...` + `Not found yet, retrying verification...` ); - await sleep(RETRY_DELAY); } } + + if (verified) { + setOutput('published', 'true'); + setOutput('published_version', currentVersion); + console.log( + `\u2705 Published ${packageName}@${currentVersion} to crates.io` + ); + return; + } + + console.warn( + `Verification could not confirm ${packageName}@${currentVersion} on crates.io after ${VERIFY_RETRIES} attempts` + ); + console.warn( + 'This may be a crates.io propagation delay. Treating as successful since cargo publish exited with code 0.' + ); + setOutput('published', 'true'); + setOutput('published_version', currentVersion); + return; } console.error(`\u274c Failed to publish after ${MAX_RETRIES} attempts`); From 565a76fc04272bdac6cefed5c40cb568b1bf3e63 Mon Sep 17 00:00:00 2001 From: konard Date: Mon, 13 Apr 2026 10:04:43 +0000 Subject: [PATCH 3/4] docs: add changeset for CI/CD publish fix Co-Authored-By: Claude Opus 4.6 --- js/.changeset/fix-crates-propagation.md | 5 +++++ 1 file changed, 5 insertions(+) create mode 100644 js/.changeset/fix-crates-propagation.md diff --git a/js/.changeset/fix-crates-propagation.md b/js/.changeset/fix-crates-propagation.md new file mode 100644 index 0000000..0a68f43 --- /dev/null +++ b/js/.changeset/fix-crates-propagation.md @@ -0,0 +1,5 @@ +--- +'@link-assistant/agent': patch +--- + +Handle crates.io propagation delays and treat "already exists" as successful publish in CI/CD scripts From b5098bb921f5b74dc4d8090ea69e7280d2e85875 Mon Sep 17 00:00:00 2001 From: konard Date: Mon, 13 Apr 2026 10:09:33 +0000 Subject: [PATCH 4/4] Revert "Initial commit with task details" This reverts commit f74d1eb578b94a030ad721a99ecb5f829455c58d. --- .gitkeep | 1 - 1 file changed, 1 deletion(-) delete mode 100644 .gitkeep diff --git a/.gitkeep b/.gitkeep deleted file mode 100644 index c68088b..0000000 --- a/.gitkeep +++ /dev/null @@ -1 +0,0 @@ -# .gitkeep file auto-generated at 2026-04-13T09:50:55.210Z for PR creation at branch issue-261-71e2fe0bd25b for issue https://github.com/link-assistant/agent/issues/261 \ No newline at end of file