Skip to content

fix: retry Helm downloads and extend install timeout - #115

Open
bugkeep wants to merge 1 commit into
casosorg:masterfrom
bugkeep:fix/helm-download-reliability
Open

fix: retry Helm downloads and extend install timeout#115
bugkeep wants to merge 1 commit into
casosorg:masterfrom
bugkeep:fix/helm-download-reliability

Conversation

@bugkeep

@bugkeep bugkeep commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

Problem

App Store installs can fail on transient Helm repository or OCI registry errors because chart metadata and archives are downloaded once without retry or reuse. Opening the install dialog and then submitting the install can also download the same large artifact twice.

The install path also uses a fixed 10-minute readiness timeout while an active task is considered abandoned after 11 minutes. A healthy large installation can therefore be marked stale while Helm is still running, allowing another request to install the same release. When Helm eventually fails, the release and partially created resources are not cleaned up automatically.

What changed

  • retry classic repository index/chart downloads and OCI tag/pull operations for transient network errors, HTTP 408/429, and HTTP 5xx responses
  • apply a two-minute deadline to each download attempt and use context-aware retry delays
  • cache indexes and chart archives in process for 15 minutes, bounded to 32 entries and 128 MiB
  • add helmInstallTimeout, defaulting to 20 minutes
  • wait for application resources and jobs, and enable Helm atomic install cleanup on timeout or failure
  • refresh active task timestamps once per minute so healthy long-running installs are not expired; when a process stops, the existing 11-minute stale-task recovery remains effective

Compatibility and failure behavior

The configuration is backward compatible: an omitted value uses the new 20-minute default, while an invalid or non-positive duration is rejected before installation. Retries are finite and only cover transient failures. Authentication errors, missing charts, invalid values, and other permanent failures return immediately.

The cache is process-local, time-limited, and memory-bounded. No API payload, database schema, workflow, application-specific values, or image source is changed.

Validation

  • go test ./...
  • go vet ./...
  • go build ./...
  • fork CI passed Running Go tests, Back-end, Front-end, Go-Linter, and UI tests
  • a local task-heartbeat test passed 20 consecutive runs and was removed before commit, following the repository rule against uploading Go test files
  • Airflow OCI chart: first load 5.00s, cached load 62ms
  • hello-world chart: installed in 43.7s, reached deployed, exposed its endpoint, and uninstalled cleanly
  • Airflow timeout path: diagnostics were recorded and atomic cleanup removed the failed release resources, namespace, and provisioned volume
  • GitLab default chart: the application-specific required-value error was reported before cluster resources were created

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant