Problem
For HTTP(S) HelmRepository objects, source-controller downloads the full index.yaml on every reconcile interval, unconditionally: CacheIndex → DownloadIndex issues a plain GET, and only after the download completes is the digest compared against the current artifact to decide the reconcile is a no-op (helmrepository_controller.go#L464-L468). There is no If-None-Match / If-Modified-Since anywhere in the fetch path (the only ETag references in the codebase are the S3/GCS test mocks).
Real-world indexes are large (grafana ~2 MB, prometheus-community ~4 MB, bitnami historically >10 MB), and the cost is linear in the number of objects: N HelmRepository objects pointing at the same or different repos re-download N full indexes per interval, forever, even when nothing changed. In multi-tenant setups where each tenant namespace declares its own HelmRepository (cross-namespace refs disabled), N grows with the tenant count.
Measurements
Bench: k3s + source-controller v1.9.3, defaults; one HTTP helm repo with a 6.0 MB index; N HelmRepository objects with interval: 1m; server-side byte/request counters; idle steady state (index unchanged), 5-minute windows.
| N objects |
index GETs / 5 min |
bytes / 5 min |
projected / day |
| 1 |
5 |
31.0 MB |
1,440 GETs / 8.9 GB |
| 10 |
50 |
309.7 MB |
14,400 GETs / 89 GB |
| 50 |
249 |
1.54 GB |
~72,000 GETs / ~444 GB |
Every one of those bytes is thrown away: the digest comparison concludes "unchanged" after each download.
Proposal
Remember the ETag / Last-Modified response headers per object (an in-memory map keyed by object UID is sufficient — a controller restart just pays one full fetch), and send If-None-Match / If-Modified-Since on the next reconcile. On 304 Not Modified, short-circuit to the existing no-op path without touching the body. When the server sends no validators, behaviour is unchanged.
GitHub Pages, S3/CloudFront, Cloudflare, nginx/static hosting and Harbor all emit validators for static index files, so the common cases are covered.
This mirrors what the controller already does for the other source kinds — GitRepository checks advertised refs before cloning, OCIRepository resolves the manifest digest via HEAD before pulling — HelmRepository/HTTP is the only kind that must download the full payload to discover nothing changed.
Happy to send the PR.
Problem
For HTTP(S)
HelmRepositoryobjects, source-controller downloads the fullindex.yamlon every reconcile interval, unconditionally:CacheIndex→DownloadIndexissues a plain GET, and only after the download completes is the digest compared against the current artifact to decide the reconcile is a no-op (helmrepository_controller.go#L464-L468). There is noIf-None-Match/If-Modified-Sinceanywhere in the fetch path (the only ETag references in the codebase are the S3/GCS test mocks).Real-world indexes are large (grafana ~2 MB, prometheus-community ~4 MB, bitnami historically >10 MB), and the cost is linear in the number of objects: N HelmRepository objects pointing at the same or different repos re-download N full indexes per interval, forever, even when nothing changed. In multi-tenant setups where each tenant namespace declares its own HelmRepository (cross-namespace refs disabled), N grows with the tenant count.
Measurements
Bench: k3s + source-controller v1.9.3, defaults; one HTTP helm repo with a 6.0 MB index; N HelmRepository objects with
interval: 1m; server-side byte/request counters; idle steady state (index unchanged), 5-minute windows.Every one of those bytes is thrown away: the digest comparison concludes "unchanged" after each download.
Proposal
Remember the
ETag/Last-Modifiedresponse headers per object (an in-memory map keyed by object UID is sufficient — a controller restart just pays one full fetch), and sendIf-None-Match/If-Modified-Sinceon the next reconcile. On304 Not Modified, short-circuit to the existing no-op path without touching the body. When the server sends no validators, behaviour is unchanged.GitHub Pages, S3/CloudFront, Cloudflare, nginx/static hosting and Harbor all emit validators for static index files, so the common cases are covered.
This mirrors what the controller already does for the other source kinds — GitRepository checks advertised refs before cloning, OCIRepository resolves the manifest digest via HEAD before pulling — HelmRepository/HTTP is the only kind that must download the full payload to discover nothing changed.
Happy to send the PR.