You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
In a tenant self-service multi-tenancy model each tenant namespace declares its own source objects pointing at the same platform repositories — cross-namespace refs are disabled by the lockdown, so object count grows with tenant count while the set of distinct sources stays small. We benchmarked what that costs source-controller today (numbers inside) and wrote up a proposal for deduplicating probes, fetches and stored artifacts across objects whose content-determining inputs are identical.
The proposal is below in RFC format — happy to resubmit it as a PR to fluxcd/flux2rfcs/ if that is the preferred route, and to work on the implementation. Related: #2113 (conditional requests for HelmRepository, per-object, composes with this).
RFC-0000 Source Fetch and Probe Deduplication
Status: provisional
Creation date: 2026-07-15
Last update: 2026-07-15
Summary
Introduce opt-in deduplication of upstream traffic and stored artifacts in
source-controller for source objects whose content-determining inputs are
identical: same URL, same ref, same resolved credentials, same
artifact-shaping spec fields. Such objects form a dedup group served by
one shared prober running at the group's effective interval (the minimum
of the members' intervals), one shared fetch per new revision, and one
stored copy per distinct artifact. Objects remain fully independent at the
API level — per-object status, conditions, events and artifact URLs are
unchanged — while upstream request volume, download bytes, storage and CPU
become proportional to the number of distinct sources instead of the
number of objects.
Motivation
In a tenant self-service model, each tenant namespace declares its own GitRepository / OCIRepository / HelmRepository. Cross-namespace sourceRefs are disabled (--no-cross-namespace-refs, per the multi-tenancy
lockdown guidance), so the recommended "one shared source in flux-system"
pattern is unavailable by design: tenants own their namespaces, not flux-system. Tenants typically point at the same platform repositories, so
the number of identical source objects grows linearly with the tenant count.
source-controller treats every object as fully independent. The existing
optimizations are strictly per-object: GitRepository checks advertised refs
before cloning (LastObservedCommit), OCIRepository resolves the manifest
digest via HEAD before pulling. Nothing is shared across objects, so the
total cost against the same upstream is linear in N — and that is exactly
what git-provider and registry rate limits meter.
Idle probes: request volume is linear in N; for HelmRepository/HTTP
every "probe" is a full index download (~444 GB/day projected at N=50,
all discarded after a digest comparison).
Change amplification: one commit pushed at N=50 produced 50 downloads
of the same ~1.05 MB pack (52.5 MB transferred) followed by 50 identical
checkout/ignore/tar/hash cycles.
Storage: artifact storage holds 50 copies of an identical 9.26 MB
artifact set (462.9 MB).
CPU: controller CPU at idle grows linearly (5m / 28m / 69m for
N=1/10/50).
Queue latency: the N re-fetches serialize behind --concurrent
(default 2) workers, so the gap between the first and the last tenant
observing a new revision also grows with N.
Rate limits: N × (probes + fetches) count against the same upstream
quota regardless of sharding or controller resources.
With dedup groups, the N=50 row collapses to the N=1 row for upstream
traffic, downloads, storage and fetch CPU.
Goals
Probe each distinct source once per effective interval, fetch each new
revision once, store each distinct artifact once — regardless of how many
objects reference it.
Objects whose dedup key (below) is identical form a group. Three
cooperating mechanisms operate on groups:
Shared prober. One prober per group polls the upstream (advertised
refs for git, manifest HEAD for OCI, index download for helm) at the
group's effective interval = min(spec.interval of live members), and
publishes the result (resolved revision or error) as the group state.
The minimum — not the GCD — is deliberate: an object's contract is
"observed data no older than my interval". Probing at min(intervals)
means every member consumes data at most min ≤ its own interval old, so
every member's freshness contract holds. A GCD schedule fires more
often than min (gcd ≤ min, e.g. intervals 2m/3m → gcd 1m) and buys
nothing: aligning probes with each member's tick marks is unnecessary
when staleness is already bounded by min.
Per-object reconciles keep running at their own cadence but become local: read the object's own secretRef (a local API read, no
upstream traffic), recompute the dedup key, consume the group state,
update own status/conditions/events. A changed secret or spec
immediately changes the key, moving the object to another group (or its
own) from the next reconcile. Interval changes re-derive the group's
effective interval.
Shared fetch. When the group prober resolves a revision R that is not
yet in the artifact store for (key, R), it performs one fetch and
packages one artifact. Concurrent demands for the same (key, R) collapse
(singleflight). A change followed by N member reconciles produces one
download, not N.
Content-addressed artifact storage. Artifact tarballs are stored by
digest and reference-counted; per-object artifact paths (the existing /<kind>/<namespace>/<name>/<file> URL contract) become links into the
store. Existing per-object retention/GC decrements references; a blob is
deleted when the last reference is gone.
Dedup key
The key is the tuple of every input that can influence the probe result or
the produced artifact:
kind + canonicalized URL;
ref specification (branch/tag/semver/commit; layerSelector for OCI);
digest of the resolved authentication material: the credential bytes
after reading secretRef (plus CA bundle and proxy configuration) — not
the Secret name. For provider-based auth (workload identity), the resolved
principal identity is used instead of the ephemeral token, so token
rotation does not fragment the cache;
every artifact-shaping field: ignore, sparseCheckout, recurseSubmodules, include, etc.
Key digests are computed with an HMAC using a per-controller-instance random
salt, so logged/exported key hashes cannot be used for offline credential
guessing.
Properties:
public sources (no credentials) deduplicate out of the box;
tenants holding replicated copies of the same secret deduplicate
automatically;
different credentials → different keys → no sharing: a tenant can never
read content it could not fetch itself. The key is the authorization
context, so there is no confused-deputy path by construction;
within a group the credential material is byte-identical, so the shared
probe exercises exactly the same authorization that each member would
have exercised individually — including failing for all members when the
server revokes it, which is indistinguishable from today's N identical
failures.
Signature/provenance verification (spec.verify) remains per-object: it
runs against the shared content with the object's own keyring/policy, so two
objects sharing a fetch can still differ in verification outcome.
User Stories
Platform with N self-service tenants. Each of 200 tenant namespaces
declares the same platform GitRepository (same URL, same replicated
read-only secret), intervals ranging from 1m to 10m. Today: 200 independent
pollers and, after one merge, 200 clones of the same pack and 200 identical
tarballs. With dedup: one prober at 1m, one clone per commit, one stored
artifact, 200 objects Ready — each with its own status and events.
Public charts/repos. Dozens of namespaces reference https://grafana.github.io/helm-charts or a public git repo without
credentials. All of them share one prober and one fetch per revision with
zero configuration.
Tenant with distinct credentials. One tenant uses its own deploy key for
the same URL. Its key differs; it probes and fetches independently. Nothing
changes for it — including when its credentials are revoked: its own probes
fail even though other tenants' artifacts for the same URL exist on disk.
Alternatives
One shared source + cross-namespace refs — precluded by the tenancy
lockdown this proposal targets.
Sharding (--watch-label-selector) — scales controller capacity, but
the same N probes and fetches still hit the same upstream; rate limits
are per-provider, not per-shard. Dedup and sharding compose (groups are
per-shard).
Longer intervals — trades staleness for load linearly; does not change
the shape of the problem.
External caches (git mirror, pull-through registry proxy) — additional
infrastructure to operate, TLS/auth pass-through complexity, does not
reduce controller CPU or artifact storage, and is invisible to Flux
status/metrics.
Fetch-only dedup with per-object probes (an earlier draft of this
proposal): keeps every object probing upstream itself and shares only the
downloads. Simpler (no group lifecycle), but leaves probe volume — and
for helm, full index downloads — linear in N. Group probing subsumes it;
it remains a reasonable phase-1 implementation milestone.
A GCD-based shared schedule — fires more often than min(intervals)
(gcd ≤ min) without improving any member's observed freshness; rejected,
see Proposal.
Relaxing the key to drop the credential digest when a per-object
probe already proved repo-level read access. Rejected as the default:
repo-level auth is the norm for the git protocol, but the conservative key
is safe against servers with more granular models; the relaxation could be
a later, explicitly documented mode.
Design Details
A group registry lives in source-controller between the reconcilers and fluxcd/pkg/git / OCI clients: key → {members, effective interval,
prober, last probe result, revision-indexed artifact refs}. Membership is
maintained by the per-object reconcilers (join on key computation, leave
on key change or deletion); the registry is rebuilt naturally on
controller restart from the first reconcile of each object.
The prober publishes group state; member objects are requeued on state
transitions (new revision, new error) so their statuses converge promptly
rather than waiting for their next tick. Between transitions, member
reconciles are no-ops against local state.
Probe jitter applies per group instead of per object, which also removes
the thundering-herd effect of many same-interval objects.
CAS layout: /data/blobs/sha256/<digest> plus per-object hardlinks (same
filesystem) preserving today's advertised artifact URLs; Storage already
computes digests, so packaging changes are contained.
Reference counting is derived from object status (the set of artifacts
currently referenced), making the GC crash-safe: an unreferenced blob is
only removed after no object has referenced it for one GC cycle.
Feature gate: SourceFetchDeduplication, default off. Metrics: source_dedup_groups{kind}, source_dedup_group_size histogram, source_fetch_dedup_requests_total{kind,outcome="hit|miss|inflight"},
gauges for CAS blob count/bytes and deduplicated bytes.
Failure isolation: a group probe/fetch error is fanned out to every member
as the object-level condition it would have produced individually —
identical outcome to today's N independent failures with the same
credentials, at 1/N the upstream cost. Verification and packaging errors
remain per-object.
Suggested implementation phases: (1) fetch singleflight + revision-indexed
reuse with per-object probes; (2) CAS with refcounting; (3) group probers
with effective-interval scheduling.
Implementation History
2026-07-15: benchmark demonstrating linear cost in N (numbers above);
proposal drafted.
In a tenant self-service multi-tenancy model each tenant namespace declares its own source objects pointing at the same platform repositories — cross-namespace refs are disabled by the lockdown, so object count grows with tenant count while the set of distinct sources stays small. We benchmarked what that costs source-controller today (numbers inside) and wrote up a proposal for deduplicating probes, fetches and stored artifacts across objects whose content-determining inputs are identical.
The proposal is below in RFC format — happy to resubmit it as a PR to
fluxcd/flux2rfcs/if that is the preferred route, and to work on the implementation. Related: #2113 (conditional requests forHelmRepository, per-object, composes with this).RFC-0000 Source Fetch and Probe Deduplication
Status: provisional
Creation date: 2026-07-15
Last update: 2026-07-15
Summary
Introduce opt-in deduplication of upstream traffic and stored artifacts in
source-controller for source objects whose content-determining inputs are
identical: same URL, same ref, same resolved credentials, same
artifact-shaping spec fields. Such objects form a dedup group served by
one shared prober running at the group's effective interval (the minimum
of the members' intervals), one shared fetch per new revision, and one
stored copy per distinct artifact. Objects remain fully independent at the
API level — per-object status, conditions, events and artifact URLs are
unchanged — while upstream request volume, download bytes, storage and CPU
become proportional to the number of distinct sources instead of the
number of objects.
Motivation
In a tenant self-service model, each tenant namespace declares its own
GitRepository/OCIRepository/HelmRepository. Cross-namespacesourceRefs are disabled (--no-cross-namespace-refs, per the multi-tenancylockdown guidance), so the recommended "one shared source in
flux-system"pattern is unavailable by design: tenants own their namespaces, not
flux-system. Tenants typically point at the same platform repositories, sothe number of identical source objects grows linearly with the tenant count.
source-controller treats every object as fully independent. The existing
optimizations are strictly per-object:
GitRepositorychecks advertised refsbefore cloning (
LastObservedCommit),OCIRepositoryresolves the manifestdigest via HEAD before pulling. Nothing is shared across objects, so the
total cost against the same upstream is linear in N — and that is exactly
what git-provider and registry rate limits meter.
Measured (k3s, source-controller v1.9.3, defaults,
interval: 1m, one gitrepo ~1.3 MB packed, one 6.0 MB helm
index.yaml, server-siderequest/byte counters, 5-minute idle windows):
HelmRepository/HTTPevery "probe" is a full index download (~444 GB/day projected at N=50,
all discarded after a digest comparison).
of the same ~1.05 MB pack (52.5 MB transferred) followed by 50 identical
checkout/ignore/tar/hash cycles.
artifact set (462.9 MB).
N=1/10/50).
--concurrent(default 2) workers, so the gap between the first and the last tenant
observing a new revision also grows with N.
quota regardless of sharding or controller resources.
With dedup groups, the N=50 row collapses to the N=1 row for upstream
traffic, downloads, storage and fetch CPU.
Goals
revision once, store each distinct artifact once — regardless of how many
objects reference it.
artifact URLs, failure reporting.
not fetch with its own object's credentials.
Non-Goals
HelmRepository(HelmRepository: use conditional HTTP requests (ETag / If-Modified-Since) when fetching index.yaml #2113 — separate,per-object; composes with this one — it makes the group's single index
download a 304 when unchanged).
Proposal
Objects whose dedup key (below) is identical form a group. Three
cooperating mechanisms operate on groups:
Shared prober. One prober per group polls the upstream (advertised
refs for git, manifest HEAD for OCI, index download for helm) at the
group's effective interval = min(spec.interval of live members), and
publishes the result (resolved revision or error) as the group state.
The minimum — not the GCD — is deliberate: an object's contract is
"observed data no older than my interval". Probing at min(intervals)
means every member consumes data at most min ≤ its own interval old, so
every member's freshness contract holds. A GCD schedule fires more
often than min (gcd ≤ min, e.g. intervals 2m/3m → gcd 1m) and buys
nothing: aligning probes with each member's tick marks is unnecessary
when staleness is already bounded by min.
Per-object reconciles keep running at their own cadence but become
local: read the object's own
secretRef(a local API read, noupstream traffic), recompute the dedup key, consume the group state,
update own status/conditions/events. A changed secret or spec
immediately changes the key, moving the object to another group (or its
own) from the next reconcile. Interval changes re-derive the group's
effective interval.
Shared fetch. When the group prober resolves a revision R that is not
yet in the artifact store for (key, R), it performs one fetch and
packages one artifact. Concurrent demands for the same (key, R) collapse
(singleflight). A change followed by N member reconciles produces one
download, not N.
Content-addressed artifact storage. Artifact tarballs are stored by
digest and reference-counted; per-object artifact paths (the existing
/<kind>/<namespace>/<name>/<file>URL contract) become links into thestore. Existing per-object retention/GC decrements references; a blob is
deleted when the last reference is gone.
Dedup key
The key is the tuple of every input that can influence the probe result or
the produced artifact:
layerSelectorfor OCI);after reading
secretRef(plus CA bundle and proxy configuration) — notthe Secret name. For provider-based auth (workload identity), the resolved
principal identity is used instead of the ephemeral token, so token
rotation does not fragment the cache;
ignore,sparseCheckout,recurseSubmodules,include, etc.Key digests are computed with an HMAC using a per-controller-instance random
salt, so logged/exported key hashes cannot be used for offline credential
guessing.
Properties:
automatically;
read content it could not fetch itself. The key is the authorization
context, so there is no confused-deputy path by construction;
probe exercises exactly the same authorization that each member would
have exercised individually — including failing for all members when the
server revokes it, which is indistinguishable from today's N identical
failures.
Signature/provenance verification (
spec.verify) remains per-object: itruns against the shared content with the object's own keyring/policy, so two
objects sharing a fetch can still differ in verification outcome.
User Stories
Platform with N self-service tenants. Each of 200 tenant namespaces
declares the same platform
GitRepository(same URL, same replicatedread-only secret), intervals ranging from 1m to 10m. Today: 200 independent
pollers and, after one merge, 200 clones of the same pack and 200 identical
tarballs. With dedup: one prober at 1m, one clone per commit, one stored
artifact, 200 objects Ready — each with its own status and events.
Public charts/repos. Dozens of namespaces reference
https://grafana.github.io/helm-chartsor a public git repo withoutcredentials. All of them share one prober and one fetch per revision with
zero configuration.
Tenant with distinct credentials. One tenant uses its own deploy key for
the same URL. Its key differs; it probes and fetches independently. Nothing
changes for it — including when its credentials are revoked: its own probes
fail even though other tenants' artifacts for the same URL exist on disk.
Alternatives
lockdown this proposal targets.
--watch-label-selector) — scales controller capacity, butthe same N probes and fetches still hit the same upstream; rate limits
are per-provider, not per-shard. Dedup and sharding compose (groups are
per-shard).
the shape of the problem.
infrastructure to operate, TLS/auth pass-through complexity, does not
reduce controller CPU or artifact storage, and is invisible to Flux
status/metrics.
proposal): keeps every object probing upstream itself and shares only the
downloads. Simpler (no group lifecycle), but leaves probe volume — and
for helm, full index downloads — linear in N. Group probing subsumes it;
it remains a reasonable phase-1 implementation milestone.
(gcd ≤ min) without improving any member's observed freshness; rejected,
see Proposal.
probe already proved repo-level read access. Rejected as the default:
repo-level auth is the norm for the git protocol, but the conservative key
is safe against servers with more granular models; the relaxation could be
a later, explicitly documented mode.
Design Details
fluxcd/pkg/git/ OCI clients: key → {members, effective interval,prober, last probe result, revision-indexed artifact refs}. Membership is
maintained by the per-object reconcilers (join on key computation, leave
on key change or deletion); the registry is rebuilt naturally on
controller restart from the first reconcile of each object.
transitions (new revision, new error) so their statuses converge promptly
rather than waiting for their next tick. Between transitions, member
reconciles are no-ops against local state.
the thundering-herd effect of many same-interval objects.
/data/blobs/sha256/<digest>plus per-object hardlinks (samefilesystem) preserving today's advertised artifact URLs;
Storagealreadycomputes digests, so packaging changes are contained.
currently referenced), making the GC crash-safe: an unreferenced blob is
only removed after no object has referenced it for one GC cycle.
SourceFetchDeduplication, default off. Metrics:source_dedup_groups{kind},source_dedup_group_sizehistogram,source_fetch_dedup_requests_total{kind,outcome="hit|miss|inflight"},gauges for CAS blob count/bytes and deduplicated bytes.
as the object-level condition it would have produced individually —
identical outcome to today's N independent failures with the same
credentials, at 1/N the upstream cost. Verification and packaging errors
remain per-object.
GitRepositoryandOCIRepositoryin the first iteration;HelmRepository/HTTP joins with the index download as the probe (furtherreduced by HelmRepository: use conditional HTTP requests (ETag / If-Modified-Since) when fetching index.yaml #2113);
Bucketis a candidatefollow-up.
reuse with per-object probes; (2) CAS with refcounting; (3) group probers
with effective-interval scheduling.
Implementation History
proposal drafted.