GEP-0043: Spegel Registry Support in the registry-cache extension by dimitar-kostadinov · Pull Request #44 · gardener/enhancements

dimitar-kostadinov · 2026-02-13T16:35:12Z

One-line PR description: Adds a technical steering proposal for Spegel Registry Support in the registry-cache extension (moved from Add technical steering proposal for Spegel registry extension documentation#814 because the process has changed)

Issue link: Spegel Registry Support in the registry-cache extension #43

Other comments:

gardener-prow · 2026-02-13T16:35:18Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign scheererj for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

rfranzke

Apart from the comments inline, can you explain the purpose of the registry-cache and registry-mirror extension when we have registry-spegel? Are they still of any use, and if so, when and how?

rfranzke · 2026-02-14T12:36:02Z

+...
+```
+Spegel `bootstrapper` is deployed into the Shoot namespace in the Seed cluster. In the Shoot cluster nodes the `containerd`, `spegel`, and `kubelet` systemd units are started.
+When `spegel` is starting, it gets the bootstrap peers from the `bootstrapper` and registers for image related events in `containerd`. It also lists existing content in the image store and advertises it to the DHT.


What happens when kubelet already starts to pull an image while Spegel has not yet contacted (or is still in the process of contacting) the bootstrapper?

In this case Spegel returns 404 and containerd fall back to upstream registry:

Feb 23 13:25:24 spegel[768]: {"time":"2026-02-23T13:25:24.027277587Z","level":"ERROR","source":{"function":"github.com/spegel-org/spegel/pkg/httpx.(*ServeMux).Handle.func1.1","file":"/build/pkg/httpx/mux.go","line":80},"msg":"","err":"MANIFEST_UNKNOWN could not find peer for registry.k8s.io/pause:3.10","path":"/v2/pause/manifests/3.10","status":404,"method":"HEAD","latency":"1.100958ms","ip":"::1","handler":"mirror","registry":"registry.k8s.io"} Feb 23 13:25:24 spegel[768]: {"time":"2026-02-23T13:25:24.074244712Z","level":"ERROR","source":{"function":"github.com/spegel-org/spegel/pkg/httpx.(*ServeMux).Handle.func1.1","file":"/build/pkg/httpx/mux.go","line":80},"msg":"","err":"MANIFEST_UNKNOWN could not find peer for registry.k8s.io/pause:3.10","path":"/v2/pause/manifests/3.10","status":404,"method":"HEAD","latency":"706.125µs","ip":"::1","handler":"mirror","registry":"registry.k8s.io"} Feb 23 13:25:25 spegel[768]: {"time":"2026-02-23T13:25:25.441465754Z","level":"ERROR","source":{"function":"github.com/spegel-org/spegel/pkg/routing.ensureOnline.func2","file":"/build/pkg/routing/p2p.go","line":460},"msg":"failed to run bootstrap","err":"Get \"https://sp-local--local.ingress.local.seed.local.gardener.cloud/bootstrap-nodes\": tls: failed to verify certificate: x509: certificate is valid for ingress.local, not sp-local--local.ingress.local.seed.local.gardener.cloud","attempts":6} Feb 23 13:25:24 containerd[623]: time="2026-02-23T13:25:24.035447878Z" level=debug msg=resolving host="localhost:5500" Feb 23 13:25:24 containerd[623]: time="2026-02-23T13:25:24.035467170Z" level=debug msg="do request" host="localhost:5500" request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*" request.header.user-agent=containerd/v2.1.3 request.method=HEAD url="http://localhost:5500/v2/pause/manifests/3.10?ns=registry.k8s.io" Feb 23 13:25:24 containerd[623]: time="2026-02-23T13:25:24.036964795Z" level=debug msg="fetch response received" host="localhost:5500" response.header.content-length=127 response.header.content-type=application/json response.header.date="Mon, 23 Feb 2026 13:25:24 GMT" response.status="404 Not Found" url="http://localhost:5500/v2/pause/manifests/3.10?ns=registry.k8s.io" Feb 23 13:25:24 containerd[623]: time="2026-02-23T13:25:24.036991462Z" level=info msg="trying next host after status: 404 Not Found" host="localhost:5500"

rfranzke · 2026-02-14T12:39:02Z

+- service
+- ingress
+
+The `spegel` registries send a GET request to `https://<ingress-host>/bootstrap-nodes` to get the bootstrap peers. Traffic is encrypted using mTLS.


Did you think about certificate rotation? How would this work?

The secrets manager is used for certificate creation & rotation: https://github.com/dimitar-kostadinov/gardener-extension-registry-cache/blob/6f243bbe7f4f3667b3fff7806343c60f3079f408/pkg/secrets/config.go#L84
E2E test for rotation will be implemented.

Can you elaborate? When will the certificate be replaced? Automatically? Manually on user demand? What if there is an issue with rolling out the new certificate to the worker nodes?

The certificates will be rotated when Certificate Authorities rotation is triggered (suppose on user demand).
I tested following scenario in local setup:

annotate Shoot with gardener.cloud/operation=rotate-ca-start and ensure the Spegel bootstrap works

annotate Shoot with gardener.cloud/operation=rotate-ca-complete and ensure the Spegel bootstrap works

There was an issue using the Shoot CA, but I switched to caBundle and now it works without problems.

If for some reason the certificates are not propagated to the nodes, Spegel will not be able to join the p2p cluster and the images will be pulled from the upstream registries.

Fair enough. I guess it would makes sense to use a dedicated CA for Spegel, but that's a detail.

Fair enough. I guess it would makes sense to use a dedicated CA for Spegel, but that's a detail.

Yes, currently common ca-extension-registry-cache CA is used for both registry-cache & registry-spegel. I'll set dedicated CA for Spegel.

ScheererJ

Thanks for proposing a solution to the limitations of the current registry cache extension.

ScheererJ · 2026-02-16T12:24:43Z

+
+### Risks, Downsides and Trade-offs
+
+The external `Bootstrapper` should be contributed to Spegel.


What are the alternatives, i.e. if the upstream does not want the additional bootstrapper do we need to fork spegel?

dimitar-kostadinov · 2026-02-16T13:21:49Z

Apart from the comments inline, can you explain the purpose of the registry-cache and registry-mirror extension when we have registry-spegel? Are they still of any use, and if so, when and how?

Yes, the extensions types are still valid. They can work together with Spegel p2p image cache.

registry-cache - is based on distribution/distribution project and represent a pull-through cache.
If a user wants a dedicated image cache that stores images in its own storage, independently from nodes' image cache, he can use registry-cache.
registry-cache and registry-spegel can be combined in the containerd hosts.toml config like this:

server = "https://registry-1.docker.io"

[host."http://localhost:50000"]
  capabilities = ["pull", "resolve"]

[host."https://<docker-registry-service-ip>:5000"]
  capabilities = ["pull","resolve"]
  ca = ["/etc/containerd/certs.d/ca-bundle.pem"]

registry-mirror - introduced with this PR after a user request for such feature.
It simply provides containerd configuration and can be combined with registry-spegel in the same way as registry-cache.

ialidzhikov · 2026-02-16T14:12:21Z

/kind enhancement

timebertt

Thanks for the proposal. Using Spegel via an extension that integrates well with the shoot's/nodes' lifecycle seems like a great addition to the Gardener project.
Looking forward to this contribution!

timebertt · 2026-02-16T16:04:11Z

+The content discovery in a Kubernetes cluster is based on [Kademlia DHT][content-provider-routing]. When an image content (blob, manifest or index) is available in the containerd image store, Spegel adds its `digest` to the DHT provider store, announcing that the node provides the content corresponding to the `digest`. Then, when the same `digest` is needed by another node, Spegel searches the DHT for peers that provide the content and pulls it from them.
+
+The straightforward way to deploy Spegel to a Kubernetes cluster is by using the provided helm chart. However, this has some [drawbacks](https://spegel.dev/docs/faq/#what-should-i-do-if-other-pods-are-scheduled-on-new-nodes-before-spegel) when a new node joins the cluster.
+Our goal is to be able to use Spegel for all images pulled from the kubelet, including the `registry.k8s.io/pause` image. Therefore it was decided to [run][run-spegel-on-host] Spegel registry as a systemd unit service on the node. This requires contributing a new Spegel `bootstrapper` or extending the existing [HTTP bootstrapper](https://github.com/spegel-org/spegel/blob/6f02215fa3fc1d3bbdb11fa62dfa7c07dbe3b7c2/pkg/routing/bootstrap.go#L131-L135).


Our goal is to be able to use Spegel for all images pulled from the kubelet, including the registry.k8s.io/pause image.

That's a pretty high bar, which comes at a price.
We could address many of the mentioned drawbacks by running the spegel DaemonSet in the host network, tolerating the node.gardener.cloud/critical-components-not-ready, and labelling the spegel DaemonSet itself with node.gardener.cloud/critical-component=true, correct?
With this, we wouldn't ensure all images (especially other node-critical components) are pulled through spegel. However, we would have better visibility in Kubernetes itself if not running as a systemd unit.

Our goal is to be able to cache as many images as we can.
Since Spegel is tightly coupled to containerd we can further decide to include it into machine images and to be able to cache the images pulled by the gardener-node-agent as well - see this comment for details.

Additionally, Spegel recommends using nidhogg to taint nodes. This way image pulls go through Spegel (except for spegel and pause images).

Additionally, Spegel recommends using nidhogg to taint nodes. This way image pulls go through Spegel (except for spegel and pause images).

gardener-resource-manager's node critical components controller seems to provide similar benefits.
Can you outline the differences to nidhogg?
If that's the recommended way of running Spegel, can we do it the same way or adapt our components according to your goal?

timebertt · 2026-02-16T16:06:27Z

+The `spegel` binary will be provided to the node in the `/opt/bin/` folder via the `OperatingSystemConfig` mutation. The same goes for the `spegel_metrics.sh` script used to write metrics to the node-exporter's textfile collector.
+The same approach is used for systemd units:
+
+- `spegel.service` systemd unit service depends on containerd service and must be started before the kubelet service:


Running spegel as a systemd unit has significant drawbacks: no visibility in Kubernetes (think DaemonSet/pod status, logs, etc.), easier debugging (e.g., port-forward), no monitoring (shoot system components condition), no autoscaling, no consideration of resource requests out of the box, etc.

ialidzhikov · 2026-02-17T13:13:56Z

+reviewers:
+- "@timebertt"
+- "@ScheererJ"
+- "@rfranzke"
+approvers:
+- "@timebertt"
+- "@ScheererJ"
+- "@rfranzke"


Do these fields represent the reviewers/approvers of the proposal or the follow-up implementation?

If it is about the implementation, I guess that would be me as a maintainer of the registry-cache extension/repo.

My understanding was that these lists represents the reviewers/approvers of the proposal, not necessarily the persons implementing it.

ialidzhikov · 2026-02-17T13:23:05Z

Apart from the comments inline, can you explain the purpose of the registry-cache and registry-mirror extension when we have registry-spegel? Are they still of any use, and if so, when and how?

+1 for the question. We could add a new section to the proposal describing:

registry-spegel is not replacement or v2 for the existing registry-cache extension.
they can be used together
the architectural difference between them - pull-through cache vs peer-to-peer cache
in which cases to use them (when to choose one of them over the another one)

dimitar-kostadinov · 2026-02-27T14:32:34Z

I have implemented PoCs based on DaemonSet. The first variant uses the default DNS bootstrapper, while the second variant uses the bootstrapper from this proposal.

Note

In the first variant bootstrap run will be successful once the kube-dns service becomes available (i.e. the coredns pods are running)

Note

In both variants, the ReadinessProbe of the DaemonSet pod is NOT executed because pod becomes ready when there is more than 1 peer in the p2p cluster.

The following test scenario is run for both variants:

In the local setup create a Shoot cluster with 1 node and enabled registry-spegel extension.
Add a new node to the cluster.
Collect the containerd logs from the new node and extract the OCI artifacts fetched from upstream and local Spegel registry.

Here are the test results:

variant1: 239MiB pulled from upstream and 159MiB pulled from spegel registry.
variant2: 226MiB pulled from upstream and 159MiB pulled from spegel registry.

Note

Results may vary depending on when the spegel is ready and the content is advertised.
In variant2 locally build spegel image is used.
In a real-world scenario where a CSI driver exists, the results could be even worse.

rfranzke · 2026-03-02T14:19:30Z

@dimitar-kostadinov What is your conclusion out of this? How do you propose to continue?
cc @timebertt

dimitar-kostadinov · 2026-03-02T14:28:44Z

What is your conclusion out of this? How do you propose to continue?

I need a little more time to prepare another PoC, after which we can arrange a second round to make the final decision.

rfranzke · 2026-03-02T14:43:50Z

Fair enough, then let's mark this PR as
/hold
for now. Please /unhold it once you are ready and want the folks to have another look on the doc.

timebertt · 2026-03-03T09:26:19Z

+- `http_request_duration_seconds` - The latency of the HTTP requests - histogram type.
+- `spegel_mirror_requests_total` - Total number of mirror requests - counter type.
+- `spegel_resolve_duration_seconds` - The duration for router to resolve a peer - histogram type.
+- `spegel_advertised_keys` - Number of keys advertised to be available - gauge type.


Please outline how the cache hit/miss ratio (i.e., cache efficiency) can be measured/observed.

Here is a sample query grouped by registry:

sum by (registry) (spegel_mirror_requests_total{cache="hit"}) / sum by (registry) (spegel_mirror_requests_total{cache="miss"})

I'll try to prepare a monitoring dashboard next week.

dimitar-kostadinov · 2026-03-04T13:03:42Z

For completeness, I have prepared a PoC in which all images, including gardener-node-agent, pass through the Spegel registry.
In this variant Spegel binary is included in provider-local node image and this requires some changes in gardener/gardener.

The same test scenario is used as in this comment.
From the logs is visible that all images on node2 are pulled from Spegel registry on node1.

Note

An additional 5 sec of wait time is added to the gardener-node-init.service unit to allow Spegel to advertise the content.
The size of the init OSC exceeded 16Kb, so it needs further rework to support AWS.

dimitar-kostadinov · 2026-03-06T12:47:57Z

Summary

[option 1] [current proposal] Spegel as systemd unit service with the External Bootstrapper (*)

Pros:

Covers all images pulled by kubelet (including all images from the kube-system namespace).

Cons:

No visibility in Kubernetes.
Doesn't cover the images pulled by the gardener-node-agent.
The External Bootstrapper is not yet contributed in the upstream.

[option 2] Spegel as DaemonSet with the upstream DNS Bootstrapper

Pros:

Improved visibility in Kubernetes (monitoring, debugging, autoscaling and resource reservation).

Cons:

Doesn't cover most of the images in the kube-system namespace. Doesn't cover images with size more than ~200MiB per Node compared to option 1.
Doesn't cover the images pulled by the gardener-node-agent.

[option 3] Spegel as DaemonSet with the External Bootstrapper (*)

Pros:

Improved visibility in Kubernetes (monitoring, debugging, autoscaling and resource reservation).

Cons:

Doesn't cover most of the images in the kube-system namespace. Doesn't cover images with size more than ~200MiB per Node compared to option 1.
Doesn't cover the images pulled by the gardener-node-agent.
The External Bootstrapper is not yet contributed in the upstream.

[option 4] Spegel as systemd unit service with the Spegel binary delivered by the OS with the External Bootstrapper (*)

Pros:

Covers all images pulled by kubelet (including all images from the kube-system namespace) and gardener-node-agent (gardener-node-agent, hyperkube and opentelemetry-collector).

Cons:

Requires the Spegel binary to be present in the OS image. Not able to fullfil this requirement.
No visibility in Kubernetes.
The External Bootstrapper is not yet contributed in the upstream.

(*) The External Bootstrapper is to be contributed.

TL;DR

We prefer option 1 which is described in the current revision of the proposal.

/hold cancel

@ScheererJ, can we schedule TSC round 2 for this topic? Thanks in advance!

rfranzke · 2026-03-06T16:03:53Z

@dimitar-kostadinov Did you get clarity on #44 (comment)? I guess this would be relevant to decide how to proceed. If the Spegel community is not interested in accepting this, we cannot go with your preferred option.

timebertt · 2026-03-09T07:36:02Z

TL;DR

We prefer option 1 which is described in the current revision of the proposal.

I still don't get why you don't want to start simple with the "Spegel as DaemonSet with the upstream DNS Bootstrapper" option.
It "just works", doesn't have dependencies on Spegel development, gain some experience with Spegel in production, and we can measure its efficiency over time, etc.

The listed cons are only relevant for node bootstrapping. However, most nodes live long enough to observe updates to the gardener-node-agent binary, kube-system images, etc. Even with this approach, we will be able to benefit from images cached by Spegel.

can we schedule TSC round 2 for this topic?

To me, it doesn't make sense to continue with the document's current state until you've convinced me why we shouldn't use option 2 as the first iteration of this extension.
I'm not saying we should never use this approach in the future. But I'm very much in favor of starting simple and improving over time.

gardener-ci-robot · 2026-04-08T08:33:38Z

The Gardener project currently lacks enough active contributors to adequately respond to all PRs.
This bot triages PRs according to the following rules:

After 30d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 14d of inactivity since lifecycle/rotten was applied, the PR is closed

You can:

Mark this PR as active with /lifecycle active
Mark this PR as fresh with /remove-lifecycle stale
Mark this PR as rotten with /lifecycle rotten
Close this PR with /close

/lifecycle stale

dimitar-kostadinov · 2026-04-08T08:42:53Z

/remove-lifecycle stale

ialidzhikov · 2026-04-08T11:43:41Z

@dimitar-kostadinov raised a PR to spegel some time ago for adding support for external bootstrapper - spegel-org/spegel#1250.

Let's see if/when this contribution will be accepted in order to unblock option 1. from #44 (comment).

gardener-ci-robot · 2026-05-08T12:20:19Z

The Gardener project currently lacks enough active contributors to adequately respond to all PRs.
This bot triages PRs according to the following rules:

After 30d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 14d of inactivity since lifecycle/rotten was applied, the PR is closed

You can:

Mark this PR as active with /lifecycle active
Mark this PR as fresh with /remove-lifecycle stale
Mark this PR as rotten with /lifecycle rotten
Close this PR with /close

/lifecycle stale

dimitar-kostadinov · 2026-05-08T12:31:53Z

/remove-lifecycle stale

Add technical steering proposal for spegel registry

ffc1fe9

gardener-prow Bot added do-not-merge/needs-kind Indicates a PR lacks a `kind/foo` label and requires one. cla: yes Indicates the PR's author has signed the cla-assistant.io CLA. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Feb 13, 2026

dimitar-kostadinov mentioned this pull request Feb 13, 2026

Spegel Registry Support in the registry-cache extension #43

Open

rfranzke changed the title ~~GEP-0040: Spegel Registry Support in the registry-cache extension~~ GEP-0043: Spegel Registry Support in the registry-cache extension Feb 14, 2026

rfranzke reviewed Feb 14, 2026

View reviewed changes

ScheererJ reviewed Feb 16, 2026

View reviewed changes

Address PR review feedback

7b5d152

gardener-prow Bot added kind/enhancement Enhancement, improvement, extension and removed do-not-merge/needs-kind Indicates a PR lacks a `kind/foo` label and requires one. labels Feb 16, 2026

Address PR review feedback (2)

70ad227

timebertt reviewed Feb 16, 2026

View reviewed changes

ialidzhikov reviewed Feb 17, 2026

View reviewed changes

gardener-prow Bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Mar 2, 2026

timebertt reviewed Mar 3, 2026

View reviewed changes

ialidzhikov assigned dimitar-kostadinov Mar 4, 2026

dimitar-kostadinov added 4 commits March 4, 2026 16:07

Use default ports that are not reserved by IANA

0e6a0aa

Add a list of images not covered by Spegel

5e3eda1

Rework of metrics scraping

775a218

Add clarification about the purpose of the bootstrapper

6052021

dimitar-kostadinov added 7 commits March 5, 2026 15:05

Add summary section

e8c40b3

Add resolveTags option

341a915

Add Goals and Non-Goals sections

29cfc42

Move KadDHT overview into the Appendix section

53d18b7

Add clarification for discard_unpacked_layers setting

c2c75cd

Regenerate TOC

97999dd

Address minor suggestions

85e1b21

gardener-prow Bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Mar 6, 2026

gardener-prow Bot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Apr 8, 2026

gardener-prow Bot removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Apr 8, 2026

gardener-prow Bot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label May 8, 2026

gardener-prow Bot removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label May 8, 2026


		### Risks, Downsides and Trade-offs

		The external `Bootstrapper` should be contributed to Spegel.

Conversation

dimitar-kostadinov commented Feb 13, 2026

Uh oh!

gardener-prow Bot commented Feb 13, 2026

Uh oh!

rfranzke left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rfranzke Feb 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dimitar-kostadinov Feb 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

ScheererJ left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

dimitar-kostadinov commented Feb 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ialidzhikov commented Feb 16, 2026

Uh oh!

timebertt left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

timebertt Feb 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rfranzke Feb 14, 2026 •

edited

Loading

dimitar-kostadinov Feb 16, 2026 •

edited

Loading

dimitar-kostadinov commented Feb 16, 2026 •

edited

Loading

timebertt Feb 16, 2026 •

edited

Loading

dimitar-kostadinov commented Feb 27, 2026 •

edited

Loading

dimitar-kostadinov commented Mar 6, 2026 •

edited

Loading