feat: introduce kamaji flavor

.gitignore

@@ -13,7 +13,8 @@ Dockerfile.cross
 
 infrastructure-metal-stack
 infrastructure-components.yaml
-.capms-cluster-kubeconfig.yaml
+capms-cluster.kubeconfig
+kamaji-tenant.kubeconfig
 
 # Test binary, built with `go test -c`
 *.test
@@ -33,4 +34,4 @@ go.work
 .vscode
 *.swp
 *.swo
-*~
+*~

DEVELOPMENT.md

@@ -39,9 +39,9 @@ make -C capi-lab mtu-fix
 When the control plane node was provisioned, you can obtain the kubeconfig like:
 
 ```bash
-kubectl get secret metal-test-kubeconfig -o jsonpath='{.data.value}' | base64 -d > capi-lab/.capms-cluster-kubeconfig.yaml
+kubectl get secret metal-test-kubeconfig -o jsonpath='{.data.value}' | base64 -d > capms-cluster.kubeconfig
 # alternatively:
-clusterctl get kubeconfig metal-test > capi-lab/.capms-cluster-kubeconfig.yaml
+clusterctl get kubeconfig metal-test > capms-cluster.kubeconfig
 ```
 
 The node's provider ID is provided by the [metal-ccm](https://github.com/metal-stack/metal-ccm), which needs to be deployed into the cluster:
@@ -50,11 +50,129 @@ The node's provider ID is provided by the [metal-ccm](https://github.com/metal-s
 If you want to provide service's of type load balancer through MetalLB by the metal-ccm, you need to deploy MetalLB:
 
 ```bash
-kubectl --kubeconfig capi-lab/.capms-cluster-kubeconfig.yaml apply --kustomize capi-lab/metallb
+kubectl --kubeconfig capms-cluster.kubeconfig apply --kustomize capi-lab/metallb
 ```
 
 That's it!
 
+## Running the _Kamaji_ flavor
+The _Kamaji_ flavor runs [_Kamaji_](https://github.com/clastix/kamaji) inside `kind` as the management cluster and uses `mini-lab` VMs as tenant cluster worker machines.
+It uses `cluster-api-provider-metal-stack` as the infrastructure provider for the tenant clusters, with the `metal-stack` control plane also running inside the `kind` cluster.
+_Kamaji_ is used as the control plane provider and machines are joined using [`CABPK`](https://cluster-api.sigs.k8s.io/reference/glossary.html#cabpk) and `Ignition`.
+
+![_Kamaji_ Architecture Overview](./capi-lab/mini-lab/docs/overview-kamaji.drawio.svg)
+
+_Kamaji_ is set up based on the [Kamaji on kind](https://kamaji.clastix.io/getting-started/kamaji-kind/) tutorial and the deployed `MetalLB` address pool expects 
+the network `172.18.0.0/16` for `kind`.
+
+To run the _Kamaji_ flavor, set the `MINI_LAB_FLAVOR` environment variable to `kamaji` and then run the `make -C capi-lab` command to start the `mini-lab`.
+
+```bash
+export MINI_LAB_FLAVOR=kamaji
+make -C capi-lab
+```
+
+This sets up the `mini-lab` and deploys _Kamaji_ into the `kind` cluster. 
+The management cluster is initialized with the _Kamaji_ control plane provider, installing all the necessary components for _Kamaji_ to run and manage tenant clusters.
+
+To access the `mini-lab` and run commands like `metalctl` and `kubectl`, you need to set up the environment variables by running the following command:
+```bash
+# allows access using metalctl and kubectl
+eval $(make -C capi-lab --silent dev-env)
+```
+
+Now it's time to deploy the `cluster-api-provider-metal-stack` into the _Kamaji_ management cluster.
+Install the `CAPMS` provider using the locally built image via the following `make` target (useful for development):
+
+```bash
+make push-to-capi-lab
+```
+
+For the `metal-stack` machines to be able to reach the _Kamaji_ tenant API server, a virtual IP needs to be created in the `mini_lab_ext` network (represented in `metal-stack` by the `internet-mini-lab` network).
+It will be assigned to the tenant cluster's control plane (running in the `kind` cluster) by `MetalLB`.
+This IP will be used as the control plane endpoint in the cluster configuration.
+
+```bash
+export CLUSTER_NAME=kamaji-tenant-test
+make -C capi-lab control-plane-ip
+```
+
+Now we can create a _Kamaji_ tenant cluster.
+This registers the just created IP in `MetalLB`, then applies the cluster template via `clusterctl`.
+A control plane for the tenant will be created within the `kind` cluster and made available via the VIP.
+_Kamaji_ will then use the `CAPMS` provider to provision the firewall and worker machines in the `mini-lab` 
+and join them to the tenant cluster's control plane via [`CABPK`](https://cluster-api.sigs.k8s.io/reference/glossary.html#cabpk) and `kubeadm`.
+
+```bash
+make -C capi-lab create-kamaji-tenant
+```
+
+You should now see `metal-stack` machines being provisioned. 
+First the firewall machine, then the worker machine. 
+
+After the firewall and worker machines have phoned home, the MTU needs to be fixed to ensure the workers' connectivity to the VIP.
+This is again only necessary because of the virtual network setup of the `mini-lab` and can be skipped when running on real hardware.
+Only then will `kubeadm` and the `kubelet` be able to reach the API server on the VIP, and the cluster will become healthy as soon as the node has joined.
+
+```bash
+make -C capi-lab mtu-fix
+```
+
+For the fixes to take effect, `FRR` needs to be restarted on the worker and firewall machines. 
+You can use the `console-machine` `make` target to access the machines' consoles and restart `FRR` there.
+Use `metalctl machine list` to find out the machine IDs if you are unsure which one is the firewall and which one is the worker.
+
+```bash
+# on the firewall
+make -C capi-lab/mini-lab password-machine01
+make -C capi-lab/mini-lab console-machine01
+# login using the metal user and password provided by the password-machine01 make target, then run:
+sudo systemctl restart frr
+
+# on the worker
+make -C capi-lab/mini-lab console-machine02
+sudo systemctl restart frr
+sudo systemctl restart kubeadm
+```
+
+Wait until the worker's `kubeadm` and `kubelet` services have reached the API server and the node has joined the cluster.
+
+It is already possible to retrieve the tenant cluster kubeconfig and use it to access the tenant cluster. 
+The kubeconfig is stored as a secret in the management cluster, which we can retrieve and decode.
+The following `make` target does exactly that and stores the kubeconfig in the `capi-lab` directory:
+
+```bash
+make -C capi-lab kamaji-tenant-kubeconfig
+```
+
+The API server in the kubeconfig points to the tenant cluster VIP (`203.0.113.x`). 
+We can now use the tenant kubeconfig to access the tenant cluster, e.g. to see the nodes that have joined:
+
+```bash
+kubectl --kubeconfig kamaji-tenant.kubeconfig get nodes
+```
+
+When the nodes are ready, a CNI and the [`metal-ccm`](https://github.com/metal-stack/metal-ccm) need to be deployed to the tenant cluster for it to be fully functional and allow scheduling workloads.
+
+```bash
+# deploy calico as the CNI to the tenant cluster.
+make -C capi-lab kamaji-tenant-deploy-calico
+```
+
+```bash
+# deploy the metal-ccm to the tenant cluster.
+make -C capi-lab kamaji-tenant-deploy-metal-ccm
+```
+
+All pods in the tenant cluster should now be running and the node should be ready.
+We could now deploy workloads to the tenant cluster and they would be scheduled on the worker machine and have network connectivity.
+
+Use `cleanup` to tear down the _Kamaji_ lab.
+
+```bash
+make -C capi-lab cleanup
+```
+
 ## Running E2E Tests
 
 Before being able to run the E2E or integration tests, make sure to set the following variables to the correct values:
@@ -251,7 +369,7 @@ metalctl machine console --ipmi $control_plane_machine_id
 clusterctl get kubeconfig > capms-cluster.kubeconfig
 
 # metal-ccm
-cat $repo_path/config/target-cluster/metal-ccm.yaml | envsubst | kubectl --kubeconfig capms-cluster.kubeconfig apply -f -
+kustomize build $repo_path/config/target-cluster/overlays/kubeadm | envsubst | kubectl --kubeconfig capms-cluster.kubeconfig apply -f -
 
 # cni
 kubectl --kubeconfig=capms-cluster.kubeconfig create -f https://raw.githubusercontent.com/projectcalico/calico/v3.28.2/manifests/tigera-operator.yaml

capi-lab/Makefile

@@ -4,7 +4,8 @@
 ANSIBLE_EXTRA_VARS_FILE=$(shell pwd)/mini-lab-overrides/extra-vars.yaml
 KIND_EXPERIMENTAL_DOCKER_NETWORK=mini_lab_ext
 KUBECONFIG := $(shell pwd)/mini-lab/.kubeconfig
-MINI_LAB_FLAVOR=capms
+
+MINI_LAB_FLAVOR := $(or $(MINI_LAB_FLAVOR),capms)
 
 CLUSTER_NAME ?= metal-test
 KUBERNETES_VERSION ?= 1.33.5
@@ -29,17 +30,45 @@ FIREWALL_EXTERNAL_NETWORKS ?= ["internet-mini-lab"]
 
 IMG ?= ghcr.io/metal-stack/cluster-api-metal-stack-controller:latest
 
+ifeq ($(MINI_LAB_FLAVOR),capms)
+DEPLOY_TARGET=deploy-kubeadm
+else ifeq ($(MINI_LAB_FLAVOR),kamaji)
+DEPLOY_TARGET=deploy-kamaji
+KUBERNETES_VERSION = 1.34.1
+WORKER_MACHINE_IMAGE ?= capms-ubuntu-$(KUBERNETES_VERSION)
+TENANT_NAMESPACE = kamaji-tenants
+CONTROL_PLANE_IP ?=
+PODS_CIDR = 192.168.0.0/16
+SERVICES_CIDR = 10.96.0.0/12
+else
+$(error Unknown flavor $(MINI_LAB_FLAVOR))
+endif
+
 .PHONY: up
-up: bake deploy-capi
+up: bake $(DEPLOY_TARGET)
 
 .PHONY: bake
 bake:
 	cd mini-lab && make external_network up # change directory required for gen-certs
 
-.PHONY: deploy-capi
-deploy-capi:
+.PHONY: deploy-kubeadm
+deploy-kubeadm:
 	docker compose run --rm clusterctl init --addon helm
 
+.PHONY: deploy-kamaji
+deploy-kamaji:
+	docker compose -f compose.yaml -f compose.kamaji.yaml run --rm clusterctl init --addon helm --control-plane kamaji
+	# TODO: remove once merged: https://github.com/clastix/cluster-api-control-plane-provider-kamaji/pull/323
+	# - Grant the Kamaji controller manager access to MetalStackCluster resources.
+	kubectl --kubeconfig=$(KUBECONFIG) apply -f kamaji/kamaji-rbac.yaml
+	# - Register MetalStackCluster as a supported dynamic infrastructure cluster in the Kamaji controller manager.
+	# - The following patch is quite dirty as we double the args, but works as the last value in the args list wins for duplicate keys.
+	kubectl --kubeconfig=$(KUBECONFIG) patch deployments.apps -n kamaji-system capi-kamaji-controller-manager --type=json -p='[{"op":"add","path":"/spec/template/spec/containers/0/args/-","value":"--feature-gates=DynamicInfrastructureClusterPatch=true,ExternalClusterReference=false,ExternalClusterReferenceCrossNamespace=false,SkipInfraClusterPatch=false"},{"op":"add","path":"/spec/template/spec/containers/0/args/-","value":"--dynamic-infrastructure-clusters=MetalStackCluster"}]'
+	kubectl --kubeconfig=$(KUBECONFIG) rollout status deployment -n kamaji-system capi-kamaji-controller-manager
+
+	# workaround for a route not being applied successfully in the exit container.
+	docker exec -it exit ip route add 203.0.113.128/25 via 203.0.113.128 dev mini_lab_ext
+
 .PHONY: cleanup
 cleanup:
 	make -C mini-lab cleanup
@@ -90,3 +119,41 @@ delete-sample-cluster:
 mtu-fix:
 	cd mini-lab && ssh -F files/ssh/config leaf01 'ip link set dev vtep-1001 mtu 9100 && echo done'
 	cd mini-lab && ssh -F files/ssh/config leaf02 'ip link set dev vtep-1001 mtu 9100 && echo done'
+
+.PHONY: create-kamaji-tenant
+create-kamaji-tenant:
+	$(eval CONTROL_PLANE_IP = $(shell metalctl network ip list --name "$(CLUSTER_NAME)-vip" -o template --template '{{ .ipaddress }}'))
+	$(eval METAL_NODE_NETWORK_ID = $(shell metalctl network list --project $(METAL_PROJECT_ID) -o template --template '{{ (index . 0).id }}'))
+	# let MetalLB assign the IP to the tenant cluster control plane service
+	envsubst < kamaji/metallb-tenant-pool.yaml | kubectl --kubeconfig=$(KUBECONFIG) apply -f -
+	docker compose -f compose.yaml -f compose.kamaji.yaml run --rm \
+		clusterctl generate cluster $(CLUSTER_NAME) \
+		--target-namespace $(TENANT_NAMESPACE) \
+		--worker-machine-count 1 \
+		--kubernetes-version $(KUBERNETES_VERSION) \
+		--from /templates/cluster-template-kamaji-tenant.yaml \
+		| kubectl --kubeconfig=$(KUBECONFIG) apply -f -
+
+.PHONY: delete-kamaji-tenant
+delete-kamaji-tenant:
+	docker compose -f compose.yaml -f compose.kamaji.yaml run --rm \
+		clusterctl generate cluster $(CLUSTER_NAME) \
+		--target-namespace $(TENANT_NAMESPACE) \
+		--worker-machine-count 1 \
+		--kubernetes-version $(KUBERNETES_VERSION) \
+		--from /templates/cluster-template-kamaji-tenant.yaml \
+		| kubectl --kubeconfig=$(KUBECONFIG) delete -f -
+
+.PHONY: kamaji-tenant-kubeconfig
+kamaji-tenant-kubeconfig:
+	kubectl --kubeconfig=$(KUBECONFIG) -n $(TENANT_NAMESPACE) get secret $(CLUSTER_NAME)-kubeconfig -o jsonpath='{.data.value}' | base64 -d > ../kamaji-tenant.kubeconfig
+	@echo "Tenant kubeconfig written to kamaji-tenant.kubeconfig"
+
+.PHONY: kamaji-tenant-deploy-calico
+kamaji-tenant-deploy-calico:
+	kubectl --kubeconfig=../kamaji-tenant.kubeconfig create -f https://raw.githubusercontent.com/projectcalico/calico/v3.28.2/manifests/tigera-operator.yaml || true
+	kubectl --kubeconfig=../kamaji-tenant.kubeconfig apply -f kamaji/calico.yaml
+
+.PHONY: kamaji-tenant-deploy-metal-ccm
+kamaji-tenant-deploy-metal-ccm:
+	kustomize build ../config/target-cluster/overlays/kamaji | envsubst | kubectl --kubeconfig=../kamaji-tenant.kubeconfig apply -f -

capi-lab/compose.kamaji.yaml

@@ -0,0 +1,27 @@
+---
+services:
+  clusterctl:
+    image: registry.k8s.io/cluster-api/clusterctl:v1.12.3
+    environment:
+      - EXP_KUBEADM_BOOTSTRAP_FORMAT_IGNITION=true
+      - METAL_API_HMAC=${METAL_API_HMAC}
+      - METAL_API_URL=${METAL_API_URL}
+      - METAL_API_HMAC_AUTH_TYPE=${METAL_API_HMAC_AUTH_TYPE}
+      - CLUSTER_NAME=${CLUSTER_NAME}
+      - TENANT_NAMESPACE=${TENANT_NAMESPACE}
+      - KUBERNETES_VERSION=${KUBERNETES_VERSION}
+      - CONTROL_PLANE_IP=${CONTROL_PLANE_IP}
+      - PODS_CIDR=${PODS_CIDR}
+      - SERVICES_CIDR=${SERVICES_CIDR}
+      - METAL_PARTITION=${METAL_PARTITION}
+      - METAL_PROJECT_ID=${METAL_PROJECT_ID}
+      - CONTROL_PLANE_MACHINE_IMAGE=${CONTROL_PLANE_MACHINE_IMAGE}
+      - CONTROL_PLANE_MACHINE_SIZE=${CONTROL_PLANE_MACHINE_SIZE}
+      - WORKER_MACHINE_IMAGE=${WORKER_MACHINE_IMAGE}
+      - WORKER_MACHINE_SIZE=${WORKER_MACHINE_SIZE}
+      - FIREWALL_MACHINE_IMAGE=${FIREWALL_MACHINE_IMAGE}
+      - FIREWALL_MACHINE_SIZE=${FIREWALL_MACHINE_SIZE}
+      - METAL_NODE_NETWORK_ID=${METAL_NODE_NETWORK_ID}
+      - FIREWALL_EXTERNAL_NETWORKS=${FIREWALL_EXTERNAL_NETWORKS}
+    volumes:
+      - ../config/clusterctl-templates:/templates:ro

capi-lab/kamaji/calico.yaml

@@ -0,0 +1,18 @@
+apiVersion: operator.tigera.io/v1
+kind: Installation
+metadata:
+  name: default
+spec:
+  # Configures Calico networking.
+  calicoNetwork:
+    bgp: Disabled
+    ipPools:
+    - name: default-ipv4-ippool
+      blockSize: 26
+      cidr: 192.168.0.0/16
+      encapsulation: None
+    mtu: 1440
+  cni:
+    ipam:
+      type: HostLocal
+    type: Calico

capi-lab/kamaji/kamaji-rbac.yaml

@@ -0,0 +1,21 @@
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRole
+metadata:
+  name: capi-kamaji-metalstackcluster-access
+rules:
+- apiGroups: ["infrastructure.cluster.x-k8s.io"]
+  resources: ["metalstackclusters"]
+  verbs: ["get", "list", "watch"]
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRoleBinding
+metadata:
+  name: capi-kamaji-metalstackcluster-access
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: ClusterRole
+  name: capi-kamaji-metalstackcluster-access
+subjects:
+- kind: ServiceAccount
+  name: capi-kamaji-controller-manager
+  namespace: kamaji-system

capi-lab/kamaji/metallb-tenant-pool.yaml

@@ -0,0 +1,19 @@
+---
+apiVersion: metallb.io/v1beta1
+kind: IPAddressPool
+metadata:
+  name: $CLUSTER_NAME-vip
+  namespace: metallb-system
+spec:
+  addresses:
+    - $CONTROL_PLANE_IP/32
+  autoAssign: false
+---
+apiVersion: metallb.io/v1beta1
+kind: L2Advertisement
+metadata:
+  name: $CLUSTER_NAME-vip
+  namespace: metallb-system
+spec:
+  ipAddressPools:
+    - $CLUSTER_NAME-vip