From ebe7ecc0ca6929afa80af682655572d3b7e5c518 Mon Sep 17 00:00:00 2001 From: Gabriele Quaresima Date: Wed, 5 Aug 2026 11:11:18 +0200 Subject: [PATCH 1/5] docs(samples): add multi-server/multi-cluster OpenTelemetry sample Adds operator/config/samples/opentelemetry-multi, deploying two Klio servers and three CNPG clusters (one in a different namespace, backed cross-namespace by one of the servers) against the same OTel/Prometheus/ Grafana stack the single-server sample uses. The existing sample only ever exercises one server and one cluster, so it can't validate the Grafana dashboard's namespace/server/cluster template variables or its per-tier and per-cluster aggregations. Validated end to end on Kind: all three clusters reconcile, stream WAL and complete backups, including the cross-namespace client whose certificates are bridged via copy-cross-namespace-secrets.sh (cert-manager Issuers are namespace scoped, so that certificate has to be generated in the server's namespace and copied over rather than issued locally). Closes #26 Assisted-by: Claude Signed-off-by: Gabriele Quaresima --- .../samples/opentelemetry-multi/README.md | 118 ++++++++++++++++++ .../opentelemetry-multi/backups-example.yaml | 44 +++++++ .../opentelemetry-multi/cluster_a.yaml | 60 +++++++++ .../cluster_a_klio_client_auth.yaml | 19 +++ .../opentelemetry-multi/cluster_b.yaml | 60 +++++++++ .../cluster_b_klio_client_auth.yaml | 19 +++ .../cluster_c_klio_client_auth.yaml | 23 ++++ .../cluster_klio_otel_config.yaml | 21 ++++ .../copy-cross-namespace-secrets.sh | 45 +++++++ .../samples/opentelemetry-multi/issuer.yaml | 7 ++ .../samples/opentelemetry-multi/jaeger.yaml | 71 +++++++++++ .../opentelemetry-multi/klio_encryption.yaml | 18 +++ .../opentelemetry-multi/klio_server_a.yaml | 93 ++++++++++++++ .../klio_server_a_certificate.yaml | 23 ++++ .../opentelemetry-multi/klio_server_b.yaml | 93 ++++++++++++++ .../klio_server_b_certificate.yaml | 28 +++++ .../opentelemetry-multi/klio_server_ca.yaml | 23 ++++ .../klio_server_ca_issuer.yaml | 7 ++ .../klio_server_otel_client_certificate.yaml | 23 ++++ .../klio_server_otel_config.yaml | 28 +++++ .../opentelemetry-multi/kustomization.yaml | 32 +++++ .../opentelemetry-multi/otel_collector.yaml | 83 ++++++++++++ .../otel_collector_certificate.yaml | 13 ++ .../otel_collector_clusterrole.yaml | 33 +++++ .../otel_collector_clusterrolebinding.yaml | 12 ++ .../otel_collector_sa.yaml | 4 + .../otel_collector_svc_monitor.yaml | 18 +++ .../opentelemetry-multi/team-c/cluster_c.yaml | 68 ++++++++++ .../team-c/cluster_klio_otel_config.yaml | 20 +++ .../team-c/kustomization.yaml | 9 ++ .../opentelemetry-multi/team-c/namespace.yaml | 4 + 31 files changed, 1119 insertions(+) create mode 100644 operator/config/samples/opentelemetry-multi/README.md create mode 100644 operator/config/samples/opentelemetry-multi/backups-example.yaml create mode 100644 operator/config/samples/opentelemetry-multi/cluster_a.yaml create mode 100644 operator/config/samples/opentelemetry-multi/cluster_a_klio_client_auth.yaml create mode 100644 operator/config/samples/opentelemetry-multi/cluster_b.yaml create mode 100644 operator/config/samples/opentelemetry-multi/cluster_b_klio_client_auth.yaml create mode 100644 operator/config/samples/opentelemetry-multi/cluster_c_klio_client_auth.yaml create mode 100644 operator/config/samples/opentelemetry-multi/cluster_klio_otel_config.yaml create mode 100755 operator/config/samples/opentelemetry-multi/copy-cross-namespace-secrets.sh create mode 100644 operator/config/samples/opentelemetry-multi/issuer.yaml create mode 100644 operator/config/samples/opentelemetry-multi/jaeger.yaml create mode 100644 operator/config/samples/opentelemetry-multi/klio_encryption.yaml create mode 100644 operator/config/samples/opentelemetry-multi/klio_server_a.yaml create mode 100644 operator/config/samples/opentelemetry-multi/klio_server_a_certificate.yaml create mode 100644 operator/config/samples/opentelemetry-multi/klio_server_b.yaml create mode 100644 operator/config/samples/opentelemetry-multi/klio_server_b_certificate.yaml create mode 100644 operator/config/samples/opentelemetry-multi/klio_server_ca.yaml create mode 100644 operator/config/samples/opentelemetry-multi/klio_server_ca_issuer.yaml create mode 100644 operator/config/samples/opentelemetry-multi/klio_server_otel_client_certificate.yaml create mode 100644 operator/config/samples/opentelemetry-multi/klio_server_otel_config.yaml create mode 100644 operator/config/samples/opentelemetry-multi/kustomization.yaml create mode 100644 operator/config/samples/opentelemetry-multi/otel_collector.yaml create mode 100644 operator/config/samples/opentelemetry-multi/otel_collector_certificate.yaml create mode 100644 operator/config/samples/opentelemetry-multi/otel_collector_clusterrole.yaml create mode 100644 operator/config/samples/opentelemetry-multi/otel_collector_clusterrolebinding.yaml create mode 100644 operator/config/samples/opentelemetry-multi/otel_collector_sa.yaml create mode 100644 operator/config/samples/opentelemetry-multi/otel_collector_svc_monitor.yaml create mode 100644 operator/config/samples/opentelemetry-multi/team-c/cluster_c.yaml create mode 100644 operator/config/samples/opentelemetry-multi/team-c/cluster_klio_otel_config.yaml create mode 100644 operator/config/samples/opentelemetry-multi/team-c/kustomization.yaml create mode 100644 operator/config/samples/opentelemetry-multi/team-c/namespace.yaml diff --git a/operator/config/samples/opentelemetry-multi/README.md b/operator/config/samples/opentelemetry-multi/README.md new file mode 100644 index 00000000..886da605 --- /dev/null +++ b/operator/config/samples/opentelemetry-multi/README.md @@ -0,0 +1,118 @@ +# README + +This directory contains a sample environment with **two Klio servers** and +**three CloudNativePG clusters** distributed across them, one of which lives +in a different namespace than the servers. It exercises the same +OpenTelemetry + Prometheus + Grafana stack as +`operator/config/samples/opentelemetry/`, but is meant to validate the +Grafana dashboard's `$namespace`, `$server` and `$cluster` template +variables and its per-tier/per-cluster aggregations against more than one +server or cluster, something the single-server sample can't exercise. + +## Topology + +| Cluster | Namespace | Backed by | +|-------------|-----------|-----------| +| cluster-a | default | klio-a | +| cluster-b | default | klio-b | +| cluster-c | team-c | klio-b | + +`klio-b` intentionally backs clusters in two different namespaces, since a +single shared backup server serving multiple application namespaces is a +realistic multi-tenant deployment and the case most likely to expose +dashboard attribution bugs. + +## Prerequisites + +A running Kubernetes cluster with the following operators installed: + +- CloudNativePG +- Klio +- cert-manager +- OpenTelemetry +- Prometheus + +`jq` must also be available locally (used by `copy-cross-namespace-secrets.sh`). + +## Deploying a Kubernetes cluster with the required operators + +Assuming an environment with CloudNativePG, Klio and cert-manager +created through the CloudNativePG `hack/setup-cluster.sh` script and +the klio task + +```shell +KIND_CLUSTER_NAME=$(kind get clusters | grep pg-operator-e2e) task integration:deploy-to-kind +``` + +you can install the OpenTelemetry operator by running: + +```shell +kubectl apply -f https://github.com/open-telemetry/opentelemetry-operator/releases/latest/download/opentelemetry-operator.yaml +``` + +You can install Prometheus using the Prometheus community Helm chart and +the CloudNativePG example configuration: + +```shell +helm repo add prometheus-community https://prometheus-community.github.io/helm-charts +``` + +```shell +helm upgrade --install \ + -f https://raw.githubusercontent.com/cloudnative-pg/cloudnative-pg/main/docs/src/samples/monitoring/kube-stack-config.yaml \ + prometheus-community prometheus-community/kube-prometheus-stack +``` + +## Deploying the sample configuration + +1. Deploy the two servers, the two same-namespace clusters (cluster-a, + cluster-b) and the shared OTel/Jaeger stack, all in `default`: + + ```shell + kubectl apply -k operator/config/samples/opentelemetry-multi + ``` + + Wait for `klio-a`, `klio-b`, `cluster-a` and `cluster-b` to become ready + before continuing. + +1. cert-manager `Issuer`s are namespace-scoped, so the client certificate + cluster-c needs (to authenticate to klio-b) can only be generated in + `default`, where the `klio-server-ca` Issuer lives. Copy that + certificate, plus the (pinned, not CA-validated — see the script's + comments) server and OTel collector certificates, into the `team-c` + namespace: + + ```shell + ./operator/config/samples/opentelemetry-multi/copy-cross-namespace-secrets.sh + ``` + +1. Deploy cluster-c into `team-c`: + + ```shell + kubectl apply -k operator/config/samples/opentelemetry-multi/team-c + ``` + +## Validating the Grafana dashboard + +Trigger one base backup per cluster (needed before any backup/snapshot panel +has data to show): + +```shell +kubectl apply -f operator/config/samples/opentelemetry-multi/backups-example.yaml +``` + +Once all three backups complete and Prometheus has scraped a +metrics-collection cycle, open the Klio Grafana dashboard and confirm: + +- The `$namespace` variable offers both `default` and `team-c`. +- The `$server` variable offers both `klio-a-klio-0` and `klio-b-klio-0` + (the value is each server's pod hostname). +- The `$cluster` variable offers `cluster-a`, `cluster-b` and `cluster-c`, + and narrows correctly when `$namespace`/`$server` are filtered (e.g. + selecting `$namespace=team-c` should only ever offer `cluster-c`). +- Per-cluster and per-server panels correctly attribute data instead of + aggregating everything together. + +Any panel that fails to distinguish between clusters/servers/namespaces +here is a dashboard bug to file separately; this sample's job is only to +make that determination possible. diff --git a/operator/config/samples/opentelemetry-multi/backups-example.yaml b/operator/config/samples/opentelemetry-multi/backups-example.yaml new file mode 100644 index 00000000..b33de6e6 --- /dev/null +++ b/operator/config/samples/opentelemetry-multi/backups-example.yaml @@ -0,0 +1,44 @@ +# Triggers one base backup on each of the three clusters, so the Grafana +# dashboard has backup/snapshot data to show for all three (namespace, +# server, cluster) combinations. Apply after cluster-a, cluster-b and +# cluster-c are all healthy: +# +# kubectl apply -f operator/config/samples/opentelemetry-multi/backups-example.yaml +apiVersion: postgresql.cnpg.io/v1 +kind: Backup +metadata: + name: cluster-a-backup-1 + namespace: default +spec: + method: plugin + target: primary + cluster: + name: cluster-a + pluginConfiguration: + name: klio.cnpg.io +--- +apiVersion: postgresql.cnpg.io/v1 +kind: Backup +metadata: + name: cluster-b-backup-1 + namespace: default +spec: + method: plugin + target: primary + cluster: + name: cluster-b + pluginConfiguration: + name: klio.cnpg.io +--- +apiVersion: postgresql.cnpg.io/v1 +kind: Backup +metadata: + name: cluster-c-backup-1 + namespace: team-c +spec: + method: plugin + target: primary + cluster: + name: cluster-c + pluginConfiguration: + name: klio.cnpg.io diff --git a/operator/config/samples/opentelemetry-multi/cluster_a.yaml b/operator/config/samples/opentelemetry-multi/cluster_a.yaml new file mode 100644 index 00000000..56738186 --- /dev/null +++ b/operator/config/samples/opentelemetry-multi/cluster_a.yaml @@ -0,0 +1,60 @@ +apiVersion: postgresql.cnpg.io/v1 +kind: Cluster +metadata: + name: cluster-a +spec: + instances: 3 + imagePullPolicy: Always + + projectedVolumeTemplate: + sources: + - secret: + name: otel-collector-tls + items: + - key: ca.crt + path: ca.crt + - secret: + name: klio-server-otel-client-tls + items: + - key: tls.crt + path: tls.crt + - key: tls.key + path: tls.key + + postgresql: + pg_hba: + - local replication all peer map=local + + plugins: + - name: klio.cnpg.io + enabled: true + parameters: + pluginConfigurationRef: client-config-cluster-a + + storage: + size: 1Gi +--- +apiVersion: klio.cnpg.io/v1alpha1 +kind: PluginConfiguration +metadata: + name: client-config-cluster-a +spec: + serverAddress: klio-a.default + clientSecretName: cluster-a-klio-user + serverSecretName: klio-a-tls + clusterName: cluster-a + containers: + - name: klio-plugin + env: + - name: OTEL_SERVICE_NAME + value: "klio-plugin" + envFrom: + - configMapRef: + name: cluster-klio-otel-config + - name: klio-restore + env: + - name: OTEL_SERVICE_NAME + value: "klio-restore" + envFrom: + - configMapRef: + name: cluster-klio-otel-config diff --git a/operator/config/samples/opentelemetry-multi/cluster_a_klio_client_auth.yaml b/operator/config/samples/opentelemetry-multi/cluster_a_klio_client_auth.yaml new file mode 100644 index 00000000..9cc7e7b0 --- /dev/null +++ b/operator/config/samples/opentelemetry-multi/cluster_a_klio_client_auth.yaml @@ -0,0 +1,19 @@ +apiVersion: cert-manager.io/v1 +kind: Certificate +metadata: + name: cluster-a-klio-user +spec: + commonName: klio@cluster-a + secretName: cluster-a-klio-user + + duration: 2160h # 90d + renewBefore: 360h # 15d + + isCA: false + usages: + - client auth + + issuerRef: + name: klio-server-ca + kind: Issuer + group: cert-manager.io diff --git a/operator/config/samples/opentelemetry-multi/cluster_b.yaml b/operator/config/samples/opentelemetry-multi/cluster_b.yaml new file mode 100644 index 00000000..e75dd557 --- /dev/null +++ b/operator/config/samples/opentelemetry-multi/cluster_b.yaml @@ -0,0 +1,60 @@ +apiVersion: postgresql.cnpg.io/v1 +kind: Cluster +metadata: + name: cluster-b +spec: + instances: 3 + imagePullPolicy: Always + + projectedVolumeTemplate: + sources: + - secret: + name: otel-collector-tls + items: + - key: ca.crt + path: ca.crt + - secret: + name: klio-server-otel-client-tls + items: + - key: tls.crt + path: tls.crt + - key: tls.key + path: tls.key + + postgresql: + pg_hba: + - local replication all peer map=local + + plugins: + - name: klio.cnpg.io + enabled: true + parameters: + pluginConfigurationRef: client-config-cluster-b + + storage: + size: 1Gi +--- +apiVersion: klio.cnpg.io/v1alpha1 +kind: PluginConfiguration +metadata: + name: client-config-cluster-b +spec: + serverAddress: klio-b.default + clientSecretName: cluster-b-klio-user + serverSecretName: klio-b-tls + clusterName: cluster-b + containers: + - name: klio-plugin + env: + - name: OTEL_SERVICE_NAME + value: "klio-plugin" + envFrom: + - configMapRef: + name: cluster-klio-otel-config + - name: klio-restore + env: + - name: OTEL_SERVICE_NAME + value: "klio-restore" + envFrom: + - configMapRef: + name: cluster-klio-otel-config diff --git a/operator/config/samples/opentelemetry-multi/cluster_b_klio_client_auth.yaml b/operator/config/samples/opentelemetry-multi/cluster_b_klio_client_auth.yaml new file mode 100644 index 00000000..ee37d034 --- /dev/null +++ b/operator/config/samples/opentelemetry-multi/cluster_b_klio_client_auth.yaml @@ -0,0 +1,19 @@ +apiVersion: cert-manager.io/v1 +kind: Certificate +metadata: + name: cluster-b-klio-user +spec: + commonName: klio@cluster-b + secretName: cluster-b-klio-user + + duration: 2160h # 90d + renewBefore: 360h # 15d + + isCA: false + usages: + - client auth + + issuerRef: + name: klio-server-ca + kind: Issuer + group: cert-manager.io diff --git a/operator/config/samples/opentelemetry-multi/cluster_c_klio_client_auth.yaml b/operator/config/samples/opentelemetry-multi/cluster_c_klio_client_auth.yaml new file mode 100644 index 00000000..d47440be --- /dev/null +++ b/operator/config/samples/opentelemetry-multi/cluster_c_klio_client_auth.yaml @@ -0,0 +1,23 @@ +# cluster-c lives in the "team-c" namespace (see ../team-c), but the +# klio-server-ca Issuer is namespace-scoped and only exists here in +# "default". This Certificate is therefore generated here and its resulting +# secret is copied into "team-c" by ../copy-cross-namespace-secrets.sh. +apiVersion: cert-manager.io/v1 +kind: Certificate +metadata: + name: cluster-c-klio-user +spec: + commonName: klio@cluster-c + secretName: cluster-c-klio-user + + duration: 2160h # 90d + renewBefore: 360h # 15d + + isCA: false + usages: + - client auth + + issuerRef: + name: klio-server-ca + kind: Issuer + group: cert-manager.io diff --git a/operator/config/samples/opentelemetry-multi/cluster_klio_otel_config.yaml b/operator/config/samples/opentelemetry-multi/cluster_klio_otel_config.yaml new file mode 100644 index 00000000..9b9da243 --- /dev/null +++ b/operator/config/samples/opentelemetry-multi/cluster_klio_otel_config.yaml @@ -0,0 +1,21 @@ +apiVersion: v1 +kind: ConfigMap +metadata: + name: cluster-klio-otel-config +data: + OTEL_RESOURCE_DETECTORS: "telemetry.sdk,host,os.type,process.executable.name" + # Because traces use the gRPC protocol, Klio also enables Kopia's own + # OTLP/gRPC trace exporter during backup operations: Kopia spans + # (service.name=kopia) are sent to the same collector, reusing the + # OTEL_EXPORTER_OTLP_* settings below (endpoint and TLS). Kopia can only + # export traces over gRPC. + OTEL_TRACES_EXPORTER: "otlp" + OTEL_METRICS_EXPORTER: "otlp" + OTEL_EXPORTER_OTLP_PROTOCOL: "grpc" + OTEL_EXPORTER_OTLP_ENDPOINT: "https://otel-collector:4317" + OTEL_EXPORTER_OTLP_COMPRESSION: "gzip" + OTEL_EXPORTER_OTLP_TIMEOUT: "10000" + OTEL_EXPORTER_OTLP_INSECURE: "false" + OTEL_EXPORTER_OTLP_CERTIFICATE: "/projected/ca.crt" + OTEL_EXPORTER_OTLP_CLIENT_CERTIFICATE: "/projected/tls.crt" + OTEL_EXPORTER_OTLP_CLIENT_KEY: "/projected/tls.key" diff --git a/operator/config/samples/opentelemetry-multi/copy-cross-namespace-secrets.sh b/operator/config/samples/opentelemetry-multi/copy-cross-namespace-secrets.sh new file mode 100755 index 00000000..52c68136 --- /dev/null +++ b/operator/config/samples/opentelemetry-multi/copy-cross-namespace-secrets.sh @@ -0,0 +1,45 @@ +#!/usr/bin/env bash +# Copies the secrets cluster-c (namespace "team-c") needs from klio-b and the +# shared OTel collector, both of which live in "default". cert-manager's +# Issuer resources are namespace-scoped, so the client certificate for +# cluster-c has to be generated in "default" (where the klio-server-ca Issuer +# lives, see cluster_c_klio_client_auth.yaml) and then copied here; the +# server/collector certificates are pinned by exact bytes rather than +# validated through a CA (see core/internal/client/klioclient/grpcclient/connection.go), +# so only their public half needs copying. +# +# Run this after `kubectl apply -k .` and before `kubectl apply -k team-c`. +set -euo pipefail + +SOURCE_NS=default +DEST_NS=team-c + +echo "Waiting for cert-manager to issue the secrets in ${SOURCE_NS}..." +kubectl wait --for=create secret/klio-b-tls -n "${SOURCE_NS}" --timeout=120s +kubectl wait --for=create secret/cluster-c-klio-user -n "${SOURCE_NS}" --timeout=120s +kubectl wait --for=create secret/otel-collector-tls -n "${SOURCE_NS}" --timeout=120s +kubectl wait --for=create secret/klio-server-otel-client-tls -n "${SOURCE_NS}" --timeout=120s + +kubectl create namespace "${DEST_NS}" --dry-run=client -o yaml | kubectl apply -f - + +echo "Copying cluster-c's client certificate (full secret, incl. private key)..." +kubectl get secret cluster-c-klio-user -n "${SOURCE_NS}" -o json | + jq 'del(.metadata.namespace,.metadata.resourceVersion,.metadata.uid,.metadata.creationTimestamp,.metadata.ownerReferences,.metadata.annotations)' | + kubectl apply -n "${DEST_NS}" -f - + +echo "Copying klio-b's server certificate (public cert only, pinned by the client)..." +kubectl get secret klio-b-tls -n "${SOURCE_NS}" -o jsonpath='{.data.tls\.crt}' | base64 -d | + kubectl create secret generic klio-b-tls -n "${DEST_NS}" --from-file=tls.crt=/dev/stdin \ + --dry-run=client -o yaml | kubectl apply -f - + +echo "Copying the OTel collector's certificate (public cert only, used as the trust anchor)..." +kubectl get secret otel-collector-tls -n "${SOURCE_NS}" -o jsonpath='{.data.ca\.crt}' | base64 -d | + kubectl create secret generic otel-collector-ca -n "${DEST_NS}" --from-file=ca.crt=/dev/stdin \ + --dry-run=client -o yaml | kubectl apply -f - + +echo "Copying the shared OTel client certificate (full secret; not verified by the collector)..." +kubectl get secret klio-server-otel-client-tls -n "${SOURCE_NS}" -o json | + jq 'del(.metadata.namespace,.metadata.resourceVersion,.metadata.uid,.metadata.creationTimestamp,.metadata.ownerReferences,.metadata.annotations)' | + kubectl apply -n "${DEST_NS}" -f - + +echo "Done. You can now run: kubectl apply -k team-c" diff --git a/operator/config/samples/opentelemetry-multi/issuer.yaml b/operator/config/samples/opentelemetry-multi/issuer.yaml new file mode 100644 index 00000000..981dfb78 --- /dev/null +++ b/operator/config/samples/opentelemetry-multi/issuer.yaml @@ -0,0 +1,7 @@ +# cert-manager self-signed issuer, used to generate the certificates in this example. +apiVersion: cert-manager.io/v1 +kind: Issuer +metadata: + name: selfsigned-issuer +spec: + selfSigned: {} diff --git a/operator/config/samples/opentelemetry-multi/jaeger.yaml b/operator/config/samples/opentelemetry-multi/jaeger.yaml new file mode 100644 index 00000000..13f29bb5 --- /dev/null +++ b/operator/config/samples/opentelemetry-multi/jaeger.yaml @@ -0,0 +1,71 @@ +# This is a simple Jaeger all-in-one deployment for testing purposes. +# It is NOT recommended for production use. +apiVersion: apps/v1 +kind: Deployment +metadata: + labels: + app: jaeger + name: jaeger +spec: + replicas: 1 + selector: + matchLabels: + app: jaeger + strategy: {} + template: + metadata: + labels: + app: jaeger + spec: + automountServiceAccountToken: false + containers: + - image: cr.jaegertracing.io/jaegertracing/jaeger:2.20.0@sha256:46a886260e04002d8f45e213fc39063fa11a50446048fdaa64786fc0840cb9f8 + name: all-in-one + ports: + - containerPort: 16686 + protocol: TCP + - containerPort: 4317 + protocol: TCP + - containerPort: 4318 + protocol: TCP + - containerPort: 5778 + protocol: TCP + - containerPort: 9411 + protocol: TCP + resources: + requests: + cpu: 100m + memory: 128Mi + ephemeral-storage: 100Mi +status: {} +--- +apiVersion: v1 +kind: Service +metadata: + labels: + app: jaeger + name: jaeger +spec: + ports: + - name: port-1 + port: 16686 + protocol: TCP + targetPort: 16686 + - name: port-2 + port: 4317 + protocol: TCP + targetPort: 4317 + - name: port-3 + port: 4318 + protocol: TCP + targetPort: 4318 + - name: port-4 + port: 5778 + protocol: TCP + targetPort: 5778 + - name: port-5 + port: 9411 + protocol: TCP + targetPort: 9411 + selector: + app: jaeger diff --git a/operator/config/samples/opentelemetry-multi/klio_encryption.yaml b/operator/config/samples/opentelemetry-multi/klio_encryption.yaml new file mode 100644 index 00000000..02f050f5 --- /dev/null +++ b/operator/config/samples/opentelemetry-multi/klio_encryption.yaml @@ -0,0 +1,18 @@ +# Shared encryption identity for both klio-a and klio-b. Reusing the same +# identity across servers is fine for this demo; production deployments +# would normally give each server its own. +apiVersion: v1 +stringData: + secret-key: AGE-SECRET-KEY-1P7TWJ8H837DPVLSRVHAP0ZCZVHXJRTUS7S0A5VZAR49URW0Z9Q0SFW4HVM + public-key: age16y2zu5qptkzateyuarcahs0am75qjpu0xd2ty2dvm5jusnfzxg0scjh5uu + encryption-key: | + -----BEGIN AGE ENCRYPTED FILE----- + YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSAyMzJERW1QTUhjNGdlNmdi + WmFLZ3FLL1lZclduUWEvZkJncGdJZmtVd1FzCmZwSHVTcWNMRFN6a2xpY0NHNVRv + MlltSlNRY1B4bmNoOUxuZzZ5RENSSmsKLS0tIGNYUUNzSGVqNUZhS3VodnZ1cXNP + T3JHS3JRSUNkR1RyeEROYnZYNnNSV00KUd3HyxvvqCyW6hC1sWNDn/xeptHsoz/e + hxCgWoSJvfSfNk174njMwEeW+ic= + -----END AGE ENCRYPTED FILE----- +kind: Secret +metadata: + name: klio-encryption diff --git a/operator/config/samples/opentelemetry-multi/klio_server_a.yaml b/operator/config/samples/opentelemetry-multi/klio_server_a.yaml new file mode 100644 index 00000000..ff71ec2e --- /dev/null +++ b/operator/config/samples/opentelemetry-multi/klio_server_a.yaml @@ -0,0 +1,93 @@ +apiVersion: klio.cnpg.io/v1alpha1 +kind: Server +metadata: + name: klio-a +spec: + queue: + pvcTemplate: + resources: + requests: + storage: 1Gi + accessModes: + - ReadWriteOnce + tier1: + cache: + pvcTemplate: + resources: + requests: + storage: 1Gi + accessModes: + - ReadWriteOnce + + data: + pvcTemplate: + resources: + requests: + storage: 1Gi + accessModes: + - ReadWriteOnce + + encryptionKeyFile: + fileReference: + volume: + secret: + secretName: klio-encryption + path: encryption-key + + identityFile: + fileReference: + volume: + secret: + secretName: klio-encryption + path: secret-key + + caSecretName: klio-server-ca + tlsSecretName: klio-a-tls + + image: registry.dev:5000/klio-testing:dev + imagePullPolicy: Always + + template: + spec: + containers: + - name: server + env: + # Use the downward API to get pod and node information + # These environment variables will be used in the OTEL_RESOURCE_ATTRIBUTES + # to provide more context about the telemetry data for the otel collector k8sattributes processor + - name: NODE_NAME + valueFrom: + fieldRef: + fieldPath: spec.nodeName + - name: POD_UID + valueFrom: + fieldRef: + fieldPath: metadata.uid + # OTEL_SERVICE_NAME and OTEL_RESOURCE_ATTRIBUTES are set specifically for each container + - name: OTEL_SERVICE_NAME + value: "klio-server-a" + - name: OTEL_RESOURCE_ATTRIBUTES + value: "k8s.node.name=$(NODE_NAME),k8s.pod.uid=$(POD_UID)" + envFrom: + # Use a common config map for all shared OTEL settings + - configMapRef: + name: klio-server-otel-config + volumeMounts: + - mountPath: /otel + name: otel + volumes: + - name: otel + projected: + sources: + - secret: + name: otel-collector-tls + items: + - key: ca.crt + path: ca.crt + - secret: + name: klio-server-otel-client-tls + items: + - key: tls.crt + path: tls.crt + - key: tls.key + path: tls.key diff --git a/operator/config/samples/opentelemetry-multi/klio_server_a_certificate.yaml b/operator/config/samples/opentelemetry-multi/klio_server_a_certificate.yaml new file mode 100644 index 00000000..2f6debab --- /dev/null +++ b/operator/config/samples/opentelemetry-multi/klio_server_a_certificate.yaml @@ -0,0 +1,23 @@ +apiVersion: cert-manager.io/v1 +kind: Certificate +metadata: + name: klio-a +spec: + secretName: klio-a-tls + commonName: klio-a + dnsNames: + - klio-a + - klio-a.default + - klio-a.default.svc + + duration: 2160h # 90d + renewBefore: 360h # 15d + + isCA: false + usages: + - server auth + + issuerRef: + name: selfsigned-issuer + kind: Issuer + group: cert-manager.io diff --git a/operator/config/samples/opentelemetry-multi/klio_server_b.yaml b/operator/config/samples/opentelemetry-multi/klio_server_b.yaml new file mode 100644 index 00000000..3b9ac8bf --- /dev/null +++ b/operator/config/samples/opentelemetry-multi/klio_server_b.yaml @@ -0,0 +1,93 @@ +apiVersion: klio.cnpg.io/v1alpha1 +kind: Server +metadata: + name: klio-b +spec: + queue: + pvcTemplate: + resources: + requests: + storage: 1Gi + accessModes: + - ReadWriteOnce + tier1: + cache: + pvcTemplate: + resources: + requests: + storage: 1Gi + accessModes: + - ReadWriteOnce + + data: + pvcTemplate: + resources: + requests: + storage: 1Gi + accessModes: + - ReadWriteOnce + + encryptionKeyFile: + fileReference: + volume: + secret: + secretName: klio-encryption + path: encryption-key + + identityFile: + fileReference: + volume: + secret: + secretName: klio-encryption + path: secret-key + + caSecretName: klio-server-ca + tlsSecretName: klio-b-tls + + image: registry.dev:5000/klio-testing:dev + imagePullPolicy: Always + + template: + spec: + containers: + - name: server + env: + # Use the downward API to get pod and node information + # These environment variables will be used in the OTEL_RESOURCE_ATTRIBUTES + # to provide more context about the telemetry data for the otel collector k8sattributes processor + - name: NODE_NAME + valueFrom: + fieldRef: + fieldPath: spec.nodeName + - name: POD_UID + valueFrom: + fieldRef: + fieldPath: metadata.uid + # OTEL_SERVICE_NAME and OTEL_RESOURCE_ATTRIBUTES are set specifically for each container + - name: OTEL_SERVICE_NAME + value: "klio-server-b" + - name: OTEL_RESOURCE_ATTRIBUTES + value: "k8s.node.name=$(NODE_NAME),k8s.pod.uid=$(POD_UID)" + envFrom: + # Use a common config map for all shared OTEL settings + - configMapRef: + name: klio-server-otel-config + volumeMounts: + - mountPath: /otel + name: otel + volumes: + - name: otel + projected: + sources: + - secret: + name: otel-collector-tls + items: + - key: ca.crt + path: ca.crt + - secret: + name: klio-server-otel-client-tls + items: + - key: tls.crt + path: tls.crt + - key: tls.key + path: tls.key diff --git a/operator/config/samples/opentelemetry-multi/klio_server_b_certificate.yaml b/operator/config/samples/opentelemetry-multi/klio_server_b_certificate.yaml new file mode 100644 index 00000000..18c67786 --- /dev/null +++ b/operator/config/samples/opentelemetry-multi/klio_server_b_certificate.yaml @@ -0,0 +1,28 @@ +# klio-b is reached both from cluster-b (same namespace, "default") and from +# cluster-c (namespace "team-c", see ../team-c), so its dnsNames must cover +# both the short in-namespace form and the fully-qualified cross-namespace +# form used as PluginConfiguration.serverAddress by cluster-c. +apiVersion: cert-manager.io/v1 +kind: Certificate +metadata: + name: klio-b +spec: + secretName: klio-b-tls + commonName: klio-b + dnsNames: + - klio-b + - klio-b.default + - klio-b.default.svc + - klio-b.default.svc.cluster.local + + duration: 2160h # 90d + renewBefore: 360h # 15d + + isCA: false + usages: + - server auth + + issuerRef: + name: selfsigned-issuer + kind: Issuer + group: cert-manager.io diff --git a/operator/config/samples/opentelemetry-multi/klio_server_ca.yaml b/operator/config/samples/opentelemetry-multi/klio_server_ca.yaml new file mode 100644 index 00000000..07d2a8d3 --- /dev/null +++ b/operator/config/samples/opentelemetry-multi/klio_server_ca.yaml @@ -0,0 +1,23 @@ +# Root CA shared by both Klio servers (klio-a, klio-b) in this sample, and by +# the cross-namespace client certificate generated for cluster-c (see +# ../copy-cross-namespace-secrets.sh). Sharing a single CA lets any client +# certificate it signs be trusted by either server's caSecretName. +apiVersion: cert-manager.io/v1 +kind: Certificate +metadata: + name: klio-server-ca +spec: + commonName: klio-server + secretName: klio-server-ca + + duration: 2160h # 90d + renewBefore: 360h # 15d + + isCA: true + usages: + - cert sign + + issuerRef: + name: selfsigned-issuer + kind: Issuer + group: cert-manager.io diff --git a/operator/config/samples/opentelemetry-multi/klio_server_ca_issuer.yaml b/operator/config/samples/opentelemetry-multi/klio_server_ca_issuer.yaml new file mode 100644 index 00000000..0c06112f --- /dev/null +++ b/operator/config/samples/opentelemetry-multi/klio_server_ca_issuer.yaml @@ -0,0 +1,7 @@ +apiVersion: cert-manager.io/v1 +kind: Issuer +metadata: + name: klio-server-ca +spec: + ca: + secretName: klio-server-ca diff --git a/operator/config/samples/opentelemetry-multi/klio_server_otel_client_certificate.yaml b/operator/config/samples/opentelemetry-multi/klio_server_otel_client_certificate.yaml new file mode 100644 index 00000000..e28f9213 --- /dev/null +++ b/operator/config/samples/opentelemetry-multi/klio_server_otel_client_certificate.yaml @@ -0,0 +1,23 @@ +apiVersion: cert-manager.io/v1 +kind: Certificate +metadata: + name: klio-server-otel-client +spec: + secretName: klio-server-otel-client-tls + commonName: klio + dnsNames: + - klio + - klio.default + - klio.default.svc + + duration: 2160h # 90d + renewBefore: 360h # 15d + + isCA: false + usages: + - server auth + + issuerRef: + name: selfsigned-issuer + kind: Issuer + group: cert-manager.io diff --git a/operator/config/samples/opentelemetry-multi/klio_server_otel_config.yaml b/operator/config/samples/opentelemetry-multi/klio_server_otel_config.yaml new file mode 100644 index 00000000..9d78c2de --- /dev/null +++ b/operator/config/samples/opentelemetry-multi/klio_server_otel_config.yaml @@ -0,0 +1,28 @@ +apiVersion: v1 +kind: ConfigMap +metadata: + name: klio-server-otel-config +data: + OTEL_RESOURCE_DETECTORS: "telemetry.sdk,host,os.type,process.executable.name" + # Because traces use the gRPC protocol, Klio also enables Kopia's own + # OTLP/gRPC trace exporter: Kopia spans (service.name=kopia) are sent to the + # same collector, reusing the OTEL_EXPORTER_OTLP_TRACES_* settings below + # (endpoint and TLS). Kopia can only export traces over gRPC. + OTEL_TRACES_EXPORTER: "otlp" + OTEL_EXPORTER_OTLP_TRACES_PROTOCOL: "grpc" + OTEL_EXPORTER_OTLP_TRACES_ENDPOINT: "https://otel-collector:4317" + OTEL_EXPORTER_OTLP_TRACES_COMPRESSION: "gzip" + OTEL_EXPORTER_OTLP_TRACES_TIMEOUT: "10000" + OTEL_EXPORTER_OTLP_TRACES_INSECURE: "false" + OTEL_EXPORTER_OTLP_TRACES_CERTIFICATE: "/otel/ca.crt" + OTEL_EXPORTER_OTLP_TRACES_CLIENT_CERTIFICATE: "/otel/tls.crt" + OTEL_EXPORTER_OTLP_TRACES_CLIENT_KEY: "/otel/tls.key" + OTEL_METRICS_EXPORTER: "otlp" + OTEL_METRIC_EXPORT_INTERVAL: "60000" + OTEL_EXPORTER_OTLP_METRICS_PROTOCOL: "grpc" + OTEL_EXPORTER_OTLP_METRICS_ENDPOINT: "https://otel-collector:4317" + OTEL_EXPORTER_OTLP_METRICS_TIMEOUT: "60000" + OTEL_EXPORTER_OTLP_METRICS_INSECURE: "false" + OTEL_EXPORTER_OTLP_METRICS_CERTIFICATE: "/otel/ca.crt" + OTEL_EXPORTER_OTLP_METRICS_CLIENT_CERTIFICATE: "/otel/tls.crt" + OTEL_EXPORTER_OTLP_METRICS_CLIENT_KEY: "/otel/tls.key" diff --git a/operator/config/samples/opentelemetry-multi/kustomization.yaml b/operator/config/samples/opentelemetry-multi/kustomization.yaml new file mode 100644 index 00000000..7baafcce --- /dev/null +++ b/operator/config/samples/opentelemetry-multi/kustomization.yaml @@ -0,0 +1,32 @@ +# REQUIRES: +# kubectl apply -f https://github.com/open-telemetry/opentelemetry-operator/releases/latest/download/opentelemetry-operator.yaml +# +# Deploys two Klio servers (klio-a, klio-b) and two CNPG clusters +# (cluster-a -> klio-a, cluster-b -> klio-b) into the "default" namespace, +# sharing one OTel collector / Jaeger / Prometheus stack. See ../team-c for +# the third cluster, which lives in a different namespace and is backed by +# klio-b. +resources: + - cluster_klio_otel_config.yaml + - cluster_a.yaml + - cluster_a_klio_client_auth.yaml + - cluster_b.yaml + - cluster_b_klio_client_auth.yaml + - cluster_c_klio_client_auth.yaml + - issuer.yaml + - jaeger.yaml + - klio_encryption.yaml + - klio_server_a.yaml + - klio_server_a_certificate.yaml + - klio_server_b.yaml + - klio_server_b_certificate.yaml + - klio_server_ca.yaml + - klio_server_ca_issuer.yaml + - klio_server_otel_client_certificate.yaml + - klio_server_otel_config.yaml + - otel_collector.yaml + - otel_collector_certificate.yaml + - otel_collector_clusterrole.yaml + - otel_collector_clusterrolebinding.yaml + - otel_collector_sa.yaml + - otel_collector_svc_monitor.yaml diff --git a/operator/config/samples/opentelemetry-multi/otel_collector.yaml b/operator/config/samples/opentelemetry-multi/otel_collector.yaml new file mode 100644 index 00000000..852b9e94 --- /dev/null +++ b/operator/config/samples/opentelemetry-multi/otel_collector.yaml @@ -0,0 +1,83 @@ +# OpenTelemetry Collector instance +# A full-fledged configuration is outside of the scope of this example. +# This example receives OTLP data over gRPC with TLS, adds Kubernetes attributes, +# and exports to a Jaeger instance (without TLS, for simplicity) and to Prometheus. +# It also exports to a debug exporter, which prints the data to the collector's logs. +# OpenTelemetry Operator is required to run this: +# kubectl apply -f https://github.com/open-telemetry/opentelemetry-operator/releases/latest/download/opentelemetry-operator.yaml +apiVersion: opentelemetry.io/v1beta1 +kind: OpenTelemetryCollector +metadata: + name: otel +spec: + image: "otel/opentelemetry-collector-contrib:0.157.0@sha256:f2f01157055a9b2aab9df7118e1f1c9abf345e99b23bc7a2bc791db374a7d0f6" + config: + receivers: + # otlp receiver to receive data in OTLP format over gRPC. TLS is enabled. + # Must match the settings in the clients. + otlp: + protocols: + grpc: + endpoint: 0.0.0.0:4317 + tls: + cert_file: /etc/otel/certs/tls.crt + key_file: /etc/otel/certs/tls.key + processors: + memory_limiter: + check_interval: 1s + limit_percentage: 75 + spike_limit_percentage: 15 + batch: + send_batch_size: 10000 + timeout: 10s + k8sattributes: + auth_type: 'serviceAccount' + extract: + metadata: # extracted from the pod + - k8s.namespace.name + - k8s.node.name + - k8s.pod.name + - k8s.pod.start_time + - k8s.pod.uid + - container.image.name + - container.image.tag + - k8s.container.name + pod_association: # How to associate the data to a pod (order matters) + - sources: # First try to use the value of the resource attribute k8s.pod.ip + - from: resource_attribute + name: k8s.pod.ip + - sources: # Then try to use the value of the resource attribute k8s.pod.uid + - from: resource_attribute + name: k8s.pod.uid + - sources: # If neither of those work, use the request's connection to get the pod IP. + - from: connection + exporters: + debug: + verbosity: detailed + prometheus: + endpoint: "0.0.0.0:9464" + send_timestamps: true + resource_to_telemetry_conversion: + enabled: true + otlp: + endpoint: jaeger:4317 + tls: + insecure: true + service: + pipelines: + traces: + receivers: [ otlp ] + processors: [ memory_limiter, batch, k8sattributes ] + exporters: [ debug, otlp ] + metrics: + receivers: [ otlp ] + processors: [ memory_limiter, batch, k8sattributes ] + exporters: [ debug, prometheus ] + volumeMounts: + - name: otel-tls + mountPath: /etc/otel/certs + readOnly: true + volumes: + - name: otel-tls + secret: + secretName: otel-collector-tls diff --git a/operator/config/samples/opentelemetry-multi/otel_collector_certificate.yaml b/operator/config/samples/opentelemetry-multi/otel_collector_certificate.yaml new file mode 100644 index 00000000..05ae0e8e --- /dev/null +++ b/operator/config/samples/opentelemetry-multi/otel_collector_certificate.yaml @@ -0,0 +1,13 @@ +apiVersion: cert-manager.io/v1 +kind: Certificate +metadata: + name: otel-collector +spec: + secretName: otel-collector-tls + issuerRef: + name: selfsigned-issuer + kind: Issuer + dnsNames: + - otel-collector.default.svc.cluster.local + - otel-collector.default.svc + - otel-collector diff --git a/operator/config/samples/opentelemetry-multi/otel_collector_clusterrole.yaml b/operator/config/samples/opentelemetry-multi/otel_collector_clusterrole.yaml new file mode 100644 index 00000000..e4cdbac1 --- /dev/null +++ b/operator/config/samples/opentelemetry-multi/otel_collector_clusterrole.yaml @@ -0,0 +1,33 @@ +# Role with permissions to read pods, namespaces, and replicasets. +# Required for the k8sattributes processor to work + +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRole +metadata: + name: otel-collector +rules: + - apiGroups: + - '' + resources: + - 'pods' + - 'namespaces' + verbs: + - 'get' + - 'list' + - 'watch' + - apiGroups: + - 'apps' + resources: + - 'replicasets' + verbs: + - 'get' + - 'list' + - 'watch' + - apiGroups: + - 'extensions' + resources: + - 'replicasets' + verbs: + - 'get' + - 'list' + - 'watch' diff --git a/operator/config/samples/opentelemetry-multi/otel_collector_clusterrolebinding.yaml b/operator/config/samples/opentelemetry-multi/otel_collector_clusterrolebinding.yaml new file mode 100644 index 00000000..37c13b64 --- /dev/null +++ b/operator/config/samples/opentelemetry-multi/otel_collector_clusterrolebinding.yaml @@ -0,0 +1,12 @@ +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRoleBinding +metadata: + name: otel-collector +subjects: + - kind: ServiceAccount + name: otel-collector + namespace: default +roleRef: + kind: ClusterRole + name: otel-collector + apiGroup: rbac.authorization.k8s.io diff --git a/operator/config/samples/opentelemetry-multi/otel_collector_sa.yaml b/operator/config/samples/opentelemetry-multi/otel_collector_sa.yaml new file mode 100644 index 00000000..fcba07ec --- /dev/null +++ b/operator/config/samples/opentelemetry-multi/otel_collector_sa.yaml @@ -0,0 +1,4 @@ +apiVersion: v1 +kind: ServiceAccount +metadata: + name: otel-collector diff --git a/operator/config/samples/opentelemetry-multi/otel_collector_svc_monitor.yaml b/operator/config/samples/opentelemetry-multi/otel_collector_svc_monitor.yaml new file mode 100644 index 00000000..9d465a12 --- /dev/null +++ b/operator/config/samples/opentelemetry-multi/otel_collector_svc_monitor.yaml @@ -0,0 +1,18 @@ +# Expose the OpenTelemetry Collector prometheys exporter to Prometheus using a ServiceMonitor. +apiVersion: monitoring.coreos.com/v1 +kind: ServiceMonitor +metadata: + name: otel-collector +spec: + endpoints: + - path: /metrics + port: prometheus + scheme: http + interval: 30s + scrapeTimeout: 10s + selector: + matchLabels: + app.kubernetes.io/component: opentelemetry-collector + app.kubernetes.io/instance: default.otel + app.kubernetes.io/name: otel-collector + operator.opentelemetry.io/collector-service-type: base diff --git a/operator/config/samples/opentelemetry-multi/team-c/cluster_c.yaml b/operator/config/samples/opentelemetry-multi/team-c/cluster_c.yaml new file mode 100644 index 00000000..0a2b7d8c --- /dev/null +++ b/operator/config/samples/opentelemetry-multi/team-c/cluster_c.yaml @@ -0,0 +1,68 @@ +# cluster-c is backed by klio-b, which lives in the "default" namespace, not +# here. This exercises a single Klio server serving clusters from more than +# one namespace. serverSecretName/clientSecretName/the OTel client secret +# below are NOT created by this file: they are copied in from "default" by +# ../copy-cross-namespace-secrets.sh, which must be run before applying this +# kustomization. +apiVersion: postgresql.cnpg.io/v1 +kind: Cluster +metadata: + name: cluster-c + namespace: team-c +spec: + instances: 3 + imagePullPolicy: Always + + projectedVolumeTemplate: + sources: + - secret: + name: otel-collector-ca + items: + - key: ca.crt + path: ca.crt + - secret: + name: klio-server-otel-client-tls + items: + - key: tls.crt + path: tls.crt + - key: tls.key + path: tls.key + + postgresql: + pg_hba: + - local replication all peer map=local + + plugins: + - name: klio.cnpg.io + enabled: true + parameters: + pluginConfigurationRef: client-config-cluster-c + + storage: + size: 1Gi +--- +apiVersion: klio.cnpg.io/v1alpha1 +kind: PluginConfiguration +metadata: + name: client-config-cluster-c + namespace: team-c +spec: + serverAddress: klio-b.default.svc.cluster.local + clientSecretName: cluster-c-klio-user + serverSecretName: klio-b-tls + clusterName: cluster-c + containers: + - name: klio-plugin + env: + - name: OTEL_SERVICE_NAME + value: "klio-plugin" + envFrom: + - configMapRef: + name: cluster-klio-otel-config + - name: klio-restore + env: + - name: OTEL_SERVICE_NAME + value: "klio-restore" + envFrom: + - configMapRef: + name: cluster-klio-otel-config diff --git a/operator/config/samples/opentelemetry-multi/team-c/cluster_klio_otel_config.yaml b/operator/config/samples/opentelemetry-multi/team-c/cluster_klio_otel_config.yaml new file mode 100644 index 00000000..d7cbbdad --- /dev/null +++ b/operator/config/samples/opentelemetry-multi/team-c/cluster_klio_otel_config.yaml @@ -0,0 +1,20 @@ +# Same settings as ../cluster_klio_otel_config.yaml, but the endpoint is +# fully-qualified since the otel-collector Service lives in "default", not +# in this namespace. +apiVersion: v1 +kind: ConfigMap +metadata: + name: cluster-klio-otel-config + namespace: team-c +data: + OTEL_RESOURCE_DETECTORS: "telemetry.sdk,host,os.type,process.executable.name" + OTEL_TRACES_EXPORTER: "otlp" + OTEL_METRICS_EXPORTER: "otlp" + OTEL_EXPORTER_OTLP_PROTOCOL: "grpc" + OTEL_EXPORTER_OTLP_ENDPOINT: "https://otel-collector.default.svc.cluster.local:4317" + OTEL_EXPORTER_OTLP_COMPRESSION: "gzip" + OTEL_EXPORTER_OTLP_TIMEOUT: "10000" + OTEL_EXPORTER_OTLP_INSECURE: "false" + OTEL_EXPORTER_OTLP_CERTIFICATE: "/projected/ca.crt" + OTEL_EXPORTER_OTLP_CLIENT_CERTIFICATE: "/projected/tls.crt" + OTEL_EXPORTER_OTLP_CLIENT_KEY: "/projected/tls.key" diff --git a/operator/config/samples/opentelemetry-multi/team-c/kustomization.yaml b/operator/config/samples/opentelemetry-multi/team-c/kustomization.yaml new file mode 100644 index 00000000..72c31da7 --- /dev/null +++ b/operator/config/samples/opentelemetry-multi/team-c/kustomization.yaml @@ -0,0 +1,9 @@ +# Apply ../kustomization.yaml first, then run ../copy-cross-namespace-secrets.sh +# to populate the secrets this kustomization expects to already exist +# (klio-b-tls, cluster-c-klio-user, otel-collector-ca, +# klio-server-otel-client-tls), then apply this one. +namespace: team-c +resources: + - namespace.yaml + - cluster_klio_otel_config.yaml + - cluster_c.yaml diff --git a/operator/config/samples/opentelemetry-multi/team-c/namespace.yaml b/operator/config/samples/opentelemetry-multi/team-c/namespace.yaml new file mode 100644 index 00000000..3375cd25 --- /dev/null +++ b/operator/config/samples/opentelemetry-multi/team-c/namespace.yaml @@ -0,0 +1,4 @@ +apiVersion: v1 +kind: Namespace +metadata: + name: team-c From a25fe46a619e646d0c9eb0df2ee7ff257058499c Mon Sep 17 00:00:00 2001 From: Gabriele Quaresima Date: Wed, 5 Aug 2026 12:06:26 +0200 Subject: [PATCH 2/5] docs(samples): exercise a duplicate Klio server name across namespaces Adds a fourth deployment to opentelemetry-multi: team-e runs its own Klio server also named "klio-a" (see ../klio_server_a.yaml in "default"). Since a Server's StatefulSet pod name is derived from the Server's own name alone, both servers' pods are named "klio-a-klio-0", giving them an identical host_name label -- a case the dashboard's $server variable cannot disambiguate on its own. bootstrap-remote-server.sh bootstraps this scenario: it copies the shared CA's public certificate (so team-e's independently self-signed server can validate clients signed by the same klio-server-ca) and the OTel collector trust anchor, plus one cluster's client certificate generated in "default" (cert-manager Issuers are namespace-scoped). Confirmed live on Kind: both servers run independently and cluster-e backs up successfully. Querying Prometheus directly shows two distinct series for host_name=klio-a-klio-0 (uptime ~9778s in default vs ~299s in team-e, disambiguated only by k8s_namespace_name); the dashboard's actual "Server uptime" panel query, with $namespace left at its default "All" and $server narrowed to klio-a-klio-0, collapses both into a single value and silently picks one. Assisted-by: Claude Signed-off-by: Gabriele Quaresima --- .../bootstrap-remote-server.sh | 56 +++++++++++ .../cluster_e_klio_client_auth.yaml | 26 ++++++ .../opentelemetry-multi/kustomization.yaml | 8 +- .../opentelemetry-multi/team-e/cluster_e.yaml | 62 +++++++++++++ .../team-e/cluster_klio_otel_config.yaml | 20 ++++ .../opentelemetry-multi/team-e/issuer.yaml | 10 ++ .../team-e/klio_encryption.yaml | 18 ++++ .../team-e/klio_server_a.yaml | 92 +++++++++++++++++++ .../team-e/klio_server_a_certificate.yaml | 29 ++++++ .../team-e/klio_server_otel_config.yaml | 28 ++++++ .../team-e/kustomization.yaml | 15 +++ .../opentelemetry-multi/team-e/namespace.yaml | 4 + 12 files changed, 365 insertions(+), 3 deletions(-) create mode 100755 operator/config/samples/opentelemetry-multi/bootstrap-remote-server.sh create mode 100644 operator/config/samples/opentelemetry-multi/cluster_e_klio_client_auth.yaml create mode 100644 operator/config/samples/opentelemetry-multi/team-e/cluster_e.yaml create mode 100644 operator/config/samples/opentelemetry-multi/team-e/cluster_klio_otel_config.yaml create mode 100644 operator/config/samples/opentelemetry-multi/team-e/issuer.yaml create mode 100644 operator/config/samples/opentelemetry-multi/team-e/klio_encryption.yaml create mode 100644 operator/config/samples/opentelemetry-multi/team-e/klio_server_a.yaml create mode 100644 operator/config/samples/opentelemetry-multi/team-e/klio_server_a_certificate.yaml create mode 100644 operator/config/samples/opentelemetry-multi/team-e/klio_server_otel_config.yaml create mode 100644 operator/config/samples/opentelemetry-multi/team-e/kustomization.yaml create mode 100644 operator/config/samples/opentelemetry-multi/team-e/namespace.yaml diff --git a/operator/config/samples/opentelemetry-multi/bootstrap-remote-server.sh b/operator/config/samples/opentelemetry-multi/bootstrap-remote-server.sh new file mode 100755 index 00000000..7ea7f3ee --- /dev/null +++ b/operator/config/samples/opentelemetry-multi/bootstrap-remote-server.sh @@ -0,0 +1,56 @@ +#!/usr/bin/env bash +# Bootstraps a namespace that runs its OWN local Klio server (self-signed, +# not copied from "default") which nonetheless needs to: (a) validate client +# certs signed by the shared klio-server-ca, and (b) export OTel data to the +# shared collector in "default". Unlike copy-cross-namespace-secrets.sh +# (which pins a specific EXISTING server's own cert for a remote client), +# this copies the CA's public cert so any locally-issued server certificate +# can validate against it, plus the shared OTel collector/client certs, plus +# one cluster's client cert (generated in "default", where klio-server-ca +# lives, then copied here since Issuers are namespace-scoped). +# +# Usage: bootstrap-remote-server.sh +# +# Run this after `kubectl apply -k .` and before applying the destination +# namespace's kustomization, e.g.: +# ./bootstrap-remote-server.sh team-e cluster-e-klio-user +set -euo pipefail + +if [ $# -ne 2 ]; then + echo "usage: $0 " >&2 + exit 1 +fi + +SOURCE_NS=default +DEST_NS="$1" +CLIENT_SECRET_NAME="$2" + +echo "Waiting for cert-manager to issue the secrets in ${SOURCE_NS}..." +kubectl wait --for=create secret/klio-server-ca -n "${SOURCE_NS}" --timeout=120s +kubectl wait --for=create "secret/${CLIENT_SECRET_NAME}" -n "${SOURCE_NS}" --timeout=120s +kubectl wait --for=create secret/otel-collector-tls -n "${SOURCE_NS}" --timeout=120s +kubectl wait --for=create secret/klio-server-otel-client-tls -n "${SOURCE_NS}" --timeout=120s + +kubectl create namespace "${DEST_NS}" --dry-run=client -o yaml | kubectl apply -f - + +echo "Copying the shared CA's public certificate (so a locally-issued server cert can validate clients signed by it)..." +kubectl get secret klio-server-ca -n "${SOURCE_NS}" -o jsonpath='{.data.tls\.crt}' | base64 -d | + kubectl create secret generic klio-server-ca -n "${DEST_NS}" --from-file=tls.crt=/dev/stdin \ + --dry-run=client -o yaml | kubectl apply -f - + +echo "Copying the OTel collector's certificate (public cert only, used as the trust anchor)..." +kubectl get secret otel-collector-tls -n "${SOURCE_NS}" -o jsonpath='{.data.ca\.crt}' | base64 -d | + kubectl create secret generic otel-collector-ca -n "${DEST_NS}" --from-file=ca.crt=/dev/stdin \ + --dry-run=client -o yaml | kubectl apply -f - + +echo "Copying the shared OTel client certificate (full secret; not verified by the collector)..." +kubectl get secret klio-server-otel-client-tls -n "${SOURCE_NS}" -o json | + jq 'del(.metadata.namespace,.metadata.resourceVersion,.metadata.uid,.metadata.creationTimestamp,.metadata.ownerReferences,.metadata.annotations)' | + kubectl apply -n "${DEST_NS}" -f - + +echo "Copying the cluster's client certificate (full secret, incl. private key)..." +kubectl get secret "${CLIENT_SECRET_NAME}" -n "${SOURCE_NS}" -o json | + jq 'del(.metadata.namespace,.metadata.resourceVersion,.metadata.uid,.metadata.creationTimestamp,.metadata.ownerReferences,.metadata.annotations)' | + kubectl apply -n "${DEST_NS}" -f - + +echo "Done. You can now apply ${DEST_NS}'s kustomization." diff --git a/operator/config/samples/opentelemetry-multi/cluster_e_klio_client_auth.yaml b/operator/config/samples/opentelemetry-multi/cluster_e_klio_client_auth.yaml new file mode 100644 index 00000000..441c0444 --- /dev/null +++ b/operator/config/samples/opentelemetry-multi/cluster_e_klio_client_auth.yaml @@ -0,0 +1,26 @@ +# Client cert for cluster-e, backed by a SECOND Klio server also named +# "klio-a" (see ../team-e), living in a different namespace than the +# original klio-a in "default". Since a Server's StatefulSet pod name +# equals the Server's own name, this second klio-a's pod is ALSO named +# "klio-a-klio-0" -- an identical host_name to the original, but in a +# different namespace. Generated here because the klio-server-ca Issuer is +# namespace-scoped; copied into team-e by bootstrap-remote-server.sh. +apiVersion: cert-manager.io/v1 +kind: Certificate +metadata: + name: cluster-e-klio-user +spec: + commonName: klio@cluster-e + secretName: cluster-e-klio-user + + duration: 2160h # 90d + renewBefore: 360h # 15d + + isCA: false + usages: + - client auth + + issuerRef: + name: klio-server-ca + kind: Issuer + group: cert-manager.io diff --git a/operator/config/samples/opentelemetry-multi/kustomization.yaml b/operator/config/samples/opentelemetry-multi/kustomization.yaml index 7baafcce..bce79d75 100644 --- a/operator/config/samples/opentelemetry-multi/kustomization.yaml +++ b/operator/config/samples/opentelemetry-multi/kustomization.yaml @@ -3,9 +3,10 @@ # # Deploys two Klio servers (klio-a, klio-b) and two CNPG clusters # (cluster-a -> klio-a, cluster-b -> klio-b) into the "default" namespace, -# sharing one OTel collector / Jaeger / Prometheus stack. See ../team-c for -# the third cluster, which lives in a different namespace and is backed by -# klio-b. +# sharing one OTel collector / Jaeger / Prometheus stack. See: +# ../team-c for a third cluster in a different namespace, backed by klio-b +# ../team-e for a second, independent server also named "klio-a" (so its +# StatefulSet pod, and therefore host_name, collides with the original) resources: - cluster_klio_otel_config.yaml - cluster_a.yaml @@ -13,6 +14,7 @@ resources: - cluster_b.yaml - cluster_b_klio_client_auth.yaml - cluster_c_klio_client_auth.yaml + - cluster_e_klio_client_auth.yaml - issuer.yaml - jaeger.yaml - klio_encryption.yaml diff --git a/operator/config/samples/opentelemetry-multi/team-e/cluster_e.yaml b/operator/config/samples/opentelemetry-multi/team-e/cluster_e.yaml new file mode 100644 index 00000000..5c8b8eb2 --- /dev/null +++ b/operator/config/samples/opentelemetry-multi/team-e/cluster_e.yaml @@ -0,0 +1,62 @@ +apiVersion: postgresql.cnpg.io/v1 +kind: Cluster +metadata: + name: cluster-e + namespace: team-e +spec: + instances: 3 + imagePullPolicy: Always + + projectedVolumeTemplate: + sources: + - secret: + name: otel-collector-ca + items: + - key: ca.crt + path: ca.crt + - secret: + name: klio-server-otel-client-tls + items: + - key: tls.crt + path: tls.crt + - key: tls.key + path: tls.key + + postgresql: + pg_hba: + - local replication all peer map=local + + plugins: + - name: klio.cnpg.io + enabled: true + parameters: + pluginConfigurationRef: client-config-cluster-e + + storage: + size: 1Gi +--- +apiVersion: klio.cnpg.io/v1alpha1 +kind: PluginConfiguration +metadata: + name: client-config-cluster-e + namespace: team-e +spec: + serverAddress: klio-a + clientSecretName: cluster-e-klio-user + serverSecretName: klio-a-tls + clusterName: cluster-e + containers: + - name: klio-plugin + env: + - name: OTEL_SERVICE_NAME + value: "klio-plugin" + envFrom: + - configMapRef: + name: cluster-klio-otel-config + - name: klio-restore + env: + - name: OTEL_SERVICE_NAME + value: "klio-restore" + envFrom: + - configMapRef: + name: cluster-klio-otel-config diff --git a/operator/config/samples/opentelemetry-multi/team-e/cluster_klio_otel_config.yaml b/operator/config/samples/opentelemetry-multi/team-e/cluster_klio_otel_config.yaml new file mode 100644 index 00000000..bc232a0e --- /dev/null +++ b/operator/config/samples/opentelemetry-multi/team-e/cluster_klio_otel_config.yaml @@ -0,0 +1,20 @@ +# Same settings as ../cluster_klio_otel_config.yaml, but the endpoint is +# fully-qualified since the otel-collector Service lives in "default", not +# in this namespace. +apiVersion: v1 +kind: ConfigMap +metadata: + name: cluster-klio-otel-config + namespace: team-e +data: + OTEL_RESOURCE_DETECTORS: "telemetry.sdk,host,os.type,process.executable.name" + OTEL_TRACES_EXPORTER: "otlp" + OTEL_METRICS_EXPORTER: "otlp" + OTEL_EXPORTER_OTLP_PROTOCOL: "grpc" + OTEL_EXPORTER_OTLP_ENDPOINT: "https://otel-collector.default.svc.cluster.local:4317" + OTEL_EXPORTER_OTLP_COMPRESSION: "gzip" + OTEL_EXPORTER_OTLP_TIMEOUT: "10000" + OTEL_EXPORTER_OTLP_INSECURE: "false" + OTEL_EXPORTER_OTLP_CERTIFICATE: "/projected/ca.crt" + OTEL_EXPORTER_OTLP_CLIENT_CERTIFICATE: "/projected/tls.crt" + OTEL_EXPORTER_OTLP_CLIENT_KEY: "/projected/tls.key" diff --git a/operator/config/samples/opentelemetry-multi/team-e/issuer.yaml b/operator/config/samples/opentelemetry-multi/team-e/issuer.yaml new file mode 100644 index 00000000..04d57c15 --- /dev/null +++ b/operator/config/samples/opentelemetry-multi/team-e/issuer.yaml @@ -0,0 +1,10 @@ +# Local self-signed issuer for this namespace's OWN Klio server identity +# (cert-manager Issuers are namespace-scoped, so ../issuer.yaml in "default" +# can't be reused here). +apiVersion: cert-manager.io/v1 +kind: Issuer +metadata: + name: selfsigned-issuer + namespace: team-e +spec: + selfSigned: {} diff --git a/operator/config/samples/opentelemetry-multi/team-e/klio_encryption.yaml b/operator/config/samples/opentelemetry-multi/team-e/klio_encryption.yaml new file mode 100644 index 00000000..67d851a3 --- /dev/null +++ b/operator/config/samples/opentelemetry-multi/team-e/klio_encryption.yaml @@ -0,0 +1,18 @@ +# Same content as ../klio_encryption.yaml: reusing the same encryption +# identity across servers is fine for this demo. +apiVersion: v1 +stringData: + secret-key: AGE-SECRET-KEY-1P7TWJ8H837DPVLSRVHAP0ZCZVHXJRTUS7S0A5VZAR49URW0Z9Q0SFW4HVM + public-key: age16y2zu5qptkzateyuarcahs0am75qjpu0xd2ty2dvm5jusnfzxg0scjh5uu + encryption-key: | + -----BEGIN AGE ENCRYPTED FILE----- + YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSAyMzJERW1QTUhjNGdlNmdi + WmFLZ3FLL1lZclduUWEvZkJncGdJZmtVd1FzCmZwSHVTcWNMRFN6a2xpY0NHNVRv + MlltSlNRY1B4bmNoOUxuZzZ5RENSSmsKLS0tIGNYUUNzSGVqNUZhS3VodnZ1cXNP + T3JHS3JRSUNkR1RyeEROYnZYNnNSV00KUd3HyxvvqCyW6hC1sWNDn/xeptHsoz/e + hxCgWoSJvfSfNk174njMwEeW+ic= + -----END AGE ENCRYPTED FILE----- +kind: Secret +metadata: + name: klio-encryption + namespace: team-e diff --git a/operator/config/samples/opentelemetry-multi/team-e/klio_server_a.yaml b/operator/config/samples/opentelemetry-multi/team-e/klio_server_a.yaml new file mode 100644 index 00000000..0e5aff49 --- /dev/null +++ b/operator/config/samples/opentelemetry-multi/team-e/klio_server_a.yaml @@ -0,0 +1,92 @@ +# Reuses the name "klio-a" (see ../klio_server_a.yaml in "default"). Its +# StatefulSet pod is therefore also named "klio-a-klio-0", identical to the +# original klio-a's host_name -- see team-e/README note in ../README.md. +apiVersion: klio.cnpg.io/v1alpha1 +kind: Server +metadata: + name: klio-a + namespace: team-e +spec: + queue: + pvcTemplate: + resources: + requests: + storage: 1Gi + accessModes: + - ReadWriteOnce + tier1: + cache: + pvcTemplate: + resources: + requests: + storage: 1Gi + accessModes: + - ReadWriteOnce + + data: + pvcTemplate: + resources: + requests: + storage: 1Gi + accessModes: + - ReadWriteOnce + + encryptionKeyFile: + fileReference: + volume: + secret: + secretName: klio-encryption + path: encryption-key + + identityFile: + fileReference: + volume: + secret: + secretName: klio-encryption + path: secret-key + + caSecretName: klio-server-ca + tlsSecretName: klio-a-tls + + image: registry.dev:5000/klio-testing:dev + imagePullPolicy: Always + + template: + spec: + containers: + - name: server + env: + - name: NODE_NAME + valueFrom: + fieldRef: + fieldPath: spec.nodeName + - name: POD_UID + valueFrom: + fieldRef: + fieldPath: metadata.uid + - name: OTEL_SERVICE_NAME + value: "klio-server-a-team-e" + - name: OTEL_RESOURCE_ATTRIBUTES + value: "k8s.node.name=$(NODE_NAME),k8s.pod.uid=$(POD_UID)" + envFrom: + - configMapRef: + name: klio-server-otel-config + volumeMounts: + - mountPath: /otel + name: otel + volumes: + - name: otel + projected: + sources: + - secret: + name: otel-collector-ca + items: + - key: ca.crt + path: ca.crt + - secret: + name: klio-server-otel-client-tls + items: + - key: tls.crt + path: tls.crt + - key: tls.key + path: tls.key diff --git a/operator/config/samples/opentelemetry-multi/team-e/klio_server_a_certificate.yaml b/operator/config/samples/opentelemetry-multi/team-e/klio_server_a_certificate.yaml new file mode 100644 index 00000000..8d363c2f --- /dev/null +++ b/operator/config/samples/opentelemetry-multi/team-e/klio_server_a_certificate.yaml @@ -0,0 +1,29 @@ +# A SECOND server named "klio-a" (same name as the one in "default"), in a +# different namespace. Its StatefulSet pod is therefore also named +# "klio-a-klio-0" -- an identical host_name to the original klio-a, testing +# whether $server (built from host_name alone) can disambiguate two +# same-named servers across namespaces. +apiVersion: cert-manager.io/v1 +kind: Certificate +metadata: + name: klio-a + namespace: team-e +spec: + secretName: klio-a-tls + commonName: klio-a + dnsNames: + - klio-a + - klio-a.team-e + - klio-a.team-e.svc + + duration: 2160h # 90d + renewBefore: 360h # 15d + + isCA: false + usages: + - server auth + + issuerRef: + name: selfsigned-issuer + kind: Issuer + group: cert-manager.io diff --git a/operator/config/samples/opentelemetry-multi/team-e/klio_server_otel_config.yaml b/operator/config/samples/opentelemetry-multi/team-e/klio_server_otel_config.yaml new file mode 100644 index 00000000..757acbcb --- /dev/null +++ b/operator/config/samples/opentelemetry-multi/team-e/klio_server_otel_config.yaml @@ -0,0 +1,28 @@ +# Same settings as ../klio_server_otel_config.yaml, but the endpoint is +# fully-qualified since the otel-collector Service lives in "default", not +# in this namespace. +apiVersion: v1 +kind: ConfigMap +metadata: + name: klio-server-otel-config + namespace: team-e +data: + OTEL_RESOURCE_DETECTORS: "telemetry.sdk,host,os.type,process.executable.name" + OTEL_TRACES_EXPORTER: "otlp" + OTEL_EXPORTER_OTLP_TRACES_PROTOCOL: "grpc" + OTEL_EXPORTER_OTLP_TRACES_ENDPOINT: "https://otel-collector.default.svc.cluster.local:4317" + OTEL_EXPORTER_OTLP_TRACES_COMPRESSION: "gzip" + OTEL_EXPORTER_OTLP_TRACES_TIMEOUT: "10000" + OTEL_EXPORTER_OTLP_TRACES_INSECURE: "false" + OTEL_EXPORTER_OTLP_TRACES_CERTIFICATE: "/otel/ca.crt" + OTEL_EXPORTER_OTLP_TRACES_CLIENT_CERTIFICATE: "/otel/tls.crt" + OTEL_EXPORTER_OTLP_TRACES_CLIENT_KEY: "/otel/tls.key" + OTEL_METRICS_EXPORTER: "otlp" + OTEL_METRIC_EXPORT_INTERVAL: "60000" + OTEL_EXPORTER_OTLP_METRICS_PROTOCOL: "grpc" + OTEL_EXPORTER_OTLP_METRICS_ENDPOINT: "https://otel-collector.default.svc.cluster.local:4317" + OTEL_EXPORTER_OTLP_METRICS_TIMEOUT: "60000" + OTEL_EXPORTER_OTLP_METRICS_INSECURE: "false" + OTEL_EXPORTER_OTLP_METRICS_CERTIFICATE: "/otel/ca.crt" + OTEL_EXPORTER_OTLP_METRICS_CLIENT_CERTIFICATE: "/otel/tls.crt" + OTEL_EXPORTER_OTLP_METRICS_CLIENT_KEY: "/otel/tls.key" diff --git a/operator/config/samples/opentelemetry-multi/team-e/kustomization.yaml b/operator/config/samples/opentelemetry-multi/team-e/kustomization.yaml new file mode 100644 index 00000000..5f76a4cd --- /dev/null +++ b/operator/config/samples/opentelemetry-multi/team-e/kustomization.yaml @@ -0,0 +1,15 @@ +# Apply ../kustomization.yaml first, then run: +# ../bootstrap-remote-server.sh team-e cluster-e-klio-user +# to populate the secrets this kustomization expects to already exist +# (klio-server-ca, otel-collector-ca, klio-server-otel-client-tls, +# cluster-e-klio-user), then apply this one. +namespace: team-e +resources: + - namespace.yaml + - issuer.yaml + - klio_server_a_certificate.yaml + - klio_encryption.yaml + - klio_server_otel_config.yaml + - cluster_klio_otel_config.yaml + - klio_server_a.yaml + - cluster_e.yaml diff --git a/operator/config/samples/opentelemetry-multi/team-e/namespace.yaml b/operator/config/samples/opentelemetry-multi/team-e/namespace.yaml new file mode 100644 index 00000000..1e94b068 --- /dev/null +++ b/operator/config/samples/opentelemetry-multi/team-e/namespace.yaml @@ -0,0 +1,4 @@ +apiVersion: v1 +kind: Namespace +metadata: + name: team-e From 1fc4f305e3a3b1dff59602f8c5e109fa8fc2f821 Mon Sep 17 00:00:00 2001 From: Jonathan Battiato Date: Fri, 7 Aug 2026 19:27:27 +0200 Subject: [PATCH 3/5] docs(samples): unify opentelemetry samples into base/single/multi layout Merges operator/config/samples/opentelemetry and opentelemetry-multi into one tree (base/single/multi) so the shared OTel/Jaeger/CA infrastructure is defined once instead of duplicated byte-for-byte across both directories, which had already drifted (e.g. mismatched otel-collector-contrib versions). multi's per-instance servers and clusters now reuse single's Server/Cluster definitions via kustomize resource+patch composition instead of duplicating their spec, and team-e is renamed to team-d for consistency with the other team-* namespaces. The klio-server-ca Issuer becomes a ClusterIssuer so cluster-c's and cluster-d's client-auth certificates can be requested directly in their own namespace instead of being generated in "default" and copied over by script; the two remaining scripts now only copy secrets that are genuinely pinned by exact bytes rather than CA-validated. Also fixes two related sample bugs: a missing cluster-d Backup in multi/backups-example.yaml, and the complete absence of a backup example for the "single" scenario. Co-Authored-By: Claude Sonnet 5 Signed-off-by: Jonathan Battiato --- .../web/docs/user/grafana-dashboards.md | 4 +- observability/grafana/README.md | 2 +- .../samples/opentelemetry-multi/README.md | 118 -------------- .../opentelemetry-multi/cluster_a.yaml | 60 -------- .../opentelemetry-multi/cluster_b.yaml | 60 -------- .../cluster_c_klio_client_auth.yaml | 23 --- .../cluster_e_klio_client_auth.yaml | 26 ---- .../opentelemetry-multi/klio_encryption.yaml | 18 --- .../opentelemetry-multi/klio_server_a.yaml | 93 ----------- .../klio_server_a_certificate.yaml | 23 --- .../opentelemetry-multi/klio_server_b.yaml | 93 ----------- .../klio_server_b_certificate.yaml | 28 ---- .../opentelemetry-multi/klio_server_ca.yaml | 23 --- .../klio_server_ca_issuer.yaml | 7 - .../opentelemetry-multi/kustomization.yaml | 34 ----- .../opentelemetry-multi/otel_collector.yaml | 83 ---------- .../opentelemetry-multi/team-c/cluster_c.yaml | 68 --------- .../team-c/kustomization.yaml | 9 -- .../opentelemetry-multi/team-e/cluster_e.yaml | 62 -------- .../team-e/klio_server_a.yaml | 92 ----------- .../team-e/klio_server_a_certificate.yaml | 29 ---- .../team-e/kustomization.yaml | 15 -- .../config/samples/opentelemetry/README.md | 144 +++++++++++++++++- .../base}/cluster_klio_otel_config.yaml | 0 .../base}/issuer.yaml | 0 .../base}/jaeger.yaml | 0 .../{ => base}/klio_encryption.yaml | 3 + .../opentelemetry/base/klio_server_ca.yaml | 31 ++++ .../base/klio_server_ca_issuer.yaml | 16 ++ .../klio_server_otel_client_certificate.yaml | 0 .../base}/klio_server_otel_config.yaml | 0 .../{ => base}/kustomization.yaml | 12 +- .../{ => base}/otel_collector.yaml | 0 .../base}/otel_collector_certificate.yaml | 0 .../base}/otel_collector_clusterrole.yaml | 0 .../otel_collector_clusterrolebinding.yaml | 0 .../base}/otel_collector_sa.yaml | 0 .../base}/otel_collector_svc_monitor.yaml | 0 .../cluster_klio_otel_config.yaml | 21 --- .../config/samples/opentelemetry/issuer.yaml | 7 - .../config/samples/opentelemetry/jaeger.yaml | 71 --------- .../samples/opentelemetry/klio_server_ca.yaml | 19 --- .../opentelemetry/klio_server_ca_issuer.yaml | 7 - .../klio_server_otel_client_certificate.yaml | 23 --- .../klio_server_otel_config.yaml | 28 ---- .../multi}/backups-example.yaml | 23 ++- .../multi}/bootstrap-remote-server.sh | 32 ++-- .../multi}/cluster_a_klio_client_auth.yaml | 2 +- .../multi}/cluster_b_klio_client_auth.yaml | 2 +- .../multi}/copy-cross-namespace-secrets.sh | 23 ++- .../opentelemetry/multi/kustomization.yaml | 16 ++ .../multi/team-a/kustomization.yaml | 68 +++++++++ .../multi/team-b/kustomization.yaml | 75 +++++++++ .../team-c/cluster_c_klio_client_auth.yaml | 25 +++ .../team-c/cluster_klio_otel_config.yaml | 0 .../multi/team-c/kustomization.yaml | 57 +++++++ .../multi}/team-c/namespace.yaml | 0 .../team-d/cluster_d_klio_client_auth.yaml | 28 ++++ .../team-d}/cluster_klio_otel_config.yaml | 2 +- .../multi/team-d}/issuer.yaml | 2 +- .../multi/team-d}/klio_encryption.yaml | 2 +- .../team-d}/klio_server_otel_config.yaml | 2 +- .../multi/team-d/kustomization.yaml | 91 +++++++++++ .../multi/team-d}/namespace.yaml | 2 +- .../otel_collector_certificate.yaml | 13 -- .../otel_collector_clusterrole.yaml | 33 ---- .../otel_collector_clusterrolebinding.yaml | 12 -- .../opentelemetry/otel_collector_sa.yaml | 4 - .../otel_collector_svc_monitor.yaml | 18 --- .../opentelemetry/single/backups-example.yaml | 16 ++ .../{ => single/cluster}/cluster.yaml | 0 .../single/cluster/kustomization.yaml | 6 + .../cluster}/plugin_configuration.yaml | 0 .../cluster_klio_client_auth.yaml | 2 +- .../opentelemetry/single/kustomization.yaml | 9 ++ .../single/server/kustomization.yaml | 6 + .../server/server.yaml} | 0 .../server/server_certificate.yaml} | 0 78 files changed, 645 insertions(+), 1278 deletions(-) delete mode 100644 operator/config/samples/opentelemetry-multi/README.md delete mode 100644 operator/config/samples/opentelemetry-multi/cluster_a.yaml delete mode 100644 operator/config/samples/opentelemetry-multi/cluster_b.yaml delete mode 100644 operator/config/samples/opentelemetry-multi/cluster_c_klio_client_auth.yaml delete mode 100644 operator/config/samples/opentelemetry-multi/cluster_e_klio_client_auth.yaml delete mode 100644 operator/config/samples/opentelemetry-multi/klio_encryption.yaml delete mode 100644 operator/config/samples/opentelemetry-multi/klio_server_a.yaml delete mode 100644 operator/config/samples/opentelemetry-multi/klio_server_a_certificate.yaml delete mode 100644 operator/config/samples/opentelemetry-multi/klio_server_b.yaml delete mode 100644 operator/config/samples/opentelemetry-multi/klio_server_b_certificate.yaml delete mode 100644 operator/config/samples/opentelemetry-multi/klio_server_ca.yaml delete mode 100644 operator/config/samples/opentelemetry-multi/klio_server_ca_issuer.yaml delete mode 100644 operator/config/samples/opentelemetry-multi/kustomization.yaml delete mode 100644 operator/config/samples/opentelemetry-multi/otel_collector.yaml delete mode 100644 operator/config/samples/opentelemetry-multi/team-c/cluster_c.yaml delete mode 100644 operator/config/samples/opentelemetry-multi/team-c/kustomization.yaml delete mode 100644 operator/config/samples/opentelemetry-multi/team-e/cluster_e.yaml delete mode 100644 operator/config/samples/opentelemetry-multi/team-e/klio_server_a.yaml delete mode 100644 operator/config/samples/opentelemetry-multi/team-e/klio_server_a_certificate.yaml delete mode 100644 operator/config/samples/opentelemetry-multi/team-e/kustomization.yaml rename operator/config/samples/{opentelemetry-multi => opentelemetry/base}/cluster_klio_otel_config.yaml (100%) rename operator/config/samples/{opentelemetry-multi => opentelemetry/base}/issuer.yaml (100%) rename operator/config/samples/{opentelemetry-multi => opentelemetry/base}/jaeger.yaml (100%) rename operator/config/samples/opentelemetry/{ => base}/klio_encryption.yaml (76%) create mode 100644 operator/config/samples/opentelemetry/base/klio_server_ca.yaml create mode 100644 operator/config/samples/opentelemetry/base/klio_server_ca_issuer.yaml rename operator/config/samples/{opentelemetry-multi => opentelemetry/base}/klio_server_otel_client_certificate.yaml (100%) rename operator/config/samples/{opentelemetry-multi => opentelemetry/base}/klio_server_otel_config.yaml (100%) rename operator/config/samples/opentelemetry/{ => base}/kustomization.yaml (62%) rename operator/config/samples/opentelemetry/{ => base}/otel_collector.yaml (100%) rename operator/config/samples/{opentelemetry-multi => opentelemetry/base}/otel_collector_certificate.yaml (100%) rename operator/config/samples/{opentelemetry-multi => opentelemetry/base}/otel_collector_clusterrole.yaml (100%) rename operator/config/samples/{opentelemetry-multi => opentelemetry/base}/otel_collector_clusterrolebinding.yaml (100%) rename operator/config/samples/{opentelemetry-multi => opentelemetry/base}/otel_collector_sa.yaml (100%) rename operator/config/samples/{opentelemetry-multi => opentelemetry/base}/otel_collector_svc_monitor.yaml (100%) delete mode 100644 operator/config/samples/opentelemetry/cluster_klio_otel_config.yaml delete mode 100644 operator/config/samples/opentelemetry/issuer.yaml delete mode 100644 operator/config/samples/opentelemetry/jaeger.yaml delete mode 100644 operator/config/samples/opentelemetry/klio_server_ca.yaml delete mode 100644 operator/config/samples/opentelemetry/klio_server_ca_issuer.yaml delete mode 100644 operator/config/samples/opentelemetry/klio_server_otel_client_certificate.yaml delete mode 100644 operator/config/samples/opentelemetry/klio_server_otel_config.yaml rename operator/config/samples/{opentelemetry-multi => opentelemetry/multi}/backups-example.yaml (61%) rename operator/config/samples/{opentelemetry-multi => opentelemetry/multi}/bootstrap-remote-server.sh (62%) rename operator/config/samples/{opentelemetry-multi => opentelemetry/multi}/cluster_a_klio_client_auth.yaml (92%) rename operator/config/samples/{opentelemetry-multi => opentelemetry/multi}/cluster_b_klio_client_auth.yaml (92%) rename operator/config/samples/{opentelemetry-multi => opentelemetry/multi}/copy-cross-namespace-secrets.sh (63%) create mode 100644 operator/config/samples/opentelemetry/multi/kustomization.yaml create mode 100644 operator/config/samples/opentelemetry/multi/team-a/kustomization.yaml create mode 100644 operator/config/samples/opentelemetry/multi/team-b/kustomization.yaml create mode 100644 operator/config/samples/opentelemetry/multi/team-c/cluster_c_klio_client_auth.yaml rename operator/config/samples/{opentelemetry-multi => opentelemetry/multi}/team-c/cluster_klio_otel_config.yaml (100%) create mode 100644 operator/config/samples/opentelemetry/multi/team-c/kustomization.yaml rename operator/config/samples/{opentelemetry-multi => opentelemetry/multi}/team-c/namespace.yaml (100%) create mode 100644 operator/config/samples/opentelemetry/multi/team-d/cluster_d_klio_client_auth.yaml rename operator/config/samples/{opentelemetry-multi/team-e => opentelemetry/multi/team-d}/cluster_klio_otel_config.yaml (97%) rename operator/config/samples/{opentelemetry-multi/team-e => opentelemetry/multi/team-d}/issuer.yaml (93%) rename operator/config/samples/{opentelemetry-multi/team-e => opentelemetry/multi/team-d}/klio_encryption.yaml (97%) rename operator/config/samples/{opentelemetry-multi/team-e => opentelemetry/multi/team-d}/klio_server_otel_config.yaml (98%) create mode 100644 operator/config/samples/opentelemetry/multi/team-d/kustomization.yaml rename operator/config/samples/{opentelemetry-multi/team-e => opentelemetry/multi/team-d}/namespace.yaml (73%) delete mode 100644 operator/config/samples/opentelemetry/otel_collector_certificate.yaml delete mode 100644 operator/config/samples/opentelemetry/otel_collector_clusterrole.yaml delete mode 100644 operator/config/samples/opentelemetry/otel_collector_clusterrolebinding.yaml delete mode 100644 operator/config/samples/opentelemetry/otel_collector_sa.yaml delete mode 100644 operator/config/samples/opentelemetry/otel_collector_svc_monitor.yaml create mode 100644 operator/config/samples/opentelemetry/single/backups-example.yaml rename operator/config/samples/opentelemetry/{ => single/cluster}/cluster.yaml (100%) create mode 100644 operator/config/samples/opentelemetry/single/cluster/kustomization.yaml rename operator/config/samples/opentelemetry/{ => single/cluster}/plugin_configuration.yaml (100%) rename operator/config/samples/opentelemetry/{ => single}/cluster_klio_client_auth.yaml (93%) create mode 100644 operator/config/samples/opentelemetry/single/kustomization.yaml create mode 100644 operator/config/samples/opentelemetry/single/server/kustomization.yaml rename operator/config/samples/opentelemetry/{klio_server.yaml => single/server/server.yaml} (100%) rename operator/config/samples/opentelemetry/{klio_server_certificate.yaml => single/server/server_certificate.yaml} (100%) diff --git a/documentation/web/docs/user/grafana-dashboards.md b/documentation/web/docs/user/grafana-dashboards.md index 4a02de38..c7536216 100644 --- a/documentation/web/docs/user/grafana-dashboards.md +++ b/documentation/web/docs/user/grafana-dashboards.md @@ -100,7 +100,7 @@ When you route metrics through an OpenTelemetry Collector, enable `resource_to_telemetry_conversion` on the Prometheus exporter so that resource attributes such as the pod and namespace become Prometheus labels. The sample collector under -`operator/config/samples/opentelemetry/otel_collector.yaml` already does +`operator/config/samples/opentelemetry/base/otel_collector.yaml` already does this. ::: @@ -138,7 +138,7 @@ helm upgrade --install \ Ensure Prometheus scrapes Klio's metrics by deploying a `ServiceMonitor` (or a `PodMonitor`, if the collector's `Service` has no labels) for the OpenTelemetry collector's Prometheus exporter — see -`operator/config/samples/opentelemetry/otel_collector_svc_monitor.yaml`. +`operator/config/samples/opentelemetry/base/otel_collector_svc_monitor.yaml`. Port-forward Grafana and log in with `admin` / `prom-operator`: diff --git a/observability/grafana/README.md b/observability/grafana/README.md index c7a1efab..6263232c 100644 --- a/observability/grafana/README.md +++ b/observability/grafana/README.md @@ -91,7 +91,7 @@ helm upgrade --install \ Make Prometheus scrape Klio's metrics by deploying a `ServiceMonitor` — or a `PodMonitor` if the collector's `Service` has no labels — for the OTel collector (see -[`operator/config/samples/opentelemetry/otel_collector_svc_monitor.yaml`](../../operator/config/samples/opentelemetry/otel_collector_svc_monitor.yaml) +[`operator/config/samples/opentelemetry/base/otel_collector_svc_monitor.yaml`](../../operator/config/samples/opentelemetry/base/otel_collector_svc_monitor.yaml) and the [OpenTelemetry guide](../../documentation/web/docs/user/opentelemetry.md)). Port-forward Grafana (log in with `admin` / `prom-operator`): diff --git a/operator/config/samples/opentelemetry-multi/README.md b/operator/config/samples/opentelemetry-multi/README.md deleted file mode 100644 index 886da605..00000000 --- a/operator/config/samples/opentelemetry-multi/README.md +++ /dev/null @@ -1,118 +0,0 @@ -# README - -This directory contains a sample environment with **two Klio servers** and -**three CloudNativePG clusters** distributed across them, one of which lives -in a different namespace than the servers. It exercises the same -OpenTelemetry + Prometheus + Grafana stack as -`operator/config/samples/opentelemetry/`, but is meant to validate the -Grafana dashboard's `$namespace`, `$server` and `$cluster` template -variables and its per-tier/per-cluster aggregations against more than one -server or cluster, something the single-server sample can't exercise. - -## Topology - -| Cluster | Namespace | Backed by | -|-------------|-----------|-----------| -| cluster-a | default | klio-a | -| cluster-b | default | klio-b | -| cluster-c | team-c | klio-b | - -`klio-b` intentionally backs clusters in two different namespaces, since a -single shared backup server serving multiple application namespaces is a -realistic multi-tenant deployment and the case most likely to expose -dashboard attribution bugs. - -## Prerequisites - -A running Kubernetes cluster with the following operators installed: - -- CloudNativePG -- Klio -- cert-manager -- OpenTelemetry -- Prometheus - -`jq` must also be available locally (used by `copy-cross-namespace-secrets.sh`). - -## Deploying a Kubernetes cluster with the required operators - -Assuming an environment with CloudNativePG, Klio and cert-manager -created through the CloudNativePG `hack/setup-cluster.sh` script and -the klio task - -```shell -KIND_CLUSTER_NAME=$(kind get clusters | grep pg-operator-e2e) task integration:deploy-to-kind -``` - -you can install the OpenTelemetry operator by running: - -```shell -kubectl apply -f https://github.com/open-telemetry/opentelemetry-operator/releases/latest/download/opentelemetry-operator.yaml -``` - -You can install Prometheus using the Prometheus community Helm chart and -the CloudNativePG example configuration: - -```shell -helm repo add prometheus-community https://prometheus-community.github.io/helm-charts -``` - -```shell -helm upgrade --install \ - -f https://raw.githubusercontent.com/cloudnative-pg/cloudnative-pg/main/docs/src/samples/monitoring/kube-stack-config.yaml \ - prometheus-community prometheus-community/kube-prometheus-stack -``` - -## Deploying the sample configuration - -1. Deploy the two servers, the two same-namespace clusters (cluster-a, - cluster-b) and the shared OTel/Jaeger stack, all in `default`: - - ```shell - kubectl apply -k operator/config/samples/opentelemetry-multi - ``` - - Wait for `klio-a`, `klio-b`, `cluster-a` and `cluster-b` to become ready - before continuing. - -1. cert-manager `Issuer`s are namespace-scoped, so the client certificate - cluster-c needs (to authenticate to klio-b) can only be generated in - `default`, where the `klio-server-ca` Issuer lives. Copy that - certificate, plus the (pinned, not CA-validated — see the script's - comments) server and OTel collector certificates, into the `team-c` - namespace: - - ```shell - ./operator/config/samples/opentelemetry-multi/copy-cross-namespace-secrets.sh - ``` - -1. Deploy cluster-c into `team-c`: - - ```shell - kubectl apply -k operator/config/samples/opentelemetry-multi/team-c - ``` - -## Validating the Grafana dashboard - -Trigger one base backup per cluster (needed before any backup/snapshot panel -has data to show): - -```shell -kubectl apply -f operator/config/samples/opentelemetry-multi/backups-example.yaml -``` - -Once all three backups complete and Prometheus has scraped a -metrics-collection cycle, open the Klio Grafana dashboard and confirm: - -- The `$namespace` variable offers both `default` and `team-c`. -- The `$server` variable offers both `klio-a-klio-0` and `klio-b-klio-0` - (the value is each server's pod hostname). -- The `$cluster` variable offers `cluster-a`, `cluster-b` and `cluster-c`, - and narrows correctly when `$namespace`/`$server` are filtered (e.g. - selecting `$namespace=team-c` should only ever offer `cluster-c`). -- Per-cluster and per-server panels correctly attribute data instead of - aggregating everything together. - -Any panel that fails to distinguish between clusters/servers/namespaces -here is a dashboard bug to file separately; this sample's job is only to -make that determination possible. diff --git a/operator/config/samples/opentelemetry-multi/cluster_a.yaml b/operator/config/samples/opentelemetry-multi/cluster_a.yaml deleted file mode 100644 index 56738186..00000000 --- a/operator/config/samples/opentelemetry-multi/cluster_a.yaml +++ /dev/null @@ -1,60 +0,0 @@ -apiVersion: postgresql.cnpg.io/v1 -kind: Cluster -metadata: - name: cluster-a -spec: - instances: 3 - imagePullPolicy: Always - - projectedVolumeTemplate: - sources: - - secret: - name: otel-collector-tls - items: - - key: ca.crt - path: ca.crt - - secret: - name: klio-server-otel-client-tls - items: - - key: tls.crt - path: tls.crt - - key: tls.key - path: tls.key - - postgresql: - pg_hba: - - local replication all peer map=local - - plugins: - - name: klio.cnpg.io - enabled: true - parameters: - pluginConfigurationRef: client-config-cluster-a - - storage: - size: 1Gi ---- -apiVersion: klio.cnpg.io/v1alpha1 -kind: PluginConfiguration -metadata: - name: client-config-cluster-a -spec: - serverAddress: klio-a.default - clientSecretName: cluster-a-klio-user - serverSecretName: klio-a-tls - clusterName: cluster-a - containers: - - name: klio-plugin - env: - - name: OTEL_SERVICE_NAME - value: "klio-plugin" - envFrom: - - configMapRef: - name: cluster-klio-otel-config - - name: klio-restore - env: - - name: OTEL_SERVICE_NAME - value: "klio-restore" - envFrom: - - configMapRef: - name: cluster-klio-otel-config diff --git a/operator/config/samples/opentelemetry-multi/cluster_b.yaml b/operator/config/samples/opentelemetry-multi/cluster_b.yaml deleted file mode 100644 index e75dd557..00000000 --- a/operator/config/samples/opentelemetry-multi/cluster_b.yaml +++ /dev/null @@ -1,60 +0,0 @@ -apiVersion: postgresql.cnpg.io/v1 -kind: Cluster -metadata: - name: cluster-b -spec: - instances: 3 - imagePullPolicy: Always - - projectedVolumeTemplate: - sources: - - secret: - name: otel-collector-tls - items: - - key: ca.crt - path: ca.crt - - secret: - name: klio-server-otel-client-tls - items: - - key: tls.crt - path: tls.crt - - key: tls.key - path: tls.key - - postgresql: - pg_hba: - - local replication all peer map=local - - plugins: - - name: klio.cnpg.io - enabled: true - parameters: - pluginConfigurationRef: client-config-cluster-b - - storage: - size: 1Gi ---- -apiVersion: klio.cnpg.io/v1alpha1 -kind: PluginConfiguration -metadata: - name: client-config-cluster-b -spec: - serverAddress: klio-b.default - clientSecretName: cluster-b-klio-user - serverSecretName: klio-b-tls - clusterName: cluster-b - containers: - - name: klio-plugin - env: - - name: OTEL_SERVICE_NAME - value: "klio-plugin" - envFrom: - - configMapRef: - name: cluster-klio-otel-config - - name: klio-restore - env: - - name: OTEL_SERVICE_NAME - value: "klio-restore" - envFrom: - - configMapRef: - name: cluster-klio-otel-config diff --git a/operator/config/samples/opentelemetry-multi/cluster_c_klio_client_auth.yaml b/operator/config/samples/opentelemetry-multi/cluster_c_klio_client_auth.yaml deleted file mode 100644 index d47440be..00000000 --- a/operator/config/samples/opentelemetry-multi/cluster_c_klio_client_auth.yaml +++ /dev/null @@ -1,23 +0,0 @@ -# cluster-c lives in the "team-c" namespace (see ../team-c), but the -# klio-server-ca Issuer is namespace-scoped and only exists here in -# "default". This Certificate is therefore generated here and its resulting -# secret is copied into "team-c" by ../copy-cross-namespace-secrets.sh. -apiVersion: cert-manager.io/v1 -kind: Certificate -metadata: - name: cluster-c-klio-user -spec: - commonName: klio@cluster-c - secretName: cluster-c-klio-user - - duration: 2160h # 90d - renewBefore: 360h # 15d - - isCA: false - usages: - - client auth - - issuerRef: - name: klio-server-ca - kind: Issuer - group: cert-manager.io diff --git a/operator/config/samples/opentelemetry-multi/cluster_e_klio_client_auth.yaml b/operator/config/samples/opentelemetry-multi/cluster_e_klio_client_auth.yaml deleted file mode 100644 index 441c0444..00000000 --- a/operator/config/samples/opentelemetry-multi/cluster_e_klio_client_auth.yaml +++ /dev/null @@ -1,26 +0,0 @@ -# Client cert for cluster-e, backed by a SECOND Klio server also named -# "klio-a" (see ../team-e), living in a different namespace than the -# original klio-a in "default". Since a Server's StatefulSet pod name -# equals the Server's own name, this second klio-a's pod is ALSO named -# "klio-a-klio-0" -- an identical host_name to the original, but in a -# different namespace. Generated here because the klio-server-ca Issuer is -# namespace-scoped; copied into team-e by bootstrap-remote-server.sh. -apiVersion: cert-manager.io/v1 -kind: Certificate -metadata: - name: cluster-e-klio-user -spec: - commonName: klio@cluster-e - secretName: cluster-e-klio-user - - duration: 2160h # 90d - renewBefore: 360h # 15d - - isCA: false - usages: - - client auth - - issuerRef: - name: klio-server-ca - kind: Issuer - group: cert-manager.io diff --git a/operator/config/samples/opentelemetry-multi/klio_encryption.yaml b/operator/config/samples/opentelemetry-multi/klio_encryption.yaml deleted file mode 100644 index 02f050f5..00000000 --- a/operator/config/samples/opentelemetry-multi/klio_encryption.yaml +++ /dev/null @@ -1,18 +0,0 @@ -# Shared encryption identity for both klio-a and klio-b. Reusing the same -# identity across servers is fine for this demo; production deployments -# would normally give each server its own. -apiVersion: v1 -stringData: - secret-key: AGE-SECRET-KEY-1P7TWJ8H837DPVLSRVHAP0ZCZVHXJRTUS7S0A5VZAR49URW0Z9Q0SFW4HVM - public-key: age16y2zu5qptkzateyuarcahs0am75qjpu0xd2ty2dvm5jusnfzxg0scjh5uu - encryption-key: | - -----BEGIN AGE ENCRYPTED FILE----- - YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSAyMzJERW1QTUhjNGdlNmdi - WmFLZ3FLL1lZclduUWEvZkJncGdJZmtVd1FzCmZwSHVTcWNMRFN6a2xpY0NHNVRv - MlltSlNRY1B4bmNoOUxuZzZ5RENSSmsKLS0tIGNYUUNzSGVqNUZhS3VodnZ1cXNP - T3JHS3JRSUNkR1RyeEROYnZYNnNSV00KUd3HyxvvqCyW6hC1sWNDn/xeptHsoz/e - hxCgWoSJvfSfNk174njMwEeW+ic= - -----END AGE ENCRYPTED FILE----- -kind: Secret -metadata: - name: klio-encryption diff --git a/operator/config/samples/opentelemetry-multi/klio_server_a.yaml b/operator/config/samples/opentelemetry-multi/klio_server_a.yaml deleted file mode 100644 index ff71ec2e..00000000 --- a/operator/config/samples/opentelemetry-multi/klio_server_a.yaml +++ /dev/null @@ -1,93 +0,0 @@ -apiVersion: klio.cnpg.io/v1alpha1 -kind: Server -metadata: - name: klio-a -spec: - queue: - pvcTemplate: - resources: - requests: - storage: 1Gi - accessModes: - - ReadWriteOnce - tier1: - cache: - pvcTemplate: - resources: - requests: - storage: 1Gi - accessModes: - - ReadWriteOnce - - data: - pvcTemplate: - resources: - requests: - storage: 1Gi - accessModes: - - ReadWriteOnce - - encryptionKeyFile: - fileReference: - volume: - secret: - secretName: klio-encryption - path: encryption-key - - identityFile: - fileReference: - volume: - secret: - secretName: klio-encryption - path: secret-key - - caSecretName: klio-server-ca - tlsSecretName: klio-a-tls - - image: registry.dev:5000/klio-testing:dev - imagePullPolicy: Always - - template: - spec: - containers: - - name: server - env: - # Use the downward API to get pod and node information - # These environment variables will be used in the OTEL_RESOURCE_ATTRIBUTES - # to provide more context about the telemetry data for the otel collector k8sattributes processor - - name: NODE_NAME - valueFrom: - fieldRef: - fieldPath: spec.nodeName - - name: POD_UID - valueFrom: - fieldRef: - fieldPath: metadata.uid - # OTEL_SERVICE_NAME and OTEL_RESOURCE_ATTRIBUTES are set specifically for each container - - name: OTEL_SERVICE_NAME - value: "klio-server-a" - - name: OTEL_RESOURCE_ATTRIBUTES - value: "k8s.node.name=$(NODE_NAME),k8s.pod.uid=$(POD_UID)" - envFrom: - # Use a common config map for all shared OTEL settings - - configMapRef: - name: klio-server-otel-config - volumeMounts: - - mountPath: /otel - name: otel - volumes: - - name: otel - projected: - sources: - - secret: - name: otel-collector-tls - items: - - key: ca.crt - path: ca.crt - - secret: - name: klio-server-otel-client-tls - items: - - key: tls.crt - path: tls.crt - - key: tls.key - path: tls.key diff --git a/operator/config/samples/opentelemetry-multi/klio_server_a_certificate.yaml b/operator/config/samples/opentelemetry-multi/klio_server_a_certificate.yaml deleted file mode 100644 index 2f6debab..00000000 --- a/operator/config/samples/opentelemetry-multi/klio_server_a_certificate.yaml +++ /dev/null @@ -1,23 +0,0 @@ -apiVersion: cert-manager.io/v1 -kind: Certificate -metadata: - name: klio-a -spec: - secretName: klio-a-tls - commonName: klio-a - dnsNames: - - klio-a - - klio-a.default - - klio-a.default.svc - - duration: 2160h # 90d - renewBefore: 360h # 15d - - isCA: false - usages: - - server auth - - issuerRef: - name: selfsigned-issuer - kind: Issuer - group: cert-manager.io diff --git a/operator/config/samples/opentelemetry-multi/klio_server_b.yaml b/operator/config/samples/opentelemetry-multi/klio_server_b.yaml deleted file mode 100644 index 3b9ac8bf..00000000 --- a/operator/config/samples/opentelemetry-multi/klio_server_b.yaml +++ /dev/null @@ -1,93 +0,0 @@ -apiVersion: klio.cnpg.io/v1alpha1 -kind: Server -metadata: - name: klio-b -spec: - queue: - pvcTemplate: - resources: - requests: - storage: 1Gi - accessModes: - - ReadWriteOnce - tier1: - cache: - pvcTemplate: - resources: - requests: - storage: 1Gi - accessModes: - - ReadWriteOnce - - data: - pvcTemplate: - resources: - requests: - storage: 1Gi - accessModes: - - ReadWriteOnce - - encryptionKeyFile: - fileReference: - volume: - secret: - secretName: klio-encryption - path: encryption-key - - identityFile: - fileReference: - volume: - secret: - secretName: klio-encryption - path: secret-key - - caSecretName: klio-server-ca - tlsSecretName: klio-b-tls - - image: registry.dev:5000/klio-testing:dev - imagePullPolicy: Always - - template: - spec: - containers: - - name: server - env: - # Use the downward API to get pod and node information - # These environment variables will be used in the OTEL_RESOURCE_ATTRIBUTES - # to provide more context about the telemetry data for the otel collector k8sattributes processor - - name: NODE_NAME - valueFrom: - fieldRef: - fieldPath: spec.nodeName - - name: POD_UID - valueFrom: - fieldRef: - fieldPath: metadata.uid - # OTEL_SERVICE_NAME and OTEL_RESOURCE_ATTRIBUTES are set specifically for each container - - name: OTEL_SERVICE_NAME - value: "klio-server-b" - - name: OTEL_RESOURCE_ATTRIBUTES - value: "k8s.node.name=$(NODE_NAME),k8s.pod.uid=$(POD_UID)" - envFrom: - # Use a common config map for all shared OTEL settings - - configMapRef: - name: klio-server-otel-config - volumeMounts: - - mountPath: /otel - name: otel - volumes: - - name: otel - projected: - sources: - - secret: - name: otel-collector-tls - items: - - key: ca.crt - path: ca.crt - - secret: - name: klio-server-otel-client-tls - items: - - key: tls.crt - path: tls.crt - - key: tls.key - path: tls.key diff --git a/operator/config/samples/opentelemetry-multi/klio_server_b_certificate.yaml b/operator/config/samples/opentelemetry-multi/klio_server_b_certificate.yaml deleted file mode 100644 index 18c67786..00000000 --- a/operator/config/samples/opentelemetry-multi/klio_server_b_certificate.yaml +++ /dev/null @@ -1,28 +0,0 @@ -# klio-b is reached both from cluster-b (same namespace, "default") and from -# cluster-c (namespace "team-c", see ../team-c), so its dnsNames must cover -# both the short in-namespace form and the fully-qualified cross-namespace -# form used as PluginConfiguration.serverAddress by cluster-c. -apiVersion: cert-manager.io/v1 -kind: Certificate -metadata: - name: klio-b -spec: - secretName: klio-b-tls - commonName: klio-b - dnsNames: - - klio-b - - klio-b.default - - klio-b.default.svc - - klio-b.default.svc.cluster.local - - duration: 2160h # 90d - renewBefore: 360h # 15d - - isCA: false - usages: - - server auth - - issuerRef: - name: selfsigned-issuer - kind: Issuer - group: cert-manager.io diff --git a/operator/config/samples/opentelemetry-multi/klio_server_ca.yaml b/operator/config/samples/opentelemetry-multi/klio_server_ca.yaml deleted file mode 100644 index 07d2a8d3..00000000 --- a/operator/config/samples/opentelemetry-multi/klio_server_ca.yaml +++ /dev/null @@ -1,23 +0,0 @@ -# Root CA shared by both Klio servers (klio-a, klio-b) in this sample, and by -# the cross-namespace client certificate generated for cluster-c (see -# ../copy-cross-namespace-secrets.sh). Sharing a single CA lets any client -# certificate it signs be trusted by either server's caSecretName. -apiVersion: cert-manager.io/v1 -kind: Certificate -metadata: - name: klio-server-ca -spec: - commonName: klio-server - secretName: klio-server-ca - - duration: 2160h # 90d - renewBefore: 360h # 15d - - isCA: true - usages: - - cert sign - - issuerRef: - name: selfsigned-issuer - kind: Issuer - group: cert-manager.io diff --git a/operator/config/samples/opentelemetry-multi/klio_server_ca_issuer.yaml b/operator/config/samples/opentelemetry-multi/klio_server_ca_issuer.yaml deleted file mode 100644 index 0c06112f..00000000 --- a/operator/config/samples/opentelemetry-multi/klio_server_ca_issuer.yaml +++ /dev/null @@ -1,7 +0,0 @@ -apiVersion: cert-manager.io/v1 -kind: Issuer -metadata: - name: klio-server-ca -spec: - ca: - secretName: klio-server-ca diff --git a/operator/config/samples/opentelemetry-multi/kustomization.yaml b/operator/config/samples/opentelemetry-multi/kustomization.yaml deleted file mode 100644 index bce79d75..00000000 --- a/operator/config/samples/opentelemetry-multi/kustomization.yaml +++ /dev/null @@ -1,34 +0,0 @@ -# REQUIRES: -# kubectl apply -f https://github.com/open-telemetry/opentelemetry-operator/releases/latest/download/opentelemetry-operator.yaml -# -# Deploys two Klio servers (klio-a, klio-b) and two CNPG clusters -# (cluster-a -> klio-a, cluster-b -> klio-b) into the "default" namespace, -# sharing one OTel collector / Jaeger / Prometheus stack. See: -# ../team-c for a third cluster in a different namespace, backed by klio-b -# ../team-e for a second, independent server also named "klio-a" (so its -# StatefulSet pod, and therefore host_name, collides with the original) -resources: - - cluster_klio_otel_config.yaml - - cluster_a.yaml - - cluster_a_klio_client_auth.yaml - - cluster_b.yaml - - cluster_b_klio_client_auth.yaml - - cluster_c_klio_client_auth.yaml - - cluster_e_klio_client_auth.yaml - - issuer.yaml - - jaeger.yaml - - klio_encryption.yaml - - klio_server_a.yaml - - klio_server_a_certificate.yaml - - klio_server_b.yaml - - klio_server_b_certificate.yaml - - klio_server_ca.yaml - - klio_server_ca_issuer.yaml - - klio_server_otel_client_certificate.yaml - - klio_server_otel_config.yaml - - otel_collector.yaml - - otel_collector_certificate.yaml - - otel_collector_clusterrole.yaml - - otel_collector_clusterrolebinding.yaml - - otel_collector_sa.yaml - - otel_collector_svc_monitor.yaml diff --git a/operator/config/samples/opentelemetry-multi/otel_collector.yaml b/operator/config/samples/opentelemetry-multi/otel_collector.yaml deleted file mode 100644 index 852b9e94..00000000 --- a/operator/config/samples/opentelemetry-multi/otel_collector.yaml +++ /dev/null @@ -1,83 +0,0 @@ -# OpenTelemetry Collector instance -# A full-fledged configuration is outside of the scope of this example. -# This example receives OTLP data over gRPC with TLS, adds Kubernetes attributes, -# and exports to a Jaeger instance (without TLS, for simplicity) and to Prometheus. -# It also exports to a debug exporter, which prints the data to the collector's logs. -# OpenTelemetry Operator is required to run this: -# kubectl apply -f https://github.com/open-telemetry/opentelemetry-operator/releases/latest/download/opentelemetry-operator.yaml -apiVersion: opentelemetry.io/v1beta1 -kind: OpenTelemetryCollector -metadata: - name: otel -spec: - image: "otel/opentelemetry-collector-contrib:0.157.0@sha256:f2f01157055a9b2aab9df7118e1f1c9abf345e99b23bc7a2bc791db374a7d0f6" - config: - receivers: - # otlp receiver to receive data in OTLP format over gRPC. TLS is enabled. - # Must match the settings in the clients. - otlp: - protocols: - grpc: - endpoint: 0.0.0.0:4317 - tls: - cert_file: /etc/otel/certs/tls.crt - key_file: /etc/otel/certs/tls.key - processors: - memory_limiter: - check_interval: 1s - limit_percentage: 75 - spike_limit_percentage: 15 - batch: - send_batch_size: 10000 - timeout: 10s - k8sattributes: - auth_type: 'serviceAccount' - extract: - metadata: # extracted from the pod - - k8s.namespace.name - - k8s.node.name - - k8s.pod.name - - k8s.pod.start_time - - k8s.pod.uid - - container.image.name - - container.image.tag - - k8s.container.name - pod_association: # How to associate the data to a pod (order matters) - - sources: # First try to use the value of the resource attribute k8s.pod.ip - - from: resource_attribute - name: k8s.pod.ip - - sources: # Then try to use the value of the resource attribute k8s.pod.uid - - from: resource_attribute - name: k8s.pod.uid - - sources: # If neither of those work, use the request's connection to get the pod IP. - - from: connection - exporters: - debug: - verbosity: detailed - prometheus: - endpoint: "0.0.0.0:9464" - send_timestamps: true - resource_to_telemetry_conversion: - enabled: true - otlp: - endpoint: jaeger:4317 - tls: - insecure: true - service: - pipelines: - traces: - receivers: [ otlp ] - processors: [ memory_limiter, batch, k8sattributes ] - exporters: [ debug, otlp ] - metrics: - receivers: [ otlp ] - processors: [ memory_limiter, batch, k8sattributes ] - exporters: [ debug, prometheus ] - volumeMounts: - - name: otel-tls - mountPath: /etc/otel/certs - readOnly: true - volumes: - - name: otel-tls - secret: - secretName: otel-collector-tls diff --git a/operator/config/samples/opentelemetry-multi/team-c/cluster_c.yaml b/operator/config/samples/opentelemetry-multi/team-c/cluster_c.yaml deleted file mode 100644 index 0a2b7d8c..00000000 --- a/operator/config/samples/opentelemetry-multi/team-c/cluster_c.yaml +++ /dev/null @@ -1,68 +0,0 @@ -# cluster-c is backed by klio-b, which lives in the "default" namespace, not -# here. This exercises a single Klio server serving clusters from more than -# one namespace. serverSecretName/clientSecretName/the OTel client secret -# below are NOT created by this file: they are copied in from "default" by -# ../copy-cross-namespace-secrets.sh, which must be run before applying this -# kustomization. -apiVersion: postgresql.cnpg.io/v1 -kind: Cluster -metadata: - name: cluster-c - namespace: team-c -spec: - instances: 3 - imagePullPolicy: Always - - projectedVolumeTemplate: - sources: - - secret: - name: otel-collector-ca - items: - - key: ca.crt - path: ca.crt - - secret: - name: klio-server-otel-client-tls - items: - - key: tls.crt - path: tls.crt - - key: tls.key - path: tls.key - - postgresql: - pg_hba: - - local replication all peer map=local - - plugins: - - name: klio.cnpg.io - enabled: true - parameters: - pluginConfigurationRef: client-config-cluster-c - - storage: - size: 1Gi ---- -apiVersion: klio.cnpg.io/v1alpha1 -kind: PluginConfiguration -metadata: - name: client-config-cluster-c - namespace: team-c -spec: - serverAddress: klio-b.default.svc.cluster.local - clientSecretName: cluster-c-klio-user - serverSecretName: klio-b-tls - clusterName: cluster-c - containers: - - name: klio-plugin - env: - - name: OTEL_SERVICE_NAME - value: "klio-plugin" - envFrom: - - configMapRef: - name: cluster-klio-otel-config - - name: klio-restore - env: - - name: OTEL_SERVICE_NAME - value: "klio-restore" - envFrom: - - configMapRef: - name: cluster-klio-otel-config diff --git a/operator/config/samples/opentelemetry-multi/team-c/kustomization.yaml b/operator/config/samples/opentelemetry-multi/team-c/kustomization.yaml deleted file mode 100644 index 72c31da7..00000000 --- a/operator/config/samples/opentelemetry-multi/team-c/kustomization.yaml +++ /dev/null @@ -1,9 +0,0 @@ -# Apply ../kustomization.yaml first, then run ../copy-cross-namespace-secrets.sh -# to populate the secrets this kustomization expects to already exist -# (klio-b-tls, cluster-c-klio-user, otel-collector-ca, -# klio-server-otel-client-tls), then apply this one. -namespace: team-c -resources: - - namespace.yaml - - cluster_klio_otel_config.yaml - - cluster_c.yaml diff --git a/operator/config/samples/opentelemetry-multi/team-e/cluster_e.yaml b/operator/config/samples/opentelemetry-multi/team-e/cluster_e.yaml deleted file mode 100644 index 5c8b8eb2..00000000 --- a/operator/config/samples/opentelemetry-multi/team-e/cluster_e.yaml +++ /dev/null @@ -1,62 +0,0 @@ -apiVersion: postgresql.cnpg.io/v1 -kind: Cluster -metadata: - name: cluster-e - namespace: team-e -spec: - instances: 3 - imagePullPolicy: Always - - projectedVolumeTemplate: - sources: - - secret: - name: otel-collector-ca - items: - - key: ca.crt - path: ca.crt - - secret: - name: klio-server-otel-client-tls - items: - - key: tls.crt - path: tls.crt - - key: tls.key - path: tls.key - - postgresql: - pg_hba: - - local replication all peer map=local - - plugins: - - name: klio.cnpg.io - enabled: true - parameters: - pluginConfigurationRef: client-config-cluster-e - - storage: - size: 1Gi ---- -apiVersion: klio.cnpg.io/v1alpha1 -kind: PluginConfiguration -metadata: - name: client-config-cluster-e - namespace: team-e -spec: - serverAddress: klio-a - clientSecretName: cluster-e-klio-user - serverSecretName: klio-a-tls - clusterName: cluster-e - containers: - - name: klio-plugin - env: - - name: OTEL_SERVICE_NAME - value: "klio-plugin" - envFrom: - - configMapRef: - name: cluster-klio-otel-config - - name: klio-restore - env: - - name: OTEL_SERVICE_NAME - value: "klio-restore" - envFrom: - - configMapRef: - name: cluster-klio-otel-config diff --git a/operator/config/samples/opentelemetry-multi/team-e/klio_server_a.yaml b/operator/config/samples/opentelemetry-multi/team-e/klio_server_a.yaml deleted file mode 100644 index 0e5aff49..00000000 --- a/operator/config/samples/opentelemetry-multi/team-e/klio_server_a.yaml +++ /dev/null @@ -1,92 +0,0 @@ -# Reuses the name "klio-a" (see ../klio_server_a.yaml in "default"). Its -# StatefulSet pod is therefore also named "klio-a-klio-0", identical to the -# original klio-a's host_name -- see team-e/README note in ../README.md. -apiVersion: klio.cnpg.io/v1alpha1 -kind: Server -metadata: - name: klio-a - namespace: team-e -spec: - queue: - pvcTemplate: - resources: - requests: - storage: 1Gi - accessModes: - - ReadWriteOnce - tier1: - cache: - pvcTemplate: - resources: - requests: - storage: 1Gi - accessModes: - - ReadWriteOnce - - data: - pvcTemplate: - resources: - requests: - storage: 1Gi - accessModes: - - ReadWriteOnce - - encryptionKeyFile: - fileReference: - volume: - secret: - secretName: klio-encryption - path: encryption-key - - identityFile: - fileReference: - volume: - secret: - secretName: klio-encryption - path: secret-key - - caSecretName: klio-server-ca - tlsSecretName: klio-a-tls - - image: registry.dev:5000/klio-testing:dev - imagePullPolicy: Always - - template: - spec: - containers: - - name: server - env: - - name: NODE_NAME - valueFrom: - fieldRef: - fieldPath: spec.nodeName - - name: POD_UID - valueFrom: - fieldRef: - fieldPath: metadata.uid - - name: OTEL_SERVICE_NAME - value: "klio-server-a-team-e" - - name: OTEL_RESOURCE_ATTRIBUTES - value: "k8s.node.name=$(NODE_NAME),k8s.pod.uid=$(POD_UID)" - envFrom: - - configMapRef: - name: klio-server-otel-config - volumeMounts: - - mountPath: /otel - name: otel - volumes: - - name: otel - projected: - sources: - - secret: - name: otel-collector-ca - items: - - key: ca.crt - path: ca.crt - - secret: - name: klio-server-otel-client-tls - items: - - key: tls.crt - path: tls.crt - - key: tls.key - path: tls.key diff --git a/operator/config/samples/opentelemetry-multi/team-e/klio_server_a_certificate.yaml b/operator/config/samples/opentelemetry-multi/team-e/klio_server_a_certificate.yaml deleted file mode 100644 index 8d363c2f..00000000 --- a/operator/config/samples/opentelemetry-multi/team-e/klio_server_a_certificate.yaml +++ /dev/null @@ -1,29 +0,0 @@ -# A SECOND server named "klio-a" (same name as the one in "default"), in a -# different namespace. Its StatefulSet pod is therefore also named -# "klio-a-klio-0" -- an identical host_name to the original klio-a, testing -# whether $server (built from host_name alone) can disambiguate two -# same-named servers across namespaces. -apiVersion: cert-manager.io/v1 -kind: Certificate -metadata: - name: klio-a - namespace: team-e -spec: - secretName: klio-a-tls - commonName: klio-a - dnsNames: - - klio-a - - klio-a.team-e - - klio-a.team-e.svc - - duration: 2160h # 90d - renewBefore: 360h # 15d - - isCA: false - usages: - - server auth - - issuerRef: - name: selfsigned-issuer - kind: Issuer - group: cert-manager.io diff --git a/operator/config/samples/opentelemetry-multi/team-e/kustomization.yaml b/operator/config/samples/opentelemetry-multi/team-e/kustomization.yaml deleted file mode 100644 index 5f76a4cd..00000000 --- a/operator/config/samples/opentelemetry-multi/team-e/kustomization.yaml +++ /dev/null @@ -1,15 +0,0 @@ -# Apply ../kustomization.yaml first, then run: -# ../bootstrap-remote-server.sh team-e cluster-e-klio-user -# to populate the secrets this kustomization expects to already exist -# (klio-server-ca, otel-collector-ca, klio-server-otel-client-tls, -# cluster-e-klio-user), then apply this one. -namespace: team-e -resources: - - namespace.yaml - - issuer.yaml - - klio_server_a_certificate.yaml - - klio_encryption.yaml - - klio_server_otel_config.yaml - - cluster_klio_otel_config.yaml - - klio_server_a.yaml - - cluster_e.yaml diff --git a/operator/config/samples/opentelemetry/README.md b/operator/config/samples/opentelemetry/README.md index 14fb8cdb..778b2270 100644 --- a/operator/config/samples/opentelemetry/README.md +++ b/operator/config/samples/opentelemetry/README.md @@ -1,7 +1,40 @@ # README -This directory contains sample configurations for the a klio server and a -CNPG cluster with OpenTelemetry enabled. +This directory contains two sample environments exercising Klio's +OpenTelemetry integration, both sharing the OTel collector / Jaeger / +Prometheus stack defined in `base/`: + +- [`single/`](single): one Klio server and one CNPG cluster. Start here if + you just want to see OpenTelemetry wired up. +- [`multi/`](multi): two Klio servers and four CNPG clusters distributed + across them, one of which lives in a different namespace than the + servers. It reuses `single/`'s Server and Cluster/PluginConfiguration + definitions (see `multi/team-a`) rather than duplicating them, and exists + to validate the Grafana dashboard's `$namespace`, `$server` and `$cluster` + template variables and its per-tier/per-cluster aggregations against more + than one server, cluster or namespace — something `single/` can't + exercise. + +## Topology + +`multi/`'s four clusters: + +| Cluster | Namespace | Backed by | +|-------------|-----------|-----------| +| cluster-a | default | klio-a | +| cluster-b | default | klio-b | +| cluster-c | team-c | klio-b | +| cluster-d | team-d | klio-a (independent server also named "klio-a") | + +`klio-b` intentionally backs clusters in two different namespaces, since a +single shared backup server serving multiple application namespaces is a +realistic multi-tenant deployment and the case most likely to expose +dashboard attribution bugs. `cluster-d` is backed by a *second*, independent +Klio server that happens to also be named "klio-a" (see `multi/team-d`): +since a Server's StatefulSet pod name is derived from the Server's own name +alone, both servers' pods are named "klio-a-klio-0", giving them an +identical host_name label — a case the dashboard's `$server` variable +cannot disambiguate on its own. ## Prerequisites @@ -13,6 +46,27 @@ A running Kubernetes cluster with the following operators installed: - OpenTelemetry - Prometheus +`jq` must also be available locally (used by `multi/copy-cross-namespace-secrets.sh` +and `multi/bootstrap-remote-server.sh`). + +All of `multi`'s client certificates (cluster-a's through cluster-d's) are +issued through a `ClusterIssuer`, which resolves its backing CA secret in +cert-manager's `--cluster-resource-namespace` rather than in the namespace +of the `Certificate` requesting it. This matters only for cluster-c's and +cluster-d's, which request their certificate from `team-c`/`team-d` +directly instead of `default` — cluster-a's and cluster-b's already live in +`default`, so the same lookup is a same-namespace no-op for them. This +sample assumes cert-manager's cluster resource namespace is `default` +(where `base/klio_server_ca.yaml` is deployed), so install cert-manager +accordingly, e.g. with the Jetstack Helm chart: + +```shell +helm upgrade --install cert-manager jetstack/cert-manager \ + --namespace cert-manager --create-namespace \ + --set crds.enabled=true \ + --set clusterResourceNamespace=default +``` + ## Deploying a Kubernetes cluster with the required operators Assuming an environment with CloudNativePG, Klio and cert-manager @@ -42,8 +96,90 @@ helm upgrade --install \ prometheus-community prometheus-community/kube-prometheus-stack ``` -## Deploying the sample configuration +## Deploying the "single" sample + +```shell +kubectl apply -k operator/config/samples/opentelemetry/single +``` + +Wait for `klio` and `cluster-example` to become ready, then trigger a base +backup so the Grafana dashboard has backup/snapshot data to show: + +```shell +kubectl apply -f operator/config/samples/opentelemetry/single/backups-example.yaml +``` + +## Deploying the "multi" sample + +1. Deploy the two servers, the two same-namespace clusters (cluster-a, + cluster-b) and the shared OTel/Jaeger stack, all in `default`: + + ```shell + kubectl apply -k operator/config/samples/opentelemetry/multi + ``` + + Wait for `klio-a`, `klio-b`, `cluster-a` and `cluster-b` to become ready + before continuing. + +1. cluster-c's own client certificate is requested directly in `team-c` + through the cluster-scoped `klio-server-ca` `ClusterIssuer`, so it needs + no copying. klio-b's server certificate and the OTel collector's + certificate are pinned by exact bytes rather than CA-validated (see the + script's comments), so those still have to be copied from `default` + into the `team-c` namespace: + + ```shell + ./operator/config/samples/opentelemetry/multi/copy-cross-namespace-secrets.sh + ``` + +1. Deploy cluster-c into `team-c`: + + ```shell + kubectl apply -k operator/config/samples/opentelemetry/multi/team-c + ``` + +1. cluster-d's own client certificate is likewise requested directly in + `team-d` through the `klio-server-ca` `ClusterIssuer`. `team-d`'s server + (its own "klio-a") is independently self-signed rather than a copy of + `default`'s CA-issued certificate, but it still needs to validate + clients signed by the shared `klio-server-ca` and to export telemetry to + the shared collector. Copy the CA's public certificate and the OTel + collector's trust anchor into the `team-d` namespace: + + ```shell + ./operator/config/samples/opentelemetry/multi/bootstrap-remote-server.sh team-d + ``` + +1. Deploy cluster-d (and its own klio-a server) into `team-d`: + + ```shell + kubectl apply -k operator/config/samples/opentelemetry/multi/team-d + ``` + +## Validating the Grafana dashboard with the "multi" sample + +Trigger one base backup per cluster (needed before any backup/snapshot panel +has data to show): ```shell -kubectl apply -k operator/config/samples/opentelemetry +kubectl apply -f operator/config/samples/opentelemetry/multi/backups-example.yaml ``` + +Once all backups complete and Prometheus has scraped a metrics-collection +cycle, open the Klio Grafana dashboard and confirm: + +- The `$namespace` variable offers `default`, `team-c` and `team-d`. +- The `$server` variable offers `klio-a-klio-0` (twice, once per namespace) + and `klio-b-klio-0` (the value is each server's pod hostname). +- The `$cluster` variable offers `cluster-a`, `cluster-b`, `cluster-c` and + `cluster-d`, and narrows correctly when `$namespace`/`$server` are + filtered (e.g. selecting `$namespace=team-c` should only ever offer + `cluster-c`). +- Per-cluster and per-server panels correctly attribute data instead of + aggregating everything together, and in particular that the two + same-named `klio-a` servers (in `default` and in `team-d`) are not + conflated. + +Any panel that fails to distinguish between clusters/servers/namespaces +here is a dashboard bug to file separately; this sample's job is only to +make that determination possible. \ No newline at end of file diff --git a/operator/config/samples/opentelemetry-multi/cluster_klio_otel_config.yaml b/operator/config/samples/opentelemetry/base/cluster_klio_otel_config.yaml similarity index 100% rename from operator/config/samples/opentelemetry-multi/cluster_klio_otel_config.yaml rename to operator/config/samples/opentelemetry/base/cluster_klio_otel_config.yaml diff --git a/operator/config/samples/opentelemetry-multi/issuer.yaml b/operator/config/samples/opentelemetry/base/issuer.yaml similarity index 100% rename from operator/config/samples/opentelemetry-multi/issuer.yaml rename to operator/config/samples/opentelemetry/base/issuer.yaml diff --git a/operator/config/samples/opentelemetry-multi/jaeger.yaml b/operator/config/samples/opentelemetry/base/jaeger.yaml similarity index 100% rename from operator/config/samples/opentelemetry-multi/jaeger.yaml rename to operator/config/samples/opentelemetry/base/jaeger.yaml diff --git a/operator/config/samples/opentelemetry/klio_encryption.yaml b/operator/config/samples/opentelemetry/base/klio_encryption.yaml similarity index 76% rename from operator/config/samples/opentelemetry/klio_encryption.yaml rename to operator/config/samples/opentelemetry/base/klio_encryption.yaml index 1938bc40..a60d4b7e 100644 --- a/operator/config/samples/opentelemetry/klio_encryption.yaml +++ b/operator/config/samples/opentelemetry/base/klio_encryption.yaml @@ -1,3 +1,6 @@ +# Shared encryption identity for every Klio server in these samples. +# Reusing the same identity across servers is fine for this demo; production +# deployments would normally give each server its own. apiVersion: v1 stringData: secret-key: AGE-SECRET-KEY-1P7TWJ8H837DPVLSRVHAP0ZCZVHXJRTUS7S0A5VZAR49URW0Z9Q0SFW4HVM diff --git a/operator/config/samples/opentelemetry/base/klio_server_ca.yaml b/operator/config/samples/opentelemetry/base/klio_server_ca.yaml new file mode 100644 index 00000000..43cc936e --- /dev/null +++ b/operator/config/samples/opentelemetry/base/klio_server_ca.yaml @@ -0,0 +1,31 @@ +# Root CA shared by every Klio server in these samples, and by every +# client-auth certificate issued through the klio-server-ca ClusterIssuer +# (see ../base/klio_server_ca_issuer.yaml), including cluster-c's and +# cluster-d's — requested directly in their own namespaces (see +# ../multi/team-c, ../multi/team-d) rather than generated in "default" and +# copied cross-namespace. Sharing a single CA lets any client certificate it +# signs be trusted by any server's caSecretName. +# +# The resulting secret must live in cert-manager's +# --cluster-resource-namespace (see the README's Prerequisites section) for +# the ClusterIssuer to find it — this sample assumes that's "default", +# where ../base (and therefore this Certificate) is deployed. +apiVersion: cert-manager.io/v1 +kind: Certificate +metadata: + name: klio-server-ca +spec: + commonName: klio-server + secretName: klio-server-ca + + duration: 2160h # 90d + renewBefore: 360h # 15d + + isCA: true + usages: + - cert sign + + issuerRef: + name: selfsigned-issuer + kind: Issuer + group: cert-manager.io diff --git a/operator/config/samples/opentelemetry/base/klio_server_ca_issuer.yaml b/operator/config/samples/opentelemetry/base/klio_server_ca_issuer.yaml new file mode 100644 index 00000000..1dd72613 --- /dev/null +++ b/operator/config/samples/opentelemetry/base/klio_server_ca_issuer.yaml @@ -0,0 +1,16 @@ +# A ClusterIssuer (not a namespace-scoped Issuer) so client-auth +# certificates can be requested directly in whatever namespace their +# cluster lives in (see ../multi/team-c, ../multi/team-d) instead of being +# generated in "default" and copied cross-namespace. +# +# cert-manager looks up a ClusterIssuer's `ca.secretName` in its configured +# --cluster-resource-namespace (see the README's Prerequisites section), +# which must therefore be set to "default" for this to resolve to the +# klio-server-ca secret created by ../base/klio_server_ca.yaml. +apiVersion: cert-manager.io/v1 +kind: ClusterIssuer +metadata: + name: klio-server-ca +spec: + ca: + secretName: klio-server-ca diff --git a/operator/config/samples/opentelemetry-multi/klio_server_otel_client_certificate.yaml b/operator/config/samples/opentelemetry/base/klio_server_otel_client_certificate.yaml similarity index 100% rename from operator/config/samples/opentelemetry-multi/klio_server_otel_client_certificate.yaml rename to operator/config/samples/opentelemetry/base/klio_server_otel_client_certificate.yaml diff --git a/operator/config/samples/opentelemetry-multi/klio_server_otel_config.yaml b/operator/config/samples/opentelemetry/base/klio_server_otel_config.yaml similarity index 100% rename from operator/config/samples/opentelemetry-multi/klio_server_otel_config.yaml rename to operator/config/samples/opentelemetry/base/klio_server_otel_config.yaml diff --git a/operator/config/samples/opentelemetry/kustomization.yaml b/operator/config/samples/opentelemetry/base/kustomization.yaml similarity index 62% rename from operator/config/samples/opentelemetry/kustomization.yaml rename to operator/config/samples/opentelemetry/base/kustomization.yaml index fe7c2feb..b4817b7e 100644 --- a/operator/config/samples/opentelemetry/kustomization.yaml +++ b/operator/config/samples/opentelemetry/base/kustomization.yaml @@ -1,16 +1,17 @@ +# Shared infrastructure for both the "single" and "multi" OpenTelemetry +# samples: the OTel collector, Jaeger, the self-signed CA/issuer chain, the +# shared encryption identity, and the default-namespace OTel ConfigMaps used +# by the Klio server(s) and cluster(s) that live in "default". +# # REQUIRES: # kubectl apply -f https://github.com/open-telemetry/opentelemetry-operator/releases/latest/download/opentelemetry-operator.yaml resources: - cluster_klio_otel_config.yaml - - cluster.yaml - - cluster_klio_client_auth.yaml - issuer.yaml - jaeger.yaml - klio_encryption.yaml - - klio_server.yaml - klio_server_ca.yaml - klio_server_ca_issuer.yaml - - klio_server_certificate.yaml - klio_server_otel_client_certificate.yaml - klio_server_otel_config.yaml - otel_collector.yaml @@ -18,5 +19,4 @@ resources: - otel_collector_clusterrole.yaml - otel_collector_clusterrolebinding.yaml - otel_collector_sa.yaml - - otel_collector_svc_monitor.yaml - - plugin_configuration.yaml + - otel_collector_svc_monitor.yaml \ No newline at end of file diff --git a/operator/config/samples/opentelemetry/otel_collector.yaml b/operator/config/samples/opentelemetry/base/otel_collector.yaml similarity index 100% rename from operator/config/samples/opentelemetry/otel_collector.yaml rename to operator/config/samples/opentelemetry/base/otel_collector.yaml diff --git a/operator/config/samples/opentelemetry-multi/otel_collector_certificate.yaml b/operator/config/samples/opentelemetry/base/otel_collector_certificate.yaml similarity index 100% rename from operator/config/samples/opentelemetry-multi/otel_collector_certificate.yaml rename to operator/config/samples/opentelemetry/base/otel_collector_certificate.yaml diff --git a/operator/config/samples/opentelemetry-multi/otel_collector_clusterrole.yaml b/operator/config/samples/opentelemetry/base/otel_collector_clusterrole.yaml similarity index 100% rename from operator/config/samples/opentelemetry-multi/otel_collector_clusterrole.yaml rename to operator/config/samples/opentelemetry/base/otel_collector_clusterrole.yaml diff --git a/operator/config/samples/opentelemetry-multi/otel_collector_clusterrolebinding.yaml b/operator/config/samples/opentelemetry/base/otel_collector_clusterrolebinding.yaml similarity index 100% rename from operator/config/samples/opentelemetry-multi/otel_collector_clusterrolebinding.yaml rename to operator/config/samples/opentelemetry/base/otel_collector_clusterrolebinding.yaml diff --git a/operator/config/samples/opentelemetry-multi/otel_collector_sa.yaml b/operator/config/samples/opentelemetry/base/otel_collector_sa.yaml similarity index 100% rename from operator/config/samples/opentelemetry-multi/otel_collector_sa.yaml rename to operator/config/samples/opentelemetry/base/otel_collector_sa.yaml diff --git a/operator/config/samples/opentelemetry-multi/otel_collector_svc_monitor.yaml b/operator/config/samples/opentelemetry/base/otel_collector_svc_monitor.yaml similarity index 100% rename from operator/config/samples/opentelemetry-multi/otel_collector_svc_monitor.yaml rename to operator/config/samples/opentelemetry/base/otel_collector_svc_monitor.yaml diff --git a/operator/config/samples/opentelemetry/cluster_klio_otel_config.yaml b/operator/config/samples/opentelemetry/cluster_klio_otel_config.yaml deleted file mode 100644 index 9b9da243..00000000 --- a/operator/config/samples/opentelemetry/cluster_klio_otel_config.yaml +++ /dev/null @@ -1,21 +0,0 @@ -apiVersion: v1 -kind: ConfigMap -metadata: - name: cluster-klio-otel-config -data: - OTEL_RESOURCE_DETECTORS: "telemetry.sdk,host,os.type,process.executable.name" - # Because traces use the gRPC protocol, Klio also enables Kopia's own - # OTLP/gRPC trace exporter during backup operations: Kopia spans - # (service.name=kopia) are sent to the same collector, reusing the - # OTEL_EXPORTER_OTLP_* settings below (endpoint and TLS). Kopia can only - # export traces over gRPC. - OTEL_TRACES_EXPORTER: "otlp" - OTEL_METRICS_EXPORTER: "otlp" - OTEL_EXPORTER_OTLP_PROTOCOL: "grpc" - OTEL_EXPORTER_OTLP_ENDPOINT: "https://otel-collector:4317" - OTEL_EXPORTER_OTLP_COMPRESSION: "gzip" - OTEL_EXPORTER_OTLP_TIMEOUT: "10000" - OTEL_EXPORTER_OTLP_INSECURE: "false" - OTEL_EXPORTER_OTLP_CERTIFICATE: "/projected/ca.crt" - OTEL_EXPORTER_OTLP_CLIENT_CERTIFICATE: "/projected/tls.crt" - OTEL_EXPORTER_OTLP_CLIENT_KEY: "/projected/tls.key" diff --git a/operator/config/samples/opentelemetry/issuer.yaml b/operator/config/samples/opentelemetry/issuer.yaml deleted file mode 100644 index 981dfb78..00000000 --- a/operator/config/samples/opentelemetry/issuer.yaml +++ /dev/null @@ -1,7 +0,0 @@ -# cert-manager self-signed issuer, used to generate the certificates in this example. -apiVersion: cert-manager.io/v1 -kind: Issuer -metadata: - name: selfsigned-issuer -spec: - selfSigned: {} diff --git a/operator/config/samples/opentelemetry/jaeger.yaml b/operator/config/samples/opentelemetry/jaeger.yaml deleted file mode 100644 index 13f29bb5..00000000 --- a/operator/config/samples/opentelemetry/jaeger.yaml +++ /dev/null @@ -1,71 +0,0 @@ -# This is a simple Jaeger all-in-one deployment for testing purposes. -# It is NOT recommended for production use. -apiVersion: apps/v1 -kind: Deployment -metadata: - labels: - app: jaeger - name: jaeger -spec: - replicas: 1 - selector: - matchLabels: - app: jaeger - strategy: {} - template: - metadata: - labels: - app: jaeger - spec: - automountServiceAccountToken: false - containers: - - image: cr.jaegertracing.io/jaegertracing/jaeger:2.20.0@sha256:46a886260e04002d8f45e213fc39063fa11a50446048fdaa64786fc0840cb9f8 - name: all-in-one - ports: - - containerPort: 16686 - protocol: TCP - - containerPort: 4317 - protocol: TCP - - containerPort: 4318 - protocol: TCP - - containerPort: 5778 - protocol: TCP - - containerPort: 9411 - protocol: TCP - resources: - requests: - cpu: 100m - memory: 128Mi - ephemeral-storage: 100Mi -status: {} ---- -apiVersion: v1 -kind: Service -metadata: - labels: - app: jaeger - name: jaeger -spec: - ports: - - name: port-1 - port: 16686 - protocol: TCP - targetPort: 16686 - - name: port-2 - port: 4317 - protocol: TCP - targetPort: 4317 - - name: port-3 - port: 4318 - protocol: TCP - targetPort: 4318 - - name: port-4 - port: 5778 - protocol: TCP - targetPort: 5778 - - name: port-5 - port: 9411 - protocol: TCP - targetPort: 9411 - selector: - app: jaeger diff --git a/operator/config/samples/opentelemetry/klio_server_ca.yaml b/operator/config/samples/opentelemetry/klio_server_ca.yaml deleted file mode 100644 index f8d1eb3c..00000000 --- a/operator/config/samples/opentelemetry/klio_server_ca.yaml +++ /dev/null @@ -1,19 +0,0 @@ -apiVersion: cert-manager.io/v1 -kind: Certificate -metadata: - name: klio-server-ca -spec: - commonName: klio-server - secretName: klio-server-ca - - duration: 2160h # 90d - renewBefore: 360h # 15d - - isCA: true - usages: - - cert sign - - issuerRef: - name: selfsigned-issuer - kind: Issuer - group: cert-manager.io diff --git a/operator/config/samples/opentelemetry/klio_server_ca_issuer.yaml b/operator/config/samples/opentelemetry/klio_server_ca_issuer.yaml deleted file mode 100644 index 0c06112f..00000000 --- a/operator/config/samples/opentelemetry/klio_server_ca_issuer.yaml +++ /dev/null @@ -1,7 +0,0 @@ -apiVersion: cert-manager.io/v1 -kind: Issuer -metadata: - name: klio-server-ca -spec: - ca: - secretName: klio-server-ca diff --git a/operator/config/samples/opentelemetry/klio_server_otel_client_certificate.yaml b/operator/config/samples/opentelemetry/klio_server_otel_client_certificate.yaml deleted file mode 100644 index e28f9213..00000000 --- a/operator/config/samples/opentelemetry/klio_server_otel_client_certificate.yaml +++ /dev/null @@ -1,23 +0,0 @@ -apiVersion: cert-manager.io/v1 -kind: Certificate -metadata: - name: klio-server-otel-client -spec: - secretName: klio-server-otel-client-tls - commonName: klio - dnsNames: - - klio - - klio.default - - klio.default.svc - - duration: 2160h # 90d - renewBefore: 360h # 15d - - isCA: false - usages: - - server auth - - issuerRef: - name: selfsigned-issuer - kind: Issuer - group: cert-manager.io diff --git a/operator/config/samples/opentelemetry/klio_server_otel_config.yaml b/operator/config/samples/opentelemetry/klio_server_otel_config.yaml deleted file mode 100644 index 9d78c2de..00000000 --- a/operator/config/samples/opentelemetry/klio_server_otel_config.yaml +++ /dev/null @@ -1,28 +0,0 @@ -apiVersion: v1 -kind: ConfigMap -metadata: - name: klio-server-otel-config -data: - OTEL_RESOURCE_DETECTORS: "telemetry.sdk,host,os.type,process.executable.name" - # Because traces use the gRPC protocol, Klio also enables Kopia's own - # OTLP/gRPC trace exporter: Kopia spans (service.name=kopia) are sent to the - # same collector, reusing the OTEL_EXPORTER_OTLP_TRACES_* settings below - # (endpoint and TLS). Kopia can only export traces over gRPC. - OTEL_TRACES_EXPORTER: "otlp" - OTEL_EXPORTER_OTLP_TRACES_PROTOCOL: "grpc" - OTEL_EXPORTER_OTLP_TRACES_ENDPOINT: "https://otel-collector:4317" - OTEL_EXPORTER_OTLP_TRACES_COMPRESSION: "gzip" - OTEL_EXPORTER_OTLP_TRACES_TIMEOUT: "10000" - OTEL_EXPORTER_OTLP_TRACES_INSECURE: "false" - OTEL_EXPORTER_OTLP_TRACES_CERTIFICATE: "/otel/ca.crt" - OTEL_EXPORTER_OTLP_TRACES_CLIENT_CERTIFICATE: "/otel/tls.crt" - OTEL_EXPORTER_OTLP_TRACES_CLIENT_KEY: "/otel/tls.key" - OTEL_METRICS_EXPORTER: "otlp" - OTEL_METRIC_EXPORT_INTERVAL: "60000" - OTEL_EXPORTER_OTLP_METRICS_PROTOCOL: "grpc" - OTEL_EXPORTER_OTLP_METRICS_ENDPOINT: "https://otel-collector:4317" - OTEL_EXPORTER_OTLP_METRICS_TIMEOUT: "60000" - OTEL_EXPORTER_OTLP_METRICS_INSECURE: "false" - OTEL_EXPORTER_OTLP_METRICS_CERTIFICATE: "/otel/ca.crt" - OTEL_EXPORTER_OTLP_METRICS_CLIENT_CERTIFICATE: "/otel/tls.crt" - OTEL_EXPORTER_OTLP_METRICS_CLIENT_KEY: "/otel/tls.key" diff --git a/operator/config/samples/opentelemetry-multi/backups-example.yaml b/operator/config/samples/opentelemetry/multi/backups-example.yaml similarity index 61% rename from operator/config/samples/opentelemetry-multi/backups-example.yaml rename to operator/config/samples/opentelemetry/multi/backups-example.yaml index b33de6e6..acc58e71 100644 --- a/operator/config/samples/opentelemetry-multi/backups-example.yaml +++ b/operator/config/samples/opentelemetry/multi/backups-example.yaml @@ -1,9 +1,9 @@ -# Triggers one base backup on each of the three clusters, so the Grafana -# dashboard has backup/snapshot data to show for all three (namespace, -# server, cluster) combinations. Apply after cluster-a, cluster-b and -# cluster-c are all healthy: +# Triggers one base backup on each of the four clusters, so the Grafana +# dashboard has backup/snapshot data to show for all four (namespace, +# server, cluster) combinations. Apply after cluster-a, cluster-b, +# cluster-c and cluster-d are all healthy: # -# kubectl apply -f operator/config/samples/opentelemetry-multi/backups-example.yaml +# kubectl apply -f operator/config/samples/opentelemetry/multi/backups-example.yaml apiVersion: postgresql.cnpg.io/v1 kind: Backup metadata: @@ -42,3 +42,16 @@ spec: name: cluster-c pluginConfiguration: name: klio.cnpg.io +--- +apiVersion: postgresql.cnpg.io/v1 +kind: Backup +metadata: + name: cluster-d-backup-1 + namespace: team-d +spec: + method: plugin + target: primary + cluster: + name: cluster-d + pluginConfiguration: + name: klio.cnpg.io diff --git a/operator/config/samples/opentelemetry-multi/bootstrap-remote-server.sh b/operator/config/samples/opentelemetry/multi/bootstrap-remote-server.sh similarity index 62% rename from operator/config/samples/opentelemetry-multi/bootstrap-remote-server.sh rename to operator/config/samples/opentelemetry/multi/bootstrap-remote-server.sh index 7ea7f3ee..18046d89 100755 --- a/operator/config/samples/opentelemetry-multi/bootstrap-remote-server.sh +++ b/operator/config/samples/opentelemetry/multi/bootstrap-remote-server.sh @@ -2,32 +2,33 @@ # Bootstraps a namespace that runs its OWN local Klio server (self-signed, # not copied from "default") which nonetheless needs to: (a) validate client # certs signed by the shared klio-server-ca, and (b) export OTel data to the -# shared collector in "default". Unlike copy-cross-namespace-secrets.sh -# (which pins a specific EXISTING server's own cert for a remote client), -# this copies the CA's public cert so any locally-issued server certificate -# can validate against it, plus the shared OTel collector/client certs, plus -# one cluster's client cert (generated in "default", where klio-server-ca -# lives, then copied here since Issuers are namespace-scoped). +# shared collector in "default". This copies the CA's public cert so any +# locally-issued server certificate can validate against it, plus the shared +# OTel collector/client certs — all three are pinned by exact bytes rather +# than validated through a CA (see +# core/internal/client/klioclient/grpcclient/connection.go), so there is no +# ClusterIssuer shortcut for them. The destination namespace's own client +# certificate (e.g. cluster_d_klio_client_auth.yaml) needs no such copy: it +# is requested directly there via the cluster-scoped klio-server-ca +# ClusterIssuer. # -# Usage: bootstrap-remote-server.sh +# Usage: bootstrap-remote-server.sh # # Run this after `kubectl apply -k .` and before applying the destination # namespace's kustomization, e.g.: -# ./bootstrap-remote-server.sh team-e cluster-e-klio-user +# ./bootstrap-remote-server.sh team-d set -euo pipefail -if [ $# -ne 2 ]; then - echo "usage: $0 " >&2 +if [ $# -ne 1 ]; then + echo "usage: $0 " >&2 exit 1 fi SOURCE_NS=default DEST_NS="$1" -CLIENT_SECRET_NAME="$2" echo "Waiting for cert-manager to issue the secrets in ${SOURCE_NS}..." kubectl wait --for=create secret/klio-server-ca -n "${SOURCE_NS}" --timeout=120s -kubectl wait --for=create "secret/${CLIENT_SECRET_NAME}" -n "${SOURCE_NS}" --timeout=120s kubectl wait --for=create secret/otel-collector-tls -n "${SOURCE_NS}" --timeout=120s kubectl wait --for=create secret/klio-server-otel-client-tls -n "${SOURCE_NS}" --timeout=120s @@ -48,9 +49,4 @@ kubectl get secret klio-server-otel-client-tls -n "${SOURCE_NS}" -o json | jq 'del(.metadata.namespace,.metadata.resourceVersion,.metadata.uid,.metadata.creationTimestamp,.metadata.ownerReferences,.metadata.annotations)' | kubectl apply -n "${DEST_NS}" -f - -echo "Copying the cluster's client certificate (full secret, incl. private key)..." -kubectl get secret "${CLIENT_SECRET_NAME}" -n "${SOURCE_NS}" -o json | - jq 'del(.metadata.namespace,.metadata.resourceVersion,.metadata.uid,.metadata.creationTimestamp,.metadata.ownerReferences,.metadata.annotations)' | - kubectl apply -n "${DEST_NS}" -f - - -echo "Done. You can now apply ${DEST_NS}'s kustomization." +echo "Done. You can now apply ${DEST_NS}'s kustomization." \ No newline at end of file diff --git a/operator/config/samples/opentelemetry-multi/cluster_a_klio_client_auth.yaml b/operator/config/samples/opentelemetry/multi/cluster_a_klio_client_auth.yaml similarity index 92% rename from operator/config/samples/opentelemetry-multi/cluster_a_klio_client_auth.yaml rename to operator/config/samples/opentelemetry/multi/cluster_a_klio_client_auth.yaml index 9cc7e7b0..ea8c9c58 100644 --- a/operator/config/samples/opentelemetry-multi/cluster_a_klio_client_auth.yaml +++ b/operator/config/samples/opentelemetry/multi/cluster_a_klio_client_auth.yaml @@ -15,5 +15,5 @@ spec: issuerRef: name: klio-server-ca - kind: Issuer + kind: ClusterIssuer group: cert-manager.io diff --git a/operator/config/samples/opentelemetry-multi/cluster_b_klio_client_auth.yaml b/operator/config/samples/opentelemetry/multi/cluster_b_klio_client_auth.yaml similarity index 92% rename from operator/config/samples/opentelemetry-multi/cluster_b_klio_client_auth.yaml rename to operator/config/samples/opentelemetry/multi/cluster_b_klio_client_auth.yaml index ee37d034..7d8906dc 100644 --- a/operator/config/samples/opentelemetry-multi/cluster_b_klio_client_auth.yaml +++ b/operator/config/samples/opentelemetry/multi/cluster_b_klio_client_auth.yaml @@ -15,5 +15,5 @@ spec: issuerRef: name: klio-server-ca - kind: Issuer + kind: ClusterIssuer group: cert-manager.io diff --git a/operator/config/samples/opentelemetry-multi/copy-cross-namespace-secrets.sh b/operator/config/samples/opentelemetry/multi/copy-cross-namespace-secrets.sh similarity index 63% rename from operator/config/samples/opentelemetry-multi/copy-cross-namespace-secrets.sh rename to operator/config/samples/opentelemetry/multi/copy-cross-namespace-secrets.sh index 52c68136..447ef020 100755 --- a/operator/config/samples/opentelemetry-multi/copy-cross-namespace-secrets.sh +++ b/operator/config/samples/opentelemetry/multi/copy-cross-namespace-secrets.sh @@ -1,12 +1,13 @@ #!/usr/bin/env bash # Copies the secrets cluster-c (namespace "team-c") needs from klio-b and the -# shared OTel collector, both of which live in "default". cert-manager's -# Issuer resources are namespace-scoped, so the client certificate for -# cluster-c has to be generated in "default" (where the klio-server-ca Issuer -# lives, see cluster_c_klio_client_auth.yaml) and then copied here; the -# server/collector certificates are pinned by exact bytes rather than -# validated through a CA (see core/internal/client/klioclient/grpcclient/connection.go), -# so only their public half needs copying. +# shared OTel collector, both of which live in "default". klio-b's server +# certificate and the OTel collector's certificate are pinned by exact bytes +# rather than validated through a CA (see +# core/internal/client/klioclient/grpcclient/connection.go), so only their +# public half needs copying — there is no ClusterIssuer shortcut for these, +# unlike cluster-c's own client certificate (see +# cluster_c_klio_client_auth.yaml), which is requested directly in "team-c" +# via the cluster-scoped klio-server-ca ClusterIssuer and needs no copy. # # Run this after `kubectl apply -k .` and before `kubectl apply -k team-c`. set -euo pipefail @@ -16,17 +17,11 @@ DEST_NS=team-c echo "Waiting for cert-manager to issue the secrets in ${SOURCE_NS}..." kubectl wait --for=create secret/klio-b-tls -n "${SOURCE_NS}" --timeout=120s -kubectl wait --for=create secret/cluster-c-klio-user -n "${SOURCE_NS}" --timeout=120s kubectl wait --for=create secret/otel-collector-tls -n "${SOURCE_NS}" --timeout=120s kubectl wait --for=create secret/klio-server-otel-client-tls -n "${SOURCE_NS}" --timeout=120s kubectl create namespace "${DEST_NS}" --dry-run=client -o yaml | kubectl apply -f - -echo "Copying cluster-c's client certificate (full secret, incl. private key)..." -kubectl get secret cluster-c-klio-user -n "${SOURCE_NS}" -o json | - jq 'del(.metadata.namespace,.metadata.resourceVersion,.metadata.uid,.metadata.creationTimestamp,.metadata.ownerReferences,.metadata.annotations)' | - kubectl apply -n "${DEST_NS}" -f - - echo "Copying klio-b's server certificate (public cert only, pinned by the client)..." kubectl get secret klio-b-tls -n "${SOURCE_NS}" -o jsonpath='{.data.tls\.crt}' | base64 -d | kubectl create secret generic klio-b-tls -n "${DEST_NS}" --from-file=tls.crt=/dev/stdin \ @@ -42,4 +37,4 @@ kubectl get secret klio-server-otel-client-tls -n "${SOURCE_NS}" -o json | jq 'del(.metadata.namespace,.metadata.resourceVersion,.metadata.uid,.metadata.creationTimestamp,.metadata.ownerReferences,.metadata.annotations)' | kubectl apply -n "${DEST_NS}" -f - -echo "Done. You can now run: kubectl apply -k team-c" +echo "Done. You can now run: kubectl apply -k team-c" \ No newline at end of file diff --git a/operator/config/samples/opentelemetry/multi/kustomization.yaml b/operator/config/samples/opentelemetry/multi/kustomization.yaml new file mode 100644 index 00000000..b8a32ac3 --- /dev/null +++ b/operator/config/samples/opentelemetry/multi/kustomization.yaml @@ -0,0 +1,16 @@ +# Deploys two Klio servers (klio-a, klio-b) and two CNPG clusters +# (cluster-a -> klio-a, cluster-b -> klio-b) into the "default" namespace, +# reusing ../single's Server/Cluster definitions (see ./team-a, ./team-b), +# and shares the OTel collector / Jaeger / Prometheus stack defined in +# ../base. ./team-a and ./team-b are named for consistency with ./team-c +# and ./team-d, but — unlike those two — both still deploy into "default", +# not a namespace of their own. See: +# ./team-c for a third cluster in a different namespace, backed by klio-b +# ./team-d for a second, independent server also named "klio-a" (so its +# StatefulSet pod, and therefore host_name, collides with the original) +resources: + - ../base + - team-a + - team-b + - cluster_a_klio_client_auth.yaml + - cluster_b_klio_client_auth.yaml \ No newline at end of file diff --git a/operator/config/samples/opentelemetry/multi/team-a/kustomization.yaml b/operator/config/samples/opentelemetry/multi/team-a/kustomization.yaml new file mode 100644 index 00000000..89761ac9 --- /dev/null +++ b/operator/config/samples/opentelemetry/multi/team-a/kustomization.yaml @@ -0,0 +1,68 @@ +# Instantiates ../../single's Klio server and CNPG cluster a second time, +# as "klio-a"/"cluster-a" in the "default" namespace. Only the fields that +# derive from the resource's own name are patched here; everything else +# (PVC templates, volumes, cert usages/issuerRef, plugin container list...) +# is inherited unchanged from ../../single. +namespace: default +resources: + - ../../single/server + - ../../single/cluster +patches: + - target: + kind: Server + name: klio + patch: |- + - op: replace + path: /metadata/name + value: klio-a + - op: replace + path: /spec/tlsSecretName + value: klio-a-tls + - op: replace + path: /spec/template/spec/containers/0/env/2/value + value: klio-server-a + - target: + kind: Certificate + name: klio-server + patch: |- + - op: replace + path: /metadata/name + value: klio-a + - op: replace + path: /spec/secretName + value: klio-a-tls + - op: replace + path: /spec/commonName + value: klio-a + - op: replace + path: /spec/dnsNames + value: [klio-a, klio-a.default, klio-a.default.svc] + - target: + kind: Cluster + name: cluster-example + patch: |- + - op: replace + path: /metadata/name + value: cluster-a + - op: replace + path: /spec/plugins/0/parameters/pluginConfigurationRef + value: client-config-cluster-a + - target: + kind: PluginConfiguration + name: client-config-cluster-example + patch: |- + - op: replace + path: /metadata/name + value: client-config-cluster-a + - op: replace + path: /spec/serverAddress + value: klio-a.default + - op: replace + path: /spec/clientSecretName + value: cluster-a-klio-user + - op: replace + path: /spec/serverSecretName + value: klio-a-tls + - op: replace + path: /spec/clusterName + value: cluster-a \ No newline at end of file diff --git a/operator/config/samples/opentelemetry/multi/team-b/kustomization.yaml b/operator/config/samples/opentelemetry/multi/team-b/kustomization.yaml new file mode 100644 index 00000000..6db4639f --- /dev/null +++ b/operator/config/samples/opentelemetry/multi/team-b/kustomization.yaml @@ -0,0 +1,75 @@ +# Instantiates ../../single's Klio server and CNPG cluster a third time, as +# "klio-b"/"cluster-b" in the "default" namespace. klio-b is also reached +# from cluster-c (namespace "team-c", see ../team-c), so its dnsNames must +# cover both the short in-namespace form and the fully-qualified +# cross-namespace form used as PluginConfiguration.serverAddress by +# cluster-c. +namespace: default +resources: + - ../../single/server + - ../../single/cluster +patches: + - target: + kind: Server + name: klio + patch: |- + - op: replace + path: /metadata/name + value: klio-b + - op: replace + path: /spec/tlsSecretName + value: klio-b-tls + - op: replace + path: /spec/template/spec/containers/0/env/2/value + value: klio-server-b + - target: + kind: Certificate + name: klio-server + patch: |- + - op: replace + path: /metadata/name + value: klio-b + - op: replace + path: /spec/secretName + value: klio-b-tls + - op: replace + path: /spec/commonName + value: klio-b + - op: replace + path: /spec/dnsNames + value: + [ + klio-b, + klio-b.default, + klio-b.default.svc, + klio-b.default.svc.cluster.local, + ] + - target: + kind: Cluster + name: cluster-example + patch: |- + - op: replace + path: /metadata/name + value: cluster-b + - op: replace + path: /spec/plugins/0/parameters/pluginConfigurationRef + value: client-config-cluster-b + - target: + kind: PluginConfiguration + name: client-config-cluster-example + patch: |- + - op: replace + path: /metadata/name + value: client-config-cluster-b + - op: replace + path: /spec/serverAddress + value: klio-b.default + - op: replace + path: /spec/clientSecretName + value: cluster-b-klio-user + - op: replace + path: /spec/serverSecretName + value: klio-b-tls + - op: replace + path: /spec/clusterName + value: cluster-b \ No newline at end of file diff --git a/operator/config/samples/opentelemetry/multi/team-c/cluster_c_klio_client_auth.yaml b/operator/config/samples/opentelemetry/multi/team-c/cluster_c_klio_client_auth.yaml new file mode 100644 index 00000000..b9f8b9e1 --- /dev/null +++ b/operator/config/samples/opentelemetry/multi/team-c/cluster_c_klio_client_auth.yaml @@ -0,0 +1,25 @@ +# cluster-c's client certificate, requested directly in "team-c" via the +# cluster-scoped klio-server-ca ClusterIssuer (see ../../base). Unlike +# klio-b's own certificate (which cluster-c's PluginConfiguration pins by +# exact bytes, see ../copy-cross-namespace-secrets.sh), this is a genuinely +# separate certificate signed by the shared CA — no cross-namespace copy +# needed. +apiVersion: cert-manager.io/v1 +kind: Certificate +metadata: + name: cluster-c-klio-user +spec: + commonName: klio@cluster-c + secretName: cluster-c-klio-user + + duration: 2160h # 90d + renewBefore: 360h # 15d + + isCA: false + usages: + - client auth + + issuerRef: + name: klio-server-ca + kind: ClusterIssuer + group: cert-manager.io \ No newline at end of file diff --git a/operator/config/samples/opentelemetry-multi/team-c/cluster_klio_otel_config.yaml b/operator/config/samples/opentelemetry/multi/team-c/cluster_klio_otel_config.yaml similarity index 100% rename from operator/config/samples/opentelemetry-multi/team-c/cluster_klio_otel_config.yaml rename to operator/config/samples/opentelemetry/multi/team-c/cluster_klio_otel_config.yaml diff --git a/operator/config/samples/opentelemetry/multi/team-c/kustomization.yaml b/operator/config/samples/opentelemetry/multi/team-c/kustomization.yaml new file mode 100644 index 00000000..4ea58e3f --- /dev/null +++ b/operator/config/samples/opentelemetry/multi/team-c/kustomization.yaml @@ -0,0 +1,57 @@ +# Apply ../kustomization.yaml first, then run ../copy-cross-namespace-secrets.sh +# to populate the secrets this kustomization expects to already exist +# (klio-b-tls, otel-collector-ca, klio-server-otel-client-tls), then apply +# this one. +# +# cluster-c reuses ../../single's Cluster/PluginConfiguration (see ../team-a +# for how that reuse looks); it is backed by klio-b, which lives in the +# "default" namespace, not here. This exercises a single Klio server +# serving clusters from more than one namespace. serverSecretName above +# (klio-b-tls) and the OTel client secret are NOT created by this +# kustomization: klio-b's certificate is pinned by exact bytes rather than +# CA-validated (see connection.go, referenced from +# ../copy-cross-namespace-secrets.sh), so its public half has to be copied +# in from "default" by that script, which must be run before applying this +# kustomization. cluster-c's OWN client certificate +# (cluster_c_klio_client_auth.yaml) has no such requirement: it is signed +# by the cluster-scoped klio-server-ca ClusterIssuer (see ../../base) and +# requested directly in this namespace. +namespace: team-c +resources: + - namespace.yaml + - cluster_klio_otel_config.yaml + - cluster_c_klio_client_auth.yaml + - ../../single/cluster +patches: + - target: + kind: Cluster + name: cluster-example + patch: |- + - op: replace + path: /metadata/name + value: cluster-c + - op: replace + path: /spec/plugins/0/parameters/pluginConfigurationRef + value: client-config-cluster-c + - op: replace + path: /spec/projectedVolumeTemplate/sources/0/secret/name + value: otel-collector-ca + - target: + kind: PluginConfiguration + name: client-config-cluster-example + patch: |- + - op: replace + path: /metadata/name + value: client-config-cluster-c + - op: replace + path: /spec/serverAddress + value: klio-b.default.svc.cluster.local + - op: replace + path: /spec/clientSecretName + value: cluster-c-klio-user + - op: replace + path: /spec/serverSecretName + value: klio-b-tls + - op: replace + path: /spec/clusterName + value: cluster-c diff --git a/operator/config/samples/opentelemetry-multi/team-c/namespace.yaml b/operator/config/samples/opentelemetry/multi/team-c/namespace.yaml similarity index 100% rename from operator/config/samples/opentelemetry-multi/team-c/namespace.yaml rename to operator/config/samples/opentelemetry/multi/team-c/namespace.yaml diff --git a/operator/config/samples/opentelemetry/multi/team-d/cluster_d_klio_client_auth.yaml b/operator/config/samples/opentelemetry/multi/team-d/cluster_d_klio_client_auth.yaml new file mode 100644 index 00000000..a846b41b --- /dev/null +++ b/operator/config/samples/opentelemetry/multi/team-d/cluster_d_klio_client_auth.yaml @@ -0,0 +1,28 @@ +# Client cert for cluster-d, backed by a SECOND Klio server also named +# "klio-a" (see ../team-d), living in a different namespace than the +# original klio-a in "default". Since a Server's StatefulSet pod name +# equals the Server's own name, this second klio-a's pod is ALSO named +# "klio-a-klio-0" -- an identical host_name to the original, but in a +# different namespace. Requested directly here via the cluster-scoped +# klio-server-ca ClusterIssuer (see ../../base) — no cross-namespace copy +# needed for this one, unlike the CA's own public cert and the OTel trust +# anchor (see ../bootstrap-remote-server.sh). +apiVersion: cert-manager.io/v1 +kind: Certificate +metadata: + name: cluster-d-klio-user +spec: + commonName: klio@cluster-d + secretName: cluster-d-klio-user + + duration: 2160h # 90d + renewBefore: 360h # 15d + + isCA: false + usages: + - client auth + + issuerRef: + name: klio-server-ca + kind: ClusterIssuer + group: cert-manager.io \ No newline at end of file diff --git a/operator/config/samples/opentelemetry-multi/team-e/cluster_klio_otel_config.yaml b/operator/config/samples/opentelemetry/multi/team-d/cluster_klio_otel_config.yaml similarity index 97% rename from operator/config/samples/opentelemetry-multi/team-e/cluster_klio_otel_config.yaml rename to operator/config/samples/opentelemetry/multi/team-d/cluster_klio_otel_config.yaml index bc232a0e..fff353d4 100644 --- a/operator/config/samples/opentelemetry-multi/team-e/cluster_klio_otel_config.yaml +++ b/operator/config/samples/opentelemetry/multi/team-d/cluster_klio_otel_config.yaml @@ -5,7 +5,7 @@ apiVersion: v1 kind: ConfigMap metadata: name: cluster-klio-otel-config - namespace: team-e + namespace: team-d data: OTEL_RESOURCE_DETECTORS: "telemetry.sdk,host,os.type,process.executable.name" OTEL_TRACES_EXPORTER: "otlp" diff --git a/operator/config/samples/opentelemetry-multi/team-e/issuer.yaml b/operator/config/samples/opentelemetry/multi/team-d/issuer.yaml similarity index 93% rename from operator/config/samples/opentelemetry-multi/team-e/issuer.yaml rename to operator/config/samples/opentelemetry/multi/team-d/issuer.yaml index 04d57c15..57400914 100644 --- a/operator/config/samples/opentelemetry-multi/team-e/issuer.yaml +++ b/operator/config/samples/opentelemetry/multi/team-d/issuer.yaml @@ -5,6 +5,6 @@ apiVersion: cert-manager.io/v1 kind: Issuer metadata: name: selfsigned-issuer - namespace: team-e + namespace: team-d spec: selfSigned: {} diff --git a/operator/config/samples/opentelemetry-multi/team-e/klio_encryption.yaml b/operator/config/samples/opentelemetry/multi/team-d/klio_encryption.yaml similarity index 97% rename from operator/config/samples/opentelemetry-multi/team-e/klio_encryption.yaml rename to operator/config/samples/opentelemetry/multi/team-d/klio_encryption.yaml index 67d851a3..e6c2a9e7 100644 --- a/operator/config/samples/opentelemetry-multi/team-e/klio_encryption.yaml +++ b/operator/config/samples/opentelemetry/multi/team-d/klio_encryption.yaml @@ -15,4 +15,4 @@ stringData: kind: Secret metadata: name: klio-encryption - namespace: team-e + namespace: team-d diff --git a/operator/config/samples/opentelemetry-multi/team-e/klio_server_otel_config.yaml b/operator/config/samples/opentelemetry/multi/team-d/klio_server_otel_config.yaml similarity index 98% rename from operator/config/samples/opentelemetry-multi/team-e/klio_server_otel_config.yaml rename to operator/config/samples/opentelemetry/multi/team-d/klio_server_otel_config.yaml index 757acbcb..33e1568d 100644 --- a/operator/config/samples/opentelemetry-multi/team-e/klio_server_otel_config.yaml +++ b/operator/config/samples/opentelemetry/multi/team-d/klio_server_otel_config.yaml @@ -5,7 +5,7 @@ apiVersion: v1 kind: ConfigMap metadata: name: klio-server-otel-config - namespace: team-e + namespace: team-d data: OTEL_RESOURCE_DETECTORS: "telemetry.sdk,host,os.type,process.executable.name" OTEL_TRACES_EXPORTER: "otlp" diff --git a/operator/config/samples/opentelemetry/multi/team-d/kustomization.yaml b/operator/config/samples/opentelemetry/multi/team-d/kustomization.yaml new file mode 100644 index 00000000..c2bf152c --- /dev/null +++ b/operator/config/samples/opentelemetry/multi/team-d/kustomization.yaml @@ -0,0 +1,91 @@ +# Apply ../kustomization.yaml first, then run: +# ../bootstrap-remote-server.sh team-d +# to populate the secrets this kustomization expects to already exist +# (klio-server-ca, otel-collector-ca, klio-server-otel-client-tls), then +# apply this one. Those three are all pinned-by-exact-bytes copies of +# secrets that live in "default" (see the script's comments); cluster-d's +# OWN client certificate (cluster_d_klio_client_auth.yaml) has no such +# requirement, since it is signed by the cluster-scoped klio-server-ca +# ClusterIssuer (see ../../base) and requested directly in this namespace. +# +# Reuses ../../single's Server and Cluster/PluginConfiguration (see +# ../team-a for how that reuse looks). The server keeps the name "klio-a" +# (same as ../team-a's), deliberately colliding with it: since a Server's +# StatefulSet pod name equals the Server's own name, this second klio-a's +# pod is ALSO named "klio-a-klio-0" -- an identical host_name to the +# original, but in a different namespace. The cluster is named "cluster-d". +namespace: team-d +resources: + - namespace.yaml + - issuer.yaml + - klio_encryption.yaml + - klio_server_otel_config.yaml + - cluster_klio_otel_config.yaml + - cluster_d_klio_client_auth.yaml + - ../../single/server + - ../../single/cluster +patches: + - target: + kind: Server + name: klio + patch: |- + - op: replace + path: /metadata/name + value: klio-a + - op: replace + path: /spec/tlsSecretName + value: klio-a-tls + - op: replace + path: /spec/template/spec/containers/0/env/2/value + value: klio-server-a-team-d + - op: replace + path: /spec/template/spec/volumes/0/projected/sources/0/secret/name + value: otel-collector-ca + - target: + kind: Certificate + name: klio-server + patch: |- + - op: replace + path: /metadata/name + value: klio-a + - op: replace + path: /spec/secretName + value: klio-a-tls + - op: replace + path: /spec/commonName + value: klio-a + - op: replace + path: /spec/dnsNames + value: [klio-a, klio-a.team-d, klio-a.team-d.svc] + - target: + kind: Cluster + name: cluster-example + patch: |- + - op: replace + path: /metadata/name + value: cluster-d + - op: replace + path: /spec/plugins/0/parameters/pluginConfigurationRef + value: client-config-cluster-d + - op: replace + path: /spec/projectedVolumeTemplate/sources/0/secret/name + value: otel-collector-ca + - target: + kind: PluginConfiguration + name: client-config-cluster-example + patch: |- + - op: replace + path: /metadata/name + value: client-config-cluster-d + - op: replace + path: /spec/serverAddress + value: klio-a + - op: replace + path: /spec/clientSecretName + value: cluster-d-klio-user + - op: replace + path: /spec/serverSecretName + value: klio-a-tls + - op: replace + path: /spec/clusterName + value: cluster-d \ No newline at end of file diff --git a/operator/config/samples/opentelemetry-multi/team-e/namespace.yaml b/operator/config/samples/opentelemetry/multi/team-d/namespace.yaml similarity index 73% rename from operator/config/samples/opentelemetry-multi/team-e/namespace.yaml rename to operator/config/samples/opentelemetry/multi/team-d/namespace.yaml index 1e94b068..641c44af 100644 --- a/operator/config/samples/opentelemetry-multi/team-e/namespace.yaml +++ b/operator/config/samples/opentelemetry/multi/team-d/namespace.yaml @@ -1,4 +1,4 @@ apiVersion: v1 kind: Namespace metadata: - name: team-e + name: team-d diff --git a/operator/config/samples/opentelemetry/otel_collector_certificate.yaml b/operator/config/samples/opentelemetry/otel_collector_certificate.yaml deleted file mode 100644 index 05ae0e8e..00000000 --- a/operator/config/samples/opentelemetry/otel_collector_certificate.yaml +++ /dev/null @@ -1,13 +0,0 @@ -apiVersion: cert-manager.io/v1 -kind: Certificate -metadata: - name: otel-collector -spec: - secretName: otel-collector-tls - issuerRef: - name: selfsigned-issuer - kind: Issuer - dnsNames: - - otel-collector.default.svc.cluster.local - - otel-collector.default.svc - - otel-collector diff --git a/operator/config/samples/opentelemetry/otel_collector_clusterrole.yaml b/operator/config/samples/opentelemetry/otel_collector_clusterrole.yaml deleted file mode 100644 index e4cdbac1..00000000 --- a/operator/config/samples/opentelemetry/otel_collector_clusterrole.yaml +++ /dev/null @@ -1,33 +0,0 @@ -# Role with permissions to read pods, namespaces, and replicasets. -# Required for the k8sattributes processor to work - -apiVersion: rbac.authorization.k8s.io/v1 -kind: ClusterRole -metadata: - name: otel-collector -rules: - - apiGroups: - - '' - resources: - - 'pods' - - 'namespaces' - verbs: - - 'get' - - 'list' - - 'watch' - - apiGroups: - - 'apps' - resources: - - 'replicasets' - verbs: - - 'get' - - 'list' - - 'watch' - - apiGroups: - - 'extensions' - resources: - - 'replicasets' - verbs: - - 'get' - - 'list' - - 'watch' diff --git a/operator/config/samples/opentelemetry/otel_collector_clusterrolebinding.yaml b/operator/config/samples/opentelemetry/otel_collector_clusterrolebinding.yaml deleted file mode 100644 index 37c13b64..00000000 --- a/operator/config/samples/opentelemetry/otel_collector_clusterrolebinding.yaml +++ /dev/null @@ -1,12 +0,0 @@ -apiVersion: rbac.authorization.k8s.io/v1 -kind: ClusterRoleBinding -metadata: - name: otel-collector -subjects: - - kind: ServiceAccount - name: otel-collector - namespace: default -roleRef: - kind: ClusterRole - name: otel-collector - apiGroup: rbac.authorization.k8s.io diff --git a/operator/config/samples/opentelemetry/otel_collector_sa.yaml b/operator/config/samples/opentelemetry/otel_collector_sa.yaml deleted file mode 100644 index fcba07ec..00000000 --- a/operator/config/samples/opentelemetry/otel_collector_sa.yaml +++ /dev/null @@ -1,4 +0,0 @@ -apiVersion: v1 -kind: ServiceAccount -metadata: - name: otel-collector diff --git a/operator/config/samples/opentelemetry/otel_collector_svc_monitor.yaml b/operator/config/samples/opentelemetry/otel_collector_svc_monitor.yaml deleted file mode 100644 index 9d465a12..00000000 --- a/operator/config/samples/opentelemetry/otel_collector_svc_monitor.yaml +++ /dev/null @@ -1,18 +0,0 @@ -# Expose the OpenTelemetry Collector prometheys exporter to Prometheus using a ServiceMonitor. -apiVersion: monitoring.coreos.com/v1 -kind: ServiceMonitor -metadata: - name: otel-collector -spec: - endpoints: - - path: /metrics - port: prometheus - scheme: http - interval: 30s - scrapeTimeout: 10s - selector: - matchLabels: - app.kubernetes.io/component: opentelemetry-collector - app.kubernetes.io/instance: default.otel - app.kubernetes.io/name: otel-collector - operator.opentelemetry.io/collector-service-type: base diff --git a/operator/config/samples/opentelemetry/single/backups-example.yaml b/operator/config/samples/opentelemetry/single/backups-example.yaml new file mode 100644 index 00000000..293db777 --- /dev/null +++ b/operator/config/samples/opentelemetry/single/backups-example.yaml @@ -0,0 +1,16 @@ +# Triggers one base backup on cluster-example, so the Grafana dashboard has +# backup/snapshot data to show. Apply after cluster-example is healthy: +# +# kubectl apply -f operator/config/samples/opentelemetry/single/backups-example.yaml +apiVersion: postgresql.cnpg.io/v1 +kind: Backup +metadata: + name: cluster-example-backup-1 + namespace: default +spec: + method: plugin + target: primary + cluster: + name: cluster-example + pluginConfiguration: + name: klio.cnpg.io \ No newline at end of file diff --git a/operator/config/samples/opentelemetry/cluster.yaml b/operator/config/samples/opentelemetry/single/cluster/cluster.yaml similarity index 100% rename from operator/config/samples/opentelemetry/cluster.yaml rename to operator/config/samples/opentelemetry/single/cluster/cluster.yaml diff --git a/operator/config/samples/opentelemetry/single/cluster/kustomization.yaml b/operator/config/samples/opentelemetry/single/cluster/kustomization.yaml new file mode 100644 index 00000000..ca3e2665 --- /dev/null +++ b/operator/config/samples/opentelemetry/single/cluster/kustomization.yaml @@ -0,0 +1,6 @@ +# The CNPG cluster + its PluginConfiguration, as their own buildable unit so +# ../../multi can reuse them (as cluster-a, cluster-b, ...) without +# duplicating their spec. See ../../multi/team-a for how that reuse looks. +resources: + - cluster.yaml + - plugin_configuration.yaml diff --git a/operator/config/samples/opentelemetry/plugin_configuration.yaml b/operator/config/samples/opentelemetry/single/cluster/plugin_configuration.yaml similarity index 100% rename from operator/config/samples/opentelemetry/plugin_configuration.yaml rename to operator/config/samples/opentelemetry/single/cluster/plugin_configuration.yaml diff --git a/operator/config/samples/opentelemetry/cluster_klio_client_auth.yaml b/operator/config/samples/opentelemetry/single/cluster_klio_client_auth.yaml similarity index 93% rename from operator/config/samples/opentelemetry/cluster_klio_client_auth.yaml rename to operator/config/samples/opentelemetry/single/cluster_klio_client_auth.yaml index b3f8935b..a572cde5 100644 --- a/operator/config/samples/opentelemetry/cluster_klio_client_auth.yaml +++ b/operator/config/samples/opentelemetry/single/cluster_klio_client_auth.yaml @@ -15,5 +15,5 @@ spec: issuerRef: name: klio-server-ca - kind: Issuer + kind: ClusterIssuer group: cert-manager.io diff --git a/operator/config/samples/opentelemetry/single/kustomization.yaml b/operator/config/samples/opentelemetry/single/kustomization.yaml new file mode 100644 index 00000000..43989f76 --- /dev/null +++ b/operator/config/samples/opentelemetry/single/kustomization.yaml @@ -0,0 +1,9 @@ +# One Klio server + one CNPG cluster, sharing the OTel collector / Jaeger / +# Prometheus stack defined in ../base. See ../multi for a scenario covering +# more than one server/cluster/namespace, which reuses ./server and +# ./cluster rather than duplicating them. +resources: + - ../base + - server + - cluster + - cluster_klio_client_auth.yaml \ No newline at end of file diff --git a/operator/config/samples/opentelemetry/single/server/kustomization.yaml b/operator/config/samples/opentelemetry/single/server/kustomization.yaml new file mode 100644 index 00000000..c60d99f7 --- /dev/null +++ b/operator/config/samples/opentelemetry/single/server/kustomization.yaml @@ -0,0 +1,6 @@ +# The Klio server + its self-signed Certificate, as their own buildable unit +# so ../../multi can reuse them (as klio-a, klio-b, ...) without duplicating +# their spec. See ../../multi/team-a for how that reuse looks. +resources: + - server.yaml + - server_certificate.yaml diff --git a/operator/config/samples/opentelemetry/klio_server.yaml b/operator/config/samples/opentelemetry/single/server/server.yaml similarity index 100% rename from operator/config/samples/opentelemetry/klio_server.yaml rename to operator/config/samples/opentelemetry/single/server/server.yaml diff --git a/operator/config/samples/opentelemetry/klio_server_certificate.yaml b/operator/config/samples/opentelemetry/single/server/server_certificate.yaml similarity index 100% rename from operator/config/samples/opentelemetry/klio_server_certificate.yaml rename to operator/config/samples/opentelemetry/single/server/server_certificate.yaml From 9dbb414542fcb6d23125de4721fc17fd5f627e0e Mon Sep 17 00:00:00 2001 From: Jonathan Battiato Date: Fri, 7 Aug 2026 20:05:37 +0200 Subject: [PATCH 4/5] docs(samples): add a PodMonitor for each CNPG cluster in the OTel samples Each cluster now gets its own cnpg.io/cluster-scoped PodMonitor, bundled into single/cluster (the reusable cluster unit multi/team-a through team-d already build on) and patched per instance like the rest of that unit's resources, following the same pattern used in the sibling klio-benchmark repo. Without it, Prometheus never sees CloudNativePG's own cnpg_pg_stat_replication_* metrics, which the Grafana dashboard's WAL Replication Lag row depends on. Co-Authored-By: Claude Sonnet 5 Signed-off-by: Jonathan Battiato --- operator/config/samples/opentelemetry/README.md | 7 ++++++- .../opentelemetry/multi/team-a/kustomization.yaml | 10 ++++++++++ .../opentelemetry/multi/team-b/kustomization.yaml | 10 ++++++++++ .../opentelemetry/multi/team-c/kustomization.yaml | 10 ++++++++++ .../opentelemetry/multi/team-d/kustomization.yaml | 10 ++++++++++ .../single/cluster/cluster_pod_monitor.yaml | 13 +++++++++++++ .../opentelemetry/single/cluster/kustomization.yaml | 1 + 7 files changed, 60 insertions(+), 1 deletion(-) create mode 100644 operator/config/samples/opentelemetry/single/cluster/cluster_pod_monitor.yaml diff --git a/operator/config/samples/opentelemetry/README.md b/operator/config/samples/opentelemetry/README.md index 778b2270..15cddc9c 100644 --- a/operator/config/samples/opentelemetry/README.md +++ b/operator/config/samples/opentelemetry/README.md @@ -2,7 +2,12 @@ This directory contains two sample environments exercising Klio's OpenTelemetry integration, both sharing the OTel collector / Jaeger / -Prometheus stack defined in `base/`: +Prometheus stack defined in `base/`. Every CNPG cluster also gets its own +`PodMonitor` (see `single/cluster/cluster_pod_monitor.yaml`), so Prometheus +scrapes CloudNativePG's own `cnpg_pg_stat_replication_*` metrics in +addition to what the OTel collector exports — the Grafana dashboard's WAL +Replication Lag row needs this (see +`documentation/web/docs/user/grafana-dashboards.md`): - [`single/`](single): one Klio server and one CNPG cluster. Start here if you just want to see OpenTelemetry wired up. diff --git a/operator/config/samples/opentelemetry/multi/team-a/kustomization.yaml b/operator/config/samples/opentelemetry/multi/team-a/kustomization.yaml index 89761ac9..2aa5f5f9 100644 --- a/operator/config/samples/opentelemetry/multi/team-a/kustomization.yaml +++ b/operator/config/samples/opentelemetry/multi/team-a/kustomization.yaml @@ -65,4 +65,14 @@ patches: value: klio-a-tls - op: replace path: /spec/clusterName + value: cluster-a + - target: + kind: PodMonitor + name: cluster-example + patch: |- + - op: replace + path: /metadata/name + value: cluster-a + - op: replace + path: /spec/selector/matchLabels/cnpg.io~1cluster value: cluster-a \ No newline at end of file diff --git a/operator/config/samples/opentelemetry/multi/team-b/kustomization.yaml b/operator/config/samples/opentelemetry/multi/team-b/kustomization.yaml index 6db4639f..c8de9414 100644 --- a/operator/config/samples/opentelemetry/multi/team-b/kustomization.yaml +++ b/operator/config/samples/opentelemetry/multi/team-b/kustomization.yaml @@ -72,4 +72,14 @@ patches: value: klio-b-tls - op: replace path: /spec/clusterName + value: cluster-b + - target: + kind: PodMonitor + name: cluster-example + patch: |- + - op: replace + path: /metadata/name + value: cluster-b + - op: replace + path: /spec/selector/matchLabels/cnpg.io~1cluster value: cluster-b \ No newline at end of file diff --git a/operator/config/samples/opentelemetry/multi/team-c/kustomization.yaml b/operator/config/samples/opentelemetry/multi/team-c/kustomization.yaml index 4ea58e3f..4973e2e9 100644 --- a/operator/config/samples/opentelemetry/multi/team-c/kustomization.yaml +++ b/operator/config/samples/opentelemetry/multi/team-c/kustomization.yaml @@ -55,3 +55,13 @@ patches: - op: replace path: /spec/clusterName value: cluster-c + - target: + kind: PodMonitor + name: cluster-example + patch: |- + - op: replace + path: /metadata/name + value: cluster-c + - op: replace + path: /spec/selector/matchLabels/cnpg.io~1cluster + value: cluster-c diff --git a/operator/config/samples/opentelemetry/multi/team-d/kustomization.yaml b/operator/config/samples/opentelemetry/multi/team-d/kustomization.yaml index c2bf152c..8861e3b3 100644 --- a/operator/config/samples/opentelemetry/multi/team-d/kustomization.yaml +++ b/operator/config/samples/opentelemetry/multi/team-d/kustomization.yaml @@ -88,4 +88,14 @@ patches: value: klio-a-tls - op: replace path: /spec/clusterName + value: cluster-d + - target: + kind: PodMonitor + name: cluster-example + patch: |- + - op: replace + path: /metadata/name + value: cluster-d + - op: replace + path: /spec/selector/matchLabels/cnpg.io~1cluster value: cluster-d \ No newline at end of file diff --git a/operator/config/samples/opentelemetry/single/cluster/cluster_pod_monitor.yaml b/operator/config/samples/opentelemetry/single/cluster/cluster_pod_monitor.yaml new file mode 100644 index 00000000..fa21a9c9 --- /dev/null +++ b/operator/config/samples/opentelemetry/single/cluster/cluster_pod_monitor.yaml @@ -0,0 +1,13 @@ +# Exposes the CNPG cluster's own metrics (including cnpg_pg_stat_replication_*, +# which the Grafana dashboard's WAL Replication Lag row reads) to Prometheus. +# See ../../../../../documentation/web/docs/user/grafana-dashboards.md. +apiVersion: monitoring.coreos.com/v1 +kind: PodMonitor +metadata: + name: cluster-example +spec: + selector: + matchLabels: + cnpg.io/cluster: cluster-example + podMetricsEndpoints: + - port: metrics \ No newline at end of file diff --git a/operator/config/samples/opentelemetry/single/cluster/kustomization.yaml b/operator/config/samples/opentelemetry/single/cluster/kustomization.yaml index ca3e2665..979cb9b4 100644 --- a/operator/config/samples/opentelemetry/single/cluster/kustomization.yaml +++ b/operator/config/samples/opentelemetry/single/cluster/kustomization.yaml @@ -4,3 +4,4 @@ resources: - cluster.yaml - plugin_configuration.yaml + - cluster_pod_monitor.yaml From d1de89ab9ad408e0b12e64882390068db624405f Mon Sep 17 00:00:00 2001 From: Gabriele Quaresima Date: Mon, 10 Aug 2026 15:25:17 +0200 Subject: [PATCH 5/5] fix(samples): make ClusterIssuer's cluster-resource-namespace an explicit setup step The opentelemetry sample's klio-server-ca ClusterIssuer only resolves its CA secret from cert-manager's --cluster-resource-namespace, which this sample assumes is "default". The only concrete setup instructions the README gives (task integration:deploy-to-kind) install cert-manager without that flag, defaulting to the "cert-manager" namespace instead; the override lived only as a prose aside under Prerequisites, so a reader following the documented flow ends up with a permanently unresolvable ClusterIssuer. Confirmed live: on a cluster set up via task integration:deploy-to-kind, klio-server-ca's ClusterIssuer reports "secrets \"klio-server-ca\" not found", every client certificate in the sample (single and multi alike) gets stuck at Ready: False, and every dependent cluster never leaves "Setting up primary". Re-running the corrected helm upgrade command below flips the ClusterIssuer to Ready and the stuck certificates issue immediately. Moves the fix into an explicit step right after the deploy-to-kind command instead of a Prerequisites aside, and corrects that aside's claim that the namespace mismatch "matters only for cluster-c's and cluster-d's": ClusterIssuer resolution is a single, fixed lookup for the whole cert-manager installation, not scoped per requesting namespace, so it affects cluster-a's and cluster-b's certificates identically. Assisted-by: Claude Signed-off-by: Gabriele Quaresima --- .../config/samples/opentelemetry/README.md | 36 ++++++++++--------- 1 file changed, 19 insertions(+), 17 deletions(-) diff --git a/operator/config/samples/opentelemetry/README.md b/operator/config/samples/opentelemetry/README.md index 15cddc9c..6a19b7ab 100644 --- a/operator/config/samples/opentelemetry/README.md +++ b/operator/config/samples/opentelemetry/README.md @@ -55,22 +55,14 @@ A running Kubernetes cluster with the following operators installed: and `multi/bootstrap-remote-server.sh`). All of `multi`'s client certificates (cluster-a's through cluster-d's) are -issued through a `ClusterIssuer`, which resolves its backing CA secret in -cert-manager's `--cluster-resource-namespace` rather than in the namespace -of the `Certificate` requesting it. This matters only for cluster-c's and -cluster-d's, which request their certificate from `team-c`/`team-d` -directly instead of `default` — cluster-a's and cluster-b's already live in -`default`, so the same lookup is a same-namespace no-op for them. This -sample assumes cert-manager's cluster resource namespace is `default` -(where `base/klio_server_ca.yaml` is deployed), so install cert-manager -accordingly, e.g. with the Jetstack Helm chart: - -```shell -helm upgrade --install cert-manager jetstack/cert-manager \ - --namespace cert-manager --create-namespace \ - --set crds.enabled=true \ - --set clusterResourceNamespace=default -``` +issued through a `ClusterIssuer`, which always resolves its backing CA +secret in cert-manager's `--cluster-resource-namespace`, regardless of +which namespace the requesting `Certificate` lives in: this is a single, +fixed lookup location for the whole cert-manager installation, not a +per-request one, so it affects cluster-a's and cluster-b's certificates +(both in `default`) exactly as much as cluster-c's and cluster-d's. This +sample assumes that namespace is `default` (where `base/klio_server_ca.yaml` +is deployed); see the next section for the command that configures it. ## Deploying a Kubernetes cluster with the required operators @@ -82,6 +74,16 @@ the klio task KIND_CLUSTER_NAME=$(kind get clusters | grep pg-operator-e2e) task integration:deploy-to-kind ``` +That task's cert-manager install does not set `--cluster-resource-namespace`, +so it defaults to the `cert-manager` namespace, not `default`. Reconfigure +it, or every client certificate in this sample fails to issue: + +```shell +helm upgrade cert-manager jetstack/cert-manager \ + --namespace cert-manager --reuse-values \ + --set clusterResourceNamespace=default +``` + you can install the OpenTelemetry operator by running: ```shell @@ -187,4 +189,4 @@ cycle, open the Klio Grafana dashboard and confirm: Any panel that fails to distinguish between clusters/servers/namespaces here is a dashboard bug to file separately; this sample's job is only to -make that determination possible. \ No newline at end of file +make that determination possible.