docs(samples): add multi-server/multi-cluster OpenTelemetry sample

documentation/web/docs/user/grafana-dashboards.md

@@ -100,7 +100,7 @@ When you route metrics through an OpenTelemetry Collector, enable
 `resource_to_telemetry_conversion` on the Prometheus exporter so that
 resource attributes such as the pod and namespace become Prometheus labels.
 The sample collector under
-`operator/config/samples/opentelemetry/otel_collector.yaml` already does
+`operator/config/samples/opentelemetry/base/otel_collector.yaml` already does
 this.
 :::
 
@@ -138,7 +138,7 @@ helm upgrade --install \
 Ensure Prometheus scrapes Klio's metrics by deploying a `ServiceMonitor` (or a
 `PodMonitor`, if the collector's `Service` has no labels) for the
 OpenTelemetry collector's Prometheus exporter — see
-`operator/config/samples/opentelemetry/otel_collector_svc_monitor.yaml`.
+`operator/config/samples/opentelemetry/base/otel_collector_svc_monitor.yaml`.
 
 Port-forward Grafana and log in with `admin` / `prom-operator`:
 

observability/grafana/README.md

@@ -91,7 +91,7 @@ helm upgrade --install \
 Make Prometheus scrape Klio's metrics by deploying a `ServiceMonitor` — or a
 `PodMonitor` if the collector's `Service` has no labels — for the OTel
 collector (see
-[`operator/config/samples/opentelemetry/otel_collector_svc_monitor.yaml`](../../operator/config/samples/opentelemetry/otel_collector_svc_monitor.yaml)
+[`operator/config/samples/opentelemetry/base/otel_collector_svc_monitor.yaml`](../../operator/config/samples/opentelemetry/base/otel_collector_svc_monitor.yaml)
 and the [OpenTelemetry guide](../../documentation/web/docs/user/opentelemetry.md)).
 
 Port-forward Grafana (log in with `admin` / `prom-operator`):

operator/config/samples/opentelemetry/README.md

@@ -1,7 +1,45 @@
 # README
 
-This directory contains sample configurations for the a klio server and a
-CNPG cluster with OpenTelemetry enabled.
+This directory contains two sample environments exercising Klio's
+OpenTelemetry integration, both sharing the OTel collector / Jaeger /
+Prometheus stack defined in `base/`. Every CNPG cluster also gets its own
+`PodMonitor` (see `single/cluster/cluster_pod_monitor.yaml`), so Prometheus
+scrapes CloudNativePG's own `cnpg_pg_stat_replication_*` metrics in
+addition to what the OTel collector exports — the Grafana dashboard's WAL
+Replication Lag row needs this (see
+`documentation/web/docs/user/grafana-dashboards.md`):
+
+- [`single/`](single): one Klio server and one CNPG cluster. Start here if
+  you just want to see OpenTelemetry wired up.
+- [`multi/`](multi): two Klio servers and four CNPG clusters distributed
+  across them, one of which lives in a different namespace than the
+  servers. It reuses `single/`'s Server and Cluster/PluginConfiguration
+  definitions (see `multi/team-a`) rather than duplicating them, and exists
+  to validate the Grafana dashboard's `$namespace`, `$server` and `$cluster`
+  template variables and its per-tier/per-cluster aggregations against more
+  than one server, cluster or namespace — something `single/` can't
+  exercise.
+
+## Topology
+
+`multi/`'s four clusters:
+
+| Cluster     | Namespace | Backed by |
+|-------------|-----------|-----------|
+| cluster-a   | default   | klio-a    |
+| cluster-b   | default   | klio-b    |
+| cluster-c   | team-c    | klio-b    |
+| cluster-d   | team-d    | klio-a (independent server also named "klio-a") |
+
+`klio-b` intentionally backs clusters in two different namespaces, since a
+single shared backup server serving multiple application namespaces is a
+realistic multi-tenant deployment and the case most likely to expose
+dashboard attribution bugs. `cluster-d` is backed by a *second*, independent
+Klio server that happens to also be named "klio-a" (see `multi/team-d`):
+since a Server's StatefulSet pod name is derived from the Server's own name
+alone, both servers' pods are named "klio-a-klio-0", giving them an
+identical host_name label — a case the dashboard's `$server` variable
+cannot disambiguate on its own.
 
 ## Prerequisites
 
@@ -13,6 +51,19 @@ A running Kubernetes cluster with the following operators installed:
 - OpenTelemetry
 - Prometheus
 
+`jq` must also be available locally (used by `multi/copy-cross-namespace-secrets.sh`
+and `multi/bootstrap-remote-server.sh`).
+
+All of `multi`'s client certificates (cluster-a's through cluster-d's) are
+issued through a `ClusterIssuer`, which always resolves its backing CA
+secret in cert-manager's `--cluster-resource-namespace`, regardless of
+which namespace the requesting `Certificate` lives in: this is a single,
+fixed lookup location for the whole cert-manager installation, not a
+per-request one, so it affects cluster-a's and cluster-b's certificates
+(both in `default`) exactly as much as cluster-c's and cluster-d's. This
+sample assumes that namespace is `default` (where `base/klio_server_ca.yaml`
+is deployed); see the next section for the command that configures it.
+
 ## Deploying a Kubernetes cluster with the required operators
 
 Assuming an environment with CloudNativePG, Klio and cert-manager
@@ -23,6 +74,16 @@ the klio task
 KIND_CLUSTER_NAME=$(kind get clusters | grep pg-operator-e2e) task integration:deploy-to-kind
 ```
 
+That task's cert-manager install does not set `--cluster-resource-namespace`,
+so it defaults to the `cert-manager` namespace, not `default`. Reconfigure
+it, or every client certificate in this sample fails to issue:
+
+```shell
+helm upgrade cert-manager jetstack/cert-manager \
+  --namespace cert-manager --reuse-values \
+  --set clusterResourceNamespace=default
+```
+
 you can install the OpenTelemetry operator by running:
 
 ```shell
@@ -42,8 +103,90 @@ helm upgrade --install \
   prometheus-community prometheus-community/kube-prometheus-stack
 ```
 
-## Deploying the sample configuration
+## Deploying the "single" sample
 
 ```shell
-kubectl apply -k operator/config/samples/opentelemetry
+kubectl apply -k operator/config/samples/opentelemetry/single
 ```
+
+Wait for `klio` and `cluster-example` to become ready, then trigger a base
+backup so the Grafana dashboard has backup/snapshot data to show:
+
+```shell
+kubectl apply -f operator/config/samples/opentelemetry/single/backups-example.yaml
+```
+
+## Deploying the "multi" sample
+
+1. Deploy the two servers, the two same-namespace clusters (cluster-a,
+   cluster-b) and the shared OTel/Jaeger stack, all in `default`:
+
+   ```shell
+   kubectl apply -k operator/config/samples/opentelemetry/multi
+   ```
+
+   Wait for `klio-a`, `klio-b`, `cluster-a` and `cluster-b` to become ready
+   before continuing.
+
+1. cluster-c's own client certificate is requested directly in `team-c`
+   through the cluster-scoped `klio-server-ca` `ClusterIssuer`, so it needs
+   no copying. klio-b's server certificate and the OTel collector's
+   certificate are pinned by exact bytes rather than CA-validated (see the
+   script's comments), so those still have to be copied from `default`
+   into the `team-c` namespace:
+
+   ```shell
+   ./operator/config/samples/opentelemetry/multi/copy-cross-namespace-secrets.sh
+   ```
+
+1. Deploy cluster-c into `team-c`:
+
+   ```shell
+   kubectl apply -k operator/config/samples/opentelemetry/multi/team-c
+   ```
+
+1. cluster-d's own client certificate is likewise requested directly in
+   `team-d` through the `klio-server-ca` `ClusterIssuer`. `team-d`'s server
+   (its own "klio-a") is independently self-signed rather than a copy of
+   `default`'s CA-issued certificate, but it still needs to validate
+   clients signed by the shared `klio-server-ca` and to export telemetry to
+   the shared collector. Copy the CA's public certificate and the OTel
+   collector's trust anchor into the `team-d` namespace:
+
+   ```shell
+   ./operator/config/samples/opentelemetry/multi/bootstrap-remote-server.sh team-d
+   ```
+
+1. Deploy cluster-d (and its own klio-a server) into `team-d`:
+
+   ```shell
+   kubectl apply -k operator/config/samples/opentelemetry/multi/team-d
+   ```
+
+## Validating the Grafana dashboard with the "multi" sample
+
+Trigger one base backup per cluster (needed before any backup/snapshot panel
+has data to show):
+
+```shell
+kubectl apply -f operator/config/samples/opentelemetry/multi/backups-example.yaml
+```
+
+Once all backups complete and Prometheus has scraped a metrics-collection
+cycle, open the Klio Grafana dashboard and confirm:
+
+- The `$namespace` variable offers `default`, `team-c` and `team-d`.
+- The `$server` variable offers `klio-a-klio-0` (twice, once per namespace)
+  and `klio-b-klio-0` (the value is each server's pod hostname).
+- The `$cluster` variable offers `cluster-a`, `cluster-b`, `cluster-c` and
+  `cluster-d`, and narrows correctly when `$namespace`/`$server` are
+  filtered (e.g. selecting `$namespace=team-c` should only ever offer
+  `cluster-c`).
+- Per-cluster and per-server panels correctly attribute data instead of
+  aggregating everything together, and in particular that the two
+  same-named `klio-a` servers (in `default` and in `team-d`) are not
+  conflated.
+
+Any panel that fails to distinguish between clusters/servers/namespaces
+here is a dashboard bug to file separately; this sample's job is only to
+make that determination possible.

operator/config/samples/opentelemetry/base/klio_encryption.yaml

@@ -0,0 +1,18 @@
+# Shared encryption identity for every Klio server in these samples.
+# Reusing the same identity across servers is fine for this demo; production
+# deployments would normally give each server its own.
+apiVersion: v1
+stringData:
+  secret-key: AGE-SECRET-KEY-1P7TWJ8H837DPVLSRVHAP0ZCZVHXJRTUS7S0A5VZAR49URW0Z9Q0SFW4HVM
+  public-key: age16y2zu5qptkzateyuarcahs0am75qjpu0xd2ty2dvm5jusnfzxg0scjh5uu
+  encryption-key: |
+    -----BEGIN AGE ENCRYPTED FILE-----
+    YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSAyMzJERW1QTUhjNGdlNmdi
+    WmFLZ3FLL1lZclduUWEvZkJncGdJZmtVd1FzCmZwSHVTcWNMRFN6a2xpY0NHNVRv
+    MlltSlNRY1B4bmNoOUxuZzZ5RENSSmsKLS0tIGNYUUNzSGVqNUZhS3VodnZ1cXNP
+    T3JHS3JRSUNkR1RyeEROYnZYNnNSV00KUd3HyxvvqCyW6hC1sWNDn/xeptHsoz/e
+    hxCgWoSJvfSfNk174njMwEeW+ic=
+    -----END AGE ENCRYPTED FILE-----
+kind: Secret
+metadata:
+  name: klio-encryption

operator/config/samples/opentelemetry/base/klio_server_ca.yaml

@@ -0,0 +1,31 @@
+# Root CA shared by every Klio server in these samples, and by every
+# client-auth certificate issued through the klio-server-ca ClusterIssuer
+# (see ../base/klio_server_ca_issuer.yaml), including cluster-c's and
+# cluster-d's — requested directly in their own namespaces (see
+# ../multi/team-c, ../multi/team-d) rather than generated in "default" and
+# copied cross-namespace. Sharing a single CA lets any client certificate it
+# signs be trusted by any server's caSecretName.
+#
+# The resulting secret must live in cert-manager's
+# --cluster-resource-namespace (see the README's Prerequisites section) for
+# the ClusterIssuer to find it — this sample assumes that's "default",
+# where ../base (and therefore this Certificate) is deployed.
+apiVersion: cert-manager.io/v1
+kind: Certificate
+metadata:
+  name: klio-server-ca
+spec:
+  commonName: klio-server
+  secretName: klio-server-ca
+
+  duration: 2160h # 90d
+  renewBefore: 360h # 15d
+
+  isCA: true
+  usages:
+    - cert sign
+
+  issuerRef:
+    name: selfsigned-issuer
+    kind: Issuer
+    group: cert-manager.io

operator/config/samples/opentelemetry/base/klio_server_ca_issuer.yaml

@@ -0,0 +1,16 @@
+# A ClusterIssuer (not a namespace-scoped Issuer) so client-auth
+# certificates can be requested directly in whatever namespace their
+# cluster lives in (see ../multi/team-c, ../multi/team-d) instead of being
+# generated in "default" and copied cross-namespace.
+#
+# cert-manager looks up a ClusterIssuer's `ca.secretName` in its configured
+# --cluster-resource-namespace (see the README's Prerequisites section),
+# which must therefore be set to "default" for this to resolve to the
+# klio-server-ca secret created by ../base/klio_server_ca.yaml.
+apiVersion: cert-manager.io/v1
+kind: ClusterIssuer
+metadata:
+  name: klio-server-ca
+spec:
+  ca:
+    secretName: klio-server-ca

operator/config/samples/opentelemetry/kustomization.yaml → operator/config/samples/opentelemetry/base/kustomization.yaml

@@ -1,22 +1,22 @@
+# Shared infrastructure for both the "single" and "multi" OpenTelemetry
+# samples: the OTel collector, Jaeger, the self-signed CA/issuer chain, the
+# shared encryption identity, and the default-namespace OTel ConfigMaps used
+# by the Klio server(s) and cluster(s) that live in "default".
+#
 # REQUIRES:
 # kubectl apply -f https://github.com/open-telemetry/opentelemetry-operator/releases/latest/download/opentelemetry-operator.yaml
 resources:
   - cluster_klio_otel_config.yaml
-  - cluster.yaml
-  - cluster_klio_client_auth.yaml
   - issuer.yaml
   - jaeger.yaml
   - klio_encryption.yaml
-  - klio_server.yaml
   - klio_server_ca.yaml
   - klio_server_ca_issuer.yaml
-  - klio_server_certificate.yaml
   - klio_server_otel_client_certificate.yaml
   - klio_server_otel_config.yaml
   - otel_collector.yaml
   - otel_collector_certificate.yaml
   - otel_collector_clusterrole.yaml
   - otel_collector_clusterrolebinding.yaml
   - otel_collector_sa.yaml
-  - otel_collector_svc_monitor.yaml
-  - plugin_configuration.yaml
+  - otel_collector_svc_monitor.yaml

operator/config/samples/opentelemetry/multi/backups-example.yaml

@@ -0,0 +1,57 @@
+# Triggers one base backup on each of the four clusters, so the Grafana
+# dashboard has backup/snapshot data to show for all four (namespace,
+# server, cluster) combinations. Apply after cluster-a, cluster-b,
+# cluster-c and cluster-d are all healthy:
+#
+#   kubectl apply -f operator/config/samples/opentelemetry/multi/backups-example.yaml
+apiVersion: postgresql.cnpg.io/v1
+kind: Backup
+metadata:
+  name: cluster-a-backup-1
+  namespace: default
+spec:
+  method: plugin
+  target: primary
+  cluster:
+    name: cluster-a
+  pluginConfiguration:
+    name: klio.cnpg.io
+---
+apiVersion: postgresql.cnpg.io/v1
+kind: Backup
+metadata:
+  name: cluster-b-backup-1
+  namespace: default
+spec:
+  method: plugin
+  target: primary
+  cluster:
+    name: cluster-b
+  pluginConfiguration:
+    name: klio.cnpg.io
+---
+apiVersion: postgresql.cnpg.io/v1
+kind: Backup
+metadata:
+  name: cluster-c-backup-1
+  namespace: team-c
+spec:
+  method: plugin
+  target: primary
+  cluster:
+    name: cluster-c
+  pluginConfiguration:
+    name: klio.cnpg.io
+---
+apiVersion: postgresql.cnpg.io/v1
+kind: Backup
+metadata:
+  name: cluster-d-backup-1
+  namespace: team-d
+spec:
+  method: plugin
+  target: primary
+  cluster:
+    name: cluster-d
+  pluginConfiguration:
+    name: klio.cnpg.io