diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml new file mode 100644 index 0000000..354bf44 --- /dev/null +++ b/.github/workflows/ci.yml @@ -0,0 +1,50 @@ +name: CI + +on: + pull_request: + branches: [main] + push: + branches: [main] + +jobs: + # ── Job 1: Helm lint & render ────────────────────────────────────────────── + helm-lint: + name: Helm lint + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + + - name: Install Helm + uses: azure/setup-helm@v4 + with: + version: "latest" + + - name: Lint openbao-init (local values) + run: helm lint apps/openbao-init/ -f apps/openbao-init/values-local.yaml + + - name: Lint openbao-init (scaleway values) + run: helm lint apps/openbao-init/ -f apps/openbao-init/values-scaleway.yaml + + - name: Render bootstrap chart (local) + run: helm template boot bootstrap/ --set env=local --set revision=main + + - name: Render clusters/local chart + run: helm template loc clusters/local/ --set revision=main + + - name: Render clusters/scaleway chart + run: helm template loc clusters/scaleway/ --set revision=main + + # ── Job 2: Infisical absent ──────────────────────────────────────────────── + infisical-absent: + name: Infisical absent + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + + - name: Assert no Infisical references in active GitOps manifests + run: | + if git grep -ri infisical apps/ platform/ clusters/ bootstrap/ 2>/dev/null; then + echo "ERROR: Infisical references found in GitOps manifests" + exit 1 + fi + echo "OK: No Infisical references found" diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..4bcadc1 --- /dev/null +++ b/.gitignore @@ -0,0 +1,21 @@ +# OS / editor +.DS_Store +Thumbs.db +*.swp +*.swo +.vscode/ +.idea/ + +# Secrets (never commit) +*.secret.yaml +*.secret.yml +kubeconfig* +*.key +*.pem +secrets/ + +# Helm +*.tgz + +# Temp +tmp/ diff --git a/.specify/feature.json b/.specify/feature.json new file mode 100644 index 0000000..2834fd2 --- /dev/null +++ b/.specify/feature.json @@ -0,0 +1,3 @@ +{ + "feature_directory": "specs/001-replace-infisical-oss-secrets" +} diff --git a/CLAUDE.md b/CLAUDE.md index 4a6c752..87ce86b 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -31,6 +31,7 @@ minikube status # cluster run kubectl --context minikube get pods -n argocd # ArgoCD healthy? kubectl --context minikube get applications -n argocd # apps synced? kubectl --context minikube get pods -n ingress-nginx # ingress up? +kubectl --context minikube get clustersecretstore openbao # OpenBao + ESO ready? ``` ## Validation commands @@ -40,16 +41,21 @@ Always target the local cluster explicitly to avoid accidental execution against ```bash # Lint a Helm chart helm lint apps/demo/ -f apps/demo/values-local.yaml +helm lint apps/openbao-init/ -f apps/openbao-init/values-local.yaml +helm lint apps/openbao-init/ -f apps/openbao-init/values-scaleway.yaml # Render an app chart to inspect output helm template demo apps/demo/ -f apps/demo/values-local.yaml +helm template openbao-init apps/openbao-init/ -f apps/openbao-init/values-local.yaml # Render the app-of-apps charts (confirm revision propagation) helm template boot bootstrap/ --set env=local --set revision= -helm template loc clusters/local/ --set revision= +helm template loc clusters/local/ --set revision= --set repoURL= +helm template loc clusters/scaleway/ --set revision= --set repoURL= # Validate a platform Application manifest kubectl --context minikube apply --dry-run=client -f platform/local/kube-prometheus-stack.yml +kubectl --context minikube apply --dry-run=client -f platform/local/openbao.yml ``` ## Adding things @@ -63,3 +69,39 @@ kubectl --context minikube apply --dry-run=client -f platform/local/kube-prometh 4. `bootstrap/templates/staging.yaml` guarded by `{{- if eq .Values.env "staging" }}` as the entry point for external provisioners **New platform tool:** Add a file to `platform//` following `platform/local/ingress-nginx.yml`. + + +## Active Feature + +**Feature**: Replace Infisical with OSS Secrets Backend +**Plan**: [specs/001-replace-infisical-oss-secrets/plan.md](specs/001-replace-infisical-oss-secrets/plan.md) +**Tasks**: [specs/001-replace-infisical-oss-secrets/tasks.md](specs/001-replace-infisical-oss-secrets/tasks.md) +**Status**: Tasks generated — ready for `/speckit-implement` +**Tool**: OpenBao (MPL 2.0) — decided 2026-06-21 + +### Quick reference + +| Artifact | Path | +|----------|------| +| Spec | `specs/001-replace-infisical-oss-secrets/spec.md` | +| Plan | `specs/001-replace-infisical-oss-secrets/plan.md` | +| Research | `specs/001-replace-infisical-oss-secrets/research.md` | +| Data model | `specs/001-replace-infisical-oss-secrets/data-model.md` | +| Contracts | `specs/001-replace-infisical-oss-secrets/contracts/` | +| Quickstart | `specs/001-replace-infisical-oss-secrets/quickstart.md` | +| Tasks | `specs/001-replace-infisical-oss-secrets/tasks.md` | + +### MVP scope (US1, Phases 1–3, tasks T001–T020) + +1. Scaffold `apps/openbao-init/` Helm chart (T001–T004) +2. Write `platform//openbao.yml` + `external-secrets.yml` (T005–T009) +3. Write init Job, RBAC, policies, ClusterSecretStore, cluster templates, sync waves (T010–T018) +4. Validate: `helm lint apps/openbao-init/` + local Scenario 1 smoke test (T019–T020) + +### Key constraints + +- `scaleway-s3-credentials` Secret created by infra Terraform, not this repo — init Job reads it +- S3 backend: no HA on Scaleway (no DynamoDB locking) → single replica +- Init Job must be idempotent; root token revoked after first init +- All external Helm chart versions must be pinned explicitly + diff --git a/README.md b/README.md index a5165e5..e38f05f 100644 --- a/README.md +++ b/README.md @@ -172,6 +172,33 @@ The `platform` App-of-Apps picks it up automatically. --- +## Local development workflow + +**Smoke test** (destroy + rebuild everything from scratch): +```bash +# from the infrastructure repo +mise run reset && mise run dev +``` + +ArgoCD takes a few minutes to reconcile all apps — be patient. The difference between "still syncing" and "actually stuck" is visible in: +```bash +kubectl --context minikube get applications -n argocd +``` + +**Important:** before running, check `infrastructure/02-cluster/local/nico.auto.tfvars` — it contains the `gitops_revision` variable that controls which branch of this repo ArgoCD will deploy from. If you're testing a feature branch, set it there: +```hcl +gitops_revision = "feat/your-branch" +``` +If it points to `main` and your changes aren't merged yet, ArgoCD will deploy the old code. + +**Validate OpenBao is up:** +```bash +kubectl --context minikube get clustersecretstore openbao +# READY=True → OpenBao running, unsealed, ESO connected +``` + +--- + ## Accessing the demo app (local) After `mise run reset && mise run dev` in the infrastructure repo: diff --git a/TODO.md b/TODO.md new file mode 100644 index 0000000..61b4e7a --- /dev/null +++ b/TODO.md @@ -0,0 +1,71 @@ +# TODO + +## OpenBao — ce qui reste à faire + +### 1. Sortir du mode `dev` + +`platform/local/openbao.yml` a encore `dev: enabled: true`. +Ce mode est in-memory, sans Raft, sans seal — inutilisable en prod. + +- [ ] Passer en mode standalone avec Raft réel (supprimer `dev: enabled: true`, garder la config `standalone.config` déjà présente) + +--- + +### 2. Auto-unseal — **Option C choisie : AWS KMS natif** + +Scaleway KMS n'est pas supporté nativement par OpenBao (pas d'API AWS KMS-compatible, pas de PKCS#11). On part sur AWS KMS natif (hors Scaleway). + +- [x] **Terraform KMS** — `infrastructure/03-backup/scaleway/kms.tf` : clé KMS symétrique (rotation auto, `prevent_destroy`), user IAM dédié single-purpose (Encrypt/Decrypt/DescribeKey via key policy, aucune policy IAM), access key statique. Outputs : `openbao_unseal_kms_key_id`, `openbao_unseal_aws_region`, `openbao_unseal_access_key_id`, `openbao_unseal_secret_access_key`. + - ⚠️ **Apply = admin/local uniquement** (`aws sso login`). Le rôle CI backup est S3-only et la permissions-boundary interdit les users IAM + access keys → un push touchant `kms.tf` fait échouer le CI `backup_scaleway`. +- [ ] **Côté gitops** — ajouter le stanza `seal "awskms"` dans `standalone.config` de `platform//openbao.yml` (`region`, `kms_key_id`) + injecter `AWS_ACCESS_KEY_ID` / `AWS_SECRET_ACCESS_KEY` dans le pod OpenBao via un Secret (`extraSecretEnvironmentVars` ou env). Chicken-and-egg : ce Secret ne peut PAS venir d'OpenBao lui-même → source externe (bootstrap cluster terraform, ou Infisical→ESO pendant la transition). +- [ ] **Init** — avec auto-unseal, `bao operator init` génère toujours des recovery keys (Shamir de secours) : les stocker hors-cluster. + +--- + +### 3. Snapshot agent — prérequis OpenBao + +La config S3 est posée (`platform/local/openbao.yml`), mais le CronJob ne peut pas s'authentifier à OpenBao sans : + +- [ ] Activer le Kubernetes auth method dans OpenBao : + ```bash + bao auth enable kubernetes + bao write auth/kubernetes/config kubernetes_host=https://kubernetes.default.svc + ``` +- [ ] Créer la policy `snapshot` (droit sur `sys/storage/raft/snapshot`) +- [ ] Créer le role `snapshot` lié au ServiceAccount du CronJob : + ```bash + bao write auth/kubernetes/role/snapshot \ + bound_service_account_names=openbao-snapshot-agent \ + bound_service_account_namespaces=openbao \ + policies=snapshot \ + ttl=1h + ``` + +--- + +### 4. Restauration au démarrage (non natif) — **implémenté, en attente de validation E2E** + +Le snapshot agent fait des backups, mais ne restaure pas au boot. +Pour un cluster éphémère, il faut un job d'init qui : + +- [x] Vérifie si OpenBao est fraîchement initialisé (`bao operator init -status`) +- [x] Télécharge le dernier snapshot depuis `s3://backup-dev-id` +- [x] Exécute `bao operator raft snapshot restore ` + +Réalisé via `apps/openbao-init/` : Job-hook ArgoCD (`hook: Sync`, +`hook-delete-policy: BeforeHookCreation`) qui miroir le script de backup upstream +dans la même image `openbao-snapshot-agent` (`bao` + `s3cmd`). Idempotent : ne touche +jamais un cluster déjà initialisé. Sur cluster vide → `bao operator init` (token root +jetable, juste le temps du restore) puis `snapshot restore`. **Local uniquement** +(scaleway hors scope, pas encore de snapshots). + +- [ ] Smoke test : `mise run reset && mise run dev`, vérifier qu'un secret témoin + survit au reset. Vérifier aussi que le nœud se re-unseal seul après restore + (même clé KMS) — sinon redémarrer `openbao-0`. + +--- + +## Ce qui est fait + +- [x] `snapshotAgent` configuré dans `platform/local/openbao.yml` — pousse vers `s3://backup-dev-id` (Scaleway fr-par) toutes les 5 min +- [x] Credentials S3 injectés via Secret `scaleway-s3-credentials` (créé par Terraform, pas dans git) diff --git a/apps/openbao-init/Chart.yaml b/apps/openbao-init/Chart.yaml new file mode 100644 index 0000000..28a5aaa --- /dev/null +++ b/apps/openbao-init/Chart.yaml @@ -0,0 +1,6 @@ +apiVersion: v2 +name: openbao-init +description: Restores the latest OpenBao snapshot from S3 on a fresh (empty) cluster, via an ArgoCD sync hook +type: application +version: 0.1.0 +appVersion: "0.3.0" diff --git a/apps/openbao-init/templates/restore-configmap.yaml b/apps/openbao-init/templates/restore-configmap.yaml new file mode 100644 index 0000000..93d577b --- /dev/null +++ b/apps/openbao-init/templates/restore-configmap.yaml @@ -0,0 +1,63 @@ +{{- if .Values.restore.enabled }} +apiVersion: v1 +kind: ConfigMap +metadata: + name: openbao-init-restore + namespace: {{ .Release.Namespace }} + annotations: + # Normal resource (see serviceaccount.yaml). Wave 0 → exists before the Job. + argocd.argoproj.io/sync-wave: "0" +data: + restore.sh: | + #!/usr/bin/env sh + # Restore the latest OpenBao snapshot from S3 on a fresh (empty) cluster. + # Mirrors the upstream backup script (bao-snapshot.sh), same image. + # Idempotent: never touches an already-initialized cluster. + set -eu + + # 1. Wait for the OpenBao API. `openbao` syncs in parallel (separate + # Application); its pod comes up 0/1 (uninitialized) but the `openbao` + # Service publishes not-ready addresses, so the API is reachable. `bao + # status` returns rc 1 on a connection error, rc 2 when reachable-but- + # sealed/uninitialized. Budget ~10 min for a cold start. + i=0 + while true; do + # Capture the real exit code: `|| rc=$?` both shields `set -e` and avoids + # the `if`-without-`else` trap that would reset $? to 0. rc 0 = unsealed, + # rc 2 = reachable but sealed/uninitialized — both mean the API is up. + rc=0 + bao status >/dev/null 2>&1 || rc=$? + if [ "$rc" = "0" ] || [ "$rc" = "2" ]; then break; fi + i=$((i + 1)) + if [ "$i" -gt 120 ]; then echo "timeout waiting for OpenBao API"; exit 1; fi + echo "waiting for OpenBao API..." + sleep 5 + done + + # 2. Idempotence guard: exit 0 of `init -status` means already initialized. + # Never clobber a live cluster (protects against selfHeal re-syncs). + if bao operator init -status >/dev/null 2>&1; then + echo "OpenBao already initialized — nothing to restore." + exit 0 + fi + + # 3. Locate the latest snapshot. Names sort chronologically. + LATEST=$(s3cmd ls "$S3_URI/" --host="$S3_HOST" --host-bucket="$S3_BUCKET" $S3CMD_EXTRA_FLAG \ + | sort | tail -1 | awk '{print $4}') + if [ -z "$LATEST" ]; then + echo "No snapshot found in S3 — leaving node uninitialized for manual bootstrap." + exit 0 + fi + echo "Latest snapshot: $LATEST" + + # 4. Throwaway init: works on empty storage and yields a root token valid long + # enough to perform the restore. The restore replaces all storage (auth, + # policies, secrets) so this token becomes void afterwards — by design. + ROOT=$(bao operator init | grep -i 'Initial Root Token' | awk '{print $NF}') + export BAO_TOKEN="$ROOT" + + s3cmd get "$LATEST" /tmp/restore.snapshot \ + --host="$S3_HOST" --host-bucket="$S3_BUCKET" $S3CMD_EXTRA_FLAG + bao operator raft snapshot restore /tmp/restore.snapshot + echo "Restored $LATEST (throwaway root token is now void)." +{{- end }} diff --git a/apps/openbao-init/templates/restore-job.yaml b/apps/openbao-init/templates/restore-job.yaml new file mode 100644 index 0000000..e3fca1e --- /dev/null +++ b/apps/openbao-init/templates/restore-job.yaml @@ -0,0 +1,73 @@ +{{- if .Values.restore.enabled }} +apiVersion: batch/v1 +kind: Job +metadata: + name: openbao-init-restore + namespace: {{ .Release.Namespace }} + annotations: + # Normal resource, wave 1 (after SA + ConfigMap). On a fresh cluster the Job + # doesn't exist → OutOfSync → ArgoCD creates and runs it. Re-runs on a live + # cluster are prevented by the idempotence guard in restore.sh. + argocd.argoproj.io/sync-wave: "1" + # A Job's pod template is immutable; Replace lets ArgoCD recreate it cleanly + # when the spec (e.g. restore.sh) changes, instead of failing on the patch. + argocd.argoproj.io/sync-options: Replace=true +spec: + backoffLimit: 2 + template: + metadata: + labels: + app.kubernetes.io/name: openbao-init + component: restore + spec: + serviceAccountName: openbao-init + restartPolicy: Never + securityContext: + runAsNonRoot: true + runAsUser: 100 + runAsGroup: 1000 + fsGroup: 1000 + seccompProfile: + type: RuntimeDefault + containers: + - name: restore + image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}" + imagePullPolicy: {{ .Values.image.pullPolicy }} + command: ["sh", "/scripts/restore.sh"] + securityContext: + allowPrivilegeEscalation: false + capabilities: + drop: ["ALL"] + env: + - name: BAO_ADDR + value: {{ .Values.baoAddr | quote }} + - name: S3_HOST + value: {{ .Values.s3.host | quote }} + - name: S3_BUCKET + value: {{ .Values.s3.bucket | quote }} + - name: S3_URI + value: {{ .Values.s3.uri | quote }} + - name: S3CMD_EXTRA_FLAG + value: {{ .Values.s3.extraFlag | quote }} + - name: AWS_ACCESS_KEY_ID + valueFrom: + secretKeyRef: + name: {{ .Values.s3CredentialsSecret }} + key: AWS_ACCESS_KEY_ID + - name: AWS_SECRET_ACCESS_KEY + valueFrom: + secretKeyRef: + name: {{ .Values.s3CredentialsSecret }} + key: AWS_SECRET_ACCESS_KEY + volumeMounts: + - name: scripts + mountPath: /scripts + - name: tmp + mountPath: /tmp + volumes: + - name: scripts + configMap: + name: openbao-init-restore + - name: tmp + emptyDir: {} +{{- end }} diff --git a/apps/openbao-init/templates/serviceaccount.yaml b/apps/openbao-init/templates/serviceaccount.yaml new file mode 100644 index 0000000..ed40370 --- /dev/null +++ b/apps/openbao-init/templates/serviceaccount.yaml @@ -0,0 +1,11 @@ +{{- if .Values.restore.enabled }} +apiVersion: v1 +kind: ServiceAccount +metadata: + name: openbao-init + namespace: {{ .Release.Namespace }} + annotations: + # Normal resource (not a hook): an all-hooks Application never triggers a sync + # and its hooks never run. Wave 0 → created before the Job (wave 1). + argocd.argoproj.io/sync-wave: "0" +{{- end }} diff --git a/apps/openbao-init/values-local.yaml b/apps/openbao-init/values-local.yaml new file mode 100644 index 0000000..3297c46 --- /dev/null +++ b/apps/openbao-init/values-local.yaml @@ -0,0 +1,7 @@ +# Local cluster overrides. Mirrors platform/local/openbao.yml snapshotAgent.config +# so the restore reads from the same bucket the backups are pushed to. +s3: + host: s3.fr-par.scw.cloud + bucket: backup-dev-id + uri: s3://backup-dev-id + extraFlag: "--server-side-encryption -v" diff --git a/apps/openbao-init/values-scaleway.yaml b/apps/openbao-init/values-scaleway.yaml new file mode 100644 index 0000000..3297c46 --- /dev/null +++ b/apps/openbao-init/values-scaleway.yaml @@ -0,0 +1,7 @@ +# Local cluster overrides. Mirrors platform/local/openbao.yml snapshotAgent.config +# so the restore reads from the same bucket the backups are pushed to. +s3: + host: s3.fr-par.scw.cloud + bucket: backup-dev-id + uri: s3://backup-dev-id + extraFlag: "--server-side-encryption -v" diff --git a/apps/openbao-init/values.yaml b/apps/openbao-init/values.yaml new file mode 100644 index 0000000..146ef2d --- /dev/null +++ b/apps/openbao-init/values.yaml @@ -0,0 +1,27 @@ +# Baseline values for the OpenBao restore-at-boot job. +# Per-environment S3 settings live in values-.yaml. + +# Set to false to render the chart as a no-op (e.g. environments without snapshots). +restore: + enabled: true + +# Image carrying BOTH `bao` and `s3cmd` — same image the snapshot agent uses, +# so the restore job mirrors the upstream backup script (bao-snapshot.sh). +image: + repository: ghcr.io/openbao/openbao-snapshot-agent + tag: "0.3.0" + pullPolicy: IfNotPresent + +# In-cluster OpenBao API. Matches the snapshot agent's BAO_ADDR. +baoAddr: http://openbao.openbao.svc:8200 + +# S3 credentials Secret (AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY). +# Created out-of-band by terraform, shared with the snapshot agent. +s3CredentialsSecret: scaleway-s3-credentials + +# S3 location of the snapshots — overridden per environment. +s3: + host: "" + bucket: "" + uri: "" + extraFlag: "" diff --git a/clusters/local/templates/openbao-init.yaml b/clusters/local/templates/openbao-init.yaml new file mode 100644 index 0000000..f1802f8 --- /dev/null +++ b/clusters/local/templates/openbao-init.yaml @@ -0,0 +1,33 @@ +apiVersion: argoproj.io/v1alpha1 +kind: Application +metadata: + name: openbao-init + namespace: argocd + annotations: + # Wave 0 (parallel with `platform`, NOT after it): on a from-scratch cluster + # openbao starts uninitialized → sealed → 0/1 NotReady → the `openbao` app + # never reaches Healthy until restored. Gating this behind platform's health + # would deadlock. Running in parallel lets the restore job initialize openbao + # while it is still 0/1 (the `openbao` Service publishes not-ready addresses). + argocd.argoproj.io/sync-wave: "0" +spec: + project: default + + source: + repoURL: {{ .Values.repoURL }} + targetRevision: {{ .Values.revision }} + path: apps/openbao-init + helm: + valueFiles: + - values-local.yaml + + destination: + server: https://kubernetes.default.svc + namespace: openbao + + syncPolicy: + automated: + prune: true + selfHeal: true + syncOptions: + - CreateNamespace=true diff --git a/clusters/scaleway/templates/openbao-init.yaml b/clusters/scaleway/templates/openbao-init.yaml new file mode 100644 index 0000000..7cb72cf --- /dev/null +++ b/clusters/scaleway/templates/openbao-init.yaml @@ -0,0 +1,28 @@ +apiVersion: argoproj.io/v1alpha1 +kind: Application +metadata: + name: openbao-init + namespace: argocd + annotations: + argocd.argoproj.io/sync-wave: "2" +spec: + project: default + + source: + repoURL: {{ .Values.repoURL }} + targetRevision: {{ .Values.revision }} + path: apps/openbao-init + helm: + valueFiles: + - values-scaleway.yaml + + destination: + server: https://kubernetes.default.svc + namespace: openbao + + syncPolicy: + automated: + prune: true + selfHeal: true + syncOptions: + - CreateNamespace=true diff --git a/mise.toml b/mise.toml new file mode 100644 index 0000000..0d22634 --- /dev/null +++ b/mise.toml @@ -0,0 +1,2 @@ +[tools] +helm = "latest" diff --git a/platform/local/external-secrets.yml b/platform/local/external-secrets.yml new file mode 100644 index 0000000..e1db44c --- /dev/null +++ b/platform/local/external-secrets.yml @@ -0,0 +1,26 @@ +apiVersion: argoproj.io/v1alpha1 +kind: Application +metadata: + name: external-secrets + namespace: argocd + annotations: + argocd.argoproj.io/sync-wave: "1" +spec: + project: default + + source: + repoURL: https://external-secrets.io + chart: external-secrets + targetRevision: 2.6.0 + + destination: + server: https://kubernetes.default.svc + namespace: external-secrets + + syncPolicy: + automated: + prune: true + selfHeal: true + syncOptions: + - CreateNamespace=true + - ServerSideApply=true diff --git a/platform/local/openbao.yml b/platform/local/openbao.yml new file mode 100644 index 0000000..ffc04b3 --- /dev/null +++ b/platform/local/openbao.yml @@ -0,0 +1,75 @@ +apiVersion: argoproj.io/v1alpha1 +kind: Application +metadata: + name: openbao + namespace: argocd + annotations: + argocd.argoproj.io/sync-wave: "0" +spec: + project: default + + source: + repoURL: https://openbao.github.io/openbao-helm + chart: openbao + targetRevision: 0.28.4 + helm: + values: | + server: + # dev: + # enabled: true + ha: + enabled: false + # AWS credentials the awskms seal authenticates with. OpenBao must read + # these at startup to unseal — they cannot come from OpenBao itself + # (chicken-and-egg). Secret created out-of-band by 02-cluster/local + # terraform; values originate from 03-backup/scaleway kms.tf. + extraSecretEnvironmentVars: + - envName: AWS_ACCESS_KEY_ID + secretName: openbao-unseal-aws + secretKey: AWS_ACCESS_KEY_ID + - envName: AWS_SECRET_ACCESS_KEY + secretName: openbao-unseal-aws + secretKey: AWS_SECRET_ACCESS_KEY + standalone: + config: | + ui = true + listener "tcp" { + address = "0.0.0.0:8200" + tls_disable = true + } + storage "raft" { + path = "/openbao/data" + node_id = "openbao-0" + } + # Auto-unseal via AWS KMS. region + kms_key_id are non-secret + # (the IAM creds above gate access). On first `bao operator init` + # this yields RECOVERY keys, not unseal keys — store them safely. + seal "awskms" { + region = "eu-west-3" + kms_key_id = "367e11de-75f4-4df0-a038-b6cf06ae2ead" + } + + + + snapshotAgent: + enabled: true + schedule: "0 * * * *" # Every hours, at 0min. + s3CredentialsSecret: "scaleway-s3-credentials" + config: + s3Host: "s3.fr-par.scw.cloud" + s3Bucket: "backup-dev-id" + s3Uri: "s3://backup-dev-id" + s3ExpireDays: "14" + s3cmdExtraFlag: "--server-side-encryption -v" + # baoAuthPath: "kubernetes" + # baoRole: "snapshot" + destination: + server: https://kubernetes.default.svc + namespace: openbao + + syncPolicy: + automated: + prune: true + selfHeal: true + syncOptions: + - CreateNamespace=true diff --git a/platform/scaleway/external-secrets.yml b/platform/scaleway/external-secrets.yml new file mode 100644 index 0000000..e1db44c --- /dev/null +++ b/platform/scaleway/external-secrets.yml @@ -0,0 +1,26 @@ +apiVersion: argoproj.io/v1alpha1 +kind: Application +metadata: + name: external-secrets + namespace: argocd + annotations: + argocd.argoproj.io/sync-wave: "1" +spec: + project: default + + source: + repoURL: https://external-secrets.io + chart: external-secrets + targetRevision: 2.6.0 + + destination: + server: https://kubernetes.default.svc + namespace: external-secrets + + syncPolicy: + automated: + prune: true + selfHeal: true + syncOptions: + - CreateNamespace=true + - ServerSideApply=true diff --git a/platform/scaleway/openbao.yml b/platform/scaleway/openbao.yml new file mode 100644 index 0000000..897fe9d --- /dev/null +++ b/platform/scaleway/openbao.yml @@ -0,0 +1,41 @@ +apiVersion: argoproj.io/v1alpha1 +kind: Application +metadata: + name: openbao + namespace: argocd + annotations: + argocd.argoproj.io/sync-wave: "0" +spec: + project: default + + source: + repoURL: https://openbao.github.io/openbao-helm + chart: openbao + targetRevision: 0.28.4 + helm: + values: | + server: + ha: + enabled: false + standalone: + config: | + ui = true + listener "tcp" { + address = "0.0.0.0:8200" + tls_disable = true + } + storage "raft" { + path = "/openbao/data" + node_id = "openbao-0" + } + + destination: + server: https://kubernetes.default.svc + namespace: openbao + + syncPolicy: + automated: + prune: true + selfHeal: true + syncOptions: + - CreateNamespace=true diff --git a/specs/001-replace-infisical-oss-secrets/checklists/requirements.md b/specs/001-replace-infisical-oss-secrets/checklists/requirements.md new file mode 100644 index 0000000..ef0ab25 --- /dev/null +++ b/specs/001-replace-infisical-oss-secrets/checklists/requirements.md @@ -0,0 +1,35 @@ +# Specification Quality Checklist: Replace Infisical with OSS Secrets Backend + +**Purpose**: Validate specification completeness and quality before proceeding to planning +**Created**: 2026-06-21 +**Feature**: [spec.md](../spec.md) + +## Content Quality + +- [x] No implementation details (languages, frameworks, APIs) +- [x] Focused on user value and business needs +- [x] Written for non-technical stakeholders +- [x] All mandatory sections completed + +## Requirement Completeness + +- [x] No [NEEDS CLARIFICATION] markers remain +- [x] Requirements are testable and unambiguous +- [x] Success criteria are measurable +- [x] Success criteria are technology-agnostic (no implementation details) +- [x] All acceptance scenarios are defined +- [x] Edge cases are identified +- [x] Scope is clearly bounded +- [x] Dependencies and assumptions identified + +## Feature Readiness + +- [x] All functional requirements have clear acceptance criteria +- [x] User scenarios cover primary flows +- [x] Feature meets measurable outcomes defined in Success Criteria +- [x] No implementation details leak into specification + +## Notes + +- Tool choice (OpenBao / Vault OSS / Scaleway Secret Manager) is explicitly deferred to `/speckit-plan` per FR-008 and constitution Principle II — this is intentional, not a gap. +- All items pass. Spec is ready for `/speckit-plan`. diff --git a/specs/001-replace-infisical-oss-secrets/contracts/argocd-applications.md b/specs/001-replace-infisical-oss-secrets/contracts/argocd-applications.md new file mode 100644 index 0000000..8d3797a --- /dev/null +++ b/specs/001-replace-infisical-oss-secrets/contracts/argocd-applications.md @@ -0,0 +1,184 @@ +# Contract: ArgoCD Application Manifests + +**Feature**: 001-replace-infisical-oss-secrets +**Date**: 2026-06-21 + +This document defines the expected shape of ArgoCD Application manifests introduced by this feature. These are the GitOps "contracts" — the declarative interface between the gitops repo and ArgoCD. + +--- + +## Application: openbao (platform tool) + +Deployed under `platform//openbao.yml`. Points to the official OpenBao Helm chart. + +```yaml +# platform/local/openbao.yml (and platform/scaleway/openbao.yml with env-specific overrides) +apiVersion: argoproj.io/v1alpha1 +kind: Application +metadata: + name: openbao + namespace: argocd +spec: + project: default + source: + repoURL: https://openbao.github.io/openbao-helm + chart: openbao + targetRevision: # pin to exact version + helm: + values: | + server: + ha: + enabled: false # S3 backend does not support HA on Scaleway + standalone: + config: | + ui = true + listener "tcp" { + address = "0.0.0.0:8200" + tls_disable = true # TLS terminated at ingress or cluster mesh + } + storage "s3" { + bucket = "" + region = "fr-par" + endpoint = "https://s3.fr-par.scw.cloud" + s3_force_path_style = true + # credentials sourced from env vars: AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY + } + extraEnv: + - name: AWS_ACCESS_KEY_ID + valueFrom: + secretKeyRef: + name: scaleway-s3-credentials + key: access_key + - name: AWS_SECRET_ACCESS_KEY + valueFrom: + secretKeyRef: + name: scaleway-s3-credentials + key: secret_key + destination: + server: https://kubernetes.default.svc + namespace: openbao + syncPolicy: + automated: + prune: true + selfHeal: true + syncOptions: + - CreateNamespace=true +``` + +**Constraints**: +- `targetRevision` MUST be pinned to a specific chart version (no `latest` or `*`) +- `tls_disable = true` is acceptable for local; staging should configure TLS at ingress level +- `scaleway-s3-credentials` Secret MUST exist before first ArgoCD sync (created by infra Terraform) + +--- + +## Application: openbao-init (init/unseal job) + +Deployed under `clusters//templates/openbao-init.yaml`. Points to an in-house Helm chart at `apps/openbao-init/`. + +```yaml +# clusters/local/templates/openbao-init.yaml +apiVersion: argoproj.io/v1alpha1 +kind: Application +metadata: + name: openbao-init + namespace: argocd +spec: + project: default + source: + repoURL: {{ .Values.repoURL }} + targetRevision: {{ .Values.revision }} + path: apps/openbao-init + helm: + valueFiles: + - values-local.yaml + destination: + server: https://kubernetes.default.svc + namespace: openbao + syncPolicy: + automated: + prune: true + selfHeal: true + syncOptions: + - CreateNamespace=true +``` + +**Constraints**: +- Must deploy after the `openbao` Application (ArgoCD sync-wave or dependency annotation) +- The init Job itself is an ArgoCD PostSync hook within the `apps/openbao-init` chart + +--- + +## Application: external-secrets (ESO) + +Deployed under `platform//external-secrets.yml`. Points to the official ESO Helm chart. + +```yaml +# platform/local/external-secrets.yml +apiVersion: argoproj.io/v1alpha1 +kind: Application +metadata: + name: external-secrets + namespace: argocd +spec: + project: default + source: + repoURL: https://charts.external-secrets.io + chart: external-secrets + targetRevision: + destination: + server: https://kubernetes.default.svc + namespace: external-secrets + syncPolicy: + automated: + prune: true + selfHeal: true + syncOptions: + - CreateNamespace=true +``` + +--- + +## Sync Ordering + +ArgoCD sync-wave annotations control deployment order: + +| Wave | Application | Why | +|------|-------------|-----| +| 0 | `openbao` (platform) | Server must start before init job | +| 1 | `openbao-init` | Init/unseal after server ready | +| 2 | `external-secrets` (ESO) | ClusterSecretStore configured after server unsealed | +| 3 | `` workloads | ExternalSecrets resolved after ESO + server ready | + +Sync wave is set via `metadata.annotations: argocd.argoproj.io/sync-wave: "N"` on each Application manifest. + +--- + +## ClusterSecretStore Contract + +After ESO is deployed and OpenBao is unsealed, a `ClusterSecretStore` CR is created to connect ESO to OpenBao: + +```yaml +apiVersion: external-secrets.io/v1beta1 +kind: ClusterSecretStore +metadata: + name: openbao +spec: + provider: + vault: + server: "http://openbao.openbao.svc.cluster.local:8200" + path: "kv" + version: "v2" + auth: + kubernetes: + mountPath: "kubernetes" + role: "external-secrets" + serviceAccountRef: + name: external-secrets + namespace: external-secrets +``` + +**Constraints**: +- `server` URL uses the cluster-internal DNS name (never an external URL for in-cluster use) +- `role` must match a Kubernetes auth role configured by the init Job +- The `ClusterSecretStore` is managed as part of the `apps/openbao-init` chart (post-unseal configuration) diff --git a/specs/001-replace-infisical-oss-secrets/contracts/secrets-consumption.md b/specs/001-replace-infisical-oss-secrets/contracts/secrets-consumption.md new file mode 100644 index 0000000..46e337f --- /dev/null +++ b/specs/001-replace-infisical-oss-secrets/contracts/secrets-consumption.md @@ -0,0 +1,121 @@ +# Contract: Secrets Consumption Interface + +**Feature**: 001-replace-infisical-oss-secrets +**Date**: 2026-06-21 + +This document defines how application workloads consume secrets from the OSS secrets backend. This is the stable interface that application teams depend on. + +--- + +## KV Secret Path Convention + +All application secrets are stored in the KV v2 engine under a namespaced path: + +``` +kv/apps// +``` + +Examples: +``` +kv/apps/demo/db-password +kv/apps/demo/api-key +kv/apps/monitoring/grafana-admin-password +``` + +**Rules**: +- Path prefix `kv/apps/` is the standard application secrets mount +- `` matches the ArgoCD Application name and Kubernetes namespace name +- Secret names use kebab-case +- No nested subpaths beyond `kv/apps//` (flat structure per app) + +--- + +## ExternalSecret Usage Pattern + +Application teams create an `ExternalSecret` CR in their namespace to sync a secret: + +```yaml +apiVersion: external-secrets.io/v1beta1 +kind: ExternalSecret +metadata: + name: + namespace: +spec: + refreshInterval: "1h" + secretStoreRef: + name: openbao + kind: ClusterSecretStore + target: + name: # name of the resulting v1/Secret + creationPolicy: Owner + data: + - secretKey: # key in the resulting v1/Secret + remoteRef: + key: apps// # path in KV v2 + property: # field within the KV secret +``` + +**Example** — syncing a database password for the `demo` app: +```yaml +apiVersion: external-secrets.io/v1beta1 +kind: ExternalSecret +metadata: + name: demo-db-credentials + namespace: demo +spec: + refreshInterval: "1h" + secretStoreRef: + name: openbao + kind: ClusterSecretStore + target: + name: demo-db-credentials + creationPolicy: Owner + data: + - secretKey: password + remoteRef: + key: apps/demo/db-credentials + property: password + - secretKey: username + remoteRef: + key: apps/demo/db-credentials + property: username +``` + +The resulting `v1/Secret` named `demo-db-credentials` is then consumed by pods via `envFrom` or `env.valueFrom.secretKeyRef` as standard Kubernetes patterns. + +--- + +## Policy Naming Convention + +Each application has a read-only policy named `-read`: + +```hcl +# Policy: demo-read +path "kv/data/apps/demo/*" { + capabilities = ["read", "list"] +} +``` + +Policies are created by the init Job at bootstrap time from a config map in `apps/openbao-init/`. + +--- + +## Auth Role Convention + +Each application namespace has a Kubernetes auth role: + +| Role name | Bound SA | Bound namespace | Policy | +|-----------|----------|----------------|--------| +| `external-secrets` | `external-secrets` | `external-secrets` | All app read policies | +| `-workload` | `default` | `` | `-read` | + +The `external-secrets` role is the primary one used by ESO. Per-app workload roles are optional for direct Vault agent usage. + +--- + +## Out of Scope (this feature) + +- Secret rotation automation (future feature) +- Dynamic database credentials (future feature) +- PKI / certificate issuance (future feature) +- Encryption-as-a-service (transit engine) (future feature) diff --git a/specs/001-replace-infisical-oss-secrets/data-model.md b/specs/001-replace-infisical-oss-secrets/data-model.md new file mode 100644 index 0000000..fe2a7e4 --- /dev/null +++ b/specs/001-replace-infisical-oss-secrets/data-model.md @@ -0,0 +1,157 @@ +# Data Model: OSS Secrets Backend + +**Feature**: 001-replace-infisical-oss-secrets +**Date**: 2026-06-21 + +This document describes the logical entities and their Kubernetes/GitOps representations for the OSS secrets backend deployment. + +--- + +## Logical Entities + +### SecretsBackend (OpenBao / Vault instance) + +The core runtime: a single-replica pod serving the secrets API. + +| Attribute | Value / Constraint | +|-----------|-------------------| +| Deployment | ArgoCD Application → external Helm chart | +| Replicas | 1 (S3 backend does not support HA on Scaleway) | +| Namespace | `openbao` | +| Ports | 8200 (API/UI), 8201 (cluster — unused in single-replica) | +| Storage | S3 backend (Scaleway Object Storage) | +| State | Persisted to S3 on every write; survives cluster destroy | +| Seal state | Starts sealed; unsealed by init Job on every boot | + +### Secret + +A named key/value pair managed by the backend. + +| Attribute | Constraint | +|-----------|-----------| +| Path | Hierarchical (e.g. `kv/apps/demo/db-password`) | +| Versioning | KV v2 engine — retains version history | +| Access control | Governed by Policy entities | +| Consumer | External Secrets Operator ExternalSecret CR | + +### Policy + +Defines what paths a given identity can read/write. + +| Attribute | Constraint | +|-----------|-----------| +| Scope | Per-namespace or per-application | +| Format | HCL (stored in-cluster; managed via init Job or Vault provider) | +| Minimum | One read-only policy per consuming namespace | + +### AuthMethod + +How workloads and operators authenticate to the backend. + +| Method | Use case | +|--------|---------| +| Kubernetes auth | In-cluster workloads and ESO; bound to ServiceAccount | +| Token | Init Job bootstrap only; root token stored in S3, rotated immediately | + +### InitJob (Kubernetes Job) + +An ArgoCD post-sync hook Job that bootstraps or unseals the backend on every cluster boot. + +| Attribute | Value | +|-----------|-------| +| Trigger | ArgoCD PostSync hook on the secrets-backend Application | +| Function | Init (first boot) or unseal (subsequent boots) | +| S3 path | `openbao/init-keys/.json` | +| Credentials | `scaleway-s3-credentials` Secret in `openbao` namespace | +| Idempotent | Yes — no-op if already initialised and unsealed | + +### S3BucketState + +The external persistence layer. One S3 path per environment. + +| Attribute | Value | +|-----------|-------| +| Provider | Scaleway Object Storage (fr-par) | +| Bucket | Shared with Task 1 S3 backup bucket (separate prefix) | +| Prefix — backend data | `openbao/data/` (written by OpenBao storage backend) | +| Prefix — init keys | `openbao/init-keys/` (written by init Job) | +| Encryption | SSE with Scaleway-managed keys | +| Access | Limited to init Job ServiceAccount (IAM policy) | + +### ExternalSecretsCR + +The Kubernetes Custom Resource that syncs a secret from the backend into a native `v1/Secret`. + +| Attribute | Value | +|-----------|-------| +| Kind | `ExternalSecret` (External Secrets Operator) | +| Source | SecretStore pointing to OpenBao Kubernetes auth | +| Target | Native `v1/Secret` in the consuming namespace | +| Refresh interval | Configurable per secret (default: 1h) | + +--- + +## Kubernetes Resource Hierarchy + +``` +Namespace: openbao +├── Deployment: openbao # main server pod (1 replica) +├── Service: openbao # ClusterIP :8200 +├── ServiceAccount: openbao # for Kubernetes auth +├── ConfigMap: openbao-config # HCL config (S3 backend, listener) +├── Secret: scaleway-s3-credentials # created by infra Terraform +└── Job: openbao-init (PostSync hook) # init/unseal on each boot + +Namespace: openbao-system (ESO) +├── Deployment: external-secrets # ESO controller +├── ClusterSecretStore: openbao # cluster-wide ESO SecretStore +└── (per app namespace) ExternalSecret # syncs secret into v1/Secret + +Namespace: (e.g. demo) +└── Secret: # synced by ESO from OpenBao +``` + +--- + +## State Transitions + +### First Boot (initialisation) +``` +OpenBao Pod starts → sealed (no data in S3) + → InitJob: detect "not initialized" + → Call /v1/sys/init (5 key shares, threshold 3) + → Store root token + unseal keys to S3 (openbao/init-keys/.json) + → Call /v1/sys/unseal × 3 (with 3 of 5 key shares) + → OpenBao: unsealed ✅ + → InitJob: enable KV v2, Kubernetes auth method, create policies + → InitJob: revoke root token (use AppRole or K8s auth for subsequent access) +``` + +### Subsequent Boots (cluster recreate or pod restart) +``` +OpenBao Pod starts → sealed (data in S3, keys in S3) + → InitJob: detect "initialized but sealed" + → Read unseal keys from S3 + → Call /v1/sys/unseal × 3 + → OpenBao: unsealed ✅ + → InitJob: no-op on KV/auth (already configured) +``` + +### Secret Consumption +``` +Workload pod scheduled + → ESO ExternalSecret reconciles + → ESO authenticates to OpenBao via Kubernetes auth (ServiceAccount token) + → ESO reads secret from KV v2 path + → ESO writes/updates v1/Secret in workload namespace + → Workload reads from v1/Secret (env var or volume mount) +``` + +--- + +## Validation Rules + +- `openbao/init-keys/.json` MUST exist in S3 before cluster recreate can self-heal (created on first init) +- `scaleway-s3-credentials` Secret MUST be present in `openbao` namespace before ArgoCD sync runs the PostSync hook +- Init Job MUST be idempotent — multiple runs must not re-initialise an already-initialised backend +- Root token MUST be revoked after initial setup; subsequent admin access via Kubernetes auth + admin policy diff --git a/specs/001-replace-infisical-oss-secrets/plan.md b/specs/001-replace-infisical-oss-secrets/plan.md new file mode 100644 index 0000000..f2d879c --- /dev/null +++ b/specs/001-replace-infisical-oss-secrets/plan.md @@ -0,0 +1,162 @@ +# Implementation Plan: Replace Infisical with OSS Secrets Backend + +**Branch**: `001-replace-infisical-oss-secrets` | **Date**: 2026-06-21 | **Spec**: [spec.md](spec.md) + +**Input**: Feature specification from `specs/001-replace-infisical-oss-secrets/spec.md` + +## Summary + +Deploy an OSS secrets backend (OpenBao recommended, final tool chosen by human per constitution Principle II) as an ArgoCD-managed platform Application. Backend state is persisted to Scaleway Object Storage (S3) so secrets survive full cluster destroy/recreate. A PostSync Kubernetes Job handles init and unseal on every cluster boot from S3-stored keys. External Secrets Operator (ESO) bridges the backend to workloads via native Kubernetes Secrets. Infisical is not deployed in this repo; no live migration is required. + +## Technical Context + +**Language/Version**: YAML / HCL config only (no application code — GitOps repo) + +**Primary Dependencies**: +- OpenBao Helm chart (or HashiCorp Vault Helm chart — human decides, see research.md) +- External Secrets Operator Helm chart (`https://charts.external-secrets.io`) +- Scaleway Object Storage (S3-compatible, fr-par region) — provisioned by Task 1 + +**Storage**: Scaleway Object Storage (S3 backend for OpenBao state + `openbao/init-keys/` prefix for unseal keys) + +**Testing**: `helm lint`, `helm template`, `kubectl apply --dry-run=client`, CI pipeline (minikube ephemeral cluster) + +**Target Platform**: Kubernetes — `local` (minikube) and `scaleway` (Kapsule) + +**Project Type**: GitOps / platform infrastructure + +**Performance Goals**: Secret retrieval < 5 seconds from workload perspective (SC-002) + +**Constraints**: +- S3 backend does not support HA on Scaleway (no DynamoDB-compatible locking) → single replica +- `scaleway-s3-credentials` Kubernetes Secret must be created by infra Terraform before ArgoCD first sync +- Auto-unseal via Scaleway KMS is not currently viable (no PKCS#11/KMIP endpoint) + +**Scale/Scope**: `local` + `scaleway` environments; single-replica secrets backend per cluster + +## Constitution Check + +*GATE: Must pass before implementation. Re-checked after design.* + +| Principle | Status | Notes | +|-----------|--------|-------| +| I. OSS-first | ✅ if OpenBao; ⚠️ if Vault BSL; ❌ if Scaleway SM | OpenBao (MPL 2.0) satisfies fully. Vault BSL acceptable for self-hosted. Scaleway SM requires explicit human approval. | +| II. Human decides tooling | ✅ | Three alternatives presented in research.md; human selects before implementation | +| III. Enterprise-grade | ✅ | Secrets management with audit log, policy-based access, namespace isolation | +| IV. Validation by steps | ✅ | Depends on Task 1 (S3 bucket) ✅; CI validates restore scenario (SC-001) | +| V. GitOps-first | ✅ | All config in this repo; deployed via ArgoCD; zero manual kubectl applies | +| VI. Scaleway-sovereign | ✅ | State on Scaleway S3; backend self-hosted on Scaleway Kapsule | +| VII. CI-verifiable acceptance | ✅ | Destroy/recreate scenario executable in CI (see quickstart.md) | + +**Gate result**: PASS — no violations, assuming OSS tool (OpenBao or Vault BSL) is selected. + +## Project Structure + +### Documentation (this feature) + +```text +specs/001-replace-infisical-oss-secrets/ +├── plan.md # This file +├── spec.md # Feature specification +├── research.md # Phase 0: tool comparison, auto-unseal analysis +├── data-model.md # Phase 1: entities, K8s resource hierarchy, state transitions +├── quickstart.md # Phase 1: end-to-end validation scenarios (incl. destroy/recreate) +├── contracts/ +│ ├── argocd-applications.md # ArgoCD Application manifest shapes + sync ordering +│ └── secrets-consumption.md # ExternalSecret pattern, KV path convention, policies +└── tasks.md # Phase 2 output (/speckit-tasks command) +``` + +### Source Code Layout (gitops repo root) + +```text +platform/ +├── local/ +│ ├── ingress-nginx.yml # existing +│ ├── kube-prometheus-stack.yml # existing +│ ├── openbao.yml # NEW: OpenBao ArgoCD Application (external Helm) +│ └── external-secrets.yml # NEW: ESO ArgoCD Application (external Helm) +└── scaleway/ + ├── kube-prometheus-stack.yml # existing + ├── openbao.yml # NEW: env-specific overrides (same chart, scaleway values) + └── external-secrets.yml # NEW: ESO for scaleway + +apps/ +└── openbao-init/ # NEW: in-house Helm chart for init/unseal Job + ├── Chart.yaml + ├── values.yaml # defaults: openbao address, S3 bucket name + ├── values-local.yaml # local overrides + ├── values-scaleway.yaml # scaleway overrides + └── templates/ + ├── serviceaccount.yaml # SA for the init Job + ├── role.yaml # RBAC to read scaleway-s3-credentials Secret + ├── rolebinding.yaml + ├── configmap-policies.yaml # OpenBao policies (HCL) per namespace + ├── job-init.yaml # PostSync hook Job (init + unseal + configure) + └── clustersecretstore.yaml # ESO ClusterSecretStore pointing to OpenBao + +clusters/ +├── local/ +│ └── templates/ +│ ├── demo.yaml # existing +│ ├── platform.yaml # existing (picks up new openbao.yml + eso.yml) +│ └── openbao-init.yaml # NEW: ArgoCD Application for apps/openbao-init +└── scaleway/ + └── templates/ + ├── platform.yaml # existing (picks up new platform/scaleway/ files) + └── openbao-init.yaml # NEW: ArgoCD Application for apps/openbao-init +``` + +**Structure Decision**: Platform-level tools (OpenBao, ESO) follow the `platform//` pattern (single YAML, external Helm chart), consistent with `ingress-nginx` and `kube-prometheus-stack`. The init/unseal Job is custom logic → in-house Helm chart under `apps/openbao-init/`. This matches the `apps/demo/` pattern for in-house charts. + +## Complexity Tracking + +No constitution violations. No complexity justification required. + +--- + +## Implementation Phases (for `/speckit-tasks`) + +### Phase A — Tool Decision ✅ DECIDED + +**OpenBao selected (2026-06-21)** — MPL 2.0, CNCF sandbox, Vault-compatible API. +Helm chart: `https://openbao.github.io/openbao-helm`, chart name: `openbao`. + +### Phase B — Platform Application Manifests + +1. Write `platform/local/openbao.yml` — ArgoCD Application pointing to chosen Helm chart +2. Write `platform/scaleway/openbao.yml` — same chart, env-specific values +3. Write `platform/local/external-secrets.yml` — ESO Helm chart +4. Write `platform/scaleway/external-secrets.yml` +5. Validate with `helm lint` and `kubectl apply --dry-run=client` + +### Phase C — Init/Unseal Helm Chart (`apps/openbao-init/`) + +1. Scaffold Helm chart: `Chart.yaml`, `values.yaml`, `values-local.yaml`, `values-scaleway.yaml` +2. Write PostSync hook Job: shell script logic for init/unseal (reads from S3, calls OpenBao API) +3. Write ServiceAccount + RBAC for the Job +4. Write `configmap-policies.yaml` with per-app HCL policies +5. Write `clustersecretstore.yaml` (ESO ClusterSecretStore) +6. Validate with `helm lint apps/openbao-init/` and `helm template` + +### Phase D — Cluster Application Templates + +1. Write `clusters/local/templates/openbao-init.yaml` — ArgoCD Application for the init chart +2. Write `clusters/scaleway/templates/openbao-init.yaml` +3. Add ArgoCD sync-wave annotations to control ordering (openbao → openbao-init → ESO → apps) +4. Validate with `helm template loc clusters/local/ --set revision=` + +### Phase E — Local Validation (SC-001 through SC-005) + +1. Start minikube, manually create `scaleway-s3-credentials` Secret +2. Push branch, let ArgoCD reconcile (or manual sync) +3. Run Scenario 1 from quickstart.md: verify OpenBao unsealed, write canary secret +4. Run Scenario 2: `minikube delete`, `minikube start`, verify canary secret restored +5. Run Scenario 3: verify ESO syncs secret to workload namespace +6. Run Scenario 4: verify no Infisical resources + +### Phase F — CI Pipeline (SC-007 / constitution Principle VII) + +1. Write GitHub Actions workflow (or extend existing) to execute destroy/recreate scenario +2. Pipeline uses CI-injected Scaleway credentials for the `scaleway-s3-credentials` Secret +3. Assert all success criteria in CI (no manual validation) diff --git a/specs/001-replace-infisical-oss-secrets/quickstart.md b/specs/001-replace-infisical-oss-secrets/quickstart.md new file mode 100644 index 0000000..62ddde1 --- /dev/null +++ b/specs/001-replace-infisical-oss-secrets/quickstart.md @@ -0,0 +1,175 @@ +# Quickstart: Validate OSS Secrets Backend + +**Feature**: 001-replace-infisical-oss-secrets +**Date**: 2026-06-21 + +Step-by-step guide to validate the secrets backend end-to-end, including the cluster destroy/recreate scenario required by SC-001. + +--- + +## Prerequisites + +1. Minikube running: `minikube status` +2. ArgoCD running: `kubectl --context minikube get pods -n argocd` +3. Scaleway S3 bucket (from Task 1) accessible with valid credentials +4. The `scaleway-s3-credentials` Kubernetes Secret exists in the `openbao` namespace (created by infra Terraform or manually for local testing): + +```bash +kubectl --context minikube create namespace openbao --dry-run=client -o yaml | kubectl apply -f - +kubectl --context minikube create secret generic scaleway-s3-credentials \ + --namespace openbao \ + --from-literal=access_key= \ + --from-literal=secret_key= +``` + +--- + +## Scenario 1: Initial Deployment (User Story 1 — P1) + +**Goal**: Verify secrets backend is deployed, initialised, and accessible. + +```bash +# 1. Sync the platform Application (triggers openbao deployment) +kubectl --context minikube get applications -n argocd + +# 2. Wait for openbao pod to be running +kubectl --context minikube get pods -n openbao -w + +# 3. Verify init Job completed successfully +kubectl --context minikube get jobs -n openbao +kubectl --context minikube logs job/openbao-init -n openbao + +# 4. Check OpenBao seal status (should report: Sealed false) +kubectl --context minikube exec -n openbao deploy/openbao -- \ + bao status + +# 5. Write a test secret (requires a valid token from the init Job logs or S3) +export VAULT_ADDR="http://$(kubectl --context minikube get svc openbao -n openbao -o jsonpath='{.spec.clusterIP}'):8200" +bao kv put kv/apps/test/smoke-test value="hello" + +# 6. Read it back +bao kv get kv/apps/test/smoke-test +``` + +**Expected**: `value = hello` printed. No errors. `bao status` shows `Sealed: false`. + +--- + +## Scenario 2: Destroy and Recreate (User Story 2 — P1 for SC-001) + +**Goal**: Verify secrets survive full cluster lifecycle. + +```bash +# Step 1: Write a canary secret before destroy +bao kv put kv/apps/test/canary value="survive-destroy-$(date +%s)" +export CANARY=$(bao kv get -field=value kv/apps/test/canary) +echo "Canary: $CANARY" + +# Step 2: Destroy the cluster +minikube delete + +# Step 3: Recreate cluster and bootstrap ArgoCD +# (follow infra repo bootstrap procedure — applies Terraform or minikube + ArgoCD seed) +minikube start +# ... re-apply ArgoCD seed Application pointing to this gitops repo ... + +# Step 4: Ensure S3 credentials Secret exists in the new cluster (infra bootstrap should create it) +kubectl --context minikube get secret scaleway-s3-credentials -n openbao + +# Step 5: Wait for ArgoCD to reconcile and init Job to complete +kubectl --context minikube get jobs -n openbao -w +kubectl --context minikube logs job/openbao-init -n openbao + +# Step 6: Verify OpenBao is unsealed +kubectl --context minikube exec -n openbao deploy/openbao -- bao status + +# Step 7: Read back canary secret — must match pre-destroy value +bao kv get -field=value kv/apps/test/canary +# Expected output: same value as $CANARY above +``` + +**Expected**: Canary secret value matches. `bao status` shows `Sealed: false`. No manual intervention required between step 2 and step 7. + +--- + +## Scenario 3: ESO Secret Sync (Integration Validation) + +**Goal**: Verify External Secrets Operator syncs a secret into a workload namespace. + +```bash +# 1. Apply a test ExternalSecret (see contracts/secrets-consumption.md for format) +kubectl --context minikube apply -f - </` pattern +- Durable state must survive full cluster destroy/recreate on Scaleway Object Storage (S3-compatible) +- Must auto-recover without operator intervention after cluster recreate +- OSS-first (constitution Principle I); human decides tooling (Principle II) + +--- + +## Decision 1: Tool Choice + +*Human decision required before implementation can proceed.* + +### Option A — OpenBao + +- **Decision**: Use OpenBao as the secrets backend +- **Rationale**: True MPL 2.0 open-source fork of HashiCorp Vault, created after Vault's BSL license change in 2023. Drop-in API-compatible with Vault. Active CNCF sandbox project. Free for all uses with no commercial restrictions. +- **License**: Mozilla Public License 2.0 — satisfies constitution Principle I (OSS-first) with zero ambiguity +- **Helm chart**: `https://openbao.github.io/openbao-helm` → chart `openbao` +- **S3 backend**: Supported via Vault-compatible S3 storage configuration (works with Scaleway Object Storage using `s3_force_path_style = true`) +- **Ecosystem**: Smaller than HashiCorp Vault but growing; compatible with Vault clients, SDKs, and External Secrets Operator +- **Risk**: Less mature than Vault; smaller community, fewer battle-tested production deployments + +### Option B — HashiCorp Vault OSS + +- **Decision**: Use HashiCorp Vault (latest OSS build under BSL 1.1) +- **Rationale**: The most widely adopted secrets backend in the Kubernetes ecosystem. Largest community, most documentation, richest integrations. BSL 1.1 permits self-hosted use within your own organisation without restriction. +- **License**: Business Source License 1.1 (since v1.14, 2023) — self-hosted internal use is permitted; redistribution as a service is not. Satisfies constitution Principle I in spirit (no paid tier for our use case) but not in letter (not "open source" by OSI definition). +- **Helm chart**: `https://helm.releases.hashicorp.com` → chart `vault` +- **S3 backend**: Fully supported, production-tested +- **Ecosystem**: Largest in class; External Secrets Operator, Vault Agent Injector, CSI provider, many tutorials +- **Risk**: BSL ambiguity if the organisation's use case grows; HashiCorp (now IBM) may tighten terms + +### Option C — Scaleway Secret Manager + +- **Decision**: Use Scaleway's managed Secret Manager, integrated via External Secrets Operator +- **Rationale**: Fully managed, zero operational burden. Native Scaleway sovereignty (Principle VI). No cluster-level deployment needed; secrets live outside the cluster lifecycle by design. +- **License**: Proprietary SaaS (Scaleway managed service) +- **Integration**: External Secrets Operator (ESO) with Scaleway provider +- **Pricing**: Charged per API call + per-secret storage (consult current Scaleway pricing page — not included here as prices change) +- **Risk**: Violates constitution Principle I (not open-source, not self-hosted). Ongoing per-secret cost. Vendor-dependent. +- **Note**: Option C REQUIRES explicit human approval per constitution Principle I ("Aucun service tiers payant n'est adopté sans l'accord explicite de l'humain"). Do not select without confirming cost acceptability. + +### Comparison Matrix + +| Criterion | OpenBao (A) | Vault OSS/BSL (B) | Scaleway SM (C) | +|-----------|-------------|-------------------|-----------------| +| OSS-first (P.I) | ✅ MPL 2.0 | ⚠️ BSL 1.1 | ❌ Proprietary | +| Human approval needed | No | No | Yes (cost + non-OSS) | +| Self-hosted | Yes | Yes | No (managed) | +| Paid tier risk | None | None | Yes (API billing) | +| S3 state backend | ✅ | ✅ | N/A (managed) | +| Auto-unseal options | KMS / init job | KMS / init job | N/A | +| Helm chart maturity | Medium | High | N/A (ESO) | +| API compatibility | Vault-compatible | Reference | Scaleway API | +| Community size | Medium (CNCF) | Large | Small | +| Scaleway sovereign (P.VI) | Self-hosted on SCW | Self-hosted on SCW | ✅ Managed on SCW | +| **Constitution compliance** | **Full** | **Partial** | **Requires approval** | + +**Decision (confirmed 2026-06-21)**: **Option A — OpenBao**. MPL 2.0 fully satisfies constitution Principle I with no ambiguity. API-compatible with Vault so tooling (ESO, Vault Agent) works unchanged. Active CNCF project with a clear governance path. + +--- + +## Decision 2: Storage Backend Strategy + +*Decision: Use S3 storage backend (Scaleway Object Storage)* + +OpenBao and Vault both support the `s3` storage stanza, which is compatible with any S3-API-compliant object store. Scaleway Object Storage is S3-compatible and will be used as the primary storage backend. + +**Configuration parameters** (Scaleway Object Storage, Paris region): +```hcl +storage "s3" { + bucket = "" + region = "fr-par" + endpoint = "https://s3.fr-par.scw.cloud" + access_key = "" + secret_key = "" + s3_force_path_style = true +} +``` + +**Limitation**: The S3 storage backend does not support high-availability (HA) locking without DynamoDB. Scaleway Object Storage does not offer a DynamoDB-compatible locking API. This means HA mode (`ha_enabled = true`) is not supported with the S3 backend on Scaleway. A single-replica deployment is required. + +**Alternative considered — Raft (integrated storage)**: +- Raft is self-contained, no S3 dependency for runtime +- Requires PersistentVolumeClaim (PVC) that would be lost on full cluster destroy +- On cluster recreate, a snapshot restore from S3 is needed (more complex) +- Rejected for this use case: S3 backend is simpler and directly satisfies "state persisted to S3" + +--- + +## Decision 3: Auto-Unseal / Init Strategy + +*Decision: S3-backed init Job (Option 3A below)* + +The core challenge: on a fresh cluster, OpenBao starts in a sealed state. It needs unseal keys to become operational. Human intervention would break SC-001 (CI-verifiable auto-restore). Three options evaluated: + +### Option 3A: S3-backed init Job (recommended) + +A Kubernetes Job runs as a post-sync ArgoCD hook. It: +1. Checks if OpenBao is already initialised (`/v1/sys/health`) +2. If **not initialised**: calls `/v1/sys/init` → stores root token + unseal keys as JSON in an S3 object (prefixed `openbao/init-keys/`) +3. If **initialised but sealed**: reads the keys JSON from S3 → calls `/v1/sys/unseal` with each key share + +**S3 credentials bootstrap problem**: The init Job needs S3 credentials on a fresh cluster before ArgoCD has run. Solution: the infra repo's Terraform creates a `scaleway-s3-credentials` Kubernetes Secret in the `openbao` namespace during cluster bootstrap (before ArgoCD seed Application is applied). This is a cross-repo dependency documented in the quickstart. + +**Security note**: Unseal keys in S3 are unencrypted at rest (S3 object). Mitigation: use S3 bucket versioning + SSE (server-side encryption with Scaleway-managed keys), restrict bucket policy to the init Job's service account. + +### Option 3B: Scaleway Key Manager (KMS auto-unseal) + +OpenBao supports PKCS#11 and KMIP transit seals. Scaleway Key Manager (SKM) exposes an API but does not (as of 2026-06) expose a PKCS#11 or KMIP endpoint compatible with Vault/OpenBao transit seal. Therefore **not currently viable without a custom proxy**. + +### Option 3C: Bank-Vaults Operator + +[Bank-Vaults](https://bank-vaults.dev/) is an OSS operator (Apache 2.0) that handles Vault/OpenBao lifecycle including init, unseal, and backup/restore. Adds an additional operator dependency but provides a production-grade solution. + +**Trade-off**: Adds complexity (one more operator, CRD, controller) but handles edge cases (partial unseal, key rotation) better than Option 3A. + +**Option 3A selected for v1** (simpler, fewer moving parts, constitution Principle I on complexity). Bank-Vaults can be evaluated later. + +--- + +## Decision 4: Kubernetes Integration (How Workloads Consume Secrets) + +*Decision: External Secrets Operator (ESO)* + +Since Infisical is not deployed, there are no existing workload integrations to migrate. For new integrations, two patterns are viable: + +| Pattern | How it works | Trade-off | +|---------|-------------|-----------| +| **External Secrets Operator** | ESO `ExternalSecret` CR syncs secrets from OpenBao into native `Secret` objects | Works with any workload; decoupled from vault agent | +| **Vault Agent Injector** | Sidecar injects secrets as files into pods via annotations | Tighter coupling; richer templating | +| **CSI provider** | Mount secrets as volumes via Secrets Store CSI Driver | Native volume mount; no sidecar | + +**Recommendation**: External Secrets Operator (ESO) — most decoupled, cloud-agnostic (can later switch secret backend without changing workloads), active CNCF project. ESO will be a separate platform Application. + +--- + +## Resolved Unknowns + +| Unknown | Resolution | +|---------|-----------| +| Tool choice | Presented above — human decides | +| Storage backend | S3 (Scaleway Object Storage) — supports cluster recreate natively | +| Auto-unseal | S3-backed init Job — no additional paid services needed | +| HA mode | Not supported with S3 backend on Scaleway; single replica | +| Workload integration | External Secrets Operator (ESO) | +| S3 credentials bootstrap | Terraform in infra repo creates `scaleway-s3-credentials` Secret before ArgoCD seed | +| Infisical migration | No live migration needed — Infisical never deployed in this repo | +| Scaleway KMS auto-unseal | Not viable currently (no PKCS#11/KMIP endpoint) | + +--- + +## Cross-Repo Dependency + +The S3-backed init Job requires Scaleway S3 credentials (access key + secret key) to be present as a Kubernetes Secret at the time of OpenBao's first boot. This Secret must be created by the infra repo's Terraform before the ArgoCD seed Application runs. This dependency must be documented in the infra repo's bootstrap sequence. diff --git a/specs/001-replace-infisical-oss-secrets/spec.md b/specs/001-replace-infisical-oss-secrets/spec.md new file mode 100644 index 0000000..dadd262 --- /dev/null +++ b/specs/001-replace-infisical-oss-secrets/spec.md @@ -0,0 +1,109 @@ +# Feature Specification: Replace Infisical with OSS Secrets Backend + +**Feature Branch**: `001-replace-infisical-oss-secrets` + +**Created**: 2026-06-21 + +**Status**: Draft + +**Input**: User description: "Task 2 — Remplacer Infisical par un backend de secrets OSS. Done quand : Infisical retiré, backend secrets OSS en place, état persisté hors cluster éphémère (S3 Scaleway), secrets restaurables après destroy/recreate cluster." + +## User Scenarios & Testing *(mandatory)* + +### User Story 1 - Secrets Accessible via New Backend (Priority: P1) + +An operator needs to store and retrieve secrets from the new OSS secrets backend running in the cluster. The backend is deployed via ArgoCD (GitOps) and accessible by workloads. This is the foundational capability: without it, no workload can function. + +**Why this priority**: Core capability — everything else depends on the backend being operational. + +**Independent Test**: Can be validated by deploying the backend chart, writing a test secret, and reading it back from a pod in the cluster. + +**Acceptance Scenarios**: + +1. **Given** the OSS secrets backend is deployed via ArgoCD, **When** an operator writes a secret to the backend API, **Then** the secret is stored and retrievable from within the cluster. +2. **Given** the backend is running, **When** a workload requests a secret, **Then** the secret is injected within 5 seconds. +3. **Given** the backend is unreachable, **When** a workload requests a secret, **Then** the workload receives an appropriate error and the backend failure is observable in cluster monitoring. + +--- + +### User Story 2 - Secrets Survive Cluster Destroy/Recreate (Priority: P2) + +An operator destroys the ephemeral cluster and recreates it. All secrets previously stored in the backend must be automatically restored from Scaleway Object Storage (S3) without any manual re-entry. + +**Why this priority**: This is the key resilience requirement — persistent state is the primary problem with ephemeral clusters, and the whole reason Infisical is being replaced. + +**Independent Test**: Destroy cluster, recreate it, verify secrets are available without human intervention. CI pipeline executes this flow. + +**Acceptance Scenarios**: + +1. **Given** secrets are stored in the backend and the backend's state is persisted to Scaleway S3, **When** the cluster is destroyed and recreated, **Then** the backend is restored and all previously stored secrets are accessible. +2. **Given** the cluster is freshly created, **When** ArgoCD reconciles the backend Application, **Then** the backend automatically unseals/initialises from the Scaleway S3 state without operator input. +3. **Given** the S3 bucket is empty (first deploy), **When** the backend initialises, **Then** a fresh secrets store is created and the initial credentials/unseal keys are persisted to S3. + +--- + +### User Story 3 - Infisical Fully Removed (Priority: P3) + +Infisical pods, CRDs, Helm releases, and all GitOps manifests referencing Infisical are absent from the cluster and from the `gitops` repository. + +**Why this priority**: Cleanup is mandatory for the "Done when" criterion but does not block P1/P2 functionality. + +**Independent Test**: `kubectl get pods -A | grep infisical` returns empty; `git grep -r infisical` (case-insensitive) in the gitops repo returns no active manifests. + +**Acceptance Scenarios**: + +1. **Given** Infisical was previously deployed, **When** Infisical removal manifests are applied via ArgoCD, **Then** no Infisical pods, CRDs, or Helm resources remain in the cluster. +2. **Given** the gitops repo has been updated, **When** ArgoCD syncs, **Then** no Application or Helm release for Infisical exists in the cluster. +3. **Given** workloads previously sourced secrets from Infisical, **When** Infisical is removed, **Then** those workloads retrieve their secrets from the new backend without interruption. + +--- + +### Edge Cases + +- What happens when the Scaleway S3 bucket is temporarily unavailable during backend startup (network partition)? +- How does the backend behave if the S3 state is corrupted or partially written (mid-write crash)? +- What happens if the cluster is destroyed before the first S3 state flush completes? +- How are initial unseal/root credentials stored so they survive cluster recreation but are not exposed in Git? + +## Requirements *(mandatory)* + +### Functional Requirements + +- **FR-001**: The secrets backend MUST be an open-source tool available under a licence that requires no paid tier for self-hosted operation (constitution Principle I). +- **FR-002**: The secrets backend MUST be deployed as a GitOps-managed ArgoCD Application in the `gitops` repository, following the existing `platform//` or `apps//` conventions. +- **FR-003**: The secrets backend MUST persist its state to the Scaleway Object Storage bucket provisioned by Task 1, such that all secrets survive a full cluster destroy/recreate cycle. +- **FR-004**: On cluster recreate, the backend MUST automatically restore from S3 state without requiring manual operator input (i.e., auto-unseal or equivalent mechanism). +- **FR-005**: The backend MUST expose a secrets API accessible from within-cluster workloads (pod-level network reach). +- **FR-006**: All Infisical manifests, Helm releases, CRDs, and ArgoCD Applications MUST be removed from the `gitops` repo and from the cluster. +- **FR-007**: Workloads that previously sourced secrets from Infisical MUST be migrated to source secrets from the new backend without manual secret re-entry (secrets migrated, not lost). +- **FR-008**: The tool choice (OpenBao vs HashiCorp Vault OSS vs Scaleway Secret Manager) SHALL be deferred to the `/speckit-plan` phase where alternatives are presented and the human makes the final decision (constitution Principle II). +- **FR-009**: The backend configuration (addresses, auth methods, policies) MUST be fully version-controlled in the `gitops` repo (constitution Principle V). +- **FR-010**: CI MUST include a step that validates secrets are restorable after cluster recreate (constitution Principle VII). + +### Key Entities + +- **Secrets Backend**: The chosen OSS tool instance (tool TBD at plan phase); owns the secrets API, access control, and audit log. +- **Backend State**: The durable, encrypted blob persisted to Scaleway S3; contains all secrets and backend configuration; survives cluster lifecycle. +- **S3 Bucket**: Scaleway Object Storage bucket from Task 1; acts as the external persistence layer for backend state. +- **Secret**: A named key/value (or key/file) pair managed by the backend; consumed by workloads via injection or API call. +- **ArgoCD Application**: GitOps resource in `gitops` repo that declares the backend's desired state; reconciled by ArgoCD. + +## Success Criteria *(mandatory)* + +### Measurable Outcomes + +- **SC-001**: After cluster destroy and recreate, 100% of previously stored secrets are available to workloads without manual re-entry — verifiable by a CI pipeline that writes secrets, destroys/recreates the cluster, and asserts secret presence. +- **SC-002**: Secret retrieval by a workload completes in under 5 seconds under normal operating conditions. +- **SC-003**: Zero Infisical resources (pods, CRDs, Helm releases, ArgoCD Applications) remain in the cluster after migration — verifiable by `kubectl get all -A` and `helm list -A` returning no Infisical results. +- **SC-004**: Zero Infisical references remain in active GitOps manifests in the `gitops` repo — verifiable by `git grep -ri infisical` returning no matches in `apps/`, `platform/`, `clusters/`, or `bootstrap/`. +- **SC-005**: The backend is deployed exclusively via ArgoCD (GitOps), with no manually applied manifests — verifiable by ArgoCD reporting all backend resources as Synced. +- **SC-006**: The tool used is 100% open-source with no paid-tier requirement for the deployed functionality (constitution Principle I) — verifiable by licence review at plan phase. + +## Assumptions + +- The Scaleway S3 bucket provisioned by Task 1 (S3 backup foundation) is available, accessible from within the cluster, and has appropriate write permissions for the backend. +- The tool choice (OpenBao / HashiCorp Vault OSS / Scaleway Secret Manager) is deferred to `/speckit-plan`; Scaleway Secret Manager is a candidate but may conflict with Principle I (OSS-first) pending licence/cost review. +- Existing Infisical integrations use a known, documented method (e.g. `InfisicalSecret` CRDs or environment variable injection) that can be mapped to the new backend's equivalent — the exact migration path will be defined at plan time. +- The cluster environments in scope are `local` (minikube) and `staging` (Scaleway Kapsule), consistent with the current `gitops` repo structure. +- Auto-unseal credentials (e.g. Vault/OpenBao transit key or KMS reference) will be stored in S3 alongside the backend state, or injected via a mechanism that does not require human presence on cluster restart. +- A production environment is not yet in scope; only `local` and `staging` need to be validated. diff --git a/specs/001-replace-infisical-oss-secrets/tasks.md b/specs/001-replace-infisical-oss-secrets/tasks.md new file mode 100644 index 0000000..cfdb212 --- /dev/null +++ b/specs/001-replace-infisical-oss-secrets/tasks.md @@ -0,0 +1,226 @@ +# Tasks: Replace Infisical with OSS Secrets Backend (OpenBao) + +**Input**: Design documents from `specs/001-replace-infisical-oss-secrets/` + +**Prerequisites**: plan.md ✅, spec.md ✅, research.md ✅, data-model.md ✅, contracts/ ✅, quickstart.md ✅ + +**Tool decision**: OpenBao (MPL 2.0) — confirmed 2026-06-21 + +**No tests requested** — validation is done via the quickstart.md scenarios and the CI pipeline. + +## Format: `[ID] [P?] [Story] Description` + +- **[P]**: Can run in parallel (different files, no incomplete dependencies) +- **[Story]**: Which user story this task belongs to ([US1], [US2], [US3]) +- Exact file paths in every description + +--- + +## Phase 1: Setup — Init Chart Scaffold + +**Purpose**: Create the `apps/openbao-init/` Helm chart structure. All subsequent phases add templates into this chart. + +- [x] T001 Create `apps/openbao-init/Chart.yaml` — name: `openbao-init`, version: `0.1.0`, description: "OpenBao init, unseal, and ESO configuration" +- [x] T002 Create `apps/openbao-init/values.yaml` with defaults: `openbao.address`, `openbao.namespace`, `s3.bucket`, `s3.region`, `s3.endpoint`, `s3.keysPrefix`, `policies` list (empty), `eso.clusterSecretStoreName` +- [x] T003 [P] Create `apps/openbao-init/values-local.yaml` — overrides: `openbao.address: http://openbao.openbao.svc.cluster.local:8200`, `s3.bucket: `, `s3.endpoint: https://s3.fr-par.scw.cloud` +- [x] T004 [P] Create `apps/openbao-init/values-scaleway.yaml` — same bucket/endpoint as local (same Scaleway S3), different `s3.keysPrefix: openbao/init-keys/scaleway.json` to avoid collision with local env keys + +**Checkpoint**: `apps/openbao-init/` exists with chart metadata and all values files + +--- + +## Phase 2: Foundational — Platform Manifests (OpenBao + ESO) + +**Purpose**: Declare OpenBao and External Secrets Operator as ArgoCD Applications for both environments. These are prerequisites for any secrets to be accessible. + +**⚠️ CRITICAL**: US1 cannot proceed until this phase is complete (OpenBao must be declared before the init job can target it) + +- [x] T005 Write `platform/local/openbao.yml` — ArgoCD Application pointing to `https://openbao.github.io/openbao-helm` chart `openbao`; destination namespace `openbao`; Helm values inline: `server.ha.enabled: false`, `server.standalone.config` with S3 storage stanza (bucket, region, endpoint `https://s3.fr-par.scw.cloud`, `s3_force_path_style = true`, credentials via env vars from `scaleway-s3-credentials` Secret); `server.extraEnv` with `AWS_ACCESS_KEY_ID`/`AWS_SECRET_ACCESS_KEY` from `scaleway-s3-credentials`; `syncOptions: [CreateNamespace=true]` +- [x] T006 [P] Write `platform/scaleway/openbao.yml` — identical structure to `platform/local/openbao.yml`; same S3 endpoint (Scaleway fr-par); pin same chart version +- [x] T007 [P] Write `platform/local/external-secrets.yml` — ArgoCD Application pointing to `https://charts.external-secrets.io` chart `external-secrets`; destination namespace `external-secrets`; no custom values needed for v1; `syncOptions: [CreateNamespace=true]` +- [x] T008 [P] Write `platform/scaleway/external-secrets.yml` — identical to `platform/local/external-secrets.yml`; pin same chart version +- [x] T009 Validate all four platform manifests: run `kubectl --context minikube apply --dry-run=client -f platform/local/openbao.yml`, same for the other three files; fix any schema errors + +**Checkpoint**: Four platform manifests pass `--dry-run=client`; ArgoCD can render them once deployed + +--- + +## Phase 3: User Story 1 — Secrets Accessible via New Backend (Priority: P1) 🎯 MVP + +**Goal**: OpenBao is deployed via ArgoCD, the init Job unseals it on boot, and a secret can be written and read. + +**Independent Test**: Scenario 1 from `specs/001-replace-infisical-oss-secrets/quickstart.md` — deploy, verify `bao status` shows `Sealed: false`, write `kv/apps/test/smoke-test value=hello`, read it back successfully. + +### Init Job RBAC + +- [x] T010 [US1] Write `apps/openbao-init/templates/serviceaccount.yaml` — ServiceAccount named `openbao-init` in `{{ .Release.Namespace }}`; used by the init Job to access the `scaleway-s3-credentials` Secret +- [x] T011 [P] [US1] Write `apps/openbao-init/templates/role.yaml` — Role named `openbao-init`; rules: `get` on `secrets` resource, resource name `scaleway-s3-credentials` +- [x] T012 [P] [US1] Write `apps/openbao-init/templates/rolebinding.yaml` — RoleBinding binding `openbao-init` Role to `openbao-init` ServiceAccount + +### Policies ConfigMap + +- [x] T013 [US1] Write `apps/openbao-init/templates/configmap-policies.yaml` — ConfigMap named `openbao-policies`; data keys are policy names (e.g. `demo-read.hcl`); each value is an HCL policy granting `["read", "list"]` on `kv/data/apps//*`; policies driven by `{{ .Values.policies }}` list in values so new apps can be added without template changes + +### Init/Unseal Job + +- [x] T014 [US1] Write `apps/openbao-init/templates/job-init.yaml` — Kubernetes Job with `argocd.argoproj.io/hook: PostSync` and `argocd.argoproj.io/hook-delete-policy: HookSucceeded` annotations; image: `alpine/k8s` or equivalent with `curl`, `jq`, `aws-cli`; serviceAccountName: `openbao-init`; env vars: `OPENBAO_ADDR` from values, `AWS_ACCESS_KEY_ID`/`AWS_SECRET_ACCESS_KEY` from `scaleway-s3-credentials` Secret; shell script logic (in a ConfigMap or inline): + 1. Wait for OpenBao to respond on `:8200/v1/sys/health` (retry loop, 60s timeout) + 2. Read `bao status -format=json`; extract `.initialized` and `.sealed` + 3. **If not initialized**: call `bao operator init -key-shares=5 -key-threshold=3 -format=json`; write JSON to S3 at `{{ .Values.s3.keysPrefix }}`; extract 3 unseal keys and unseal; enable KV v2 (`bao secrets enable -path=kv kv-v2`); enable Kubernetes auth (`bao auth enable kubernetes`; configure with in-cluster API server); apply all policies from ConfigMap; create `external-secrets` Kubernetes auth role bound to `external-secrets` ServiceAccount in `external-secrets` namespace with all policies; revoke root token + 4. **If initialized but sealed**: read JSON from S3; extract 3 unseal key shares; call `bao operator unseal` 3 times + +### ESO ClusterSecretStore + +- [x] T015 [P] [US1] Write `apps/openbao-init/templates/clustersecretstore.yaml` — `ClusterSecretStore` CR named `{{ .Values.eso.clusterSecretStoreName }}`; provider: vault; server: `{{ .Values.openbao.address }}`; path: `kv`; version: `v2`; auth.kubernetes.mountPath: `kubernetes`; auth.kubernetes.role: `external-secrets`; auth.kubernetes.serviceAccountRef: `external-secrets` SA in `external-secrets` namespace + +### Cluster Application Templates + +- [x] T016 [US1] Write `clusters/local/templates/openbao-init.yaml` — ArgoCD Application for `apps/openbao-init` chart; `source.repoURL: {{ .Values.repoURL }}`; `source.targetRevision: {{ .Values.revision }}`; `source.path: apps/openbao-init`; `source.helm.valueFiles: [values-local.yaml]`; destination namespace: `openbao`; automated sync with prune + selfHeal + CreateNamespace=true +- [x] T017 [P] [US1] Write `clusters/scaleway/templates/openbao-init.yaml` — identical to T016 but `source.helm.valueFiles: [values-scaleway.yaml]` + +### Sync Wave Ordering + +- [x] T018 [US1] Add `argocd.argoproj.io/sync-wave` annotations to all six affected manifests: + - `platform/local/openbao.yml` and `platform/scaleway/openbao.yml` → wave `"0"` (server starts first) + - `clusters/local/templates/openbao-init.yaml` and `clusters/scaleway/templates/openbao-init.yaml` → wave `"1"` (init job after server) + - `platform/local/external-secrets.yml` and `platform/scaleway/external-secrets.yml` → wave `"2"` (ESO after init configures ClusterSecretStore) + +### Validation + +- [x] T019 [US1] Validate init chart: run `helm lint apps/openbao-init/ -f apps/openbao-init/values-local.yaml` and `helm template openbao-init apps/openbao-init/ -f apps/openbao-init/values-local.yaml`; fix any lint errors +- [ ] T020 [US1] Local smoke test — Scenario 1 from `specs/001-replace-infisical-oss-secrets/quickstart.md`: start minikube, manually create `scaleway-s3-credentials` Secret, push branch, trigger ArgoCD sync, verify `bao status` shows `Sealed: false`, write `kv/apps/test/smoke-test value=hello`, read it back + +**Checkpoint**: `bao kv get kv/apps/test/smoke-test` returns `value = hello`. US1 is complete and independently validated. + +--- + +## Phase 4: User Story 2 — Secrets Survive Cluster Destroy/Recreate (Priority: P2) + +**Goal**: A canary secret written before cluster destruction is automatically available after cluster recreate — no operator input required. + +**Independent Test**: Scenario 2 from `specs/001-replace-infisical-oss-secrets/quickstart.md` — write canary, `minikube delete`, `minikube start`, ArgoCD re-syncs, canary value matches pre-destroy value. CI pipeline executes this automatically. + +### CI Pipeline + +- [x] T021 [US2] Create `.github/workflows/ci.yml` with two jobs: + - **job `helm-lint`**: runs on every PR; steps: checkout → `helm lint apps/openbao-init/ -f apps/openbao-init/values-local.yaml` → `helm lint apps/openbao-init/ -f apps/openbao-init/values-scaleway.yaml` → `helm template boot bootstrap/ --set env=local --set revision=main` → `helm template loc clusters/local/ --set revision=main` + - **job `secrets-restore`**: runs on every PR; needs: minikube, kubectl, argocd CLI, `bao` CLI, `aws` CLI; secrets: `SCW_ACCESS_KEY_ID`, `SCW_SECRET_ACCESS_KEY`, `SCW_S3_BUCKET`; steps: (1) start minikube, (2) install ArgoCD, (3) apply seed Application pointing to this repo + branch, (4) create `scaleway-s3-credentials` Secret in `openbao` namespace, (5) wait for ArgoCD sync + init Job completion, (6) write canary secret `kv/apps/test/canary value=survive-`, (7) `minikube delete`, (8) `minikube start`, (9) install ArgoCD, (10) re-apply seed Application, (11) create `scaleway-s3-credentials` Secret, (12) wait for sync + init Job, (13) assert canary value matches step 6, (14) assert `bao status` shows `Sealed: false` +- [ ] T022 [US2] Local validation — Scenario 2 from `specs/001-replace-infisical-oss-secrets/quickstart.md`: manually execute destroy/recreate cycle; confirm canary secret restored; document any deviations + +**Checkpoint**: CI `secrets-restore` job passes. SC-001 verified. US2 complete. + +--- + +## Phase 5: User Story 3 — Infisical Fully Removed (Priority: P3) + +**Goal**: Confirm no Infisical artifacts exist in the gitops repo or the cluster; enforce this permanently in CI. + +**Independent Test**: Scenario 4 from `specs/001-replace-infisical-oss-secrets/quickstart.md` — `git grep -ri infisical apps/ platform/ clusters/ bootstrap/` returns empty; `kubectl get pods -A | grep -i infisical` returns empty. + +### CI Enforcement + +- [x] T023 [US3] Add `infisical-absent` job to `.github/workflows/ci.yml`: (1) `git grep -ri infisical apps/ platform/ clusters/ bootstrap/` must exit non-zero (no matches); (2) on the `secrets-restore` job's cluster: `kubectl get pods -A -o name | grep -i infisical` must return empty; fail the job if either check finds a match +- [x] T024 [US3] Local validation — Scenario 4 from `specs/001-replace-infisical-oss-secrets/quickstart.md`: run the grep and kubectl checks locally; confirm all pass + +**Checkpoint**: `infisical-absent` CI job passes. SC-003 and SC-004 verified. US3 complete. + +--- + +## Phase 6: Polish & Cross-Cutting Concerns + +**Purpose**: ESO end-to-end integration, cluster chart rendering, documentation. + +- [x] T025 [P] Validate cluster chart rendering for local: `helm template boot bootstrap/ --set env=local --set revision=main` and `helm template loc clusters/local/ --set revision=main`; confirm `openbao-init.yaml` Application appears in output +- [x] T026 [P] Validate cluster chart rendering for scaleway: `helm template loc clusters/scaleway/ --set revision=main`; confirm `openbao-init.yaml` Application appears in output +- [ ] T027 Local ESO integration test — Scenario 3 from `specs/001-replace-infisical-oss-secrets/quickstart.md`: apply test `ExternalSecret` in `default` namespace; wait 15s; verify `kubectl get secret test-secret -n default` exists and value matches what was written to OpenBao +- [x] T028 Update `CLAUDE.md` validation commands section with the new `openbao-init` lint/template commands (mirror the existing `helm lint apps/demo/` pattern) + +**Checkpoint**: All six CI jobs pass on a PR against main. All four quickstart scenarios pass locally. + +--- + +## Dependencies & Execution Order + +### Phase Dependencies + +- **Phase 1 (Setup)**: No dependencies — start immediately +- **Phase 2 (Foundational)**: Depends on Phase 1 (chart scaffold must exist) — blocks US1 +- **Phase 3 (US1 MVP)**: Depends on Phase 2 completion — T010–T019 can largely run in parallel within the phase +- **Phase 4 (US2)**: Depends on Phase 3 checkpoint (destroy/recreate requires a working US1 deployment) +- **Phase 5 (US3)**: Depends on Phase 2 (platform manifests must exist to confirm Infisical is absent); CI job can be written in parallel with Phase 4 +- **Phase 6 (Polish)**: Depends on Phases 3–5 complete + +### User Story Dependencies + +| Story | Depends on | Can parallelise with | +|-------|-----------|---------------------| +| US1 (P1) | Phase 2 complete | — | +| US2 (P2) | US1 checkpoint (T020 passed) | — | +| US3 (P3) | Phase 2 complete (for CI enforcement) | US2 (CI file already exists after T021) | + +### Within-Phase Parallel Opportunities + +**Phase 2**: T006, T007, T008 can all run in parallel once T005 establishes the manifest pattern. + +**Phase 3**: After T001–T004 (scaffold) and T010–T012 (RBAC): +- T013 (policies ConfigMap) and T015 (ClusterSecretStore) can run in parallel +- T016 and T017 (cluster templates) can run in parallel +- T018 sync-wave edits touch multiple files but are independent per-file + +--- + +## Parallel Examples + +### Parallel: Phase 2 Platform Manifests +``` +T005 → T006 [P] (scaleway openbao.yml, same pattern) + → T007 [P] (local external-secrets.yml, independent) + → T008 [P] (scaleway external-secrets.yml, independent) +``` + +### Parallel: Phase 3 RBAC + ConfigMap +``` +After T010 (ServiceAccount): + T011 [P] role.yaml + T012 [P] rolebinding.yaml + T013 configmap-policies.yaml (independent) + T015 [P] clustersecretstore.yaml (independent) +``` + +### Parallel: Phase 3 Cluster Templates +``` +T016 [US1] clusters/local/templates/openbao-init.yaml +T017 [P][US1] clusters/scaleway/templates/openbao-init.yaml +(different files, no dependencies on each other) +``` + +--- + +## Implementation Strategy + +### MVP (US1 only — Phases 1–3) + +1. Phase 1: Scaffold chart (T001–T004) +2. Phase 2: Platform manifests (T005–T009) +3. Phase 3: Init job + RBAC + cluster templates + sync waves (T010–T019) +4. **STOP and VALIDATE**: Run Scenario 1 from quickstart.md (T020) +5. If smoke test passes → US1 is shipped; secrets backend is operational + +### Incremental Delivery + +1. MVP (Phases 1–3) → OpenBao operational ✅ +2. Phase 4 (US2) → Destroy/recreate proven in CI ✅ +3. Phase 5 (US3) → Infisical-absent gate in CI ✅ +4. Phase 6 (Polish) → All validation scenarios pass ✅ + +--- + +## Notes + +- All manifests must use `{{ .Values.repoURL }}` and `{{ .Values.revision }}` for in-house charts (see `clusters/local/templates/demo.yaml` as reference) +- Pin external Helm chart versions explicitly — never use floating refs +- The `scaleway-s3-credentials` Secret is created by infra Terraform, not by this repo — the init Job **reads** it but never creates it +- Init Job must be idempotent: multiple runs must not re-initialise an already-initialised OpenBao +- Root token must be revoked at the end of the init script; subsequent access uses Kubernetes auth +- See `contracts/argocd-applications.md` for exact manifest shapes and sync-wave table +- See `contracts/secrets-consumption.md` for KV path convention and policy naming