From 99367a2319e29802db625df81866926f9bcec055 Mon Sep 17 00:00:00 2001 From: Antonia Gaete Date: Fri, 5 Jun 2026 16:28:47 -0400 Subject: [PATCH 01/11] feat: secrets management skill Signed-off-by: Antonia Gaete --- lfx-v2-secrets/SKILL.md | 542 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 542 insertions(+) create mode 100644 lfx-v2-secrets/SKILL.md diff --git a/lfx-v2-secrets/SKILL.md b/lfx-v2-secrets/SKILL.md new file mode 100644 index 0000000..e1db8a7 --- /dev/null +++ b/lfx-v2-secrets/SKILL.md @@ -0,0 +1,542 @@ +--- +name: lfx-v2-secrets +description: > + Guide an agent through wiring up secrets for LFX V2 microservices using External Secrets + Operator (ESO) + IRSA. Supports two modes: (1) full setup for new services touching + lfx-v2-opentofu, lfx-secrets-management, the service Helm chart, and lfx-v2-argocd; + (2) adding secrets to existing services already configured with ESO. Use this skill + whenever someone says "set up secrets", "wire up ESO", "add a secret to this service", + "IRSA configuration", "External Secrets for V2", or any mention of AWS Secrets Manager + integration with Kubernetes for LFX V2 services. +allowed-tools: Bash, Read, Glob, Grep, AskUserQuestion, WebFetch +--- + + + + + +# LFX V2 Secrets Setup Guide + +Secrets for LFX V2 microservices are managed through **External Secrets Operator (ESO)** +combined with **IAM Roles for Service Accounts (IRSA)** on AWS. This provides a secure, +GitOps-driven way to sync secrets from AWS Secrets Manager into Kubernetes. + +> **For the AI**: This skill has two modes. Mode 1 (new service) touches four repos +> and requires coordinated changes across infrastructure layers. Mode 2 (existing service) +> is much smaller. Always ask the user which applies before proceeding. + +--- + +## Understanding the Architecture + +### How It Works + +1. A **Kubernetes ServiceAccount** is annotated with an **IRSA role ARN** +2. ESO's **SecretStore** uses that ServiceAccount's JWT token to authenticate to AWS +3. ESO watches **ExternalSecret** manifests and syncs matching secrets from AWS SM into K8s Secrets +4. Application deployments reference the K8s Secret via environment variable or volume mount +5. Local development skips ESO entirely and injects secret values directly via `environment` in values + +### Key Constants + +These values are fixed and apply across all V2 services: + +| Item | Value | +|------|-------| +| AWS Region | `us-west-2` | +| K8s Secret name | `{{ .Chart.Name }}` (e.g., `lfx-v2-invite-service`) | +| SecretStore name | `{{ .Chart.Name }}` | +| IAM account — dev | `788942260905` | +| IAM account — staging | `844790888233` | +| IAM account — prod | `372256339901` | +| IRSA role ARN pattern | `arn:aws:iam:::role/lfx-v2-` | +| AWS SM path pattern | `cloud/<3rd-party-service>/` | +| ServiceAccount annotation key | `eks.amazonaws.com/role-arn` | +| ESO JWT auth field | `spec.provider.aws.auth.jwt.serviceAccountRef` | + +--- + +## Mode 1: New Service (Full Setup) + +Use this mode when a brand-new V2 service needs secrets wired up end-to-end from scratch. +This touches **four repos** and requires changes in a specific order. + +### Step 1: Prepare Information + +Ask the user to collect: + +1. **Service name** — fully qualified name (e.g., `lfx-v2-invite-service`, `lfx-v2-email-service`) +2. **Namespace name** — the namespace in Kubernetes the service is deployed in +3. **List of secrets** — secrets can come from two source types; collect the relevant details for each: + + **1Password sources** (e.g., API keys, SMTP credentials, JWT secrets): + - **Secret name** (e.g., "JWT Secret", "SMTP Credentials") + - **Third-party service** that provides the secret (e.g., `litellm`, `github`) + - **1Password item name** — exact name as it appears in the vault + - **Field names in 1Password** — the exact field names as they appear in the source vault + + **Auth0 sources** (e.g., M2M client credentials, BFF client secrets): + - **Auth0 client name** — exact display name in Auth0 (e.g., `LFX V2 Invite Service`) + - **Credential type** — `auth0` (produces `client_id` + `client_secret`) or `auth0_jwt` (produces `client_id` + `client_public_key` + `client_private_key`; standard for LFX V2 microservices) + - **Field rename mapping** (optional) — if env var names differ from defaults (e.g., `client_id` → `auth0_client_id`) + - **Auto-rotate** — yes/no; default `true` for `auth0_jwt` V2 services + - **AWS SM path** — follows `auth0/` convention (e.g., `auth0/LFX_V2_Invite_Service`) + +4. **Which environments need this secret** — `development`, `staging`, `production` +5. **Service tag** — applies to all secrets going to AWS SM; use the `eso_service_tag` value from this service's entry in `iam-service-accounts-definitions.yaml` in `lfx-v2-opentofu` (set in Step 2) — do not ask the user for this + +**1Password example:** + +```text +Service: lfx-v2-invite-service +Namespace: invite-service +Secrets: + - Atlassian API Key (service: atlassian, field: atlassian_api_key) - all envs + - JWT Secret (service: jwt, field: jwt_secret) — all envs +``` + +**Auth0 examples:** + +`auth0` (client_secret, simpler M2M or BFF clients): + +```text +Service: lfx-v2-invite-service +Namespace: invite-service +Auth0 secrets: + - Invite Service BFF + type: auth0 + client: "LFX V2 Invite BFF" + path: auth0/LFX_V2_Invite_BFF +``` + +`auth0_jwt` (JWT private key, standard for LFX V2 microservices): + +```text +Service: lfx-v2-invite-service +Namespace: invite-service +Auth0 secrets: + - Invite Service M2M + type: auth0_jwt + client: "LFX V2 Invite Service" + auto_rotate: true + rename_fields: client_id → auth0_client_id, client_private_key → auth0_client_private_key + path: auth0/LFX_V2_Invite_Service +``` + +### Step 2: Create IAM Service Account in `lfx-v2-opentofu` + +In the [lfx-v2-opentofu](https://github.com/linuxfoundation/lfx-v2-opentofu) repo, +edit `iam-service-accounts-definitions.yaml` and add: + +```yaml +service_account_roles: + lfx-v2-: + namespace: "" +``` + +Example for invite service: + +```yaml +service_account_roles: + lfx-v2-invite-service: + namespace: "invite-service" +``` + +> **Defaults**: All fields default to the role key (e.g., `lfx-v2-invite-service`): `namespace`, +> `service_account`, and `eso_service_tag`. Only specify a field when its value differs +> from the role key. + +### Step 3: Create Sync Entries in `lfx-secrets-management` + +In the [lfx-secrets-management](https://github.com/linuxfoundation/lfx-secrets-management) repo, +add an entry for each secret to the appropriate file under `secrets/lfx/`. Check the existing +files to find where similar secrets live — auth0 credentials go in `auth0_clients.yml`, +most one-off service credentials go in `cloud.yml`, but third-party services may have their +own file (e.g., `litellm.yml`). When unsure, grep for the third-party service name across +`secrets/lfx/`. + +> **Important**: All secrets must be stored as JSON in AWS SM, even single-field ones. +> Two equivalent ways to express this: +> +> ```yaml +> # implicit JSON (list form) +> fields: +> - +> +> # explicit JSON (scalar + flag) +> fields: +> store_as_json: true +> ``` + +```yaml +: + tags: [lfx_v2, , ] + environments: [development, staging, production] + source: + onepassword: + vaults: + development: LFX V2 - Development + staging: LFX V2 - Staging + production: LFX V2 - Production + item: <1Password Item Name> + fields: + - + destinations: + - aws_secretsmanager: + tags: + service-: enabled + path: /<3rd-party-service>/ +``` + +Example for invite service JWT secret: + +```yaml +LFX V2 Invite Service JWT Secret: + tags: [lfx_v2, invite, jwt] + environments: [development, staging, production] + source: + onepassword: + vaults: + development: LFX V2 - Development + staging: LFX V2 - Staging + production: LFX V2 - Production + item: "LFX V2 Invite Service - JWT Secret" + fields: + - jwt_secret_key + destinations: + - aws_secretsmanager: + tags: + service-lfx-v2-invite-service: enabled + path: "cloud/invite/jwt" +``` + +> **Tips**: +> +> - Each secret in the lfx-secrets-management source becomes a separate AWS SM path entry +> - The `path` convention is `cloud//` +> - Use the `environments` list to sync to all three environments in parallel +> - The `source.onepassword.item` should match exactly the name in 1Password vaults +> - The field names should be descriptive enough to avoid duplicates (`litellm_api_key`, not just `api_key`) + +> **Important**: After the `lfx-secrets-management` PR is merged, manually trigger the +> [Deploy workflow](https://github.com/linuxfoundation/lfx-secrets-management/actions/workflows/deploy.yml) +> to push the secret to AWS SM. Use the most specific secret config tag (the `tags:` field +> in the YAML entry, e.g. `litellm` or `pcc`) — not the AWS resource tag — to avoid +> re-deploying or rotating unrelated secrets. The `lfx-v2-argocd` PR must not merge until +> the deploy has completed — ArgoCD will fail to sync if the secret doesn't exist in AWS SM yet. +> +> **Auth0 JWT secrets** (`auth0_jwt` source type) are rotated on every deploy. Before +> triggering the workflow for any entry tagged with `auth0_jwt`, check with CloudOps to +> confirm you are only rotating the intended service's credentials. + +### Step 4: Create Helm Chart and Custom Resource Files + +#### 4a. `serviceaccount.yaml` in the service Helm chart + +In the service repo's Helm chart (e.g., `lfx-v2-invite-service`), create +`charts/lfx-v2-/templates/serviceaccount.yaml`: + +```yaml +# Copyright The Linux Foundation and each contributor to LFX. +# SPDX-License-Identifier: MIT +{{- if .Values.serviceAccount.create }} +--- +apiVersion: v1 +kind: ServiceAccount +metadata: + name: {{ .Values.serviceAccount.name | default .Chart.Name }} + namespace: {{ .Release.Namespace }} + labels: + app: {{ .Chart.Name }} + {{- with .Values.serviceAccount.annotations }} + annotations: + {{- toYaml . | nindent 4 }} + {{- end }} +{{- end }} +``` + +Add to `charts/lfx-v2-/values.yaml`: + +```yaml +serviceAccount: + create: true + name: "lfx-v2-" + annotations: {} +``` + +#### 4b. Custom resources in `lfx-v2-argocd` + +The `SecretStore` and `ExternalSecret` are **static YAML files** (not Helm templates) placed in +`lfx-v2-argocd/custom-resources/lfx-v2-/`. + +Create `custom-resources/lfx-v2-/SecretStore.yaml`: + +```yaml +# Copyright The Linux Foundation and each contributor to LFX. +# SPDX-License-Identifier: MIT +--- +apiVersion: external-secrets.io/v1 +kind: SecretStore +metadata: + name: lfx-v2- + namespace: +spec: + provider: + aws: + auth: + jwt: + serviceAccountRef: + name: lfx-v2- + region: us-west-2 + service: SecretsManager +``` + +Create `custom-resources/lfx-v2-/ExternalSecret.yaml`: + +```yaml +# Copyright The Linux Foundation and each contributor to LFX. +# SPDX-License-Identifier: MIT +--- +apiVersion: external-secrets.io/v1 +kind: ExternalSecret +metadata: + name: lfx-v2- + namespace: +spec: + secretStoreRef: + kind: SecretStore + name: lfx-v2- + target: + creationPolicy: Owner + name: lfx-v2--secrets + refreshInterval: 10m + dataFrom: + - find: + conversionStrategy: Default + decodingStrategy: None + tags: + service-lfx-v2-: enabled + rewrite: + - merge: + conflictPolicy: Error + into: '' + strategy: Extract +``` + +> **Tag-based discovery**: ESO finds and merges all AWS SM secrets tagged +> `service-lfx-v2-: enabled` into a single Kubernetes Secret named +> `lfx-v2--secrets`. No manual `data` list is needed — new secrets are picked up +> automatically after the next sync. + +#### 4c. IRSA annotation in `lfx-v2-argocd` per-environment values + +In `values/dev/lfx-v2-.yaml` (repeat for staging and prod with the matching account ID): + +```yaml +# Copyright The Linux Foundation and each contributor to LFX. +# SPDX-License-Identifier: MIT +--- +serviceAccount: + annotations: + eks.amazonaws.com/role-arn: arn:aws:iam::788942260905:role/lfx-v2- + automountServiceAccountToken: true +``` + +| Environment | Account ID | File | +|-------------|-----------|------| +| Development | `788942260905` | `values/dev/lfx-v2-.yaml` | +| Staging | `844790888233` | `values/staging/lfx-v2-.yaml` | +| Production | `372256339901` | `values/prod/lfx-v2-.yaml` | + +### Step 5: Wire Secrets into Service Environment in `lfx-v2-argocd` + +In `values/global/lfx-v2-.yaml`, add an `environment` block that maps each secret +field to an environment variable. Reference the Kubernetes Secret created by the ExternalSecret +(`lfx-v2--secrets`) and use the field name as the key. + +```yaml +# Copyright The Linux Foundation and each contributor to LFX. +# SPDX-License-Identifier: MIT +--- +environment: + SECRET_NAME: + valueFrom: + secretKeyRef: + name: lfx-v2--secrets + key: + ANOTHER_SECRET_NAME: + valueFrom: + secretKeyRef: + name: lfx-v2--secrets + key: +``` + +--- + +## Mode 2: Add Secret to Existing Service + +Use this mode when adding a new secret to a service that already has ESO + IRSA configured. + +### Step 1: Add Entry to `lfx-secrets-management` + +Follow the same pattern as Mode 1, Step 3. Add an entry to the appropriate file under +`secrets/lfx/` — check existing files or grep for the third-party service name to find +the right one. + +```yaml +: + tags: [lfx_v2, , ] + environments: [development, staging, production] + source: + onepassword: + vaults: + development: LFX V2 - Development + staging: LFX V2 - Staging + production: LFX V2 - Production + item: <1Password Item Name> + fields: + - + destinations: + - aws_secretsmanager: + tags: + service-: enabled + path: /<3rd-party-service>/ +``` + +For auth0 sources, use the same pattern as Mode 1, Step 3 and add to `auth0_clients.yml`. + +> **Important**: After the `lfx-secrets-management` PR is merged, manually trigger the +> [Deploy workflow](https://github.com/linuxfoundation/lfx-secrets-management/actions/workflows/deploy.yml) +> to push the secret to AWS SM. Use the most specific secret config tag (the `tags:` field +> in the YAML entry, e.g. `litellm` or `pcc`) — not the AWS resource tag — to avoid +> re-deploying or rotating unrelated secrets. The `lfx-v2-argocd` PR must not merge until +> the deploy has completed — ArgoCD will fail to sync if the secret doesn't exist in AWS SM yet. +> +> **Auth0 JWT secrets** (`auth0_jwt` source type) are rotated on every deploy. Before +> triggering the workflow for any entry tagged with `auth0_jwt`, check with CloudOps to +> confirm you are only rotating the intended service's credentials. + +### Step 2: Wire Secret into Service Environment in `lfx-v2-argocd` + +In `values/global/lfx-v2-.yaml`, add the new environment variable to the existing +`environment` block: + +```yaml +environment: + NEW_ENV_VAR: + valueFrom: + secretKeyRef: + name: lfx-v2--secrets + key: +``` + +> Tag-based discovery means the new secret is picked up automatically — no changes to +> `ExternalSecret.yaml` are needed as long as the AWS SM tag matches the service. + +--- + +## Verification Checklist + +After completing either mode, verify the setup: + +**File checklist — all repos involved:** + +- [ ] `lfx-v2-opentofu`: `iam-service-accounts-definitions.yaml` has service entry *(Mode 1 only)* +- [ ] `lfx-secrets-management`: appropriate file under `secrets/lfx/` has sync entry for each secret; Deploy workflow run after merge +- [ ] Service Helm chart *(Mode 1 only)*: + - [ ] `templates/serviceaccount.yaml` created + - [ ] `values.yaml` has `serviceAccount` block +- [ ] `lfx-v2-argocd`: + - [ ] `custom-resources/lfx-v2-/SecretStore.yaml` created *(Mode 1 only)* + - [ ] `custom-resources/lfx-v2-/ExternalSecret.yaml` created *(Mode 1 only)* + - [ ] `values/dev/lfx-v2-.yaml` has IRSA role ARN + `automountServiceAccountToken: true` *(Mode 1 only)* + - [ ] `values/staging/lfx-v2-.yaml` has IRSA role ARN + `automountServiceAccountToken: true` *(Mode 1 only)* + - [ ] `values/prod/lfx-v2-.yaml` has IRSA role ARN + `automountServiceAccountToken: true` *(Mode 1 only)* + - [ ] `values/global/lfx-v2-.yaml` has `environment` block with `secretKeyRef` entries for all secrets + +**Configuration checks:** + +- [ ] IRSA role ARN format is correct: `arn:aws:iam:::role/lfx-v2-` +- [ ] All account IDs are correct (dev=`788942260905`, staging=`844790888233`, prod=`372256339901`) +- [ ] AWS SM secret config tags (`tags:` field) are specific enough to scope the deploy workflow run +- [ ] AWS SM resource tag on each secret matches `eso_service_tag`: `service-: enabled` +- [ ] All secrets are stored as JSON (list form fields or `store_as_json: true`) +- [ ] `ExternalSecret` target name is `lfx-v2--secrets` and `secretKeyRef` names match + +**1Password setup** *(1Password sources only)*: + +- [ ] Items exist in all required vaults (LFX V2 - Development/Staging/Production) +- [ ] Item names match exactly what's in the `lfx-secrets-management` `item:` field +- [ ] Field names match exactly what's in the `fields:` list + +--- + +## Reference Implementations + +Real examples in the codebase: + +| Service | What It Added | References | +|---------|---------------|-----------| +| Email Service | SMTP credentials | `lfx-v2-email-service` chart, lfx-v2-argocd values | +| Invite Service | JWT secret | `lfx-v2-invite-service` chart (LFXV2-1783), lfx-v2-argocd values | +| Committee Service | Auth0 JWT client secret (`auth0_jwt`, auto-rotate, renamed fields) | `secrets/lfx/auth0_clients.yml` in lfx-secrets-management | + +Check these repos for the exact file structure and conventions used in production. + +--- + +## Common Workflows + +### Adding a JWT secret to a new service + +1. User asks: "Set up JWT secret for invite-service" +2. Collect: service name, 1Password item name, field names, environments +3. Follow Mode 1 steps in order +4. Verify using the checklist above +5. Open PRs for `lfx-v2-opentofu`, `lfx-secrets-management`, service Helm chart, and `lfx-v2-argocd` +6. Merge `lfx-secrets-management` PR, trigger Deploy workflow, then merge remaining PRs + +### Adding SMTP credentials to an existing service + +1. User asks: "Add SMTP secret to email-service" +2. Follow Mode 2 steps +3. Verify using the checklist above +4. Open PRs for `lfx-secrets-management` and `lfx-v2-argocd` +5. Merge `lfx-secrets-management` PR, trigger Deploy workflow, then merge `lfx-v2-argocd` PR + +### Adding an Auth0 client key pair to an existing service + +1. Users asks: "Add the LFX V2 Persona Service auth0 client to lfx-v2-persona-service" +2. Add sync entry in lfx-secrets-management +3. Verify using the checklist above +4. Submit secrets PR +5. Coordinate with the Platform Engineering team to deploy the auto-rotated secret +6. Wire into deployment in values charts +7. Verify using the checklist above +8. Submit argocd PR + +### Debugging: "Pods can't read the secret" + +Check in order: + +1. **Pod events** — `kubectl describe pod ` to see if the SecretStore mounted +2. **ESO logs** — `kubectl logs -n external-secrets-system deployment/external-secrets` +3. **AWS SM permissions** — verify IRSA role has `SecretsManager:GetSecretValue` on the path +4. **Secret exists in AWS SM** — lfx-secrets-management automation has synced the secret +5. **ExternalSecret status** — `kubectl describe externalsecret ` shows sync status +6. **Topology/firewalling** — pod can reach AWS API endpoint (check SecurityGroup, NACL, DNS) + +--- + +## Communication Style + +This skill serves both platform engineers and application developers: + +- **For experienced infrastructure engineers**: Use technical terms freely (IRSA, JWT auth, ESO operator). +- **For application developers touching secrets for the first time**: Explain what ESO is + (*"it automatically copies secrets from AWS into Kubernetes"*) and IRSA (*"it proves + your pod is who it claims to be when talking to AWS"*). +- **For non-technical users**: Avoid "Kubernetes", "IRSA", "operator", "manifest". Instead say + "cloud setup", "permissions", "secure secret storage", "automated sync". + +Always finish with the verification checklist so the user can confirm everything is wired correctly. From 2a6341653cee4add5baead3c60ab4159a44d40e8 Mon Sep 17 00:00:00 2001 From: Antonia Gaete Date: Fri, 5 Jun 2026 16:44:10 -0400 Subject: [PATCH 02/11] copilot suggestions Signed-off-by: Antonia Gaete --- lfx-v2-secrets/SKILL.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/lfx-v2-secrets/SKILL.md b/lfx-v2-secrets/SKILL.md index e1db8a7..126d8ab 100644 --- a/lfx-v2-secrets/SKILL.md +++ b/lfx-v2-secrets/SKILL.md @@ -44,7 +44,7 @@ These values are fixed and apply across all V2 services: | Item | Value | |------|-------| | AWS Region | `us-west-2` | -| K8s Secret name | `{{ .Chart.Name }}` (e.g., `lfx-v2-invite-service`) | +| K8s Secret name | `lfx-v2--secrets` (e.g., `lfx-v2-invite-service-secrets`) | | SecretStore name | `{{ .Chart.Name }}` | | IAM account — dev | `788942260905` | | IAM account — staging | `844790888233` | @@ -252,6 +252,7 @@ metadata: annotations: {{- toYaml . | nindent 4 }} {{- end }} + automountServiceAccountToken: {{ .Values.serviceAccount.automountServiceAccountToken | default true }} {{- end }} ``` @@ -262,6 +263,7 @@ serviceAccount: create: true name: "lfx-v2-" annotations: {} + automountServiceAccountToken: true ``` #### 4b. Custom resources in `lfx-v2-argocd` @@ -506,7 +508,7 @@ Check these repos for the exact file structure and conventions used in productio ### Adding an Auth0 client key pair to an existing service -1. Users asks: "Add the LFX V2 Persona Service auth0 client to lfx-v2-persona-service" +1. User asks: "Add the LFX V2 Persona Service auth0 client to lfx-v2-persona-service" 2. Add sync entry in lfx-secrets-management 3. Verify using the checklist above 4. Submit secrets PR From 6a0fc59fdbeb6a5991ecbb7428a357447c98f387 Mon Sep 17 00:00:00 2001 From: Antonia Gaete Date: Fri, 5 Jun 2026 16:52:11 -0400 Subject: [PATCH 03/11] changes after testing skill Signed-off-by: Antonia Gaete --- lfx-v2-secrets/SKILL.md | 108 +++++++++++++++++++++++++++++++++++++++- 1 file changed, 107 insertions(+), 1 deletion(-) diff --git a/lfx-v2-secrets/SKILL.md b/lfx-v2-secrets/SKILL.md index 126d8ab..458dda0 100644 --- a/lfx-v2-secrets/SKILL.md +++ b/lfx-v2-secrets/SKILL.md @@ -218,6 +218,60 @@ LFX V2 Invite Service JWT Secret: > - The `source.onepassword.item` should match exactly the name in 1Password vaults > - The field names should be descriptive enough to avoid duplicates (`litellm_api_key`, not just `api_key`) +**Auth0 template** (`auth0` — client_secret, `auth0_clients.yml`): + +> For LFX V2 services, always rename fields to a descriptive name prefixed with `auth0_` +> (e.g. `auth0_client_id`, `auth0_client_secret`) so keys are unambiguous in the merged K8s Secret. + +```yaml +: + tags: [auth0, ] + envs: [development, staging, production] + source: + auth0: + client_name: + rename_fields: + client_id: auth0_client_id + client_secret: auth0_client_secret + destinations: + - onepassword: + item: auth0 + field_types: + auth0_client_id: text + auth0_client_secret: password + - aws_secretsmanager: + path: auth0/ + tags: + service-: enabled +``` + +**Auth0 template** (`auth0_jwt` — JWT private key, standard for LFX V2 microservices, `auth0_clients.yml`): + +```yaml +: + tags: [auth0_jwt, ] + envs: [development, staging, production] + auto_rotate: true + source: + auth0_jwt: + client_name: + rename_fields: + client_id: + client_public_key: + client_private_key: + destinations: + - onepassword: + item: auth0 + field_types: + : text + : text + : password + - aws_secretsmanager: + path: auth0/ + tags: + service-: enabled +``` + > **Important**: After the `lfx-secrets-management` PR is merged, manually trigger the > [Deploy workflow](https://github.com/linuxfoundation/lfx-secrets-management/actions/workflows/deploy.yml) > to push the secret to AWS SM. Use the most specific secret config tag (the `tags:` field @@ -405,7 +459,59 @@ the right one. path: /<3rd-party-service>/ ``` -For auth0 sources, use the same pattern as Mode 1, Step 3 and add to `auth0_clients.yml`. +**Auth0 template** (`auth0` — client_secret, `auth0_clients.yml`): + +> For LFX V2 services, always rename fields to a descriptive name prefixed with `auth0_` +> (e.g. `auth0_client_id`, `auth0_client_secret`) so keys are unambiguous in the merged K8s Secret. + +```yaml +: + tags: [auth0, ] + envs: [development, staging, production] + source: + auth0: + client_name: + rename_fields: + client_id: auth0_client_id + client_secret: auth0_client_secret + destinations: + - onepassword: + item: auth0 + field_types: + auth0_client_id: text + auth0_client_secret: password + - aws_secretsmanager: + path: auth0/ + tags: + service-: enabled +``` + +**Auth0 template** (`auth0_jwt` — JWT private key, standard for LFX V2 microservices, `auth0_clients.yml`): + +```yaml +: + tags: [auth0_jwt, ] + envs: [development, staging, production] + auto_rotate: true + source: + auth0_jwt: + client_name: + rename_fields: + client_id: + client_public_key: + client_private_key: + destinations: + - onepassword: + item: auth0 + field_types: + : text + : text + : password + - aws_secretsmanager: + path: auth0/ + tags: + service-: enabled +``` > **Important**: After the `lfx-secrets-management` PR is merged, manually trigger the > [Deploy workflow](https://github.com/linuxfoundation/lfx-secrets-management/actions/workflows/deploy.yml) From b6e3fa8d32d0e5615760b51c2c6aa7f345423665 Mon Sep 17 00:00:00 2001 From: Antonia Gaete Date: Mon, 15 Jun 2026 10:48:28 -0700 Subject: [PATCH 04/11] examples are just examples, not options Signed-off-by: Antonia Gaete --- lfx-v2-secrets/SKILL.md | 18 +++++++++++------- 1 file changed, 11 insertions(+), 7 deletions(-) diff --git a/lfx-v2-secrets/SKILL.md b/lfx-v2-secrets/SKILL.md index 458dda0..8c08e2e 100644 --- a/lfx-v2-secrets/SKILL.md +++ b/lfx-v2-secrets/SKILL.md @@ -63,13 +63,17 @@ This touches **four repos** and requires changes in a specific order. ### Step 1: Prepare Information -Ask the user to collect: +**Before asking the user anything**, read `iam-service-account-definitions.yaml` in `lfx-v2-opentofu` +and look up the service's entry. Note the `namespace` and `eso_service_tag` values — both default +to the role key if not set. You will need these in Steps 3 and 4 and must not ask the user for them. + +Then ask the user to collect: 1. **Service name** — fully qualified name (e.g., `lfx-v2-invite-service`, `lfx-v2-email-service`) -2. **Namespace name** — the namespace in Kubernetes the service is deployed in -3. **List of secrets** — secrets can come from two source types; collect the relevant details for each: +2. **Namespace name** — read from `iam-service-account-definitions.yaml` (see above); confirm with the user only if it is missing from the file +3. **List of secrets** — secrets can come from several source types; the examples below cover the most common ones, but accept any source the user describes: - **1Password sources** (e.g., API keys, SMTP credentials, JWT secrets): + **1Password sources** (e.g., API keys, SMTP credentials, JWT secrets, etc): - **Secret name** (e.g., "JWT Secret", "SMTP Credentials") - **Third-party service** that provides the secret (e.g., `litellm`, `github`) - **1Password item name** — exact name as it appears in the vault @@ -83,7 +87,7 @@ Ask the user to collect: - **AWS SM path** — follows `auth0/` convention (e.g., `auth0/LFX_V2_Invite_Service`) 4. **Which environments need this secret** — `development`, `staging`, `production` -5. **Service tag** — applies to all secrets going to AWS SM; use the `eso_service_tag` value from this service's entry in `iam-service-accounts-definitions.yaml` in `lfx-v2-opentofu` (set in Step 2) — do not ask the user for this +5. **Service tag** — read from `iam-service-account-definitions.yaml` as `eso_service_tag` (defaults to the role key if not set) — do not ask the user for this **1Password example:** @@ -126,7 +130,7 @@ Auth0 secrets: ### Step 2: Create IAM Service Account in `lfx-v2-opentofu` In the [lfx-v2-opentofu](https://github.com/linuxfoundation/lfx-v2-opentofu) repo, -edit `iam-service-accounts-definitions.yaml` and add: +edit `iam-service-account-definitions.yaml` and add: ```yaml service_account_roles: @@ -549,7 +553,7 @@ After completing either mode, verify the setup: **File checklist — all repos involved:** -- [ ] `lfx-v2-opentofu`: `iam-service-accounts-definitions.yaml` has service entry *(Mode 1 only)* +- [ ] `lfx-v2-opentofu`: `iam-service-account-definitions.yaml` has service entry *(Mode 1 only)* - [ ] `lfx-secrets-management`: appropriate file under `secrets/lfx/` has sync entry for each secret; Deploy workflow run after merge - [ ] Service Helm chart *(Mode 1 only)*: - [ ] `templates/serviceaccount.yaml` created From 669bd033625a3fe8d690d9f033a5f56f8441647b Mon Sep 17 00:00:00 2001 From: Antonia Gaete Date: Mon, 15 Jun 2026 14:17:03 -0700 Subject: [PATCH 05/11] add note to make changes to self serve branch values Signed-off-by: Antonia Gaete --- lfx-v2-secrets/SKILL.md | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/lfx-v2-secrets/SKILL.md b/lfx-v2-secrets/SKILL.md index 8c08e2e..a9ac3a5 100644 --- a/lfx-v2-secrets/SKILL.md +++ b/lfx-v2-secrets/SKILL.md @@ -56,6 +56,14 @@ These values are fixed and apply across all V2 services: --- +## Branching + +Before making any changes, create a branch in each repo being modified. Use the format +`/`. Never commit directly to `main`. The username is the git +username (typically the part before `@` in the email). Always sign off commits. + +--- + ## Mode 1: New Service (Full Setup) Use this mode when a brand-new V2 service needs secrets wired up end-to-end from scratch. @@ -408,12 +416,18 @@ serviceAccount: | Staging | `844790888233` | `values/staging/lfx-v2-.yaml` | | Production | `372256339901` | `values/prod/lfx-v2-.yaml` | +> **lfx-self-serve**: If the service is `lfx-self-serve`, any update to a values file in +> `lfx-v2-argocd` must also include the corresponding update to `lfx-self-serve-branch`. + ### Step 5: Wire Secrets into Service Environment in `lfx-v2-argocd` In `values/global/lfx-v2-.yaml`, add an `environment` block that maps each secret field to an environment variable. Reference the Kubernetes Secret created by the ExternalSecret (`lfx-v2--secrets`) and use the field name as the key. +> **lfx-self-serve**: If the service is `lfx-self-serve`, also update `lfx-self-serve-branch` +> whenever this values file is modified. + ```yaml # Copyright The Linux Foundation and each contributor to LFX. # SPDX-License-Identifier: MIT @@ -545,6 +559,9 @@ environment: > Tag-based discovery means the new secret is picked up automatically — no changes to > `ExternalSecret.yaml` are needed as long as the AWS SM tag matches the service. +> **lfx-self-serve**: If the service is `lfx-self-serve`, also update `lfx-self-serve-branch` +> whenever this values file is modified. + --- ## Verification Checklist @@ -565,6 +582,7 @@ After completing either mode, verify the setup: - [ ] `values/staging/lfx-v2-.yaml` has IRSA role ARN + `automountServiceAccountToken: true` *(Mode 1 only)* - [ ] `values/prod/lfx-v2-.yaml` has IRSA role ARN + `automountServiceAccountToken: true` *(Mode 1 only)* - [ ] `values/global/lfx-v2-.yaml` has `environment` block with `secretKeyRef` entries for all secrets + - [ ] **lfx-self-serve only**: `lfx-self-serve-branch` updated alongside any `lfx-self-serve` values file change **Configuration checks:** From 22f3134da01f79f72fe36842576c5cc492c1f6a4 Mon Sep 17 00:00:00 2001 From: Antonia Gaete Date: Mon, 15 Jun 2026 14:39:42 -0700 Subject: [PATCH 06/11] copilot suggestions Signed-off-by: Antonia Gaete --- lfx-v2-secrets/SKILL.md | 21 +++++++++++---------- 1 file changed, 11 insertions(+), 10 deletions(-) diff --git a/lfx-v2-secrets/SKILL.md b/lfx-v2-secrets/SKILL.md index a9ac3a5..ce893e7 100644 --- a/lfx-v2-secrets/SKILL.md +++ b/lfx-v2-secrets/SKILL.md @@ -45,7 +45,7 @@ These values are fixed and apply across all V2 services: |------|-------| | AWS Region | `us-west-2` | | K8s Secret name | `lfx-v2--secrets` (e.g., `lfx-v2-invite-service-secrets`) | -| SecretStore name | `{{ .Chart.Name }}` | +| SecretStore name | `lfx-v2-` — matches `metadata.name` in `SecretStore.yaml` and `spec.secretStoreRef.name` in `ExternalSecret.yaml` | | IAM account — dev | `788942260905` | | IAM account — staging | `844790888233` | | IAM account — prod | `372256339901` | @@ -200,26 +200,27 @@ own file (e.g., `litellm.yml`). When unsure, grep for the third-party service na path: /<3rd-party-service>/ ``` -Example for invite service JWT secret: +Example for Supabase API key: ```yaml -LFX V2 Invite Service JWT Secret: - tags: [lfx_v2, invite, jwt] - environments: [development, staging, production] +Supabase API Key: + tags: [supabase, supabase_api_key, lfx_v2] + envs: [core, development, staging, production] source: onepassword: vaults: development: LFX V2 - Development - staging: LFX V2 - Staging production: LFX V2 - Production - item: "LFX V2 Invite Service - JWT Secret" + staging: LFX V2 - Staging + item: LFX v2 supabase fields: - - jwt_secret_key + - url + - api_key destinations: - aws_secretsmanager: tags: - service-lfx-v2-invite-service: enabled - path: "cloud/invite/jwt" + service: pcc + path: cloud/supabase/api_key ``` > **Tips**: From 1f8aff4be6061a9114672c3d2eb8be3c8c72caf1 Mon Sep 17 00:00:00 2001 From: Antonia Gaete Date: Mon, 15 Jun 2026 14:41:55 -0700 Subject: [PATCH 07/11] copilot suggestions Signed-off-by: Antonia Gaete --- lfx-v2-secrets/SKILL.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/lfx-v2-secrets/SKILL.md b/lfx-v2-secrets/SKILL.md index ce893e7..f993026 100644 --- a/lfx-v2-secrets/SKILL.md +++ b/lfx-v2-secrets/SKILL.md @@ -71,9 +71,10 @@ This touches **four repos** and requires changes in a specific order. ### Step 1: Prepare Information -**Before asking the user anything**, read `iam-service-account-definitions.yaml` in `lfx-v2-opentofu` -and look up the service's entry. Note the `namespace` and `eso_service_tag` values — both default -to the role key if not set. You will need these in Steps 3 and 4 and must not ask the user for them. +**Before asking the user anything**, read `iam-service-account-definitions.yaml` in `lfx-v2-opentofu`. +If the service already has an entry, note the `namespace` and `eso_service_tag` values — both default +to the role key if not set. If it does not exist yet (common for a brand-new service), you will add +it in Step 2 and then use those values in Steps 3 and 4. Then ask the user to collect: From 8285cd1fffaa768f6e55bf1cdbef2bc871a46a22 Mon Sep 17 00:00:00 2001 From: Antonia Gaete Date: Mon, 15 Jun 2026 14:55:53 -0700 Subject: [PATCH 08/11] more copilot suggestions Signed-off-by: Antonia Gaete --- {lfx-v2-secrets => skills/lfx-v2-secrets}/SKILL.md | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) rename {lfx-v2-secrets => skills/lfx-v2-secrets}/SKILL.md (98%) diff --git a/lfx-v2-secrets/SKILL.md b/skills/lfx-v2-secrets/SKILL.md similarity index 98% rename from lfx-v2-secrets/SKILL.md rename to skills/lfx-v2-secrets/SKILL.md index f993026..e086f49 100644 --- a/lfx-v2-secrets/SKILL.md +++ b/skills/lfx-v2-secrets/SKILL.md @@ -105,7 +105,7 @@ Service: lfx-v2-invite-service Namespace: invite-service Secrets: - Atlassian API Key (service: atlassian, field: atlassian_api_key) - all envs - - JWT Secret (service: jwt, field: jwt_secret) — all envs + - Supabase API Key (service: pcc, fields: url, api_key) - all envs ``` **Auth0 examples:** @@ -184,7 +184,7 @@ own file (e.g., `litellm.yml`). When unsure, grep for the third-party service na ```yaml : tags: [lfx_v2, , ] - environments: [development, staging, production] + envs: [development, staging, production] source: onepassword: vaults: @@ -206,7 +206,7 @@ Example for Supabase API key: ```yaml Supabase API Key: tags: [supabase, supabase_api_key, lfx_v2] - envs: [core, development, staging, production] + envs: [development, staging, production] source: onepassword: vaults: @@ -455,6 +455,10 @@ Use this mode when adding a new secret to a service that already has ESO + IRSA ### Step 1: Add Entry to `lfx-secrets-management` +**Before writing the entry**, read `iam-service-account-definitions.yaml` in `lfx-v2-opentofu` +and confirm the service's `eso_service_tag` (defaults to the role key if not set). You will need +this for the `service-: enabled` AWS SM resource tag — do not ask the user for it. + Follow the same pattern as Mode 1, Step 3. Add an entry to the appropriate file under `secrets/lfx/` — check existing files or grep for the third-party service name to find the right one. From a27b36a147c985e3154ddfed53876e92e8374313 Mon Sep 17 00:00:00 2001 From: Antonia Gaete Date: Mon, 15 Jun 2026 15:11:59 -0700 Subject: [PATCH 09/11] more copilot suggestions Signed-off-by: Antonia Gaete --- skills/lfx-v2-secrets/SKILL.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/skills/lfx-v2-secrets/SKILL.md b/skills/lfx-v2-secrets/SKILL.md index e086f49..8acbcfe 100644 --- a/skills/lfx-v2-secrets/SKILL.md +++ b/skills/lfx-v2-secrets/SKILL.md @@ -220,7 +220,7 @@ Supabase API Key: destinations: - aws_secretsmanager: tags: - service: pcc + service-pcc: enabled path: cloud/supabase/api_key ``` @@ -228,7 +228,7 @@ Supabase API Key: > > - Each secret in the lfx-secrets-management source becomes a separate AWS SM path entry > - The `path` convention is `cloud//` -> - Use the `environments` list to sync to all three environments in parallel +> - Use the `envs` list to sync to all three environments in parallel > - The `source.onepassword.item` should match exactly the name in 1Password vaults > - The field names should be descriptive enough to avoid duplicates (`litellm_api_key`, not just `api_key`) @@ -320,7 +320,7 @@ metadata: annotations: {{- toYaml . | nindent 4 }} {{- end }} - automountServiceAccountToken: {{ .Values.serviceAccount.automountServiceAccountToken | default true }} +automountServiceAccountToken: {{ .Values.serviceAccount.automountServiceAccountToken | default true }} {{- end }} ``` @@ -385,7 +385,7 @@ spec: conversionStrategy: Default decodingStrategy: None tags: - service-lfx-v2-: enabled + service-: enabled rewrite: - merge: conflictPolicy: Error @@ -394,7 +394,7 @@ spec: ``` > **Tag-based discovery**: ESO finds and merges all AWS SM secrets tagged -> `service-lfx-v2-: enabled` into a single Kubernetes Secret named +> `service-: enabled` into a single Kubernetes Secret named > `lfx-v2--secrets`. No manual `data` list is needed — new secrets are picked up > automatically after the next sync. @@ -466,7 +466,7 @@ the right one. ```yaml : tags: [lfx_v2, , ] - environments: [development, staging, production] + envs: [development, staging, production] source: onepassword: vaults: From 056ebafbee20e02ae075b7153d40cff0a9a209ec Mon Sep 17 00:00:00 2001 From: Antonia Gaete Date: Mon, 15 Jun 2026 15:39:24 -0700 Subject: [PATCH 10/11] check opentofu from github, clarify service names Signed-off-by: Antonia Gaete --- skills/lfx-v2-secrets/SKILL.md | 131 ++++++++++++++++++--------------- 1 file changed, 71 insertions(+), 60 deletions(-) diff --git a/skills/lfx-v2-secrets/SKILL.md b/skills/lfx-v2-secrets/SKILL.md index 8acbcfe..c5592a4 100644 --- a/skills/lfx-v2-secrets/SKILL.md +++ b/skills/lfx-v2-secrets/SKILL.md @@ -44,12 +44,12 @@ These values are fixed and apply across all V2 services: | Item | Value | |------|-------| | AWS Region | `us-west-2` | -| K8s Secret name | `lfx-v2--secrets` (e.g., `lfx-v2-invite-service-secrets`) | -| SecretStore name | `lfx-v2-` — matches `metadata.name` in `SecretStore.yaml` and `spec.secretStoreRef.name` in `ExternalSecret.yaml` | +| K8s Secret name | `-secrets` (e.g., `lfx-v2-committee-service-secrets`) | +| SecretStore name | `` — matches `metadata.name` in `SecretStore.yaml` and `spec.secretStoreRef.name` in `ExternalSecret.yaml` | | IAM account — dev | `788942260905` | | IAM account — staging | `844790888233` | | IAM account — prod | `372256339901` | -| IRSA role ARN pattern | `arn:aws:iam:::role/lfx-v2-` | +| IRSA role ARN pattern | `arn:aws:iam:::role/` | | AWS SM path pattern | `cloud/<3rd-party-service>/` | | ServiceAccount annotation key | `eks.amazonaws.com/role-arn` | | ESO JWT auth field | `spec.provider.aws.auth.jwt.serviceAccountRef` | @@ -71,16 +71,24 @@ This touches **four repos** and requires changes in a specific order. ### Step 1: Prepare Information -**Before asking the user anything**, read `iam-service-account-definitions.yaml` in `lfx-v2-opentofu`. -If the service already has an entry, note the `namespace` and `eso_service_tag` values — both default -to the role key if not set. If it does not exist yet (common for a brand-new service), you will add -it in Step 2 and then use those values in Steps 3 and 4. +Identify `` — the fully qualified service name including the `lfx-v2-` prefix (e.g., +`lfx-v2-committee-service`). If the user did not include it in their request, ask for it now +before proceeding. This is used directly in all resource names: K8s Secret is `-secrets`, +role ARN ends in ``, etc. -Then ask the user to collect: +Then fetch `iam-service-account-definitions.yaml` directly from GitHub: +``` +https://raw.githubusercontent.com/linuxfoundation/lfx-v2-opentofu/main/iam-service-account-definitions.yaml +``` +If the fetch fails (e.g., auth error), ask the user to paste the relevant entry. + +Look up the entry for ``: +- **`namespace`** — note the value; defaults to `` if not set. Confirm with the user only if the entry is missing entirely. +- **`eso_service_tag`** — note the value; defaults to `` if not set. If the file is inaccessible or the entry is missing, ask the user to confirm the tag (suggest `` as the default). -1. **Service name** — fully qualified name (e.g., `lfx-v2-invite-service`, `lfx-v2-email-service`) -2. **Namespace name** — read from `iam-service-account-definitions.yaml` (see above); confirm with the user only if it is missing from the file -3. **List of secrets** — secrets can come from several source types; the examples below cover the most common ones, but accept any source the user describes: +Then ask the user for: + +1. **List of secrets** — secrets can come from several source types; the examples below cover the most common ones, but accept any source the user describes: **1Password sources** (e.g., API keys, SMTP credentials, JWT secrets, etc): - **Secret name** (e.g., "JWT Secret", "SMTP Credentials") @@ -95,8 +103,7 @@ Then ask the user to collect: - **Auto-rotate** — yes/no; default `true` for `auth0_jwt` V2 services - **AWS SM path** — follows `auth0/` convention (e.g., `auth0/LFX_V2_Invite_Service`) -4. **Which environments need this secret** — `development`, `staging`, `production` -5. **Service tag** — read from `iam-service-account-definitions.yaml` as `eso_service_tag` (defaults to the role key if not set) — do not ask the user for this +2. **Which environments need this secret** — `development`, `staging`, `production` **1Password example:** @@ -113,27 +120,27 @@ Secrets: `auth0` (client_secret, simpler M2M or BFF clients): ```text -Service: lfx-v2-invite-service -Namespace: invite-service +Service: lfx-v2-committee-service +Namespace: committee-service Auth0 secrets: - - Invite Service BFF + - Committee Service BFF type: auth0 - client: "LFX V2 Invite BFF" - path: auth0/LFX_V2_Invite_BFF + client: "LFX V2 Committee BFF" + path: auth0/LFX_V2_Committee_BFF ``` `auth0_jwt` (JWT private key, standard for LFX V2 microservices): ```text -Service: lfx-v2-invite-service -Namespace: invite-service +Service: lfx-v2-committee-service +Namespace: committee-service Auth0 secrets: - - Invite Service M2M + - Committee Service M2M type: auth0_jwt - client: "LFX V2 Invite Service" + client: "LFX V2 Committee Service" auto_rotate: true rename_fields: client_id → auth0_client_id, client_private_key → auth0_client_private_key - path: auth0/LFX_V2_Invite_Service + path: auth0/LFX_V2_Committee_Service ``` ### Step 2: Create IAM Service Account in `lfx-v2-opentofu` @@ -143,19 +150,19 @@ edit `iam-service-account-definitions.yaml` and add: ```yaml service_account_roles: - lfx-v2-: + : namespace: "" ``` -Example for invite service: +Example for committee service: ```yaml service_account_roles: - lfx-v2-invite-service: - namespace: "invite-service" + lfx-v2-committee-service: + namespace: "committee-service" ``` -> **Defaults**: All fields default to the role key (e.g., `lfx-v2-invite-service`): `namespace`, +> **Defaults**: All fields default to the role key (i.e., ``): `namespace`, > `service_account`, and `eso_service_tag`. Only specify a field when its value differs > from the role key. @@ -227,7 +234,7 @@ Supabase API Key: > **Tips**: > > - Each secret in the lfx-secrets-management source becomes a separate AWS SM path entry -> - The `path` convention is `cloud//` +> - The `path` convention is `cloud/<3rd-party-service>/` > - Use the `envs` list to sync to all three environments in parallel > - The `source.onepassword.item` should match exactly the name in 1Password vaults > - The field names should be descriptive enough to avoid duplicates (`litellm_api_key`, not just `api_key`) @@ -302,7 +309,7 @@ Supabase API Key: #### 4a. `serviceaccount.yaml` in the service Helm chart In the service repo's Helm chart (e.g., `lfx-v2-invite-service`), create -`charts/lfx-v2-/templates/serviceaccount.yaml`: +`charts//templates/serviceaccount.yaml`: ```yaml # Copyright The Linux Foundation and each contributor to LFX. @@ -324,12 +331,12 @@ automountServiceAccountToken: {{ .Values.serviceAccount.automountServiceAccountT {{- end }} ``` -Add to `charts/lfx-v2-/values.yaml`: +Add to `charts//values.yaml`: ```yaml serviceAccount: create: true - name: "lfx-v2-" + name: "" annotations: {} automountServiceAccountToken: true ``` @@ -337,9 +344,9 @@ serviceAccount: #### 4b. Custom resources in `lfx-v2-argocd` The `SecretStore` and `ExternalSecret` are **static YAML files** (not Helm templates) placed in -`lfx-v2-argocd/custom-resources/lfx-v2-/`. +`lfx-v2-argocd/custom-resources//`. -Create `custom-resources/lfx-v2-/SecretStore.yaml`: +Create `custom-resources//SecretStore.yaml`: ```yaml # Copyright The Linux Foundation and each contributor to LFX. @@ -348,7 +355,7 @@ Create `custom-resources/lfx-v2-/SecretStore.yaml`: apiVersion: external-secrets.io/v1 kind: SecretStore metadata: - name: lfx-v2- + name: namespace: spec: provider: @@ -356,12 +363,12 @@ spec: auth: jwt: serviceAccountRef: - name: lfx-v2- + name: region: us-west-2 service: SecretsManager ``` -Create `custom-resources/lfx-v2-/ExternalSecret.yaml`: +Create `custom-resources//ExternalSecret.yaml`: ```yaml # Copyright The Linux Foundation and each contributor to LFX. @@ -370,15 +377,15 @@ Create `custom-resources/lfx-v2-/ExternalSecret.yaml`: apiVersion: external-secrets.io/v1 kind: ExternalSecret metadata: - name: lfx-v2- + name: namespace: spec: secretStoreRef: kind: SecretStore - name: lfx-v2- + name: target: creationPolicy: Owner - name: lfx-v2--secrets + name: -secrets refreshInterval: 10m dataFrom: - find: @@ -395,12 +402,12 @@ spec: > **Tag-based discovery**: ESO finds and merges all AWS SM secrets tagged > `service-: enabled` into a single Kubernetes Secret named -> `lfx-v2--secrets`. No manual `data` list is needed — new secrets are picked up +> `-secrets`. No manual `data` list is needed — new secrets are picked up > automatically after the next sync. #### 4c. IRSA annotation in `lfx-v2-argocd` per-environment values -In `values/dev/lfx-v2-.yaml` (repeat for staging and prod with the matching account ID): +In `values/dev/.yaml` (repeat for staging and prod with the matching account ID): ```yaml # Copyright The Linux Foundation and each contributor to LFX. @@ -408,24 +415,24 @@ In `values/dev/lfx-v2-.yaml` (repeat for staging and prod with the matc --- serviceAccount: annotations: - eks.amazonaws.com/role-arn: arn:aws:iam::788942260905:role/lfx-v2- + eks.amazonaws.com/role-arn: arn:aws:iam::788942260905:role/ automountServiceAccountToken: true ``` | Environment | Account ID | File | |-------------|-----------|------| -| Development | `788942260905` | `values/dev/lfx-v2-.yaml` | -| Staging | `844790888233` | `values/staging/lfx-v2-.yaml` | -| Production | `372256339901` | `values/prod/lfx-v2-.yaml` | +| Development | `788942260905` | `values/dev/.yaml` | +| Staging | `844790888233` | `values/staging/.yaml` | +| Production | `372256339901` | `values/prod/.yaml` | > **lfx-self-serve**: If the service is `lfx-self-serve`, any update to a values file in > `lfx-v2-argocd` must also include the corresponding update to `lfx-self-serve-branch`. ### Step 5: Wire Secrets into Service Environment in `lfx-v2-argocd` -In `values/global/lfx-v2-.yaml`, add an `environment` block that maps each secret +In `values/global/.yaml`, add an `environment` block that maps each secret field to an environment variable. Reference the Kubernetes Secret created by the ExternalSecret -(`lfx-v2--secrets`) and use the field name as the key. +(`-secrets`) and use the field name as the key. > **lfx-self-serve**: If the service is `lfx-self-serve`, also update `lfx-self-serve-branch` > whenever this values file is modified. @@ -438,12 +445,12 @@ environment: SECRET_NAME: valueFrom: secretKeyRef: - name: lfx-v2--secrets + name: -secrets key: ANOTHER_SECRET_NAME: valueFrom: secretKeyRef: - name: lfx-v2--secrets + name: -secrets key: ``` @@ -550,7 +557,7 @@ the right one. ### Step 2: Wire Secret into Service Environment in `lfx-v2-argocd` -In `values/global/lfx-v2-.yaml`, add the new environment variable to the existing +In `values/global/.yaml`, add the new environment variable to the existing `environment` block: ```yaml @@ -558,10 +565,14 @@ environment: NEW_ENV_VAR: valueFrom: secretKeyRef: - name: lfx-v2--secrets + name: -secrets key: ``` +> Before writing `secretKeyRef.name`, verify the K8s Secret name from the existing +> `ExternalSecret.yaml` in `lfx-v2-argocd/custom-resources//` — +> check `spec.target.name`. It is typically `-secrets` but must match exactly. + > Tag-based discovery means the new secret is picked up automatically — no changes to > `ExternalSecret.yaml` are needed as long as the AWS SM tag matches the service. @@ -582,22 +593,22 @@ After completing either mode, verify the setup: - [ ] `templates/serviceaccount.yaml` created - [ ] `values.yaml` has `serviceAccount` block - [ ] `lfx-v2-argocd`: - - [ ] `custom-resources/lfx-v2-/SecretStore.yaml` created *(Mode 1 only)* - - [ ] `custom-resources/lfx-v2-/ExternalSecret.yaml` created *(Mode 1 only)* - - [ ] `values/dev/lfx-v2-.yaml` has IRSA role ARN + `automountServiceAccountToken: true` *(Mode 1 only)* - - [ ] `values/staging/lfx-v2-.yaml` has IRSA role ARN + `automountServiceAccountToken: true` *(Mode 1 only)* - - [ ] `values/prod/lfx-v2-.yaml` has IRSA role ARN + `automountServiceAccountToken: true` *(Mode 1 only)* - - [ ] `values/global/lfx-v2-.yaml` has `environment` block with `secretKeyRef` entries for all secrets + - [ ] `custom-resources//SecretStore.yaml` created *(Mode 1 only)* + - [ ] `custom-resources//ExternalSecret.yaml` created *(Mode 1 only)* + - [ ] `values/dev/.yaml` has IRSA role ARN + `automountServiceAccountToken: true` *(Mode 1 only)* + - [ ] `values/staging/.yaml` has IRSA role ARN + `automountServiceAccountToken: true` *(Mode 1 only)* + - [ ] `values/prod/.yaml` has IRSA role ARN + `automountServiceAccountToken: true` *(Mode 1 only)* + - [ ] `values/global/.yaml` has `environment` block with `secretKeyRef` entries for all secrets - [ ] **lfx-self-serve only**: `lfx-self-serve-branch` updated alongside any `lfx-self-serve` values file change **Configuration checks:** -- [ ] IRSA role ARN format is correct: `arn:aws:iam:::role/lfx-v2-` +- [ ] IRSA role ARN format is correct: `arn:aws:iam:::role/` - [ ] All account IDs are correct (dev=`788942260905`, staging=`844790888233`, prod=`372256339901`) - [ ] AWS SM secret config tags (`tags:` field) are specific enough to scope the deploy workflow run - [ ] AWS SM resource tag on each secret matches `eso_service_tag`: `service-: enabled` - [ ] All secrets are stored as JSON (list form fields or `store_as_json: true`) -- [ ] `ExternalSecret` target name is `lfx-v2--secrets` and `secretKeyRef` names match +- [ ] `ExternalSecret` target name is `-secrets` and `secretKeyRef` names match **1Password setup** *(1Password sources only)*: From 64f946075e054f3c87d4376de04b50d14a1ab589 Mon Sep 17 00:00:00 2001 From: Antonia Gaete Date: Tue, 16 Jun 2026 13:34:47 -0700 Subject: [PATCH 11/11] update skill based on new file and path naming Signed-off-by: Antonia Gaete --- skills/lfx-v2-secrets/SKILL.md | 474 +++++++++++++++------------------ 1 file changed, 210 insertions(+), 264 deletions(-) diff --git a/skills/lfx-v2-secrets/SKILL.md b/skills/lfx-v2-secrets/SKILL.md index c5592a4..fd59b72 100644 --- a/skills/lfx-v2-secrets/SKILL.md +++ b/skills/lfx-v2-secrets/SKILL.md @@ -2,12 +2,12 @@ name: lfx-v2-secrets description: > Guide an agent through wiring up secrets for LFX V2 microservices using External Secrets - Operator (ESO) + IRSA. Supports two modes: (1) full setup for new services touching - lfx-v2-opentofu, lfx-secrets-management, the service Helm chart, and lfx-v2-argocd; - (2) adding secrets to existing services already configured with ESO. Use this skill - whenever someone says "set up secrets", "wire up ESO", "add a secret to this service", - "IRSA configuration", "External Secrets for V2", or any mention of AWS Secrets Manager - integration with Kubernetes for LFX V2 services. + Operator (ESO) + IRSA. Handles both new services (full infrastructure setup) and existing + services (add a secret to an already-configured service) by checking whether the ESO + objects exist before deciding which steps to run. Use this skill whenever someone says + "set up secrets", "wire up ESO", "add a secret to this service", "IRSA configuration", + "External Secrets for V2", or any mention of AWS Secrets Manager integration with + Kubernetes for LFX V2 services. allowed-tools: Bash, Read, Glob, Grep, AskUserQuestion, WebFetch --- @@ -21,10 +21,6 @@ Secrets for LFX V2 microservices are managed through **External Secrets Operator combined with **IAM Roles for Service Accounts (IRSA)** on AWS. This provides a secure, GitOps-driven way to sync secrets from AWS Secrets Manager into Kubernetes. -> **For the AI**: This skill has two modes. Mode 1 (new service) touches four repos -> and requires coordinated changes across infrastructure layers. Mode 2 (existing service) -> is much smaller. Always ask the user which applies before proceeding. - --- ## Understanding the Architecture @@ -50,7 +46,7 @@ These values are fixed and apply across all V2 services: | IAM account — staging | `844790888233` | | IAM account — prod | `372256339901` | | IRSA role ARN pattern | `arn:aws:iam:::role/` | -| AWS SM path pattern | `cloud/<3rd-party-service>/` | +| AWS SM path pattern | `<3rd-party-service>//[]` | | ServiceAccount annotation key | `eks.amazonaws.com/role-arn` | | ESO JWT auth field | `spec.provider.aws.auth.jwt.serviceAccountRef` | @@ -64,34 +60,19 @@ username (typically the part before `@` in the email). Always sign off commits. --- -## Mode 1: New Service (Full Setup) - -Use this mode when a brand-new V2 service needs secrets wired up end-to-end from scratch. -This touches **four repos** and requires changes in a specific order. - -### Step 1: Prepare Information +## Step 1: Gather Information from the User Identify `` — the fully qualified service name including the `lfx-v2-` prefix (e.g., `lfx-v2-committee-service`). If the user did not include it in their request, ask for it now before proceeding. This is used directly in all resource names: K8s Secret is `-secrets`, role ARN ends in ``, etc. -Then fetch `iam-service-account-definitions.yaml` directly from GitHub: -``` -https://raw.githubusercontent.com/linuxfoundation/lfx-v2-opentofu/main/iam-service-account-definitions.yaml -``` -If the fetch fails (e.g., auth error), ask the user to paste the relevant entry. +If not already provided in the initial request, ask the user for: -Look up the entry for ``: -- **`namespace`** — note the value; defaults to `` if not set. Confirm with the user only if the entry is missing entirely. -- **`eso_service_tag`** — note the value; defaults to `` if not set. If the file is inaccessible or the entry is missing, ask the user to confirm the tag (suggest `` as the default). - -Then ask the user for: +1. **List of secrets** — LFX V2 secrets come from either 1Password or Auth0: -1. **List of secrets** — secrets can come from several source types; the examples below cover the most common ones, but accept any source the user describes: - - **1Password sources** (e.g., API keys, SMTP credentials, JWT secrets, etc): - - **Secret name** (e.g., "JWT Secret", "SMTP Credentials") + **1Password sources** (e.g., API keys, SMTP credentials, etc): + - **Secret name** (e.g., "LinkedIn Credentials", "SMTP Credentials") - **Third-party service** that provides the secret (e.g., `litellm`, `github`) - **1Password item name** — exact name as it appears in the vault - **Field names in 1Password** — the exact field names as they appear in the source vault @@ -99,10 +80,11 @@ Then ask the user for: **Auth0 sources** (e.g., M2M client credentials, BFF client secrets): - **Auth0 client name** — exact display name in Auth0 (e.g., `LFX V2 Invite Service`) - **Credential type** — `auth0` (produces `client_id` + `client_secret`) or `auth0_jwt` (produces `client_id` + `client_public_key` + `client_private_key`; standard for LFX V2 microservices) - - **Field rename mapping** (optional) — if env var names differ from defaults (e.g., `client_id` → `auth0_client_id`) - **Auto-rotate** — yes/no; default `true` for `auth0_jwt` V2 services - **AWS SM path** — follows `auth0/` convention (e.g., `auth0/LFX_V2_Invite_Service`) + > Field renames are applied automatically — always prefix with `auth0_` (e.g., `client_id` → `auth0_client_id`, `client_secret` → `auth0_client_secret`, `client_private_key` → `auth0_client_private_key`). Do not ask the user for these. + 2. **Which environments need this secret** — `development`, `staging`, `production` **1Password example:** @@ -111,8 +93,8 @@ Then ask the user for: Service: lfx-v2-invite-service Namespace: invite-service Secrets: - - Atlassian API Key (service: atlassian, field: atlassian_api_key) - all envs - - Supabase API Key (service: pcc, fields: url, api_key) - all envs + - Atlassian API Key (3rd party service: atlassian, 1Password field: atlassian_api_key) - all envs + - Supabase API Key (3rd party service: supabase, 1Password fields: url, api_key) - all envs ``` **Auth0 examples:** @@ -139,174 +121,83 @@ Auth0 secrets: type: auth0_jwt client: "LFX V2 Committee Service" auto_rotate: true - rename_fields: client_id → auth0_client_id, client_private_key → auth0_client_private_key path: auth0/LFX_V2_Committee_Service ``` -### Step 2: Create IAM Service Account in `lfx-v2-opentofu` +--- -In the [lfx-v2-opentofu](https://github.com/linuxfoundation/lfx-v2-opentofu) repo, -edit `iam-service-account-definitions.yaml` and add: +## Step 2: Check Whether ESO Is Already Configured -```yaml -service_account_roles: - : - namespace: "" -``` +Before making any changes, look up the service's infrastructure details and determine whether +`SecretStore` and `ExternalSecret` objects already exist. -Example for committee service: +### 2a. Fetch IAM service account definitions -```yaml -service_account_roles: - lfx-v2-committee-service: - namespace: "committee-service" +Fetch `iam-service-account-definitions.yaml` directly from GitHub: +``` +https://raw.githubusercontent.com/linuxfoundation/lfx-v2-opentofu/main/iam-service-account-definitions.yaml ``` +If the fetch fails (e.g., auth error), ask the user to provide the service's namespace and eso_service_tag. -> **Defaults**: All fields default to the role key (i.e., ``): `namespace`, -> `service_account`, and `eso_service_tag`. Only specify a field when its value differs -> from the role key. +Look up the entry for ``: +- **`namespace`** — note the value; defaults to `` if not set. Confirm with the user only if the entry is missing entirely. +- **`eso_service_tag`** — note the value; defaults to `` if not set. If the file is inaccessible or the entry is missing, ask the user to confirm the tag (suggest `` as the default). -### Step 3: Create Sync Entries in `lfx-secrets-management` +### 2b. Check for existing ESO objects -In the [lfx-secrets-management](https://github.com/linuxfoundation/lfx-secrets-management) repo, -add an entry for each secret to the appropriate file under `secrets/lfx/`. Check the existing -files to find where similar secrets live — auth0 credentials go in `auth0_clients.yml`, -most one-off service credentials go in `cloud.yml`, but third-party services may have their -own file (e.g., `litellm.yml`). When unsure, grep for the third-party service name across -`secrets/lfx/`. +Determine whether `SecretStore` and `ExternalSecret` objects already exist for this service +by fetching from GitHub. The service repo name matches the service name (e.g., `lfx-v2-committee-service` +lives at `github.com/linuxfoundation/lfx-v2-committee-service`). -> **Important**: All secrets must be stored as JSON in AWS SM, even single-field ones. -> Two equivalent ways to express this: -> -> ```yaml -> # implicit JSON (list form) -> fields: -> - -> -> # explicit JSON (scalar + flag) -> fields: -> store_as_json: true -> ``` +Fetch these three URLs (a 404 means the file doesn't exist yet): -```yaml -: - tags: [lfx_v2, , ] - envs: [development, staging, production] - source: - onepassword: - vaults: - development: LFX V2 - Development - staging: LFX V2 - Staging - production: LFX V2 - Production - item: <1Password Item Name> - fields: - - - destinations: - - aws_secretsmanager: - tags: - service-: enabled - path: /<3rd-party-service>/ ``` +# ESO custom resources in lfx-v2-argocd +https://raw.githubusercontent.com/linuxfoundation/lfx-v2-argocd/main/custom-resources//SecretStore.yaml +https://raw.githubusercontent.com/linuxfoundation/lfx-v2-argocd/main/custom-resources//ExternalSecret.yaml -Example for Supabase API key: - -```yaml -Supabase API Key: - tags: [supabase, supabase_api_key, lfx_v2] - envs: [development, staging, production] - source: - onepassword: - vaults: - development: LFX V2 - Development - production: LFX V2 - Production - staging: LFX V2 - Staging - item: LFX v2 supabase - fields: - - url - - api_key - destinations: - - aws_secretsmanager: - tags: - service-pcc: enabled - path: cloud/supabase/api_key +# ServiceAccount in the service Helm chart +https://raw.githubusercontent.com/linuxfoundation//main/charts//templates/serviceaccount.yaml ``` -> **Tips**: -> -> - Each secret in the lfx-secrets-management source becomes a separate AWS SM path entry -> - The `path` convention is `cloud/<3rd-party-service>/` -> - Use the `envs` list to sync to all three environments in parallel -> - The `source.onepassword.item` should match exactly the name in 1Password vaults -> - The field names should be descriptive enough to avoid duplicates (`litellm_api_key`, not just `api_key`) +If any fetch returns a non-404 auth error, fall back to checking the local filesystem if the +repo is checked out, or ask the user whether ESO is already configured for this service. -**Auth0 template** (`auth0` — client_secret, `auth0_clients.yml`): +**If all three files are present** → skip to [Step 4](#step-4-add-entry-to-lfx-secrets-management). -> For LFX V2 services, always rename fields to a descriptive name prefixed with `auth0_` -> (e.g. `auth0_client_id`, `auth0_client_secret`) so keys are unambiguous in the merged K8s Secret. +**If any are missing** → continue with Step 3 to set up the infrastructure first. + +--- + +## Step 3: Set Up ESO Infrastructure (New Services Only) + +Run this step only when Step 2 found that ESO is not yet configured. This touches three repos +and must be done before writing any secrets. + +### Step 3a: Add IAM Service Account Entry in `lfx-v2-opentofu` + +In the [lfx-v2-opentofu](https://github.com/linuxfoundation/lfx-v2-opentofu) repo, +edit `iam-service-account-definitions.yaml` and add: ```yaml -: - tags: [auth0, ] - envs: [development, staging, production] - source: - auth0: - client_name: - rename_fields: - client_id: auth0_client_id - client_secret: auth0_client_secret - destinations: - - onepassword: - item: auth0 - field_types: - auth0_client_id: text - auth0_client_secret: password - - aws_secretsmanager: - path: auth0/ - tags: - service-: enabled +service_account_roles: + : + namespace: "" ``` -**Auth0 template** (`auth0_jwt` — JWT private key, standard for LFX V2 microservices, `auth0_clients.yml`): +Example for committee service: ```yaml -: - tags: [auth0_jwt, ] - envs: [development, staging, production] - auto_rotate: true - source: - auth0_jwt: - client_name: - rename_fields: - client_id: - client_public_key: - client_private_key: - destinations: - - onepassword: - item: auth0 - field_types: - : text - : text - : password - - aws_secretsmanager: - path: auth0/ - tags: - service-: enabled +service_account_roles: + lfx-v2-committee-service: + namespace: "committee-service" ``` -> **Important**: After the `lfx-secrets-management` PR is merged, manually trigger the -> [Deploy workflow](https://github.com/linuxfoundation/lfx-secrets-management/actions/workflows/deploy.yml) -> to push the secret to AWS SM. Use the most specific secret config tag (the `tags:` field -> in the YAML entry, e.g. `litellm` or `pcc`) — not the AWS resource tag — to avoid -> re-deploying or rotating unrelated secrets. The `lfx-v2-argocd` PR must not merge until -> the deploy has completed — ArgoCD will fail to sync if the secret doesn't exist in AWS SM yet. -> -> **Auth0 JWT secrets** (`auth0_jwt` source type) are rotated on every deploy. Before -> triggering the workflow for any entry tagged with `auth0_jwt`, check with CloudOps to -> confirm you are only rotating the intended service's credentials. - -### Step 4: Create Helm Chart and Custom Resource Files +> **Defaults**: All fields default to the role key (i.e., ``): `namespace`, +> `service_account`, and `eso_service_tag`. Only specify a field when its value differs +> from the role key. -#### 4a. `serviceaccount.yaml` in the service Helm chart +### Step 3b: Create ServiceAccount in the Service Helm Chart In the service repo's Helm chart (e.g., `lfx-v2-invite-service`), create `charts//templates/serviceaccount.yaml`: @@ -341,7 +232,7 @@ serviceAccount: automountServiceAccountToken: true ``` -#### 4b. Custom resources in `lfx-v2-argocd` +### Step 3c: Create Custom Resources in `lfx-v2-argocd` The `SecretStore` and `ExternalSecret` are **static YAML files** (not Helm templates) placed in `lfx-v2-argocd/custom-resources//`. @@ -405,7 +296,7 @@ spec: > `-secrets`. No manual `data` list is needed — new secrets are picked up > automatically after the next sync. -#### 4c. IRSA annotation in `lfx-v2-argocd` per-environment values +### Step 3d: Add IRSA Annotation in `lfx-v2-argocd` Per-Environment Values In `values/dev/.yaml` (repeat for staging and prod with the matching account ID): @@ -425,54 +316,32 @@ serviceAccount: | Staging | `844790888233` | `values/staging/.yaml` | | Production | `372256339901` | `values/prod/.yaml` | -> **lfx-self-serve**: If the service is `lfx-self-serve`, any update to a values file in -> `lfx-v2-argocd` must also include the corresponding update to `lfx-self-serve-branch`. - -### Step 5: Wire Secrets into Service Environment in `lfx-v2-argocd` - -In `values/global/.yaml`, add an `environment` block that maps each secret -field to an environment variable. Reference the Kubernetes Secret created by the ExternalSecret -(`-secrets`) and use the field name as the key. - -> **lfx-self-serve**: If the service is `lfx-self-serve`, also update `lfx-self-serve-branch` -> whenever this values file is modified. - -```yaml -# Copyright The Linux Foundation and each contributor to LFX. -# SPDX-License-Identifier: MIT --- -environment: - SECRET_NAME: - valueFrom: - secretKeyRef: - name: -secrets - key: - ANOTHER_SECRET_NAME: - valueFrom: - secretKeyRef: - name: -secrets - key: -``` - ---- - -## Mode 2: Add Secret to Existing Service -Use this mode when adding a new secret to a service that already has ESO + IRSA configured. +## Step 4: Add Entry to `lfx-secrets-management` -### Step 1: Add Entry to `lfx-secrets-management` +In the [lfx-secrets-management](https://github.com/linuxfoundation/lfx-secrets-management) repo, +add an entry for each secret to `secrets/lfx/.yml` — one file per LFX V2 service. +If the file doesn't exist yet, create it. If it already exists, append the new entry. -**Before writing the entry**, read `iam-service-account-definitions.yaml` in `lfx-v2-opentofu` -and confirm the service's `eso_service_tag` (defaults to the role key if not set). You will need -this for the `service-: enabled` AWS SM resource tag — do not ask the user for it. +> **Important**: All secrets must be stored as JSON in AWS SM, even single-field ones. +> Two equivalent ways to express this: +> +> ```yaml +> # implicit JSON (list form) +> fields: +> - +> +> # explicit JSON (scalar + flag) +> fields: +> store_as_json: true +> ``` -Follow the same pattern as Mode 1, Step 3. Add an entry to the appropriate file under -`secrets/lfx/` — check existing files or grep for the third-party service name to find -the right one. +**1Password template:** ```yaml : - tags: [lfx_v2, , ] + tags: [lfx_v2, , <3rd_party_service_tag>] envs: [development, staging, production] source: onepassword: @@ -487,10 +356,42 @@ the right one. - aws_secretsmanager: tags: service-: enabled - path: /<3rd-party-service>/ + path: <3rd-party-service>/ +``` + +Example for Supabase API key: + +```yaml +Supabase API Key: + tags: [lfx_v2, lfx-self-serve, supabase] + envs: [development, staging, production] + source: + onepassword: + vaults: + development: LFX V2 - Development + production: LFX V2 - Production + staging: LFX V2 - Staging + item: LFX v2 supabase + fields: + - url + - api_key + destinations: + - aws_secretsmanager: + tags: + service-pcc: enabled + path: supabase/lfx-self-serve ``` -**Auth0 template** (`auth0` — client_secret, `auth0_clients.yml`): +> **Tips**: +> +> - Each secret becomes a separate AWS SM path entry +> - The `path` must include the service name: `<3rd-party-service>/` (e.g., `atlassian/lfx-v2-committee-service`) +> - The `tags` list must include the service tag (`eso_service_tag` from Step 2a) so the secret is identifiable by service +> - Use the `envs` list to sync to all three environments in parallel +> - The `source.onepassword.item` should match exactly the name in 1Password vaults +> - Field names should be descriptive enough to avoid duplicates (`litellm_api_key`, not just `api_key`) + +**Auth0 template** (`auth0` — client_secret): > For LFX V2 services, always rename fields to a descriptive name prefixed with `auth0_` > (e.g. `auth0_client_id`, `auth0_client_secret`) so keys are unambiguous in the merged K8s Secret. @@ -517,7 +418,7 @@ the right one. service-: enabled ``` -**Auth0 template** (`auth0_jwt` — JWT private key, standard for LFX V2 microservices, `auth0_clients.yml`): +**Auth0 template** (`auth0_jwt` — JWT private key): ```yaml : @@ -544,60 +445,103 @@ the right one. service-: enabled ``` -> **Important**: After the `lfx-secrets-management` PR is merged, manually trigger the +> **Naming review**: Before finalising the entry, review the secret name, tags, and +> destination path against this test: could a reasonable person look at each value and +> have a reasonable idea of what the secret is and where it came from — without needing +> to read the source or ask anyone? If not, rename before proceeding. For example: +> - Secret name: `Atlassian API Key` ✓ — `Key` ✗ +> - Tags: `[lfx_v2, atlassian, lfx-v2-committee-service]` ✓ — `[lfx_v2, key]` ✗ +> - Path: `atlassian/lfx-v2-committee-service` ✓ — `api_key` ✗ + +> **Important**: After the `lfx-secrets-management` PR is merged, the secret must be deployed +> to AWS SM before the `lfx-v2-argocd` PR can merge — ArgoCD will fail to sync if the secret +> doesn't exist yet. +> +> **For 1Password secrets**: trigger the > [Deploy workflow](https://github.com/linuxfoundation/lfx-secrets-management/actions/workflows/deploy.yml) -> to push the secret to AWS SM. Use the most specific secret config tag (the `tags:` field -> in the YAML entry, e.g. `litellm` or `pcc`) — not the AWS resource tag — to avoid -> re-deploying or rotating unrelated secrets. The `lfx-v2-argocd` PR must not merge until -> the deploy has completed — ArgoCD will fail to sync if the secret doesn't exist in AWS SM yet. +> manually: +> 1. Go to the Deploy workflow page linked above +> 2. Click **Run workflow** +> 3. In the tag field, enter the most specific tag from the `tags:` field in your YAML entry +> (e.g. `litellm`, `atlassian`) — not the AWS resource tag — to avoid re-deploying or +> rotating unrelated secrets +> 4. Confirm the workflow completes successfully before merging the `lfx-v2-argocd` PR > -> **Auth0 JWT secrets** (`auth0_jwt` source type) are rotated on every deploy. Before -> triggering the workflow for any entry tagged with `auth0_jwt`, check with CloudOps to -> confirm you are only rotating the intended service's credentials. +> If you're not comfortable triggering the workflow yourself, ask the Platform Engineering team +> to run it for you. +> +> **For Auth0 JWT secrets** (`auth0_jwt` source type): do not trigger the workflow yourself — +> these secrets are rotated on every deploy. Ask the Platform Engineering team to deploy it +> and coordinate the timing so only the intended service's credentials are rotated. + +--- -### Step 2: Wire Secret into Service Environment in `lfx-v2-argocd` +## Step 5: Wire Secrets into Service Environment in `lfx-v2-argocd` -In `values/global/.yaml`, add the new environment variable to the existing -`environment` block: +Add an `environment` block (or extend the existing one) that maps each secret field to an +environment variable. Reference the Kubernetes Secret created by the ExternalSecret +(`-secrets`) and use the field name as the key. + +- If the secret is deployed to **all environments**, add it to `values/global/.yaml` +- If the secret is deployed to **specific environments only**, add it to each relevant + per-environment file (`values/dev/.yaml`, `values/staging/.yaml`, + `values/prod/.yaml`) instead + +> Before writing `secretKeyRef.name`, verify the K8s Secret name from `ExternalSecret.yaml` +> in `lfx-v2-argocd/custom-resources//` — check `spec.target.name`. It is typically +> `-secrets` but must match exactly. + +> Tag-based discovery means new secrets are picked up automatically — no changes to +> `ExternalSecret.yaml` are needed as long as the AWS SM tag matches the service. + +> **lfx-self-serve**: If the service is `lfx-self-serve` and `values/dev/lfx-self-serve.yaml` +> or `values/global/lfx-self-serve.yaml` are modified, apply the same change to +> `values/dev/lfx-self-serve-branch.yaml` as well. The K8s Secret for `lfx-self-serve` +> is named `pcc-secrets` — use that as `secretKeyRef.name` instead of `lfx-self-serve-secrets`. +> Note: the `eso_service_tag` for `lfx-self-serve` is `pcc` (so the AWS SM resource tag is +> `service-pcc: enabled` and the K8s Secret is `pcc-secrets`), but the service tag in the +> `tags:` list should be `lfx-self-serve` — the human-readable service name. ```yaml +# Copyright The Linux Foundation and each contributor to LFX. +# SPDX-License-Identifier: MIT +--- environment: - NEW_ENV_VAR: + SECRET_NAME: valueFrom: secretKeyRef: name: -secrets key: + ANOTHER_SECRET_NAME: + valueFrom: + secretKeyRef: + name: -secrets + key: ``` -> Before writing `secretKeyRef.name`, verify the K8s Secret name from the existing -> `ExternalSecret.yaml` in `lfx-v2-argocd/custom-resources//` — -> check `spec.target.name`. It is typically `-secrets` but must match exactly. - -> Tag-based discovery means the new secret is picked up automatically — no changes to -> `ExternalSecret.yaml` are needed as long as the AWS SM tag matches the service. - -> **lfx-self-serve**: If the service is `lfx-self-serve`, also update `lfx-self-serve-branch` -> whenever this values file is modified. +> **Note**: The `environment:` block is always named `environment`, but its nesting varies — +> some services have it at the top level, others under `app:`. Always check the existing +> values file before adding entries and match the structure already in use. --- ## Verification Checklist -After completing either mode, verify the setup: +After completing all applicable steps, verify the setup: **File checklist — all repos involved:** -- [ ] `lfx-v2-opentofu`: `iam-service-account-definitions.yaml` has service entry *(Mode 1 only)* +- [ ] `lfx-v2-opentofu`: `iam-service-account-definitions.yaml` has service entry *(new services only)* - [ ] `lfx-secrets-management`: appropriate file under `secrets/lfx/` has sync entry for each secret; Deploy workflow run after merge -- [ ] Service Helm chart *(Mode 1 only)*: +- [ ] Service Helm chart *(new services only)*: - [ ] `templates/serviceaccount.yaml` created - [ ] `values.yaml` has `serviceAccount` block - [ ] `lfx-v2-argocd`: - - [ ] `custom-resources//SecretStore.yaml` created *(Mode 1 only)* - - [ ] `custom-resources//ExternalSecret.yaml` created *(Mode 1 only)* - - [ ] `values/dev/.yaml` has IRSA role ARN + `automountServiceAccountToken: true` *(Mode 1 only)* - - [ ] `values/staging/.yaml` has IRSA role ARN + `automountServiceAccountToken: true` *(Mode 1 only)* - - [ ] `values/prod/.yaml` has IRSA role ARN + `automountServiceAccountToken: true` *(Mode 1 only)* + - [ ] `custom-resources//SecretStore.yaml` created *(new services only)* + - [ ] `custom-resources//ExternalSecret.yaml` created *(new services only)* + - [ ] `values/dev/.yaml` has IRSA role ARN + `automountServiceAccountToken: true` *(new services only)* + - [ ] `values/staging/.yaml` has IRSA role ARN + `automountServiceAccountToken: true` *(new services only)* + - [ ] `values/prod/.yaml` has IRSA role ARN + `automountServiceAccountToken: true` *(new services only)* - [ ] `values/global/.yaml` has `environment` block with `secretKeyRef` entries for all secrets - [ ] **lfx-self-serve only**: `lfx-self-serve-branch` updated alongside any `lfx-self-serve` values file change @@ -637,30 +581,32 @@ Check these repos for the exact file structure and conventions used in productio ### Adding a JWT secret to a new service 1. User asks: "Set up JWT secret for invite-service" -2. Collect: service name, 1Password item name, field names, environments -3. Follow Mode 1 steps in order -4. Verify using the checklist above -5. Open PRs for `lfx-v2-opentofu`, `lfx-secrets-management`, service Helm chart, and `lfx-v2-argocd` -6. Merge `lfx-secrets-management` PR, trigger Deploy workflow, then merge remaining PRs +2. Collect: service name, 1Password item name, field names, environments (Step 1) +3. Check for existing ESO objects — none found, proceed with Step 3 +4. Follow Steps 3–5 in order +5. Verify using the checklist above +6. Open PRs for `lfx-v2-opentofu`, `lfx-secrets-management`, service Helm chart, and `lfx-v2-argocd` +7. Merge `lfx-secrets-management` PR, trigger Deploy workflow, then merge remaining PRs ### Adding SMTP credentials to an existing service 1. User asks: "Add SMTP secret to email-service" -2. Follow Mode 2 steps -3. Verify using the checklist above -4. Open PRs for `lfx-secrets-management` and `lfx-v2-argocd` -5. Merge `lfx-secrets-management` PR, trigger Deploy workflow, then merge `lfx-v2-argocd` PR +2. Collect: service name, 1Password item name, field names, environments (Step 1) +3. Check for existing ESO objects — found, skip Step 3 +4. Follow Steps 4–5 +5. Verify using the checklist above +6. Open PRs for `lfx-secrets-management` and `lfx-v2-argocd` +7. Merge `lfx-secrets-management` PR, trigger Deploy workflow, then merge `lfx-v2-argocd` PR ### Adding an Auth0 client key pair to an existing service 1. User asks: "Add the LFX V2 Persona Service auth0 client to lfx-v2-persona-service" -2. Add sync entry in lfx-secrets-management -3. Verify using the checklist above -4. Submit secrets PR -5. Coordinate with the Platform Engineering team to deploy the auto-rotated secret -6. Wire into deployment in values charts -7. Verify using the checklist above -8. Submit argocd PR +2. Check for existing ESO objects — found, skip Step 3 +3. Add sync entry in lfx-secrets-management (Step 4) +4. Coordinate with the Platform Engineering team to deploy the auto-rotated secret +5. Wire into deployment in values charts (Step 5) +6. Verify using the checklist above +7. Submit argocd PR ### Debugging: "Pods can't read the secret"