From 1b07a3e05831155ee3661713a3919ad993c3f209 Mon Sep 17 00:00:00 2001 From: Siddhesh Ghadi Date: Wed, 15 Jul 2026 15:17:48 +0530 Subject: [PATCH] Add argo knowledge skill Signed-off-by: Siddhesh Ghadi --- gitops/OWNERS | 10 + gitops/README.md | 22 + gitops/argo-knowledge/SKILL.md | 809 +++++++++++++++++ gitops/argo-knowledge/evals/evals.json | 157 ++++ .../argo-knowledge/references/agent-mode.md | 89 ++ .../argo-knowledge/references/app-projects.md | 300 +++++++ .../argo-knowledge/references/applications.md | 375 ++++++++ .../references/applicationsets.md | 403 +++++++++ .../references/best-practices.md | 392 +++++++++ gitops/argo-knowledge/references/events.md | 818 ++++++++++++++++++ .../references/gitops-promoter.md | 297 +++++++ .../references/image-updater.md | 393 +++++++++ .../references/multi-tenancy.md | 229 +++++ .../references/notifications.md | 505 +++++++++++ gitops/argo-knowledge/references/openshift.md | 272 ++++++ .../references/repo-patterns.md | 388 +++++++++ gitops/argo-knowledge/references/rollouts.md | 551 ++++++++++++ gitops/argo-knowledge/references/workflows.md | 675 +++++++++++++++ 18 files changed, 6685 insertions(+) create mode 100644 gitops/OWNERS create mode 100644 gitops/README.md create mode 100644 gitops/argo-knowledge/SKILL.md create mode 100644 gitops/argo-knowledge/evals/evals.json create mode 100644 gitops/argo-knowledge/references/agent-mode.md create mode 100644 gitops/argo-knowledge/references/app-projects.md create mode 100644 gitops/argo-knowledge/references/applications.md create mode 100644 gitops/argo-knowledge/references/applicationsets.md create mode 100644 gitops/argo-knowledge/references/best-practices.md create mode 100644 gitops/argo-knowledge/references/events.md create mode 100644 gitops/argo-knowledge/references/gitops-promoter.md create mode 100644 gitops/argo-knowledge/references/image-updater.md create mode 100644 gitops/argo-knowledge/references/multi-tenancy.md create mode 100644 gitops/argo-knowledge/references/notifications.md create mode 100644 gitops/argo-knowledge/references/openshift.md create mode 100644 gitops/argo-knowledge/references/repo-patterns.md create mode 100644 gitops/argo-knowledge/references/rollouts.md create mode 100644 gitops/argo-knowledge/references/workflows.md diff --git a/gitops/OWNERS b/gitops/OWNERS new file mode 100644 index 0000000..6f76f9e --- /dev/null +++ b/gitops/OWNERS @@ -0,0 +1,10 @@ +# See the OWNERS docs: https://git.k8s.io/community/contributors/guide/owners.md + +approvers: + - alimobrem + - svghadi +reviewers: + - alimobrem + - svghadi + +component: "GitOps Skills" \ No newline at end of file diff --git a/gitops/README.md b/gitops/README.md new file mode 100644 index 0000000..b810ef5 --- /dev/null +++ b/gitops/README.md @@ -0,0 +1,22 @@ +# GitOps skills + +This directory contains skills that help AI agents design, generate, review, +and troubleshoot GitOps-based application delivery. + +The skills may cover: + +- General GitOps concepts and repository patterns +- Argo CD +- OpenShift GitOps Operator +- Argo Rollouts +- Argo Workflows +- Argo Events +- Multi-cluster and multi-tenant delivery + +## Available skills + +### argo-knowledge + +Provides knowledge and guidance for the Argo ecosystem and OpenShift GitOps, +including manifest generation, architecture guidance, resource explanations, +best practices, and common configuration mistakes. \ No newline at end of file diff --git a/gitops/argo-knowledge/SKILL.md b/gitops/argo-knowledge/SKILL.md new file mode 100644 index 0000000..6979540 --- /dev/null +++ b/gitops/argo-knowledge/SKILL.md @@ -0,0 +1,809 @@ +--- +name: argo-knowledge +description: > + Comprehensive knowledge base for the Argo ecosystem including Argo CD, Argo Rollouts, + Argo Workflows, and Argo Events. Use this skill when generating, reviewing, debugging, + or explaining Argo CRDs (Application, ApplicationSet, AppProject, Rollout, AnalysisTemplate, + Workflow, WorkflowTemplate, CronWorkflow, EventSource, Sensor, EventBus), configuring + notifications, image updater, sync policies, progressive delivery strategies, or + designing GitOps repository patterns. Also use when the user asks about Argo best practices, + common mistakes, or needs to choose between Argo approaches (e.g., app-of-apps vs + ApplicationSet, canary vs blue-green, steps vs DAG workflows). +license: MIT +--- + +# Argo Knowledge Base + +## Rules + +1. **Always use correct apiVersion and kind.** Every Argo CRD uses `apiVersion: argoproj.io/v1alpha1`. Never invent CRDs or apiVersions that do not exist. +2. **Load references on-demand.** Only read reference files when the user's question requires detailed field-level knowledge. Do not preload all references. +3. **Max 1-2 reference files per question.** Keep context focused. If a question spans more than two reference areas, answer the primary question first and offer to elaborate. +4. **Validate YAML before presenting.** Every YAML example must be syntactically valid and use real field names from the Argo CRD schemas. +5. **Prefer canonical patterns.** Use the numbered patterns below as starting points. Adapt to the user's specifics rather than inventing from scratch. +6. **State trade-offs.** When recommending an approach, briefly note what you give up. + +## What is Argo + +The Argo project is a set of Kubernetes-native tools for running and managing jobs and applications on Kubernetes. + +- **Argo CD** — Declarative GitOps continuous delivery. Watches Git repos and syncs Kubernetes resources to match the desired state. Supports Helm, Kustomize, plain YAML, Jsonnet, and plugin-based config management tools. Provides a UI, CLI, and API for managing applications across multiple clusters. + +- **Argo Rollouts** — Progressive delivery controller. Extends Kubernetes Deployments with canary releases, blue-green deployments, experimentation, and automated analysis-driven promotion/rollback. Integrates with service meshes and ingress controllers for traffic management. + +- **Argo Workflows** — Container-native workflow engine for Kubernetes. Runs DAG and step-based workflows where each step is a container. Used for CI/CD pipelines, data processing, ML pipelines, and infrastructure automation. + +- **Argo Events** — Event-driven workflow automation. Connects external event sources (webhooks, message queues, cloud events, cron schedules) to triggers that create Argo Workflows or any Kubernetes resource. + +**How they relate:** Argo CD deploys your applications via GitOps. Argo Rollouts handles the progressive delivery strategy during those deployments. Argo Workflows orchestrates complex multi-step jobs. Argo Events wires external events to trigger Workflows or other actions. Together they form a complete GitOps + progressive delivery + automation platform. + +## CRD Table + +| Kind | apiVersion | Project | Purpose | +|------|-----------|---------|---------| +| Application | argoproj.io/v1alpha1 | Argo CD | Defines a single application to sync from Git to a cluster | +| AppProject | argoproj.io/v1alpha1 | Argo CD | RBAC boundary: restricts sources, destinations, and resources | +| ApplicationSet | argoproj.io/v1alpha1 | Argo CD | Generates multiple Applications from templates + generators | +| Rollout | argoproj.io/v1alpha1 | Rollouts | Progressive delivery replacement for Deployment | +| AnalysisTemplate | argoproj.io/v1alpha1 | Rollouts | Defines metric queries for automated rollout analysis | +| ClusterAnalysisTemplate | argoproj.io/v1alpha1 | Rollouts | Cluster-scoped AnalysisTemplate | +| AnalysisRun | argoproj.io/v1alpha1 | Rollouts | Instance of an AnalysisTemplate execution (auto-created) | +| Experiment | argoproj.io/v1alpha1 | Rollouts | Runs multiple ReplicaSet versions simultaneously for comparison | +| Workflow | argoproj.io/v1alpha1 | Workflows | A single workflow execution | +| WorkflowTemplate | argoproj.io/v1alpha1 | Workflows | Reusable workflow definition (namespace-scoped) | +| ClusterWorkflowTemplate | argoproj.io/v1alpha1 | Workflows | Reusable workflow definition (cluster-scoped) | +| CronWorkflow | argoproj.io/v1alpha1 | Workflows | Scheduled workflow execution | +| EventSource | argoproj.io/v1alpha1 | Events | Defines external event sources to consume | +| EventBus | argoproj.io/v1alpha1 | Events | Message transport layer between EventSources and Sensors | +| Sensor | argoproj.io/v1alpha1 | Events | Listens to EventBus, applies filters, fires triggers | + +## How Argo CD Works + +### Sync Loop + +1. **Desired state:** Argo CD reads manifests from a Git repo (source). The source can be Helm charts, Kustomize overlays, plain YAML directories, Jsonnet, or custom config management plugins. +2. **Live state:** Argo CD queries the target Kubernetes cluster (destination) for the current state of resources it manages. +3. **Diff:** Argo CD compares desired vs live state. Resources that differ are marked `OutOfSync`. +4. **Sync:** When triggered (manually or via automated sync policy), Argo CD applies the desired state to the cluster using `kubectl apply`, server-side apply, or `kubectl create/replace` depending on sync options. +5. **Health assessment:** After sync, Argo CD evaluates resource health using built-in health checks (Deployments, StatefulSets, Services, Ingresses, etc.) and custom health checks (Lua scripts in `argocd-cm`). Resources are marked Healthy, Progressing, Degraded, Suspended, or Missing. + +### Resource Tracking + +Argo CD tracks which resources belong to an Application using one of three methods: + +- **`label`** — Adds `app.kubernetes.io/instance: ` label. Simple but can conflict with other tools using this label. +- **`annotation`** — Adds `argocd.argoproj.io/tracking-id` annotation. Avoids label conflicts. Default in modern Argo CD. +- **`annotation+label`** — Uses both. Maximum compatibility. + +Configured via `resource.trackingMethod` in `argocd-cm` ConfigMap. + +### Multi-Cluster + +Argo CD manages applications across multiple clusters from a single control plane: + +- The cluster running Argo CD is the **in-cluster** target (referenced as `https://kubernetes.default.svc`). +- External clusters are added via `argocd cluster add `, which creates a ServiceAccount + ClusterRoleBinding on the target cluster and stores credentials as a Secret in the Argo CD namespace. +- Cluster Secrets have label `argocd.argoproj.io/secret-type: cluster` and contain `server`, `name`, `config` (with `bearerToken`, `tlsClientConfig`). + +## Decision Trees + +### Application vs ApplicationSet + +Use **Application** when: +- You have a single application or a small, fixed number of applications +- Each application has unique configuration that doesn't follow a pattern +- You want direct, explicit control over each application + +Use **ApplicationSet** when: +- You need to generate many Applications from a pattern (e.g., per-directory, per-cluster, per-team) +- Applications share a common template with parameterized differences +- You want Applications auto-created/deleted when directories or clusters appear/disappear +- You need progressive rollout across environments (rollingSync) + +### Rollout vs Deployment + +Use **Deployment** when: +- You want simple rolling updates +- You don't need traffic shaping, analysis, or manual promotion gates +- The application is non-critical or internal-only + +Use **Rollout** when: +- You need canary or blue-green deployment strategies +- You want automated analysis (metrics-based promotion/rollback) +- You need traffic percentage control via service mesh or ingress +- You need manual approval gates between rollout steps +- You want experiment-based A/B testing + +### Which ApplicationSet Generator + +- **Git directory** — One Application per directory in a monorepo. Best for: environments or services organized as directories. +- **Git file** — One Application per JSON/YAML config file in a repo. Best for: externalized app configs with arbitrary fields. +- **List** — Explicitly enumerated Applications. Best for: small, fixed sets with no dynamic discovery. +- **Cluster** — One Application per registered Argo CD cluster. Best for: deploying the same app to all clusters matching a selector. +- **Matrix** — Cartesian product of two generators. Best for: deploy N apps to M clusters. +- **Merge** — Combine generator outputs, overriding fields from a secondary generator. Best for: cluster-specific overrides on top of a base config. +- **Pull request** — One Application per open PR. Best for: PR preview environments. +- **SCM provider** — One Application per repo in a GitHub org or GitLab group. Best for: auto-onboarding repos. + +## Canonical YAML Patterns + +### Pattern 1: Application with Helm Source + +```yaml +apiVersion: argoproj.io/v1alpha1 +kind: Application +metadata: + name: my-app + namespace: argocd + finalizers: + - resources-finalizer.argocd.argoproj.io +spec: + project: default + source: + repoURL: https://charts.example.com + chart: my-app + targetRevision: 1.2.3 + helm: + releaseName: my-app + valuesObject: + replicaCount: 3 + image: + repository: registry.example.com/my-app + tag: latest + ingress: + enabled: true + hosts: + - my-app.example.com + parameters: + - name: service.type + value: ClusterIP + destination: + server: https://kubernetes.default.svc + namespace: my-app + syncPolicy: + automated: + prune: true + selfHeal: true + syncOptions: + - CreateNamespace=true + - ServerSideApply=true + retry: + limit: 5 + backoff: + duration: 5s + factor: 2 + maxDuration: 3m +``` + +### Pattern 2: Application with Kustomize Source + +```yaml +apiVersion: argoproj.io/v1alpha1 +kind: Application +metadata: + name: my-app-prod + namespace: argocd +spec: + project: default + source: + repoURL: https://github.com/org/k8s-configs.git + targetRevision: main + path: apps/my-app/overlays/production + kustomize: + namePrefix: prod- + commonLabels: + environment: production + images: + - registry.example.com/my-app:v2.1.0 + destination: + server: https://kubernetes.default.svc + namespace: production + syncPolicy: + automated: + prune: true + selfHeal: true + syncOptions: + - CreateNamespace=true +``` + +### Pattern 3: ApplicationSet with Git Directory Generator + +```yaml +apiVersion: argoproj.io/v1alpha1 +kind: ApplicationSet +metadata: + name: cluster-addons + namespace: argocd +spec: + goTemplate: true + goTemplateOptions: + - missingkey=error + generators: + - git: + repoURL: https://github.com/org/cluster-addons.git + revision: main + directories: + - path: addons/* + - path: addons/experimental-* + exclude: true + preserveResourcesOnDeletion: true + template: + metadata: + name: 'addon-{{.path.basename}}' + labels: + envLabel: staging + spec: + project: cluster-addons + source: + repoURL: https://github.com/org/cluster-addons.git + targetRevision: main + path: '{{.path.path}}' + destination: + server: https://kubernetes.default.svc + namespace: '{{.path.basename}}' + syncPolicy: + automated: + prune: true + selfHeal: true + syncOptions: + - CreateNamespace=true + strategy: + type: RollingSync + rollingSync: + steps: + - matchExpressions: + - key: envLabel + operator: In + values: + - staging + maxUpdate: 100% + - matchExpressions: + - key: envLabel + operator: In + values: + - production + maxUpdate: 25% +``` + +### Pattern 4: ApplicationSet with Cluster Generator + Matrix + +```yaml +apiVersion: argoproj.io/v1alpha1 +kind: ApplicationSet +metadata: + name: platform-services + namespace: argocd +spec: + goTemplate: true + goTemplateOptions: + - missingkey=error + preserveResourcesOnDeletion: true + generators: + - matrix: + generators: + - clusters: + selector: + matchLabels: + tier: production + values: + revision: main + - list: + elements: + - app: ingress-nginx + namespace: ingress-nginx + path: platform/ingress-nginx + - app: cert-manager + namespace: cert-manager + path: platform/cert-manager + - app: monitoring + namespace: monitoring + path: platform/monitoring + template: + metadata: + name: '{{.name}}-{{.app}}' + spec: + project: platform + source: + repoURL: https://github.com/org/platform-config.git + targetRevision: '{{.values.revision}}' + path: '{{.path}}' + destination: + server: '{{.server}}' + namespace: '{{.namespace}}' + syncPolicy: + automated: + prune: true + selfHeal: true + syncOptions: + - CreateNamespace=true + - ServerSideApply=true +``` + +### Pattern 5: Rollout with Canary Strategy + AnalysisTemplate + +```yaml +apiVersion: argoproj.io/v1alpha1 +kind: Rollout +metadata: + name: my-app + namespace: my-app +spec: + replicas: 5 + revisionHistoryLimit: 3 + selector: + matchLabels: + app: my-app + template: + metadata: + labels: + app: my-app + spec: + containers: + - name: my-app + image: registry.example.com/my-app:v2.0.0 + ports: + - containerPort: 8080 + resources: + requests: + cpu: 100m + memory: 128Mi + limits: + cpu: 500m + memory: 512Mi + strategy: + canary: + maxSurge: 1 + maxUnavailable: 0 + steps: + - setWeight: 10 + - pause: { duration: 2m } + - setWeight: 30 + - analysis: + templates: + - templateName: success-rate + args: + - name: service-name + value: my-app + - setWeight: 60 + - pause: { duration: 5m } + - setWeight: 100 + canaryService: my-app-canary + stableService: my-app-stable + trafficRouting: + istio: + virtualServices: + - name: my-app-vsvc + routes: + - primary +--- +apiVersion: argoproj.io/v1alpha1 +kind: AnalysisTemplate +metadata: + name: success-rate + namespace: my-app +spec: + args: + - name: service-name + metrics: + - name: success-rate + interval: 60s + count: 5 + successCondition: result[0] >= 0.95 + failureLimit: 2 + provider: + prometheus: + address: http://prometheus.monitoring:9090 + query: | + sum(rate(http_requests_total{service="{{args.service-name}}", status=~"2.."}[5m])) + / + sum(rate(http_requests_total{service="{{args.service-name}}"}[5m])) +``` + +### Pattern 6: Rollout with Blue-Green Strategy + +```yaml +apiVersion: argoproj.io/v1alpha1 +kind: Rollout +metadata: + name: my-app + namespace: my-app +spec: + replicas: 3 + revisionHistoryLimit: 3 + selector: + matchLabels: + app: my-app + template: + metadata: + labels: + app: my-app + spec: + containers: + - name: my-app + image: registry.example.com/my-app:v2.0.0 + ports: + - containerPort: 8080 + strategy: + blueGreen: + activeService: my-app-active + previewService: my-app-preview + autoPromotionEnabled: false + scaleDownDelaySeconds: 30 + prePromotionAnalysis: + templates: + - templateName: smoke-test + args: + - name: preview-url + value: http://my-app-preview.my-app.svc.cluster.local + postPromotionAnalysis: + templates: + - templateName: success-rate + args: + - name: service-name + value: my-app +``` + +### Pattern 7: Workflow DAG Pattern + +```yaml +apiVersion: argoproj.io/v1alpha1 +kind: Workflow +metadata: + generateName: ci-pipeline- + namespace: argo +spec: + entrypoint: ci-pipeline + serviceAccountName: argo-workflow + arguments: + parameters: + - name: repo-url + value: https://github.com/org/my-app.git + - name: revision + value: main + - name: image + value: registry.example.com/my-app + templates: + - name: ci-pipeline + dag: + tasks: + - name: checkout + template: git-clone + arguments: + parameters: + - name: repo-url + value: '{{workflow.parameters.repo-url}}' + - name: revision + value: '{{workflow.parameters.revision}}' + - name: unit-test + template: run-tests + dependencies: + - checkout + - name: lint + template: run-lint + dependencies: + - checkout + - name: build-image + template: build-push + dependencies: + - unit-test + - lint + arguments: + parameters: + - name: image + value: '{{workflow.parameters.image}}' + - name: git-clone + inputs: + parameters: + - name: repo-url + - name: revision + container: + image: alpine/git:latest + command: [sh, -c] + args: + - git clone --branch {{inputs.parameters.revision}} {{inputs.parameters.repo-url}} /work + volumeMounts: + - name: work + mountPath: /work + - name: run-tests + container: + image: golang:1.22 + command: [sh, -c] + args: + - cd /work && go test ./... + volumeMounts: + - name: work + mountPath: /work + - name: run-lint + container: + image: golangci/golangci-lint:latest + command: [sh, -c] + args: + - cd /work && golangci-lint run + volumeMounts: + - name: work + mountPath: /work + - name: build-push + inputs: + parameters: + - name: image + container: + image: gcr.io/kaniko-project/executor:latest + args: + - --context=/work + - --destination={{inputs.parameters.image}}:{{workflow.uid}} + volumeMounts: + - name: work + mountPath: /work + volumeClaimTemplates: + - metadata: + name: work + spec: + accessModes: [ReadWriteOnce] + resources: + requests: + storage: 1Gi +``` + +### Pattern 8: EventSource + Sensor Triggering a Workflow + +```yaml +apiVersion: argoproj.io/v1alpha1 +kind: EventSource +metadata: + name: github-webhook + namespace: argo-events +spec: + service: + ports: + - port: 12000 + targetPort: 12000 + github: + push-events: + repositories: + - owner: my-org + names: + - my-app + webhook: + endpoint: /push + port: "12000" + method: POST + events: + - push + apiToken: + name: github-token + key: token + webhookSecret: + name: github-token + key: webhook-secret + contentType: json + active: true +--- +apiVersion: argoproj.io/v1alpha1 +kind: EventBus +metadata: + name: default + namespace: argo-events +spec: + jetstream: + version: latest + replicas: 3 + persistence: + storageClassName: standard + accessMode: ReadWriteOnce + volumeSize: 20Gi +--- +apiVersion: argoproj.io/v1alpha1 +kind: Sensor +metadata: + name: github-sensor + namespace: argo-events +spec: + dependencies: + - name: github-push + eventSourceName: github-webhook + eventName: push-events + filters: + data: + - path: body.ref + type: string + value: + - refs/heads/main + triggers: + - template: + name: trigger-ci + argoWorkflow: + operation: submit + source: + resource: + apiVersion: argoproj.io/v1alpha1 + kind: Workflow + metadata: + generateName: ci-triggered- + namespace: argo + spec: + entrypoint: build + arguments: + parameters: + - name: repo-url + - name: revision + templates: + - name: build + container: + image: golang:1.22 + command: [sh, -c] + args: + - | + echo "Building {{workflow.parameters.repo-url}} at {{workflow.parameters.revision}}" + parameters: + - src: + dependencyName: github-push + dataKey: body.repository.clone_url + dest: spec.arguments.parameters.0.value + - src: + dependencyName: github-push + dataKey: body.after + dest: spec.arguments.parameters.1.value +``` + +### Pattern 9: Notifications Config (argocd-notifications-cm) + +```yaml +apiVersion: v1 +kind: ConfigMap +metadata: + name: argocd-notifications-cm + namespace: argocd +data: + service.slack: | + token: $slack-token + signingSecret: $slack-signing-secret + template.app-sync-succeeded: | + message: | + Application {{.app.metadata.name}} has been successfully synced. + Revision: {{.app.status.sync.revision}} + Project: {{.app.spec.project}} + slack: + attachments: | + [{ + "color": "#18be52", + "title": "{{.app.metadata.name}} synced", + "title_link": "{{.context.argocdUrl}}/applications/{{.app.metadata.name}}", + "fields": [ + {"title": "Project", "value": "{{.app.spec.project}}", "short": true}, + {"title": "Revision", "value": "{{.app.status.sync.revision | trunc 7}}", "short": true} + ] + }] + template.app-sync-failed: | + message: | + Application {{.app.metadata.name}} sync failed. + Error: {{.app.status.operationState.message}} + slack: + attachments: | + [{ + "color": "#E96D76", + "title": "{{.app.metadata.name}} sync failed", + "title_link": "{{.context.argocdUrl}}/applications/{{.app.metadata.name}}", + "fields": [ + {"title": "Project", "value": "{{.app.spec.project}}", "short": true}, + {"title": "Error", "value": "{{.app.status.operationState.message}}", "short": false} + ] + }] + template.app-health-degraded: | + message: | + Application {{.app.metadata.name}} is degraded. + slack: + attachments: | + [{ + "color": "#f4c030", + "title": "{{.app.metadata.name}} degraded", + "title_link": "{{.context.argocdUrl}}/applications/{{.app.metadata.name}}" + }] + trigger.on-sync-succeeded: | + - when: app.status.operationState.phase in ['Succeeded'] + send: [app-sync-succeeded] + trigger.on-sync-failed: | + - when: app.status.operationState.phase in ['Error', 'Failed'] + send: [app-sync-failed] + trigger.on-health-degraded: | + - when: app.status.health.status == 'Degraded' + send: [app-health-degraded] +--- +apiVersion: v1 +kind: Secret +metadata: + name: argocd-notifications-secret + namespace: argocd +type: Opaque +stringData: + slack-token: xoxb-XXXXXXXXX + slack-signing-secret: XXXXXXXXX +``` + +**Application annotation to subscribe:** + +```yaml +metadata: + annotations: + notifications.argoproj.io/subscribe.on-sync-failed.slack: my-channel + notifications.argoproj.io/subscribe.on-health-degraded.slack: my-channel + notifications.argoproj.io/subscribe.on-sync-succeeded.slack: deployments +``` + +### Pattern 10: Image Updater Annotations + +```yaml +apiVersion: argoproj.io/v1alpha1 +kind: Application +metadata: + name: my-app + namespace: argocd + annotations: + argocd-image-updater.argoproj.io/image-list: myapp=registry.example.com/my-app + argocd-image-updater.argoproj.io/myapp.update-strategy: semver + argocd-image-updater.argoproj.io/myapp.allow-tags: "regexp:^v[0-9]+\\.[0-9]+\\.[0-9]+$" + argocd-image-updater.argoproj.io/myapp.ignore-tags: "latest,dev-*" + argocd-image-updater.argoproj.io/myapp.helm.image-name: image.repository + argocd-image-updater.argoproj.io/myapp.helm.image-tag: image.tag + argocd-image-updater.argoproj.io/myapp.pull-secret: pullsecret:argocd/registry-creds + argocd-image-updater.argoproj.io/write-back-method: git + argocd-image-updater.argoproj.io/write-back-target: kustomization + argocd-image-updater.argoproj.io/git-branch: main +spec: + project: default + source: + repoURL: https://github.com/org/k8s-configs.git + targetRevision: main + path: apps/my-app/overlays/production + destination: + server: https://kubernetes.default.svc + namespace: my-app + syncPolicy: + automated: + prune: true + selfHeal: true +``` + +## Common Mistakes + +1. **Missing `namespace: argocd` on Application metadata.** Applications must live in the Argo CD namespace (or a namespace explicitly allowed via `--application-namespaces`). Omitting this causes Argo CD to never see the Application. + +2. **Using `helm.values` (string) instead of `helm.valuesObject` (map).** `helm.values` is a multiline YAML string which is error-prone. `helm.valuesObject` is a structured map and should be preferred. + +3. **Setting `automated.prune: true` without understanding consequences.** This deletes resources from the cluster that are no longer in Git. If you accidentally remove a manifest from Git, prune will delete the live resource. Use `prune: false` until you trust the workflow, or use `preserveResourcesOnDeletion` on ApplicationSets. + +4. **Forgetting `CreateNamespace=true` sync option.** Argo CD does not create namespaces by default. Without this option, syncing to a non-existent namespace fails. + +5. **Using `targetRevision: HEAD` in production.** This follows the default branch and makes deployments unpredictable. Pin to a specific tag, commit SHA, or semver range. + +6. **Rollout without matching Services.** Canary and blue-green strategies require correctly configured `stableService` and `canaryService`/`previewService`. Forgetting these Services causes traffic routing failures. + +7. **AnalysisTemplate with no `count` or `interval`.** Without these, the analysis runs once and completes immediately, which is rarely useful. Set `count` and `interval` for continuous monitoring during rollout. + +8. **ApplicationSet with `goTemplate: true` but using `{{name}}` syntax.** Go templates use `{{.name}}`, not `{{name}}` (which is fasttemplate syntax). Mixing them produces empty strings silently. + +9. **EventSource and Sensor in different namespaces without cross-namespace EventBus config.** They must reference the same EventBus. By default, they look for an EventBus named `default` in their own namespace. + +10. **Workflow `activeDeadlineSeconds` not set.** Without a deadline, stuck workflows run forever, consuming cluster resources. Always set a reasonable deadline. + +11. **Notification trigger `when` expression referencing wrong fields.** Common mistake: `app.status.sync.status` vs `app.status.operationState.phase`. Sync status is Synced/OutOfSync. Operation phase is Succeeded/Failed/Error/Running. + +12. **Image updater `write-back-method: git` without write access.** The image updater needs push access to the Git repo. Without it, updates silently fail. Check the image updater logs. + +## Reference Index + +| Topic | Reference File | When to Load | +|-------|---------------|-------------| +| Application spec, sync policies, sync waves, health checks, ignore differences, multi-cluster | `references/applications.md` | Questions about Application configuration, sync behavior, health, or multi-cluster | +| ApplicationSet generators, templates, progressive syncs | `references/applicationsets.md` | Questions about ApplicationSet generators, templating, or progressive rollout | +| AppProject RBAC, source/destination restrictions, sync windows | `references/app-projects.md` | Questions about RBAC, project policies, sync windows | +| Rollout canary/blue-green, traffic management, AnalysisTemplate, experiments | `references/rollouts.md` | Questions about progressive delivery, canary/blue-green, analysis metrics | +| Workflow DAG/steps, parameters, artifacts, CronWorkflow | `references/workflows.md` | Questions about workflow orchestration, scheduling, parameters | +| EventSource types, EventBus, Sensor triggers, filters | `references/events.md` | Questions about event-driven automation | +| Notification services, templates, triggers, subscriptions | `references/notifications.md` | Questions about alerting and notifications | +| Image updater annotations, strategies, write-back | `references/image-updater.md` | Questions about automated image updates | +| App-of-apps, monorepo, multi-repo patterns | `references/repo-patterns.md` | Questions about repository structure and GitOps patterns | +| Sync policy, RBAC, health checks, secret management | `references/best-practices.md` | General architecture and operational best practices | +| Agent mode, principal/agent architecture, managed/autonomous modes, hub-and-spoke | `references/agent-mode.md` | Questions about argocd-agent, multi-cluster at scale, air-gapped, edge deployments | +| Applications in any namespace, multi-instance, multi-tenancy, Autopilot | `references/multi-tenancy.md` | Questions about multi-tenancy, team isolation, --application-namespaces, Autopilot bootstrap | +| OpenShift GitOps Operator, ArgoCD CRD, Routes, SCCs, OAuth, managed namespaces | `references/openshift.md` | Questions about running Argo on OpenShift, operator-managed instances, OpenShift-specific patterns | +| GitOps Promoter, PromotionStrategy, environment promotion, commit status gating | `references/gitops-promoter.md` | Questions about environment promotion, PR-based gating, branch-per-environment, gitops-promoter CRDs | diff --git a/gitops/argo-knowledge/evals/evals.json b/gitops/argo-knowledge/evals/evals.json new file mode 100644 index 0000000..e890f49 --- /dev/null +++ b/gitops/argo-knowledge/evals/evals.json @@ -0,0 +1,157 @@ +{ + "skill_name": "argo-knowledge", + "evals": [ + { + "id": 1, + "prompt": "Generate a multi-source Application called 'webapp' in the argocd namespace that combines a Helm chart from oci://registry.example.com/charts/webapp at version 2.x with environment-specific values from a Git repo https://github.com/org/config.git (branch main, path envs/production/values.yaml). Deploy to namespace 'webapp-prod' on in-cluster. Use automated sync with selfHeal but WITHOUT prune (the team wants manual prune control). Use valuesObject (not the deprecated string values field) for any inline Helm values. Set server-side apply and respect ignore differences for /spec/replicas on Deployments.", + "expected_output": "A multi-source Application using spec.sources[] array (not spec.source singular), with one Helm source and one Git source providing values via $values ref, automated sync without prune, ServerSideApply, and ignoreDifferences.", + "files": [], + "expectations": [ + "Output uses spec.sources (plural array) not spec.source (singular) — multi-source Applications require the array form", + "Output contains exactly two entries in the sources array", + "First source is Helm with repoURL: oci://registry.example.com/charts/webapp and chart field", + "Helm source uses targetRevision with semver constraint 2.x or similar", + "Second source is Git with repoURL: https://github.com/org/config.git", + "Git source has ref.targetRevision: main and path containing envs/production", + "Helm source references the Git source values via $values syntax (e.g., $values/envs/production/values.yaml)", + "Output does NOT use helm.values as a string — uses helm.valuesObject or the $values ref pattern", + "Sync policy has automated with selfHeal: true but prune: false or prune omitted", + "syncOptions includes ServerSideApply=true", + "ignoreDifferences entry for group apps kind Deployment with jsonPointers /spec/replicas", + "Destination namespace is webapp-prod with server https://kubernetes.default.svc" + ] + }, + { + "id": 2, + "prompt": "Create an ApplicationSet using the merge generator that combines a cluster generator (selecting clusters labeled tier=production) with a git file generator reading config from https://github.com/org/fleet-config.git at path 'clusters/{{name}}/config.json'. The merge key should be 'name' so cluster-specific config from Git overrides the cluster generator defaults. Use goTemplate with goTemplateOptions missingkey=error. Each generated Application should deploy from an OCI Helm chart oci://ghcr.io/org/app at the version specified in the config file's 'appVersion' field. Set preserveResourcesOnDeletion at the spec level. Add an ignoreApplicationDifferences entry to ignore changes to spec.source.targetRevision so the image updater can modify it without causing drift.", + "expected_output": "An ApplicationSet with merge generator combining cluster and git file generators, goTemplateOptions, OCI Helm source, preserveResourcesOnDeletion at spec level, and ignoreApplicationDifferences.", + "files": [], + "expectations": [ + "Output contains kind: ApplicationSet with apiVersion: argoproj.io/v1alpha1", + "Output has goTemplate: true AND goTemplateOptions containing missingkey=error", + "Output uses a merge generator with mergeKeys containing 'name'", + "Merge generator contains a clusters generator with selector matchLabels tier: production", + "Merge generator contains a git file generator with repoURL https://github.com/org/fleet-config.git", + "Git file generator path references clusters/{{.name}}/config.json or equivalent Go template syntax", + "Template source uses oci:// prefix for the chart URL", + "Template source targetRevision references the appVersion field from the config (e.g., {{.appVersion}})", + "preserveResourcesOnDeletion: true is at spec level (sibling of generators/template), NOT under strategy", + "ignoreApplicationDifferences is present with jsonPointers or jqPathExpressions targeting spec.source.targetRevision" + ] + }, + { + "id": 3, + "prompt": "Generate an Argo Rollout for 'checkout-svc' in namespace 'checkout' with blue-green strategy. Active service is 'checkout-active', preview service is 'checkout-preview'. Do NOT auto-promote — require manual approval. Add a prePromotionAnalysis that runs an AnalysisTemplate called 'smoke-test' which uses a Kubernetes Job provider (not Prometheus). The Job should run an image 'registry.example.com/smoke-tests:latest' that curls the preview service and exits 0 on success. The analysis should have a single measurement with no interval (run once). Also configure scaleDownDelaySeconds to 600 so the old ReplicaSet stays around for 10 minutes after promotion. Generate both the Rollout and the AnalysisTemplate.", + "expected_output": "A Rollout with blue-green strategy, manual promotion, prePromotionAnalysis referencing a Job-based AnalysisTemplate, and scaleDownDelaySeconds.", + "files": [], + "expectations": [ + "Output contains kind: Rollout with strategy.blueGreen (not canary)", + "Blue-green config has activeService: checkout-active and previewService: checkout-preview", + "autoPromotionEnabled is explicitly set to false", + "prePromotionAnalysis references templates with templateName: smoke-test", + "scaleDownDelaySeconds is set to 600", + "Output contains a separate AnalysisTemplate kind with name: smoke-test", + "AnalysisTemplate metric uses provider.job (not prometheus, not web)", + "Job spec container image is registry.example.com/smoke-tests:latest", + "Job container command or args references the preview service URL or hostname", + "AnalysisTemplate metric has count: 1 or count omitted (single measurement, no interval)" + ] + }, + { + "id": 4, + "prompt": "Set up Argo CD notifications with TWO notification services: 1) Slack for the 'platform-alerts' channel on sync failures with a template using Slack Block Kit (not legacy attachments) — use a section block with mrkdwn text showing app name, sync status, and error message, plus an actions block with a button linking to the ArgoCD UI. 2) GitHub commit status updates on the 'infra' repo https://github.com/org/infra using a GitHub App (appID and installationID from secret, not a PAT). The GitHub trigger should fire on every sync operation and set the commit status to success/failure/pending based on the operation phase. Use oncePer: app.status.operationState.syncResult.revision to avoid duplicate statuses for the same commit. Show all ConfigMap entries, Secret entries, and the Application annotations needed.", + "expected_output": "Complete notification setup with Slack Block Kit template, GitHub App commit status, oncePer dedup, and proper annotations.", + "files": [], + "expectations": [ + "ConfigMap contains service.slack config referencing $slack-token", + "ConfigMap contains service.github config with appID and installationID referencing secret keys", + "Secret contains slack-token key", + "Secret contains github-app-privateKey or similar key for GitHub App authentication", + "Slack template uses blocks array with section type and mrkdwn (not legacy attachments format)", + "Slack template includes an actions block with a button element linking to argocdUrl", + "GitHub trigger uses oncePer referencing app.status.operationState.syncResult.revision", + "GitHub trigger condition maps operation phases to commit status states (success/failure/pending)", + "Application annotation for Slack: notifications.argoproj.io/subscribe.on-sync-failed.slack", + "Application annotation for GitHub: notifications.argoproj.io/subscribe referencing the github trigger and service" + ] + }, + { + "id": 5, + "prompt": "I'm deploying Argo CD on OpenShift using the OpenShift GitOps Operator. Generate: 1) An ArgoCD custom resource (the operator CRD, not raw Argo CD manifests) in the openshift-gitops namespace with HA disabled, Route enabled with reencrypt TLS, OpenShift OAuth via Dex, RBAC that maps 'cluster-admins' group to admin role with default readonly policy, resource exclusions for Tekton TaskRun/PipelineRun and OLM InstallPlan, and notifications enabled. 2) A namespace-scoped ArgoCD instance for the 'platform-team' namespace with a Route and sourceNamespaces set. 3) Show the label needed on a target namespace 'platform-staging' so the namespace-scoped instance can manage it. 4) Explain what happens differently on OpenShift vs vanilla Kubernetes regarding SCCs when running Argo Rollouts pods.", + "expected_output": "ArgoCD CRD (argoproj.io/v1beta1) with OpenShift-specific configuration, namespace-scoped instance, managed-by label, and SCC explanation.", + "files": [], + "expectations": [ + "Uses apiVersion argoproj.io/v1beta1 kind ArgoCD (not v1alpha1, not raw Deployment manifests)", + "Cluster-scoped instance in openshift-gitops namespace (not argocd)", + "spec.server.route.enabled true with tls.termination reencrypt", + "spec.dex.openShiftOAuth true for OAuth integration", + "RBAC policy maps cluster-admins to role:admin", + "RBAC defaultPolicy is role:readonly", + "resourceExclusions includes tekton.dev TaskRun and PipelineRun", + "resourceExclusions includes operators.coreos.com InstallPlan", + "spec.notifications.enabled true", + "Namespace-scoped instance in platform-team namespace with route and sourceNamespaces", + "Target namespace label argocd.argoproj.io/managed-by pointing to the instance namespace", + "Explains that OpenShift uses SecurityContextConstraints (SCCs) instead of PodSecurityPolicies and Rollout pods need appropriate SCC bindings" + ] + }, + { + "id": 6, + "prompt": "I need to set up Argo CD for a multi-tenant platform where 3 teams (frontend, backend, data) each manage their own Applications in their own namespaces. I want a single Argo CD instance, not separate instances per team. Each team should only be able to deploy to their own namespaces and access their own Git repos. The frontend team uses namespace 'team-frontend-apps', the backend team uses 'team-backend-apps', and the data team uses 'team-data-apps'. Generate: 1) The argocd-cmd-params-cm ConfigMap change to enable this, 2) An AppProject for the frontend team that restricts them to their namespace, repos matching 'https://github.com/org/frontend-*', and only allows Deployments, Services, ConfigMaps, and Ingresses (no cluster-scoped resources), 3) A sample Application in the team-frontend-apps namespace referencing the project, 4) Show the RBAC policy that allows the 'frontend-devs' SSO group to get/sync apps in their project and namespace only.", + "expected_output": "ConfigMap with application.namespaces, AppProject with sourceNamespaces restricting to team namespace, Application in non-argocd namespace, and RBAC with three-part // syntax.", + "files": [], + "expectations": [ + "ConfigMap argocd-cmd-params-cm with data.application.namespaces listing team namespaces", + "AppProject has spec.sourceNamespaces containing team-frontend-apps", + "AppProject has sourceRepos restricted to https://github.com/org/frontend-*", + "AppProject has destinations restricted to frontend team namespaces only", + "AppProject has clusterResourceWhitelist that is empty or absent (no cluster-scoped resources)", + "AppProject has namespaceResourceWhitelist listing only Deployment, Service, ConfigMap, Ingress", + "Sample Application metadata.namespace is team-frontend-apps (NOT argocd)", + "Application spec.project references the frontend team project", + "RBAC policy uses three-part syntax: project/namespace/app (not legacy two-part)", + "RBAC policy scopes frontend-devs to only their project and namespace", + "Does NOT suggest running multiple Argo CD instances — uses single instance pattern", + "Mentions switching resource tracking to annotation or annotation+label for long composite names" + ] + }, + { + "id": 7, + "prompt": "We have 200 edge clusters behind restrictive firewalls where the control plane cannot reach the cluster API servers. We need GitOps on every cluster with centralized observability. Each edge cluster should be fully autonomous — if it loses connectivity to the control plane, it must continue reconciling. When connected, Application status should sync back to the control plane for monitoring. Describe the architecture and components needed. What is the recommended approach? Generate a high-level deployment plan with the components on the control plane cluster vs each edge cluster.", + "expected_output": "Description of argocd-agent with principal on control plane and agent on each workload cluster, autonomous mode, fully autonomous pattern with local Argo CD stack, and the communication model.", + "files": [], + "expectations": [ + "Recommends argocd-agent (not traditional multi-cluster Argo CD)", + "Describes the principal component running on the control plane", + "Describes the agent component running on each edge/workload cluster", + "Specifies autonomous mode (not managed mode) for edge clusters", + "Recommends the fully autonomous pattern with local Argo CD stack (application-controller + repo-server + redis) on each edge cluster", + "Explains that agents initiate connections outbound to the principal (not control plane reaching into clusters)", + "Mentions mTLS or certificate-based authentication between agent and principal", + "Explains that edge clusters continue reconciling independently during connectivity loss", + "States that Application status syncs back to control plane for observability when connected", + "Does NOT suggest storing cluster credentials on the control plane" + ] + }, + { + "id": 8, + "prompt": "I want to set up environment promotion for my app across dev, staging, and production using gitops-promoter. The app repo is on GitHub (org: acme-corp, repo: payments). I want automatic promotion from dev to staging (gated on Argo CD app health), but manual approval for production (gated on both app health and a performance-test check). Also add a security-scan proposed commit status that must pass before merging into any environment. Generate all the resources needed: ScmProvider, GitRepository, PromotionStrategy, and an ArgoCD CommitStatus to report app health. Use a GitHub App for authentication.", + "expected_output": "Complete gitops-promoter setup with ScmProvider (GitHub App), GitRepository, PromotionStrategy with 3 environments and commit status gating, and ArgocdCommitStatus for health reporting.", + "files": [], + "expectations": [ + "Uses apiVersion promoter.argoproj.io/v1alpha1 for all promoter CRDs", + "ScmProvider with github field containing appID and installationID", + "ScmProvider references a Secret for githubAppPrivateKey", + "GitRepository with github field containing owner: acme-corp and name: payments", + "GitRepository references the ScmProvider via scmProviderRef", + "PromotionStrategy with 3 environments: environment/development, environment/staging, environment/production", + "Production environment has autoMerge: false for manual approval", + "Dev and staging have autoMerge: true (explicit or default)", + "activeCommitStatuses includes argocd-app-health at the strategy level", + "Production environment has additional activeCommitStatus for performance-test", + "proposedCommitStatuses includes security-scan at the strategy level", + "ArgocdCommitStatus resource referencing the GitRepository and selecting the app" + ] + } + ] +} diff --git a/gitops/argo-knowledge/references/agent-mode.md b/gitops/argo-knowledge/references/agent-mode.md new file mode 100644 index 0000000..dc39945 --- /dev/null +++ b/gitops/argo-knowledge/references/agent-mode.md @@ -0,0 +1,89 @@ +# Argo CD Agent Mode + +Reference for the argocd-agent project — a hub-and-spoke architecture for managing +multiple clusters where the agent initiates connections to the control plane, eliminating +the need for the control plane to have direct API access to workload clusters. + +**Project:** [argoproj-labs/argocd-agent](https://github.com/argoproj-labs/argocd-agent) +**Docs:** [argocd-agent.readthedocs.io](https://argocd-agent.readthedocs.io/latest/) +**Status:** GA in Red Hat OpenShift GitOps 1.19; upstream still pre-v1 + +## Architecture + +### Components + +| Component | Runs On | Role | +|-----------|---------|------| +| Principal | Control plane cluster | Accepts agent connections, serves as config hub and observability aggregation point | +| Agent | Each workload cluster | Initiates gRPC connection to principal, manages local Argo CD reconciliation | +| Local Argo CD | Each workload cluster | application-controller + repo-server + redis running locally for autonomous operation | + +### Communication Model + +- **Agent-initiated only** — connections always flow from workload clusters to the control plane, never the reverse +- **gRPC bi-directional streaming** — both parties send and receive messages over the same connection +- **mTLS everywhere** — all communications encrypted and certificate-authenticated +- **Lightweight** — synchronizes only Argo CD config objects (Applications, AppProjects, repo config), not cluster resources +- **Resilient** — designed for unreliable connectivity; workload clusters continue reconciling independently when disconnected + +### What Gets Synchronized + +The agent syncs Argo CD configuration resources between control plane and workload clusters: +- `Application` specs and status +- `AppProject` configurations +- Repository credentials and configuration +- Does NOT sync arbitrary Kubernetes resources — reconciliation happens locally + +## Operational Modes + +### Managed Mode + +- Application specs originate on the **control plane** and are distributed to workload clusters +- Control plane is the source of truth for what should be deployed +- Equivalent to traditional Argo CD but without requiring cluster credentials on the control plane +- Best for: centralized platform teams managing fleet deployments + +### Autonomous Mode + +- Application specs are defined **locally on workload clusters** +- Changes sync back to the control plane for observability only +- Workload clusters operate independently even during connectivity loss +- Best for: edge deployments, air-gapped environments, teams needing local autonomy + +### Mixed Mode + +- Different clusters can operate in different modes within the same fleet +- Example: production clusters in managed mode, edge clusters in autonomous mode + +## Fully Autonomous Pattern (Recommended) + +The recommended production deployment runs a complete Argo CD stack on each workload cluster: + +- **application-controller** — local reconciliation, no dependency on control plane +- **repo-server** — local manifest generation from Git/OCI/Helm sources +- **redis** — local caching + +The control plane serves as a configuration hub and observability aggregation point. +Workload clusters continue all GitOps operations during control plane maintenance, +upgrades, or outages. + +## When to Use Agent Mode + +| Scenario | Traditional Argo CD | Agent Mode | +|----------|-------------------|------------| +| Small fleet (< 10 clusters) | Works well | Overkill | +| Large fleet (50+ clusters) | Resource strain on control plane | Designed for this | +| Air-gapped / restricted networks | Requires VPN or firewall rules | Agent initiates outbound only | +| Edge / IoT / remote sites | Impractical — unreliable connectivity | Built for intermittent connectivity | +| Multi-cloud | Each cloud needs inbound rules | Outbound-only from each cloud | +| Compliance requires no stored credentials | Must store cluster creds centrally | No cluster credentials on control plane | + +## Comparison with Traditional Multi-Cluster + +| Aspect | Traditional | Agent Mode | +|--------|------------|------------| +| Credential storage | Control plane stores kubeconfig/tokens for every cluster | No cluster credentials on control plane | +| Network direction | Control plane → cluster API (inbound to cluster) | Agent → control plane (outbound from cluster) | +| Failure mode | Control plane down = no sync for any cluster | Each cluster continues independently | +| Resource scaling | Linear with cluster count on control plane | Compute distributed to workload clusters | +| Security surface | Concentrated — control plane has access to all clusters | Distributed — each agent has minimal permissions | diff --git a/gitops/argo-knowledge/references/app-projects.md b/gitops/argo-knowledge/references/app-projects.md new file mode 100644 index 0000000..98d3694 --- /dev/null +++ b/gitops/argo-knowledge/references/app-projects.md @@ -0,0 +1,300 @@ +# AppProject Reference + +## Overview + +AppProject defines a logical grouping of Applications with RBAC, source/destination restrictions, and sync windows. Every Application must reference a project. The `default` project is created automatically and allows all sources/destinations. + +```yaml +apiVersion: argoproj.io/v1alpha1 +kind: AppProject +metadata: + name: + namespace: argocd +spec: + description: + sourceRepos: [] + sourceNamespaces: [] + destinations: [] + clusterResourceWhitelist: [] + clusterResourceBlacklist: [] + namespaceResourceWhitelist: [] + namespaceResourceBlacklist: [] + orphanedResources: {} + roles: [] + syncWindows: [] + signatureKeys: [] + permitOnlyProjectScopedClusters: false +``` + +## Source Restrictions + +### sourceRepos + +Restrict which Git repositories or Helm chart repositories Applications can use: + +```yaml +spec: + sourceRepos: + - https://github.com/org/repo.git # Specific repo + - https://github.com/org/* # Wildcard org + - https://charts.example.com # Helm repo + - '*' # Allow all (default project) +``` + +### sourceNamespaces + +Allow Applications to be created in namespaces other than the Argo CD namespace: + +```yaml +spec: + sourceNamespaces: + - team-a-argocd + - team-b-argocd +``` + +Requires `--application-namespaces` on the Argo CD controller. + +## Destination Restrictions + +Control which clusters and namespaces Applications can deploy to: + +```yaml +spec: + destinations: + - server: https://kubernetes.default.svc + namespace: team-a-* # Wildcard namespace + - server: https://prod.example.com + namespace: production + name: prod-cluster # Optional cluster name + - server: '*' + namespace: '*' # Allow all (default project) +``` + +**Fields:** +- `server` — Cluster API URL. Use `*` for all clusters. +- `namespace` — Target namespace. Use `*` for all namespaces. Wildcards supported. +- `name` — Cluster name (alternative to server). + +If both `server` and `name` are specified, both must match. + +## Cluster Resource Allow/Deny Lists + +Control which cluster-scoped resources (ClusterRole, Namespace, etc.) Applications can manage: + +```yaml +spec: + # Allow specific cluster-scoped resources + clusterResourceWhitelist: + - group: '' + kind: Namespace + - group: rbac.authorization.k8s.io + kind: ClusterRole + - group: rbac.authorization.k8s.io + kind: ClusterRoleBinding + + # Deny specific cluster-scoped resources (applied after whitelist) + clusterResourceBlacklist: + - group: '' + kind: ResourceQuota +``` + +- Empty `clusterResourceWhitelist` = no cluster-scoped resources allowed +- `group: '*'`, `kind: '*'` = allow all + +## Namespace Resource Allow/Deny Lists + +Control which namespace-scoped resources Applications can manage: + +```yaml +spec: + # Allow only specific resources + namespaceResourceWhitelist: + - group: '' + kind: ConfigMap + - group: '' + kind: Secret + - group: apps + kind: Deployment + - group: '' + kind: Service + + # Deny specific resources (applied after whitelist) + namespaceResourceBlacklist: + - group: '' + kind: LimitRange +``` + +Default: all namespace-scoped resources are allowed (if no whitelist specified). + +## Orphaned Resource Monitoring + +Detect resources in a namespace that aren't managed by any Application: + +```yaml +spec: + orphanedResources: + warn: true # Show warning in UI/CLI + ignore: + - group: "" + kind: ConfigMap + name: kube-root-ca.crt # Ignore specific resource + - group: "" + kind: ServiceAccount + name: default # Ignore default SA + - group: "" + kind: Endpoints # Ignore all Endpoints +``` + +- `warn: true` — Display orphaned resources as warnings in the Application UI +- `ignore` — Exclude specific resources from orphan detection. Supports `group`, `kind`, and optional `name`. + +## Roles and Policies (Project-Level RBAC) + +Define roles with specific permissions within a project: + +```yaml +spec: + roles: + - name: developer + description: Developer access + policies: + - p, proj:my-project:developer, applications, get, my-project/*, allow + - p, proj:my-project:developer, applications, sync, my-project/*, allow + - p, proj:my-project:developer, applications, action/*, my-project/*, allow + groups: + - my-org:developers # SSO group binding + jwtTokens: + - iat: 1535390316 # API token (managed via CLI) + + - name: ops + description: Operations team + policies: + - p, proj:my-project:ops, applications, *, my-project/*, allow + - p, proj:my-project:ops, logs, get, my-project/*, allow + groups: + - my-org:ops-team +``` + +**Policy format:** `p, , , , /, ` + +**Resources and actions:** +| Resource | Actions | +|----------|---------| +| applications | get, create, update, delete, sync, override, action/// | +| logs | get | +| exec | create | +| repositories | get, create, update, delete | +| clusters | get, create, update, delete | + +## Sync Windows + +Restrict when syncs can occur: + +```yaml +spec: + syncWindows: + - kind: allow # "allow" or "deny" + schedule: '0 8 * * 1-5' # Cron schedule (UTC) + duration: 10h # Window duration + applications: + - '*' # Apply to all apps in project + namespaces: + - production + clusters: + - prod-* + manualSync: true # Also restrict manual syncs (default: false) + timeZone: America/New_York # Optional timezone + + - kind: deny + schedule: '0 0 * * 5' # Deny on Fridays midnight UTC + duration: 48h # Until Sunday midnight UTC + applications: + - '*' + manualSync: false # Allow manual syncs during deny window +``` + +**Logic:** +- If only `allow` windows exist: syncs are blocked outside all allow windows +- If only `deny` windows exist: syncs are allowed outside all deny windows +- If both exist: sync is allowed when it falls inside an allow window AND outside all deny windows +- `manualSync: true` means the window applies to manual syncs too (default is automated only) + +## Signature Verification + +Require GPG signature verification on Git commits: + +```yaml +spec: + signatureKeys: + - keyID: ABCDEF1234567890 + - keyID: 1234567890ABCDEF +``` + +When configured, only commits signed by listed keys can be synced. Unsigned or differently-signed commits are rejected. + +GPG public keys must be imported into the `argocd-gpg-keys-cm` ConfigMap. + +## Complete Example + +```yaml +apiVersion: argoproj.io/v1alpha1 +kind: AppProject +metadata: + name: team-platform + namespace: argocd +spec: + description: Platform team project + + sourceRepos: + - https://github.com/org/platform-*.git + - https://charts.bitnami.com/bitnami + + destinations: + - server: https://kubernetes.default.svc + namespace: platform-* + - server: https://prod.example.com + namespace: platform-* + + clusterResourceWhitelist: + - group: '' + kind: Namespace + - group: rbac.authorization.k8s.io + kind: ClusterRole + - group: rbac.authorization.k8s.io + kind: ClusterRoleBinding + + namespaceResourceBlacklist: + - group: '' + kind: LimitRange + - group: '' + kind: ResourceQuota + + orphanedResources: + warn: true + ignore: + - group: "" + kind: ConfigMap + name: kube-root-ca.crt + + roles: + - name: admin + policies: + - p, proj:team-platform:admin, applications, *, team-platform/*, allow + groups: + - org:platform-admins + - name: viewer + policies: + - p, proj:team-platform:viewer, applications, get, team-platform/*, allow + groups: + - org:platform-viewers + + syncWindows: + - kind: deny + schedule: '0 18 * * 5' + duration: 62h + applications: + - '*' + clusters: + - prod-* + manualSync: false +``` diff --git a/gitops/argo-knowledge/references/applications.md b/gitops/argo-knowledge/references/applications.md new file mode 100644 index 0000000..51edc0c --- /dev/null +++ b/gitops/argo-knowledge/references/applications.md @@ -0,0 +1,375 @@ +# Argo CD Applications Reference + +## Application Spec Structure + +```yaml +apiVersion: argoproj.io/v1alpha1 +kind: Application +metadata: + name: + namespace: argocd # Must be the Argo CD namespace + labels: {} + annotations: {} + finalizers: + - resources-finalizer.argocd.argoproj.io # Cascade-delete managed resources +spec: + project: # AppProject reference (default: "default") + + source: # Single source (use sources[] for multi-source) + repoURL: # Git URL or Helm chart repo URL + targetRevision: # Branch, tag, commit SHA, or semver range + path: # For Git repos: path to manifests directory + chart: # For Helm repos: chart name (mutually exclusive with path) + helm: # Helm-specific options + releaseName: + valuesObject: {} # Structured values (preferred over values string) + values: | # Multi-line YAML string values (use valuesObject instead) + valueFiles: # List of values files relative to path + - values-prod.yaml + parameters: # Individual parameter overrides + - name: + value: + forceString: false # Force value to be treated as string + fileParameters: # File-based parameters + - name: + path: + ignoreMissingValueFiles: false # Don't error on missing values files + skipCrds: false # Skip CRD installation + passCredentials: false # Pass credentials to all domains + version: v3 # Helm version (v3 default) + kustomize: # Kustomize-specific options + namePrefix: + nameSuffix: + commonLabels: {} + commonAnnotations: {} + images: # Override kustomize images + - : + forceCommonLabels: false + forceCommonAnnotations: false + version: + directory: # Plain directory options + recurse: false # Recurse into subdirectories + jsonnet: # Jsonnet-specific options + extVars: [] + tlas: [] + libs: [] + exclude: # Exclude files matching glob + include: # Only include files matching glob + plugin: # Config management plugin + name: + env: + - name: + value: + parameters: + - name: + string: + + sources: [] # Multi-source (list of source objects) + + destination: + server: # Cluster API server URL + name: # OR cluster name (mutually exclusive with server) + namespace: # Target namespace + + syncPolicy: + automated: # Automated sync settings + prune: false # Delete resources not in Git + selfHeal: false # Re-sync when live state drifts + allowEmpty: false # Allow syncing when source produces zero manifests + syncOptions: [] # List of sync options (see below) + managedNamespaceMetadata: # Metadata for auto-created namespaces + labels: {} + annotations: {} + retry: + limit: # Max retry attempts (0 = no retry, -1 = infinite) + backoff: + duration: 5s + factor: 2 + maxDuration: 3m + + ignoreDifferences: [] # Ignore specific field diffs (see below) + + info: # Informational metadata (displayed in UI) + - name: + value: + + revisionHistoryLimit: 10 # Number of sync history entries to keep +``` + +## Sync Policies + +### Automated Sync + +```yaml +syncPolicy: + automated: + prune: true # DELETE resources from cluster that are no longer in Git + selfHeal: true # Re-sync when someone manually modifies a resource in the cluster + allowEmpty: false # If true, allows sync when Git source produces zero resources +``` + +- **prune** — When a manifest is removed from Git, the corresponding resource is deleted from the cluster. Without this, removed manifests leave orphaned resources. +- **selfHeal** — When a resource is manually modified in the cluster (kubectl edit, etc.), Argo CD reverts it to match Git. Polling interval is configurable via `timeout.reconciliation` in `argocd-cm` (default: 180s). +- **allowEmpty** — Safety guard. If your source suddenly produces zero manifests (broken Helm chart, empty directory), this prevents deleting everything. Keep `false` in production. + +### Manual Sync + +Omit `syncPolicy.automated` entirely. Users must click "Sync" in the UI or run `argocd app sync `. + +## Sync Options + +Applied as a list of `Key=Value` strings under `syncPolicy.syncOptions`: + +| Option | Default | Description | +|--------|---------|-------------| +| `CreateNamespace=true` | false | Create the destination namespace if it doesn't exist | +| `ServerSideApply=true` | false | Use server-side apply instead of client-side. Avoids annotation size limits. Required for CRDs >256KB | +| `Replace=true` | false | Use `kubectl replace` instead of `kubectl apply`. Destructive — replaces entire resource | +| `PruneLast=true` | false | Delete removed resources after all other resources are synced and healthy | +| `PrunePropagationPolicy=foreground` | background | Deletion propagation: `foreground`, `background`, or `orphan` | +| `ApplyOutOfSyncOnly=true` | false | Only apply resources that are out of sync (optimization for large apps) | +| `RespectIgnoreDifferences=true` | false | Use ignoreDifferences during sync (not just diff display). Prevents reverting ignored fields | +| `Validate=false` | true | Skip schema validation during apply | +| `SkipDryRunOnMissingResource=true` | false | Skip dry-run for resource types not installed on the cluster | +| `FailOnSharedResource=true` | false | Fail sync if a resource is managed by another Application | + +Sync options can also be set per-resource via annotation: + +```yaml +metadata: + annotations: + argocd.argoproj.io/sync-options: ServerSideApply=true,PruneLast=true +``` + +## Sync Waves and Hooks + +### Sync Waves + +Control the order resources are applied within a sync operation: + +```yaml +metadata: + annotations: + argocd.argoproj.io/sync-wave: "5" +``` + +- Waves are integers (negative allowed). Lower numbers sync first. +- Default wave is 0. Resources within the same wave sync in parallel. +- Argo CD waits for all resources in a wave to be healthy before moving to the next wave. +- Typical pattern: `-1` for namespaces/CRDs, `0` for core resources, `1` for dependent resources, `5` for post-deploy jobs. + +### Sync Hooks + +Run Jobs or Pods at specific phases of the sync lifecycle: + +```yaml +metadata: + annotations: + argocd.argoproj.io/hook: PreSync + argocd.argoproj.io/hook-delete-policy: HookSucceeded +``` + +**Hook phases:** +- `PreSync` — Before the sync (e.g., database migrations, schema changes) +- `Sync` — During the sync (applied alongside manifests in the same wave) +- `PostSync` — After all resources are synced and healthy (e.g., smoke tests, notifications) +- `SyncFail` — When sync fails (e.g., cleanup, alerting) +- `Skip` — Skip this resource during sync entirely + +**Delete policies:** +- `HookSucceeded` — Delete hook resource after it succeeds +- `HookFailed` — Delete hook resource after it fails +- `BeforeHookCreation` — Delete existing hook resource before creating a new one (default) + +## Health Checks + +### Built-in Health Checks + +Argo CD has built-in health assessments for standard Kubernetes resources: + +| Resource | Healthy When | +|----------|-------------| +| Deployment | All replicas updated and available | +| StatefulSet | All replicas updated and ready | +| DaemonSet | All desired pods scheduled and available | +| ReplicaSet | All replicas available | +| Pod | Running and ready | +| Service | Exists (always healthy) | +| Ingress | Has at least one address assigned | +| PVC | Bound | +| Job | Succeeded | +| PDB | CurrentHealthy >= DesiredHealthy | + +**Health statuses:** +- `Healthy` — Resource is operating normally +- `Progressing` — Resource is not yet healthy but is making progress (e.g., Deployment rolling out) +- `Degraded` — Resource has failed or errored +- `Suspended` — Resource is paused (e.g., suspended CronJob, Rollout paused) +- `Missing` — Resource does not exist in the cluster + +### Custom Health Checks (Lua) + +Add custom health checks for CRDs or override built-in checks in `argocd-cm`: + +```yaml +apiVersion: v1 +kind: ConfigMap +metadata: + name: argocd-cm + namespace: argocd +data: + resource.customizations.health.argoproj.io_Rollout: | + hs = {} + if obj.status ~= nil then + if obj.status.phase == "Healthy" then + hs.status = "Healthy" + hs.message = "Rollout is healthy" + elseif obj.status.phase == "Paused" then + hs.status = "Suspended" + hs.message = obj.status.message + elseif obj.status.phase == "Degraded" then + hs.status = "Degraded" + hs.message = obj.status.message + else + hs.status = "Progressing" + hs.message = "Rollout in progress" + end + end + return hs +``` + +## Resource Tracking Methods + +Configured in `argocd-cm` via `resource.trackingMethod`: + +| Method | Label Added | Annotation Added | Pros | Cons | +|--------|------------|-----------------|------|------| +| `label` | `app.kubernetes.io/instance` | — | Simple, visible via `kubectl` | Conflicts with Helm/other tools using same label | +| `annotation` | — | `argocd.argoproj.io/tracking-id` | No label conflicts | Not visible via label selectors | +| `annotation+label` | Both | Both | Maximum compatibility | Two tracking markers | + +Default is `annotation` in Argo CD 2.6+. + +## Ignore Differences + +Prevent specific fields from showing as OutOfSync: + +```yaml +spec: + ignoreDifferences: + - group: apps + kind: Deployment + jsonPointers: + - /spec/replicas # Ignore HPA-managed replicas + - group: "*" + kind: "*" + managedFieldsManagers: + - kube-controller-manager # Ignore fields managed by controller + - group: admissionregistration.k8s.io + kind: MutatingWebhookConfiguration + jqPathExpressions: + - .webhooks[]?.clientConfig.caBundle # Ignore injected CA bundles +``` + +**Methods:** +- `jsonPointers` — RFC 6901 JSON pointers. Exact field paths. E.g., `/spec/replicas`. +- `jqPathExpressions` — jq expressions for more complex matching. E.g., `.spec.template.spec.containers[]?.resources`. +- `managedFieldsManagers` — Ignore all fields managed by a specific field manager. Useful for controller-managed fields. + +To make ignored differences also ignored during sync (not just display), add `RespectIgnoreDifferences=true` to sync options. + +System-level ignore differences (applying to all Applications) can be configured in `argocd-cm`: + +```yaml +data: + resource.customizations.ignoreDifferences.all: | + jsonPointers: + - /metadata/resourceVersion + - /metadata/generation +``` + +## Multi-Cluster Management + +### Adding Clusters + +```bash +# Add a cluster using a kubeconfig context +argocd cluster add --name + +# Add with specific service account +argocd cluster add --service-account argocd-manager + +# Add with project restrictions +argocd cluster add --project my-project +``` + +### Cluster Secrets + +Each external cluster is stored as a Secret: + +```yaml +apiVersion: v1 +kind: Secret +metadata: + name: cluster-prod-east + namespace: argocd + labels: + argocd.argoproj.io/secret-type: cluster +type: Opaque +stringData: + name: prod-east + server: https://prod-east.example.com:6443 + config: | + { + "bearerToken": "", + "tlsClientConfig": { + "insecure": false, + "caData": "" + } + } +``` + +### In-Cluster Reference + +The cluster where Argo CD runs is always available as: +- `server: https://kubernetes.default.svc` + +No Secret needed. + +## Orphaned Resource Monitoring + +Detects resources in a namespace that are not managed by any Application: + +```yaml +# In AppProject spec +spec: + orphanedResources: + warn: true # Show warning in UI for orphaned resources + ignore: # Exclude certain resources from orphan detection + - group: "" + kind: ConfigMap + name: kube-root-ca.crt + - group: "" + kind: ServiceAccount + name: default +``` + +## Multi-Source Applications + +Combine multiple sources (e.g., Helm chart + separate values repo): + +```yaml +spec: + sources: + - repoURL: https://charts.example.com + chart: my-app + targetRevision: 1.2.3 + helm: + valueFiles: + - $values/apps/my-app/values-prod.yaml + - repoURL: https://github.com/org/config-repo.git + targetRevision: main + ref: values # Reference name used as $values above +``` + +The `ref` field creates a named reference. The `$ref` prefix in `valueFiles` resolves to the path of the referenced source. diff --git a/gitops/argo-knowledge/references/applicationsets.md b/gitops/argo-knowledge/references/applicationsets.md new file mode 100644 index 0000000..709d57f --- /dev/null +++ b/gitops/argo-knowledge/references/applicationsets.md @@ -0,0 +1,403 @@ +# ApplicationSet Reference + +## Overview + +ApplicationSet is a controller that generates Argo CD Applications from a template combined with one or more generators. It enables managing many Applications at scale. + +```yaml +apiVersion: argoproj.io/v1alpha1 +kind: ApplicationSet +metadata: + name: + namespace: argocd +spec: + goTemplate: true # Use Go templates (recommended) + goTemplateOptions: + - missingkey=error # Fail on missing template keys + generators: [] # List of generators + template: # Application template + metadata: + name: '' + labels: {} + annotations: {} + spec: + project: + source: {} + destination: {} + syncPolicy: {} + preserveResourcesOnDeletion: false # Keep generated Applications' resources when deleting AppSet + strategy: {} # Progressive sync strategy + templatePatch: "" # JSON merge patch applied to template +``` + +## Go Template vs Fasttemplate + +| Feature | Fasttemplate | Go Template | +|---------|-------------|-------------| +| Syntax | `{{name}}` | `{{.name}}` | +| Functions | None | Full Go template functions (sprig) | +| Conditionals | No | `{{if}}` / `{{else}}` / `{{end}}` | +| Loops | No | `{{range}}` | +| String ops | No | `upper`, `lower`, `replace`, `trimSuffix`, etc. | +| Enable | Default (legacy) | `goTemplate: true` | + +**Always use Go templates** for new ApplicationSets. Fasttemplate is legacy. + +Go template pitfall: `{{.path.basename}}` works, but `{{.path[0]}}` does not. Use `index .path 0` for array access. + +## Generators + +### Git Directory Generator + +Generates one Application per directory matching a path pattern: + +```yaml +generators: + - git: + repoURL: https://github.com/org/repo.git + revision: main + directories: + - path: apps/* # Include all dirs under apps/ + - path: apps/excluded-app # Exclude specific dirs + exclude: true + - path: apps/experimental-* # Exclude by pattern + exclude: true +``` + +**Template variables available:** +- `{{.path.path}}` — Full path (e.g., `apps/my-app`) +- `{{.path.basename}}` — Directory name (e.g., `my-app`) +- `{{.path.basenameNormalized}}` — DNS-safe name (e.g., `my-app`) +- `{{.path[N]}}` — Nth path segment (use `index .path N`) + +### Git File Generator + +Generates one Application per JSON/YAML file: + +```yaml +generators: + - git: + repoURL: https://github.com/org/repo.git + revision: main + files: + - path: config/**/config.json +``` + +Example `config.json`: +```json +{ + "appName": "my-app", + "namespace": "production", + "replicaCount": 3, + "cluster": { + "server": "https://prod.example.com" + } +} +``` + +Template usage: `{{.appName}}`, `{{.cluster.server}}` + +### List Generator + +Explicit list of parameter sets: + +```yaml +generators: + - list: + elements: + - cluster: staging + url: https://staging.example.com + values: + environment: staging + replicas: "1" + - cluster: production + url: https://prod.example.com + values: + environment: production + replicas: "3" +``` + +Template: `{{.cluster}}`, `{{.url}}`, `{{.values.environment}}` + +### Cluster Generator + +Generates one Application per registered Argo CD cluster: + +```yaml +generators: + - clusters: + selector: + matchLabels: + environment: production + tier: frontend + values: + revision: release-2.0 + namespace: frontend +``` + +**Template variables:** +- `{{.name}}` — Cluster name +- `{{.server}}` — Cluster API server URL +- `{{.metadata.labels.}}` — Cluster labels +- `{{.metadata.annotations.}}` — Cluster annotations +- `{{.values.}}` — Values defined in the generator + +Note: The in-cluster (where Argo CD runs) has `name: in-cluster` and `server: https://kubernetes.default.svc`. + +### Cluster Decision Resource Generator + +Uses an external custom resource to determine cluster selection: + +```yaml +generators: + - clusterDecisionResource: + configMapRef: my-placement-decision + name: placement-decision + requeueAfterSeconds: 180 +``` + +Used with Open Cluster Management (OCM) Placement decisions. + +### Matrix Generator + +Cartesian product of two generators — every combination of outputs: + +```yaml +generators: + - matrix: + generators: + - clusters: + selector: + matchLabels: + tier: production + - git: + repoURL: https://github.com/org/apps.git + revision: main + directories: + - path: apps/* +``` + +This produces one Application for each `(cluster, directory)` pair. Template has access to all variables from both generators. + +**Nesting rules:** +- Matrix can contain any two generators +- Nested matrix (matrix within matrix) is supported up to one level +- The inner generators cannot be matrix or merge generators + +### Merge Generator + +Combines outputs from multiple generators, merging by key: + +```yaml +generators: + - merge: + mergeKeys: + - server + generators: + - clusters: + selector: + matchLabels: + tier: production + values: + replicas: "3" + helmRelease: stable + - list: + elements: + - server: https://prod-east.example.com + values: + replicas: "5" # Override for this cluster +``` + +The merge generator lets you define defaults via one generator and overrides via another, merging on a shared key. + +### Pull Request Generator + +Creates one Application per open pull request: + +```yaml +generators: + - pullRequest: + github: + owner: my-org + repo: my-app + tokenRef: + secretName: github-token + key: token + labels: + - preview # Only PRs with this label + requeueAfterSeconds: 60 +``` + +**Template variables:** +- `{{.number}}` — PR number +- `{{.branch}}` — Source branch name +- `{{.branch_slug}}` — DNS-safe branch name +- `{{.head_sha}}` — Head commit SHA +- `{{.head_short_sha}}` — Short head commit SHA +- `{{.labels}}` — PR labels + +**Supported providers:** GitHub, GitLab, Bitbucket Server, Bitbucket Cloud + +### SCM Provider Generator + +Discovers repositories from a SCM organization/group: + +```yaml +generators: + - scmProvider: + github: + organization: my-org + tokenRef: + secretName: github-token + key: token + filters: + - repositoryMatch: "^service-.*" + pathsExist: + - deploy/kustomization.yaml + - labelMatch: "deploy-with-argocd" + cloneProtocol: https +``` + +**Template variables:** +- `{{.organization}}` — Org name +- `{{.repository}}` — Repo name +- `{{.url}}` — Clone URL +- `{{.branch}}` — Default branch +- `{{.labels}}` — Repo topics/labels + +**Supported providers:** GitHub, GitLab, Bitbucket Server, Bitbucket Cloud, Azure DevOps, Gitea + +### Plugin Generator + +Calls an external HTTP endpoint to generate parameters: + +```yaml +generators: + - plugin: + configMapRef: my-plugin-cm + requeueAfterSeconds: 300 + input: + parameters: + team: platform +``` + +The plugin ConfigMap contains the endpoint URL and token. The endpoint returns a JSON array of parameter sets. + +## Template Override + +Per-generator template overrides the top-level template: + +```yaml +generators: + - list: + elements: + - name: special-app + namespace: special + template: + metadata: + name: 'override-{{.name}}' # Overrides top-level template + spec: + project: special-project +template: + metadata: + name: 'default-{{.name}}' # Used by other generators + spec: + project: default +``` + +## Progressive Syncs (RollingSync) + +Roll out generated Applications in stages: + +```yaml +spec: + strategy: + type: RollingSync + rollingSync: + steps: + - matchExpressions: + - key: envLabel + operator: In + values: + - dev + maxUpdate: 100% # Deploy all dev apps first + - matchExpressions: + - key: envLabel + operator: In + values: + - staging + maxUpdate: 100% # Then all staging + - matchExpressions: + - key: envLabel + operator: In + values: + - production + maxUpdate: 25% # Then 25% of prod at a time +``` + +**Step fields:** +- `matchExpressions` — Label selectors to match Applications generated by this step. Labels come from `template.metadata.labels`. +- `maxUpdate` — How many Applications to sync in parallel. Integer count or percentage string. + +Applications must have labels matching the `matchExpressions` for steps to apply. Unmatched Applications sync in the final implicit step. + +## Sync Policy for ApplicationSets + +Controls what happens when the ApplicationSet or its generators change: + +```yaml +spec: + syncPolicy: + preserveResourcesOnDeletion: true # When AppSet is deleted, keep managed Applications' cluster resources + applicationsSync: create-only # Only create, never update existing Applications + # applicationsSync: create-update # Create and update (default) + # applicationsSync: create-delete # Create and delete, but don't update +``` + +## preserveResourcesOnDeletion + +```yaml +spec: + preserveResourcesOnDeletion: true +``` + +When the ApplicationSet is deleted: +- `false` (default) — All generated Applications and their cluster resources are deleted +- `true` — Generated Applications are deleted, but their managed cluster resources are left in place (orphaned) + +This is a safety mechanism. Use `true` in production to prevent accidental mass deletion. + +## templatePatch + +Apply a JSON merge patch to the generated Application template based on generator parameters: + +```yaml +spec: + goTemplate: true + generators: + - list: + elements: + - name: my-app + autoSync: "true" + template: + metadata: + name: '{{.name}}' + spec: + project: default + source: + repoURL: https://github.com/org/repo.git + path: 'apps/{{.name}}' + targetRevision: main + destination: + server: https://kubernetes.default.svc + templatePatch: | + {{- if eq .autoSync "true" }} + spec: + syncPolicy: + automated: + prune: true + selfHeal: true + {{- end }} +``` + +This is useful for conditionally adding fields to the Application spec based on generator parameters without duplicating templates. diff --git a/gitops/argo-knowledge/references/best-practices.md b/gitops/argo-knowledge/references/best-practices.md new file mode 100644 index 0000000..d678288 --- /dev/null +++ b/gitops/argo-knowledge/references/best-practices.md @@ -0,0 +1,392 @@ +# Argo Best Practices Reference + +## Sync Policy Recommendations + +### Production Applications + +```yaml +syncPolicy: + automated: + prune: true # Enable — but only after testing in lower envs + selfHeal: true # Prevent manual drift + allowEmpty: false # Safety: never sync zero manifests + syncOptions: + - CreateNamespace=true + - ServerSideApply=true # Handles large CRDs, avoids annotation limits + - PruneLast=true # Delete removed resources after new ones are healthy + - RespectIgnoreDifferences=true # Don't revert fields you explicitly ignore + retry: + limit: 5 + backoff: + duration: 5s + factor: 2 + maxDuration: 3m +``` + +### Checklist + +- [ ] `prune: true` tested in dev/staging before production +- [ ] `selfHeal: true` enabled to prevent drift +- [ ] `allowEmpty: false` (default) — never disable in production +- [ ] `retry` configured to handle transient failures +- [ ] `PruneLast=true` to avoid downtime during resource replacement +- [ ] `ServerSideApply=true` for CRDs and large resources +- [ ] `targetRevision` pinned to a tag or SHA in production (never `HEAD`) +- [ ] `ignoreDifferences` configured for controller-managed fields (e.g., replicas with HPA) +- [ ] `RespectIgnoreDifferences=true` if using `ignoreDifferences` + +## Resource Tracking Method Selection + +| Method | When to Use | +|--------|------------| +| `annotation` (default) | Most cases. No conflicts with other tools. | +| `label` | You need to query Argo-managed resources via label selectors. | +| `annotation+label` | Migrating from label to annotation tracking. Both tools need to find the resources. | + +**Recommendation:** Use `annotation` (default). Only use `label` or `annotation+label` if you have a specific need. + +Configured in `argocd-cm`: +```yaml +data: + resource.trackingMethod: annotation +``` + +## Health Check Customization + +### When to Add Custom Health Checks + +- CRDs that Argo CD doesn't know about (custom operators, third-party CRDs) +- Override default health logic (e.g., treat a specific condition as healthy) +- Rollout, AnalysisRun, and other Argo CRDs (built-in support, but you may need to tune) + +### Best Practices + +- [ ] Custom health checks for all CRDs your Applications manage +- [ ] Health checks should be conservative — prefer `Progressing` over `Healthy` when uncertain +- [ ] Test health check Lua scripts against real resource states +- [ ] Document custom health checks in your platform runbook + +### Example: Custom Health Check for a CRD + +```yaml +# argocd-cm ConfigMap +data: + resource.customizations.health.myorg.io_MyResource: | + hs = {} + if obj.status ~= nil and obj.status.conditions ~= nil then + for _, condition in ipairs(obj.status.conditions) do + if condition.type == "Ready" then + if condition.status == "True" then + hs.status = "Healthy" + hs.message = condition.message + elseif condition.status == "False" then + hs.status = "Degraded" + hs.message = condition.message + else + hs.status = "Progressing" + hs.message = "Waiting for Ready condition" + end + return hs + end + end + end + hs.status = "Progressing" + hs.message = "No status available" + return hs +``` + +## RBAC Layering + +### Architecture + +``` +SSO Provider (OIDC/SAML) + └── Groups (org:team-a, org:platform-admins) + └── Argo CD RBAC (argocd-rbac-cm) + └── AppProject Roles + └── Source/Destination Restrictions +``` + +### Layer 1: Global RBAC (argocd-rbac-cm) + +```yaml +apiVersion: v1 +kind: ConfigMap +metadata: + name: argocd-rbac-cm + namespace: argocd +data: + policy.default: role:readonly # Default for authenticated users + policy.csv: | + # Platform admins — full access + g, org:platform-admins, role:admin + + # Team roles — scoped to projects + p, role:team-deployer, applications, sync, */*, allow + p, role:team-deployer, applications, get, */*, allow + p, role:team-deployer, applications, action/*, */*, allow + g, org:deployers, role:team-deployer + + # Read-only for everyone else + p, role:readonly, applications, get, */*, allow + p, role:readonly, logs, get, */*, allow + scopes: '[groups, email]' +``` + +### Layer 2: Project-Level RBAC + +```yaml +apiVersion: argoproj.io/v1alpha1 +kind: AppProject +metadata: + name: team-a +spec: + roles: + - name: admin + policies: + - p, proj:team-a:admin, applications, *, team-a/*, allow + groups: + - org:team-a-leads + - name: deployer + policies: + - p, proj:team-a:deployer, applications, get, team-a/*, allow + - p, proj:team-a:deployer, applications, sync, team-a/*, allow + groups: + - org:team-a +``` + +### Layer 3: Source/Destination Restrictions + +```yaml +spec: + sourceRepos: + - https://github.com/org/team-a-*.git + destinations: + - server: https://kubernetes.default.svc + namespace: team-a-* + clusterResourceWhitelist: + - group: '' + kind: Namespace +``` + +### Checklist + +- [ ] SSO configured with group claims +- [ ] `policy.default: role:readonly` in `argocd-rbac-cm` +- [ ] Platform admins mapped to `role:admin` +- [ ] Per-team AppProjects with source/destination restrictions +- [ ] Project roles mapped to SSO groups +- [ ] Sync windows for production projects +- [ ] No team has `clusterResourceWhitelist: [{group: '*', kind: '*'}]` unless justified + +## ApplicationSet vs App-of-Apps Selection + +| Criteria | Use ApplicationSet | Use App-of-Apps | +|----------|-------------------|----------------| +| # Applications | > 20 | < 20 | +| Pattern-based | Yes (all apps look similar) | No (each app is unique) | +| Auto-discovery needed | Yes | No | +| Multi-cluster | Yes (cluster generator) | Manual per cluster | +| Progressive rollout | Yes (rollingSync) | Manual (sync waves) | +| Per-app customization | Limited (template + patch) | Full (each is a manifest) | +| Bootstrap | No (needs Argo CD running) | Yes (root app bootstraps) | + +**Common pattern:** Use app-of-apps to bootstrap the cluster (install Argo CD, CRDs, platform components) and ApplicationSets for application workloads. + +## Rollout Adoption Strategy + +### Migration Path: Deployment to Rollout + +1. **Start with `Rollout` that mirrors your current `Deployment`:** + ```yaml + apiVersion: argoproj.io/v1alpha1 + kind: Rollout + spec: + strategy: + canary: + steps: + - setWeight: 100 # Immediate 100% — same as Deployment + ``` + +2. **Add a simple canary step:** + ```yaml + steps: + - setWeight: 20 + - pause: { duration: 5m } + - setWeight: 100 + ``` + +3. **Add analysis:** + ```yaml + steps: + - setWeight: 20 + - analysis: + templates: + - templateName: success-rate + - setWeight: 100 + ``` + +4. **Add traffic routing (if using service mesh/ingress controller).** + +### Checklist + +- [ ] Install Argo Rollouts controller +- [ ] Create canary and stable Services +- [ ] Replace `Deployment` with `Rollout` (same pod template) +- [ ] Start with simple weight-based steps, no analysis +- [ ] Add AnalysisTemplates with `dryRun` first to validate metrics without blocking rollouts +- [ ] Remove `dryRun` once metrics are validated +- [ ] Configure traffic routing if available +- [ ] Set up Rollouts dashboard/notifications + +## Workflow Resource Management + +### Checklist + +- [ ] **Always set `activeDeadlineSeconds`** — stuck workflows consume resources forever +- [ ] **Configure `podGC`** — completed pods accumulate without it +- [ ] **Set `ttlStrategy`** — auto-delete completed workflow CRs +- [ ] **Use `retryStrategy`** with `backoff` for transient failures +- [ ] **Limit `parallelism`** — prevent thundering herd +- [ ] **Use `resourceQuota`** in the workflow namespace +- [ ] **Archive workflows** to PostgreSQL/MySQL for long-term storage +- [ ] **Use `workflowTemplateRef`** — avoid duplicating templates +- [ ] **Set `podPriorityClassName`** for production workflows + +### Resource Defaults + +Configure defaults in the Workflow Controller ConfigMap: + +```yaml +# workflow-controller-configmap +data: + workflowDefaults: | + spec: + activeDeadlineSeconds: 7200 + ttlStrategy: + secondsAfterCompletion: 86400 + secondsAfterSuccess: 3600 + secondsAfterFailure: 259200 + podGC: + strategy: OnPodSuccess + deleteDelayDuration: 120s + securityContext: + runAsNonRoot: true + runAsUser: 1000 +``` + +## Secret Management Approaches + +### Option 1: Sealed Secrets (Bitnami) + +```yaml +# Encrypt secrets client-side, store encrypted SealedSecret in Git +apiVersion: bitnami.com/v1alpha1 +kind: SealedSecret +metadata: + name: my-secret + namespace: my-app +spec: + encryptedData: + password: AgByz+... # Encrypted with cluster's public key +``` + +- **Pro:** Secrets stored in Git (encrypted). Works with any GitOps workflow. +- **Con:** Cluster-specific encryption. Re-seal needed for each cluster. + +### Option 2: External Secrets Operator + +```yaml +apiVersion: external-secrets.io/v1beta1 +kind: ExternalSecret +metadata: + name: my-secret + namespace: my-app +spec: + refreshInterval: 1h + secretStoreRef: + name: vault-backend + kind: ClusterSecretStore + target: + name: my-secret + data: + - secretKey: password + remoteRef: + key: secret/my-app + property: password +``` + +- **Pro:** Secrets live in a vault (Vault, AWS SM, GCP SM, Azure KV). Git only has references. +- **Con:** Requires External Secrets Operator + secret store infrastructure. + +### Option 3: Vault CSI Provider + +```yaml +# Mount secrets as volumes via CSI driver +spec: + volumes: + - name: secrets + csi: + driver: secrets-store.csi.k8s.io + readOnly: true + volumeAttributes: + secretProviderClass: vault-secrets +``` + +- **Pro:** No Kubernetes Secret objects created. Secrets injected directly. +- **Con:** Requires CSI driver + Vault Agent. More complex setup. + +### Option 4: SOPS + Kustomize/Helm Secrets Plugin + +Encrypt values in-place within YAML files using SOPS: + +```bash +sops --encrypt --in-place secrets.yaml +``` + +Use Argo CD config management plugin to decrypt during sync. + +- **Pro:** Encrypted files in Git. Works with existing tooling. +- **Con:** Requires config management plugin setup. Key management complexity. + +### Recommendation + +| Team Size | Infrastructure | Recommendation | +|-----------|---------------|---------------| +| Small | Simple | Sealed Secrets | +| Medium | Cloud-native | External Secrets Operator | +| Large | Enterprise | Vault + External Secrets or CSI | +| Any | Already using SOPS | SOPS plugin | + +### Checklist + +- [ ] Never store plaintext secrets in Git +- [ ] Choose one approach and standardize across all teams +- [ ] Automate secret rotation +- [ ] Audit secret access +- [ ] Test secret sync with Argo CD (ensure CMP or operator is healthy before Application sync) + +## Argo CD High Availability + +### Checklist + +- [ ] Run at least 2 replicas of `argocd-server` +- [ ] Run at least 2 replicas of `argocd-repo-server` +- [ ] Run single `argocd-application-controller` with sharding for large-scale +- [ ] Redis HA (Sentinel or Redis Cluster) for caching +- [ ] Increase `--repo-server-timeout-seconds` for large repos +- [ ] Configure `resource.exclusions` to skip resources you don't need to track +- [ ] Set `--app-resync` interval based on scale (default 180s may be too frequent at scale) +- [ ] Monitor Argo CD metrics (Prometheus endpoint at `:8082/metrics`) + +## Multi-Tenancy Checklist + +- [ ] One AppProject per team/tenant +- [ ] Source repos restricted per project +- [ ] Destination namespaces restricted per project +- [ ] Cluster-scoped resource access denied by default +- [ ] SSO groups mapped to project roles +- [ ] Sync windows for production namespaces +- [ ] Network policies between tenant namespaces +- [ ] Resource quotas per namespace +- [ ] Separate Argo CD instances for hard multi-tenancy (if needed) diff --git a/gitops/argo-knowledge/references/events.md b/gitops/argo-knowledge/references/events.md new file mode 100644 index 0000000..bcb6614 --- /dev/null +++ b/gitops/argo-knowledge/references/events.md @@ -0,0 +1,818 @@ +# Argo Events Reference + +## Architecture + +Argo Events has three core components: + +1. **EventSource** — Consumes events from external systems and publishes them to the EventBus +2. **EventBus** — Message transport layer (NATS JetStream or Kafka) +3. **Sensor** — Subscribes to EventBus, applies filters/transforms, and fires triggers + +Flow: External Event -> EventSource -> EventBus -> Sensor -> Trigger Action + +## EventSource + +```yaml +apiVersion: argoproj.io/v1alpha1 +kind: EventSource +metadata: + name: + namespace: +spec: + eventBusName: default # EventBus to publish to (default: "default") + replicas: 1 # Number of EventSource pods + service: # Service config for webhook-based sources + ports: + - port: 12000 + targetPort: 12000 + template: # Pod template overrides + metadata: + labels: {} + spec: + containers: + - resources: {} + serviceAccountName: + tolerations: [] + nodeSelector: {} + + # Event source type configs (one or more): + : + : + # source-specific config +``` + +### EventSource Types + +#### Webhook + +```yaml +spec: + webhook: + deploy-hook: + endpoint: /deploy + port: "12000" + method: POST +``` + +#### GitHub + +```yaml +spec: + github: + push-events: + repositories: + - owner: my-org + names: + - my-repo + webhook: + endpoint: /github + port: "12000" + method: POST + url: https://events.example.com # External URL for webhook registration + events: + - push + - pull_request + - create + - delete + apiToken: + name: github-token + key: token + webhookSecret: + name: github-token + key: webhook-secret + contentType: json + active: true + insecure: false + deleteHookOnFinish: false +``` + +#### GitLab + +```yaml +spec: + gitlab: + merge-request: + gitlab_base_url: https://gitlab.example.com + projectID: "12345" + webhook: + endpoint: /gitlab + port: "12000" + method: POST + events: + - MergeRequestsEvents + - PushEvents + - TagPushEvents + accessToken: + name: gitlab-token + key: token + secretToken: + name: gitlab-token + key: webhook-secret + enableSSLVerification: true + deleteHookOnFinish: false +``` + +#### Kafka + +```yaml +spec: + kafka: + order-events: + url: kafka-broker:9092 + topic: orders + consumerGroup: + groupName: argo-events + rebalanceStrategy: range # range, sticky, roundrobin + partition: "0" # Optional specific partition + version: "2.0.0" + tls: + caCertSecret: + name: kafka-tls + key: ca.crt + sasl: + mechanism: PLAIN + userSecret: + name: kafka-creds + key: username + passwordSecret: + name: kafka-creds + key: password + jsonBody: true # Parse body as JSON + connectionBackoff: + duration: 10s + factor: 2 + steps: 5 +``` + +#### SNS + +```yaml +spec: + sns: + notifications: + topicArn: arn:aws:sns:us-east-1:123456789:my-topic + webhook: + endpoint: /sns + port: "12000" + method: POST + accessKey: + name: aws-creds + key: access-key + secretKey: + name: aws-creds + key: secret-key + region: us-east-1 +``` + +#### SQS + +```yaml +spec: + sqs: + messages: + region: us-east-1 + queue: my-queue + waitTimeSeconds: 20 + accessKey: + name: aws-creds + key: access-key + secretKey: + name: aws-creds + key: secret-key + jsonBody: true +``` + +#### NATS + +```yaml +spec: + nats: + events: + url: nats://nats.nats:4222 + subject: my-subject + jsonBody: true + auth: + token: + name: nats-token + key: token +``` + +#### Redis + +```yaml +spec: + redis: + events: + hostAddress: redis.redis:6379 + db: 0 + channels: + - my-channel + password: + name: redis-creds + key: password + jsonBody: true +``` + +#### Calendar (Cron) + +```yaml +spec: + calendar: + hourly: + schedule: "0 * * * *" # Cron expression + interval: 1h # OR interval + timezone: America/New_York + exclusionDates: + - 2024-12-25 + persistence: + catchup: + enabled: true # Fire missed events on restart + configMap: + name: calendar-state + createIfNotExist: true +``` + +#### Resource (Kubernetes Watch) + +```yaml +spec: + resource: + pod-changes: + namespace: default + group: "" + version: v1 + resource: pods + eventTypes: + - ADD + - UPDATE + - DELETE + filter: + labels: + - key: app + operation: "==" + value: my-app + afterAction: true # Fire after action completes +``` + +#### Webhook (Generic) + +```yaml +spec: + webhook: + generic-hook: + endpoint: /webhook + port: "12000" + method: POST +``` + +#### Slack + +```yaml +spec: + slack: + slash-commands: + token: + name: slack-token + key: token + signingSecret: + name: slack-token + key: signing-secret + webhook: + endpoint: /slack + port: "12000" + method: POST +``` + +#### AMQP (RabbitMQ) + +```yaml +spec: + amqp: + events: + url: amqp://guest:guest@rabbitmq:5672/ + exchangeName: my-exchange + exchangeType: topic + routingKey: "events.#" + jsonBody: true + connectionBackoff: + duration: 10s + factor: 2 + steps: 5 +``` + +#### Pub/Sub (GCP) + +```yaml +spec: + pubsub: + events: + projectID: my-gcp-project + topicProjectID: my-gcp-project + topic: my-topic + subscriptionID: argo-events-sub + credentialSecret: + name: gcp-creds + key: serviceAccountKey + jsonBody: true +``` + +#### Minio / S3-compatible + +```yaml +spec: + minio: + file-upload: + endpoint: minio.minio:9000 + bucket: + name: my-bucket + events: + - s3:ObjectCreated:* + - s3:ObjectRemoved:* + filter: + prefix: uploads/ + suffix: .csv + accessKey: + name: minio-creds + key: accessKey + secretKey: + name: minio-creds + key: secretKey + insecure: true +``` + +#### Pulsar + +```yaml +spec: + pulsar: + events: + url: pulsar://pulsar:6650 + topics: + - persistent://public/default/my-topic + type: exclusive + jsonBody: true +``` + +All supported EventSource types: webhook, github, gitlab, bitbucket, bitbucketserver, slack, sns, sqs, kafka, amqp, nats, redis, pubsub, emitter, calendar, file, resource, stripe, azure-events-hub, azure-service-bus, azure-queue-storage, minio, pulsar, generic, gerrit, sftp. + +## EventBus + +### NATS JetStream (recommended) + +```yaml +apiVersion: argoproj.io/v1alpha1 +kind: EventBus +metadata: + name: default + namespace: argo-events +spec: + jetstream: + version: latest # NATS version + replicas: 3 # 3 for HA + persistence: + storageClassName: standard + accessMode: ReadWriteOnce + volumeSize: 20Gi + streamConfig: | + maxAge: 72h # Message retention + maxBytes: 1073741824 # 1GB + replicas: 3 + settings: | + max_payload: 1048576 # 1MB max message + containerTemplate: + resources: + requests: + cpu: 100m + memory: 256Mi + metadata: + labels: {} + annotations: {} +``` + +### Kafka + +```yaml +apiVersion: argoproj.io/v1alpha1 +kind: EventBus +metadata: + name: default + namespace: argo-events +spec: + kafka: + url: kafka-broker:9092 + topic: argo-events + version: "2.6.0" + tls: + caCertSecret: + name: kafka-tls + key: ca.crt + sasl: + mechanism: PLAIN + userSecret: + name: kafka-creds + key: username + passwordSecret: + name: kafka-creds + key: password + consumerGroup: + groupName: argo-events + rebalanceStrategy: sticky +``` + +### NATS (native — legacy) + +```yaml +spec: + nats: + native: + replicas: 3 + auth: token +``` + +### NATS (exotic — external) + +```yaml +spec: + nats: + exotic: + url: nats://external-nats:4222 + clusterID: argo-events + auth: + token: + name: nats-token + key: token +``` + +## Sensor + +```yaml +apiVersion: argoproj.io/v1alpha1 +kind: Sensor +metadata: + name: + namespace: +spec: + eventBusName: default + replicas: 1 + template: + metadata: + labels: {} + spec: + containers: + - resources: {} + serviceAccountName: + + dependencies: # Events to subscribe to + - name: + eventSourceName: + eventName: + filters: {} # Optional filters (see below) + transform: {} # Optional transformation (see below) + filtersLogicalOperator: and # "and" or "or" for combining filters + + triggers: # Actions to perform + - template: + name: + conditions: # Optional: which dependencies activate this trigger + : {} # Trigger config (see below) + retryStrategy: + steps: 3 + duration: 10s + factor: 2 + rateLimit: + unit: minute + requestsPerUnit: 5 +``` + +## Sensor Dependency Filters + +### Data Filter + +Filter based on event payload fields: + +```yaml +dependencies: + - name: github-push + eventSourceName: github-webhook + eventName: push-events + filters: + data: + - path: body.ref + type: string + value: + - refs/heads/main + - refs/heads/release-* + - path: body.commits.#.modified.# + type: string + value: + - "src/**" + comparator: ">=" # >=, >, =, !=, <, <= +``` + +### Time Filter + +```yaml +filters: + time: + start: "08:00:00" + stop: "18:00:00" +``` + +### Context Filter + +Filter on CloudEvents context attributes: + +```yaml +filters: + context: + type: github.push + source: my-org/my-repo + subject: main +``` + +### Expression Filter (CEL) + +```yaml +filters: + exprs: + - expr: body.action in ["opened", "synchronize"] + fields: + - name: body.action + path: body.action + - expr: int(body.pull_request.number) > 0 + fields: + - name: body.pull_request.number + path: body.pull_request.number +``` + +### Combining Filters + +```yaml +dependencies: + - name: filtered-event + eventSourceName: my-source + eventName: my-event + filtersLogicalOperator: and # All filters must match + filters: + data: + - path: body.action + type: string + value: ["push"] + time: + start: "08:00:00" + stop: "18:00:00" + context: + type: github.push +``` + +## Event Transformation + +### Data Transformation + +Transform event data before passing to trigger: + +```yaml +dependencies: + - name: github-push + eventSourceName: github-webhook + eventName: push-events + transform: + jq: ".body | {repo: .repository.full_name, branch: .ref, sha: .after}" +``` + +Or using Lua: + +```yaml +transform: + script: | + event = obj.body + return { + repo = event.repository.full_name, + branch = event.ref, + sha = event.after + } +``` + +## Sensor Triggers + +### Argo Workflow Trigger + +```yaml +triggers: + - template: + name: run-ci + argoWorkflow: + operation: submit # submit, resubmit, retry, resume, suspend, stop, terminate + source: + resource: + apiVersion: argoproj.io/v1alpha1 + kind: Workflow + metadata: + generateName: ci- + namespace: argo + spec: + entrypoint: main + arguments: + parameters: + - name: repo + - name: sha + templates: + - name: main + container: + image: alpine + command: [echo] + args: ["Building {{workflow.parameters.repo}} at {{workflow.parameters.sha}}"] + parameters: + - src: + dependencyName: github-push + dataKey: body.repository.clone_url + dest: spec.arguments.parameters.0.value + - src: + dependencyName: github-push + dataKey: body.after + dest: spec.arguments.parameters.1.value +``` + +### HTTP Trigger + +```yaml +triggers: + - template: + name: call-webhook + http: + url: https://api.example.com/deploy + method: POST + headers: + Content-Type: application/json + Authorization: "Bearer ${secret:my-secret:token}" + payload: + - src: + dependencyName: my-dep + dataKey: body + dest: body + secureHeaders: + - name: Authorization + valueFrom: + secretKeyRef: + name: api-token + key: token + timeout: 30 +``` + +### Kubernetes Resource Trigger + +```yaml +triggers: + - template: + name: create-job + k8s: + operation: create # create, update, patch + source: + resource: + apiVersion: batch/v1 + kind: Job + metadata: + generateName: process- + namespace: default + spec: + template: + spec: + containers: + - name: worker + image: worker:latest + restartPolicy: Never + parameters: + - src: + dependencyName: my-dep + dataKey: body.data + dest: spec.template.spec.containers.0.env.0.value + operation: append +``` + +### Slack Trigger + +```yaml +triggers: + - template: + name: notify-slack + slack: + channel: deployments + message: "New deployment triggered for {{.Input.body.repository.name}}" + slackToken: + name: slack-token + key: token + parameters: + - src: + dependencyName: github-push + dataKey: body.repository.name + dest: message +``` + +### AWS Lambda Trigger + +```yaml +triggers: + - template: + name: invoke-lambda + awsLambda: + functionName: process-event + region: us-east-1 + accessKey: + name: aws-creds + key: access-key + secretKey: + name: aws-creds + key: secret-key + payload: + - src: + dependencyName: my-dep + dataKey: body + dest: event +``` + +### Log Trigger + +```yaml +triggers: + - template: + name: log-event + log: + intervalSeconds: 0 # Log every event +``` + +### NATS Trigger + +```yaml +triggers: + - template: + name: publish-nats + nats: + url: nats://nats:4222 + subject: processed-events + payload: + - src: + dependencyName: my-dep + dataKey: body + dest: data +``` + +### Kafka Trigger + +```yaml +triggers: + - template: + name: publish-kafka + kafka: + url: kafka:9092 + topic: processed-events + partition: 0 + payload: + - src: + dependencyName: my-dep + dataKey: body + dest: data +``` + +## Trigger Conditions + +Control which dependencies activate a trigger using boolean logic: + +```yaml +triggers: + - template: + name: deploy-production + conditions: "github-push && approval" # Both must fire + argoWorkflow: + # ... + + - template: + name: deploy-staging + conditions: "github-push" # Only github-push needed + argoWorkflow: + # ... +``` + +Operators: `&&` (AND), `||` (OR), `!` (NOT), parentheses for grouping. + +## Trigger Policies + +```yaml +triggers: + - template: + name: my-trigger + policy: + k8s: + backoff: + steps: 3 + duration: 5s + factor: 2 + errorOnBackoffTimeout: true + status: + allow: + - 200 + - 201 +``` + +## Cross-Namespace Event Delivery + +EventSource and Sensor must be in the same namespace, or you must configure the EventBus to allow cross-namespace subscriptions. The simplest approach: deploy EventSource, EventBus, and Sensor in the same namespace. diff --git a/gitops/argo-knowledge/references/gitops-promoter.md b/gitops/argo-knowledge/references/gitops-promoter.md new file mode 100644 index 0000000..a909f70 --- /dev/null +++ b/gitops/argo-knowledge/references/gitops-promoter.md @@ -0,0 +1,297 @@ +# GitOps Promoter + +Reference for the gitops-promoter project — a Kubernetes-native environment promotion +tool that uses Git branches, pull requests, and commit statuses to gate promotions +through environment sequences. + +**Project:** [argoproj-labs/gitops-promoter](https://github.com/argoproj-labs/gitops-promoter) +**Docs:** [gitops-promoter.readthedocs.io](https://gitops-promoter.readthedocs.io/en/latest/) +**Status:** Experimental (v0.31.0), active development, 461 stars +**API Group:** `promoter.argoproj.io/v1alpha1` + +## Core Concepts + +### How Promotion Works + +1. A hydrator (CI pipeline, Argo CD, etc.) pushes rendered manifests to a `-next` branch + (e.g., `environment/development-next`) +2. GitOps Promoter opens a PR from `-next` to the environment branch (`environment/development`) +3. Commit statuses gate the merge (security scans, health checks, tests) +4. When all checks pass and `autoMerge: true`, the PR merges automatically +5. The promoter then creates a PR for the next environment in the sequence +6. This continues until the change reaches the final environment (e.g., production) + +### Key Principles + +- **Drift-free:** promotion happens via Git merges, not file rewrites +- **PR-based gating:** every promotion is a reviewable pull request +- **Commit status gates:** both "active" (running in env) and "proposed" (pre-merge) checks +- **No fragile file changes:** the promoter doesn't modify user-facing files +- **Branch-per-environment:** each environment has its own Git branch + +## CRDs + +| Kind | apiVersion | Purpose | +|------|-----------|---------| +| PromotionStrategy | promoter.argoproj.io/v1alpha1 | Defines the environment sequence and gating rules | +| ChangeTransferPolicy | promoter.argoproj.io/v1alpha1 | Controls how changes transfer between branches (auto-generated) | +| ScmProvider | promoter.argoproj.io/v1alpha1 | SCM credentials and provider config (namespace-scoped) | +| ClusterScmProvider | promoter.argoproj.io/v1alpha1 | Cluster-scoped SCM provider config | +| GitRepository | promoter.argoproj.io/v1alpha1 | Repository reference (owner, name, SCM provider ref) | +| CommitStatus | promoter.argoproj.io/v1alpha1 | Set commit status on a SHA (pending/success/failure) | +| PullRequest | promoter.argoproj.io/v1alpha1 | Manage PR lifecycle (open/merged/closed) | +| RevertCommit | promoter.argoproj.io/v1alpha1 | Revert a specific commit | + +## Supported SCM Providers + +| Provider | Secret Key | ScmProvider Field | +|----------|-----------|------------------| +| GitHub | `githubAppPrivateKey` (GitHub App) | `github: {appID, installationID}` | +| GitLab | `token` (Access Token, Developer role, api + write_repository) | `gitlab: {}` | +| Forgejo / Codeberg | `token` (read/write repo) | `forgejo: {}` | +| Gitea | `token` (read/write repo) | `gitea: {domain}` | +| Bitbucket Cloud | `token` (repo access token, read/write repos + PRs) | `bitbucketCloud: {}` | +| Azure DevOps | `token` (PAT, Code read/write) | `azureDevOps: {organization}` | + +## PromotionStrategy + +The core CRD that defines the promotion pipeline: + +```yaml +apiVersion: promoter.argoproj.io/v1alpha1 +kind: PromotionStrategy +metadata: + name: my-app +spec: + gitRepositoryRef: + name: my-repo + + # Commit statuses that gate ALL environments + activeCommitStatuses: + - key: argocd-app-health + proposedCommitStatuses: + - key: security-scan + + environments: + - branch: environment/development + # autoMerge defaults to true + - branch: environment/staging + - branch: environment/production + autoMerge: false # require manual merge for production + activeCommitStatuses: + - key: performance-test # additional gate for prod only + proposedCommitStatuses: + - key: deployment-freeze # block during freeze windows +``` + +### Environment Fields + +| Field | Type | Default | Description | +|-------|------|---------|-------------| +| `branch` | string | required | Active branch name for this environment | +| `autoMerge` | bool | `true` | Auto-merge PRs when all checks pass | +| `activeCommitStatuses` | list | `[]` | Additional checks for running deployments (per-env) | +| `proposedCommitStatuses` | list | `[]` | Additional checks before merge (per-env) | + +### Commit Status Types + +- **Active:** checked while a commit is live in an environment. If active status fails, + the commit won't promote to the next environment. Use for: Argo CD app health, + synthetic monitoring, error rate checks. +- **Proposed:** checked before merging a PR. If proposed status fails, the PR stays + open. Use for: security scans, integration tests, deployment freezes. + +### Monorepo Support + +For monorepos with multiple apps sharing environment branches, use `activePath`: + +```yaml +spec: + activePath: apps/payments + gitRepositoryRef: + name: my-monorepo + environments: + - branch: environment/development + - branch: environment/production +``` + +Proposed branches become `environment/development-next/apps/payments`, enabling +independent promotion per app on shared active branches. + +## SCM Provider Setup + +### GitHub (Recommended) + +```yaml +apiVersion: v1 +kind: Secret +metadata: + name: github-app-secret +type: Opaque +stringData: + githubAppPrivateKey: | + -----BEGIN RSA PRIVATE KEY----- + + -----END RSA PRIVATE KEY----- +--- +apiVersion: promoter.argoproj.io/v1alpha1 +kind: ScmProvider +metadata: + name: github +spec: + secretRef: + name: github-app-secret + github: + appID: 123456 + installationID: 789012 +--- +apiVersion: promoter.argoproj.io/v1alpha1 +kind: GitRepository +metadata: + name: my-repo +spec: + scmProviderRef: + kind: ScmProvider + name: github + github: + owner: my-org + name: my-app +``` + +### GitLab + +```yaml +apiVersion: v1 +kind: Secret +metadata: + name: gitlab-token +type: Opaque +stringData: + token: +--- +apiVersion: promoter.argoproj.io/v1alpha1 +kind: ScmProvider +metadata: + name: gitlab +spec: + secretRef: + name: gitlab-token + gitlab: {} +--- +apiVersion: promoter.argoproj.io/v1alpha1 +kind: GitRepository +metadata: + name: my-repo +spec: + scmProviderRef: + kind: ScmProvider + name: gitlab + gitlab: + name: my-app + namespace: my-group/my-subgroup + projectId: 12345 +``` + +## Integration with Argo CD + +GitOps Promoter is designed to work alongside Argo CD: + +1. **Argo CD watches environment branches** — each environment branch (`environment/dev`, + `environment/prod`) is a source for an Argo CD Application or ApplicationSet +2. **Argo CD reports health via CommitStatus** — use the `argocd-app-health` commit status + key as an `activeCommitStatus` to gate promotions on Argo CD sync/health +3. **ArgoCD CommitStatus CRD** — the promoter includes an `ArgocdCommitStatus` resource + that automatically reports Argo CD Application health as a commit status + +### Example: Argo CD Application per Environment + +```yaml +apiVersion: argoproj.io/v1alpha1 +kind: ApplicationSet +metadata: + name: my-app + namespace: argocd +spec: + goTemplate: true + generators: + - list: + elements: + - env: development + branch: environment/development + namespace: my-app-dev + - env: staging + branch: environment/staging + namespace: my-app-staging + - env: production + branch: environment/production + namespace: my-app-prod + template: + metadata: + name: 'my-app-{{ .env }}' + spec: + project: default + source: + repoURL: https://github.com/my-org/my-app.git + targetRevision: '{{ .branch }}' + path: . + destination: + server: https://kubernetes.default.svc + namespace: '{{ .namespace }}' + syncPolicy: + automated: + prune: true + selfHeal: true +``` + +### Example: ArgoCD CommitStatus for Health Gating + +```yaml +apiVersion: promoter.argoproj.io/v1alpha1 +kind: ArgocdCommitStatus +metadata: + name: my-app-health +spec: + gitRepositoryRef: + name: my-repo + applicationSelector: + matchLabels: + app.kubernetes.io/name: my-app +``` + +This automatically creates CommitStatus resources reflecting the Argo CD Application +health, which the PromotionStrategy uses as an `activeCommitStatus` gate. + +## Installation + +### Manifest + +```bash +kubectl apply -f https://github.com/argoproj-labs/gitops-promoter/releases/download/v0.31.0/install.yaml +``` + +### Helm + +```bash +helm repo add gitops-promoter https://argoproj-labs.github.io/gitops-promoter +helm install gitops-promoter gitops-promoter/gitops-promoter -n gitops-promoter-system --create-namespace +``` + +### Dashboard UI + +```bash +# Download CLI from releases page +gitops-promoter dashboard +# Opens at http://localhost:8080 +``` + +## Important Notes + +- **Branch naming convention is hard-coded:** proposed branches are always + `-next` (e.g., `environment/development-next`) +- **Don't auto-delete staging branches:** disable auto-deletion on `-next` branches + or add branch protection rules for `environment/*-next` +- **Webhook recommended:** without webhooks, set lower reconciliation intervals via + `ControllerConfiguration` (`promotionStrategyRequeueDuration`, `changeTransferPolicyRequeueDuration`) +- **GitLab limitation:** can't update existing commit status descriptions without a state transition +- **All resources must be in the same namespace:** ScmProvider, GitRepository, and + PromotionStrategy must coexist in one namespace diff --git a/gitops/argo-knowledge/references/image-updater.md b/gitops/argo-knowledge/references/image-updater.md new file mode 100644 index 0000000..6314de9 --- /dev/null +++ b/gitops/argo-knowledge/references/image-updater.md @@ -0,0 +1,393 @@ +# Argo CD Image Updater Reference + +## Overview + +Argo CD Image Updater is a tool that automatically updates container image versions in Argo CD Applications. It watches container registries for new image versions and updates the Application accordingly. + +## How It Works + +1. Image Updater watches Applications with image-updater annotations +2. It queries container registries for available image tags +3. Based on the update strategy, it selects the newest/best tag +4. It updates the Application via parameter overrides (argocd method) or Git commit (git method) + +## Annotation Reference + +All annotations are placed on the Application resource's metadata. + +### Image List + +```yaml +argocd-image-updater.argoproj.io/image-list: =[:] +``` + +Examples: +```yaml +# Single image with alias +argocd-image-updater.argoproj.io/image-list: myapp=registry.example.com/my-app + +# Multiple images +argocd-image-updater.argoproj.io/image-list: >- + frontend=registry.example.com/frontend, + backend=registry.example.com/backend + +# With semver constraint +argocd-image-updater.argoproj.io/image-list: myapp=registry.example.com/my-app:~1.2 + +# Semver constraints +# ~1.2 = >=1.2.0, <1.3.0 +# ^1.2 = >=1.2.0, <2.0.0 +# 1.x = >=1.0.0, <2.0.0 +# >=1.2 = >=1.2.0 +# 1.2.x = >=1.2.0, <1.3.0 +``` + +### Update Strategy + +```yaml +argocd-image-updater.argoproj.io/.update-strategy: +``` + +| Strategy | Description | Use When | +|----------|-------------|----------| +| `semver` | Select highest semver tag matching constraint | Tags follow semantic versioning | +| `latest` | Select most recently built image | Tags are arbitrary (git SHAs, timestamps) | +| `digest` | Select most recently pushed digest | Always pull latest, track by digest | +| `name` | Select alphabetically last tag | Tags are sortable strings (e.g., dates) | + +Default: `semver` + +### Tag Filtering + +```yaml +# Allow only tags matching regex +argocd-image-updater.argoproj.io/.allow-tags: "regexp:^v[0-9]+\\.[0-9]+\\.[0-9]+$" + +# Shorthand: fn prefix for named functions +argocd-image-updater.argoproj.io/.allow-tags: "regexp:^(main|release)-[a-f0-9]{7}$" + +# Ignore specific tags +argocd-image-updater.argoproj.io/.ignore-tags: "latest,dev,nightly" + +# Ignore tags matching pattern +argocd-image-updater.argoproj.io/.ignore-tags: "regexp:^.*-rc[0-9]+$" +``` + +### Pull Secret + +```yaml +# Reference a Kubernetes Secret +argocd-image-updater.argoproj.io/.pull-secret: pullsecret:/ + +# Reference a Secret in the Argo CD namespace +argocd-image-updater.argoproj.io/.pull-secret: pullsecret:argocd/registry-creds + +# Use environment variable +argocd-image-updater.argoproj.io/.pull-secret: env:DOCKER_PASSWORD + +# Use script +argocd-image-updater.argoproj.io/.pull-secret: ext:/path/to/script +``` + +Secret format for `pullsecret`: +```yaml +apiVersion: v1 +kind: Secret +metadata: + name: registry-creds + namespace: argocd +type: kubernetes.io/dockerconfigjson +data: + .dockerconfigjson: +``` + +### Helm Integration + +Map images to Helm value parameters: + +```yaml +# Set the image name and tag via Helm values +argocd-image-updater.argoproj.io/.helm.image-name: image.repository +argocd-image-updater.argoproj.io/.helm.image-tag: image.tag + +# For charts using a single image value +argocd-image-updater.argoproj.io/.helm.image-spec: image +``` + +This translates to Helm parameter overrides: +``` +--set image.repository=registry.example.com/my-app +--set image.tag=v1.2.3 +``` + +### Kustomize Integration + +```yaml +# Override the original image name in kustomization.yaml +argocd-image-updater.argoproj.io/.kustomize.image-name: original-image-name + +# Example: if kustomization.yaml has "nginx:1.19", and you want to override: +argocd-image-updater.argoproj.io/image-list: myapp=registry.example.com/my-nginx +argocd-image-updater.argoproj.io/myapp.kustomize.image-name: nginx +``` + +### Write-Back Method + +```yaml +# Method: argocd (parameter overrides) or git (commit to repo) +argocd-image-updater.argoproj.io/write-back-method: git + +# Git target for write-back +# Format: [:] +argocd-image-updater.argoproj.io/write-back-target: kustomization +# OR +argocd-image-updater.argoproj.io/write-back-target: helmvalues:path/to/values.yaml +# OR (default for git method) +argocd-image-updater.argoproj.io/write-back-target: argocd + +# Git branch for write-back (default: same as targetRevision) +argocd-image-updater.argoproj.io/git-branch: main + +# Custom commit message template +argocd-image-updater.argoproj.io/write-back-method: "git:message=build: update {{range .Changes}}{{.Image}} to {{.NewTag}}{{end}}" +``` + +## Update Strategies Detailed + +### semver + +Selects the highest tag matching semver constraints: + +```yaml +annotations: + argocd-image-updater.argoproj.io/image-list: app=reg.io/app:~1.2 + argocd-image-updater.argoproj.io/app.update-strategy: semver +``` + +Tags must be valid semver (with optional `v` prefix). Non-semver tags are ignored. + +Constraint syntax: +- `~1.2.3` — Patch-level changes: `>=1.2.3, <1.3.0` +- `^1.2.3` — Minor-level changes: `>=1.2.3, <2.0.0` +- `1.x` — Any `1.*.*` version +- `>=1.2, <2.0` — Explicit range +- No constraint — Latest semver tag + +### latest + +Selects the most recently built image by creation timestamp: + +```yaml +annotations: + argocd-image-updater.argoproj.io/image-list: app=reg.io/app + argocd-image-updater.argoproj.io/app.update-strategy: latest + argocd-image-updater.argoproj.io/app.allow-tags: "regexp:^main-[a-f0-9]{7}$" +``` + +Requires registry API support for image metadata. Works with most registries. + +### digest + +Tracks the latest digest for a specific tag: + +```yaml +annotations: + argocd-image-updater.argoproj.io/image-list: app=reg.io/app:latest + argocd-image-updater.argoproj.io/app.update-strategy: digest +``` + +Useful when a mutable tag (like `latest`) is updated in-place. Updates the Application to use `image@sha256:...`. + +### name + +Selects the alphabetically last tag: + +```yaml +annotations: + argocd-image-updater.argoproj.io/image-list: app=reg.io/app + argocd-image-updater.argoproj.io/app.update-strategy: name + argocd-image-updater.argoproj.io/app.allow-tags: "regexp:^20[0-9]{6}-[0-9]+$" +``` + +Useful for date-based tags like `20240115-1`, `20240116-2`. + +## Registries Configuration + +Configure container registries in the `argocd-image-updater-config` ConfigMap: + +```yaml +apiVersion: v1 +kind: ConfigMap +metadata: + name: argocd-image-updater-config + namespace: argocd +data: + registries.conf: | + registries: + - name: Docker Hub + prefix: docker.io + api_url: https://registry-1.docker.io + credentials: secret:argocd/dockerhub-creds#creds + default: true + defaultns: library + + - name: GitHub Container Registry + prefix: ghcr.io + api_url: https://ghcr.io + credentials: secret:argocd/ghcr-creds#creds + + - name: ECR + prefix: 123456789.dkr.ecr.us-east-1.amazonaws.com + api_url: https://123456789.dkr.ecr.us-east-1.amazonaws.com + credentials: ext:/scripts/ecr-login.sh + credsexpire: 10h + + - name: Private Registry + prefix: registry.example.com + api_url: https://registry.example.com + credentials: pullsecret:argocd/registry-creds + insecure: false + ping: true + + log.level: info + applications_api: argocd + git.commit-signing-key: "" + git.commit-signing-method: "" + git.commit-message-template: | + build: update image(s) + + {{ range .Changes -}} + updates image {{ .Image }} tag '{{ .OldTag }}' to '{{ .NewTag }}' + {{ end -}} +``` + +## Write-Back Methods Detailed + +### argocd (Parameter Overrides) + +Default method. Updates the Application's parameter overrides directly: + +```yaml +annotations: + argocd-image-updater.argoproj.io/write-back-method: argocd +``` + +- Modifies the Application resource in Kubernetes +- Changes are visible in `app.spec.source.helm.parameters` or `app.spec.source.kustomize.images` +- No Git commits +- Changes are lost if the Application is recreated from Git +- Fastest method + +### git (Git Commit) + +Commits image updates to the Git repository: + +```yaml +annotations: + argocd-image-updater.argoproj.io/write-back-method: git + argocd-image-updater.argoproj.io/git-branch: main +``` + +**For Helm:** + +With `write-back-target: argocd` (default for git method), creates/updates `.argocd-source-.yaml`: + +```yaml +# .argocd-source-my-app.yaml (auto-generated) +helm: + parameters: + - name: image.tag + value: v1.2.4 + forcestring: true +``` + +With `write-back-target: helmvalues:`, modifies the values file directly. + +**For Kustomize:** + +With `write-back-target: kustomization`, updates the `kustomization.yaml` images section: + +```yaml +# kustomization.yaml (modified by image updater) +images: + - name: original-image + newName: registry.example.com/my-app + newTag: v1.2.4 +``` + +### Git Credentials for Write-Back + +Image updater uses the same credentials configured for the Application's source repo in Argo CD. Ensure the credentials have write (push) access. + +For SSH: +```yaml +annotations: + argocd-image-updater.argoproj.io/write-back-method: git + argocd-image-updater.argoproj.io/git-branch: main +``` + +Argo CD repo credentials must have push access. + +## Complete Example: Helm Application with Image Updater + +```yaml +apiVersion: argoproj.io/v1alpha1 +kind: Application +metadata: + name: my-app + namespace: argocd + annotations: + # Image list with semver constraint + argocd-image-updater.argoproj.io/image-list: >- + app=registry.example.com/my-app:~1.x, + sidecar=registry.example.com/sidecar:^2.0 + + # App image config + argocd-image-updater.argoproj.io/app.update-strategy: semver + argocd-image-updater.argoproj.io/app.allow-tags: "regexp:^v?[0-9]+\\.[0-9]+\\.[0-9]+$" + argocd-image-updater.argoproj.io/app.helm.image-name: app.image.repository + argocd-image-updater.argoproj.io/app.helm.image-tag: app.image.tag + argocd-image-updater.argoproj.io/app.pull-secret: pullsecret:argocd/registry-creds + + # Sidecar image config + argocd-image-updater.argoproj.io/sidecar.update-strategy: semver + argocd-image-updater.argoproj.io/sidecar.helm.image-name: sidecar.image.repository + argocd-image-updater.argoproj.io/sidecar.helm.image-tag: sidecar.image.tag + + # Write-back via Git + argocd-image-updater.argoproj.io/write-back-method: git + argocd-image-updater.argoproj.io/write-back-target: helmvalues:values.yaml + argocd-image-updater.argoproj.io/git-branch: main +spec: + project: default + source: + repoURL: https://github.com/org/k8s-configs.git + targetRevision: main + path: apps/my-app + helm: + valueFiles: + - values.yaml + destination: + server: https://kubernetes.default.svc + namespace: my-app + syncPolicy: + automated: + prune: true + selfHeal: true +``` + +## Troubleshooting + +### Check Image Updater Logs + +```bash +kubectl -n argocd logs deployment/argocd-image-updater +``` + +### Common Issues + +1. **"could not get tags"** — Registry credentials missing or invalid. Check `pull-secret` annotation and registry config. +2. **"no updates found"** — `allow-tags` regex doesn't match any tags, or semver constraint too restrictive. +3. **"git push failed"** — Repository credentials don't have write access for `write-back-method: git`. +4. **Image updater ignoring Application** — Missing `image-list` annotation, or Application not in a watched namespace. +5. **Updates not triggering sync** — With `write-back-method: argocd`, ensure automated sync is enabled. With `write-back-method: git`, ensure Argo CD detects the commit. diff --git a/gitops/argo-knowledge/references/multi-tenancy.md b/gitops/argo-knowledge/references/multi-tenancy.md new file mode 100644 index 0000000..bf610a5 --- /dev/null +++ b/gitops/argo-knowledge/references/multi-tenancy.md @@ -0,0 +1,229 @@ +# Argo CD Multi-Tenancy and Multi-Instance Patterns + +Reference for running Argo CD in multi-tenant environments — from single-instance with +AppProjects to multiple instances and the Applications-in-any-namespace feature. + +## Applications in Any Namespace (v2.5+) + +Allows Application resources to live outside the `argocd` namespace, enabling teams +to manage their own Applications declaratively in their own namespaces. + +**Docs:** [argo-cd.readthedocs.io/en/stable/operator-manual/app-any-namespace/](https://argo-cd.readthedocs.io/en/stable/operator-manual/app-any-namespace/) + +### Prerequisites + +- **Cluster-scoped Argo CD installation** — does NOT work with namespace-scoped installs +- **Switch resource tracking** — change from `label` (default) to `annotation` or `annotation+label` because + composite names (`/`) can exceed the 63-char label limit + +### Enabling + +Option 1 — Startup flag on both `argocd-server` and `argocd-application-controller`: +``` +--application-namespaces=team-a,team-b,app-* +``` + +Option 2 — ConfigMap `argocd-cmd-params-cm`: +```yaml +apiVersion: v1 +kind: ConfigMap +metadata: + name: argocd-cmd-params-cm + namespace: argocd +data: + application.namespaces: team-a, team-b, app-* +``` + +Supports shell-style wildcards (`app-*`) and regex (`/^((?!excluded).)*$/`). + +After changing, restart workloads: +```bash +kubectl rollout restart -n argocd deployment argocd-server +kubectl rollout restart -n argocd statefulset argocd-application-controller +``` + +### AppProject sourceNamespaces + +Each AppProject must explicitly allow namespaces via `.spec.sourceNamespaces`: + +```yaml +apiVersion: argoproj.io/v1alpha1 +kind: AppProject +metadata: + name: team-frontend + namespace: argocd +spec: + sourceNamespaces: + - team-frontend-apps + - team-frontend-staging + sourceRepos: + - https://github.com/org/frontend-*.git + destinations: + - server: https://kubernetes.default.svc + namespace: frontend-* +``` + +An Application in `team-frontend-apps` can only reference `team-frontend` project. +Referencing any other project is a permission violation. + +### Application Naming + +Applications are now referenced as `/`: +```bash +argocd app get team-frontend-apps/webapp +argocd app sync team-frontend-apps/webapp +``` + +For apps in `argocd` namespace, the prefix is optional (backward compatible). + +### RBAC Changes + +RBAC syntax becomes `//`: +``` +p, frontend-devs, applications, get, team-frontend/team-frontend-apps/*, allow +p, frontend-devs, applications, sync, team-frontend/team-frontend-apps/*, allow +``` + +Wildcard across namespaces: +``` +p, frontend-devs, applications, get, team-frontend/*/*, allow +``` + +### Security Constraints + +- **Never add user-controlled namespaces to the `default` AppProject** — this bypasses isolation +- **Never grant `sourceNamespaces` access to the `argocd` namespace** in a tenant AppProject +- **ApplicationSets cannot generate Applications cross-namespace** (issue #11104) +- **Extend RBAC for argocd-server** to cover tenant namespaces: + ```bash + kubectl apply -k examples/k8s-rbac/argocd-server-applications/ + ``` + +## Multi-Tenancy Patterns + +### Pattern 1: Single Instance + AppProjects (Recommended) + +One Argo CD instance, team isolation via AppProjects. + +``` +argocd/ # Single Argo CD install +├── argocd-server +├── argocd-application-controller (--application-namespaces=team-*) +├── argocd-repo-server +└── argocd-redis + +team-frontend-apps/ # Team namespace +├── Application: webapp +├── Application: api-gateway +└── (references AppProject: team-frontend) + +team-backend-apps/ # Team namespace +├── Application: payments +├── Application: users +└── (references AppProject: team-backend) +``` + +**Checklist:** +- [ ] One AppProject per team with restricted sourceRepos and destinations +- [ ] `sourceNamespaces` configured per project +- [ ] SSO groups mapped to project roles +- [ ] Cluster resource access denied by default (explicit allowlist) +- [ ] Sync windows for production namespaces +- [ ] Resource quotas per team namespace +- [ ] Network policies between tenant namespaces +- [ ] `--application-namespaces` configured with specific namespaces (not `*`) + +### Pattern 2: Multiple Instances (Hard Isolation) + +Separate Argo CD deployments per team in dedicated namespaces. + +``` +argocd-team-a/ # Team A's Argo CD +├── argocd-server +├── argocd-application-controller +├── argocd-repo-server +└── argocd-redis + +argocd-team-b/ # Team B's Argo CD +├── argocd-server +├── argocd-application-controller +├── argocd-repo-server +└── argocd-redis +``` + +**When to use:** +- Regulatory/compliance requires complete isolation +- Teams need different Argo CD versions +- Independent upgrade cycles required +- Blast radius must be zero between teams + +**Trade-offs:** +- Higher resource consumption (full Argo CD stack per team) +- Higher operational overhead (upgrades, monitoring, RBAC per instance) +- CRDs are cluster-wide — all instances share the same CRD definitions +- Each instance manages its own ConfigMaps/Secrets in its namespace + +### Comparison + +| Aspect | Single + AppProjects | Multiple Instances | +|--------|---------------------|-------------------| +| Isolation | Soft (RBAC) | Hard (process) | +| Resource cost | Low | High (N × full stack) | +| Operational overhead | Low | High | +| Upgrade coordination | One upgrade | N independent upgrades | +| Recommended for | Most organizations | Regulated / compliance | +| Max teams | 100+ | < 10 | + +## Argo CD Autopilot + +**Project:** [argoproj-labs/argocd-autopilot](https://github.com/argoproj-labs/argocd-autopilot) +**Status:** Pre-v1 (v0.4.20), under active development, not production-ready +**Supports:** Raw YAML and Kustomize (Helm not yet supported) + +Autopilot is an opinionated CLI for bootstrapping Argo CD with a self-managing GitOps +repository structure. + +### What It Does + +1. `argocd-autopilot repo bootstrap` — installs Argo CD on a cluster and creates a + self-managing Application in Git (Argo CD manages its own installation via GitOps) +2. `argocd-autopilot project create` — creates AppProjects with directory structure +3. `argocd-autopilot app create` — adds applications with base + overlay pattern +4. After bootstrap, Autopilot only needs Git access (no direct cluster access) + +### Repository Structure + +``` +bootstrap/ +├── argo-cd/ # Argo CD installation manifests +└── cluster-resources/ # Cluster-scoped resources +projects/ +├── team-a.yaml # AppProject definitions +└── team-b.yaml +apps/ +├── webapp/ +│ ├── base/ # Shared config +│ └── overlays/ +│ ├── staging/ +│ └── production/ +└── api/ + ├── base/ + └── overlays/ +``` + +### Disaster Recovery + +The primary value proposition: bootstrap a new cluster from the Git repo and all +projects, applications, and configuration are automatically restored. + +```bash +argocd-autopilot repo bootstrap \ + --repo https://github.com/org/gitops.git \ + --token $GITHUB_TOKEN +``` + +### Limitations + +- Pre-v1, not production-ready +- No Helm support yet (only raw YAML and Kustomize) +- Opinionated structure — may not fit existing repo layouts diff --git a/gitops/argo-knowledge/references/notifications.md b/gitops/argo-knowledge/references/notifications.md new file mode 100644 index 0000000..7447c7e --- /dev/null +++ b/gitops/argo-knowledge/references/notifications.md @@ -0,0 +1,505 @@ +# Argo CD Notifications Reference + +## Overview + +Argo CD Notifications is built into Argo CD (since v2.6). It monitors Application resources and sends notifications based on configurable triggers and templates. Configuration lives in two resources in the Argo CD namespace: + +- **`argocd-notifications-cm`** ConfigMap — Services, templates, triggers +- **`argocd-notifications-secret`** Secret — Tokens, passwords, API keys + +## ConfigMap Structure + +```yaml +apiVersion: v1 +kind: ConfigMap +metadata: + name: argocd-notifications-cm + namespace: argocd +data: + # Service configurations + service.: | + + + # Templates + template.: | + + + # Triggers + trigger.: | + + + # Default triggers (applied to all subscribed apps) + defaultTriggers: | + - on-sync-succeeded + - on-health-degraded + + # Global context variables + context: | + argocdUrl: https://argocd.example.com + environmentName: production +``` + +## Service Configuration + +### Slack + +```yaml +data: + service.slack: | + token: $slack-token + signingSecret: $slack-signing-secret + icon: ":argocd:" + username: ArgoCD +``` + +Secret: +```yaml +stringData: + slack-token: xoxb-XXXXXXXXX + slack-signing-secret: XXXXXXXXX +``` + +### Microsoft Teams + +```yaml +data: + service.teams: | + recipientUrls: + my-channel: https://outlook.office.com/webhook/XXXXXXX +``` + +### GitHub (Commit Status) + +```yaml +data: + service.github: | + appID: 12345 + installationID: 67890 + privateKey: $github-private-key +``` + +### Webhook + +```yaml +data: + service.webhook.: | + url: https://api.example.com/argocd-events + headers: + - name: Content-Type + value: application/json + - name: Authorization + value: "Bearer $webhook-token" + insecureSkipVerify: false +``` + +### Email (SMTP) + +```yaml +data: + service.email: | + host: smtp.example.com + port: 587 + from: argocd@example.com + username: $email-username + password: $email-password + html: true +``` + +### Grafana + +```yaml +data: + service.grafana: | + apiUrl: https://grafana.example.com/api + apiKey: $grafana-api-key +``` + +### Opsgenie + +```yaml +data: + service.opsgenie: | + apiUrl: https://api.opsgenie.com + apiKeys: + default: $opsgenie-api-key +``` + +### PagerDuty + +```yaml +data: + service.pagerduty: | + token: $pagerduty-token + from: argocd@example.com +``` + +### Rocket.Chat + +```yaml +data: + service.rocketchat: | + serverUrl: https://rocketchat.example.com + token: $rocketchat-token + userId: $rocketchat-user-id +``` + +### Google Chat + +```yaml +data: + service.googlechat: | + webhooks: + my-space: https://chat.googleapis.com/v1/spaces/XXXXXX/messages?key=XXXXX&token=XXXXX +``` + +### Matrix + +```yaml +data: + service.matrix: | + homeserverUrl: https://matrix.example.com + accessToken: $matrix-access-token + roomId: "!XXXXXXXXX:example.com" +``` + +### Telegram + +```yaml +data: + service.telegram: | + token: $telegram-bot-token +``` + +## Template Configuration + +### Basic Template + +```yaml +data: + template.app-sync-succeeded: | + message: | + Application {{.app.metadata.name}} has been successfully synced. + Revision: {{.app.status.sync.revision}} +``` + +### Slack Template with Blocks + +```yaml +data: + template.app-deployed: | + message: "{{.app.metadata.name}} deployed to {{.app.spec.destination.namespace}}" + slack: + attachments: | + [{ + "color": "#18be52", + "title": "{{.app.metadata.name}}", + "title_link": "{{.context.argocdUrl}}/applications/{{.app.metadata.name}}", + "fields": [ + {"title": "Project", "value": "{{.app.spec.project}}", "short": true}, + {"title": "Namespace", "value": "{{.app.spec.destination.namespace}}", "short": true}, + {"title": "Revision", "value": "{{.app.status.sync.revision | trunc 7}}", "short": true}, + {"title": "Status", "value": "{{.app.status.health.status}}", "short": true} + ] + }] + blocks: | + [{ + "type": "header", + "text": { + "type": "plain_text", + "text": "{{.app.metadata.name}} Deployed" + } + }, + { + "type": "section", + "fields": [ + {"type": "mrkdwn", "text": "*Project:* {{.app.spec.project}}"}, + {"type": "mrkdwn", "text": "*Status:* {{.app.status.health.status}}"} + ] + }] + groupingKey: "{{.app.metadata.name}}" + notifyBroadcast: false + deliveryPolicy: Post # Post, PostAndUpdate +``` + +### Email Template + +```yaml +data: + template.app-sync-failed-email: | + email: + subject: "ArgoCD: {{.app.metadata.name}} sync failed" + message: | +

Application {{.app.metadata.name}} sync failed

+

Project: {{.app.spec.project}}

+

Error: {{.app.status.operationState.message}}

+

View in ArgoCD

+``` + +### Teams Template + +```yaml +data: + template.app-sync-failed-teams: | + teams: + themeColor: "#FF0000" + title: "{{.app.metadata.name}} Sync Failed" + summary: "ArgoCD application {{.app.metadata.name}} sync has failed" + sections: | + [{ + "facts": [ + {"name": "Application", "value": "{{.app.metadata.name}}"}, + {"name": "Project", "value": "{{.app.spec.project}}"}, + {"name": "Error", "value": "{{.app.status.operationState.message}}"} + ] + }] + potentialAction: | + [{ + "@type": "OpenUri", + "name": "Open in ArgoCD", + "targets": [{"os": "default", "uri": "{{.context.argocdUrl}}/applications/{{.app.metadata.name}}"}] + }] +``` + +### GitHub Commit Status Template + +```yaml +data: + template.app-deployed-github: | + message: "ArgoCD deployed {{.app.metadata.name}}" + github: + repoURLPath: "{{.app.spec.source.repoURL}}" + revisionPath: "{{.app.status.operationState.syncResult.revision}}" + status: + state: success # error, failure, pending, success + label: "argocd/{{.app.metadata.name}}" + targetURL: "{{.context.argocdUrl}}/applications/{{.app.metadata.name}}" +``` + +### Webhook Template + +```yaml +data: + template.webhook-notification: | + webhook: + my-webhook: + method: POST + path: /argocd/events + body: | + { + "app": "{{.app.metadata.name}}", + "project": "{{.app.spec.project}}", + "status": "{{.app.status.health.status}}", + "sync": "{{.app.status.sync.status}}", + "revision": "{{.app.status.sync.revision}}" + } +``` + +## Template Functions + +Available in Go templates: + +| Function | Description | Example | +|----------|-------------|---------| +| `trunc N` | Truncate string | `{{.app.status.sync.revision \| trunc 7}}` | +| `now` | Current time | `{{now}}` | +| `toUpper` | Uppercase | `{{.app.metadata.name \| toUpper}}` | +| `toLower` | Lowercase | `{{.app.metadata.name \| toLower}}` | +| `replace` | String replace | `{{.app.metadata.name \| replace "-" "_"}}` | +| `default` | Default value | `{{.app.spec.project \| default "default"}}` | + +## Template Variables + +| Variable | Description | +|----------|-------------| +| `.app.metadata.name` | Application name | +| `.app.metadata.namespace` | Application namespace | +| `.app.metadata.annotations` | Application annotations | +| `.app.metadata.labels` | Application labels | +| `.app.spec.project` | Project name | +| `.app.spec.source.repoURL` | Source repository URL | +| `.app.spec.source.path` | Source path | +| `.app.spec.source.targetRevision` | Target revision | +| `.app.spec.destination.server` | Destination cluster | +| `.app.spec.destination.namespace` | Destination namespace | +| `.app.status.sync.status` | Sync status (Synced, OutOfSync) | +| `.app.status.sync.revision` | Current synced revision | +| `.app.status.health.status` | Health status (Healthy, Degraded, Progressing, etc.) | +| `.app.status.health.message` | Health message | +| `.app.status.operationState.phase` | Operation phase (Succeeded, Failed, Error, Running) | +| `.app.status.operationState.message` | Operation message (error details) | +| `.app.status.operationState.syncResult.revision` | Synced revision | +| `.context.argocdUrl` | ArgoCD server URL (from context config) | +| `.serviceType` | Notification service type | +| `.recipient` | Notification recipient | + +## Trigger Configuration + +### Basic Trigger + +```yaml +data: + trigger.on-sync-succeeded: | + - when: app.status.operationState.phase in ['Succeeded'] + oncePer: app.status.sync.revision + send: + - app-sync-succeeded + + trigger.on-sync-failed: | + - when: app.status.operationState.phase in ['Error', 'Failed'] + send: + - app-sync-failed + + trigger.on-health-degraded: | + - when: app.status.health.status == 'Degraded' + send: + - app-health-degraded + + trigger.on-sync-status-unknown: | + - when: app.status.sync.status == 'Unknown' + send: + - app-sync-status-unknown +``` + +### Trigger Fields + +| Field | Description | +|-------|-------------| +| `when` | Condition expression (see below) | +| `send` | List of template names to send | +| `oncePer` | Dedup key expression — only trigger once per unique value of this expression | + +### Trigger Condition Expressions + +Conditions use [expr](https://expr.medv.io/) syntax: + +``` +# Phase checks +app.status.operationState.phase in ['Succeeded'] +app.status.operationState.phase in ['Error', 'Failed'] + +# Health checks +app.status.health.status == 'Degraded' +app.status.health.status == 'Healthy' +app.status.health.status != 'Healthy' + +# Sync status +app.status.sync.status == 'OutOfSync' +app.status.sync.status == 'Synced' + +# Combined conditions +app.status.operationState.phase in ['Succeeded'] && app.status.health.status == 'Healthy' + +# Time-based +time.Now().Sub(time.Parse(app.status.operationState.startedAt)).Minutes() > 10 + +# Resource count +len(app.status.resources.filter(r, r.health.status == 'Degraded')) > 0 +``` + +### Default Triggers + +Applied automatically to any Application that subscribes to a service without specifying triggers: + +```yaml +data: + defaultTriggers: | + - on-sync-succeeded + - on-sync-failed + - on-health-degraded +``` + +## Subscription via Annotations + +Subscribe an Application to notifications by adding annotations: + +```yaml +metadata: + annotations: + # Format: notifications.argoproj.io/subscribe..: + notifications.argoproj.io/subscribe.on-sync-failed.slack: my-channel + notifications.argoproj.io/subscribe.on-sync-succeeded.slack: deployments + notifications.argoproj.io/subscribe.on-health-degraded.slack: alerts + notifications.argoproj.io/subscribe.on-sync-failed.email: team@example.com + notifications.argoproj.io/subscribe.on-sync-failed.teams: my-channel + notifications.argoproj.io/subscribe.on-sync-succeeded.webhook.my-webhook: "" + notifications.argoproj.io/subscribe.on-deployed.googlechat: my-space + notifications.argoproj.io/subscribe.on-sync-failed.telegram: "-1234567890" +``` + +Multiple recipients: comma-separated values. + +```yaml +notifications.argoproj.io/subscribe.on-sync-failed.slack: alerts,team-channel +``` + +## Built-in Default Triggers and Templates + +Argo CD ships with these commonly used defaults: + +| Trigger | Condition | Template | +|---------|-----------|----------| +| `on-created` | New Application created | `app-created` | +| `on-deleted` | Application deleted | `app-deleted` | +| `on-deployed` | App synced and healthy | `app-deployed` | +| `on-health-degraded` | Health degraded | `app-health-degraded` | +| `on-sync-failed` | Sync operation failed | `app-sync-failed` | +| `on-sync-running` | Sync in progress | `app-sync-running` | +| `on-sync-status-unknown` | Sync status unknown | `app-sync-status-unknown` | +| `on-sync-succeeded` | Sync succeeded | `app-sync-succeeded` | + +These are available out of the box. Custom triggers/templates in `argocd-notifications-cm` override or extend them. + +## Complete Example: Slack Notifications for Sync Failures + +```yaml +apiVersion: v1 +kind: Secret +metadata: + name: argocd-notifications-secret + namespace: argocd +type: Opaque +stringData: + slack-token: xoxb-YOUR-BOT-TOKEN +--- +apiVersion: v1 +kind: ConfigMap +metadata: + name: argocd-notifications-cm + namespace: argocd +data: + context: | + argocdUrl: https://argocd.example.com + + service.slack: | + token: $slack-token + + template.app-sync-failed: | + message: | + :x: Application {{.app.metadata.name}} sync failed! + Project: {{.app.spec.project}} + Error: {{.app.status.operationState.message}} + slack: + attachments: | + [{ + "color": "#E96D76", + "title": "{{.app.metadata.name}}", + "title_link": "{{.context.argocdUrl}}/applications/{{.app.metadata.name}}", + "fields": [ + {"title": "Project", "value": "{{.app.spec.project}}", "short": true}, + {"title": "Revision", "value": "{{.app.status.sync.revision | trunc 7}}", "short": true}, + {"title": "Error", "value": "{{.app.status.operationState.message}}", "short": false} + ] + }] + + trigger.on-sync-failed: | + - when: app.status.operationState.phase in ['Error', 'Failed'] + send: + - app-sync-failed +``` + +Application annotation: +```yaml +metadata: + annotations: + notifications.argoproj.io/subscribe.on-sync-failed.slack: alerts-channel +``` diff --git a/gitops/argo-knowledge/references/openshift.md b/gitops/argo-knowledge/references/openshift.md new file mode 100644 index 0000000..0d4f315 --- /dev/null +++ b/gitops/argo-knowledge/references/openshift.md @@ -0,0 +1,272 @@ +# OpenShift GitOps + +Reference for running the Argo ecosystem on OpenShift — covers the OpenShift GitOps +Operator, the `ArgoCD` CRD, Routes, SCCs, and platform-specific patterns. + +## OpenShift GitOps Operator + +OpenShift GitOps is Red Hat's supported distribution of Argo CD, installed via OLM +(Operator Lifecycle Manager). It manages Argo CD instances declaratively through the +`ArgoCD` custom resource. + +**Key differences from upstream Argo CD:** +- Installed via OperatorHub, not Helm or raw manifests +- Default namespace is `openshift-gitops` (not `argocd`) +- Managed via `ArgoCD` CRD (not ConfigMaps directly) +- Includes Argo CD Agent Mode (GA in OpenShift GitOps 1.19) +- Integrated with OpenShift OAuth for SSO out of the box +- Uses `Route` objects for UI access (not `Ingress`) + +### ArgoCD CRD + +The operator watches for `ArgoCD` custom resources and reconciles Argo CD components: + +```yaml +apiVersion: argoproj.io/v1beta1 +kind: ArgoCD +metadata: + name: argocd + namespace: openshift-gitops +spec: + server: + autoscale: + enabled: true + route: + enabled: true + tls: + termination: reencrypt + controller: + resources: + limits: + cpu: "2" + memory: 4Gi + requests: + cpu: 500m + memory: 1Gi + repo: + resources: + limits: + cpu: "1" + memory: 1Gi + requests: + cpu: 250m + memory: 256Mi + redis: + resources: + limits: + cpu: 500m + memory: 256Mi + ha: + enabled: false + rbac: + defaultPolicy: role:readonly + policy: | + g, cluster-admins, role:admin + g, dev-team, role:readonly + scopes: '[groups]' + resourceExclusions: | + - apiGroups: + - tekton.dev + kinds: + - TaskRun + - PipelineRun + applicationSet: + resources: + limits: + cpu: "1" + memory: 1Gi + notifications: + enabled: true +``` + +### Cluster-Scoped vs Namespace-Scoped Instances + +**Cluster-scoped** (default in `openshift-gitops` namespace): +- Can manage resources across all namespaces +- Required for "Applications in any namespace" feature +- The default instance created by the operator is cluster-scoped + +**Namespace-scoped** (team instances): +- Created in any namespace by users with appropriate RBAC +- Can only manage resources in explicitly granted namespaces +- Ideal for team-level GitOps isolation + +```yaml +apiVersion: argoproj.io/v1beta1 +kind: ArgoCD +metadata: + name: team-argocd + namespace: team-frontend +spec: + server: + route: + enabled: true + sourceNamespaces: + - team-frontend + - team-frontend-staging +``` + +To extend a namespace-scoped instance to manage other namespaces, label the target +namespace: + +```yaml +apiVersion: v1 +kind: Namespace +metadata: + name: team-frontend-staging + labels: + argocd.argoproj.io/managed-by: team-frontend +``` + +## OpenShift-Specific Patterns + +### Routes Instead of Ingress + +OpenShift uses `Route` objects. When generating Application manifests, prefer `Route` +over `Ingress` unless targeting multi-cloud: + +```yaml +apiVersion: route.openshift.io/v1 +kind: Route +metadata: + name: frontend +spec: + to: + kind: Service + name: frontend + port: + targetPort: 8080 + tls: + termination: edge + insecureEdgeTerminationPolicy: Redirect +``` + +When auditing repos, check if Applications deploy both `Route` and `Ingress` — this +is usually redundant. + +### SecurityContextConstraints (SCCs) + +OpenShift replaces PodSecurityPolicies with SCCs. When deploying Argo Rollouts or +Workflows on OpenShift: + +- **Rollout pods** need appropriate SCC. The `restricted-v2` SCC (default) works for + most containers. If Rollout pods need elevated permissions, create a dedicated + ServiceAccount and bind it to the required SCC. +- **Workflow pods** often need broader permissions (artifact upload, Docker builds). + Use a dedicated ServiceAccount with `nonroot-v2` or custom SCC. +- **AnalysisTemplate Job pods** inherit the Rollout's ServiceAccount unless overridden. + Verify the SA has the right SCC for the analysis container. + +```yaml +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRoleBinding +metadata: + name: workflow-sa-scc +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: system:openshift:scc:nonroot-v2 +subjects: + - kind: ServiceAccount + name: workflow-sa + namespace: argo +``` + +### DeploymentConfig Considerations + +Legacy OpenShift workloads may use `DeploymentConfig` instead of `Deployment`. Argo CD +can manage DeploymentConfigs, but: + +- **Health checks** — Argo CD has a built-in health check for DeploymentConfig +- **Rollouts** — Argo Rollouts does NOT support DeploymentConfig. Migrate to + `Deployment` before adopting Rollouts +- **Resource tracking** — DeploymentConfig uses a different rollout mechanism + (`oc rollout`) that can conflict with Argo CD sync + +### OAuth Integration + +OpenShift GitOps integrates with OpenShift OAuth by default. The ArgoCD CR can configure +Dex with OpenShift as an OIDC provider: + +```yaml +spec: + dex: + openShiftOAuth: true + resources: + limits: + cpu: 500m + memory: 256Mi + rbac: + defaultPolicy: role:readonly + policy: | + g, cluster-admins, role:admin + scopes: '[groups]' +``` + +Groups from OpenShift are passed through to Argo CD RBAC policies via the `scopes` +field. Map OpenShift groups to Argo CD roles in the `policy` field. + +### Resource Exclusions + +OpenShift clusters have many operator-managed resources that create noise in Argo CD. +Common exclusions: + +```yaml +spec: + resourceExclusions: | + - apiGroups: + - tekton.dev + kinds: + - TaskRun + - PipelineRun + - apiGroups: + - compliance.openshift.io + kinds: + - ComplianceCheckResult + - ComplianceRemediation + - apiGroups: + - operators.coreos.com + kinds: + - InstallPlan + - CatalogSource +``` + +### Managed Namespaces + +The OpenShift GitOps operator creates managed namespaces by default: +- `openshift-gitops` — cluster-scoped Argo CD instance +- The operator automatically grants the Argo CD instance permissions over namespaces + labeled with `argocd.argoproj.io/managed-by` + +When auditing, check that: +- Production namespaces have the `managed-by` label pointing to the correct ArgoCD instance +- No namespace is managed by multiple ArgoCD instances (causes conflicts) +- The `openshift-gitops` namespace is not used for team Applications (use separate namespaces) + +## OpenShift GitOps Agent (GA in 1.19) + +The Argo CD Agent is GA in OpenShift GitOps 1.19. OpenShift-specific considerations: + +- **Installation:** Deployed via the OpenShift GitOps operator on both hub and spoke clusters +- **Hub cluster:** Runs the principal component alongside the ArgoCD instance +- **Spoke clusters:** Run the agent component with a local Argo CD stack +- **Authentication:** Uses OpenShift certificate infrastructure for mTLS +- **Network:** Agent initiates outbound connections — works with OpenShift's default + network policies and egress controls + +See `references/agent-mode.md` for the full agent architecture. + +## Audit Checklist for OpenShift + +When auditing Argo CD repos targeting OpenShift: + +- [ ] **ArgoCD CRD used** — check for `apiVersion: argoproj.io/v1beta1 kind: ArgoCD` (operator-managed) +- [ ] **Route configured** — ArgoCD CR has `spec.server.route.enabled: true` +- [ ] **TLS termination** — Route uses `reencrypt` or `edge`, not `passthrough` without good reason +- [ ] **OAuth enabled** — `spec.dex.openShiftOAuth: true` for SSO +- [ ] **Resource exclusions** — Tekton, compliance, OLM resources excluded to reduce noise +- [ ] **No DeploymentConfig with Rollouts** — flag DeploymentConfig in repos using Argo Rollouts +- [ ] **SCC bindings for Workflow ServiceAccounts** — verify pods won't fail due to SCC restrictions +- [ ] **Managed-by labels** — target namespaces labeled for the correct ArgoCD instance +- [ ] **No conflicting management** — no namespace managed by multiple ArgoCD instances +- [ ] **RBAC maps OpenShift groups** — ArgoCD RBAC `policy` references actual OpenShift groups diff --git a/gitops/argo-knowledge/references/repo-patterns.md b/gitops/argo-knowledge/references/repo-patterns.md new file mode 100644 index 0000000..851805d --- /dev/null +++ b/gitops/argo-knowledge/references/repo-patterns.md @@ -0,0 +1,388 @@ +# GitOps Repository Patterns Reference + +## Overview + +The structure of your Git repositories determines how Argo CD discovers, manages, and deploys applications. Choosing the right pattern affects team autonomy, blast radius, scalability, and operational complexity. + +## Pattern 1: App of Apps + +A root Application manages child Applications. The root Application points to a directory containing Application manifests. + +### Structure + +``` +gitops-repo/ +├── root-app.yaml # Root Application (bootstraps everything) +└── apps/ + ├── app-a.yaml # Child Application manifest + ├── app-b.yaml # Child Application manifest + ├── app-c.yaml # Child Application manifest + └── platform/ + ├── cert-manager.yaml + ├── ingress-nginx.yaml + └── monitoring.yaml +``` + +### Root Application + +```yaml +apiVersion: argoproj.io/v1alpha1 +kind: Application +metadata: + name: root-app + namespace: argocd +spec: + project: default + source: + repoURL: https://github.com/org/gitops-repo.git + targetRevision: main + path: apps + destination: + server: https://kubernetes.default.svc + namespace: argocd # Child Applications created in argocd namespace + syncPolicy: + automated: + prune: true + selfHeal: true +``` + +### Child Application + +```yaml +# apps/app-a.yaml +apiVersion: argoproj.io/v1alpha1 +kind: Application +metadata: + name: app-a + namespace: argocd + finalizers: + - resources-finalizer.argocd.argoproj.io +spec: + project: default + source: + repoURL: https://github.com/org/app-a.git + targetRevision: main + path: deploy/overlays/production + destination: + server: https://kubernetes.default.svc + namespace: app-a + syncPolicy: + automated: + prune: true + selfHeal: true + syncOptions: + - CreateNamespace=true +``` + +### When to Use + +- Small to medium number of applications (< 50) +- Each application needs unique, hand-crafted configuration +- You want full visibility into each Application manifest in Git +- Bootstrap/initial cluster setup + +### Trade-offs + +- **Pro:** Full control over each Application; easy to understand +- **Pro:** Sync waves on child Applications control deploy order +- **Con:** Manual maintenance of each Application YAML +- **Con:** No auto-discovery of new applications +- **Con:** Can become unwieldy at scale (hundreds of files) + +## Pattern 2: ApplicationSet + +Use ApplicationSet generators to auto-create Applications from patterns. + +### Structure + +``` +gitops-repo/ +├── applicationset.yaml # ApplicationSet definition +└── apps/ + ├── app-a/ + │ ├── kustomization.yaml + │ └── deployment.yaml + ├── app-b/ + │ ├── kustomization.yaml + │ └── deployment.yaml + └── app-c/ + ├── kustomization.yaml + └── deployment.yaml +``` + +### ApplicationSet Definition + +```yaml +apiVersion: argoproj.io/v1alpha1 +kind: ApplicationSet +metadata: + name: apps + namespace: argocd +spec: + goTemplate: true + goTemplateOptions: + - missingkey=error + generators: + - git: + repoURL: https://github.com/org/gitops-repo.git + revision: main + directories: + - path: apps/* + template: + metadata: + name: '{{.path.basename}}' + spec: + project: default + source: + repoURL: https://github.com/org/gitops-repo.git + targetRevision: main + path: '{{.path.path}}' + destination: + server: https://kubernetes.default.svc + namespace: '{{.path.basename}}' + syncPolicy: + automated: + prune: true + selfHeal: true + syncOptions: + - CreateNamespace=true +``` + +### When to Use + +- Many applications following a common pattern +- Auto-discovery of new applications (add a directory = get an Application) +- Multi-cluster deployments (cluster generator) +- Dynamic environments (PR preview environments) + +### Trade-offs + +- **Pro:** Auto-discovery; add a directory and Application appears +- **Pro:** Consistent configuration across all generated Applications +- **Pro:** Progressive syncs for controlled rollout +- **Con:** Less flexibility per Application (all share a template) +- **Con:** Harder to debug template rendering issues +- **Con:** `preserveResourcesOnDeletion` should be `true` in production + +## Pattern 3: Monorepo + +Single repository containing all application manifests, organized by directory. + +### Structure + +``` +k8s-configs/ +├── base/ # Shared base manifests +│ ├── app-a/ +│ │ ├── kustomization.yaml +│ │ ├── deployment.yaml +│ │ └── service.yaml +│ └── app-b/ +│ ├── kustomization.yaml +│ ├── deployment.yaml +│ └── service.yaml +├── overlays/ +│ ├── dev/ +│ │ ├── app-a/ +│ │ │ ├── kustomization.yaml +│ │ │ └── patch-replicas.yaml +│ │ └── app-b/ +│ │ └── kustomization.yaml +│ ├── staging/ +│ │ ├── app-a/ +│ │ │ └── kustomization.yaml +│ │ └── app-b/ +│ │ └── kustomization.yaml +│ └── production/ +│ ├── app-a/ +│ │ ├── kustomization.yaml +│ │ └── patch-replicas.yaml +│ └── app-b/ +│ └── kustomization.yaml +└── platform/ # Platform services + ├── cert-manager/ + ├── ingress-nginx/ + └── monitoring/ +``` + +### Application per Environment + +```yaml +apiVersion: argoproj.io/v1alpha1 +kind: Application +metadata: + name: app-a-production + namespace: argocd +spec: + project: default + source: + repoURL: https://github.com/org/k8s-configs.git + targetRevision: main + path: overlays/production/app-a + destination: + server: https://kubernetes.default.svc + namespace: app-a +``` + +### When to Use + +- Centralized platform team managing all configurations +- Kustomize-based overlays for environment differences +- Want atomic commits across multiple applications +- Tight coupling between application configs + +### Trade-offs + +- **Pro:** Single source of truth; atomic cross-app changes +- **Pro:** Works naturally with Kustomize bases/overlays +- **Pro:** Easy to audit and review all changes in one place +- **Con:** Large repo can be slow to clone/sync +- **Con:** Broad blast radius — bad commit affects everything +- **Con:** Access control is repo-wide (use CODEOWNERS for review gating) + +## Pattern 4: Multi-Repo + +Separate Git repositories per team or service. + +### Structure + +``` +# Repo: org/app-a +app-a/ +├── src/ # Application source code +├── Dockerfile +└── deploy/ + ├── base/ + │ ├── kustomization.yaml + │ ├── deployment.yaml + │ └── service.yaml + └── overlays/ + ├── dev/ + ├── staging/ + └── production/ + +# Repo: org/app-b +app-b/ +├── src/ +├── Dockerfile +└── deploy/ + └── ... + +# Repo: org/platform-config +platform-config/ +├── cert-manager/ +├── ingress-nginx/ +└── monitoring/ +``` + +### When to Use + +- Multiple autonomous teams owning their deployment configs +- Different release cadences per service +- Teams need independent Git access control +- Microservices architecture with clear ownership boundaries + +### Trade-offs + +- **Pro:** Team autonomy; each team owns their deploy config +- **Pro:** Independent release cycles +- **Pro:** Fine-grained Git access control +- **Con:** Cross-cutting changes require multiple PRs +- **Con:** Harder to ensure consistency across repos +- **Con:** Need a registry pattern (app-of-apps or ApplicationSet) to discover repos + +## Pattern 5: Environment Branch + +Separate Git branches per environment (dev, staging, production). + +### Structure + +``` +# Branch: main (or dev) +├── apps/ +│ ├── app-a/ +│ │ ├── deployment.yaml +│ │ └── service.yaml +│ └── app-b/ +│ └── ... + +# Branch: staging +├── apps/ +│ ├── app-a/ +│ │ ├── deployment.yaml # Different image tag +│ │ └── service.yaml +│ └── app-b/ +│ └── ... + +# Branch: production +├── apps/ +│ └── ... +``` + +### Application per Branch + +```yaml +# Dev +apiVersion: argoproj.io/v1alpha1 +kind: Application +metadata: + name: app-a-dev +spec: + source: + repoURL: https://github.com/org/configs.git + targetRevision: main + path: apps/app-a + +# Production +apiVersion: argoproj.io/v1alpha1 +kind: Application +metadata: + name: app-a-prod +spec: + source: + repoURL: https://github.com/org/configs.git + targetRevision: production + path: apps/app-a +``` + +### When to Use + +- Simple promotion model (merge dev -> staging -> production) +- Teams familiar with branch-based workflows +- Small number of environments + +### Trade-offs + +- **Pro:** Simple mental model; promotion = merge/cherry-pick +- **Pro:** Easy to diff between environments (branch diff) +- **Con:** Merge conflicts between branches +- **Con:** Branch drift is hard to detect and fix +- **Con:** Doesn't scale well with many environments +- **Con:** Generally **not recommended** — prefer directory-based overlays + +## Choosing a Pattern + +| Factor | App of Apps | ApplicationSet | Monorepo | Multi-Repo | Env Branch | +|--------|-----------|---------------|---------|-----------|-----------| +| Scale (# apps) | Small-Med | Large | Med-Large | Large | Small | +| Auto-discovery | No | Yes | No | No | No | +| Team autonomy | Low | Low | Low | High | Medium | +| Consistency | Manual | Enforced | Manual | Varies | Manual | +| Multi-cluster | Manual | Built-in | Manual | Manual | Manual | +| Complexity | Low | Medium | Low | Medium | Low | +| Recommended | Bootstrap | At scale | Centralized | Distributed teams | Avoid | + +### Common Combinations + +1. **ApplicationSet + Monorepo** — Git directory generator discovers apps from a monorepo. Best for platform teams. +2. **ApplicationSet + Multi-Repo** — SCM provider generator discovers repos from a GitHub org. Best for distributed teams. +3. **App-of-Apps + Multi-Repo** — Root app bootstraps child apps pointing to team repos. Good for medium-scale. +4. **ApplicationSet (cluster generator) + Monorepo** — Deploy same apps to all clusters. Best for multi-cluster platforms. + +## Anti-Patterns + +1. **Storing secrets in Git** — Use sealed-secrets, external-secrets, or Vault CSI provider instead. +2. **One giant Application** — Break into logical units. Each Application should map to a team or service boundary. +3. **Environment branches for more than 3 environments** — Use Kustomize overlays or Helm values per environment instead. +4. **Mixing app source code and deploy manifests in the same commit flow** — Separate CI (build) from CD (deploy). Use image updater or CI-triggered Git commits. +5. **No AppProject restrictions** — Always use AppProjects to limit blast radius, especially in multi-tenant setups. diff --git a/gitops/argo-knowledge/references/rollouts.md b/gitops/argo-knowledge/references/rollouts.md new file mode 100644 index 0000000..eb86165 --- /dev/null +++ b/gitops/argo-knowledge/references/rollouts.md @@ -0,0 +1,551 @@ +# Argo Rollouts Reference + +## Overview + +Argo Rollouts extends Kubernetes with advanced deployment strategies: canary, blue-green, experimentation, and automated analysis. A Rollout replaces a Deployment and manages ReplicaSets with traffic shaping and metric-based promotion. + +## Rollout Spec Structure + +```yaml +apiVersion: argoproj.io/v1alpha1 +kind: Rollout +metadata: + name: + namespace: +spec: + replicas: + revisionHistoryLimit: # Default: 10 + selector: + matchLabels: {} + template: # Pod template (identical to Deployment) + metadata: + labels: {} + spec: + containers: [] + workloadRef: # Alternative: reference an existing Deployment + apiVersion: apps/v1 + kind: Deployment + name: + scaleDown: onsuccess # When to scale down the Deployment: never, onsuccess, progressively + strategy: + canary: {} # OR + blueGreen: {} + minReadySeconds: + progressDeadlineSeconds: # Default: 600 + progressDeadlineAbort: false # Abort instead of error on deadline + restartAt: # Schedule a restart + analysis: {} # Background analysis (runs entire lifecycle) +``` + +## Canary Strategy + +```yaml +strategy: + canary: + maxSurge: 1 # Max pods above desired during update (int or %) + maxUnavailable: 0 # Max unavailable pods during update (int or %) + canaryService: # Service routing to canary pods + stableService: # Service routing to stable pods + scaleDownDelaySeconds: 30 # Delay before scaling down old RS + scaleDownDelayRevisionLimit: 2 # Max old RS to keep before delay + abortScaleDownDelaySeconds: 30 # Delay before scaling down after abort + dynamicStableScale: false # Scale stable RS based on traffic weight + canaryMetadata: # Extra metadata for canary pods + labels: {} + annotations: {} + stableMetadata: # Extra metadata for stable pods + labels: {} + annotations: {} + steps: [] # Rollout steps (see below) + trafficRouting: {} # Traffic management (see below) + analysis: {} # Inline analysis (see below) + antiAffinity: {} # Anti-affinity between canary and stable + pingPong: # Ping-pong deployment pattern + pingService: + pongService: +``` + +### Canary Steps + +```yaml +steps: + # Set traffic weight + - setWeight: 10 + + # Pause (timed or indefinite) + - pause: { duration: 5m } # Timed pause + - pause: {} # Indefinite (manual promote required) + + # Scale canary independently of weight + - setCanaryScale: + replicas: 2 # Exact replica count + # OR + weight: 20 # Percentage of spec.replicas + matchTrafficWeight: true # Scale canary to match current traffic weight + + # Run analysis + - analysis: + templates: + - templateName: success-rate + - templateName: latency-check + clusterScope: true # Use ClusterAnalysisTemplate + args: + - name: service-name + value: my-app + - name: threshold + value: "0.95" + dryRun: # Run analysis but don't fail rollout + - metricName: experimental-metric + + # Run experiment + - experiment: + duration: 30m + templates: + - name: baseline + specRef: stable + replicas: 1 + - name: canary + specRef: canary + replicas: 1 + analyses: + - name: compare + templateName: compare-metrics + args: + - name: baseline-hash + value: '{{templates.baseline.podTemplateHash}}' + - name: canary-hash + value: '{{templates.canary.podTemplateHash}}' + + # Set header-based routing (for traffic management that supports it) + - setHeaderRoute: + name: smoke-test-header + match: + - headerName: X-Canary + headerValue: + exact: "true" + + # Set mirror traffic + - setMirrorRoute: + name: mirror-traffic + percentage: 50 + match: + - method: + exact: GET +``` + +## Blue-Green Strategy + +```yaml +strategy: + blueGreen: + activeService: # Service pointing to active (live) version + previewService: # Service pointing to preview (new) version + autoPromotionEnabled: true # Auto-promote after analysis (default: true) + autoPromotionSeconds: 60 # Wait N seconds before auto-promotion + scaleDownDelaySeconds: 30 # Wait before scaling down old version + scaleDownDelayRevisionLimit: 1 # Max old RS to keep during delay + abortScaleDownDelaySeconds: 30 # Scale-down delay after abort + antiAffinity: {} + activeMetadata: # Extra metadata for active pods + labels: {} + previewMetadata: # Extra metadata for preview pods + labels: {} + + prePromotionAnalysis: # Analysis before promoting preview to active + templates: + - templateName: smoke-test + args: + - name: preview-url + value: http://preview-svc.ns.svc.cluster.local + + postPromotionAnalysis: # Analysis after promotion + templates: + - templateName: success-rate + args: + - name: service-name + value: my-app +``` + +**Blue-green flow:** +1. New ReplicaSet created with preview pods +2. `previewService` updated to point to preview pods +3. `prePromotionAnalysis` runs against preview +4. If analysis passes (or `autoPromotionEnabled: true` + `autoPromotionSeconds` elapsed), promote +5. `activeService` switches to new pods +6. `postPromotionAnalysis` runs +7. Old ReplicaSet scales down after `scaleDownDelaySeconds` + +## Traffic Management Integrations + +### Istio + +```yaml +trafficRouting: + istio: + virtualServices: + - name: my-app-vsvc + routes: + - primary # Route name within VirtualService + destinationRule: + name: my-app-destrule + canarySubsetName: canary + stableSubsetName: stable +``` + +Argo Rollouts automatically modifies the VirtualService weight split and DestinationRule subsets. + +### AWS ALB Ingress + +```yaml +trafficRouting: + alb: + ingress: my-app-ingress # Ingress resource name + rootService: my-app-root # Root service (optional) + servicePort: 443 # Service port + annotationPrefix: alb.ingress.kubernetes.io # Custom prefix (optional) + stickinessConfig: + enabled: true + durationSeconds: 3600 +``` + +### Nginx Ingress + +```yaml +trafficRouting: + nginx: + stableIngress: my-app-ingress # Existing Ingress for stable + additionalIngressAnnotations: # Annotations for canary Ingress + canary-by-header: X-Canary + canary-by-header-value: "true" + annotationPrefix: nginx.ingress.kubernetes.io +``` + +### Traefik + +```yaml +trafficRouting: + traefik: + weightedTraefikServiceName: my-app-traefik # TraefikService name +``` + +### SMI (Service Mesh Interface) + +```yaml +trafficRouting: + smi: + rootService: my-app # Root service + trafficSplitName: my-app-split # TrafficSplit name +``` + +## AnalysisTemplate + +Defines metrics to evaluate during rollouts: + +```yaml +apiVersion: argoproj.io/v1alpha1 +kind: AnalysisTemplate +metadata: + name: + namespace: +spec: + args: + - name: + value: # Optional default + - name: # Required arg (must be provided at runtime) + + metrics: + - name: + interval: 60s # How often to run the measurement + count: 10 # Total measurements to make + initialDelay: 60s # Wait before first measurement + successCondition: result[0] >= 0.95 # CEL or Go template expression + failureCondition: result[0] < 0.8 + failureLimit: 3 # Max failures before analysis fails + inconclusiveLimit: 3 # Max inconclusive before failing + consecutiveErrorLimit: 4 # Max consecutive errors + provider: {} # Metric provider (see below) + + dryRun: # Run but don't affect rollout decision + - metricName: + + measurementRetention: # How many measurements to keep + - metricName: + limit: 10 +``` + +### Metric Providers + +#### Prometheus + +```yaml +provider: + prometheus: + address: http://prometheus.monitoring:9090 + query: | + sum(rate(http_requests_total{service="{{args.service-name}}", status=~"2.."}[5m])) + / + sum(rate(http_requests_total{service="{{args.service-name}}"}[5m])) + timeout: 30 # Query timeout in seconds + insecure: false + headers: + - key: Authorization + value: "Bearer {{args.api-token}}" +``` + +#### Datadog + +```yaml +provider: + datadog: + interval: 5m + query: | + avg:app.request.error_rate{service:{{args.service-name}}} + apiVersion: v2 +``` + +#### New Relic + +```yaml +provider: + newRelic: + profile: default + query: | + SELECT percentage(count(*), WHERE httpResponseCode < 500) + FROM Transaction WHERE appName = '{{args.service-name}}' +``` + +#### Web (HTTP) + +```yaml +provider: + web: + url: "https://api.example.com/health/{{args.service-name}}" + method: GET + headers: + - key: Authorization + value: "Bearer {{args.api-token}}" + timeoutSeconds: 30 + jsonPath: "{$.healthy}" + insecure: false +``` + +#### Job + +```yaml +provider: + job: + metadata: + labels: + app: analysis + spec: + backoffLimit: 1 + template: + spec: + containers: + - name: test + image: curlimages/curl:latest + command: [sh, -c] + args: + - | + curl -sf http://{{args.service-name}}/health + restartPolicy: Never +``` + +#### Kayenta (Automated Canary Analysis) + +```yaml +provider: + kayenta: + address: https://kayenta.example.com + application: my-app + canaryConfigName: my-canary-config + metricsAccountName: prometheus + storageAccountName: s3 + threshold: + pass: 90 + marginal: 75 + scopes: + - name: default + controlScope: + scope: baseline + experimentScope: + scope: canary +``` + +#### CloudWatch + +```yaml +provider: + cloudWatch: + interval: 5m + metricDataQueries: + - id: error_rate + metricStat: + metric: + namespace: MyApp + metricName: ErrorRate + dimensions: + - name: ServiceName + value: "{{args.service-name}}" + period: 300 + stat: Average +``` + +#### Graphite + +```yaml +provider: + graphite: + address: http://graphite.monitoring:80 + query: "summarize(stats.my-app.errors, '5min', 'sum')" +``` + +#### InfluxDB + +```yaml +provider: + influxdb: + profile: default + query: | + from(bucket: "metrics") + |> range(start: -5m) + |> filter(fn: (r) => r._measurement == "http_requests" and r.service == "{{args.service-name}}") +``` + +## ClusterAnalysisTemplate + +Cluster-scoped version of AnalysisTemplate. Referenced with `clusterScope: true`: + +```yaml +apiVersion: argoproj.io/v1alpha1 +kind: ClusterAnalysisTemplate +metadata: + name: global-success-rate +spec: + args: + - name: service-name + metrics: + - name: success-rate + # ... same as AnalysisTemplate +``` + +## AnalysisRun + +Auto-created by the Rollouts controller when an analysis step executes. Not user-created. + +```yaml +apiVersion: argoproj.io/v1alpha1 +kind: AnalysisRun +metadata: + name: my-app-6cf78b4d5-2-success-rate + namespace: my-app + ownerReferences: + - apiVersion: argoproj.io/v1alpha1 + kind: Rollout + name: my-app +status: + phase: Successful # Running, Successful, Failed, Error, Inconclusive + metricResults: + - name: success-rate + phase: Successful + measurements: + - phase: Successful + value: "0.98" + startedAt: "2024-01-01T00:00:00Z" + finishedAt: "2024-01-01T00:01:00Z" +``` + +## Experiments + +Run side-by-side comparison of versions: + +```yaml +apiVersion: argoproj.io/v1alpha1 +kind: Experiment +metadata: + name: my-experiment +spec: + duration: 1h # How long to run + progressDeadlineSeconds: 300 + templates: + - name: baseline + specRef: stable # Use stable RS template + replicas: 1 + selector: + matchLabels: + app: my-app + experiment: baseline + metadata: + labels: + app: my-app + experiment: baseline + - name: canary + specRef: canary # Use canary RS template + replicas: 1 + selector: + matchLabels: + app: my-app + experiment: canary + metadata: + labels: + app: my-app + experiment: canary + analyses: + - name: compare + templateName: compare-latencies + args: + - name: baseline-hash + value: '{{templates.baseline.podTemplateHash}}' + - name: canary-hash + value: '{{templates.canary.podTemplateHash}}' +``` + +## Rollout Commands + +```bash +# Promote a paused rollout to next step +kubectl argo rollouts promote my-app + +# Full promote (skip all remaining steps) +kubectl argo rollouts promote --full my-app + +# Abort a rollout (revert to stable) +kubectl argo rollouts abort my-app + +# Retry an aborted rollout +kubectl argo rollouts retry rollout my-app + +# Restart (trigger a new rollout with same image) +kubectl argo rollouts restart my-app + +# Set image (trigger a new rollout) +kubectl argo rollouts set image my-app my-app=registry.example.com/my-app:v2 + +# Undo (rollback to previous revision) +kubectl argo rollouts undo my-app + +# Watch status +kubectl argo rollouts status my-app --watch + +# Get detailed info +kubectl argo rollouts get rollout my-app +``` + +## Background Analysis + +Run analysis for the entire lifecycle of a rollout (not just a single step): + +```yaml +spec: + strategy: + canary: + analysis: + templates: + - templateName: continuous-success-rate + args: + - name: service-name + value: my-app + startingStep: 1 # Start after first step +``` + +The background analysis runs from `startingStep` until the rollout completes or analysis fails. If it fails, the rollout is aborted. diff --git a/gitops/argo-knowledge/references/workflows.md b/gitops/argo-knowledge/references/workflows.md new file mode 100644 index 0000000..23c65e8 --- /dev/null +++ b/gitops/argo-knowledge/references/workflows.md @@ -0,0 +1,675 @@ +# Argo Workflows Reference + +## Overview + +Argo Workflows is a container-native workflow engine for Kubernetes. Each step in a workflow runs as a container in a pod. Workflows support steps (sequential/parallel), DAGs, parameters, artifacts, retries, and scheduling. + +## Workflow Spec Structure + +```yaml +apiVersion: argoproj.io/v1alpha1 +kind: Workflow +metadata: + name: my-workflow # Fixed name (one-shot) + generateName: my-workflow- # OR generate unique name (preferred for repeated runs) + namespace: argo + labels: {} + annotations: {} +spec: + entrypoint: # Required: name of the starting template + serviceAccountName: # ServiceAccount for pod creation + automountServiceAccountToken: true + + arguments: # Workflow-level input parameters and artifacts + parameters: + - name: + value: + artifacts: [] + + templates: [] # Template definitions (see below) + + volumes: [] # Pod-level volumes (available to all templates) + volumeClaimTemplates: [] # PVC templates (created per workflow) + + ttlStrategy: # Auto-delete completed workflows + secondsAfterCompletion: 3600 # Delete 1h after completion + secondsAfterSuccess: 1800 # Delete 30m after success + secondsAfterFailure: 86400 # Keep failures for 24h + + activeDeadlineSeconds: 3600 # Max workflow runtime (ALWAYS SET THIS) + podGC: # Garbage collect completed pods + strategy: OnPodCompletion # OnPodCompletion, OnPodSuccess, OnWorkflowCompletion, OnWorkflowSuccess + deleteDelayDuration: 60s + labelSelector: {} + + retryStrategy: # Global retry strategy + limit: 3 + retryPolicy: Always # Always, OnFailure, OnError, OnTransientError + backoff: + duration: 5s + factor: 2 + maxDuration: 1m + + nodeSelector: {} + tolerations: [] + affinity: {} + securityContext: {} + parallelism: # Max parallel pods + priority: # Scheduling priority + schedulerName: + + onExit: # Exit handler template (runs after workflow completes regardless of status) + + hooks: # Lifecycle hooks + exit: + template: + running: + template: + expression: workflow.status == "Running" + + synchronization: # Concurrency control + semaphore: + configMapKeyRef: + name: my-semaphore + key: workflow-limit + mutex: + name: my-mutex + + artifactRepositoryRef: + configMap: artifact-repositories + key: default + + workflowTemplateRef: # Reference a WorkflowTemplate instead of inline templates + name: + clusterScope: false +``` + +## Template Types + +### Container Template + +Runs a single container: + +```yaml +templates: + - name: build + inputs: + parameters: + - name: image-tag + container: + image: golang:1.22 + command: [go, build, -o, /output/app, .] + workingDir: /src + env: + - name: GOPROXY + value: https://proxy.golang.org + resources: + requests: + cpu: 500m + memory: 512Mi + limits: + cpu: "2" + memory: 2Gi + volumeMounts: + - name: workdir + mountPath: /src + outputs: + artifacts: + - name: binary + path: /output/app +``` + +### Script Template + +Container with an inline script: + +```yaml +templates: + - name: gen-random + script: + image: python:3.12-slim + command: [python] + source: | + import random + result = random.randint(1, 100) + print(result) +``` + +The stdout of the script is captured as the template's output result (accessible via `{{steps..outputs.result}}` or `{{tasks..outputs.result}}`). + +### Resource Template + +Performs CRUD operations on Kubernetes resources: + +```yaml +templates: + - name: create-configmap + resource: + action: create # create, patch, apply, delete + manifest: | + apiVersion: v1 + kind: ConfigMap + metadata: + name: my-config + namespace: default + data: + key: "{{inputs.parameters.value}}" + successCondition: status.phase == Succeeded # For async resources + failureCondition: status.phase == Failed +``` + +### Suspend Template + +Pauses workflow execution: + +```yaml +templates: + - name: manual-approval + suspend: + duration: "0" # "0" = indefinite (manual resume) +``` + +Resume: `argo resume ` + +### HTTP Template + +Makes HTTP requests: + +```yaml +templates: + - name: call-api + http: + url: "https://api.example.com/deploy" + method: POST + headers: + - name: Content-Type + value: application/json + - name: Authorization + value: "Bearer {{inputs.parameters.token}}" + body: | + {"version": "{{inputs.parameters.version}}"} + successCondition: response.statusCode == 200 + timeoutSeconds: 30 +``` + +## Steps-Based Workflows + +Sequential steps, with optional parallelism within a step: + +```yaml +templates: + - name: pipeline + steps: + - - name: checkout # Step 1 (sequential) + template: git-clone + - - name: test # Step 2 (parallel within step) + template: run-tests + - name: lint + template: run-lint + - - name: build # Step 3 (sequential, after step 2 completes) + template: build-image + when: "{{steps.test.outputs.result}} == passed" +``` + +- Each outer list item is a sequential step. +- Each inner list item runs in parallel within that step. +- `when` conditionals control step execution. + +## DAG-Based Workflows + +Directed acyclic graph with explicit dependencies: + +```yaml +templates: + - name: pipeline + dag: + tasks: + - name: checkout + template: git-clone + - name: unit-test + template: run-tests + dependencies: [checkout] + - name: integration-test + template: run-integration + dependencies: [checkout] + - name: lint + template: run-lint + dependencies: [checkout] + - name: build + template: build-image + dependencies: [unit-test, lint] + arguments: + parameters: + - name: test-result + value: "{{tasks.unit-test.outputs.result}}" + - name: deploy + template: deploy-app + dependencies: [build, integration-test] +``` + +Tasks run as soon as all their dependencies are satisfied. No explicit dependency = runs immediately. + +## Parameters + +### Workflow-Level Parameters + +```yaml +spec: + arguments: + parameters: + - name: environment + value: staging # Default value + - name: image-tag # No default = required at submit time + entrypoint: main + templates: + - name: main + container: + image: my-app:{{workflow.parameters.image-tag}} + env: + - name: ENV + value: "{{workflow.parameters.environment}}" +``` + +Submit with parameters: `argo submit workflow.yaml -p environment=production -p image-tag=v2.0` + +### Template-Level Parameters + +```yaml +templates: + - name: deploy + inputs: + parameters: + - name: env + - name: replicas + default: "3" + container: + image: kubectl:latest + command: [sh, -c] + args: + - kubectl scale deployment my-app --replicas={{inputs.parameters.replicas}} -n {{inputs.parameters.env}} +``` + +### Output Parameters + +```yaml +templates: + - name: get-version + container: + image: alpine:latest + command: [sh, -c] + args: + - echo "v2.1.0" > /tmp/version.txt + outputs: + parameters: + - name: version + valueFrom: + path: /tmp/version.txt + default: "unknown" +``` + +Access: `{{steps.get-version.outputs.parameters.version}}` or `{{tasks.get-version.outputs.parameters.version}}` + +## Artifacts + +### S3 + +```yaml +templates: + - name: produce-artifact + outputs: + artifacts: + - name: report + path: /tmp/report.html + s3: + endpoint: s3.amazonaws.com + bucket: my-bucket + key: reports/{{workflow.uid}}/report.html + accessKeySecret: + name: s3-credentials + key: accessKey + secretKeySecret: + name: s3-credentials + key: secretKey +``` + +### GCS + +```yaml +artifacts: + - name: data + path: /data/output + gcs: + bucket: my-bucket + key: data/{{workflow.uid}}/output + serviceAccountKeySecret: + name: gcs-credentials + key: serviceAccountKey +``` + +### Git + +```yaml +artifacts: + - name: source + path: /src + git: + repo: https://github.com/org/repo.git + revision: main + usernameSecret: + name: git-creds + key: username + passwordSecret: + name: git-creds + key: password +``` + +### Raw + +```yaml +artifacts: + - name: config + path: /config/app.yaml + raw: + data: | + server: + port: 8080 + host: 0.0.0.0 +``` + +## Volume-Based Workflows + +### PVC Template + +```yaml +spec: + volumeClaimTemplates: + - metadata: + name: workdir + spec: + accessModes: [ReadWriteOnce] + resources: + requests: + storage: 5Gi + storageClassName: standard + + templates: + - name: step1 + container: + image: alpine + command: [sh, -c] + args: ["echo 'data' > /mnt/data.txt"] + volumeMounts: + - name: workdir + mountPath: /mnt + - name: step2 + container: + image: alpine + command: [cat, /mnt/data.txt] + volumeMounts: + - name: workdir + mountPath: /mnt +``` + +### EmptyDir + +```yaml +spec: + volumes: + - name: shared + emptyDir: {} + templates: + - name: step + container: + volumeMounts: + - name: shared + mountPath: /shared +``` + +## Resource Limits and Lifecycle + +```yaml +spec: + activeDeadlineSeconds: 7200 # Max 2 hours for entire workflow + templates: + - name: step + activeDeadlineSeconds: 600 # Max 10 min for this template + retryStrategy: + limit: 3 + retryPolicy: OnFailure # Always, OnFailure, OnError, OnTransientError + backoff: + duration: 10s + factor: 2 + maxDuration: 5m + affinity: + nodeAntiAffinity: {} # Retry on different node + timeout: 300s # Template timeout + container: + resources: + requests: + cpu: 100m + memory: 128Mi + limits: + cpu: "1" + memory: 1Gi +``` + +## WorkflowTemplate References + +### Namespace-Scoped + +```yaml +apiVersion: argoproj.io/v1alpha1 +kind: WorkflowTemplate +metadata: + name: ci-template + namespace: argo +spec: + arguments: + parameters: + - name: repo + entrypoint: main + templates: + - name: main + dag: + tasks: + - name: build + template: build-step + - name: build-step + container: + image: golang:1.22 + command: [go, build] +``` + +Reference from a Workflow: + +```yaml +apiVersion: argoproj.io/v1alpha1 +kind: Workflow +metadata: + generateName: ci-run- +spec: + workflowTemplateRef: + name: ci-template + arguments: + parameters: + - name: repo + value: https://github.com/org/app.git +``` + +Or reference individual templates: + +```yaml +templates: + - name: main + steps: + - - name: build + templateRef: + name: ci-template + template: build-step + clusterScope: false +``` + +### Cluster-Scoped + +```yaml +apiVersion: argoproj.io/v1alpha1 +kind: ClusterWorkflowTemplate +metadata: + name: shared-ci-template +spec: + templates: + - name: build + # ... +``` + +Reference with `clusterScope: true`. + +## CronWorkflow + +```yaml +apiVersion: argoproj.io/v1alpha1 +kind: CronWorkflow +metadata: + name: nightly-cleanup + namespace: argo +spec: + schedule: "0 2 * * *" # Cron expression (daily at 2am) + timezone: America/New_York # Optional timezone + concurrencyPolicy: Replace # Allow, Forbid, Replace + startingDeadlineSeconds: 60 # Max seconds after missed schedule + successfulJobsHistoryLimit: 3 # Completed workflows to keep + failedJobsHistoryLimit: 3 # Failed workflows to keep + suspend: false # Pause scheduling + + workflowSpec: # Inline workflow spec + entrypoint: cleanup + activeDeadlineSeconds: 3600 + templates: + - name: cleanup + container: + image: my-cleanup:latest + command: [/cleanup.sh] + + # OR reference a WorkflowTemplate + workflowSpec: + workflowTemplateRef: + name: cleanup-template +``` + +## Suspend/Resume, Stop/Terminate + +```bash +# Suspend a running workflow (pauses at next node boundary) +argo suspend my-workflow + +# Resume a suspended workflow +argo resume my-workflow + +# Stop a workflow (finish running nodes, then mark Failed) +argo stop my-workflow --message "stopping for maintenance" + +# Terminate a workflow (kill running nodes immediately) +argo terminate my-workflow +``` + +## Archive and Garbage Collection + +### Workflow Archive (PostgreSQL/MySQL) + +Configured in the Workflow Controller ConfigMap: + +```yaml +persistence: + archive: true + postgresql: + host: postgres.argo + port: 5432 + database: argo + tableName: argo_workflows + userNameSecret: + name: argo-postgres + key: username + passwordSecret: + name: argo-postgres + key: password +``` + +### Garbage Collection + +```yaml +# Per-workflow TTL +spec: + ttlStrategy: + secondsAfterCompletion: 3600 + secondsAfterSuccess: 600 + secondsAfterFailure: 86400 + +# Controller-level default +workflowDefaults: + spec: + ttlStrategy: + secondsAfterCompletion: 86400 + podGC: + strategy: OnPodSuccess +``` + +## Workflow-of-Workflows Pattern + +A parent workflow creates child workflows using the resource template: + +```yaml +templates: + - name: orchestrator + dag: + tasks: + - name: run-etl + template: submit-workflow + arguments: + parameters: + - name: workflow-template + value: etl-workflow + - name: run-ml + template: submit-workflow + dependencies: [run-etl] + arguments: + parameters: + - name: workflow-template + value: ml-workflow + + - name: submit-workflow + inputs: + parameters: + - name: workflow-template + resource: + action: create + manifest: | + apiVersion: argoproj.io/v1alpha1 + kind: Workflow + metadata: + generateName: child- + spec: + workflowTemplateRef: + name: {{inputs.parameters.workflow-template}} + successCondition: status.phase == Succeeded + failureCondition: status.phase in (Failed, Error) +``` + +The parent workflow waits for each child workflow to complete before continuing. + +## Common Variables + +| Variable | Description | +|----------|-------------| +| `{{workflow.name}}` | Workflow name | +| `{{workflow.namespace}}` | Workflow namespace | +| `{{workflow.uid}}` | Workflow UID | +| `{{workflow.parameters.}}` | Workflow-level parameter | +| `{{workflow.status}}` | Workflow status | +| `{{steps..outputs.result}}` | Step output (script stdout) | +| `{{steps..outputs.parameters.}}` | Step output parameter | +| `{{tasks..outputs.result}}` | DAG task output | +| `{{tasks..outputs.parameters.}}` | DAG task output parameter | +| `{{inputs.parameters.}}` | Template input parameter | +| `{{inputs.artifacts..path}}` | Input artifact mount path | +| `{{pod.name}}` | Current pod name | +| `{{retries}}` | Current retry count |