Skip to content

feat: add GitOps deployment via OpenShift GitOps (ArgoCD) #314

Merged
luis5tb merged 10 commits into
RHEcosystemAppEng:mainfrom
luis5tb:gitops-argocd-deployment
Jul 15, 2026
Merged

feat: add GitOps deployment via OpenShift GitOps (ArgoCD) #314
luis5tb merged 10 commits into
RHEcosystemAppEng:mainfrom
luis5tb:gitops-argocd-deployment

Conversation

@luis5tb

@luis5tb luis5tb commented Jul 2, 2026

Copy link
Copy Markdown
Collaborator

Summary

Add ArgoCD-managed (OpenShift GitOps) deployment for two targets: OpenShift (syncs existing Helm chart) and Google Cloud (triggers Cloud Build for Cloud Run). PR-based workflow: change image tags or config → merge → ArgoCD auto-syncs → redeployment.

Also adds comprehensive multi-instance support for deploying multiple agent instances to the same GCP project, and parameterizes dashboard names and Redis config so instances don't collide.

Google Cloud target (deploy/gitops/google-cloud/)

  • Helm chart that ArgoCD manages on OpenShift to trigger GCP deployments
  • ConfigMap maps all Cloud Build substitutions from values.yaml to cloudbuild.yaml variables
  • PostSync hook Job authenticates to GCP and runs gcloud builds submit
  • Jobs hardened: runAsNonRoot, seccompProfile, resource limits, optional git token for private repos
  • secrets.yaml.example documents the required secrets format

Setup scripts (deploy/gitops/setup/)

  • install-gitops-operator.sh — installs the OpenShift GitOps operator with health checks
  • setup-gcp-sa.sh — creates GCP service account, assigns roles, generates key, and uploads the key to GCP Secret Manager

Multi-instance deployment support

  • Comprehensive guide for deploying staging + production (or multi-tenant) instances to the same GCP project
  • GCP resource name length limits documented (SA: 30 chars, VPC connector: <21 effective chars)
  • Clarifies setup.sh is one-time infra provisioning, separate from Cloud Build pipeline
  • Same-namespace deployment option with concrete setup-gcp-sa.sh commands and matching values-override.yaml keys
  • Load balancer and DNS setup guide (static IPs first, DNS after, no chicken-and-egg)
  • Pub/Sub: shared topic per project, per-instance subscription, product-level event filtering via serviceControlServiceName
  • LB sub-resources documented as derived from loadBalancer.name prefix (16 resources across 2 services)
  • GitOps deploy SA (lightspeed-gitops) added to collisions table
  • Cross-reference added to the GitOps companion repo

Dashboard parameterization (cloudbuild.yaml)

  • deploy-dashboards step now substitutes _SERVICE_NAME into dashboard display names and labels before creating
  • Prevents instances from overwriting each other's Cloud Monitoring dashboards
  • Example: Lightspeed (lightspeed-agent-staging) — Agent Operations

Redis / rate limiting parameterization

New configurable fields across all deployment paths (deploy.sh, deploy-cloudbuild.sh, GitOps/ArgoCD):

Field Helm Value Default
Rate limit key prefix rateLimit.keyPrefix lightspeed:ratelimit
Redis URL secret name rateLimit.redisUrlSecret rate-limit-redis-url
Redis CA cert secret name rateLimit.redisCaCertSecret redis-ca-cert

Enables separate rate limit counters and separate Redis instances per deployment. Defaults unchanged — existing single-instance deployments are unaffected.

Files modified: service.yaml, marketplace-handler.yaml (placeholders), cloudbuild.yaml (substitutions + sed), deploy.sh (env vars + sed), deploy-cloudbuild.sh (forwarding), values.yaml, deployment-config.yaml, deploy-job.yaml (Helm chart), setup.sh (Redis instance name in instructions).

Cloud Run deployment updates

  • cloudbuild.yaml gains _GEMINI_MODEL, _MCP_DEBUG, _AGENT_LOGGING_DETAIL, _RATE_LIMIT_KEY_PREFIX, _REDIS_URL_SECRET, _REDIS_CA_CERT_SECRET substitutions
  • deploy.sh applies model, MCP debug, log level, and Redis config env vars to Cloud Run services
  • service.yaml and marketplace-handler.yaml use placeholders for all configurable values

Other changes

  • Fix gcp-sa-secret.yaml to respect secrets.create: false in OpenShift chart
  • OpenShift values.yaml gains geminiModel field; agent deployment template passes it through

GitOps companion repo

  • Added cross-reference to multi-instance guide in google-cloud/README.md (commit)

Test plan

  • Verify Helm chart templates render: helm template test deploy/gitops/google-cloud/ --set project.id=test
  • Verify deploy Job is triggered as a PostSync hook after ArgoCD sync
  • Verify setup scripts run cleanly on a fresh OpenShift cluster
  • End-to-end: change image tag in values-override.yaml → merge → confirm Cloud Build triggers
  • Verify multi-instance deployment with distinct service names in the same GCP project
  • Verify dashboard names are unique per instance after deployment
  • Verify Redis rate limit isolation with different rateLimit.keyPrefix values

🤖 Generated with Claude Code

@luis5tb
luis5tb force-pushed the gitops-argocd-deployment branch from fe3a1e7 to 5f519bc Compare July 2, 2026 11:13

@r2dedios r2dedios left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a few observations, but looks great :)

Comment thread deploy/gitops/google-cloud/templates/external-secret.yaml Outdated
Comment thread deploy/gitops/google-cloud/templates/deploy-job.yaml
Comment thread deploy/gitops/google-cloud/templates/deploy-job.yaml Outdated
Comment thread deploy/gitops/google-cloud/templates/external-secret.yaml Outdated
Comment thread deploy/gitops/google-cloud/templates/secret-store.yaml Outdated
Comment thread deploy/gitops/google-cloud/templates/secret-store.yaml Outdated
Comment thread deploy/gitops/google-cloud/application.yaml Outdated
Comment thread deploy/gitops/google-cloud/application.yaml Outdated
@luis5tb
luis5tb force-pushed the gitops-argocd-deployment branch 13 times, most recently from fd4573b to 2caf03e Compare July 8, 2026 15:37
@luis5tb luis5tb changed the title [WIP] feat: add GitOps deployment via OpenShift GitOps (ArgoCD) feat: add GitOps deployment via OpenShift GitOps (ArgoCD) Jul 8, 2026
r2dedios
r2dedios previously approved these changes Jul 10, 2026

@r2dedios r2dedios left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great Work! 🚀

@r2dedios r2dedios left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a couple of questions

Comment thread deploy/cloudrun/service.yaml
Comment thread deploy/gitops/google-cloud/values.yaml
@luis5tb
luis5tb force-pushed the gitops-argocd-deployment branch from 25c5592 to 1ce7396 Compare July 10, 2026 13:23
@luis5tb
luis5tb requested a review from r2dedios July 10, 2026 13:24
@yuvalk

yuvalk commented Jul 10, 2026

Copy link
Copy Markdown
Collaborator

All fixes are implemented, tested, and committed locally on gitops-argocd-deployment (not pushed). Here's the review, formatted to paste into GitHub:


Review of #314 — GitOps deployment via OpenShift GitOps (ArgoCD)

Reviewed by Claude (Anthropic) via automated code review at the requester's direction. Each comment includes a fix commit created on the PR branch — SHAs below, not yet pushed.

🔴 Critical

1. PostSync deploy Job cannot start — runAsNonRoot: true with no runAsUser, but the image runs as root
deploy/gitops/google-cloud/templates/deploy-job.yaml (pod securityContext)
google/cloud-sdk:slim ships no USER directive (verified via skopeo inspectUser field is empty, defaults to UID 0). With runAsNonRoot: true and no runAsUser set, the kubelet refuses to start the container (CreateContainerConfigError: container has runAsNonRoot and image will run as root). The entire mechanism this PR builds — the PostSync hook that triggers Cloud Build — cannot run as shipped, despite the PR description claiming "Jobs hardened: runAsNonRoot..." and checking off an end-to-end test.
Fix: ac6afc4 — pins runAsUser: 1000, runAsGroup: 1000, fsGroup: 1000.

🟠 Important

2. run.admin grant likely has no effect — gcloud builds submit doesn't pin --service-account
deploy/gitops/setup/setup-gcp-sa.sh, deploy/gitops/google-cloud/templates/deploy-job.yaml
Per Google's own docs, Cloud Build executes build steps under whichever service account is passed via --service-account — or the project's default executor SA if omitted — never under the identity that called gcloud builds submit. Since deploy-job.yaml's gcloud builds submit call doesn't pass --service-account, the roles/run.admin/roles/serviceusage.serviceUsageConsumer granted to lightspeed-gitops in setup-gcp-sa.sh don't apply to the identity actually running gcloud run services replace. The pipeline was silently depending on the project's default executor SA already having sufficient permissions from elsewhere.
Fix: c5f5160 — pins --service-account on the build submission and adds the required self-actAs grant, updates docs.

3. IAM role grants silently swallow failures
deploy/gitops/setup/setup-gcp-sa.sh
Every add-iam-policy-binding call ends in --quiet || true. A failed grant (insufficient caller permissions, API not enabled) produces no warning, and the script's closing summary unconditionally declares "Setup Complete" and lists all roles as granted regardless of what actually succeeded.
Fix: c5f5160 (same commit as #2) — tracks failures explicitly, exits non-zero with a clear summary instead.

4. GEMINI_MODEL default silently changed from gemini-2.5-flash to gemini-3.5-flash, undocumented
cloudbuild.yaml, deploy/cloudrun/deploy.sh, deploy/gitops/google-cloud/values.yaml
Three files introduce gemini-3.5-flash as the new deploy-time default, with no mention in the PR description. src/lightspeed_agent/config/settings.py:75 and the docs (CLAUDE.md, docs/configuration.md) still document gemini-2.5-flash as the default — this leaves docs/code out of sync and is an undocumented behavior change riding along in a "GitOps config" PR.
Fix: d8d1ac2 — reverts the deploy-time default to gemini-2.5-flash, consistent with app config and docs. If a model bump is intended, it should be its own PR that also updates settings.py and the docs.

5. Git PAT exposure is broader than the existing warning acknowledges
deploy/gitops/google-cloud/templates/deploy-job.yaml
The in-container git clone embeds the token directly in the URL, exposing it via process arguments (ps//proc/*/cmdline) to anything else with visibility into the pod. Separately, the injected Cloud Build clone step embeds the token in step args, which persist in the Build resource itself (visible via gcloud builds describe/Console/API to anyone with cloudbuild.builds.viewer) — a broader and longer-lived exposure than the existing comment's "Cloud Logging access" framing suggests.
Fix: 6ea500c — switches the in-container clone to a credential helper (token never on that process's argv), and rewrites the warning to correctly describe the residual Cloud-Build-side exposure with the concrete Secret-Manager-based fix, flagged as follow-up work since it needs new GCP resource provisioning to implement and validate properly.

6. Unused RBAC grant on the deploy Job's ServiceAccount
deploy/gitops/google-cloud/templates/serviceaccount.yaml
The Role grants get/list/watch on all ConfigMaps in the namespace, but the deploy Job's container command never calls the Kubernetes API — its config is injected via envFrom, which the kubelet resolves at pod admission using its own credentials, not the pod's ServiceAccount token. This is a dead permission that gives anything with access to the pod's token read access to every ConfigMap in a shared namespace.
Fix: 194c1a7 — removes the Role/RoleBinding, keeps the bare ServiceAccount.

7. New deploy.sh branching logic has no test coverage
deploy/cloudrun/deploy.sh, tests/shell/
CLAUDE.md explicitly requires test updates for deploy.sh changes. The new MCP_DEBUG if/else (insert vs. delete the --debug placeholder via sed) shipped with zero new tests — existing coverage only calls deploy_agent generically via an unrelated LB test, so CI's function-coverage check didn't catch the gap.
Fix: ba73f65 — adds two bats tests asserting both branches; verified passing locally (25/25 tests green).

🟡 Suggestions

8. Helm Job name has no truncation headroom for its appended suffix
deploy/gitops/google-cloud/templates/_helpers.tpl
chart.fullname truncates to the full 63-char K8s label limit, but deploy-job.yaml and deployment-config.yaml append further suffixes (-deploy-{revision}, -deploy-config) on top of it. For sufficiently long release names — plausible given this PR's own multi-instance/multi-environment guidance — the combined name exceeds 63 chars and breaks pod creation (the auto-generated job-name label). Not triggered by the PR's own documented example names, but a real latent edge case.
Fix: 5ddc8d5 — truncates to 49 chars, verified a deliberately long release name now renders a 58-char Job name.

9. GIT_REPO isn't regex-validated before being spliced into injected YAML
deploy/gitops/google-cloud/templates/deploy-job.yaml
GIT_BRANCH and _SCAN_SEVERITY are validated before being spliced into single-quoted injected YAML, but GIT_REPO (and the token-bearing CLONE_URL derived from it) wasn't — inconsistent with the defensive pattern already used in the same file. Low real-world risk since it's Helm-values-controlled.
Fix: 50898c5 — adds the same validation pattern.

10. PR description's test-plan checklist references a path that doesn't exist in this repo
oc apply -f deploy/gitops/openshift/application.yaml is checked off, but per this PR's own new CLAUDE.md section, Application CRs live only in the separate GitOps repo — deploy/gitops/openshift/ doesn't exist here. Likely just a copy-paste artifact in the description, not a code issue — no fix commit for this one.

✅ Strengths

  • Container hardening on the deploy Job is otherwise well composed: allowPrivilegeEscalation: false, capabilities: drop: [ALL], readOnlyRootFilesystem: true, seccompProfile: RuntimeDefault, and HOME=/tmp + the tmp emptyDir correctly gives gcloud/git a writable home under a read-only rootfs.
  • setup-gcp-sa.sh handles the SA key file carefully: umask 077 + chmod 600, deletes it only after confirming the K8s secret exists, and scopes iam.serviceAccountUser to the specific Cloud Run runtime SA rather than project-wide.
  • _SCAN_SEVERITY and (now) GIT_BRANCH/GIT_REPO injection into the generated cloudbuild.yaml is defensively validated.
  • ${VAR}/$${VAR} Cloud Build escaping is correct end-to-end for the new GEMINI_MODEL/MCP_DEBUG_FLAG substitutions.
  • .gitignore correctly excludes deploy/gitops/google-cloud/secrets.yaml from being committed.

Verdict: Request Changes

The Critical finding (#1) means the deploy Job as shipped cannot run at all — this alone blocks merge. #2#7 are real correctness/security gaps worth fixing before this becomes the standard deployment path. All fixes above are implemented as separate commits on this branch, each verified (helm template/helm lint, shellcheck, bash -n, and the full bats suite — 25/25 passing).


@yuvalk

yuvalk commented Jul 10, 2026

Copy link
Copy Markdown
Collaborator

luis5tb#8

@IlonaShishov

Copy link
Copy Markdown
Collaborator

PR summary need an update :)

@luis5tb
luis5tb force-pushed the gitops-argocd-deployment branch from 1ce7396 to e02f82a Compare July 13, 2026 05:59
Comment thread deploy/gitops/google-cloud/values.yaml Outdated
@luis5tb
luis5tb force-pushed the gitops-argocd-deployment branch 6 times, most recently from 5242be4 to b787439 Compare July 13, 2026 15:56
@luis5tb
luis5tb force-pushed the gitops-argocd-deployment branch 6 times, most recently from fa8de9e to addf255 Compare July 14, 2026 15:25
luis5tb and others added 9 commits July 14, 2026 17:57
Add ArgoCD-managed (OpenShift GitOps) deployment for two targets:
- OpenShift: ArgoCD Application syncs the existing Helm chart
- Google Cloud: Helm chart where PostSync hook triggers Cloud Build

Includes:
- Helm chart with security-hardened deploy Job (runAsNonRoot,
  readOnlyRootFilesystem, seccompProfile, pinned image digest)
- Setup scripts (ArgoCD operator install, GCP SA creation)
- Input validation for SCAN_SEVERITY and GIT_BRANCH
- Git token handling via Secret Manager availableSecrets
- Base GitOps documentation
Parameterize all deployment-configurable values for multi-instance:
- Agent model, MCP debug/timeout, logging detail
- Dashboard names with service_name substitution
- Redis config (key prefix, URL secret, CA cert secret)
- Database and DCR secret names
- MCP timeout and SSE read timeout

All values threaded consistently through cloudbuild.yaml, deploy.sh,
deploy-cloudbuild.sh, service.yaml, marketplace-handler.yaml, and the
GitOps Helm chart (values, ConfigMap, deploy Job).
Automate creation of Cloud SQL instance, Redis/Memorystore instance,
VPC connector, and database secret population. Adds wait_for_sa()
with exponential backoff to fix IAM propagation race condition.
- Multi-instance deployment guide (staging + prod on same GCP project)
- GCP resource name collision table and length constraints
- Shared vs separate infrastructure decision guide
- Cloud Build SA roles reference table
- setup.sh to values-override.yaml field mapping table
- Cross-cluster deployment with credential rotation
- Load balancer and DNS setup ordering
GCP Pub/Sub notifications often omit the product field. When filtering
is configured, fetch it from the Procurement API via the entitlement
ID. 403 responses indicate the SA lacks permission for that product
(belongs to a different instance), so the event is safely skipped.

Includes tests for matching, non-matching, and 403 scenarios.
- deploy-cloudbuild.sh: add forwarding for GEMINI_MODEL,
  AGENT_LOGGING_DETAIL, and MCP_DEBUG substitutions
- setup.sh: use stdlib-only DCR key generation (no cryptography dep),
  add VPC_NETWORK env var for VPC connector creation
- values.yaml: shorter VPC connector default (ls-redis-conn), remove
  unused deploy.namespace, document handler tag behavior

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Use fail-closed: skip events when product can't be determined
  (Pub/Sub retries handle transient API failures)
- Wrap credentials.refresh() in asyncio.to_thread() to avoid
  blocking the event loop
- Add explicit _NOT_AUTHORIZED check with clear log message
- Use Settings type annotation instead of Any
- Remove redundant function-level imports
- Add tests for fail-closed path and _fetch_entitlement_product
  unit tests covering all code paths

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Thread _OTEL_SERVICE_NAME through the GitOps Helm chart to support
PR RHEcosystemAppEng#328's configurable OTel service name for dashboard filtering
and metrics scoping in multi-instance deployments.

- GitOps: falls back to services.agent.name when otelServiceName
  is not explicitly set, so multi-instance deployments get unique
  OTel names automatically
- deploy.sh: add OTEL_SERVICE_NAME env var with SERVICE_NAME fallback

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@luis5tb
luis5tb force-pushed the gitops-argocd-deployment branch from addf255 to db85e1a Compare July 14, 2026 15:58
IlonaShishov
IlonaShishov previously approved these changes Jul 15, 2026

@IlonaShishov IlonaShishov left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some minor comments to address (optional)
LGTM

Comment thread deploy/openshift/values.yaml
Comment thread deploy/gitops/google-cloud/values.yaml
Comment thread deploy/openshift/values.yaml
The old default 'lightspeed-redis-conn' (23 effective chars) exceeded
the documented GCP limit of <21 characters for VPC connector names.
Changed to 'ls-redis-conn' (16 effective chars) across all scripts,
config, and documentation to ensure compliance and leave room for
environment prefixes.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@luis5tb
luis5tb merged commit 857a8c9 into RHEcosystemAppEng:main Jul 15, 2026
15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants