[firestartr-bootstrap] Add non-SaaS dedicated deployment path (initial support: Azure)

[jalvarezit]

ADR 001: Non-SaaS Dedicated Deployment Path — Azure Bootstrap and Decoupled Secret Store Machinery

• Status: Accepted
• Date: July 24, 2026
• Deciders: Platform Engineering Team

---

1. Context and Problem Statement

Currently, the Firestartr bootstrap process (implemented via Dagger) is heavily tied to AWS SaaS deployments.

1. Explicit Unsupported State: Section 3.2 of the bootstrap README.md explicitly flags Azure support as (currently not supported).
2. Hardcoded AWS Secret Store: In kubernetes.go, the bootstrap pipeline directly mounts aws_secretstore.yaml, preventing deployment against non-AWS cloud providers.
3. AWS Parameter Store Assumption: Secret resolution assumes hierarchical paths (/firestartr/<customer>/...), which are invalid in Azure Key Vault (only [a-zA-Z0-9-] allowed).
4. SaaS-only Repo Targeting: Commands like CmdPushDeployment and CmdPushArgo unconditionally target shared SaaS repositories (firestartr-<env>/app-firestartr, firestartr-<env>/state-argocd, firestartr-<env>/state-sys-services) rather than the customer's own GitHub org.
5. Bare Cluster Assumption Gap: The pipeline assumes a pre-provisioned cluster (ESO already running, ingress in place). Dedicated deployments start from a truly bare AKS cluster.

To enable bootstrapping a dedicated Azure deployment, we must decouple the secret store implementation, introduce a deployment-mode concept, handle Key Vault structural differences, and make all cluster service installation and repo targeting conditional on the deployment model.

---

2. Decision Drivers

• Multi-Cloud Support: Enable Firestartr pipelines against Azure subscriptions.
• Dedicated Deployment Support: Allow customers to run Firestartr on their own infrastructure (not the shared SaaS platform).
• Provider Parity: Secrets, Terraform state backends, and GitHub integrations must work identically regardless of underlying cloud.
• Extensibility: Eliminate hardcoded cloud provider and SaaS-org references so future providers and topologies can be added cleanly.
• Bare Cluster Bootstrap: The pipeline must be self-sufficient — installing all required cluster services from scratch.

---

3. Deployment Model: SaaS vs Dedicated

A new field deploymentMode is introduced in Bootstrapfile.yaml:

deploymentMode: saas | dedicated

Invariant: SaaS deployments are always on AWS. Dedicated deployments are always on non-AWS providers (currently Azure). This invariant is enforced by convention, not code.

Behavioral differences

Concern	SaaS (AWS)	Dedicated (Azure)
Repo targeting	firestartr-<env>/<repo>	<bootstrap.Org>/<repo>
Cluster services	ESO only (pre-provisioned infra)	All five services installed by pipeline
Webhook URL	<customer>.events[.<env>].firestartr.dev	<customer>.events.<domain>
Secret refs	AWS Parameter Store paths	Azure Key Vault dash-delimited names
Deployment values	values.tmpl (AWS/ALB/IAM)	azure_values.tmpl (Azure/nginx/MI)
Env concept	Present	Absent — always treated as production

---

4. Azure Requirements

4.1 Bootstrapfile.yaml additions

deploymentMode: dedicated
domain: "azure-pre.firestartr.dev"   # fully-qualified base domain, encodes env if needed

The webhook URL for dedicated mode is computed as:

https://<customer>.events.<domain>

No env suffix is appended — dedicated deployments have no environment concept.

4.2 Credentialsfile.yaml schema for Azure

cloudProvider:
  name: azure
  config:
    tenant_id: "<azure-tenant-id>"
    subscription_id: "<azure-subscription-id>"
    # Runtime identity — firestartr-mi (User-Assigned Managed Identity)
    # Used in deployed AKS state via Workload Identity (no secret)
    client_id: "<firestartr-mi-client-id>"
    # Bootstrap identity — dedicated App Registration (Service Principal)
    # Used only during bootstrap in the local kind cluster; delete SP after bootstrap
    bootstrap_client_id: "<bootstrap-sp-application-id>"
    bootstrap_client_secret: "<bootstrap-sp-client-secret>"
    storage_account_name: "tfstatedev<customer>"
    container_name: "tfstate"
    resource_group_name: "rg-firestartr-dev"   # also used for DNS zone resource group
    key_vault_name: "firestartr-kv"
  source: hashicorp/azurerm
  type: azurerm
  version: "~> 3.0"
github:
  prefappBotPat: "<prefapp-bot-pat>"
  operatorPat: "<operator-pat>"

4.3 Two-phase credential split

Bootstrap uses two separate Azure identities with distinct client_id values. They cannot be the same identity because:

• A Managed Identity has no client secret — it only supports Workload Identity federation (requires an OIDC issuer) or direct attachment to Azure compute. Neither is available on a local kind cluster.
• A Service Principal (App Registration) supports client secrets and can authenticate from anywhere, including a local kind cluster.
• Neither ESO's Azure KV provider nor the Terraform azurerm provider support raw Bearer tokens (e.g. from az account get-access-token) as a substitute for SP credentials. ESO has no accessToken auth type; Terraform has no access_token / ARM_ACCESS_TOKEN field.

Phase	Identity	Auth mechanism	Where used
Bootstrap-time (kind/local cluster)	Bootstrap App Registration	Service Principal — bootstrap_client_id + bootstrap_client_secret	azure_secret.tmpl, azure_secretstore.tmpl, initial_crs.tmpl (FirestartrProviderConfig)
Deployed state (state-sys-services)	firestartr-mi	Workload Identity (OIDC) — client_id, no secret	azure_values.tmpl, all cluster service Helm values

The bootstrap App Registration is a one-time throwaway credential: it is created before bootstrap, used exclusively during that single run, and the entire App Registration must be deleted afterwards.

4.4 Key Vault secret naming convention

Azure Key Vault secret names use [a-zA-Z0-9-] only (no forward slashes):

• fs-pem, fs-app-id, fs-<org>-installation-id
• fs-admin-pem, fs-admin-app-id, fs-admin-<org>-installation-id
• fs-argocd-pem, fs-argocd-app-id, fs-argocd-<org>-installation-id
• fs-state-pem, fs-state-app-id, fs-state-<org>-installation-id
• fs-checks-pem, fs-checks-app-id, fs-checks-<org>-installation-id
• fs-import-pem, fs-import-app-id, fs-import-<org>-installation-id
• github-webhook-secret, prefapp-bot-pat, firestartr-cli-version

---

5. Proposed Architectural Changes

A. deploymentMode field and routing

Bootstrapfile.yaml gains deploymentMode: saas | dedicated. All branching in the pipeline is gated on this field. Helper: isDedicatedDeployment() bool.

B. Repo targeting (dedicated mode)

All functions that hardcode firestartr-<env> as the GitHub org are made deployment-mode-aware:

File	Current	Dedicated
deployment.go	firestartr-<env>/app-firestartr	<bootstrap.Org>/state-sys-services
argocd.go	firestartr-<env>/state-argocd	<bootstrap.Org>/state-argocd
argocd_config_secrets.go	firestartr-<env>/state-sys-services	<bootstrap.Org>/state-sys-services
validations.go	firestartr-<env> (webhook org check)	<bootstrap.Org>

state-sys-services and state-argocd are auto-injected as components when deploymentMode: dedicated, so the pipeline creates them without requiring the operator to declare them in Bootstrapfile.yaml.

C. state-sys-services file structure (dedicated)

The deployment platform is named firestartr-aks (hardcoded constant). The bootstrap pipeline pushes initial configuration for all sys-services:

firestartr-aks/
  firestartr.yaml                          # tenant/release descriptor (from tenant.tmpl)
  firestartr/values.yaml                   # Helm values (from azure_values.tmpl)
  argo-configuration-secrets.yaml          # ArgoCD secrets release descriptor
  argo-configuration-secrets/values.yaml   # ArgoCD secrets Helm values
  argocd.yaml
  argocd/values.yaml
  nginx.yaml
  nginx/values.yaml
  cert-manager.yaml
  cert-manager/values.yaml
  external-dns.yaml
  external-dns/values.yaml

D. Cluster services installation (dedicated only)

InstallHelmAndExternalSecrets is renamed InstallClusterServices. For dedicated mode it installs all five services via Helm:

1. External Secrets Operator — unchanged from current implementation
2. nginx ingress controller — standard nginx Helm chart
3. cert-manager — standard cert-manager Helm chart
4. external-dns — Azure DNS provider; authenticates via Workload Identity (firestartr-mi) to manage records in the delegated zone at resource_group_name
5. ArgoCD — standard ArgoCD Helm chart

For saas mode the function installs ESO only (no change from current behavior).

E. Secret store injection (kubernetes.go)

Already partially implemented. Bootstrap-time SecretStore for Azure uses the bootstrap App Registration credentials (not firestartr-mi):

• azure_secret.tmpl — creates azure-creds Kubernetes Secret with bootstrapClientId, bootstrapClientSecret, tenantId
• azure_secretstore.tmpl — ESO SecretStore using authSecretRef pointing to azure-creds

F. New azure_values.tmpl

New Helm values template for the firestartr-controller chart (dedicated mode). Uses client_id (firestartr-mi) for Workload Identity — not the bootstrap SP. Key differences from values.tmpl:

# Auth: Workload Identity (no clientSecret, no roleArn)
general:
  auth:
    provider: azure
    config:
      azureIdentity:
        clientId: "{{ .CloudProvider.Config.ClientId }}"
        tenantId: "{{ .CloudProvider.Config.TenantId }}"

externalSecrets:
  auth:
    provider: azure
    config:
      azureIdentity:
        clientId: "{{ .CloudProvider.Config.ClientId }}"
        tenantId: "{{ .CloudProvider.Config.TenantId }}"
        vaultUrl: "https://{{ .CloudProvider.Config.KeyVaultName }}.vault.azure.net"

# Ingress: nginx (no ALB annotations)
argoEvents:
  ingress:
    enabled: true
    annotations:
      kubernetes.io/ingress.class: nginx

# Terraform backend: Azure Blob Storage with OIDC (no clientSecret)
providerConfigs:
  tfstate-{{ .Customer }}:
    config: |
      {
        "storage_account_name": "{{ .CloudProvider.Config.StorageAccountName }}",
        "container_name": "{{ .CloudProvider.Config.ContainerName }}",
        "resource_group_name": "{{ .CloudProvider.Config.ResourceGroupName }}",
        "subscription_id": "{{ .CloudProvider.Config.SubscriptionId }}",
        "tenant_id": "{{ .CloudProvider.Config.TenantId }}",
        "client_id": "{{ .CloudProvider.Config.ClientId }}",
        "use_oidc": true
      }
    type: azurerm
    source: "hashicorp/azurerm"
    version: "~> 3.0"

# Secret store key refs: Azure KV dash-delimited names
controller:
  secretStoreKeysRefs:
    githubAppPem: "fs-pem"
    githubAppId: "fs-app-id"

# Datadog: disabled for dedicated
controller:
  datadog:
    enabled: false

G. RenderDeploymentAzure (dedicated mode)

A new RenderDeploymentAzure function is introduced. Unlike RenderDeployment:

• Does not call ValidateSTSCredentials (no AWS STS)
• Renders azure_values.tmpl instead of values.tmpl
• Targets firestartr-aks/firestartr.yaml and firestartr-aks/firestartr/values.yaml
• PR targets <bootstrap.Org>/state-sys-services at repo root

CmdPushDeployment dispatches to RenderDeploymentAzure when isDedicatedDeployment().

H. ArgoCD configuration (dedicated mode)

argocd_config_secrets.go changes for dedicated:

• Secret refs use Azure KV names: fs-argocd-app-id, fs-argocd-<org>-installation-id, fs-argocd-pem
• File patched: firestartr-aks/argo-configuration-secrets/values.yaml
• Repo: <bootstrap.Org>/state-sys-services

argocd.go changes for dedicated:

• ArgoCD Application namespace: argocd
• Repo cloned/PR target: <bootstrap.Org>/state-argocd

---

6. Work Breakdown

Task	File / Component	Description
1. deploymentMode field	types.go, main.go	Add DeploymentMode string to Bootstrap struct; add isDedicatedDeployment() helper
2. domain field	types.go, main.go	Add Domain string to Bootstrap struct; update webhook URL calculation for dedicated mode
3. Split credential fields	types.go, schemas/credentials-file.json	Add BootstrapClientId + BootstrapClientSecret fields to ConfigProvider; ClientId becomes the runtime MI client ID; update JSON schema validation
4. Rename + expand cluster services	operator.go	Rename InstallHelmAndExternalSecrets → InstallClusterServices; add nginx, cert-manager, external-dns (WI), ArgoCD for dedicated path
5. azure_values.tmpl	templates/deployment/	New Azure Helm values template (uses ClientId for MI/WI)
6. azure_tenant.tmpl	templates/deployment/	New Azure tenant/release descriptor template
7. state-sys-services templates	templates/deployment/	Templates for all sys-service release descriptors and values under firestartr-aks/
8. RenderDeploymentAzure	deployment.go	New render function; dispatch in CmdPushDeployment
9. Repo targeting	argocd.go, argocd_config_secrets.go, deployment.go, validations.go	Branch on isDedicatedDeployment() to use <bootstrap.Org> instead of firestartr-<env>
10. ArgoCD secret refs	argocd_config_secrets.go	Azure KV names for dedicated path
11. ArgoCD namespace	argocd.go	Use argocd namespace for dedicated
12. Auto-inject repos	main.go or import.go	Auto-add state-sys-services and state-argocd to components when dedicated
13. initial_crs.tmpl bootstrap SP	templates/initial_crs.tmpl	In dedicated mode, the FirestartrProviderConfig for the cloud backend must use BootstrapClientId + BootstrapClientSecret (SP credentials) for Terraform state auth in the kind cluster, not the MI ClientId
14. Documentation	README.md	Add Azure dedicated setup section; remove "(currently not supported)" from section 3.2; document bootstrap SP creation and deletion steps

---

7. Consequences

Positive

• Dedicated Azure Deployments Unlocked: Full bootstrap lifecycle works on a bare AKS cluster.
• Clean Topology Separation: deploymentMode makes SaaS vs dedicated explicit and easy to extend.
• Self-Sufficient Pipeline: A single bootstrap run takes a bare cluster to fully operational state.
• Workload Identity: No long-lived credentials in the deployed state.

Negative / Risks

• Pre-flight Requirements: Operator must create firestartr-mi (Managed Identity), create a bootstrap App Registration (Service Principal), configure OIDC federation on the AKS cluster for firestartr-mi, grant the bootstrap SP Key Vault Secrets Officer and Storage Blob Data Contributor on the relevant resources, and populate Azure Key Vault secrets before running bootstrap.
• Bootstrap SP Lifecycle: The bootstrap App Registration is a one-time credential. The entire App Registration must be deleted after bootstrap completes — not just the client secret. Leaving it in place is an unnecessary persistent attack surface.
• firestartr-aks is hardcoded: Renaming the deployment platform requires a code change until this is made configurable.