feat(keycloak): run Keycloak on Flatcar

aws/keycloak/README.md

@@ -1,55 +1,61 @@
 # aws/keycloak
 
-OpenTofu stack for the first AWS-hosted Keycloak instance in the `lab`
-account.
+OpenTofu stack for the AWS-hosted Keycloak instance in the `lab` account.
 
 This stack creates:
 
-- a dedicated `t4g.small` Amazon Linux 2023 EC2 instance
+- a dedicated `t4g.small` Flatcar Container Linux EC2 instance
+- a dedicated encrypted gp3 data volume mounted at `/var/lib/keycloak`
 - an IAM role and instance profile for SSM management
 - a security group that exposes HTTPS only to lab CIDRs
 - a private Route 53 `A` record for `id.glab.lol`
 - scoped Route 53 permissions for ACME DNS-01 validation in `acme.glab.lol`
-- a GitHub token broker Lambda, sourced from
-  `meigma/github-token-broker`, for short-lived `GilmanLab/secrets` access
+- a GitHub token broker Lambda, sourced from `meigma/github-token-broker`, for short-lived `GilmanLab/secrets` access
 - Keycloak instance-role permission to invoke the token broker
-- an SSM-driven Docker Compose deployment for Postgres, Keycloak, and Traefik
+- Keycloak instance-role permission to decrypt only SOPS secrets with `Repo=GilmanLab/secrets` and `Scope=keycloak`
+- Ignition-managed systemd units for Postgres, Keycloak, Traefik, and bootstrap secret fetches
 
-This stack intentionally stops at starting Keycloak. It does **not** configure
-realms, GitHub OIDC, clients, backups, Synology sync, Kubernetes OIDC, Argo CD,
-Grafana, IAM Identity Center federation, or `keycloak-config-cli`.
+This stack intentionally stops at starting Keycloak. It does **not** configure realms, GitHub OIDC, clients, backups, Synology sync, Kubernetes OIDC, Argo CD, Grafana, IAM Identity Center federation, or `keycloak-config-cli`.
 
-Traefik obtains the `id.glab.lol` certificate from Let's Encrypt through
-DNS-01. Cloudflare delegates `_acme-challenge.id.glab.lol` to the public
-Route 53 `acme.glab.lol` zone, and the Keycloak instance role may mutate only
-the delegated TXT record for this hostname.
+Traefik obtains the `id.glab.lol` certificate from Let's Encrypt through DNS-01. Cloudflare delegates `_acme-challenge.id.glab.lol` to the public Route 53 `acme.glab.lol` zone, and the Keycloak instance role may mutate only the delegated TXT record for this hostname.
+
+## Bootstrap model
+
+Flatcar receives raw Ignition JSON through EC2 user data. Ignition writes non-secret runtime config and helper scripts under `/etc/glab/keycloak`, then enables the systemd units.
+
+`glab-keycloak-bootstrap.service` runs the pinned `labctl` container:
+
+```sh
+secrets get services/keycloak/bootstrap.sops.yaml \
+  --source github \
+  --field /stack_env \
+  --output /run/glab/keycloak/stack.env \
+  --aws-region us-west-2 \
+  --broker-function glab-github-token-broker
+```
+
+Plaintext bootstrap material is written only under `/run/glab/keycloak`. The persistent data volume stores Postgres state in `/var/lib/keycloak/postgres` and ACME state in `/var/lib/keycloak/acme`; the Postgres directory is owned by the container's Postgres UID.
 
 ## Prerequisites
 
 - OpenTofu `>= 1.10`
 - `just`
 - `AWS_PROFILE` set to the `lab` account admin profile
-- `GLAB_AWS_STATE_BUCKET` set to the pre-created S3 backend bucket in the
-  `lab` account
+- `GLAB_AWS_STATE_BUCKET` set to the pre-created S3 backend bucket in the `lab` account
 - the `aws/lab-foundation` and `aws/subnet-router` stacks already applied
 - the GitHub App SSM parameters created outside Terraform:
   `/glab/bootstrap/github-app/client-id`,
   `/glab/bootstrap/github-app/installation-id`, and
   `/glab/bootstrap/github-app/private-key-pem`
-- `gh` and `sha256sum` on the apply host so the broker module can download and
-  verify the pinned release asset
-- Cloudflare delegates `acme.glab.lol` to the `aws/lab-foundation`
-  `acme_zone_name_servers` output and sets
-  `_acme-challenge.id.glab.lol` as a CNAME to
-  `_acme-challenge.id.acme.glab.lol`
+- `gh` and `sha256sum` on the apply host so the broker module can download and verify the pinned release asset
+- Cloudflare delegates `acme.glab.lol` to the `aws/lab-foundation` `acme_zone_name_servers` output and sets `_acme-challenge.id.glab.lol` as a CNAME to `_acme-challenge.id.acme.glab.lol`
+- `secrets/services/keycloak/bootstrap.sops.yaml` exists in `GilmanLab/secrets` with SOPS KMS context `Repo=GilmanLab/secrets` and `Scope=keycloak`
 
-The expected local operator flow is to export both values via `direnv`.
+The expected local operator flow is to export both AWS values via `direnv`.
 
 ## Legacy token broker cleanup
 
-The old `aws/github-token-broker` stack used the same default Lambda name,
-`glab-github-token-broker`, and was originally published from
-`GilmanLab/platform`. Do not apply both stacks at the same time.
+The old `aws/github-token-broker` stack used the same default Lambda name, `glab-github-token-broker`, and was originally published from `GilmanLab/platform`. Do not apply both stacks at the same time.
 
 If the old Lambda is still active, retire it before applying this stack:
 
@@ -69,9 +75,7 @@ tofu plan -destroy \
 tofu apply destroy-broker-only.tfplan
 ```
 
-Then apply `aws/keycloak`. This removes the legacy Lambda resources while
-leaving the GitHub Actions OIDC provider alone; only run a full destroy after
-confirming that provider has no shared use.
+Then apply `aws/keycloak`. This removes the legacy Lambda resources while leaving the GitHub Actions OIDC provider alone; only run a full destroy after confirming that provider has no shared use.
 
 ## Usage
 
@@ -82,15 +86,19 @@ just plan
 just apply
 ```
 
-`just init` uses `GLAB_AWS_STATE_BUCKET` to finish the otherwise-partial S3
-backend configuration. The backend bucket itself is part of the manual AWS
-bootstrap and is intentionally not managed by this stack.
+`just init` uses `GLAB_AWS_STATE_BUCKET` to finish the otherwise-partial S3 backend configuration. The backend bucket itself is part of the manual AWS bootstrap and is intentionally not managed by this stack.
 
-After apply, use SSM to inspect the host:
+After apply, use SSM to inspect the host without printing secret values:
 
 ```sh
 aws ssm send-command \
   --document-name AWS-RunShellScript \
   --targets Key=instanceids,Values="$(tofu output -raw instance_id)" \
-  --parameters commands='["docker compose --env-file /opt/keycloak/stack.env -f /opt/keycloak/compose.yml ps","curl -fsS http://127.0.0.1:9000/health/ready"]'
+  --parameters commands='[
+    "systemctl is-active glab-keycloak-bootstrap.service glab-keycloak-postgres.service glab-keycloak.service glab-keycloak-traefik.service",
+    "findmnt /var/lib/keycloak",
+    "stat -c %a\\ %s\\ %n /run/glab/keycloak/stack.env",
+    "cut -d= -f1 /run/glab/keycloak/stack.env | sort",
+    "curl -fsS http://127.0.0.1:9000/health/ready"
+  ]'
 ```

aws/keycloak/compute.tf

@@ -1,9 +1,10 @@
 resource "aws_instance" "keycloak" {
-  ami                         = data.aws_ssm_parameter.al2023_arm64.value
+  ami                         = var.flatcar_ami_id
   associate_public_ip_address = true
   iam_instance_profile        = aws_iam_instance_profile.keycloak.name
   instance_type               = var.instance_type
   subnet_id                   = data.aws_subnet.public.id
+  user_data                   = local.ignition_config
   user_data_replace_on_change = true
   vpc_security_group_ids      = [aws_security_group.keycloak.id]
 
@@ -24,3 +25,20 @@ resource "aws_instance" "keycloak" {
     Name = var.instance_name
   })
 }
+
+resource "aws_ebs_volume" "keycloak_data" {
+  availability_zone = data.aws_subnet.public.availability_zone
+  encrypted         = true
+  size              = var.data_volume_size
+  type              = "gp3"
+
+  tags = merge(local.common_tags, {
+    Name = "${var.instance_name}-data"
+  })
+}
+
+resource "aws_volume_attachment" "keycloak_data" {
+  device_name = var.data_volume_device_name
+  instance_id = aws_instance.keycloak.id
+  volume_id   = aws_ebs_volume.keycloak_data.id
+}

aws/keycloak/data.tf

@@ -1,5 +1,3 @@
-data "aws_caller_identity" "current" {}
-
 data "aws_iam_policy_document" "keycloak_assume_role" {
   statement {
     actions = ["sts:AssumeRole"]
@@ -11,16 +9,31 @@ data "aws_iam_policy_document" "keycloak_assume_role" {
   }
 }
 
-data "aws_iam_policy_document" "keycloak_bootstrap_parameters" {
+data "aws_iam_policy_document" "keycloak_sops_decrypt" {
   statement {
-    sid = "AllowReadWriteKeycloakBootstrapParameters"
+    sid = "AllowDecryptKeycloakSopsSecrets"
     actions = [
-      "ssm:GetParameter",
-      "ssm:PutParameter",
+      "kms:Decrypt",
     ]
     resources = [
-      local.ssm_parameter_path_arn,
+      var.sops_kms_key_arn,
     ]
+
+    condition {
+      test     = "StringEquals"
+      variable = "kms:EncryptionContext:Repo"
+      values = [
+        var.sops_kms_context_repo,
+      ]
+    }
+
+    condition {
+      test     = "StringEquals"
+      variable = "kms:EncryptionContext:Scope"
+      values = [
+        var.sops_kms_context_scope,
+      ]
+    }
   }
 }
 
@@ -102,10 +115,6 @@ data "aws_route_table" "public" {
   }
 }
 
-data "aws_ssm_parameter" "al2023_arm64" {
-  name = var.ami_ssm_parameter_name
-}
-
 data "aws_instance" "subnet_router" {
   filter {
     name   = "tag:Name"

aws/keycloak/iam.tf

@@ -7,18 +7,18 @@ resource "aws_iam_role" "keycloak" {
   })
 }
 
-resource "aws_iam_role_policy" "keycloak_bootstrap_parameters" {
-  name   = "${var.iam_role_name}-bootstrap-parameters"
-  policy = data.aws_iam_policy_document.keycloak_bootstrap_parameters.json
-  role   = aws_iam_role.keycloak.id
-}
-
 resource "aws_iam_role_policy" "keycloak_acme_route53" {
   name   = "${var.iam_role_name}-acme-route53"
   policy = data.aws_iam_policy_document.keycloak_acme_route53.json
   role   = aws_iam_role.keycloak.id
 }
 
+resource "aws_iam_role_policy" "keycloak_sops_decrypt" {
+  name   = "${var.iam_role_name}-sops-keycloak-decrypt"
+  policy = data.aws_iam_policy_document.keycloak_sops_decrypt.json
+  role   = aws_iam_role.keycloak.id
+}
+
 resource "aws_iam_role_policy_attachment" "keycloak_ssm_managed_instance_core" {
   policy_arn = "arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore"
   role       = aws_iam_role.keycloak.name