Unable to Run Workshop on My Own AWS Account – CloudFormation Stack Stuck During Setup

[Gall-oDrone]

Hi, I'm currently following the Platform Engineering on EKS Workshop in self-paced mode on my own AWS account. I'm encountering an issue that prevents me from progressing beyond the Setting up the Visual Studio Code Server step.
Issue Details:

Workshop Link: [Platform Engineering on EKS – Setting up the Visual Studio Code Server](https://catalog.us-east-1.prod.workshops.aws/platform-engineering-on-eks/en-US/10-start-workshop/12-self-paced#setting-up-the-visual-studio-code-server)

Problem: The CloudFormation stack deployment gets stuck in an UPDATE_IN_PROGRESS or CREATE_IN_PROGRESS state and never completes. I waited over 90 minutes, but there was no change in status or error shown.

CloudFormation behavior:

    No explicit failure message

    Some resources seem to stay in an in-progress state indefinitely

    No rollback or timeout occurs automatically

Environment:

AWS Region: us-west-2

Running in my own AWS account

IAM permissions are aligned with the workshop prerequisites

Steps Taken:

Started workshop in self-paced mode using the link provided.

Chose to deploy in my own AWS account.

Followed steps up to Setting up the Visual Studio Code Server.

Deployed the CloudFormation template as instructed.

Waited over 90 minutes without completion or error. Altough stack stucks on 'Waiting Condition'

What I’ve Tried:

Verified that IAM roles and permissions are sufficient.

Checked for limits or quota issues in my AWS account (none found).

Attempted to look at CloudFormation events, but no specific resource shows an error.

Tried re-deploying the stack from scratch with the same result and on a new account.

Request:

Could you please help diagnose why the stack might be stuck during deployment, or suggest any known workarounds? I’d like to proceed with the rest of the workshop, but I’m currently blocked at this early stage.

Thanks!

Let me know if you want to include logs or screenshots, or tailor it to a specific repo.

[zjaco13]

Hi @Gall-oDrone, we've been able to track down the issue to the release of new versions of the Terraform provider for Helm and AWS. We will be applying a fix shortly. Thank you for the issue!

[zjaco13]

Hi @Gall-oDrone, this should be resolved! Please let me know otherwise.

[Gall-oDrone]

Hi @zjaco13,

Thanks for the update and the hotfix reference. I wanted to confirm: was the fix applied only to the Terraform code, or was it also addressed in the modern-engineering-workshop CloudFormation stack?

I’m currently facing the same issue described in the original comment while following the self-paced steps from:
Platform Engineering on EKS – Self-Paced Workshop.

Specifically, my CloudFormation stack remains stuck in UPDATE_IN_PROGRESS on the "Waiting Condition" step without any error messages. Could you clarify whether the CloudFormation template was updated as part of the fix, or if any additional manual steps are required on our end?

Thank you for your help!

[zjaco13]

Hey @Gall-oDrone, it was not addressed in the self guided cloudformation stack. I've updated the stack to pin the tf versions as well. Thank you! Please ping me if it is still stuck.

[mavogel]

Hi @zjaco13 ,
AFAICS the link to the self-paced version unfortunately seems to be deleted: Platform Engineering on EKS – Self-Paced Workshop

I was trying the riv25 branch, which got stuck with the error

'module.eks_monitoring_spoke_prod.module.helm_addon.helm_release.addon[0]: Creating...
Warning: Helm release "" was created but has a failed status. Use the `helm` command to investigate the error, correct it,
then run Terraform again.
  with module.eks_monitoring_spoke_prod.module.helm_addon.helm_release.addon[0],
  on .terraform/modules/eks_monitoring_spoke_prod.helm_addon/modules/kubernetes-addons/helm-addon/main.tf line 1, in resource
 "helm_release" "addon":
   1: resource "helm_release" "addon" {
Error: 1 error occurred:
    * Internal error occurred: Internal error occurred: conversion webhook for opentelemetry.io/v1alpha1,
Kind=OpenTelemetryCollector failed: Post
"https://opentelemetry-operator-webhook.opentelemetry-operator-system.svc:443/convert?timeout=30s": no endpoints available
for service "opentelemetry-operator-webhook"
  with module.eks_monitoring_spoke_prod.module.helm_addon.helm_release.addon[0],
  on .terraform/modules/eks_monitoring_spoke_prod.helm_addon/modules/kubernetes-addons/helm-addon/main.tf line 1, in resource
 "helm_release" "addon":
   1: resource "helm_release" "addon" {'

and with some AI help (I did not fully verify yet) I got the explanation

Root cause: The OpenTelemetry operator is not being deployed to the spoke clusters via GitOps, but the Terraform eks_monitoring_spoke_prod module requires it to create OpenTelemetryCollector resources.

  The spoke clusters don't have their own addons.yaml override, and the operator enablement depends on cluster labels.

I am using the commit 7ba06ead7d0b15fe491968a175c23bd2dc4f28b6

Questions:

• is there a plan to update the WS workshop for a self-paced version in the near future? (if no, I would continue debugging as I would like to do the workshop and learn)
• or even merge the riv25 branch into main? (maybe @allamand knows 😃 )

Looking forward to you reply

[allamand]

@mavogel, sorry about that, we are working on providing managed capability to the workshop on riv25 and will merge that into main soon after. Normally the on your own will be bringer back also in the workshop very soon. The cloudformation should work, but I’ll test it to confirm it point to the latest version as the publishing system is little different. I’ll come back quickly here.

[allamand]

I created this PR : #412 that update the Readme with valid instructions to deploy the workshop in us-west-2 only for now.

curl https://ws-assets-prod-iad-r-pdx-f3b3f9f1a7d6a3d0.s3.us-west-2.amazonaws.com/daa2a765-04db-4399-aaa7-fddc8d07e9e1/peeks-workshop-team-stack-self.json --output peeks-workshop.json

bucketName="peeks-workshop-$(uuidgen | tr -d - | tr '[:upper:]' '[:lower:]')"
aws s3api create-bucket \
    --bucket "$bucketName" \
    --region us-west-2 \
    --create-bucket-configuration LocationConstraint=us-west-2

aws cloudformation deploy --stack-name peeks-workshop \
    --template-file ./peeks-workshop.json \
    --parameter-overrides \
        ParticipantAssumedRoleArn=$(aws sts get-caller-identity --query Arn --output text) \
    --capabilities CAPABILITY_NAMED_IAM \
    --s3-bucket $bucketName

let us know if you want the Cfn for other regions as well

[mavogel]

Great, thank you @allamand, now the clusters deploy successfully on second retry with the otel collectors in the spoke clusters. Probably #185 will standardise it then 😃

However I bumped into some more...

on create

Unfortunately, now I got stuck in the IDE initialization IdeInitScriptWaitCondition:

Root Cause Chain:

1. IDE Init Script runs ArgoCD sync operations via SSM RunCommand on EC2 instance i-0281example
2. ArgoCD apps stuck in "Progressing" state:
   - flux-peeks-spoke-prod - Flux controllers (source-controller, image-reflector) not becoming Healthy
   - argo-rollouts-peeks-spoke-prod - IAM Role resource stuck in Degraded state
   - metrics-server-peeks-spoke-prod - Deployment stuck in Progressing
3. ArgoCD CLI timeouts:
   rpc error: code = Canceled desc = context canceled (multiple times over 30 mins)
rpc error: code = Unknown desc = POST .../argocd/application.ApplicationService/Get failed with status code 504
4. GitRepository resources in Degraded state:
   source.toolkit.fluxcd.io GitRepository flux-system platform-gitops-repo Synced Degraded
source.toolkit.fluxcd.io GitRepository flux-system workloads-gitops-repo Synced Degraded
5. Script exceeded 31m45s timeout → Signaled CFN with FAILURE → Rollback triggered

Key Issues Identified:

  | Issue                          | Component             | Impact                                  |
  |--------------------------------|-----------------------|-----------------------------------------|
  | ArgoCD sync timeout            | ArgoCD CLI            | RPC calls canceled after ~3min each     |
  | Flux GitRepository Degraded    | flux-peeks-spoke-prod | Git sync failing                        |
  | IAM Role Degraded              | argo-rollouts         | ACK IAM controller not reconciling      |
  | Deployments stuck Progressing  | Multiple apps         | Pods not reaching Ready state           |

Is this known? If not, today I faced some network issue to us-west-2 and had to retry lots of commands locally when debugging. I would retry it again in the new year with --disable-rollback to see where argo / flux reconcile fails

on destroy

had to manually removed the SecGroups for the 3 ingress NLBs:

1. the NLBs
2. which had the ENIs attached
3. which was associated with SecGroups

which would have caused terraform destroy to timeout. Maybe a known issue

[allamand]

@mavogel sorry you have some issues. Did you get the idea cloudfront url and password ?

We know that sometimes all argocd app may have trouble to sync properly, but the platform/infra/terraform/scripts/0-init.sh script which run in ssm should handle this normally.

If you have access to the ide we have another command argocd-sync that try to reconcile a known limitation when argocd when using multiple sources on same repository using the $values, which need sometimes to terminate operation and retry.

Not sure exactly which issue you got this time, kiro-cli is also installed and help debug thoses..

There are also dependencies between app so if a critical is missing other may not work also.

I have created a stack also when posting this will review the success status and let you know

[allamand]

the stack were working fine on my side

@mavogel don't hesitate to dm me when you are on it

[mavogel]

After f2f with @allamand I can confirm that #420 fixes this issue. Tested on eu-central-1. Minor bugs with hardcoded us-west-2 regions in e.g. the k8s ingress configs are reported :)