pebcak? not working out of the box.

[F0gC10ud]

Hello,
I just found your project and I am grateful for the work you are doing! I was previously a user of https://github.com/docker-archive/compose-cli and was sad when it started to die.

Keep up the great work!

The bug[s]?

I am assuming I am doing something wrong, but I have tried on my linux desktop in a docker container, on both master and tag v1.1.6. I still get errors with most of the things I try do. (I still haven't managed to deploy anything yet. )

Reproducing issue

out of the box e2e-examples/nginx_webapp

Out of the box using tag v1.1.6, I get this error

#  I get the same error if I use "render --format yaml -d out"
python -m venv venv
source  venv/bin/activate
pip install ecs-composex
git clone https://github.com/compose-x/ecs_composex
cd ecs_composex/e2e-examples/nginx_webapp
ecs-compose-x up -n test -f docker-compose.yaml -f aws-compose-x.yaml  -f docker-compose.override.yaml 

Error

See full log further below

raise KeyError(
KeyError: ('x-route53.PublicZone - Requires either or both Properties or MacroParameters. Got neither', dict_keys(['ZoneName'])

Work envs

1. Linux Python 3.11.2 master and v1.1.6
2. Docker/python:3.10-slim (3.10.16) master and v1.1.6

Logs

Hold my beer, full log

$ python -m venv venv
$ source  venv/bin/activate

(venv) $ pip install ecs-composex
Collecting ecs-composex
  Using cached ecs_composex-1.1.6-py3-none-any.whl (522 kB)
Collecting Jinja2<4.0.0,>=3.1.2
  Using cached jinja2-3.1.5-py3-none-any.whl (134 kB)
Collecting PyYAML<7.0,>=6.0
  Using cached PyYAML-6.0.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (762 kB)
Collecting boto3<2.0,>=1.26
  Using cached boto3-1.35.98-py3-none-any.whl (139 kB)
Collecting compose-x-common<2.0,>=1.4
  Using cached compose_x_common-1.4.10-py3-none-any.whl (43 kB)
Collecting compose-x-render<2.0,>=1.1
  Using cached compose_x_render-1.1.0-py3-none-any.whl (24 kB)
Collecting docker<8.0,>=6.0.1
  Using cached docker-7.1.0-py3-none-any.whl (147 kB)
Collecting importlib-resources<7.0,>=6.4
  Using cached importlib_resources-6.5.2-py3-none-any.whl (37 kB)
Collecting jsonschema>=4.21
  Using cached jsonschema-4.23.0-py3-none-any.whl (88 kB)
Collecting requests<3.0,>=2.28
  Using cached requests-2.32.3-py3-none-any.whl (64 kB)
Collecting retry2<0.10,>=0.9
  Using cached retry2-0.9.5-py2.py3-none-any.whl (6.0 kB)
Collecting tabulate<0.9,>=0.8
  Using cached tabulate-0.8.10-py3-none-any.whl (29 kB)
Collecting troposphere<5.0,>=4.5.3
  Using cached troposphere-4.8.3-py3-none-any.whl (530 kB)
Collecting troposphere-awscommunity-applicationautoscaling-scheduledaction<0.2.0,>=0.1.1
  Using cached troposphere_awscommunity_applicationautoscaling_scheduledaction-0.1.1-py3-none-any.whl (5.4 kB)
Collecting botocore<1.36.0,>=1.35.98
  Using cached botocore-1.35.98-py3-none-any.whl (13.3 MB)
Collecting jmespath<2.0.0,>=0.7.1
  Using cached jmespath-1.0.1-py3-none-any.whl (20 kB)
Collecting s3transfer<0.11.0,>=0.10.0
  Using cached s3transfer-0.10.4-py3-none-any.whl (83 kB)
Collecting flatdict<5.0.0,>=4.0.1
  Using cached flatdict-4.0.1-py3-none-any.whl
Collecting python-dateutil<3.0.0,>=2.8.2
  Using cached python_dateutil-2.9.0.post0-py2.py3-none-any.whl (229 kB)
Collecting argparse<2.0.0,>=1.4.0
  Using cached argparse-1.4.0-py2.py3-none-any.whl (23 kB)
Collecting urllib3>=1.26.0
  Using cached urllib3-2.3.0-py3-none-any.whl (128 kB)
Collecting MarkupSafe>=2.0
  Using cached MarkupSafe-3.0.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (23 kB)
Collecting attrs>=22.2.0
  Using cached attrs-24.3.0-py3-none-any.whl (63 kB)
Collecting jsonschema-specifications>=2023.03.6
  Using cached jsonschema_specifications-2024.10.1-py3-none-any.whl (18 kB)
Collecting referencing>=0.28.4
  Using cached referencing-0.35.1-py3-none-any.whl (26 kB)
Collecting rpds-py>=0.7.1
  Using cached rpds_py-0.22.3-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (381 kB)
Collecting charset-normalizer<4,>=2
  Using cached charset_normalizer-3.4.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (143 kB)
Collecting idna<4,>=2.5
  Using cached idna-3.10-py3-none-any.whl (70 kB)
Collecting certifi>=2017.4.17
  Using cached certifi-2024.12.14-py3-none-any.whl (164 kB)
Collecting decorator>=3.4.2
  Using cached decorator-5.1.1-py3-none-any.whl (9.1 kB)
Collecting cfn-flip>=1.0.2
  Using cached cfn_flip-1.3.0-py3-none-any.whl (21 kB)
Collecting Click
  Using cached click-8.1.8-py3-none-any.whl (98 kB)
Collecting six
  Using cached six-1.17.0-py2.py3-none-any.whl (11 kB)
Installing collected packages: flatdict, argparse, urllib3, tabulate, six, rpds-py, PyYAML, MarkupSafe, jmespath, importlib-resources, idna, decorator, Click, charset-normalizer, certifi, attrs, retry2, requests, referencing, python-dateutil, Jinja2, cfn-flip, troposphere, jsonschema-specifications, docker, botocore, troposphere-awscommunity-applicationautoscaling-scheduledaction, s3transfer, jsonschema, boto3, compose-x-common, compose-x-render, ecs-composex
Successfully installed Click-8.1.8 Jinja2-3.1.5 MarkupSafe-3.0.2 PyYAML-6.0.2 argparse-1.4.0 attrs-24.3.0 boto3-1.35.98 botocore-1.35.98 certifi-2024.12.14 cfn-flip-1.3.0 charset-normalizer-3.4.1 compose-x-common-1.4.10 compose-x-render-1.1.0 decorator-5.1.1 docker-7.1.0 ecs-composex-1.1.6 flatdict-4.0.1 idna-3.10 importlib-resources-6.5.2 jmespath-1.0.1 jsonschema-4.23.0 jsonschema-specifications-2024.10.1 python-dateutil-2.9.0.post0 referencing-0.35.1 requests-2.32.3 retry2-0.9.5 rpds-py-0.22.3 s3transfer-0.10.4 six-1.17.0 tabulate-0.8.10 troposphere-4.8.3 troposphere-awscommunity-applicationautoscaling-scheduledaction-0.1.1 urllib3-2.3.0



(venv) $ git clone https://github.com/compose-x/ecs_composex
Cloning into 'ecs_composex'...
remote: Enumerating objects: 15303, done.
  <snip>
Resolving deltas: 100% (11660/11660), done.

(venv) $ cd ecs_composex/e2e-examples/nginx_webapp

(venv) ecs_composex/e2e-examples/nginx_webapp$ ecs-compose-x up -n test -f docker-compose.yaml -f aws-compose-x.yaml  -f docker-compose.override.yaml 

2025-01-14 14:17:03 [    INFO] volumes.app - Mapped to frontend
2025-01-14 14:17:03 [    INFO] services.frontend - No Launch Type defined. Using default: FARGATE
2025-01-14 14:17:04 [    INFO] Service families to process ['frontend']
2025-01-14 14:17:04 [    INFO] No cluster information provided. Creating a new one
Loaded x-route53 route53 route53 /scratch/venv/lib/python3.11/site-packages/ecs_composex/route53
Loaded x-cloudmap cloudmap cloudmap /scratch/venv/lib/python3.11/site-packages/ecs_composex/cloudmap
Loaded x-elbv2 elbv2 elbv2 /scratch/venv/lib/python3.11/site-packages/ecs_composex/elbv2
2025-01-14 14:17:04 [    INFO] x-elbv2.public-alb - Adding target frontend:frontend
2025-01-14 14:17:04 [    INFO] LB public-alb only has a unique service. LB will be deployed with the service stack.
2025-01-14 14:17:04 [    INFO] Processing x-route53
Traceback (most recent call last):
  File "/scratch/venv/bin/ecs-compose-x", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/scratch/venv/lib/python3.11/site-packages/ecs_composex/cli.py", line 213, in main
    root_stack = generate_full_template(settings)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/scratch/venv/lib/python3.11/site-packages/ecs_composex/ecs_composex.py", line 257, in generate_full_template
    add_x_resources(settings)
  File "/scratch/venv/lib/python3.11/site-packages/ecs_composex/ecs_composex.py", line 181, in add_x_resources
    x_stack = module.stack_class(
              ^^^^^^^^^^^^^^^^^^^
  File "/scratch/venv/lib/python3.11/site-packages/ecs_composex/route53/route53_stack.py", line 208, in __init__
    if module.new_resources:
       ^^^^^^^^^^^^^^^^^^^^
  File "/scratch/venv/lib/python3.11/site-packages/ecs_composex/mods_manager.py", line 123, in new_resources
    raise KeyError(
KeyError: ('x-route53.PublicZone - Requires either or both Properties or MacroParameters. Got neither', dict_keys(['ZoneName']))

---

PEBKAC, ou une case 18"?

If its just me here are other details

fwiw

1. I am using ca-centra-1, and I noticed it was absent from the source I grep'd through
2. in the case of x-vpc, for the most, it will never work, regardless of what type
3. I can build stacks with boto3 and awscli ( so my tokens work )
4. After removing most of the aws stuff I was able to deploy a stack that failed and was rolled back

When trying to build a config from scratch or using compose-x-labs I will often get these errors

1. is not of type 'object'
2. is not valid under any of the given schemas

I copied this from the doc, and got the error is not valid under any of the given schemas

x-vpc:
  Use:
    VpcId:  vpc-12345678901234567
    AppSubnets:
    - subnet-12345678901234567
    - subnet-12345678901234567
    - subnet-12345678901234567
    StorageSubnets:
    - subnet-12345678901234567
    - subnet-12345678901234567
    - subnet-12345678901234567    
    PublicSubnets:
    - subnet-12345678901234567
    - subnet-12345678901234567
    - subnet-12345678901234567

[raffidahmad]

@F0gC10ud The starter example here should work as is: https://docs.compose-x.io/examples/nginx_simple.html

And your x-vpc syntax is wrong as well

x-vpc:
  Lookup:
    VpcId:
      Tags:
        - Name: xyz-vpc
    AppSubnets:
      Tags:
        - Name: xyz-subnet-private1-${AWS_REGION}a
    StorageSubnets:
      Tags:
        - Name: xyz-subnet-private1-${AWS_REGION}a
    PublicSubnets:
      Tags:
        - Name: xyz-subnet-public1-${AWS_REGION}a

[JohnPreston]

The use of VPC and other x- has been removed a while back due to the pain and suffering it can cause. Lookup allows to do a few things as it gathers the metadata about the resources themselves. Yes, that means you need to be able to scan resources and have tagged these, but that's also a best practice and costs nothing to do the tagging itself.

The compose-cli was I hoped a good replacement for this project but it was lacking some essentials, I found the syntax for overrides was difficult and generally compose-x had gone a lot further already. While it doesn't work like for like to use the two, you should be able to achieve similar results.

One tweak I would do from @raffidahmad answer (thanks btw man!)

x-vpc:
  Lookup:
    VpcId:
      Tags:
        Name: xyz-vpc
    AppSubnets:
      Tags:
        Name: xyz-subnet-private1-${AWS_REGION}a
    StorageSubnets:
      Tags:
        Name: xyz-subnet-private1-${AWS_REGION}a
    PublicSubnets:
      Tags:
        Name: xyz-subnet-public1-${AWS_REGION}a

Tags appear in most places as a List which was a bad decision from me at the time, allowing duplicates (which is not possible). So moving to a dict/object structure will get your IDEs to yell at you and flag duplicate entries :)

[F0gC10ud]

Hey guys, thanks for your help! I am grateful for your time.

if I can get this to work I will be very excited to use this tool, so thanks.

The VPC question

wrt to the vpc stuff, just to be clear, in case I am getting this wrong.
if I want to use my VPC named "myapp" and a subnet named "myapp-private1a"

Am I getting this right?

x-vpc:
  Lookup:
    VpcId:
      Tags:
        Name: myapp
    AppSubnets:
      Tags:
        Name: myapp-private1a-ca-central-1a

---

The pebcak :)

Following the suggestion, to try https://docs.compose-x.io/examples/nginx_simple.html

Using the docker command to avoid an local env related issues

I am getting one of the errors I get when trying some of the compose-x-labs. jsonschema/validators.py often fails to validate.

validators.py", line 451, in validate
    raise error
jsonschema.exceptions.ValidationError: [{'name': 'frontend:frontend', 'port': 80, 'protocol': 'HTTP', 'healthcheck': '80:HTTP:/:200'}] is not of type 'object'

the files and logs

just to show you that I copied exactly what is on the web site.

The logs when using docker

docker run -u $(id -u):$(id -u) -it --rm -v ~/.aws:/tmp/.aws -e HOME=/tmp -v $PWD:/tmp public.ecr.aws/compose-x/compose-x:latest plan -f docker-compose.yaml -f aws-compose-x.yaml -n frontend-app

2025-01-15 21:05:47 [    INFO] services.frontend - No Launch Type defined. Using default: FARGATE
2025-01-15 21:05:47 [    INFO] Service families to process ['frontend']
2025-01-15 21:05:47 [    INFO] No cluster information provided. Creating a new one
Loaded x-elbv2 elbv2 elbv2 /usr/local/lib/python3.10/site-packages/ecs_composex/elbv2
2025-01-15 21:05:47 [   ERROR] elbv2.public-alb - Definition is not conform to schema.
Traceback (most recent call last):
  File "/usr/local/bin/ecs-compose-x", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.10/site-packages/ecs_composex/cli.py", line 213, in main
    root_stack = generate_full_template(settings)
  File "/usr/local/lib/python3.10/site-packages/ecs_composex/ecs_composex.py", line 248, in generate_full_template
    settings.mod_manager.init_mods_resources(settings)
  File "/usr/local/lib/python3.10/site-packages/ecs_composex/mods_manager.py", line 307, in init_mods_resources
    module.set_resources(settings)
  File "/usr/local/lib/python3.10/site-packages/ecs_composex/mods_manager.py", line 266, in set_resources
    new_definition = self.resource_class(
  File "/usr/local/lib/python3.10/site-packages/ecs_composex/elbv2/elbv2_stack/elbv2.py", line 75, in __init__
    super().__init__(name, definition, module, settings)
  File "/usr/local/lib/python3.10/site-packages/ecs_composex/compose/x_resources/network_x_resources.py", line 37, in __init__
    super().__init__(name, definition, module, settings)
  File "/usr/local/lib/python3.10/site-packages/ecs_composex/compose/x_resources/services_resources.py", line 39, in __init__
    super().__init__(name, definition, module, settings)
  File "/usr/local/lib/python3.10/site-packages/ecs_composex/compose/x_resources/__init__.py", line 79, in __init__
    self.validate_schema(name, definition, module.mod_key)
  File "/usr/local/lib/python3.10/site-packages/ecs_composex/compose/x_resources/__init__.py", line 191, in validate_schema
    _eval.validate(definition)
  File "/usr/local/lib/python3.10/site-packages/jsonschema/validators.py", line 451, in validate
    raise error
jsonschema.exceptions.ValidationError: [{'name': 'frontend:frontend', 'port': 80, 'protocol': 'HTTP', 'healthcheck': '80:HTTP:/:200'}] is not of type 'object'

Failed validating 'type' in schema['properties']['Services']:
    {'type': 'object',
     'additionalProperties': False,
     'patternProperties': {'x-': {},
                           '^[a-zA-Z0-9\\:_\\-]+$': {'$ref': '#/definitions/TargetDef'}}}

On instance['Services']:
    [{'name': 'frontend:frontend',
      'port': 80,
      'protocol': 'HTTP',
      'healthcheck': '80:HTTP:/:200'}]

aws-compose-x.yaml click to view

services:
  frontend:
    x-ecr:
      InterpolateWithDigest: true

# We need DNS information. We indicate which DNS zone to use publicly and which one to use in the VPC.
# Given that we do not indicate Lookup, the new DNS Zones will be created.

x-dns:
  PublicZone:
    Name: mydomain.net # Create a new public route53 zone.
  PrivateNamespace:
    Name: cluster.internal # Create a new AWS CloudMap service discovery instance associated with the VPC

# We create an ALB and send traffic to our frontend. Note that the listener is not encrypted at this point.
# To use encryption we need n ACM certificate and set the listener protocol to HTTPS

x-elbv2:
  public-alb:
    Properties:
      Scheme: internet-facing
      Type: application
    Services:
      - name: frontend:frontend
        port: 80
        protocol: HTTP
        healthcheck: 80:HTTP:/:200 # We expect port 80 with HTTP protocol to work and we expect a 200 OK return
    Listeners:
      - Port: 80
        Protocol: HTTP
        Targets:
          - name: frontend:frontend
            access: /

docker-compose.yaml click to view

version: "3.8"
volumes:
  app:

services:
  frontend:
    image: ${REGISTRY_URI}frontend:${TAG:-latest}
    ports:
      - protocol: tcp
        target: 80
    build:
      context: .
      dockerfile: Dockerfile
    deploy:
      resources:
        limits:
          cpus: 0.5
          memory: 256MB
        reservations:
          cpus: 0.1
          memory: 128MB

[JohnPreston]

I need to update a bunch of example files as it stands. Check for more up to date examples in the labs.
The error you are getting above means you have a list [] instead of a dict/mapping {}

[JohnPreston]

I will update the docs to reflect the correct syntax. But generally, check the use-cases/ directory.ies as they contain all the test cases and the spec files which give you the expected input in a JSON Schema format.