[FR] x-vpc: Allow Lookup.Identifier usage

[jakub-qg]

Is your feature request related to a problem? Please describe.

Make the syntax work:

x-vpc:
  Lookup:
    VpcId:
      Identifier: vpc-xxx

Output (if there are no VPC with tags in whole AWS acct):

2025-03-25 11:39:47 [    INFO] Processing x-vpc
Traceback (most recent call last):
  File "/Users/user/.local/bin/compose-x", line 8, in <module>
    sys.exit(main())
             ~~~~^^
  File "/Users/user/.local/pipx/venvs/ecs-composex/lib/python3.13/site-packages/ecs_composex/cli.py", line 213, in main
    root_stack = generate_full_template(settings)
  File "/Users/user/.local/pipx/venvs/ecs-composex/lib/python3.13/site-packages/ecs_composex/ecs_composex.py", line 257, in generate_full_template
    add_x_resources(settings)
    ~~~~~~~~~~~~~~~^^^^^^^^^^
  File "/Users/user/.local/pipx/venvs/ecs-composex/lib/python3.13/site-packages/ecs_composex/ecs_composex.py", line 181, in add_x_resources
    x_stack = module.stack_class(
        module.mapping_key,
    ...<2 lines>...
        Parameters={ROOT_STACK_NAME.title: Ref(AWS_STACK_NAME)},
    )
  File "/Users/user/.local/pipx/venvs/ecs-composex/lib/python3.13/site-packages/ecs_composex/vpc/vpc_stack.py", line 371, in __init__
    self.vpc_resource.lookup_vpc()
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/Users/user/.local/pipx/venvs/ecs-composex/lib/python3.13/site-packages/ecs_composex/vpc/vpc_stack.py", line 173, in lookup_vpc
    vpc_settings = lookup_x_vpc_settings(self)
  File "/Users/user/.local/pipx/venvs/ecs-composex/lib/python3.13/site-packages/ecs_composex/vpc/vpc_aws.py", line 106, in lookup_x_vpc_settings
    vpc_arn = find_aws_resource_arn_from_tags_api(
        vpc_resource.lookup[VPC_ID.title],
    ...<2 lines>...
        allow_multi=False,
    )
  File "/Users/user/.local/pipx/venvs/ecs-composex/lib/python3.13/site-packages/ecs_composex/common/aws.py", line 237, in find_aws_resource_arn_from_tags_api
    return handle_search_results(
        arns, name, res_types, aws_resource_search, allow_multi=allow_multi
    )
  File "/Users/user/.local/pipx/venvs/ecs-composex/lib/python3.13/site-packages/ecs_composex/common/aws.py", line 176, in handle_search_results
    raise LookupError(
    ...<3 lines>...
    )
LookupError: ('No resources were found with the provided tags and information', None, 'ec2:vpc')

Output (if there are two tagged VPCs but using "Identifier"):

LookupError: ('More than one resource None:ec2:vpc was found with the current tags.Found', ['arn:aws:ec2:us-east-2:123:vpc/vpc-xxx', 'arn:aws:ec2:us-east-2:123:vpc/vpc-yyy'])

Not sure if this is a bug or feature. Docs are unclear.

Lookups docs suggest this should be possible:

Identifier | The Service resource ID (i.e. vpc-abcd1234)

x-vpc docs nor test cases ever mention this field

Reasoning:

• Network resources shared via RAM shares do not have Tags by default (unless some effort is made), ID/Arn lookup would be great.

Describe the solution you'd like
Allow ID/ARN lookup

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

[JohnPreston]

Hello :)
So this would use the Identifier as the tag key and : vpc-xxx as the tag value to use. So if that pair is not set, you won't find it. Lookup works using Tags as the mean to find the resources.

x-vpc:
  Lookup:
    VpcId:
      Tags:
        Identifier: vpc-xxx

Should work and is the recommended way to use Lookup (finds items via tags api). The tags you select here for the VPC in particular though must resolve to a single resource. For Subnets, the tags can match up to multiple subnets of course.

There is a way to give the ARN indeed of the resource you are looking for but that's not the recommended way. Let me dig it though.

[jakub-qg]

Thank you, however that syntax is not working.

x-vpc:
  Lookup:
    VpcId:
      Tags:
        Identifier: vpc-030xxxxxxxxxx

Also tried:

      Tags:
        - Identifier: vpc-030xxxxxxxxxx

LookupError: ('No resources were found with the provided tags and information', None, 'ec2:vpc')

Well, at least I finally tagged the network resources!

[JohnPreston]

Hmm okay. Let me investigate but, weird...

…

On Tue, Mar 25, 2025, 13:13 jakub-qg ***@***.***> wrote: Thank you, however that syntax is not working. x-vpc: Lookup: VpcId: Tags: Identifier: vpc-030xxxxxxxxxx Also tried: Tags: - Identifier: vpc-030xxxxxxxxxx LookupError: ('No resources were found with the provided tags and information', None, 'ec2:vpc') Well, at least I finally tagged the network resources! — Reply to this email directly, view it on GitHub <#798 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAJNZNWB3MWEXKUA7KU5ARL2WFB5TAVCNFSM6AAAAABZXIWGKGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDONJRGIZDANZSGU> . You are receiving this because you were assigned.Message ID: ***@***.***> [image: jakub-qg]*jakub-qg* left a comment (compose-x/ecs_composex#798) <#798 (comment)> Thank you, however that syntax is not working. x-vpc: Lookup: VpcId: Tags: Identifier: vpc-030xxxxxxxxxx Also tried: Tags: - Identifier: vpc-030xxxxxxxxxx LookupError: ('No resources were found with the provided tags and information', None, 'ec2:vpc') Well, at least I finally tagged the network resources! — Reply to this email directly, view it on GitHub <#798 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAJNZNWB3MWEXKUA7KU5ARL2WFB5TAVCNFSM6AAAAABZXIWGKGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDONJRGIZDANZSGU> . You are receiving this because you were assigned.Message ID: ***@***.***>

[JohnPreston]

Thank you, however that syntax is not working.

x-vpc:
  Lookup:
    VpcId:
      Tags:
        Identifier: vpc-030xxxxxxxxxx

Also tried:

      Tags:
        - Identifier: vpc-030xxxxxxxxxx

LookupError: ('No resources were found with the provided tags and information', None, 'ec2:vpc')

Well, at least I finally tagged the network resources!

Well so, if your network resources are not tagged, this explains this error message, there is nothing to find :D

[jakub-qg]

To clarify, the Identifier tag didn't work for looking up the vpc by name. I'd expect such functionality to exist, this is a common pattern we use with e.g. Terraform (we pass the network res names/ids via variables)

[JohnPreston]

Lookup uses the Tags as the primary focal point to find resources. It uses the AWS tagging API rather than, i.e. go to EC2, describe all VPCs, and find the one that matches all the Tags. The excellent Tagging Resource API does that better and faster. Then yes, once compose-x has the ARN of the resource, doing all sorts of further describes on the resource itself, becomes much easier.

Resources in AWS have an ID that is regional, but the real identifier that works all the time is the ARN. Therefore, in the code you can pass the ARN directly. But not the shorthand ID which is regional. I haven't yet had the case of multi-region use-cases but ARNs just remove the guessing altogether.

"Name" is not a thing (not consistently for all resources anyways, and I always recommend not to give things a name, let CFN generate it), there is a massively common pattern used by AWS customers to use the tag key Name to identify resources.

So, in Lookup, if you did

x-vpc:
  Lookup:
    VpcId:
      Arn: arn:aws:ec2:region::account/vpc/vpc-id # on top of my head

Then that would work, as you are saying "Use the ARN, don't bother looking up (See https://github.com/compose-x/ecs_composex/blob/main/ecs_composex/compose/x_resources/__init__.py#L249)

Otherwise, yes, your resources, network or otherwise, must be tagged appropriately for this to work. I will check in the docs, as this has had a few evolution and however the docs contain the JSON schemas of what is expected to be set by the user, this does not yet render into the most human friendly docs to go through.

I hope this sheds some light on this. I will spin a VPC up etc. and give you a working example of what works with the latest version of the code.

[mikejgray]

I wasn't able to get the lookup syntax working with just giving it an Arn, unfortunately, since that would be my preference. But I did get this tag syntax working:

x-vpc:
  Lookup:
    VpcId:
      Tags:
        Name: "Customer Dashboard"
        env: prod
    PublicSubnets:
      Tags:
        "aws-cdk:subnet-type": Public
        env: prod
    AppSubnets:
      Tags:
        "aws-cdk:subnet-type": Private
        env: prod
    StorageSubnets:
      Tags:
        "aws-cdk:subnet-type": Private
        env: prod

Some quirk in the tag API required that I include both tags for the VPC even though only one VPC has that particular Name tag. Not a bug in this tool, but apparently in AWS.

Throwing in my +1 for being able to reference explicitly by a VPC ID!

[JohnPreston]

Hello @mikejgray
FYI, I have added the functionality for the vpc lookup via ID or ARN, although I still highly recommend to use Tags in priority, otherwise ARN over ID.
Let me know if you get some time to test the branch otherwise I will merge it by end of June. Thanks!