Skip to content

fix(ps80-binary-tarball): tear down molecule EC2 instances on failure#1138

Open
nogueiraanderson wants to merge 1 commit into
Percona-QA:masterfrom
nogueiraanderson:fix-ps80-tarball-molecule-teardown
Open

fix(ps80-binary-tarball): tear down molecule EC2 instances on failure#1138
nogueiraanderson wants to merge 1 commit into
Percona-QA:masterfrom
nogueiraanderson:fix-ps80-tarball-molecule-teardown

Conversation

@nogueiraanderson
Copy link
Copy Markdown

Bug

  • The ps80-binary-tarball molecule scenarios never tear down their EC2 instances: each molecule/<os>/molecule.yml has destroy commented out of test_sequence and no destroy playbook wired, and playbooks/destroy.yml does not exist.
  • A failing OS run therefore leaves its instance running. 48 on-demand VMs leaked across builds 191-198 of the test-ps-binary-tarball job (6 per build) and were terminated manually.

Fix

  • Add playbooks/destroy.yml that terminates by the unique Name tag (<os>-${BUILD_NUMBER}-${JOB_NAME}, scoped to the platform region). It reads molecule_yml.platforms[].name, the same value create.yml tags as Name, so teardown works even when molecule instance-config was never written (early create/SSH failures).
  • Re-enable destroy as the final step of test_sequence and add destroy_sequence across all 9 EC2 scenarios, so molecule tears down on success and on verify/converge failure.

Notes

  • Termination keys on the run-unique Name, so it cannot affect another build or job. The pipeline post { always } molecule destroy is the backstop and now resolves to a working destroy playbook.

…nce leak

The ps80-binary-tarball molecule scenarios launch on-demand EC2 instances via
the create playbook but do not reliably tear them down, leaking tagged
on-demand VMs per build of the test-ps-binary-tarball job:

- Several scenarios (oracle-8/9, rhel-10, debian-12, ubuntu-jammy) had destroy
  commented out of both test_sequence and destroy_sequence and no destroy
  playbook wired, so 'molecule test' never destroyed and the post-build
  'molecule destroy' had no playbook to run.
- The scenarios that did wire a destroy pointed at destroy-noble-arm.yml, which
  terminates purely from molecule instance-config (instance_id); when that file
  is absent (a separate post 'molecule destroy' invocation with a fresh
  ephemeral dir) it sets skip_instances and terminates nothing.

Fix:
- Add playbooks/destroy.yml that terminates this run's instances
  deterministically by the unique Name tag (osprefix-BUILD_NUMBER-JOB_NAME set
  by the create playbook), independent of molecule instance-config.
- Wire provisioner.playbooks.destroy = ../../playbooks/destroy.yml uniformly in
  all 9 ec2-driver OS scenarios.
- Re-enable destroy as the last step of test_sequence and add
  destroy_sequence: [destroy] in all 9 scenarios.
Copy link
Copy Markdown
Collaborator

@kaushikpuneet07 kaushikpuneet07 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to Me

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants