[vpc] [rhel-8.4] cloud-init failure during catalog rhel-8.4 provisioning

[fernflower]

Reproduced with catalog rhel-8.4 one.

ibmcloud is instance-create ivasilev-rhel-8-4-from-catalog-1 testing-farm-us-east-vpc us-east-2 bz2-2x8 0767-0458a498-e138-4e3a-b466-b74a9ac1cf38 --image ibm-l1bm-rhel8-4-minimal-s390x-byol-2 --allow-ip-spoofing=false  --keys ivasilev 

The command doesn't issue any errors and vm gets to running state, but it is never reachable by ssh.
Rebooting the machine while having the console attached reveals the following cloud-init trace

[    3.622460] cloud-init[1005]: Cloud-init v. 20.3-10.el8_4.5 running 'init' at Wed, 27 Nov 2024 20:03:59 +0000. Up 3.57 seconds.
[    3.622646] cloud-init[1005]: ci-info: +++++++++++++++++++++++Net device info++++++++++++++++++++++++
[    3.622765] cloud-init[1005]: ci-info: +--------+------+-----------+-----------+-------+------------+
[    3.622838] cloud-init[1005]: ci-info: | Device |  Up  |  Address  |    Mask   | Scope | Hw-Address |
[    3.622914] cloud-init[1005]: ci-info: +--------+------+-----------+-----------+-------+------------+
[    3.622998] cloud-init[1005]: ci-info: |   lo   | True | 127.0.0.1 | 255.0.0.0 |  host |     .      |
[    3.623126] cloud-init[1005]: ci-info: |   lo   | True |  ::1/128  |     .     |  host |     .      |
[    3.623208] cloud-init[1005]: ci-info: +--------+------+-----------+-----------+-------+------------+
[    3.623287] cloud-init[1005]: ci-info: +++++++++++++++++++Route IPv6 info+++++++++++++++++++
[    3.623403] cloud-init[1005]: ci-info: +-------+-------------+---------+-----------+-------+
[    3.623582] cloud-init[1005]: ci-info: | Route | Destination | Gateway | Interface | Flags |
[    3.623661] cloud-init[1005]: ci-info: +-------+-------------+---------+-----------+-------+
[    3.623736] cloud-init[1005]: ci-info: +-------+-------------+---------+-----------+-------+
[    3.623912] cloud-init[1005]: 2024-11-27 20:03:59,395 - util.py[WARNING]: No instance datasource found! Likely bad things to come!
[FAILED] Failed to start Initial cloud-init job (metadata service crawler).
See 'systemctl status cloud-init.service' for details.
         Starting Permit User Sessions...
[  OK  ] Reached target Cloud-config availability.
[  OK  ] Reached target Network is Online.
         Starting System Logging Service...
         Starting Crash recovery kernel arming...
         Starting Apply the settings specified in cloud-config...
         Starting OpenSSH server daemon...
[  OK  ] Started Permit User Sessions.
[  OK  ] Started Command Scheduler.
[  OK  ] Started Job spooling tools.
[  OK  ] Started Serial Getty on ttysclp0.
[  OK  ] Started Getty on tty1.
[  OK  ] Reached target Login Prompts.
[  OK  ] Started OpenSSH server daemon.
[  OK  ] Started System Logging Service.
[  OK  ] Reached target Multi-User System.
         Starting Update UTMP about System Runlevel Changes...
[  OK  ] Started Update UTMP about System Runlevel Changes.
[    3.971802] cloud-init[1055]: Can not apply stage config, no datasource found! Likely bad things to come!
[    3.971990] cloud-init[1055]: ------------------------------------------------------------
[    3.972041] cloud-init[1055]: Traceback (most recent call last):
[    3.972092] cloud-init[1055]:   File "/usr/lib/python3.6/site-packages/cloudinit/cmd/main.py", line 485, in main_modules
[    3.972140] cloud-init[1055]:     init.fetch(existing="trust")
[    3.972199] cloud-init[1055]:   File "/usr/lib/python3.6/site-packages/cloudinit/stages.py", line 351, in fetch
[    3.972249] cloud-init[1055]:     return self._get_data_source(existing=existing)
[    3.972289] cloud-init[1055]:   File "/usr/lib/python3.6/site-packages/cloudinit/stages.py", line 261, in _get_data_source
[    3.972342] cloud-init[1055]:     pkg_list, self.reporter)
[    3.972397] cloud-init[1055]:   File "/usr/lib/python3.6/site-packages/cloudinit/sources/__init__.py", line 778, in find_source
[    3.972446] cloud-init[1055]:     raise DataSourceNotFoundException(msg)
[    3.972502] cloud-init[1055]: cloudinit.sources.DataSourceNotFoundException: Did not find any data source, searched classes: ()
[    3.972545] cloud-init[1055]: ------------------------------------------------------------
[FAILED] Failed to start Apply the settings specified in cloud-config.
See 'systemctl status cloud-config.service' for details.
         Starting Execute cloud user/final scripts...
[  OK  ] Started Crash recovery kernel arming.
[    4.303195] cloud-init[1283]: Can not apply stage final, no datasource found! Likely bad things to come!
[    4.303393] cloud-init[1283]: ------------------------------------------------------------
[    4.303457] cloud-init[1283]: Traceback (most recent call last):
[    4.303519] cloud-init[1283]:   File "/usr/lib/python3.6/site-packages/cloudinit/cmd/main.py", line 485, in main_modules
[    4.303585] cloud-init[1283]:     init.fetch(existing="trust")
[    4.303644] cloud-init[1283]:   File "/usr/lib/python3.6/site-packages/cloudinit/stages.py", line 351, in fetch
[    4.303722] cloud-init[1283]:     return self._get_data_source(existing=existing)
[    4.303787] cloud-init[1283]:   File "/usr/lib/python3.6/site-packages/cloudinit/stages.py", line 261, in _get_data_source
[    4.303863] cloud-init[1283]:     pkg_list, self.reporter)
[    4.303945] cloud-init[1283]:   File "/usr/lib/python3.6/site-packages/cloudinit/sources/__init__.py", line 778, in find_source
[    4.304010] cloud-init[1283]:     raise DataSourceNotFoundException(msg)
[    4.304074] cloud-init[1283]: cloudinit.sources.DataSourceNotFoundException: Did not find any data source, searched classes: ()
[    4.304115] cloud-init[1283]: ------------------------------------------------------------
[FAILED] Failed to start Execute cloud user/final scripts.
See 'systemctl status cloud-final.service' for details.

[steveclay]

@sreekarbvibm do you have any thoughts regarding this issue?

[sreekarbvibm]

Hey @steveclay
From looking at the above log, it doesn't seems to be an issue from the VPC CLI plugin but rather network/compute team might be the right stake holders to address the issue.

Thank you