Integration tests are very flaky #186
Description
Bug Description
I have noticed that the integration tests test_charm.py and test_upgrade.py fail quite often as the github-image-builder subprocess command fails to run. For example, in this CI run, exceptions.BuilderRunError is raised.
To Reproduce
Rerun the CI multiple times.
Environment
Relevant log output
Traceback (most recent call last):
File "/var/lib/juju/agents/unit-image-builder-operator-531c043c-0/charm/src/builder.py", line 527, in _run
stdout = subprocess.check_output( # pylint: disable=unexpected-keyword-arg # nosec:B603
File "/usr/lib/python3.10/subprocess.py", line 421, in check_output
return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
File "/usr/lib/python3.10/subprocess.py", line 526, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/usr/bin/run-one', '/usr/bin/sudo', '--preserve-env', '/home/ubuntu/.local/bin/github-runner-image-builder', '--os-cloud', 'builder', 'run', 'image-builder-operator-531c043c-noble-x64', '--flavor', 'shared.xsmall', '--keep-revisions', '5', '--network', 'net_stg-ps7-github-runner-test-amd64', '--prefix', 'image-builder-operator-531c043c', '--upload-clouds', 'stg-ps7-github-runner-test-amd64_project_stg-ps7-github-runner-test-amd64', '--arch', 'x64', '--base-image', 'noble', '--script-url', 'https://raw.githubusercontent.com/canonical/github-runner-image-builder/refs/heads/main/tests/integration/testdata/test_script.sh', '--proxy', 'egress.ps7.internal:3128']' returned non-zero exit status 1.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/lib/python3.10/multiprocessing/pool.py", line 125, in worker
result = (True, func(*args, **kwds))
File "/usr/lib/python3.10/multiprocessing/pool.py", line 48, in mapstar
return list(map(*args))
File "/var/lib/juju/agents/unit-image-builder-operator-531c043c-0/charm/venv/tenacity/__init__.py", line 338, in wrapped_f
return copy(f, *args, **kw)
File "/var/lib/juju/agents/unit-image-builder-operator-531c043c-0/charm/venv/tenacity/__init__.py", line 477, in __call__
do = self.iter(retry_state=retry_state)
File "/var/lib/juju/agents/unit-image-builder-operator-531c043c-0/charm/venv/tenacity/__init__.py", line 378, in iter
result = action(retry_state)
File "/var/lib/juju/agents/unit-image-builder-operator-531c043c-0/charm/venv/tenacity/__init__.py", line 420, in exc_check
raise retry_exc.reraise()
File "/var/lib/juju/agents/unit-image-builder-operator-531c043c-0/charm/venv/tenacity/__init__.py", line 187, in reraise
raise self.last_attempt.result()
File "/usr/lib/python3.10/concurrent/futures/_base.py", line 451, in result
return self.__get_result()
File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
raise self._exception
File "/var/lib/juju/agents/unit-image-builder-operator-531c043c-0/charm/venv/tenacity/__init__.py", line 480, in __call__
result = fn(*args, **kwargs)
File "/var/lib/juju/agents/unit-image-builder-operator-531c043c-0/charm/src/builder.py", line 558, in _run
raise BuilderRunError from exc
exceptions.BuilderRunError
Its always this error all the time.Additional context
No response