[Failing Test]: GroupByEncryptedKeyTest.test_gbk_with_gbek_option_fake_secret_manager_roundtrips is flaky

[tvalentyn]

What happened?

Sample failure:

...
2025-10-15T20:57:50.7556019Z E         File "/runner/_work/beam/beam/sdks/python/test-suites/tox/py312/build/srcs/sdks/python/target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/apache_beam/transforms/util.py", line 408, in get_secret_bytes
2025-10-15T20:57:50.7556900Z E           raise RuntimeError(
2025-10-15T20:57:50.7558261Z E       RuntimeError: Failed to retrieve secret bytes for secret projects/apache-beam-testing/secrets/gbek_util_secret_tests_1015_205647_py312/versions/latest with exception 404 Secret [projects/844138762903/secrets/gbek_util_secret_tests_1015_205647_py312] not found or has no versions. [while running 'assert_that/Group/CoGroupByKeyImpl/GroupByKey/Group by encrypted key/ParDo(_EncryptMessage)']
2025-10-15T20:57:50.7559537Z 
2025-10-15T20:57:50.7560240Z target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/apache_beam/runners/portability/portable_runner.py:568: RuntimeError

Issue Failure

Failure: Test is flaky

Issue Priority

Priority: 1 (unhealthy code / failing or flaky postcommit so we cannot be sure the product is healthy)

Issue Components

• Component: Python SDK
• Component: Java SDK
• Component: Go SDK
• Component: Typescript SDK
• Component: IO connector
• Component: Beam YAML
• Component: Beam examples
• Component: Beam playground
• Component: Beam katas
• Component: Website
• Component: Infrastructure
• Component: Spark Runner
• Component: Flink Runner
• Component: Samza Runner
• Component: Twister2 Runner
• Component: Hazelcast Jet Runner
• Component: Google Cloud Dataflow Runner

[tvalentyn]

Full logs:

Details

2025-10-15T20:57:50.7394290Z _ GroupByEncryptedKeyTest.test_gbk_with_gbek_option_fake_secret_manager_roundtrips _
2025-10-15T20:57:50.7435633Z [gw0] linux -- Python 3.12.10 /runner/_work/beam/beam/sdks/python/test-suites/tox/py312/build/srcs/sdks/python/target/.tox-py312-cloud/py312-cloud/bin/python
2025-10-15T20:57:50.7436919Z
2025-10-15T20:57:50.7437725Z self = <apache_beam.transforms.util_test.GroupByEncryptedKeyTest testMethod=test_gbk_with_gbek_option_fake_secret_manager_roundtrips>
2025-10-15T20:57:50.7438641Z
2025-10-15T20:57:50.7439007Z @unittest.skipIf(secretmanager is None, 'GCP dependencies are not installed')
2025-10-15T20:57:50.7439740Z def test_gbk_with_gbek_option_fake_secret_manager_roundtrips(self):
2025-10-15T20:57:50.7440587Z options = PipelineOptions()
2025-10-15T20:57:50.7441075Z options.view_as(SetupOptions).gbek = self.secret_option
2025-10-15T20:57:50.7441512Z
2025-10-15T20:57:50.7441870Z > with beam.Pipeline(options=options) as pipeline:
2025-10-15T20:57:50.7442319Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-10-15T20:57:50.7442572Z
2025-10-15T20:57:50.7442739Z apache_beam/transforms/util_test.py:371:
2025-10-15T20:57:50.7443233Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
2025-10-15T20:57:50.7444044Z target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/apache_beam/pipeline.py:672: in exit
2025-10-15T20:57:50.7444815Z self.result.wait_until_finish()
2025-10-15T20:57:50.7445320Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
2025-10-15T20:57:50.7445656Z
2025-10-15T20:57:50.7446118Z self = <apache_beam.runners.portability.portable_runner.PipelineResult object at 0x7a457c2c7cb0>
2025-10-15T20:57:50.7446824Z duration = None
2025-10-15T20:57:50.7447012Z
2025-10-15T20:57:50.7447204Z def wait_until_finish(self, duration=None):
2025-10-15T20:57:50.7447630Z """
2025-10-15T20:57:50.7448183Z :param duration: The maximum time in milliseconds to wait for the result of
2025-10-15T20:57:50.7449014Z the execution. If None or zero, will wait until the pipeline finishes.
2025-10-15T20:57:50.7449980Z :return: The result of the pipeline, i.e. PipelineResult.
2025-10-15T20:57:50.7451701Z """
2025-10-15T20:57:50.7452069Z def read_messages() -> None:
2025-10-15T20:57:50.7452509Z previous_state = -1
2025-10-15T20:57:50.7452966Z for message in self._message_stream:
2025-10-15T20:57:50.7453500Z if message.HasField('message_response'):
2025-10-15T20:57:50.7454014Z logging.log(
2025-10-15T20:57:50.7454531Z MESSAGE_LOG_LEVELS[message.message_response.importance],
2025-10-15T20:57:50.7455116Z "%s",
2025-10-15T20:57:50.7455526Z message.message_response.message_text)
2025-10-15T20:57:50.7456008Z else:
2025-10-15T20:57:50.7456424Z current_state = message.state_response.state
2025-10-15T20:57:50.7456914Z if current_state != previous_state:
2025-10-15T20:57:50.7457325Z _LOGGER.info(
2025-10-15T20:57:50.7457701Z "Job state changed to %s",
2025-10-15T20:57:50.7458255Z self.runner_api_state_to_pipeline_state(current_state))
2025-10-15T20:57:50.7458839Z previous_state = current_state
2025-10-15T20:57:50.7459279Z self._messages.append(message)
2025-10-15T20:57:50.7459678Z
2025-10-15T20:57:50.7460216Z message_thread = threading.Thread(
2025-10-15T20:57:50.7460826Z target=read_messages, name='wait_until_finish_read')
2025-10-15T20:57:50.7461350Z message_thread.daemon = True
2025-10-15T20:57:50.7461779Z message_thread.start()
2025-10-15T20:57:50.7462147Z
2025-10-15T20:57:50.7462453Z if duration:
2025-10-15T20:57:50.7463167Z state_thread = threading.Thread(
2025-10-15T20:57:50.7463809Z target=functools.partial(self._observe_state, message_thread),
2025-10-15T20:57:50.7464484Z name='wait_until_finish_state_observer')
2025-10-15T20:57:50.7464988Z state_thread.daemon = True
2025-10-15T20:57:50.7465392Z state_thread.start()
2025-10-15T20:57:50.7465766Z start_time = time.time()
2025-10-15T20:57:50.7466192Z duration_secs = duration / 1000
2025-10-15T20:57:50.7466709Z while (time.time() - start_time < duration_secs and
2025-10-15T20:57:50.7467252Z state_thread.is_alive()):
2025-10-15T20:57:50.7467660Z time.sleep(1)
2025-10-15T20:57:50.7468019Z else:
2025-10-15T20:57:50.7468358Z self._observe_state(message_thread)
2025-10-15T20:57:50.7468766Z
2025-10-15T20:57:50.7469095Z if self._runtime_exception:
2025-10-15T20:57:50.7469532Z > raise self._runtime_exception
2025-10-15T20:57:50.7470652Z E RuntimeError: Pipeline job-057 failed in state FAILED: bundle inst008 stage-013 failed:Traceback (most recent call last):
2025-10-15T20:57:50.7472436Z E File "/runner/_work/beam/beam/sdks/python/test-suites/tox/py312/build/srcs/sdks/python/target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/apache_beam/transforms/util.py", line 403, in get_secret_bytes
2025-10-15T20:57:50.7473981Z E response = client.access_secret_version(
2025-10-15T20:57:50.7474513Z E ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-10-15T20:57:50.7476425Z E File "/runner/_work/beam/beam/sdks/python/test-suites/tox/py312/build/srcs/sdks/python/target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/google/cloud/secretmanager_v1/services/secret_manager_service/client.py", line 1857, in access_secret_version
2025-10-15T20:57:50.7478394Z E response = rpc(
2025-10-15T20:57:50.7478771Z E ^^^^
2025-10-15T20:57:50.7480492Z E File "/runner/_work/beam/beam/sdks/python/test-suites/tox/py312/build/srcs/sdks/python/target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/google/api_core/gapic_v1/method.py", line 131, in call
2025-10-15T20:57:50.7482069Z E return wrapped_func(*args, **kwargs)
2025-10-15T20:57:50.7482563Z E ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-10-15T20:57:50.7484154Z E File "/runner/_work/beam/beam/sdks/python/test-suites/tox/py312/build/srcs/sdks/python/target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/google/api_core/retry/retry_unary.py", line 294, in retry_wrapped_func
2025-10-15T20:57:50.7486136Z E return retry_target(
2025-10-15T20:57:50.7486535Z E ^^^^^^^^^^^^^
2025-10-15T20:57:50.7488046Z E File "/runner/_work/beam/beam/sdks/python/test-suites/tox/py312/build/srcs/sdks/python/target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/google/api_core/retry/retry_unary.py", line 156, in retry_target
2025-10-15T20:57:50.7489702Z E next_sleep = _retry_error_helper(
2025-10-15T20:57:50.7490691Z E ^^^^^^^^^^^^^^^^^^^^
2025-10-15T20:57:50.7492277Z E File "/runner/_work/beam/beam/sdks/python/test-suites/tox/py312/build/srcs/sdks/python/target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/google/api_core/retry/retry_base.py", line 214, in _retry_error_helper
2025-10-15T20:57:50.7493958Z E raise final_exc from source_exc
2025-10-15T20:57:50.7495532Z E File "/runner/_work/beam/beam/sdks/python/test-suites/tox/py312/build/srcs/sdks/python/target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/google/api_core/retry/retry_unary.py", line 147, in retry_target
2025-10-15T20:57:50.7497104Z E result = target()
2025-10-15T20:57:50.7497535Z E ^^^^^^^^
2025-10-15T20:57:50.7499335Z E File "/runner/_work/beam/beam/sdks/python/test-suites/tox/py312/build/srcs/sdks/python/target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/google/api_core/timeout.py", line 130, in func_with_timeout
2025-10-15T20:57:50.7501238Z E return func(*args, **kwargs)
2025-10-15T20:57:50.7501721Z E ^^^^^^^^^^^^^^^^^^^^^
2025-10-15T20:57:50.7503310Z E File "/runner/_work/beam/beam/sdks/python/test-suites/tox/py312/build/srcs/sdks/python/target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/google/api_core/grpc_helpers.py", line 77, in error_remapped_callable
2025-10-15T20:57:50.7505009Z E raise exceptions.from_grpc_error(exc) from exc
2025-10-15T20:57:50.7506191Z E google.api_core.exceptions.NotFound: 404 Secret [projects/844138762903/secrets/gbek_util_secret_tests_1015_205647_py312] not found or has no versions.
2025-10-15T20:57:50.7507281Z E
2025-10-15T20:57:50.7507804Z E During handling of the above exception, another exception occurred:
2025-10-15T20:57:50.7508431Z E
2025-10-15T20:57:50.7508787Z E Traceback (most recent call last):
2025-10-15T20:57:50.7509700Z E File "apache_beam/runners/common.py", line 1562, in apache_beam.runners.common.DoFnRunner._invoke_lifecycle_method
2025-10-15T20:57:50.7510824Z E lifecycle_method()
2025-10-15T20:57:50.7511621Z E File "apache_beam/runners/common.py", line 602, in apache_beam.runners.common.DoFnInvoker.invoke_setup
2025-10-15T20:57:50.7512572Z E self.signature.setup_lifecycle_method.method_value()
2025-10-15T20:57:50.7514166Z E File "/runner/_work/beam/beam/sdks/python/test-suites/tox/py312/build/srcs/sdks/python/target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/apache_beam/transforms/util.py", line 428, in setup
2025-10-15T20:57:50.7515796Z E self._hmac_key = self.hmac_key_secret.get_secret_bytes()
2025-10-15T20:57:50.7516405Z E ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-10-15T20:57:50.7518059Z E File "/runner/_work/beam/beam/sdks/python/test-suites/tox/py312/build/srcs/sdks/python/target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/apache_beam/transforms/util.py", line 408, in get_secret_bytes
2025-10-15T20:57:50.7519671Z E raise RuntimeError(
2025-10-15T20:57:50.7521972Z E RuntimeError: Failed to retrieve secret bytes for secret projects/apache-beam-testing/secrets/gbek_util_secret_tests_1015_205647_py312/versions/latest with exception 404 Secret [projects/844138762903/secrets/gbek_util_secret_tests_1015_205647_py312] not found or has no versions.
2025-10-15T20:57:50.7524112Z E
2025-10-15T20:57:50.7524629Z E During handling of the above exception, another exception occurred:
2025-10-15T20:57:50.7525261Z E
2025-10-15T20:57:50.7525627Z E Traceback (most recent call last):
2025-10-15T20:57:50.7527274Z E File "/runner/_work/beam/beam/sdks/python/test-suites/tox/py312/build/srcs/sdks/python/target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/apache_beam/runners/worker/sdk_worker.py", line 316, in _execute
2025-10-15T20:57:50.7528889Z E response = task()
2025-10-15T20:57:50.7529312Z E ^^^^^^
2025-10-15T20:57:50.7531121Z E File "/runner/_work/beam/beam/sdks/python/test-suites/tox/py312/build/srcs/sdks/python/target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/apache_beam/runners/worker/sdk_worker.py", line 390, in
2025-10-15T20:57:50.7532913Z E lambda: self.create_worker().do_instruction(request), request)
2025-10-15T20:57:50.7533552Z E ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-10-15T20:57:50.7534958Z E File "/runner/_work/beam/beam/sdks/python/test-suites/tox/py312/build/srcs/sdks/python/target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/apache_beam/runners/worker/sdk_worker.py", line 666, in do_instruction
2025-10-15T20:57:50.7536300Z E return getattr(self, request_type)(
2025-10-15T20:57:50.7536761Z E ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-10-15T20:57:50.7538590Z E File "/runner/_work/beam/beam/sdks/python/test-suites/tox/py312/build/srcs/sdks/python/target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/apache_beam/runners/worker/sdk_worker.py", line 697, in process_bundle
2025-10-15T20:57:50.7540459Z E bundle_processor = self.bundle_processor_cache.get(
2025-10-15T20:57:50.7540980Z E ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-10-15T20:57:50.7541873Z E File "/runner/_work/beam/beam/sdks/python/test-suites/tox/py312/build/srcs/sdks/python/target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/apache_beam/runners/worker/sdk_worker.py", line 517, in get
2025-10-15T20:57:50.7542806Z E processor = bundle_processor.BundleProcessor(
2025-10-15T20:57:50.7543152Z E ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-10-15T20:57:50.7544106Z E File "/runner/_work/beam/beam/sdks/python/test-suites/tox/py312/build/srcs/sdks/python/target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/apache_beam/runners/worker/bundle_processor.py", line 1135, in init
2025-10-15T20:57:50.7545095Z E op.setup(self.data_sampler)
2025-10-15T20:57:50.7545643Z E File "apache_beam/runners/worker/operations.py", line 875, in apache_beam.runners.worker.operations.DoOperation.setup
2025-10-15T20:57:50.7546183Z E with self.scoped_start_state:
2025-10-15T20:57:50.7546708Z E File "apache_beam/runners/worker/operations.py", line 925, in apache_beam.runners.worker.operations.DoOperation.setup
2025-10-15T20:57:50.7547257Z E self.dofn_runner.setup()
2025-10-15T20:57:50.7547707Z E File "apache_beam/runners/common.py", line 1568, in apache_beam.runners.common.DoFnRunner.setup
2025-10-15T20:57:50.7548249Z E self._invoke_lifecycle_method(self.do_fn_invoker.invoke_setup)
2025-10-15T20:57:50.7548859Z E File "apache_beam/runners/common.py", line 1564, in apache_beam.runners.common.DoFnRunner._invoke_lifecycle_method
2025-10-15T20:57:50.7549375Z E self._reraise_augmented(exn)
2025-10-15T20:57:50.7550091Z E File "apache_beam/runners/common.py", line 1609, in apache_beam.runners.common.DoFnRunner._reraise_augmented
2025-10-15T20:57:50.7550766Z E raise new_exn
2025-10-15T20:57:50.7551237Z E File "apache_beam/runners/common.py", line 1562, in apache_beam.runners.common.DoFnRunner._invoke_lifecycle_method
2025-10-15T20:57:50.7551747Z E lifecycle_method()
2025-10-15T20:57:50.7552418Z E File "apache_beam/runners/common.py", line 602, in apache_beam.runners.common.DoFnInvoker.invoke_setup
2025-10-15T20:57:50.7552982Z E self.signature.setup_lifecycle_method.method_value()
2025-10-15T20:57:50.7553888Z E File "/runner/_work/beam/beam/sdks/python/test-suites/tox/py312/build/srcs/sdks/python/target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/apache_beam/transforms/util.py", line 428, in setup
2025-10-15T20:57:50.7554783Z E self._hmac_key = self.hmac_key_secret.get_secret_bytes()
2025-10-15T20:57:50.7555131Z E ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-10-15T20:57:50.7556019Z E File "/runner/_work/beam/beam/sdks/python/test-suites/tox/py312/build/srcs/sdks/python/target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/apache_beam/transforms/util.py", line 408, in get_secret_bytes
2025-10-15T20:57:50.7556900Z E raise RuntimeError(
2025-10-15T20:57:50.7558261Z E RuntimeError: Failed to retrieve secret bytes for secret projects/apache-beam-testing/secrets/gbek_util_secret_tests_1015_205647_py312/versions/latest with exception 404 Secret [projects/844138762903/secrets/gbek_util_secret_tests_1015_205647_py312] not found or has no versions. [while running 'assert_that/Group/CoGroupByKeyImpl/GroupByKey/Group by encrypted key/ParDo(_EncryptMessage)']
2025-10-15T20:57:50.7559537Z
2025-10-15T20:57:50.7560240Z target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/apache_beam/runners/portability/portable_runner.py:568: RuntimeError
2025-10-15T20:57:50.7561370Z ----------------------------- Captured stderr call -----------------------------
2025-10-15T20:57:50.7561748Z Traceback (most recent call last):
2025-10-15T20:57:50.7562723Z File "/runner/_work/beam/beam/sdks/python/test-suites/tox/py312/build/srcs/sdks/python/target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/apache_beam/transforms/util.py", line 403, in get_secret_bytes
2025-10-15T20:57:50.7563682Z response = client.access_secret_version(
2025-10-15T20:57:50.7563979Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-10-15T20:57:50.7565046Z File "/runner/_work/beam/beam/sdks/python/test-suites/tox/py312/build/srcs/sdks/python/target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/google/cloud/secretmanager_v1/services/secret_manager_service/client.py", line 1857, in access_secret_version
2025-10-15T20:57:50.7566092Z response = rpc(
2025-10-15T20:57:50.7566309Z ^^^^
2025-10-15T20:57:50.7567132Z File "/runner/_work/beam/beam/sdks/python/test-suites/tox/py312/build/srcs/sdks/python/target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/google/api_core/gapic_v1/method.py", line 131, in call
2025-10-15T20:57:50.7568008Z return wrapped_func(*args, **kwargs)
2025-10-15T20:57:50.7568270Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-10-15T20:57:50.7569184Z File "/runner/_work/beam/beam/sdks/python/test-suites/tox/py312/build/srcs/sdks/python/target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/google/api_core/retry/retry_unary.py", line 294, in retry_wrapped_func
2025-10-15T20:57:50.7570702Z return retry_target(
2025-10-15T20:57:50.7570949Z ^^^^^^^^^^^^^
2025-10-15T20:57:50.7571807Z File "/runner/_work/beam/beam/sdks/python/test-suites/tox/py312/build/srcs/sdks/python/target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/google/api_core/retry/retry_unary.py", line 156, in retry_target
2025-10-15T20:57:50.7572701Z next_sleep = _retry_error_helper(
2025-10-15T20:57:50.7572976Z ^^^^^^^^^^^^^^^^^^^^
2025-10-15T20:57:50.7573885Z File "/runner/_work/beam/beam/sdks/python/test-suites/tox/py312/build/srcs/sdks/python/target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/google/api_core/retry/retry_base.py", line 214, in _retry_error_helper
2025-10-15T20:57:50.7574813Z raise final_exc from source_exc
2025-10-15T20:57:50.7575708Z File "/runner/_work/beam/beam/sdks/python/test-suites/tox/py312/build/srcs/sdks/python/target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/google/api_core/retry/retry_unary.py", line 147, in retry_target
2025-10-15T20:57:50.7576862Z result = target()
2025-10-15T20:57:50.7577093Z ^^^^^^^^
2025-10-15T20:57:50.7577927Z File "/runner/_work/beam/beam/sdks/python/test-suites/tox/py312/build/srcs/sdks/python/target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/google/api_core/timeout.py", line 130, in func_with_timeout
2025-10-15T20:57:50.7578799Z return func(*args, **kwargs)
2025-10-15T20:57:50.7579054Z ^^^^^^^^^^^^^^^^^^^^^
2025-10-15T20:57:50.7580104Z File "/runner/_work/beam/beam/sdks/python/test-suites/tox/py312/build/srcs/sdks/python/target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/google/api_core/grpc_helpers.py", line 77, in error_remapped_callable
2025-10-15T20:57:50.7581129Z raise exceptions.from_grpc_error(exc) from exc
2025-10-15T20:57:50.7581790Z google.api_core.exceptions.NotFound: 404 Secret [projects/844138762903/secrets/gbek_util_secret_tests_1015_205647_py312] not found or has no versions.
2025-10-15T20:57:50.7582318Z
2025-10-15T20:57:50.7582514Z During handling of the above exception, another exception occurred:
2025-10-15T20:57:50.7582776Z
2025-10-15T20:57:50.7582881Z Traceback (most recent call last):
2025-10-15T20:57:50.7583426Z File "apache_beam/runners/common.py", line 1562, in apache_beam.runners.common.DoFnRunner._invoke_lifecycle_method
2025-10-15T20:57:50.7583923Z lifecycle_method()
2025-10-15T20:57:50.7584540Z File "apache_beam/runners/common.py", line 602, in apache_beam.runners.common.DoFnInvoker.invoke_setup
2025-10-15T20:57:50.7585077Z self.signature.setup_lifecycle_method.method_value()
2025-10-15T20:57:50.7585987Z File "/runner/_work/beam/beam/sdks/python/test-suites/tox/py312/build/srcs/sdks/python/target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/apache_beam/transforms/util.py", line 428, in setup
2025-10-15T20:57:50.7586879Z self._hmac_key = self.hmac_key_secret.get_secret_bytes()
2025-10-15T20:57:50.7587205Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-10-15T20:57:50.7588128Z File "/runner/_work/beam/beam/sdks/python/test-suites/tox/py312/build/srcs/sdks/python/target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/apache_beam/transforms/util.py", line 408, in get_secret_bytes
2025-10-15T20:57:50.7589020Z raise RuntimeError(
2025-10-15T20:57:50.7590397Z RuntimeError: Failed to retrieve secret bytes for secret projects/apache-beam-testing/secrets/gbek_util_secret_tests_1015_205647_py312/versions/latest with exception 404 Secret [projects/844138762903/secrets/gbek_util_secret_tests_1015_205647_py312] not found or has no versions.
2025-10-15T20:57:50.7591439Z
2025-10-15T20:57:50.7591637Z During handling of the above exception, another exception occurred:
2025-10-15T20:57:50.7591897Z
2025-10-15T20:57:50.7592013Z Traceback (most recent call last):
2025-10-15T20:57:50.7592948Z File "/runner/_work/beam/beam/sdks/python/test-suites/tox/py312/build/srcs/sdks/python/target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/apache_beam/runners/worker/sdk_worker.py", line 316, in _execute
2025-10-15T20:57:50.7593837Z response = task()
2025-10-15T20:57:50.7594059Z ^^^^^^
2025-10-15T20:57:50.7594905Z File "/runner/_work/beam/beam/sdks/python/test-suites/tox/py312/build/srcs/sdks/python/target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/apache_beam/runners/worker/sdk_worker.py", line 390, in
2025-10-15T20:57:50.7595867Z lambda: self.create_worker().do_instruction(request), request)
2025-10-15T20:57:50.7596425Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-10-15T20:57:50.7597501Z File "/runner/_work/beam/beam/sdks/python/test-suites/tox/py312/build/srcs/sdks/python/target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/apache_beam/runners/worker/sdk_worker.py", line 666, in do_instruction
2025-10-15T20:57:50.7598411Z return getattr(self, request_type)(
2025-10-15T20:57:50.7598873Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-10-15T20:57:50.7599963Z File "/runner/_work/beam/beam/sdks/python/test-suites/tox/py312/build/srcs/sdks/python/target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/apache_beam/runners/worker/sdk_worker.py", line 697, in process_bundle
2025-10-15T20:57:50.7601339Z bundle_processor = self.bundle_processor_cache.get(
2025-10-15T20:57:50.7601671Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-10-15T20:57:50.7602568Z File "/runner/_work/beam/beam/sdks/python/test-suites/tox/py312/build/srcs/sdks/python/target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/apache_beam/runners/worker/sdk_worker.py", line 517, in get
2025-10-15T20:57:50.7603515Z processor = bundle_processor.BundleProcessor(
2025-10-15T20:57:50.7603827Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-10-15T20:57:50.7604762Z File "/runner/_work/beam/beam/sdks/python/test-suites/tox/py312/build/srcs/sdks/python/target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/apache_beam/runners/worker/bundle_processor.py", line 1135, in init
2025-10-15T20:57:50.7605690Z op.setup(self.data_sampler)
2025-10-15T20:57:50.7606255Z File "apache_beam/runners/worker/operations.py", line 875, in apache_beam.runners.worker.operations.DoOperation.setup
2025-10-15T20:57:50.7606792Z with self.scoped_start_state:
2025-10-15T20:57:50.7607340Z File "apache_beam/runners/worker/operations.py", line 925, in apache_beam.runners.worker.operations.DoOperation.setup
2025-10-15T20:57:50.7607863Z self.dofn_runner.setup()
2025-10-15T20:57:50.7608813Z File "apache_beam/runners/common.py", line 1568, in apache_beam.runners.common.DoFnRunner.setup
2025-10-15T20:57:50.7609363Z self._invoke_lifecycle_method(self.do_fn_invoker.invoke_setup)
2025-10-15T20:57:50.7610251Z File "apache_beam/runners/common.py", line 1564, in apache_beam.runners.common.DoFnRunner._invoke_lifecycle_method
2025-10-15T20:57:50.7611001Z self._reraise_augmented(exn)
2025-10-15T20:57:50.7611494Z File "apache_beam/runners/common.py", line 1609, in apache_beam.runners.common.DoFnRunner._reraise_augmented
2025-10-15T20:57:50.7611977Z raise new_exn
2025-10-15T20:57:50.7612458Z File "apache_beam/runners/common.py", line 1562, in apache_beam.runners.common.DoFnRunner._invoke_lifecycle_method
2025-10-15T20:57:50.7612968Z lifecycle_method()
2025-10-15T20:57:50.7613427Z File "apache_beam/runners/common.py", line 602, in apache_beam.runners.common.DoFnInvoker.invoke_setup
2025-10-15T20:57:50.7613950Z self.signature.setup_lifecycle_method.method_value()
2025-10-15T20:57:50.7614872Z File "/runner/_work/beam/beam/sdks/python/test-suites/tox/py312/build/srcs/sdks/python/target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/apache_beam/transforms/util.py", line 428, in setup
2025-10-15T20:57:50.7615755Z self._hmac_key = self.hmac_key_secret.get_secret_bytes()
2025-10-15T20:57:50.7616078Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-10-15T20:57:50.7616977Z File "/runner/_work/beam/beam/sdks/python/test-suites/tox/py312/build/srcs/sdks/python/target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/apache_beam/transforms/util.py", line 408, in get_secret_bytes
2025-10-15T20:57:50.7617852Z raise RuntimeError(
2025-10-15T20:57:50.7619177Z RuntimeError: Failed to retrieve secret bytes for secret projects/apache-beam-testing/secrets/gbek_util_secret_tests_1015_205647_py312/versions/latest with exception 404 Secret [projects/844138762903/secrets/gbek_util_secret_tests_1015_205647_py312] not found or has no versions. [while running 'assert_that/Group/CoGroupByKeyImpl/GroupByKey/Group by encrypted key/ParDo(_EncryptMessage)']
2025-10-15T20:57:50.7620997Z
2025-10-15T20:57:50.7621129Z Traceback (most recent call last):
2025-10-15T20:57:50.7622041Z File "/runner/_work/beam/beam/sdks/python/test-suites/tox/py312/build/srcs/sdks/python/target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/apache_beam/transforms/util.py", line 403, in get_secret_bytes
2025-10-15T20:57:50.7623126Z response = client.access_secret_version(
2025-10-15T20:57:50.7623415Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-10-15T20:57:50.7624470Z File "/runner/_work/beam/beam/sdks/python/test-suites/tox/py312/build/srcs/sdks/python/target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/google/cloud/secretmanager_v1/services/secret_manager_service/client.py", line 1857, in access_secret_version
2025-10-15T20:57:50.7625508Z response = rpc(
2025-10-15T20:57:50.7625722Z ^^^^
2025-10-15T20:57:50.7626546Z File "/runner/_work/beam/beam/sdks/python/test-suites/tox/py312/build/srcs/sdks/python/target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/google/api_core/gapic_v1/method.py", line 131, in call
2025-10-15T20:57:50.7627412Z return wrapped_func(*args, **kwargs)
2025-10-15T20:57:50.7627672Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-10-15T20:57:50.7628578Z File "/runner/_work/beam/beam/sdks/python/test-suites/tox/py312/build/srcs/sdks/python/target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/google/api_core/retry/retry_unary.py", line 294, in retry_wrapped_func
2025-10-15T20:57:50.7629475Z return retry_target(
2025-10-15T20:57:50.7629700Z ^^^^^^^^^^^^^
2025-10-15T20:57:50.7631040Z File "/runner/_work/beam/beam/sdks/python/test-suites/tox/py312/build/srcs/sdks/python/target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/google/api_core/retry/retry_unary.py", line 156, in retry_target
2025-10-15T20:57:50.7631937Z next_sleep = _retry_error_helper(
2025-10-15T20:57:50.7632378Z ^^^^^^^^^^^^^^^^^^^^
2025-10-15T20:57:50.7633298Z File "/runner/_work/beam/beam/sdks/python/test-suites/tox/py312/build/srcs/sdks/python/target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/google/api_core/retry/retry_base.py", line 214, in _retry_error_helper
2025-10-15T20:57:50.7634223Z raise final_exc from source_exc
2025-10-15T20:57:50.7635118Z File "/runner/_work/beam/beam/sdks/python/test-suites/tox/py312/build/srcs/sdks/python/target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/google/api_core/retry/retry_unary.py", line 147, in retry_target
2025-10-15T20:57:50.7636067Z result = target()
2025-10-15T20:57:50.7636287Z ^^^^^^^^
2025-10-15T20:57:50.7637121Z File "/runner/_work/beam/beam/sdks/python/test-suites/tox/py312/build/srcs/sdks/python/target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/google/api_core/timeout.py", line 130, in func_with_timeout
2025-10-15T20:57:50.7637988Z return func(*args, **kwargs)
2025-10-15T20:57:50.7638240Z ^^^^^^^^^^^^^^^^^^^^^
2025-10-15T20:57:50.7639127Z File "/runner/_work/beam/beam/sdks/python/test-suites/tox/py312/build/srcs/sdks/python/target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/google/api_core/grpc_helpers.py", line 77, in error_remapped_callable
2025-10-15T20:57:50.7640611Z raise exceptions.from_grpc_error(exc) from exc
2025-10-15T20:57:50.7641288Z google.api_core.exceptions.NotFound: 404 Secret [projects/844138762903/secrets/gbek_util_secret_tests_1015_205647_py312] not found or has no versions.
2025-10-15T20:57:50.7641806Z
2025-10-15T20:57:50.7641984Z During handling of the above exception, another exception occurred:
2025-10-15T20:57:50.7642262Z
2025-10-15T20:57:50.7642363Z Traceback (most recent call last):
2025-10-15T20:57:50.7642907Z File "apache_beam/runners/common.py", line 1562, in apache_beam.runners.common.DoFnRunner._invoke_lifecycle_method
2025-10-15T20:57:50.7643546Z lifecycle_method()
2025-10-15T20:57:50.7644288Z File "apache_beam/runners/common.py", line 602, in apache_beam.runners.common.DoFnInvoker.invoke_setup
2025-10-15T20:57:50.7644822Z self.signature.setup_lifecycle_method.method_value()
2025-10-15T20:57:50.7645733Z File "/runner/_work/beam/beam/sdks/python/test-suites/tox/py312/build/srcs/sdks/python/target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/apache_beam/transforms/util.py", line 465, in setup
2025-10-15T20:57:50.7646948Z hmac_key = self.hmac_key_secret.get_secret_bytes()
2025-10-15T20:57:50.7647264Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-10-15T20:57:50.7648162Z File "/runner/_work/beam/beam/sdks/python/test-suites/tox/py312/build/srcs/sdks/python/target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/apache_beam/transforms/util.py", line 408, in get_secret_bytes
2025-10-15T20:57:50.7649043Z raise RuntimeError(
2025-10-15T20:57:50.7650443Z RuntimeError: Failed to retrieve secret bytes for secret projects/apache-beam-testing/secrets/gbek_util_secret_tests_1015_205647_py312/versions/latest with exception 404 Secret [projects/844138762903/secrets/gbek_util_secret_tests_1015_205647_py312] not found or has no versions.
2025-10-15T20:57:50.7651390Z
2025-10-15T20:57:50.7651584Z During handling of the above exception, another exception occurred:
2025-10-15T20:57:50.7651845Z
2025-10-15T20:57:50.7651964Z Traceback (most recent call last):
2025-10-15T20:57:50.7653074Z File "/runner/_work/beam/beam/sdks/python/test-suites/tox/py312/build/srcs/sdks/python/target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/apache_beam/runners/worker/sdk_worker.py", line 316, in _execute
2025-10-15T20:57:50.7654029Z response = task()
2025-10-15T20:57:50.7654256Z ^^^^^^
2025-10-15T20:57:50.7655535Z File "/runner/_work/beam/beam/sdks/python/test-suites/tox/py312/build/srcs/sdks/python/target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/apache_beam/runners/worker/sdk_worker.py", line 390, in
2025-10-15T20:57:50.7656977Z lambda: self.create_worker().do_instruction(request), request)
2025-10-15T20:57:50.7657352Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-10-15T20:57:50.7658321Z File "/runner/_work/beam/beam/sdks/python/test-suites/tox/py312/build/srcs/sdks/python/target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/apache_beam/runners/worker/sdk_worker.py", line 666, in do_instruction
2025-10-15T20:57:50.7659250Z return getattr(self, request_type)(
2025-10-15T20:57:50.7659531Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-10-15T20:57:50.7661015Z File "/runner/_work/beam/beam/sdks/python/test-suites/tox/py312/build/srcs/sdks/python/target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/apache_beam/runners/worker/sdk_worker.py", line 697, in process_bundle
2025-10-15T20:57:50.7661978Z bundle_processor = self.bundle_processor_cache.get(
2025-10-15T20:57:50.7662296Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-10-15T20:57:50.7663226Z File "/runner/_work/beam/beam/sdks/python/test-suites/tox/py312/build/srcs/sdks/python/target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/apache_beam/runners/worker/sdk_worker.py", line 517, in get
2025-10-15T20:57:50.7664150Z processor = bundle_processor.BundleProcessor(
2025-10-15T20:57:50.7664461Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-10-15T20:57:50.7665405Z File "/runner/_work/beam/beam/sdks/python/test-suites/tox/py312/build/srcs/sdks/python/target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/apache_beam/runners/worker/bundle_processor.py", line 1135, in init
2025-10-15T20:57:50.7666340Z op.setup(self.data_sampler)
2025-10-15T20:57:50.7666904Z File "apache_beam/runners/worker/operations.py", line 875, in apache_beam.runners.worker.operations.DoOperation.setup
2025-10-15T20:57:50.7667440Z with self.scoped_start_state:
2025-10-15T20:57:50.7667985Z File "apache_beam/runners/worker/operations.py", line 925, in apache_beam.runners.worker.operations.DoOperation.setup
2025-10-15T20:57:50.7668526Z self.dofn_runner.setup()
2025-10-15T20:57:50.7668982Z File "apache_beam/runners/common.py", line 1568, in apache_beam.runners.common.DoFnRunner.setup
2025-10-15T20:57:50.7669504Z self._invoke_lifecycle_method(self.do_fn_invoker.invoke_setup)
2025-10-15T20:57:50.7670616Z File "apache_beam/runners/common.py", line 1564, in apache_beam.runners.common.DoFnRunner._invoke_lifecycle_method
2025-10-15T20:57:50.7671357Z self._reraise_augmented(exn)
2025-10-15T20:57:50.7671873Z File "apache_beam/runners/common.py", line 1609, in apache_beam.runners.common.DoFnRunner._reraise_augmented
2025-10-15T20:57:50.7672354Z raise new_exn
2025-10-15T20:57:50.7672854Z File "apache_beam/runners/common.py", line 1562, in apache_beam.runners.common.DoFnRunner._invoke_lifecycle_method
2025-10-15T20:57:50.7673365Z lifecycle_method()
2025-10-15T20:57:50.7673828Z File "apache_beam/runners/common.py", line 602, in apache_beam.runners.common.DoFnInvoker.invoke_setup
2025-10-15T20:57:50.7674356Z self.signature.setup_lifecycle_method.method_value()
2025-10-15T20:57:50.7675443Z File "/runner/_work/beam/beam/sdks/python/test-suites/tox/py312/build/srcs/sdks/python/target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/apache_beam/transforms/util.py", line 465, in setup
2025-10-15T20:57:50.7676315Z hmac_key = self.hmac_key_secret.get_secret_bytes()
2025-10-15T20:57:50.7676636Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-10-15T20:57:50.7677520Z File "/runner/_work/beam/beam/sdks/python/test-suites/tox/py312/build/srcs/sdks/python/target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/apache_beam/transforms/util.py", line 408, in get_secret_bytes
2025-10-15T20:57:50.7678459Z raise RuntimeError(
2025-10-15T20:57:50.7679958Z RuntimeError: Failed to retrieve secret bytes for secret projects/apache-beam-testing/secrets/gbek_util_secret_tests_1015_205647_py312/versions/latest with exception 404 Secret [projects/844138762903/secrets/gbek_util_secret_tests_1015_205647_py312] not found or has no versions. [while running 'GroupByKey/Group by encrypted key/ParDo(_DecryptMessage)']
2025-10-15T20:57:50.7681441Z
2025-10-15T20:57:50.7681632Z ------------------------------ Captured log call -------------------------------
2025-10-15T20:57:50.7683232Z ERROR apache_beam.runners.common:bundle_processor.py:1135 Failed to retrieve secret bytes for secret projects/apache-beam-testing/secrets/gbek_util_secret_tests_1015_205647_py312/versions/latest with exception 404 Secret [projects/844138762903/secrets/gbek_util_secret_tests_1015_205647_py312] not found or has no versions. [while running 'assert_that/Group/CoGroupByKeyImpl/GroupByKey/Group by encrypted key/ParDo(_EncryptMessage)']
2025-10-15T20:57:50.7684734Z Traceback (most recent call last):
2025-10-15T20:57:50.7685642Z File "/runner/_work/beam/beam/sdks/python/test-suites/tox/py312/build/srcs/sdks/python/target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/apache_beam/transforms/util.py", line 403, in get_secret_bytes
2025-10-15T20:57:50.7686544Z response = client.access_secret_version(
2025-10-15T20:57:50.7686825Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-10-15T20:57:50.7687896Z File "/runner/_work/beam/beam/sdks/python/test-suites/tox/py312/build/srcs/sdks/python/target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/google/cloud/secretmanager_v1/services/secret_manager_service/client.py", line 1857, in access_secret_version
2025-10-15T20:57:50.7688933Z response = rpc(
2025-10-15T20:57:50.7689137Z ^^^^
2025-10-15T20:57:50.7690220Z File "/runner/_work/beam/beam/sdks/python/test-suites/tox/py312/build/srcs/sdks/python/target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/google/api_core/gapic_v1/method.py", line 131, in call
2025-10-15T20:57:50.7691276Z return wrapped_func(*args, **kwargs)
2025-10-15T20:57:50.7691532Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-10-15T20:57:50.7692440Z File "/runner/_work/beam/beam/sdks/python/test-suites/tox/py312/build/srcs/sdks/python/target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/google/api_core/retry/retry_unary.py", line 294, in retry_wrapped_func
2025-10-15T20:57:50.7693360Z return retry_target(
2025-10-15T20:57:50.7693594Z ^^^^^^^^^^^^^
2025-10-15T20:57:50.7694438Z File "/runner/_work/beam/beam/sdks/python/test-suites/tox/py312/build/srcs/sdks/python/target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/google/api_core/retry/retry_unary.py", line 156, in retry_target
2025-10-15T20:57:50.7695490Z next_sleep = _retry_error_helper(
2025-10-15T20:57:50.7695752Z ^^^^^^^^^^^^^^^^^^^^
2025-10-15T20:57:50.7696637Z File "/runner/_work/beam/beam/sdks/python/test-suites/tox/py312/build/srcs/sdks/python/target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/google/api_core/retry/retry_base.py", line 214, in _retry_error_helper
2025-10-15T20:57:50.7697870Z raise final_exc from source_exc
2025-10-15T20:57:50.7698777Z File "/runner/_work/beam/beam/sdks/python/test-suites/tox/py312/build/srcs/sdks/python/target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/google/api_core/retry/retry_unary.py", line 147, in retry_target
2025-10-15T20:57:50.7699640Z result = target()
2025-10-15T20:57:50.7700135Z ^^^^^^^^
2025-10-15T20:57:50.7701296Z File "/runner/_work/beam/beam/sdks/python/test-suites/tox/py312/build/srcs/sdks/python/target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/google/api_core/timeout.py", line 130, in func_with_timeout
2025-10-15T20:57:50.7702180Z return func(*args, **kwargs)
2025-10-15T20:57:50.7702421Z ^^^^^^^^^^^^^^^^^^^^^
2025-10-15T20:57:50.7703326Z File "/runner/_work/beam/beam/sdks/python/test-suites/tox/py312/build/srcs/sdks/python/target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/google/api_core/grpc_helpers.py", line 77, in error_remapped_callable
2025-10-15T20:57:50.7704275Z raise exceptions.from_grpc_error(exc) from exc
2025-10-15T20:57:50.7705091Z google.api_core.exceptions.NotFound: 404 Secret [projects/844138762903/secrets/gbek_util_secret_tests_1015_205647_py312] not found or has no versions.
2025-10-15T20:57:50.7705610Z
2025-10-15T20:57:50.7705803Z During handling of the above exception, another exception occurred:
2025-10-15T20:57:50.7706062Z
2025-10-15T20:57:50.7706162Z Traceback (most recent call last):
2025-10-15T20:57:50.7706704Z File "apache_beam/runners/common.py", line 1562, in apache_beam.runners.common.DoFnRunner._invoke_lifecycle_method
2025-10-15T20:57:50.7707189Z lifecycle_method()
2025-10-15T20:57:50.7707635Z File "apache_beam/runners/common.py", line 602, in apache_beam.runners.common.DoFnInvoker.invoke_setup
2025-10-15T20:57:50.7708145Z self.signature.setup_lifecycle_method.method_value()
2025-10-15T20:57:50.7709046Z File "/runner/_work/beam/beam/sdks/python/test-suites/tox/py312/build/srcs/sdks/python/target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/apache_beam/transforms/util.py", line 428, in setup
2025-10-15T20:57:50.7710161Z self._hmac_key = self.hmac_key_secret.get_secret_bytes()
2025-10-15T20:57:50.7710699Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-10-15T20:57:50.7711605Z File "/runner/_work/beam/beam/sdks/python/test-suites/tox/py312/build/srcs/sdks/python/target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/apache_beam/transforms/util.py", line 408, in get_secret_bytes
2025-10-15T20:57:50.7712497Z raise RuntimeError(
2025-10-15T20:57:50.7713500Z RuntimeError: Failed to retrieve secret bytes for secret projects/apache-beam-testing/secrets/gbek_util_secret_tests_1015_205647_py312/versions/latest with exception 404 Secret [projects/844138762903/secrets/gbek_util_secret_tests_1015_205647_py312] not found or has no versions.
2025-10-15T20:57:50.7714769Z ERROR apache_beam.runners.worker.sdk_worker:sdk_worker.py:320 Error processing instruction inst008. Original traceback is
2025-10-15T20:57:50.7715288Z Traceback (most recent call last):
2025-10-15T20:57:50.7716184Z File "/runner/_work/beam/beam/sdks/python/test-suites/tox/py312/build/srcs/sdks/python/target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/apache_beam/transforms/util.py", line 403, in get_secret_bytes
2025-10-15T20:57:50.7717069Z response = client.access_secret_version(
2025-10-15T20:57:50.7717341Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-10-15T20:57:50.7718567Z File "/runner/_work/beam/beam/sdks/python/test-suites/tox/py312/build/srcs/sdks/python/target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/google/cloud/secretmanager_v1/services/secret_manager_service/client.py", line 1857, in access_secret_version
2025-10-15T20:57:50.7719610Z response = rpc(
2025-10-15T20:57:50.7719979Z ^^^^
2025-10-15T20:57:50.7721152Z File "/runner/_work/beam/beam/sdks/python/test-suites/tox/py312/build/srcs/sdks/python/target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/google/api_core/gapic_v1/method.py", line 131, in call
2025-10-15T20:57:50.7722031Z return wrapped_func(*args, **kwargs)
2025-10-15T20:57:50.7722298Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-10-15T20:57:50.7723186Z File "/runner/_work/beam/beam/sdks/python/test-suites/tox/py312/build/srcs/sdks/python/target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/google/api_core/retry/retry_unary.py", line 294, in retry_wrapped_func
2025-10-15T20:57:50.7724069Z return retry_target(
2025-10-15T20:57:50.7724287Z ^^^^^^^^^^^^^
2025-10-15T20:57:50.7725121Z File "/runner/_work/beam/beam/sdks/python/test-suites/tox/py312/build/srcs/sdks/python/target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/google/api_core/retry/retry_unary.py", line 156, in retry_target
2025-10-15T20:57:50.7726000Z next_sleep = _retry_error_helper(
2025-10-15T20:57:50.7726256Z ^^^^^^^^^^^^^^^^^^^^
2025-10-15T20:57:50.7727305Z File "/runner/_work/beam/beam/sdks/python/test-suites/tox/py312/build/srcs/sdks/python/target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/google/api_core/retry/retry_base.py", line 214, in _retry_error_helper
2025-10-15T20:57:50.7728219Z raise final_exc from source_exc
2025-10-15T20:57:50.7729096Z File "/runner/_work/beam/beam/sdks/python/test-suites/tox/py312/build/srcs/sdks/python/target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/google/api_core/retry/retry_unary.py", line 147, in retry_target
2025-10-15T20:57:50.7730378Z result = target()
2025-10-15T20:57:50.7730726Z ^^^^^^^^
2025-10-15T20:57:50.7731549Z File "/runner/_work/beam/beam/sdks/python/test-suites/tox/py312/build/srcs/sdks/python/target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/google/api_core/timeout.py", line 130, in func_with_timeout
2025-10-15T20:57:50.7732402Z return func(*args, **kwargs)
2025-10-15T20:57:50.7732641Z ^^^^^^^^^^^^^^^^^^^^^
2025-10-15T20:57:50.7733526Z File "/runner/_work/beam/beam/sdks/python/test-suites/tox/py312/build/srcs/sdks/python/target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/google/api_core/grpc_helpers.py", line 77, in error_remapped_callable
2025-10-15T20:57:50.7734457Z raise exceptions.from_grpc_error(exc) from exc
2025-10-15T20:57:50.7735082Z google.api_core.exceptions.NotFound: 404 Secret [projects/844138762903/secrets/gbek_util_secret_tests_1015_205647_py312] not found or has no versions.
2025-10-15T20:57:50.7735605Z
2025-10-15T20:57:50.7735778Z During handling of the above exception, another exception occurred:
2025-10-15T20:57:50.7736042Z
2025-10-15T20:57:50.7736143Z Traceback (most recent call last):
2025-10-15T20:57:50.7736667Z File "apache_beam/runners/common.py", line 1562, in apache_beam.runners.common.DoFnRunner._invoke_lifecycle_method
2025-10-15T20:57:50.7737150Z lifecycle_method()
2025-10-15T20:57:50.7737595Z File "apache_beam/runners/common.py", line 602, in apache_beam.runners.common.DoFnInvoker.invoke_setup
2025-10-15T20:57:50.7738091Z self.signature.setup_lifecycle_method.method_value()
2025-10-15T20:57:50.7738996Z File "/runner/_work/beam/beam/sdks/python/test-suites/tox/py312/build/srcs/sdks/python/target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/apache_beam/transforms/util.py", line 428, in setup
2025-10-15T20:57:50.7740079Z self._hmac_key = self.hmac_key_secret.get_secret_bytes()
2025-10-15T20:57:50.7740676Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-10-15T20:57:50.7741837Z File "/runner/_work/beam/beam/sdks/python/test-suites/tox/py312/build/srcs/sdks/python/target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/apache_beam/transforms/util.py", line 408, in get_secret_bytes
2025-10-15T20:57:50.7742733Z raise RuntimeError(
2025-10-15T20:57:50.7743708Z RuntimeError: Failed to retrieve secret bytes for secret projects/apache-beam-testing/secrets/gbek_util_secret_tests_1015_205647_py312/versions/latest with exception 404 Secret [projects/844138762903/secrets/gbek_util_secret_tests_1015_205647_py312] not found or has no versions.
2025-10-15T20:57:50.7744626Z
2025-10-15T20:57:50.7744812Z During handling of the above exception, another exception occurred:
2025-10-15T20:57:50.7745064Z
2025-10-15T20:57:50.7745169Z Traceback (most recent call last):
2025-10-15T20:57:50.7746055Z File "/runner/_work/beam/beam/sdks/python/test-suites/tox/py312/build/srcs/sdks/python/target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/apache_beam/runners/worker/sdk_worker.py", line 316, in _execute
2025-10-15T20:57:50.7746931Z response = task()
2025-10-15T20:57:50.7747129Z ^^^^^^
2025-10-15T20:57:50.7747963Z File "/runner/_work/beam/beam/sdks/python/test-suites/tox/py312/build/srcs/sdks/python/target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/apache_beam/runners/worker/sdk_worker.py", line 390, in
2025-10-15T20:57:50.7748894Z lambda: self.create_worker().do_instruction(request), request)
2025-10-15T20:57:50.7749229Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-10-15T20:57:50.7750773Z File "/runner/_work/beam/beam/sdks/python/test-suites/tox/py312/build/srcs/sdks/python/target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/apache_beam/runners/worker/sdk_worker.py", line 666, in do_instruction
2025-10-15T20:57:50.7751737Z return getattr(self, request_type)(
2025-10-15T20:57:50.7752001Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-10-15T20:57:50.7752901Z File "/runner/_work/beam/beam/sdks/python/test-suites/tox/py312/build/srcs/sdks/python/target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/apache_beam/runners/worker/sdk_worker.py", line 697, in process_bundle
2025-10-15T20:57:50.7753835Z bundle_processor = self.bundle_processor_cache.get(
2025-10-15T20:57:50.7754144Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-10-15T20:57:50.7755027Z File "/runner/_work/beam/beam/sdks/python/test-suites/tox/py312/build/srcs/sdks/python/target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/apache_beam/runners/worker/sdk_worker.py", line 517, in get
2025-10-15T20:57:50.7755926Z processor = bundle_processor.BundleProcessor(
2025-10-15T20:57:50.7756236Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-10-15T20:57:50.7757150Z File "/runner/_work/beam/beam/sdks/python/test-suites/tox/py312/build/srcs/sdks/python/target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/apache_beam/runners/worker/bundle_processor.py", line 1135, in init
2025-10-15T20:57:50.7758064Z op.setup(self.data_sampler)
2025-10-15T20:57:50.7758604Z File "apache_beam/runners/worker/operations.py", line 875, in apache_beam.runners.worker.operations.DoOperation.setup
2025-10-15T20:57:50.7759136Z with self.scoped_start_state:
2025-10-15T20:57:50.7759673Z File "apache_beam/runners/worker/operations.py", line 925, in apache_beam.runners.worker.operations.DoOperation.setup
2025-10-15T20:57:50.7763263Z self.dofn_runner.setup()
2025-10-15T20:57:50.7764323Z File "apache_beam/runners/common.py", line 1568, in apache_beam.runners.common.DoFnRunner.setup
2025-10-15T20:57:50.7765287Z self._invoke_lifecycle_method(self.do_fn_invoker.invoke_setup)
2025-10-15T20:57:50.7766541Z File "apache_beam/runners/common.py", line 1564, in apache_beam.runners.common.DoFnRunner._invoke_lifecycle_method
2025-10-15T20:57:50.7767481Z self._reraise_augmented(exn)
2025-10-15T20:57:50.7768405Z File "apache_beam/runners/common.py", line 1609, in apache_beam.runners.common.DoFnRunner._reraise_augmented
2025-10-15T20:57:50.7770387Z raise new_exn
2025-10-15T20:57:50.7771458Z File "apache_beam/runners/common.py", line 1562, in apache_beam.runners.common.DoFnRunner._invoke_lifecycle_method
2025-10-15T20:57:50.7772380Z lifecycle_method()
2025-10-15T20:57:50.7773243Z File "apache_beam/runners/common.py", line 602, in apache_beam.runners.common.DoFnInvoker.invoke_setup
2025-10-15T20:57:50.7774080Z self.signature.setup_lifecycle_method.method_value()
2025-10-15T20:57:50.7775613Z File "/runner/_work/beam/beam/sdks/python/test-suites/tox/py312/build/srcs/sdks/python/target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/apache_beam/transforms/util.py", line 428, in setup
2025-10-15T20:57:50.7777088Z self._hmac_key = self.hmac_key_secret.get_secret_bytes()
2025-10-15T20:57:50.7777584Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-10-15T20:57:50.7779082Z File "/runner/_work/beam/beam/sdks/python/test-suites/tox/py312/build/srcs/sdks/python/target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/apache_beam/transforms/util.py", line 408, in get_secret_bytes
2025-10-15T20:57:50.7780960Z raise RuntimeError(
2025-10-15T20:57:50.7783472Z RuntimeError: Failed to retrieve secret bytes for secret projects/apache-beam-testing/secrets/gbek_util_secret_tests_1015_205647_py312/versions/latest with exception 404 Secret [projects/844138762903/secrets/gbek_util_secret_tests_1015_205647_py312] not found or has no versions. [while running 'assert_that/Group/CoGroupByKeyImpl/GroupByKey/Group by encrypted key/ParDo(_EncryptMessage)']
2025-10-15T20:57:50.7785295Z
2025-10-15T20:57:50.7785300Z
2025-10-15T20:57:50.7786954Z ERROR apache_beam.runners.common:bundle_processor.py:1135 Failed to retrieve secret bytes for secret projects/apache-beam-testing/secrets/gbek_util_secret_tests_1015_205647_py312/versions/latest with exception 404 Secret [projects/844138762903/secrets/gbek_util_secret_tests_1015_205647_py312] not found or has no versions. [while running 'GroupByKey/Group by encrypted key/ParDo(_DecryptMessage)']
2025-10-15T20:57:50.7788853Z Traceback (most recent call last):
2025-10-15T20:57:50.7790309Z File "/runner/_work/beam/beam/sdks/python/test-suites/tox/py312/build/srcs/sdks/python/target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/apache_beam/transforms/util.py", line 403, in get_secret_bytes
2025-10-15T20:57:50.7791663Z response = client.access_secret_version(
2025-10-15T20:57:50.7792013Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-10-15T20:57:50.7793423Z File "/runner/_work/beam/beam/sdks/python/test-suites/tox/py312/build/srcs/sdks/python/target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/google/cloud/secretmanager_v1/services/secret_manager_service/client.py", line 1857, in access_secret_version
2025-10-15T20:57:50.7794673Z response = rpc(
2025-10-15T20:57:50.7795050Z ^^^^
2025-10-15T20:57:50.7796015Z File "/runner/_work/beam/beam/sdks/python/test-suites/tox/py312/build/srcs/sdks/python/target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/google/api_core/gapic_v1/method.py", line 131, in call
2025-10-15T20:57:50.7797064Z return wrapped_func(*args, **kwargs)
2025-10-15T20:57:50.7797357Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-10-15T20:57:50.7798525Z File "/runner/_work/beam/beam/sdks/python/test-suites/tox/py312/build/srcs/sdks/python/target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/google/api_core/retry/retry_unary.py", line 294, in retry_wrapped_func
2025-10-15T20:57:50.7799932Z return retry_target(
2025-10-15T20:57:50.7800323Z ^^^^^^^^^^^^^
2025-10-15T20:57:50.7801364Z File "/runner/_work/beam/beam/sdks/python/test-suites/tox/py312/build/srcs/sdks/python/target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/google/api_core/retry/retry_unary.py", line 156, in retry_target
2025-10-15T20:57:50.7802447Z next_sleep = _retry_error_helper(
2025-10-15T20:57:50.7803010Z ^^^^^^^^^^^^^^^^^^^^
2025-10-15T20:57:50.7803939Z File "/runner/_work/beam/beam/sdks/python/test-suites/tox/py312/build/srcs/sdks/python/target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/google/api_core/retry/retry_base.py", line 214, in _retry_error_helper
2025-10-15T20:57:50.7804876Z raise final_exc from source_exc
2025-10-15T20:57:50.7805791Z File "/runner/_work/beam/beam/sdks/python/test-suites/tox/py312/build/srcs/sdks/python/target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/google/api_core/retry/retry_unary.py", line 147, in retry_target
2025-10-15T20:57:50.7806691Z result = target()
2025-10-15T20:57:50.7806902Z ^^^^^^^^
2025-10-15T20:57:50.7807748Z File "/runner/_work/beam/beam/sdks/python/test-suites/tox/py312/build/srcs/sdks/python/target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/google/api_core/timeout.py", line 130, in func_with_timeout
2025-10-15T20:57:50.7808925Z return func(*args, **kwargs)
2025-10-15T20:57:50.7809179Z ^^^^^^^^^^^^^^^^^^^^^
2025-10-15T20:57:50.7810473Z File "/runner/_work/beam/beam/sdks/python/test-suites/tox/py312/build/srcs/sdks/python/target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/google/api_core/grpc_helpers.py", line 77, in error_remapped_callable
2025-10-15T20:57:50.7811679Z raise exceptions.from_grpc_error(exc) from exc
2025-10-15T20:57:50.7812348Z google.api_core.exceptions.NotFound: 404 Secret [projects/844138762903/secrets/gbek_util_secret_tests_1015_205647_py312] not found or has no versions.
2025-10-15T20:57:50.7812924Z
2025-10-15T20:57:50.7813302Z During handling of the above exception, another exception occurred:
2025-10-15T20:57:50.7813572Z
2025-10-15T20:57:50.7813668Z Traceback (most recent call last):
2025-10-15T20:57:50.7814191Z File "apache_beam/runners/common.py", line 1562, in apache_beam.runners.common.DoFnRunner._invoke_lifecycle_method
2025-10-15T20:57:50.7814678Z lifecycle_method()
2025-10-15T20:57:50.7815122Z File "apache_beam/runners/common.py", line 602, in apache_beam.runners.common.DoFnInvoker.invoke_setup
2025-10-15T20:57:50.7815625Z self.signature.setup_lifecycle_method.method_value()
2025-10-15T20:57:50.7816547Z File "/runner/_work/beam/beam/sdks/python/test-suites/tox/py312/build/srcs/sdks/python/target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/apache_beam/transforms/util.py", line 465, in setup
2025-10-15T20:57:50.7817403Z hmac_key = self.hmac_key_secret.get_secret_bytes()
2025-10-15T20:57:50.7817695Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-10-15T20:57:50.7818580Z File "/runner/_work/beam/beam/sdks/python/test-suites/tox/py312/build/srcs/sdks/python/target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/apache_beam/transforms/util.py", line 408, in get_secret_bytes
2025-10-15T20:57:50.7819438Z raise RuntimeError(
2025-10-15T20:57:50.7820943Z RuntimeError: Failed to retrieve secret bytes for secret projects/apache-beam-testing/secrets/gbek_util_secret_tests_1015_205647_py312/versions/latest with exception 404 Secret [projects/844138762903/secrets/gbek_util_secret_tests_1015_205647_py312] not found or has no versions.
2025-10-15T20:57:50.7822270Z ERROR apache_beam.runners.worker.sdk_worker:sdk_worker.py:320 Error processing instruction inst007. Original traceback is
2025-10-15T20:57:50.7822787Z Traceback (most recent call last):
2025-10-15T20:57:50.7823723Z File "/runner/_work/beam/beam/sdks/python/test-suites/tox/py312/build/srcs/sdks/python/target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/apache_beam/transforms/util.py", line 403, in get_secret_bytes
2025-10-15T20:57:50.7824606Z response = client.access_secret_version(
2025-10-15T20:57:50.7824872Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-10-15T20:57:50.7825918Z File "/runner/_work/beam/beam/sdks/python/test-suites/tox/py312/build/srcs/sdks/python/target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/google/cloud/secretmanager_v1/services/secret_manager_service/client.py", line 1857, in access_secret_version
2025-10-15T20:57:50.7827146Z response = rpc(
2025-10-15T20:57:50.7827337Z ^^^^
2025-10-15T20:57:50.7828138Z File "/runner/_work/beam/beam/sdks/python/test-suites/tox/py312/build/srcs/sdks/python/target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/google/api_core/gapic_v1/method.py", line 131, in call
2025-10-15T20:57:50.7828985Z return wrapped_func(*args, **kwargs)
2025-10-15T20:57:50.7829232Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-10-15T20:57:50.7830184Z File "/runner/_work/beam/beam/sdks/python/test-suites/tox/py312/build/srcs/sdks/python/target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/google/api_core/retry/retry_unary.py", line 294, in retry_wrapped_func
2025-10-15T20:57:50.7830332Z return retry_target(
2025-10-15T20:57:50.7830475Z ^^^^^^^^^^^^^
2025-10-15T20:57:50.7831381Z File "/runner/_work/beam/beam/sdks/python/test-suites/tox/py312/build/srcs/sdks/python/target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/google/api_core/retry/retry_unary.py", line 156, in retry_target
2025-10-15T20:57:50.7831507Z next_sleep = _retry_error_helper(
2025-10-15T20:57:50.7831587Z ^^^^^^^^^^^^^^^^^^^^
2025-10-15T20:57:50.7832325Z File "/runner/_work/beam/beam/sdks/python/test-suites/tox/py312/build/srcs/sdks/python/target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/google/api_core/retry/retry_base.py", line 214, in _retry_error_helper
2025-10-15T20:57:50.7832440Z raise final_exc from source_exc
2025-10-15T20:57:50.7833349Z File "/runner/_work/beam/beam/sdks/python/test-suites/tox/py312/build/srcs/sdks/python/target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/google/api_core/retry/retry_unary.py", line 147, in retry_target
2025-10-15T20:57:50.7833452Z result = target()
2025-10-15T20:57:50.7833524Z ^^^^^^^^
2025-10-15T20:57:50.7834223Z File "/runner/_work/beam/beam/sdks/python/test-suites/tox/py312/build/srcs/sdks/python/target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/google/api_core/timeout.py", line 130, in func_with_timeout
2025-10-15T20:57:50.7834320Z return func(*args, **kwargs)
2025-10-15T20:57:50.7834404Z ^^^^^^^^^^^^^^^^^^^^^
2025-10-15T20:57:50.7835137Z File "/runner/_work/beam/beam/sdks/python/test-suites/tox/py312/build/srcs/sdks/python/target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/google/api_core/grpc_helpers.py", line 77, in error_remapped_callable
2025-10-15T20:57:50.7835286Z raise exceptions.from_grpc_error(exc) from exc
2025-10-15T20:57:50.7835725Z google.api_core.exceptions.NotFound: 404 Secret [projects/844138762903/secrets/gbek_util_secret_tests_1015_205647_py312] not found or has no versions.
2025-10-15T20:57:50.7835732Z
2025-10-15T20:57:50.7835914Z During handling of the above exception, another exception occurred:
2025-10-15T20:57:50.7835919Z
2025-10-15T20:57:50.7836010Z Traceback (most recent call last):
2025-10-15T20:57:50.7836383Z File "apache_beam/runners/common.py", line 1562, in apache_beam.runners.common.DoFnRunner._invoke_lifecycle_method
2025-10-15T20:57:50.7836470Z lifecycle_method()
2025-10-15T20:57:50.7836794Z File "apache_beam/runners/common.py", line 602, in apache_beam.runners.common.DoFnInvoker.invoke_setup
2025-10-15T20:57:50.7836939Z self.signature.setup_lifecycle_method.method_value()
2025-10-15T20:57:50.7837621Z File "/runner/_work/beam/beam/sdks/python/test-suites/tox/py312/build/srcs/sdks/python/target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/apache_beam/transforms/util.py", line 465, in setup
2025-10-15T20:57:50.7837756Z hmac_key = self.hmac_key_secret.get_secret_bytes()
2025-10-15T20:57:50.7837856Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-10-15T20:57:50.7838570Z File "/runner/_work/beam/beam/sdks/python/test-suites/tox/py312/build/srcs/sdks/python/target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/apache_beam/transforms/util.py", line 408, in get_secret_bytes
2025-10-15T20:57:50.7838862Z raise RuntimeError(
2025-10-15T20:57:50.7839698Z RuntimeError: Failed to retrieve secret bytes for secret projects/apache-beam-testing/secrets/gbek_util_secret_tests_1015_205647_py312/versions/latest with exception 404 Secret [projects/844138762903/secrets/gbek_util_secret_tests_1015_205647_py312] not found or has no versions.
2025-10-15T20:57:50.7839714Z
2025-10-15T20:57:50.7840075Z During handling of the above exception, another exception occurred:
2025-10-15T20:57:50.7840085Z
2025-10-15T20:57:50.7840251Z Traceback (most recent call last):
2025-10-15T20:57:50.7841173Z File "/runner/_work/beam/beam/sdks/python/test-suites/tox/py312/build/srcs/sdks/python/target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/apache_beam/runners/worker/sdk_worker.py", line 316, in _execute
2025-10-15T20:57:50.7841276Z response = task()
2025-10-15T20:57:50.7841355Z ^^^^^^
2025-10-15T20:57:50.7842078Z File "/runner/_work/beam/beam/sdks/python/test-suites/tox/py312/build/srcs/sdks/python/target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/apache_beam/runners/worker/sdk_worker.py", line 390, in
2025-10-15T20:57:50.7842249Z lambda: self.create_worker().do_instruction(request), request)
2025-10-15T20:57:50.7842355Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-10-15T20:57:50.7843141Z File "/runner/_work/beam/beam/sdks/python/test-suites/tox/py312/build/srcs/sdks/python/target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/apache_beam/runners/worker/sdk_worker.py", line 666, in do_instruction
2025-10-15T20:57:50.7843428Z return getattr(self, request_type)(
2025-10-15T20:57:50.7843514Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-10-15T20:57:50.7844256Z File "/runner/_work/beam/beam/sdks/python/test-suites/tox/py312/build/srcs/sdks/python/target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/apache_beam/runners/worker/sdk_worker.py", line 697, in process_bundle
2025-10-15T20:57:50.7844394Z bundle_processor = self.bundle_processor_cache.get(
2025-10-15T20:57:50.7844491Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-10-15T20:57:50.7845181Z File "/runner/_work/beam/beam/sdks/python/test-suites/tox/py312/build/srcs/sdks/python/target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/apache_beam/runners/worker/sdk_worker.py", line 517, in get
2025-10-15T20:57:50.7845318Z processor = bundle_processor.BundleProcessor(
2025-10-15T20:57:50.7845404Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-10-15T20:57:50.7846234Z File "/runner/_work/beam/beam/sdks/python/test-suites/tox/py312/build/srcs/sdks/python/target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/apache_beam/runners/worker/bundle_processor.py", line 1135, in init
2025-10-15T20:57:50.7846352Z op.setup(self.data_sampler)
2025-10-15T20:57:50.7846731Z File "apache_beam/runners/worker/operations.py", line 875, in apache_beam.runners.worker.operations.DoOperation.setup
2025-10-15T20:57:50.7846839Z with self.scoped_start_state:
2025-10-15T20:57:50.7847206Z File "apache_beam/runners/worker/operations.py", line 925, in apache_beam.runners.worker.operations.DoOperation.setup
2025-10-15T20:57:50.7847306Z self.dofn_runner.setup()
2025-10-15T20:57:50.7847593Z File "apache_beam/runners/common.py", line 1568, in apache_beam.runners.common.DoFnRunner.setup
2025-10-15T20:57:50.7847762Z self._invoke_lifecycle_method(self.do_fn_invoker.invoke_setup)
2025-10-15T20:57:50.7848102Z File "apache_beam/runners/common.py", line 1564, in apache_beam.runners.common.DoFnRunner._invoke_lifecycle_method
2025-10-15T20:57:50.7848208Z self._reraise_augmented(exn)
2025-10-15T20:57:50.7848530Z File "apache_beam/runners/common.py", line 1609, in apache_beam.runners.common.DoFnRunner._reraise_augmented
2025-10-15T20:57:50.7848624Z raise new_exn
2025-10-15T20:57:50.7848961Z File "apache_beam/runners/common.py", line 1562, in apache_beam.runners.common.DoFnRunner._invoke_lifecycle_method
2025-10-15T20:57:50.7849183Z lifecycle_method()
2025-10-15T20:57:50.7849487Z File "apache_beam/runners/common.py", line 602, in apache_beam.runners.common.DoFnInvoker.invoke_setup
2025-10-15T20:57:50.7849640Z self.signature.setup_lifecycle_method.method_value()
2025-10-15T20:57:50.7852563Z File "/runner/_work/beam/beam/sdks/python/test-suites/tox/py312/build/srcs/sdks/python/target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/apache_beam/transforms/util.py", line 465, in setup
2025-10-15T20:57:50.7853124Z hmac_key = self.hmac_key_secret.get_secret_bytes()
2025-10-15T20:57:50.7853293Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-10-15T20:57:50.7854278Z File "/runner/_work/beam/beam/sdks/python/test-suites/tox/py312/build/srcs/sdks/python/target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/apache_beam/transforms/util.py", line 408, in get_secret_bytes
2025-10-15T20:57:50.7854403Z raise RuntimeError(
2025-10-15T20:57:50.7855765Z RuntimeError: Failed to retrieve secret bytes for secret projects/apache-beam-testing/secrets/gbek_util_secret_tests_1015_205647_py312/versions/latest with exception 404 Secret [projects/844138762903/secrets/gbek_util_secret_tests_1015_205647_py312] not found or has no versions. [while running 'GroupByKey/Group by encrypted key/ParDo(_DecryptMessage)']
2025-10-15T20:57:50.7855797Z
2025-10-15T20:57:50.7855803Z
2025-10-15T20:57:50.7856143Z ERROR root:portable_runner.py:535 bundle inst008 stage-013 failed:Traceback (most recent call last):
2025-10-15T20:57:50.7857518Z File "/runner/_work/beam/beam/sdks/python/test-suites/tox/py312/build/srcs/sdks/python/target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/apache_beam/transforms/util.py", line 403, in get_secret_bytes
2025-10-15T20:57:50.7857686Z response = client.access_secret_version(
2025-10-15T20:57:50.7857784Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-10-15T20:57:50.7858927Z File "/runner/_work/beam/beam/sdks/python/test-suites/tox/py312/build/srcs/sdks/python/target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/google/cloud/secretmanager_v1/services/secret_manager_service/client.py", line 1857, in access_secret_version
2025-10-15T20:57:50.7859044Z response = rpc(
2025-10-15T20:57:50.7859135Z ^^^^
2025-10-15T20:57:50.7860420Z File "/runner/_work/beam/beam/sdks/python/test-suites/tox/py312/build/srcs/sdks/python/target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/google/api_core/gapic_v1/method.py", line 131, in call
2025-10-15T20:57:50.7860615Z return wrapped_func(*args, **kwargs)
2025-10-15T20:57:50.7860751Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-10-15T20:57:50.7861766Z File "/runner/_work/beam/beam/sdks/python/test-suites/tox/py312/build/srcs/sdks/python/target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/google/api_core/retry/retry_unary.py", line 294, in retry_wrapped_func
2025-10-15T20:57:50.7861874Z return retry_target(
2025-10-15T20:57:50.7861969Z ^^^^^^^^^^^^^
2025-10-15T20:57:50.7862905Z File "/runner/_work/beam/beam/sdks/python/test-suites/tox/py312/build/srcs/sdks/python/target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/google/api_core/retry/retry_unary.py", line 156, in retry_target
2025-10-15T20:57:50.7863031Z next_sleep = _retry_error_helper(
2025-10-15T20:57:50.7863123Z ^^^^^^^^^^^^^^^^^^^^
2025-10-15T20:57:50.7864123Z File "/runner/_work/beam/beam/sdks/python/test-suites/tox/py312/build/srcs/sdks/python/target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/google/api_core/retry/retry_base.py", line 214, in _retry_error_helper
2025-10-15T20:57:50.7864260Z raise final_exc from source_exc
2025-10-15T20:57:50.7865154Z File "/runner/_work/beam/beam/sdks/python/test-suites/tox/py312/build/srcs/sdks/python/target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/google/api_core/retry/retry_unary.py", line 147, in retry_target
2025-10-15T20:57:50.7865265Z result = target()
2025-10-15T20:57:50.7865629Z ^^^^^^^^
2025-10-15T20:57:50.7866513Z File "/runner/_work/beam/beam/sdks/python/test-suites/tox/py312/build/srcs/sdks/python/target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/google/api_core/timeout.py", line 130, in func_with_timeout
2025-10-15T20:57:50.7866623Z return func(*args, **kwargs)
2025-10-15T20:57:50.7866727Z ^^^^^^^^^^^^^^^^^^^^^
2025-10-15T20:57:50.7867654Z File "/runner/_work/beam/beam/sdks/python/test-suites/tox/py312/build/srcs/sdks/python/target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/google/api_core/grpc_helpers.py", line 77, in error_remapped_callable
2025-10-15T20:57:50.7867834Z raise exceptions.from_grpc_error(exc) from exc
2025-10-15T20:57:50.7868371Z google.api_core.exceptions.NotFound: 404 Secret [projects/844138762903/secrets/gbek_util_secret_tests_1015_205647_py312] not found or has no versions.
2025-10-15T20:57:50.7868379Z
2025-10-15T20:57:50.7868598Z During handling of the above exception, another exception occurred:
2025-10-15T20:57:50.7868612Z
2025-10-15T20:57:50.7868723Z Traceback (most recent call last):
2025-10-15T20:57:50.7869201Z File "apache_beam/runners/common.py", line 1562, in apache_beam.runners.common.DoFnRunner._invoke_lifecycle_method
2025-10-15T20:57:50.7869302Z lifecycle_method()
2025-10-15T20:57:50.7869689Z File "apache_beam/runners/common.py", line 602, in apache_beam.runners.common.DoFnInvoker.invoke_setup
2025-10-15T20:57:50.7870087Z self.signature.setup_lifecycle_method.method_value()
2025-10-15T20:57:50.7871434Z File "/runner/_work/beam/beam/sdks/python/test-suites/tox/py312/build/srcs/sdks/python/target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/apache_beam/transforms/util.py", line 428, in setup
2025-10-15T20:57:50.7871626Z self._hmac_key = self.hmac_key_secret.get_secret_bytes()
2025-10-15T20:57:50.7871745Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-10-15T20:57:50.7872623Z File "/runner/_work/beam/beam/sdks/python/test-suites/tox/py312/build/srcs/sdks/python/target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/apache_beam/transforms/util.py", line 408, in get_secret_bytes
2025-10-15T20:57:50.7872747Z raise RuntimeError(
2025-10-15T20:57:50.7873721Z RuntimeError: Failed to retrieve secret bytes for secret projects/apache-beam-testing/secrets/gbek_util_secret_tests_1015_205647_py312/versions/latest with exception 404 Secret [projects/844138762903/secrets/gbek_util_secret_tests_1015_205647_py312] not found or has no versions.
2025-10-15T20:57:50.7873737Z
2025-10-15T20:57:50.7873922Z During handling of the above exception, another exception occurred:
2025-10-15T20:57:50.7873928Z
2025-10-15T20:57:50.7874037Z Traceback (most recent call last):
2025-10-15T20:57:50.7874796Z File "/runner/_work/beam/beam/sdks/python/test-suites/tox/py312/build/srcs/sdks/python/target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/apache_beam/runners/worker/sdk_worker.py", line 316, in _execute
2025-10-15T20:57:50.7874895Z response = task()
2025-10-15T20:57:50.7874971Z ^^^^^^
2025-10-15T20:57:50.7875714Z File "/runner/_work/beam/beam/sdks/python/test-suites/tox/py312/build/srcs/sdks/python/target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/apache_beam/runners/worker/sdk_worker.py", line 390, in
2025-10-15T20:57:50.7875878Z lambda: self.create_worker().do_instruction(request), request)
2025-10-15T20:57:50.7875979Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-10-15T20:57:50.7876747Z File "/runner/_work/beam/beam/sdks/python/test-suites/tox/py312/build/srcs/sdks/python/target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/apache_beam/runners/worker/sdk_worker.py", line 666, in do_instruction
2025-10-15T20:57:50.7876855Z return getattr(self, request_type)(
2025-10-15T20:57:50.7876934Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-10-15T20:57:50.7877694Z File "/runner/_work/beam/beam/sdks/python/test-suites/tox/py312/build/srcs/sdks/python/target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/apache_beam/runners/worker/sdk_worker.py", line 697, in process_bundle
2025-10-15T20:57:50.7878020Z bundle_processor = self.bundle_processor_cache.get(
2025-10-15T20:57:50.7878120Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-10-15T20:57:50.7878839Z File "/runner/_work/beam/beam/sdks/python/test-suites/tox/py312/build/srcs/sdks/python/target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/apache_beam/runners/worker/sdk_worker.py", line 517, in get
2025-10-15T20:57:50.7878979Z processor = bundle_processor.BundleProcessor(
2025-10-15T20:57:50.7879072Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-10-15T20:57:50.7880077Z File "/runner/_work/beam/beam/sdks/python/test-suites/tox/py312/build/srcs/sdks/python/target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/apache_beam/runners/worker/bundle_processor.py", line 1135, in init
2025-10-15T20:57:50.7880225Z op.setup(self.data_sampler)
2025-10-15T20:57:50.7880856Z File "apache_beam/runners/worker/operations.py", line 875, in apache_beam.runners.worker.operations.DoOperation.setup
2025-10-15T20:57:50.7881026Z with self.scoped_start_state:
2025-10-15T20:57:50.7881445Z File "apache_beam/runners/worker/operations.py", line 925, in apache_beam.runners.worker.operations.DoOperation.setup
2025-10-15T20:57:50.7881548Z self.dofn_runner.setup()
2025-10-15T20:57:50.7881842Z File "apache_beam/runners/common.py", line 1568, in apache_beam.runners.common.DoFnRunner.setup
2025-10-15T20:57:50.7882023Z self._invoke_lifecycle_method(self.do_fn_invoker.invoke_setup)
2025-10-15T20:57:50.7882604Z File "apache_beam/runners/common.py", line 1564, in apache_beam.runners.common.DoFnRunner._invoke_lifecycle_method
2025-10-15T20:57:50.7882735Z self._reraise_augmented(exn)
2025-10-15T20:57:50.7883146Z File "apache_beam/runners/common.py", line 1609, in apache_beam.runners.common.DoFnRunner._reraise_augmented
2025-10-15T20:57:50.7883244Z raise new_exn
2025-10-15T20:57:50.7883595Z File "apache_beam/runners/common.py", line 1562, in apache_beam.runners.common.DoFnRunner._invoke_lifecycle_method
2025-10-15T20:57:50.7883684Z lifecycle_method()
2025-10-15T20:57:50.7883995Z File "apache_beam/runners/common.py", line 602, in apache_beam.runners.common.DoFnInvoker.invoke_setup
2025-10-15T20:57:50.7884145Z self.signature.setup_lifecycle_method.method_value()
2025-10-15T20:57:50.7884837Z File "/runner/_work/beam/beam/sdks/python/test-suites/tox/py312/build/srcs/sdks/python/target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/apache_beam/transforms/util.py", line 428, in setup
2025-10-15T20:57:50.7884984Z self._hmac_key = self.hmac_key_secret.get_secret_bytes()
2025-10-15T20:57:50.7885074Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-10-15T20:57:50.7885798Z File "/runner/_work/beam/beam/sdks/python/test-suites/tox/py312/build/srcs/sdks/python/target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/apache_beam/transforms/util.py", line 408, in get_secret_bytes
2025-10-15T20:57:50.7885894Z raise RuntimeError(
2025-10-15T20:57:50.7887090Z RuntimeError: Failed to retrieve secret bytes for secret projects/apache-beam-testing/secrets/gbek_util_secret_tests_1015_205647_py312/versions/latest with exception 404 Secret [projects/844138762903/secrets/gbek_util_secret_tests_1015_205647_py312] not found or has no versions. [while running 'assert_that/Group/CoGroupByKeyImpl/GroupByKey/Group by encrypted key/ParDo(_EncryptMessage)']

[damccorm]

Thanks, do you have a link to the workflow where you saw this?

[damccorm]

Actually, I bet I know what the issue is. I tried to fix this flakiness with #36526 but I think I need to move the setup/teardown to be a class level operation

[tvalentyn]

A different failure mode:

2025-10-15T21:59:09.8236405Z E google.api_core.exceptions.AlreadyExists: 409 Secret [projects/844138762903/secrets/gbek_util_secret_tests_1015_215511_py312] already exists.

Details

```

2025-10-15T21:59:09.7128542Z =================================== FAILURES ===================================
2025-10-15T21:59:09.7153248Z _______ GroupByEncryptedKeyTest.test_gbek_fake_secret_manager_roundtrips _______
2025-10-15T21:59:09.7156065Z [gw1] linux -- Python 3.12.10 /runner/work/beam/beam/sdks/python/test-suites/tox/py312/build/srcs/sdks/python/target/.tox-py312-cloud/py312-cloud/bin/python
2025-10-15T21:59:09.7157854Z
2025-10-15T21:59:09.7158735Z args = (name: "projects/apache-beam-testing/secrets/gbek_util_secret_tests_1015_215511_py312"
2025-10-15T21:59:09.7160137Z ,)
2025-10-15T21:59:09.7162670Z kwargs = {'metadata': [('x-goog-request-params', 'name=projects/apache-beam-testing/secrets/gbek_util_secret_tests_1015_215511...x-goog-api-client', 'gl-python/3.12.10 grpc/1.65.5 gax/2.26.0 gapic/2.25.0 pb/5.29.5')], 'timeout': 59.997403144836426}
2025-10-15T21:59:09.7165120Z
2025-10-15T21:59:09.7165584Z @functools.wraps(callable_)
2025-10-15T21:59:09.7166448Z def error_remapped_callable(*args, **kwargs):
2025-10-15T21:59:09.7167381Z try:
2025-10-15T21:59:09.7167979Z > return callable_(*args, **kwargs)
2025-10-15T21:59:09.7168846Z ^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-10-15T21:59:09.7169492Z
2025-10-15T21:59:09.7170463Z target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/google/api_core/grpc_helpers.py:75:
2025-10-15T21:59:09.7172060Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
2025-10-15T21:59:09.7173586Z target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/grpc/_interceptor.py:277: in call
2025-10-15T21:59:09.7175109Z response, ignored_call = self._with_call(
2025-10-15T21:59:09.7177299Z target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/grpc/_interceptor.py:332: in _with_call
2025-10-15T21:59:09.7178915Z return call.result(), call
2025-10-15T21:59:09.7179629Z ^^^^^^^^^^^^^
2025-10-15T21:59:09.7180899Z target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/grpc/_channel.py:440: in result
2025-10-15T21:59:09.7182239Z raise self
2025-10-15T21:59:09.7183529Z target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/grpc/_interceptor.py:315: in continuation
2025-10-15T21:59:09.7197268Z response, call = self._thunk(new_method).with_call(
2025-10-15T21:59:09.7198717Z target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/grpc/_channel.py:1198: in with_call
2025-10-15T21:59:09.7200321Z return _end_unary_response_blocking(state, call, True, None)
2025-10-15T21:59:09.7201299Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-10-15T21:59:09.7202290Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
2025-10-15T21:59:09.7202900Z
2025-10-15T21:59:09.7203382Z state = <grpc._channel._RPCState object at 0x7fb9546c8800>
2025-10-15T21:59:09.7204660Z call = <grpc._cython.cygrpc.SegregatedCall object at 0x7fb986e82140>
2025-10-15T21:59:09.7205936Z with_call = True, deadline = None
2025-10-15T21:59:09.7206438Z
2025-10-15T21:59:09.7206717Z def _end_unary_response_blocking(
2025-10-15T21:59:09.7207437Z state: _RPCState,
2025-10-15T21:59:09.7208246Z call: cygrpc.SegregatedCall,
2025-10-15T21:59:09.7209105Z with_call: bool,
2025-10-15T21:59:09.7209768Z deadline: Optional[float],
2025-10-15T21:59:09.7210915Z ) -> Union[ResponseType, Tuple[ResponseType, grpc.Call]]:
2025-10-15T21:59:09.7211992Z if state.code is grpc.StatusCode.OK:
2025-10-15T21:59:09.7212807Z if with_call:
2025-10-15T21:59:09.7213786Z rendezvous = _MultiThreadedRendezvous(state, call, None, deadline)
2025-10-15T21:59:09.7215077Z return state.response, rendezvous
2025-10-15T21:59:09.7216045Z else:
2025-10-15T21:59:09.7216779Z return state.response
2025-10-15T21:59:09.7217550Z else:
2025-10-15T21:59:09.7218446Z > raise InactiveRpcError(state) # pytype: disable=not-instantiable
2025-10-15T21:59:09.7219555Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-10-15T21:59:09.7221263Z E grpc.channel.InactiveRpcError: <InactiveRpcError of RPC that terminated with:
2025-10-15T21:59:09.7222591Z E status = StatusCode.NOT_FOUND
2025-10-15T21:59:09.7224082Z E details = "Secret [projects/844138762903/secrets/gbek_util_secret_tests_1015_215511_py312] not found."
2025-10-15T21:59:09.7248032Z E debug_error_string = "UNKNOWN:Error received from peer ipv4:142.250.152.95:443 {created_time:"2025-10-15T21:55:11.542009081+00:00", grpc_status:5, grpc_message:"Secret [projects/844138762903/secrets/gbek_util_secret_tests_1015_215511_py312] not found."}"
2025-10-15T21:59:09.7250787Z E >
2025-10-15T21:59:09.7251155Z
2025-10-15T21:59:09.7252239Z target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/grpc/channel.py:1006: InactiveRpcError
2025-10-15T21:59:09.7253465Z
2025-10-15T21:59:09.7254152Z The above exception was the direct cause of the following exception:
2025-10-15T21:59:09.7255010Z
2025-10-15T21:59:09.7257046Z self = <apache_beam.transforms.util_test.GroupByEncryptedKeyTest testMethod=test_gbek_fake_secret_manager_roundtrips>
2025-10-15T21:59:09.7258460Z
2025-10-15T21:59:09.7258792Z def setUp(self):
2025-10-15T21:59:09.7259439Z if secretmanager is not None:
2025-10-15T21:59:09.7260365Z self.project_id = 'apache-beam-testing'
2025-10-15T21:59:09.7261437Z py_version = f'py{sys.version_info.major}{sys.version_info.minor}'
2025-10-15T21:59:09.7262781Z secret_postfix = datetime.now().strftime('%m%d%H%M%S') + py_version
2025-10-15T21:59:09.7264100Z self.secret_id = 'gbek_util_secret_tests' + secret_postfix
2025-10-15T21:59:09.7265863Z self.client = secretmanager.SecretManagerServiceClient()
2025-10-15T21:59:09.7267068Z self.project_path = f'projects/{self.project_id}'
2025-10-15T21:59:09.7268249Z self.secret_path = f'{self.project_path}/secrets/{self.secret_id}'
2025-10-15T21:59:09.7269448Z try:
2025-10-15T21:59:09.7270261Z > self.client.get_secret(request={'name': self.secret_path})
2025-10-15T21:59:09.7271108Z
2025-10-15T21:59:09.7271417Z apache_beam/transforms/util_test.py:329:
2025-10-15T21:59:09.7272321Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
2025-10-15T21:59:09.7274531Z target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/google/cloud/secretmanager_v1/services/secret_manager_service/client.py:1265: in get_secret
2025-10-15T21:59:09.7298054Z response = rpc(
2025-10-15T21:59:09.7299600Z target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/google/api_core/gapic_v1/method.py:131: in call
2025-10-15T21:59:09.7301364Z return wrapped_func(*args, **kwargs)
2025-10-15T21:59:09.7302203Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-10-15T21:59:09.7303784Z target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/google/api_core/timeout.py:130: in func_with_timeout
2025-10-15T21:59:09.7305637Z return func(*args, **kwargs)
2025-10-15T21:59:09.7306388Z ^^^^^^^^^^^^^^^^^^^^^
2025-10-15T21:59:09.7307302Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
2025-10-15T21:59:09.7308062Z
2025-10-15T21:59:09.7308913Z args = (name: "projects/apache-beam-testing/secrets/gbek_util_secret_tests_1015_215511_py312"
2025-10-15T21:59:09.7310167Z ,)
2025-10-15T21:59:09.7312638Z kwargs = {'metadata': [('x-goog-request-params', 'name=projects/apache-beam-testing/secrets/gbek_util_secret_tests_1015_215511...x-goog-api-client', 'gl-python/3.12.10 grpc/1.65.5 gax/2.26.0 gapic/2.25.0 pb/5.29.5')], 'timeout': 59.997403144836426}
2025-10-15T21:59:09.7315083Z
2025-10-15T21:59:09.7315501Z @functools.wraps(callable)
2025-10-15T21:59:09.7316369Z def error_remapped_callable(*args, **kwargs):
2025-10-15T21:59:09.7317157Z try:
2025-10-15T21:59:09.7317799Z return callable(*args, **kwargs)
2025-10-15T21:59:09.7318780Z except grpc.RpcError as exc:
2025-10-15T21:59:09.7320110Z > raise exceptions.from_grpc_error(exc) from exc
2025-10-15T21:59:09.7321928Z E google.api_core.exceptions.NotFound: 404 Secret [projects/844138762903/secrets/gbek_util_secret_tests_1015_215511_py312] not found.
2025-10-15T21:59:09.7323403Z
2025-10-15T21:59:09.7324451Z target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/google/api_core/grpc_helpers.py:77: NotFound
2025-10-15T21:59:09.7326425Z
2025-10-15T21:59:09.7326985Z During handling of the above exception, another exception occurred:
2025-10-15T21:59:09.7327779Z
2025-10-15T21:59:09.7328138Z args = (parent: "projects/apache-beam-testing"
2025-10-15T21:59:09.7329307Z secret_id: "gbek_util_secret_tests_1015_215511_py312"
2025-10-15T21:59:09.7330115Z secret {
2025-10-15T21:59:09.7330683Z replication {
2025-10-15T21:59:09.7331272Z automatic {
2025-10-15T21:59:09.7332006Z }
2025-10-15T21:59:09.7332563Z }
2025-10-15T21:59:09.7333033Z }
2025-10-15T21:59:09.7333471Z ,)
2025-10-15T21:59:09.7335591Z kwargs = {'metadata': [('x-goog-request-params', 'parent=projects/apache-beam-testing'), ('x-goog-api-client', 'gl-python/3.12.10 grpc/1.65.5 gax/2.26.0 gapic/2.25.0 pb/5.29.5')], 'timeout': 60.0}
2025-10-15T21:59:09.7337519Z
2025-10-15T21:59:09.7337787Z @functools.wraps(callable)
2025-10-15T21:59:09.7338719Z def error_remapped_callable(*args, **kwargs):
2025-10-15T21:59:09.7339521Z try:
2025-10-15T21:59:09.7340188Z > return callable(*args, **kwargs)
2025-10-15T21:59:09.7341024Z ^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-10-15T21:59:09.7341510Z
2025-10-15T21:59:09.7342428Z target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/google/api_core/grpc_helpers.py:75:
2025-10-15T21:59:09.7344431Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
2025-10-15T21:59:09.7346109Z target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/grpc/_interceptor.py:277: in call
2025-10-15T21:59:09.7347617Z response, ignored_call = self._with_call(
2025-10-15T21:59:09.7349190Z target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/grpc/_interceptor.py:332: in _with_call
2025-10-15T21:59:09.7350743Z return call.result(), call
2025-10-15T21:59:09.7351446Z ^^^^^^^^^^^^^
2025-10-15T21:59:09.7352698Z target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/grpc/_channel.py:440: in result
2025-10-15T21:59:09.7354050Z raise self
2025-10-15T21:59:09.7395775Z target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/grpc/_interceptor.py:315: in continuation
2025-10-15T21:59:09.7397671Z response, call = self._thunk(new_method).with_call(
2025-10-15T21:59:09.7399412Z target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/grpc/_channel.py:1198: in with_call
2025-10-15T21:59:09.7401115Z return _end_unary_response_blocking(state, call, True, None)
2025-10-15T21:59:09.7402098Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-10-15T21:59:09.7403035Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
2025-10-15T21:59:09.7403687Z
2025-10-15T21:59:09.7404166Z state = <grpc._channel._RPCState object at 0x7fb9546ca780>
2025-10-15T21:59:09.7405554Z call = <grpc._cython.cygrpc.SegregatedCall object at 0x7fb984112780>
2025-10-15T21:59:09.7406711Z with_call = True, deadline = None
2025-10-15T21:59:09.7407197Z
2025-10-15T21:59:09.7407574Z def _end_unary_response_blocking(
2025-10-15T21:59:09.7408307Z state: _RPCState,
2025-10-15T21:59:09.7409057Z call: cygrpc.SegregatedCall,
2025-10-15T21:59:09.7409873Z with_call: bool,
2025-10-15T21:59:09.7410535Z deadline: Optional[float],
2025-10-15T21:59:09.7411479Z ) -> Union[ResponseType, Tuple[ResponseType, grpc.Call]]:
2025-10-15T21:59:09.7412533Z if state.code is grpc.StatusCode.OK:
2025-10-15T21:59:09.7413356Z if with_call:
2025-10-15T21:59:09.7414352Z rendezvous = _MultiThreadedRendezvous(state, call, None, deadline)
2025-10-15T21:59:09.7415701Z return state.response, rendezvous
2025-10-15T21:59:09.7417019Z else:
2025-10-15T21:59:09.7417658Z return state.response
2025-10-15T21:59:09.7418360Z else:
2025-10-15T21:59:09.8111878Z > raise _InactiveRpcError(state) # pytype: disable=not-instantiable
2025-10-15T21:59:09.8113967Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-10-15T21:59:09.8116029Z E grpc._channel._InactiveRpcError: <InactiveRpcError of RPC that terminated with:
2025-10-15T21:59:09.8117548Z E status = StatusCode.ALREADY_EXISTS
2025-10-15T21:59:09.8121142Z E details = "Secret [projects/844138762903/secrets/gbek_util_secret_tests_1015_215511_py312] already exists."
2025-10-15T21:59:09.8124810Z E debug_error_string = "UNKNOWN:Error received from peer ipv4:142.250.152.95:443 {created_time:"2025-10-15T21:55:11.639695647+00:00", grpc_status:6, grpc_message:"Secret [projects/844138762903/secrets/gbek_util_secret_tests_1015_215511_py312] already exists."}"
2025-10-15T21:59:09.8135956Z E >
2025-10-15T21:59:09.8136354Z
2025-10-15T21:59:09.8137377Z target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/grpc/channel.py:1006: InactiveRpcError
2025-10-15T21:59:09.8138635Z
2025-10-15T21:59:09.8139213Z The above exception was the direct cause of the following exception:
2025-10-15T21:59:09.8140074Z
2025-10-15T21:59:09.8141185Z self = <apache_beam.transforms.util_test.GroupByEncryptedKeyTest testMethod=test_gbek_fake_secret_manager_roundtrips>
2025-10-15T21:59:09.8142628Z
2025-10-15T21:59:09.8142878Z def setUp(self):
2025-10-15T21:59:09.8143541Z if secretmanager is not None:
2025-10-15T21:59:09.8144442Z self.project_id = 'apache-beam-testing'
2025-10-15T21:59:09.8154780Z py_version = f'py{sys.version_info.major}{sys.version_info.minor}'
2025-10-15T21:59:09.8158156Z secret_postfix = datetime.now().strftime('%m%d%H%M%S') + py_version
2025-10-15T21:59:09.8159453Z self.secret_id = 'gbek_util_secret_tests' + secret_postfix
2025-10-15T21:59:09.8160757Z self.client = secretmanager.SecretManagerServiceClient()
2025-10-15T21:59:09.8161952Z self.project_path = f'projects/{self.project_id}'
2025-10-15T21:59:09.8163143Z self.secret_path = f'{self.project_path}/secrets/{self.secret_id}'
2025-10-15T21:59:09.8164172Z try:
2025-10-15T21:59:09.8164914Z self.client.get_secret(request={'name': self.secret_path})
2025-10-15T21:59:09.8166055Z except Exception:
2025-10-15T21:59:09.8166825Z > self.client.create_secret(
2025-10-15T21:59:09.8167654Z request={
2025-10-15T21:59:09.8168411Z 'parent': self.project_path,
2025-10-15T21:59:09.8169368Z 'secret_id': self.secret_id,
2025-10-15T21:59:09.8170173Z 'secret': {
2025-10-15T21:59:09.8170840Z 'replication': {
2025-10-15T21:59:09.8171675Z 'automatic': {}
2025-10-15T21:59:09.8172451Z }
2025-10-15T21:59:09.8173052Z }
2025-10-15T21:59:09.8173634Z })
2025-10-15T21:59:09.8173966Z
2025-10-15T21:59:09.8174275Z apache_beam/transforms/util_test.py:331:
2025-10-15T21:59:09.8175175Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
2025-10-15T21:59:09.8177326Z target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/google/cloud/secretmanager_v1/services/secret_manager_service/client.py:1024: in create_secret
2025-10-15T21:59:09.8179388Z response = rpc(
2025-10-15T21:59:09.8180843Z target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/google/api_core/gapic_v1/method.py:131: in call
2025-10-15T21:59:09.8182577Z return wrapped_func(*args, **kwargs)
2025-10-15T21:59:09.8183412Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-10-15T21:59:09.8184953Z target/.tox-py312-cloud/py312-cloud/lib/python3.12/site-packages/google/api_core/timeout.py:130: in func_with_timeout
2025-10-15T21:59:09.8217345Z return func(*args, **kwargs)
2025-10-15T21:59:09.8218534Z ^^^^^^^^^^^^^^^^^^^^^
2025-10-15T21:59:09.8219398Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
2025-10-15T21:59:09.8220070Z
2025-10-15T21:59:09.8220422Z args = (parent: "projects/apache-beam-testing"
2025-10-15T21:59:09.8221449Z secret_id: "gbek_util_secret_tests_1015_215511_py312"
2025-10-15T21:59:09.8222381Z secret {
2025-10-15T21:59:09.8222952Z replication {
2025-10-15T21:59:09.8223506Z automatic {
2025-10-15T21:59:09.8224006Z }
2025-10-15T21:59:09.8224483Z }
2025-10-15T21:59:09.8225066Z }
2025-10-15T21:59:09.8225768Z ,)
2025-10-15T21:59:09.8227723Z kwargs = {'metadata': [('x-goog-request-params', 'parent=projects/apache-beam-testing'), ('x-goog-api-client', 'gl-python/3.12.10 grpc/1.65.5 gax/2.26.0 gapic/2.25.0 pb/5.29.5')], 'timeout': 60.0}
2025-10-15T21:59:09.8229640Z
2025-10-15T21:59:09.8230021Z @functools.wraps(callable)
2025-10-15T21:59:09.8230913Z def error_remapped_callable(*args, **kwargs):
2025-10-15T21:59:09.8231841Z try:
2025-10-15T21:59:09.8232476Z return callable(*args, **kwargs)
2025-10-15T21:59:09.8233366Z except grpc.RpcError as exc:
2025-10-15T21:59:09.8234323Z > raise exceptions.from_grpc_error(exc) from exc
2025-10-15T21:59:09.8236405Z E google.api_core.exceptions.AlreadyExists: 409 Secret [projects/844138762903/secrets/gbek_util_secret_tests_1015_215511_py312] already exists.

</<details>

This one is from https://github.com/apache/beam/actions/runs/18543048965/job/52855054053?pr=36367

[tvalentyn]

Thanks, do you have a link to the workflow where you saw this?

it was one of the failures in #36518

[damccorm]

Letting tests run, but I think #36546 should solve this