[Failing Test]: beam_PreCommit_Yaml_Xlang_Direct fails consistently (io expansion-service shadowJar not found)

[tkaymak]

What happened?

The beam_PreCommit_Yaml_Xlang_Direct job (:sdks:python:yamlIntegrationTests) fails consistently on master — the last several scheduled runs are all red. It is not an intermittent flake.

The external-Java yaml integration tests fail because the IO expansion-service shadowJar is missing at runtime:

RuntimeError: .../sdks/java/io/expansion-service/build/libs/beam-sdks-java-io-expansion-service-2.75.0-SNAPSHOT.jar not found. Please build the server with ...
ValueError: Invalid transform specification at "IcebergAddFiles" ... jar not found

Affected tests include (all using the external Java provider): IcebergTest, Iceberg_Add_FilesTest, Iceberg_Add_Files_BatchTest, MongodbTest, FlattenTest, PartitionTest (WriteToJson_ExternalJavaProvider). The DatadogTest::test_only failure (Mismatch in recorded Datadog events!) also appears in the same runs.

The yamlIntegrationTests gradle task dependsOn :sdks:java:io:expansion-service:build, but the tests look up the shadowJar specifically, which build does not necessarily produce — so the jar is absent when the test resolves the external provider.

Recent failing runs on master:

• https://github.com/apache/beam/actions/runs/27549683373 (2026-06-15)
• https://github.com/apache/beam/actions/runs/27530225891 (2026-06-15)
• https://github.com/apache/beam/actions/runs/27516024051 (2026-06-14)

Issue Priority

Priority: 2 (default / most failing tests should be filed as P2)

Issue Components

• Component: Python SDK
• Component: Java SDK