Skip to content

Fix PreCommit Python Transforms job #36031

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
Amar3tto wants to merge 1 commit into from
Closed

Fix PreCommit Python Transforms job #36031

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 19 additions & 2 deletions sdks/python/apache_beam/transforms/external_test.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,7 @@
from apache_beam import Pipeline
from apache_beam.coders import RowCoder
from apache_beam.options.pipeline_options import PipelineOptions
from apache_beam.pipeline import PipelineVisitor
from apache_beam.portability.api import beam_expansion_api_pb2
from apache_beam.portability.api import external_transforms_pb2
from apache_beam.portability.api import schema_pb2
Expand Down Expand Up @@ -349,6 +350,22 @@ def test_external_transform_finder_non_leaf(self):
self.assertTrue(pipeline.contains_external_transforms)

def test_external_transform_finder_leaf(self):
def has_external(p):
class Finder(PipelineVisitor):
def __init__(self):
self.matches = []

def enter_composite_transform(self, node):
if isinstance(node.transform, beam.ExternalTransform):
self.matches.append(node.full_label)

def visit_transform(self, node):
self.enter_composite_transform(node)

finder = Finder()
p.visit(finder)
return finder.matches

pipeline = beam.Pipeline()
_ = (
pipeline
Expand All @@ -357,9 +374,9 @@ def test_external_transform_finder_leaf(self):
'beam:transforms:xlang:test:nooutput',
ImplicitSchemaPayloadBuilder({'data': '0'}),
expansion_service.ExpansionServiceServicer()))
pipeline.run().wait_until_finish()

self.assertTrue(pipeline.contains_external_transforms)
found = has_external(pipeline)
self.assertTrue(found, f"No ExternalTransform found; saw: {found}")
Comment on lines +378 to +379

@damccorm damccorm Sep 4, 2025

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this removes the purpose of what we're testing here - we are trying to make sure that Beam can correctly identify external transforms when we run it.

@damccorm damccorm Sep 4, 2025

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the core problem is that this is running on Prism, but Prism theoretically should be excluding this -

beam/sdks/python/apache_beam/runners/direct/direct_runner.py

Line 148 in 67eaded

if isinstance(transform, beam.ExternalTransform):

I'm not sure why it isn't yet, but taking a look

@damccorm damccorm Sep 4, 2025

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Letting tests run now, but I validated #36049 should work locally and I think it is a better fix.

@Amar3tto Amar3tto Sep 4, 2025

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, let's try.
Also the issue usually relates only to this particular 'no-output' transform


def test_sanitize_java_traceback(self):
error_string = '''
Expand Down
4 changes: 3 additions & 1 deletion sdks/python/apache_beam/transforms/util_test.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -762,7 +762,7 @@ def process(self, element):
equal_to(expected_windows),
label='before_identity',
reify_windows=True)
_ = (
after = (
before_identity
| 'window' >> beam.WindowInto(
beam.transforms.util._IdentityWindowFn(
Expand All @@ -772,6 +772,8 @@ def process(self, element):
# contain a window of None. IdentityWindowFn should
# raise an exception.
| 'add_timestamps2' >> beam.ParDo(AddTimestampDoFn()))
# Terminal consumer to prevent pruning:
_ = after | 'count_per_element' >> beam.combiners.Count.PerElement()


class ReshuffleTest(unittest.TestCase):
Expand Down
Loading