Skip to content

[Task]: Support standard portable coder 'beam:coder:schema:v1' in all SDKs #38513

Description

@Abacn

What needs to happen?

The URN is introduced into Beam model in #37376 and after #37631 it started to get emitted from Java pipelines. However it's not recognized in runners written in other languages including Python:

    [context.coders.get_by_id(c) for c in coder_proto.component_coder_ids],
  File "/runner/_work/beam/beam/sdks/python/apache_beam/runners/pipeline_context.py", line 103, in get_by_id
    self._id_to_obj[id] = self._obj_type.from_runner_api(
  File "/runner/_work/beam/beam/sdks/python/apache_beam/coders/coders.py", line 361, in from_runner_api
    parameter_type, constructor = cls._known_urns[coder_proto.spec.urn]
KeyError: 'beam:coder:schema:v1'

Since this is defined in Beam model, we should fix the gaps in all SDK languages

Issue Priority

Priority: 2 (default / most normal work should be filed as P2)

Issue Components

  • Component: Python SDK
  • Component: Java SDK
  • Component: Go SDK
  • Component: Typescript SDK
  • Component: IO connector
  • Component: Beam YAML
  • Component: Beam examples
  • Component: Beam playground
  • Component: Beam katas
  • Component: Website
  • Component: Infrastructure
  • Component: Spark Runner
  • Component: Flink Runner
  • Component: Prism Runner
  • Component: Twister2 Runner
  • Component: Hazelcast Jet Runner
  • Component: Google Cloud Dataflow Runner

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions