Skip to content

Auto-infer BigQuery schema from schema'd PCollections for FILE_LOADS - #39662

Open
nikitagrover19 wants to merge 2 commits into
apache:masterfrom
nikitagrover19:file-loads-schema-inference-28162
Open

Auto-infer BigQuery schema from schema'd PCollections for FILE_LOADS#39662
nikitagrover19 wants to merge 2 commits into
apache:masterfrom
nikitagrover19:file-loads-schema-inference-28162

Conversation

@nikitagrover19

@nikitagrover19 nikitagrover19 commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

Fixes #28162

What

WriteToBigQuery with method=FILE_LOADS previously required an explicit
schema, even when the input PCollection already carried a Beam schema - for example a PCollection produced by
ReadFromBigQuery(..., output_type='BEAM_ROW'), or any PCollection of
NamedTuples, dataclasses, or Beam Rows. The STORAGE_WRITE_API method
already auto-infers a schema in this situation; this brings FILE_LOADS
(both JSON and AVRO temp file formats) in line with that behavior.

How

  • Added beam_schema_to_bq_table_schema() in bigquery_schema_tools.py,
    the reverse of the existing generate_user_type_from_bq_schema(). It
    converts a Beam schema_pb2.Schema into a BigQuery TableSchema dict,
    handling atomic types, logical types (Timestamp, Date, Decimal),
    nested/repeated fields, and nullability, and raises a clear ValueError
    for types with no BigQuery equivalent (e.g. MapType).
  • WriteToBigQuery.expand()'s FILE_LOADS branch now tries
    schema_from_element_type(pcoll.element_type) when self.schema is None, before the existing "schema required for AVRO" check, so AVRO
    loads benefit from inference too, not just JSON.
  • Falls back to the prior behavior (schema stays None) when the
    PCollection has no attached Beam schema (e.g. plain dicts), so this is
    purely additive and non-breaking.

Testing

  • Unit tests for the schema conversion covering atomic types, timestamps,
    decimals, nullable fields, repeated fields, nested records, repeated
    records, and the unsupported-MapType error case
    (bigquery_schema_tools_test.py).
  • Integration-style tests on WriteToBigQuery confirming: schema
    auto-inference for JSON file loads, schema auto-inference for AVRO file
    loads (previously always raised "A schema must be provided" here),
    and that plain-dict PCollections are unaffected
    (bigquery_test.py).
  • Ran the existing TestWriteToBigQuery test class locally to check for
    regressions; all passing except one pre-existing test that requires
    real GCP credentials unavailable in my local environment (unrelated to
    this change).

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

  • Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
  • Update CHANGES.md with noteworthy changes.
  • If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

Build python source distribution and wheels
Python tests
Java tests
Go tests

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

nikitagrover19 and others added 2 commits August 6, 2026 23:41
Adds beam_schema_to_bq_table_schema() to convert a Beam schema into a
BigQuery TableSchema dict, and uses it in WriteToBigQuery's FILE_LOADS
path (JSON and AVRO) when no explicit schema is provided but the input
PCollection carries a Beam schema (e.g. NamedTuples, dataclasses, or
Beam Rows). Mirrors the existing STORAGE_WRITE_API auto-inference.

Fixes apache#28162
@nikitagrover19
nikitagrover19 marked this pull request as draft August 6, 2026 18:46
@nikitagrover19
nikitagrover19 marked this pull request as ready for review August 6, 2026 19:05
@github-actions

github-actions Bot commented Aug 7, 2026

Copy link
Copy Markdown
Contributor

Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment assign set of reviewers

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: WriteToBigQuery FILE_LOADS method should support writing schema'd PCollections.

1 participant