KeyCommitTooLarge logging improvements by rwiggles · Pull Request #39316 · apache/beam

rwiggles · 2026-07-14T00:28:07Z

Support logging the key during a KeyCommitTooLarge if EnableHotKeyLogging is turned on. Also switch to logging the dfe name instead of the computation name and fix an overflow in logging the sharding key.

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
Update CHANGES.md with noteworthy changes.
If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

…ging is turned on. Also switch to logging the dfe name instead of the computation name and fix an overflow in logging the sharding key

gemini-code-assist · 2026-07-14T00:28:16Z

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request enhances the logging capabilities for KeyCommitTooLargeException in the Dataflow worker. By allowing the inclusion of the specific key in error messages, it improves debuggability for large commit issues. The change introduces a configurable flag to toggle this behavior, ensuring it can be enabled selectively via worker options or experiment flags.

Highlights

Hot Key Logging: Introduced an optional hot key logging feature that includes the key in the KeyCommitTooLargeException error message when enabled.
Configuration: Added a new boolean flag 'hotKeyLoggingEnabled' to the StreamingWorkScheduler, configurable via worker options or an experiment flag.
Testing: Expanded StreamingDataflowWorkerTest to verify that the key is correctly included or excluded from error messages based on the configuration.

New Features

🧠 You can now enable Memory (public preview) to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist

Code Review

This pull request adds support for hot key logging in KeyCommitTooLargeException by passing a hotKeyLoggingEnabled flag and updates the streaming work scheduler and tests accordingly. However, the changes introduce several compilation issues due to importing standard instead of vendored dependencies (Guava, Protobuf, and gRPC) across multiple files, as well as importing DataflowRunner into the worker module, which causes a circular dependency. The feedback advises replacing these with their vendored counterparts and checking the experiments list directly from the options.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

github-actions · 2026-07-14T01:38:14Z

Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment assign set of reviewers

…ache/beam/runners/dataflow/worker/streaming/KeyCommitTooLargeException.java Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

…ache/beam/runners/dataflow/worker/windmill/work/processing/StreamingWorkScheduler.java Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

…ache/beam/runners/dataflow/worker/StreamingDataflowWorkerTest.java Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

Bumps [actions/checkout](https://github.com/actions/checkout) from 6 to 7. - [Release notes](https://github.com/actions/checkout/releases) - [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md) - [Commits](actions/checkout@v6...v7) --- updated-dependencies: - dependency-name: actions/checkout dependency-version: '7' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

…ache#39333) Bumps [cloud.google.com/go/datastore](https://github.com/googleapis/google-cloud-go) from 1.24.0 to 1.25.0. - [Release notes](https://github.com/googleapis/google-cloud-go/releases) - [Changelog](https://github.com/googleapis/google-cloud-go/blob/main/documentai/CHANGES.md) - [Commits](googleapis/google-cloud-go@kms/v1.24.0...kms/v1.25.0) --- updated-dependencies: - dependency-name: cloud.google.com/go/datastore dependency-version: 1.25.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

…#39334) Bumps [github.com/aws/aws-sdk-go-v2/feature/s3/manager](https://github.com/aws/aws-sdk-go-v2) from 1.22.32 to 1.22.33. - [Release notes](https://github.com/aws/aws-sdk-go-v2/releases) - [Commits](aws/aws-sdk-go-v2@feature/s3/manager/v1.22.32...feature/s3/manager/v1.22.33) --- updated-dependencies: - dependency-name: github.com/aws/aws-sdk-go-v2/feature/s3/manager dependency-version: 1.22.33 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

)

…39337)

…closes apache#39188)

rwiggles · 2026-07-17T20:08:38Z

PreCommit Java Dataflow Non-portable Worker is perma red but I was able to get a passing run locally. Issue is being tracked at: #38286

arunpandianp

a couple of nits. rest looks good to me.

arunpandianp · 2026-07-22T21:34:12Z

-
-    // Check this explicitly since the estimated commit bytes weren't actually
+    // Check this explicitly since the estimated commit bytes weren't actuallyExpand commentComment
+    // on line L1340


fix comment

…ache/beam/runners/dataflow/worker/StreamingModeExecutionContext.java Co-authored-by: Arun Pandian <arunpandianp@gmail.com>

github-actions · 2026-07-22T22:51:04Z

Assigning reviewers:

R: @shunping added as fallback since no labels match configuration

Note: If you would like to opt out of this review, comment assign to next reviewer.

Available commands:

stop reviewer notifications - opt out of the automated review tooling
remind me after tests pass - tag the comment author after tests pass
waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

rwiggles · 2026-07-28T21:52:21Z

Example of what this looks like

rwiggles and others added 2 commits July 14, 2026 00:22

Support logging the key during a KeyCommitTooLarge if EnableHotKeyLog…

c831c33

…ging is turned on. Also switch to logging the dfe name instead of the computation name and fix an overflow in logging the sharding key

Merge branch 'apache:master' into master

986fae8

github-actions Bot added runners dataflow labels Jul 14, 2026

gemini-code-assist Bot reviewed Jul 14, 2026

View reviewed changes

rwiggles and others added 10 commits July 14, 2026 11:01

Update runners/google-cloud-dataflow-java/worker/src/main/java/org/ap…

25d7bbc

…ache/beam/runners/dataflow/worker/streaming/KeyCommitTooLargeException.java Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

Update runners/google-cloud-dataflow-java/worker/src/main/java/org/ap…

2513ffe

…ache/beam/runners/dataflow/worker/windmill/work/processing/StreamingWorkScheduler.java Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

Update runners/google-cloud-dataflow-java/worker/src/test/java/org/ap…

647a815

…ache/beam/runners/dataflow/worker/StreamingDataflowWorkerTest.java Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

gemini review responses

369aff8

Merge branch 'apache:master' into master

8f55913

Fix imports

15751e2

one more

d3a4d1c

one more attempt

d5babb0

one more attempt

9f611b5

formatting fix

a5bd7d0

arunpandianp reviewed Jul 15, 2026

View reviewed changes

Comment thread .../main/java/org/apache/beam/runners/dataflow/worker/streaming/KeyCommitTooLargeException.java Outdated

dependabot Bot and others added 10 commits July 15, 2026 22:19

Fix DataflowV1 test failure by fixing getSimpleName access (apache#39330

4e1732b

)

add jdkAddOpenModules for Dataflow V2 (apache#39306)

2bea70b

Add Spark JVM --add-opens for (Nexmark, TPC-DS, PortableJar) (apache#…

1e6b61b

…39337)

add Spark job-server --add-opens (apache#39339)

163cd12

fix: add retries and query parameter encoding for GitHub API requests (…

dca9622

…closes apache#39188)

fix: add Apache license header to test_sending.py

7b88d16

Move validation to StreamingModeExecutionContext

5939496

github-actions Bot added python java labels Jul 17, 2026

github-actions Bot added go build docker portability labels Jul 17, 2026

remove unused import

a3b7163

Merge branch 'master' into master

c31b3e9

github-actions Bot removed python java go build docker portability labels Jul 17, 2026

arunpandianp reviewed Jul 19, 2026

View reviewed changes

arunpandianp suggested changes Jul 19, 2026

View reviewed changes

rwiggles added 2 commits July 22, 2026 19:19

respond to comments

fffa499

bugfix

883e7e7

rwiggles requested a review from arunpandianp July 22, 2026 21:27

arunpandianp reviewed Jul 22, 2026

View reviewed changes

rwiggles and others added 2 commits July 22, 2026 14:47

Update runners/google-cloud-dataflow-java/worker/src/main/java/org/ap…

30b5b36

…ache/beam/runners/dataflow/worker/StreamingModeExecutionContext.java Co-authored-by: Arun Pandian <arunpandianp@gmail.com>

respond to comments

c716973

rwiggles requested a review from arunpandianp July 22, 2026 21:53

arunpandianp approved these changes Jul 22, 2026

View reviewed changes

github-actions Bot added the Next Action: Reviewers label Jul 22, 2026

Abacn merged commit 5e49938 into apache:master Jul 23, 2026
16 checks passed

Navigation Menu

Uh oh!

KeyCommitTooLarge logging improvements - #39316

KeyCommitTooLarge logging improvements#39316
Abacn merged 28 commits into
apache:masterfrom
rwiggles:master

rwiggles commented Jul 14, 2026 •

edited

Loading

Uh oh!

gemini-code-assist Bot commented Jul 14, 2026

Uh oh!

gemini-code-assist Bot left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Jul 14, 2026

Uh oh!

Uh oh!

rwiggles commented Jul 17, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

arunpandianp left a comment

Uh oh!

arunpandianp Jul 22, 2026

Uh oh!

Uh oh!

github-actions Bot commented Jul 22, 2026

Uh oh!

Uh oh!

rwiggles commented Jul 28, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Uh oh!

Conversation

rwiggles commented Jul 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

GitHub Actions Tests Status (on master branch)

Uh oh!

gemini-code-assist Bot commented Jul 14, 2026

Summary of Changes

Highlights

Footnotes

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Jul 14, 2026

Uh oh!

Uh oh!

rwiggles commented Jul 17, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

arunpandianp left a comment

Choose a reason for hiding this comment

Uh oh!

arunpandianp Jul 22, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

github-actions Bot commented Jul 22, 2026

Uh oh!

Uh oh!

rwiggles commented Jul 28, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

rwiggles commented Jul 14, 2026 •

edited

Loading