From 3ddfe07aa865b89bb968e2c3abfe53ba075e1e19 Mon Sep 17 00:00:00 2001 From: Yi Hu Date: Thu, 30 Apr 2026 11:01:17 -0400 Subject: [PATCH 1/2] Revert "remove processContext usage across examples (java and kotlin) (#37937)" This reverts commit 4151dded54fc3b120e8a2b382ce64f9eee576a06. --- .agent/skills/beam-dofn-modernizer/SKILL.md | 207 ------------------ .../iceberg/IcebergBatchWriteExample.java | 15 +- .../beam/examples/SchemaTransformExample.java | 8 +- .../beam/examples/SqlTransformExample.java | 8 +- .../examples/ApproximateQuantilesExample.java | 8 +- .../examples/CoCombineTransformExample.java | 17 +- .../beam/examples/CoGroupByKeyExample.java | 8 +- .../apache/beam/examples/CombineExample.java | 8 +- .../apache/beam/examples/CountExample.java | 8 +- .../beam/examples/CountPerKeyExample.java | 8 +- .../apache/beam/examples/CreateExample.java | 8 +- .../beam/examples/DebuggingWordCount.java | 13 +- .../apache/beam/examples/DistinctExample.java | 8 +- .../beam/examples/FlatMapElementsExample.java | 8 +- .../examples/GroupIntoBatchesExample.java | 8 +- .../examples/KafkaPassengerCountJson.java | 19 +- .../apache/beam/examples/KafkaStreaming.java | 31 ++- .../beam/examples/KafkaWordCountAvro.java | 9 +- .../beam/examples/KafkaWordCountJson.java | 9 +- .../org/apache/beam/examples/KeysExample.java | 8 +- .../apache/beam/examples/KvSwapExample.java | 8 +- .../apache/beam/examples/LatestExample.java | 8 +- .../beam/examples/MapElementsExample.java | 8 +- .../org/apache/beam/examples/MaxExample.java | 8 +- .../beam/examples/MaxPerKeyExample.java | 8 +- .../org/apache/beam/examples/MeanExample.java | 8 +- .../beam/examples/MeanPerKeyExample.java | 8 +- .../org/apache/beam/examples/MinExample.java | 8 +- .../beam/examples/MinPerKeyExample.java | 8 +- .../beam/examples/PartitionExample.java | 8 +- .../beam/examples/RateLimiterSimple.java | 8 +- .../apache/beam/examples/RegexExample.java | 8 +- .../apache/beam/examples/SampleExample.java | 8 +- .../org/apache/beam/examples/SumExample.java | 8 +- .../beam/examples/SumPerKeyExample.java | 8 +- .../apache/beam/examples/ToStringExample.java | 8 +- .../org/apache/beam/examples/TopExample.java | 8 +- .../apache/beam/examples/ValuesExample.java | 8 +- .../org/apache/beam/examples/ViewExample.java | 19 +- .../apache/beam/examples/WindowExample.java | 8 +- .../beam/examples/complete/AutoComplete.java | 60 +++-- .../complete/StreamingWordExtract.java | 16 +- .../apache/beam/examples/complete/TfIdf.java | 75 +++---- .../complete/TopWikipediaSessions.java | 30 +-- .../examples/complete/TrafficMaxLaneFlow.java | 30 +-- .../beam/examples/complete/TrafficRoutes.java | 46 ++-- .../transforms/DataProtectors.java | 21 +- .../transforms/io/TokenizationBigQueryIO.java | 7 +- .../transforms/io/TokenizationBigTableIO.java | 6 +- .../datatokenization/utils/CsvConverters.java | 107 +++------ .../utils/ErrorConverters.java | 30 +-- .../examples/complete/game/GameStats.java | 45 ++-- .../complete/game/HourlyTeamScore.java | 6 +- .../examples/complete/game/LeaderBoard.java | 18 +- .../complete/game/StatefulTeamScore.java | 18 +- .../examples/complete/game/UserScore.java | 14 +- .../complete/game/utils/WriteToBigQuery.java | 22 +- .../complete/game/utils/WriteToText.java | 21 +- .../game/utils/WriteWindowedToBigQuery.java | 15 +- .../cookbook/BigQueryStreamingTornadoes.java | 21 +- .../examples/cookbook/BigQueryTornadoes.java | 16 +- .../cookbook/CombinePerKeyExamples.java | 14 +- .../examples/cookbook/FilterExamples.java | 31 ++- .../beam/examples/cookbook/JoinExamples.java | 32 ++- .../examples/cookbook/MaxPerKeyExamples.java | 17 +- .../cookbook/MinimalBigQueryTornadoes.java | 18 +- .../examples/cookbook/TriggerExample.java | 47 ++-- .../beam/examples/snippets/Snippets.java | 40 ++-- .../subprocess/ExampleEchoPipeline.java | 10 +- .../examples/cookbook/TriggerExampleTest.java | 8 +- .../subprocess/ExampleEchoPipelineTest.java | 10 +- .../examples/kotlin/DebuggingWordCount.kt | 12 +- .../kotlin/cookbook/BigQueryTornadoes.kt | 15 +- .../kotlin/cookbook/CombinePerKeyExamples.kt | 13 +- .../kotlin/cookbook/FilterExamples.kt | 30 ++- .../examples/kotlin/cookbook/JoinExamples.kt | 31 ++- .../kotlin/cookbook/MaxPerKeyExamples.kt | 15 +- .../kotlin/cookbook/TriggerExample.kt | 42 ++-- .../beam/examples/kotlin/snippets/Snippets.kt | 13 +- 79 files changed, 547 insertions(+), 1062 deletions(-) delete mode 100644 .agent/skills/beam-dofn-modernizer/SKILL.md diff --git a/.agent/skills/beam-dofn-modernizer/SKILL.md b/.agent/skills/beam-dofn-modernizer/SKILL.md deleted file mode 100644 index d45fc3fdc794..000000000000 --- a/.agent/skills/beam-dofn-modernizer/SKILL.md +++ /dev/null @@ -1,207 +0,0 @@ ---- -# Licensed to the Apache Software Foundation (ASF) under one -# or more contributor license agreements. See the NOTICE file -# distributed with this work for additional information -# regarding copyright ownership. The ASF licenses this file -# to you under the Apache License, Version 2.0 (the -# "License"); you may not use this file except in compliance -# with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, -# software distributed under the License is distributed on an -# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -# KIND, either express or implied. See the License for the -# specific language governing permissions and limitations -# under the License. ---- -name: beam-dofn-modernizer -description: Rewrite Apache Beam DoFn methods (@ProcessElement, @OnTimer, @OnWindowExpiration) to remove legacy ProcessContext or OnTimerContext usage. Use this skill when you encounter DoFn methods that use context.element(), context.output(), etc., and need to modernize them using parameter injection (@Element, @Timestamp, @Pane, OutputReceiver, MultiOutputReceiver). ---- - -# Modernizing Apache Beam DoFns - -Apache Beam has moved towards parameter injection in `DoFn` methods to improve readability and allow for more efficient execution. This skill helps you migrate legacy `ProcessContext` and `OnTimerContext` usage to modern annotated parameters. - -## Core Mappings - -When rewriting a `@ProcessElement` or `@OnTimer` method, replace the context argument with the corresponding parameters based on the usage: - -| Legacy Context Usage (e.g. `ProcessContext c`) | Modern Parameter Replacement | -| :--- | :--- | -| `c.element()` | `@Element T element` | -| `c.timestamp()` | `@Timestamp Instant timestamp` | -| `c.pane()` | `PaneInfo pane` | -| `c.window()` | `BoundedWindow window` | -| `c.sideInput(PCollectionView view)` | `@SideInput("viewName") T value` | -| `c.getPipelineOptions()` | `PipelineOptions options` | -| `c.output(value)` | `OutputReceiver receiver` then `receiver.output(value)` | -| `c.output(tag, value)` | `MultiOutputReceiver receiver` then `receiver.get(tag).output(value)` | -| `c.outputWithTimestamp(value, ts)` | `OutputReceiver receiver` then `receiver.outputWithTimestamp(value, ts)` | - -## Method Signature Changes - -### @ProcessElement - -**Legacy:** -```java -@ProcessElement -public void processElement(ProcessContext c) { - T element = c.element(); - c.output(transform(element)); -} -``` - -**Modern:** -```java -@ProcessElement -public void processElement( - @Element T element, - @Timestamp Instant timestamp, - OutputReceiver receiver) { - receiver.output(transform(element)); -} -``` - -### @OnTimer - -**Legacy:** -```java -@OnTimer("timerId") -public void onTimer(OnTimerContext c) { - c.output(someValue); -} -``` - -**Modern:** -```java -@OnTimer("timerId") -public void onTimer( - @Timestamp Instant timestamp, - BoundedWindow window, - OutputReceiver receiver) { - receiver.output(someValue); -} -``` - -## Best Practices - -1. **Specific OutputReceiver**: If the method only outputs to the main output, use `OutputReceiver`. If it outputs to multiple tags, use `MultiOutputReceiver`. -2. **Element Type**: Ensure the `@Element` parameter type matches the input type of the `DoFn`. -3. **Imports**: Don't forget to add imports for: - * `org.apache.beam.sdk.transforms.DoFn.Element` - * `org.apache.beam.sdk.transforms.DoFn.Timestamp` - * `org.apache.beam.sdk.transforms.DoFn.OutputReceiver` - * `org.apache.beam.sdk.transforms.DoFn.MultiOutputReceiver` (if needed) - * `org.apache.beam.sdk.values.PCollectionView` (if using `@SideInput`) - * `org.apache.beam.sdk.transforms.DoFn.SideInput` - * `org.apache.beam.sdk.transforms.windowing.PaneInfo` -4. **Side Inputs**: When using `@SideInput`, make sure to use the correct name that matches the one passed to `ParDo.withSideInput("name", view)`. -5. **Parameter Naming and Redundant Variables**: Use descriptive names for the `@Element` parameter (e.g., `record`, `line`, `row`) instead of a generic `element` if it improves readability. Do not create a redundant local variable to copy the element (e.g., `MyType elm = element;`), use the parameter directly. - -## Example Conversion - -### Before: -```java -@ProcessElement -public void processElement(ProcessContext c) { - KV element = c.element(); - Instant ts = c.timestamp(); - if (element.getValue() > threshold) { - c.output(element.getKey()); - c.output(specialTag, element.getValue()); - } -} -``` - -### After: -```java -@ProcessElement -public void processElement( - @Element KV element, - @Timestamp Instant timestamp, - MultiOutputReceiver receiver) { - if (element.getValue() > threshold) { - receiver.get(mainTag).output(element.getKey()); - receiver.get(specialTag).output(element.getValue()); - } -} -``` -> [!NOTE] -> If you only have one output, use `OutputReceiver receiver` and `receiver.output(element.getKey())`. - -## Side Input Conversion - -Modernizing side inputs involves removing the `PCollectionView` from the `DoFn` constructor and using `@SideInput` parameter injection instead. - -### Before (Legacy): - -**PTransform/Pipeline side:** -```java -PCollectionView myView = ...; -input.apply(ParDo.of(new MyFn(myView)).withSideInputs(myView)); -``` - -**DoFn side:** -```java -class MyFn extends DoFn { - private final PCollectionView view; - MyFn(PCollectionView view) { this.view = view; } - - @ProcessElement - public void processElement(ProcessContext c) { - String value = c.sideInput(view); - // ... - } -} -``` - -### Nullable Side Inputs - -If a side input is optional and a `DoFn` has conditional logic based on whether the side input is present, it is best to split the `DoFn` into two separate classes: one that requires the side input and one that does not. This avoids creating complex, conditional `DoFn`s and ensures type safety. - -**PTransform/Pipeline side:** -```java -PCollectionView myView = ...; -input.apply(ParDo.of(new MyFn(myView)).withSideInputs(myView)); -//or -input.apply(ParDo.of(new MyFn(null))); // to introduce null -``` -**DoFn side:** -```java -class MyFn extends DoFn { - private final PCollectionView view; - MyFn(PCollectionView view) { this.view = view; } - - @ProcessElement - public void processElement(ProcessContext c) { - String value = null; - if (this.view != null) { // can do conditional side input - value = c.sideInput(this.view); - } - - // ... - } -} -``` - -### After (Modern): - -**PTransform/Pipeline side:** -```java -PCollectionView myView = ...; -input.apply(ParDo.of(new MyFn()).withSideInput("sideInputName", myView)); -``` - -**DoFn side:** -```java -class MyFn extends DoFn { - @ProcessElement - public void processElement( - @Element T element, - @SideInput("sideInputName") String value) { - // value is injected directly - } -} -``` diff --git a/examples/java/iceberg/src/main/java/org/apache/beam/examples/iceberg/IcebergBatchWriteExample.java b/examples/java/iceberg/src/main/java/org/apache/beam/examples/iceberg/IcebergBatchWriteExample.java index 167d3017c3d9..2a5f85e524ed 100644 --- a/examples/java/iceberg/src/main/java/org/apache/beam/examples/iceberg/IcebergBatchWriteExample.java +++ b/examples/java/iceberg/src/main/java/org/apache/beam/examples/iceberg/IcebergBatchWriteExample.java @@ -28,8 +28,6 @@ import org.apache.beam.sdk.options.Validation; import org.apache.beam.sdk.schemas.Schema; import org.apache.beam.sdk.transforms.DoFn; -import org.apache.beam.sdk.transforms.DoFn.Element; -import org.apache.beam.sdk.transforms.DoFn.OutputReceiver; import org.apache.beam.sdk.transforms.MapElements; import org.apache.beam.sdk.transforms.PTransform; import org.apache.beam.sdk.transforms.ParDo; @@ -76,8 +74,9 @@ private static Row flattenAnalyticsRow(Row row) { static class ExtractBrowserTransactionsFn extends DoFn> { @ProcessElement - public void processElement(@Element Row row, OutputReceiver> receiver) { - receiver.output( + public void processElement(ProcessContext c) { + Row row = c.element(); + c.output( KV.of( Preconditions.checkStateNotNull(row.getString("browser")), Preconditions.checkStateNotNull(row.getInt64("transactions")))); @@ -86,13 +85,13 @@ public void processElement(@Element Row row, OutputReceiver> re static class FormatCountsFn extends DoFn, Row> { @ProcessElement - public void processElement(@Element KV element, OutputReceiver receiver) { + public void processElement(ProcessContext c) { Row row = Row.withSchema(AGGREGATED_SCHEMA) - .withFieldValue("browser", element.getKey()) - .withFieldValue("transaction_count", element.getValue()) + .withFieldValue("browser", c.element().getKey()) + .withFieldValue("transaction_count", c.element().getValue()) .build(); - receiver.output(row); + c.output(row); } } diff --git a/examples/java/sql/src/main/java/org/apache/beam/examples/SchemaTransformExample.java b/examples/java/sql/src/main/java/org/apache/beam/examples/SchemaTransformExample.java index 84155683e9cc..5b8e4a34cf2c 100644 --- a/examples/java/sql/src/main/java/org/apache/beam/examples/SchemaTransformExample.java +++ b/examples/java/sql/src/main/java/org/apache/beam/examples/SchemaTransformExample.java @@ -42,8 +42,6 @@ import org.apache.beam.sdk.schemas.transforms.Select; import org.apache.beam.sdk.transforms.Create; import org.apache.beam.sdk.transforms.DoFn; -import org.apache.beam.sdk.transforms.DoFn.Element; -import org.apache.beam.sdk.transforms.DoFn.OutputReceiver; import org.apache.beam.sdk.transforms.Max; import org.apache.beam.sdk.transforms.Min; import org.apache.beam.sdk.transforms.ParDo; @@ -103,9 +101,9 @@ public LogOutput(String prefix) { } @ProcessElement - public void processElement(@Element T element, OutputReceiver receiver) throws Exception { - LOG.info("{}{}", prefix, element); - receiver.output(element); + public void processElement(ProcessContext c) throws Exception { + LOG.info("{}{}", prefix, c.element()); + c.output(c.element()); } } } diff --git a/examples/java/sql/src/main/java/org/apache/beam/examples/SqlTransformExample.java b/examples/java/sql/src/main/java/org/apache/beam/examples/SqlTransformExample.java index 9ca2dda313c6..9c2302c3de2f 100644 --- a/examples/java/sql/src/main/java/org/apache/beam/examples/SqlTransformExample.java +++ b/examples/java/sql/src/main/java/org/apache/beam/examples/SqlTransformExample.java @@ -41,8 +41,6 @@ import org.apache.beam.sdk.schemas.Schema; import org.apache.beam.sdk.transforms.Create; import org.apache.beam.sdk.transforms.DoFn; -import org.apache.beam.sdk.transforms.DoFn.Element; -import org.apache.beam.sdk.transforms.DoFn.OutputReceiver; import org.apache.beam.sdk.transforms.ParDo; import org.apache.beam.sdk.values.PCollection; import org.apache.beam.sdk.values.Row; @@ -97,9 +95,9 @@ public LogOutput(String prefix) { } @ProcessElement - public void processElement(@Element T element, OutputReceiver receiver) throws Exception { - LOG.info("{}{}", prefix, element); - receiver.output(element); + public void processElement(ProcessContext c) throws Exception { + LOG.info("{}{}", prefix, c.element()); + c.output(c.element()); } } } diff --git a/examples/java/src/main/java/org/apache/beam/examples/ApproximateQuantilesExample.java b/examples/java/src/main/java/org/apache/beam/examples/ApproximateQuantilesExample.java index 0f968e88d392..9e2a96b1eca9 100644 --- a/examples/java/src/main/java/org/apache/beam/examples/ApproximateQuantilesExample.java +++ b/examples/java/src/main/java/org/apache/beam/examples/ApproximateQuantilesExample.java @@ -24,8 +24,6 @@ import org.apache.beam.sdk.transforms.ApproximateQuantiles; import org.apache.beam.sdk.transforms.Create; import org.apache.beam.sdk.transforms.DoFn; -import org.apache.beam.sdk.transforms.DoFn.Element; -import org.apache.beam.sdk.transforms.DoFn.OutputReceiver; import org.apache.beam.sdk.transforms.ParDo; import org.apache.beam.sdk.values.PCollection; import org.slf4j.Logger; @@ -72,9 +70,9 @@ public LogOutput(String prefix) { } @ProcessElement - public void processElement(@Element T element, OutputReceiver receiver) throws Exception { - LOG.info("{}{}", prefix, element); - receiver.output(element); + public void processElement(ProcessContext c) throws Exception { + LOG.info("{}{}", prefix, c.element()); + c.output(c.element()); } } } diff --git a/examples/java/src/main/java/org/apache/beam/examples/CoCombineTransformExample.java b/examples/java/src/main/java/org/apache/beam/examples/CoCombineTransformExample.java index 1aafd50a1621..0ee7d7bcac89 100644 --- a/examples/java/src/main/java/org/apache/beam/examples/CoCombineTransformExample.java +++ b/examples/java/src/main/java/org/apache/beam/examples/CoCombineTransformExample.java @@ -46,8 +46,6 @@ import org.apache.beam.sdk.transforms.CombineFns; import org.apache.beam.sdk.transforms.Create; import org.apache.beam.sdk.transforms.DoFn; -import org.apache.beam.sdk.transforms.DoFn.Element; -import org.apache.beam.sdk.transforms.DoFn.OutputReceiver; import org.apache.beam.sdk.transforms.Max; import org.apache.beam.sdk.transforms.Min; import org.apache.beam.sdk.transforms.ParDo; @@ -187,16 +185,13 @@ public Long apply(Long input) { new DoFn< KV, KV>>>() { @ProcessElement - public void processElement( - @Element KV element, - OutputReceiver>>> receiver) - throws Exception { - CombineFns.CoCombineResult e = element.getValue(); + public void processElement(ProcessContext c) throws Exception { + CombineFns.CoCombineResult e = c.element().getValue(); ArrayList> o = new ArrayList>(); o.add(KV.of(minTag.getId(), e.get(minTag))); o.add(KV.of(maxTag.getId(), e.get(maxTag))); o.add(KV.of(sumTag.getId(), e.get(sumTag))); - receiver.output(KV.of(element.getKey(), o)); + c.output(KV.of(c.element().getKey(), o)); } })); @@ -215,9 +210,9 @@ public LogOutput(String prefix) { } @ProcessElement - public void processElement(@Element T element, OutputReceiver receiver) throws Exception { - LOG.info("{}{}", prefix, element); - receiver.output(element); + public void processElement(ProcessContext c) throws Exception { + LOG.info("{}{}", prefix, c.element()); + c.output(c.element()); } } } diff --git a/examples/java/src/main/java/org/apache/beam/examples/CoGroupByKeyExample.java b/examples/java/src/main/java/org/apache/beam/examples/CoGroupByKeyExample.java index 46905ad71daf..c77708b5de20 100644 --- a/examples/java/src/main/java/org/apache/beam/examples/CoGroupByKeyExample.java +++ b/examples/java/src/main/java/org/apache/beam/examples/CoGroupByKeyExample.java @@ -22,8 +22,6 @@ import org.apache.beam.sdk.options.PipelineOptionsFactory; import org.apache.beam.sdk.transforms.Create; import org.apache.beam.sdk.transforms.DoFn; -import org.apache.beam.sdk.transforms.DoFn.Element; -import org.apache.beam.sdk.transforms.DoFn.OutputReceiver; import org.apache.beam.sdk.transforms.ParDo; import org.apache.beam.sdk.transforms.join.CoGbkResult; import org.apache.beam.sdk.transforms.join.CoGroupByKey; @@ -86,9 +84,9 @@ public LogOutput(String prefix) { } @ProcessElement - public void processElement(@Element T element, OutputReceiver receiver) throws Exception { - LOG.info("{}{}", prefix, element); - receiver.output(element); + public void processElement(ProcessContext c) throws Exception { + LOG.info("{}{}", prefix, c.element()); + c.output(c.element()); } } } diff --git a/examples/java/src/main/java/org/apache/beam/examples/CombineExample.java b/examples/java/src/main/java/org/apache/beam/examples/CombineExample.java index 5f6901ea5ec4..24bed27c2360 100644 --- a/examples/java/src/main/java/org/apache/beam/examples/CombineExample.java +++ b/examples/java/src/main/java/org/apache/beam/examples/CombineExample.java @@ -23,8 +23,6 @@ import org.apache.beam.sdk.transforms.Combine; import org.apache.beam.sdk.transforms.Create; import org.apache.beam.sdk.transforms.DoFn; -import org.apache.beam.sdk.transforms.DoFn.Element; -import org.apache.beam.sdk.transforms.DoFn.OutputReceiver; import org.apache.beam.sdk.transforms.ParDo; import org.apache.beam.sdk.transforms.Sum; import org.apache.beam.sdk.values.PCollection; @@ -70,9 +68,9 @@ public LogOutput(String prefix) { } @ProcessElement - public void processElement(@Element T element, OutputReceiver receiver) throws Exception { - LOG.info("{}{}", prefix, element); - receiver.output(element); + public void processElement(ProcessContext c) throws Exception { + LOG.info("{}{}", prefix, c.element()); + c.output(c.element()); } } } diff --git a/examples/java/src/main/java/org/apache/beam/examples/CountExample.java b/examples/java/src/main/java/org/apache/beam/examples/CountExample.java index f95ec3fe4759..cb0bd0ecf943 100644 --- a/examples/java/src/main/java/org/apache/beam/examples/CountExample.java +++ b/examples/java/src/main/java/org/apache/beam/examples/CountExample.java @@ -23,8 +23,6 @@ import org.apache.beam.sdk.transforms.Count; import org.apache.beam.sdk.transforms.Create; import org.apache.beam.sdk.transforms.DoFn; -import org.apache.beam.sdk.transforms.DoFn.Element; -import org.apache.beam.sdk.transforms.DoFn.OutputReceiver; import org.apache.beam.sdk.transforms.ParDo; import org.apache.beam.sdk.values.PCollection; import org.slf4j.Logger; @@ -65,9 +63,9 @@ public LogOutput(String prefix) { } @ProcessElement - public void processElement(@Element T element, OutputReceiver receiver) throws Exception { - LOG.info("{}{}", prefix, element); - receiver.output(element); + public void processElement(ProcessContext c) throws Exception { + LOG.info("{}{}", prefix, c.element()); + c.output(c.element()); } } } diff --git a/examples/java/src/main/java/org/apache/beam/examples/CountPerKeyExample.java b/examples/java/src/main/java/org/apache/beam/examples/CountPerKeyExample.java index f02c262ccbab..9bf9bf1ef00f 100644 --- a/examples/java/src/main/java/org/apache/beam/examples/CountPerKeyExample.java +++ b/examples/java/src/main/java/org/apache/beam/examples/CountPerKeyExample.java @@ -23,8 +23,6 @@ import org.apache.beam.sdk.transforms.Count; import org.apache.beam.sdk.transforms.Create; import org.apache.beam.sdk.transforms.DoFn; -import org.apache.beam.sdk.transforms.DoFn.Element; -import org.apache.beam.sdk.transforms.DoFn.OutputReceiver; import org.apache.beam.sdk.transforms.ParDo; import org.apache.beam.sdk.values.KV; import org.apache.beam.sdk.values.PCollection; @@ -69,9 +67,9 @@ public LogOutput(String prefix) { } @ProcessElement - public void processElement(@Element T element, OutputReceiver receiver) throws Exception { - LOG.info("{}{}", prefix, element); - receiver.output(element); + public void processElement(ProcessContext c) throws Exception { + LOG.info("{}{}", prefix, c.element()); + c.output(c.element()); } } } diff --git a/examples/java/src/main/java/org/apache/beam/examples/CreateExample.java b/examples/java/src/main/java/org/apache/beam/examples/CreateExample.java index b5eb5ba3e9de..5943ffa489d4 100644 --- a/examples/java/src/main/java/org/apache/beam/examples/CreateExample.java +++ b/examples/java/src/main/java/org/apache/beam/examples/CreateExample.java @@ -27,8 +27,6 @@ import org.apache.beam.sdk.options.PipelineOptionsFactory; import org.apache.beam.sdk.transforms.Create; import org.apache.beam.sdk.transforms.DoFn; -import org.apache.beam.sdk.transforms.DoFn.Element; -import org.apache.beam.sdk.transforms.DoFn.OutputReceiver; import org.apache.beam.sdk.transforms.ParDo; import org.apache.beam.sdk.values.KV; import org.apache.beam.sdk.values.PCollection; @@ -81,9 +79,9 @@ public LogOutput(String prefix) { } @ProcessElement - public void processElement(@Element T element, OutputReceiver receiver) throws Exception { - LOG.info("{}{}", prefix, element); - receiver.output(element); + public void processElement(ProcessContext c) throws Exception { + LOG.info("{}{}", prefix, c.element()); + c.output(c.element()); } } } diff --git a/examples/java/src/main/java/org/apache/beam/examples/DebuggingWordCount.java b/examples/java/src/main/java/org/apache/beam/examples/DebuggingWordCount.java index 07c5be76d753..7c54e238da33 100644 --- a/examples/java/src/main/java/org/apache/beam/examples/DebuggingWordCount.java +++ b/examples/java/src/main/java/org/apache/beam/examples/DebuggingWordCount.java @@ -46,8 +46,6 @@ import org.apache.beam.sdk.options.PipelineOptionsFactory; import org.apache.beam.sdk.testing.PAssert; import org.apache.beam.sdk.transforms.DoFn; -import org.apache.beam.sdk.transforms.DoFn.Element; -import org.apache.beam.sdk.transforms.DoFn.OutputReceiver; import org.apache.beam.sdk.transforms.ParDo; import org.apache.beam.sdk.values.KV; import org.apache.beam.sdk.values.PCollection; @@ -117,19 +115,18 @@ public FilterTextFn(String pattern) { private final Counter unmatchedWords = Metrics.counter(FilterTextFn.class, "unmatchedWords"); @ProcessElement - public void processElement( - @Element KV element, OutputReceiver> receiver) { - if (filter.matcher(element.getKey()).matches()) { + public void processElement(ProcessContext c) { + if (filter.matcher(c.element().getKey()).matches()) { // Log at the "DEBUG" level each element that we match. When executing this pipeline // these log lines will appear only if the log level is set to "DEBUG" or lower. - LOG.debug("Matched: {}", element.getKey()); + LOG.debug("Matched: {}", c.element().getKey()); matchedWords.inc(); - receiver.output(element); + c.output(c.element()); } else { // Log at the "TRACE" level each element that is not matched. Different log levels // can be used to control the verbosity of logging providing an effective mechanism // to filter less important information. - LOG.trace("Did not match: {}", element.getKey()); + LOG.trace("Did not match: {}", c.element().getKey()); unmatchedWords.inc(); } } diff --git a/examples/java/src/main/java/org/apache/beam/examples/DistinctExample.java b/examples/java/src/main/java/org/apache/beam/examples/DistinctExample.java index 28fd4af87066..d3ff9d663f44 100644 --- a/examples/java/src/main/java/org/apache/beam/examples/DistinctExample.java +++ b/examples/java/src/main/java/org/apache/beam/examples/DistinctExample.java @@ -23,8 +23,6 @@ import org.apache.beam.sdk.transforms.Create; import org.apache.beam.sdk.transforms.Distinct; import org.apache.beam.sdk.transforms.DoFn; -import org.apache.beam.sdk.transforms.DoFn.Element; -import org.apache.beam.sdk.transforms.DoFn.OutputReceiver; import org.apache.beam.sdk.transforms.ParDo; import org.apache.beam.sdk.values.PCollection; import org.slf4j.Logger; @@ -70,9 +68,9 @@ public LogOutput(String prefix) { } @ProcessElement - public void processElement(@Element T element, OutputReceiver receiver) throws Exception { - LOG.info("{}{}", prefix, element); - receiver.output(element); + public void processElement(ProcessContext c) throws Exception { + LOG.info("{}{}", prefix, c.element()); + c.output(c.element()); } } } diff --git a/examples/java/src/main/java/org/apache/beam/examples/FlatMapElementsExample.java b/examples/java/src/main/java/org/apache/beam/examples/FlatMapElementsExample.java index 4e9493ac1cf2..71d05ac7ade3 100644 --- a/examples/java/src/main/java/org/apache/beam/examples/FlatMapElementsExample.java +++ b/examples/java/src/main/java/org/apache/beam/examples/FlatMapElementsExample.java @@ -24,8 +24,6 @@ import org.apache.beam.sdk.options.PipelineOptionsFactory; import org.apache.beam.sdk.transforms.Create; import org.apache.beam.sdk.transforms.DoFn; -import org.apache.beam.sdk.transforms.DoFn.Element; -import org.apache.beam.sdk.transforms.DoFn.OutputReceiver; import org.apache.beam.sdk.transforms.FlatMapElements; import org.apache.beam.sdk.transforms.InferableFunction; import org.apache.beam.sdk.transforms.ParDo; @@ -80,9 +78,9 @@ public LogOutput(String prefix) { } @ProcessElement - public void processElement(@Element T element, OutputReceiver receiver) throws Exception { - LOG.info("{}{}", prefix, element); - receiver.output(element); + public void processElement(ProcessContext c) throws Exception { + LOG.info("{}{}", prefix, c.element()); + c.output(c.element()); } } } diff --git a/examples/java/src/main/java/org/apache/beam/examples/GroupIntoBatchesExample.java b/examples/java/src/main/java/org/apache/beam/examples/GroupIntoBatchesExample.java index 212079579ee0..78e898b9c173 100644 --- a/examples/java/src/main/java/org/apache/beam/examples/GroupIntoBatchesExample.java +++ b/examples/java/src/main/java/org/apache/beam/examples/GroupIntoBatchesExample.java @@ -22,8 +22,6 @@ import org.apache.beam.sdk.options.PipelineOptionsFactory; import org.apache.beam.sdk.transforms.Create; import org.apache.beam.sdk.transforms.DoFn; -import org.apache.beam.sdk.transforms.DoFn.Element; -import org.apache.beam.sdk.transforms.DoFn.OutputReceiver; import org.apache.beam.sdk.transforms.GroupIntoBatches; import org.apache.beam.sdk.transforms.ParDo; import org.apache.beam.sdk.values.KV; @@ -76,9 +74,9 @@ public LogOutput(String prefix) { } @ProcessElement - public void processElement(@Element T element, OutputReceiver receiver) throws Exception { - LOG.info("{}{}", prefix, element); - receiver.output(element); + public void processElement(ProcessContext c) throws Exception { + LOG.info("{}{}", prefix, c.element()); + c.output(c.element()); } } } diff --git a/examples/java/src/main/java/org/apache/beam/examples/KafkaPassengerCountJson.java b/examples/java/src/main/java/org/apache/beam/examples/KafkaPassengerCountJson.java index 5b26455e3c5f..20f70232ae94 100644 --- a/examples/java/src/main/java/org/apache/beam/examples/KafkaPassengerCountJson.java +++ b/examples/java/src/main/java/org/apache/beam/examples/KafkaPassengerCountJson.java @@ -55,8 +55,6 @@ import org.apache.beam.sdk.options.PipelineOptionsFactory; import org.apache.beam.sdk.transforms.Combine; import org.apache.beam.sdk.transforms.DoFn; -import org.apache.beam.sdk.transforms.DoFn.Element; -import org.apache.beam.sdk.transforms.DoFn.OutputReceiver; import org.apache.beam.sdk.transforms.ParDo; import org.apache.beam.sdk.transforms.Sum; import org.apache.beam.sdk.transforms.Values; @@ -111,13 +109,10 @@ public static void main(String[] args) { ParDo.of( new DoFn>() { @ProcessElement - public void processElement( - @Element String element, OutputReceiver> receiver) - throws JsonProcessingException { + public void processElement(ProcessContext c) throws JsonProcessingException { final VendorToPassengerDTO result = - om.readValue(element, new TypeReference() {}); - receiver.output( - KV.of(result.getVendorIdField(), result.getPassengerCountField())); + om.readValue(c.element(), new TypeReference() {}); + c.output(KV.of(result.getVendorIdField(), result.getPassengerCountField())); } })) .apply( @@ -129,11 +124,11 @@ public void processElement( new DoFn, KV>() { @ProcessElement public void processElement( - OutputReceiver> out, - @Element KV element) { + ProcessContext c, OutputReceiver> out) { System.out.printf( - "Vendor: %s, Passengers: %s%n", element.getKey(), element.getValue()); - out.output(element); + "Vendor: %s, Passengers: %s%n", + c.element().getKey(), c.element().getValue()); + out.output(c.element()); } })); p.run().waitUntilFinish(); diff --git a/examples/java/src/main/java/org/apache/beam/examples/KafkaStreaming.java b/examples/java/src/main/java/org/apache/beam/examples/KafkaStreaming.java index 327d321d8ad3..f0b09226865a 100644 --- a/examples/java/src/main/java/org/apache/beam/examples/KafkaStreaming.java +++ b/examples/java/src/main/java/org/apache/beam/examples/KafkaStreaming.java @@ -49,8 +49,6 @@ import org.apache.beam.sdk.options.PipelineOptionsFactory; import org.apache.beam.sdk.transforms.Combine; import org.apache.beam.sdk.transforms.DoFn; -import org.apache.beam.sdk.transforms.DoFn.Element; -import org.apache.beam.sdk.transforms.DoFn.OutputReceiver; import org.apache.beam.sdk.transforms.PTransform; import org.apache.beam.sdk.transforms.ParDo; import org.apache.beam.sdk.transforms.Sum; @@ -177,9 +175,8 @@ static class RandomUserScoreGeneratorFn extends DoFn private static final Random RANDOM = new Random(); @ProcessElement - public void processElement( - @Element Object element, OutputReceiver> receiver) { - receiver.output(generate()); + public void processElement(ProcessContext c) { + c.output(generate()); } public KV generate() { @@ -293,17 +290,17 @@ public Map extractOutput(Map accumulator) { static class LogResults extends DoFn, Map> { @ProcessElement - public void processElement( - PaneInfo pane, - IntervalWindow w, - @Element Map scores, - OutputReceiver> receiver) - throws Exception { + public void processElement(ProcessContext c, IntervalWindow w) throws Exception { + Map map = c.element(); + if (map == null) { + c.output(c.element()); + return; + } String startTime = w.start().toString(dateTimeFormatter); String endTime = w.end().toString(dateTimeFormatter); - PaneInfo.Timing timing = pane.getTiming(); + PaneInfo.Timing timing = c.pane().getTiming(); switch (timing) { case EARLY: @@ -319,7 +316,7 @@ public void processElement( throw new RuntimeException("Unknown timing value"); } - for (Map.Entry entry : scores.entrySet()) { + for (Map.Entry entry : map.entrySet()) { System.out.printf("%10s: %-10s%n", entry.getKey(), entry.getValue()); } @@ -329,7 +326,7 @@ public void processElement( System.out.println(); } - receiver.output(scores); + c.output(c.element()); } } @@ -343,9 +340,9 @@ public PCollection expand(PCollection input) { static class LogErrorFn extends DoFn { @ProcessElement - public void processElement(@Element BadRecord badRecord, OutputReceiver receiver) { - System.out.println(badRecord); - receiver.output(badRecord); + public void processElement(@Element BadRecord record, OutputReceiver receiver) { + System.out.println(record); + receiver.output(record); } } } diff --git a/examples/java/src/main/java/org/apache/beam/examples/KafkaWordCountAvro.java b/examples/java/src/main/java/org/apache/beam/examples/KafkaWordCountAvro.java index fad5668945e7..9e2da248017d 100644 --- a/examples/java/src/main/java/org/apache/beam/examples/KafkaWordCountAvro.java +++ b/examples/java/src/main/java/org/apache/beam/examples/KafkaWordCountAvro.java @@ -51,8 +51,6 @@ import org.apache.beam.sdk.options.PipelineOptionsFactory; import org.apache.beam.sdk.transforms.Count; import org.apache.beam.sdk.transforms.DoFn; -import org.apache.beam.sdk.transforms.DoFn.Element; -import org.apache.beam.sdk.transforms.DoFn.OutputReceiver; import org.apache.beam.sdk.transforms.MapElements; import org.apache.beam.sdk.transforms.ParDo; import org.apache.beam.sdk.transforms.SimpleFunction; @@ -105,11 +103,10 @@ public static void main(String[] args) { ParDo.of( new DoFn() { @ProcessElement - public void processElement( - @Element String element, OutputReceiver receiver) { - for (String word : element.split(TOKENIZER_PATTERN, 0)) { + public void processElement(ProcessContext c) { + for (String word : c.element().split(TOKENIZER_PATTERN, 0)) { if (!word.isEmpty()) { - receiver.output(word); + c.output(word); } } } diff --git a/examples/java/src/main/java/org/apache/beam/examples/KafkaWordCountJson.java b/examples/java/src/main/java/org/apache/beam/examples/KafkaWordCountJson.java index 036cc6702123..355614b43869 100644 --- a/examples/java/src/main/java/org/apache/beam/examples/KafkaWordCountJson.java +++ b/examples/java/src/main/java/org/apache/beam/examples/KafkaWordCountJson.java @@ -52,8 +52,6 @@ import org.apache.beam.sdk.options.PipelineOptionsFactory; import org.apache.beam.sdk.transforms.Count; import org.apache.beam.sdk.transforms.DoFn; -import org.apache.beam.sdk.transforms.DoFn.Element; -import org.apache.beam.sdk.transforms.DoFn.OutputReceiver; import org.apache.beam.sdk.transforms.MapElements; import org.apache.beam.sdk.transforms.ParDo; import org.apache.beam.sdk.transforms.SimpleFunction; @@ -106,11 +104,10 @@ public static void main(String[] args) { ParDo.of( new DoFn() { @ProcessElement - public void processElement( - @Element String element, OutputReceiver receiver) { - for (String word : element.split(TOKENIZER_PATTERN, 0)) { + public void processElement(ProcessContext c) { + for (String word : c.element().split(TOKENIZER_PATTERN, 0)) { if (!word.isEmpty()) { - receiver.output(word); + c.output(word); } } } diff --git a/examples/java/src/main/java/org/apache/beam/examples/KeysExample.java b/examples/java/src/main/java/org/apache/beam/examples/KeysExample.java index c9dc40170a57..155834bc0a43 100644 --- a/examples/java/src/main/java/org/apache/beam/examples/KeysExample.java +++ b/examples/java/src/main/java/org/apache/beam/examples/KeysExample.java @@ -22,8 +22,6 @@ import org.apache.beam.sdk.options.PipelineOptionsFactory; import org.apache.beam.sdk.transforms.Create; import org.apache.beam.sdk.transforms.DoFn; -import org.apache.beam.sdk.transforms.DoFn.Element; -import org.apache.beam.sdk.transforms.DoFn.OutputReceiver; import org.apache.beam.sdk.transforms.Keys; import org.apache.beam.sdk.transforms.ParDo; import org.apache.beam.sdk.values.KV; @@ -72,9 +70,9 @@ public LogOutput(String prefix) { } @ProcessElement - public void processElement(@Element T element, OutputReceiver receiver) throws Exception { - LOG.info("{}{}", prefix, element); - receiver.output(element); + public void processElement(ProcessContext c) throws Exception { + LOG.info("{}{}", prefix, c.element()); + c.output(c.element()); } } } diff --git a/examples/java/src/main/java/org/apache/beam/examples/KvSwapExample.java b/examples/java/src/main/java/org/apache/beam/examples/KvSwapExample.java index 192dc893795b..090779de7c36 100644 --- a/examples/java/src/main/java/org/apache/beam/examples/KvSwapExample.java +++ b/examples/java/src/main/java/org/apache/beam/examples/KvSwapExample.java @@ -22,8 +22,6 @@ import org.apache.beam.sdk.options.PipelineOptionsFactory; import org.apache.beam.sdk.transforms.Create; import org.apache.beam.sdk.transforms.DoFn; -import org.apache.beam.sdk.transforms.DoFn.Element; -import org.apache.beam.sdk.transforms.DoFn.OutputReceiver; import org.apache.beam.sdk.transforms.KvSwap; import org.apache.beam.sdk.transforms.ParDo; import org.apache.beam.sdk.values.KV; @@ -71,9 +69,9 @@ public LogOutput(String prefix) { } @ProcessElement - public void processElement(@Element T element, OutputReceiver receiver) throws Exception { - LOG.info("{}{}", prefix, element); - receiver.output(element); + public void processElement(ProcessContext c) throws Exception { + LOG.info("{}{}", prefix, c.element()); + c.output(c.element()); } } } diff --git a/examples/java/src/main/java/org/apache/beam/examples/LatestExample.java b/examples/java/src/main/java/org/apache/beam/examples/LatestExample.java index ef4c50d24c73..5e9662a2f1ab 100644 --- a/examples/java/src/main/java/org/apache/beam/examples/LatestExample.java +++ b/examples/java/src/main/java/org/apache/beam/examples/LatestExample.java @@ -22,8 +22,6 @@ import org.apache.beam.sdk.options.PipelineOptionsFactory; import org.apache.beam.sdk.transforms.Create; import org.apache.beam.sdk.transforms.DoFn; -import org.apache.beam.sdk.transforms.DoFn.Element; -import org.apache.beam.sdk.transforms.DoFn.OutputReceiver; import org.apache.beam.sdk.transforms.Latest; import org.apache.beam.sdk.transforms.ParDo; import org.apache.beam.sdk.transforms.WithTimestamps; @@ -86,9 +84,9 @@ public LogOutput(String prefix) { } @ProcessElement - public void processElement(@Element T element, OutputReceiver receiver) throws Exception { - LOG.info("{}{}", prefix, element); - receiver.output(element); + public void processElement(ProcessContext c) throws Exception { + LOG.info("{}{}", prefix, c.element()); + c.output(c.element()); } } } diff --git a/examples/java/src/main/java/org/apache/beam/examples/MapElementsExample.java b/examples/java/src/main/java/org/apache/beam/examples/MapElementsExample.java index 60908cf83a48..93d885750ae8 100644 --- a/examples/java/src/main/java/org/apache/beam/examples/MapElementsExample.java +++ b/examples/java/src/main/java/org/apache/beam/examples/MapElementsExample.java @@ -22,8 +22,6 @@ import org.apache.beam.sdk.options.PipelineOptionsFactory; import org.apache.beam.sdk.transforms.Create; import org.apache.beam.sdk.transforms.DoFn; -import org.apache.beam.sdk.transforms.DoFn.Element; -import org.apache.beam.sdk.transforms.DoFn.OutputReceiver; import org.apache.beam.sdk.transforms.MapElements; import org.apache.beam.sdk.transforms.ParDo; import org.apache.beam.sdk.transforms.SimpleFunction; @@ -78,9 +76,9 @@ public LogOutput(String prefix) { } @ProcessElement - public void processElement(@Element T element, OutputReceiver receiver) throws Exception { - LOG.info("{}{}", prefix, element); - receiver.output(element); + public void processElement(ProcessContext c) throws Exception { + LOG.info("{}{}", prefix, c.element()); + c.output(c.element()); } } } diff --git a/examples/java/src/main/java/org/apache/beam/examples/MaxExample.java b/examples/java/src/main/java/org/apache/beam/examples/MaxExample.java index 845b107a2214..9173d11754a3 100644 --- a/examples/java/src/main/java/org/apache/beam/examples/MaxExample.java +++ b/examples/java/src/main/java/org/apache/beam/examples/MaxExample.java @@ -22,8 +22,6 @@ import org.apache.beam.sdk.options.PipelineOptionsFactory; import org.apache.beam.sdk.transforms.Create; import org.apache.beam.sdk.transforms.DoFn; -import org.apache.beam.sdk.transforms.DoFn.Element; -import org.apache.beam.sdk.transforms.DoFn.OutputReceiver; import org.apache.beam.sdk.transforms.Max; import org.apache.beam.sdk.transforms.ParDo; import org.apache.beam.sdk.values.PCollection; @@ -65,9 +63,9 @@ public LogOutput(String prefix) { } @ProcessElement - public void processElement(@Element T element, OutputReceiver receiver) throws Exception { - LOG.info("{}{}", prefix, element); - receiver.output(element); + public void processElement(ProcessContext c) throws Exception { + LOG.info("{}{}", prefix, c.element()); + c.output(c.element()); } } } diff --git a/examples/java/src/main/java/org/apache/beam/examples/MaxPerKeyExample.java b/examples/java/src/main/java/org/apache/beam/examples/MaxPerKeyExample.java index 5ad156a343e1..f5eda8179929 100644 --- a/examples/java/src/main/java/org/apache/beam/examples/MaxPerKeyExample.java +++ b/examples/java/src/main/java/org/apache/beam/examples/MaxPerKeyExample.java @@ -22,8 +22,6 @@ import org.apache.beam.sdk.options.PipelineOptionsFactory; import org.apache.beam.sdk.transforms.Create; import org.apache.beam.sdk.transforms.DoFn; -import org.apache.beam.sdk.transforms.DoFn.Element; -import org.apache.beam.sdk.transforms.DoFn.OutputReceiver; import org.apache.beam.sdk.transforms.Max; import org.apache.beam.sdk.transforms.ParDo; import org.apache.beam.sdk.values.KV; @@ -69,9 +67,9 @@ public LogOutput(String prefix) { } @ProcessElement - public void processElement(@Element T element, OutputReceiver receiver) throws Exception { - LOG.info("{}{}", prefix, element); - receiver.output(element); + public void processElement(ProcessContext c) throws Exception { + LOG.info("{}{}", prefix, c.element()); + c.output(c.element()); } } } diff --git a/examples/java/src/main/java/org/apache/beam/examples/MeanExample.java b/examples/java/src/main/java/org/apache/beam/examples/MeanExample.java index fbb51636253a..a31907977dfb 100644 --- a/examples/java/src/main/java/org/apache/beam/examples/MeanExample.java +++ b/examples/java/src/main/java/org/apache/beam/examples/MeanExample.java @@ -22,8 +22,6 @@ import org.apache.beam.sdk.options.PipelineOptionsFactory; import org.apache.beam.sdk.transforms.Create; import org.apache.beam.sdk.transforms.DoFn; -import org.apache.beam.sdk.transforms.DoFn.Element; -import org.apache.beam.sdk.transforms.DoFn.OutputReceiver; import org.apache.beam.sdk.transforms.Mean; import org.apache.beam.sdk.transforms.ParDo; import org.apache.beam.sdk.values.PCollection; @@ -65,9 +63,9 @@ public LogOutput(String prefix) { } @ProcessElement - public void processElement(@Element T element, OutputReceiver receiver) throws Exception { - LOG.info("{}{}", prefix, element); - receiver.output(element); + public void processElement(ProcessContext c) throws Exception { + LOG.info("{}{}", prefix, c.element()); + c.output(c.element()); } } } diff --git a/examples/java/src/main/java/org/apache/beam/examples/MeanPerKeyExample.java b/examples/java/src/main/java/org/apache/beam/examples/MeanPerKeyExample.java index 97f598d4abb4..aecccee067e4 100644 --- a/examples/java/src/main/java/org/apache/beam/examples/MeanPerKeyExample.java +++ b/examples/java/src/main/java/org/apache/beam/examples/MeanPerKeyExample.java @@ -22,8 +22,6 @@ import org.apache.beam.sdk.options.PipelineOptionsFactory; import org.apache.beam.sdk.transforms.Create; import org.apache.beam.sdk.transforms.DoFn; -import org.apache.beam.sdk.transforms.DoFn.Element; -import org.apache.beam.sdk.transforms.DoFn.OutputReceiver; import org.apache.beam.sdk.transforms.Mean; import org.apache.beam.sdk.transforms.ParDo; import org.apache.beam.sdk.values.KV; @@ -69,9 +67,9 @@ public LogOutput(String prefix) { } @ProcessElement - public void processElement(@Element T element, OutputReceiver receiver) throws Exception { - LOG.info("{}{}", prefix, element); - receiver.output(element); + public void processElement(ProcessContext c) throws Exception { + LOG.info("{}{}", prefix, c.element()); + c.output(c.element()); } } } diff --git a/examples/java/src/main/java/org/apache/beam/examples/MinExample.java b/examples/java/src/main/java/org/apache/beam/examples/MinExample.java index db46f1323121..a76bcdc5ee3f 100644 --- a/examples/java/src/main/java/org/apache/beam/examples/MinExample.java +++ b/examples/java/src/main/java/org/apache/beam/examples/MinExample.java @@ -22,8 +22,6 @@ import org.apache.beam.sdk.options.PipelineOptionsFactory; import org.apache.beam.sdk.transforms.Create; import org.apache.beam.sdk.transforms.DoFn; -import org.apache.beam.sdk.transforms.DoFn.Element; -import org.apache.beam.sdk.transforms.DoFn.OutputReceiver; import org.apache.beam.sdk.transforms.Min; import org.apache.beam.sdk.transforms.ParDo; import org.apache.beam.sdk.values.PCollection; @@ -65,9 +63,9 @@ public LogOutput(String prefix) { } @ProcessElement - public void processElement(@Element T element, OutputReceiver receiver) throws Exception { - LOG.info("{}{}", prefix, element); - receiver.output(element); + public void processElement(ProcessContext c) throws Exception { + LOG.info("{}{}", prefix, c.element()); + c.output(c.element()); } } } diff --git a/examples/java/src/main/java/org/apache/beam/examples/MinPerKeyExample.java b/examples/java/src/main/java/org/apache/beam/examples/MinPerKeyExample.java index e8c3c51d3c23..d3c0312feaf3 100644 --- a/examples/java/src/main/java/org/apache/beam/examples/MinPerKeyExample.java +++ b/examples/java/src/main/java/org/apache/beam/examples/MinPerKeyExample.java @@ -22,8 +22,6 @@ import org.apache.beam.sdk.options.PipelineOptionsFactory; import org.apache.beam.sdk.transforms.Create; import org.apache.beam.sdk.transforms.DoFn; -import org.apache.beam.sdk.transforms.DoFn.Element; -import org.apache.beam.sdk.transforms.DoFn.OutputReceiver; import org.apache.beam.sdk.transforms.Min; import org.apache.beam.sdk.transforms.ParDo; import org.apache.beam.sdk.values.KV; @@ -69,9 +67,9 @@ public LogOutput(String prefix) { } @ProcessElement - public void processElement(@Element T element, OutputReceiver receiver) throws Exception { - LOG.info("LogOutput: {} {}", prefix, element); - receiver.output(element); + public void processElement(ProcessContext c) throws Exception { + LOG.info("LogOutput: {} {}", prefix, c.element()); + c.output(c.element()); } } } diff --git a/examples/java/src/main/java/org/apache/beam/examples/PartitionExample.java b/examples/java/src/main/java/org/apache/beam/examples/PartitionExample.java index c202aedf76c0..b34f2bdd16bf 100644 --- a/examples/java/src/main/java/org/apache/beam/examples/PartitionExample.java +++ b/examples/java/src/main/java/org/apache/beam/examples/PartitionExample.java @@ -32,8 +32,6 @@ import org.apache.beam.sdk.options.PipelineOptionsFactory; import org.apache.beam.sdk.transforms.Create; import org.apache.beam.sdk.transforms.DoFn; -import org.apache.beam.sdk.transforms.DoFn.Element; -import org.apache.beam.sdk.transforms.DoFn.OutputReceiver; import org.apache.beam.sdk.transforms.ParDo; import org.apache.beam.sdk.transforms.Partition; import org.apache.beam.sdk.values.PCollection; @@ -194,9 +192,9 @@ public LogOutput(String prefix) { } @ProcessElement - public void processElement(@Element T element, OutputReceiver receiver) throws Exception { - LOG.info("{}{}", prefix, element); - receiver.output(element); + public void processElement(ProcessContext c) throws Exception { + LOG.info("{}{}", prefix, c.element()); + c.output(c.element()); } } } diff --git a/examples/java/src/main/java/org/apache/beam/examples/RateLimiterSimple.java b/examples/java/src/main/java/org/apache/beam/examples/RateLimiterSimple.java index 57b7a0848499..3ec8fcec0bd8 100644 --- a/examples/java/src/main/java/org/apache/beam/examples/RateLimiterSimple.java +++ b/examples/java/src/main/java/org/apache/beam/examples/RateLimiterSimple.java @@ -31,8 +31,6 @@ import org.apache.beam.sdk.options.PipelineOptionsFactory; import org.apache.beam.sdk.transforms.Create; import org.apache.beam.sdk.transforms.DoFn; -import org.apache.beam.sdk.transforms.DoFn.Element; -import org.apache.beam.sdk.transforms.DoFn.OutputReceiver; import org.apache.beam.sdk.transforms.ParDo; import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.base.Preconditions; import org.checkerframework.checker.nullness.qual.Nullable; @@ -100,8 +98,8 @@ public void teardown() { } @ProcessElement - public void processElement(@Element String element, OutputReceiver receiver) - throws Exception { + public void processElement(ProcessContext c) throws Exception { + String element = c.element(); try { Preconditions.checkNotNull(rateLimiter).allow(1); } catch (Exception e) { @@ -111,7 +109,7 @@ public void processElement(@Element String element, OutputReceiver recei // Simulate external API call LOG.info("Processing: {}", element); Thread.sleep(100); - receiver.output("Processed: " + element); + c.output("Processed: " + element); } } // [END RateLimiterSimpleJava] diff --git a/examples/java/src/main/java/org/apache/beam/examples/RegexExample.java b/examples/java/src/main/java/org/apache/beam/examples/RegexExample.java index d80bc9fedbaa..a0d467718bed 100644 --- a/examples/java/src/main/java/org/apache/beam/examples/RegexExample.java +++ b/examples/java/src/main/java/org/apache/beam/examples/RegexExample.java @@ -22,8 +22,6 @@ import org.apache.beam.sdk.options.PipelineOptionsFactory; import org.apache.beam.sdk.transforms.Create; import org.apache.beam.sdk.transforms.DoFn; -import org.apache.beam.sdk.transforms.DoFn.Element; -import org.apache.beam.sdk.transforms.DoFn.OutputReceiver; import org.apache.beam.sdk.transforms.ParDo; import org.apache.beam.sdk.transforms.Regex; import org.apache.beam.sdk.values.PCollection; @@ -77,9 +75,9 @@ public LogOutput(String prefix) { } @ProcessElement - public void processElement(@Element T element, OutputReceiver receiver) throws Exception { - LOG.info("{}{}", prefix, element); - receiver.output(element); + public void processElement(ProcessContext c) throws Exception { + LOG.info("{}{}", prefix, c.element()); + c.output(c.element()); } } } diff --git a/examples/java/src/main/java/org/apache/beam/examples/SampleExample.java b/examples/java/src/main/java/org/apache/beam/examples/SampleExample.java index 8d9db0cf3a86..ed1d90606d3b 100644 --- a/examples/java/src/main/java/org/apache/beam/examples/SampleExample.java +++ b/examples/java/src/main/java/org/apache/beam/examples/SampleExample.java @@ -22,8 +22,6 @@ import org.apache.beam.sdk.options.PipelineOptionsFactory; import org.apache.beam.sdk.transforms.Create; import org.apache.beam.sdk.transforms.DoFn; -import org.apache.beam.sdk.transforms.DoFn.Element; -import org.apache.beam.sdk.transforms.DoFn.OutputReceiver; import org.apache.beam.sdk.transforms.ParDo; import org.apache.beam.sdk.transforms.Sample; import org.apache.beam.sdk.values.KV; @@ -78,9 +76,9 @@ public LogOutput(String prefix) { } @ProcessElement - public void processElement(@Element T element, OutputReceiver receiver) throws Exception { - LOG.info("{}{}", prefix, element); - receiver.output(element); + public void processElement(ProcessContext c) throws Exception { + LOG.info("{}{}", prefix, c.element()); + c.output(c.element()); } } } diff --git a/examples/java/src/main/java/org/apache/beam/examples/SumExample.java b/examples/java/src/main/java/org/apache/beam/examples/SumExample.java index a571d8dc5596..00fcc8697926 100644 --- a/examples/java/src/main/java/org/apache/beam/examples/SumExample.java +++ b/examples/java/src/main/java/org/apache/beam/examples/SumExample.java @@ -22,8 +22,6 @@ import org.apache.beam.sdk.options.PipelineOptionsFactory; import org.apache.beam.sdk.transforms.Create; import org.apache.beam.sdk.transforms.DoFn; -import org.apache.beam.sdk.transforms.DoFn.Element; -import org.apache.beam.sdk.transforms.DoFn.OutputReceiver; import org.apache.beam.sdk.transforms.ParDo; import org.apache.beam.sdk.transforms.Sum; import org.apache.beam.sdk.values.PCollection; @@ -65,9 +63,9 @@ public LogOutput(String prefix) { } @ProcessElement - public void processElement(@Element T element, OutputReceiver receiver) throws Exception { - LOG.info("{}{}", prefix, element); - receiver.output(element); + public void processElement(ProcessContext c) throws Exception { + LOG.info("{}{}", prefix, c.element()); + c.output(c.element()); } } } diff --git a/examples/java/src/main/java/org/apache/beam/examples/SumPerKeyExample.java b/examples/java/src/main/java/org/apache/beam/examples/SumPerKeyExample.java index a2c92ef0c80e..45d4a9ffd852 100644 --- a/examples/java/src/main/java/org/apache/beam/examples/SumPerKeyExample.java +++ b/examples/java/src/main/java/org/apache/beam/examples/SumPerKeyExample.java @@ -22,8 +22,6 @@ import org.apache.beam.sdk.options.PipelineOptionsFactory; import org.apache.beam.sdk.transforms.Create; import org.apache.beam.sdk.transforms.DoFn; -import org.apache.beam.sdk.transforms.DoFn.Element; -import org.apache.beam.sdk.transforms.DoFn.OutputReceiver; import org.apache.beam.sdk.transforms.ParDo; import org.apache.beam.sdk.transforms.Sum; import org.apache.beam.sdk.values.KV; @@ -69,9 +67,9 @@ public LogOutput(String prefix) { } @ProcessElement - public void processElement(@Element T element, OutputReceiver receiver) throws Exception { - LOG.info("{}{}", prefix, element); - receiver.output(element); + public void processElement(ProcessContext c) throws Exception { + LOG.info("{}{}", prefix, c.element()); + c.output(c.element()); } } } diff --git a/examples/java/src/main/java/org/apache/beam/examples/ToStringExample.java b/examples/java/src/main/java/org/apache/beam/examples/ToStringExample.java index 54e783231908..23e3db6cfd96 100644 --- a/examples/java/src/main/java/org/apache/beam/examples/ToStringExample.java +++ b/examples/java/src/main/java/org/apache/beam/examples/ToStringExample.java @@ -22,8 +22,6 @@ import org.apache.beam.sdk.options.PipelineOptionsFactory; import org.apache.beam.sdk.transforms.Create; import org.apache.beam.sdk.transforms.DoFn; -import org.apache.beam.sdk.transforms.DoFn.Element; -import org.apache.beam.sdk.transforms.DoFn.OutputReceiver; import org.apache.beam.sdk.transforms.ParDo; import org.apache.beam.sdk.transforms.ToString; import org.apache.beam.sdk.values.KV; @@ -76,9 +74,9 @@ public LogOutput(String prefix) { } @ProcessElement - public void processElement(@Element T element, OutputReceiver receiver) throws Exception { - LOG.info("{}{}", prefix, element); - receiver.output(element); + public void processElement(ProcessContext c) throws Exception { + LOG.info("{}{}", prefix, c.element()); + c.output(c.element()); } } } diff --git a/examples/java/src/main/java/org/apache/beam/examples/TopExample.java b/examples/java/src/main/java/org/apache/beam/examples/TopExample.java index f602aa345569..520af0f66550 100644 --- a/examples/java/src/main/java/org/apache/beam/examples/TopExample.java +++ b/examples/java/src/main/java/org/apache/beam/examples/TopExample.java @@ -23,8 +23,6 @@ import org.apache.beam.sdk.options.PipelineOptionsFactory; import org.apache.beam.sdk.transforms.Create; import org.apache.beam.sdk.transforms.DoFn; -import org.apache.beam.sdk.transforms.DoFn.Element; -import org.apache.beam.sdk.transforms.DoFn.OutputReceiver; import org.apache.beam.sdk.transforms.ParDo; import org.apache.beam.sdk.transforms.Top; import org.apache.beam.sdk.values.PCollection; @@ -66,9 +64,9 @@ public LogOutput(String prefix) { } @ProcessElement - public void processElement(@Element T element, OutputReceiver receiver) throws Exception { - LOG.info("{}{}", prefix, element); - receiver.output(element); + public void processElement(ProcessContext c) throws Exception { + LOG.info("{}{}", prefix, c.element()); + c.output(c.element()); } } } diff --git a/examples/java/src/main/java/org/apache/beam/examples/ValuesExample.java b/examples/java/src/main/java/org/apache/beam/examples/ValuesExample.java index 1b9839ac03f0..3fc9e84fcb39 100644 --- a/examples/java/src/main/java/org/apache/beam/examples/ValuesExample.java +++ b/examples/java/src/main/java/org/apache/beam/examples/ValuesExample.java @@ -22,8 +22,6 @@ import org.apache.beam.sdk.options.PipelineOptionsFactory; import org.apache.beam.sdk.transforms.Create; import org.apache.beam.sdk.transforms.DoFn; -import org.apache.beam.sdk.transforms.DoFn.Element; -import org.apache.beam.sdk.transforms.DoFn.OutputReceiver; import org.apache.beam.sdk.transforms.ParDo; import org.apache.beam.sdk.transforms.Values; import org.apache.beam.sdk.values.KV; @@ -72,9 +70,9 @@ public LogOutput(String prefix) { } @ProcessElement - public void processElement(@Element T element, OutputReceiver receiver) throws Exception { - LOG.info("{}{}", prefix, element); - receiver.output(element); + public void processElement(ProcessContext c) throws Exception { + LOG.info("{}{}", prefix, c.element()); + c.output(c.element()); } } } diff --git a/examples/java/src/main/java/org/apache/beam/examples/ViewExample.java b/examples/java/src/main/java/org/apache/beam/examples/ViewExample.java index d40d6cc333ef..01c1b1c3f31d 100644 --- a/examples/java/src/main/java/org/apache/beam/examples/ViewExample.java +++ b/examples/java/src/main/java/org/apache/beam/examples/ViewExample.java @@ -23,9 +23,6 @@ import org.apache.beam.sdk.options.PipelineOptionsFactory; import org.apache.beam.sdk.transforms.Create; import org.apache.beam.sdk.transforms.DoFn; -import org.apache.beam.sdk.transforms.DoFn.Element; -import org.apache.beam.sdk.transforms.DoFn.OutputReceiver; -import org.apache.beam.sdk.transforms.DoFn.SideInput; import org.apache.beam.sdk.transforms.ParDo; import org.apache.beam.sdk.transforms.View; import org.apache.beam.sdk.values.KV; @@ -86,8 +83,10 @@ public static void main(String[] args) { @ProcessElement public void processElement( @Element KV person, - @SideInput("citiesToCountries") Map citiesToCountries, - OutputReceiver> out) { + OutputReceiver> out, + ProcessContext context) { + Map citiesToCountries = + context.sideInput(citiesToCountriesView); String city = person.getValue(); String country = citiesToCountries.get(city); if (country == null) { @@ -96,7 +95,7 @@ public void processElement( out.output(KV.of(person.getKey(), country)); } }) - .withSideInput("citiesToCountries", citiesToCountriesView)); + .withSideInputs(citiesToCountriesView)); // [END main_section] output.apply("Log", ParDo.of(new LogOutput<>("Output: "))); @@ -113,11 +112,9 @@ public LogOutput(String prefix) { } @ProcessElement - public void processElement( - @Element KV element, OutputReceiver> receiver) - throws Exception { - LOG.info("{}{}", prefix, element); - receiver.output(element); + public void processElement(ProcessContext c) throws Exception { + LOG.info("{}{}", prefix, c.element()); + c.output(c.element()); } } } diff --git a/examples/java/src/main/java/org/apache/beam/examples/WindowExample.java b/examples/java/src/main/java/org/apache/beam/examples/WindowExample.java index d6a3dff2fa04..244a42bc9613 100644 --- a/examples/java/src/main/java/org/apache/beam/examples/WindowExample.java +++ b/examples/java/src/main/java/org/apache/beam/examples/WindowExample.java @@ -25,8 +25,6 @@ import org.apache.beam.sdk.transforms.Count; import org.apache.beam.sdk.transforms.Create; import org.apache.beam.sdk.transforms.DoFn; -import org.apache.beam.sdk.transforms.DoFn.Element; -import org.apache.beam.sdk.transforms.DoFn.OutputReceiver; import org.apache.beam.sdk.transforms.ParDo; import org.apache.beam.sdk.transforms.windowing.FixedWindows; import org.apache.beam.sdk.transforms.windowing.Window; @@ -89,9 +87,9 @@ public LogOutput(String prefix) { } @ProcessElement - public void processElement(@Element T element, OutputReceiver receiver) throws Exception { - LOG.info("{}{}", prefix, element); - receiver.output(element); + public void processElement(ProcessContext c) throws Exception { + LOG.info("{}{}", prefix, c.element()); + c.output(c.element()); } } } diff --git a/examples/java/src/main/java/org/apache/beam/examples/complete/AutoComplete.java b/examples/java/src/main/java/org/apache/beam/examples/complete/AutoComplete.java index 24a3c1b16dcf..11114043e830 100644 --- a/examples/java/src/main/java/org/apache/beam/examples/complete/AutoComplete.java +++ b/examples/java/src/main/java/org/apache/beam/examples/complete/AutoComplete.java @@ -52,8 +52,6 @@ import org.apache.beam.sdk.testing.PAssert; import org.apache.beam.sdk.transforms.Count; import org.apache.beam.sdk.transforms.DoFn; -import org.apache.beam.sdk.transforms.DoFn.Element; -import org.apache.beam.sdk.transforms.DoFn.OutputReceiver; import org.apache.beam.sdk.transforms.Filter; import org.apache.beam.sdk.transforms.Flatten; import org.apache.beam.sdk.transforms.PTransform; @@ -133,11 +131,10 @@ public PCollection>> expand(PCollection, CompletionCandidate>() { @ProcessElement - public void processElement( - @Element KV element, - OutputReceiver receiver) { - receiver.output( - new CompletionCandidate(element.getKey(), element.getValue())); + public void processElement(ProcessContext c) { + c.output( + new CompletionCandidate( + c.element().getKey(), c.element().getValue())); } })); @@ -213,11 +210,9 @@ public int partitionFor(KV> elem, int numParti private static class FlattenTops extends DoFn>, CompletionCandidate> { @ProcessElement - public void processElement( - @Element KV> element, - OutputReceiver receiver) { - for (CompletionCandidate cc : element.getValue()) { - receiver.output(cc); + public void processElement(ProcessContext c) { + for (CompletionCandidate cc : c.element().getValue()) { + c.output(cc); } } } @@ -272,12 +267,10 @@ public AllPrefixes(int minPrefix, int maxPrefix) { } @ProcessElement - public void processElement( - @Element CompletionCandidate element, - OutputReceiver> receiver) { - String word = element.value; + public void processElement(ProcessContext c) { + String word = c.element().value; for (int i = minPrefix; i <= Math.min(word.length(), maxPrefix); i++) { - receiver.output(KV.of(word.substring(0, i), element)); + c.output(KV.of(word.substring(0, i), c.element())); } } } @@ -339,24 +332,23 @@ public String toString() { /** Takes as input a set of strings, and emits each #hashtag found therein. */ static class ExtractHashtags extends DoFn { @ProcessElement - public void processElement(@Element String element, OutputReceiver receiver) { - Matcher m = Pattern.compile("#\\S+").matcher(element); + public void processElement(ProcessContext c) { + Matcher m = Pattern.compile("#\\S+").matcher(c.element()); while (m.find()) { - receiver.output(m.group().substring(1)); + c.output(m.group().substring(1)); } } } static class FormatForBigquery extends DoFn>, TableRow> { @ProcessElement - public void processElement( - @Element KV> element, OutputReceiver receiver) { + public void processElement(ProcessContext c) { List completions = new ArrayList<>(); - for (CompletionCandidate cc : element.getValue()) { + for (CompletionCandidate cc : c.element().getValue()) { completions.add(new TableRow().set("count", cc.getCount()).set("tag", cc.getValue())); } - TableRow row = new TableRow().set("prefix", element.getKey()).set("tags", completions); - receiver.output(row); + TableRow row = new TableRow().set("prefix", c.element().getKey()).set("tags", completions); + c.output(row); } /** Defines the BigQuery schema used for the output. */ @@ -394,16 +386,15 @@ public FormatForDatastore(String kind, String ancestorKey) { } @ProcessElement - public void processElement( - @Element KV> element, OutputReceiver receiver) { + public void processElement(ProcessContext c) { Entity.Builder entityBuilder = Entity.newBuilder(); com.google.datastore.v1.Key key = - makeKey(makeKey(kind, ancestorKey).build(), kind, element.getKey()).build(); + makeKey(makeKey(kind, ancestorKey).build(), kind, c.element().getKey()).build(); entityBuilder.setKey(key); List candidates = new ArrayList<>(); Map properties = new HashMap<>(); - for (CompletionCandidate tag : element.getValue()) { + for (CompletionCandidate tag : c.element().getValue()) { Entity.Builder tagEntity = Entity.newBuilder(); properties.put("tag", makeValue(tag.value).build()); properties.put("count", makeValue(tag.count).build()); @@ -411,7 +402,7 @@ public void processElement( } properties.put("candidates", makeValue(candidates).build()); entityBuilder.putAllProperties(properties); - receiver.output(entityBuilder.build()); + c.output(entityBuilder.build()); } } @@ -536,12 +527,11 @@ public static void runAutocompletePipeline(Options options) throws IOException { ParDo.of( new DoFn>, Long>() { @ProcessElement - public void process( - @Element KV> elm, - OutputReceiver receiver) { + public void process(ProcessContext c) { + KV> elm = c.element(); Long listHash = - elm.getValue().stream().mapToLong(cc -> cc.hashCode()).sum(); - receiver.output(Long.valueOf(elm.getKey().hashCode()) + listHash); + c.element().getValue().stream().mapToLong(cc -> cc.hashCode()).sum(); + c.output(Long.valueOf(elm.getKey().hashCode()) + listHash); } })) .apply(Sum.longsGlobally()); diff --git a/examples/java/src/main/java/org/apache/beam/examples/complete/StreamingWordExtract.java b/examples/java/src/main/java/org/apache/beam/examples/complete/StreamingWordExtract.java index ea5f2e5399e7..e4ce5e3eb17e 100644 --- a/examples/java/src/main/java/org/apache/beam/examples/complete/StreamingWordExtract.java +++ b/examples/java/src/main/java/org/apache/beam/examples/complete/StreamingWordExtract.java @@ -34,8 +34,6 @@ import org.apache.beam.sdk.options.PipelineOptionsFactory; import org.apache.beam.sdk.options.StreamingOptions; import org.apache.beam.sdk.transforms.DoFn; -import org.apache.beam.sdk.transforms.DoFn.Element; -import org.apache.beam.sdk.transforms.DoFn.OutputReceiver; import org.apache.beam.sdk.transforms.ParDo; /** @@ -57,12 +55,12 @@ public class StreamingWordExtract { /** A {@link DoFn} that tokenizes lines of text into individual words. */ static class ExtractWords extends DoFn { @ProcessElement - public void processElement(@Element String element, OutputReceiver receiver) { - String[] words = element.split(ExampleUtils.TOKENIZER_PATTERN, -1); + public void processElement(ProcessContext c) { + String[] words = c.element().split(ExampleUtils.TOKENIZER_PATTERN, -1); for (String word : words) { if (!word.isEmpty()) { - receiver.output(word); + c.output(word); } } } @@ -71,8 +69,8 @@ public void processElement(@Element String element, OutputReceiver recei /** A {@link DoFn} that uppercases a word. */ static class Uppercase extends DoFn { @ProcessElement - public void processElement(@Element String element, OutputReceiver receiver) { - receiver.output(element.toUpperCase()); + public void processElement(ProcessContext c) { + c.output(c.element().toUpperCase()); } } @@ -80,8 +78,8 @@ public void processElement(@Element String element, OutputReceiver recei static class StringToRowConverter extends DoFn { /** In this example, put the whole string into single BigQuery field. */ @ProcessElement - public void processElement(@Element String element, OutputReceiver receiver) { - receiver.output(new TableRow().set("string_field", element)); + public void processElement(ProcessContext c) { + c.output(new TableRow().set("string_field", c.element())); } static TableSchema getSchema() { diff --git a/examples/java/src/main/java/org/apache/beam/examples/complete/TfIdf.java b/examples/java/src/main/java/org/apache/beam/examples/complete/TfIdf.java index a61bd4c88651..b3e5fd04fa7e 100644 --- a/examples/java/src/main/java/org/apache/beam/examples/complete/TfIdf.java +++ b/examples/java/src/main/java/org/apache/beam/examples/complete/TfIdf.java @@ -56,9 +56,6 @@ import org.apache.beam.sdk.transforms.Count; import org.apache.beam.sdk.transforms.Distinct; import org.apache.beam.sdk.transforms.DoFn; -import org.apache.beam.sdk.transforms.DoFn.Element; -import org.apache.beam.sdk.transforms.DoFn.OutputReceiver; -import org.apache.beam.sdk.transforms.DoFn.SideInput; import org.apache.beam.sdk.transforms.Flatten; import org.apache.beam.sdk.transforms.Keys; import org.apache.beam.sdk.transforms.PTransform; @@ -239,11 +236,9 @@ public PCollection>> expand( ParDo.of( new DoFn, KV>() { @ProcessElement - public void processElement( - @Element KV element, - OutputReceiver> receiver) { - URI uri = element.getKey(); - String line = element.getValue(); + public void processElement(ProcessContext c) { + URI uri = c.element().getKey(); + String line = c.element().getValue(); for (String word : line.split("\\W+", -1)) { // Log INFO messages when the word “love” is found. if ("love".equalsIgnoreCase(word)) { @@ -251,7 +246,7 @@ public void processElement( } if (!word.isEmpty()) { - receiver.output(KV.of(uri, word.toLowerCase())); + c.output(KV.of(uri, word.toLowerCase())); } } } @@ -286,13 +281,11 @@ public void processElement( ParDo.of( new DoFn, Long>, KV>>() { @ProcessElement - public void processElement( - @Element KV, Long> element, - OutputReceiver>> receiver) { - URI uri = element.getKey().getKey(); - String word = element.getKey().getValue(); - Long occurrences = element.getValue(); - receiver.output(KV.of(uri, KV.of(word, occurrences))); + public void processElement(ProcessContext c) { + URI uri = c.element().getKey().getKey(); + String word = c.element().getKey().getValue(); + Long occurrences = c.element().getValue(); + c.output(KV.of(uri, KV.of(word, occurrences))); } })); @@ -329,18 +322,16 @@ public void processElement( ParDo.of( new DoFn, KV>>() { @ProcessElement - public void processElement( - @Element KV element, - OutputReceiver>> receiver) { - URI uri = element.getKey(); - Long wordTotal = element.getValue().getOnly(wordTotalsTag); + public void processElement(ProcessContext c) { + URI uri = c.element().getKey(); + Long wordTotal = c.element().getValue().getOnly(wordTotalsTag); for (KV wordAndCount : - element.getValue().getAll(wordCountsTag)) { + c.element().getValue().getAll(wordCountsTag)) { String word = wordAndCount.getKey(); Long wordCount = wordAndCount.getValue(); Double termFrequency = wordCount.doubleValue() / wordTotal.doubleValue(); - receiver.output(KV.of(word, KV.of(uri, termFrequency))); + c.output(KV.of(word, KV.of(uri, termFrequency))); } } })); @@ -357,19 +348,17 @@ public void processElement( ParDo.of( new DoFn, KV>() { @ProcessElement - public void processElement( - @SideInput("totalDocuments") Long documentTotal, - @Element KV element, - OutputReceiver> receiver) { - String word = element.getKey(); - Long documentCount = element.getValue(); + public void processElement(ProcessContext c) { + String word = c.element().getKey(); + Long documentCount = c.element().getValue(); + Long documentTotal = c.sideInput(totalDocuments); Double documentFrequency = documentCount.doubleValue() / documentTotal.doubleValue(); - receiver.output(KV.of(word, documentFrequency)); + c.output(KV.of(word, documentFrequency)); } }) - .withSideInput("totalDocuments", totalDocuments)); + .withSideInputs(totalDocuments)); // Join the term frequency and document frequency // collections, each keyed on the word. @@ -391,17 +380,15 @@ public void processElement( ParDo.of( new DoFn, KV>>() { @ProcessElement - public void processElement( - @Element KV element, - OutputReceiver>> receiver) { - String word = element.getKey(); - Double df = element.getValue().getOnly(dfTag); + public void processElement(ProcessContext c) { + String word = c.element().getKey(); + Double df = c.element().getValue().getOnly(dfTag); - for (KV uriAndTf : element.getValue().getAll(tfTag)) { + for (KV uriAndTf : c.element().getValue().getAll(tfTag)) { URI uri = uriAndTf.getKey(); Double tf = uriAndTf.getValue(); Double tfIdf = tf * Math.log(1 / df); - receiver.output(KV.of(word, KV.of(uri, tfIdf))); + c.output(KV.of(word, KV.of(uri, tfIdf))); } } })); @@ -432,15 +419,13 @@ public PDone expand(PCollection>> wordToUriAndTfIdf) ParDo.of( new DoFn>, String>() { @ProcessElement - public void processElement( - @Element KV> element, - OutputReceiver receiver) { - receiver.output( + public void processElement(ProcessContext c) { + c.output( String.format( "%s,\t%s,\t%f", - element.getKey(), - element.getValue().getKey(), - element.getValue().getValue())); + c.element().getKey(), + c.element().getValue().getKey(), + c.element().getValue().getValue())); } })) .apply(TextIO.write().to(output).withSuffix(".csv")); diff --git a/examples/java/src/main/java/org/apache/beam/examples/complete/TopWikipediaSessions.java b/examples/java/src/main/java/org/apache/beam/examples/complete/TopWikipediaSessions.java index d156777e0f97..b06cd8da9d43 100644 --- a/examples/java/src/main/java/org/apache/beam/examples/complete/TopWikipediaSessions.java +++ b/examples/java/src/main/java/org/apache/beam/examples/complete/TopWikipediaSessions.java @@ -48,8 +48,6 @@ import org.apache.beam.sdk.options.Validation; import org.apache.beam.sdk.transforms.Count; import org.apache.beam.sdk.transforms.DoFn; -import org.apache.beam.sdk.transforms.DoFn.Element; -import org.apache.beam.sdk.transforms.DoFn.OutputReceiver; import org.apache.beam.sdk.transforms.MapElements; import org.apache.beam.sdk.transforms.PTransform; import org.apache.beam.sdk.transforms.ParDo; @@ -95,7 +93,8 @@ public class TopWikipediaSessions { /** Extracts user and timestamp from a TableRow representing a Wikipedia edit. */ static class ExtractUserAndTimestamp extends DoFn { @ProcessElement - public void processElement(@Element TableRow row, OutputReceiver receiver) { + public void processElement(ProcessContext c) { + TableRow row = c.element(); int timestamp; // TODO(BEAM-5390): Avoid this workaround. try { @@ -106,7 +105,7 @@ public void processElement(@Element TableRow row, OutputReceiver receive String userName = (String) row.get("contributor_username"); if (userName != null) { // Sets the implicit timestamp field to be used in windowing. - receiver.outputWithTimestamp(userName, new Instant(timestamp * 1000L)); + c.outputWithTimestamp(userName, new Instant(timestamp * 1000L)); } } } @@ -144,24 +143,18 @@ public PCollection>> expand(PCollection> static class SessionsToStringsDoFn extends DoFn, KV> { @ProcessElement - public void processElement( - BoundedWindow window, - @Element KV element, - OutputReceiver> receiver) { - receiver.output(KV.of(element.getKey() + " : " + window, element.getValue())); + public void processElement(ProcessContext c, BoundedWindow window) { + c.output(KV.of(c.element().getKey() + " : " + window, c.element().getValue())); } } static class FormatOutputDoFn extends DoFn>, String> { @ProcessElement - public void processElement( - BoundedWindow window, - @Element List> element, - OutputReceiver receiver) { - for (KV item : element) { + public void processElement(ProcessContext c, BoundedWindow window) { + for (KV item : c.element()) { String session = item.getKey(); long count = item.getValue(); - receiver.output(session + " : " + count + " : " + ((IntervalWindow) window).start()); + c.output(session + " : " + count + " : " + ((IntervalWindow) window).start()); } } } @@ -194,11 +187,10 @@ public PCollection expand(PCollection input) { ParDo.of( new DoFn() { @ProcessElement - public void processElement( - @Element String element, OutputReceiver receiver) { - if (Math.abs((long) element.hashCode()) + public void processElement(ProcessContext c) { + if (Math.abs((long) c.element().hashCode()) <= Integer.MAX_VALUE * samplingThreshold) { - receiver.output(element); + c.output(c.element()); } } })) diff --git a/examples/java/src/main/java/org/apache/beam/examples/complete/TrafficMaxLaneFlow.java b/examples/java/src/main/java/org/apache/beam/examples/complete/TrafficMaxLaneFlow.java index b22e3fed6698..6f75e2e03d99 100644 --- a/examples/java/src/main/java/org/apache/beam/examples/complete/TrafficMaxLaneFlow.java +++ b/examples/java/src/main/java/org/apache/beam/examples/complete/TrafficMaxLaneFlow.java @@ -58,9 +58,6 @@ import org.apache.beam.sdk.options.PipelineOptionsFactory; import org.apache.beam.sdk.transforms.Combine; import org.apache.beam.sdk.transforms.DoFn; -import org.apache.beam.sdk.transforms.DoFn.Element; -import org.apache.beam.sdk.transforms.DoFn.OutputReceiver; -import org.apache.beam.sdk.transforms.DoFn.Timestamp; import org.apache.beam.sdk.transforms.PTransform; import org.apache.beam.sdk.transforms.ParDo; import org.apache.beam.sdk.transforms.SerializableFunction; @@ -185,14 +182,13 @@ static class ExtractTimestamps extends DoFn { DateTimeFormat.forPattern("MM/dd/yyyy HH:mm:ss"); @ProcessElement - public void processElement(@Element String element, OutputReceiver receiver) - throws Exception { - String[] items = element.split(",", -1); + public void processElement(DoFn.ProcessContext c) throws Exception { + String[] items = c.element().split(",", -1); if (items.length > 0) { try { String timestamp = items[0]; - receiver.outputWithTimestamp(element, new Instant(dateTimeFormat.parseMillis(timestamp))); + c.outputWithTimestamp(c.element(), new Instant(dateTimeFormat.parseMillis(timestamp))); } catch (IllegalArgumentException e) { // Skip the invalid input. } @@ -210,9 +206,8 @@ public void processElement(@Element String element, OutputReceiver recei static class ExtractFlowInfoFn extends DoFn> { @ProcessElement - public void processElement( - @Element String element, OutputReceiver> receiver) { - String[] items = element.split(",", -1); + public void processElement(ProcessContext c) { + String[] items = c.element().split(",", -1); if (items.length < 48) { // Skip the invalid input. return; @@ -241,7 +236,7 @@ public void processElement( laneAvgOccupancy, laneAvgSpeed, totalFlow); - receiver.output(KV.of(stationId, laneInfo)); + c.output(KV.of(stationId, laneInfo)); } } } @@ -275,15 +270,12 @@ public LaneInfo apply(Iterable input) { */ static class FormatMaxesFn extends DoFn, TableRow> { @ProcessElement - public void processElement( - @Element KV element, - @Timestamp Instant timestamp, - OutputReceiver receiver) { + public void processElement(ProcessContext c) { - LaneInfo laneInfo = element.getValue(); + LaneInfo laneInfo = c.element().getValue(); TableRow row = new TableRow() - .set("station_id", element.getKey()) + .set("station_id", c.element().getKey()) .set("direction", laneInfo.getDirection()) .set("freeway", laneInfo.getFreeway()) .set("lane_max_flow", laneInfo.getLaneFlow()) @@ -292,8 +284,8 @@ public void processElement( .set("avg_speed", laneInfo.getLaneAS()) .set("total_flow", laneInfo.getTotalFlow()) .set("recorded_timestamp", laneInfo.getRecordedTimestamp()) - .set("window_timestamp", timestamp.toString()); - receiver.output(row); + .set("window_timestamp", c.timestamp().toString()); + c.output(row); } /** Defines the BigQuery schema used for the output. */ diff --git a/examples/java/src/main/java/org/apache/beam/examples/complete/TrafficRoutes.java b/examples/java/src/main/java/org/apache/beam/examples/complete/TrafficRoutes.java index 38ba4322f3e6..958415626863 100644 --- a/examples/java/src/main/java/org/apache/beam/examples/complete/TrafficRoutes.java +++ b/examples/java/src/main/java/org/apache/beam/examples/complete/TrafficRoutes.java @@ -63,9 +63,6 @@ import org.apache.beam.sdk.options.Description; import org.apache.beam.sdk.options.PipelineOptionsFactory; import org.apache.beam.sdk.transforms.DoFn; -import org.apache.beam.sdk.transforms.DoFn.Element; -import org.apache.beam.sdk.transforms.DoFn.OutputReceiver; -import org.apache.beam.sdk.transforms.DoFn.Timestamp; import org.apache.beam.sdk.transforms.GroupByKey; import org.apache.beam.sdk.transforms.PTransform; import org.apache.beam.sdk.transforms.ParDo; @@ -190,13 +187,12 @@ static class ExtractTimestamps extends DoFn { DateTimeFormat.forPattern("MM/dd/yyyy HH:mm:ss"); @ProcessElement - public void processElement(@Element String element, OutputReceiver receiver) - throws Exception { - String[] items = element.split(","); + public void processElement(DoFn.ProcessContext c) throws Exception { + String[] items = c.element().split(","); String timestamp = tryParseTimestamp(items); if (timestamp != null) { try { - receiver.outputWithTimestamp(element, new Instant(dateTimeFormat.parseMillis(timestamp))); + c.outputWithTimestamp(c.element(), new Instant(dateTimeFormat.parseMillis(timestamp))); } catch (IllegalArgumentException e) { // Skip the invalid input. } @@ -211,11 +207,8 @@ public void processElement(@Element String element, OutputReceiver recei static class ExtractStationSpeedFn extends DoFn> { @ProcessElement - public void processElement( - @Timestamp Instant timestamp, - @Element String element, - OutputReceiver> receiver) { - String[] items = element.split(","); + public void processElement(ProcessContext c) { + String[] items = c.element().split(","); String stationType = tryParseStationType(items); // For this analysis, use only 'main line' station types if ("ML".equals(stationType)) { @@ -223,10 +216,11 @@ public void processElement( String stationId = tryParseStationId(items); // For this simple example, filter out everything but some hardwired routes. if (avgSpeed != null && stationId != null && sdStations.containsKey(stationId)) { - StationSpeed stationSpeed = new StationSpeed(stationId, avgSpeed, timestamp.getMillis()); + StationSpeed stationSpeed = + new StationSpeed(stationId, avgSpeed, c.timestamp().getMillis()); // The tuple key is the 'route' name stored in the 'sdStations' hash. KV outputValue = KV.of(sdStations.get(stationId), stationSpeed); - receiver.output(outputValue); + c.output(outputValue); } } } @@ -240,16 +234,13 @@ public void processElement( */ static class GatherStats extends DoFn>, KV> { @ProcessElement - public void processElement( - @Element KV> element, - OutputReceiver> receiver) - throws IOException { - String route = element.getKey(); + public void processElement(ProcessContext c) throws IOException { + String route = c.element().getKey(); double speedSum = 0.0; int speedCount = 0; int speedups = 0; int slowdowns = 0; - List infoList = Lists.newArrayList(element.getValue()); + List infoList = Lists.newArrayList(c.element().getValue()); // StationSpeeds sort by embedded timestamp. Collections.sort(infoList); Map prevSpeeds = new HashMap<>(); @@ -277,25 +268,22 @@ public void processElement( double speedAvg = speedSum / speedCount; boolean slowdownEvent = slowdowns >= 2 * speedups; RouteInfo routeInfo = new RouteInfo(route, speedAvg, slowdownEvent); - receiver.output(KV.of(route, routeInfo)); + c.output(KV.of(route, routeInfo)); } } /** Format the results of the slowdown calculations to a TableRow, to save to BigQuery. */ static class FormatStatsFn extends DoFn, TableRow> { @ProcessElement - public void processElement( - @Element KV element, - @Timestamp Instant timestamp, - OutputReceiver receiver) { - RouteInfo routeInfo = element.getValue(); + public void processElement(ProcessContext c) { + RouteInfo routeInfo = c.element().getValue(); TableRow row = new TableRow() .set("avg_speed", routeInfo.getAvgSpeed()) .set("slowdown_event", routeInfo.getSlowdownEvent()) - .set("route", element.getKey()) - .set("window_timestamp", timestamp.toString()); - receiver.output(row); + .set("route", c.element().getKey()) + .set("window_timestamp", c.timestamp().toString()); + c.output(row); } /** Defines the BigQuery schema used for the output. */ diff --git a/examples/java/src/main/java/org/apache/beam/examples/complete/datatokenization/transforms/DataProtectors.java b/examples/java/src/main/java/org/apache/beam/examples/complete/datatokenization/transforms/DataProtectors.java index 6e5321dc54ab..cf097c8ea979 100644 --- a/examples/java/src/main/java/org/apache/beam/examples/complete/datatokenization/transforms/DataProtectors.java +++ b/examples/java/src/main/java/org/apache/beam/examples/complete/datatokenization/transforms/DataProtectors.java @@ -37,9 +37,6 @@ import org.apache.beam.sdk.schemas.Schema.Field; import org.apache.beam.sdk.schemas.Schema.FieldType; import org.apache.beam.sdk.transforms.DoFn; -import org.apache.beam.sdk.transforms.DoFn.Element; -import org.apache.beam.sdk.transforms.DoFn.MultiOutputReceiver; -import org.apache.beam.sdk.transforms.DoFn.OutputReceiver; import org.apache.beam.sdk.transforms.GroupIntoBatches; import org.apache.beam.sdk.transforms.PTransform; import org.apache.beam.sdk.transforms.ParDo; @@ -202,24 +199,20 @@ public void close() { @ProcessElement @SuppressWarnings("argument") - public void process( - @Element KV> element, - OutputReceiver mainReceiver, - MultiOutputReceiver multiReceiver) { + public void process(@Element KV> element, ProcessContext context) { Iterable rows = element.getValue(); try { for (Row outputRow : getTokenizedRow(rows)) { - mainReceiver.output(outputRow); + context.output(outputRow); } } catch (Exception e) { for (Row outputRow : rows) { - multiReceiver - .get(failureTag) - .output( - FailsafeElement.of(outputRow, outputRow) - .setErrorMessage(e.getMessage()) - .setStacktrace(Throwables.getStackTraceAsString(e))); + context.output( + failureTag, + FailsafeElement.of(outputRow, outputRow) + .setErrorMessage(e.getMessage()) + .setStacktrace(Throwables.getStackTraceAsString(e))); } } } diff --git a/examples/java/src/main/java/org/apache/beam/examples/complete/datatokenization/transforms/io/TokenizationBigQueryIO.java b/examples/java/src/main/java/org/apache/beam/examples/complete/datatokenization/transforms/io/TokenizationBigQueryIO.java index 667e26fa9c2d..fe8f4c1afad8 100644 --- a/examples/java/src/main/java/org/apache/beam/examples/complete/datatokenization/transforms/io/TokenizationBigQueryIO.java +++ b/examples/java/src/main/java/org/apache/beam/examples/complete/datatokenization/transforms/io/TokenizationBigQueryIO.java @@ -26,8 +26,6 @@ import org.apache.beam.sdk.io.gcp.bigquery.InsertRetryPolicy; import org.apache.beam.sdk.io.gcp.bigquery.WriteResult; import org.apache.beam.sdk.transforms.DoFn; -import org.apache.beam.sdk.transforms.DoFn.Element; -import org.apache.beam.sdk.transforms.DoFn.OutputReceiver; import org.apache.beam.sdk.transforms.ParDo; import org.apache.beam.sdk.values.PCollection; import org.apache.beam.sdk.values.Row; @@ -92,8 +90,9 @@ public static FailsafeElement wrapBigQueryInsertError( public static class RowToTableRowFn extends DoFn { @ProcessElement - public void processElement(@Element Row row, OutputReceiver receiver) { - receiver.output(BigQueryUtils.toTableRow(row)); + public void processElement(ProcessContext context) { + Row row = context.element(); + context.output(BigQueryUtils.toTableRow(row)); } } } diff --git a/examples/java/src/main/java/org/apache/beam/examples/complete/datatokenization/transforms/io/TokenizationBigTableIO.java b/examples/java/src/main/java/org/apache/beam/examples/complete/datatokenization/transforms/io/TokenizationBigTableIO.java index 435620a4b011..d7d1c3e97232 100644 --- a/examples/java/src/main/java/org/apache/beam/examples/complete/datatokenization/transforms/io/TokenizationBigTableIO.java +++ b/examples/java/src/main/java/org/apache/beam/examples/complete/datatokenization/transforms/io/TokenizationBigTableIO.java @@ -75,10 +75,8 @@ static class TransformToBigTableFormat extends DoFn>> out, - PipelineOptions pipelineOptions) { - DataTokenizationOptions options = pipelineOptions.as(DataTokenizationOptions.class); + @Element Row in, OutputReceiver>> out, ProcessContext c) { + DataTokenizationOptions options = c.getPipelineOptions().as(DataTokenizationOptions.class); // Mapping every field in provided Row to Mutation.SetCell, which will create/update // cell content with provided data Set mutations = diff --git a/examples/java/src/main/java/org/apache/beam/examples/complete/datatokenization/utils/CsvConverters.java b/examples/java/src/main/java/org/apache/beam/examples/complete/datatokenization/utils/CsvConverters.java index 9fead4e1a7fd..d827e4b30cb3 100644 --- a/examples/java/src/main/java/org/apache/beam/examples/complete/datatokenization/utils/CsvConverters.java +++ b/examples/java/src/main/java/org/apache/beam/examples/complete/datatokenization/utils/CsvConverters.java @@ -264,6 +264,8 @@ public static Builder newBuilder() { @Override public PCollectionTuple expand(PCollectionTuple lines) { + PCollectionView headersView = null; + // Convert csv lines into Failsafe elements so that we can recover over multiple transforms. PCollection> lineFailsafeElements = lines @@ -283,14 +285,16 @@ public PCollectionTuple expand(PCollectionTuple lines) { return lineFailsafeElements.apply( "LineToDocumentUsingSchema", - ParDo.of(new FailsafeElementToJsonFn(schema, delimiter(), udfDeadletterTag())) + ParDo.of( + new FailsafeElementToJsonFn( + headersView, schema, delimiter(), udfDeadletterTag())) .withOutputTags(udfOutputTag(), TupleTagList.of(udfDeadletterTag()))); } // Run if using headers - PCollectionView headersView = - lines.get(headerTag()).apply(Sample.any(1)).apply(View.asSingleton()); + headersView = lines.get(headerTag()).apply(Sample.any(1)).apply(View.asSingleton()); + PCollectionView finalHeadersView = headersView; lines .get(headerTag()) .apply( @@ -298,24 +302,23 @@ public PCollectionTuple expand(PCollectionTuple lines) { ParDo.of( new DoFn() { @ProcessElement - public void processElement( - @SideInput("finalHeadersView") String headers, - @Element String element) { - if (!element.equals(headers)) { + public void processElement(ProcessContext c) { + String headers = c.sideInput(finalHeadersView); + if (!c.element().equals(headers)) { LOG.error("Headers do not match, consistency cannot be guaranteed"); throw new RuntimeException( "Headers do not match, consistency cannot be guaranteed"); } } }) - .withSideInput("finalHeadersView", headersView)); + .withSideInputs(finalHeadersView)); return lineFailsafeElements.apply( "LineToDocumentWithHeaders", ParDo.of( - new FailsafeElementToJsonWithHeadersFn( - jsonSchemaPath(), delimiter(), udfDeadletterTag())) - .withSideInput("finalHeadersView", headersView) + new FailsafeElementToJsonFn( + headersView, jsonSchemaPath(), delimiter(), udfDeadletterTag())) + .withSideInputs(headersView) .withOutputTags(udfOutputTag(), TupleTagList.of(udfDeadletterTag()))); } @@ -354,90 +357,45 @@ public static class FailsafeElementToJsonFn @Nullable public final String jsonSchema; public final String delimiter; public final TupleTag> udfDeadletterTag; + @Nullable private final PCollectionView headersView; private Counter successCounter = Metrics.counter(FailsafeElementToJsonFn.class, SUCCESSFUL_TO_JSON_COUNTER); private Counter failedCounter = Metrics.counter(FailsafeElementToJsonFn.class, FAILED_TO_JSON_COUNTER); FailsafeElementToJsonFn( + PCollectionView headersView, String jsonSchema, String delimiter, TupleTag> udfDeadletterTag) { + this.headersView = headersView; this.jsonSchema = jsonSchema; this.delimiter = delimiter; this.udfDeadletterTag = udfDeadletterTag; } @ProcessElement - public void processElement( - @Element FailsafeElement element, - OutputReceiver> receiver, - MultiOutputReceiver multiReceiver) { - List header = null; - List record = Arrays.asList(element.getOriginalPayload().split(this.delimiter)); - - try { - String json = buildJsonString(header, record, this.jsonSchema); - receiver.output(FailsafeElement.of(element.getOriginalPayload(), json)); - successCounter.inc(); - } catch (Exception e) { - failedCounter.inc(); - multiReceiver - .get(this.udfDeadletterTag) - .output( - FailsafeElement.of(element) - .setErrorMessage(e.getMessage()) - .setStacktrace(Throwables.getStackTraceAsString(e))); - } - } - } - - public static class FailsafeElementToJsonWithHeadersFn - extends DoFn, FailsafeElement> { - - @Nullable public final String jsonSchema; - public final String delimiter; - public final TupleTag> udfDeadletterTag; - private Counter successCounter = - Metrics.counter(FailsafeElementToJsonWithHeadersFn.class, SUCCESSFUL_TO_JSON_COUNTER); - private Counter failedCounter = - Metrics.counter(FailsafeElementToJsonWithHeadersFn.class, FAILED_TO_JSON_COUNTER); - - FailsafeElementToJsonWithHeadersFn( - String jsonSchema, - String delimiter, - TupleTag> udfDeadletterTag) { - this.jsonSchema = jsonSchema; - this.delimiter = delimiter; - this.udfDeadletterTag = udfDeadletterTag; - } - - @ProcessElement - public void processElement( - @Element FailsafeElement element, - OutputReceiver> receiver, - MultiOutputReceiver multiReceiver, - @SideInput("finalHeadersView") String headersStr) { + public void processElement(ProcessContext context) { + FailsafeElement element = context.element(); List header = null; - if (headersStr != null) { - header = Arrays.asList(headersStr.split(this.delimiter)); + if (this.headersView != null) { + header = Arrays.asList(context.sideInput(this.headersView).split(this.delimiter)); } List record = Arrays.asList(element.getOriginalPayload().split(this.delimiter)); try { String json = buildJsonString(header, record, this.jsonSchema); - receiver.output(FailsafeElement.of(element.getOriginalPayload(), json)); + context.output(FailsafeElement.of(element.getOriginalPayload(), json)); successCounter.inc(); } catch (Exception e) { failedCounter.inc(); - multiReceiver - .get(this.udfDeadletterTag) - .output( - FailsafeElement.of(element) - .setErrorMessage(e.getMessage()) - .setStacktrace(Throwables.getStackTraceAsString(e))); + context.output( + this.udfDeadletterTag, + FailsafeElement.of(element) + .setErrorMessage(e.getMessage()) + .setStacktrace(Throwables.getStackTraceAsString(e))); } } } @@ -449,9 +407,9 @@ public void processElement( static class LineToFailsafeElementFn extends DoFn> { @ProcessElement - public void processElement( - @Element String message, OutputReceiver> receiver) { - receiver.output(FailsafeElement.of(message, message)); + public void processElement(ProcessContext context) { + String message = context.element(); + context.output(FailsafeElement.of(message, message)); } } @@ -552,12 +510,13 @@ static class GetCsvHeadersFn extends DoFn { } @ProcessElement - public void processElement(@Element ReadableFile file, MultiOutputReceiver outputReceiver) { + public void processElement(ProcessContext context, MultiOutputReceiver outputReceiver) { + ReadableFile f = context.element(); String headers; List records = null; String delimiter = String.valueOf(this.csvFormat.getDelimiter()); try { - String csvFileString = file.readFullyAsUTF8String(); + String csvFileString = f.readFullyAsUTF8String(); StringReader reader = new StringReader(csvFileString); CSVParser parser = CSVParser.parse(reader, this.csvFormat.withFirstRecordAsHeader()); records = diff --git a/examples/java/src/main/java/org/apache/beam/examples/complete/datatokenization/utils/ErrorConverters.java b/examples/java/src/main/java/org/apache/beam/examples/complete/datatokenization/utils/ErrorConverters.java index 99981cb17b84..01377add0858 100644 --- a/examples/java/src/main/java/org/apache/beam/examples/complete/datatokenization/utils/ErrorConverters.java +++ b/examples/java/src/main/java/org/apache/beam/examples/complete/datatokenization/utils/ErrorConverters.java @@ -29,9 +29,6 @@ import org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO.Write.WriteDisposition; import org.apache.beam.sdk.io.gcp.bigquery.WriteResult; import org.apache.beam.sdk.transforms.DoFn; -import org.apache.beam.sdk.transforms.DoFn.Element; -import org.apache.beam.sdk.transforms.DoFn.OutputReceiver; -import org.apache.beam.sdk.transforms.DoFn.Timestamp; import org.apache.beam.sdk.transforms.MapElements; import org.apache.beam.sdk.transforms.PTransform; import org.apache.beam.sdk.transforms.ParDo; @@ -46,7 +43,6 @@ import org.checkerframework.checker.nullness.qual.Nullable; import org.joda.time.DateTimeZone; import org.joda.time.Duration; -import org.joda.time.Instant; import org.joda.time.format.DateTimeFormat; import org.joda.time.format.DateTimeFormatter; @@ -127,17 +123,16 @@ public FailedStringToCsvRowFn() { } @ProcessElement - public void processElement( - @Element FailsafeElement failsafeElement, - @Timestamp Instant timestamp, - OutputReceiver receiver) { + public void processElement(ProcessContext context) { + FailsafeElement failsafeElement = context.element(); ArrayList outputRow = new ArrayList<>(); final String message = failsafeElement.getOriginalPayload(); // Format the timestamp for insertion - String timestampStr = TIMESTAMP_FORMATTER.print(timestamp.toDateTime(DateTimeZone.UTC)); + String timestamp = + TIMESTAMP_FORMATTER.print(context.timestamp().toDateTime(DateTimeZone.UTC)); - outputRow.add(timestampStr); + outputRow.add(timestamp); outputRow.add(MoreObjects.firstNonNull(failsafeElement.getErrorMessage(), "")); // Only set the payload if it's populated on the message. @@ -145,7 +140,7 @@ public void processElement( outputRow.add(message); } - receiver.output(String.join(csvDelimiter, outputRow)); + context.output(String.join(csvDelimiter, outputRow)); } } @@ -203,20 +198,19 @@ public static class FailedStringToTableRowFn DateTimeFormat.forPattern("yyyy-MM-dd HH:mm:ss.SSSSSS"); @ProcessElement - public void processElement( - @Timestamp Instant timestamp, - @Element FailsafeElement failsafeElement, - OutputReceiver receiver) { + public void processElement(ProcessContext context) { + FailsafeElement failsafeElement = context.element(); final String message = failsafeElement.getOriginalPayload(); // Format the timestamp for insertion - String timestampStr = TIMESTAMP_FORMATTER.print(timestamp.toDateTime(DateTimeZone.UTC)); + String timestamp = + TIMESTAMP_FORMATTER.print(context.timestamp().toDateTime(DateTimeZone.UTC)); // Build the table row @SuppressWarnings("nullness") // TableRow.set not annotated but does accept nulls final TableRow failedRow = new TableRow() - .set("timestamp", timestampStr) + .set("timestamp", timestamp) .set("errorMessage", failsafeElement.getErrorMessage()) .set("stacktrace", failsafeElement.getStacktrace()); @@ -227,7 +221,7 @@ public void processElement( .set("payloadBytes", message.getBytes(StandardCharsets.UTF_8)); } - receiver.output(failedRow); + context.output(failedRow); } } diff --git a/examples/java/src/main/java/org/apache/beam/examples/complete/game/GameStats.java b/examples/java/src/main/java/org/apache/beam/examples/complete/game/GameStats.java index 0018201f3686..a3ed04bb1c48 100644 --- a/examples/java/src/main/java/org/apache/beam/examples/complete/game/GameStats.java +++ b/examples/java/src/main/java/org/apache/beam/examples/complete/game/GameStats.java @@ -34,9 +34,6 @@ import org.apache.beam.sdk.options.PipelineOptionsFactory; import org.apache.beam.sdk.transforms.Combine; import org.apache.beam.sdk.transforms.DoFn; -import org.apache.beam.sdk.transforms.DoFn.Element; -import org.apache.beam.sdk.transforms.DoFn.OutputReceiver; -import org.apache.beam.sdk.transforms.DoFn.SideInput; import org.apache.beam.sdk.transforms.MapElements; import org.apache.beam.sdk.transforms.Mean; import org.apache.beam.sdk.transforms.PTransform; @@ -129,23 +126,21 @@ public PCollection> expand(PCollection> Metrics.counter("main", "SpammerUsers"); @ProcessElement - public void processElement( - @SideInput("globalMeanScore") Double gmc, - @Element KV element, - OutputReceiver> receiver) { - Integer score = element.getValue(); + public void processElement(ProcessContext c) { + Integer score = c.element().getValue(); + Double gmc = c.sideInput(globalMeanScore); if (score > (gmc * SCORE_WEIGHT)) { LOG.info( "user {} spammer score {} with mean {}", - element.getKey(), + c.element().getKey(), score, gmc); numSpammerUsers.inc(); - receiver.output(element); + c.output(c.element()); } } }) - .withSideInput("globalMeanScore", globalMeanScore)); + .withSideInputs(globalMeanScore)); return filtered; } } @@ -154,10 +149,10 @@ public void processElement( /** Calculate and output an element's session duration. */ private static class UserSessionInfoFn extends DoFn, Integer> { @ProcessElement - public void processElement(BoundedWindow window, OutputReceiver receiver) { + public void processElement(ProcessContext c, BoundedWindow window) { IntervalWindow w = (IntervalWindow) window; int duration = new Duration(w.start(), w.end()).toPeriod().toStandardMinutes().getMinutes(); - receiver.output(duration); + c.output(duration); } } @@ -197,21 +192,22 @@ public interface Options extends LeaderBoard.Options { configureWindowedWrite() { Map>> tableConfigure = new HashMap<>(); tableConfigure.put( - "team", new WriteToBigQuery.FieldInfo<>("STRING", (e, w, t, p) -> e.getKey())); + "team", new WriteToBigQuery.FieldInfo<>("STRING", (c, w) -> c.element().getKey())); tableConfigure.put( - "total_score", new WriteToBigQuery.FieldInfo<>("INTEGER", (e, w, t, p) -> e.getValue())); + "total_score", + new WriteToBigQuery.FieldInfo<>("INTEGER", (c, w) -> c.element().getValue())); tableConfigure.put( "window_start", new WriteToBigQuery.FieldInfo<>( "STRING", - (e, w, t, p) -> { + (c, w) -> { IntervalWindow window = (IntervalWindow) w; return GameConstants.DATE_TIME_FORMATTER.print(window.start()); })); tableConfigure.put( "processing_time", new WriteToBigQuery.FieldInfo<>( - "STRING", (e, w, t, p) -> GameConstants.DATE_TIME_FORMATTER.print(Instant.now()))); + "STRING", (c, w) -> GameConstants.DATE_TIME_FORMATTER.print(Instant.now()))); return tableConfigure; } @@ -226,12 +222,12 @@ protected static Map> configureSession "window_start", new WriteToBigQuery.FieldInfo<>( "STRING", - (e, w, t, p) -> { + (c, w) -> { IntervalWindow window = (IntervalWindow) w; return GameConstants.DATE_TIME_FORMATTER.print(window.start()); })); tableConfigure.put( - "mean_duration", new WriteToBigQuery.FieldInfo<>("FLOAT", (e, w, t, p) -> e)); + "mean_duration", new WriteToBigQuery.FieldInfo<>("FLOAT", (c, w) -> c.element())); return tableConfigure; } @@ -292,17 +288,14 @@ public static void main(String[] args) throws Exception { ParDo.of( new DoFn() { @ProcessElement - public void processElement( - @SideInput("spammersView") Map spammers, - @Element GameActionInfo element, - OutputReceiver receiver) { + public void processElement(ProcessContext c) { // If the user is not in the spammers Map, output the data element. - if (spammers.get(element.getUser().trim()) == null) { - receiver.output(element); + if (c.sideInput(spammersView).get(c.element().getUser().trim()) == null) { + c.output(c.element()); } } }) - .withSideInput("spammersView", spammersView)) + .withSideInputs(spammersView)) // Extract and sum teamname/score pairs from the event data. .apply("ExtractTeamScore", new ExtractAndSumScore("team")) // [END DocInclude_FilterAndCalc] diff --git a/examples/java/src/main/java/org/apache/beam/examples/complete/game/HourlyTeamScore.java b/examples/java/src/main/java/org/apache/beam/examples/complete/game/HourlyTeamScore.java index a94f699282a8..c57d9ba6b8c8 100644 --- a/examples/java/src/main/java/org/apache/beam/examples/complete/game/HourlyTeamScore.java +++ b/examples/java/src/main/java/org/apache/beam/examples/complete/game/HourlyTeamScore.java @@ -112,11 +112,11 @@ public interface Options extends UserScore.Options { */ protected static Map>> configureOutput() { Map>> config = new HashMap<>(); - config.put("team", (e, w, t, p) -> e.getKey()); - config.put("total_score", (e, w, t, p) -> e.getValue()); + config.put("team", (c, w) -> c.element().getKey()); + config.put("total_score", (c, w) -> c.element().getValue()); config.put( "window_start", - (e, w, t, p) -> { + (c, w) -> { IntervalWindow window = (IntervalWindow) w; return GameConstants.DATE_TIME_FORMATTER.print(window.start()); }); diff --git a/examples/java/src/main/java/org/apache/beam/examples/complete/game/LeaderBoard.java b/examples/java/src/main/java/org/apache/beam/examples/complete/game/LeaderBoard.java index 3750b33aa791..832c0ad79e76 100644 --- a/examples/java/src/main/java/org/apache/beam/examples/complete/game/LeaderBoard.java +++ b/examples/java/src/main/java/org/apache/beam/examples/complete/game/LeaderBoard.java @@ -135,24 +135,25 @@ public interface Options extends ExampleOptions, StreamingOptions { Map>> tableConfigure = new HashMap<>(); tableConfigure.put( - "team", new WriteToBigQuery.FieldInfo<>("STRING", (e, w, t, p) -> e.getKey())); + "team", new WriteToBigQuery.FieldInfo<>("STRING", (c, w) -> c.element().getKey())); tableConfigure.put( - "total_score", new WriteToBigQuery.FieldInfo<>("INTEGER", (e, w, t, p) -> e.getValue())); + "total_score", + new WriteToBigQuery.FieldInfo<>("INTEGER", (c, w) -> c.element().getValue())); tableConfigure.put( "window_start", new WriteToBigQuery.FieldInfo<>( "STRING", - (e, w, t, p) -> { + (c, w) -> { IntervalWindow window = (IntervalWindow) w; return GameConstants.DATE_TIME_FORMATTER.print(window.start()); })); tableConfigure.put( "processing_time", new WriteToBigQuery.FieldInfo<>( - "STRING", (e, w, t, p) -> GameConstants.DATE_TIME_FORMATTER.print(Instant.now()))); + "STRING", (c, w) -> GameConstants.DATE_TIME_FORMATTER.print(Instant.now()))); tableConfigure.put( "timing", - new WriteToBigQuery.FieldInfo<>("STRING", (e, w, t, p) -> p.getTiming().toString())); + new WriteToBigQuery.FieldInfo<>("STRING", (c, w) -> c.pane().getTiming().toString())); return tableConfigure; } @@ -164,9 +165,10 @@ public interface Options extends ExampleOptions, StreamingOptions { configureBigQueryWrite() { Map>> tableConfigure = new HashMap<>(); tableConfigure.put( - "user", new WriteToBigQuery.FieldInfo<>("STRING", (e, w, t, p) -> e.getKey())); + "user", new WriteToBigQuery.FieldInfo<>("STRING", (c, w) -> c.element().getKey())); tableConfigure.put( - "total_score", new WriteToBigQuery.FieldInfo<>("INTEGER", (e, w, t, p) -> e.getValue())); + "total_score", + new WriteToBigQuery.FieldInfo<>("INTEGER", (c, w) -> c.element().getValue())); return tableConfigure; } @@ -182,7 +184,7 @@ public interface Options extends ExampleOptions, StreamingOptions { tableConfigure.put( "processing_time", new WriteToBigQuery.FieldInfo<>( - "STRING", (e, w, t, p) -> GameConstants.DATE_TIME_FORMATTER.print(Instant.now()))); + "STRING", (c, w) -> GameConstants.DATE_TIME_FORMATTER.print(Instant.now()))); return tableConfigure; } diff --git a/examples/java/src/main/java/org/apache/beam/examples/complete/game/StatefulTeamScore.java b/examples/java/src/main/java/org/apache/beam/examples/complete/game/StatefulTeamScore.java index 3caa1e619526..b28db261ab0e 100644 --- a/examples/java/src/main/java/org/apache/beam/examples/complete/game/StatefulTeamScore.java +++ b/examples/java/src/main/java/org/apache/beam/examples/complete/game/StatefulTeamScore.java @@ -37,8 +37,6 @@ import org.apache.beam.sdk.state.StateSpecs; import org.apache.beam.sdk.state.ValueState; import org.apache.beam.sdk.transforms.DoFn; -import org.apache.beam.sdk.transforms.DoFn.Element; -import org.apache.beam.sdk.transforms.DoFn.OutputReceiver; import org.apache.beam.sdk.transforms.MapElements; import org.apache.beam.sdk.transforms.ParDo; import org.apache.beam.sdk.values.KV; @@ -103,12 +101,12 @@ public interface Options extends LeaderBoard.Options { private static Map>> configureCompleteWindowedTableWrite() { Map>> tableConfigure = new HashMap<>(); - tableConfigure.put("team", new FieldInfo<>("STRING", (e, w, t, p) -> e.getKey())); - tableConfigure.put("total_score", new FieldInfo<>("INTEGER", (e, w, t, p) -> e.getValue())); + tableConfigure.put("team", new FieldInfo<>("STRING", (c, w) -> c.element().getKey())); + tableConfigure.put("total_score", new FieldInfo<>("INTEGER", (c, w) -> c.element().getValue())); tableConfigure.put( "processing_time", new FieldInfo<>( - "STRING", (e, w, t, p) -> GameConstants.DATE_TIME_FORMATTER.print(Instant.now()))); + "STRING", (c, w) -> GameConstants.DATE_TIME_FORMATTER.print(Instant.now()))); return tableConfigure; } @@ -205,11 +203,9 @@ public UpdateTeamScoreFn(int thresholdScore) { */ @ProcessElement public void processElement( - @Element KV element, - @StateId(TOTAL_SCORE) ValueState totalScore, - OutputReceiver> receiver) { - String teamName = element.getKey(); - GameActionInfo gInfo = element.getValue(); + ProcessContext c, @StateId(TOTAL_SCORE) ValueState totalScore) { + String teamName = c.element().getKey(); + GameActionInfo gInfo = c.element().getValue(); // ValueState cells do not contain a default value. If the state is possibly not written, make // sure to check for null on read. @@ -222,7 +218,7 @@ public void processElement( // the new total is 2002, and the threshold is 1000, 1999 / 1000 = 1, 2002 / 1000 = 2. // Therefore, this team passed the threshold. if (oldTotalScore / this.thresholdScore < totalScore.read() / this.thresholdScore) { - receiver.output(KV.of(teamName, totalScore.read())); + c.output(KV.of(teamName, totalScore.read())); } } } diff --git a/examples/java/src/main/java/org/apache/beam/examples/complete/game/UserScore.java b/examples/java/src/main/java/org/apache/beam/examples/complete/game/UserScore.java index 054ce7a52935..b30b4665d265 100644 --- a/examples/java/src/main/java/org/apache/beam/examples/complete/game/UserScore.java +++ b/examples/java/src/main/java/org/apache/beam/examples/complete/game/UserScore.java @@ -34,8 +34,6 @@ import org.apache.beam.sdk.options.PipelineOptionsFactory; import org.apache.beam.sdk.options.Validation; import org.apache.beam.sdk.transforms.DoFn; -import org.apache.beam.sdk.transforms.DoFn.Element; -import org.apache.beam.sdk.transforms.DoFn.OutputReceiver; import org.apache.beam.sdk.transforms.MapElements; import org.apache.beam.sdk.transforms.PTransform; import org.apache.beam.sdk.transforms.ParDo; @@ -163,18 +161,18 @@ static class ParseEventFn extends DoFn { private final Counter numParseErrors = Metrics.counter("main", "ParseErrors"); @ProcessElement - public void processElement(@Element String element, OutputReceiver receiver) { - String[] components = element.split(",", -1); + public void processElement(ProcessContext c) { + String[] components = c.element().split(",", -1); try { String user = components[0].trim(); String team = components[1].trim(); Integer score = Integer.parseInt(components[2].trim()); Long timestamp = Long.parseLong(components[3].trim()); GameActionInfo gInfo = new GameActionInfo(user, team, score, timestamp); - receiver.output(gInfo); + c.output(gInfo); } catch (ArrayIndexOutOfBoundsException | NumberFormatException e) { numParseErrors.inc(); - LOG.info("Parse error on {}", element, e); + LOG.info("Parse error on {}", c.element(), e); } } } @@ -234,8 +232,8 @@ public interface Options extends PipelineOptions { */ protected static Map>> configureOutput() { Map>> config = new HashMap<>(); - config.put("user", (e, w, t, p) -> e.getKey()); - config.put("total_score", (e, w, t, p) -> e.getValue()); + config.put("user", (c, w) -> c.element().getKey()); + config.put("total_score", (c, w) -> c.element().getValue()); return config; } diff --git a/examples/java/src/main/java/org/apache/beam/examples/complete/game/utils/WriteToBigQuery.java b/examples/java/src/main/java/org/apache/beam/examples/complete/game/utils/WriteToBigQuery.java index 5486025083c9..eef4bc932682 100644 --- a/examples/java/src/main/java/org/apache/beam/examples/complete/game/utils/WriteToBigQuery.java +++ b/examples/java/src/main/java/org/apache/beam/examples/complete/game/utils/WriteToBigQuery.java @@ -30,16 +30,11 @@ import org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO.Write.WriteDisposition; import org.apache.beam.sdk.io.gcp.bigquery.InsertRetryPolicy; import org.apache.beam.sdk.transforms.DoFn; -import org.apache.beam.sdk.transforms.DoFn.Element; -import org.apache.beam.sdk.transforms.DoFn.OutputReceiver; -import org.apache.beam.sdk.transforms.DoFn.Timestamp; import org.apache.beam.sdk.transforms.PTransform; import org.apache.beam.sdk.transforms.ParDo; import org.apache.beam.sdk.transforms.windowing.BoundedWindow; -import org.apache.beam.sdk.transforms.windowing.PaneInfo; import org.apache.beam.sdk.values.PCollection; import org.apache.beam.sdk.values.PDone; -import org.joda.time.Instant; /** * Generate, format, and write BigQuery table row information. Use provided information about the @@ -69,11 +64,11 @@ public WriteToBigQuery( } /** - * A {@link Serializable} function from an element and {@link BoundedWindow} to the value for that - * field. + * A {@link Serializable} function from a {@link DoFn.ProcessContext} and {@link BoundedWindow} to + * the value for that field. */ public interface FieldFn extends Serializable { - Object apply(InputT element, BoundedWindow window, Instant timestamp, PaneInfo pane); + Object apply(DoFn.ProcessContext context, BoundedWindow window); } /** Define a class to hold information about output table field definitions. */ @@ -101,21 +96,16 @@ FieldFn getFieldFn() { protected class BuildRowFn extends DoFn { @ProcessElement - public void processElement( - @Element InputT element, - @Timestamp Instant timestamp, - PaneInfo pane, - BoundedWindow window, - OutputReceiver receiver) { + public void processElement(ProcessContext c, BoundedWindow window) { TableRow row = new TableRow(); for (Map.Entry> entry : fieldInfo.entrySet()) { String key = entry.getKey(); FieldInfo fcnInfo = entry.getValue(); FieldFn fcn = fcnInfo.getFieldFn(); - row.set(key, fcn.apply(element, window, timestamp, pane)); + row.set(key, fcn.apply(c, window)); } - receiver.output(row); + c.output(row); } } diff --git a/examples/java/src/main/java/org/apache/beam/examples/complete/game/utils/WriteToText.java b/examples/java/src/main/java/org/apache/beam/examples/complete/game/utils/WriteToText.java index 35471163c036..330769e0c79e 100644 --- a/examples/java/src/main/java/org/apache/beam/examples/complete/game/utils/WriteToText.java +++ b/examples/java/src/main/java/org/apache/beam/examples/complete/game/utils/WriteToText.java @@ -32,9 +32,6 @@ import org.apache.beam.sdk.io.fs.ResolveOptions.StandardResolveOptions; import org.apache.beam.sdk.io.fs.ResourceId; import org.apache.beam.sdk.transforms.DoFn; -import org.apache.beam.sdk.transforms.DoFn.Element; -import org.apache.beam.sdk.transforms.DoFn.OutputReceiver; -import org.apache.beam.sdk.transforms.DoFn.Timestamp; import org.apache.beam.sdk.transforms.PTransform; import org.apache.beam.sdk.transforms.ParDo; import org.apache.beam.sdk.transforms.windowing.BoundedWindow; @@ -73,32 +70,26 @@ public WriteToText( } /** - * A {@link Serializable} function from an element and {@link BoundedWindow} to the value for that - * field. + * A {@link Serializable} function from a {@link DoFn.ProcessContext} and {@link BoundedWindow} to + * the value for that field. */ public interface FieldFn extends Serializable { - Object apply( - InputT element, BoundedWindow window, org.joda.time.Instant timestamp, PaneInfo pane); + Object apply(DoFn.ProcessContext context, BoundedWindow window); } /** Convert each key/score pair into a row as specified by fieldFn. */ protected class BuildRowFn extends DoFn { @ProcessElement - public void processElement( - @Element InputT element, - @Timestamp org.joda.time.Instant timestamp, - PaneInfo pane, - BoundedWindow window, - OutputReceiver receiver) { + public void processElement(ProcessContext c, BoundedWindow window) { List fields = new ArrayList<>(); for (Map.Entry> entry : fieldFn.entrySet()) { String key = entry.getKey(); FieldFn fcn = entry.getValue(); - fields.add(key + ": " + fcn.apply(element, window, timestamp, pane)); + fields.add(key + ": " + fcn.apply(c, window)); } String result = fields.stream().collect(Collectors.joining(", ")); - receiver.output(result); + c.output(result); } } diff --git a/examples/java/src/main/java/org/apache/beam/examples/complete/game/utils/WriteWindowedToBigQuery.java b/examples/java/src/main/java/org/apache/beam/examples/complete/game/utils/WriteWindowedToBigQuery.java index 77a59f2c3b6e..36fa18a34e0d 100644 --- a/examples/java/src/main/java/org/apache/beam/examples/complete/game/utils/WriteWindowedToBigQuery.java +++ b/examples/java/src/main/java/org/apache/beam/examples/complete/game/utils/WriteWindowedToBigQuery.java @@ -24,12 +24,8 @@ import org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO.Write.WriteDisposition; import org.apache.beam.sdk.io.gcp.bigquery.InsertRetryPolicy; import org.apache.beam.sdk.transforms.DoFn; -import org.apache.beam.sdk.transforms.DoFn.Element; -import org.apache.beam.sdk.transforms.DoFn.OutputReceiver; -import org.apache.beam.sdk.transforms.DoFn.Timestamp; import org.apache.beam.sdk.transforms.ParDo; import org.apache.beam.sdk.transforms.windowing.BoundedWindow; -import org.apache.beam.sdk.transforms.windowing.PaneInfo; import org.apache.beam.sdk.values.PCollection; import org.apache.beam.sdk.values.PDone; @@ -48,20 +44,15 @@ public WriteWindowedToBigQuery( /** Convert each key/score pair into a BigQuery TableRow. */ protected class BuildRowFn extends DoFn { @ProcessElement - public void processElement( - @Element T element, - @Timestamp org.joda.time.Instant timestamp, - PaneInfo pane, - BoundedWindow window, - OutputReceiver receiver) { + public void processElement(ProcessContext c, BoundedWindow window) { TableRow row = new TableRow(); for (Map.Entry> entry : fieldInfo.entrySet()) { String key = entry.getKey(); FieldInfo fcnInfo = entry.getValue(); - row.set(key, fcnInfo.getFieldFn().apply(element, window, timestamp, pane)); + row.set(key, fcnInfo.getFieldFn().apply(c, window)); } - receiver.output(row); + c.output(row); } } diff --git a/examples/java/src/main/java/org/apache/beam/examples/cookbook/BigQueryStreamingTornadoes.java b/examples/java/src/main/java/org/apache/beam/examples/cookbook/BigQueryStreamingTornadoes.java index d662cde1f000..3b2653b7601e 100644 --- a/examples/java/src/main/java/org/apache/beam/examples/cookbook/BigQueryStreamingTornadoes.java +++ b/examples/java/src/main/java/org/apache/beam/examples/cookbook/BigQueryStreamingTornadoes.java @@ -32,9 +32,6 @@ import org.apache.beam.sdk.options.Validation; import org.apache.beam.sdk.transforms.Count; import org.apache.beam.sdk.transforms.DoFn; -import org.apache.beam.sdk.transforms.DoFn.Element; -import org.apache.beam.sdk.transforms.DoFn.OutputReceiver; -import org.apache.beam.sdk.transforms.DoFn.Timestamp; import org.apache.beam.sdk.transforms.MapElements; import org.apache.beam.sdk.transforms.PTransform; import org.apache.beam.sdk.transforms.ParDo; @@ -88,9 +85,10 @@ public class BigQueryStreamingTornadoes { */ static class ExtractTornadoesFn extends DoFn { @ProcessElement - public void processElement(@Element TableRow row, OutputReceiver receiver) { + public void processElement(ProcessContext c) { + TableRow row = c.element(); if (Boolean.TRUE.equals(row.get("tornado"))) { - receiver.output(Integer.parseInt((String) row.get("month"))); + c.output(Integer.parseInt((String) row.get("month"))); } } } @@ -101,16 +99,13 @@ public void processElement(@Element TableRow row, OutputReceiver receiv */ static class FormatCountsFn extends DoFn, TableRow> { @ProcessElement - public void processElement( - @Element KV element, - @Timestamp Instant timestamp, - OutputReceiver receiver) { + public void processElement(ProcessContext c) { TableRow row = new TableRow() - .set("ts", timestamp.toString()) - .set("month", element.getKey()) - .set("tornado_count", element.getValue()); - receiver.output(row); + .set("ts", c.timestamp().toString()) + .set("month", c.element().getKey()) + .set("tornado_count", c.element().getValue()); + c.output(row); } } diff --git a/examples/java/src/main/java/org/apache/beam/examples/cookbook/BigQueryTornadoes.java b/examples/java/src/main/java/org/apache/beam/examples/cookbook/BigQueryTornadoes.java index 67e0661a1a42..43d720e35268 100644 --- a/examples/java/src/main/java/org/apache/beam/examples/cookbook/BigQueryTornadoes.java +++ b/examples/java/src/main/java/org/apache/beam/examples/cookbook/BigQueryTornadoes.java @@ -32,8 +32,6 @@ import org.apache.beam.sdk.options.Validation; import org.apache.beam.sdk.transforms.Count; import org.apache.beam.sdk.transforms.DoFn; -import org.apache.beam.sdk.transforms.DoFn.Element; -import org.apache.beam.sdk.transforms.DoFn.OutputReceiver; import org.apache.beam.sdk.transforms.PTransform; import org.apache.beam.sdk.transforms.ParDo; import org.apache.beam.sdk.values.KV; @@ -81,9 +79,10 @@ public class BigQueryTornadoes { */ static class ExtractTornadoesFn extends DoFn { @ProcessElement - public void processElement(@Element TableRow row, OutputReceiver receiver) { + public void processElement(ProcessContext c) { + TableRow row = c.element(); if ((Boolean) row.get("tornado")) { - receiver.output(Integer.parseInt((String) row.get("month"))); + c.output(Integer.parseInt((String) row.get("month"))); } } } @@ -94,11 +93,12 @@ public void processElement(@Element TableRow row, OutputReceiver receiv */ static class FormatCountsFn extends DoFn, TableRow> { @ProcessElement - public void processElement( - @Element KV element, OutputReceiver receiver) { + public void processElement(ProcessContext c) { TableRow row = - new TableRow().set("month", element.getKey()).set("tornado_count", element.getValue()); - receiver.output(row); + new TableRow() + .set("month", c.element().getKey()) + .set("tornado_count", c.element().getValue()); + c.output(row); } } diff --git a/examples/java/src/main/java/org/apache/beam/examples/cookbook/CombinePerKeyExamples.java b/examples/java/src/main/java/org/apache/beam/examples/cookbook/CombinePerKeyExamples.java index 5f5b5d06b1f8..2a581d769dc7 100644 --- a/examples/java/src/main/java/org/apache/beam/examples/cookbook/CombinePerKeyExamples.java +++ b/examples/java/src/main/java/org/apache/beam/examples/cookbook/CombinePerKeyExamples.java @@ -33,8 +33,6 @@ import org.apache.beam.sdk.options.Validation; import org.apache.beam.sdk.transforms.Combine; import org.apache.beam.sdk.transforms.DoFn; -import org.apache.beam.sdk.transforms.DoFn.Element; -import org.apache.beam.sdk.transforms.DoFn.OutputReceiver; import org.apache.beam.sdk.transforms.PTransform; import org.apache.beam.sdk.transforms.ParDo; import org.apache.beam.sdk.transforms.SerializableFunction; @@ -80,11 +78,12 @@ static class ExtractLargeWordsFn extends DoFn> { private final Counter smallerWords = Metrics.counter(ExtractLargeWordsFn.class, "smallerWords"); @ProcessElement - public void processElement(@Element TableRow row, OutputReceiver> receiver) { + public void processElement(ProcessContext c) { + TableRow row = c.element(); String playName = (String) row.get("corpus"); String word = (String) row.get("word"); if (word.length() >= MIN_WORD_LENGTH) { - receiver.output(KV.of(word, playName)); + c.output(KV.of(word, playName)); } else { // Track how many smaller words we're not including. This information will be // visible in the Monitoring UI. @@ -99,11 +98,10 @@ public void processElement(@Element TableRow row, OutputReceiver, TableRow> { @ProcessElement - public void processElement( - @Element KV element, OutputReceiver receiver) { + public void processElement(ProcessContext c) { TableRow row = - new TableRow().set("word", element.getKey()).set("all_plays", element.getValue()); - receiver.output(row); + new TableRow().set("word", c.element().getKey()).set("all_plays", c.element().getValue()); + c.output(row); } } diff --git a/examples/java/src/main/java/org/apache/beam/examples/cookbook/FilterExamples.java b/examples/java/src/main/java/org/apache/beam/examples/cookbook/FilterExamples.java index 26a6659a86b8..9187bb83d7da 100644 --- a/examples/java/src/main/java/org/apache/beam/examples/cookbook/FilterExamples.java +++ b/examples/java/src/main/java/org/apache/beam/examples/cookbook/FilterExamples.java @@ -31,9 +31,6 @@ import org.apache.beam.sdk.options.PipelineOptionsFactory; import org.apache.beam.sdk.options.Validation; import org.apache.beam.sdk.transforms.DoFn; -import org.apache.beam.sdk.transforms.DoFn.Element; -import org.apache.beam.sdk.transforms.DoFn.OutputReceiver; -import org.apache.beam.sdk.transforms.DoFn.SideInput; import org.apache.beam.sdk.transforms.Mean; import org.apache.beam.sdk.transforms.PTransform; import org.apache.beam.sdk.transforms.ParDo; @@ -90,7 +87,8 @@ public class FilterExamples { */ static class ProjectionFn extends DoFn { @ProcessElement - public void processElement(@Element TableRow row, OutputReceiver receiver) { + public void processElement(ProcessContext c) { + TableRow row = c.element(); // Grab year, month, day, mean_temp from the row Integer year = Integer.parseInt((String) row.get("year")); Integer month = Integer.parseInt((String) row.get("month")); @@ -103,7 +101,7 @@ public void processElement(@Element TableRow row, OutputReceiver recei .set("month", month) .set("day", day) .set("mean_temp", meanTemp); - receiver.output(outRow); + c.output(outRow); } } @@ -121,11 +119,12 @@ public FilterSingleMonthDataFn(Integer monthFilter) { } @ProcessElement - public void processElement(@Element TableRow row, OutputReceiver receiver) { + public void processElement(ProcessContext c) { + TableRow row = c.element(); Integer month; month = (Integer) row.get("month"); if (month.equals(this.monthFilter)) { - receiver.output(row); + c.output(row); } } } @@ -136,9 +135,10 @@ public void processElement(@Element TableRow row, OutputReceiver recei */ static class ExtractTempFn extends DoFn { @ProcessElement - public void processElement(@Element TableRow row, OutputReceiver receiver) { + public void processElement(ProcessContext c) { + TableRow row = c.element(); Double meanTemp = Double.parseDouble(row.get("mean_temp").toString()); - receiver.output(meanTemp); + c.output(meanTemp); } } @@ -178,17 +178,16 @@ public PCollection expand(PCollection rows) { ParDo.of( new DoFn() { @ProcessElement - public void processElement( - @SideInput("globalMeanTemp") Double gTemp, - @Element TableRow element, - OutputReceiver receiver) { - Double meanTemp = Double.parseDouble(element.get("mean_temp").toString()); + public void processElement(ProcessContext c) { + Double meanTemp = + Double.parseDouble(c.element().get("mean_temp").toString()); + Double gTemp = c.sideInput(globalMeanTemp); if (meanTemp < gTemp) { - receiver.output(element); + c.output(c.element()); } } }) - .withSideInput("globalMeanTemp", globalMeanTemp)); + .withSideInputs(globalMeanTemp)); return filteredRows; } diff --git a/examples/java/src/main/java/org/apache/beam/examples/cookbook/JoinExamples.java b/examples/java/src/main/java/org/apache/beam/examples/cookbook/JoinExamples.java index e6f8573705a2..f78df0c09461 100644 --- a/examples/java/src/main/java/org/apache/beam/examples/cookbook/JoinExamples.java +++ b/examples/java/src/main/java/org/apache/beam/examples/cookbook/JoinExamples.java @@ -26,8 +26,6 @@ import org.apache.beam.sdk.options.PipelineOptionsFactory; import org.apache.beam.sdk.options.Validation; import org.apache.beam.sdk.transforms.DoFn; -import org.apache.beam.sdk.transforms.DoFn.Element; -import org.apache.beam.sdk.transforms.DoFn.OutputReceiver; import org.apache.beam.sdk.transforms.ParDo; import org.apache.beam.sdk.transforms.join.CoGbkResult; import org.apache.beam.sdk.transforms.join.CoGroupByKey; @@ -92,14 +90,13 @@ static PCollection joinEvents( ParDo.of( new DoFn, KV>() { @ProcessElement - public void processElement( - @Element KV element, - OutputReceiver> receiver) { - String countryCode = element.getKey(); - String countryName = element.getValue().getOnly(countryInfoTag); - for (String eventInfo : element.getValue().getAll(eventInfoTag)) { + public void processElement(ProcessContext c) { + KV e = c.element(); + String countryCode = e.getKey(); + String countryName = e.getValue().getOnly(countryInfoTag); + for (String eventInfo : c.element().getValue().getAll(eventInfoTag)) { // Generate a string that combines information from both collection values - receiver.output( + c.output( KV.of( countryCode, "Country name: " + countryName + ", Event info: " + eventInfo)); @@ -114,11 +111,10 @@ public void processElement( ParDo.of( new DoFn, String>() { @ProcessElement - public void processElement( - @Element KV element, OutputReceiver receiver) { + public void processElement(ProcessContext c) { String outputstring = - "Country code: " + element.getKey() + ", " + element.getValue(); - receiver.output(outputstring); + "Country code: " + c.element().getKey() + ", " + c.element().getValue(); + c.output(outputstring); } })); return formattedResults; @@ -130,13 +126,14 @@ public void processElement( */ static class ExtractEventDataFn extends DoFn> { @ProcessElement - public void processElement(@Element TableRow row, OutputReceiver> receiver) { + public void processElement(ProcessContext c) { + TableRow row = c.element(); String countryCode = (String) row.get("ActionGeo_CountryCode"); String sqlDate = (String) row.get("SQLDATE"); String actor1Name = (String) row.get("Actor1Name"); String sourceUrl = (String) row.get("SOURCEURL"); String eventInfo = "Date: " + sqlDate + ", Actor1: " + actor1Name + ", url: " + sourceUrl; - receiver.output(KV.of(countryCode, eventInfo)); + c.output(KV.of(countryCode, eventInfo)); } } @@ -146,10 +143,11 @@ public void processElement(@Element TableRow row, OutputReceiver> { @ProcessElement - public void processElement(@Element TableRow row, OutputReceiver> receiver) { + public void processElement(ProcessContext c) { + TableRow row = c.element(); String countryCode = (String) row.get("FIPSCC"); String countryName = (String) row.get("HumanName"); - receiver.output(KV.of(countryCode, countryName)); + c.output(KV.of(countryCode, countryName)); } } diff --git a/examples/java/src/main/java/org/apache/beam/examples/cookbook/MaxPerKeyExamples.java b/examples/java/src/main/java/org/apache/beam/examples/cookbook/MaxPerKeyExamples.java index 85df56a58258..8760d562d040 100644 --- a/examples/java/src/main/java/org/apache/beam/examples/cookbook/MaxPerKeyExamples.java +++ b/examples/java/src/main/java/org/apache/beam/examples/cookbook/MaxPerKeyExamples.java @@ -30,8 +30,6 @@ import org.apache.beam.sdk.options.PipelineOptionsFactory; import org.apache.beam.sdk.options.Validation; import org.apache.beam.sdk.transforms.DoFn; -import org.apache.beam.sdk.transforms.DoFn.Element; -import org.apache.beam.sdk.transforms.DoFn.OutputReceiver; import org.apache.beam.sdk.transforms.Max; import org.apache.beam.sdk.transforms.PTransform; import org.apache.beam.sdk.transforms.ParDo; @@ -75,22 +73,23 @@ public class MaxPerKeyExamples { */ static class ExtractTempFn extends DoFn> { @ProcessElement - public void processElement( - @Element TableRow row, OutputReceiver> receiver) { + public void processElement(ProcessContext c) { + TableRow row = c.element(); Integer month = Integer.parseInt((String) row.get("month")); Double meanTemp = Double.parseDouble(row.get("mean_temp").toString()); - receiver.output(KV.of(month, meanTemp)); + c.output(KV.of(month, meanTemp)); } } /** Format the results to a TableRow, to save to BigQuery. */ static class FormatMaxesFn extends DoFn, TableRow> { @ProcessElement - public void processElement( - @Element KV element, OutputReceiver receiver) { + public void processElement(ProcessContext c) { TableRow row = - new TableRow().set("month", element.getKey()).set("max_mean_temp", element.getValue()); - receiver.output(row); + new TableRow() + .set("month", c.element().getKey()) + .set("max_mean_temp", c.element().getValue()); + c.output(row); } } diff --git a/examples/java/src/main/java/org/apache/beam/examples/cookbook/MinimalBigQueryTornadoes.java b/examples/java/src/main/java/org/apache/beam/examples/cookbook/MinimalBigQueryTornadoes.java index 0c39c600b6cc..713af1d50953 100644 --- a/examples/java/src/main/java/org/apache/beam/examples/cookbook/MinimalBigQueryTornadoes.java +++ b/examples/java/src/main/java/org/apache/beam/examples/cookbook/MinimalBigQueryTornadoes.java @@ -26,8 +26,6 @@ import org.apache.beam.sdk.options.PipelineOptionsFactory; import org.apache.beam.sdk.transforms.Count; import org.apache.beam.sdk.transforms.DoFn; -import org.apache.beam.sdk.transforms.DoFn.Element; -import org.apache.beam.sdk.transforms.DoFn.OutputReceiver; import org.apache.beam.sdk.transforms.ParDo; import org.apache.beam.sdk.values.KV; import org.apache.beam.sdk.values.PCollection; @@ -75,9 +73,10 @@ public class MinimalBigQueryTornadoes { */ static class ExtractTornadoesFn extends DoFn { @ProcessElement - public void processElement(@Element TableRow row, OutputReceiver receiver) { + public void processElement(ProcessContext c) { + TableRow row = c.element(); if ((Boolean) row.get("tornado")) { - receiver.output(Integer.parseInt((String) row.get("month"))); + c.output(Integer.parseInt((String) row.get("month"))); } } } @@ -88,9 +87,8 @@ public void processElement(@Element TableRow row, OutputReceiver receiv */ static class FormatCountsFn extends DoFn, String> { @ProcessElement - public void processElement( - @Element KV element, OutputReceiver receiver) { - receiver.output(element.getKey() + ": " + element.getValue()); + public void processElement(ProcessContext c) { + c.output(c.element().getKey() + ": " + c.element().getValue()); } } @@ -136,9 +134,9 @@ static class LogOutput extends DoFn { } @ProcessElement - public void processElement(@Element T element, OutputReceiver receiver) { - LOG.info("{}{}", prefix, element); - receiver.output(element); + public void processElement(ProcessContext c) { + LOG.info("{}{}", prefix, c.element()); + c.output(c.element()); } } } diff --git a/examples/java/src/main/java/org/apache/beam/examples/cookbook/TriggerExample.java b/examples/java/src/main/java/org/apache/beam/examples/cookbook/TriggerExample.java index 5270077ea037..cd3c6dd84157 100644 --- a/examples/java/src/main/java/org/apache/beam/examples/cookbook/TriggerExample.java +++ b/examples/java/src/main/java/org/apache/beam/examples/cookbook/TriggerExample.java @@ -38,9 +38,6 @@ import org.apache.beam.sdk.options.PipelineOptionsFactory; import org.apache.beam.sdk.options.StreamingOptions; import org.apache.beam.sdk.transforms.DoFn; -import org.apache.beam.sdk.transforms.DoFn.Element; -import org.apache.beam.sdk.transforms.DoFn.OutputReceiver; -import org.apache.beam.sdk.transforms.DoFn.Timestamp; import org.apache.beam.sdk.transforms.GroupByKey; import org.apache.beam.sdk.transforms.PTransform; import org.apache.beam.sdk.transforms.ParDo; @@ -49,7 +46,6 @@ import org.apache.beam.sdk.transforms.windowing.AfterWatermark; import org.apache.beam.sdk.transforms.windowing.BoundedWindow; import org.apache.beam.sdk.transforms.windowing.FixedWindows; -import org.apache.beam.sdk.transforms.windowing.PaneInfo; import org.apache.beam.sdk.transforms.windowing.Repeatedly; import org.apache.beam.sdk.transforms.windowing.Window; import org.apache.beam.sdk.values.KV; @@ -368,18 +364,15 @@ public PCollection expand(PCollection> flowInfo) { new DoFn>, KV>() { @ProcessElement - public void processElement( - @Element KV> element, - OutputReceiver> receiver) - throws Exception { - Iterable flows = element.getValue(); + public void processElement(ProcessContext c) throws Exception { + Iterable flows = c.element().getValue(); Integer sum = 0; Long numberOfRecords = 0L; for (Integer value : flows) { sum += value; numberOfRecords++; } - receiver.output(KV.of(element.getKey(), sum + "," + numberOfRecords)); + c.output(KV.of(c.element().getKey(), sum + "," + numberOfRecords)); } })); PCollection output = results.apply(ParDo.of(new FormatTotalFlow(triggerType))); @@ -399,27 +392,21 @@ public FormatTotalFlow(String triggerType) { } @ProcessElement - public void processElement( - PaneInfo pane, - @Timestamp Instant timestamp, - BoundedWindow window, - @Element KV element, - OutputReceiver receiver) - throws Exception { - String[] values = element.getValue().split(",", -1); + public void processElement(ProcessContext c, BoundedWindow window) throws Exception { + String[] values = c.element().getValue().split(",", -1); TableRow row = new TableRow() .set("trigger_type", triggerType) - .set("freeway", element.getKey()) + .set("freeway", c.element().getKey()) .set("total_flow", Integer.parseInt(values[0])) .set("number_of_records", Long.parseLong(values[1])) .set("window", window.toString()) - .set("isFirst", pane.isFirst()) - .set("isLast", pane.isLast()) - .set("timing", pane.getTiming().toString()) - .set("event_time", timestamp.toString()) + .set("isFirst", c.pane().isFirst()) + .set("isLast", c.pane().isLast()) + .set("timing", c.pane().getTiming().toString()) + .set("event_time", c.timestamp().toString()) .set("processing_time", Instant.now().toString()); - receiver.output(row); + c.output(row); } } @@ -431,9 +418,8 @@ static class ExtractFlowInfo extends DoFn> { private static final int VALID_NUM_FIELDS = 50; @ProcessElement - public void processElement( - @Element String element, OutputReceiver> receiver) throws Exception { - String[] laneInfo = element.split(",", -1); + public void processElement(ProcessContext c) throws Exception { + String[] laneInfo = c.element().split(",", -1); if ("timestamp".equals(laneInfo[0])) { // Header row return; @@ -449,7 +435,7 @@ public void processElement( if (totalFlow == null || totalFlow <= 0) { return; } - receiver.output(KV.of(freeway, totalFlow)); + c.output(KV.of(freeway, totalFlow)); } } @@ -523,8 +509,7 @@ public void setup() { } @ProcessElement - public void processElement(@Element String element, OutputReceiver receiver) - throws Exception { + public void processElement(ProcessContext c) throws Exception { Instant timestamp = Instant.now(); if (random.nextDouble() < THRESHOLD) { int range = MAX_DELAY - MIN_DELAY; @@ -532,7 +517,7 @@ public void processElement(@Element String element, OutputReceiver recei long delayInMillis = TimeUnit.MINUTES.toMillis(delayInMinutes); timestamp = new Instant(timestamp.getMillis() - delayInMillis); } - receiver.outputWithTimestamp(element, timestamp); + c.outputWithTimestamp(c.element(), timestamp); } } diff --git a/examples/java/src/main/java/org/apache/beam/examples/snippets/Snippets.java b/examples/java/src/main/java/org/apache/beam/examples/snippets/Snippets.java index 99aada20669b..4f24c69f74b7 100644 --- a/examples/java/src/main/java/org/apache/beam/examples/snippets/Snippets.java +++ b/examples/java/src/main/java/org/apache/beam/examples/snippets/Snippets.java @@ -76,10 +76,6 @@ import org.apache.beam.sdk.transforms.Create; import org.apache.beam.sdk.transforms.DoFn; import org.apache.beam.sdk.transforms.DoFn.BoundedPerElement; -import org.apache.beam.sdk.transforms.DoFn.Element; -import org.apache.beam.sdk.transforms.DoFn.OutputReceiver; -import org.apache.beam.sdk.transforms.DoFn.SideInput; -import org.apache.beam.sdk.transforms.DoFn.Timestamp; import org.apache.beam.sdk.transforms.Latest; import org.apache.beam.sdk.transforms.MapElements; import org.apache.beam.sdk.transforms.ParDo; @@ -544,14 +540,14 @@ public static PCollection coGroupByKeyTuple( ParDo.of( new DoFn, String>() { @ProcessElement - public void processElement( - @Element KV e, OutputReceiver receiver) { + public void processElement(ProcessContext c) { + KV e = c.element(); String name = e.getKey(); Iterable emailsIter = e.getValue().getAll(emailsTag); Iterable phonesIter = e.getValue().getAll(phonesTag); String formattedResult = Snippets.formatCoGbkResults(name, emailsIter, phonesIter); - receiver.output(formattedResult); + c.output(formattedResult); } })); // [END CoGroupByKeyTuple] @@ -646,23 +642,20 @@ public void process( new DoFn>() { @ProcessElement - public void process( - @Timestamp Instant timestamp, - @Element Long element, - @SideInput("mapIterable") Iterable> si, - OutputReceiver> receiver) { + public void process(ProcessContext c, @Timestamp Instant timestamp) { + Iterable> si = c.sideInput(mapIterable); // Take an element from the side input iterable (likely length 1) Map keyMap = si.iterator().next(); - receiver.outputWithTimestamp(KV.of(1L, element), Instant.now()); + c.outputWithTimestamp(KV.of(1L, c.element()), Instant.now()); LOG.info( "Value is {} with timestamp {}, using key A from side input with time {}.", - element, + c.element(), timestamp.toString(DateTimeFormat.forPattern("HH:mm:ss")), keyMap.get("Key_A")); } }) - .withSideInput("mapIterable", mapIterable)); + .withSideInputs(mapIterable)); p.run(); } @@ -708,9 +701,9 @@ public static void accessingValueProviderInfoAfterRunSnip1(String[] args) { // Define the DoFn that logs the ValueProvider value. @ProcessElement - public void process(PipelineOptions options) { + public void process(ProcessContext c) { - MyOptions ops = options.as(MyOptions.class); + MyOptions ops = c.getPipelineOptions().as(MyOptions.class); // This example logs the ValueProvider value, but you could store it by // pushing it to an external database. @@ -950,13 +943,11 @@ public void process(@Element String src, OutputReceiver o) { ParDo.of( new DoFn() { @ProcessElement - public void process( - @SideInput("sideInput") List sideInputValue, - OutputReceiver receiver) { - receiver.output((long) sideInputValue.size()); + public void process(ProcessContext c) { + c.output((long) c.sideInput(sideInput).size()); } }) - .withSideInput("sideInput", sideInput)); + .withSideInputs(sideInput)); // [END PeriodicallyUpdatingSideInputs] return result; } @@ -1197,10 +1188,7 @@ private static class BundleFinalization { private static class BundleFinalizationDoFn extends DoFn { // [START BundleFinalize] @ProcessElement - public void processElement( - @Element String element, - OutputReceiver receiver, - BundleFinalizer bundleFinalizer) { + public void processElement(ProcessContext c, BundleFinalizer bundleFinalizer) { // ... produce output ... bundleFinalizer.afterBundleCommit( diff --git a/examples/java/src/main/java/org/apache/beam/examples/subprocess/ExampleEchoPipeline.java b/examples/java/src/main/java/org/apache/beam/examples/subprocess/ExampleEchoPipeline.java index b7fb7b82db31..4aa20fc10dfb 100644 --- a/examples/java/src/main/java/org/apache/beam/examples/subprocess/ExampleEchoPipeline.java +++ b/examples/java/src/main/java/org/apache/beam/examples/subprocess/ExampleEchoPipeline.java @@ -28,8 +28,6 @@ import org.apache.beam.sdk.options.PipelineOptionsFactory; import org.apache.beam.sdk.transforms.Create; import org.apache.beam.sdk.transforms.DoFn; -import org.apache.beam.sdk.transforms.DoFn.Element; -import org.apache.beam.sdk.transforms.DoFn.OutputReceiver; import org.apache.beam.sdk.transforms.ParDo; import org.apache.beam.sdk.values.KV; import org.slf4j.Logger; @@ -86,13 +84,11 @@ public void setUp() throws Exception { } @ProcessElement - public void processElement( - @Element KV element, OutputReceiver> receiver) - throws Exception { + public void processElement(ProcessContext c) throws Exception { try { // Our Library takes a single command in position 0 which it will echo back in the result SubProcessCommandLineArgs commands = new SubProcessCommandLineArgs(); - Command command = new Command(0, String.valueOf(element.getValue())); + Command command = new Command(0, String.valueOf(c.element().getValue())); commands.putCommand(command); // The ProcessingKernel deals with the execution of the process @@ -101,7 +97,7 @@ public void processElement( // Run the command and work through the results List results = kernel.exec(commands); for (String s : results) { - receiver.output(KV.of(element.getKey(), s)); + c.output(KV.of(c.element().getKey(), s)); } } catch (Exception ex) { LOG.error("Error processing element ", ex); diff --git a/examples/java/src/test/java/org/apache/beam/examples/cookbook/TriggerExampleTest.java b/examples/java/src/test/java/org/apache/beam/examples/cookbook/TriggerExampleTest.java index 0d9e257fc6c0..19c83c6eb73c 100644 --- a/examples/java/src/test/java/org/apache/beam/examples/cookbook/TriggerExampleTest.java +++ b/examples/java/src/test/java/org/apache/beam/examples/cookbook/TriggerExampleTest.java @@ -29,8 +29,6 @@ import org.apache.beam.sdk.testing.TestPipeline; import org.apache.beam.sdk.transforms.Create; import org.apache.beam.sdk.transforms.DoFn; -import org.apache.beam.sdk.transforms.DoFn.Element; -import org.apache.beam.sdk.transforms.DoFn.OutputReceiver; import org.apache.beam.sdk.transforms.ParDo; import org.apache.beam.sdk.transforms.windowing.FixedWindows; import org.apache.beam.sdk.transforms.windowing.Window; @@ -148,8 +146,8 @@ static String canonicalFormat(TableRow row) { static class FormatResults extends DoFn { @ProcessElement - public void processElement(@Element TableRow element, OutputReceiver receiver) - throws Exception { + public void processElement(ProcessContext c) throws Exception { + TableRow element = c.element(); TableRow row = new TableRow() .set("trigger_type", element.get("trigger_type")) @@ -160,7 +158,7 @@ public void processElement(@Element TableRow element, OutputReceiver rec .set("isLast", element.get("isLast")) .set("timing", element.get("timing")) .set("window", element.get("window")); - receiver.output(canonicalFormat(row)); + c.output(canonicalFormat(row)); } } } diff --git a/examples/java/src/test/java/org/apache/beam/examples/subprocess/ExampleEchoPipelineTest.java b/examples/java/src/test/java/org/apache/beam/examples/subprocess/ExampleEchoPipelineTest.java index 9981b4040e3b..01055d9658a9 100644 --- a/examples/java/src/test/java/org/apache/beam/examples/subprocess/ExampleEchoPipelineTest.java +++ b/examples/java/src/test/java/org/apache/beam/examples/subprocess/ExampleEchoPipelineTest.java @@ -41,8 +41,6 @@ import org.apache.beam.sdk.testing.TestPipeline; import org.apache.beam.sdk.transforms.Create; import org.apache.beam.sdk.transforms.DoFn; -import org.apache.beam.sdk.transforms.DoFn.Element; -import org.apache.beam.sdk.transforms.DoFn.OutputReceiver; import org.apache.beam.sdk.transforms.ParDo; import org.apache.beam.sdk.values.KV; import org.apache.beam.sdk.values.PCollection; @@ -146,13 +144,11 @@ public void setUp() throws Exception { } @ProcessElement - public void processElement( - @Element KV element, OutputReceiver> receiver) - throws Exception { + public void processElement(ProcessContext c) throws Exception { try { // Our Library takes a single command in position 0 which it will echo back in the result SubProcessCommandLineArgs commands = new SubProcessCommandLineArgs(); - Command command = new Command(0, String.valueOf(element.getValue())); + Command command = new Command(0, String.valueOf(c.element().getValue())); commands.putCommand(command); // The ProcessingKernel deals with the execution of the process @@ -161,7 +157,7 @@ public void processElement( // Run the command and work through the results List results = kernel.exec(commands); for (String s : results) { - receiver.output(KV.of(element.getKey(), s)); + c.output(KV.of(c.element().getKey(), s)); } } catch (Exception ex) { LOG.error("Error processing element ", ex); diff --git a/examples/kotlin/src/main/java/org/apache/beam/examples/kotlin/DebuggingWordCount.kt b/examples/kotlin/src/main/java/org/apache/beam/examples/kotlin/DebuggingWordCount.kt index 7be4df2c8a91..05cb404ac6d2 100644 --- a/examples/kotlin/src/main/java/org/apache/beam/examples/kotlin/DebuggingWordCount.kt +++ b/examples/kotlin/src/main/java/org/apache/beam/examples/kotlin/DebuggingWordCount.kt @@ -25,8 +25,6 @@ import org.apache.beam.sdk.options.Description import org.apache.beam.sdk.options.PipelineOptionsFactory import org.apache.beam.sdk.testing.PAssert import org.apache.beam.sdk.transforms.DoFn -import org.apache.beam.sdk.transforms.DoFn.Element -import org.apache.beam.sdk.transforms.DoFn.OutputReceiver import org.apache.beam.sdk.transforms.ParDo import org.apache.beam.sdk.values.KV import org.slf4j.LoggerFactory @@ -86,18 +84,18 @@ public object DebuggingWordCount { private val unmatchedWords = Metrics.counter(FilterTextFn::class.java, "unmatchedWords") @ProcessElement - fun processElement(@Element element: KV, receiver: OutputReceiver>) { - if (filter.matcher(element.key).matches()) { + fun processElement(c: ProcessContext) { + if (filter.matcher(c.element().key).matches()) { // Log at the "DEBUG" level each element that we match. When executing this pipeline // these log lines will appear only if the log level is set to "DEBUG" or lower. - LOG.debug("Matched: ${element.key}") + LOG.debug("Matched: ${c.element().key}") matchedWords.inc() - receiver.output(element) + c.output(c.element()) } else { // Log at the "TRACE" level each element that is not matched. Different log levels // can be used to control the verbosity of logging providing an effective mechanism // to filter less important information. - LOG.trace("Did not match: ${element.key}") + LOG.trace("Did not match: ${c.element().key}") unmatchedWords.inc() } } diff --git a/examples/kotlin/src/main/java/org/apache/beam/examples/kotlin/cookbook/BigQueryTornadoes.kt b/examples/kotlin/src/main/java/org/apache/beam/examples/kotlin/cookbook/BigQueryTornadoes.kt index 8edc98872350..ec56bc659970 100644 --- a/examples/kotlin/src/main/java/org/apache/beam/examples/kotlin/cookbook/BigQueryTornadoes.kt +++ b/examples/kotlin/src/main/java/org/apache/beam/examples/kotlin/cookbook/BigQueryTornadoes.kt @@ -27,8 +27,6 @@ import org.apache.beam.sdk.io.gcp.bigquery.WriteResult import org.apache.beam.sdk.options.* import org.apache.beam.sdk.transforms.Count import org.apache.beam.sdk.transforms.DoFn -import org.apache.beam.sdk.transforms.DoFn.Element -import org.apache.beam.sdk.transforms.DoFn.OutputReceiver import org.apache.beam.sdk.transforms.PTransform import org.apache.beam.sdk.transforms.ParDo import org.apache.beam.sdk.values.KV @@ -75,9 +73,10 @@ object BigQueryTornadoes { */ internal class ExtractTornadoesFn : DoFn() { @ProcessElement - fun processElement(@Element row: TableRow, receiver: OutputReceiver) { + fun processElement(c: ProcessContext) { + val row = c.element() if (row["tornado"] as Boolean) { - receiver.output(Integer.parseInt(row["month"] as String)) + c.output(Integer.parseInt(row["month"] as String)) } } } @@ -88,11 +87,11 @@ object BigQueryTornadoes { */ internal class FormatCountsFn : DoFn, TableRow>() { @ProcessElement - fun processElement(@Element element: KV, receiver: OutputReceiver) { + fun processElement(c: ProcessContext) { val row = TableRow() - .set("month", element.key) - .set("tornado_count", element.value) - receiver.output(row) + .set("month", c.element().key) + .set("tornado_count", c.element().value) + c.output(row) } } diff --git a/examples/kotlin/src/main/java/org/apache/beam/examples/kotlin/cookbook/CombinePerKeyExamples.kt b/examples/kotlin/src/main/java/org/apache/beam/examples/kotlin/cookbook/CombinePerKeyExamples.kt index 4d89b692dd82..9e388031002d 100644 --- a/examples/kotlin/src/main/java/org/apache/beam/examples/kotlin/cookbook/CombinePerKeyExamples.kt +++ b/examples/kotlin/src/main/java/org/apache/beam/examples/kotlin/cookbook/CombinePerKeyExamples.kt @@ -26,8 +26,6 @@ import org.apache.beam.sdk.io.gcp.bigquery.WriteResult import org.apache.beam.sdk.metrics.Metrics import org.apache.beam.sdk.options.* import org.apache.beam.sdk.transforms.* -import org.apache.beam.sdk.transforms.DoFn.Element -import org.apache.beam.sdk.transforms.DoFn.OutputReceiver import org.apache.beam.sdk.values.KV import org.apache.beam.sdk.values.PCollection @@ -72,11 +70,12 @@ object CombinePerKeyExamples { private val smallerWords = Metrics.counter(ExtractLargeWordsFn::class.java, "smallerWords") @ProcessElement - fun processElement(@Element row: TableRow, receiver: OutputReceiver>) { + fun processElement(c: ProcessContext) { + val row = c.element() val playName = row["corpus"] as String val word = row["word"] as String if (word.length >= MIN_WORD_LENGTH) { - receiver.output(KV.of(word, playName)) + c.output(KV.of(word, playName)) } else { // Track how many smaller words we're not including. This information will be // visible in the Monitoring UI. @@ -91,9 +90,9 @@ object CombinePerKeyExamples { */ internal class FormatShakespeareOutputFn : DoFn, TableRow>() { @ProcessElement - fun processElement(@Element element: KV, receiver: OutputReceiver) { - val row = TableRow().set("word", element.key).set("all_plays", element.value) - receiver.output(row) + fun processElement(c: ProcessContext) { + val row = TableRow().set("word", c.element().key).set("all_plays", c.element().value) + c.output(row) } } diff --git a/examples/kotlin/src/main/java/org/apache/beam/examples/kotlin/cookbook/FilterExamples.kt b/examples/kotlin/src/main/java/org/apache/beam/examples/kotlin/cookbook/FilterExamples.kt index 218188448965..2625f5bfec10 100644 --- a/examples/kotlin/src/main/java/org/apache/beam/examples/kotlin/cookbook/FilterExamples.kt +++ b/examples/kotlin/src/main/java/org/apache/beam/examples/kotlin/cookbook/FilterExamples.kt @@ -25,9 +25,6 @@ import org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO import org.apache.beam.sdk.io.gcp.bigquery.WriteResult import org.apache.beam.sdk.options.* import org.apache.beam.sdk.transforms.* -import org.apache.beam.sdk.transforms.DoFn.Element -import org.apache.beam.sdk.transforms.DoFn.OutputReceiver -import org.apache.beam.sdk.transforms.DoFn.SideInput import org.apache.beam.sdk.values.PCollection import java.util.logging.Logger @@ -83,7 +80,8 @@ object FilterExamples { */ internal class ProjectionFn : DoFn() { @ProcessElement - fun processElement(@Element row: TableRow, receiver: OutputReceiver) { + fun processElement(c: ProcessContext) { + val row = c.element() // Grab year, month, day, mean_temp from the row val year = Integer.parseInt(row["year"] as String) val month = Integer.parseInt(row["month"] as String) @@ -96,7 +94,7 @@ object FilterExamples { .set("month", month) .set("day", day) .set("mean_temp", meanTemp) - receiver.output(outRow) + c.output(outRow) } } @@ -110,10 +108,11 @@ object FilterExamples { internal class FilterSingleMonthDataFn(private var monthFilter: Int?) : DoFn() { @ProcessElement - fun processElement(@Element row: TableRow, receiver: OutputReceiver) { + fun processElement(c: ProcessContext) { + val row = c.element() val month = row["month"] if (month == this.monthFilter) { - receiver.output(row) + c.output(row) } } } @@ -124,9 +123,10 @@ object FilterExamples { */ internal class ExtractTempFn : DoFn() { @ProcessElement - fun processElement(@Element row: TableRow, receiver: OutputReceiver) { + fun processElement(c: ProcessContext) { + val row = c.element() val meanTemp = java.lang.Double.parseDouble(row["mean_temp"].toString()) - receiver.output(meanTemp) + c.output(meanTemp) } } @@ -158,17 +158,15 @@ object FilterExamples { ParDo.of( object : DoFn() { @ProcessElement - fun processElement( - @SideInput("globalMeanTemp") gTemp: Double, - @Element element: TableRow, - receiver: OutputReceiver) { - val meanTemp = java.lang.Double.parseDouble(element["mean_temp"].toString()) + fun processElement(c: ProcessContext) { + val meanTemp = java.lang.Double.parseDouble(c.element()["mean_temp"].toString()) + val gTemp = c.sideInput(globalMeanTemp) if (meanTemp < gTemp) { - receiver.output(element) + c.output(c.element()) } } }) - .withSideInput("globalMeanTemp", globalMeanTemp)) + .withSideInputs(globalMeanTemp)) } } diff --git a/examples/kotlin/src/main/java/org/apache/beam/examples/kotlin/cookbook/JoinExamples.kt b/examples/kotlin/src/main/java/org/apache/beam/examples/kotlin/cookbook/JoinExamples.kt index 7f81629ae463..2f2215e1d96a 100644 --- a/examples/kotlin/src/main/java/org/apache/beam/examples/kotlin/cookbook/JoinExamples.kt +++ b/examples/kotlin/src/main/java/org/apache/beam/examples/kotlin/cookbook/JoinExamples.kt @@ -26,8 +26,6 @@ import org.apache.beam.sdk.options.PipelineOptions import org.apache.beam.sdk.options.PipelineOptionsFactory import org.apache.beam.sdk.options.Validation import org.apache.beam.sdk.transforms.DoFn -import org.apache.beam.sdk.transforms.DoFn.Element -import org.apache.beam.sdk.transforms.DoFn.OutputReceiver import org.apache.beam.sdk.transforms.ParDo import org.apache.beam.sdk.transforms.join.CoGbkResult import org.apache.beam.sdk.transforms.join.CoGroupByKey @@ -91,14 +89,13 @@ object JoinExamples { ParDo.of( object : DoFn, KV>() { @ProcessElement - fun processElement( - @Element element: KV, - receiver: OutputReceiver>) { - val countryCode = element.key - val countryName = element.value.getOnly(countryInfoTag) - for (ei in element.value.getAll(eventInfoTag)) { + fun processElement(c: ProcessContext) { + val e = c.element() + val countryCode = e.key + val countryName = e.value.getOnly(countryInfoTag) + for (ei in c.element().value.getAll(eventInfoTag)) { // Generate a string that combines information from both collection values - receiver.output( + c.output( KV.of( countryCode, "Country name: $countryName, Event info: $ei")) @@ -112,9 +109,9 @@ object JoinExamples { ParDo.of( object : DoFn, String>() { @ProcessElement - fun processElement(@Element element: KV, receiver: OutputReceiver) { - val outputString = "Country code: ${element.key}, ${element.value}" - receiver.output(outputString) + fun processElement(c: ProcessContext) { + val outputString = "Country code: ${c.element().key}, ${c.element().value}" + c.output(outputString) } })) } @@ -125,13 +122,14 @@ object JoinExamples { */ internal class ExtractEventDataFn : DoFn>() { @ProcessElement - fun processElement(@Element row: TableRow, receiver: OutputReceiver>) { + fun processElement(c: ProcessContext) { + val row = c.element() val countryCode = row["ActionGeo_CountryCode"] as String val sqlDate = row["SQLDATE"] as String val actor1Name = row["Actor1Name"] as String val sourceUrl = row["SOURCEURL"] as String val eventInfo = "Date: $sqlDate, Actor1: $actor1Name, url: $sourceUrl" - receiver.output(KV.of(countryCode, eventInfo)) + c.output(KV.of(countryCode, eventInfo)) } } @@ -141,10 +139,11 @@ object JoinExamples { */ internal class ExtractCountryInfoFn : DoFn>() { @ProcessElement - fun processElement(@Element row: TableRow, receiver: OutputReceiver>) { + fun processElement(c: ProcessContext) { + val row = c.element() val countryCode = row["FIPSCC"] as String val countryName = row["HumanName"] as String - receiver.output(KV.of(countryCode, countryName)) + c.output(KV.of(countryCode, countryName)) } } diff --git a/examples/kotlin/src/main/java/org/apache/beam/examples/kotlin/cookbook/MaxPerKeyExamples.kt b/examples/kotlin/src/main/java/org/apache/beam/examples/kotlin/cookbook/MaxPerKeyExamples.kt index ad16ecde9657..11418d3933cf 100644 --- a/examples/kotlin/src/main/java/org/apache/beam/examples/kotlin/cookbook/MaxPerKeyExamples.kt +++ b/examples/kotlin/src/main/java/org/apache/beam/examples/kotlin/cookbook/MaxPerKeyExamples.kt @@ -26,8 +26,6 @@ import org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO import org.apache.beam.sdk.io.gcp.bigquery.WriteResult import org.apache.beam.sdk.options.* import org.apache.beam.sdk.transforms.DoFn -import org.apache.beam.sdk.transforms.DoFn.Element -import org.apache.beam.sdk.transforms.DoFn.OutputReceiver import org.apache.beam.sdk.transforms.Max import org.apache.beam.sdk.transforms.PTransform import org.apache.beam.sdk.transforms.ParDo @@ -75,21 +73,22 @@ object MaxPerKeyExamples { */ internal class ExtractTempFn : DoFn>() { @ProcessElement - fun processElement(@Element row: TableRow, receiver: OutputReceiver>) { + fun processElement(c: ProcessContext) { + val row = c.element() val month = Integer.parseInt(row["month"] as String) val meanTemp = java.lang.Double.parseDouble(row["mean_temp"].toString()) - receiver.output(KV.of(month, meanTemp)) + c.output(KV.of(month, meanTemp)) } } /** Format the results to a TableRow, to save to BigQuery. */ internal class FormatMaxesFn : DoFn, TableRow>() { @ProcessElement - fun processElement(@Element element: KV, receiver: OutputReceiver) { + fun processElement(c: ProcessContext) { val row = TableRow() - .set("month", element.key) - .set("max_mean_temp", element.value) - receiver.output(row) + .set("month", c.element().key) + .set("max_mean_temp", c.element().value) + c.output(row) } } diff --git a/examples/kotlin/src/main/java/org/apache/beam/examples/kotlin/cookbook/TriggerExample.kt b/examples/kotlin/src/main/java/org/apache/beam/examples/kotlin/cookbook/TriggerExample.kt index bb8c0900e319..4afa7d0dfc70 100644 --- a/examples/kotlin/src/main/java/org/apache/beam/examples/kotlin/cookbook/TriggerExample.kt +++ b/examples/kotlin/src/main/java/org/apache/beam/examples/kotlin/cookbook/TriggerExample.kt @@ -34,9 +34,6 @@ import org.apache.beam.sdk.options.Description import org.apache.beam.sdk.options.PipelineOptionsFactory import org.apache.beam.sdk.options.StreamingOptions import org.apache.beam.sdk.transforms.DoFn -import org.apache.beam.sdk.transforms.DoFn.Element -import org.apache.beam.sdk.transforms.DoFn.OutputReceiver -import org.apache.beam.sdk.transforms.DoFn.Timestamp import org.apache.beam.sdk.transforms.GroupByKey import org.apache.beam.sdk.transforms.PTransform import org.apache.beam.sdk.transforms.ParDo @@ -367,17 +364,15 @@ object TriggerExample { @ProcessElement @Throws(Exception::class) - fun processElement( - @Element element: KV>, - receiver: OutputReceiver>) { - val flows = element.value + fun processElement(c: ProcessContext) { + val flows = c.element().value var sum = 0 var numberOfRecords = 0L for (value in flows) { sum += value numberOfRecords++ } - receiver.output(KV.of(element.key, "$sum,$numberOfRecords")) + c.output(KV.of(c.element().key, "$sum,$numberOfRecords")) } })) return results.apply(ParDo.of(FormatTotalFlow(triggerType))) @@ -392,25 +387,20 @@ object TriggerExample { @ProcessElement @Throws(Exception::class) - fun processElement( - @Element element: KV, - window: BoundedWindow, - pane: PaneInfo, - @Timestamp timestamp: Instant, - receiver: OutputReceiver) { - val values = element.value.split(",".toRegex()).toTypedArray() + fun processElement(c: ProcessContext, window: BoundedWindow) { + val values = c.element().value.split(",".toRegex()).toTypedArray() val row = TableRow() .set("trigger_type", triggerType) - .set("freeway", element.key) + .set("freeway", c.element().key) .set("total_flow", Integer.parseInt(values[0])) .set("number_of_records", java.lang.Long.parseLong(values[1])) .set("window", window.toString()) - .set("isFirst", pane.isFirst) - .set("isLast", pane.isLast) - .set("timing", pane.timing.toString()) - .set("event_time", timestamp.toString()) + .set("isFirst", c.pane().isFirst) + .set("isLast", c.pane().isLast) + .set("timing", c.pane().timing.toString()) + .set("event_time", c.timestamp().toString()) .set("processing_time", Instant.now().toString()) - receiver.output(row) + c.output(row) } } @@ -422,8 +412,8 @@ object TriggerExample { @ProcessElement @Throws(Exception::class) - fun processElement(@Element element: String, receiver: OutputReceiver>) { - val laneInfo = element.split(",".toRegex()).toTypedArray() + fun processElement(c: ProcessContext) { + val laneInfo = c.element().split(",".toRegex()).toTypedArray() if ("timestamp" == laneInfo[0]) { // Header row return @@ -439,7 +429,7 @@ object TriggerExample { if (totalFlow == null || totalFlow <= 0) { return } - receiver.output(KV.of(freeway, totalFlow)) + c.output(KV.of(freeway, totalFlow)) } companion object { @@ -496,7 +486,7 @@ object TriggerExample { @ProcessElement @Throws(Exception::class) - fun processElement(@Element element: String, receiver: OutputReceiver) { + fun processElement(c: ProcessContext) { var timestamp = Instant.now() if (random.nextDouble() < THRESHOLD) { val range = MAX_DELAY - MIN_DELAY @@ -504,7 +494,7 @@ object TriggerExample { val delayInMillis = TimeUnit.MINUTES.toMillis(delayInMinutes.toLong()) timestamp = Instant(timestamp.millis - delayInMillis) } - receiver.outputWithTimestamp(element, timestamp) + c.outputWithTimestamp(c.element(), timestamp) } companion object { diff --git a/examples/kotlin/src/main/java/org/apache/beam/examples/kotlin/snippets/Snippets.kt b/examples/kotlin/src/main/java/org/apache/beam/examples/kotlin/snippets/Snippets.kt index eb2099372482..d2f58c215a56 100644 --- a/examples/kotlin/src/main/java/org/apache/beam/examples/kotlin/snippets/Snippets.kt +++ b/examples/kotlin/src/main/java/org/apache/beam/examples/kotlin/snippets/Snippets.kt @@ -33,8 +33,6 @@ import org.apache.beam.sdk.io.gcp.bigquery.DynamicDestinations import org.apache.beam.sdk.io.gcp.bigquery.TableDestination import org.apache.beam.sdk.io.gcp.bigquery.WriteResult import org.apache.beam.sdk.transforms.* -import org.apache.beam.sdk.transforms.DoFn.Element -import org.apache.beam.sdk.transforms.DoFn.OutputReceiver import org.apache.beam.sdk.transforms.join.CoGbkResult import org.apache.beam.sdk.transforms.join.CoGroupByKey import org.apache.beam.sdk.transforms.join.KeyedPCollectionTuple @@ -362,12 +360,13 @@ object Snippets { ParDo.of( object : DoFn, String>() { @ProcessElement - fun processElement(@Element element: KV, receiver: OutputReceiver) { - val name = element.key - val emailsIter = element.value.getAll(emailsTag) - val phonesIter = element.value.getAll(phonesTag) + fun processElement(c: ProcessContext) { + val e = c.element() + val name = e.key + val emailsIter = e.value.getAll(emailsTag) + val phonesIter = e.value.getAll(phonesTag) val formattedResult = formatCoGbkResults(name, emailsIter, phonesIter) - receiver.output(formattedResult) + c.output(formattedResult) } })) } From c782069aca73a53f35b9959b4fd63d521f7d54f2 Mon Sep 17 00:00:00 2001 From: Yi Hu Date: Thu, 30 Apr 2026 15:20:11 +0000 Subject: [PATCH 2/2] Restore SKILLs.md as it does not affect Beam playground --- .agent/skills/beam-dofn-modernizer/SKILL.md | 207 ++++++++++++++++++++ 1 file changed, 207 insertions(+) create mode 100644 .agent/skills/beam-dofn-modernizer/SKILL.md diff --git a/.agent/skills/beam-dofn-modernizer/SKILL.md b/.agent/skills/beam-dofn-modernizer/SKILL.md new file mode 100644 index 000000000000..d45fc3fdc794 --- /dev/null +++ b/.agent/skills/beam-dofn-modernizer/SKILL.md @@ -0,0 +1,207 @@ +--- +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +--- +name: beam-dofn-modernizer +description: Rewrite Apache Beam DoFn methods (@ProcessElement, @OnTimer, @OnWindowExpiration) to remove legacy ProcessContext or OnTimerContext usage. Use this skill when you encounter DoFn methods that use context.element(), context.output(), etc., and need to modernize them using parameter injection (@Element, @Timestamp, @Pane, OutputReceiver, MultiOutputReceiver). +--- + +# Modernizing Apache Beam DoFns + +Apache Beam has moved towards parameter injection in `DoFn` methods to improve readability and allow for more efficient execution. This skill helps you migrate legacy `ProcessContext` and `OnTimerContext` usage to modern annotated parameters. + +## Core Mappings + +When rewriting a `@ProcessElement` or `@OnTimer` method, replace the context argument with the corresponding parameters based on the usage: + +| Legacy Context Usage (e.g. `ProcessContext c`) | Modern Parameter Replacement | +| :--- | :--- | +| `c.element()` | `@Element T element` | +| `c.timestamp()` | `@Timestamp Instant timestamp` | +| `c.pane()` | `PaneInfo pane` | +| `c.window()` | `BoundedWindow window` | +| `c.sideInput(PCollectionView view)` | `@SideInput("viewName") T value` | +| `c.getPipelineOptions()` | `PipelineOptions options` | +| `c.output(value)` | `OutputReceiver receiver` then `receiver.output(value)` | +| `c.output(tag, value)` | `MultiOutputReceiver receiver` then `receiver.get(tag).output(value)` | +| `c.outputWithTimestamp(value, ts)` | `OutputReceiver receiver` then `receiver.outputWithTimestamp(value, ts)` | + +## Method Signature Changes + +### @ProcessElement + +**Legacy:** +```java +@ProcessElement +public void processElement(ProcessContext c) { + T element = c.element(); + c.output(transform(element)); +} +``` + +**Modern:** +```java +@ProcessElement +public void processElement( + @Element T element, + @Timestamp Instant timestamp, + OutputReceiver receiver) { + receiver.output(transform(element)); +} +``` + +### @OnTimer + +**Legacy:** +```java +@OnTimer("timerId") +public void onTimer(OnTimerContext c) { + c.output(someValue); +} +``` + +**Modern:** +```java +@OnTimer("timerId") +public void onTimer( + @Timestamp Instant timestamp, + BoundedWindow window, + OutputReceiver receiver) { + receiver.output(someValue); +} +``` + +## Best Practices + +1. **Specific OutputReceiver**: If the method only outputs to the main output, use `OutputReceiver`. If it outputs to multiple tags, use `MultiOutputReceiver`. +2. **Element Type**: Ensure the `@Element` parameter type matches the input type of the `DoFn`. +3. **Imports**: Don't forget to add imports for: + * `org.apache.beam.sdk.transforms.DoFn.Element` + * `org.apache.beam.sdk.transforms.DoFn.Timestamp` + * `org.apache.beam.sdk.transforms.DoFn.OutputReceiver` + * `org.apache.beam.sdk.transforms.DoFn.MultiOutputReceiver` (if needed) + * `org.apache.beam.sdk.values.PCollectionView` (if using `@SideInput`) + * `org.apache.beam.sdk.transforms.DoFn.SideInput` + * `org.apache.beam.sdk.transforms.windowing.PaneInfo` +4. **Side Inputs**: When using `@SideInput`, make sure to use the correct name that matches the one passed to `ParDo.withSideInput("name", view)`. +5. **Parameter Naming and Redundant Variables**: Use descriptive names for the `@Element` parameter (e.g., `record`, `line`, `row`) instead of a generic `element` if it improves readability. Do not create a redundant local variable to copy the element (e.g., `MyType elm = element;`), use the parameter directly. + +## Example Conversion + +### Before: +```java +@ProcessElement +public void processElement(ProcessContext c) { + KV element = c.element(); + Instant ts = c.timestamp(); + if (element.getValue() > threshold) { + c.output(element.getKey()); + c.output(specialTag, element.getValue()); + } +} +``` + +### After: +```java +@ProcessElement +public void processElement( + @Element KV element, + @Timestamp Instant timestamp, + MultiOutputReceiver receiver) { + if (element.getValue() > threshold) { + receiver.get(mainTag).output(element.getKey()); + receiver.get(specialTag).output(element.getValue()); + } +} +``` +> [!NOTE] +> If you only have one output, use `OutputReceiver receiver` and `receiver.output(element.getKey())`. + +## Side Input Conversion + +Modernizing side inputs involves removing the `PCollectionView` from the `DoFn` constructor and using `@SideInput` parameter injection instead. + +### Before (Legacy): + +**PTransform/Pipeline side:** +```java +PCollectionView myView = ...; +input.apply(ParDo.of(new MyFn(myView)).withSideInputs(myView)); +``` + +**DoFn side:** +```java +class MyFn extends DoFn { + private final PCollectionView view; + MyFn(PCollectionView view) { this.view = view; } + + @ProcessElement + public void processElement(ProcessContext c) { + String value = c.sideInput(view); + // ... + } +} +``` + +### Nullable Side Inputs + +If a side input is optional and a `DoFn` has conditional logic based on whether the side input is present, it is best to split the `DoFn` into two separate classes: one that requires the side input and one that does not. This avoids creating complex, conditional `DoFn`s and ensures type safety. + +**PTransform/Pipeline side:** +```java +PCollectionView myView = ...; +input.apply(ParDo.of(new MyFn(myView)).withSideInputs(myView)); +//or +input.apply(ParDo.of(new MyFn(null))); // to introduce null +``` +**DoFn side:** +```java +class MyFn extends DoFn { + private final PCollectionView view; + MyFn(PCollectionView view) { this.view = view; } + + @ProcessElement + public void processElement(ProcessContext c) { + String value = null; + if (this.view != null) { // can do conditional side input + value = c.sideInput(this.view); + } + + // ... + } +} +``` + +### After (Modern): + +**PTransform/Pipeline side:** +```java +PCollectionView myView = ...; +input.apply(ParDo.of(new MyFn()).withSideInput("sideInputName", myView)); +``` + +**DoFn side:** +```java +class MyFn extends DoFn { + @ProcessElement + public void processElement( + @Element T element, + @SideInput("sideInputName") String value) { + // value is injected directly + } +} +```