What happened?
Problem description
The function apache_beam.io.gcp.pubsublite.ReadFromPubSubLite is not working. This can be seen as a simple data processing pipeline for reading messages from a Google Cloud Pub/Sub Lite subscription and logging the messages' content is not working for apache-beam=2.50.0 and google-cloud-pubsublite=1.8.3.
Code to reproduce issue
This is the Python file defining the pipeline:
import logging
import apache_beam
from apache_beam.io.gcp import pubsublite
from apache_beam import Pipeline
from apache_beam.options.pipeline_options import PipelineOptions
def run(pipeline_args=None):
pipeline_options = PipelineOptions(
pipeline_args, streaming=True, save_main_session=False
)
with Pipeline(options=pipeline_options) as pipeline:
elements = (
pipeline
| "Read from Pub/Sub" >> pubsublite.ReadFromPubSubLite(subscription_path="projects/<PROJECT_ID>/locations/<LOCATION>/subscriptions/<SUBSCRIPTION_NAME>")
| "Print Elements" >> apache_beam.Map(lambda x: logging.info(x.getMessage()) or x)
)
if __name__ == "__main__":
logging.getLogger().setLevel(logging.INFO)
run()
When running the pipeline locally the only logs printed are the following:
INFO:root:severity: INFO
timestamp {
seconds: 1694002848
nanos: 949000000
}
message: "Fn Harness started"
log_location: "org.apache.beam.fn.harness.FnHarness"
thread: "1"
INFO:root:severity: INFO
timestamp {
seconds: 1694002849
nanos: 579000000
}
message: "Entering instruction processing loop"
log_location: "org.apache.beam.fn.harness.FnHarness"
thread: "1"
Now I post a message on the Pub/Sub Lite topic the subscription has subscribed:
gcloud pubsub lite-topics publish <TOPIC_NAME> --location <LOCATION> --project=<PROJECT_ID> --message="Hello world"
This message is not printed despite the pipeline process still running.
Issue Priority
Priority: 1 (data loss / total loss of function)
Issue Components
What happened?
Problem description
The function
apache_beam.io.gcp.pubsublite.ReadFromPubSubLiteis not working. This can be seen as a simple data processing pipeline for reading messages from a Google Cloud Pub/Sub Lite subscription and logging the messages' content is not working forapache-beam=2.50.0andgoogle-cloud-pubsublite=1.8.3.Code to reproduce issue
This is the Python file defining the pipeline:
When running the pipeline locally the only logs printed are the following:
Now I post a message on the Pub/Sub Lite topic the subscription has subscribed:
gcloud pubsub lite-topics publish <TOPIC_NAME> --location <LOCATION> --project=<PROJECT_ID> --message="Hello world"This message is not printed despite the pipeline process still running.
Issue Priority
Priority: 1 (data loss / total loss of function)
Issue Components