Add Real-Time Mode (RTM) sub-second latency streaming demo#78
Open
jiteshsoni wants to merge 6 commits intodatabricks-solutions:mainfrom
Open
Add Real-Time Mode (RTM) sub-second latency streaming demo#78jiteshsoni wants to merge 6 commits intodatabricks-solutions:mainfrom
jiteshsoni wants to merge 6 commits intodatabricks-solutions:mainfrom
Conversation
This demo showcases Databricks Real-Time Mode for achieving sub-second
latency in streaming pipelines. It implements a stateless guardrail
pipeline that validates Ethereum blockchain events in real-time.
Features:
- RTM-enabled streaming pipeline (Kafka to Kafka)
- Sensitive data detection (PII, credentials)
- Validation rules for operational guardrails
- Dynamic topic routing (ALLOW/QUARANTINE)
- Parse error handling for malformed JSON
- End-to-end and unit tests
Requirements:
- Databricks Runtime 16.4 LTS or later
- Dedicated clusters (serverless not supported)
- outputMode("update") required for RTM
Files:
- rtm_stateless_guardrail.py - Main notebook
- cluster_config.template.json - Cluster config template
- test_rtm_guardrail.py - Unit tests
- e2e_local_test.py - End-to-end tests
- produce_test_data.py - Test data producer
- README.md - Documentation
Blog: https://canadiandataguy.com/p/unlocking-sub-second-latency-with
jiteshsoni
requested review from
QuentinAmbard,
alanreese-dbrx,
alexott,
anupkalburgi,
kwulffert23,
matthewmoorcroft,
slcc2c and
srinivasadmala
as code owners
Reflect the Databricks + Redpanda integration path that worked in practice by requiring single-user clusters for UC volume checkpoints, documenting explicit topic setup, and replaying backlog from earliest offsets during demos.
Replace the old canned validation notes with the actual staging workspace and Redpanda checks used to verify the demo, including the live routing cases that were confirmed end to end.
Extend the RTM notebook with an explicit output-topic verification section so the demo shows what was actually written to the allowed and quarantine topics, instead of only showing query status.
Point readers to the new output-topic verification and stream management sections so the README matches the notebook flow during demos and testing.
Store the Kafka output verification screenshot in the PR branch so it can be embedded directly in the pull request description.
Member
|
@jiteshsoni I can see jwt tokens, also missing readme and your name in codeowners file, can you review the code to make sure that there are no security issues. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR adds a Real-Time Mode demo that reads Ethereum-style block events from Kafka, applies a set of stateless guardrail checks, and routes each record to either an allowed or quarantine topic.
The intent is to show the full Kafka -> Databricks RTM -> Kafka loop in a way that is easy to run, inspect, and explain.
Included in the PR
rtm_stateless_guardrail.py: main notebookcluster_config.template.json: RTM-capable cluster templatetest_rtm_guardrail.py: local validation-pattern testse2e_local_test.py: local end-to-end validation logic via Databricks Connectproduce_test_data.py: sample Kafka producerREADME.md: setup and testing notesSetup Notes
A few setup details turned out to matter in practice and are now called out in the code and docs:
outputMode("update")startingOffsets = "earliest"for demos and integration tests so seeded backlog is replayedThe notebook also now includes an explicit verification section that reads the target Kafka topics back and shows what was actually written to
-allowedand-quarantine, instead of stopping at query status alone.How I tested it
Local checks
python test_rtm_guardrail.pyWorkspace / integration validation
Validated on e2-dogfood using:
rtm-guardrail-cluster(0313-063110-u4ldfaiy)875702710968733447049974860353For the final validation run I created a fresh set of Kafka topics, seeded deterministic records, ran the notebook as a Databricks submit run, and then verified the output topics directly.
Confirmed routing:
4000001->ethereum-validated-jobrun-20260401000541-allowedwithdecision=ALLOW4000002->ethereum-validated-jobrun-20260401000541-quarantinewithvalidation_reasons=["HIGH_GAS_USAGE"]4000003->ethereum-validated-jobrun-20260401000541-quarantinewithvalidation_reasons=["PII_EMAIL"]Output verification screenshot:
That verified both halves of the demo:
Related Post
Unlocking Sub-Second Latency with Spark Structured Streaming Real-Time Mode