FlareDB is a streaming database for building and running batch and streaming data pipelines. It uses Apache Beam as its programming interface. Beam provides a rich programming model for writing batch and streaming data pipelines in Java, Python, Go and SQL, while FlareDB provides a Rust based runtime to execute pipelines written with Beam SDKs.
Its based on a unified streams-and-tables architecture. Streams represent data in motion, while tables represent that same data as materialized state. FlareDB brings these concepts together in a single engine. As pipelines execute, PCollections transition naturally between streams and materialized table state, allowing FlareDB to unify data processing and storage within a single system.
For a deeper dive into FlareDB's design and execution model, check out this post: https://ganeshsivakumar.substack.com/p/flaredb.
⭐ New streaming systems don't come along that often. If you're curious to see where this project goes, consider starring the repository, it helps you keep track of updates and helps others discover it too.
The FlareDB CLI provides commands to initialize, start, and manage FlareDB instances, as well as run Apache Beam pipelines on FlareDB.
If you are on Linux or macOS , please run the following command to install the CLI:
curl --proto '=https' --tlsv1.2 -LsSf https://github.com/flare-db/flare-db/releases/download/flare-cli-v0.1.3/flare-cli-installer.sh | shFor Windows run the following command in PowerShell:
powershell -ExecutionPolicy Bypass -c "irm https://github.com/flare-db/flare-db/releases/download/flare-cli-v0.1.3/flare-cli-installer.ps1 | iex"Alternatively, you can download the CLI binary directly from the GitHub Releases page by selecting the appropriate binary for your platform.
After installing the CLI, run:
flare initThis command performs the initial setup by creating the required local directories and downloading the FlareDB binary and Apache Beam worker JAR.
The initialization only needs to be completed once. After that, you can use the flare up and flare down commands to manage the instance.
Start a local FlareDB instance with:
flare upOnce the instance is running, FlareDB is ready to accept pipeline jobs.
To run an Apache Beam pipeline on FlareDB, add the FlareDB Runner SDK as a dependency to your Beam project. The runner sdk submits the pipeline to the FlareDB instance as a Job.
Check out the WordCount example under examples/ for a complete reference.
With FlareDB running, execute the WordCount example:
# compile wordcount pipeline
mvn clean install
# run the example
mvn exec:java -Dexec.mainClass="com.flaredb.example.WordCount"The pipeline will be submitted to the local FlareDB instance and executed by the engine. Execution logs and pipeline output can be found in the logging directory created during startup.
After executing pipelines, run this command to stop FlareDB instance
flare downFlareDB V0.1.0 is the first public release of FlareDB. It lays the foundation for a streaming database and its execution engine.
The initial release supports:
- Single-node execution of Apache Beam pipelines.
- Bounded sources on the Global Window.
- Native execution of core runner transforms, including
ImpulseandGroupByKey. - Portable
DoFnexecution through the Beam SDK Harness. - Apache Beam Portability Framework implementation.
Upcoming releases will focus on expanding FlareDB's streaming execution capabilities:
- Unbounded Sources - Support for watermarks, event-time processing, windowing, and triggers.
- Splittable DoFns - Parallel work execution for I/Os.
- Stateful Processing - Implementation of the Apache Beam State API.
- Native Transforms - Additional runner-native implementations for element-wise, aggregation, and composite transforms.
- Materialized Views - Persist Beam
PCollectionsas queryable table state for serving and analytics.
FlareDB is licensed under the Apache License 2.0