These are core plans, but need to find the right "sweet spot" for these.
- Configuration for the pipeline
- Adding the right instrumentation
- Putting telemetry in
- Evaluate NATS and determine if that's a valid replacement for Kafka. Zookeeper as a dependency sucks.
- Evaluate NoSQL DB's for storing run data.
These are core plans, but need to find the right "sweet spot" for these.