A centralized management layer for Apache Kafka that enables admin-approved topic creation and dynamic consumer subscriptions.
This project replaces the default "auto-create" behavior of Kafka with a governed workflow. Topics are requested by producers, vetted by an administrator, and dynamically provisioned across the system. This ensures that the Kafka broker remains clean and only contains approved data streams.
The producer handles high-throughput data and background synchronization using three dedicated threads:
- Input Listener Thread: Continuously ingests incoming data streams and moves them to an internal queue.
- Publisher Thread: Consumes data from the internal queue and routes messages to the appropriate Kafka topics.
- Topic Watcher Thread: Periodically checks the Admin API for newly approved topics. When a topic becomes active, it enables the Publisher to begin routing data to it.
- Request Lifecycle: New topic requests are submitted via the UI and stored in SQLite with a 'pending' status.
- Approval UI: An administrator can approve or reject requests.
- Status Management: Once approved, the system triggers the physical creation of the topic in Kafka and updates the database status to 'active'.
- Controlled Environment: Configured with
auto.create.topics.enable=false. - Topic Creation: Topics are created programmatically only after the Admin approval flow is complete.
- Consumer UI: A FastAPI-driven interface that displays all currently active topics fetched from the database.
- Dynamic Subscription: Users select topics from the UI, and the consumer code dynamically subscribes to those specific streams to retrieve messages.
- Human-in-the-Loop Governance: Prevents unauthorized or accidental topic creation.
- Non-blocking Architecture: Multi-threaded producer ensures data ingestion continues while waiting for topic approvals.
- Real-time Discovery: Consumers and producers sync with the database to discover new topics without service restarts.
- Simplified Management: Centralized SQLite database tracks topic metadata and user-topic mappings.
- Language: Python
- Web Framework: FastAPI
- Database: SQLite
- Message Broker: Apache Kafka
- Concurrency: Python Threading Library