Improve Data Stream Cleanup and Management with Enhanced Tracking and Sequential IDs

[Andreufb]

This merge introduces significant enhancements to the data stream management and cleanup process, ensuring efficient resource utilization and better user control. Key changes include:

• Active Consumers and Data Streams Tracking:

  • Implemented active consumers and data streams trackers to manage and clean up resources upon API shutdown.
  • This ensures no leftover streams or consumers remain active after shutdown.

• Sequential Integer-Based Stream IDs:

  • Replaced random hash-based stream IDs with sequential integers (e.g., data_stream_1, data_stream_2).
  • This allows for clearer identification and better tracking of streams, while also preventing random ID generation.

• Enforced Maximum Stream Limit:

  • Introduced logic to enforce a maximum number of active streams using an environment variable (KAFKA_MAX_STREAMS).
  • This ensures no more streams than allowed are created, preventing overloading the system.

• New Endpoint for Stream Deletion:

  • Added an endpoint to delete streams by their integer ID (e.g., 1, 2) or by full topic name.
  • The endpoint efficiently removes streams from Kafka and updates internal trackers.

• Stream ID Reuse Mechanism:

  • Developed functionality to detect and reuse available stream IDs when streams are deleted.
  • This avoids ID exhaustion and allows for efficient recycling of IDs for newly created streams.

• Enhanced Kafka Topic Deletion:

  • Improved the Kafka topic deletion process to handle errors gracefully and remove deleted topics from both Kafka and internal trackers.
  • Logging and error handling ensure better transparency and easier debugging for users when topics fail to delete.

These improvements streamline the data stream lifecycle, offering efficient resource management, reducing the potential for stream ID exhaustion, and providing greater control over stream creation and deletion processes.

[Andreufb]

The .env_kafka has been updated, need to accept this new version.

[rbardaji]

Could you remove the IP please?