Multitopic realtime ingestion as it currently exists has some issues. In this PR, @Jackie-Jiang said
The multi-topic partition id overloading existing partition id has caused so many issues:
We are missing checks here and there
Stream with custom partition id fails (See #17260)
I think it is the time to revisit this design decision and avoid the troubles going forward. I can see 2 possible solutions:
Make partition_id part within the table name x_y where x is the topic id and y is the partition id. For single topic name, it remains the same
Leverage logical table for multiple topics, each topic a separate physical table
This is the PEP request to make those changes. I've detailed both suggestions in this doc:
https://docs.google.com/document/d/1dY6ZvaG_-FbTAQ5e1lxB4uvi8VGbR_WfCfCl5zD76A4/
Multitopic realtime ingestion as it currently exists has some issues. In this PR, @Jackie-Jiang said
The multi-topic partition id overloading existing partition id has caused so many issues:
We are missing checks here and there
Stream with custom partition id fails (See #17260)
I think it is the time to revisit this design decision and avoid the troubles going forward. I can see 2 possible solutions:
Make partition_id part within the table name x_y where x is the topic id and y is the partition id. For single topic name, it remains the same
Leverage logical table for multiple topics, each topic a separate physical table
This is the PEP request to make those changes. I've detailed both suggestions in this doc:
https://docs.google.com/document/d/1dY6ZvaG_-FbTAQ5e1lxB4uvi8VGbR_WfCfCl5zD76A4/