Follow-up to #2329.
When a native source feeds more than one Calc (for example UNION ALL over one table under Flink source reuse), the source is not fused and each Calc runs as a standalone native operator over the shared source stream. This is correct, but it repeats work per consumer.
The source emits AuronColumnarRowData, a zero-copy view over the native Arrow batch. Each standalone Calc then re-materializes that row back into a freshly allocated Arrow VectorSchemaRoot, column by column, before running its native plan (FlinkAuronCalcOperator -> FlinkArrowFFIExporter -> FlinkArrowWriter). So for N consumers the Kafka read and decode happen once, but the row-to-Arrow rebuild is paid N times, and it is a rows x cols scalar copy each time.
AuronColumnarRowData exposes only per-field getters, so there is no zero-copy re-export of the underlying batch today, and the exporter has no already-Arrow-backed fast path.
Idea: hand Arrow batches across the operator edge instead of a per-row RowData stream, so a shared native source can feed multiple native Calc operators without rebuilding the columnar buffers per consumer. This is a batch-oriented operator exchange and is larger than the multi-consumer-safety scope of #2329, hence a separate issue.
Magnitude is unmeasured. A profiler run on a source -> single Calc job (parallelism 1) would quantify the CPU share spent in FlinkArrowWriter.write / ArrowFieldWriter versus Kafka read and native compute, to confirm the win is worth the change.
Follow-up to #2329.
When a native source feeds more than one Calc (for example
UNION ALLover one table under Flink source reuse), the source is not fused and each Calc runs as a standalone native operator over the shared source stream. This is correct, but it repeats work per consumer.The source emits
AuronColumnarRowData, a zero-copy view over the native Arrow batch. Each standalone Calc then re-materializes that row back into a freshly allocated ArrowVectorSchemaRoot, column by column, before running its native plan (FlinkAuronCalcOperator->FlinkArrowFFIExporter->FlinkArrowWriter). So for N consumers the Kafka read and decode happen once, but the row-to-Arrow rebuild is paid N times, and it is a rows x cols scalar copy each time.AuronColumnarRowDataexposes only per-field getters, so there is no zero-copy re-export of the underlying batch today, and the exporter has no already-Arrow-backed fast path.Idea: hand Arrow batches across the operator edge instead of a per-row RowData stream, so a shared native source can feed multiple native Calc operators without rebuilding the columnar buffers per consumer. This is a batch-oriented operator exchange and is larger than the multi-consumer-safety scope of #2329, hence a separate issue.
Magnitude is unmeasured. A profiler run on a
source -> single Calcjob (parallelism 1) would quantify the CPU share spent inFlinkArrowWriter.write/ArrowFieldWriterversus Kafka read and native compute, to confirm the win is worth the change.