In TPC-H Q21 (SF=10, lineitem = 59,986,052 rows), the right side of the lineitem SortMergeJoin is sorted on the single-column integer key l_orderkey (Int64). The Auron native SortExec is ~8x slower than Spark's native sort on this path, because it degrades a single-column integer sort into a generic byte-comparison sort.
Root cause
| Implementation |
Sort approach |
Single-column Int64 handling |
| Spark SortExec |
prefix + radix sort (O(n)) |
radix sort on 8-byte prefix |
| Auron SortExec |
generic external sort |
RowConverter encodes Int64 into 9 bytes (1B null + 8B big-endian, sign-flipped), then byte-compares |
Auron NativeSort time : 13.8s
Spark Sort time: 1.6s
In TPC-H Q21 (SF=10, lineitem = 59,986,052 rows), the right side of the lineitem
SortMergeJoinis sorted on the single-column integer key l_orderkey (Int64). The Auron native SortExec is ~8x slower than Spark's native sort on this path, because it degrades a single-column integer sort into a generic byte-comparison sort.Root cause
Auron NativeSort time : 13.8s
Spark Sort time: 1.6s
