Current blocking tree is passed as part of the task which makes it slower and we get warnings like
26/07/24 16:59:30 WARN DAGScheduler: Broadcasting large task binary with size 5.4 MiB
Let us see if broadcasting the blocking tree and using it in the BlockFunction may yeild better results.
We need to make the code change and then verify performance on 5m to figure out if it indeed helps.
Current blocking tree is passed as part of the task which makes it slower and we get warnings like
Let us see if broadcasting the blocking tree and using it in the
BlockFunctionmay yeild better results.We need to make the code change and then verify performance on 5m to figure out if it indeed helps.