If the Spark Canary fails in Dataproc and the status for the failure is "Task was not acquired" then we are likely missing a firewall rule.
We already do a describe on the batches job, so we should be able to check the status/reason and log extra firewall related advice.
Advice is noted here: https://docs.cloud.google.com/managed-spark/docs/concepts/network-serverless#open-subnet
If the Spark Canary fails in Dataproc and the status for the failure is "Task was not acquired" then we are likely missing a firewall rule.
We already do a describe on the batches job, so we should be able to check the status/reason and log extra firewall related advice.
Advice is noted here: https://docs.cloud.google.com/managed-spark/docs/concepts/network-serverless#open-subnet