Skip to content

feat: Add connect check for Spark "Task was not acquired" error#267

Merged
nj1973 merged 3 commits into
mainfrom
257-dataproc-spark-connect-error-check
Jun 4, 2026
Merged

feat: Add connect check for Spark "Task was not acquired" error#267
nj1973 merged 3 commits into
mainfrom
257-dataproc-spark-connect-error-check

Conversation

@nj1973
Copy link
Copy Markdown
Collaborator

@nj1973 nj1973 commented Jun 3, 2026

The PR adds a simple check when Managed Spark (serverless) batches fail. It took a customer a couple of days to figute they had a missing firewall rule, hopefully the new text helps point them at the Spark pre-reqs. I didn't include an actual link to docs in the error because it's only a matter of time before the link becomes a dead link.

I also added:

  • An internal design doc for connect to help future AI session understand the tool in order to make changes.
  • Fixed setup for 3 tests. I tweaked the OFFLOAD_PREFIX on my test server and the tests didn't like it.

@nj1973 nj1973 linked an issue Jun 3, 2026 that may be closed by this pull request
@nj1973 nj1973 requested a review from ssenior45 June 3, 2026 17:06
Copy link
Copy Markdown

@ssenior45 ssenior45 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@nj1973 nj1973 merged commit c2b1d71 into main Jun 4, 2026
9 checks passed
@nj1973 nj1973 deleted the 257-dataproc-spark-connect-error-check branch June 4, 2026 07:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Dataproc Spark connect error check

2 participants