[BUG] Debug with Databricks Connect hangs on Spark actions (toPandas/collect) due to debugpy/gRPC deadlock

[gmartinezmirai]

Describe the bug

When using "Debug Current File with Databricks Connect" on serverless compute, the debugger hangs indefinitely on the first Spark action that triggers a gRPC call (e.g., .toPandas(), .collect()). The same notebook runs to completion using "Run Current File with Databricks Connect" (no debugger).

The root cause appears to be debugpy's sys.settrace thread tracing interfering with gRPC background threads used by Spark Connect. The gRPC response handler thread deadlocks under debugpy's trace hook, so the Spark action never receives its response.

Expected behavior:
Debugging should work the same as run mode — Spark actions should complete and breakpoints should be hit without hanging.

To Reproduce

1. Configure Databricks Connect with serverless compute
2. Create a notebook with any Spark action (e.g., spark.table("x").toPandas())
3. Set a breakpoint before or after the Spark action
4. Use "Debug Current File with Databricks Connect"
5. Execution hangs at the Spark action and never returns

System information:

• Databricks Extension: 2.10.6
• databricks-connect: 16.1
• debugpy: 1.8.20
• Python: 3.12
• OS: Windows 11 Enterprise 10.0.26200
• Compute: Serverless
• VS Code: 1.104.3

Databricks Extension Logs

sdk-and-extension-logs.json

Additional context

Attempted workarounds (none resolved the issue):

• Custom launch.json with justMyCode: true and subProcess: false
• gevent: true in launch config
• Attach mode (debugpy.listen() + debugpy.wait_for_client()): allows debugging up to the first Spark action after attach, then hangs on the next gRPC call