From de5c1624ec047b09334326350a2be04e87426e00 Mon Sep 17 00:00:00 2001 From: SAY-5 Date: Tue, 14 Apr 2026 20:25:02 -0700 Subject: [PATCH] fix(python): remove hardcoded OTEL_LOG_LEVEL=DEBUG that caused stderr deadlocks MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Per #3357, `python.runScript()` deadlocks any Python subprocess that writes more than ~64KB to stderr. The previous implementation hardcoded `OTEL_LOG_LEVEL: 'DEBUG'` in the subprocess environment, which caused OTEL-aware Python libraries (mlflow, opentelemetry-sdk, etc.) to emit verbose debug-level logs to stderr during import. Once stderr exceeded the OS pipe buffer, the Python process would block on its next `write()` syscall indefinitely — a silent, permanent hang with no error or timeout (unless `maxDuration` was set). Remove the hardcoded `OTEL_LOG_LEVEL: 'DEBUG'`. This brings `runScript()` in line with `run()`, `runInline()`, and the streaming variants — none of which set `OTEL_LOG_LEVEL`. The original line was almost certainly leftover dev instrumentation that shipped accidentally. Closes #3357 --- .changeset/python-stderr-deadlock-fix.md | 5 +++++ packages/python/src/index.ts | 1 - 2 files changed, 5 insertions(+), 1 deletion(-) create mode 100644 .changeset/python-stderr-deadlock-fix.md diff --git a/.changeset/python-stderr-deadlock-fix.md b/.changeset/python-stderr-deadlock-fix.md new file mode 100644 index 00000000000..3eef0c3aa96 --- /dev/null +++ b/.changeset/python-stderr-deadlock-fix.md @@ -0,0 +1,5 @@ +--- +"@trigger.dev/python": patch +--- + +Fix `python.runScript()` deadlock when the Python subprocess produces more than ~64KB of stderr output. The previous implementation hardcoded `OTEL_LOG_LEVEL: "DEBUG"` in the subprocess environment, which caused OTEL-aware Python libraries (mlflow, opentelemetry-sdk, etc.) to emit verbose debug-level logging to stderr during import. Once stderr exceeded the OS pipe buffer, the Python process would block on `write()` indefinitely. Removing the hardcoded debug log level brings `runScript()` in line with `run()`, `runInline()`, and the streaming variants — none of which set `OTEL_LOG_LEVEL`. diff --git a/packages/python/src/index.ts b/packages/python/src/index.ts index 6924802fd88..8d68db01dd4 100644 --- a/packages/python/src/index.ts +++ b/packages/python/src/index.ts @@ -97,7 +97,6 @@ export const python = { }=trigger,${Object.entries(taskContext.attributes) .map(([key, value]) => `${key}=${value}`) .join(",")}`, - OTEL_LOG_LEVEL: "DEBUG", }, }, throwOnError: false,