UnicodeDecodeError in pod log reading marks healthy GPU training runs as SYSTEM_ERROR

[morgan-wowk]

Root cause

When a Kubernetes pod completes a GPU training run with tqdm progress bars, the pod log contains multi-byte UTF-8 block glyphs (█▉▊▋▌▍▎▏, each 3 bytes: E2 96 8x). Two distinct mechanisms produce torn byte sequences in the raw log bytes that the Kubernetes API returns:

1. Concurrent writer interleaving — 64 worker processes share a single stderr fd. Python's TextIOWrapper can split one logical write() call across multiple write() syscalls. When two processes' writes interleave at byte granularity, the receiver sees an orphaned continuation byte (invalid continuation byte).

2. Response truncation at a chunk boundary — the K8s API response is split at a size boundary (~5.1 MB observed), cutting a 3-byte glyph after its first byte (unexpected end of data at 0xe2).

The Kubernetes Python client reads the response with r.data.decode('utf8') (strict, no error handling). Both variants throw UnicodeDecodeError before the run has any real failure. The exception bubbles through the _retry wrapper in internal_process_one_running_execution and, after 5 retries, marks the execution SYSTEM_ERROR — causing downstream Upload steps to be skipped on an otherwise successful training run.

This is responsible for 3,960 Bugsnag occurrences in the past 7 days.

Neither the Linux kernel nor the Kubernetes client is at fault. The kernel's PIPE_BUF atomic write guarantee applies only to writes ≤ 4096 bytes, and Python's TextIOWrapper may issue multiple syscalls for one logical write. The corruption is baked into the bytes before the K8s client sees them, so the client cannot fix it in a lossless way.

Solutions compared

Three approaches were kept as separate PRs for team comparison:

PR #277 — Defensive decode (_preload_content=False + errors="replace")
Bypass the K8s client's strict decode by using _preload_content=False, then decode ourselves with errors="replace" so U+FFFD is substituted for any malformed byte sequence. A _logger.warning fires when substitution occurs.

• Pro: handles both error classes; logs remain complete (only the torn glyph is replaced)
• Con: lossy — the torn bytes (partial tqdm glyph) are replaced rather than recovered

PR #278 — Suppress tqdm output at source
Inject TQDM_DISABLE=1 (and related env vars) into the pod's environment so block glyphs are never written to stderr in the first place.

• Pro: eliminates the root cause entirely for tqdm-based progress bars
• Con: only covers tqdm; any other multi-byte output from training code or libraries would still tear

PR #279 / #280 — Swallow log-acquisition errors + Observe search link
Catch all exceptions from log reading, return an empty string / None, and embed a pre-built Observe log search URL in the placeholder message so the user can still find the pod logs.

• Pro: execution is never incorrectly marked SYSTEM_ERROR; user has a direct link to raw logs
• Con: the execution log stored in GCS is lost; relies on Observe retention

Fix landed

PR #277 — _preload_content=False + errors="replace" was selected. It addresses both UnicodeDecodeError variants, keeps the execution log intact (minus the torn glyph bytes), and does not require changes to the training environment.

Fixes #277

[morgan-wowk]

If the K8s library wanted to, could it take any other approach other than TextIOWrapper that would guarentee non-torn log reads and not be subject to the multiprocessing issue?

No. The corruption happens upstream of the K8s client entirely — inside the container at write time. By the time the API serves the log, the torn bytes are already baked into the stored log data. The K8s client is receiving a faithful copy of corrupted bytes; no read strategy (streaming chunk by chunk, overlapping range requests using sinceBytes/limitBytes, different buffering) can recover bytes that were never atomically written in the first place.

The only lossless fix requires changing the writer side: either ensure all multi-byte sequences are written atomically (a multiprocessing-safe handler that holds a lock around each write() syscall), or eliminate multi-byte output entirely (tqdm suppress). The K8s client, as a pure reader, has no lever to pull.

Where would we implement the multiprocessing-safe handler?

In the training code, not in Tangle. It would go wherever the tqdm progress bars are configured — likely a datasets.map() call or a training loop in user-submitted pipeline components.

The handler would be a custom logging.Handler (or a thin wrapper around tqdm.tqdm.write) that acquires a multiprocessing.Lock before each write, ensuring the full byte sequence for one glyph lands in a single atomic syscall. But since this lives in user code rather than the framework, Tangle can't enforce it — which is why the defensive decode in PR #277 is the right fix at the framework layer.

So does this only impact multi-node Tangle executions?

No — it affects any execution where num_proc > 1 in a datasets.map() or similar call, even on a single node. The torn bytes come from multiple Python processes sharing the same stderr fd within one pod, which happens with multiprocessing regardless of how many nodes are involved.

Multi-node (LaunchedKubernetesJob) just has an additional surface: each pod has its own log, and _get_log_by_pod_key reads them individually — but the tearing mechanism is the same within each pod.

Multiple Tangle users get this problem. It happens more often than we'd like. We'd like to do something to help and not offload the solution to the training code / component.

The most practical option at the framework layer is to land both PRs:

#277 (defensive decode) as a permanent safety net — handles any future torn bytes regardless of source
#278 (inject TQDM_DISABLE=1) as the preventative fix for the known cause — Tangle already controls the pod spec, so it can inject this universally without touching user code
Together they give defense-in-depth: tqdm glyphs are suppressed before they can tear, and if anything else ever produces a malformed sequence, the decode won't crash.

The tradeoff with #278 is that users lose progress bars in their logs. That's worth a conversation — some users may care. You could make it opt-out via a component annotation (e.g. tangleml.com/tqdm-disable: "false") so the default is safe but power users can re-enable it.

CC: @Ark-kun

[morgan-wowk]

What I'm getting is the proper fix is at the log write within the container itself. Putting the onus on the component author. It's interesting that it happens so frequently. I was hoping there would be something we could do to help prevent it Tangle-side but it seems not.