kubernetes: swallow log-acquisition errors to prevent false SYSTEM_ERROR

[morgan-wowk]

Problem

When the Kubernetes API returns a malformed response — truncated body, broken UTF-8, broken JSON — the Kubernetes client throws inside read_namespaced_pod_log before returning any data. That exception surfaces to the orchestrator's _retry wrapper (max_retries=5). After five consecutive failures the wrapper re-raises, and the orchestrator records a SYSTEM_ERROR and skips all downstream tasks — even though the training job itself completed successfully.

The 5-retry pattern has been observed in production: the K8s API served the same malformed response on all five attempts before the retry budget was exhausted.

Fix

Catch all non-ApiException errors in the two log-read entry points:

• LaunchedKubernetesContainer.get_log — single-pod path
• LaunchedKubernetesJob._get_log_by_pod_key — multi-pod path

On failure, emit a WARNING log with the full traceback (observable in Observe) and return an empty string / None. The execution reaches its correct terminal state (SUCCEEDED or FAILED) with an empty log, rather than being incorrectly promoted to SYSTEM_ERROR.

The existing ApiException / "Bad Request" handling for PodInitializing is preserved.

Trade-off

This is a deliberate lossy choice: a log-fetch failure produces a missing log rather than a run failure. The trade-off is:

• Upside: eliminates false SYSTEM_ERROR on healthy runs; downstream tasks are no longer skipped; retry budget is preserved for real failures.
• Downside: when logs are missing the user has no in-product record of the run's output. The warning in Observe is the only signal that a log was lost.

Context — three approaches under review

PR	Approach	Logs preserved?	Run outcome
#277	Defensive decode (errors="replace")	Yes — torn glyphs replaced with ?	Run completes, logs uploaded
#278	Suppress tqdm at source (TQDM_DISABLE=1)	Yes — no non-ASCII bytes emitted	Run completes, logs uploaded
This PR	Swallow log-acquisition errors	No — empty log on fetch failure	Run completes, logs empty

#277 and #278 address the specific torn-UTF-8 failure class. This PR addresses the broader class of any malformed K8s API response, including cases where the API repeatedly returns broken data that no decode strategy can recover from.

[morgan-wowk]

• kubernetes: surface log-unavailable reason and observability link in placeholder #280
• kubernetes: swallow log-acquisition errors to prevent false SYSTEM_ERROR #279 👈 (View in Graphite)
• master

This stack of pull requests is managed by Graphite. Learn more about stacking.

[morgan-wowk]

We chose a different solution as indicated on the issue #281