[deployment,core] fix: harden local sandbox lifecycle

[FeathBow]

What does this PR do?

Hardens the LocalDeployment sandbox lifecycle for high-concurrency RL training, mirroring the Modal hardening from #47. Refs #43; it does not claim to close it. Blast radius: uni_agent/deployment/local/deployment.py plus diagnose=False on the two loguru sinks in async_logging.py.

Checklist Before Starting

• Search for similar PRs or issues and paste at least one relevant link here: Support on local deployment for SWE-Bench. #43 (target issue), [deployment] fix: harden Modal sandbox lifecycle at high concurrency #47 (Modal counterpart, merged), [deployment,docs] fix: select local OCI runtime endpoint by network path #44 (local endpoint-port fix -- semantically orthogonal, but real code overlap in deployment.py: it overlaps _start_oci_container and adjacent endpoint/network code, so whichever lands second must rebase; this branch keeps _build_run_command itself byte-identical to main, so [deployment,docs] fix: select local OCI runtime endpoint by network path #44's change to that function still grafts cleanly -- merge [deployment,docs] fix: select local OCI runtime endpoint by network path #44 first and rebase this onto it), [train] fix: add Layer-2 safety net and MoE shape match to UniAgentLoop failure path #50 (no intersection: agent_loop.py is at main here), [core] fix: replace per-run log sinks with single dispatch sink to fix leaks and HDFS stalls #63 (the only async_logging.py change here is diagnose=False on the two sink configs; the dispatch rework is untouched)
• Format the PR title as [{modules}] {type}: {description} (checked by CI)
  • {modules} may include core, interaction, model, env, tools, deployment, reward, dashboard, docs, examples, data, train, ci, build, deps, misc
  • If this PR involves multiple modules, separate them with , like [interaction, tools, docs]
  • {type} must be one of feat, fix, refactor, chore, test
  • If this PR breaks an API, config contract, workflow, or other compatibility boundary, add [BREAKING] to the beginning of the title
  • For a stacked PR series, you may prepend a progress marker such as [1/N]
  • Example: [BREAKING][deployment, docs] feat: simplify runtime env configuration

Test

• tests/deployment/ unit tests: 101 cases on this branch (the 2 config tests land in PR 2). PYTHONPATH=. pytest tests/deployment/. Python 3.11 venv. No container runtime is needed -- the tests replace subprocess.run / Popen with doubles, so no real docker/apptainer is ever invoked (an autouse fixture also stubs free-port picking). Most cases aim at failures the Docker gate cannot stage reliably (a CLI killed mid-create, a tag query that will not answer, a close() that hangs); the rest pin plain behaviour (command construction, config validation, the limiter's plumbing).
• The real-Docker E2E gate lives in PR 3 (it depends on this branch's behaviour and PR 2's config), so there is no live-Docker evidence in this PR.
• pre-commit run --all-files clean (ruff, ruff-format, mypy, compileall).

Design & Code Changes

• Serial lifecycle over one CLI slot. Concurrency is across deployments (a STARTING limiter caps how many launch at once); one deployment's lifecycle is serial -- overlapping start/stop, or a start over resources no stop reclaimed, raise. That keeps the bookkeeping small: with at most one container CLI call alive, "abandoned" is a bool, not generations threaded through a ContextVar. A cancel cannot stop a running docker run, so its future (_future) holds a single slot until someone takes the result back; _exec_bg, start(), and stop() all refuse to run over an unclaimed one, and every event-loop CLI call (run/logs/rm) goes through it so no rm overtakes the log fetch that must precede it. A start happens only over a clean deployment: start() refuses unless _has_cleanup() is false (nothing owned, in flight, unnamed, a retained runtime handle, or an unreaped process/log). The failed-attempt cleanup between retries is bounded by _CLEANWAIT, but the rm -f it launches carries a larger bound (2 x _RMWAIT + _RMPAUSE) on an uncancellable thread -- so cleanup can return having reclaimed nothing, and a retry that started anyway would turn one failed cleanup into two live containers; the loop stops and demands an explicit stop(). (The GC finalizer is the exception: it shells out directly, each OCI call bounded by _DELWAIT, best-effort.)
• Ownership by confirmed id -- no exit code proves non-creation. Every OCI run carries an exclusive --cidfile and a random uni-agent.owner=<uuid> label, injected in _with_cid after every user option so extra_run_args cannot override them (so _build_run_command stays byte-identical to main); an untaggable shape or an ambiguous bare -- is refused rather than launched. The docker CLI creates the cidfile empty, the daemon creates the container, and only then does the CLI write the id -- so a failed id-write exits non-zero and deletes the cidfile again: "non-zero, no cidfile, no stdout" can still be a live container, and the owner label is then the only ownership-safe way left to find it. Only a full 64-hex id is accepted; with none, the owner tag decides -- a single match is removed by id; two cases prove nothing of ours exists (the runtime never ran, or it ran and reported failure with an authoritative-empty query); every unresolved outcome is the state the container may exist and we cannot name it, kept, reported at ERROR, and blocking the retry and the next start() until stop() or the finalizer re-probes and removes by id. An id leaves the ledger only on a confirmed removal (_confirmed_gone), in the critical section that clears its diagnostics pointer. Removal is always by id, never by name (which could reach a foreign container).
• Teardown: bounded close, apptainer, and a teardown logging failure that cannot abort cleanup. A RemoteRuntime handle is an HTTP client, not evidence of container ownership; _has_cleanup does count a live handle as a barrier (it blocks re-entry until cleanup), but close() gets one bounded attempt (_CLOSEWAIT) and the handle is dropped afterward whatever happens -- a hung or failing close cannot barricade the deployment forever. Apptainer server processes get SIGTERM then SIGKILL with bounded waits, references cleared only on confirmed exit; a failed cleanup keeps the process and log references for the next stop(). Every teardown record -- cleanup, stop(), everything __del__ reaches -- goes through one helper that swallows a raising sink, so a torn-down logger at interpreter shutdown cannot abort a sweep and strand a container; start-only records keep the plain logger and stay loud.
• The auth token stays out of the logs, as a contract: redacted by value (never by command syntax -- the sandbox command is a user template that may place it anywhere) and only while it is still alive. Logged exception summaries go through _why; rendered chains and all container output through _redact. Nothing carrying the raw argv or a raw cause reaches a caller's own logger -- _runtime_exec runs check=False and rebuilds its error outside the except (so no __context__ leaks), and the retry loop raises an unchained, redacted summary; the loguru sinks run diagnose=False. It covers apptainer too, whose Popen.args hold the token. Tokens are registered last in _begin_attempt (after the announce, so a start that dies earlier strands nothing) and released once _has_cleanup() is false.
• Observability, trade-offs, matrix. One traceback-bearing record per non-cancelled failed attempt (id/attempt/port); the announce carries attempt/port/name; greppable "may leak" errors; invalid LOCAL_* limits fail fast with ValueError. Accepted: a cancelled docker run keeps pressuring the daemon until its 600s CLI timeout, and a stop() landing on one waits for it; a container that neither cidfile nor owner label ever names must be reclaimed by hand from the ERROR record. LocalDeployment.start's max_retries stays at main's 5 -- the value AgentEnv.start passes -- so the retry tests exercise the same retry contract AgentEnv relies on (they do not run the training path). Target matrix: Linux x86_64 + Docker on the rollout host; macOS is development-only.

Checklist Before Submitting

• Read the Contribute Guide
• Run pre-commit install && pre-commit run --all-files --show-diff-on-failure --color=always
• Add or update docs/examples for user-facing changes
• Add tests or explain why tests are not practical
• Confirm the PR title matches the required format
• Confirm the placeholder text in this template has been replaced with real content

[gemini-code-assist]

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.