feat: isolated code execution (IsolatedCodeRunner + backends) — v0.24.0

.github/workflows/isolation.yml

@@ -0,0 +1,39 @@
+name: Isolation integration
+
+# Runs the isolation backend integration tests (opt-in) against real runtimes.
+# The default matrix (test.yml) only covers the pure/unit tests; this job spins
+# up real Docker containers so the DockerBackend is verified end-to-end.
+
+on:
+  push:
+    branches: [main, develop, "feat/**"]
+    paths:
+      - "src/chuk_tool_processor/execution/isolation/**"
+      - "tests/execution/isolation/**"
+      - ".github/workflows/isolation.yml"
+  pull_request:
+    paths:
+      - "src/chuk_tool_processor/execution/isolation/**"
+      - "tests/execution/isolation/**"
+  workflow_dispatch:
+
+jobs:
+  integration:
+    runs-on: ubuntu-latest
+    env:
+      # Opt in to the backend integration tests (real Docker daemon on ubuntu).
+      CTP_TEST_ISOLATION_INTEGRATION: "1"
+    steps:
+      - uses: actions/checkout@v7
+      - uses: astral-sh/setup-uv@v8.3.2
+        with:
+          enable-cache: true
+          cache-dependency-glob: "uv.lock"
+      - run: uv python install 3.12
+      - run: uv sync --dev
+      # NB: bubblewrap is intentionally NOT installed here — GitHub runners block
+      # the netlink call bwrap uses to bring up loopback in a new net namespace
+      # ("RTM_NEWADDR: Operation not permitted"), so its integration test can't
+      # run here. It requires a real Linux host with unprivileged user namespaces.
+      - name: Isolation tests (Docker integration enabled)
+        run: uv run pytest tests/execution/isolation/ -v -o addopts=""

.pre-commit-config.yaml

@@ -13,7 +13,7 @@ repos:
       - id: detect-private-key
 
   - repo: https://github.com/astral-sh/ruff-pre-commit
-    rev: v0.7.1
+    rev: v0.15.21
     hooks:
       - id: ruff
         args: [--fix]

CHANGELOG.md

@@ -3,6 +3,34 @@
 All notable changes to this project are documented here. This project follows
 [Semantic Versioning](https://semver.org/).
 
+## [0.24.0]
+
+### Added
+
+- **`IsolatedCodeRunner`** — runs untrusted / LLM-generated code behind a real
+  OS/runtime boundary, with tool access brokered back to the host over a single
+  audited channel (JSON, never pickle). This is the safe counterpart to
+  `CodeSandbox`; see `docs/isolated_execution.md`.
+- Isolation backends behind a common `IsolationBackend` protocol:
+  `SeatbeltBackend` (macOS `sandbox-exec`), `DockerBackend` (throwaway
+  container), `BubblewrapBackend` (Linux namespaces), and `LocalProcessBackend`
+  (no isolation; dev/testing only — the runner refuses it unless
+  `allow_no_isolation=True`).
+- `IsolationLimits`, `IsolatedResult`, and the `IsolationBackend` protocol,
+  exported from `chuk_tool_processor.execution.isolation`.
+
+### Changed
+
+- Documentation now routes untrusted / LLM-generated code to
+  `IsolatedCodeRunner`, and no longer presents the subprocess `IsolatedStrategy`
+  as a security boundary — it provides crash/fault isolation for tool dispatch,
+  not isolation of an orchestration code string.
+
+### Notes
+
+- Experimental Windows (AppContainer) and WASM backends live on separate
+  branches and are not part of this release.
+
 ## [0.23.0]
 
 ### Security

README.md

@@ -58,7 +58,7 @@ Parsers (XML / OpenAI / JSON)
    Execution Strategy
    ┌──────────────────────┐
    │ • InProcess          │  ← Fast, trusted
-   │ • Isolated/Subprocess│  ← Safe, untrusted
+   │ • Isolated/Subprocess│  ← Crash isolation
    │ • Remote via MCP     │  ← Distributed
    └──────────────────────┘
 ```
@@ -160,7 +160,7 @@ results = await processor.process(json_output)
 | **Pattern Bulkheads** | Glob patterns like `"db.*": 3` for grouped concurrency limits |
 | **Scoped Registries** | Isolated registries for multi-tenant apps and testing |
 | **ExecutionContext** | Request-scoped metadata propagation (user, tenant, tracing, deadlines) |
-| **Isolated Strategy** | Subprocess execution for untrusted code (zero crash blast radius) |
+| **Isolated Strategy** | Subprocess tool execution for crash/fault isolation (zero crash blast radius). Not a security boundary — for untrusted/LLM *code*, see [`IsolatedCodeRunner`](docs/isolated_execution.md) |
 | **Redis Registry** | Distributed tool registry for multi-process/multi-machine deployments |
 
 ### Advanced Scheduling
@@ -573,6 +573,8 @@ See [ERRORS.md](docs/ERRORS.md) for complete error taxonomy.
 | [**GUARDS.md**](docs/GUARDS.md) | Runtime guards for safety, validation, and resource management |
 | [**MCP_INTEGRATION.md**](docs/MCP_INTEGRATION.md) | HTTP Streamable, STDIO, SSE, OAuth, Middleware Stack |
 | [**ADVANCED_TOPICS.md**](docs/ADVANCED_TOPICS.md) | Deferred loading, code sandbox, isolated strategy, testing |
+| [**isolated_execution.md**](docs/isolated_execution.md) | Running untrusted/LLM code behind a real boundary (`IsolatedCodeRunner`, backends) |
+| [**security.md**](docs/security.md) | Security model: `CodeSandbox` vs real isolation, running untrusted code safely |
 | [**CONFIGURATION.md**](docs/CONFIGURATION.md) | All config options and environment variables |
 | [**OBSERVABILITY.md**](docs/OBSERVABILITY.md) | OpenTelemetry, Prometheus, metrics reference |
 | [**ERRORS.md**](docs/ERRORS.md) | Error codes and handling patterns |
@@ -663,7 +665,7 @@ pip install chuk-tool-processor[all]
 **Use CHUK Tool Processor when:**
 - Your LLM calls tools or APIs
 - You need retries, timeouts, caching, or rate limits
-- You need to run untrusted tools safely
+- You need crash isolation for flaky tools, or a real boundary for untrusted/LLM code ([`IsolatedCodeRunner`](docs/isolated_execution.md))
 - Your tools are local or remote (MCP)
 - You need multi-tenant isolation
 - You want production-grade observability

docs/ADVANCED_TOPICS.md

@@ -184,7 +184,16 @@ See `examples/code_sandbox_demo.py` and `examples/advanced_tool_use_math_server.
 
 ## Using Isolated Strategy
 
-Use `IsolatedStrategy` when running untrusted, third-party, or potentially unsafe code that shouldn't share the same process as your main app.
+Use `IsolatedStrategy` for **crash/fault isolation** of tool execution — it runs each registered **tool call** in a separate worker process so a hanging or crashing tool can't take down your app.
+
+> [!WARNING]
+> `IsolatedStrategy` is **not a security boundary**. Workers run as the same OS
+> user with no seccomp/namespace/rlimit confinement, and tool arguments/results
+> cross the boundary via **pickle**. It also only governs how *registered tool
+> calls* are dispatched — it never executes an orchestration **code string**. For
+> running untrusted or LLM-generated *code*, use
+> [`IsolatedCodeRunner`](./isolated_execution.md) (real OS/container/WASM
+> isolation with brokered tool access), not this strategy and not `CodeSandbox`.
 
 ```python
 import asyncio
@@ -206,25 +215,28 @@ async def main():
 asyncio.run(main())
 ```
 
-### Security & Isolation — Threat Model
+### What IsolatedStrategy actually protects against
 
 | Aspect | Protection |
 |--------|------------|
-| **Process Isolation** | Untrusted code runs in subprocesses |
-| **Crash Blast Radius** | Zero — faults don't bring down your app |
-| **Resource Limits** | Use containers with `--cpus`, `--memory` |
-| **Network Isolation** | Egress filtering via container network policy |
+| **Crash Blast Radius** | Zero — a crashing/hanging tool doesn't bring down your app |
+| **Process separation** | Each tool call runs in a separate worker process (fault, not security, isolation) |
+| **Timeouts** | Per-call deadlines terminate stuck workers |
 | **Secrets** | Never injected by default — pass explicitly |
 
+This is **fault isolation, not a security sandbox** — see the warning above. For a
+real security boundary around untrusted code, use
+[`IsolatedCodeRunner`](./isolated_execution.md), optionally with a container/gVisor
+runtime for `--cpus`/`--memory`/egress limits.
+
 ### When to Use Each Strategy
 
-| Scenario | Strategy |
-|----------|----------|
-| Trusted internal tools | InProcessStrategy |
-| External/user-provided code | IsolatedStrategy |
-| LLM-generated code execution | IsolatedStrategy |
-| Performance-critical path | InProcessStrategy |
-| Tools that might crash | IsolatedStrategy |
+| Scenario | Use |
+|----------|-----|
+| Trusted internal tools | `InProcessStrategy` |
+| Performance-critical path | `InProcessStrategy` |
+| Tools that might crash or hang | `IsolatedStrategy` (crash isolation) |
+| Untrusted / third-party / LLM-generated **code** | [`IsolatedCodeRunner`](./isolated_execution.md) (real isolation) — *not* `IsolatedStrategy`, *not* `CodeSandbox` |
 
 ---
 

docs/CORE_CONCEPTS.md

@@ -93,8 +93,13 @@ class SearchTool:
 
 | Strategy | Use Case | Trade-offs |
 |----------|----------|------------|
-| **InProcessStrategy** | Fast, trusted tools | Speed ✅, Isolation ❌ |
-| **IsolatedStrategy** | Untrusted or risky code | Isolation ✅, Speed ❌ |
+| **InProcessStrategy** | Fast, trusted tools | Speed ✅, Crash isolation ❌ |
+| **IsolatedStrategy** | Tools that may crash/hang | Crash isolation ✅, Speed ❌ |
+
+> `IsolatedStrategy` gives **crash/fault isolation** (separate worker processes),
+> **not** a security boundary — workers are the same user and results cross via
+> pickle. For untrusted or LLM-generated *code*, use
+> [`IsolatedCodeRunner`](./isolated_execution.md), not a strategy.
 
 ### Parallel Execution
 
@@ -125,21 +130,20 @@ async def main():
         )
     )
     async with processor:
-        # Tools run in separate subprocesses (safe)
+        # Each tool call runs in a separate worker process (crash isolation)
         results = await processor.process(tool_calls)
 ```
 
-> **Note:** `IsolatedStrategy` is an alias of `SubprocessStrategy` for backwards compatibility. Use `IsolatedStrategy` for clarity—it better communicates the security boundary intent.
+> **Note:** `IsolatedStrategy` is an alias of `SubprocessStrategy` for backwards compatibility. It provides crash/fault isolation for tool dispatch, **not** a security sandbox; for untrusted/LLM code use [`IsolatedCodeRunner`](./isolated_execution.md).
 
 ### When to Use Each Strategy
 
-| Scenario | Recommended Strategy |
-|----------|---------------------|
-| Trusted internal tools | InProcessStrategy |
-| External/user-provided code | IsolatedStrategy |
-| LLM-generated code execution | IsolatedStrategy |
-| Performance-critical path | InProcessStrategy |
-| Tools that might crash | IsolatedStrategy |
+| Scenario | Recommended |
+|----------|-------------|
+| Trusted internal tools | `InProcessStrategy` |
+| Performance-critical path | `InProcessStrategy` |
+| Tools that might crash or hang | `IsolatedStrategy` (crash isolation) |
+| Untrusted / third-party / LLM-generated **code** | [`IsolatedCodeRunner`](./isolated_execution.md) — real isolation, not a strategy |
 
 ---
 

docs/isolated_execution.md

@@ -0,0 +1,165 @@
+# Isolated Code Execution
+
+`IsolatedCodeRunner` runs **untrusted or LLM-generated** Python behind a real
+OS/runtime boundary, while still letting that code call your registered tools.
+It is the safe counterpart to
+[`CodeSandbox`](./programmatic_execution.md), which runs code in-process with no
+isolation and is **trusted-code-only** (see [security.md](./security.md)).
+
+- **`CodeSandbox`** — in-process `exec()`, no boundary. Only for code you wrote.
+- **`IsolatedCodeRunner`** — code runs inside a container / macOS Seatbelt /
+  Linux bubblewrap sandbox; tools are brokered back to the host over one audited
+  channel. For code you did **not** write. (A WASM backend is in development on a
+  separate branch.)
+
+## Why not just reuse the subprocess strategy?
+
+The existing `IsolatedStrategy` (`subprocess_strategy.py`) runs *registered tool
+calls* in a `ProcessPoolExecutor` using **pickle**. That is fault isolation, not
+security isolation: same OS user, no seccomp/rlimits/namespaces, and unpickling
+data across the boundary is itself unsafe. It never executes the orchestration
+code string. `IsolatedCodeRunner` is a separate mechanism built for untrusted
+*code*.
+
+## Architecture
+
+The hard, backend-independent part is the **tool bridge**: untrusted code must
+reach host tools (which hold real credentials) without any other host access.
+
+```
+host process (trusted)                     isolated guest (untrusted)
+┌───────────────────────────┐              ┌──────────────────────────┐
+│ IsolatedCodeRunner        │              │ guest_bootstrap.py       │
+│  ├─ owns the registry     │              │  ├─ exec(user code)      │
+│  ├─ ToolBroker (RPC srv)  │◄──JSON RPC───┤  └─ async tool proxies ──┼─┐
+│  │   • list_tools()       │  1 unix fd   │      call_tool(name,kw)  │ │
+│  │   • call_tool() ───────┼─ runs REAL   └──────────────────────────┘ │
+│  │     (token+allowlist)  │  tool here          ▲                      │
+│  └─ IsolationBackend ─────┼─ spawns guest ──────┘                      │
+└───────────────────────────┘   with limits; nothing else crosses ◄─────┘
+```
+
+Invariants:
+
+- **The broker channel is the only hole.** Network, filesystem, and host
+  processes are denied by the backend; the guest can only reach the socket.
+- **JSON on the wire, never pickle.** The guest is untrusted; unpickling
+  guest-controlled bytes on the host would defeat the whole exercise.
+- **Policy is enforced host-side.** The per-run token, the tool allowlist, and
+  the `max_tool_calls` ceiling live in `ToolBroker`, not in the guest.
+- **The return value is untrusted data.** `IsolatedResult.value` is JSON
+  produced by untrusted code — validate before acting on it.
+
+## Quick start
+
+```python
+from chuk_tool_processor.execution.isolation import (
+    IsolatedCodeRunner, DockerBackend, IsolationLimits,
+)
+
+runner = IsolatedCodeRunner(
+    DockerBackend(),                       # or SeatbeltBackend(), BubblewrapBackend()
+    namespace="math",                      # tools the guest may call
+    limits=IsolationLimits(wall_timeout=30.0, allow_network=False),
+)
+
+result = await runner.run("""
+total = 0
+for i in range(1, 6):
+    r = await add(a=str(total), b=str(i))   # 'add' is a brokered host tool
+    total = r["sum"]
+return total
+""")
+
+print(result.ok, result.value, result.tool_calls)   # True 15 5
+```
+
+Pick the backend that matches where you deploy; the runner refuses a
+non-isolating backend unless you pass `allow_no_isolation=True`.
+
+## Backends
+
+| Backend | Isolation | Platform | Needs | Notes |
+|---|---|---|---|---|
+| `DockerBackend` | Strong§ | Linux Docker host | `docker`/`podman` CLI + daemon | throwaway container, `--network none`, read-only root, dropped caps, runs as host uid |
+| `SeatbeltBackend` | Strong* | macOS | `sandbox-exec` (built in) | no inet, no fs-writes outside work/tmp, secret dirs unreadable |
+| `BubblewrapBackend` | Strong¶ | Linux | `bwrap` binary | user/mount/pid/net namespaces |
+| `LocalProcessBackend` | **None** | any | — | dev/testing only; runner refuses it without `allow_no_isolation=True` |
+
+§ `DockerBackend` runs each guest in a throwaway `docker run --rm` container
+(pre-pulled image + `--pull never`, `--network none`, read-only root, `--cap-drop
+ALL`, memory/pids limits) **as the host uid**, and is CI-verified end-to-end on
+native Linux. The host↔guest broker uses a bind-mounted unix socket, which the
+VM-based file sharing in **Docker Desktop / podman-machine (macOS, Windows)**
+does not support (`connect()` returns `ENOTSUP`) — run it on a native Linux
+Docker host (servers, CI, WSL2).
+
+¶ `BubblewrapBackend` requires a Linux host with **unprivileged user namespaces**
+enabled. It is not exercised in GitHub CI because the runners block the netlink
+call `bwrap` uses to bring up loopback in a fresh network namespace
+(`RTM_NEWADDR: Operation not permitted`); verify it on a real Linux host.
+
+\* Seatbelt reliably blocks network and filesystem *writes*; read confinement is
+best-effort (broad reads with known secret dirs denied) because a strict read
+allowlist aborts CPython. The denied secret paths are configurable —
+`SeatbeltBackend(deny_read_paths=..., add_deny_read_paths=...)` — defaulting to
+`DEFAULT_DENY_READ_PATHS` (`~/.ssh`, `~/.aws`, cloud creds, keychains, …).
+`sandbox-exec` is deprecated by Apple but functional.
+
+A **WASM backend** (wasmtime/WASI — the strongest boundary by construction) is in
+development on a separate branch; it is not part of this release.
+
+Install notes: the Docker, Seatbelt, and bubblewrap backends need no Python
+dependencies (they shell out to the respective binary).
+
+## Resource limits
+
+`IsolationLimits` (all enforced as far as the backend allows):
+
+| Field | Default | Meaning |
+|---|---|---|
+| `wall_timeout` | 30s | hard wall-clock kill (always enforced) |
+| `cpu_timeout` | 15s | CPU-seconds ceiling (RLIMIT_CPU / container) |
+| `memory_bytes` | 256 MiB | memory ceiling (RLIMIT_AS / `--memory`) |
+| `max_output_bytes` | 64 KiB | captured stdout/stderr cap |
+| `max_tool_calls` | 100 | broker rejects calls beyond this |
+| `max_processes` | 64 | RLIMIT_NPROC / `--pids-limit` |
+| `allow_network` | `False` | deny all guest network except the broker channel |
+
+## Security model
+
+What the boundary is expected to stop, and where enforced:
+
+- **Arbitrary host code execution / sandbox escape** → the backend (container,
+  namespace, or Seatbelt). Even a full `CodeSandbox`-style `__subclasses__()`
+  escape only reaches the *guest's* interpreter, which has no host access beyond
+  the broker socket.
+- **Reaching tools you didn't expose** → `ToolBroker` allowlist + namespace.
+- **Tool-call flooding** → `max_tool_calls`.
+- **Network exfiltration** → `allow_network=False` (default).
+- **Reading host secrets / writing host files** → backend filesystem policy.
+- **Runaway CPU/memory/fork bombs** → limits (`wall_timeout`, `cpu_timeout`,
+  `memory_bytes`, `max_processes`).
+
+Residual risks: the broker still runs *your* tools with their real privileges on
+the guest's behalf — expose only tools that are safe to call with
+attacker-chosen arguments. Seatbelt read-confinement is best-effort. The guest's
+return value is untrusted.
+
+## Writing a custom backend
+
+Implement the `IsolationBackend` protocol:
+
+```python
+class MyBackend:
+    name = "mybackend"
+    provides_isolation = True            # False => runner requires allow_no_isolation
+
+    def is_available(self) -> bool: ...
+    async def run_guest(self, job, *, host_socket_path) -> GuestOutcome: ...
+```
+
+Most OS-level backends should subclass `SubprocessBackend` and override just
+`_wrapper_argv()` (the sandbox launcher prefix) and, if paths are remapped,
+`_guest_ctx()` — see `DockerBackend` for the remapping pattern.
+```