Add a container primitive built on namespaces and cgroups v2 - #2
Open
erichanwang wants to merge 6 commits into
Open
Add a container primitive built on namespaces and cgroups v2#2erichanwang wants to merge 6 commits into
erichanwang wants to merge 6 commits into
Conversation
neuros-container run puts a command in a real cgroup v2 leaf and, when privilege allows, a fresh mount/UTS/PID/IPC namespace set, without runc, containerd, or libcontainer underneath. Resource limits are plain cgroup files (memory.max, pids.max, cpu.weight) written by hand. The interesting part turned out to be cgroup placement, not the limit values: a cgroup already holding member processes can't enable subtree_control for children (the "no internal process" rule), and an ordinary interactive shell's own cgroup is exactly such a cgroup. The tool walks up to the nearest ancestor that already delegates the wanted controller and nests the new leaf there instead, which is what lets this run without root from a normal terminal. Namespace isolation still needs root or an unprivileged user namespace; Ubuntu 24.04's default AppArmor policy blocks the latter for unconfined processes, so on a stock install this degrades to cgroup-only limiting and says so on stderr rather than pretending to sandbox anything. tests/test_container.py exercises this for real: a 16M memory.max cgroup holding a process that touches 200MB of bytearray keeps memory.current at the 16M ceiling, and a pids.max=4 cgroup stops a 20-iteration fork loop after 3 children.
lb config was missing --mode ubuntu, so live-build defaulted to Debian's mirrors even with --distribution noble set. Debian has no noble suite, so a build from a clean checkout failed at bootstrap - a more basic problem than the missing version pins. Also fixes also-utils -> alsa-utils, which is not a real package name and would fail regardless. Nothing was pinned, so "reproducible" was not true: oh-my-zsh tracked master, GNOME extensions used mutable tags, and the package list had no versions. Pin oh-my-zsh to a commit, the Ollama installer to the v0.32.5 tag SHA, GNOME tarballs to commit SHAs, and VS Code plus Continue.dev to explicit versions. For the ~60 apt packages, point build-time mirrors at a snapshot.ubuntu.com timestamp rather than pinning each one, which is brittle when the archive rotates. --mirror-binary deliberately stays on the live archive so installed systems still receive security updates. This gives pinned-input reproducibility - same commit, same software versions. Not bit-identical ISOs: timestamps, file ordering, and initramfs generation are unaddressed, and the README now says so rather than implying more. Also still unpinned: ollama pull mistral resolves a floating tag, which would need a ~4GB pull to verify against a manifest digest. neuros-mcp spoke HTTP+JSON-RPC only, which no MCP client actually uses. Add a stdio transport, verified against the official mcp Python SDK - initialize, list_tools, and call_tool over a real handshake, not an assumed one. Add scripts/check-airgap.sh, which fails on any external URL outside a reviewed allowlist, and wire it into CI along with test_mcp.py - both existed but were never actually run. The build-iso job now runs a real lb config and resolves the package list against the pinned snapshot instead of only checking that files exist.
…pecified them --json was meant to be an alternative single-line output mode (per the tool's own docstring) but cmd_verify printed the grep-friendly text lines unconditionally, so --json emitted text followed by JSON and callers parsing that stream as pure JSON would fail. --quiet was also silencing the aggregate line, not just the per-file lines, so a quiet run gave no overall PASS/FAIL at all. Fixed cmd_verify to gate text output on both flags correctly. Two of the four failing tests were themselves wrong: test_multi_file_and_merge asserted the literal substring "verdict=FAIL\n", which the aggregate line's documented format (verdict=... source=aggregate ...) can never produce since more fields always follow on the same line -- checked the last line's prefix instead. test_missing_script_raises assumed setting only NEUROS_POLICY_SCRIPT to a bad path was enough to force a lookup failure, but the resolver's next candidate (sibling neuros-policy next to neuros-verify) is a real file in this checkout, so it always succeeded -- now mocks os.path.isfile so every candidate misses, matching what the test actually wants to exercise. Also close the air-gap guard's biggest blind spot: check-airgap.sh only greps for literal http(s):// URLs, so a shipped tool that imports a third-party HTTP client (requests/httpx) and builds its target URL at runtime is completely invisible to it, even though that's the easiest way to add a real outbound call. Added an import-statement check for those two libraries; verified zero false positives against the current tree (no shipped tool uses them, unlike stdlib urllib which is legitimately used everywhere for local Ollama calls) and that it does catch an injected `import requests`.
All 13 tests/test_*.py files pass individually and together (383 tests,
0 failures via unittest discover), so CI now runs discover instead of
listing test_nn.py/test_mcp.py by name -- every current and future
test file gates merges without another wiring PR.
check-airgap.sh gaps closed:
- Scans were -maxdepth 1, invisible to anything in a subdirectory of
the tools path. Now recursive (excluding __pycache__).
- neuros-network's socket.create_connection(("1.1.1.1", 53)) and its
nslookup subprocess call contain no http(s):// literal, so the
existing URL scan couldn't see them. Added a raw-socket/DNS-tool
scan and allowlisted the two known-intentional neuros-network calls
explicitly (1.1.1.1, google.com) instead of leaving them undetected.
- Investigated removing the neuros-speak:github.com allowlist entry
(per the stated goal that it only covers a URL printed in a help
message, never fetched) -- confirmed removing it currently breaks
the check, since the literal URL scan has no existing skip for
"URL that only appears in a print statement". Left in place rather
than silently reintroducing a false positive.
Verified zero false positives across all 86 shipped tools after each
change, and that the new detections still catch injected violations
(raw socket + nslookup to non-allowlisted hosts, in a nested subdir).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016towgcp2NVGxUBbB9Tmpv4
…s, pip A prior commit (2e4080a) pinned oh-my-zsh, the Ollama installer, GNOME extension tarballs, VS Code/Continue.dev, and the apt snapshot mirror -- verified all of those still resolve. This closes what a full grep for curl/wget/git-clone/pip/npm/Dockerfile fetches turned up beyond that: - Dockerfile: FROM ubuntu:24.04 pinned to its image digest. The bare tag is mutable -- Canonical rebuilds it in place for security patches -- so the Dockerized build container's own starting filesystem was still floating. `pip3 install requests` pinned to ==2.34.2. - .github/workflows/ci.yml: actions/checkout@v4 (x4) and docker/setup-buildx-action@v3 pinned to the commit SHA each tag currently resolves to, so CI can't silently start running different action code after an upstream tag move. `ollama pull mistral` is still unpinned, documented in README. New finding: the manifest digest is cheap to look up (registry.ollama.ai's manifest endpoint returns <1KB, not the 4GB blob) -- the original assumption that pinning required a full pull was wrong. But Ollama's pull path has no documented digest-pin syntax the way `docker pull` does, so there's still no way to verify a pinned reference would work without the 4GB pull this environment intentionally avoids. Added a comment at the call site with the exact finding for whoever picks this up with a real Ollama environment. config/package-lists/neuros.list.chroot intentionally left alone -- already covered by the snapshot.ubuntu.com pin in build.sh, and pinning ~60 individual package versions on top of that would trade a working snapshot pin for a build that breaks the moment one exact version rotates off a mirror. Verified: bash -n on the edited hook and all other hook.chroot files, `docker build --check` (confirms Dockerfile syntax and that the pinned digest resolves against the registry), full validate-build.sh (249/249 pass), and scripts/check-airgap.sh. All new SHAs/versions/digests confirmed live via GitHub API, PyPI, and `docker buildx imagetools inspect` before committing. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W6ShnQA7UMNDtUL9DiBNnN
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
neuros-container run puts a command in a real cgroup v2 leaf and, when
privilege allows, a fresh mount/UTS/PID/IPC namespace set. No runc,
containerd, or libcontainer underneath; the limits are plain cgroup
files (memory.max, pids.max, cpu.weight) written by hand.
The interesting part was cgroup placement: a cgroup already holding
member processes can't enable subtree_control for children, and an
ordinary interactive shell's own cgroup is exactly such a cgroup. The
tool walks up to the nearest ancestor that already delegates the
wanted controller and nests there instead, which is what lets this run
from a normal terminal without root.
Namespace isolation needs root or an unprivileged user namespace;
Ubuntu 24.04's default AppArmor policy blocks the latter for
unconfined processes, so on a stock install this degrades to
cgroup-only limiting and says so on stderr.
Measured on this machine (see tests/test_container.py):
bytearray keeps memory.current at the 16M ceiling instead of growing
past it
Verified: python3 tests/test_nn.py test_autofix.py test_model.py
test_mcp.py test_container.py all green, ./validate-build.sh 249/249.