diff --git a/.agents/docs/2026-08-05-ecosystem-three-tier-and-composable-distro.md b/.agents/docs/2026-08-05-ecosystem-three-tier-and-composable-distro.md new file mode 100644 index 00000000..0023e7d1 --- /dev/null +++ b/.agents/docs/2026-08-05-ecosystem-three-tier-and-composable-distro.md @@ -0,0 +1,488 @@ +# xlings 生态三分层定位与可组合发行版:讨论纪要 + +**日期**: 2026-08-05 +**类型**: 讨论纪要(discussion memo)—— 综合 2026-08-05 会话讨论,作为后续逐条深入的锚点,不是策略也不是实施计划 +**触发**: 关于"xlings 生态究竟是什么、多 glibc 能不能做、可组合发行版如何表达、预构建 vs 源码构建如何取舍、消费者依赖硬钉能否用版本语义救"的连续追问 +**关联**: +- `2026-08-05-userspace-distro-hermetic-strategy.md`(运行时边界策略,本文的**下位配套**) +- `2026-05-22-subos-sandbox-gpu-passthrough.md`(sandbox 层 GPU 设备节点透传) +- `2026-06-26-multiarch-package-description-design.md`(payload 多架构表达) +- `2026-06-21-linux-root-usability-survey.md`(权限边界) +- xim-pkgindex `docs/V2/xpackage-spec.md`(新字段/能力的 probe-first 迁移范式) + +--- + +## 0. TL;DR + +**xlings 生态定位为三分层**: +- **kernel**(宿主提供)—— 只提供 syscall 边界 +- **xlings**(用户态发行版底座)—— subos 隔离 + xim-pkgindex 包管理 + 版本/binding 解析 +- **mcpp**(生态主构建工具)—— C/C++ 是核心域,**不限于** C/C++;其他语言按需包裹/集成 + +**分发采用预构建 + 源码构建混合架构**:命中预构建即取,不命中走 mcpp 源构;主流软件通过 xlings-res 预热覆盖长尾。 + +**多 glibc / 可组合发行版通过 platform manifest 承载**,不做"任意组合"(Nix 模式对 xlings 团队规模不现实),而是"用户从官方 platform 里选一份"(Flatpak runtime / Homebrew bottle 模式)。 + +**消费者依赖从字面量硬钉演进为三轴模型**: +- `deps` 范围(装机期选择器) +- `runtime_floor`(构件期物理烙定,不可变) +- `platform_membership`(coupled-ABI 群的集合成员,短路 range 解析) + +**与 hermetic 策略正交且互补**:hermetic 谈"运行时边界",本文谈"生态角色 + 分发架构"。合并后形成 xlings 生态的完整技术定位。 + +--- + +## 1. 起点与范围 + +会话由前一份 hermetic 策略延伸而来。hermetic 策略解决"什么 `.so` 能穿越到宿主",但一个更基础的问题从未明文写过:**xlings 究竟是什么?** 生态里的很多具体决策(compat.* 归属、xim-x-* 命名、pkgindex 与 mcpp-index 的分工、xlings-res 的作用、subos 与 mcpp sandbox 的关系)因为缺一份"我们是什么"的定位文档而反复重新推导。 + +本文的目标: +- **梳理**上述讨论,把散在多个 issue/PR/记忆里的决策浮到明面 +- **命名**已经在做的模式(用户态发行版、三分层、混合分发、platform 化),让后续 review 有共同锚 +- **枚举**已识别的阻塞项与开放问题,不给结论,留给后续逐条深入 + +本文**不是**策略、不是实施计划、不承诺时间表。任何进入代码的动作都需要独立的设计文档评审。 + +--- + +## 2. 三分层定位:kernel + xlings + mcpp + +### 2.1 分工与职责 + +``` +┌─────────────────────────────────────────────────────────┐ +│ kernel (host) │ +│ syscall, /dev/*, /proc/*, vendor kmod(如 nvidia.ko)│ +│ ↑ 通过 capabilities_host 白名单穿越 │ +├─────────────────────────────────────────────────────────┤ +│ xlings │ +│ subos 隔离层 (bwrap sandbox, 独立 sysroot) │ +│ xim-pkgindex 包管理 (recipe, xvm, elfpatch) │ +│ 版本/binding 解析 (xvm.add, xvm 节点绑定) │ +│ platform manifest (新: 承载"这个 subos 是哪个发行版") │ +├─────────────────────────────────────────────────────────┤ +│ mcpp │ +│ 生态主构建工具 │ +│ C/C++ 核心域: compile, link, test, canonical build │ +│ 构建 fingerprint、缓存、增量 │ +│ 其他语言按需包裹/集成 (见 §2.3 scope 修正) │ +├─────────────────────────────────────────────────────────┤ +│ 用户项目 / 应用 │ +└─────────────────────────────────────────────────────────┘ +``` + +**语义收敛**:每一层只对上一层负责,每一层解决一类问题。当前生态里的角色混淆(比如 recipe 里塞 build 逻辑、mcpp 反过来管 subos 状态)是**层次未分开**的症状。 + +### 2.2 参照系(定位判据的来源) + +| 生态 | Kernel | 用户态底座 | 包管理器 | 构建工具 | 分发形态 | +|---|---|---|---|---|---| +| Debian/Fedora | Linux | glibc + rootfs | apt/dnf | (每包各带) | 预构建为主 | +| NixOS | Linux(自建) | glibc(自建) | nix | nix build | binary cache 命中即取,不中现构 | +| Homebrew | XNU(宿主) | libSystem(宿主) | brew | (每 formula) | bottle(预构)+ 源构混合 | +| Gentoo | Linux(宿主) | glibc(自建) | portage | ebuild | 源构为主,binhost 兜底 | +| **xlings 提议** | Linux(宿主) | xlings subos + xim-pkgindex | xlings CLI | **mcpp** | **预构命中即取,不中走 mcpp 源构** | + +**最贴近的类比**:Homebrew 精神 + Nix 分层。 +- 从 Homebrew 继承:不接管 kernel、不自建 rootfs、host 只提供 syscall 边界(呼应 hermetic 策略 non-goals) +- 从 Nix 继承:用一个构建工具(nix build ↔ mcpp)承担源→二进制的正典转换,预构建是同一 recipe 的产物缓存,不是另一套并行的东西 +- **业界没有完美先例**——Homebrew 的 bottle/source 有微差(build flag 漂移),Nix 严格但要求自建整个 rootfs。用户态发行版 + host kernel + 单一构建正典的组合是空生态位 + +### 2.3 mcpp scope 定位(用户 2026-08-05 更正) + +**mcpp 是生态的主要构建工具,C/C++ 是核心域,不限于 C/C++。** + +原会话草案曾表述为"mcpp 是 C/C++ 构建规范,非 C/C++ 走各自 upstream build system"。这个表述过窄,已按用户更正修改。修正后的定位: + +- mcpp 首先服务 C/C++——现有能力、canonical build、fingerprint 机制、hermetic build 环境都以 C/C++ 为一等公民 +- mcpp **可以且应该**承担多语言场景下的**协调/包裹**角色——具体形态(是原生集成、是插件、还是 wrapper)是**开放问题 A** +- **不承诺 mcpp 亲自实现 cargo/npm/pip 的求解器/构建管线**;这类生态的求解和构建有其成熟工具,mcpp 侧更可能是"调用它们并统一 fingerprint/缓存/hermetic 边界" + +**为什么这个边界重要**:如果不明确 scope,mcpp 会被要求覆盖所有语言生态,scope 爆炸后没人能维护;如果把 mcpp 限死为 C/C++,又切断了"xlings 生态用一套工具"的定位价值。**折中是"C/C++ 一等公民 + 多语言协调层"**,协调层的具体形态另议。 + +### 2.4 subos 与 mcpp sandbox 的关系 + +- **subos**:运行 OS 实例(bwrap-based,独立 sysroot / xvm 状态 / 包安装目录) +- **mcpp sandbox**:构建时环境(bwrap-based,提供 hermetic 构建) + +现状:两者的 sandbox 实现应已复用。**长期方向**:mcpp build 应能直接消费"当前 subos"作为构建环境——subos 是 mcpp 的输入(工具链/依赖来源),构建产物落回 subos。这样 mcpp canonical build 天然遵循 subos 的 platform 约束。 + +**开放问题**:mcpp build 究竟应"消费 subos"(在 subos 里跑)还是"消费 platform 描述"(在临时环境里按 platform 组装工具链)?两者对 fingerprint 稳定性影响不同。留作开放问题 J 的子问题。 + +--- + +## 3. 可组合发行版:subos + platform manifest + +### 3.1 subos 是容器,platform manifest 是"是哪一款" + +subos 已经是"一个用户态 OS 实例"的物理位置。但**subos 现在没有整体身份**——每个包各自声明 `xim:glibc@2.39`,合起来隐式产生了一个"发行版",但没人拥有它、没人能整体替换它。 + +**可组合发行版 = subos + platform manifest**: + +``` +distro := subos + platform_manifest +platform_manifest := { + id: "xlings-2026.08-el9", + kernel_abi_floor: "3.10.0", + glibc: "2.39", + gcc: "16.1.0", + libstdcxx: "gcc@16.1.0", + binutils: "2.42", + linux_headers: "6.6", + ... +} +``` + +一个 subos 生成时选一份 platform,之后**subos 内所有包的依赖解析都以 platform 为锚**。用户"切换发行版" = "换 subos",不是"改配置"。 + +### 3.2 platform manifest 的形态(开放问题 B) + +至少三种承载形式,尚未定: + +**形式 1 —— 显式声明包**(platform 本身是 xim-pkgindex 的一个 xpkg): +```lua +-- pkgs/x/xlings-platform-2026.08.lua +package = { + name = "xlings-platform-2026.08", + type = "platform", + members = { + { "xim:glibc", "2.39" }, + { "xim:gcc", "16.1.0" }, + { "xim:binutils", "2.42" }, + { "xim:linux-headers", "6.6" }, + ... + }, + kernel_abi_floor = "3.10.0", +} +``` + +**形式 2 —— subos 元数据**(subos 创建时选一份,记在 subos 目录里): +```json +// $XLINGS_HOME/subos//platform.json +{ "id": "xlings-2026.08-el9", ... } +``` + +**形式 3 —— `.xlings.json` 用户侧**(用户项目声明依赖的 platform): +```json +{ "platform": "xlings-2026.08-el9" } +``` + +三者可以共存(索引侧发布定义 + subos 侧固化实例 + 项目侧声明需求),但**谁是权威源**、**谁与谁一致性校验**需要单独设计。留作开放问题 B。 + +### 3.3 为什么不是"任意组合"(Nix 模式) + +Nix 允许用户任意组合"glibc 2.31 + gcc 13.3 + llvm 20 + boost 1.85",代价是 nixpkgs 社区的巨大规模(数千 committer,构建集群,binary cache)。**xlings 团队规模是两个数量级之外**。 + +务实定位:**"用户从官方 platform 里选一份"**,不是"用户自由组合"。 +- 官方发布节奏:比如季度一份 platform(2026.05 / 2026.08 / 2026.11 / ...) +- 每份 platform 是**原子集合**:内部互测通过,一起 release +- 用户"自定义发行版" = "从官方 platform 库里选" + "在 platform 之上叠加应用层包" +- 极端情况下用户可以 fork platform recipe 自建,但那是社区扩展,不是核心承诺 + +**对齐 Flatpak runtime `//24.08` 模型**,也匹配 hermetic 策略 §5.3 P2 #8 的"Release 套装"表述。 + +--- + +## 4. 多 glibc:能力 vs 现实 + +### 4.1 xvm 数据模型:已经够 + +事实核对(xim-pkgindex): +- `glibc.lua` 有 `xvm_enable = true` 且每个 .so 通过 `xvm.add(lib, {version=glibc_version, ...})` 版本绑定注册 +- gcc 已有 5 个共存版本(9.4.0 / 11.5.0 / 13.3.0 / 15.1.0 / 16.1.0),llvm 有 2 个(20.1.7 / 22.1.8) +- **glibc 现在只有 2.39 是"没做",不是"做不到"** +- 每个 subos 有独立 `subos_sysrootdir()`,glibc payload 走 xvm 独立目录——存储上完全隔离 + +### 4.2 阻塞项(优先级排序) + +即使 xvm 模型够,**加第二份 glibc 到索引里,今天只会产生孤儿包**,因为: + +1. **消费者硬钉字面量**(最硬的阻塞) + - `pkgs/g/gcc.lua:35`: `"xim:glibc@2.39", "xim:binutils@2.42"` + - `pkgs/l/llvm.lua:27`: `"xim:glibc@2.39"` + - **所有 5 个 gcc 版本、2 个 llvm 版本全部锁 2.39**——加 glibc 2.28 索引里没消费者用 + +2. **elfpatch loader 单值** + - `glibc.exports.runtime.loader = "lib64/ld-linux-x86-64.so.2"` 无版本轴 + - 预设"当前 subos 只有一个 glibc"——PT_INTERP 该写哪份必须由 subos platform 决定,不是 recipe 决定 + +3. **sysroot include 平面** + - `glibc.lua:__config_header` 塞 130 项进 `subos_sysrootdir()/usr/include`,`declare_headers` first-claimant-wins + - 两份 glibc 会在同一平面 include 目录打架 + - 需要 `usr/include/glibc-2.28/` 版本分片 + gcc `--sysroot`/`-isysroot` 联动 + +4. **binding 是 pin 不是 range** + - 无法表达"我兼容 glibc >= 2.28" + - manylinux 那种"编译对老、运行对新"的复用能力现在无法表达 + +5. **libstdc++/libc++ 与 glibc 隐式耦合** + - gcc-15 的 libstdc++ 依赖新 glibc 符号 + - 换 glibc 就得换 gcc → 佐证"platform 是原子单位"的判断 + +6. **kernel syscall 版本地板** + - 最新 glibc 会隐式要求某个 kernel(2.39 要 3.2+,2.42 要 3.10+) + - platform 应显式声明 `kernel_abi_floor`,首装时探 `uname -r` 拒绝不合规宿主 + +**前三条不修,加第二份 glibc 到索引里就是纯技术债。** + +### 4.3 platform 化 = 多 glibc 的最小可行形式 + +多 glibc 应该表现为**多 platform**,而不是"任意 glibc 组合"。用户不定义"glibc 2.31 + gcc 13.3.0 + llvm 20"这种随意组合(nixpkgs 规模),而是从 xlings 官方发布的一组 platform 里选。 + +**渐进路径**: +1. 定义 `xlings-platform-2026.08 = {glibc=2.39, gcc=16.1.0, ...}`,让当前 recipe 全都隐式属这一个 platform。**零功能变化,只是给现状命名** +2. 补 recipe DSL 里的 `${platform.*}` 变量,仅新写 recipe 用 +3. 做第二份 platform 当作压力测试(如 `xlings-platform-legacy-el7 = {glibc=2.28, gcc=11.5.0, ...}`)——此时同时暴露阻塞项 1/2/3/6 +4. compat.mesa 等新 payload 一律标 `platform: 2026.08`,不试图跨 platform 复用 + +--- + +## 5. 分发架构:预构建 + 源码构建混合 + +### 5.1 三 tier 分层 + +用户提议的 "预构建不中就源构" 有一个例外:**底座层没有源构选项**(bootstrap 循环:构 glibc 需要 gcc,构 gcc 需要 glibc)。 + +``` +Tier 0 — Bootstrap 底座 (永远预构建,不允许源构) + glibc, binutils, gcc(至少一份能自举的), linux-headers, libstdc++ + → 就是 §3 的 xim-platform-YYYY.MM + → 份数由 xlings 官方发布节奏决定 (季度一份 × 3 arch × 3 年 ≈ 36 份底座) + +Tier 1 — 工具链 / 主流库 (优先预构建,可源构兜底) + llvm, cmake, ninja, boost, freetype, libpng, mesa, ... + → 预构命中率应 > 95% (受众都是主流版本) + → 不中时 mcpp 本地构 + +Tier 2 — 长尾 / 用户项目 (源构常态,预构增值) + 用户 mcpp 项目、compat.* 库、私域包 + → 预构建是可选优化 + → 组织可架私有 substituter 加速团队 +``` + +**关键性质**:Tier 0 的存在恰好是"xlings 是发行版"最硬的一条证据——发行版的定义就是"我发布一组同步过的底座包"。 + +### 5.2 Canonical build 与 fingerprint + +用户提议的"预构不中就源构"要 sound,必须解决"等价性"问题。 + +**Canonical build**:一个包在 mcpp 侧只有一套 build 描述(compile 命令 + flags + 依赖版本),预构建 tarball 与本机源构走**同一份 recipe**。这样: +- 预构 tarball = canonical build 的 CI 产物 +- 源构产物 = 用户机上跑同一 canonical build +- 两者行为等价(不承诺字节等价——那是未来目标) + +**xim-pkgindex 现状** vs **迁移目标**: +- 现状:100+ 个 recipe 的 `install()` 里塞满 build 逻辑(makefile 参数、cmake flags、post-build patch) +- 目标:recipe 只做 metadata,build 逻辑交给 mcpp canonical build 描述 +- **这是多年重构**。务实做法:**新包 mcpp-first,老包按需迁移**(参照 xpackage-spec V2 是 V1 严格超集的模式) + +**Fingerprint**:mcpp 已有 fingerprint 概念(memory 里 `mcpp fingerprint stale binary` 提到 `target//bin/xlings` 路径结构)。扩展方向: + +``` +fingerprint = hash( + source_tarball_sha256, + toolchain_version + toolchain_fp, # 当前 subos 的 gcc/glibc 版本 + build_flags, + target_platform, # §3 的 platform id + canonical_build_recipe_version +) +``` + +查询 xlings-res:`//.tar.gz` 存在 → 下载;不存在 → mcpp 本地构 → 落 `target//`。 + +**Fingerprint 定义的字段列表与稳定性**是开放问题 J。 + +### 5.3 "命中 = 相等"的定义 + +两级选项: +- **字节级重现**(reproducible builds):同源 + 同工具链 + 同 flags → bit-identical。Nix/Bazel 承诺。工程代价高,签名可从 hash 派生 +- **行为等价**(functional equivalence):跑起来一样,字节不必一样。apt/brew 现状。代价低,但预构与源构结果不同时 bug 只在其中一条路径复现,定位困难 + +**对 xlings 现状**:行为等价起步,字节重现作为未来目标。 + +### 5.4 迁移代价与现实约束 + +1. **本地源构对宿主要求上升**:用户机装 xlings 几百 MB,但源构 mcpp 项目要 gcc + headers + 磁盘 + CPU。教育场景下源构兜底可能是"永远打不开的开关"——**xlings-res 预构建覆盖率必须 >> 95%**。这不推翻架构,但决定了 **xlings-res 预构建 CI 是生态的关键基础设施**。开放问题 F +2. **Fingerprint 传染性**:一个包的 fingerprint 依赖工具链 fingerprint,工具链依赖 libc fingerprint。**libc 换一次,fingerprint 全部失效,tier 1 所有包需重构**。对策:libc/toolchain 变更走 platform 原子发布,平时不动 +3. **信任模型统一**:预构建来自 xlings-res(CA/signing 假设),源构建来自本机(信任 upstream sha256)。两者语义等价意味着"不介意从哪拿"——已经是 Nix binary cache 信任模型 + +--- + +## 6. 版本语义:三轴模型 + +### 6.1 现在的硬钉字符串同时承担四种语义 + +`"xim:glibc@2.39"` 现在暗含四件事,没显式拆开: + +| 语义 | 含义 | 生命周期 | +|---|---|---| +| **构建期符号地板** | "gcc 这份预构件是拿 glibc 2.39 的 headers/loader 编的" | 不可变,tarball 出厂就烙死 | +| **装机期选择器** | "装 gcc 时,给我把 glibc 2.39 也装上/找到" | 可由 policy 改 | +| **兼容承诺** | "我保证这份 gcc 在 glibc 2.39 上能跑" | 人工断言,断言错了用户炸 | +| **平台成员** | "我是 xlings-platform-2026.08 的一员" | 集合成员,不是比较 | + +**Semver 范围解决的是第 2 和第 3 类**。但 glibc 的向前兼容是**单向的**: +- 对 2.28 编译的二进制 → 在 2.39 上跑得动 +- 对 2.39 编译的二进制 → 在 2.28 上跑不动(缺 GLIBC_2.30+) + +写 `"xim:glibc@>=2.28"` 只有在**这份 tarball 真的是拿 2.28 编的**时才是真话。tarball 拿 2.39 编,再声明 `>=2.28`——resolver 满心欢喜给 2.28 环境,运行时炸——**范围声明与物理事实脱节比硬钉更危险**,因为默认"绿"实际"红"。 + +**对 glibc 这类的正确模型是最小版本地板,不是范围**。 + +### 6.2 拆开:三轴模型 + +```lua +package = { + name = "gcc", + version = "16.1.0", + + -- 装机期(resolver 看这个,可以是范围/平台变量) + deps = { + { "xim:glibc", version = ">=2.39", reason = "gcc 16 build-time floor" }, + { "xim:binutils", version = "^2.42", reason = "ld 兼容" }, + }, + + -- 构件期(不可变,elfpatch/loader-选择/CI hermetic 都读这里) + runtime_floor = { + loader = "glibc@2.39", -- PT_INTERP 烙定 + libc = "glibc@2.39", -- 编译时 headers + libstdcxx = "gcc@16.1.0", -- 自带 libstdc++.so.6 + }, + + -- 平台成员(若属某个原子发布集合) + platform_membership = "xlings-2026.08", +} +``` + +**关键性质**: +- `deps` 里的范围**只在 mcpp 源构或纯 header-only 包时是自由的**。对预构 tarball,`deps` 的范围下界必须 == `runtime_floor` 对应字段——**CI 强校验** +- `runtime_floor` 是 tarball 元数据,发布时确定,**从此再不能改**——就像 PT_INTERP 一样物理烙定 +- `platform_membership` 存在时,resolver **短路**——直接锁 platform 定义的版本,不走 `deps` 范围解析。这条路径吃掉 80% 的现实场景 + +### 6.3 按包类型分级采用 + +| 包类型 | 举例 | 版本语义 | 理由 | +|---|---|---|---| +| **coupled-ABI 底座** | glibc, gcc, libstdc++, libc++ | 只用 platform 成员 | 三者符号 ABI 相互烙定,任意范围都是幻觉 | +| **系统运行时** | libpng, freetype, openssl | 最小版本地板 (`>=X.Y`) | 通常向前兼容,反向不成立;上界几乎不需要 | +| **纯 header-only C/C++ 库** | fmt, spdlog, catch2 | 完整语义范围 (`^X.Y.Z`) | 无 ABI,只在 build-time 解析 | +| **应用层预构件** | gh, ollama, godot | 单版本 pin (现状) | 上游各自发一个,没有多版本选择必要 | +| **mcpp 源构包** (未来) | 用户项目、compat.mesa 源版 | 完整语义范围 | 源构环境已知,范围能真被 solver 利用 | + +**推论**:"引入 semver 范围"不是全局决策,是按类型渐进的政策。 + +### 6.4 Silent-success 陷阱 + +xlings 生态踩过多次"从不发生和已经成功产生一样的日志"(memory: silent-success pattern)。版本范围的天然坑就是这一族: +- 声明 `>=2.28`,预构建实际拿 2.39 编——CI 里全都是 2.39,永远绿 +- 用户在 2.28 环境装,报 `GLIBC_2.30 not found`——错误信息不指向 recipe 的 range 声明,而是指向 loader + +**防坑必须的两件事**: +1. `runtime_floor` 与 `deps` 下界一致性,**CI 强校验**——范围声明与物理事实必须联动 +2. 范围声明的**每个"点"都必须 CI 覆盖**——`>=2.28, <3.0` 意味着 CI 至少跑过 2.28/2.35/2.39 三个采样点的 install + smoke test + +只做第一件,范围诚实但受限;只做第二件,CI 矩阵爆炸。**两件必须同时做**,而**平台成员机制**恰好把 CI 矩阵砍下来:成员天然把"跑几个采样点"变成"跑几个 platform",数量可控。CI 采样点矩阵设计是开放问题 H。 + +### 6.5 迁移(V2 spec probe pattern 复用) + +xpackage-spec V2 §"Adopting a capability" 已写清:新字段/新语法必须通过 probe 让老 client 走 legacy 分支,不能靠 `min_xlings` 挡门。三轴模型同样适用: + +```lua +if pkgindex.version_constraints then + deps = { { "xim:glibc", version = ">=2.39" } } +else + deps = { "xim:glibc@2.39" } -- 老 client 退化到硬钉最低点 +end +``` + +代价:每次 range 变更都要维护 legacy 侧硬钉,直到 dropping 老 client 那天。 + +--- + +## 7. 与 hermetic 策略的关系 + +正交、互补: + +| | hermetic 策略 | 三分层 + 分发架构 | +|---|---|---| +| **谈的是** | 运行时**边界**(什么 `.so` 能穿越到宿主) | 生态**角色** + 分发**架构** | +| **主要产物** | `capabilities_host` 元数据 + bwrap 空 host CI | mcpp canonical build + fingerprint 分发协议 + platform manifest | +| **单独价值** | 收敛"跨越宿主"的口径 | 收敛"我们是什么 + 如何来到用户机器上"的口径 | +| **联动价值** | mcpp 本地构出的二进制**天然遵循 hermetic**(build 环境本身 hermetic);预构建 tarball CI 用 bwrap 空 host 验证 hermetic 合规 | + +**建议**:把三分层定位写成上位文档,hermetic 策略作为其下位的**运行时边界章节**;分发架构作为下位的**分发章节**。合并后生态里所有决策(新增包、新增依赖、新增 CI check)都能找到锚。 + +--- + +## 8. 阻塞项汇总(优先级排序) + +从"要做多 glibc / 可组合发行版"角度倒推,阻塞项排序: + +| # | 阻塞项 | 影响 | 相关章节 | +|---|---|---|---| +| 1 | 消费者硬钉字面量 | 加第二份 glibc 无人使用 | §4.2 | +| 2 | 无 platform manifest 概念 | subos 无整体身份,无从"选一份" | §3 | +| 3 | elfpatch loader 单值 | PT_INTERP 无法按 subos platform 选择 | §4.2 | +| 4 | sysroot include 平面 | 多 glibc headers 打架 | §4.2 | +| 5 | recipe 里塞 build 逻辑 | 无法做 canonical build → 预构/源构等价性无从谈起 | §5.2 | +| 6 | 无 fingerprint 分发协议 | "命中即取,不中现构"无判定 | §5.2 | +| 7 | binding 是 pin 不是 range | 无法表达跨版本兼容 | §6 | +| 8 | 无 CI 一致性校验(range vs runtime_floor) | Silent-success 陷阱 | §6.4 | +| 9 | libstdc++ 与 glibc 隐式耦合无声明 | 换任一即换另一,但没人显式表达这个约束 | §4.2 §6.3 | +| 10 | kernel syscall 地板无声明 | 最新 glibc 隐式要求某 kernel,无人校验 | §4.2 | + +--- + +## 9. 最小可行下一步(仅建议顺序,不承诺时间) + +1. **写定位文档**(本文的继任者):把三分层 + hermetic 合成一份 `docs/design/ecosystem-positioning.md`。**零代码,当天可做** +2. **命名现状为 platform-2026.08**:定义 `xlings-platform-2026.08 = {现有 gcc/glibc/binutils/...}`,让当前 recipe 隐式属这份 platform。**零功能变化,零迁移** +3. **加 `runtime_floor` 字段**(spec V3):走 V2 spec 的 probe pattern 引入,recipe 侧填充,尚不强制。**逐包填,老 client 兼容** +4. **加 `deps` range 语法**(spec V3):同上,同样 probe 引入。CI 强校验 `runtime_floor` 与 `deps` 下界一致性 +5. **mcpp canonical build 探索**:选 1~2 个包做 pilot(不要选 glibc/gcc 这种硬骨头,选 libpng/fmt 这类简单库),验证 fingerprint 稳定性 + 预构/源构等价性 +6. **第二份 platform 作为压力测试**:当上面 5 步走通,做一份 `xlings-platform-legacy-el7`,一次性暴露多 glibc 剩余阻塞项(elfpatch / sysroot / libstdc++ 耦合) +7. **compat.mesa 等 hermetic 策略新 payload**:严格标 `platform_membership`,不跨 platform 复用 + +--- + +## 10. 开放问题(留待后续逐条讨论) + +- **A. mcpp scope 到底延伸到多远?** C/C++ 一等公民 + 多语言协调层——协调层是原生集成、插件、还是 wrapper?对 cargo/npm/pip 的态度分别是什么? +- **B. platform manifest 的 CLI/配置形态**:形式 1(xpkg)/形式 2(subos 元数据)/形式 3(`.xlings.json`)如何选?权威源是谁?一致性校验谁做? +- **C. compat.* vs xim-x-* 命名归属**:上一份分析建议归 xim-pkgindex,但保留 `compat.` 前缀会打破现有命名分层 +- **D. macOS / Windows 上 hermetic 与 platform 的对应**:macOS 有 system frameworks 与 libSystem,Windows 有 UCRT。platform 概念如何跨平台一致? +- **E. xlings-res 存储 / 带宽预算**:多 platform × 多 arch × tier 1 全预构 → 存储和 CDN 成本量级? +- **F. 源构建 UX 在教育场景的可行性**:mcpp 本地构对宿主 CPU/磁盘/网络的最低要求,教育场景能否兜底? +- **G. 老 recipe 迁移政策**:canonical build 迁移是强制(deadline)还是渐进(新包新政策)? +- **H. Range 的 CI 采样点矩阵设计**:每个 `>=X.Y` 声明至少覆盖哪几个点?矩阵如何砍? +- **I. Bootstrap 层如何独立更新**:比如 glibc 出安全补丁 2.39.1,能不能不动 platform 集合的其它成员单独发? +- **J. Fingerprint 定义的字段列表与稳定性**:哪些字段进 fingerprint?工具链版本进入的话,platform 微调 fingerprint 全炸,如何权衡? +- **K. mcpp 消费 subos vs 消费 platform 描述**:mcpp build 究竟在 subos 内跑,还是在临时环境按 platform 组装工具链? +- **L. 私有 substituter 支持**:组织内的私域预构建缓存,如何在信任模型 + fingerprint 协议里表达? +- **M. platform 弃用政策**:一份 platform release 后多久 EOL?EOL 后旧 subos 是不可迁移的? + +--- + +## 11. 决策未定与假设标注 + +本文所有具体形态(字段名、目录布局、CLI 语法)都是**示例**,不是决策。真正的决策在后续 §10 各开放问题单独讨论时才作出。 + +**已经作出的假设(可讨论)**: +- xlings 定位为"用户态发行版"(呼应 hermetic 策略 TL;DR)——如果这个定位本身要改,本文整体重来 +- mcpp 是生态主要构建工具(用户 2026-08-05 更正确认) +- kernel 由宿主提供,不接管(呼应 hermetic 策略 non-goals) +- Bootstrap 底座必须预构建(bootstrap 循环的技术必然) +- coupled-ABI 群走 platform 而非 range(basd on glibc symver 单向兼容的物理事实) + +**故意不作的假设**(留待后续): +- mcpp 对非 C/C++ 语言的具体覆盖形态 +- platform manifest 承载的确切数据结构 +- fingerprint 具体字段 +- range 语法(采纳 semver 还是自定义) +- 迁移 deadline + +--- + +## 12. 一句话总括 + +> **xlings 是"用户态 Linux 发行版",三分层 = kernel(host) + xlings(底座) + mcpp(主构建工具)。分发采用预构 + 源构混合(canonical build 保等价性,fingerprint 判命中)。多 glibc / 可组合发行版通过 platform manifest 承载,不做自由组合。消费者依赖从硬钉演进为 deps range + runtime_floor + platform_membership 三轴模型。与 hermetic 策略正交互补,合并后形成完整生态定位。** diff --git a/.agents/docs/2026-08-05-subos-minimum-design.md b/.agents/docs/2026-08-05-subos-minimum-design.md new file mode 100644 index 00000000..2d0e08d1 --- /dev/null +++ b/.agents/docs/2026-08-05-subos-minimum-design.md @@ -0,0 +1,577 @@ +# xlings subos slice 1 详细设计:Configuration 基质补完 + +**日期**: 2026-08-05 +**类型**: 详细设计(detailed design)—— slice 1 可实施规格 +**范围**: xlings 侧新增最小机制,让 `compat.mesa` 端到端可装可跑;不引入 platform 抽象、不引入新 CLI 动词 +**关联**: +- `2026-08-05-userspace-distro-hermetic-strategy.md`(运行时边界策略) +- `2026-08-05-ecosystem-three-tier-and-composable-distro.md`(生态定位讨论) + +**状态**:所有 schema 与 CLI 决策已定案(2026-08-05 会话收敛);实施细节标 [OPEN] 待具体施工时定 + +--- + +## 0. TL;DR + +**问题**:issue #352 类场景——xlings 的 subos 缺 Configuration 基质,GL/Vulkan/字体等子系统的发现协议(env vars + config 目录)没有 per-subos 承载。 + +**Slice 1 目标**:在 xlings 侧引入最小机制补完 Configuration 基质,让 `compat.mesa` payload 装进 subos 后 GLFW 用户程序能跑起来。 + +**核心决策**: +- Manifest 新增单块 `subos_info`(对齐 `xlings subos info` CLI) +- 字段:`schema_version` / `runtime` / `envs` / `created_at` / `created_by` +- **不引入 platform 抽象**——`runtime` 就是一个 xvm binding 字符串,self-describing +- **不新增 CLI 动词**——复用 `xlings subos use --shell` 与 `--cmd`,slice 1 扩展它们的行为 +- 新 xpkg API:`subos.env{}` 让 pkg config() 声明 env + +**Slice 1 明确不做**: +- Platform manifest 概念 +- 多 runtime(多 glibc 等)—— 生态阻塞项另议,slice 1 只需字段就位 +- Capabilities_host / activation_hooks +- `xlings subos export/import/diff/snapshot/migrate` +- 跨 shell/OS 完备性(bash 优先,fish/zsh 之后补) +- 自动 schema upgrade + +--- + +## 1. 分析框架:三层基质(分析工具,不是 schema 分块) + +用户程序在 subos 里跑起来需要三层基质齐备: + +| 层 | 内容 | xlings 现状 | Slice 1 | +|---|---|---|---| +| **1. Bootstrap** | PT_INTERP + CRT + libc/runtime | ✅ glibc pkg + elfpatch | 不动 | +| **2. Discovery** | PATH + RPATH + xvm binding | ✅ xvm.add/xvm.files + elfpatch | 不动 | +| **3. Configuration** | env vars + config 发现约定 | 🟡 无 per-subos 承载 | **补完** | + +三层是分析工具,不是 schema 组织形式——schema 只有一个 `subos_info` 块。 + +--- + +## 2. 最小 subos 不变量 + +`xlings subos new ` 完成后,以下条件必须同时成立(doctor 用同一套判据检查): + +**文件系统**: +- I1. `$XLINGS_HOME/subos//` 目录存在 +- I2. `$XLINGS_HOME/subos//.xlings.json` 文件存在且合法 JSON +- I3. xvm 状态目录已初始化(即使为空) + +**Manifest 结构**: +- I4. 文件顶级存在 `subos_info` 对象 +- I5. `subos_info.schema_version` == `1` +- I6. `subos_info.runtime` 是非空字符串,格式 `@` +- I7. `subos_info.envs` 是对象(可空 `{}`) +- I8. `subos_info.created_at` / `created_by` 是非空字符串 + +**注册状态**: +- I9. `$XLINGS_HOME/.xlings.json` 的 `subos` 注册表包含 `` 条目 + +**基质就绪**(逻辑判据,非文件检查): +- I10. `subos_info.runtime` 引用的 pkg 在 xvm 里已注册(Bootstrap 可用) +- I11. `envs` 段每个 key 对应的 pkg 在 xvm 里已装(env 与 xvm 一致) + +**违反任一 → doctor 报错并给出修复动作**。Reporter 与 repairer 用同一函数,遵循 xlings 生态"reporter/repairer 谓词一致"原则。 + +**"空集合 ≠ 缺失"原则**:所有可增长段(`envs`)在 `new` 时初始化为空对象/数组,不省略 key。避免 reader 处理 optional。 + +--- + +## 3. subos_info schema(完整) + +### 3.1 位置 + +**Per-subos** `.xlings.json`: +- Home 全局 subos:`$XLINGS_HOME/subos//.xlings.json` +- 项目 subos:`/.xlings/subos//.xlings.json` + +**不出现在**: +- 项目根 `.xlings.json`(那里 `subos` 是字符串选择器) +- Home 根 `.xlings.json`(那里 `subos` 是注册表 map) + +### 3.2 完整结构 + +```json +{ + "workspace": { ... }, // 现有段,不动 + + "subos_info": { // slice 1 新增 + "schema_version": 1, + "runtime": "glibc@2.39", + "envs": { }, + "created_at": "2026-08-05T14:23:11Z", + "created_by": "xlings 0.4.71" + } +} +``` + +### 3.3 字段规约 + +| 字段 | 类型 | 必填 | 说明 | +|---|---|---|---| +| `schema_version` | integer | ✅ | 当前 = 1,未来通过 probe pattern 演化 | +| `runtime` | string | ✅ | xvm binding 格式 `@`,self-describing | +| `envs` | object | ✅ | Per-pkg env 声明,可空 `{}` | +| `created_at` | string (ISO-8601) | ✅ | wall clock,信息字段 | +| `created_by` | string | ✅ | 创建时 xlings 版本,信息字段 | + +### 3.4 runtime 字段:self-describing 约定 + +``` +runtime: "glibc@2.39" → Linux + glibc 家族 +runtime: "musl@1.2.5" → Linux + musl 家族 +runtime: "wasi-libc@0.1" → WASI 家族(未来) +runtime: "macos_sdk@14.5" → macOS 家族(未来) +runtime: "ucrt@10.0.22000" → Windows 家族(未来) +``` + +**家族信息通过 pkg 名字前缀由 xlings 侧辅助函数派生**,不作为独立字段冗余存储: + +``` +family_of(runtime_binding) → + glibc@* → linux--glibc + musl@* → linux--musl + wasi-libc@* → wasm32-wasi + macos_sdk@* → darwin- + ucrt@* → windows--ucrt + 默认 → "unknown" +``` + +映射表在 xlings 侧代码中维护,新 OS 加入时更新。**Slice 1 只需 `glibc@*` 一项**。 + +### 3.5 envs 段结构 + +```json +"envs": { + "compat.mesa@25.0.0": [ + { "var": "LIBGL_DRIVERS_PATH", "op": "set", "value": "${pkgdir}/lib/dri" }, + { "var": "__EGL_VENDOR_LIBRARY_DIRS", "op": "set", "value": "${pkgdir}/share/glvnd/egl_vendor.d" }, + { "var": "XDG_DATA_DIRS", "op": "prepend", "value": "${pkgdir}/share" } + ] +} +``` + +**Schema**: +- Key:pkg binding,格式 `@` +- Value:数组,每项 `{ var: string, op: enum, value: string }` + +**op 允许取值**: +| op | 语义 | Slice 1 | +|---|---|---| +| `set` | 无条件设置(覆盖既有值) | ✅ 实现 | +| `prepend` | 前置到既有值(冒号分隔) | ✅ 实现 | +| `set-if-unset` | 仅在未设时设置 | ⬜ 未来 | +| `append` | 追加到既有值 | ⬜ 未来 | + +**Slice 1 只实现 `set` 与 `prepend`**——足够覆盖 GL/Vulkan/fontconfig 类场景。 + +### 3.6 value 占位符 + +在 emit 时(activate 或 run)展开: + +| 占位符 | 展开为 | +|---|---| +| `${pkgdir}` | 该 pkg 的 install 目录绝对路径 | +| `${subosdir}` | 该 subos 的根目录绝对路径 | +| `${home}` | 用户 home 目录 | +| `${xlings_home}` | `$XLINGS_HOME` | + +**占位符是可迁移性的关键**——manifest 内 value 不含机器特定绝对路径。 + +### 3.7 Provider-scoped 所有权 + +`envs` 以 pkg binding 为 key,与 xvm.add / xvm.files 同一所有权模型: +- pkg install 时其 config() 通过 `subos.env{}` 写入 +- pkg uninstall 时 xlings 自动删除该 key 整段 +- 不允许 cross-pkg 声明(A pkg config() 声明 B pkg 的 env) + +### 3.8 冲突语义(activate 时) + +同一变量被多个 pkg 声明时: + +| 场景 | 规则 | +|---|---| +| 多个 `set` 同变量 | doctor 报 warning;运行期以 **binding 序后到者**为准 | +| 多个 `prepend` | **binding 序后到者**在前(靠近变量头) | +| `set` 与 `prepend` 混合 | `set` 生效,`prepend` 忽略,doctor 报 warning | +| 用户已在 shell 里 export | **用户值优先**,pkg 声明不覆盖(见 UC-1) | + +> **施工修正(2026-08-05)**:本节原写"以**装机顺序**后到者为准"。**装机顺序不在 +> manifest 里**,要实现它就得加一个字段,而那个字段唯一的作用是让结果取决于历史 —— +> 两台机器持有逐字节相同的 manifest 会导出不同的值。这与 SH-1(可分享性)直接矛盾: +> 一份只在写它的机器上成立的描述不算描述。 +> +> 改为按 **binding 字典序**。它由数据本身决定,可复现,且让 manifest 成为完整答案。 +> 已由单测 `DoesNotDependOnDeclarationOrder` 钉住。 + +**冲突检测在 activate 时进行**,不在 write 时(简化 write 路径,写多次不 fail)。 + +--- + +## 4. xlings subos new(最小改动) + +### 4.1 CLI 签名 + +现有: +``` +xlings subos new [--storage ] [--image-size ] [--from ] +``` + +Slice 1 新增: +``` +xlings subos new [--runtime ] [其他现有 flag] + +--runtime subos 的 runtime pkg binding,如 "glibc@2.39" + 缺省:xlings 内嵌 fallback(当前为 "glibc@2.39") +``` + +**不新增其他 flag**——`--platform` / `default_platform` 不做。 + +### 4.2 执行流程 + +``` +xlings subos new [--runtime ] + +Step 1: 前置校验 + - 未占用、命名合法 + - Home 目录可写 + +Step 2: 解析 runtime + --runtime CLI 参数 → 若无 → xlings 编译时嵌入默认(slice 1: "glibc@2.39") + → 记为 R + +Step 3: 校验 R 可用 + - R 对应的 pkg 在索引里存在 + - 若目标机未装 → prompt "install now? [Y/n]"(默认 Yes) + +Step 4: 物理创建 + - mkdir -p $XLINGS_HOME/subos// + - 初始化 xvm 状态子目录 + - 若需要 → 装 R + +Step 5: 写 subos_info 到 subos//.xlings.json + { + "workspace": {}, + "subos_info": { + "schema_version": 1, + "runtime": "", + "envs": {}, + "created_at": "", + "created_by": "xlings " + } + } + +Step 6: 更新 home .xlings.json subos 注册表 + +Step 7: 验证不变量 I1~I11,任一失败回滚 + +Step 8: 输出提示 + "Created subos '' with runtime . + Enter with: xlings subos use + Run one command: xlings subos use --cmd '' + Emit env script: xlings subos use --shell" +``` + +### 4.3 错误场景 + +| 场景 | 退出码 | 提示 | +|---|---|---| +| `` 已存在 | 1 | "use another name or `subos remove`" | +| `--runtime` 引用的 pkg 索引里无 | 2 | "no such runtime pkg; check `xlings install --search`" | +| Runtime 与 host arch 不容 | 3 | 提示 arch 详情 | +| Runtime 未装且用户拒绝安装 | 4 | 提示手动安装 | +| Step 4~6 部分成功后失败 | 5 | 反向回滚;若回滚失败,提示手动清理路径 | + +失败必须显式,不 silent-success(memory: silent-success pattern)。 + +--- + +## 5. subos.env{} xpkg API + +### 5.1 Lua API 签名 + +```lua +subos.env{ + var = "LIBGL_DRIVERS_PATH", + op = "set", -- 或 "prepend" + value = "${pkgdir}/lib/dri", + binding = "compat.mesa@" .. pkginfo.version() +} +``` + +### 5.2 参数 + +| 参数 | 类型 | 必填 | 说明 | +|---|---|---|---| +| `var` | string | ✅ | 环境变量名 | +| `op` | string | ✅ | `set` 或 `prepend`(slice 1 只这两种) | +| `value` | string | ✅ | 值,支持 §3.6 占位符 | +| `binding` | string | ✅ | Provider 标识,必须与本 pkg 一致(cross-pkg 拒绝) | + +### 5.3 语义 + +- 调用时 append 到 `subos_info.envs[binding]` 数组 +- 幂等:同 binding 内 (var, op, value) 三元组已存在则不重复写 +- 写入原子:走 atomic write 路径(避免 flock 场景;memory: atomic write vs flock) +- **write 时不做冲突检测**——只在 activate 时统一按 §3.8 规则解冲突 + +### 5.4 老 client 兼容(V2 spec probe 模式) + +> **施工修正(2026-08-05)**:本节原写的是 `if subos.env then`。**那个探针在所有 +> client 上恒为真**,必须用 `type(...)`: +> +> ```lua +> if type(subos.env) == "function" then +> subos.env{ var = "...", op = "set", value = "...", binding = binding } +> else +> -- 老 client 无此能力,走原有路径 +> end +> ``` +> +> 原因在 libxpkg `prelude.lua::import()`:未知模块返回 permissive proxy stub, +> 它的 `__index` 对任意 key 都返回一个 truthy 的可调用 table。老 client 没有 +> `xim.libxpkg.subos`,`subos.env` 于是是个 truthy proxy —— recipe 走"新分支", +> 调用静默无效,装完什么也没发生。 +> +> V2 spec 里 `if xvm.files then` 成立,只因为 `xvm` 是**已存在的模块**,老 client +> 上那个 *field* 才真的是 `nil`。缺失的 *module* 永远不是。stub 是带 `__call` 的 +> **table**,真函数才是 `function`,`type()` 是唯一能区分两者的判据。 +> +> 已由 e2e 用真实旧二进制钉住(见 §11)。规则已写进 +> `xim-pkgindex/docs/V2/xpackage-spec.md`。 + +### 5.5 卸载路径 + +pkg `uninstall()` 无需显式清理 env——xlings 侧 uninstall 流程会按 binding 删除对应 envs key。 + +--- + +## 6. 复用 `xlings subos use`(不新增命令) + +### 6.1 现有 CLI(不动) + +``` +xlings subos use + --global Persist the active SubOS + --shell [KIND] Emit shell activation code + --sandbox [BACKEND] Enable sandbox + --cmd Run one command + --keep / --no-keep Namespace keeper + --ttl Keeper idle timeout + --gpu Expose GPU devices +``` + +### 6.2 Slice 1 扩展:`--shell` 输出内容 + +现有 `--shell` 已经 emit shell 激活源码。**Slice 1 在其输出里追加 subos_info.envs 展开的 export 语句**: + +Bash 输出示例: +```bash +# 现有 xvm binding 相关 export(不动) +export PATH="..." + +# Slice 1 新增:subos_info.envs 展开 +export LIBGL_DRIVERS_PATH="/home/user/.xlings/subos/default/pkg/compat.mesa-25.0.0/lib/dri" +export __EGL_VENDOR_LIBRARY_DIRS="/home/user/.xlings/subos/default/pkg/compat.mesa-25.0.0/share/glvnd/egl_vendor.d" +export XDG_DATA_DIRS="/home/user/.xlings/subos/default/pkg/compat.mesa-25.0.0/share${XDG_DATA_DIRS:+:$XDG_DATA_DIRS}" +``` + +**注入规则**: +1. 读 subos_info.envs,按 binding 迭代 +2. 按 §3.8 冲突规则汇总 +3. 展开 §3.6 占位符 +4. 探测目标 shell(bash / fish / zsh),emit 对应语法 +5. **不覆盖用户已 export 的变量**——通过 `${VAR:+...}`(bash)、`$VAR`(fish)保持既有值 + +### 6.3 Slice 1 扩展:`--cmd` 环境注入 + +现有 `--cmd ` 已经在 subos 上下文里执行命令。**Slice 1 在 exec 前把 subos_info.envs 计算后的 env dict 注入进程**: + +- 保留当前进程 env 作为基线 +- 叠加 envs 段计算结果(用户已设的 env 优先,pkg 声明不覆盖) +- `execvpe(cmd, argv, effective_env)`——不启动 shell + +### 6.4 Shell 覆盖(Slice 1 范围) + +| Shell | Slice 1 支持 | +|---|---| +| bash | ✅ 主 emitter | +| zsh | ✅(与 bash 兼容语法) | +| fish | ✅(独立 emitter) | +| PowerShell/cmd | ⬜ 未来 | + +用户可通过 `--shell [KIND]` 显式指定(已在现有 CLI)。 + +### 6.5 UC-2 输出实际生效清单 + +`--shell` stderr 输出实际注入的变量清单(不进 eval,只做提示): + +``` +[xlings] subos: default (runtime glibc@2.39) +[xlings] env: 3 vars from 1 packages +[xlings] compat.mesa: LIBGL_DRIVERS_PATH, __EGL_VENDOR_LIBRARY_DIRS, XDG_DATA_DIRS +``` + +避免用户 `eval` 后不清楚加了什么。 + +--- + +## 7. 卸载 + doctor 集成 + +### 7.1 pkg uninstall 路径 + +Xlings 侧 uninstall 流程新增一步: + +``` +uninstall(pkg): + 1. 调用 pkg uninstall() hook(既有) + 2. 从 xvm 撤销 binding(既有) + 3. [新增] 从 subos_info.envs 移除 @ 为 key 的整段 + 4. Atomic write .xlings.json + 5. 若移除后 envs 为空 → 保留 {}(遵循 I7) +``` + +### 7.2 doctor 检查项 + +| # | 检查 | 判据 | 修复动作 | +|---|---|---|---| +| D1 | Manifest 结构完整 | I1~I8 全成立 | 引导用户重建 subos | +| D2 | envs binding 与 xvm 一致 | 每个 envs key 对应 pkg 在 xvm 已装 | 移除孤儿 envs 段 or 提示重装 | +| D3 | envs value 占位符可解 | `${pkgdir}` 等指向目录存在 | 提示重装对应 pkg | +| D4 | envs 变量无冲突 | 同 var 无多个 `set` | warning(非 error) | +| D5 | runtime 引用一致 | subos_info.runtime 引用的 pkg 在 xvm 已装 | 提示装 runtime | + +**Reporter 与 repairer 用同一函数**(memory: reporter/repairer predicate drift),不允许两处描述判据。 + +--- + +## 8. 契约条款 + +### 8.1 SH(可分享性) + +- **SH-1** 值可迁移:`subos_info` 内所有 value 必须用占位符,禁止绝对路径、机器特定 ID +- **SH-2** envs 是 derived cache:pkg config() 是权威源;recreate 时重跑 config() 而非直接 apply + +### 8.2 UC(用户覆盖权) + +- **UC-1** 用户 env 优先:用户在 shell 里 export 的变量,activate/run 不覆盖 +- **UC-2** 无静默选择:activate 输出实际生效清单到 stderr +- **UC-3** 遵循 v2 spec probe pattern:新字段/API 通过 probe 兼容老 client,不用 `min_xlings` + +--- + +## 9. Slice 1 边界 + +### 9.1 已完成(不动) + +- glibc pkg + elfpatch PT_INTERP/RPATH(Bootstrap) +- xvm.add / xvm.files provider-scoped 注册(Discovery) +- xvm shim + PATH(Discovery) +- `xlings subos new/list/remove/info/use/stop` 基础 CLI +- `~/.xlings.json` subos 注册表 + activeSubos 机制 +- doctor 框架 + +### 9.2 Slice 1 新增(xlings 侧) + +代码估算 ~800~1200 行: + +| 组件 | 估算 | +|---|---| +| `subos_info` 读写 + schema 校验 | ~200 | +| `subos.env{}` xpkg API + provider-scoped 记账 | ~150 | +| `xlings subos use --shell` env emitter (bash/zsh/fish) | ~200 | +| `xlings subos use --cmd` env 注入 | ~100 | +| `xlings subos new --runtime` + 不变量校验 | ~150 | +| Uninstall envs 清理 | ~60 | +| Doctor D1~D5 检查项 | ~120 | +| 单元 + 集成测试 | ~250 | + +### 9.3 Slice 1 新增(xim-pkgindex 侧) + +- `compat.mesa` recipe(独立设计,见 task #8~9) +- 一份预构建 tarball 上传 xlings-res +- dlopen 闭包一次性人工审计 +- 一份 bwrap 空 host smoke test + +### 9.4 明确不做(non-goals) + +- Platform manifest / 可组合发行版 +- 多 runtime(多 glibc / musl 并存)—— 生态阻塞项另议 +- `capabilities_host` / `activation_hooks` 段 +- `xlings subos export/import/diff/snapshot/migrate` CLI +- 用户 `--force` flag(UC-1 是原则,CLI 后续 slice 补) +- 自动 schema upgrade +- PowerShell / cmd shell 支持 +- macOS / Windows 上的 runtime(仅数据模型预留,不实现) +- NVIDIA 闭源栈 payload(slice 2) + +--- + +## 10. 开放问题 + +| # | 问题 | 阻塞谁 | +|---|---|---| +| O1 | `--shell fish` 与 bash 的语义等价 fixture 是否需要? | 测试 | +| O2 | Runtime 未装且用户拒绝 `--yes` 时的具体退出码语义 | subos new UX | +| O3 | Doctor D4(env 冲突)是 warning 还是 error?判据严格性统一 | doctor | +| O4 | `compat.mesa` payload 实际 size(可能超 800MB,是否分包)| pkgindex 侧 slice 1 | +| O5 | subos 迁移/克隆场景下 `${pkgdir}` 是否需要在 subos_info 里持久化解析结果? | 未来 subos 迁移能力 | + +不阻塞 slice 1 起步,施工时定。 + +--- + +## 11. 施工顺序建议 + +1. **Task #2 定案**:本 spec §3 schema 已定,可直接进入实施 +2. **Task #4**:实现 `subos.env{}` API + 写入路径(2~3 天) +3. **Task #3**:`xlings subos new --runtime` + 不变量校验(2~3 天) +4. **Task #5**:`xlings subos use --shell` env emitter (bash 优先,~5 天) +5. **Task #6**:`xlings subos use --cmd` env 注入(2 天) +6. **Task #7**:uninstall 清理 + doctor D1~D5(3~5 天) +7. 并行 **Task #8~10**:pkgindex 侧 compat.mesa(2~3 周) +8. 集成 + e2e smoke test(1 周) + +**总估算 4~6 周**。 + +--- + +## 13. 施工实况(2026-08-05) + +Slice 1 的 xlings + libxpkg 两侧已实现并验证。落地拆分见 +`2026-08-05-subos-slice1-landing-plan.md`。 + +### 13.1 与本设计的三处偏离 + +| # | 设计原文 | 实际 | 原因 | +|---|---|---|---| +| 1 | 探针 `if subos.env then` | `type(subos.env) == "function"` | 前者恒真,见 §5.4 修正 | +| 2 | 冲突按装机顺序 | 按 binding 序 | 装机顺序不在 manifest 里,见 §3.8 修正 | +| 3 | (未提及)`default` subos | `self init` 也写 manifest | 它不走 `subos::create`,否则新能力在**所有人实际在用的那个 subos 上**全程静默无效,同时成为老 home 的迁移路径 | + +### 13.2 设计未覆盖、施工中必须补的 + +- **早退分支**:`process_xvm_operations_` 在没有 xvm 注册项时会提前 return。 + 只声明 env、不注册任何 xvm 节点的包会被静默丢弃 —— env 消费必须放在早退**之前**。 +- **doctor 渲染器的 `default: break;`**:新 FindingKind 不加 case 就不打印, + 查得出、报不出。 +- **卸载按包名而非精确 binding 匹配**:removal 会经命名空间和 group 成员解析版本, + 安装时记录的 binding 在卸载点无法可靠重建;同一包装过两个版本时会留下另一段。 + +### 13.3 覆盖 + +- 单测 `tests/unit/test_subos_manifest.cpp` — 29 例(schema/不变量/占位符/冲突/确定性) +- libxpkg `tests/test_executor.cpp` — 6 例(op 收集/校验/探针语义) +- E2E-60 `subos_env_declaration_test.sh` — 装→声明落盘→`--shell`→`--cmd`→用户覆盖→doctor→卸载清段 +- E2E-61 `subos_env_probe_compat_test.sh` — 真实旧二进制上的双读数差分 + +### 13.4 仍未做 + +§9.4 全部照旧。§10 的 O1(fish/bash 等价 fixture)已由 E2E-60 覆盖 bash 侧, +fish emitter 只有构造正确性、无行为断言;O4、O5 仍开放,随 compat.mesa 一起定。 + +--- + +## 12. 一句话总括 + +> **Slice 1 = 一个 `subos_info` 块 + 一个 `subos.env{}` API + `xlings subos use --shell/--cmd` 两处扩展。总代码 ~800~1200 行。补完 Configuration 基质,让 compat.mesa 端到端可跑;所有更大的野心(platform、多 runtime、canonical build 等)明确挂到后续 slice,不阻塞当前工作。** diff --git a/.agents/docs/2026-08-05-subos-slice1-landing-plan.md b/.agents/docs/2026-08-05-subos-slice1-landing-plan.md new file mode 100644 index 00000000..d9b90a20 --- /dev/null +++ b/.agents/docs/2026-08-05-subos-slice1-landing-plan.md @@ -0,0 +1,176 @@ +# subos slice 1 落地计划:任务拆分与跨仓依赖 + +**日期**: 2026-08-05 +**类型**: 落地计划(landing plan) +**上游**: `2026-08-05-subos-minimum-design.md`(详细设计,schema/CLI 已定案) +**目标**: 把 slice 1 从设计变成已发布、可验证的能力 + +--- + +## 0. 设计文档的一处修正(施工前必须先改) + +设计文档 §5.4 给出的老 client 探针是: + +```lua +if subos.env then ... else ... end +``` + +**这个探针在所有 client 上恒为真,是错的。** + +原因在 libxpkg 的 `prelude.lua::import()`:未知模块返回一个 permissive +proxy stub —— 它的 `__index` 对任意 key 都返回一个可调用的 proxy。老 client +没有 `xim.libxpkg.subos`,`import()` 于是返回 stub,`subos.env` 是一个 truthy +的 proxy table。recipe 会走"新路径",调用静默无效,装完什么也没发生。这正是 +xlings 生态反复出现的 silent-success 形态。 + +V2 spec 里 `if xvm.files then` 之所以成立,是因为 `xvm` 是**已存在的模块**—— +老 client 上 `xvm` 是真模块,`xvm.files` 才真的是 `nil`。**新模块不适用这个 +写法。** + +**新模块的探针必须判类型**: + +```lua +if type(subos.env) == "function" then ... end +``` + +proxy 是带 `__call` 元方法的 **table**,真函数才是 `function`,两者可区分。 + +这条规则要写进 `xim-pkgindex/docs/V2/xpackage-spec.md`,并由一个跑真实旧 +xlings 二进制的 e2e 断言(模板:`tests/e2e/xvm_files_probe_compat_test.sh`)。 + +--- + +## 1. 为什么必须动 libxpkg(而不是只动 xlings) + +考虑过三条路,只有一条站得住: + +| 方案 | 结论 | +|---|---| +| `subos.env` 放 xim-pkgindex 的 `libs/`(像 `xim.pkgindex.sysroot`) | ❌ index 是滚动更新的,模块在所有 client 上都存在,而消费 op 的 C++ 在 xlings 里。探针恒真 → silent-success | +| xlings 在执行 recipe 前往 `_LIBXPKG_MODULES` 注入模块 | ❌ 模块表在 libxpkg 的 `load_stdlib` 里写死,没有宿主注入口 | +| **放 libxpkg** | ✅ Lua 函数与消费它的 C++ 静态链接进同一个 xlings 二进制,"函数在不在"就等于"这个 client 支不支持"。单一真相源 | + +所以 **libxpkg 0.0.48 是硬前置**,不是可选项。 + +--- + +## 2. 跨仓依赖图 + +``` +┌─ libxpkg ─────────────────────────────────────────────┐ +│ #12 subos.lua + XvmOp{var,value,mode} + 模块注册 + 测试 │ +└───────────────┬───────────────────────┬───────────────┘ + │ │ + ┌────────────▼──────────┐ ┌────────▼────────────────┐ + │ #13 发布链(串行) │ │ #14 本地 registry 播种 │ + │ bump 0.0.48 → merge │ │ ~/.mcpp/registry/data/ │ + │ → tag → gtc 镜像 │ │ …/0.0.48 + .mcpp_ok │ + │ → mcpp-index 条目 │ │ (让 xlings 侧立刻能编) │ + │ → publish-artifact │ └────────┬────────────────┘ + └────────────┬──────────┘ │ + │ ┌────────▼──────────────────┐ + │ │ #15 manifest.cppm 核心 │ + │ │ schema/不变量/占位符/合并 │ + │ └────────┬──────────────────┘ + │ │ + │ ┌──────┬───────┴──────┬──────────┐ + │ ▼ ▼ ▼ ▼ + │ #16 #17 #18 #19 + │ new install/ use doctor + │ --runtime uninstall --shell D1–D5 + │ env ops --cmd + │ └──────┴───────┬──────┴──────────┘ + │ ▼ + │ #20 测试(unit + e2e) + │ │ + #21 规范/文档(可全程并行)──────────┤ + │ │ + └───────────┬───────────┘ + ▼ + #22 版本号 + 单 PR + CI 全绿 + ▼ + #23 release + gtc 补 gitcode 资源 + ▼ + #24 xlings subos 真实验证 +``` + +**并行窗口**:#14 一旦完成,#15→#16/#17/#18/#19 与 #13 的发布链完全并行。 +#21 全程可并行。真正的串行瓶颈只有 #12 → #14 → #15 → 四路扇出 → #20 → #22。 + +--- + +## 3. 各仓改动面 + +### 3.1 openxlings/libxpkg(#12, #13) + +| 文件 | 改动 | +|---|---| +| `src/lua-stdlib/xim/libxpkg/subos.lua` | 新增。`M.env{var, op, value, binding}` | +| `src/xpkg-executor.cppm` | `XvmOp` 加 `var` / `value` / `mode`;`xvm_operations()` 读取;`load_stdlib` 注册 `subos` 模块 | +| `xmake.lua` | embed 列表加 `subos_lua` | +| `src/xpkg-lua-stdlib.cppm` | 由 xmake 从 `.lua` **自动生成**,不手改 | +| `tests/test_executor.cpp` | subos_env op 收集用例 | +| `mcpp.toml` | `0.0.47` → `0.0.48` | + +**命名冲突处理**:用户面参数是 `op = "set"|"prepend"`,而 `XvmOp.op` 已经 +表示 op 类别。内部 entry 用 `op = "subos_env"` + `mode = "set"|"prepend"`, +用户面不变。 + +### 3.2 openxlings/xlings(#14–#20, #22, #23) + +| 文件 | 改动 | +|---|---| +| `src/core/subos/manifest.cppm` | **新增**,slice 1 的地基 | +| `src/core/subos.cppm` | `create()` 写 subos_info;`run()` 解析 `--runtime`;`use_emit_shell` / `use_spawn_shell` 注入 env | +| `src/core/xim/installer.cppm` | 消费 `subos_env` op;uninstall 按 binding 清段 | +| `src/core/xself/doctor.cppm` | D1–D5 | +| `src/cli/spec.cppm` | `subos new --runtime` 帮助文本 | +| `mcpp.toml` | 版本号 + `mcpplibs.xpkg = "0.0.48"` | +| `tests/unit/`、`tests/e2e/` | 见 #20 | + +### 3.3 openxlings/xim-pkgindex(#21, #24) + +| 文件 | 改动 | +|---|---| +| `docs/V2/xpackage-spec.md` | `subos.env` API + **新模块探针规则** | +| 验证载体 recipe | #24 需要一个真实声明 env 的包 | + +### 3.4 mcpplibs/mcpp-index(#13 的一环) + +`pkgs/x/xpkg.lua` 加 0.0.48 条目,GLOBAL + CN 双 URL + sha256,**三个平台块都要写**。 + +--- + +## 4. 已知的坑(来自既往教训,不是推测) + +1. **`bump-index` 和 `mirror-binaries` 都吞掉自己的失败**(结尾 `|| echo "…(non-blocking)"`)。两个都可能报绿而什么都没干。#23 必须查产物:xlings-res 的版本条目 + `latest.ref`,以及用 **GET(不是 HEAD)** 验证 gitcode 资源。 +2. **mcpp index 是 artifact 不是 git clone**。合并到 mcpp-index 后要等 `publish-artifact.yml`,客户端还有 TTL —— `rm -rf ~/.mcpp/registry/data/` 强制刷新。手改缓存里的 `pkgs/**` 无效。 +3. **`mcpp build` 不重建测试二进制,只有 `mcpp test` 会**。改完测试跑 `mcpp build` 会留下过期 `test_main`,红相位可能假绿。 +4. **libxpkg CI 会静默腐烂**,失败读起来像网络错误(`fetch 'mcpplibs.capi.lua@0.0.3' failed`)。`test_executor` 另有 4 个既存 elfpatch 失败,CI 用 `--gtest_filter=-ExecutorTest.ApplyElfpatchAuto_*` 过滤。 +5. **隔离 home 测试不要预置 `data/`** —— 会让 index 变成 symlink,差分断言变得不可证伪。 +6. **绝不写入被 flock 的文件**:rename 换掉 inode 会静默破坏锁。subos_info 的原子写要与 home config 的锁路径区分清楚。 +7. **`quick_install` 忽略 `XLINGS_HOME`**,验证 release 要用 tarball 自带的 `self install`。 + +--- + +## 5. 本次落地的范围边界 + +**做**:libxpkg 0.0.48 + xlings slice 1 全部(设计文档 §9.2)+ 规范文档 + 发布 + 真实验证。 + +**不做**(设计文档 §9.4 已明确,外加): +- `compat.mesa` payload 本身(#8–#10,设计文档 §11 估 2–3 周,独立并行轨) +- 多 runtime 并存 +- platform 抽象 + +#24 的验证载体因此不是 compat.mesa,而是一个小体量的真实 recipe —— 目的是证明 +**机制**端到端通,payload 规模是另一件事。 + +--- + +## 6. 完成判据 + +- [ ] libxpkg 0.0.48 已发布,mcpp-index 可解析,gitcode 资源 GET 可下载 +- [ ] xlings 单 PR 含全部 slice 1 实现 + 测试 + 版本号,CI 全绿 +- [ ] 新老 client 探针差分测试通过(老二进制走 legacy 分支且不 silent-success) +- [ ] release 产物已验证:版本条目、`latest.ref`、gitcode 资源三者一致 +- [ ] `xlings subos` 真实跑通:new → install → use → env 生效 → uninstall 清段 → doctor 干净 diff --git a/.agents/docs/2026-08-05-userspace-distro-hermetic-strategy.md b/.agents/docs/2026-08-05-userspace-distro-hermetic-strategy.md new file mode 100644 index 00000000..e0fdd158 --- /dev/null +++ b/.agents/docs/2026-08-05-userspace-distro-hermetic-strategy.md @@ -0,0 +1,304 @@ +# xlings 作为用户态发行版:hermetic 策略与 GPU/GLIBC 边界设计 + +**日期**: 2026-08-05 +**类型**: 策略 (strategy) — 尚未细化到实施计划,先立框架 +**触发**: mcpp-community/mcpp#352(Fedora 44 上 GLFW/OpenGL 程序静默 exit 255,三层根因:观测性 + `compat.glx-runtime` symlink 到宿主 `/usr/lib` 32-bit 库 + 沙盒 glibc 2.39 与宿主 Mesa 要求 GLIBC_2.43 冲突) +**关联**: +- `2026-05-22-subos-sandbox-gpu-passthrough.md`(sandbox 内 GPU 设备节点透传,本文的**下层配套**) +- `2026-06-21-linux-root-usability-survey.md`(用户身份/权限边界) +- `2026-06-26-multiarch-package-description-design.md`(payload 的多架构表达,本文所有新包都要沿用) +- mcpp-index `pkgs/c/compat.glx-runtime.lua`(现存"半 hermetic"的典型样本) +- xim-pkgindex `pkgs/g/glibc.lua`(只有 2.39,`XLINGS_RES` payload) + +--- + +## 0. TL;DR + +**将 xlings 明确定位为"用户态 Linux 发行版",执行"能不依赖宿主就不依赖宿主"策略。** +物理上不可自带的资源(kernel syscall、wire protocol、GPU vendor 私有栈)显式列入 `capabilities_host` 白名单,除此之外一律走 xlings 内自建 payload。glibc 版本冲突是"任何宿主 `.so` 被拽进 xlings loader 进程"的**症状**,而非独立问题;策略解决"依赖宿主的范围",症状自动消失。 + +**近期最优选择(不锁死路径)**: +- Intel/AMD GPU 场景 → 走 hermetic-full(自带 Mesa/libdrm/GLVND) +- NVIDIA 场景 → 走版本化 payload(参考 Flatpak `org.freedesktop.Platform.GL.nvidia-XXX-YY`) +- Kernel module 编译**不接管**(留给宿主/DKMS),xlings 只负责 userspace 侧对齐 + +**保留的未来迁移路径**(不现在做,但**不封堵**): +- linker namespace 分层(参考 Steam pressure-vessel `libcapsule`)—— 长期解 NVIDIA 无版本 payload 兜底 +- 接管 kernel module 编译(参考 NixOS `hardware.nvidia.package`)—— 若 xlings 自建 kernel/subos 深化时再评 +- Release 套装模型(参考 NixOS channel / Flatpak runtime `//24.08`)—— 版本轴纪律 + +**明确不做**: +- 不做完整 Linux distro(不发行 init、systemd、包管理器替代)—— xlings 是**"用户态 app 侧的发行版"**,不是 rootfs 侧的发行版 +- 不接管 kernel(不发行 kernel、不管 firmware 装载、不管 udev) + +--- + +## 1. 深度分析:问题与现状 + +### 1.1 issue #352 的三层根因(具体表症) + +| 层 | 归属 | 缺陷性质 | 修复形态 | +|---|---|---|---| +| L1 观测性静默 | `sudoevolve/EUI-NEO` `glfw_app_main.cpp` | 无 `glfwSetErrorCallback` + 各失败路径无 stderr | 上游 30 行 PR | +| L2 glx-runtime 布局假设错 | `mcpplibs/mcpp-index` `compat.glx-runtime.lua:63-127` | 候选目录顺序把 `/usr/lib` 放最后,`ln -sf` 覆盖 64-bit symlink 为 32-bit | 单 patch | +| L3 glibc 版本鸿沟 | `xim-pkgindex` `glibc.lua` + mcpp 引擎 | 沙盒 glibc 2.39 vs 宿主 Mesa 要求 GLIBC_2.43;`LD_LIBRARY_PATH=/usr/lib64` 变通亦不可用(ld.so 与 libc.so.6 版本必须成对) | **架构级** | + +**L1、L2 是叶子,L3 是骨** —— 只补 L1/L2 会让用户从"静默 exit 255"变成"至少能看到 GLIBC_2.43 not found",但**不解决根本问题**。 + +### 1.2 L3 的真实性质:glibc 是症状,不是成因 + +- 崩溃的必要条件是:xlings loader(`xim-x-glibc/2.39/ld-linux-x86-64.so.2`)加载了**一份别人编译时锁死 glibc 版本的 `.so`** +- issue #352 的具体路径:用户 exe `dlopen("libGLX.so.0")` → symlink → 宿主 `libGLX_mesa.so.0` → 后者 `DT_NEEDED libc.so.6` 且 symver 表含 `GLIBC_2.43` → 我们的 2.39 libc 提不出该符号 → SIGABRT +- **如果 loader 全程只加载"xlings 自建、都链到 2.39"的 `.so`,`GLIBC_*` 这个词根本不会出现在任何 error message 里** + +这个观察的**推论**: +- 问题不是"glibc 太老要追新",而是"何时应该跨越到宿主库"这个边界不清晰 +- 追 glibc 版本是**追不完的**(发行版每半年一版,rawhide 永远领先),且**老 mcpp 二进制不受益**(PT_INTERP 编译时烙定) +- **正解是收敛"跨越边界"的地方**,不是提高沙盒 glibc 的版本 + +### 1.3 生态里已经在"实质做 hermetic"但没写成明文 + +现有历史决策(散落在若干 issue/PR)已经**在无意贯彻这个策略**,只是从未被明确提炼: + +| 记录 | 决策 | 隐含语义 | +|---|---|---| +| xim-pkgindex termux musl DNS | musl-static `getaddrinfo` 读不到 `$PREFIX/etc/resolv.conf` → 手写 UDP DNS | 不依赖宿主 `/etc/resolv.conf` | +| mcpp e2e elfpatch 前缀替换 | 测试 elfpatch 曾"写穿 `ln -sf`"烙死 `mktemp -d` 路径 → 改为前缀替换 | payload 不能借宿主路径当锚 | +| mcpp macOS 静态 libc++ SIOF | 归档成员 `.init_array.` 排最后 → 静态 vs 动态是分界线 | 自带 C++ 运行时的 ABI 独立性 | +| xim-pkgindex llvm22 slim asset | slim 删了 `libatomic.so.1`、libc++ 硬依赖它、沙盒 glibc loader**不回退系统** → 崩 | loader 不回退系统 = 隐式 hermetic | +| mcpp Windows DLL deploy | PE 无 RPATH → 拷 DLL 到 exe 旁,`mcpp run` 会塞 PATH 掩盖漏洞 → 直接执行 `.exe` 才是发布态 | Windows 上早就是 hermetic | + +**结论**:"用户态发行版 + hermetic 优先"不是新方向,是**给已经在做的事一个明确的名字和判据**,以后新决策不用重新推导。 + +--- + +## 2. 策略陈述:边界与优先级 + +### 2.1 一句话原则 + +> **只有 kernel syscall 和显式声明的 wire protocol 允许穿越到宿主;所有 `.so`(除 GPU vendor 私有栈)必须来自 xlings 自建 payload。** + +### 2.2 允许穿越的边界(**枚举而非例外**) + +| 边界类型 | 具体 | 为什么无法自带 | +|---|---|---| +| kernel syscall | 全部 syscall,含 `/dev/dri/*` ioctl、`/dev/snd/*`、futex、io_uring、`/dev/nvidia*` ioctl | 内核 ABI,唯一入口 | +| wire protocol(socket 字节流)| X11(`/tmp/.X11-unix/`)、Wayland(`$XDG_RUNTIME_DIR/wayland-*`)、D-Bus(`/run/user/*/bus`)、PulseAudio/PipeWire(socket)、CUPS | 协议稳定,进程外通信,**不引入宿主 libc** | +| 硬件/固件 | `/lib/firmware`(kernel 自动加载) | 硬件绑定 | +| GPU vendor 强制耦合 | NVIDIA 专有栈(`libGLX_nvidia.so`、`libEGL_nvidia.so`、`libcuda.so` 等) | vendor 用私有 IOCTL 与 `nvidia.ko` 校验版本;`.so` 与内核模块必须严格对齐 | + +### 2.3 不允许穿越的(即使"就一次也不行") + +- 任何 `/usr/lib*` `/lib*` 下的 `.so`,含 `libc`、`libssl`、`libGL`(Mesa)、`libX11`、`libpulse`、`libdbus`、`libcups`、`libfontconfig`、`libfreetype`、`libgio` … +- `/usr/share/*` 下的**运行期数据**:zoneinfo、locale、CA bundle、fontconfig cache、图标主题 +- xlings 有观点的 `/etc/*` 项:`/etc/ssl/certs`(信任链应归 xlings 管)、`/etc/resolv.conf`(DNS 已在 musl 场景踩过) + +### 2.4 每一条穿越必须**在包元数据里写理由** + +新增字段 `capabilities_host`,每一项带 `reason`: + +```lua +capabilities_host = { + { "kernel.drm", reason = "GPU ioctl,唯一入口" }, + { "wire.wayland", reason = "compositor 是宿主实例" }, + { "hw.nvidia-gpu", reason = "vendor 私有 IOCTL 版本绑定" }, -- 仅在探到 nvidia.ko 时激活 +} +``` + +引擎侧在 target 分析阶段:**任何 dlopen 目标不在包自身链接闭包 ∪ `capabilities_host` 隐含允许集**,报错拒绝。当前 `compat.glx-runtime.lua:76-80` 那种"直接 `ln -sf /usr/lib64/*`"就会在这一步被拦下。 + +--- + +## 3. 场景枚举(不只是 GPU) + +hermetic 策略的判据要在多个场景上自洽,不是只解 GPU: + +| 场景 | 现状 | hermetic 策略下的形态 | +|---|---|---| +| **GPU/GL** (issue #352) | `compat.glx-runtime` symlink 宿主 | Intel/AMD 自带 Mesa;NVIDIA 版本化 payload | +| **音频** (PulseAudio/PipeWire) | 未系统性处理;示例包偶发使用 | `compat.libpulse`(client-side)+ `capabilities_host = wire.pulse`(server 是宿主) | +| **字体/i18n** | 依赖宿主 `/usr/share/fonts`、`fontconfig` | `compat.fontconfig` + `xim-x-fonts-default` payload(挑一套 Noto/DejaVu) | +| **HTTPS/CA** | 静态 OpenSSL 编译时 OPENSSLDIR=/etc/ssl → Debian 挂 | `xim-x-ca-certificates` payload(Mozilla bundle),客户端库改从这里读 | +| **DNS** | musl-static 已改手写 UDP | 已 hermetic,补写策略文档确认此为**原型** | +| **DBus** | 应用 dlopen 宿主 libdbus | `compat.libdbus`(client)+ `wire.dbus` | +| **打印** (CUPS) | 未涉及 | 若涉及则 `compat.libcups` + `wire.cups` | +| **timezone/locale** | 用宿主 `/usr/share/zoneinfo` `/usr/share/locale` | `xim-x-tzdata` + `xim-x-locale` 小 payload | +| **CLI 工具运行** | 已 hermetic(自带 glibc/gcc/llvm) | 现状即目标态,无变化 | + +**重要**:并非所有场景要一次性做完。策略给的是**判据**;每次新包 review 时按判据决定 —— 这样以后不会有人再走"这次借一下 host"的短路。 + +--- + +## 4. 方案对比:GPU 边界四条路 + +聚焦最难的 GPU 场景,横向对比业界四种解法: + +### 4.1 Flatpak — 版本化 runtime,一驱动一份 + +- **机制**:extension `org.freedesktop.Platform.GL.nvidia-575-42-01` 等,一版一份;首次运行探 `/proc/driver/nvidia/version` 拉对应 +- **关键细节**:extension 里的 `.so` **重新链接到 Flatpak runtime 的 glibc**,不是 NVIDIA `.run` 原样;所以沙盒里 loader 加载它 glibc 天然对齐 +- **代价**:CDN 存几百份(每份 ~200MB)、老驱动 extension 永远不能删 +- **优点**:工程简单、5+ 年验证、有现成 CI 模板可参考 + +### 4.2 NixOS — 驱动是 nix derivation + +- **机制**:`hardware.nvidia.package` 从 NVIDIA `.run` 抽源码,**用 nixpkgs 的 gcc + kernel headers 一起 build**,产出 `nvidia.ko` + 所有 `.so`,挂 `/run/opengl-driver/lib/` +- **glibc 对齐方式**:整个 NixOS 是**同一版 nixpkgs channel**,glibc 全局一致 +- **代价**:xlings 必须接管 kernel module 编译(每 kernel 版本 × 每 driver 版本的构建矩阵)+ nixpkgs 级别的社区规模 +- **不适合近期**:接管 kernel module 是 xlings 团队规模两个数量级之外的承诺 + +### 4.3 传统发行版(Debian/Fedora)— 没沙盒,自然没冲突 + +- **机制**:`nvidia-driver-575` / `akmod-nvidia`,DKMS 在每次 kernel 升级时重 build,userspace `.so` 按发行版 glibc 编,落 `/usr/lib*` +- **glibc 对齐方式**:整个系统只有一个 glibc,不存在两份 +- **代价**:xlings 需要成为**真正的发行版**(rootfs 级、init 级、包管理器级),超出定位 + +### 4.4 Steam pressure-vessel + libcapsule — linker namespace 分层 + +- **机制**:利用 glibc `dlmopen` + `LM_ID_NEWLM`,让**同一进程内同时容纳两份 glibc** —— 沙盒 glibc 归应用,宿主 glibc 归 GL 驱动,由 `libcapsule` proxy 做跨 namespace marshaling +- **glibc 对齐方式**:根本不需要对齐,通过 linker namespace 隔离 +- **代价**:GL 近千函数需生成 proxy stub(可自动化,但每次 GL 版本升要跟);目前只 Valve/Collabora 在维护 +- **技术上最优雅**,但**工程量对小团队不现实** —— 作为未来兜底方案保留 + +### 4.5 决策矩阵 + +| 方案 | 存储代价 | 首次运行延迟 | 工程复杂度 | Kernel 侧责任 | xlings 定位契合度 | +|---|---|---|---|---|---| +| Flatpak | 高(几百份) | 高(下 ~200MB) | **低** | 无 | ★★★★ | +| NixOS | 中 | 无 | 中 | **接管 `.ko` 编译** | ★★(超定位) | +| 传统发行版 | 低 | 无 | 低(政策成本高) | 全接管 | ★(超定位) | +| pressure-vessel | 低(宿主单份) | 无 | **极高** | 无 | ★★★★★(但远期) | + +**近期最优 = Flatpak-style,兼顾工程可行与定位契合**。 + +**长期不封堵 pressure-vessel-style**,一旦 GLVND 抽象已就位、包元数据已齐备,`libcapsule` 只是"实现细节替换",不需要改架构。这就是"最优选择 + 未来可迁移"的具体含义。 + +--- + +## 5. 推荐近期方案(P0 → P1) + +### 5.1 P0 — 三个最小落地(2-4 周) + +1. **明文写策略**(本文即是初稿),在 xlings 主 repo `docs/design/` 下发一份 policy doc +2. **补 L1/L2 止血**: + - L1:向 sudoevolve/EUI-NEO 上游提 PR,`glfwSetErrorCallback` + 各失败点 stderr(**约 30 行**) + - L2:改 `compat.glx-runtime.lua` 候选目录顺序 + ELF class 探测(约 10 行)—— **不解决 L3,但让用户至少能看到真实错误** +3. **发 `compat.mesa` payload**(Intel/AMD 路径的第一个 non-trivial 落地): + - 内容:libgallium + libGLX_mesa + libEGL_mesa + libdrm(必要项)+ shader compiler 依赖的 llvm-runtime + - 大小预算:< 500MB 压缩后 + - 依赖闭包必须审:任何 `.so` `DT_NEEDED` 指向宿主 = 缺包 + - **capabilities**:`opengl.glvnd`、`opengl.mesa.driver` + - **capabilities_host**:`kernel.drm`(附 reason) + +### 5.2 P1 — 4-6 周 + +4. **加 `capabilities_host` 元数据字段** + 索引侧校验: + - `xim-pkgindex` 与 `mcpp-index` 的 lua 包描述加字段 + - 校验脚本(参考 `check_version_pins.sh`、`check_cross_package_refs.lua` 的做法):静态扫描每个包的 dlopen list,与 `capabilities_host` 允许集比对,不匹配报错 +5. **NVIDIA 分支**(Flatpak-style): + - `xim-x-nvidia-driver-575` `xim-x-nvidia-driver-570` 等一组 payload + - xlings-res CI 自动从 NVIDIA `.run` 抽包 → 重链到 xim-x-glibc → 发 release + - 首装时探测:读 `/proc/driver/nvidia/version` → 拉对应 + - **kernel module 不接管**:如果用户 kernel 上没有 `nvidia.ko`,报明确错误("请通过发行版安装 nvidia driver kernel module"),不试图代劳 +6. **CI 空-host 校验**: + - bwrap 构一个 host 只有 `/dev /proc /sys /tmp` 的 rootfs + - 在其中跑 `mcpp build && mcpp test && mcpp run`(选一组 examples) + - 任何 `LD_DEBUG=libs` 显示的 search path 命中 `/usr/lib*` 都失败 + - 参考 mcpp `link-argv-max-arg-strlen` 教训:**没有真实覆盖 = 没有验证** + +### 5.3 P2 — 3-6 个月 + +7. **其他 hermetic 缺口**(章节 3 的表)按优先级补:CA bundle、fontconfig、pulseaudio client → dbus → cups +8. **Release 套装模型**(参考 Flatpak runtime `//24.08`): + - 引入 `xlings-platform-2026.08` 这样的**同步版本集合** + - Mesa + libdrm + libX11 + xim-x-glibc + libc++ 等作为**一组**发布,内部互测通过 + - `.xlings.json` 可 pin 平台版本而非每个包单独 pin +9. **考虑 linker namespace 兜底**(P3,不承诺): + - 如果 NVIDIA payload 矩阵开始成本失控(比如 Chinese fork 分支太多),或用户"就是插了没见过的 GPU"的场景增多,启动 `libcapsule` 抽象研究 + +--- + +## 6. 未来可迁移路径(**不封堵、不预实现**) + +以下都是**在近期方案架构上可增量演化**的迁移方向,现在**不做**但**不封堵**: + +### 6.1 → NixOS-style 接管 kernel module + +- **触发条件**:xlings subos 深化到自建 kernel(现在只是 sandbox 层),或**社区规模足以维护 driver × kernel 编译矩阵** +- **迁移复杂度**:`xim-x-nvidia-driver-XXX` payload 现有元数据结构可原地扩展,增加 `kmod` 段,由 xlings 侧编译 +- **保留的选择**:即使做了,也可以保留 Flatpak-style 作为 fallback(用户没有 build toolchain 时) + +### 6.2 → pressure-vessel-style linker namespace + +- **触发条件**:NVIDIA payload 矩阵爆炸,或 pre-built 二进制场景增多 +- **迁移复杂度**:GLVND 层已经就位(方案 4.1 已在做 GLVND 分发),`libcapsule` 只替换分发层实现 +- **保留的选择**:可以只对 GL 一族做 capsule 化,其他包(音频、DBus)仍用普通 hermetic + +### 6.3 → Release 套装 → Rolling / LTS 双通道 + +- **触发条件**:企业/教育场景要求"两年不动" +- **迁移复杂度**:`xim-platform-YYYY.MM` 的语义已经支持"套装",加个 `.lts` 标签即可 +- **保留的选择**:rolling 用户零感知 + +### 6.4 → 完整 rootfs distro + +- **触发条件**:xlings 定位主动扩大(未来讨论,不现在承诺) +- **迁移复杂度**:显著,涉及 init/包管理器/kernel +- **本文明确**:P0-P2 不做这一步 + +--- + +## 7. 明确不做(non-goals) + +以下事项即使技术可行也**不做**,避免任务蔓延: + +- **不发行 kernel**:kernel 由宿主提供,`capabilities_host = kernel.*` 是永久边界 +- **不接管 NVIDIA kernel module 编译**(近期):DKMS 交给宿主发行版 +- **不发行 systemd/init**:xlings 是用户态,不管进程 1 +- **不做 Flatpak-style 应用沙盒**:xlings 是**开发工具生态**,不是应用分发;subos 有 sandbox 但那是隔离,不是应用运行时 +- **不追 glibc 版本**:2.39 就是 2.39,不发 2.40/2.41/...(除非有独立强 need);策略从"追新"转向"收敛边界" + +--- + +## 8. 开放问题(等实施前决策) + +1. **NVIDIA payload 命名 schema**: + - Flatpak: `nvidia-575-42-01`(3 段) + - 是否需要区分 open-kernel-module vs proprietary?(NVIDIA 从 R515 起有 open kernel module,但 userspace 仍闭源) + - 建议参考 Flatpak 命名,后续按需扩展 +2. **capabilities_host 声明的 review 边界**: + - 谁批准新增一条? + - 是否需要每季度 audit 一次 `capabilities_host` 集合避免膨胀? +3. **兼容"混合环境"用户**: + - 一台机器同时有 Intel iGPU 和 NVIDIA dGPU(Optimus) + - GLVND 分发能处理(靠 `__NV_PRIME_RENDER_OFFLOAD` 或 `DRI_PRIME`),但需要 payload 都装齐 + - 探测策略:探到 `/proc/driver/nvidia/version` 就装 NVIDIA payload,同时也装 Mesa payload +4. **payload 首次下载体验**: + - Mesa payload ~500MB、NVIDIA payload ~200MB,首次运行 GUI 项目会明显停顿 + - 是否 seed 到 xlings quick_install?或让用户显式 `xlings install compat.mesa`? + - 建议参考 Flatpak:第一次 `mcpp run` 触发下载,显示进度条,明确告知 +5. **`compat.mesa` 与 `xim-x-mesa` 的归属**: + - mcpp-index 归属 → 用户项目侧 + - xim-pkgindex 归属 → 工具链侧 + - GL 库属于 runtime,理论上两侧都可以;建议放 **xim-pkgindex**(更接近"用户态发行版底座") +6. **老 mcpp 二进制的兼容**: + - 已发布 mcpp 二进制 PT_INTERP 是烙定的 + - 新策略下产出的二进制才 hermetic + - 需要**发版说明明确**:某个 mcpp 版本起 GUI 项目走新路径 + +--- + +## 9. 与已有 xlings 文档的关系 + +- **不冲突**:本文是**上位策略**,不推翻任何已有实施设计 +- **补齐**: + - `2026-05-22-subos-sandbox-gpu-passthrough.md` 解决"sandbox 里能看到 GPU 设备节点",本文解决"看到之后跑什么 GL 库"—— 一上一下,配套 + - `2026-06-26-multiarch-package-description-design.md` 提供 payload 的多架构表达,本文新增的所有 payload 沿用 + - `2026-06-21-linux-root-usability-survey.md` 讨论权限边界,本文的 `capabilities_host` 是"资源边界"版本,理念一致 +- **更新**:本文落地后,`compat.glx-runtime.lua` 那种"直接 symlink 宿主"的模式应逐步淘汰,mcpp-index 的 `docs/CONTRIBUTING` 应加"新包不得直接 symlink 宿主 `.so`"约束 + +--- + +## 10. 一句话总括 + +> **xlings 是"用户态 Linux 发行版",执行"能不依赖宿主就不依赖宿主"策略。glibc 冲突是策略未明确时的症状;把边界写清楚,症状自然消失。近期抄 Flatpak-style 的做法把 GPU 边界收敛;长期不封堵 NixOS-style / pressure-vessel-style / rolling+LTS 等演化路径。** diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index b972a780..d4677112 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -50,14 +50,36 @@ jobs: xlings-deps-${{ runner.os }}- - name: Cache mcpp sandbox and project packages + id: mcpp-cache uses: actions/cache@v4 with: path: | ~/.mcpp .mcpp - key: mcpp-${{ runner.os }}-${{ hashFiles('mcpp.toml', 'mcpp.lock', '.xlings.json') }}-${{ env.BOOTSTRAP_XLINGS_VERSION }} + # `mcpp-v2-`: the v1 keyspace holds entries saved by runs that + # built against a fallback-restored BMI set and failed. Those are + # poisoned at their exact key, so the guard below -- which only fires + # on an INEXACT restore -- can never reach them. Retiring the prefix + # is what discards them; the guard is what stops it recurring. + key: mcpp-v2-${{ runner.os }}-${{ hashFiles('mcpp.toml', 'mcpp.lock', '.xlings.json') }}-${{ env.BOOTSTRAP_XLINGS_VERSION }} restore-keys: | - mcpp-${{ runner.os }}- + mcpp-v2-${{ runner.os }}- + # An inexact restore (the `restore-keys` fallback) brings BMIs built + # under a different dependency set. Mixing them with anything rebuilt + # here fails as `import 'std' has CRC mismatch` — which reads like a + # compiler bug and is a cache restored for the wrong inputs. Measured on + # PR #480, where `mcpplibs.xpkg` went 0.0.47 → 0.0.48: the fallback + # cache's tinyhttps/cmdline BMIs no longer matched the `std` the fresh + # xpkg build produced. + # + # Only the BMIs go. The payload store under registry/data/xpkgs is + # dependency-independent and is what makes the ~800 MB cache worth + # restoring at all. + - name: Drop stale BMIs when the cache key did not match exactly + if: steps.mcpp-cache.outputs.cache-hit != 'true' + shell: bash + run: | + rm -rf ~/.mcpp/bmi ~/.mcpp/build-cache .mcpp/bmi .mcpp/build-cache - name: Prepare bootstrap package index run: | @@ -206,14 +228,31 @@ jobs: xlings-deps-${{ runner.os }}- - name: Cache mcpp sandbox and project packages + id: mcpp-cache uses: actions/cache@v4 with: path: | ~/.mcpp .mcpp - key: mcpp-${{ runner.os }}-${{ hashFiles('mcpp.toml', 'mcpp.lock', '.xlings.json') }}-${{ env.BOOTSTRAP_XLINGS_VERSION }} + # `mcpp-v2-`: the v1 keyspace holds entries saved by runs that + # built against a fallback-restored BMI set and failed. Those are + # poisoned at their exact key, so the guard below -- which only fires + # on an INEXACT restore -- can never reach them. Retiring the prefix + # is what discards them; the guard is what stops it recurring. + key: mcpp-v2-${{ runner.os }}-${{ hashFiles('mcpp.toml', 'mcpp.lock', '.xlings.json') }}-${{ env.BOOTSTRAP_XLINGS_VERSION }} restore-keys: | - mcpp-${{ runner.os }}- + mcpp-v2-${{ runner.os }}- + # An inexact restore brings BMIs built under a different dependency set. + # Mixing them with anything rebuilt here fails as `import 'std' has CRC + # mismatch`, which reads like a compiler bug. Only the BMIs go -- the + # payload store is dependency-independent and is what makes the ~800 MB + # cache worth restoring. + - name: Drop stale BMIs when the cache key did not match exactly + if: steps.mcpp-cache.outputs.cache-hit != 'true' + shell: bash + run: | + rm -rf ~/.mcpp/bmi ~/.mcpp/build-cache .mcpp/bmi .mcpp/build-cache + - name: Prepare bootstrap package index run: | @@ -297,14 +336,31 @@ jobs: xlings-deps-${{ runner.os }}- - name: Cache mcpp sandbox and project packages + id: mcpp-cache uses: actions/cache@v4 with: path: | ~\.mcpp .mcpp - key: mcpp-${{ runner.os }}-${{ hashFiles('mcpp.toml', 'mcpp.lock', '.xlings.json') }}-${{ env.BOOTSTRAP_XLINGS_VERSION }} + # `mcpp-v2-`: the v1 keyspace holds entries saved by runs that + # built against a fallback-restored BMI set and failed. Those are + # poisoned at their exact key, so the guard below -- which only fires + # on an INEXACT restore -- can never reach them. Retiring the prefix + # is what discards them; the guard is what stops it recurring. + key: mcpp-v2-${{ runner.os }}-${{ hashFiles('mcpp.toml', 'mcpp.lock', '.xlings.json') }}-${{ env.BOOTSTRAP_XLINGS_VERSION }} restore-keys: | - mcpp-${{ runner.os }}- + mcpp-v2-${{ runner.os }}- + # An inexact restore brings BMIs built under a different dependency set. + # Mixing them with anything rebuilt here fails as `import 'std' has CRC + # mismatch`, which reads like a compiler bug. Only the BMIs go -- the + # payload store is dependency-independent and is what makes the ~800 MB + # cache worth restoring. + - name: Drop stale BMIs when the cache key did not match exactly + if: steps.mcpp-cache.outputs.cache-hit != 'true' + shell: bash + run: | + rm -rf ~/.mcpp/bmi ~/.mcpp/build-cache .mcpp/bmi .mcpp/build-cache + - name: Prepare bootstrap package index shell: pwsh diff --git a/.github/workflows/xlings-ci-linux-e2e.yml b/.github/workflows/xlings-ci-linux-e2e.yml index 35d40841..2806fd07 100644 --- a/.github/workflows/xlings-ci-linux-e2e.yml +++ b/.github/workflows/xlings-ci-linux-e2e.yml @@ -58,14 +58,36 @@ jobs: xlings-deps-${{ runner.os }}- - name: Cache mcpp sandbox and project packages + id: mcpp-cache uses: actions/cache@v4 with: path: | ~/.mcpp .mcpp - key: mcpp-${{ runner.os }}-${{ hashFiles('mcpp.toml', 'mcpp.lock', '.xlings.json') }}-${{ env.BOOTSTRAP_XLINGS_VERSION }} + # `mcpp-v2-`: the v1 keyspace holds entries saved by runs that + # built against a fallback-restored BMI set and failed. Those are + # poisoned at their exact key, so the guard below -- which only fires + # on an INEXACT restore -- can never reach them. Retiring the prefix + # is what discards them; the guard is what stops it recurring. + key: mcpp-v2-${{ runner.os }}-${{ hashFiles('mcpp.toml', 'mcpp.lock', '.xlings.json') }}-${{ env.BOOTSTRAP_XLINGS_VERSION }} restore-keys: | - mcpp-${{ runner.os }}- + mcpp-v2-${{ runner.os }}- + # An inexact restore (the `restore-keys` fallback) brings BMIs built + # under a different dependency set. Mixing them with anything rebuilt + # here fails as `import 'std' has CRC mismatch` — which reads like a + # compiler bug and is a cache restored for the wrong inputs. Measured on + # PR #480, where `mcpplibs.xpkg` went 0.0.47 → 0.0.48: the fallback + # cache's tinyhttps/cmdline BMIs no longer matched the `std` the fresh + # xpkg build produced. + # + # Only the BMIs go. The payload store under registry/data/xpkgs is + # dependency-independent and is what makes the ~800 MB cache worth + # restoring at all. + - name: Drop stale BMIs when the cache key did not match exactly + if: steps.mcpp-cache.outputs.cache-hit != 'true' + shell: bash + run: | + rm -rf ~/.mcpp/bmi ~/.mcpp/build-cache .mcpp/bmi .mcpp/build-cache - name: Prepare bootstrap package index run: | diff --git a/.github/workflows/xlings-ci-linux-root.yml b/.github/workflows/xlings-ci-linux-root.yml index a7c20ed7..a4377c62 100644 --- a/.github/workflows/xlings-ci-linux-root.yml +++ b/.github/workflows/xlings-ci-linux-root.yml @@ -55,14 +55,36 @@ jobs: xlings-deps-${{ runner.os }}- - name: Cache mcpp sandbox and project packages + id: mcpp-cache uses: actions/cache@v4 with: path: | ~/.mcpp .mcpp - key: mcpp-${{ runner.os }}-${{ hashFiles('mcpp.toml', 'mcpp.lock', '.xlings.json') }}-${{ env.BOOTSTRAP_XLINGS_VERSION }} + # `mcpp-v2-`: the v1 keyspace holds entries saved by runs that + # built against a fallback-restored BMI set and failed. Those are + # poisoned at their exact key, so the guard below -- which only fires + # on an INEXACT restore -- can never reach them. Retiring the prefix + # is what discards them; the guard is what stops it recurring. + key: mcpp-v2-${{ runner.os }}-${{ hashFiles('mcpp.toml', 'mcpp.lock', '.xlings.json') }}-${{ env.BOOTSTRAP_XLINGS_VERSION }} restore-keys: | - mcpp-${{ runner.os }}- + mcpp-v2-${{ runner.os }}- + # An inexact restore (the `restore-keys` fallback) brings BMIs built + # under a different dependency set. Mixing them with anything rebuilt + # here fails as `import 'std' has CRC mismatch` — which reads like a + # compiler bug and is a cache restored for the wrong inputs. Measured on + # PR #480, where `mcpplibs.xpkg` went 0.0.47 → 0.0.48: the fallback + # cache's tinyhttps/cmdline BMIs no longer matched the `std` the fresh + # xpkg build produced. + # + # Only the BMIs go. The payload store under registry/data/xpkgs is + # dependency-independent and is what makes the ~800 MB cache worth + # restoring at all. + - name: Drop stale BMIs when the cache key did not match exactly + if: steps.mcpp-cache.outputs.cache-hit != 'true' + shell: bash + run: | + rm -rf ~/.mcpp/bmi ~/.mcpp/build-cache .mcpp/bmi .mcpp/build-cache - name: Prepare bootstrap package index run: | diff --git a/.github/workflows/xlings-ci-linux.yml b/.github/workflows/xlings-ci-linux.yml index 6e865b1f..1f4e444a 100644 --- a/.github/workflows/xlings-ci-linux.yml +++ b/.github/workflows/xlings-ci-linux.yml @@ -60,14 +60,36 @@ jobs: xlings-deps-${{ runner.os }}- - name: Cache mcpp sandbox and project packages + id: mcpp-cache uses: actions/cache@v4 with: path: | ~/.mcpp .mcpp - key: mcpp-${{ runner.os }}-${{ hashFiles('mcpp.toml', 'mcpp.lock', '.xlings.json') }}-${{ env.BOOTSTRAP_XLINGS_VERSION }} + # `mcpp-v2-`: the v1 keyspace holds entries saved by runs that + # built against a fallback-restored BMI set and failed. Those are + # poisoned at their exact key, so the guard below -- which only fires + # on an INEXACT restore -- can never reach them. Retiring the prefix + # is what discards them; the guard is what stops it recurring. + key: mcpp-v2-${{ runner.os }}-${{ hashFiles('mcpp.toml', 'mcpp.lock', '.xlings.json') }}-${{ env.BOOTSTRAP_XLINGS_VERSION }} restore-keys: | - mcpp-${{ runner.os }}- + mcpp-v2-${{ runner.os }}- + # An inexact restore (the `restore-keys` fallback) brings BMIs built + # under a different dependency set. Mixing them with anything rebuilt + # here fails as `import 'std' has CRC mismatch` — which reads like a + # compiler bug and is a cache restored for the wrong inputs. Measured on + # PR #480, where `mcpplibs.xpkg` went 0.0.47 → 0.0.48: the fallback + # cache's tinyhttps/cmdline BMIs no longer matched the `std` the fresh + # xpkg build produced. + # + # Only the BMIs go. The payload store under registry/data/xpkgs is + # dependency-independent and is what makes the ~800 MB cache worth + # restoring at all. + - name: Drop stale BMIs when the cache key did not match exactly + if: steps.mcpp-cache.outputs.cache-hit != 'true' + shell: bash + run: | + rm -rf ~/.mcpp/bmi ~/.mcpp/build-cache .mcpp/bmi .mcpp/build-cache - name: Prepare bootstrap package index run: | diff --git a/.github/workflows/xlings-ci-macos.yml b/.github/workflows/xlings-ci-macos.yml index ca885e8b..ce0a111b 100644 --- a/.github/workflows/xlings-ci-macos.yml +++ b/.github/workflows/xlings-ci-macos.yml @@ -50,14 +50,36 @@ jobs: xlings-deps-${{ runner.os }}- - name: Cache mcpp sandbox and project packages + id: mcpp-cache uses: actions/cache@v4 with: path: | ~/.mcpp .mcpp - key: mcpp-${{ runner.os }}-dt110-${{ hashFiles('mcpp.toml', 'mcpp.lock', '.xlings.json') }}-${{ env.BOOTSTRAP_XLINGS_VERSION }} + # `mcpp-v2-`: the v1 keyspace holds entries saved by runs that + # built against a fallback-restored BMI set and failed. Those are + # poisoned at their exact key, so the guard below -- which only fires + # on an INEXACT restore -- can never reach them. Retiring the prefix + # is what discards them; the guard is what stops it recurring. + key: mcpp-v2-${{ runner.os }}-dt110-${{ hashFiles('mcpp.toml', 'mcpp.lock', '.xlings.json') }}-${{ env.BOOTSTRAP_XLINGS_VERSION }} restore-keys: | - mcpp-${{ runner.os }}-dt110- + mcpp-v2-${{ runner.os }}-dt110- + # An inexact restore (the `restore-keys` fallback) brings BMIs built + # under a different dependency set. Mixing them with anything rebuilt + # here fails as `import 'std' has CRC mismatch` — which reads like a + # compiler bug and is a cache restored for the wrong inputs. Measured on + # PR #480, where `mcpplibs.xpkg` went 0.0.47 → 0.0.48: the fallback + # cache's tinyhttps/cmdline BMIs no longer matched the `std` the fresh + # xpkg build produced. + # + # Only the BMIs go. The payload store under registry/data/xpkgs is + # dependency-independent and is what makes the ~800 MB cache worth + # restoring at all. + - name: Drop stale BMIs when the cache key did not match exactly + if: steps.mcpp-cache.outputs.cache-hit != 'true' + shell: bash + run: | + rm -rf ~/.mcpp/bmi ~/.mcpp/build-cache .mcpp/bmi .mcpp/build-cache - name: Prepare bootstrap package index run: | diff --git a/.github/workflows/xlings-ci-windows.yml b/.github/workflows/xlings-ci-windows.yml index 78bf3078..b9c6bfc7 100644 --- a/.github/workflows/xlings-ci-windows.yml +++ b/.github/workflows/xlings-ci-windows.yml @@ -50,14 +50,36 @@ jobs: xlings-deps-${{ runner.os }}- - name: Cache mcpp sandbox and project packages + id: mcpp-cache uses: actions/cache@v4 with: path: | ~\.mcpp .mcpp - key: mcpp-${{ runner.os }}-${{ hashFiles('mcpp.toml', 'mcpp.lock', '.xlings.json') }}-${{ env.BOOTSTRAP_XLINGS_VERSION }} + # `mcpp-v2-`: the v1 keyspace holds entries saved by runs that + # built against a fallback-restored BMI set and failed. Those are + # poisoned at their exact key, so the guard below -- which only fires + # on an INEXACT restore -- can never reach them. Retiring the prefix + # is what discards them; the guard is what stops it recurring. + key: mcpp-v2-${{ runner.os }}-${{ hashFiles('mcpp.toml', 'mcpp.lock', '.xlings.json') }}-${{ env.BOOTSTRAP_XLINGS_VERSION }} restore-keys: | - mcpp-${{ runner.os }}- + mcpp-v2-${{ runner.os }}- + # An inexact restore (the `restore-keys` fallback) brings BMIs built + # under a different dependency set. Mixing them with anything rebuilt + # here fails as `import 'std' has CRC mismatch` — which reads like a + # compiler bug and is a cache restored for the wrong inputs. Measured on + # PR #480, where `mcpplibs.xpkg` went 0.0.47 → 0.0.48: the fallback + # cache's tinyhttps/cmdline BMIs no longer matched the `std` the fresh + # xpkg build produced. + # + # Only the BMIs go. The payload store under registry/data/xpkgs is + # dependency-independent and is what makes the ~800 MB cache worth + # restoring at all. + - name: Drop stale BMIs when the cache key did not match exactly + if: steps.mcpp-cache.outputs.cache-hit != 'true' + shell: bash + run: | + rm -rf ~/.mcpp/bmi ~/.mcpp/build-cache .mcpp/bmi .mcpp/build-cache - name: Prepare bootstrap package index shell: pwsh diff --git a/docs/generated/command-reference.md b/docs/generated/command-reference.md index 68d5270c..fa32f6f6 100644 --- a/docs/generated/command-reference.md +++ b/docs/generated/command-reference.md @@ -64,7 +64,7 @@ Manage SubOS environments Create a SubOS -Options: `--storage ` — shared, tmpfs or image; `--image-size ` — Image size; `--from ` — Fork source +Options: `--storage ` — shared, tmpfs or image; `--image-size ` — Image size; `--from ` — Fork source; `--runtime ` — Runtime binding, e.g. glibc@2.39 ## `xlings subos use ` diff --git a/docs/spec/xlings-json-schema.md b/docs/spec/xlings-json-schema.md index 2fe34a6e..fb7e91a7 100644 --- a/docs/spec/xlings-json-schema.md +++ b/docs/spec/xlings-json-schema.md @@ -159,9 +159,48 @@ | 字段 | 类型 | 说明 | |------|------|------| | `workspace` | `object` | 已安装工具的版本状态（见下方格式） | +| `subos_info` | `object` | SubOS 自身的描述：运行时与环境声明（见下方） | | `storage` | `string` | 存储模式：`"shared"`、`"tmpfs"`、`"image"` | | `imageSize` | `string` | 当 storage 为 `image` 时的磁盘映像大小（如 `"4G"`） | +### subos_info（2026.8.5+） + +`workspace` 记录的是这个 SubOS **装了什么**；`subos_info` 记录的是它**是什么** —— +二进制针对哪个运行时构建，以及进入它的进程需要哪些环境变量。 + +```json +"subos_info": { + "schema_version": 1, + "runtime": "glibc@2.39", + "envs": { + "compat.mesa@25.0.0": [ + { "var": "LIBGL_DRIVERS_PATH", "op": "set", "value": "${pkgdir}/lib/dri" }, + { "var": "XDG_DATA_DIRS", "op": "prepend", "value": "${pkgdir}/share" } + ] + }, + "created_at": "2026-08-05T14:23:11Z", + "created_by": "xlings 2026.8.5.1" +} +``` + +| 字段 | 类型 | 说明 | +|------|------|------| +| `schema_version` | `integer` | 当前为 `1` | +| `runtime` | `string` | 运行时 binding `@`，自描述（`glibc@2.39` 即 Linux/glibc）。由 `xlings subos new --runtime` 指定 | +| `envs` | `object` | 以**声明包的 binding** 为键。包卸载时 xlings 删除整段；recipe 不写清理代码 | +| `created_at` / `created_by` | `string` | 创建时间与创建者版本 | + +`envs` 的值由包在 `config()` 里通过 `subos.env{}` 写入（见 xim-pkgindex 的 +xpackage-spec V2）。**值必须使用占位符** —— `${pkgdir}` / `${subosdir}` / +`${home}` / `${xlings_home}` —— 写死绝对路径会让这份描述只在写它的机器上成立。 +占位符在进入 SubOS 时展开。 + +`envs` 为空时保留 `{}`,不省略键：缺失和空是两种写法、同一个意思,会让每个读者都要处理两遍。 + +进入 SubOS(`xlings subos use`)时这些变量被注入进程；**用户自己已 export 的值优先**, +`set` 不覆盖它,`prepend` 仍然与它拼接。`xlings self doctor` 检查这一段的完整性、 +与已装包的一致性、占位符可解性,以及是否有多个包争抢同一变量。 + ### workspace 条目格式（SubOS） SubOS 工作区中每个工具支持三种值形式： @@ -228,6 +267,13 @@ SubOS 工作区中每个工具支持三种值形式： { "storage": "image", "imageSize": "4G", + "subos_info": { + "schema_version": 1, + "runtime": "glibc@2.39", + "envs": {}, + "created_at": "2026-08-05T14:23:11Z", + "created_by": "xlings 2026.8.5.1" + }, "workspace": { "gcc": { "active": "16.1.0", diff --git a/mcpp.lock b/mcpp.lock index dd0ef538..6518a6b1 100644 --- a/mcpp.lock +++ b/mcpp.lock @@ -33,7 +33,7 @@ hash = "fnv1a:3465dd0bd5d7aa20" [package."mcpplibs.xpkg"] namespace = "mcpplibs" -version = "0.0.47" -source = "index+mcpplibs@0.0.47" -hash = "fnv1a:0cd8e39df748a206" +version = "0.0.48" +source = "index+mcpplibs@0.0.48" +hash = "fnv1a:2e16f253753e1665" diff --git a/mcpp.toml b/mcpp.toml index 78fd25d3..49d31f93 100644 --- a/mcpp.toml +++ b/mcpp.toml @@ -1,6 +1,6 @@ [package] name = "xlings" -version = "2026.8.4.2" +version = "2026.8.5.1" description = "Universal package management infrastructure tool with SubOS isolation" license = "Apache-2.0" repo = "https://github.com/openxlings/xlings" @@ -39,7 +39,7 @@ libarchive = "3.8.7" [dependencies.mcpplibs] cmdline = "0.0.2" -xpkg = "0.0.47" +xpkg = "0.0.48" tinyhttps = "0.2.9" capi.lua = "0.0.3" diff --git a/src/cli/spec.cppm b/src/cli/spec.cppm index a805335d..95b513c6 100644 --- a/src/cli/spec.cppm +++ b/src/cli/spec.cppm @@ -63,7 +63,7 @@ const CommandSpec& root() { {"config", "Show or modify configuration", {}, {}, {{"--lang ", "Set language"}, {"--mirror ", "Set mirror"}, {"--add-xpkg ", "Add package recipe"}, {"--index-repo ", "Add index repository"}}, {}}, {"subos", "Manage SubOS environments", {}, {}, {}, { - {"new", "Create a SubOS", {}, {{"name", "SubOS name", true}}, {{"--storage ", "shared, tmpfs or image"}, {"--image-size ", "Image size"}, {"--from ", "Fork source"}}, {}}, + {"new", "Create a SubOS", {}, {{"name", "SubOS name", true}}, {{"--storage ", "shared, tmpfs or image"}, {"--image-size ", "Image size"}, {"--from ", "Fork source"}, {"--runtime ", "Runtime binding, e.g. glibc@2.39"}}, {}}, {"use", "Enter a SubOS", {}, {{"name", "SubOS name", true}}, {{"--global", "Persist the active SubOS"}, {"--shell [KIND]", "Emit shell activation code"}, {"--sandbox [BACKEND]", "Enable sandbox (bwrap or proot on Linux)"}, {"--cmd ", "Run one command"}, {"--keep", "Keep the namespace keeper"}, {"--no-keep", "Disable the namespace keeper"}, {"--ttl ", "Keeper idle timeout"}, {"--gpu", "Expose GPU devices (bwrap only)"}}, {}}, {"list", "List SubOS environments", {"ls"}, {}, {}, {}}, {"remove", "Remove a SubOS", {"rm"}, {{"name", "SubOS name", true}}, {}, {}}, diff --git a/src/core/config.cppm b/src/core/config.cppm index 8cfd17fa..a1716798 100644 --- a/src/core/config.cppm +++ b/src/core/config.cppm @@ -13,7 +13,7 @@ import xlings.core.xvm.db; namespace xlings { export struct Info { - static constexpr std::string_view VERSION = "2026.8.4.2"; + static constexpr std::string_view VERSION = "2026.8.5.1"; static constexpr std::string_view REPO = "https://github.com/openxlings/xlings"; }; diff --git a/src/core/subos.cppm b/src/core/subos.cppm index e39d2d23..2a2993a5 100644 --- a/src/core/subos.cppm +++ b/src/core/subos.cppm @@ -31,6 +31,7 @@ import xlings.core.xim.commands; // auto_install_backend_ needs cmd_install import xlings.core.subos.keeper; import xlings.core.subos.gpu; import xlings.core.subos.sandbox; +import xlings.core.subos.manifest; // Leaf module (std + json only). Same source for "is this a global option" // that the CLI validator uses, so the two cannot disagree about `--yes`. import xlings.cli.spec; @@ -140,11 +141,48 @@ void update_current_symlink_(EventStream& stream, // ───────────────────────────────────────────────────────────────────── +// Give a subos directory a `subos_info` block, or leave the one it has. +// +// Idempotent, and called from every path that produces a subos directory +// (create, new_from, and the migration of a subos made before this block +// existed). A subos without it violates invariant I4 and cannot be described, +// checked or entered with its environment. +// +// The block is added even when `.xlings.json` already exists, which is the +// difference from the surrounding code: the file predates the block, so +// "the file is there" does not mean "the subos describes itself". +bool ensure_subos_info_(const fs::path& dir, std::string_view runtime) { + auto json = read_config_json_(dir / ".xlings.json"); + if (!json.is_object()) json = nlohmann::json::object(); + if (!json.contains("workspace")) json["workspace"] = nlohmann::json::object(); + + // Only replace a block that is absent or unusable. Rewriting a valid one + // would discard the envs a package declared into it. + if (manifest::validate_block(json).empty()) return true; + + json[std::string(manifest::BLOCK)] = manifest::make_block( + runtime, std::format("xlings {}", Info::VERSION)); + try { + write_config_json_(dir / ".xlings.json", json); + } catch (const std::exception& e) { + log::error("failed to write subos manifest {}: {}", + (dir / ".xlings.json").string(), e.what()); + return false; + } + return true; +} + // Create a subos. V6: storage mode is a creation-time property // (`--storage image|tmpfs|shared`). Non-shared modes force sandbox // entry at use-time. The sandbox-private dirs are laid down lazily. +// +// `runtime` is the subos's declared runtime binding ("glibc@2.39"): what its +// binaries are built against. Empty means the built-in default. It is a +// creation-time property because changing it after the fact would invalidate +// every payload already installed. export int create(const std::string& name, const fs::path& customDir, sandbox::StorageMode storage, const std::string& imageSize, + const std::string& runtime, EventStream& stream) { auto& p = Config::paths(); @@ -157,6 +195,21 @@ export int create(const std::string& name, const fs::path& customDir, return 1; } + // Checked before anything is laid down. A malformed runtime that only + // surfaced at write time would leave a registered subos that cannot + // satisfy its own invariants. + const std::string effectiveRuntime = + runtime.empty() ? std::string(manifest::DEFAULT_RUNTIME) : runtime; + if (!manifest::is_binding(effectiveRuntime)) { + stream.emit(ErrorEvent{ + .code = ErrorCode::InvalidInput, + .message = "invalid --runtime '" + effectiveRuntime + + "' (expected @, e.g. glibc@2.39)", + .recoverable = false, + }); + return 1; + } + for (char c : name) { if (!std::isalnum(static_cast(c)) && c != '_' && c != '-') { stream.emit(ErrorEvent{ @@ -197,7 +250,17 @@ export int create(const std::string& name, const fs::path& customDir, j["storage"] = sandbox::storage_to_string_(storage); if (storage == sandbox::StorageMode::Image) j["imageSize"] = imageSize; + j[std::string(manifest::BLOCK)] = manifest::make_block( + effectiveRuntime, std::format("xlings {}", Info::VERSION)); write_config_json_(subosConfig, j); + } else if (!ensure_subos_info_(dir, effectiveRuntime)) { + stream.emit(ErrorEvent{ + .code = ErrorCode::Internal, + .message = "failed to write the subos manifest for '" + name + "'", + .recoverable = false, + .hint = "check write permission on " + subosConfig.string(), + }); + return 1; } // Image mode: create sparse ext4 image @@ -266,18 +329,58 @@ export int create(const std::string& name, const fs::path& customDir, return 1; } + // The subos is registered; check it can actually satisfy the invariants + // before saying so. A creation that reports success and leaves a subos + // that doctor immediately condemns is the failure mode this whole slice + // exists to remove -- "it happened" and "it worked" must not look alike. + if (auto findings = manifest::validate(dir); !findings.empty()) { + std::string detail; + for (const auto& f : findings) { + if (!detail.empty()) detail += "; "; + detail += std::string(manifest::describe(f.kind)); + if (!f.detail.empty()) detail += " (" + f.detail + ")"; + } + // Roll back to the state before the command: the registry entry first, + // since that is what makes the name unusable a second time. + (void)update_home_config(p.homeDir, [&](nlohmann::json& json) { + if (json.contains("subos") && json["subos"].is_object()) + json["subos"].erase(name); + return true; + }); + std::error_code rmec; + fs::remove_all(dir, rmec); + stream.emit(ErrorEvent{ + .code = ErrorCode::Internal, + .message = "subos '" + name + "' did not come out valid: " + detail, + .recoverable = false, + .hint = rmec + ? "rolled back the registry entry, but " + dir.string() + + " could not be removed -- delete it before retrying" + : "nothing was left behind; retry, or report this", + }); + return 1; + } + nlohmann::json payload; payload["name"] = name; payload["dir"] = dir.string(); payload["storage"] = sandbox::storage_to_string_(storage); + payload["runtime"] = effectiveRuntime; stream.emit(DataEvent{"subos_created", payload.dump()}); return 0; } -// Back-compat overload (no storage argument → shared). +// Back-compat overloads. Callers that predate the runtime argument get the +// built-in default, and callers that predate storage get shared as before. +export int create(const std::string& name, const fs::path& customDir, + sandbox::StorageMode storage, const std::string& imageSize, + EventStream& stream) { + return create(name, customDir, storage, imageSize, "", stream); +} + export int create(const std::string& name, const fs::path& customDir, EventStream& stream) { - return create(name, customDir, sandbox::StorageMode::Shared, "50G", stream); + return create(name, customDir, sandbox::StorageMode::Shared, "50G", "", stream); } // ───────────────────────────────────────────────────────────────────── @@ -424,7 +527,8 @@ fs::path locate_base_pkg_(const PkgRef& ref) { export int new_from(const std::string& name, const fs::path& customDir, sandbox::StorageMode storage, const std::string& imageSize, - const std::string& fromSpec, EventStream& stream) { + const std::string& fromSpec, const std::string& runtime, + EventStream& stream) { auto& p = Config::paths(); fs::path baseDir; @@ -498,7 +602,8 @@ export int new_from(const std::string& name, const fs::path& customDir, // Create target subos via standard `create`. This sets up // bin/lib/usr/generations, writes initial .xlings.json, optionally // creates home.img, and registers the subos. - if (auto rc = create(name, customDir, storage, imageSize, stream); rc != 0) { + if (auto rc = create(name, customDir, storage, imageSize, runtime, stream); + rc != 0) { return rc; } @@ -526,6 +631,18 @@ export int new_from(const std::string& name, const fs::path& customDir, subosCfg["imageSize"] = imageSize; else subosCfg.erase("imageSize"); + // Same restoration, same reason. copy_tree_ replaced the manifest create() + // wrote with the base's, and a base built before subos_info existed has + // none -- which would leave the fork registered and failing its own + // invariants. A base that does carry one keeps it: it describes the very + // content that was just copied in, envs included. + if (!subosCfg.contains("workspace")) + subosCfg["workspace"] = nlohmann::json::object(); + if (!manifest::validate_block(subosCfg).empty()) { + subosCfg[std::string(manifest::BLOCK)] = manifest::make_block( + runtime.empty() ? manifest::DEFAULT_RUNTIME : runtime, + std::format("xlings {}", Info::VERSION)); + } write_config_json_(subosCfgPath, subosCfg); // Re-mint subos shims (they may have been clobbered by copy_tree_ @@ -623,6 +740,87 @@ inline std::string rebuild_path_for_subos_(const std::string& orig_path, return out; } +// Where a provider's payload lives, for `${pkgdir}`. +// +// A binding is `@` and carries no namespace, while the store +// directory is `-x-` (or a bare `` for the primary index). So +// the bare spelling is tried first and the namespaced ones are found by scan. +// The scan is over the handful of bindings that actually appear in `envs`, +// not over the whole store. +// +// Returning empty is meaningful: expansion then leaves `${pkgdir}` in place +// rather than collapsing the value to a host path, and doctor D3 reports it. +inline fs::path pkgdir_for_binding_(std::string_view binding) { + const auto at = binding.find('@'); + if (at == std::string_view::npos) return {}; + const std::string name(binding.substr(0, at)); + const std::string version(binding.substr(at + 1)); + if (name.empty() || version.empty()) return {}; + + const auto store = Config::paths().dataDir / "xpkgs"; + std::error_code ec; + + if (auto direct = store / name / version; fs::is_directory(direct, ec)) + return direct; + + if (!fs::is_directory(store, ec)) return {}; + const auto suffix = "-x-" + name; + for (const auto& entry : platform::dir_entries(store)) { + if (!entry.is_directory(ec)) continue; + if (!entry.path().filename().string().ends_with(suffix)) continue; + if (auto candidate = entry.path() / version; + fs::is_directory(candidate, ec)) { + return candidate; + } + } + return {}; +} + +inline manifest::Placeholders placeholders_for_(const fs::path& subosDir) { + return manifest::Placeholders{ + .subosdir = subosDir, + .home = platform::get_home_dir(), + .xlings_home = Config::paths().homeDir, + .pkgdir_of = pkgdir_for_binding_, + }; +} + +// The variables a subos exports, resolved and ready to apply. +// +// Empty for a subos with no declarations, which is every subos until a package +// makes one -- so both call sites below stay silent in the common case. +inline std::vector subos_env_for_(const std::string& name) { + const auto dir = Config::subos_dir(name); + auto doc = manifest::read_document(dir); + if (!doc) return {}; + return manifest::resolve(manifest::parse(*doc), placeholders_for_(dir)); +} + +// UC-2: say what was injected. +// +// To stderr, always -- the `--shell` path's stdout is eval'd by the caller's +// shell, and this is a report, not code. Without it the user pipes an opaque +// blob into `eval` and cannot tell which package changed what. +inline void report_injected_env_(const std::string& subosName, + const std::vector& vars) { + if (vars.empty()) return; + std::set providers; + for (const auto& v : vars) + providers.insert(v.providers.begin(), v.providers.end()); + + std::println(stderr, "[xlings] subos {}: {} env var(s) from {} package(s)", + subosName, vars.size(), providers.size()); + for (const auto& v : vars) { + if (v.unresolved) { + std::println(stderr, "[xlings] {} — unresolved path, skipped " + "(run `xlings self doctor`)", v.var); + } else if (v.conflicted) { + std::println(stderr, "[xlings] {} — declared by {} packages, " + "conflicting", v.var, v.providers.size()); + } + } +} + } // namespace use_detail_ // Internal — not exported. `xlings subos use --global ` and @@ -695,6 +893,17 @@ int use_emit_shell(const std::string& name, bool is_pwsh = (shell_kind == "pwsh" || shell_kind == "powershell" || shell_kind == "ps1" || shell_kind == "ps"); + // The subos's own declared environment (GL driver paths, EGL vendor dirs, + // and whatever else a package needs a *user's* binary to see). Emitted + // after the xvm/PATH lines so a declaration cannot displace them. + // + // UC-1 -- a variable the user already exported wins. The emitted code + // tests the live variable rather than what this process happens to see: + // `--shell` output is frequently captured once and eval'd later, in a + // shell whose environment has moved on. + const auto envVars = use_detail_::subos_env_for_(name); + use_detail_::report_injected_env_(name, envVars); + if (is_fish) { std::println(R"(set -gx XLINGS_ACTIVE_SUBOS "{}";)", name); std::println(R"(set -gx XLINGS_BIN "{}";)", bin_dir.string()); @@ -702,6 +911,20 @@ int use_emit_shell(const std::string& name, // bin. fish's $PATH is a list, so we use string match -v. std::println(R"(set -gx PATH "{}" (string match -v -r "^{}/subos/[^/]+/bin$" -- $PATH);)", bin_dir.string(), p.homeDir.string()); + for (const auto& v : envVars) { + if (v.unresolved) continue; + // R"SH(...)SH": the fish source below contains `)"`, which ends a + // plain R"(...)" literal early -- and the truncation compiles, + // because what is left is still a valid string. + if (v.op == manifest::OP_PREPEND) { + std::println( + R"SH(if set -q {0}; set -gx {0} "{1}:${0}"; else; set -gx {0} "{1}"; end;)SH", + v.var, v.value); + } else { + std::println(R"SH(if not set -q {0}; set -gx {0} "{1}"; end;)SH", + v.var, v.value); + } + } return 0; } if (is_pwsh) { @@ -709,6 +932,19 @@ int use_emit_shell(const std::string& name, std::println(R"($env:XLINGS_BIN = '{}')", bin_dir.string()); std::println(R"($env:Path = '{}' + ';' + (($env:Path -split ';') -notmatch '^{}\\subos\\[^\\]+\\bin$' -join ';'))", bin_dir.string(), p.homeDir.string()); + for (const auto& v : envVars) { + if (v.unresolved) continue; + // ';' rather than ':' -- these are path lists, and on Windows the + // separator is the one the platform's own tools split on. + if (v.op == manifest::OP_PREPEND) { + std::println( + R"($env:{0} = if ($env:{0}) {{ '{1}' + ';' + $env:{0} }} else {{ '{1}' }})", + v.var, v.value); + } else { + std::println(R"(if (-not $env:{0}) {{ $env:{0} = '{1}' }})", + v.var, v.value); + } + } return 0; } // POSIX (sh/bash/zsh) default @@ -718,6 +954,17 @@ int use_emit_shell(const std::string& name, std::println(R"(export XLINGS_ACTIVE_SUBOS="{}";)", name); std::println(R"(export XLINGS_BIN="{}";)", bin_dir.string()); std::println(R"(export PATH="{}";)", new_path); + for (const auto& v : envVars) { + if (v.unresolved) continue; + if (v.op == manifest::OP_PREPEND) { + // ${VAR:+:$VAR} appends the separator only when VAR is non-empty, + // so an unset variable does not become a trailing ':' -- which an + // empty PATH-list element reads as "the current directory". + std::println(R"(export {0}="{1}${{{0}:+:${0}}}";)", v.var, v.value); + } else { + std::println(R"(: "${{{0}:={1}}}"; export {0};)", v.var, v.value); + } + } return 0; } @@ -797,6 +1044,30 @@ int use_spawn_shell(const std::string& name, EventStream& stream, platform::set_env_variable("XLINGS_BIN", bin_dir.string()); platform::set_env_variable("PATH", new_path); + // The subos's declared environment, applied to this process before it is + // replaced -- so the shell (or the single `--cmd`) inherits it, and so + // does every user binary run inside. This is the path that matters for + // issue #352: nothing xlings wraps needs LIBGL_DRIVERS_PATH, the user's + // own GL program does. + // + // UC-1 -- a variable already set in this environment is the user's, and + // `set` leaves it alone. `prepend` still contributes, since composing is + // what prepend means. + { + const auto envVars = use_detail_::subos_env_for_(name); + use_detail_::report_injected_env_(name, envVars); + for (const auto& v : envVars) { + if (v.unresolved) continue; + const auto existing = utils::get_env_or_default(v.var); + if (v.op == manifest::OP_PREPEND) { + platform::set_env_variable( + v.var, existing.empty() ? v.value : v.value + ":" + existing); + } else if (existing.empty()) { + platform::set_env_variable(v.var, v.value); + } + } + } + nlohmann::json payload; payload["name"] = name; payload["mode"] = "spawn"; @@ -1013,9 +1284,20 @@ export int run(int argc, char* argv[], EventStream& stream) { // (auto-installs the base xpkg if missing); bare name is treated // as a local subos to fork from. std::string fromSpec; + // --runtime : what this subos's binaries are built against + // ("glibc@2.39"). Creation-time, because changing it later would + // invalidate every payload already installed. Absent → the built-in + // default, so existing invocations keep working unchanged. + std::string runtime; for (int i = 3; i < argc; ++i) { std::string a = argv[i]; - if (a == "--storage" && i + 1 < argc) { + if (a == "--runtime" && i + 1 < argc) { + runtime = argv[++i]; + } + else if (a.rfind("--runtime=", 0) == 0) { + runtime = a.substr(10); + } + else if (a == "--storage" && i + 1 < argc) { auto s = std::string(argv[++i]); if (s == "image") storage = sandbox::StorageMode::Image; else if (s == "tmpfs") storage = sandbox::StorageMode::Tmpfs; @@ -1048,9 +1330,9 @@ export int run(int argc, char* argv[], EventStream& stream) { return 1; } if (!fromSpec.empty()) { - return new_from(name, {}, storage, imageSize, fromSpec, stream); + return new_from(name, {}, storage, imageSize, fromSpec, runtime, stream); } - return create(name, {}, storage, imageSize, stream); + return create(name, {}, storage, imageSize, runtime, stream); } if (sub == "use") { // Flags supported: diff --git a/src/core/subos/manifest.cppm b/src/core/subos/manifest.cppm new file mode 100644 index 00000000..65d3808b --- /dev/null +++ b/src/core/subos/manifest.cppm @@ -0,0 +1,488 @@ +export module xlings.core.subos.manifest; + +import std; + +import xlings.libs.json; +import xlings.platform; + +// `subos_info` — what a subos is, recorded in the subos's own `.xlings.json`. +// +// A subos already had a directory, a bin/, an xvm scope and a registry entry. +// What it did not have was a statement of *what it is*: which runtime its +// binaries were built against, and which environment its processes need. Both +// were implicit in whatever happened to be installed, which is why a subos +// could not be described, checked, or reproduced. +// +// Layer this closes: a program needs bootstrap (PT_INTERP + CRT + libc), +// discovery (PATH + RPATH), and configuration (env vars). xlings had the first +// two — glibc + elfpatch, xvm + shims — and nothing for the third. That is the +// gap behind mcpp-community/mcpp#352: a GLFW binary that links fine and exits +// 255 because no one told it where the GL drivers are. +// +// Placement is the per-subos `.xlings.json`, not a new file and not the home +// or project one. Those two already use the key `subos` for other things (a +// selector string at project level, a registry map at home level), and a subos +// describing itself belongs in the subos. +// +// This module is deliberately free of Config/xvm imports: it takes paths and a +// binding→dir resolver from the caller. That keeps invariant checking and +// placeholder expansion testable without a home on disk, and it is what lets +// the doctor reporter and the repairer share one predicate instead of +// describing the rules twice and drifting apart. + +export namespace xlings::subos::manifest { + +namespace fs = std::filesystem; + +inline constexpr int SCHEMA_VERSION = 1; +inline constexpr std::string_view BLOCK = "subos_info"; +// The runtime a subos gets when the caller names none. A constant rather than +// a lookup: slice 1 ships one runtime, and inventing a "pick the newest libc +// present" rule would make two homes with the same command produce different +// subos. +inline constexpr std::string_view DEFAULT_RUNTIME = "glibc@2.39"; + +inline constexpr std::string_view OP_SET = "set"; +inline constexpr std::string_view OP_PREPEND = "prepend"; + +// ── data ──────────────────────────────────────────────────────────────── + +// One variable a package asks its subos to export. +struct EnvDecl { + std::string var; + std::string op; // OP_SET | OP_PREPEND + std::string value; // may contain ${...} placeholders +}; + +// A provider's whole section. Keyed by binding so uninstalling the package +// removes exactly what it added — the same provider-scoped ownership xvm.add +// and xvm.files already use. +struct Provider { + std::string binding; // "@" + std::vector decls; +}; + +struct Info { + int schema_version = 0; + std::string runtime; + std::vector envs; // sorted by binding; see resolve() + std::string created_at; + std::string created_by; +}; + +// ── runtime family ────────────────────────────────────────────────────── + +// The runtime string is self-describing: "glibc@2.39" says Linux/glibc without +// a second field to disagree with it. Families are derived here rather than +// stored, so a manifest cannot claim a family its runtime contradicts. +// +// Slice 1 needs only the glibc row. The rest are listed to make the shape of +// the mapping explicit — a new OS adds a row, not a schema field. +std::string family_of(std::string_view runtime, std::string_view arch = "x86_64") { + const auto at = runtime.find('@'); + const auto name = runtime.substr(0, at == std::string_view::npos + ? runtime.size() : at); + if (name == "glibc") return std::format("linux-{}-glibc", arch); + if (name == "musl") return std::format("linux-{}-musl", arch); + if (name == "wasi-libc") return "wasm32-wasi"; + if (name == "macos_sdk") return std::format("darwin-{}", arch); + if (name == "ucrt") return std::format("windows-{}-ucrt", arch); + return "unknown"; +} + +// "@", both halves non-empty. Used for `runtime` and for every +// envs key. +bool is_binding(std::string_view s) { + const auto at = s.find('@'); + return at != std::string_view::npos && at > 0 && at + 1 < s.size(); +} + +std::string_view binding_name(std::string_view binding) { + const auto at = binding.find('@'); + return at == std::string_view::npos ? binding : binding.substr(0, at); +} + +// ── invariants ────────────────────────────────────────────────────────── + +enum class Defect { + DirMissing, // I1 + ConfigMissing, // I2 + ConfigUnreadable, // I2 + BlockMissing, // I4 + SchemaUnsupported, // I5 + RuntimeMalformed, // I6 + EnvsMalformed, // I7 + EnvDeclMalformed, // I7 + ProvenanceMissing, // I8 +}; + +struct Finding { + Defect kind; + std::string detail; +}; + +std::string_view describe(Defect d) { + switch (d) { + case Defect::DirMissing: return "subos directory is missing"; + case Defect::ConfigMissing: return "subos has no .xlings.json"; + case Defect::ConfigUnreadable: return ".xlings.json is not readable JSON"; + case Defect::BlockMissing: return "no subos_info block"; + case Defect::SchemaUnsupported: return "unsupported subos_info schema_version"; + case Defect::RuntimeMalformed: return "runtime is not @"; + case Defect::EnvsMalformed: return "envs is not an object of provider sections"; + case Defect::EnvDeclMalformed: return "env declaration is malformed"; + case Defect::ProvenanceMissing: return "created_at / created_by missing"; + } + return "unknown defect"; +} + +fs::path config_path(const fs::path& subosDir) { + return subosDir / ".xlings.json"; +} + +// Read the whole document. An unreadable or malformed file yields nullopt, and +// the caller must not paper over it with an empty object: for the home config +// "corrupt means absent" is the historical behavior, but here it would let +// `--fix` rewrite a file it never managed to read. +std::optional read_document(const fs::path& subosDir) { + const auto path = config_path(subosDir); + std::error_code ec; + if (!fs::exists(path, ec) || ec) return std::nullopt; + try { + auto content = platform::read_file_to_string(path.string()); + auto parsed = nlohmann::json::parse(content, nullptr, false); + if (parsed.is_discarded() || !parsed.is_object()) return std::nullopt; + return parsed; + } catch (...) { return std::nullopt; } +} + +// I4–I8, over a document already in hand. Split out from validate() so a +// caller that has just built a document in memory can check it before writing. +std::vector validate_block(const nlohmann::json& doc) { + std::vector out; + + if (!doc.contains(std::string(BLOCK)) || !doc[std::string(BLOCK)].is_object()) { + out.push_back({Defect::BlockMissing, std::string(BLOCK)}); + return out; // nothing below can be checked + } + const auto& b = doc[std::string(BLOCK)]; + + if (!b.contains("schema_version") || !b["schema_version"].is_number_integer() + || b["schema_version"].get() != SCHEMA_VERSION) { + out.push_back({Defect::SchemaUnsupported, + b.contains("schema_version") ? b["schema_version"].dump() + : "absent"}); + } + + const auto runtime = b.value("runtime", std::string{}); + if (!is_binding(runtime)) { + out.push_back({Defect::RuntimeMalformed, + runtime.empty() ? "absent" : runtime}); + } + + // "An empty collection is not a missing one": `envs` is written as {} at + // creation and stays {} when the last provider is removed, so no reader + // has to treat absent and empty as the same thing. + if (!b.contains("envs") || !b["envs"].is_object()) { + out.push_back({Defect::EnvsMalformed, + b.contains("envs") ? b["envs"].type_name() : "absent"}); + } else { + for (auto it = b["envs"].begin(); it != b["envs"].end(); ++it) { + if (!is_binding(it.key())) { + out.push_back({Defect::EnvsMalformed, + std::format("provider key '{}' is not @", + it.key())}); + continue; + } + if (!it.value().is_array()) { + out.push_back({Defect::EnvsMalformed, + std::format("provider '{}' is not an array", it.key())}); + continue; + } + for (const auto& d : it.value()) { + const auto var = d.is_object() ? d.value("var", std::string{}) + : std::string{}; + const auto op = d.is_object() ? d.value("op", std::string{}) + : std::string{}; + if (var.empty() || (op != OP_SET && op != OP_PREPEND)) { + out.push_back({Defect::EnvDeclMalformed, + std::format("{}: {}", it.key(), d.dump())}); + } + } + } + } + + if (b.value("created_at", std::string{}).empty() + || b.value("created_by", std::string{}).empty()) { + out.push_back({Defect::ProvenanceMissing, ""}); + } + return out; +} + +// I1–I8 for a subos on disk. The single predicate the doctor reporter and the +// repairer both call — two descriptions of the same rule is how a reporter +// ends up flagging what `--fix` will not touch. +std::vector validate(const fs::path& subosDir) { + std::error_code ec; + if (!fs::is_directory(subosDir, ec) || ec) + return {{Defect::DirMissing, subosDir.string()}}; + if (!fs::exists(config_path(subosDir), ec) || ec) + return {{Defect::ConfigMissing, config_path(subosDir).string()}}; + + auto doc = read_document(subosDir); + if (!doc) return {{Defect::ConfigUnreadable, config_path(subosDir).string()}}; + return validate_block(*doc); +} + +// ── parse / build ─────────────────────────────────────────────────────── + +// Providers come back sorted by binding. That ordering is the resolution order +// (see resolve) and it is derived from the data rather than from install +// history, so two homes holding the same manifest resolve it identically. +Info parse(const nlohmann::json& doc) { + Info info; + if (!doc.contains(std::string(BLOCK)) || !doc[std::string(BLOCK)].is_object()) + return info; + const auto& b = doc[std::string(BLOCK)]; + + info.schema_version = b.value("schema_version", 0); + info.runtime = b.value("runtime", std::string{}); + info.created_at = b.value("created_at", std::string{}); + info.created_by = b.value("created_by", std::string{}); + + if (b.contains("envs") && b["envs"].is_object()) { + for (auto it = b["envs"].begin(); it != b["envs"].end(); ++it) { + if (!it.value().is_array()) continue; + Provider p{.binding = it.key()}; + for (const auto& d : it.value()) { + if (!d.is_object()) continue; + EnvDecl e{ + .var = d.value("var", std::string{}), + .op = d.value("op", std::string{}), + .value = d.value("value", std::string{}), + }; + if (e.var.empty()) continue; + if (e.op != OP_SET && e.op != OP_PREPEND) continue; + p.decls.push_back(std::move(e)); + } + info.envs.push_back(std::move(p)); + } + } + std::ranges::sort(info.envs, {}, &Provider::binding); + return info; +} + +std::string utc_now_iso() { + const auto now = std::chrono::system_clock::to_time_t( + std::chrono::system_clock::now()); + char buf[32]; + std::strftime(buf, sizeof(buf), "%Y-%m-%dT%H:%M:%SZ", std::gmtime(&now)); + return buf; +} + +// The block a freshly created subos gets. `envs` is an explicit empty object. +nlohmann::json make_block(std::string_view runtime, std::string_view createdBy) { + nlohmann::json b; + b["schema_version"] = SCHEMA_VERSION; + b["runtime"] = std::string(runtime); + b["envs"] = nlohmann::json::object(); + b["created_at"] = utc_now_iso(); + b["created_by"] = std::string(createdBy); + return b; +} + +// ── env declarations ──────────────────────────────────────────────────── + +// Record one declaration under its provider. Idempotent on the whole +// (var, op, value) triple: config() runs again on every dependent install, and +// a re-run must not grow the section. +// +// Returns whether the document changed, so a caller can skip a write. +bool add_env(nlohmann::json& doc, std::string_view binding, const EnvDecl& decl) { + auto& b = doc[std::string(BLOCK)]; + if (!b.is_object()) b = nlohmann::json::object(); + if (!b.contains("envs") || !b["envs"].is_object()) + b["envs"] = nlohmann::json::object(); + + auto& section = b["envs"][std::string(binding)]; + if (!section.is_array()) section = nlohmann::json::array(); + + for (const auto& existing : section) { + if (existing.value("var", std::string{}) == decl.var + && existing.value("op", std::string{}) == decl.op + && existing.value("value", std::string{}) == decl.value) { + return false; + } + } + section.push_back({{"var", decl.var}, {"op", decl.op}, {"value", decl.value}}); + return true; +} + +// Drop a provider's whole section — the uninstall counterpart of add_env. The +// package never writes cleanup code for this; ownership is by binding, so +// removing the key removes exactly what that package added and nothing else. +// +// The `envs` object itself stays, empty, per the invariant above. +bool remove_provider(nlohmann::json& doc, std::string_view binding) { + if (!doc.contains(std::string(BLOCK))) return false; + auto& b = doc[std::string(BLOCK)]; + if (!b.is_object() || !b.contains("envs") || !b["envs"].is_object()) + return false; + return b["envs"].erase(std::string(binding)) > 0; +} + +// Every provider whose package name matches, regardless of version. Uninstall +// knows the name and version it removed, but a home that installed the same +// package twice under different versions can hold a stale section from the +// earlier one; matching on name is what lets doctor see it. +std::vector providers_named(const nlohmann::json& doc, + std::string_view name) { + std::vector out; + if (!doc.contains(std::string(BLOCK))) return out; + const auto& b = doc[std::string(BLOCK)]; + if (!b.is_object() || !b.contains("envs") || !b["envs"].is_object()) return out; + for (auto it = b["envs"].begin(); it != b["envs"].end(); ++it) + if (binding_name(it.key()) == name) out.push_back(it.key()); + return out; +} + +// ── placeholders ──────────────────────────────────────────────────────── + +// What a value may refer to. `${pkgdir}` differs per provider, so it arrives +// as a resolver rather than a path — and the resolver is the caller's, which +// is what keeps this module independent of the version database. +struct Placeholders { + fs::path subosdir; + fs::path home; + fs::path xlings_home; + std::function pkgdir_of; +}; + +// Expand ${...} in a declared value. +// +// Placeholders are why a manifest is portable at all: a value carrying +// /home/alice/... describes one machine, and a subos description that only +// works on the machine that wrote it is not a description. +// +// An unknown or unresolvable placeholder is left verbatim rather than replaced +// with an empty string. Empty would turn "${pkgdir}/lib/dri" into "/lib/dri" — +// a real path, on the host, outside the subos. Leaving the text intact makes +// the failure visible to doctor D3 instead of pointing a driver search at /. +std::string expand(std::string_view value, std::string_view binding, + const Placeholders& ph) { + std::string out; + out.reserve(value.size()); + + for (std::size_t i = 0; i < value.size();) { + if (value[i] != '$' || i + 1 >= value.size() || value[i + 1] != '{') { + out += value[i++]; + continue; + } + const auto close = value.find('}', i + 2); + if (close == std::string_view::npos) { // unterminated: verbatim + out += value.substr(i); + break; + } + const auto name = value.substr(i + 2, close - i - 2); + fs::path resolved; + bool known = true; + if (name == "subosdir") resolved = ph.subosdir; + else if (name == "home") resolved = ph.home; + else if (name == "xlings_home") resolved = ph.xlings_home; + else if (name == "pkgdir") resolved = ph.pkgdir_of ? ph.pkgdir_of(binding) + : fs::path{}; + else known = false; + + if (!known || resolved.empty()) out += value.substr(i, close - i + 1); + else out += resolved.string(); + i = close + 1; + } + return out; +} + +// True once expansion left any `${...}` behind — i.e. something could not be +// resolved. Doctor D3 reports on this rather than on the directory existing, +// because an unexpanded value is a defect regardless of what is on disk. +bool has_unresolved(std::string_view expanded) { + return expanded.find("${") != std::string_view::npos; +} + +// ── resolution ────────────────────────────────────────────────────────── + +// One variable as it will actually be exported. +struct Resolved { + std::string var; + std::string op; // the winning op + std::string value; // expanded, prepends already joined + std::vector providers; // every binding that declared this var + bool conflicted = false; + bool unresolved = false; +}; + +// Fold the manifest into the variables to export. +// +// Conflict rules, when more than one provider names the same variable: +// * several `set` — the last provider wins, and it is a conflict +// * several `prepend` — all contribute, later providers land nearer the front +// * `set` and `prepend` mixed — `set` wins, the prepends are dropped, conflict +// +// "Later" means later in binding order, not install order. Install order is +// not recorded in the manifest and recording it would add a field whose only +// job is to make the outcome depend on history — two homes with byte-identical +// manifests would then export different values. Sorting by binding keeps the +// manifest the whole answer. Every conflict is reported (doctor D4) rather than +// resolved quietly. +std::vector resolve(const Info& info, const Placeholders& ph) { + struct Acc { + std::string setValue; + bool hasSet = false; + std::vector prepends; // front-most last + std::vector providers; + bool unresolved = false; + std::size_t order = 0; // first appearance, for stable output + }; + std::map acc; + std::size_t seen = 0; + + for (const auto& p : info.envs) { // already sorted by binding + for (const auto& d : p.decls) { + auto expanded = expand(d.value, p.binding, ph); + auto& a = acc[d.var]; + if (a.providers.empty()) a.order = seen++; + a.providers.push_back(p.binding); + if (has_unresolved(expanded)) a.unresolved = true; + if (d.op == OP_SET) { a.setValue = std::move(expanded); a.hasSet = true; } + else { a.prepends.push_back(std::move(expanded)); } + } + } + + std::vector out; + out.reserve(acc.size()); + for (auto& [var, a] : acc) { + Resolved r{.var = var, .providers = a.providers, .unresolved = a.unresolved}; + // More than one declaration for a variable is a conflict unless they + // are all prepends, which compose by construction. + r.conflicted = a.providers.size() > 1 + && !(!a.hasSet && !a.prepends.empty()); + if (a.hasSet) { + r.op = std::string(OP_SET); + r.value = std::move(a.setValue); + if (!a.prepends.empty()) r.conflicted = true; + } else { + r.op = std::string(OP_PREPEND); + // Later provider nearer the front, matching "the newest thing + // installed is found first". + for (auto it = a.prepends.rbegin(); it != a.prepends.rend(); ++it) { + if (!r.value.empty()) r.value += ':'; + r.value += *it; + } + } + out.push_back(std::move(r)); + } + // Stable, and stable for a reason: this list is echoed to the user and + // diffed in tests, so it must not reorder because a map rehashed. + std::ranges::sort(out, {}, &Resolved::var); + return out; +} + +} // namespace xlings::subos::manifest diff --git a/src/core/xim/installer.cppm b/src/core/xim/installer.cppm index fabcab3b..6a14513b 100644 --- a/src/core/xim/installer.cppm +++ b/src/core/xim/installer.cppm @@ -25,6 +25,7 @@ import xlings.core.xvm.bindings; import xlings.core.xvm.removal; import xlings.core.xvm.registration; import xlings.core.xvm.errors; +import xlings.core.subos.manifest; import xlings.core.xvm.commands; import xlings.core.xvm.shim; import xlings.core.xim.libxpkg.types.script; @@ -1436,11 +1437,105 @@ void detach_current_subos_(const std::string& target, } } +// Record a package's `subos.env` declarations into the subos it installs into. +// +// Provider-scoped, exactly like xvm registrations: everything lands under the +// declaring package's binding, and uninstall drops that key. A recipe never +// writes cleanup for it. +// +// Cross-package declarations are refused. A package may describe its own +// runtime needs; letting it write a section owned by another name would make +// uninstall unable to clean up (the owner's key is what removal keys on) and +// would hand any recipe the ability to edit any other's environment. +bool apply_subos_env_ops_(const std::vector& operations, + const PlanNode& node) { + namespace mf = xlings::subos::manifest; + + std::vector declarations; + for (const auto& op : operations) + if (op.op == "subos_env") declarations.push_back(&op); + if (declarations.empty()) return true; + + const auto canonical = node.canonicalName.empty() + ? canonical_package_name(node.namespaceName, node.name) + : node.canonicalName; + + const auto subosDir = Config::xvm_artifact_subos_dir(); + auto docPath = mf::config_path(subosDir); + auto doc = mf::read_document(subosDir); + if (!doc) { + // Absent is recoverable (an old subos predates the block); unreadable + // is not, and must not be papered over by starting from {} -- that + // would discard a workspace we never managed to parse. + std::error_code ec; + if (std::filesystem::exists(docPath, ec)) { + log::error("[xim] {} is not readable JSON; refusing to record " + "subos env declarations for {}@{}", + docPath.string(), canonical, node.version); + return false; + } + nlohmann::json fresh; + fresh["workspace"] = nlohmann::json::object(); + fresh[std::string(mf::BLOCK)] = mf::make_block( + mf::DEFAULT_RUNTIME, std::format("xlings {}", Info::VERSION)); + doc = std::move(fresh); + } + if (!doc->contains(std::string(mf::BLOCK)) + || !doc->at(std::string(mf::BLOCK)).is_object()) { + (*doc)[std::string(mf::BLOCK)] = mf::make_block( + mf::DEFAULT_RUNTIME, std::format("xlings {}", Info::VERSION)); + } + + bool changed = false; + for (const auto* op : declarations) { + if (!mf::is_binding(op->binding)) { + log::error("[xim] {}@{} declared env '{}' with binding '{}' " + "(expected @); nothing recorded", + canonical, node.version, op->var, op->binding); + return false; + } + const auto owner = mf::binding_name(op->binding); + if (owner != node.name && owner != canonical) { + log::error("[xim] {}@{} declared env '{}' for '{}', which it does " + "not own; nothing recorded", + canonical, node.version, op->var, op->binding); + return false; + } + changed |= mf::add_env(*doc, op->binding, + {.var = op->var, .op = op->mode, .value = op->value}); + } + if (!changed) return true; + + if (auto findings = mf::validate_block(*doc); !findings.empty()) { + log::error("[xim] recording env declarations for {}@{} would leave an " + "invalid subos manifest: {}", canonical, node.version, + mf::describe(findings.front().kind)); + return false; + } + + try { + platform::write_string_to_file(docPath.string(), doc->dump(2)); + } catch (const std::exception& e) { + log::error("[xim] failed to write {}: {}", docPath.string(), e.what()); + return false; + } + log::debug("[xim] recorded {} subos env declaration(s) for {}@{}", + declarations.size(), canonical, node.version); + return true; +} + bool process_xvm_operations_(const PlanNode& node, const std::filesystem::path& dataDir, mcpplibs::xpkg::PackageExecutor& executor, bool useAfterInstall) { auto xvm_ops = executor.xvm_operations(); + + // Before the early return below. A package that declares only env and + // registers nothing with xvm has an empty registration batch, and would + // otherwise install cleanly with its declarations dropped on the floor -- + // the exact shape where "nothing happened" and "it worked" look alike. + if (!apply_subos_env_ops_(xvm_ops, node)) return false; + auto& paths = Config::paths(); auto& scopedDb = Config::versions_mut(); auto& scopedWorkspace = Config::workspace_mut(); @@ -2791,6 +2886,50 @@ public: removalResult.error().version)); } + // Drop this package's subos env section. Keyed by binding, so it takes + // exactly what the package added and leaves every other provider's + // alone -- the recipe's uninstall() writes nothing for this. + // + // Matched by package *name* rather than the exact binding: the removal + // above resolves versions through namespaces and group members, so the + // binding recorded at install time is not reliably reconstructible + // here. A home that installed two versions of the same package can + // hold a section for each, and leaving the other one behind would + // point a driver search at a payload that is being deleted. + { + namespace mf = xlings::subos::manifest; + if (auto doc = mf::read_document(artifactSubosDir)) { + bool changed = false; + for (const auto& binding : + mf::providers_named(*doc, executingProvider)) { + changed |= mf::remove_provider(*doc, binding); + } + // providers_named matches the bare name; a namespaced install + // records the canonical one, so ask again under it. + if (executingProvider != detachTarget) { + for (const auto& binding : + mf::providers_named(*doc, detachTarget)) { + changed |= mf::remove_provider(*doc, binding); + } + } + if (changed) { + try { + platform::write_string_to_file( + mf::config_path(artifactSubosDir).string(), + doc->dump(2)); + } catch (const std::exception& e) { + // Not fatal to the uninstall -- the payload is already + // going -- but it must be said. A stale section points + // at a directory that no longer exists, which doctor + // D2/D3 will report. + log::warn("[xim] could not clear subos env " + "declarations for {}: {}", + executingProvider, e.what()); + } + } + } + } + for (const auto& op : xvm_ops) { if (op.op == "remove_headers") { xvm::remove_headers(op.includedir, sysroot_include); diff --git a/src/core/xself/doctor.cppm b/src/core/xself/doctor.cppm index 1e64be0c..e31b68d2 100644 --- a/src/core/xself/doctor.cppm +++ b/src/core/xself/doctor.cppm @@ -23,6 +23,8 @@ import xlings.core.xvm.owner; import xlings.core.xself.repair; import xlings.core.xim.payload; // classify_payload_platform import xlings.core.profile; +import xlings.core.subos.manifest; +import xlings.platform.target; // platform::host().arch for the runtime family namespace xlings::xself { @@ -128,6 +130,27 @@ enum class FindingKind { SysrootDangling, BindingState, OtherSubos, + // The subos does not describe itself: no `subos_info` block, or one that + // cannot be read. Every configuration-layer feature is inert without it, + // and inert looks exactly like "no package needed anything". + SubosManifest, + // An `envs` section owned by a package that is not installed here. Left + // behind by an uninstall that could not write the manifest, or copied in + // by a fork from a home where that package existed. The values point into + // a payload directory that is not there. + SubosEnvOrphan, + // A declared value that still contains `${...}` after expansion, so the + // variable would be exported pointing at a literal placeholder. Reported + // rather than exported: see manifest::expand on why it is not blanked. + SubosEnvUnresolved, + // One variable claimed by several packages. Resolution is deterministic, + // so this is not a breakage -- but one of the two packages is not getting + // what it asked for, and only a human can say which should. + SubosEnvConflict, + // The subos names a runtime that is not installed in it. The payloads + // built against it may still run off the host's libc, which is precisely + // the hermetic boundary the runtime field exists to make checkable. + SubosRuntimeMissing, }; enum class FindingLevel { @@ -385,11 +408,187 @@ std::string activation_conflict_(const DoctorState& st, return reason; } +// D1–D5 over the active subos's `subos_info`. +// +// One function, called by detection and again by the repair pass. The repair +// acts on what this returns rather than on its own reading of the rules -- +// a reporter and a repairer that each describe the criteria end up describing +// them differently, and then fight. +// +// Nothing here needs the version DB except D2 and D5, which take it as an +// argument, so this stays checkable against a directory. +std::vector detect_subos_manifest_(const xvm::VersionDB& db, + const fs::path& subosDir, + const std::string& subosName) { + namespace mf = xlings::subos::manifest; + std::vector out; + + // D1 — structure. Anything wrong here makes the rest unreadable, so it is + // the only finding produced when it fires. + if (auto structural = mf::validate(subosDir); !structural.empty()) { + std::string detail; + for (const auto& f : structural) { + if (!detail.empty()) detail += "; "; + detail += std::string(mf::describe(f.kind)); + if (!f.detail.empty()) detail += " (" + f.detail + ")"; + } + const bool unreadable = std::ranges::any_of( + structural, [](const auto& f) { + return f.kind == mf::Defect::ConfigUnreadable; + }); + out.push_back({ + .kind = FindingKind::SubosManifest, + .level = FindingLevel::Error, + .target = subosName, + .detail = detail, + // An unreadable file is not repaired: rewriting it would discard a + // workspace no one has managed to parse. Everything else is a + // missing or unusable block, which `--fix` can add. + .remedy = unreadable + ? std::format("inspect {}", + Config::display_path(mf::config_path(subosDir))) + : "xlings self doctor --fix", + }); + return out; + } + + auto doc = mf::read_document(subosDir); + if (!doc) return out; // D1 already covered this + const auto info = mf::parse(*doc); + + const auto installed = [&](std::string_view binding) { + const auto at = binding.find('@'); + if (at == std::string_view::npos) return false; + const std::string name(binding.substr(0, at)); + const std::string version(binding.substr(at + 1)); + const auto* vi = xvm::get_vinfo(db, name); + if (!vi) return false; + // Namespaced installs record `:`, so a bare match is not + // enough -- compare the version tail. + return std::ranges::any_of(vi->versions, [&](const auto& entry) { + const auto& key = entry.first; + if (key == version) return true; + const auto colon = key.find(':'); + return colon != std::string::npos && key.substr(colon + 1) == version; + }); + }; + + // D2 — an envs section whose owner is not installed here. + for (const auto& provider : info.envs) { + if (installed(provider.binding)) continue; + out.push_back({ + .kind = FindingKind::SubosEnvOrphan, + .level = FindingLevel::Error, + .target = subosName, + .version = provider.binding, + .detail = std::format( + "subos '{}' exports {} variable(s) for '{}', which is not " + "installed here", subosName, provider.decls.size(), + provider.binding), + .remedy = "xlings self doctor --fix", + }); + } + + // D3/D4 — over the resolved set, so both see exactly what activation will. + const auto resolved = mf::resolve(info, mf::Placeholders{ + .subosdir = subosDir, + .home = platform::get_home_dir(), + .xlings_home = Config::paths().homeDir, + .pkgdir_of = [](std::string_view binding) -> fs::path { + const auto at = binding.find('@'); + if (at == std::string_view::npos) return {}; + const std::string name(binding.substr(0, at)); + const std::string version(binding.substr(at + 1)); + const auto store = Config::paths().dataDir / "xpkgs"; + std::error_code ec; + if (auto direct = store / name / version; + fs::is_directory(direct, ec)) { + return direct; + } + if (!fs::is_directory(store, ec)) return {}; + const auto suffix = "-x-" + name; + for (const auto& entry : platform::dir_entries(store)) { + if (!entry.is_directory(ec)) continue; + if (!entry.path().filename().string().ends_with(suffix)) continue; + if (auto candidate = entry.path() / version; + fs::is_directory(candidate, ec)) { + return candidate; + } + } + return {}; + }, + }); + + for (const auto& v : resolved) { + if (v.unresolved) { + out.push_back({ + .kind = FindingKind::SubosEnvUnresolved, + .level = FindingLevel::Error, + .target = subosName, + .version = v.var, + .detail = std::format( + "{} would export an unexpanded path; its provider's " + "payload is missing", v.var), + // Reinstalling the provider is what puts the payload back. + // Only one is named even when several declared the variable: + // a remedy the user can paste beats an exhaustive one. + .remedy = v.providers.empty() ? std::string{} + : std::format("xlings install {}", v.providers.front()), + }); + } + if (v.conflicted) { + std::string names; + for (const auto& b : v.providers) { + if (!names.empty()) names += ", "; + names += b; + } + out.push_back({ + .kind = FindingKind::SubosEnvConflict, + // A warning, not an error: resolution is deterministic and the + // subos works. What is wrong is that one of these packages is + // silently not getting what it asked for. + .level = FindingLevel::Warning, + .target = subosName, + .version = v.var, + .detail = std::format("{} is claimed by {}", v.var, names), + }); + } + } + + // D5 — the declared runtime is not installed here. + if (mf::is_binding(info.runtime) && !installed(info.runtime)) { + out.push_back({ + .kind = FindingKind::SubosRuntimeMissing, + // Warning: binaries generally still run, off the host's libc. + // That is the hermetic boundary being crossed silently, which is + // the thing worth saying rather than failing over. + .level = FindingLevel::Warning, + .target = subosName, + .version = info.runtime, + .detail = std::format( + "subos '{}' declares runtime {} ({}), which is not installed " + "here", subosName, info.runtime, + mf::family_of(info.runtime, platform::host().arch)), + .remedy = std::format("xlings install {}", info.runtime), + }); + } + return out; +} + Scan detect_(const DoctorState& st, const CoordinateProbe& probe) { auto& p = Config::paths(); Scan scan; const auto add = [&](Finding f) { scan.findings.push_back(std::move(f)); }; + // The subos this run is actually in. Other subos are not inspected from + // here for the same reason their payloads are not repaired: a second + // shell may be inside one right now. + for (auto&& f : detect_subos_manifest_(st.db, p.subosDir, + p.activeSubos.empty() ? "default" + : p.activeSubos)) { + add(std::move(f)); + } + // The PATH an aliased command would inherit. Read once, and read from // THIS process: doctor is normally started from the user's shell, so this // is the same PATH the shim would get. It is not guaranteed to be -- a @@ -1267,6 +1466,66 @@ void repair_local_(const DoctorState& st, const Scan& scan, out.notes.emplace_back(std::move(label), std::move(text)); }; + // Subos manifest repairs. Driven by the findings detection produced, not + // by a second reading of the rules -- so what `--fix` touches is exactly + // what was reported. + { + namespace mf = xlings::subos::manifest; + const bool wantsBlock = std::ranges::any_of( + scan.findings, [](const Finding& f) { + return f.kind == FindingKind::SubosManifest + && f.remedy == "xlings self doctor --fix"; + }); + std::vector orphans; + for (const auto& f : scan.findings) + if (f.kind == FindingKind::SubosEnvOrphan) orphans.push_back(f.version); + + if (wantsBlock || !orphans.empty()) { + auto doc = mf::read_document(p.subosDir); + nlohmann::json document = doc ? *doc : nlohmann::json::object(); + if (!doc && fs::exists(mf::config_path(p.subosDir))) { + // Unreadable rather than absent. Detection already said so and + // offered no `--fix` remedy; do not overwrite it here either. + note(glyph::mark(glyph::failed, "subos manifest"), + std::format("{} is not readable JSON; left untouched", + Config::display_path( + mf::config_path(p.subosDir)))); + } else { + bool changed = false; + if (!document.contains("workspace")) + document["workspace"] = nlohmann::json::object(); + if (wantsBlock) { + document[std::string(mf::BLOCK)] = mf::make_block( + mf::DEFAULT_RUNTIME, + std::format("xlings {}", Info::VERSION)); + changed = true; + note(glyph::mark(glyph::bullet, "subos manifest"), + std::format("described subos '{}' (runtime {})", + p.activeSubos, mf::DEFAULT_RUNTIME)); + } + for (const auto& binding : orphans) { + if (!mf::remove_provider(document, binding)) continue; + changed = true; + note(glyph::mark(glyph::bullet, "subos env dropped"), + std::format("{} is not installed here", binding)); + } + if (changed) { + try { + platform::write_string_to_file( + mf::config_path(p.subosDir).string(), + document.dump(2)); + } catch (const std::exception& e) { + note(glyph::mark(glyph::failed, "subos manifest"), + std::format("could not write {}: {}", + Config::display_path( + mf::config_path(p.subosDir)), + e.what())); + } + } + } + } + } + for (const auto& f : scan.findings) { if (f.kind == FindingKind::MissingShim) { if (!fs::exists(st.xlingsBin)) continue; @@ -2026,6 +2285,32 @@ void render_(const Scan& scan, const RepairReport& repair, bool fix, case FindingKind::ReleaseAnchor: if (verbose) add(glyph::mark(glyph::note, "release anchor"), f.detail); break; + case FindingKind::SubosManifest: + add(glyph::mark(glyph::failed, "subos manifest"), f.detail); + // The remedy for an unreadable file is not `--fix`, so it has + // to be printed rather than left to the generic footer. + if (!f.remedy.empty() && f.remedy != "xlings self doctor --fix") + add(" " + glyph::mark(glyph::remedy, "run"), f.remedy); + break; + case FindingKind::SubosEnvOrphan: + add(glyph::mark(glyph::failed, "subos env orphan"), f.detail); + break; + case FindingKind::SubosEnvUnresolved: + add(glyph::mark(glyph::failed, "subos env unresolved"), f.detail); + if (!f.remedy.empty()) + add(" " + glyph::mark(glyph::remedy, "run"), f.remedy); + break; + case FindingKind::SubosEnvConflict: + // Never collapsed into a count, unlike the notice categories: + // there is one per contested variable, the number is small, + // and which packages disagree is the whole content. + add(glyph::mark(glyph::warn, "subos env conflict"), f.detail); + break; + case FindingKind::SubosRuntimeMissing: + add(glyph::mark(glyph::warn, "subos runtime"), f.detail); + if (!f.remedy.empty()) + add(" " + glyph::mark(glyph::remedy, "run"), f.remedy); + break; default: break; } } diff --git a/src/core/xself/init.cppm b/src/core/xself/init.cppm index 51971696..60daf0b0 100644 --- a/src/core/xself/init.cppm +++ b/src/core/xself/init.cppm @@ -11,6 +11,7 @@ import xlings.platform; import xlings.core.xself.compat; // Generated at build time from config/shell/*.{sh,fish,ps1}; see mcpp.toml. import xlings.core.xself.profile_resources; +import xlings.core.subos.manifest; namespace xlings::xself { @@ -166,6 +167,38 @@ static void write_if_missing_(const fs::path& path, std::string_view content) { platform::write_string_to_file(path.string(), std::string(content)); } +// Give a subos directory a valid `subos_info`, preserving everything else in +// the file. Idempotent: a block that already validates is left alone, so the +// envs packages declared into it survive. +static void ensure_subos_manifest_(const fs::path& subos_dir) { + namespace mf = xlings::subos::manifest; + auto path = subos_dir / ".xlings.json"; + + nlohmann::json json = nlohmann::json::object(); + if (fs::exists(path)) { + try { + auto parsed = nlohmann::json::parse( + platform::read_file_to_string(path.string()), nullptr, false); + if (parsed.is_discarded() || !parsed.is_object()) { + // Unreadable, not absent. Overwriting would throw away a + // workspace we cannot see; leave it for doctor to report. + log::warn("[xlings:self]: {} is not readable JSON; leaving it " + "alone (run `xlings self doctor`)", + Config::display_path(path)); + return; + } + json = std::move(parsed); + } catch (...) { return; } + } + if (!json.contains("workspace")) json["workspace"] = nlohmann::json::object(); + if (mf::validate_block(json).empty()) return; + + json[std::string(mf::BLOCK)] = + mf::make_block(mf::DEFAULT_RUNTIME, std::format("xlings {}", Info::VERSION)); + ensure_parent_dirs_(path); + platform::write_string_to_file(path.string(), json.dump(2)); +} + // Extract the value following `# xlings-profile-version: ` on any line of // `text`. Returns an empty string when the marker is absent, which we // interpret as "legacy v1" — anything older than the time we started @@ -286,7 +319,16 @@ bool ensure_home_layout(const fs::path& home_dir) { auto current_link = home_dir / "subos" / "current"; platform::create_directory_link(current_link, default_subos); - write_if_missing_(default_subos / ".xlings.json", "{\"workspace\":{}}"); + // Not write_if_missing_: every home that predates `subos_info` already has + // this file, so "missing" is exactly the case that never fires on the + // subos that matters most. `default` is where an ordinary user installs + // everything, and a default without the block would make the whole + // configuration layer inert on upgrade while looking installed. + // + // This is also the migration: `self init` runs on install and update, so an + // old home repairs its default subos on the next either. Other subos in + // that home are `subos doctor --fix`'s job -- init does not enumerate them. + ensure_subos_manifest_(default_subos); write_if_missing_(home_dir / "data" / "xim-index-repos" / "xim-indexrepos.json", "{}"); // Profile content lives in xlings.core.xself.profile_resources. We use // the version-aware writer so users who installed an older xlings get diff --git a/tests/e2e/run_all.sh b/tests/e2e/run_all.sh index 16774774..27d16517 100755 --- a/tests/e2e/run_all.sh +++ b/tests/e2e/run_all.sh @@ -109,6 +109,8 @@ TESTS=( "E2E-57 |subos_use_process_model_test.sh||" "E2E-58 |arch_evidence_contract_test.sh||" "E2E-59 |index_version_contract_test.sh||" + "E2E-60 |subos_env_declaration_test.sh||" + "E2E-61 |subos_env_probe_compat_test.sh||" ) PASS=0; FAIL=0; SOFTFAIL=0 diff --git a/tests/e2e/subos_env_declaration_test.sh b/tests/e2e/subos_env_declaration_test.sh new file mode 100755 index 00000000..f93ac6f6 --- /dev/null +++ b/tests/e2e/subos_env_declaration_test.sh @@ -0,0 +1,254 @@ +#!/usr/bin/env bash +# E2E: subos.env — a package declares an environment variable, and a program +# the package does not own can see it. +# +# This is the whole of subos slice 1 end to end. The layer being tested is +# configuration: PATH and RPATH get a binary loaded, and neither can tell it +# where to find a GL driver. That is mcpp-community/mcpp#352 — a GLFW binary +# that links fine and exits 255 because LIBGL_DRIVERS_PATH points nowhere. +# +# What has to hold: +# 1. install — the declaration lands in the subos manifest, keyed by +# the declaring package +# 2. --shell — eval'ing the emitted code sets the variable +# 3. --cmd — a command run in the subos inherits it +# 4. user override — a value the user exported already wins (UC-1) +# 5. uninstall — the section goes, and `envs` stays as {} +# 6. doctor — clean afterwards, and it catches an orphaned section +# +# The probe rule has its own test (E2E: subos_env_probe_compat_test.sh); this +# one assumes the capability is present. +# +# Design: .agents/docs/2026-08-05-subos-minimum-design.md + +set -uo pipefail + +# shellcheck source=./project_test_lib.sh +source "$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/project_test_lib.sh" + +require_fixture_index + +RUNTIME_DIR="$ROOT_DIR/tests/e2e/runtime/subos_env" +LOCAL_INDEX_DIR="$RUNTIME_DIR/xim-pkgindex" +HOME_DIR="$RUNTIME_DIR/home" + +cleanup() { rm -rf "$RUNTIME_DIR"; } +trap cleanup EXIT +cleanup +mkdir -p "$RUNTIME_DIR" + +BIN="$(find_xlings_bin)" +log "client: $("$BIN" --version 2>&1 | head -1)" + +# ── a package that declares two variables ─────────────────────────────── +cp -r "$FIXTURE_INDEX_DIR" "$LOCAL_INDEX_DIR" +printf 'xim_indexrepos = {}\n' > "$LOCAL_INDEX_DIR/xim-indexrepos.lua" +rm -f "$LOCAL_INDEX_DIR/.xlings-index-cache.json" +mkdir -p "$LOCAL_INDEX_DIR/pkgs/e" + +cat > "$LOCAL_INDEX_DIR/pkgs/e/envfixture.lua" <<'LUA' +package = { + spec = "1", + name = "envfixture", + description = "Local fixture for tests/e2e/subos_env_declaration_test.sh", + authors = {"xlings-ci"}, + licenses = {"MIT"}, + type = "package", + archs = {"x86_64"}, + status = "stable", + categories = {"test-fixture"}, + + xpm = { + linux = { ["1.0.0"] = {} }, + macosx = { ["1.0.0"] = {} }, + windows = { ["1.0.0"] = {} }, + }, +} + +import("xim.libxpkg.pkginfo") +import("xim.libxpkg.xvm") +import("xim.libxpkg.subos") + +function install() + local dir = pkginfo.install_dir() + os.tryrm(dir) + os.mkdir(path.join(dir, "bin")) + os.mkdir(path.join(dir, "drivers")) + os.mkdir(path.join(dir, "share")) + io.writefile(path.join(dir, "bin", "envfixture"), "#!/bin/sh\necho envfixture\n") + os.exec("chmod +x " .. path.join(dir, "bin", "envfixture")) + return true +end + +function config() + xvm.add(package.name, { bindir = path.join(pkginfo.install_dir(), "bin") }) + + -- type(), not truthiness: import() hands back a permissive proxy for a + -- module the client does not ship, and every key on it is truthy. + if type(subos.env) == "function" then + local binding = package.name .. "@" .. pkginfo.version() + subos.env{ var = "E2E_DRIVERS_PATH", op = "set", + value = "${pkgdir}/drivers", binding = binding } + subos.env{ var = "E2E_DATA_DIRS", op = "prepend", + value = "${pkgdir}/share", binding = binding } + end + return true +end + +function uninstall() + -- Nothing here on purpose. The env section is provider-scoped, and xlings + -- drops it with the package; a recipe writing its own cleanup would be + -- the second owner of that state. + return true +end +LUA + +# ── an isolated home ──────────────────────────────────────────────────── +mkdir -p "$HOME_DIR/subos/default/bin" "$HOME_DIR/data/xim-index-repos" +cat > "$HOME_DIR/.xlings.json" < "$HOME_DIR/data/xim-index-repos/xim-indexrepos.json" +cp "$BIN" "$HOME_DIR/xlings" + +# env -i: the variables under test must come from the subos, not be inherited +# from whatever shell is running the suite. +x() { ( cd /tmp && env -i HOME="$HOME" PATH=/usr/bin:/bin \ + XLINGS_HOME="$HOME_DIR" "$BIN" "$@" ) } + +x self init >/dev/null 2>&1 || true + +MANIFEST="$HOME_DIR/subos/default/.xlings.json" +[[ -f "$MANIFEST" ]] || fail "self init produced no subos manifest" + +# ── 1. install records the declaration ────────────────────────────────── +OUT="$(x install envfixture@1.0.0 -y 2>&1)" || { echo "$OUT" >&2; fail "install failed"; } + +python3 - "$MANIFEST" <<'PY' || exit 1 +import json, sys +info = json.load(open(sys.argv[1])).get("subos_info") +if info is None: + raise SystemExit("no subos_info block after install") +envs = info.get("envs", {}) +key = "envfixture@1.0.0" +if key not in envs: + raise SystemExit(f"no section for {key}; envs={envs}") +got = {d["var"]: (d["op"], d["value"]) for d in envs[key]} +want = { + "E2E_DRIVERS_PATH": ("set", "${pkgdir}/drivers"), + "E2E_DATA_DIRS": ("prepend", "${pkgdir}/share"), +} +if got != want: + raise SystemExit(f"recorded {got}, expected {want}") +# The stored value must stay a placeholder. A manifest holding this machine's +# absolute paths describes this machine, and a subos description that only +# works where it was written is not a description. +print("[project-e2e] ✓ recorded, and still portable") +PY + +# Installing again must not grow the section — config() re-runs on every +# dependent install. +x install envfixture@1.0.0 -y >/dev/null 2>&1 +COUNT="$(python3 -c ' +import json,sys +print(len(json.load(open(sys.argv[1]))["subos_info"]["envs"]["envfixture@1.0.0"]))' "$MANIFEST")" +[[ "$COUNT" == "2" ]] || fail "reinstall grew the section to $COUNT declarations" +log " ✓ re-install is idempotent" + +# ── 2. --shell emits code that sets them ──────────────────────────────── +SHELL_OUT="$(x subos use default --shell sh 2>/dev/null)" +EVALED="$( env -i PATH=/usr/bin:/bin bash -c "eval '$SHELL_OUT' + echo \"DRIVERS=\$E2E_DRIVERS_PATH\" + echo \"DATA=\$E2E_DATA_DIRS\"" )" +grep -q "DRIVERS=.*/drivers$" <<<"$EVALED" \ + || { echo "$EVALED" >&2; fail "--shell did not set E2E_DRIVERS_PATH"; } +grep -q "DATA=.*/share$" <<<"$EVALED" \ + || { echo "$EVALED" >&2; fail "--shell did not set E2E_DATA_DIRS"; } +# The expanded path must exist — the placeholder resolved to a real payload. +DRIVERS_PATH="$(sed -n 's/^DRIVERS=//p' <<<"$EVALED")" +[[ -d "$DRIVERS_PATH" ]] || fail "E2E_DRIVERS_PATH=$DRIVERS_PATH is not a directory" +log " ✓ --shell exports both, expanded to a real payload" + +# UC-2: the user is told what was injected, on stderr so the stdout stays +# eval-safe. +REPORT="$(x subos use default --shell sh 2>&1 >/dev/null)" +grep -q "2 env var(s) from 1 package(s)" <<<"$REPORT" \ + || { echo "$REPORT" >&2; fail "--shell did not report what it injected"; } +log " ✓ --shell reports the injected set on stderr" + +# ── 3. --cmd inherits them ────────────────────────────────────────────── +CMD_OUT="$(x subos use default --cmd 'echo CMD=$E2E_DRIVERS_PATH' 2>/dev/null)" +grep -q "CMD=.*/drivers$" <<<"$CMD_OUT" \ + || { echo "$CMD_OUT" >&2; fail "--cmd did not inject E2E_DRIVERS_PATH"; } +log " ✓ --cmd injects into the process environment" + +# ── 4. the user's own value wins (UC-1) ───────────────────────────────── +OVERRIDE="$( cd /tmp && env -i HOME="$HOME" PATH=/usr/bin:/bin \ + XLINGS_HOME="$HOME_DIR" \ + E2E_DRIVERS_PATH=/user/choice E2E_DATA_DIRS=/user/share \ + "$BIN" subos use default \ + --cmd 'echo "D=$E2E_DRIVERS_PATH"; echo "X=$E2E_DATA_DIRS"' 2>/dev/null )" +grep -q "^D=/user/choice$" <<<"$OVERRIDE" \ + || { echo "$OVERRIDE" >&2; fail "a 'set' declaration overwrote the user's value"; } +# prepend still composes — that is what prepend means — but it must not +# discard what was there. +grep -q "^X=.*/share:/user/share$" <<<"$OVERRIDE" \ + || { echo "$OVERRIDE" >&2; fail "'prepend' did not compose with the user's value"; } +log " ✓ the user's value survives 'set' and is kept by 'prepend'" + +# ── 5. doctor is clean ────────────────────────────────────────────────── +DOC="$(x self doctor 2>&1)" +grep -qiE "subos env (orphan|unresolved)" <<<"$DOC" \ + && { echo "$DOC" >&2; fail "doctor reports a defect on a healthy subos"; } +log " ✓ doctor is clean while the package is installed" + +# ...and catches a section whose owner is gone. Removing the payload behind +# the package's back is what an interrupted uninstall leaves. +python3 - "$MANIFEST" <<'PY' +import json, sys +p = sys.argv[1] +d = json.load(open(p)) +d["subos_info"]["envs"]["ghost@9.9.9"] = [ + {"var": "GHOST", "op": "set", "value": "${pkgdir}/lib"}] +json.dump(d, open(p, "w"), indent=2) +PY +DOC="$(x self doctor 2>&1)" +grep -qi "subos env orphan" <<<"$DOC" \ + || { echo "$DOC" >&2; fail "doctor missed an orphaned env section"; } +x self doctor --fix >/dev/null 2>&1 +python3 -c ' +import json,sys +envs = json.load(open(sys.argv[1]))["subos_info"]["envs"] +assert "ghost@9.9.9" not in envs, "--fix left the orphan behind" +assert "envfixture@1.0.0" in envs, "--fix took the healthy section too"' "$MANIFEST" \ + || fail "doctor --fix did not repair exactly the orphan" +log " ✓ doctor detects and repairs an orphaned section, and only that" + +# ── 6. uninstall drops the section ────────────────────────────────────── +x remove envfixture -y >/dev/null 2>&1 || x uninstall envfixture -y >/dev/null 2>&1 + +python3 - "$MANIFEST" <<'PY' || exit 1 +import json, sys +info = json.load(open(sys.argv[1]))["subos_info"] +envs = info["envs"] +if "envfixture@1.0.0" in envs: + raise SystemExit(f"uninstall left the section behind: {envs}") +# "envs" itself must stay, as {}. An absent key and an empty one would be two +# states meaning the same thing, and every reader would have to handle both. +if not isinstance(envs, dict): + raise SystemExit(f"envs is no longer an object: {envs!r}") +print("[project-e2e] ✓ section removed, envs kept as {}") +PY + +# And the variable is gone from what a new shell would get. +AFTER="$(x subos use default --cmd 'echo AFTER=${E2E_DRIVERS_PATH:-}' 2>/dev/null)" +grep -q "AFTER=" <<<"$AFTER" \ + || { echo "$AFTER" >&2; fail "the variable is still injected after uninstall"; } +log " ✓ the variable is no longer injected" + +log "E2E subos.env declaration lifecycle: PASS" diff --git a/tests/e2e/subos_env_probe_compat_test.sh b/tests/e2e/subos_env_probe_compat_test.sh new file mode 100755 index 00000000..cbcf0b7f --- /dev/null +++ b/tests/e2e/subos_env_probe_compat_test.sh @@ -0,0 +1,200 @@ +#!/usr/bin/env bash +# E2E: the `subos.env` capability probe, run against a REAL old xlings. +# +# `xim.libxpkg.subos` is a NEW MODULE, and that changes the probe rule the V2 +# spec gives for a new *function* on an existing module. +# +# if xvm.files then ... end -- correct: `xvm` exists on old +# clients, so the field really is nil +# if subos.env then ... end -- WRONG for a new module +# if type(subos.env) == "function" -- correct for a new module +# +# import() answers an unknown module with a permissive proxy stub: every key +# read off it returns a truthy, callable table. So on a client that predates +# the module, `if subos.env then` is TRUE, the recipe takes the new branch, +# calls it, and the call evaporates. The install succeeds, nothing is +# configured, and nothing says so. +# +# Reading the prelude says all this. That is not enough — the same reasoning +# said `xvm.files` was safe, and it was only safe by accident of `xvm` already +# existing. So this runs one recipe through a real released binary and the +# current build and asserts what each observed: +# +# old client → truthiness TRUE, type() FALSE → legacy branch +# new client → truthiness TRUE, type() TRUE → subos.env branch +# +# The first line is the finding. If it ever reads FALSE the rule can be +# relaxed; until then it must stay, and this is what proves it. +# +# XLINGS_OLD_BIN can point at an already-downloaded old binary; otherwise the +# test fetches one, and skips (exit 0) if there is no network. + +set -uo pipefail + +# shellcheck source=./project_test_lib.sh +source "$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/project_test_lib.sh" + +require_fixture_index + +OLD_VERSION="${XLINGS_OLD_VERSION:-2026.8.4.2}" +RUNTIME_DIR="$ROOT_DIR/tests/e2e/runtime/subos_env_probe" +LOCAL_INDEX_DIR="$RUNTIME_DIR/xim-pkgindex" + +cleanup() { rm -rf "$RUNTIME_DIR"; } +trap cleanup EXIT +cleanup +mkdir -p "$RUNTIME_DIR" + +NEW_BIN="$(find_xlings_bin)" + +OLD_BIN="${XLINGS_OLD_BIN:-}" +if [[ -z "$OLD_BIN" ]]; then + TARBALL="$RUNTIME_DIR/old.tar.gz" + URL="https://github.com/openxlings/xlings/releases/download/v${OLD_VERSION}/xlings-${OLD_VERSION}-linux-x86_64.tar.gz" + if ! curl -fsSL --max-time 120 -o "$TARBALL" "$URL"; then + log "SKIP: cannot fetch xlings $OLD_VERSION (no network?)" + exit 0 + fi + tar xzf "$TARBALL" -C "$RUNTIME_DIR" + OLD_BIN="$(find "$RUNTIME_DIR" -type f -name xlings -perm -u+x | head -1)" +fi +[[ -x "$OLD_BIN" ]] || { log "SKIP: no usable old binary"; exit 0; } +log "old client: $("$OLD_BIN" --version 2>&1 | head -1)" +log "new client: $("$NEW_BIN" --version 2>&1 | head -1)" + +# ── one recipe, recording what each client observed ───────────────────── +cp -r "$FIXTURE_INDEX_DIR" "$LOCAL_INDEX_DIR" +printf 'xim_indexrepos = {}\n' > "$LOCAL_INDEX_DIR/xim-indexrepos.lua" +rm -f "$LOCAL_INDEX_DIR/.xlings-index-cache.json" +mkdir -p "$LOCAL_INDEX_DIR/pkgs/s" + +cat > "$LOCAL_INDEX_DIR/pkgs/s/subosprobe.lua" <<'LUA' +package = { + spec = "1", + name = "subosprobe", + description = "Local fixture for tests/e2e/subos_env_probe_compat_test.sh", + authors = {"xlings-ci"}, + licenses = {"MIT"}, + type = "package", + archs = {"x86_64"}, + status = "stable", + categories = {"test-fixture"}, + + xpm = { + linux = { ["1.0.0"] = {} }, + macosx = { ["1.0.0"] = {} }, + windows = { ["1.0.0"] = {} }, + }, +} + +import("xim.libxpkg.pkginfo") +import("xim.libxpkg.xvm") +import("xim.libxpkg.subos") + +function install() + local dir = pkginfo.install_dir() + os.tryrm(dir) + os.mkdir(path.join(dir, "bin")) + os.mkdir(path.join(dir, "drivers")) + io.writefile(path.join(dir, "bin", "subosprobe"), "#!/bin/sh\necho probe\n") + os.exec("chmod +x " .. path.join(dir, "bin", "subosprobe")) + return true +end + +function config() + xvm.add(package.name, { bindir = path.join(pkginfo.install_dir(), "bin") }) + + local dir = pkginfo.install_dir() + -- Both readings, written down. The gap between them IS the finding. + io.writefile(path.join(dir, "TRUTHY"), + tostring(subos.env ~= nil)) + io.writefile(path.join(dir, "TYPED"), + tostring(type(subos.env) == "function")) + + if type(subos.env) == "function" then + io.writefile(path.join(dir, "BRANCH"), "subos.env") + subos.env{ var = "E2E_PROBE_PATH", op = "set", + value = "${pkgdir}/drivers", + binding = package.name .. "@" .. pkginfo.version() } + else + io.writefile(path.join(dir, "BRANCH"), "legacy") + end + return true +end + +function uninstall() + return true +end +LUA + +run_with() { # + ( cd /tmp && env -i HOME="$HOME" PATH=/usr/bin:/bin XLINGS_HOME="$2" "$1" "${@:3}" ) +} + +prepare_home() { # + mkdir -p "$1/subos/default/bin" "$1/data/xim-index-repos" + cat > "$1/.xlings.json" < "$1/data/xim-index-repos/xim-indexrepos.json" +} + +read_marker() { # + local f + f="$(find "$1" -name "$2" -type f 2>/dev/null | head -1)" + [[ -n "$f" ]] && cat "$f" || echo "" +} + +# old: truthy true, typed false, legacy branch +# new: truthy true, typed true, subos.env branch +for pair in "old:$OLD_BIN:true:false:legacy" "new:$NEW_BIN:true:true:subos.env"; do + IFS=: read -r label bin wantTruthy wantTyped wantBranch <<<"$pair" + HOME_DIR="$RUNTIME_DIR/home-$label" + mkdir -p "$HOME_DIR" + cp "$bin" "$HOME_DIR/xlings" + prepare_home "$HOME_DIR" + run_with "$bin" "$HOME_DIR" self init >/dev/null 2>&1 || true + + OUT="$(run_with "$bin" "$HOME_DIR" install subosprobe@1.0.0 -y 2>&1)" + if grep -qiE "unsupported registration node kind|config hook failed" <<<"$OUT"; then + echo "$OUT" >&2 + fail "$label client failed to install — the probe does not gate cleanly" + fi + + gotTruthy="$(read_marker "$HOME_DIR" TRUTHY)" + gotTyped="$(read_marker "$HOME_DIR" TYPED)" + gotBranch="$(read_marker "$HOME_DIR" BRANCH)" + + [[ "$gotTruthy" == "$wantTruthy" ]] || { + echo "$OUT" >&2 + fail "$label: 'subos.env ~= nil' was $gotTruthy, expected $wantTruthy" + } + [[ "$gotTyped" == "$wantTyped" ]] || { + echo "$OUT" >&2 + fail "$label: 'type(subos.env)==function' was $gotTyped, expected $wantTyped" + } + [[ "$gotBranch" == "$wantBranch" ]] || { + echo "$OUT" >&2 + fail "$label client took the '$gotBranch' branch, expected '$wantBranch'" + } + log " ✓ $label → truthy=$gotTruthy typed=$gotTyped branch=$gotBranch" +done + +# The old client must not have silently produced a manifest section, and the +# new one must have. +OLD_MANIFEST="$RUNTIME_DIR/home-old/subos/default/.xlings.json" +NEW_MANIFEST="$RUNTIME_DIR/home-new/subos/default/.xlings.json" + +if [[ -f "$OLD_MANIFEST" ]] && grep -q "E2E_PROBE_PATH" "$OLD_MANIFEST"; then + fail "the old client recorded an env declaration it cannot apply" +fi +grep -q "E2E_PROBE_PATH" "$NEW_MANIFEST" \ + || fail "the new client did not record the declaration" + +log " ✓ only the client that can apply the declaration recorded one" +log "E2E subos.env probe compatibility: PASS" diff --git a/tests/unit/test_subos_manifest.cpp b/tests/unit/test_subos_manifest.cpp new file mode 100644 index 00000000..7c298764 --- /dev/null +++ b/tests/unit/test_subos_manifest.cpp @@ -0,0 +1,345 @@ +// Unit tests for the `subos_info` manifest block (subos slice 1). +// +// Design: .agents/docs/2026-08-05-subos-minimum-design.md +// Landing plan: .agents/docs/2026-08-05-subos-slice1-landing-plan.md +// +// The module deliberately takes paths and a resolver instead of reaching for +// Config, so everything here runs without a home on disk. +#include + +import std; +import xlings.core.subos.manifest; +import xlings.libs.json; + +namespace m = xlings::subos::manifest; +namespace fs = std::filesystem; + +namespace { + +m::Placeholders test_placeholders() { + return m::Placeholders{ + .subosdir = "/x/subos/default", + .home = "/home/u", + .xlings_home = "/x", + .pkgdir_of = [](std::string_view binding) -> fs::path { + if (binding == "compat.mesa@25.0.0") return "/x/pkgs/compat.mesa/25.0.0"; + if (binding == "fontconfig@2.15.0") return "/x/pkgs/fontconfig/2.15.0"; + return {}; // unknown provider → unresolvable, on purpose + }, + }; +} + +nlohmann::json doc_with(const nlohmann::json& envs, + std::string runtime = "glibc@2.39") { + nlohmann::json d; + d["workspace"] = nlohmann::json::object(); + d["subos_info"] = { + {"schema_version", m::SCHEMA_VERSION}, + {"runtime", std::move(runtime)}, + {"envs", envs}, + {"created_at", "2026-08-05T14:23:11Z"}, + {"created_by", "xlings test"}, + }; + return d; +} + +bool has(const std::vector& fs, m::Defect d) { + return std::ranges::any_of(fs, [&](const auto& f) { return f.kind == d; }); +} + +} // namespace + +// ── runtime family ─────────────────────────────────────────────────── + +TEST(SubosManifestRuntime, DerivesFamilyFromThePackageName) { + EXPECT_EQ(m::family_of("glibc@2.39"), "linux-x86_64-glibc"); + EXPECT_EQ(m::family_of("musl@1.2.5"), "linux-x86_64-musl"); + EXPECT_EQ(m::family_of("glibc@2.39", "aarch64"), "linux-aarch64-glibc"); + EXPECT_EQ(m::family_of("wasi-libc@0.1"), "wasm32-wasi"); +} + +// A family that is derived cannot contradict the runtime it came from, which +// is the whole reason it is not a stored field. +TEST(SubosManifestRuntime, UnknownRuntimeIsNamedRatherThanGuessed) { + EXPECT_EQ(m::family_of("something-else@1.0"), "unknown"); + EXPECT_EQ(m::family_of(""), "unknown"); +} + +TEST(SubosManifestRuntime, BindingShapeRequiresBothHalves) { + EXPECT_TRUE(m::is_binding("glibc@2.39")); + EXPECT_FALSE(m::is_binding("glibc")); + EXPECT_FALSE(m::is_binding("glibc@")); + EXPECT_FALSE(m::is_binding("@2.39")); +} + +// ── invariants ─────────────────────────────────────────────────────── + +TEST(SubosManifestValidate, AcceptsAWellFormedBlock) { + EXPECT_TRUE(m::validate_block(doc_with(nlohmann::json::object())).empty()); +} + +TEST(SubosManifestValidate, ReportsAMissingBlock) { + nlohmann::json d; + d["workspace"] = nlohmann::json::object(); + EXPECT_TRUE(has(m::validate_block(d), m::Defect::BlockMissing)); +} + +// An empty envs object is the correct state for a subos with no declarations, +// and must not be confused with a missing one — the whole point of writing {} +// at creation is that no reader has to handle "absent". +TEST(SubosManifestValidate, EmptyEnvsIsValidButAbsentEnvsIsNot) { + EXPECT_TRUE(m::validate_block(doc_with(nlohmann::json::object())).empty()); + + auto d = doc_with(nlohmann::json::object()); + d["subos_info"].erase("envs"); + EXPECT_TRUE(has(m::validate_block(d), m::Defect::EnvsMalformed)); +} + +TEST(SubosManifestValidate, RejectsAMalformedRuntime) { + EXPECT_TRUE(has(m::validate_block(doc_with(nlohmann::json::object(), "glibc")), + m::Defect::RuntimeMalformed)); +} + +TEST(SubosManifestValidate, RejectsAProviderKeyThatIsNotABinding) { + auto d = doc_with({{"compat.mesa", nlohmann::json::array()}}); + EXPECT_TRUE(has(m::validate_block(d), m::Defect::EnvsMalformed)); +} + +TEST(SubosManifestValidate, RejectsADeclarationWithAnOpThisSliceCannotApply) { + auto d = doc_with({{"compat.mesa@25.0.0", nlohmann::json::array({ + {{"var", "PATH"}, {"op", "append"}, {"value", "x"}}, + })}}); + EXPECT_TRUE(has(m::validate_block(d), m::Defect::EnvDeclMalformed)); +} + +TEST(SubosManifestValidate, RejectsAFutureSchemaVersion) { + auto d = doc_with(nlohmann::json::object()); + d["subos_info"]["schema_version"] = m::SCHEMA_VERSION + 1; + EXPECT_TRUE(has(m::validate_block(d), m::Defect::SchemaUnsupported)); +} + +TEST(SubosManifestValidate, ReportsMissingProvenance) { + auto d = doc_with(nlohmann::json::object()); + d["subos_info"]["created_by"] = ""; + EXPECT_TRUE(has(m::validate_block(d), m::Defect::ProvenanceMissing)); +} + +// validate() on disk, including the case that must not be treated as "absent": +// a file that exists but does not parse. Silently reading it as {} would let a +// repair rewrite a document it never managed to read. +TEST(SubosManifestValidate, DistinguishesAbsentFromUnreadable) { + auto dir = fs::temp_directory_path() / "xlings_subos_manifest_validate"; + fs::remove_all(dir); + fs::create_directories(dir); + + EXPECT_TRUE(has(m::validate(dir), m::Defect::ConfigMissing)); + + std::ofstream(m::config_path(dir)) << "{ this is not json"; + EXPECT_TRUE(has(m::validate(dir), m::Defect::ConfigUnreadable)); + + std::ofstream(m::config_path(dir)) << doc_with(nlohmann::json::object()).dump(2); + EXPECT_TRUE(m::validate(dir).empty()); + + fs::remove_all(dir); +} + +TEST(SubosManifestValidate, ReportsAMissingDirectory) { + EXPECT_TRUE(has(m::validate(fs::temp_directory_path() / "xlings_no_such_subos"), + m::Defect::DirMissing)); +} + +// ── declarations ───────────────────────────────────────────────────── + +TEST(SubosManifestEnv, RecordsUnderTheProviderBinding) { + auto d = doc_with(nlohmann::json::object()); + EXPECT_TRUE(m::add_env(d, "compat.mesa@25.0.0", + {"LIBGL_DRIVERS_PATH", "set", "${pkgdir}/lib/dri"})); + ASSERT_TRUE(d["subos_info"]["envs"].contains("compat.mesa@25.0.0")); + EXPECT_EQ(d["subos_info"]["envs"]["compat.mesa@25.0.0"].size(), 1u); +} + +// config() runs again on every dependent package's install. Without this the +// section would grow a duplicate row per rebuild. +TEST(SubosManifestEnv, IsIdempotentOnTheWholeTriple) { + auto d = doc_with(nlohmann::json::object()); + const m::EnvDecl decl{"LIBGL_DRIVERS_PATH", "set", "${pkgdir}/lib/dri"}; + + EXPECT_TRUE(m::add_env(d, "compat.mesa@25.0.0", decl)); + EXPECT_FALSE(m::add_env(d, "compat.mesa@25.0.0", decl)); + EXPECT_EQ(d["subos_info"]["envs"]["compat.mesa@25.0.0"].size(), 1u); + + // A different value for the same variable is a new declaration, not a + // duplicate — the manifest records what was asked for, and resolve() + // decides what wins. + EXPECT_TRUE(m::add_env(d, "compat.mesa@25.0.0", + {"LIBGL_DRIVERS_PATH", "set", "${pkgdir}/lib/other"})); + EXPECT_EQ(d["subos_info"]["envs"]["compat.mesa@25.0.0"].size(), 2u); +} + +TEST(SubosManifestEnv, RemovingAProviderTakesItsWholeSectionAndNothingElse) { + auto d = doc_with(nlohmann::json::object()); + m::add_env(d, "compat.mesa@25.0.0", {"LIBGL_DRIVERS_PATH", "set", "a"}); + m::add_env(d, "fontconfig@2.15.0", {"FONTCONFIG_PATH", "set", "b"}); + + EXPECT_TRUE(m::remove_provider(d, "compat.mesa@25.0.0")); + EXPECT_FALSE(d["subos_info"]["envs"].contains("compat.mesa@25.0.0")); + EXPECT_TRUE(d["subos_info"]["envs"].contains("fontconfig@2.15.0")); + + // Removing what is not there is a no-op, not a failure: uninstall runs for + // packages that never declared anything. + EXPECT_FALSE(m::remove_provider(d, "compat.mesa@25.0.0")); +} + +TEST(SubosManifestEnv, EnvsSurvivesAsAnEmptyObjectAfterTheLastRemoval) { + auto d = doc_with(nlohmann::json::object()); + m::add_env(d, "compat.mesa@25.0.0", {"LIBGL_DRIVERS_PATH", "set", "a"}); + m::remove_provider(d, "compat.mesa@25.0.0"); + + ASSERT_TRUE(d["subos_info"]["envs"].is_object()); + EXPECT_TRUE(d["subos_info"]["envs"].empty()); + EXPECT_TRUE(m::validate_block(d).empty()); +} + +TEST(SubosManifestEnv, FindsStaleProvidersOfTheSamePackageAcrossVersions) { + auto d = doc_with(nlohmann::json::object()); + m::add_env(d, "compat.mesa@25.0.0", {"A", "set", "1"}); + m::add_env(d, "compat.mesa@24.0.0", {"A", "set", "2"}); + m::add_env(d, "fontconfig@2.15.0", {"B", "set", "3"}); + + auto found = m::providers_named(d, "compat.mesa"); + EXPECT_EQ(found.size(), 2u); +} + +// ── placeholders ───────────────────────────────────────────────────── + +TEST(SubosManifestExpand, SubstitutesEveryKnownPlaceholder) { + auto ph = test_placeholders(); + EXPECT_EQ(m::expand("${pkgdir}/lib/dri", "compat.mesa@25.0.0", ph), + "/x/pkgs/compat.mesa/25.0.0/lib/dri"); + EXPECT_EQ(m::expand("${subosdir}/usr", "any@1", ph), "/x/subos/default/usr"); + EXPECT_EQ(m::expand("${home}/.cache", "any@1", ph), "/home/u/.cache"); + EXPECT_EQ(m::expand("${xlings_home}/data", "any@1", ph), "/x/data"); +} + +TEST(SubosManifestExpand, HandlesSeveralPlaceholdersAndPlainText) { + auto ph = test_placeholders(); + EXPECT_EQ(m::expand("a:${home}/b:${xlings_home}/c", "any@1", ph), + "a:/home/u/b:/x/c"); +} + +// The reason unresolvable does not mean empty: "${pkgdir}/lib/dri" collapsing +// to "/lib/dri" is a real path on the host, outside the subos, and a driver +// search would follow it. Leaving the text intact keeps the failure visible. +TEST(SubosManifestExpand, LeavesAnUnresolvableProviderVerbatim) { + auto ph = test_placeholders(); + const auto out = m::expand("${pkgdir}/lib/dri", "never.installed@9.9", ph); + EXPECT_EQ(out, "${pkgdir}/lib/dri"); + EXPECT_TRUE(m::has_unresolved(out)); +} + +TEST(SubosManifestExpand, LeavesUnknownAndUnterminatedPlaceholdersVerbatim) { + auto ph = test_placeholders(); + EXPECT_EQ(m::expand("${nope}/x", "any@1", ph), "${nope}/x"); + EXPECT_EQ(m::expand("${unterminated", "any@1", ph), "${unterminated"); + EXPECT_EQ(m::expand("plain", "any@1", ph), "plain"); +} + +// ── resolution ─────────────────────────────────────────────────────── + +TEST(SubosManifestResolve, ExpandsAndSortsDeclarations) { + auto d = doc_with(nlohmann::json::object()); + m::add_env(d, "compat.mesa@25.0.0", + {"LIBGL_DRIVERS_PATH", "set", "${pkgdir}/lib/dri"}); + m::add_env(d, "compat.mesa@25.0.0", + {"__EGL_VENDOR_LIBRARY_DIRS", "set", "${pkgdir}/share/glvnd"}); + + auto resolved = m::resolve(m::parse(d), test_placeholders()); + ASSERT_EQ(resolved.size(), 2u); + EXPECT_EQ(resolved[0].var, "LIBGL_DRIVERS_PATH"); + EXPECT_EQ(resolved[0].value, "/x/pkgs/compat.mesa/25.0.0/lib/dri"); + EXPECT_EQ(resolved[1].var, "__EGL_VENDOR_LIBRARY_DIRS"); + EXPECT_FALSE(resolved[0].conflicted); +} + +TEST(SubosManifestResolve, PrependsComposeWithoutBeingAConflict) { + auto d = doc_with(nlohmann::json::object()); + m::add_env(d, "compat.mesa@25.0.0", {"XDG_DATA_DIRS", "prepend", "${pkgdir}/share"}); + m::add_env(d, "fontconfig@2.15.0", {"XDG_DATA_DIRS", "prepend", "${pkgdir}/share"}); + + auto resolved = m::resolve(m::parse(d), test_placeholders()); + ASSERT_EQ(resolved.size(), 1u); + EXPECT_EQ(resolved[0].op, "prepend"); + EXPECT_FALSE(resolved[0].conflicted); + // Later provider nearer the front: "the newest thing installed is found + // first". Ordering is by binding, so it does not depend on install history. + EXPECT_EQ(resolved[0].value, + "/x/pkgs/fontconfig/2.15.0/share:/x/pkgs/compat.mesa/25.0.0/share"); + EXPECT_EQ(resolved[0].providers.size(), 2u); +} + +TEST(SubosManifestResolve, TwoSetsOnOneVariableAreReportedAsAConflict) { + auto d = doc_with(nlohmann::json::object()); + m::add_env(d, "compat.mesa@25.0.0", {"MESA_LOADER_DRIVER_OVERRIDE", "set", "a"}); + m::add_env(d, "fontconfig@2.15.0", {"MESA_LOADER_DRIVER_OVERRIDE", "set", "b"}); + + auto resolved = m::resolve(m::parse(d), test_placeholders()); + ASSERT_EQ(resolved.size(), 1u); + EXPECT_TRUE(resolved[0].conflicted); + // Last in binding order wins. The value matters less than the fact that it + // is the same on every machine holding this manifest. + EXPECT_EQ(resolved[0].value, "b"); +} + +TEST(SubosManifestResolve, SetBeatsPrependAndSaysSo) { + auto d = doc_with(nlohmann::json::object()); + m::add_env(d, "compat.mesa@25.0.0", {"XDG_DATA_DIRS", "prepend", "p"}); + m::add_env(d, "fontconfig@2.15.0", {"XDG_DATA_DIRS", "set", "s"}); + + auto resolved = m::resolve(m::parse(d), test_placeholders()); + ASSERT_EQ(resolved.size(), 1u); + EXPECT_EQ(resolved[0].op, "set"); + EXPECT_EQ(resolved[0].value, "s"); + EXPECT_TRUE(resolved[0].conflicted); +} + +// Resolution must be a function of the manifest alone. Two homes holding the +// same bytes have to export the same values, or a shared subos description +// describes nothing. +TEST(SubosManifestResolve, DoesNotDependOnDeclarationOrder) { + auto forwards = doc_with(nlohmann::json::object()); + m::add_env(forwards, "a.pkg@1.0", {"V", "prepend", "one"}); + m::add_env(forwards, "b.pkg@1.0", {"V", "prepend", "two"}); + + auto backwards = doc_with(nlohmann::json::object()); + m::add_env(backwards, "b.pkg@1.0", {"V", "prepend", "two"}); + m::add_env(backwards, "a.pkg@1.0", {"V", "prepend", "one"}); + + auto ph = test_placeholders(); + EXPECT_EQ(m::resolve(m::parse(forwards), ph)[0].value, + m::resolve(m::parse(backwards), ph)[0].value); +} + +TEST(SubosManifestResolve, FlagsAValueItCouldNotExpand) { + auto d = doc_with(nlohmann::json::object()); + m::add_env(d, "never.installed@9.9", {"V", "set", "${pkgdir}/lib"}); + + auto resolved = m::resolve(m::parse(d), test_placeholders()); + ASSERT_EQ(resolved.size(), 1u); + EXPECT_TRUE(resolved[0].unresolved); +} + +// ── creation ───────────────────────────────────────────────────────── + +TEST(SubosManifestBlock, NewBlockSatisfiesItsOwnInvariants) { + nlohmann::json d; + d["workspace"] = nlohmann::json::object(); + d["subos_info"] = m::make_block(m::DEFAULT_RUNTIME, "xlings test"); + + EXPECT_TRUE(m::validate_block(d).empty()); + + auto info = m::parse(d); + EXPECT_EQ(info.schema_version, m::SCHEMA_VERSION); + EXPECT_EQ(info.runtime, m::DEFAULT_RUNTIME); + EXPECT_TRUE(info.envs.empty()); + EXPECT_FALSE(info.created_at.empty()); +}