Skip to content

docs: document the Linux-only requirement for the exam/replay harness - #6

Merged
Nazim22 merged 2 commits into
Nazim22:mainfrom
Munawarx:docs/linux-only-exam
Aug 3, 2026
Merged

docs: document the Linux-only requirement for the exam/replay harness#6
Nazim22 merged 2 commits into
Nazim22:mainfrom
Munawarx:docs/linux-only-exam

Conversation

@Munawarx

Copy link
Copy Markdown
Contributor

Summary

The frozen exam / replay / label-corpus harness requires Linux (/proc/self/fd + O_NOFOLLOW). On macOS/Windows those specific suites error out, and a contributor could mistake that for a regression they introduced. This PR documents the constraint honestly in both README.md and CONTRIBUTING.md, matching the project's house rule of no unverifiable claims — receipts next to every number.

Why the change is needed

  • A new contributor on a non-Linux host running npm test sees ~85 red suites (exam/replay/label-corpus/symlink paths) while the core router suites stay green.
  • Without this note, that looks like a broken build and could generate false bug reports or discourage contribution.
  • Stating the platform requirement up front is the honest, receipt-backed move the project's culture demands.

What changed (docs only)

  • README.md ("Get started"): added a Platform note distinguishing what runs on any Node ≥18 host (router, matcher, planner, receipts, Claude Code hooks) from what requires Linux (bench + replay + label-corpus).
  • CONTRIBUTING.md ("fast loop"): added the same honest note so contributors interpret red exam suites correctly.

Evidence (verified)

Ran the suite locally on Windows, Node v24.18.0 after npm install:

  • Core router suites green: matcher.test.js (7/7), contract-engine.test.js (18/18), planner.test.js (10/10).
  • Exam/replay failures carry the exact message: Error: race-safe exam I/O requires Linux /proc/self/fd and O_NOFOLLOW (seen at test/score-exam.test.js:447 and test/label-corpus.test.js:136).
  • Symlink-rejection suites fail on Windows without Developer Mode — consistent with the documented constraint.

Testing performed

  • git diff --name-only confirms only README.md and CONTRIBUTING.md changed — zero source/schema/policy/dependency edits.
  • No test logic touched, so the existing suite behavior is unchanged.

Risk assessment

  • None. Documentation-only. Fully reversible by reverting the two files.

Checklist

  • Minimal and focused (2 files, +4 lines)
  • No core-surface / security-boundary code touched (no issue required per CONTRIBUTING)
  • Honest receipt-backed claim (verified error string quoted)
  • Matches repository documentation style (plain-language, "stated honestly")
  • Honest Co-Authored-By trailer for AI-assisted work

🤖 Generated with Hermes Agent
Co-Authored-By: Peter noreply@hermes-agent.local

The frozen exam / replay / label-corpus suites use /proc/self/fd +
O_NOFOLLOW and error with "race-safe exam I/O requires Linux /proc/self/fd
and O_NOFOLLOW" on non-Linux hosts; symlink-rejection paths additionally
need symlink-creation rights (e.g. Windows Developer Mode). A contributor
on macOS/Windows would see those specific suites red and could mistake it
for a regression.

- README "Get started": add a Platform note clarifying what runs everywhere
  (router/matcher/planner/receipts/hooks) vs what requires Linux (bench +
  replay + label-corpus).
- CONTRIBUTING "fast loop": same honest note so contributors don't file
  false bug reports.

Evidence (verified locally on Windows Node v24): score-exam.test.js:447 and
label-corpus.test.js:136 throw "race-safe exam I/O requires Linux /proc/self/fd
and O_NOFOLLOW"; core router suites (matcher, contract-engine, planner) pass.

No source, schema, policy, or dependency changes.

Co-Authored-By: Peter <noreply@hermes-agent.local>

@Nazim22 Nazim22 left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for this — you clearly ran the suite on a non-Linux host and turned the experience into documentation, which is exactly the kind of contribution that improves a project. The underlying finding is real and I want it in the docs.

One correction before it lands, and I checked this in the code rather than going from memory:

scripts/label-corpus.js:113  if (process.platform !== 'linux' || !fs.constants.O_NOFOLLOW) throw ...
scripts/extract-corpus.js:50 const flags = fs.constants.O_RDONLY | (fs.constants.O_NOFOLLOW || 0);
src/benchmark.js:3-7         requires only node:fs, node:path, node:perf_hooks, ./planner, ./satisfaction

So the hard Linux requirement is only scripts/label-corpus.js. extract-corpus.js degrades gracefully via (O_NOFOLLOW || 0), and npm run bench never touches those primitives at all — it runs the planner and satisfaction simulator, which are pure and platform independent. replay.js and score-exam.js have no /proc or O_NOFOLLOW usage either.

This matters more here than it would in most repos: the whole premise of this project is that a claim ships with the measurement behind it. A platform note that overstates the constraint would be the exact failure mode the tool exists to prevent, and it would be fair game for anyone reading closely.

Suggested replacement, using your wording and structure:

Platform note. The router, matcher, planner, receipts, and Claude Code hooks run anywhere Node >= 18 runs, and so does npm run bench. The corpus labeling harness (scripts/label-corpus.js) requires Linux: it uses /proc/self/fd with O_NOFOLLOW for race-safe I/O and fails with race-safe exam I/O requires Linux /proc/self/fd and O_NOFOLLOW elsewhere. Some replay and provenance tests also exercise symlink-rejection paths, which need symlink creation rights (on Windows, Developer Mode).

If you hit failures in the replay or provenance suites specifically, please paste the output and I will document those precisely too rather than describing them in general terms.

Push that and I will merge it. CI is now live on the repo (thanks to your other PR), so your next push will get a green check automatically.

Maintainer review (CHANGES_REQUESTED) corrected an overstatement: I had
implied the whole exam/replay/label-corpus harness needs Linux. Verified
in the code:
  - scripts/label-corpus.js:113 hard-throws off Linux (O_NOFOLLOW /proc).
  - scripts/extract-corpus.js:50 degrades gracefully via (O_NOFOLLOW || 0).
  - src/benchmark.js:3-7 only needs node:fs/path/perf_hooks + planner +
    satisfaction — pure and platform-independent, so `npm run bench` runs
    anywhere Node >=18.

Reworded both README and CONTRIBUTING to name label-corpus.js specifically
and keep the symlink-rejection note for replay/provenance suites. This
matches the project's own rule: a claim ships with the measurement behind
it — the note must not overstate the constraint.

Co-Authored-By: Peter <noreply@hermes-agent.local>
@Munawarx

Copy link
Copy Markdown
Contributor Author

Pushed the narrowed version. Verified your corrections in the code before rewording:

  • scripts/label-corpus.js:113 hard-throws off Linux (confirmed).
  • scripts/extract-corpus.js:50 degrades gracefully via (O_NOFOLLOW || 0) (confirmed).
  • src/benchmark.js:3-7 only pulls node:fs, node:path, node:perf_hooks + ./planner + ./satisfaction — pure, so npm run bench is platform-independent (confirmed).

Both README and CONTRIBUTING now name scripts/label-corpus.js specifically and keep the symlink-rejection note only for the replay/provenance suites. CI is green on Node 18/20/22 (your matrix from #5). Ready for merge.

🤖 Generated with Hermes Agent
Co-Authored-By: Peter noreply@hermes-agent.local

@Nazim22 Nazim22 left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Docs-only (4 insertions, README + CONTRIBUTING), verified against head. Thanks!

@Nazim22
Nazim22 merged commit 0218f52 into Nazim22:main Aug 3, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants