Skip to content

Functional client-side sandbox comparison site#40

Closed
chrisns wants to merge 2 commits into
mainfrom
feat/reporting-site
Closed

Functional client-side sandbox comparison site#40
chrisns wants to merge 2 commits into
mainfrom
feat/reporting-site

Conversation

@chrisns

@chrisns chrisns commented Jul 13, 2026

Copy link
Copy Markdown
Collaborator

A static, client-side GitHub Pages page that presents the probe's reports so someone can (a) compare AI-agent sandboxes to make a trust decision, (b) track degradations/improvements as harness and supporting-tech versions move, and (c) absorb new harnesses with no code change. Functional pass only — styling is deliberately plain, ready to hand to design.

What's here

  • site/index.html + vanilla app.js + placeholder style.css. All derivation (fingerprint-collapse, baseline-normalization, categorization, flips, exposure) is client-side from one concatenated reports file; falls back to sample-data.json in dev.
  • Views: baseline-normalized three-state matrix (🟥 leaked / 🟩 blocked / ⬜ n/a) with enforcement + root badges and ▲/▼ change markers; exposure-over-time ECharts step-line (CDN + SRI, progressive enhancement) with version-release markers and multi-identity overlay; per-capability heatmap; attributed flip-log; cell drill-down.
  • scripts/gen-sample-data.mjs — generates the dev fixture (history with real regressions/improvements to exercise every view).
  • Docs: CONTEXT.md (glossary), docs/adr/0001-* (pipeline decision + why not Actions artifacts), docs/reporting-site-plan.md (build checklist).

Extendability (holds by construction)

  • New harness → new row/line, automatic (identity from data).
  • New finding type → mapped category or an Other column, never dropped.

Not in this PR (tracked in the plan doc)

  • Track 1–3: probe list-targets, the soft-seeder, and the scan-matrix aggregate/publish job. The site is developed against the fixture until those land.
  • The design system.

https://claude.ai/code/session_01PtFjYK8MdnDMmFYVWPDRrn

Static GitHub Pages report over the probe's data: a baseline-normalized
three-state comparison matrix, an exposure-over-time chart with version-release
markers, a per-capability heatmap, and an attributed flip-log — all derived
client-side from one concatenated reports file (dev fixture: sample-data.json).

Design intentionally left plain; this is the functional pass for a design
system to build on. Records the domain glossary (CONTEXT.md), the pipeline
decision (ADR 0001), and the build checklist (docs/reporting-site-plan.md).

Claude-Session: https://claude.ai/code/session_01PtFjYK8MdnDMmFYVWPDRrn
A probe task can run and find nothing, emitting the finding type with an empty
value (e.g. DNS blocked -> external_host_dns_resolution: []). The renderer
counted a present finding as a leak regardless, so a blocked capability rendered
as leaked. Caught against real Seatbelt data, where strict confinement zeroes
the network findings rather than dropping them. Gate on a non-empty value.

Claude-Session: https://claude.ai/code/session_01PtFjYK8MdnDMmFYVWPDRrn
@chrisns

chrisns commented Jul 13, 2026

Copy link
Copy Markdown
Collaborator Author

Validated against real probe data, not just the fixture: seeded a throwaway $HOME, ran a real `direct` baseline and real Seatbelt-confined scans (`sandbox-exec`), and pointed the page at the resulting `all-reports.json`. Seatbelt cut sensitive reads 31→5 and (in the strict config) zeroed network — the page rendered the real baseline-vs-sandbox diff correctly.

That exercise caught a renderer bug now fixed here (bf1451e): a task that runs and finds nothing emits the finding type with an empty value (e.g. external_host_dns_resolution: []), which was being counted as a leak. Gated on a non-empty value — blocked capabilities now render as blocked. The data pipeline that produces real all-reports.json is #41.

@chrisns chrisns closed this Jul 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant