feat(skill): Sigma.js viz fallback + batched community-label reconciliation

CHANGELOG.md

@@ -4,6 +4,15 @@ Full release notes with details on each version: [GitHub Releases](https://githu
 
 ## Unreleased
 
+- Feat: Sigma.js + graphology fallback for large `graph.html` views, plus batched/reconciled community labeling at scale. vis-network's live client-side forceAtlas2 physics simulation is genuinely slow/laggy past a few hundred nodes regardless of hardware — verified in production on a 1083-community monorepo graph (apa, 2026-07-03). Step 6 of the skill now generates a separate WebGL Sigma.js render (`graph_sigma.html`) with the layout precomputed offline in Python (`nx.forceatlas2_layout`, no client-side physics) once the view exceeds ~300 nodes; the full recipe lives in the new `references/sigma-viz.md`, shipped to every split-platform skill variant. Step 5's community labeling also gains a batched-subagent flow for 100+ communities (sample each community's top-degree nodes, dispatch one `general-purpose` subagent per ~15-20-community batch in parallel) plus a required cross-batch label-reconciliation pass — a naive single-pass label run on the same graph left 338 of 1083 communities (31%) sharing a label with another before reconciliation.
+  The Sigma view also fixes a node-sizing bug (`autoRescale` must be disabled alongside `itemSizesReference: "positions"`, per sigma's own docs — otherwise sizes stay anchored to the pre-rescale coordinate frame and don't track zoom) and recalibrates node size against the layout's own median nearest-neighbor distance instead of a fixed constant, so communities stay legible instead of overlapping into a blob as node count grows. It now also surfaces each community's dominant content kind as an icon (code/document/paper/image/rationale/concept, via `@sigma/node-image`'s `NodePictogramProgram`) and dominant top-level module as color, plus a left-side panel to filter by entity kind and by relation type (grouped into calls/structure/imports/references/documentation-and-concepts/groups, covering both code and non-code relations) and a clickable module legend to isolate one part of the codebase at a time.
+  Two more critical rendering bugs were caught and fixed only by actually opening a generated file in a real browser (Chrome via Playwright), not by reading the code: (1) disabling `autoRescale` also disables sigma's automatic "fit the camera to the graph" on load, so without an explicit `fitViewportToNodes` call (from `@sigma/utils`) the camera pointed at empty space and the page rendered as a solid black screen with nothing visible; (2) the `@sigma/node-image` and `@sigma/utils` ESM imports failed at module-load time because esm.sh resolved their `sigma` peer dependency to a broken literal path instead of the pinned version — that import throws before any of the script runs, which is the more fundamental cause of the black screen (the camera fix alone wasn't sufficient). Both `@sigma/*` imports now pin the dependency explicitly (`?deps=sigma@3.0.3`). A third bug (icon SVGs missing explicit `width`/`height`, which `@sigma/node-image`'s raster path needs to size a data-URI icon) was caught by an independent second-opinion review before it shipped. Verified end-to-end against the real 20,097-node/1083-community production graph that motivated this feature: 342 aggregated communities render correctly with icons, module colors, and a populated filter panel, zero console errors.
+  A further usability pass, driven by feedback after actually using the rendered graph: (1) the default forceAtlas2 parameters packed 87% of nodes within 15% of the bounding-box center on the same real production graph — genuinely unreadable, not just suboptimal — fixed with `linlog=True`, a `node_size` repulsion halo proportional to member count, and a much higher `scaling_ratio`; (2) edges were a flat gray regardless of relation — they're now colored AND labeled by their dominant relation bucket (`renderEdgeLabels: true`; sigma has no independent edge-label zoom threshold, but ties edge-label visibility to node-label visibility, which manages clutter automatically); (3) nodes are now draggable for manual layout tidying (sigma's own `downNode`/`moveBody`/`upNode`/`upStage` pattern, with the `setCustomBBox` freeze required to stop one dragged node from panning every other node), with a reset button that restores original positions; (4) the click panel now lists each community's actual representative source files, linked via `file://` when `.graphify_root` confirms the graph is being viewed on the machine that generated it; (5) `doc_ref` (a newer AST-derived file_type not in the original six-value schema) now gets an icon instead of silently miscategorizing as code; (6) the filter panel's entity-kind icons and relation-type rows now render visibly (white icons, not black-on-black; a color swatch per relation type matching its edge color).
+  The layout spread was tuned twice: the first pass (`gravity=0.5, scaling_ratio=20`) measured better on a numeric spread proxy but was still reported as too bunched to read labels without manually dragging nodes apart — a second pass (`gravity=0.15, scaling_ratio=80`) actually fixes it, verified by reading real labels at the default fit-all zoom on the 342-community production graph, not by trusting the proxy metric a second time. Node labels also gained their own small background box (sigma's default label renderer is bare `fillText` with no settings-level background option; `defaultDrawNodeLabel` is the documented override point for a fully custom renderer) since plain white text was illegible over a dense, colorful tangle of edges.
+  The click panel was reworked again based on direct feedback: the file list is no longer capped with a "+N more" — Step 1 now sends every distinct file, rendered as a scrollable, filterable list (a filter textbox above a scrollable div, not a flat dump) since a 300-member community can have dozens of files. Clicking a file now opens a movable dialog (dragged by its header, plain screen-pixel CSS `left`/`top`, unrelated to sigma's graph/camera coordinate spaces) showing an embedded content preview — Step 1 reads a bounded preview (3000 characters) from disk for the first 8 files of each community's list, since embedding full content for every file in every community would bloat the self-contained HTML far more than paths alone do. Embedding real file content surfaced a new instance of the script-tag-collision risk: a previewed HTML/JS file (e.g. a Vite `index.html` with `<script type="module" src="...">`) routinely contains a literal `</script`, which used to be a "check after the fact and hand-patch" step — Step 3 now escapes it unconditionally in the serialized JSON before embedding, since previews make the collision a routine occurrence rather than a rare edge case. The entity-properties panel that had been a separate floating box (getting clipped by the viewport independently of the main panel) is now nested inside the main sidebar, sharing its width and scroll.
+  A good spread still isn't *structure* — a well-separated layout with every edge at full opacity still reads as a hairball, since nothing visually ties related things together beyond a color dot. Three additions address that: (1) Step 1 boosts the layout-only edge weight (a separate `layout_weight` attribute, not the real `weight` that drives rendered thickness) for same-module edges before running forceAtlas2, so modules cluster spatially instead of scattering by topology alone — verified by the ratio of within-module to between-module node distance, 0.96 (no boost) down to 0.48 at the chosen factor, with higher factors not improving further; (2) a soft translucent hull is drawn behind each module's cluster, computed with a small pure-Python convex-hull implementation (not worth adding scipy for one shape per module) and rendered on a 2D canvas layer behind sigma's own WebGL canvas, redrawn on every `afterRender` event so it tracks pan/zoom; (3) edges below a weight percentile are hidden by default (a "show all edges" checkbox opts back in; anything touching the highlighted node always shows regardless of weight) — this surfaced a real gap where nothing ever called `applyReducers()` at startup, so the previous filter/highlight reducers had been silently inert until the first user interaction the whole time (harmless before, since their default state was a no-op; not harmless for a real default-hiding behavior).
+  The sidebar panel also had a genuine viewport-anchoring bug: `position: absolute` let it drift out of sync with the actual visible viewport whenever the page gained any scroll, which read as "the panel is truncated on my laptop" on a short screen — fixed to `position: fixed`, plus a visible custom scrollbar (macOS hides scrollbars until actively scrolling, which made an already-working scroll region look like a dead end) and `height` instead of `max-height` so the panel always spans the full viewport as a proper sidebar instead of shrinking to fit whatever content it currently holds.
+  Resizing the browser window *after* the page had already loaded (not just choosing a window size before opening it) left the graph and hulls pinned at their original size and position, with dead space filling the rest of the window — sigma's own internal resize listener updates the canvas/WebGL viewport but never re-fits the camera. The first fix attempt called `fitViewportToNodes` synchronously from a `window.resize` handler, which was worse than the bug it fixed: it read node display data mid-flight through sigma's own pending (one-frame-deferred) re-normalization pass and got back stale, un-normalized graph coordinates, pointing the camera far outside the graph and rendering a blank canvas. The fix (root-caused with sigma's own source, not by guessing) is to defer the re-fit one frame with `requestAnimationFrame` so it runs after sigma's own resize/re-normalization has actually finished; verified against the real 20,097-node/1083-community production graph by loading at one window size and resizing larger, confirming the graph fills the new viewport with no dead space and no rendering corruption.
 - Fix: `--update`-style section writes to `CLAUDE.md`/`AGENTS.md` no longer corrupt or drop content (#1688, thanks @bdfinst). `_replace_or_append_section` located its managed block by substring (`marker in content`) and `next(... if marker in line)`, so a heading that appeared as a substring of another line (or duplicate headings) matched the wrong offset and the rewrite could truncate the file. It now matches the section heading exactly (`line.strip() == marker`), appends when absent, and prefers the last exact match when several exist, so unrelated content is preserved.
 - Fix: token estimation no longer crashes on files containing tiktoken special-token text like `<|endoftext|>` (#1685, thanks @Kyzcreig). `_TOKENIZER.encode(content)` raises `ValueError` by default when the text contains a special token, which aborted packing on docs/corpora that merely mention these strings. Both `encode` sites now pass `disallowed_special=()` so such text is tokenized as ordinary bytes.
 - Fix: the Ollama backend no longer multiplies a hang by the retry count (#1686, thanks @Kyzcreig). A stalled local model would wedge for `timeout * (max_retries + 1)`, which with the default 6 retries turned one long stall into a very long one. Ollama now defaults to zero client-side retries (a local model that stalls will not un-stall on retry); set `GRAPHIFY_MAX_RETRIES` to opt back in. Other backends are unchanged. Note: the underlying stall is non-deterministic and driven by the model server, so this bounds the wait rather than eliminating the hang.

graphify/skill-agents.md

@@ -474,8 +474,24 @@ Substitute `IS_DIRECTED` and `INPUT_PATH` as in Step 4. If a `GRAPH HEALTH WARNI
 
 Read `graphify-out/.graphify_analysis.json`. For each community key, look at its node labels and write a 2-5 word plain-language name (e.g. "Attention Mechanism", "Training Pipeline", "Data Loading").
 
+**At scale (100+ communities), do NOT hand-label one at a time and do NOT fall back to a naive path-prefix heuristic (e.g. title-casing the common directory prefix).** Both break down past a handful of communities — the heuristic in particular produces duplicated-segment junk ("Backend Backend", "Frontend Frontend") and word-soup names that look plausible until you actually read them. Verified in production on a 1000+ community monorepo graph (apa, 2026-07-03): a heuristic pass needed a full redo.
+
+What works instead, for communities with >=20 nodes (the ones that actually surface in God Nodes / navigation):
+1. For each such community, sample its ~15 highest-degree (most connected) member nodes — not the full member list, communities can have 300+ nodes.
+2. Batch communities into groups of ~15-20 per subagent (write each batch's samples to a small JSON file under `graphify-out/`).
+3. Dispatch one `general-purpose` subagent per batch (all in the same message, in parallel — same pattern as Step 3 Part B), each told to read its batch file and write a JSON `{community_id: name}` map based on genuine reasoning about the sampled node labels/paths, not just repeating directory names.
+4. Merge all batch results.
+
+For the long tail of small communities (<20 nodes) where subagent cost isn't justified, a path-prefix heuristic is an acceptable lighter-weight fallback — but dedupe repeated words/segments, strip trailing file extensions, and spot-check a random sample of ~15 outputs across different sizes before trusting it. If duplicated-word or generic ("Community N") names appear in the sample, fix the heuristic before applying it to the rest.
+
 Then regenerate the report and save the labels for the visualizer:
 
+**Reconcile the FULL label set for cross-batch duplicates before regenerating the report — this is a required final pass, not optional polish.** Subagent batches only see their own slice and the small-community heuristic naturally collapses siblings from the same directory (e.g. many different test-file clusters under the same folder), so duplicates are expected even after the steps above — verified in production: 338 of 1083 communities (31%) shared a label with at least one other community before reconciliation (apa monorepo, 2026-07-03). After all labels are assembled:
+1. Group community IDs by their current label.
+2. For every group with >1 member, pick a distinguishing keyword from each member's OWN node labels (word-frequency count, excluding stopwords and words already in the shared base label) and append it, e.g. `"Agent Gateway MCP Client" -> "Agent Gateway MCP Client — Server"` / `"— Additional"`.
+3. If a collision still remains after that (rare — e.g. multiple communities extracted from the very same source image), append a numeric suffix `" (1)"`, `" (2)"` as a last resort.
+4. Re-check for zero duplicates before moving on.
+
 ```bash
 $(cat graphify-out/.graphify_python) -c "
 import sys, json
@@ -531,6 +547,8 @@ graphify export html  # auto-aggregates to community view if graph > 5000 nodes
 # or: graphify export html --no-viz
 ```
 
+**If the resulting view (aggregated community graph, or the raw graph for small corpora) has more than ~300 nodes, ALSO generate a Sigma.js + graphology version and present that as the primary interactive file instead of vis-network's `graph.html`.** vis-network runs a live single-threaded JS forceAtlas2 physics simulation on load (200 iterations before it disables itself) — verified in production this is genuinely slow/laggy past a few hundred nodes (apa monorepo, 1083-community aggregated view, 2026-07-03), independent of hardware. Sigma.js (WebGL rendering) with the layout **precomputed offline in Python** (`nx.forceatlas2_layout`, no client-side physics at all) fixes this at the root rather than swapping one JS physics engine for another. See `references/sigma-viz.md` for the full recipe (meta-graph construction, offline layout, self-contained HTML template loading sigma/graphology from esm.sh). Below the 300-node threshold, vis-network's `graph.html` is fine as-is — the extra build step isn't worth it.
+
 ### Steps 6b-8 - Wiki, Neo4j, FalkorDB, SVG, GraphML, MCP, benchmark (only on their flags)
 
 These run only when their flag is present (`--wiki`, `--neo4j`/`--neo4j-push`, `--falkordb`/`--falkordb-push`, `--svg`, `--graphml`, `--mcp`) or, for the token-reduction benchmark, when `total_words` exceeds 5,000. A default run with no export flags skips all of them. See `references/exports.md` for each one. Run any `--wiki` export before Step 9 cleanup so `.graphify_labels.json` is still available.