From 74a1457290482bab705729c6323a19e5e9d6b19b Mon Sep 17 00:00:00 2001 From: Fabian Wilde Date: Mon, 6 Jul 2026 13:23:27 +0200 Subject: [PATCH 1/9] feat(skill): Sigma.js viz fallback + batched community-label reconciliation vis-network's client-side forceAtlas2 physics simulation is verified slow/laggy past a few hundred nodes regardless of hardware (apa monorepo, 1083-community aggregated view, 2026-07-03). Step 6 now generates a separate WebGL Sigma.js render with the layout precomputed offline in Python once a view exceeds ~300 nodes, documented in a new shared reference (references/sigma-viz.md) shipped to every split-platform skill variant via tools/skillgen. Step 5's community labeling also gains a batched-subagent flow for 100+ communities (sample top-degree nodes per community, dispatch one subagent per ~15-20-community batch in parallel) plus a required cross-batch label-reconciliation pass, since batches only see their own slice and naturally produce duplicate labels across siblings (338 of 1083 communities, 31%, collided before reconciliation on the same run). Ported from local edits to the installed ~/.claude/skills/graphify/ skill made in a prior session against a production run, which were not yet committed to the repo. --- CHANGELOG.md | 2 + graphify/skill-agents.md | 18 ++ graphify/skill-amp.md | 18 ++ graphify/skill-claw.md | 18 ++ graphify/skill-codex.md | 18 ++ graphify/skill-copilot.md | 18 ++ graphify/skill-droid.md | 18 ++ graphify/skill-kilo.md | 18 ++ graphify/skill-kiro.md | 18 ++ graphify/skill-opencode.md | 18 ++ graphify/skill-pi.md | 18 ++ graphify/skill-trae.md | 18 ++ graphify/skill-vscode.md | 18 ++ graphify/skill-windows.md | 18 ++ graphify/skill.md | 18 ++ .../skills/agents/references/sigma-viz.md | 219 ++++++++++++++++++ graphify/skills/amp/references/sigma-viz.md | 219 ++++++++++++++++++ .../skills/claude/references/sigma-viz.md | 219 ++++++++++++++++++ graphify/skills/claw/references/sigma-viz.md | 219 ++++++++++++++++++ graphify/skills/codex/references/sigma-viz.md | 219 ++++++++++++++++++ .../skills/copilot/references/sigma-viz.md | 219 ++++++++++++++++++ graphify/skills/droid/references/sigma-viz.md | 219 ++++++++++++++++++ graphify/skills/kilo/references/sigma-viz.md | 219 ++++++++++++++++++ graphify/skills/kiro/references/sigma-viz.md | 219 ++++++++++++++++++ .../skills/opencode/references/sigma-viz.md | 219 ++++++++++++++++++ graphify/skills/pi/references/sigma-viz.md | 219 ++++++++++++++++++ graphify/skills/trae/references/sigma-viz.md | 219 ++++++++++++++++++ .../skills/vscode/references/sigma-viz.md | 219 ++++++++++++++++++ .../skills/windows/references/sigma-viz.md | 219 ++++++++++++++++++ tests/test_install_references.py | 9 +- tests/test_skillgen.py | 10 +- .../expected/graphify__skill-agents.md | 18 ++ .../skillgen/expected/graphify__skill-amp.md | 18 ++ .../skillgen/expected/graphify__skill-claw.md | 18 ++ .../expected/graphify__skill-codex.md | 18 ++ .../expected/graphify__skill-copilot.md | 18 ++ .../expected/graphify__skill-droid.md | 18 ++ .../skillgen/expected/graphify__skill-kilo.md | 18 ++ .../skillgen/expected/graphify__skill-kiro.md | 18 ++ .../expected/graphify__skill-opencode.md | 18 ++ tools/skillgen/expected/graphify__skill-pi.md | 18 ++ .../skillgen/expected/graphify__skill-trae.md | 18 ++ .../expected/graphify__skill-vscode.md | 18 ++ .../expected/graphify__skill-windows.md | 18 ++ tools/skillgen/expected/graphify__skill.md | 18 ++ ...__skills__agents__references__sigma-viz.md | 219 ++++++++++++++++++ ...ify__skills__amp__references__sigma-viz.md | 219 ++++++++++++++++++ ...__skills__claude__references__sigma-viz.md | 219 ++++++++++++++++++ ...fy__skills__claw__references__sigma-viz.md | 219 ++++++++++++++++++ ...y__skills__codex__references__sigma-viz.md | 219 ++++++++++++++++++ ..._skills__copilot__references__sigma-viz.md | 219 ++++++++++++++++++ ...y__skills__droid__references__sigma-viz.md | 219 ++++++++++++++++++ ...fy__skills__kilo__references__sigma-viz.md | 219 ++++++++++++++++++ ...fy__skills__kiro__references__sigma-viz.md | 219 ++++++++++++++++++ ...skills__opencode__references__sigma-viz.md | 219 ++++++++++++++++++ ...hify__skills__pi__references__sigma-viz.md | 219 ++++++++++++++++++ ...fy__skills__trae__references__sigma-viz.md | 219 ++++++++++++++++++ ...__skills__vscode__references__sigma-viz.md | 219 ++++++++++++++++++ ..._skills__windows__references__sigma-viz.md | 219 ++++++++++++++++++ tools/skillgen/fragments/core/core.md | 18 ++ .../fragments/references/shared/sigma-viz.md | 219 ++++++++++++++++++ tools/skillgen/gen.py | 3 +- 62 files changed, 6888 insertions(+), 9 deletions(-) create mode 100644 graphify/skills/agents/references/sigma-viz.md create mode 100644 graphify/skills/amp/references/sigma-viz.md create mode 100644 graphify/skills/claude/references/sigma-viz.md create mode 100644 graphify/skills/claw/references/sigma-viz.md create mode 100644 graphify/skills/codex/references/sigma-viz.md create mode 100644 graphify/skills/copilot/references/sigma-viz.md create mode 100644 graphify/skills/droid/references/sigma-viz.md create mode 100644 graphify/skills/kilo/references/sigma-viz.md create mode 100644 graphify/skills/kiro/references/sigma-viz.md create mode 100644 graphify/skills/opencode/references/sigma-viz.md create mode 100644 graphify/skills/pi/references/sigma-viz.md create mode 100644 graphify/skills/trae/references/sigma-viz.md create mode 100644 graphify/skills/vscode/references/sigma-viz.md create mode 100644 graphify/skills/windows/references/sigma-viz.md create mode 100644 tools/skillgen/expected/graphify__skills__agents__references__sigma-viz.md create mode 100644 tools/skillgen/expected/graphify__skills__amp__references__sigma-viz.md create mode 100644 tools/skillgen/expected/graphify__skills__claude__references__sigma-viz.md create mode 100644 tools/skillgen/expected/graphify__skills__claw__references__sigma-viz.md create mode 100644 tools/skillgen/expected/graphify__skills__codex__references__sigma-viz.md create mode 100644 tools/skillgen/expected/graphify__skills__copilot__references__sigma-viz.md create mode 100644 tools/skillgen/expected/graphify__skills__droid__references__sigma-viz.md create mode 100644 tools/skillgen/expected/graphify__skills__kilo__references__sigma-viz.md create mode 100644 tools/skillgen/expected/graphify__skills__kiro__references__sigma-viz.md create mode 100644 tools/skillgen/expected/graphify__skills__opencode__references__sigma-viz.md create mode 100644 tools/skillgen/expected/graphify__skills__pi__references__sigma-viz.md create mode 100644 tools/skillgen/expected/graphify__skills__trae__references__sigma-viz.md create mode 100644 tools/skillgen/expected/graphify__skills__vscode__references__sigma-viz.md create mode 100644 tools/skillgen/expected/graphify__skills__windows__references__sigma-viz.md create mode 100644 tools/skillgen/fragments/references/shared/sigma-viz.md diff --git a/CHANGELOG.md b/CHANGELOG.md index f01b4fe4c..60833c012 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,8 @@ Full release notes with details on each version: [GitHub Releases](https://githu ## Unreleased +- Feat: Sigma.js + graphology fallback for large `graph.html` views, plus batched/reconciled community labeling at scale. vis-network's live client-side forceAtlas2 physics simulation is genuinely slow/laggy past a few hundred nodes regardless of hardware — verified in production on a 1083-community monorepo graph (apa, 2026-07-03). Step 6 of the skill now generates a separate WebGL Sigma.js render (`graph_sigma.html`) with the layout precomputed offline in Python (`nx.forceatlas2_layout`, no client-side physics) once the view exceeds ~300 nodes; the full recipe lives in the new `references/sigma-viz.md`, shipped to every split-platform skill variant. Step 5's community labeling also gains a batched-subagent flow for 100+ communities (sample each community's top-degree nodes, dispatch one `general-purpose` subagent per ~15-20-community batch in parallel) plus a required cross-batch label-reconciliation pass — a naive single-pass label run on the same graph left 338 of 1083 communities (31%) sharing a label with another before reconciliation. + - Fix: the `query` reference doc's inline vocab/fallback snippets now read and write files with `encoding="utf-8"` (#1619 A2, thanks @edtrackai). On Windows (default cp1252) the bare `read_text()`/`write_text()` calls crashed on exactly the cross-language corpora the doc demonstrates (e.g. Cyrillic labels like `обработчик`). Fixed across all generated skill variants. - Fix: `graphify update`/`watch` no longer leaves stale sources after a deletion or a destination-only rename (#1623 / #1622, thanks @oleksii-tumanov). When the last supported file was deleted, or a rename reported only its destination in `changed_paths`, the removed source's nodes lingered in `graph.json`. The rebuild now reconciles extractor-backed sources against the files still present (code and document sources, subdirectory roots, legacy markers, symlinks, hyperedges) while preserving semantic and out-of-scope records. diff --git a/graphify/skill-agents.md b/graphify/skill-agents.md index 2e19931fa..0d02e4a31 100644 --- a/graphify/skill-agents.md +++ b/graphify/skill-agents.md @@ -474,8 +474,24 @@ Substitute `IS_DIRECTED` and `INPUT_PATH` as in Step 4. If a `GRAPH HEALTH WARNI Read `graphify-out/.graphify_analysis.json`. For each community key, look at its node labels and write a 2-5 word plain-language name (e.g. "Attention Mechanism", "Training Pipeline", "Data Loading"). +**At scale (100+ communities), do NOT hand-label one at a time and do NOT fall back to a naive path-prefix heuristic (e.g. title-casing the common directory prefix).** Both break down past a handful of communities — the heuristic in particular produces duplicated-segment junk ("Backend Backend", "Frontend Frontend") and word-soup names that look plausible until you actually read them. Verified in production on a 1000+ community monorepo graph (apa, 2026-07-03): a heuristic pass needed a full redo. + +What works instead, for communities with >=20 nodes (the ones that actually surface in God Nodes / navigation): +1. For each such community, sample its ~15 highest-degree (most connected) member nodes — not the full member list, communities can have 300+ nodes. +2. Batch communities into groups of ~15-20 per subagent (write each batch's samples to a small JSON file under `graphify-out/`). +3. Dispatch one `general-purpose` subagent per batch (all in the same message, in parallel — same pattern as Step 3 Part B), each told to read its batch file and write a JSON `{community_id: name}` map based on genuine reasoning about the sampled node labels/paths, not just repeating directory names. +4. Merge all batch results. + +For the long tail of small communities (<20 nodes) where subagent cost isn't justified, a path-prefix heuristic is an acceptable lighter-weight fallback — but dedupe repeated words/segments, strip trailing file extensions, and spot-check a random sample of ~15 outputs across different sizes before trusting it. If duplicated-word or generic ("Community N") names appear in the sample, fix the heuristic before applying it to the rest. + Then regenerate the report and save the labels for the visualizer: +**Reconcile the FULL label set for cross-batch duplicates before regenerating the report — this is a required final pass, not optional polish.** Subagent batches only see their own slice and the small-community heuristic naturally collapses siblings from the same directory (e.g. many different test-file clusters under the same folder), so duplicates are expected even after the steps above — verified in production: 338 of 1083 communities (31%) shared a label with at least one other community before reconciliation (apa monorepo, 2026-07-03). After all labels are assembled: +1. Group community IDs by their current label. +2. For every group with >1 member, pick a distinguishing keyword from each member's OWN node labels (word-frequency count, excluding stopwords and words already in the shared base label) and append it, e.g. `"Agent Gateway MCP Client" -> "Agent Gateway MCP Client — Server"` / `"— Additional"`. +3. If a collision still remains after that (rare — e.g. multiple communities extracted from the very same source image), append a numeric suffix `" (1)"`, `" (2)"` as a last resort. +4. Re-check for zero duplicates before moving on. + ```bash $(cat graphify-out/.graphify_python) -c " import sys, json @@ -531,6 +547,8 @@ graphify export html # auto-aggregates to community view if graph > 5000 nodes # or: graphify export html --no-viz ``` +**If the resulting view (aggregated community graph, or the raw graph for small corpora) has more than ~300 nodes, ALSO generate a Sigma.js + graphology version and present that as the primary interactive file instead of vis-network's `graph.html`.** vis-network runs a live single-threaded JS forceAtlas2 physics simulation on load (200 iterations before it disables itself) — verified in production this is genuinely slow/laggy past a few hundred nodes (apa monorepo, 1083-community aggregated view, 2026-07-03), independent of hardware. Sigma.js (WebGL rendering) with the layout **precomputed offline in Python** (`nx.forceatlas2_layout`, no client-side physics at all) fixes this at the root rather than swapping one JS physics engine for another. See `references/sigma-viz.md` for the full recipe (meta-graph construction, offline layout, self-contained HTML template loading sigma/graphology from esm.sh). Below the 300-node threshold, vis-network's `graph.html` is fine as-is — the extra build step isn't worth it. + ### Steps 6b-8 - Wiki, Neo4j, FalkorDB, SVG, GraphML, MCP, benchmark (only on their flags) These run only when their flag is present (`--wiki`, `--neo4j`/`--neo4j-push`, `--falkordb`/`--falkordb-push`, `--svg`, `--graphml`, `--mcp`) or, for the token-reduction benchmark, when `total_words` exceeds 5,000. A default run with no export flags skips all of them. See `references/exports.md` for each one. Run any `--wiki` export before Step 9 cleanup so `.graphify_labels.json` is still available. diff --git a/graphify/skill-amp.md b/graphify/skill-amp.md index 2e19931fa..0d02e4a31 100644 --- a/graphify/skill-amp.md +++ b/graphify/skill-amp.md @@ -474,8 +474,24 @@ Substitute `IS_DIRECTED` and `INPUT_PATH` as in Step 4. If a `GRAPH HEALTH WARNI Read `graphify-out/.graphify_analysis.json`. For each community key, look at its node labels and write a 2-5 word plain-language name (e.g. "Attention Mechanism", "Training Pipeline", "Data Loading"). +**At scale (100+ communities), do NOT hand-label one at a time and do NOT fall back to a naive path-prefix heuristic (e.g. title-casing the common directory prefix).** Both break down past a handful of communities — the heuristic in particular produces duplicated-segment junk ("Backend Backend", "Frontend Frontend") and word-soup names that look plausible until you actually read them. Verified in production on a 1000+ community monorepo graph (apa, 2026-07-03): a heuristic pass needed a full redo. + +What works instead, for communities with >=20 nodes (the ones that actually surface in God Nodes / navigation): +1. For each such community, sample its ~15 highest-degree (most connected) member nodes — not the full member list, communities can have 300+ nodes. +2. Batch communities into groups of ~15-20 per subagent (write each batch's samples to a small JSON file under `graphify-out/`). +3. Dispatch one `general-purpose` subagent per batch (all in the same message, in parallel — same pattern as Step 3 Part B), each told to read its batch file and write a JSON `{community_id: name}` map based on genuine reasoning about the sampled node labels/paths, not just repeating directory names. +4. Merge all batch results. + +For the long tail of small communities (<20 nodes) where subagent cost isn't justified, a path-prefix heuristic is an acceptable lighter-weight fallback — but dedupe repeated words/segments, strip trailing file extensions, and spot-check a random sample of ~15 outputs across different sizes before trusting it. If duplicated-word or generic ("Community N") names appear in the sample, fix the heuristic before applying it to the rest. + Then regenerate the report and save the labels for the visualizer: +**Reconcile the FULL label set for cross-batch duplicates before regenerating the report — this is a required final pass, not optional polish.** Subagent batches only see their own slice and the small-community heuristic naturally collapses siblings from the same directory (e.g. many different test-file clusters under the same folder), so duplicates are expected even after the steps above — verified in production: 338 of 1083 communities (31%) shared a label with at least one other community before reconciliation (apa monorepo, 2026-07-03). After all labels are assembled: +1. Group community IDs by their current label. +2. For every group with >1 member, pick a distinguishing keyword from each member's OWN node labels (word-frequency count, excluding stopwords and words already in the shared base label) and append it, e.g. `"Agent Gateway MCP Client" -> "Agent Gateway MCP Client — Server"` / `"— Additional"`. +3. If a collision still remains after that (rare — e.g. multiple communities extracted from the very same source image), append a numeric suffix `" (1)"`, `" (2)"` as a last resort. +4. Re-check for zero duplicates before moving on. + ```bash $(cat graphify-out/.graphify_python) -c " import sys, json @@ -531,6 +547,8 @@ graphify export html # auto-aggregates to community view if graph > 5000 nodes # or: graphify export html --no-viz ``` +**If the resulting view (aggregated community graph, or the raw graph for small corpora) has more than ~300 nodes, ALSO generate a Sigma.js + graphology version and present that as the primary interactive file instead of vis-network's `graph.html`.** vis-network runs a live single-threaded JS forceAtlas2 physics simulation on load (200 iterations before it disables itself) — verified in production this is genuinely slow/laggy past a few hundred nodes (apa monorepo, 1083-community aggregated view, 2026-07-03), independent of hardware. Sigma.js (WebGL rendering) with the layout **precomputed offline in Python** (`nx.forceatlas2_layout`, no client-side physics at all) fixes this at the root rather than swapping one JS physics engine for another. See `references/sigma-viz.md` for the full recipe (meta-graph construction, offline layout, self-contained HTML template loading sigma/graphology from esm.sh). Below the 300-node threshold, vis-network's `graph.html` is fine as-is — the extra build step isn't worth it. + ### Steps 6b-8 - Wiki, Neo4j, FalkorDB, SVG, GraphML, MCP, benchmark (only on their flags) These run only when their flag is present (`--wiki`, `--neo4j`/`--neo4j-push`, `--falkordb`/`--falkordb-push`, `--svg`, `--graphml`, `--mcp`) or, for the token-reduction benchmark, when `total_words` exceeds 5,000. A default run with no export flags skips all of them. See `references/exports.md` for each one. Run any `--wiki` export before Step 9 cleanup so `.graphify_labels.json` is still available. diff --git a/graphify/skill-claw.md b/graphify/skill-claw.md index b3542438d..5c337677e 100644 --- a/graphify/skill-claw.md +++ b/graphify/skill-claw.md @@ -477,8 +477,24 @@ Substitute `IS_DIRECTED` and `INPUT_PATH` as in Step 4. If a `GRAPH HEALTH WARNI Read `graphify-out/.graphify_analysis.json`. For each community key, look at its node labels and write a 2-5 word plain-language name (e.g. "Attention Mechanism", "Training Pipeline", "Data Loading"). +**At scale (100+ communities), do NOT hand-label one at a time and do NOT fall back to a naive path-prefix heuristic (e.g. title-casing the common directory prefix).** Both break down past a handful of communities — the heuristic in particular produces duplicated-segment junk ("Backend Backend", "Frontend Frontend") and word-soup names that look plausible until you actually read them. Verified in production on a 1000+ community monorepo graph (apa, 2026-07-03): a heuristic pass needed a full redo. + +What works instead, for communities with >=20 nodes (the ones that actually surface in God Nodes / navigation): +1. For each such community, sample its ~15 highest-degree (most connected) member nodes — not the full member list, communities can have 300+ nodes. +2. Batch communities into groups of ~15-20 per subagent (write each batch's samples to a small JSON file under `graphify-out/`). +3. Dispatch one `general-purpose` subagent per batch (all in the same message, in parallel — same pattern as Step 3 Part B), each told to read its batch file and write a JSON `{community_id: name}` map based on genuine reasoning about the sampled node labels/paths, not just repeating directory names. +4. Merge all batch results. + +For the long tail of small communities (<20 nodes) where subagent cost isn't justified, a path-prefix heuristic is an acceptable lighter-weight fallback — but dedupe repeated words/segments, strip trailing file extensions, and spot-check a random sample of ~15 outputs across different sizes before trusting it. If duplicated-word or generic ("Community N") names appear in the sample, fix the heuristic before applying it to the rest. + Then regenerate the report and save the labels for the visualizer: +**Reconcile the FULL label set for cross-batch duplicates before regenerating the report — this is a required final pass, not optional polish.** Subagent batches only see their own slice and the small-community heuristic naturally collapses siblings from the same directory (e.g. many different test-file clusters under the same folder), so duplicates are expected even after the steps above — verified in production: 338 of 1083 communities (31%) shared a label with at least one other community before reconciliation (apa monorepo, 2026-07-03). After all labels are assembled: +1. Group community IDs by their current label. +2. For every group with >1 member, pick a distinguishing keyword from each member's OWN node labels (word-frequency count, excluding stopwords and words already in the shared base label) and append it, e.g. `"Agent Gateway MCP Client" -> "Agent Gateway MCP Client — Server"` / `"— Additional"`. +3. If a collision still remains after that (rare — e.g. multiple communities extracted from the very same source image), append a numeric suffix `" (1)"`, `" (2)"` as a last resort. +4. Re-check for zero duplicates before moving on. + ```bash $(cat graphify-out/.graphify_python) -c " import sys, json @@ -534,6 +550,8 @@ graphify export html # auto-aggregates to community view if graph > 5000 nodes # or: graphify export html --no-viz ``` +**If the resulting view (aggregated community graph, or the raw graph for small corpora) has more than ~300 nodes, ALSO generate a Sigma.js + graphology version and present that as the primary interactive file instead of vis-network's `graph.html`.** vis-network runs a live single-threaded JS forceAtlas2 physics simulation on load (200 iterations before it disables itself) — verified in production this is genuinely slow/laggy past a few hundred nodes (apa monorepo, 1083-community aggregated view, 2026-07-03), independent of hardware. Sigma.js (WebGL rendering) with the layout **precomputed offline in Python** (`nx.forceatlas2_layout`, no client-side physics at all) fixes this at the root rather than swapping one JS physics engine for another. See `references/sigma-viz.md` for the full recipe (meta-graph construction, offline layout, self-contained HTML template loading sigma/graphology from esm.sh). Below the 300-node threshold, vis-network's `graph.html` is fine as-is — the extra build step isn't worth it. + ### Steps 6b-8 - Wiki, Neo4j, FalkorDB, SVG, GraphML, MCP, benchmark (only on their flags) These run only when their flag is present (`--wiki`, `--neo4j`/`--neo4j-push`, `--falkordb`/`--falkordb-push`, `--svg`, `--graphml`, `--mcp`) or, for the token-reduction benchmark, when `total_words` exceeds 5,000. A default run with no export flags skips all of them. See `references/exports.md` for each one. Run any `--wiki` export before Step 9 cleanup so `.graphify_labels.json` is still available. diff --git a/graphify/skill-codex.md b/graphify/skill-codex.md index e6e411ea0..f28ef4cd3 100644 --- a/graphify/skill-codex.md +++ b/graphify/skill-codex.md @@ -474,8 +474,24 @@ Substitute `IS_DIRECTED` and `INPUT_PATH` as in Step 4. If a `GRAPH HEALTH WARNI Read `graphify-out/.graphify_analysis.json`. For each community key, look at its node labels and write a 2-5 word plain-language name (e.g. "Attention Mechanism", "Training Pipeline", "Data Loading"). +**At scale (100+ communities), do NOT hand-label one at a time and do NOT fall back to a naive path-prefix heuristic (e.g. title-casing the common directory prefix).** Both break down past a handful of communities — the heuristic in particular produces duplicated-segment junk ("Backend Backend", "Frontend Frontend") and word-soup names that look plausible until you actually read them. Verified in production on a 1000+ community monorepo graph (apa, 2026-07-03): a heuristic pass needed a full redo. + +What works instead, for communities with >=20 nodes (the ones that actually surface in God Nodes / navigation): +1. For each such community, sample its ~15 highest-degree (most connected) member nodes — not the full member list, communities can have 300+ nodes. +2. Batch communities into groups of ~15-20 per subagent (write each batch's samples to a small JSON file under `graphify-out/`). +3. Dispatch one `general-purpose` subagent per batch (all in the same message, in parallel — same pattern as Step 3 Part B), each told to read its batch file and write a JSON `{community_id: name}` map based on genuine reasoning about the sampled node labels/paths, not just repeating directory names. +4. Merge all batch results. + +For the long tail of small communities (<20 nodes) where subagent cost isn't justified, a path-prefix heuristic is an acceptable lighter-weight fallback — but dedupe repeated words/segments, strip trailing file extensions, and spot-check a random sample of ~15 outputs across different sizes before trusting it. If duplicated-word or generic ("Community N") names appear in the sample, fix the heuristic before applying it to the rest. + Then regenerate the report and save the labels for the visualizer: +**Reconcile the FULL label set for cross-batch duplicates before regenerating the report — this is a required final pass, not optional polish.** Subagent batches only see their own slice and the small-community heuristic naturally collapses siblings from the same directory (e.g. many different test-file clusters under the same folder), so duplicates are expected even after the steps above — verified in production: 338 of 1083 communities (31%) shared a label with at least one other community before reconciliation (apa monorepo, 2026-07-03). After all labels are assembled: +1. Group community IDs by their current label. +2. For every group with >1 member, pick a distinguishing keyword from each member's OWN node labels (word-frequency count, excluding stopwords and words already in the shared base label) and append it, e.g. `"Agent Gateway MCP Client" -> "Agent Gateway MCP Client — Server"` / `"— Additional"`. +3. If a collision still remains after that (rare — e.g. multiple communities extracted from the very same source image), append a numeric suffix `" (1)"`, `" (2)"` as a last resort. +4. Re-check for zero duplicates before moving on. + ```bash $(cat graphify-out/.graphify_python) -c " import sys, json @@ -531,6 +547,8 @@ graphify export html # auto-aggregates to community view if graph > 5000 nodes # or: graphify export html --no-viz ``` +**If the resulting view (aggregated community graph, or the raw graph for small corpora) has more than ~300 nodes, ALSO generate a Sigma.js + graphology version and present that as the primary interactive file instead of vis-network's `graph.html`.** vis-network runs a live single-threaded JS forceAtlas2 physics simulation on load (200 iterations before it disables itself) — verified in production this is genuinely slow/laggy past a few hundred nodes (apa monorepo, 1083-community aggregated view, 2026-07-03), independent of hardware. Sigma.js (WebGL rendering) with the layout **precomputed offline in Python** (`nx.forceatlas2_layout`, no client-side physics at all) fixes this at the root rather than swapping one JS physics engine for another. See `references/sigma-viz.md` for the full recipe (meta-graph construction, offline layout, self-contained HTML template loading sigma/graphology from esm.sh). Below the 300-node threshold, vis-network's `graph.html` is fine as-is — the extra build step isn't worth it. + ### Steps 6b-8 - Wiki, Neo4j, FalkorDB, SVG, GraphML, MCP, benchmark (only on their flags) These run only when their flag is present (`--wiki`, `--neo4j`/`--neo4j-push`, `--falkordb`/`--falkordb-push`, `--svg`, `--graphml`, `--mcp`) or, for the token-reduction benchmark, when `total_words` exceeds 5,000. A default run with no export flags skips all of them. See `references/exports.md` for each one. Run any `--wiki` export before Step 9 cleanup so `.graphify_labels.json` is still available. diff --git a/graphify/skill-copilot.md b/graphify/skill-copilot.md index b3542438d..5c337677e 100644 --- a/graphify/skill-copilot.md +++ b/graphify/skill-copilot.md @@ -477,8 +477,24 @@ Substitute `IS_DIRECTED` and `INPUT_PATH` as in Step 4. If a `GRAPH HEALTH WARNI Read `graphify-out/.graphify_analysis.json`. For each community key, look at its node labels and write a 2-5 word plain-language name (e.g. "Attention Mechanism", "Training Pipeline", "Data Loading"). +**At scale (100+ communities), do NOT hand-label one at a time and do NOT fall back to a naive path-prefix heuristic (e.g. title-casing the common directory prefix).** Both break down past a handful of communities — the heuristic in particular produces duplicated-segment junk ("Backend Backend", "Frontend Frontend") and word-soup names that look plausible until you actually read them. Verified in production on a 1000+ community monorepo graph (apa, 2026-07-03): a heuristic pass needed a full redo. + +What works instead, for communities with >=20 nodes (the ones that actually surface in God Nodes / navigation): +1. For each such community, sample its ~15 highest-degree (most connected) member nodes — not the full member list, communities can have 300+ nodes. +2. Batch communities into groups of ~15-20 per subagent (write each batch's samples to a small JSON file under `graphify-out/`). +3. Dispatch one `general-purpose` subagent per batch (all in the same message, in parallel — same pattern as Step 3 Part B), each told to read its batch file and write a JSON `{community_id: name}` map based on genuine reasoning about the sampled node labels/paths, not just repeating directory names. +4. Merge all batch results. + +For the long tail of small communities (<20 nodes) where subagent cost isn't justified, a path-prefix heuristic is an acceptable lighter-weight fallback — but dedupe repeated words/segments, strip trailing file extensions, and spot-check a random sample of ~15 outputs across different sizes before trusting it. If duplicated-word or generic ("Community N") names appear in the sample, fix the heuristic before applying it to the rest. + Then regenerate the report and save the labels for the visualizer: +**Reconcile the FULL label set for cross-batch duplicates before regenerating the report — this is a required final pass, not optional polish.** Subagent batches only see their own slice and the small-community heuristic naturally collapses siblings from the same directory (e.g. many different test-file clusters under the same folder), so duplicates are expected even after the steps above — verified in production: 338 of 1083 communities (31%) shared a label with at least one other community before reconciliation (apa monorepo, 2026-07-03). After all labels are assembled: +1. Group community IDs by their current label. +2. For every group with >1 member, pick a distinguishing keyword from each member's OWN node labels (word-frequency count, excluding stopwords and words already in the shared base label) and append it, e.g. `"Agent Gateway MCP Client" -> "Agent Gateway MCP Client — Server"` / `"— Additional"`. +3. If a collision still remains after that (rare — e.g. multiple communities extracted from the very same source image), append a numeric suffix `" (1)"`, `" (2)"` as a last resort. +4. Re-check for zero duplicates before moving on. + ```bash $(cat graphify-out/.graphify_python) -c " import sys, json @@ -534,6 +550,8 @@ graphify export html # auto-aggregates to community view if graph > 5000 nodes # or: graphify export html --no-viz ``` +**If the resulting view (aggregated community graph, or the raw graph for small corpora) has more than ~300 nodes, ALSO generate a Sigma.js + graphology version and present that as the primary interactive file instead of vis-network's `graph.html`.** vis-network runs a live single-threaded JS forceAtlas2 physics simulation on load (200 iterations before it disables itself) — verified in production this is genuinely slow/laggy past a few hundred nodes (apa monorepo, 1083-community aggregated view, 2026-07-03), independent of hardware. Sigma.js (WebGL rendering) with the layout **precomputed offline in Python** (`nx.forceatlas2_layout`, no client-side physics at all) fixes this at the root rather than swapping one JS physics engine for another. See `references/sigma-viz.md` for the full recipe (meta-graph construction, offline layout, self-contained HTML template loading sigma/graphology from esm.sh). Below the 300-node threshold, vis-network's `graph.html` is fine as-is — the extra build step isn't worth it. + ### Steps 6b-8 - Wiki, Neo4j, FalkorDB, SVG, GraphML, MCP, benchmark (only on their flags) These run only when their flag is present (`--wiki`, `--neo4j`/`--neo4j-push`, `--falkordb`/`--falkordb-push`, `--svg`, `--graphml`, `--mcp`) or, for the token-reduction benchmark, when `total_words` exceeds 5,000. A default run with no export flags skips all of them. See `references/exports.md` for each one. Run any `--wiki` export before Step 9 cleanup so `.graphify_labels.json` is still available. diff --git a/graphify/skill-droid.md b/graphify/skill-droid.md index d1147b903..afa602ccc 100644 --- a/graphify/skill-droid.md +++ b/graphify/skill-droid.md @@ -474,8 +474,24 @@ Substitute `IS_DIRECTED` and `INPUT_PATH` as in Step 4. If a `GRAPH HEALTH WARNI Read `graphify-out/.graphify_analysis.json`. For each community key, look at its node labels and write a 2-5 word plain-language name (e.g. "Attention Mechanism", "Training Pipeline", "Data Loading"). +**At scale (100+ communities), do NOT hand-label one at a time and do NOT fall back to a naive path-prefix heuristic (e.g. title-casing the common directory prefix).** Both break down past a handful of communities — the heuristic in particular produces duplicated-segment junk ("Backend Backend", "Frontend Frontend") and word-soup names that look plausible until you actually read them. Verified in production on a 1000+ community monorepo graph (apa, 2026-07-03): a heuristic pass needed a full redo. + +What works instead, for communities with >=20 nodes (the ones that actually surface in God Nodes / navigation): +1. For each such community, sample its ~15 highest-degree (most connected) member nodes — not the full member list, communities can have 300+ nodes. +2. Batch communities into groups of ~15-20 per subagent (write each batch's samples to a small JSON file under `graphify-out/`). +3. Dispatch one `general-purpose` subagent per batch (all in the same message, in parallel — same pattern as Step 3 Part B), each told to read its batch file and write a JSON `{community_id: name}` map based on genuine reasoning about the sampled node labels/paths, not just repeating directory names. +4. Merge all batch results. + +For the long tail of small communities (<20 nodes) where subagent cost isn't justified, a path-prefix heuristic is an acceptable lighter-weight fallback — but dedupe repeated words/segments, strip trailing file extensions, and spot-check a random sample of ~15 outputs across different sizes before trusting it. If duplicated-word or generic ("Community N") names appear in the sample, fix the heuristic before applying it to the rest. + Then regenerate the report and save the labels for the visualizer: +**Reconcile the FULL label set for cross-batch duplicates before regenerating the report — this is a required final pass, not optional polish.** Subagent batches only see their own slice and the small-community heuristic naturally collapses siblings from the same directory (e.g. many different test-file clusters under the same folder), so duplicates are expected even after the steps above — verified in production: 338 of 1083 communities (31%) shared a label with at least one other community before reconciliation (apa monorepo, 2026-07-03). After all labels are assembled: +1. Group community IDs by their current label. +2. For every group with >1 member, pick a distinguishing keyword from each member's OWN node labels (word-frequency count, excluding stopwords and words already in the shared base label) and append it, e.g. `"Agent Gateway MCP Client" -> "Agent Gateway MCP Client — Server"` / `"— Additional"`. +3. If a collision still remains after that (rare — e.g. multiple communities extracted from the very same source image), append a numeric suffix `" (1)"`, `" (2)"` as a last resort. +4. Re-check for zero duplicates before moving on. + ```bash $(cat graphify-out/.graphify_python) -c " import sys, json @@ -531,6 +547,8 @@ graphify export html # auto-aggregates to community view if graph > 5000 nodes # or: graphify export html --no-viz ``` +**If the resulting view (aggregated community graph, or the raw graph for small corpora) has more than ~300 nodes, ALSO generate a Sigma.js + graphology version and present that as the primary interactive file instead of vis-network's `graph.html`.** vis-network runs a live single-threaded JS forceAtlas2 physics simulation on load (200 iterations before it disables itself) — verified in production this is genuinely slow/laggy past a few hundred nodes (apa monorepo, 1083-community aggregated view, 2026-07-03), independent of hardware. Sigma.js (WebGL rendering) with the layout **precomputed offline in Python** (`nx.forceatlas2_layout`, no client-side physics at all) fixes this at the root rather than swapping one JS physics engine for another. See `references/sigma-viz.md` for the full recipe (meta-graph construction, offline layout, self-contained HTML template loading sigma/graphology from esm.sh). Below the 300-node threshold, vis-network's `graph.html` is fine as-is — the extra build step isn't worth it. + ### Steps 6b-8 - Wiki, Neo4j, FalkorDB, SVG, GraphML, MCP, benchmark (only on their flags) These run only when their flag is present (`--wiki`, `--neo4j`/`--neo4j-push`, `--falkordb`/`--falkordb-push`, `--svg`, `--graphml`, `--mcp`) or, for the token-reduction benchmark, when `total_words` exceeds 5,000. A default run with no export flags skips all of them. See `references/exports.md` for each one. Run any `--wiki` export before Step 9 cleanup so `.graphify_labels.json` is still available. diff --git a/graphify/skill-kilo.md b/graphify/skill-kilo.md index c45578165..b4892b39c 100644 --- a/graphify/skill-kilo.md +++ b/graphify/skill-kilo.md @@ -477,8 +477,24 @@ Substitute `IS_DIRECTED` and `INPUT_PATH` as in Step 4. If a `GRAPH HEALTH WARNI Read `graphify-out/.graphify_analysis.json`. For each community key, look at its node labels and write a 2-5 word plain-language name (e.g. "Attention Mechanism", "Training Pipeline", "Data Loading"). +**At scale (100+ communities), do NOT hand-label one at a time and do NOT fall back to a naive path-prefix heuristic (e.g. title-casing the common directory prefix).** Both break down past a handful of communities — the heuristic in particular produces duplicated-segment junk ("Backend Backend", "Frontend Frontend") and word-soup names that look plausible until you actually read them. Verified in production on a 1000+ community monorepo graph (apa, 2026-07-03): a heuristic pass needed a full redo. + +What works instead, for communities with >=20 nodes (the ones that actually surface in God Nodes / navigation): +1. For each such community, sample its ~15 highest-degree (most connected) member nodes — not the full member list, communities can have 300+ nodes. +2. Batch communities into groups of ~15-20 per subagent (write each batch's samples to a small JSON file under `graphify-out/`). +3. Dispatch one `general-purpose` subagent per batch (all in the same message, in parallel — same pattern as Step 3 Part B), each told to read its batch file and write a JSON `{community_id: name}` map based on genuine reasoning about the sampled node labels/paths, not just repeating directory names. +4. Merge all batch results. + +For the long tail of small communities (<20 nodes) where subagent cost isn't justified, a path-prefix heuristic is an acceptable lighter-weight fallback — but dedupe repeated words/segments, strip trailing file extensions, and spot-check a random sample of ~15 outputs across different sizes before trusting it. If duplicated-word or generic ("Community N") names appear in the sample, fix the heuristic before applying it to the rest. + Then regenerate the report and save the labels for the visualizer: +**Reconcile the FULL label set for cross-batch duplicates before regenerating the report — this is a required final pass, not optional polish.** Subagent batches only see their own slice and the small-community heuristic naturally collapses siblings from the same directory (e.g. many different test-file clusters under the same folder), so duplicates are expected even after the steps above — verified in production: 338 of 1083 communities (31%) shared a label with at least one other community before reconciliation (apa monorepo, 2026-07-03). After all labels are assembled: +1. Group community IDs by their current label. +2. For every group with >1 member, pick a distinguishing keyword from each member's OWN node labels (word-frequency count, excluding stopwords and words already in the shared base label) and append it, e.g. `"Agent Gateway MCP Client" -> "Agent Gateway MCP Client — Server"` / `"— Additional"`. +3. If a collision still remains after that (rare — e.g. multiple communities extracted from the very same source image), append a numeric suffix `" (1)"`, `" (2)"` as a last resort. +4. Re-check for zero duplicates before moving on. + ```bash $(cat graphify-out/.graphify_python) -c " import sys, json @@ -534,6 +550,8 @@ graphify export html # auto-aggregates to community view if graph > 5000 nodes # or: graphify export html --no-viz ``` +**If the resulting view (aggregated community graph, or the raw graph for small corpora) has more than ~300 nodes, ALSO generate a Sigma.js + graphology version and present that as the primary interactive file instead of vis-network's `graph.html`.** vis-network runs a live single-threaded JS forceAtlas2 physics simulation on load (200 iterations before it disables itself) — verified in production this is genuinely slow/laggy past a few hundred nodes (apa monorepo, 1083-community aggregated view, 2026-07-03), independent of hardware. Sigma.js (WebGL rendering) with the layout **precomputed offline in Python** (`nx.forceatlas2_layout`, no client-side physics at all) fixes this at the root rather than swapping one JS physics engine for another. See `references/sigma-viz.md` for the full recipe (meta-graph construction, offline layout, self-contained HTML template loading sigma/graphology from esm.sh). Below the 300-node threshold, vis-network's `graph.html` is fine as-is — the extra build step isn't worth it. + ### Steps 6b-8 - Wiki, Neo4j, FalkorDB, SVG, GraphML, MCP, benchmark (only on their flags) These run only when their flag is present (`--wiki`, `--neo4j`/`--neo4j-push`, `--falkordb`/`--falkordb-push`, `--svg`, `--graphml`, `--mcp`) or, for the token-reduction benchmark, when `total_words` exceeds 5,000. A default run with no export flags skips all of them. See `references/exports.md` for each one. Run any `--wiki` export before Step 9 cleanup so `.graphify_labels.json` is still available. diff --git a/graphify/skill-kiro.md b/graphify/skill-kiro.md index b3542438d..5c337677e 100644 --- a/graphify/skill-kiro.md +++ b/graphify/skill-kiro.md @@ -477,8 +477,24 @@ Substitute `IS_DIRECTED` and `INPUT_PATH` as in Step 4. If a `GRAPH HEALTH WARNI Read `graphify-out/.graphify_analysis.json`. For each community key, look at its node labels and write a 2-5 word plain-language name (e.g. "Attention Mechanism", "Training Pipeline", "Data Loading"). +**At scale (100+ communities), do NOT hand-label one at a time and do NOT fall back to a naive path-prefix heuristic (e.g. title-casing the common directory prefix).** Both break down past a handful of communities — the heuristic in particular produces duplicated-segment junk ("Backend Backend", "Frontend Frontend") and word-soup names that look plausible until you actually read them. Verified in production on a 1000+ community monorepo graph (apa, 2026-07-03): a heuristic pass needed a full redo. + +What works instead, for communities with >=20 nodes (the ones that actually surface in God Nodes / navigation): +1. For each such community, sample its ~15 highest-degree (most connected) member nodes — not the full member list, communities can have 300+ nodes. +2. Batch communities into groups of ~15-20 per subagent (write each batch's samples to a small JSON file under `graphify-out/`). +3. Dispatch one `general-purpose` subagent per batch (all in the same message, in parallel — same pattern as Step 3 Part B), each told to read its batch file and write a JSON `{community_id: name}` map based on genuine reasoning about the sampled node labels/paths, not just repeating directory names. +4. Merge all batch results. + +For the long tail of small communities (<20 nodes) where subagent cost isn't justified, a path-prefix heuristic is an acceptable lighter-weight fallback — but dedupe repeated words/segments, strip trailing file extensions, and spot-check a random sample of ~15 outputs across different sizes before trusting it. If duplicated-word or generic ("Community N") names appear in the sample, fix the heuristic before applying it to the rest. + Then regenerate the report and save the labels for the visualizer: +**Reconcile the FULL label set for cross-batch duplicates before regenerating the report — this is a required final pass, not optional polish.** Subagent batches only see their own slice and the small-community heuristic naturally collapses siblings from the same directory (e.g. many different test-file clusters under the same folder), so duplicates are expected even after the steps above — verified in production: 338 of 1083 communities (31%) shared a label with at least one other community before reconciliation (apa monorepo, 2026-07-03). After all labels are assembled: +1. Group community IDs by their current label. +2. For every group with >1 member, pick a distinguishing keyword from each member's OWN node labels (word-frequency count, excluding stopwords and words already in the shared base label) and append it, e.g. `"Agent Gateway MCP Client" -> "Agent Gateway MCP Client — Server"` / `"— Additional"`. +3. If a collision still remains after that (rare — e.g. multiple communities extracted from the very same source image), append a numeric suffix `" (1)"`, `" (2)"` as a last resort. +4. Re-check for zero duplicates before moving on. + ```bash $(cat graphify-out/.graphify_python) -c " import sys, json @@ -534,6 +550,8 @@ graphify export html # auto-aggregates to community view if graph > 5000 nodes # or: graphify export html --no-viz ``` +**If the resulting view (aggregated community graph, or the raw graph for small corpora) has more than ~300 nodes, ALSO generate a Sigma.js + graphology version and present that as the primary interactive file instead of vis-network's `graph.html`.** vis-network runs a live single-threaded JS forceAtlas2 physics simulation on load (200 iterations before it disables itself) — verified in production this is genuinely slow/laggy past a few hundred nodes (apa monorepo, 1083-community aggregated view, 2026-07-03), independent of hardware. Sigma.js (WebGL rendering) with the layout **precomputed offline in Python** (`nx.forceatlas2_layout`, no client-side physics at all) fixes this at the root rather than swapping one JS physics engine for another. See `references/sigma-viz.md` for the full recipe (meta-graph construction, offline layout, self-contained HTML template loading sigma/graphology from esm.sh). Below the 300-node threshold, vis-network's `graph.html` is fine as-is — the extra build step isn't worth it. + ### Steps 6b-8 - Wiki, Neo4j, FalkorDB, SVG, GraphML, MCP, benchmark (only on their flags) These run only when their flag is present (`--wiki`, `--neo4j`/`--neo4j-push`, `--falkordb`/`--falkordb-push`, `--svg`, `--graphml`, `--mcp`) or, for the token-reduction benchmark, when `total_words` exceeds 5,000. A default run with no export flags skips all of them. See `references/exports.md` for each one. Run any `--wiki` export before Step 9 cleanup so `.graphify_labels.json` is still available. diff --git a/graphify/skill-opencode.md b/graphify/skill-opencode.md index 684e0e223..e923319b3 100644 --- a/graphify/skill-opencode.md +++ b/graphify/skill-opencode.md @@ -469,8 +469,24 @@ Substitute `IS_DIRECTED` and `INPUT_PATH` as in Step 4. If a `GRAPH HEALTH WARNI Read `graphify-out/.graphify_analysis.json`. For each community key, look at its node labels and write a 2-5 word plain-language name (e.g. "Attention Mechanism", "Training Pipeline", "Data Loading"). +**At scale (100+ communities), do NOT hand-label one at a time and do NOT fall back to a naive path-prefix heuristic (e.g. title-casing the common directory prefix).** Both break down past a handful of communities — the heuristic in particular produces duplicated-segment junk ("Backend Backend", "Frontend Frontend") and word-soup names that look plausible until you actually read them. Verified in production on a 1000+ community monorepo graph (apa, 2026-07-03): a heuristic pass needed a full redo. + +What works instead, for communities with >=20 nodes (the ones that actually surface in God Nodes / navigation): +1. For each such community, sample its ~15 highest-degree (most connected) member nodes — not the full member list, communities can have 300+ nodes. +2. Batch communities into groups of ~15-20 per subagent (write each batch's samples to a small JSON file under `graphify-out/`). +3. Dispatch one `general-purpose` subagent per batch (all in the same message, in parallel — same pattern as Step 3 Part B), each told to read its batch file and write a JSON `{community_id: name}` map based on genuine reasoning about the sampled node labels/paths, not just repeating directory names. +4. Merge all batch results. + +For the long tail of small communities (<20 nodes) where subagent cost isn't justified, a path-prefix heuristic is an acceptable lighter-weight fallback — but dedupe repeated words/segments, strip trailing file extensions, and spot-check a random sample of ~15 outputs across different sizes before trusting it. If duplicated-word or generic ("Community N") names appear in the sample, fix the heuristic before applying it to the rest. + Then regenerate the report and save the labels for the visualizer: +**Reconcile the FULL label set for cross-batch duplicates before regenerating the report — this is a required final pass, not optional polish.** Subagent batches only see their own slice and the small-community heuristic naturally collapses siblings from the same directory (e.g. many different test-file clusters under the same folder), so duplicates are expected even after the steps above — verified in production: 338 of 1083 communities (31%) shared a label with at least one other community before reconciliation (apa monorepo, 2026-07-03). After all labels are assembled: +1. Group community IDs by their current label. +2. For every group with >1 member, pick a distinguishing keyword from each member's OWN node labels (word-frequency count, excluding stopwords and words already in the shared base label) and append it, e.g. `"Agent Gateway MCP Client" -> "Agent Gateway MCP Client — Server"` / `"— Additional"`. +3. If a collision still remains after that (rare — e.g. multiple communities extracted from the very same source image), append a numeric suffix `" (1)"`, `" (2)"` as a last resort. +4. Re-check for zero duplicates before moving on. + ```bash $(cat graphify-out/.graphify_python) -c " import sys, json @@ -526,6 +542,8 @@ graphify export html # auto-aggregates to community view if graph > 5000 nodes # or: graphify export html --no-viz ``` +**If the resulting view (aggregated community graph, or the raw graph for small corpora) has more than ~300 nodes, ALSO generate a Sigma.js + graphology version and present that as the primary interactive file instead of vis-network's `graph.html`.** vis-network runs a live single-threaded JS forceAtlas2 physics simulation on load (200 iterations before it disables itself) — verified in production this is genuinely slow/laggy past a few hundred nodes (apa monorepo, 1083-community aggregated view, 2026-07-03), independent of hardware. Sigma.js (WebGL rendering) with the layout **precomputed offline in Python** (`nx.forceatlas2_layout`, no client-side physics at all) fixes this at the root rather than swapping one JS physics engine for another. See `references/sigma-viz.md` for the full recipe (meta-graph construction, offline layout, self-contained HTML template loading sigma/graphology from esm.sh). Below the 300-node threshold, vis-network's `graph.html` is fine as-is — the extra build step isn't worth it. + ### Steps 6b-8 - Wiki, Neo4j, FalkorDB, SVG, GraphML, MCP, benchmark (only on their flags) These run only when their flag is present (`--wiki`, `--neo4j`/`--neo4j-push`, `--falkordb`/`--falkordb-push`, `--svg`, `--graphml`, `--mcp`) or, for the token-reduction benchmark, when `total_words` exceeds 5,000. A default run with no export flags skips all of them. See `references/exports.md` for each one. Run any `--wiki` export before Step 9 cleanup so `.graphify_labels.json` is still available. diff --git a/graphify/skill-pi.md b/graphify/skill-pi.md index b3542438d..5c337677e 100644 --- a/graphify/skill-pi.md +++ b/graphify/skill-pi.md @@ -477,8 +477,24 @@ Substitute `IS_DIRECTED` and `INPUT_PATH` as in Step 4. If a `GRAPH HEALTH WARNI Read `graphify-out/.graphify_analysis.json`. For each community key, look at its node labels and write a 2-5 word plain-language name (e.g. "Attention Mechanism", "Training Pipeline", "Data Loading"). +**At scale (100+ communities), do NOT hand-label one at a time and do NOT fall back to a naive path-prefix heuristic (e.g. title-casing the common directory prefix).** Both break down past a handful of communities — the heuristic in particular produces duplicated-segment junk ("Backend Backend", "Frontend Frontend") and word-soup names that look plausible until you actually read them. Verified in production on a 1000+ community monorepo graph (apa, 2026-07-03): a heuristic pass needed a full redo. + +What works instead, for communities with >=20 nodes (the ones that actually surface in God Nodes / navigation): +1. For each such community, sample its ~15 highest-degree (most connected) member nodes — not the full member list, communities can have 300+ nodes. +2. Batch communities into groups of ~15-20 per subagent (write each batch's samples to a small JSON file under `graphify-out/`). +3. Dispatch one `general-purpose` subagent per batch (all in the same message, in parallel — same pattern as Step 3 Part B), each told to read its batch file and write a JSON `{community_id: name}` map based on genuine reasoning about the sampled node labels/paths, not just repeating directory names. +4. Merge all batch results. + +For the long tail of small communities (<20 nodes) where subagent cost isn't justified, a path-prefix heuristic is an acceptable lighter-weight fallback — but dedupe repeated words/segments, strip trailing file extensions, and spot-check a random sample of ~15 outputs across different sizes before trusting it. If duplicated-word or generic ("Community N") names appear in the sample, fix the heuristic before applying it to the rest. + Then regenerate the report and save the labels for the visualizer: +**Reconcile the FULL label set for cross-batch duplicates before regenerating the report — this is a required final pass, not optional polish.** Subagent batches only see their own slice and the small-community heuristic naturally collapses siblings from the same directory (e.g. many different test-file clusters under the same folder), so duplicates are expected even after the steps above — verified in production: 338 of 1083 communities (31%) shared a label with at least one other community before reconciliation (apa monorepo, 2026-07-03). After all labels are assembled: +1. Group community IDs by their current label. +2. For every group with >1 member, pick a distinguishing keyword from each member's OWN node labels (word-frequency count, excluding stopwords and words already in the shared base label) and append it, e.g. `"Agent Gateway MCP Client" -> "Agent Gateway MCP Client — Server"` / `"— Additional"`. +3. If a collision still remains after that (rare — e.g. multiple communities extracted from the very same source image), append a numeric suffix `" (1)"`, `" (2)"` as a last resort. +4. Re-check for zero duplicates before moving on. + ```bash $(cat graphify-out/.graphify_python) -c " import sys, json @@ -534,6 +550,8 @@ graphify export html # auto-aggregates to community view if graph > 5000 nodes # or: graphify export html --no-viz ``` +**If the resulting view (aggregated community graph, or the raw graph for small corpora) has more than ~300 nodes, ALSO generate a Sigma.js + graphology version and present that as the primary interactive file instead of vis-network's `graph.html`.** vis-network runs a live single-threaded JS forceAtlas2 physics simulation on load (200 iterations before it disables itself) — verified in production this is genuinely slow/laggy past a few hundred nodes (apa monorepo, 1083-community aggregated view, 2026-07-03), independent of hardware. Sigma.js (WebGL rendering) with the layout **precomputed offline in Python** (`nx.forceatlas2_layout`, no client-side physics at all) fixes this at the root rather than swapping one JS physics engine for another. See `references/sigma-viz.md` for the full recipe (meta-graph construction, offline layout, self-contained HTML template loading sigma/graphology from esm.sh). Below the 300-node threshold, vis-network's `graph.html` is fine as-is — the extra build step isn't worth it. + ### Steps 6b-8 - Wiki, Neo4j, FalkorDB, SVG, GraphML, MCP, benchmark (only on their flags) These run only when their flag is present (`--wiki`, `--neo4j`/`--neo4j-push`, `--falkordb`/`--falkordb-push`, `--svg`, `--graphml`, `--mcp`) or, for the token-reduction benchmark, when `total_words` exceeds 5,000. A default run with no export flags skips all of them. See `references/exports.md` for each one. Run any `--wiki` export before Step 9 cleanup so `.graphify_labels.json` is still available. diff --git a/graphify/skill-trae.md b/graphify/skill-trae.md index ac16b925d..72fbd9937 100644 --- a/graphify/skill-trae.md +++ b/graphify/skill-trae.md @@ -475,8 +475,24 @@ Substitute `IS_DIRECTED` and `INPUT_PATH` as in Step 4. If a `GRAPH HEALTH WARNI Read `graphify-out/.graphify_analysis.json`. For each community key, look at its node labels and write a 2-5 word plain-language name (e.g. "Attention Mechanism", "Training Pipeline", "Data Loading"). +**At scale (100+ communities), do NOT hand-label one at a time and do NOT fall back to a naive path-prefix heuristic (e.g. title-casing the common directory prefix).** Both break down past a handful of communities — the heuristic in particular produces duplicated-segment junk ("Backend Backend", "Frontend Frontend") and word-soup names that look plausible until you actually read them. Verified in production on a 1000+ community monorepo graph (apa, 2026-07-03): a heuristic pass needed a full redo. + +What works instead, for communities with >=20 nodes (the ones that actually surface in God Nodes / navigation): +1. For each such community, sample its ~15 highest-degree (most connected) member nodes — not the full member list, communities can have 300+ nodes. +2. Batch communities into groups of ~15-20 per subagent (write each batch's samples to a small JSON file under `graphify-out/`). +3. Dispatch one `general-purpose` subagent per batch (all in the same message, in parallel — same pattern as Step 3 Part B), each told to read its batch file and write a JSON `{community_id: name}` map based on genuine reasoning about the sampled node labels/paths, not just repeating directory names. +4. Merge all batch results. + +For the long tail of small communities (<20 nodes) where subagent cost isn't justified, a path-prefix heuristic is an acceptable lighter-weight fallback — but dedupe repeated words/segments, strip trailing file extensions, and spot-check a random sample of ~15 outputs across different sizes before trusting it. If duplicated-word or generic ("Community N") names appear in the sample, fix the heuristic before applying it to the rest. + Then regenerate the report and save the labels for the visualizer: +**Reconcile the FULL label set for cross-batch duplicates before regenerating the report — this is a required final pass, not optional polish.** Subagent batches only see their own slice and the small-community heuristic naturally collapses siblings from the same directory (e.g. many different test-file clusters under the same folder), so duplicates are expected even after the steps above — verified in production: 338 of 1083 communities (31%) shared a label with at least one other community before reconciliation (apa monorepo, 2026-07-03). After all labels are assembled: +1. Group community IDs by their current label. +2. For every group with >1 member, pick a distinguishing keyword from each member's OWN node labels (word-frequency count, excluding stopwords and words already in the shared base label) and append it, e.g. `"Agent Gateway MCP Client" -> "Agent Gateway MCP Client — Server"` / `"— Additional"`. +3. If a collision still remains after that (rare — e.g. multiple communities extracted from the very same source image), append a numeric suffix `" (1)"`, `" (2)"` as a last resort. +4. Re-check for zero duplicates before moving on. + ```bash $(cat graphify-out/.graphify_python) -c " import sys, json @@ -532,6 +548,8 @@ graphify export html # auto-aggregates to community view if graph > 5000 nodes # or: graphify export html --no-viz ``` +**If the resulting view (aggregated community graph, or the raw graph for small corpora) has more than ~300 nodes, ALSO generate a Sigma.js + graphology version and present that as the primary interactive file instead of vis-network's `graph.html`.** vis-network runs a live single-threaded JS forceAtlas2 physics simulation on load (200 iterations before it disables itself) — verified in production this is genuinely slow/laggy past a few hundred nodes (apa monorepo, 1083-community aggregated view, 2026-07-03), independent of hardware. Sigma.js (WebGL rendering) with the layout **precomputed offline in Python** (`nx.forceatlas2_layout`, no client-side physics at all) fixes this at the root rather than swapping one JS physics engine for another. See `references/sigma-viz.md` for the full recipe (meta-graph construction, offline layout, self-contained HTML template loading sigma/graphology from esm.sh). Below the 300-node threshold, vis-network's `graph.html` is fine as-is — the extra build step isn't worth it. + ### Steps 6b-8 - Wiki, Neo4j, FalkorDB, SVG, GraphML, MCP, benchmark (only on their flags) These run only when their flag is present (`--wiki`, `--neo4j`/`--neo4j-push`, `--falkordb`/`--falkordb-push`, `--svg`, `--graphml`, `--mcp`) or, for the token-reduction benchmark, when `total_words` exceeds 5,000. A default run with no export flags skips all of them. See `references/exports.md` for each one. Run any `--wiki` export before Step 9 cleanup so `.graphify_labels.json` is still available. diff --git a/graphify/skill-vscode.md b/graphify/skill-vscode.md index 31a3352f7..c9592c649 100644 --- a/graphify/skill-vscode.md +++ b/graphify/skill-vscode.md @@ -473,8 +473,24 @@ Substitute `IS_DIRECTED` and `INPUT_PATH` as in Step 4. If a `GRAPH HEALTH WARNI Read `graphify-out/.graphify_analysis.json`. For each community key, look at its node labels and write a 2-5 word plain-language name (e.g. "Attention Mechanism", "Training Pipeline", "Data Loading"). +**At scale (100+ communities), do NOT hand-label one at a time and do NOT fall back to a naive path-prefix heuristic (e.g. title-casing the common directory prefix).** Both break down past a handful of communities — the heuristic in particular produces duplicated-segment junk ("Backend Backend", "Frontend Frontend") and word-soup names that look plausible until you actually read them. Verified in production on a 1000+ community monorepo graph (apa, 2026-07-03): a heuristic pass needed a full redo. + +What works instead, for communities with >=20 nodes (the ones that actually surface in God Nodes / navigation): +1. For each such community, sample its ~15 highest-degree (most connected) member nodes — not the full member list, communities can have 300+ nodes. +2. Batch communities into groups of ~15-20 per subagent (write each batch's samples to a small JSON file under `graphify-out/`). +3. Dispatch one `general-purpose` subagent per batch (all in the same message, in parallel — same pattern as Step 3 Part B), each told to read its batch file and write a JSON `{community_id: name}` map based on genuine reasoning about the sampled node labels/paths, not just repeating directory names. +4. Merge all batch results. + +For the long tail of small communities (<20 nodes) where subagent cost isn't justified, a path-prefix heuristic is an acceptable lighter-weight fallback — but dedupe repeated words/segments, strip trailing file extensions, and spot-check a random sample of ~15 outputs across different sizes before trusting it. If duplicated-word or generic ("Community N") names appear in the sample, fix the heuristic before applying it to the rest. + Then regenerate the report and save the labels for the visualizer: +**Reconcile the FULL label set for cross-batch duplicates before regenerating the report — this is a required final pass, not optional polish.** Subagent batches only see their own slice and the small-community heuristic naturally collapses siblings from the same directory (e.g. many different test-file clusters under the same folder), so duplicates are expected even after the steps above — verified in production: 338 of 1083 communities (31%) shared a label with at least one other community before reconciliation (apa monorepo, 2026-07-03). After all labels are assembled: +1. Group community IDs by their current label. +2. For every group with >1 member, pick a distinguishing keyword from each member's OWN node labels (word-frequency count, excluding stopwords and words already in the shared base label) and append it, e.g. `"Agent Gateway MCP Client" -> "Agent Gateway MCP Client — Server"` / `"— Additional"`. +3. If a collision still remains after that (rare — e.g. multiple communities extracted from the very same source image), append a numeric suffix `" (1)"`, `" (2)"` as a last resort. +4. Re-check for zero duplicates before moving on. + ```bash $(cat graphify-out/.graphify_python) -c " import sys, json @@ -530,6 +546,8 @@ graphify export html # auto-aggregates to community view if graph > 5000 nodes # or: graphify export html --no-viz ``` +**If the resulting view (aggregated community graph, or the raw graph for small corpora) has more than ~300 nodes, ALSO generate a Sigma.js + graphology version and present that as the primary interactive file instead of vis-network's `graph.html`.** vis-network runs a live single-threaded JS forceAtlas2 physics simulation on load (200 iterations before it disables itself) — verified in production this is genuinely slow/laggy past a few hundred nodes (apa monorepo, 1083-community aggregated view, 2026-07-03), independent of hardware. Sigma.js (WebGL rendering) with the layout **precomputed offline in Python** (`nx.forceatlas2_layout`, no client-side physics at all) fixes this at the root rather than swapping one JS physics engine for another. See `references/sigma-viz.md` for the full recipe (meta-graph construction, offline layout, self-contained HTML template loading sigma/graphology from esm.sh). Below the 300-node threshold, vis-network's `graph.html` is fine as-is — the extra build step isn't worth it. + ### Steps 6b-8 - Wiki, Neo4j, FalkorDB, SVG, GraphML, MCP, benchmark (only on their flags) These run only when their flag is present (`--wiki`, `--neo4j`/`--neo4j-push`, `--falkordb`/`--falkordb-push`, `--svg`, `--graphml`, `--mcp`) or, for the token-reduction benchmark, when `total_words` exceeds 5,000. A default run with no export flags skips all of them. See `references/exports.md` for each one. Run any `--wiki` export before Step 9 cleanup so `.graphify_labels.json` is still available. diff --git a/graphify/skill-windows.md b/graphify/skill-windows.md index 6ab0e027c..d947bb490 100644 --- a/graphify/skill-windows.md +++ b/graphify/skill-windows.md @@ -499,8 +499,24 @@ Substitute `IS_DIRECTED` and `INPUT_PATH` as in Step 4. If a `GRAPH HEALTH WARNI Read `graphify-out/.graphify_analysis.json`. For each community key, look at its node labels and write a 2-5 word plain-language name (e.g. "Attention Mechanism", "Training Pipeline", "Data Loading"). +**At scale (100+ communities), do NOT hand-label one at a time and do NOT fall back to a naive path-prefix heuristic (e.g. title-casing the common directory prefix).** Both break down past a handful of communities — the heuristic in particular produces duplicated-segment junk ("Backend Backend", "Frontend Frontend") and word-soup names that look plausible until you actually read them. Verified in production on a 1000+ community monorepo graph (apa, 2026-07-03): a heuristic pass needed a full redo. + +What works instead, for communities with >=20 nodes (the ones that actually surface in God Nodes / navigation): +1. For each such community, sample its ~15 highest-degree (most connected) member nodes — not the full member list, communities can have 300+ nodes. +2. Batch communities into groups of ~15-20 per subagent (write each batch's samples to a small JSON file under `graphify-out/`). +3. Dispatch one `general-purpose` subagent per batch (all in the same message, in parallel — same pattern as Step 3 Part B), each told to read its batch file and write a JSON `{community_id: name}` map based on genuine reasoning about the sampled node labels/paths, not just repeating directory names. +4. Merge all batch results. + +For the long tail of small communities (<20 nodes) where subagent cost isn't justified, a path-prefix heuristic is an acceptable lighter-weight fallback — but dedupe repeated words/segments, strip trailing file extensions, and spot-check a random sample of ~15 outputs across different sizes before trusting it. If duplicated-word or generic ("Community N") names appear in the sample, fix the heuristic before applying it to the rest. + Then regenerate the report and save the labels for the visualizer: +**Reconcile the FULL label set for cross-batch duplicates before regenerating the report — this is a required final pass, not optional polish.** Subagent batches only see their own slice and the small-community heuristic naturally collapses siblings from the same directory (e.g. many different test-file clusters under the same folder), so duplicates are expected even after the steps above — verified in production: 338 of 1083 communities (31%) shared a label with at least one other community before reconciliation (apa monorepo, 2026-07-03). After all labels are assembled: +1. Group community IDs by their current label. +2. For every group with >1 member, pick a distinguishing keyword from each member's OWN node labels (word-frequency count, excluding stopwords and words already in the shared base label) and append it, e.g. `"Agent Gateway MCP Client" -> "Agent Gateway MCP Client — Server"` / `"— Additional"`. +3. If a collision still remains after that (rare — e.g. multiple communities extracted from the very same source image), append a numeric suffix `" (1)"`, `" (2)"` as a last resort. +4. Re-check for zero duplicates before moving on. + ```bash $(cat graphify-out/.graphify_python) -c " import sys, json @@ -556,6 +572,8 @@ graphify export html # auto-aggregates to community view if graph > 5000 nodes # or: graphify export html --no-viz ``` +**If the resulting view (aggregated community graph, or the raw graph for small corpora) has more than ~300 nodes, ALSO generate a Sigma.js + graphology version and present that as the primary interactive file instead of vis-network's `graph.html`.** vis-network runs a live single-threaded JS forceAtlas2 physics simulation on load (200 iterations before it disables itself) — verified in production this is genuinely slow/laggy past a few hundred nodes (apa monorepo, 1083-community aggregated view, 2026-07-03), independent of hardware. Sigma.js (WebGL rendering) with the layout **precomputed offline in Python** (`nx.forceatlas2_layout`, no client-side physics at all) fixes this at the root rather than swapping one JS physics engine for another. See `references/sigma-viz.md` for the full recipe (meta-graph construction, offline layout, self-contained HTML template loading sigma/graphology from esm.sh). Below the 300-node threshold, vis-network's `graph.html` is fine as-is — the extra build step isn't worth it. + ### Steps 6b-8 - Wiki, Neo4j, FalkorDB, SVG, GraphML, MCP, benchmark (only on their flags) These run only when their flag is present (`--wiki`, `--neo4j`/`--neo4j-push`, `--falkordb`/`--falkordb-push`, `--svg`, `--graphml`, `--mcp`) or, for the token-reduction benchmark, when `total_words` exceeds 5,000. A default run with no export flags skips all of them. See `references/exports.md` for each one. Run any `--wiki` export before Step 9 cleanup so `.graphify_labels.json` is still available. diff --git a/graphify/skill.md b/graphify/skill.md index b3542438d..5c337677e 100644 --- a/graphify/skill.md +++ b/graphify/skill.md @@ -477,8 +477,24 @@ Substitute `IS_DIRECTED` and `INPUT_PATH` as in Step 4. If a `GRAPH HEALTH WARNI Read `graphify-out/.graphify_analysis.json`. For each community key, look at its node labels and write a 2-5 word plain-language name (e.g. "Attention Mechanism", "Training Pipeline", "Data Loading"). +**At scale (100+ communities), do NOT hand-label one at a time and do NOT fall back to a naive path-prefix heuristic (e.g. title-casing the common directory prefix).** Both break down past a handful of communities — the heuristic in particular produces duplicated-segment junk ("Backend Backend", "Frontend Frontend") and word-soup names that look plausible until you actually read them. Verified in production on a 1000+ community monorepo graph (apa, 2026-07-03): a heuristic pass needed a full redo. + +What works instead, for communities with >=20 nodes (the ones that actually surface in God Nodes / navigation): +1. For each such community, sample its ~15 highest-degree (most connected) member nodes — not the full member list, communities can have 300+ nodes. +2. Batch communities into groups of ~15-20 per subagent (write each batch's samples to a small JSON file under `graphify-out/`). +3. Dispatch one `general-purpose` subagent per batch (all in the same message, in parallel — same pattern as Step 3 Part B), each told to read its batch file and write a JSON `{community_id: name}` map based on genuine reasoning about the sampled node labels/paths, not just repeating directory names. +4. Merge all batch results. + +For the long tail of small communities (<20 nodes) where subagent cost isn't justified, a path-prefix heuristic is an acceptable lighter-weight fallback — but dedupe repeated words/segments, strip trailing file extensions, and spot-check a random sample of ~15 outputs across different sizes before trusting it. If duplicated-word or generic ("Community N") names appear in the sample, fix the heuristic before applying it to the rest. + Then regenerate the report and save the labels for the visualizer: +**Reconcile the FULL label set for cross-batch duplicates before regenerating the report — this is a required final pass, not optional polish.** Subagent batches only see their own slice and the small-community heuristic naturally collapses siblings from the same directory (e.g. many different test-file clusters under the same folder), so duplicates are expected even after the steps above — verified in production: 338 of 1083 communities (31%) shared a label with at least one other community before reconciliation (apa monorepo, 2026-07-03). After all labels are assembled: +1. Group community IDs by their current label. +2. For every group with >1 member, pick a distinguishing keyword from each member's OWN node labels (word-frequency count, excluding stopwords and words already in the shared base label) and append it, e.g. `"Agent Gateway MCP Client" -> "Agent Gateway MCP Client — Server"` / `"— Additional"`. +3. If a collision still remains after that (rare — e.g. multiple communities extracted from the very same source image), append a numeric suffix `" (1)"`, `" (2)"` as a last resort. +4. Re-check for zero duplicates before moving on. + ```bash $(cat graphify-out/.graphify_python) -c " import sys, json @@ -534,6 +550,8 @@ graphify export html # auto-aggregates to community view if graph > 5000 nodes # or: graphify export html --no-viz ``` +**If the resulting view (aggregated community graph, or the raw graph for small corpora) has more than ~300 nodes, ALSO generate a Sigma.js + graphology version and present that as the primary interactive file instead of vis-network's `graph.html`.** vis-network runs a live single-threaded JS forceAtlas2 physics simulation on load (200 iterations before it disables itself) — verified in production this is genuinely slow/laggy past a few hundred nodes (apa monorepo, 1083-community aggregated view, 2026-07-03), independent of hardware. Sigma.js (WebGL rendering) with the layout **precomputed offline in Python** (`nx.forceatlas2_layout`, no client-side physics at all) fixes this at the root rather than swapping one JS physics engine for another. See `references/sigma-viz.md` for the full recipe (meta-graph construction, offline layout, self-contained HTML template loading sigma/graphology from esm.sh). Below the 300-node threshold, vis-network's `graph.html` is fine as-is — the extra build step isn't worth it. + ### Steps 6b-8 - Wiki, Neo4j, FalkorDB, SVG, GraphML, MCP, benchmark (only on their flags) These run only when their flag is present (`--wiki`, `--neo4j`/`--neo4j-push`, `--falkordb`/`--falkordb-push`, `--svg`, `--graphml`, `--mcp`) or, for the token-reduction benchmark, when `total_words` exceeds 5,000. A default run with no export flags skips all of them. See `references/exports.md` for each one. Run any `--wiki` export before Step 9 cleanup so `.graphify_labels.json` is still available. diff --git a/graphify/skills/agents/references/sigma-viz.md b/graphify/skills/agents/references/sigma-viz.md new file mode 100644 index 000000000..7cf0ba408 --- /dev/null +++ b/graphify/skills/agents/references/sigma-viz.md @@ -0,0 +1,219 @@ +# graphify reference: Sigma.js + graphology visualization for large graphs + +Load this when Step 6's `graph.html` (vis-network) would render more than ~300 nodes — either the aggregated community view on a large corpus, or the raw graph on a smaller one that still clusters into 300+ communities. vis-network runs a live, single-threaded JS forceAtlas2 physics simulation on load (canvas 2D rendering); past a few hundred nodes this stabilization pass is genuinely slow regardless of hardware. The fix is not swapping to a different JS physics engine — it's removing client-side physics entirely: precompute the layout once in Python (fast, uses networkx's optimized implementation) and render only, with sigma.js's WebGL renderer instead of vis-network's canvas 2D renderer. + +Output file: `graphify-out/graph_sigma.html` (self-contained, opens directly like `graph.html`). + +## Step 1 — build the meta-graph and precompute layout in Python + +Adjust `MIN_COMMUNITY_SIZE` (20 is a reasonable default — communities below this rarely appear in God Nodes / navigation and just add render cost) and `INPUT_PATH`/label source to match the current run. + +```python +import json +import networkx as nx +from pathlib import Path +from collections import Counter, defaultdict + +g_data = json.loads(Path('graphify-out/graph.json').read_text(encoding='utf-8')) +labels = json.loads(Path('graphify-out/.graphify_labels.json').read_text(encoding='utf-8')) + +MIN_COMMUNITY_SIZE = 20 +comms = defaultdict(list) +for n in g_data['nodes']: + comms[n['community']].append(n['id']) + +significant = {cid: members for cid, members in comms.items() if len(members) >= MIN_COMMUNITY_SIZE} +node_to_community = {m: cid for cid, members in significant.items() for m in members} + +meta = nx.Graph() +for cid, members in significant.items(): + meta.add_node(cid, member_count=len(members), label=labels.get(str(cid), f'Community {cid}')) + +edge_counts = Counter() +for link in g_data['links']: + cu, cv = node_to_community.get(link['source']), node_to_community.get(link['target']) + if cu is not None and cv is not None and cu != cv: + edge_counts[(min(cu, cv), max(cu, cv))] += 1 +for (cu, cv), w in edge_counts.items(): + meta.add_edge(cu, cv, weight=w) + +# offline layout — no client-side physics needed at all +pos = nx.forceatlas2_layout(meta, max_iter=800, gravity=1.0, scaling_ratio=4.0, seed=42, weight='weight') + +degrees = dict(meta.degree()) +max_deg = max(degrees.values()) if degrees else 1 +max_members = max((meta.nodes[n]['member_count'] for n in meta.nodes), default=1) +xs = [float(p[0]) for p in pos.values()] +ys = [float(p[1]) for p in pos.values()] +xr, yr = (max(xs) - min(xs)) or 1, (max(ys) - min(ys)) or 1 + +nodes_out = [] +for n in meta.nodes(): + x, y = float(pos[n][0]), float(pos[n][1]) + deg, mc = int(degrees.get(n, 0)), int(meta.nodes[n]['member_count']) + nodes_out.append({ + 'key': str(n), 'label': meta.nodes[n]['label'], + 'x': round((x - min(xs)) / xr * 1000, 2), 'y': round((y - min(ys)) / yr * 1000, 2), + 'size': round(3 + 12 * (mc / max_members) ** 0.5, 2), + 't': round(deg / max_deg if max_deg else 0, 3), + 'members': mc, 'degree': deg, + }) +edges_out = [{'source': str(u), 'target': str(v), 'weight': int(d.get('weight', 1))} + for u, v, d in meta.edges(data=True)] + +Path('graphify-out/.graphify_sigma_data.json').write_text( + json.dumps({'nodes': nodes_out, 'edges': edges_out}, ensure_ascii=False), encoding='utf-8') +print(f'meta graph: {len(nodes_out)} nodes, {len(edges_out)} edges — layout precomputed') +``` + +**Important**: cast every numpy value (`forceatlas2_layout` returns numpy floats) to plain Python `float`/`int` before `json.dumps` — numpy scalars aren't JSON-serializable and will raise `TypeError: Object of type float32 is not JSON serializable`. + +## Step 2 — write the HTML template + +Write this template to `graphify-out/graph_sigma.html`, with a literal `__GRAPH_DATA__` placeholder where the data goes (substituted in Step 3 — do NOT try to embed the JSON directly while authoring the template, string-templating that much escaping inline is error-prone). + +Key implementation notes: +- **Library loading**: sigma@3 ships CJS/ESM only, no browser UMD global. Load both libraries as ES modules from `esm.sh` (`https://esm.sh/sigma@3.0.3`, `https://esm.sh/graphology@0.25.4`) via ` + + +``` + +## Step 3 — inject the data and clean up + +```python +import json +from pathlib import Path + +data = json.loads(Path('graphify-out/.graphify_sigma_data.json').read_text(encoding='utf-8')) +html = Path('graphify-out/graph_sigma.html').read_text(encoding='utf-8') +html = html.replace('__GRAPH_DATA__', json.dumps(data, ensure_ascii=False)) +Path('graphify-out/graph_sigma.html').write_text(html, encoding='utf-8') +Path('graphify-out/.graphify_sigma_data.json').unlink() +``` + +**Before telling the user it's done**, sanity-check the embedded JSON didn't break the ` + + +``` + +## Step 3 — inject the data and clean up + +```python +import json +from pathlib import Path + +data = json.loads(Path('graphify-out/.graphify_sigma_data.json').read_text(encoding='utf-8')) +html = Path('graphify-out/graph_sigma.html').read_text(encoding='utf-8') +html = html.replace('__GRAPH_DATA__', json.dumps(data, ensure_ascii=False)) +Path('graphify-out/graph_sigma.html').write_text(html, encoding='utf-8') +Path('graphify-out/.graphify_sigma_data.json').unlink() +``` + +**Before telling the user it's done**, sanity-check the embedded JSON didn't break the ` + + +``` + +## Step 3 — inject the data and clean up + +```python +import json +from pathlib import Path + +data = json.loads(Path('graphify-out/.graphify_sigma_data.json').read_text(encoding='utf-8')) +html = Path('graphify-out/graph_sigma.html').read_text(encoding='utf-8') +html = html.replace('__GRAPH_DATA__', json.dumps(data, ensure_ascii=False)) +Path('graphify-out/graph_sigma.html').write_text(html, encoding='utf-8') +Path('graphify-out/.graphify_sigma_data.json').unlink() +``` + +**Before telling the user it's done**, sanity-check the embedded JSON didn't break the ` + + +``` + +## Step 3 — inject the data and clean up + +```python +import json +from pathlib import Path + +data = json.loads(Path('graphify-out/.graphify_sigma_data.json').read_text(encoding='utf-8')) +html = Path('graphify-out/graph_sigma.html').read_text(encoding='utf-8') +html = html.replace('__GRAPH_DATA__', json.dumps(data, ensure_ascii=False)) +Path('graphify-out/graph_sigma.html').write_text(html, encoding='utf-8') +Path('graphify-out/.graphify_sigma_data.json').unlink() +``` + +**Before telling the user it's done**, sanity-check the embedded JSON didn't break the ` + + +``` + +## Step 3 — inject the data and clean up + +```python +import json +from pathlib import Path + +data = json.loads(Path('graphify-out/.graphify_sigma_data.json').read_text(encoding='utf-8')) +html = Path('graphify-out/graph_sigma.html').read_text(encoding='utf-8') +html = html.replace('__GRAPH_DATA__', json.dumps(data, ensure_ascii=False)) +Path('graphify-out/graph_sigma.html').write_text(html, encoding='utf-8') +Path('graphify-out/.graphify_sigma_data.json').unlink() +``` + +**Before telling the user it's done**, sanity-check the embedded JSON didn't break the ` + + +``` + +## Step 3 — inject the data and clean up + +```python +import json +from pathlib import Path + +data = json.loads(Path('graphify-out/.graphify_sigma_data.json').read_text(encoding='utf-8')) +html = Path('graphify-out/graph_sigma.html').read_text(encoding='utf-8') +html = html.replace('__GRAPH_DATA__', json.dumps(data, ensure_ascii=False)) +Path('graphify-out/graph_sigma.html').write_text(html, encoding='utf-8') +Path('graphify-out/.graphify_sigma_data.json').unlink() +``` + +**Before telling the user it's done**, sanity-check the embedded JSON didn't break the ` + + +``` + +## Step 3 — inject the data and clean up + +```python +import json +from pathlib import Path + +data = json.loads(Path('graphify-out/.graphify_sigma_data.json').read_text(encoding='utf-8')) +html = Path('graphify-out/graph_sigma.html').read_text(encoding='utf-8') +html = html.replace('__GRAPH_DATA__', json.dumps(data, ensure_ascii=False)) +Path('graphify-out/graph_sigma.html').write_text(html, encoding='utf-8') +Path('graphify-out/.graphify_sigma_data.json').unlink() +``` + +**Before telling the user it's done**, sanity-check the embedded JSON didn't break the ` + + +``` + +## Step 3 — inject the data and clean up + +```python +import json +from pathlib import Path + +data = json.loads(Path('graphify-out/.graphify_sigma_data.json').read_text(encoding='utf-8')) +html = Path('graphify-out/graph_sigma.html').read_text(encoding='utf-8') +html = html.replace('__GRAPH_DATA__', json.dumps(data, ensure_ascii=False)) +Path('graphify-out/graph_sigma.html').write_text(html, encoding='utf-8') +Path('graphify-out/.graphify_sigma_data.json').unlink() +``` + +**Before telling the user it's done**, sanity-check the embedded JSON didn't break the ` + + +``` + +## Step 3 — inject the data and clean up + +```python +import json +from pathlib import Path + +data = json.loads(Path('graphify-out/.graphify_sigma_data.json').read_text(encoding='utf-8')) +html = Path('graphify-out/graph_sigma.html').read_text(encoding='utf-8') +html = html.replace('__GRAPH_DATA__', json.dumps(data, ensure_ascii=False)) +Path('graphify-out/graph_sigma.html').write_text(html, encoding='utf-8') +Path('graphify-out/.graphify_sigma_data.json').unlink() +``` + +**Before telling the user it's done**, sanity-check the embedded JSON didn't break the ` + + +``` + +## Step 3 — inject the data and clean up + +```python +import json +from pathlib import Path + +data = json.loads(Path('graphify-out/.graphify_sigma_data.json').read_text(encoding='utf-8')) +html = Path('graphify-out/graph_sigma.html').read_text(encoding='utf-8') +html = html.replace('__GRAPH_DATA__', json.dumps(data, ensure_ascii=False)) +Path('graphify-out/graph_sigma.html').write_text(html, encoding='utf-8') +Path('graphify-out/.graphify_sigma_data.json').unlink() +``` + +**Before telling the user it's done**, sanity-check the embedded JSON didn't break the ` + + +``` + +## Step 3 — inject the data and clean up + +```python +import json +from pathlib import Path + +data = json.loads(Path('graphify-out/.graphify_sigma_data.json').read_text(encoding='utf-8')) +html = Path('graphify-out/graph_sigma.html').read_text(encoding='utf-8') +html = html.replace('__GRAPH_DATA__', json.dumps(data, ensure_ascii=False)) +Path('graphify-out/graph_sigma.html').write_text(html, encoding='utf-8') +Path('graphify-out/.graphify_sigma_data.json').unlink() +``` + +**Before telling the user it's done**, sanity-check the embedded JSON didn't break the ` + + +``` + +## Step 3 — inject the data and clean up + +```python +import json +from pathlib import Path + +data = json.loads(Path('graphify-out/.graphify_sigma_data.json').read_text(encoding='utf-8')) +html = Path('graphify-out/graph_sigma.html').read_text(encoding='utf-8') +html = html.replace('__GRAPH_DATA__', json.dumps(data, ensure_ascii=False)) +Path('graphify-out/graph_sigma.html').write_text(html, encoding='utf-8') +Path('graphify-out/.graphify_sigma_data.json').unlink() +``` + +**Before telling the user it's done**, sanity-check the embedded JSON didn't break the ` + + +``` + +## Step 3 — inject the data and clean up + +```python +import json +from pathlib import Path + +data = json.loads(Path('graphify-out/.graphify_sigma_data.json').read_text(encoding='utf-8')) +html = Path('graphify-out/graph_sigma.html').read_text(encoding='utf-8') +html = html.replace('__GRAPH_DATA__', json.dumps(data, ensure_ascii=False)) +Path('graphify-out/graph_sigma.html').write_text(html, encoding='utf-8') +Path('graphify-out/.graphify_sigma_data.json').unlink() +``` + +**Before telling the user it's done**, sanity-check the embedded JSON didn't break the ` + + +``` + +## Step 3 — inject the data and clean up + +```python +import json +from pathlib import Path + +data = json.loads(Path('graphify-out/.graphify_sigma_data.json').read_text(encoding='utf-8')) +html = Path('graphify-out/graph_sigma.html').read_text(encoding='utf-8') +html = html.replace('__GRAPH_DATA__', json.dumps(data, ensure_ascii=False)) +Path('graphify-out/graph_sigma.html').write_text(html, encoding='utf-8') +Path('graphify-out/.graphify_sigma_data.json').unlink() +``` + +**Before telling the user it's done**, sanity-check the embedded JSON didn't break the ` + + +``` + +## Step 3 — inject the data and clean up + +```python +import json +from pathlib import Path + +data = json.loads(Path('graphify-out/.graphify_sigma_data.json').read_text(encoding='utf-8')) +html = Path('graphify-out/graph_sigma.html').read_text(encoding='utf-8') +html = html.replace('__GRAPH_DATA__', json.dumps(data, ensure_ascii=False)) +Path('graphify-out/graph_sigma.html').write_text(html, encoding='utf-8') +Path('graphify-out/.graphify_sigma_data.json').unlink() +``` + +**Before telling the user it's done**, sanity-check the embedded JSON didn't break the ` + + +``` + +## Step 3 — inject the data and clean up + +```python +import json +from pathlib import Path + +data = json.loads(Path('graphify-out/.graphify_sigma_data.json').read_text(encoding='utf-8')) +html = Path('graphify-out/graph_sigma.html').read_text(encoding='utf-8') +html = html.replace('__GRAPH_DATA__', json.dumps(data, ensure_ascii=False)) +Path('graphify-out/graph_sigma.html').write_text(html, encoding='utf-8') +Path('graphify-out/.graphify_sigma_data.json').unlink() +``` + +**Before telling the user it's done**, sanity-check the embedded JSON didn't break the ` + + +``` + +## Step 3 — inject the data and clean up + +```python +import json +from pathlib import Path + +data = json.loads(Path('graphify-out/.graphify_sigma_data.json').read_text(encoding='utf-8')) +html = Path('graphify-out/graph_sigma.html').read_text(encoding='utf-8') +html = html.replace('__GRAPH_DATA__', json.dumps(data, ensure_ascii=False)) +Path('graphify-out/graph_sigma.html').write_text(html, encoding='utf-8') +Path('graphify-out/.graphify_sigma_data.json').unlink() +``` + +**Before telling the user it's done**, sanity-check the embedded JSON didn't break the ` + + +``` + +## Step 3 — inject the data and clean up + +```python +import json +from pathlib import Path + +data = json.loads(Path('graphify-out/.graphify_sigma_data.json').read_text(encoding='utf-8')) +html = Path('graphify-out/graph_sigma.html').read_text(encoding='utf-8') +html = html.replace('__GRAPH_DATA__', json.dumps(data, ensure_ascii=False)) +Path('graphify-out/graph_sigma.html').write_text(html, encoding='utf-8') +Path('graphify-out/.graphify_sigma_data.json').unlink() +``` + +**Before telling the user it's done**, sanity-check the embedded JSON didn't break the ` + + +``` + +## Step 3 — inject the data and clean up + +```python +import json +from pathlib import Path + +data = json.loads(Path('graphify-out/.graphify_sigma_data.json').read_text(encoding='utf-8')) +html = Path('graphify-out/graph_sigma.html').read_text(encoding='utf-8') +html = html.replace('__GRAPH_DATA__', json.dumps(data, ensure_ascii=False)) +Path('graphify-out/graph_sigma.html').write_text(html, encoding='utf-8') +Path('graphify-out/.graphify_sigma_data.json').unlink() +``` + +**Before telling the user it's done**, sanity-check the embedded JSON didn't break the ` + + +``` + +## Step 3 — inject the data and clean up + +```python +import json +from pathlib import Path + +data = json.loads(Path('graphify-out/.graphify_sigma_data.json').read_text(encoding='utf-8')) +html = Path('graphify-out/graph_sigma.html').read_text(encoding='utf-8') +html = html.replace('__GRAPH_DATA__', json.dumps(data, ensure_ascii=False)) +Path('graphify-out/graph_sigma.html').write_text(html, encoding='utf-8') +Path('graphify-out/.graphify_sigma_data.json').unlink() +``` + +**Before telling the user it's done**, sanity-check the embedded JSON didn't break the ` + + +``` + +## Step 3 — inject the data and clean up + +```python +import json +from pathlib import Path + +data = json.loads(Path('graphify-out/.graphify_sigma_data.json').read_text(encoding='utf-8')) +html = Path('graphify-out/graph_sigma.html').read_text(encoding='utf-8') +html = html.replace('__GRAPH_DATA__', json.dumps(data, ensure_ascii=False)) +Path('graphify-out/graph_sigma.html').write_text(html, encoding='utf-8') +Path('graphify-out/.graphify_sigma_data.json').unlink() +``` + +**Before telling the user it's done**, sanity-check the embedded JSON didn't break the ` + + +``` + +## Step 3 — inject the data and clean up + +```python +import json +from pathlib import Path + +data = json.loads(Path('graphify-out/.graphify_sigma_data.json').read_text(encoding='utf-8')) +html = Path('graphify-out/graph_sigma.html').read_text(encoding='utf-8') +html = html.replace('__GRAPH_DATA__', json.dumps(data, ensure_ascii=False)) +Path('graphify-out/graph_sigma.html').write_text(html, encoding='utf-8') +Path('graphify-out/.graphify_sigma_data.json').unlink() +``` + +**Before telling the user it's done**, sanity-check the embedded JSON didn't break the ` + + +``` + +## Step 3 — inject the data and clean up + +```python +import json +from pathlib import Path + +data = json.loads(Path('graphify-out/.graphify_sigma_data.json').read_text(encoding='utf-8')) +html = Path('graphify-out/graph_sigma.html').read_text(encoding='utf-8') +html = html.replace('__GRAPH_DATA__', json.dumps(data, ensure_ascii=False)) +Path('graphify-out/graph_sigma.html').write_text(html, encoding='utf-8') +Path('graphify-out/.graphify_sigma_data.json').unlink() +``` + +**Before telling the user it's done**, sanity-check the embedded JSON didn't break the ` + + +``` + +## Step 3 — inject the data and clean up + +```python +import json +from pathlib import Path + +data = json.loads(Path('graphify-out/.graphify_sigma_data.json').read_text(encoding='utf-8')) +html = Path('graphify-out/graph_sigma.html').read_text(encoding='utf-8') +html = html.replace('__GRAPH_DATA__', json.dumps(data, ensure_ascii=False)) +Path('graphify-out/graph_sigma.html').write_text(html, encoding='utf-8') +Path('graphify-out/.graphify_sigma_data.json').unlink() +``` + +**Before telling the user it's done**, sanity-check the embedded JSON didn't break the ` + + +``` + +## Step 3 — inject the data and clean up + +```python +import json +from pathlib import Path + +data = json.loads(Path('graphify-out/.graphify_sigma_data.json').read_text(encoding='utf-8')) +html = Path('graphify-out/graph_sigma.html').read_text(encoding='utf-8') +html = html.replace('__GRAPH_DATA__', json.dumps(data, ensure_ascii=False)) +Path('graphify-out/graph_sigma.html').write_text(html, encoding='utf-8') +Path('graphify-out/.graphify_sigma_data.json').unlink() +``` + +**Before telling the user it's done**, sanity-check the embedded JSON didn't break the ` + + +``` + +## Step 3 — inject the data and clean up + +```python +import json +from pathlib import Path + +data = json.loads(Path('graphify-out/.graphify_sigma_data.json').read_text(encoding='utf-8')) +html = Path('graphify-out/graph_sigma.html').read_text(encoding='utf-8') +html = html.replace('__GRAPH_DATA__', json.dumps(data, ensure_ascii=False)) +Path('graphify-out/graph_sigma.html').write_text(html, encoding='utf-8') +Path('graphify-out/.graphify_sigma_data.json').unlink() +``` + +**Before telling the user it's done**, sanity-check the embedded JSON didn't break the ` + + +``` + +## Step 3 — inject the data and clean up + +```python +import json +from pathlib import Path + +data = json.loads(Path('graphify-out/.graphify_sigma_data.json').read_text(encoding='utf-8')) +html = Path('graphify-out/graph_sigma.html').read_text(encoding='utf-8') +html = html.replace('__GRAPH_DATA__', json.dumps(data, ensure_ascii=False)) +Path('graphify-out/graph_sigma.html').write_text(html, encoding='utf-8') +Path('graphify-out/.graphify_sigma_data.json').unlink() +``` + +**Before telling the user it's done**, sanity-check the embedded JSON didn't break the ` + + +``` + +## Step 3 — inject the data and clean up + +```python +import json +from pathlib import Path + +data = json.loads(Path('graphify-out/.graphify_sigma_data.json').read_text(encoding='utf-8')) +html = Path('graphify-out/graph_sigma.html').read_text(encoding='utf-8') +html = html.replace('__GRAPH_DATA__', json.dumps(data, ensure_ascii=False)) +Path('graphify-out/graph_sigma.html').write_text(html, encoding='utf-8') +Path('graphify-out/.graphify_sigma_data.json').unlink() +``` + +**Before telling the user it's done**, sanity-check the embedded JSON didn't break the ` + + +``` + +## Step 3 — inject the data and clean up + +```python +import json +from pathlib import Path + +data = json.loads(Path('graphify-out/.graphify_sigma_data.json').read_text(encoding='utf-8')) +html = Path('graphify-out/graph_sigma.html').read_text(encoding='utf-8') +html = html.replace('__GRAPH_DATA__', json.dumps(data, ensure_ascii=False)) +Path('graphify-out/graph_sigma.html').write_text(html, encoding='utf-8') +Path('graphify-out/.graphify_sigma_data.json').unlink() +``` + +**Before telling the user it's done**, sanity-check the embedded JSON didn't break the ` safety check, search results ignoring active filters, and a camera-pan target using raw graph coordinates instead of sigma's own display-space coordinates. --- CHANGELOG.md | 1 + .../skills/agents/references/sigma-viz.md | 322 +++++++++++++++--- graphify/skills/amp/references/sigma-viz.md | 322 +++++++++++++++--- .../skills/claude/references/sigma-viz.md | 322 +++++++++++++++--- graphify/skills/claw/references/sigma-viz.md | 322 +++++++++++++++--- graphify/skills/codex/references/sigma-viz.md | 322 +++++++++++++++--- .../skills/copilot/references/sigma-viz.md | 322 +++++++++++++++--- graphify/skills/droid/references/sigma-viz.md | 322 +++++++++++++++--- graphify/skills/kilo/references/sigma-viz.md | 322 +++++++++++++++--- graphify/skills/kiro/references/sigma-viz.md | 322 +++++++++++++++--- .../skills/opencode/references/sigma-viz.md | 322 +++++++++++++++--- graphify/skills/pi/references/sigma-viz.md | 322 +++++++++++++++--- graphify/skills/trae/references/sigma-viz.md | 322 +++++++++++++++--- .../skills/vscode/references/sigma-viz.md | 322 +++++++++++++++--- .../skills/windows/references/sigma-viz.md | 322 +++++++++++++++--- ...__skills__agents__references__sigma-viz.md | 322 +++++++++++++++--- ...ify__skills__amp__references__sigma-viz.md | 322 +++++++++++++++--- ...__skills__claude__references__sigma-viz.md | 322 +++++++++++++++--- ...fy__skills__claw__references__sigma-viz.md | 322 +++++++++++++++--- ...y__skills__codex__references__sigma-viz.md | 322 +++++++++++++++--- ..._skills__copilot__references__sigma-viz.md | 322 +++++++++++++++--- ...y__skills__droid__references__sigma-viz.md | 322 +++++++++++++++--- ...fy__skills__kilo__references__sigma-viz.md | 322 +++++++++++++++--- ...fy__skills__kiro__references__sigma-viz.md | 322 +++++++++++++++--- ...skills__opencode__references__sigma-viz.md | 322 +++++++++++++++--- ...hify__skills__pi__references__sigma-viz.md | 322 +++++++++++++++--- ...fy__skills__trae__references__sigma-viz.md | 322 +++++++++++++++--- ...__skills__vscode__references__sigma-viz.md | 322 +++++++++++++++--- ..._skills__windows__references__sigma-viz.md | 322 +++++++++++++++--- .../fragments/references/shared/sigma-viz.md | 322 +++++++++++++++--- 30 files changed, 8005 insertions(+), 1334 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 60833c012..ed996c9e3 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,7 @@ Full release notes with details on each version: [GitHub Releases](https://githu ## Unreleased - Feat: Sigma.js + graphology fallback for large `graph.html` views, plus batched/reconciled community labeling at scale. vis-network's live client-side forceAtlas2 physics simulation is genuinely slow/laggy past a few hundred nodes regardless of hardware — verified in production on a 1083-community monorepo graph (apa, 2026-07-03). Step 6 of the skill now generates a separate WebGL Sigma.js render (`graph_sigma.html`) with the layout precomputed offline in Python (`nx.forceatlas2_layout`, no client-side physics) once the view exceeds ~300 nodes; the full recipe lives in the new `references/sigma-viz.md`, shipped to every split-platform skill variant. Step 5's community labeling also gains a batched-subagent flow for 100+ communities (sample each community's top-degree nodes, dispatch one `general-purpose` subagent per ~15-20-community batch in parallel) plus a required cross-batch label-reconciliation pass — a naive single-pass label run on the same graph left 338 of 1083 communities (31%) sharing a label with another before reconciliation. + The Sigma view also fixes a node-sizing bug (`autoRescale` must be disabled alongside `itemSizesReference: "positions"`, per sigma's own docs — otherwise sizes stay anchored to the pre-rescale coordinate frame and don't track zoom) and recalibrates node size against the layout's own median nearest-neighbor distance instead of a fixed constant, so communities stay legible instead of overlapping into a blob as node count grows. It now also surfaces each community's dominant content kind as an icon (code/document/paper/image/rationale/concept, via `@sigma/node-image`'s `NodePictogramProgram`) and dominant top-level module as color, plus a left-side panel to filter by entity kind and by relation type (grouped into calls/structure/imports/references/documentation-and-concepts/groups, covering both code and non-code relations) and a clickable module legend to isolate one part of the codebase at a time. - Fix: the `query` reference doc's inline vocab/fallback snippets now read and write files with `encoding="utf-8"` (#1619 A2, thanks @edtrackai). On Windows (default cp1252) the bare `read_text()`/`write_text()` calls crashed on exactly the cross-language corpora the doc demonstrates (e.g. Cyrillic labels like `обработчик`). Fixed across all generated skill variants. diff --git a/graphify/skills/agents/references/sigma-viz.md b/graphify/skills/agents/references/sigma-viz.md index 7cf0ba408..0eb161320 100644 --- a/graphify/skills/agents/references/sigma-viz.md +++ b/graphify/skills/agents/references/sigma-viz.md @@ -4,12 +4,15 @@ Load this when Step 6's `graph.html` (vis-network) would render more than ~300 n Output file: `graphify-out/graph_sigma.html` (self-contained, opens directly like `graph.html`). +Beyond the raw performance fix, this view also encodes three things vis-network's `graph.html` doesn't surface at a glance: each community's **dominant content kind** (code/document/paper/image/rationale/concept, drawn as a small icon), its **dominant module** (the top-level directory most of its members live under, drawn as color), and a **left-side filter panel** for both entity kind and relation type — so a code-heavy corpus with a handful of docs sprinkled in doesn't read as one undifferentiated blob. + ## Step 1 — build the meta-graph and precompute layout in Python Adjust `MIN_COMMUNITY_SIZE` (20 is a reasonable default — communities below this rarely appear in God Nodes / navigation and just add render cost) and `INPUT_PATH`/label source to match the current run. ```python import json +import math import networkx as nx from pathlib import Path from collections import Counter, defaultdict @@ -17,53 +20,137 @@ from collections import Counter, defaultdict g_data = json.loads(Path('graphify-out/graph.json').read_text(encoding='utf-8')) labels = json.loads(Path('graphify-out/.graphify_labels.json').read_text(encoding='utf-8')) +# Maps every relation graphify emits (AST-structural and LLM-semantic alike) to +# one of six buckets the filter panel toggles as a group. Keep this in sync +# with the relation vocabulary in graphify/extract.py if new relations are +# added there — an unmapped relation falls into 'other' rather than crashing. +RELATION_BUCKETS = { + 'calls': 'calls', 'indirect_call': 'calls', 'instantiates': 'calls', + 'contains': 'structure', 'defines': 'structure', 'method': 'structure', + 'implements': 'structure', 'inherits': 'structure', + 'imports': 'imports', 'imports_from': 'imports', 'dynamic_import': 'imports', + 're_exports': 'imports', 'depends_on': 'imports', 'crate_depends_on': 'imports', + 'requires_env': 'imports', + 'references': 'references', 'references_constant': 'references', 'uses': 'references', + 'uses_static_prop': 'references', 'bound_to': 'references', 'listened_by': 'references', + 'cites': 'docs', 'conceptually_related_to': 'docs', 'shares_data_with': 'docs', + 'semantically_similar_to': 'docs', 'rationale_for': 'docs', + 'participate_in': 'groups', 'implement': 'groups', 'form': 'groups', +} + +def top_level_dir(source_file: str) -> str: + if not source_file or '/' not in source_file: + return '(root)' + return source_file.split('/')[0] + MIN_COMMUNITY_SIZE = 20 +node_attrs = {n['id']: n for n in g_data['nodes']} comms = defaultdict(list) for n in g_data['nodes']: comms[n['community']].append(n['id']) significant = {cid: members for cid, members in comms.items() if len(members) >= MIN_COMMUNITY_SIZE} +dropped = len(comms) - len(significant) +if not significant: + # Every community is below the threshold (plausible on a corpus that + # clusters into many small communities) - fall back to all of them + # rather than crashing on an empty meta-graph downstream. + largest = max((len(m) for m in comms.values()), default=0) + print(f'No community reaches MIN_COMMUNITY_SIZE={MIN_COMMUNITY_SIZE} (largest: {largest}) - showing all {len(comms)} communities instead.') + significant = comms + dropped = 0 node_to_community = {m: cid for cid, members in significant.items() for m in members} meta = nx.Graph() for cid, members in significant.items(): - meta.add_node(cid, member_count=len(members), label=labels.get(str(cid), f'Community {cid}')) + # Dominant file_type (majority vote) drives the icon; dominant top-level + # directory drives the color. Both are approximations at the community + # level — a mixed community shows its majority kind/module, not a blend. + type_counts = Counter(node_attrs[m].get('file_type', 'code') for m in members) + dir_counts = Counter(top_level_dir(node_attrs[m].get('source_file', '')) for m in members) + meta.add_node( + cid, member_count=len(members), label=labels.get(str(cid), f'Community {cid}'), + file_type=type_counts.most_common(1)[0][0], module=dir_counts.most_common(1)[0][0], + ) edge_counts = Counter() -for link in g_data['links']: +edge_buckets = defaultdict(Counter) +# NetworkX <= 3.1 serializes edges as 'links'; some older graph.json files use +# 'edges' instead (same compatibility hazard graphify/build.py and +# graphify/affected.py already guard against) - accept either. +links_key = 'links' if 'links' in g_data else 'edges' +for link in g_data[links_key]: cu, cv = node_to_community.get(link['source']), node_to_community.get(link['target']) if cu is not None and cv is not None and cu != cv: - edge_counts[(min(cu, cv), max(cu, cv))] += 1 + key = (min(cu, cv), max(cu, cv)) + edge_counts[key] += 1 + edge_buckets[key][RELATION_BUCKETS.get(link.get('relation', ''), 'other')] += 1 + +# Hyperedges (3+ node group relations - participate_in/implement/form) carry +# no source/target, so they never appear in the loop above. Remap each to the +# distinct communities its members span and count it into every pairwise +# combination, mirroring how graphify/export.py's vis-network aggregated view +# remaps hyperedges to community IDs - otherwise 'groups' in RELATION_BUCKETS +# is unreachable and the filter panel's "Groups" checkbox always no-ops. +for he in g_data.get('hyperedges', []): + he_communities = sorted({node_to_community[m] for m in he.get('nodes', []) if m in node_to_community}) + for i in range(len(he_communities)): + for j in range(i + 1, len(he_communities)): + key = (he_communities[i], he_communities[j]) + edge_counts[key] += 1 + edge_buckets[key][RELATION_BUCKETS.get(he.get('relation', ''), 'groups')] += 1 + for (cu, cv), w in edge_counts.items(): - meta.add_edge(cu, cv, weight=w) + meta.add_edge(cu, cv, weight=w, buckets=dict(edge_buckets[(cu, cv)])) # offline layout — no client-side physics needed at all pos = nx.forceatlas2_layout(meta, max_iter=800, gravity=1.0, scaling_ratio=4.0, seed=42, weight='weight') degrees = dict(meta.degree()) -max_deg = max(degrees.values()) if degrees else 1 max_members = max((meta.nodes[n]['member_count'] for n in meta.nodes), default=1) xs = [float(p[0]) for p in pos.values()] ys = [float(p[1]) for p in pos.values()] xr, yr = (max(xs) - min(xs)) or 1, (max(ys) - min(ys)) or 1 +scaled = {n: ((float(pos[n][0]) - min(xs)) / xr * 1000, (float(pos[n][1]) - min(ys)) / yr * 1000) for n in meta.nodes()} + +# Density-aware sizing: a fixed absolute size range (e.g. "3 to 15") looks +# fine at a few dozen communities but overlaps into an unreadable blob once +# forceAtlas2 packs 300+ communities into the same normalized space — the +# same size value covers a much larger *fraction* of the available room as +# node count grows. Calibrate against the layout's own median nearest- +# neighbor distance instead, so sizes stay legible regardless of density. +def _nearest_neighbor_dists(points): + dists = [] + for i, (x1, y1) in enumerate(points): + best = min((math.hypot(x1 - x2, y1 - y2) for j, (x2, y2) in enumerate(points) if j != i), default=None) + if best is not None: + dists.append(best) + return dists + +nn = _nearest_neighbor_dists(list(scaled.values())) +median_nn = sorted(nn)[len(nn) // 2] if nn else 30.0 +SIZE_MIN = max(2.0, median_nn * 0.12) +SIZE_MAX = max(SIZE_MIN * 2, median_nn * 0.45) nodes_out = [] for n in meta.nodes(): - x, y = float(pos[n][0]), float(pos[n][1]) + x, y = scaled[n] deg, mc = int(degrees.get(n, 0)), int(meta.nodes[n]['member_count']) nodes_out.append({ 'key': str(n), 'label': meta.nodes[n]['label'], - 'x': round((x - min(xs)) / xr * 1000, 2), 'y': round((y - min(ys)) / yr * 1000, 2), - 'size': round(3 + 12 * (mc / max_members) ** 0.5, 2), - 't': round(deg / max_deg if max_deg else 0, 3), - 'members': mc, 'degree': deg, + 'x': round(x, 2), 'y': round(y, 2), + 'size': round(SIZE_MIN + (SIZE_MAX - SIZE_MIN) * (mc / max_members) ** 0.5, 2), + 'degree': deg, 'members': mc, + 'fileType': meta.nodes[n]['file_type'], 'module': meta.nodes[n]['module'], }) -edges_out = [{'source': str(u), 'target': str(v), 'weight': int(d.get('weight', 1))} - for u, v, d in meta.edges(data=True)] +edges_out = [ + {'source': str(u), 'target': str(v), 'weight': int(d.get('weight', 1)), 'buckets': d.get('buckets', {})} + for u, v, d in meta.edges(data=True) +] Path('graphify-out/.graphify_sigma_data.json').write_text( json.dumps({'nodes': nodes_out, 'edges': edges_out}, ensure_ascii=False), encoding='utf-8') -print(f'meta graph: {len(nodes_out)} nodes, {len(edges_out)} edges — layout precomputed') +print(f'meta graph: {len(nodes_out)} nodes, {len(edges_out)} edges — layout precomputed, size range {SIZE_MIN:.1f}-{SIZE_MAX:.1f}, {dropped} communities below MIN_COMMUNITY_SIZE={MIN_COMMUNITY_SIZE} dropped') ``` **Important**: cast every numpy value (`forceatlas2_layout` returns numpy floats) to plain Python `float`/`int` before `json.dumps` — numpy scalars aren't JSON-serializable and will raise `TypeError: Object of type float32 is not JSON serializable`. @@ -73,10 +160,14 @@ print(f'meta graph: {len(nodes_out)} nodes, {len(edges_out)} edges — layout pr Write this template to `graphify-out/graph_sigma.html`, with a literal `__GRAPH_DATA__` placeholder where the data goes (substituted in Step 3 — do NOT try to embed the JSON directly while authoring the template, string-templating that much escaping inline is error-prone). Key implementation notes: -- **Library loading**: sigma@3 ships CJS/ESM only, no browser UMD global. Load both libraries as ES modules from `esm.sh` (`https://esm.sh/sigma@3.0.3`, `https://esm.sh/graphology@0.25.4`) via ` tag). This used to be a check-after-the-fact step; Step 3 now escapes it unconditionally before embedding, since previews make the collision routine rather than a rare edge case. - The entity-properties panel, previously a separate floating box that could get clipped by the viewport independently of the main panel, is now nested inside the main sidebar - same width, same scroll, no separate truncation point. A parallel Sonnet review pass over the accumulated diff (this file grew quickly through many rapid incremental edits) found no functional bugs, just two cosmetic nitpicks (an unused data attribute, a stray character in placeholder text), both fixed. Verified end-to-end in a real browser (Chrome via Playwright): the label-background fix, merged panel layout, scrollable/filterable file list, and draggable preview dialog were all confirmed on a synthetic example (including an exact-coordinate drag verified via getNodeDisplayData, since eyeballed pixel coordinates missed targets repeatedly); the layout re-tune and script-tag auto-escape were confirmed on the real 20,097-node/1083-community production graph. --- CHANGELOG.md | 4 +- .../skills/agents/references/sigma-viz.md | 212 ++++++++++++++---- graphify/skills/amp/references/sigma-viz.md | 212 ++++++++++++++---- .../skills/claude/references/sigma-viz.md | 212 ++++++++++++++---- graphify/skills/claw/references/sigma-viz.md | 212 ++++++++++++++---- graphify/skills/codex/references/sigma-viz.md | 212 ++++++++++++++---- .../skills/copilot/references/sigma-viz.md | 212 ++++++++++++++---- graphify/skills/droid/references/sigma-viz.md | 212 ++++++++++++++---- graphify/skills/kilo/references/sigma-viz.md | 212 ++++++++++++++---- graphify/skills/kiro/references/sigma-viz.md | 212 ++++++++++++++---- .../skills/opencode/references/sigma-viz.md | 212 ++++++++++++++---- graphify/skills/pi/references/sigma-viz.md | 212 ++++++++++++++---- graphify/skills/trae/references/sigma-viz.md | 212 ++++++++++++++---- .../skills/vscode/references/sigma-viz.md | 212 ++++++++++++++---- .../skills/windows/references/sigma-viz.md | 212 ++++++++++++++---- ...__skills__agents__references__sigma-viz.md | 212 ++++++++++++++---- ...ify__skills__amp__references__sigma-viz.md | 212 ++++++++++++++---- ...__skills__claude__references__sigma-viz.md | 212 ++++++++++++++---- ...fy__skills__claw__references__sigma-viz.md | 212 ++++++++++++++---- ...y__skills__codex__references__sigma-viz.md | 212 ++++++++++++++---- ..._skills__copilot__references__sigma-viz.md | 212 ++++++++++++++---- ...y__skills__droid__references__sigma-viz.md | 212 ++++++++++++++---- ...fy__skills__kilo__references__sigma-viz.md | 212 ++++++++++++++---- ...fy__skills__kiro__references__sigma-viz.md | 212 ++++++++++++++---- ...skills__opencode__references__sigma-viz.md | 212 ++++++++++++++---- ...hify__skills__pi__references__sigma-viz.md | 212 ++++++++++++++---- ...fy__skills__trae__references__sigma-viz.md | 212 ++++++++++++++---- ...__skills__vscode__references__sigma-viz.md | 212 ++++++++++++++---- ..._skills__windows__references__sigma-viz.md | 212 ++++++++++++++---- .../fragments/references/shared/sigma-viz.md | 212 ++++++++++++++---- 30 files changed, 5020 insertions(+), 1132 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 157545380..687316d5b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,7 +7,9 @@ Full release notes with details on each version: [GitHub Releases](https://githu - Feat: Sigma.js + graphology fallback for large `graph.html` views, plus batched/reconciled community labeling at scale. vis-network's live client-side forceAtlas2 physics simulation is genuinely slow/laggy past a few hundred nodes regardless of hardware — verified in production on a 1083-community monorepo graph (apa, 2026-07-03). Step 6 of the skill now generates a separate WebGL Sigma.js render (`graph_sigma.html`) with the layout precomputed offline in Python (`nx.forceatlas2_layout`, no client-side physics) once the view exceeds ~300 nodes; the full recipe lives in the new `references/sigma-viz.md`, shipped to every split-platform skill variant. Step 5's community labeling also gains a batched-subagent flow for 100+ communities (sample each community's top-degree nodes, dispatch one `general-purpose` subagent per ~15-20-community batch in parallel) plus a required cross-batch label-reconciliation pass — a naive single-pass label run on the same graph left 338 of 1083 communities (31%) sharing a label with another before reconciliation. The Sigma view also fixes a node-sizing bug (`autoRescale` must be disabled alongside `itemSizesReference: "positions"`, per sigma's own docs — otherwise sizes stay anchored to the pre-rescale coordinate frame and don't track zoom) and recalibrates node size against the layout's own median nearest-neighbor distance instead of a fixed constant, so communities stay legible instead of overlapping into a blob as node count grows. It now also surfaces each community's dominant content kind as an icon (code/document/paper/image/rationale/concept, via `@sigma/node-image`'s `NodePictogramProgram`) and dominant top-level module as color, plus a left-side panel to filter by entity kind and by relation type (grouped into calls/structure/imports/references/documentation-and-concepts/groups, covering both code and non-code relations) and a clickable module legend to isolate one part of the codebase at a time. Two more critical rendering bugs were caught and fixed only by actually opening a generated file in a real browser (Chrome via Playwright), not by reading the code: (1) disabling `autoRescale` also disables sigma's automatic "fit the camera to the graph" on load, so without an explicit `fitViewportToNodes` call (from `@sigma/utils`) the camera pointed at empty space and the page rendered as a solid black screen with nothing visible; (2) the `@sigma/node-image` and `@sigma/utils` ESM imports failed at module-load time because esm.sh resolved their `sigma` peer dependency to a broken literal path instead of the pinned version — that import throws before any of the script runs, which is the more fundamental cause of the black screen (the camera fix alone wasn't sufficient). Both `@sigma/*` imports now pin the dependency explicitly (`?deps=sigma@3.0.3`). A third bug (icon SVGs missing explicit `width`/`height`, which `@sigma/node-image`'s raster path needs to size a data-URI icon) was caught by an independent second-opinion review before it shipped. Verified end-to-end against the real 20,097-node/1083-community production graph that motivated this feature: 342 aggregated communities render correctly with icons, module colors, and a populated filter panel, zero console errors. - A further usability pass, driven by feedback after actually using the rendered graph: (1) the default forceAtlas2 parameters packed 87% of nodes within 15% of the bounding-box center on the same real production graph — genuinely unreadable, not just suboptimal — fixed with `linlog=True`, a `node_size` repulsion halo proportional to member count, and a much higher `scaling_ratio`, dropping that to ~15-20% while keeping structurally related communities near each other; (2) edges were a flat gray regardless of relation — they're now colored AND labeled by their dominant relation bucket (`renderEdgeLabels: true`; sigma has no independent edge-label zoom threshold, but ties edge-label visibility to node-label visibility, which manages clutter automatically); (3) nodes are now draggable for manual layout tidying (sigma's own `downNode`/`moveBody`/`upNode`/`upStage` pattern, with the `setCustomBBox` freeze required to stop one dragged node from panning every other node), with a reset button that restores original positions; (4) the click panel now lists each community's actual representative source files (capped at 8, with a true total count), linked via `file://` when `.graphify_root` confirms the graph is being viewed on the machine that generated it; (5) `doc_ref` (a newer AST-derived file_type not in the original six-value schema) now gets an icon instead of silently miscategorizing as code; (6) the filter panel's entity-kind icons and relation-type rows now render visibly (white icons, not black-on-black; a color swatch per relation type matching its edge color). + A further usability pass, driven by feedback after actually using the rendered graph: (1) the default forceAtlas2 parameters packed 87% of nodes within 15% of the bounding-box center on the same real production graph — genuinely unreadable, not just suboptimal — fixed with `linlog=True`, a `node_size` repulsion halo proportional to member count, and a much higher `scaling_ratio`; (2) edges were a flat gray regardless of relation — they're now colored AND labeled by their dominant relation bucket (`renderEdgeLabels: true`; sigma has no independent edge-label zoom threshold, but ties edge-label visibility to node-label visibility, which manages clutter automatically); (3) nodes are now draggable for manual layout tidying (sigma's own `downNode`/`moveBody`/`upNode`/`upStage` pattern, with the `setCustomBBox` freeze required to stop one dragged node from panning every other node), with a reset button that restores original positions; (4) the click panel now lists each community's actual representative source files, linked via `file://` when `.graphify_root` confirms the graph is being viewed on the machine that generated it; (5) `doc_ref` (a newer AST-derived file_type not in the original six-value schema) now gets an icon instead of silently miscategorizing as code; (6) the filter panel's entity-kind icons and relation-type rows now render visibly (white icons, not black-on-black; a color swatch per relation type matching its edge color). + The layout spread was tuned twice: the first pass (`gravity=0.5, scaling_ratio=20`) measured better on a numeric spread proxy but was still reported as too bunched to read labels without manually dragging nodes apart — a second pass (`gravity=0.15, scaling_ratio=80`) actually fixes it, verified by reading real labels at the default fit-all zoom on the 342-community production graph, not by trusting the proxy metric a second time. Node labels also gained their own small background box (sigma's default label renderer is bare `fillText` with no settings-level background option; `defaultDrawNodeLabel` is the documented override point for a fully custom renderer) since plain white text was illegible over a dense, colorful tangle of edges. + The click panel was reworked again based on direct feedback: the file list is no longer capped with a "+N more" — Step 1 now sends every distinct file, rendered as a scrollable, filterable list (a filter textbox above a scrollable div, not a flat dump) since a 300-member community can have dozens of files. Clicking a file now opens a movable dialog (dragged by its header, plain screen-pixel CSS `left`/`top`, unrelated to sigma's graph/camera coordinate spaces) showing an embedded content preview — Step 1 reads a bounded preview (3000 characters) from disk for the first 8 files of each community's list, since embedding full content for every file in every community would bloat the self-contained HTML far more than paths alone do. Embedding real file content surfaced a new instance of the script-tag-collision risk: a previewed HTML/JS file (e.g. a Vite `index.html` with ` block and corrupt it. File previews make +# this a routine occurrence, not a rare edge case: any previewed HTML/JS/XML +# file (a Vite index.html, a block and corrupt it. File previews make +# this a routine occurrence, not a rare edge case: any previewed HTML/JS/XML +# file (a Vite index.html, a block and corrupt it. File previews make +# this a routine occurrence, not a rare edge case: any previewed HTML/JS/XML +# file (a Vite index.html, a block and corrupt it. File previews make +# this a routine occurrence, not a rare edge case: any previewed HTML/JS/XML +# file (a Vite index.html, a block and corrupt it. File previews make +# this a routine occurrence, not a rare edge case: any previewed HTML/JS/XML +# file (a Vite index.html, a block and corrupt it. File previews make +# this a routine occurrence, not a rare edge case: any previewed HTML/JS/XML +# file (a Vite index.html, a block and corrupt it. File previews make +# this a routine occurrence, not a rare edge case: any previewed HTML/JS/XML +# file (a Vite index.html, a block and corrupt it. File previews make +# this a routine occurrence, not a rare edge case: any previewed HTML/JS/XML +# file (a Vite index.html, a block and corrupt it. File previews make +# this a routine occurrence, not a rare edge case: any previewed HTML/JS/XML +# file (a Vite index.html, a block and corrupt it. File previews make +# this a routine occurrence, not a rare edge case: any previewed HTML/JS/XML +# file (a Vite index.html, a block and corrupt it. File previews make +# this a routine occurrence, not a rare edge case: any previewed HTML/JS/XML +# file (a Vite index.html, a block and corrupt it. File previews make +# this a routine occurrence, not a rare edge case: any previewed HTML/JS/XML +# file (a Vite index.html, a block and corrupt it. File previews make +# this a routine occurrence, not a rare edge case: any previewed HTML/JS/XML +# file (a Vite index.html, a block and corrupt it. File previews make +# this a routine occurrence, not a rare edge case: any previewed HTML/JS/XML +# file (a Vite index.html, a block and corrupt it. File previews make +# this a routine occurrence, not a rare edge case: any previewed HTML/JS/XML +# file (a Vite index.html, a block and corrupt it. File previews make +# this a routine occurrence, not a rare edge case: any previewed HTML/JS/XML +# file (a Vite index.html, a block and corrupt it. File previews make +# this a routine occurrence, not a rare edge case: any previewed HTML/JS/XML +# file (a Vite index.html, a block and corrupt it. File previews make +# this a routine occurrence, not a rare edge case: any previewed HTML/JS/XML +# file (a Vite index.html, a block and corrupt it. File previews make +# this a routine occurrence, not a rare edge case: any previewed HTML/JS/XML +# file (a Vite index.html, a block and corrupt it. File previews make +# this a routine occurrence, not a rare edge case: any previewed HTML/JS/XML +# file (a Vite index.html, a block and corrupt it. File previews make +# this a routine occurrence, not a rare edge case: any previewed HTML/JS/XML +# file (a Vite index.html, a block and corrupt it. File previews make +# this a routine occurrence, not a rare edge case: any previewed HTML/JS/XML +# file (a Vite index.html, a block and corrupt it. File previews make +# this a routine occurrence, not a rare edge case: any previewed HTML/JS/XML +# file (a Vite index.html, a block and corrupt it. File previews make +# this a routine occurrence, not a rare edge case: any previewed HTML/JS/XML +# file (a Vite index.html, a block and corrupt it. File previews make +# this a routine occurrence, not a rare edge case: any previewed HTML/JS/XML +# file (a Vite index.html, a block and corrupt it. File previews make +# this a routine occurrence, not a rare edge case: any previewed HTML/JS/XML +# file (a Vite index.html, a block and corrupt it. File previews make +# this a routine occurrence, not a rare edge case: any previewed HTML/JS/XML +# file (a Vite index.html, a block and corrupt it. File previews make +# this a routine occurrence, not a rare edge case: any previewed HTML/JS/XML +# file (a Vite index.html, a block and corrupt it. File previews make +# this a routine occurrence, not a rare edge case: any previewed HTML/JS/XML +# file (a Vite index.html, a