Docs overhaul: contract docstrings + rebuilt quickstart/tutorials/troubleshooting

.gitignore

@@ -9,6 +9,9 @@ wheels/
 # Virtual environments
 .venv
 
+# Built documentation (mkdocs output)
+site/
+
 # Experiment data (large, generated)
 experiments/google_and_the_mind/data/
 experiments/google_and_the_mind/graphs/

docs/api/stopwords.md

@@ -0,0 +1,3 @@
+# Stopwords
+
+::: kenon.stopwords.get_stopwords

docs/examples/comparing_two_texts.md

@@ -1,9 +1,38 @@
-# Comparing Two Texts
+# Comparing two texts
 
-See `examples/comparing_two_texts.py` for the full source.
+Build a co-occurrence graph for each of two texts on different topics, then
+quantify how much vocabulary they share. This is the building block for any
+graph-to-graph comparison (including comparing a text-derived network against
+human association norms — see the
+[word-association tutorial](../tutorials/word_association_graph.md)).
 
-This example demonstrates:
+## What it shows
 
 1. Building co-occurrence graphs for two different texts
-2. Computing Jaccard similarity of node sets
-3. Identifying vocabulary gaps between texts
+2. Computing the Jaccard similarity of their node sets
+3. Identifying the vocabulary gap (terms unique to each text)
+
+## Run it
+
+```bash
+python examples/comparing_two_texts.py
+```
+
+```text
+Text A graph: 28 nodes, 55 edges
+Text B graph: 30 nodes, 59 edges
+
+Jaccard similarity of node sets: 0.018
+
+Shared nodes (1): ['natural']
+Only in Text A (27): ['algorithm', 'architecture', ...]
+Only in Text B (29): ['biodiversity', 'cap', ...]
+
+Vocabulary gap: 56 unique terms out of 57 total
+```
+
+## Source
+
+```python title="examples/comparing_two_texts.py"
+--8<-- "examples/comparing_two_texts.py"
+```

docs/examples/multilingual_analysis.md

@@ -1,9 +1,45 @@
-# Multilingual Analysis
+# Multilingual analysis
 
-See `examples/multilingual_analysis.py` for the full source.
+kenon is language-agnostic: tokenisation is driven by whichever spaCy model you
+load, and stopwords can come from NLTK (many languages) or scikit-learn (English
+only). This example processes English and German text and compares graph density.
 
-This example demonstrates:
+## What it shows
 
-1. Tokenizing English and German texts
-2. Building co-occurrence graphs for each language
-3. Comparing graph density across languages
+1. Tokenising English and German texts with different spaCy models
+2. Selecting stopword sources per language (`sources=["nltk"]` for German)
+3. Building co-occurrence graphs for each and comparing density
+
+## Prerequisites
+
+The German half needs the German spaCy model. If it is missing, the script
+prints a clear message and skips that section rather than crashing:
+
+```bash
+python -m spacy download de_core_news_sm
+```
+
+## Run it
+
+```bash
+python examples/multilingual_analysis.py
+```
+
+```text
+Multilingual Co-occurrence Graph Analysis
+==================================================
+
+--- English ---
+  Tokens: 37, After filtering: 23
+  Graph: 22 nodes, 43 edges
+  Density: 0.1861
+  Top 5 nodes by degree: [('growth', 8), ('finding', 4), ('prestigious', 4), ('scientific', 4), ('journal', 4)]
+
+German analysis skipped: spaCy model 'de_core_news_sm' is not installed.
+```
+
+## Source
+
+```python title="examples/multilingual_analysis.py"
+--8<-- "examples/multilingual_analysis.py"
+```

docs/examples/news_article_analysis.md

@@ -1,12 +1,46 @@
-# News Article Analysis
+# News article analysis
 
-See `examples/news_article_analysis.py` for the full source.
+A complete walk-through of the core kenon workflow on a single news article:
+tokenise → remove stopwords → build a co-occurrence graph → extract the backbone
+→ rank words by centrality → compare two semantic-graph embedders.
 
-This example demonstrates:
+## What it shows
 
-1. Tokenizing a news article with lemmatization
+1. Tokenising a news article with lemmatisation
 2. Removing stopwords
-3. Building a co-occurrence graph with window=3
-4. Extracting the backbone
-5. Finding top nodes by degree centrality
-6. Building semantic graphs with CountVectorizer and TF-IDF
+3. Building a co-occurrence graph (`window=3`)
+4. Extracting the backbone with the disparity filter
+5. Finding the top nodes by degree centrality
+6. Building semantic graphs with `CountVectorizerEmbedder` and `TfidfEmbedder`
+
+## Run it
+
+```bash
+python examples/news_article_analysis.py
+```
+
+```text
+Total tokens: 158
+Tokens after stopword removal: 95
+
+Co-occurrence graph: 78 nodes, 274 edges
+Backbone: 4 nodes, 6 edges
+
+Top 10 nodes by degree centrality (backbone):
+  growth: 1.000
+  sluggish: 1.000
+  recent: 1.000
+  quarter: 1.000
+
+--- CountVectorizer Semantic Graph ---
+Nodes: 81, Edges: 555
+
+--- TF-IDF Semantic Graph ---
+Nodes: 81, Edges: 555
+```
+
+## Source
+
+```python title="examples/news_article_analysis.py"
+--8<-- "examples/news_article_analysis.py"
+```

docs/index.md

@@ -17,9 +17,11 @@ No neural models or external training data required.
 
 ## Quick links
 
-- [Quickstart](quickstart.md)
-- [API Reference](api/tokenizer.md)
-- [Examples](examples/news_article_analysis.md)
+- [Quickstart](quickstart.md) — clean install to backbone graph
+- [Tutorial: word-association graphs](tutorials/word_association_graph.md) — find paths between concepts
+- [Examples](examples/news_article_analysis.md) — full, runnable scripts
+- [Troubleshooting](troubleshooting.md) — common errors and fixes
+- [API Reference](api/tokenizer.md) — every public function and its contract
 
 ---
 

docs/quickstart.md

@@ -1,25 +1,87 @@
 # Quickstart
 
+This page takes you from a clean install to a backbone graph in one sitting.
+Every snippet below is runnable as-is.
+
 ## Installation
 
 ```bash
 uv add kenon
 python -m spacy download en_core_web_sm
 ```
 
-## Basic usage
+kenon uses a spaCy model for tokenisation. `en_core_web_sm` is the small English
+model used throughout the docs; the download step is required the first time.
+
+!!! note "Building from source"
+    kenon depends on [`chronowords`](https://pypi.org/project/chronowords/),
+    which compiles a Cython extension. If `uv add` fails with
+    `fatal error: Python.h: No such file or directory`, see the
+    [Troubleshooting](troubleshooting.md) page.
+
+## Build a co-occurrence graph
+
+A co-occurrence graph connects tokens that appear near each other within a
+sliding window. Nodes are tokens; edge weights are normalised co-occurrence
+frequencies.
 
 ```python
 from kenon import Tokenizer, get_stopwords, build_cooccurrence_graph, extract_backbone
+import networkx as nx
 
-# Tokenize
+text = (
+    "The central bank held interest rates steady on Thursday. "
+    "Inflation in the euro zone was declining but remained too high. "
+    "Financial markets reacted positively as bond yields fell and stocks rose. "
+    "Analysts said the rate-hiking cycle had likely peaked."
+)
+
+# 1. Tokenise (lemmatise so "rates"/"rate" collapse to one node)
 tokenizer = Tokenizer("en_core_web_sm", lemmatize=True)
-tokens = tokenizer.flat_tokens("Your text here.")
+tokens = tokenizer.flat_tokens(text)
 
-# Build graph
+# 2. Build the co-occurrence graph, dropping stopwords first
 stopwords = get_stopwords("english")
-graph = build_cooccurrence_graph(tokens, window=2, stopwords=stopwords)
+graph = build_cooccurrence_graph(tokens, window=3, stopwords=stopwords)
+print(f"co-occurrence graph: {graph.number_of_nodes()} nodes, {graph.number_of_edges()} edges")
+```
+
+## Extract the backbone
+
+Real co-occurrence graphs are dense and noisy. The **disparity filter**
+([Serrano et al. 2009](https://arxiv.org/pdf/0904.2389.pdf)) keeps only edges
+that are statistically significant relative to each node's other connections.
 
-# Extract backbone
-backbone = extract_backbone(graph, min_alpha_ptile=0.3, min_degree=2)
+```python
+# 3. Keep the statistically significant edges
+backbone = extract_backbone(graph, min_alpha_ptile=0.3, min_degree=1)
+print(f"backbone: {backbone.number_of_nodes()} nodes, {backbone.number_of_edges()} edges")
+
+# 4. A backbone graph is a plain networkx.Graph — use the whole networkx toolkit
+top = sorted(nx.degree_centrality(backbone).items(), key=lambda kv: kv[1], reverse=True)[:5]
+for word, score in top:
+    print(f"  {word}: {score:.2f}")
+```
+
+Running the two snippets above prints something like:
+
+```text
+co-occurrence graph: 27 nodes, 78 edges
+backbone: 6 nodes, 6 edges
+  bank: 0.40
+  central: 0.40
+  hold: 0.40
+  cycle: 0.40
+  likely: 0.40
 ```
+
+!!! tip "`min_alpha_ptile` controls aggressiveness"
+    Higher values prune more edges. On short texts like this one, start low
+    (`0.3`) and `min_degree=1`; on book-length corpora, raise both.
+
+## Where to next
+
+- [Word-association graphs tutorial](tutorials/word_association_graph.md) — find
+  paths between two concepts.
+- [Examples](examples/news_article_analysis.md) — full, runnable scripts.
+- [API Reference](api/tokenizer.md) — every public function and its contract.

docs/troubleshooting.md

@@ -0,0 +1,102 @@
+# Troubleshooting
+
+Common errors and how to resolve them. Each entry shows the failure, why it
+happens, and the fix.
+
+## `RuntimeError: spaCy model '...' is not installed`
+
+```text
+RuntimeError: spaCy model 'en_core_web_sm' is not installed.
+Run: python -m spacy download en_core_web_sm
+```
+
+`Tokenizer` loads its spaCy model lazily on first use, so this surfaces the
+first time you call `flat_tokens`, `tokenize`, or `sentencize` — not at
+construction time. Install the model named in the message:
+
+```bash
+python -m spacy download en_core_web_sm
+```
+
+For other languages, pass the model name to `Tokenizer(...)` and download the
+matching model (e.g. `de_core_news_sm` for German).
+
+## `fatal error: Python.h: No such file or directory`
+
+This appears while installing kenon, during the build of its transitive
+dependency [`chronowords`](https://pypi.org/project/chronowords/), which
+compiles a Cython extension and needs the Python development headers. System
+Pythons on most Linux distros don't ship them by default.
+
+Either install the headers for your system Python:
+
+```bash
+sudo apt install python3.12-dev   # Debian/Ubuntu; match your version
+```
+
+…or use a uv-managed Python, which bundles them:
+
+```bash
+uv python install 3.11
+uv sync --all-extras --python 3.11
+```
+
+## `ImportError: chronowords is required for PMIEmbedder`
+
+`PMIEmbedder` depends on `chronowords` (it ships as a core dependency, but can be
+absent in a stripped environment). Install it:
+
+```bash
+uv add chronowords
+```
+
+`CountVectorizerEmbedder` and `TfidfEmbedder` have no such dependency — use them
+if you only need count- or TF-IDF-based embeddings.
+
+## `ValueError: sklearn stopwords are only available for English`
+
+```python
+get_stopwords("german")  # ValueError
+```
+
+scikit-learn ships an English-only stopword list. For other languages, restrict
+the sources to NLTK:
+
+```python
+get_stopwords("german", sources=["nltk"])
+```
+
+## `ValueError: Unsupported stopword sources`
+
+`get_stopwords(..., sources=[...])` accepts only `"nltk"` and `"sklearn"`. Any
+other name (typo, wrong list) raises. Pass one or both of the supported names.
+
+## An empty or tiny backbone
+
+`extract_backbone` returns an **empty graph** when the input has no edges, and a
+small backbone is expected for short inputs — the disparity filter is designed to
+discard everything that isn't statistically significant. If you get fewer nodes
+than you want:
+
+- Lower `min_alpha_ptile` (e.g. `0.3`) to keep more edges.
+- Lower `min_degree` to `1` so weakly connected nodes survive.
+- Use a larger corpus — the filter needs enough edges per node to judge
+  significance.
+
+## `NodeNotFound` when finding paths
+
+`networkx.shortest_path(graph, src, dst)` raises `NodeNotFound` if either word is
+absent from the graph — usually because it was removed as a stopword, never
+occurred, or fell below `min_weight`. Guard before querying:
+
+```python
+if src in graph and dst in graph and nx.has_path(graph, src, dst):
+    path = nx.shortest_path(graph, src, dst, weight="distance")
+```
+
+## Loading a graph fails or behaves oddly
+
+`save_graph` / `load_graph` must use the **same `fmt`** on both ends. `graphml`
+and `gml` round-trip attributes safely and are human-readable;
+`pickle` is fastest but **executes arbitrary code on load** — never load a
+pickle file from an untrusted source. Prefer `graphml` for anything you share.