`cluster-only`/`label` never track token/cost usage from community-labeling LLM calls — `tokens = {"input": 0, "output": 0}` is a hardcoded placeholder, not an accumulator

[sub4biz]

Summary

graphify extract prints a useful tokens: X in / Y out, est. cost (~backend): $Z summary at the end —
but that summary covers only the extraction phase. cluster-only (and label) make their own LLM calls
to name communities (generate_community_labels → label_communities), and those calls are real,
metered API usage — but the token/cost total for them is never computed, tracked, or printed anywhere.

We measured this directly: for one full rebuild, graphify extract's own printed total was 6,107,552
input / 1,541,995 output tokens (7,649,547 combined). The DeepSeek dashboard's actual usage delta for the
whole rebuild (extract + cluster-only together) was 13,238,748 tokens — a 5,589,201-token gap,
attributable to cluster-only, equal to 42% of the true total spend (or, phrased the other way, 73%
on top of what extract alone reported — either framing describes the same 5.59M-token difference, cited
here explicitly to avoid ambiguity about which base the percentage is taken against). cluster-only
printed zero token information despite clearly making billed API calls (community names in the output
changed from Community N placeholders to real LLM-generated names, confirming the calls happened and
succeeded). This isn't a rounding error or an estimation quirk — it's a completely un-instrumented code
path.

Evidence

cluster-only's entire terminal output for a run that generated real community names:

Loading existing graph...
Graph: 154631 nodes, 235313 edges
Re-clustering...
Labeling communities...
[graphify] backed up semantic graph (5 files) -> 2026-07-05/
Skipped graph.html: Graph has 154631 nodes - too large for HTML viz (limit: 5000). Use --no-viz, raise
GRAPHIFY_VIZ_NODE_LIMIT, or reduce input size.
Done - 16995 communities. GRAPH_REPORT.md and graph.json updated.

No mention of tokens, cost, or API calls anywhere — compare to extract's own output, which ends with
tokens: 6,107,552 in / 1,541,995 out, est. cost (~deepseek): $1.2868.

Root cause

__main__.py:3651 (inside the cluster-only/label branch, right after the labeling step completes):

stages.mark("label")
questions = suggest_questions(G, communities, labels)
tokens = {"input": 0, "output": 0}

This is a hardcoded placeholder, not a running total updated from the actual LLM calls made just
above it. Tracing why: generate_community_labels() (llm.py:2435-2445) has the signature

def generate_community_labels(...) -> tuple[dict[int, str], str]:

— it returns (labels, source) only. Its usage internally calls label_communities(), which in turn
calls the same _call_openai_compat/backend-dispatch functions that extract uses (the ones that
populate result["input_tokens"]/result["output_tokens"] per call, per llm.py:1048-1049) — but nothing
in the call chain from label_communities() back up to generate_community_labels() back up to the
cluster-only branch in __main__.py accumulates those per-call token counts into a running total. The
information exists at the lowest level (each individual API response) and is simply never summed or
propagated upward — tokens = {"input": 0, "output": 0} is the visible symptom of that missing plumbing,
not the actual bug location.

Impact

Anyone estimating or comparing the cost of a graphify extract + cluster-only full rebuild from the
tool's own output will silently under-count by whatever fraction of the corpus's communities needed fresh
LLM labeling — in our measurement, 42% of the total token spend for that specific run. This is easy to miss
because the shape of the output (a clean tokens: X in / Y out line) looks complete and authoritative on
its own; there's no visual cue that it only covers part of the pipeline.

Suggested fix

1. Thread an accumulator through label_communities()/generate_community_labels() the same way
   extract's corpus-processing loop already does — sum result["input_tokens"]/result["output_tokens"]
   across every backend call made during labeling, return it alongside (labels, source).
2. Replace the __main__.py:3651 hardcoded tokens = {"input": 0, "output": 0} with the real accumulated
   value, and print the same tokens: X in / Y out, est. cost (~backend): $Z line at the end of
   cluster-only/label that extract already prints — same format, so users can add the two numbers
   together without guessing whether either one is already inclusive of the other.
3. Optional: for the common extract + cluster-only back-to-back pattern, a combined total across both
   commands would remove any need to add the two numbers manually — not essential if (1) and (2) land,
   just a convenience on top.

Reproduction

Not corpus-specific — reproduces on any graph with communities that need fresh labeling: run
graphify extract . followed by graphify cluster-only . --backend <any> --model <any> with a real API
key configured, on any corpus large enough to produce more than a couple of communities. Compare the
provider's own usage dashboard (before/after the cluster-only call, no other API activity in between)
against cluster-only's terminal output — the dashboard will show non-zero usage, the terminal output
will show none. No specific corpus content is required to observe the missing instrumentation itself,
only to measure the exact magnitude of the gap (which scales with community count and label-batch size).

Status check (2026-07-06, re-verified against 0.9.7)

Checked directly against the installed 0.9.7 source, not just the release notes. All three points in the
call chain are unchanged: the hardcoded tokens = {"input": 0, "output": 0} is still there
(__main__.py:3651, moved from 3645 in 0.9.6 — same line, same logic); generate_community_labels()
(llm.py:2435) still returns tuple[dict[int, str], str] with no token field, and its body still calls
label_communities(...) and returns labels, "llm" directly; label_communities() itself (llm.py:2338)
has no reference to tokens anywhere in its body — the per-call result["input_tokens"]/
result["output_tokens"] set in _call_openai_compat (llm.py:1048-1049) never leaves that function.
Not mentioned as fixed in the 0.9.7 release notes either.

Environment

• graphifyy 0.9.6, deepseek-v4-flash backend, at the time of the original measurement
• Measured via DeepSeek's own usage dashboard delta (before/after screenshots bracketing a single
  extract + cluster-only run pair on a frozen corpus, no other API activity in between) compared
  against extract's printed tokens: line

[safishamsi]

Fixed on v8 (21b851b), shipping in the next release.

You were right: tokens = {"input": 0, "output": 0} in the cluster-only path was a hardcoded placeholder, so the report's "Token cost" line always read 0 input * 0 output regardless of the labeling LLM calls. _call_llm now accumulates per-response usage into an optional accumulator that is threaded through _label_batch_with_retry -> label_communities -> generate_community_labels and surfaced in GRAPH_REPORT.md. Tokens are counted per batch (including a batch that fails to parse, since it was still billed) and merged on the main thread to avoid a data race across worker threads.

Note: backends that do not return usage in their response (currently the Claude Code CLI path) contribute nothing rather than an estimate, which keeps the number honest. Added regression tests for multi-batch accumulation and the failed-batch case. Thanks for the precise diagnosis.