feat: persistent RAG, source viewer, multi-turn chat, LLM hardening#1
Open
MaxQian888 wants to merge 21 commits into
Open
feat: persistent RAG, source viewer, multi-turn chat, LLM hardening#1MaxQian888 wants to merge 21 commits into
MaxQian888 wants to merge 21 commits into
Conversation
- README compare table: deepwiki-open uses FAISS + local files, not PostgreSQL. - model alias table now points at stable, real model IDs litellm recognises (Claude 4.5 family, GPT-4o, Gemini 1.5, DeepSeek, Qwen 2.5 via openrouter, Kimi via Moonshot, GLM-4 Plus). - add test_config.py covering alias resolution, env-var override, and fallback to provider keys.
Both expose existing config knobs that previously had no CLI surface. - --concurrency / REPOWIKI_CONCURRENCY: parallel LLM calls (default 5) - --max-context-tokens / REPOWIKI_MAX_CONTEXT_TOKENS: budget for the project-wide prompt slice; consumed by analyzer in a follow-up commit. Default 32000. Tests cover env-var parsing, invalid-value fallback, and verify the flags appear in `repowiki scan --help`.
Before, a 429 / network error returned the literal string "[LLM Error: ...]" from LLMClient. Downstream JSON parsing then silently failed and the user saw an empty wiki page with no clue why. - LLMClient.complete / .stream now raise LLMError(cause=original). - Analyzer catches LLMError per pass (overview, module, arch, guide), records to analyzer.errors, emits a "[error] ..." progress event, and returns a placeholder so the rest of the pipeline finishes. - CLI prints a yellow "Some analysis steps failed" summary at the end with a hint that successful sections are cached. Tests cover both the raise contract and the per-module degradation.
Before, a single tree_hash = sha(file_tree + every key file body) keyed all four cache slots (overview / module / arch / guide). Editing one source file invalidated all of them on re-scan. Now: - structure_hash = sha(sorted "path:size" pairs). Stable across body edits, only changes when files are added / removed / resized. Used by architecture and reading-guide passes. - overview_hash = structure_hash + bodies of README, pyproject, package.json, Cargo.toml, go.mod -- the files that genuinely change the elevator pitch. - module cache already keys by per-module content hash (unchanged). Net effect: a typical "fix a bug in one module" re-scan now only re-runs that one module's prompt, not 4 prompts. With a 10-module repo that's a 70%+ token saving on iterative re-scans.
Before: every config + entrypoint was dumped into the overview / arch / guide prompts, capped only at 4 KB per file. A monorepo with 20 config files easily blew past gpt-4o-mini's 128k window. Now: Analyzer takes max_context_tokens (default 32000, 0 = unlimited). _build_key_files_context() orders candidates by tier (config > entrypoint > rest) and PageRank within each tier, then fills until the budget is exhausted. Files that don't fit get a one-line stub so the LLM at least knows the file exists. Token counting prefers tiktoken cl100k_base when installed (it's a litellm transitive dep, so usually present), falling back to chars/4. The CLI surfaces this via --max-context-tokens.
A module like src/repowiki/core/ holding 7+ unrelated files used to become a single 'core' prompt, which produced shallow LLM output because the model couldn't pattern-match a unifying purpose. _group_into_modules() now takes split_threshold (default 10). Any top-level bucket above the threshold is recursively split by its next directory level: src/repowiki/core/* and src/repowiki/llm/* become two separate modules 'repowiki/core' and 'repowiki/llm'. Splits that degenerate to a single bucket (all files share the next segment) revert to the original name to avoid pointless renames like 'repowiki/_root'.
Frontends commonly import via aliases like 'import x from "@/lib/foo"'
where @/* maps to src/* in tsconfig.json. Without alias resolution the
PageRank graph saw zero edges from any modern Next.js / Vite project,
so 'core files by importance' was just noise on the frontend.
- _strip_jsonc(): remove // and /* */ comments and trailing commas so
tsconfig.json (which is JSONC by convention) parses with stdlib json.
- _load_ts_aliases(): walk project files for tsconfig.json /
jsconfig.json / tsconfig.*.json, extract compilerOptions.baseUrl +
paths, anchor target patterns to project-root-relative paths.
- _apply_ts_alias(): expand "@/foo/bar" against {"@/*": ["src/*"]}
back to ["src/foo/bar"].
- _resolve_import(): also try aliased candidates for JS/TS imports;
normalize candidate paths to forward slashes so Windows-built paths
match POSIX-stored project paths (this also fixed a latent bug where
pre-existing TS resolution failed on Windows).
The README has been advertising terminal Q&A since v0.1, but the CLI
just printed 'coming soon'. The web /api/.../chat endpoint already
had RAG + streaming working -- this just wires the same pieces to
the terminal:
$ repowiki chat .
> how does scan filter binary files?
Sources:
src/repowiki/core/scanner.py:113-114 (score 0.42)
...
<streamed answer>
- ingests local path or git URL (same code as `scan`)
- builds SimpleRAG index up-front
- retrieves top-K chunks per question, prints sources, streams reply
- handles LLMError / KeyboardInterrupt / 'exit' gracefully
- requires REPOWIKI_API_KEY (or provider key) before scanning so we
fail fast instead of cloning a 100 MB repo to find no key
--top-k, --model, --lang exposed; defaults match the web router.
Make CI / commit-hook usage cheap: 'repowiki scan . --since main' only re-runs the LLM for modules whose files actually changed since the named ref. Modules whose files are all unchanged emit a '[skip]' progress event and a 'skipped, unchanged since prior run' placeholder ModuleDoc. - ingest/git_diff.py: changed_paths_since(repo, ref) collects committed (ref...HEAD), working-tree, staged, and untracked paths via git diff / ls-files. Returns empty set if not a git repo or git is missing -- caller falls back to full scan. - Analyzer takes changed_paths: set[str] | None. None disables incremental mode (default). Set means: skip modules disjoint from the set. - CLI --since wires to changed_paths_since(); URL scans warn that --since is local-only and proceed with full analysis. - Skipped modules summarised at end of run. For a 50-module repo where one PR touches 2 modules, this drops LLM cost from ~50 module prompts to 2.
Symbol.line was already in the model but neither the prompt nor the builder used it, so module pages just listed bare names. Now: - build_module_prompt() asks the LLM to fill the line field on every key_symbol (1-based, 0 = unknown). - WikiBuilder._build_module_page renders 'src/foo.py:42' next to the symbol when line > 0; when line == 0 it falls back to the bare name. README updated to show all the new CLI flags now actually working (--since, -c, chat without 'coming soon').
Drop unused imports / future-annotations and re-sort import blocks that the new test files introduced. No behaviour change.
Before: 'pip install repowiki[web] && repowiki serve' on a fresh
clone served a blank page because src/repowiki/server/static/ is
gitignored and nothing built it.
- hatch_build.py: custom Hatch BuildHookInterface that runs
'npm ci && npm run build' on every wheel/sdist build. Skips if
npm is missing (warn + continue), reuses existing static/ if
it's already fresh (saves ~30s on rebuilds).
- pyproject.toml:
* register the hook with [tool.hatch.build.hooks.custom]
* force-include src/repowiki/server/static -> repowiki/server/static
so wheel ships static/ even though .gitignore excludes it
* explicit sdist include list so source builds also work
- cli.py 'serve' detects missing static/index.html at runtime and
prints actionable help (npm run build OR pip install repowiki[web]).
Verified locally: python -m build --wheel produces a 119-entry
wheel containing 84 static files (index.html + assets/*.js).
Lets downstream repos add this to .github/workflows/wiki.yml:
- uses: he-yufeng/RepoWiki/.github/actions/repowiki-scan@main
with:
output: docs/wiki
since: origin/${{ github.event.pull_request.base.ref }}
api-key: ${{ secrets.DEEPSEEK_API_KEY }}
The action installs repowiki, runs `repowiki scan` with all the
flags this branch added (since, concurrency, max-context-tokens,
model, language, format, output), and exposes a `changed-files`
output so downstream steps can decide whether to commit.
- .github/actions/repowiki-scan/action.yml: composite action with
11 inputs, mapped 1:1 to CLI flags.
- .github/workflows/wiki-on-pr.yml: full PR-bot example that
scans incrementally on PR open/sync and pushes the regenerated
markdown back to the PR branch.
- README: 'GitHub Action' section showing the minimal embed.
- tests/test_github_action.py: validates the YAMLs parse, every
declared input is actually referenced, and every --flag the
action passes is one `repowiki scan --help` accepts (so renames
on either side break loudly in CI).
The default `repowiki build` writes its output into ./wiki and a sibling wiki.zip. Both are derived artefacts and shouldn't track into version control. Likewise, src/repowiki/server/static.bak is a one-off backup of the bundled frontend that a developer can have on disk while iterating but should never end up in a commit.
The chat RAG used to rebuild from scratch on every server start, paying O(n) tokenisation even when nothing in the project had changed. This commit overhauls the index along three axes: - **BM25 scoring** replaces the previous bag-of-words count. Length normalisation (`b`) and term-saturation (`k1`) are both exposed via Config so users can tune retrieval to repo shape without code edits. - **Incremental upsert**: chunks now carry a per-file SHA so a repeat scan only re-tokenises files that genuinely changed; the rest stay untouched. `upsert_file` and `remove_file` keep the global posting list aligned and `rebuild_global()` consolidates after batches. - **SQLite snapshot** (rag_store.py) persists chunks, postings, and the per-file SHA table to ~/.repowiki/indexes.db. Cold-start reloads in milliseconds; corruption is contained via a schema version stamp that triggers a clean rebuild rather than silently misinterpreting tables. Wiki-page chunks live under the same store with a `kind="wiki"` flag so chat questions about the generated architecture page hit the page text directly. The kind flag is opt-in via REPOWIKI_RAG_INDEX_WIKI. All knobs (chunk size, overlap, top-k, min-score, k1/b, wiki indexing) are read once at startup from env vars with permissive parsing -- a typo falls back to the prior default rather than crashing on boot. Test coverage: - test_rag.py: tokenisation, identifier splitting (camelCase/snake_case), stop-word filtering, chunk boundary heuristics, basic retrieval. - test_rag_bm25.py: scoring normalisation, min-score filtering, upsert / remove correctness, wiki-kind handling, markdown chunk splitting. - test_rag_store.py: save/load round-trips, schema versioning, project isolation, snapshot replacement.
Two reliability fixes for generated wiki content. **Mermaid sanitizer** (core/mermaid.py): LLMs occasionally emit Mermaid that the renderer rejects -- missing diagram-type prefix, parentheses inside node IDs, or a sequence body claimed as a flowchart. The sanitizer patches the common cases (auto-prepend `graph TD`, rewrite non-identifier node IDs, reject kind mismatches) and returns `None` when the input is hopeless. WikiBuilder then either renders the fixed diagram or falls back to a text description via `describe_components` instead of leaving a blank box on the page. Deliberately not a real Mermaid parser -- this is a forgiving best-effort fixer covering the patterns we actually see in LLM output. **Path validation in WikiBuilder**: the LLM sometimes invents file references that don't exist in the project. We now intersect every `module.files`, `component.files`, `reading-guide step.files`, and relationship endpoint against the project's real path set. Dropped references are surfaced via `builder.warnings`, which the CLI prints at the end of a build so the user knows the LLM hallucinated and can re-run with a different model. Test coverage: - test_mermaid.py: missing-header autoprepend, sequence/flowchart mismatch rejection, node-id rewrite, empty-input handling, the text fallback renderer.
…rompt
Three related improvements to the prompt layer and analyzer.
**Chat history threading** (prompts.build_chat_prompt): the chat prompt
now accepts a list of prior `{role, content}` turns. `_trim_history`
keeps the most recent N user+assistant pairs (default 5) so multi-turn
coherence works without unbounded token growth -- the trim walks
backward so the most recent turns are always retained.
**JSON repair retry** (prompts.missing_required_keys +
build_repair_prompt): when the LLM emits invalid JSON or omits required
fields, we now ship a one-shot corrective turn that pins the prior
context and explains exactly which keys are missing. The retry is
capped at one round at the call site to keep cost bounded, but recovers
the common "extra prose around the JSON" failure mode cleanly.
**Architecture prompt no longer ships the file tree**: re-sending the
tree alongside key files mostly served as a hallucination vector --
the LLM would invent components that weren't represented by any
analyzed module. Replaced with a deterministic `module_summaries`
block (name + purpose + file count, sorted by name for stable hashing)
and a system-prompt rule that components must be derived from the
provided modules.
**Analyzer concurrency** (core/analyzer.py): module-level analysis now
runs in a bounded async pool sized by `--concurrency`, with a batched
tiktoken `encode_batch` call for the per-file token estimates instead
of per-string overhead in a loop. PageRank ordering from the dependency
graph drives the order in which we surface key files to the LLM (most
central first), so prompts hit the architecturally relevant files even
when the project is much larger than the prompt budget.
Test coverage:
- test_prompts.py: history filtering / trimming, repair-prompt context,
arch prompt no longer contains file tree, blank-string detection.
- test_analyzer_concurrency.py: bounded parallelism, exception
propagation, deterministic ordering with rankings.
**Incremental scan over HTTP** (scan.py + models.py): the scan request
now accepts an optional `since` git ref. The router resolves it into a
changed_paths set via `changed_paths_since`, mirroring the CLI flag.
Cheap fallbacks for the non-happy path: GitHub URL scans stay full
(would need a double clone to diff), an empty diff falls back to full
re-analysis, and a git error logs a warning instead of aborting.
After analysis the scan task fires `_preheat_rag()` as a background
task: it tries to reload an on-disk index, diffs file SHAs against
ProjectContext, re-chunks only what genuinely changed, drops files that
disappeared, then refreshes wiki-page chunks from the freshly built
markdown. The chat router never has to pay the indexing cost on the
request thread.
**Source snippet endpoint** (wiki.py): `GET /project/{id}/files/{path}`
now accepts `start` and `end` query params (1-based, inclusive). When
both are supplied the response includes a pre-sliced `snippet` with 10
lines of context on each side plus highlight markers, so the frontend
SourceView can render a focused view without re-parsing the full body
on the client (which is slow for 5k+ line files).
**Robust SSE chat** (chat.py): the chat endpoint hardens streaming in
several ways:
- 15s heartbeat comments (`: heartbeat\n\n`) between provider chunks
so reverse proxies / gateways don't tear down idle connections during
long first-token waits (Claude with thinking, etc.).
- LLM errors and "no API key" now stream a one-shot error frame with
the same content-type as success, so the frontend's existing parser
surfaces them in the UI instead of swallowing a 500.
- Multi-turn history threads through via `req.history`.
- Structured INFO logging on retrieve_ms, input/output tokens, cost,
and total stream_ms for cost / latency forensics.
- Snippets in references are now first-50-lines (capped 4 KB) instead
of first-200-chars, so the citation preview is meaningful.
**CORS comment**: explicit note on app.py that the dev-only allow-list
must not be widened to "*" because chat accepts user API keys via
x-api-key header.
Test coverage:
- test_scan_since.py: ScanRequest field round-trips, missing-path
graceful return, non-git directories.
The CLI chat REPL learns the same multi-turn coherence the HTTP chat endpoint just got, plus a few quality-of-life upgrades: - History is kept in process memory across questions and forwarded to `build_chat_prompt` so follow-ups like "and what does that function return?" actually have context. `:clear` / `/clear` wipes history. - Answers stream as raw text (one heartbeat dot per chunk) and then render through `rich.markdown.Markdown` once the response finishes, so code blocks, lists, and headers display properly instead of as raw markdown text. - Source citations now render as a `rich.table` for tighter alignment. - `--top-k` is bounded `IntRange(1, 50)` instead of silently accepting absurd values. Bonus: the `build` command now builds the dependency graph upfront so PageRank is computed once and shared between the analyzer (reading guide ordering) and the wiki builder (dependency page). After build, any path / Mermaid warnings collected by WikiBuilder are surfaced as a "Content sanity warnings" panel so the user knows the LLM hallucinated.
…story
Closes the loop on the backend citation + chat work.
**SourceView page** (`/project/:id/source?path=…&line=…&end=…`): a new
route reachable from any `path/file.ext:42(-67)?` citation in either
wiki content or chat references. Uses Shiki for browser-side
syntax-highlight (keeps the backend simple) and renders the focused
snippet from the new server endpoint with the target lines visually
highlighted. Falls back to the full file for small bodies.
**Shared markdown helper** (`lib/markdown.ts`): citation linkification
and Mermaid splitting were duplicated between WikiContent and
ChatView. Pulled into one tiny dependency-free module so fixes land
once. The citation regex is restricted to a whitelist of code-file
extensions to avoid mangling innocuous `Foo:42` notation in prose.
**Home page scan-since input**: the local scan form now exposes a
`Since (git ref, optional)` field that wires through to the new
`since` field on ScanRequest. Empty input remains a full scan.
**ChatView upgrades**:
- Forwards history through to the backend so multi-turn conversations
work (matches CLI behaviour).
- Renders the references panel from the existing SSE `references`
frame and links each one through to the SourceView.
- Surfaces SSE `error` frames inline instead of leaving the user
staring at an empty bubble.
- Stop button cancels the in-flight `EventSource`.
**Other surface polish**:
- App.tsx adds the SourceView route.
- WikiContent uses the shared markdown helper and renders citations as
links.
- api.ts exposes `getFileContent({start, end})` and the chat stream
helper that yields content / references / error / done frames.
- stores/wiki.ts threads `since` and the new chat history through to
the API client; chat history is persisted per-project in zustand so
navigation back to the chat preserves context.
Six interlocking fixes shaken out by running through the test plan
against a tiny smoke target on Windows + zh-CN locale.
1. **Windows GBK stdio crashes Rich tree (cli.py)**: the CLI uses
📄/⚙️/📁 glyphs that the default cp936 codepage cannot encode, so the
scan command died with `UnicodeEncodeError: 'gbk' codec can't encode
character '⚙'` before any LLM call. Stdout/stderr are now
reconfigured to UTF-8 with errors="replace"; stdin uses strict so a
bad surrogate fails loudly rather than reaching the LLM as a request
body (DeepSeek rejects messages with lone surrogates).
2. **CLI chat short-circuited on non-code follow-ups (cli.py)**: the
chat REPL returned "No relevant code found in the index." whenever
BM25 retrieval came back empty -- which it always does for short
pronoun-only follow-ups ("它返回什么？") since BM25 doesn'\''t tokenise
CJK. The history feature was therefore mechanically broken. Two
coordinated fixes:
- augment the retrieval query with the most recent user turn so
short follow-ups still match the indexed code via prior English
tokens;
- if retrieval is still empty but `history` is non-empty, call the
LLM with empty context so the conversation can continue using
the prior turns. Brand-new questions with zero retrieval still
surface the existing "no relevant code" message.
Also throttles the streaming heartbeat dots from one-per-chunk to
one-per-ten-chunks so DeepSeek'\''s char-by-char stream stops drowning
the user in noise.
3. **Server chat retrieval mirrored the same blind spot
(server/routers/chat.py)**: now augments `retrieval_query` with the
last user turn from `req.history`, keeping the HTTP and CLI paths
behaviourally aligned.
4. **React error #185 in ChatView (frontend/src/pages/ChatView.tsx)**:
`useWikiStore((s) => s.chatHistory[projectId] ?? [])` returned a
brand-new `[]` reference on every render whenever the project had
no chat history yet, failing Object.is and triggering a render
loop. Replaced with a module-level stable `EMPTY_MESSAGES` reference
so the selector returns identity-stable output.
5. **SPA deep links 404 (server/app.py)**: `/project/<id>/source` is
handled by react-router, but reloading or sharing that URL hit
`StaticFiles` which raised Starlette `HTTPException(404)` because
no such file exists in the bundle. A `SpaStaticFiles` subclass now
catches the 404 (correctly using `starlette.exceptions.HTTPException`
rather than the FastAPI subclass) and falls back to `index.html`,
while still passing through `api/*` paths untouched.
6. **RAG snapshot was effectively dead code
(server/routers/scan.py)**: scan generated `uuid4()[:8]` for the
project id, so the on-disk index keyed by project_id was orphaned
after every server restart -- the reload branch in `_preheat_rag`
never ran. Replaced with `hashlib.sha256` of the absolute path (or
URL) trimmed to 8 hex chars. Same path now yields the same id
across restarts, and the snapshot reload branch is exercised. Added
a `progress("Reloaded chat index from snapshot (N chunks)")` line
so the operator can see the fast path engaged. Test coverage in
`test_scan_project_id.py` pins stability, abs/rel canonicalisation,
URL fallback, and id-vs-id uniqueness.
Verified end-to-end:
- `repowiki scan ./<target>` on Windows zh-CN now completes; warnings
panel surfaces when the LLM emits invalid Mermaid or hallucinated
paths.
- CLI chat over two turns: Q2 "它返回什么？" correctly threads back to
Q1'\''s context via history-augmented retrieval + LLM call.
- Web UI scan -> chat -> click citation -> SourceView with Shiki
syntax highlight and the target line range highlighted.
- Server restart -> re-scan same path -> progress confirms
`Reloaded chat index from snapshot (26 chunks)`.
Lint clean (`ruff check .`), 107 tests pass, frontend builds.
Owner
|
There are good ideas in here — the persistent BM25 store and the citation source viewer especially. The trouble is the shape. At 21 commits / 49 files / ~5.4k lines spanning RAG, frontend, CLI, server, graph, and build — and with the description noting it also carries fork- If you're up for it (you offered in the description), the way to get this in is to split it into focused PRs, one theme each: "persistent BM25 index" on its own with its tests, "frontend source viewer" on its own, and so on. I'll go through them in sequence. I'll leave this open as a tracking umbrella for now, but the actual review will happen on the split PRs. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A roll-up PR that consolidates a stretch of work on top of upstream
main. Three themes:WikiBuilder, a one-shot JSON repair retry, and structured warnings the CLI shows after every build.All checks green:
ruff check .,pytest tests/ -v(102 passed),npm run buildfor the frontend.What''s in this PR
Persistent BM25 RAG (
feat(rag))k1/bexposed throughConfigso users can tune retrieval to repo shape.upsert_file/remove_filekeep the global posting list consistent.rag_store.pysnapshots chunks + postings + per-file SHAs to~/.repowiki/indexes.db. Reload is millisecond-scale; corruption is contained via a schema version stamp.kind="wiki"flag so chat questions about the architecture page hit the page text directly. Opt-in viaREPOWIKI_RAG_INDEX_WIKI.REPOWIKI_RAG_CHUNK_*,REPOWIKI_RAG_TOP_K,REPOWIKI_RAG_MIN_SCORE,REPOWIKI_RAG_BM25_K1/B) parsed permissively — typos fall back to defaults rather than crashing on boot.test_rag.py,test_rag_bm25.py,test_rag_store.pycover tokenisation, normalisation, snapshot round-trips, schema versioning, project isolation.Mermaid sanitization + path validation (
feat(wiki))core/mermaid.py: forgiving best-effort fixer that auto-prepends missing diagram headers, rewrites non-identifier node IDs, and rejects sequence/flowchart kind mismatches. ReturnsNonewhen hopeless soWikiBuilderfalls back to a text description viadescribe_componentsinstead of leaving a blank box on the page.WikiBuildernow intersects every LLM-emitted file reference (module files, component files, reading-guide steps, relationship endpoints) against the project''s real path set. Dropped references go intobuilder.warnings, surfaced after CLI builds.Multi-turn chat + JSON repair + smarter architecture prompt (
feat(llm))build_chat_promptaccepts ahistorylist of prior turns;_trim_historykeeps the most recent N user+assistant pairs (default 5) so coherence works without unbounded token growth.missing_required_keys+build_repair_prompt: when JSON parse fails or required fields are missing, ship a one-shot corrective turn pinned to the prior context. Capped at one round at the call site.--concurrency, with batchedencode_batchtoken estimation. PageRank ordering drives which files surface to the LLM first.Server: incremental scan + snippet endpoint + robust SSE (
feat(server))ScanRequest.sincemirrors the CLI flag. Resolved intochanged_pathsviachanged_paths_since; GitHub URLs stay full, empty diffs fall back gracefully, errors log a warning rather than abort._preheat_rag()runs as a background task: reload existing index, diff per-file SHAs, re-chunk only changed files, refresh wiki chunks. The chat endpoint never pays indexing cost on the request thread.GET /project/{id}/files/{path}acceptsstart/end(1-based, inclusive). When supplied it returns a pre-slicedsnippetwith ±10 lines of context plus highlight markers, so the frontend doesn''t re-parse 5k-line bodies client-side.retrieve_ms/ tokens / cost /stream_ms, and reference snippets are now first-50-lines (4 KB cap) instead of first-200-chars.CLI chat REPL (
feat(cli)):clear//clearwipes.rich.markdown.Markdownso code blocks / lists / headers display properly.rich.tablefor tight alignment;--top-kboundedIntRange(1, 50).buildcommand pre-computes the dependency graph once and shares PageRank between analyzer and wiki builder. After build,WikiBuilder.warningsare surfaced as a "Content sanity warnings" panel.Frontend source viewer + chat polish (
feat(frontend))/project/:id/source?path=…&line=…&end=…route, reachable from anypath/file.ext:42(-67)?citation in either wiki or chat. Browser-side Shiki highlight, focused snippet rendering, target-line highlight.lib/markdown.ts: extracted shared citation linkifier + Mermaid splitter soWikiContentandChatViewuse the same code path. Citation regex restricted to a whitelist of code-file extensions to avoid mangling innocuousFoo:42prose.Home.tsxexposes aSince (git ref, optional)input that wires through toScanRequest.since.ChatViewforwards history, renders the references panel from SSEreferencesframes with links toSourceView, surfaces SSE errors inline, and the stop button cancels the in-flightEventSource.stores/wiki.tspersists chat history per-project so navigation back preserves context.Housekeeping (
chore).gitignorenow excludeswiki/,wiki.zip, andsrc/repowiki/server/static.bak/— all derived artefacts that should never end up in a commit.Test plan
ruff check .— cleanpytest tests/ -v— 102 passednpm run buildinfrontend/— successful, bundles ship tosrc/repowiki/server/static/repowiki build .against a sample repo, confirm warnings panel surfaces when LLM emits invalid Mermaidrepowiki serve, scan a local path withsince, ask a question in the web UI, click a citation, confirmSourceViewopens with highlighted linesNotes for reviewers
mainthat haven''t been upstreamed yet (token budget,--sinceCLI flag, chat command, oversize-module subdivision, etc.). Happy to break this into a sequence of smaller PRs if that''s preferred — let me know.rag_store.pyis at1; bump it the first time the table layout changes and the store will rebuild rather than misinterpret.