feat(lsp): experimental in-process LSP server with hover support

[StanAngeloff]

Summary

Adds an experimental in-process LSP server that attaches to .chat buffers and provides hover inspection of every AST node. This is the foundational infrastructure for future LSP features (goto-definition for includes, diagnostics, completions, etc.).

Enable with:

require("flemma").setup({
  experimental = { lsp = true },
})

Architecture

The LSP server runs in-process via vim.lsp.start() with a Lua cmd function — no external binary, no sockets. Neovim's built-in LSP client calls directly into our dispatch table. The server advertises hoverProvider capability and handles textDocument/hover requests by:

1. Converting LSP 0-indexed positions to 1-indexed AST coordinates
2. Calling ast.find_segment_at_position(doc, lnum, col) to resolve cursor → node
3. Serializing the matched node to a structured markdown hover

Three-tier hover resolution

Every buffer position returns a hover result — there are no dead zones:

Tier	Match condition	Hover content
Segment	Cursor within a segment's position range	Kind-specific dump: expression code, thinking content + redacted/signature, tool name + ID + input JSON, tool result content + error flag, text preview (truncated at 200 chars)
Message	Cursor within message range but no segment matched (e.g. @Role: marker line)	Role, segment count, breakdown by kind (e.g. text=2, tool_use=1)
Frontmatter	Cursor within frontmatter fence block	Language, byte length, full code in a fenced block

Cursor-to-node detection (find_segment_at_position)

This is the core query function added to flemma.ast.query. It resolves a (line, col) pair to the innermost AST node:

• Column-precise matching for segments that have start_col/end_col (expressions, file references) — needed because multiple segments can share a line (e.g. Hello {{ a }} and {{ b }})
• Line-range matching for multi-line segments (thinking, tool blocks) — any line within the range hits
• Fallback mechanism for segments without column info — deferred as fallback so column-aware segments on the same line get priority
• Message-level fallback — if cursor is within a message's line range but no segment matched, returns (nil, msg) so the hover handler can show message-level info

Parser changes

• Added end_col to flemma.ast.Position type
• Parser now computes end_col for expression segments ({{ expr }}) and file reference segments (@./path), enabling column-precise hover detection on shared lines

New files

• lua/flemma/lsp.lua — In-process LSP server: dispatch table, hover handler, markdown serializers for each node type
• tests/flemma/lsp_spec.lua — 8 tests: attach/detach, hover for expression, text, thinking, tool_use, role marker, frontmatter

Modified files

• lua/flemma/config.lua — Added experimental = { lsp = false } default and flemma.config.Experimental type
• lua/flemma/init.lua — Wires lsp.setup() behind the experimental.lsp flag
• lua/flemma/ast/nodes.lua — Added end_col? integer to flemma.ast.Position
• lua/flemma/ast/query.lua — Added find_segment_at_position() with column refinement and message-level fallback
• lua/flemma/parser.lua — Computes end_col for expression and file reference segments
• tests/flemma/ast_spec.lua — 10 new tests: end_col positions, find_segment_at_position (expression, text, out-of-range, thinking, tool_use, adjacent expressions, role marker)
• Makefile — make develop now enables experimental.lsp = true and TRACE logging for manual testing

Logging

The LSP module follows codebase logging conventions:

• TRACE: Every request/notify call, full hover markdown responses
• DEBUG: Server lifecycle (create, initialize, shutdown, terminate), hover position conversion, node match details (kind + identity), response presence
• INFO: lsp.setup() activation only
• WARN: Invalid buffer, unnamed buffer URI resolution risk

Test plan

• make qa passes (luacheck, LuaLS types, inline-requires, all tests)
• Manual: open a .chat file with experimental.lsp = true, hover over text, expressions, @Role: markers, thinking blocks, tool blocks, frontmatter — each shows structured info
• Manual: hover on empty lines between messages returns nil gracefully (no errors)
• Verify no LSP client attaches when experimental.lsp = false (default)

🤖 Generated with Claude Code