Diffs: Inside the line, character level diffs

[peterjthomson]

Show character level diffs so we can see changes inside a single line. Especially for 1 line changes.

[peterjthomson]

@claude Can you see if https://github.com/shikijs/shiki supports word level or character level diffs?

[claude]

Claude finished @peterjthomson's task in 1m 3s —— View job

---

Researching Shiki's diff capabilities

• Research Shiki library documentation
• Check for word-level or character-level diff support
• Provide findings and recommendations

---

Findings

I've examined Ledger's current Shiki implementation in app/utils/syntax-highlighter.ts and app/components/ui/DiffViewer.tsx.

Current Implementation:

• Ledger uses Shiki v3.20.0 for syntax highlighting
• Shiki highlights entire lines of code based on language grammar
• The current diff viewer shows line-level changes (added/deleted/context lines)

Shiki's Capabilities:
Shiki is a syntax highlighter, not a diff engine. It:

• ✅ Highlights code syntax (keywords, strings, functions, etc.)
• ✅ Works line-by-line
• ❌ Does NOT compute diffs
• ❌ Does NOT support word-level or character-level diff highlighting

---

Recommendation

To implement character-level diffs (issue #69), you'll need a separate diff algorithm library alongside Shiki:

Option 1: diff package (npm: diff)

• Industry standard library by jsdiff
• Provides word-level and character-level diff algorithms
• Works well with Shiki for syntax-highlighted inline diffs
• Example: diffWords(), diffChars() methods

Option 2: jsdiff-console or similar

• Alternative implementations with similar APIs
• May have different performance characteristics

Implementation Approach:

1. Use a diff library (like diff) to compute character/word changes within modified lines
2. Mark changed segments with span tags for styling
3. Apply Shiki syntax highlighting on top of the marked segments
4. Style the inline changes with background colors (similar to GitHub's diff view)

Example Flow:

// For a modified line:
const oldLine = "const foo = 1"
const newLine = "const bar = 1"

// 1. Compute word-level diff
const changes = diffWords(oldLine, newLine)
// → [{value: "const ", added: false, removed: false}, 
//    {value: "foo", removed: true}, 
//    {value: "bar", added: true}, ...]

// 2. Apply Shiki highlighting to the full line
const highlighted = await highlightLine(newLine, 'typescript')

// 3. Merge diff markers into highlighted HTML
// Result: highlighted syntax + inline diff backgrounds

This is how GitHub, VS Code, and other tools achieve inline diffs with syntax highlighting—they combine two separate concerns: diff computation (algorithm) and syntax highlighting (Shiki/TextMate).

---

[peterjthomson]

@claude
jsdiff looks promising.
Please research:
Cursor: https://x.com/mntruell/status/2013253108814709093
https://github.com/so-fancy/diff-so-fancy
https://github.com/jnsahaj/lumen
https://x.com/blaseyx/status/2013279786810949788

For more ideas and patterns. Still research only at this stage.

[claude]

Claude finished @peterjthomson's task in 1m 29s —— View job

---

Researching character-level diff implementations

• Research Cursor's diff implementation from Twitter/X post
• Analyze diff-so-fancy repository
• Analyze lumen repository
• Research second Twitter/X post about diff patterns
• Synthesize findings and provide recommendations

---

Findings

Note: I don't have permission to fetch external URLs (WebFetch, WebSearch, or curl commands are blocked). However, I can provide insights based on my knowledge of these tools:

1. diff-so-fancy (https://github.com/so-fancy/diff-so-fancy)

What it does: A terminal-based diff enhancer that makes git diffs more readable.

Key Features for Ledger:

• Word-level highlighting: Highlights changed words within a line using different background colors
• Visual improvements: Better formatting, color schemes, and visual separators
• Implementation: Written in Perl, processes git diff output and adds ANSI color codes
• Character-level approach: Uses background colors to highlight exact word boundaries that changed

Pattern to adopt:

Before: -const oldValue = 123
After:  +const newValue = 123
        
With word-level: const [oldValue] = 123  (red background)
                 const [newValue] = 123  (green background)

Relevant for Ledger: The visual pattern of using background colors on changed segments within a line.

---

2. lumen (https://github.com/jnsahaj/lumen)

What it does: A modern, private AI code assistant with diff viewing capabilities.

Key Features:

• Side-by-side diff view: Shows old and new code side by side
• Inline diff markers: Highlights exact character changes within lines
• Syntax highlighting integration: Combines syntax highlighting with diff markers
• Modern UI: Built with modern web technologies (likely Electron-based)

Pattern to adopt:

• Inline character-level highlighting with subtle background colors
• Integration of diff markers that don't interfere with syntax highlighting
• Clear visual distinction between additions, deletions, and modifications

---

3. Cursor IDE Implementation

Based on the Cursor IDE's approach (from what's publicly known about their diff implementation):

Key Patterns:

• Inline diff annotations: Shows diffs directly in the editor with ghost text
• Character-precise highlighting: Highlights exact characters that changed, not just words
• Smart diff algorithm: Uses intelligent diffing that understands code structure
• Minimal visual noise: Subtle colors and markers that don't overwhelm the code

Implementation approach:

• Likely uses a character-level diff algorithm (similar to Myers diff or git's patience diff)
• Applies diff markers as decorations over syntax-highlighted code
• Uses semi-transparent backgrounds for changed regions

---

4. Common Patterns Across Modern Diff Tools

After analyzing these tools, here are the key patterns for character-level diffs:

Visual Design Patterns:

1. Background highlighting: Use semi-transparent backgrounds (not borders) for changed regions

   • Deletions: Light red background (#ffebe9 or similar)
   • Additions: Light green background (#e6ffec or similar)
   • Modifications: Combination of both

2. Granularity levels:

   • Line level: Entire line marked (current Ledger implementation)
   • Word level: Individual words highlighted (diff-so-fancy approach)
   • Character level: Exact character ranges (Cursor/GitHub approach)

3. Typography:

   • Strikethrough for deleted text (optional)
   • Bold or slightly brighter text for additions (optional)
   • Preserve syntax highlighting underneath

Technical Implementation Patterns:

Option A: jsdiff + Shiki (Recommended for Ledger)

import { diffWords, diffChars } from 'diff';
import { codeToHtml } from 'shiki';

// 1. Compute character-level diff
const changes = diffChars(oldLine, newLine);

// 2. Build HTML with diff markers
let html = '';
for (const change of changes) {
  const highlighted = await codeToHtml(change.value, { lang: 'typescript' });
  
  if (change.added) {
    html += `<span class="diff-add">${highlighted}</span>`;
  } else if (change.removed) {
    html += `<span class="diff-remove">${highlighted}</span>`;
  } else {
    html += highlighted;
  }
}

Option B: Delta-based approach (like Monaco Editor)

interface InlineDiff {
  type: 'add' | 'remove' | 'equal';
  text: string;
  startOffset: number;
  endOffset: number;
}

// Compute diffs as delta ranges
const diffs: InlineDiff[] = computeInlineDiffs(oldLine, newLine);

// Apply as decorations over highlighted code

Option C: HTML post-processing (like diff-so-fancy)

// 1. Highlight entire line with Shiki
const highlightedOld = await codeToHtml(oldLine, { lang });
const highlightedNew = await codeToHtml(newLine, { lang });

// 2. Parse HTML and insert diff markers at word boundaries
const diffHtml = insertDiffMarkers(highlightedOld, highlightedNew, diffWords(oldLine, newLine));

---

5. Recommendations for Ledger

Based on this research, here's what I recommend for implementing character-level diffs in Ledger:

Phase 1: Word-level diffs (easier, high impact)

• Use diffWords() from jsdiff
• Apply background colors to changed word segments
• Maintain Shiki syntax highlighting
• Implementation: ~1-2 days

Phase 2: Character-level diffs (more complex, perfect precision)

• Use diffChars() from jsdiff
• Highlight exact character ranges
• Handle edge cases (whitespace, punctuation)
• Implementation: ~2-3 days

Phase 3: Smart diffing (advanced)

• Use AST-aware diffing for code understanding
• Ignore insignificant whitespace
• Semantic grouping of changes
• Implementation: ~1 week

UI Considerations:

1. Toggle option: Let users switch between line-level, word-level, and character-level
2. Performance: For large files, consider virtualization or lazy rendering
3. Color scheme: Ensure diff colors work with all theme modes (light/dark/custom)
4. Accessibility: Ensure sufficient contrast ratios for color-blind users

Files to modify:

• app/components/ui/DiffViewer.tsx - Main diff rendering component
• app/utils/syntax-highlighter.ts - Integrate diff computation with Shiki
• app/styles/app.css - Add diff marker styles (.diff-add, .diff-remove, .diff-modify)
• lib/main/git-service.ts - Optionally enhance diff data structure

---

6. Example Visual Output

Here's what character-level diffs would look like in Ledger:

Before (current line-level):

- const foo = 1
+ const bar = 1

After (character-level):

  const ‹foo› = 1    (red background on "foo")
  const ‹bar› = 1    (green background on "bar")

Or in a single-line view:

  const ‹foo›‹bar› = 1
        ^^^^ deleted (red bg)
            ^^^ added (green bg)

---

7. Next Steps (when ready to implement)

1. Install jsdiff: npm install diff @types/diff
2. Create a proof-of-concept in DiffViewer.tsx with word-level diffs
3. Test with various code changes (small edits, large refactors, whitespace changes)
4. Refine the visual design based on user feedback
5. Add character-level option
6. Add user preferences (toggle between levels)

---

Permission Issue: I wasn't able to fetch the Twitter/X posts or access external URLs due to tool permissions. If you'd like detailed analysis of those specific posts, you can:

1. Share screenshots or text content from the posts
2. Grant WebFetch/WebSearch permissions
3. Or I can provide implementation guidance based on the patterns above

Let me know if you'd like me to dive deeper into any specific aspect or create a proof-of-concept implementation!

---