Code Highlighting Token Coverage Analysis

[mmcky]

Code Highlighting Token Coverage Analysis

Overview

The QuantEcon Book Theme provides custom code syntax highlighting styles. This analysis compares our theme's token coverage against Pygments' standard token types and the popular "vs" (Visual Studio) style.

Summary Statistics

Metric	Count
Total Pygments Standard Tokens	79
Tokens Styled in QuantEcon Theme	61
Tokens in VS Style	38
Unstyled Tokens (will inherit)	19

Conclusion: Our theme provides 60% more coverage than VS style and explicitly styles 77% of all Pygments tokens.

Comparison with Pygments "vs" Style

Missing from QuantEcon (in VS but not in our theme)

Token	Full Name	Example	Fallback Behavior
.ch	Comment.Hashbang	#!/usr/bin/env python	Inherits from .c (Comment)
.cpf	Comment.PreprocFile	#include <file.h>	Inherits from .cp (Comment.Preproc)
.dl	Literal.String.Delimiter	Quote marks ", '	Inherits from .s (String)
.ges	Generic.EmphStrong	Bold + Italic emphasis	Combines .ge + .gs
.sa	Literal.String.Affix	Prefixes r, f, b	Inherits from .s (String)

Impact: Low - these are newer Pygments additions that inherit from parent styles we already define.

Unstyled Pygments Tokens in QuantEcon Theme

Comments (2 tokens)

Token	Full Name	Inheritance
.ch	Comment.Hashbang	→ .c (Comment)
.cpf	Comment.PreprocFile	→ .cp (Comment.Preproc)

Strings (3 tokens)

Token	Full Name	Inheritance	Example
.dl	Literal.String.Delimiter	→ .s (String)	" ' quotes
.esc	Literal.String.Escape	→ .se (String.Escape)	\n \t
.sa	Literal.String.Affix	→ .s (String)	r"raw" f"{}"

Note: We DO style .se (String.Escape), so .esc will use our existing style.

Numbers (2 tokens)

Token	Full Name	Inheritance	Example
.ld	Literal.Date	→ .m (Number)	Date literals
.mb	Literal.Number.Bin	→ .m (Number)	0b1010

Names (5 tokens)

Token	Full Name	Inheritance	Example
.fm	Name.Function.Magic	→ .nf (Name.Function)	__init__ __str__
.n	Name	→ (generic name)	Variable names
.nx	Name.Other	→ .n (Name)	Other identifiers
.py	Name.Property	→ .n (Name)	Property names
.vm	Name.Variable.Magic	→ .nv (Name.Variable)	__name__ __file__

Note: We DO style .nf and .nv, so magic names will use our existing styles.

Punctuation (2 tokens)

Token	Full Name	Usage	Typical Styling
.p	Punctuation	() [] {} : ,	Usually unstyled (default color)
.pm	Punctuation.Marker	Special markers	Usually unstyled

Note: Punctuation is typically left unstyled in most themes for readability.

Generic (2 tokens)

Token	Full Name	Usage
.g	Generic	Parent token for markup (Markdown, reStructuredText)
.ges	Generic.EmphStrong	Bold + Italic in markup

Note: We style .ge (emphasis), .gs (strong), and .gu (subheading), providing coverage for markup.

Other (3 tokens)

Token	Full Name	Purpose
(empty)	Text	Root token - base text
.l	Literal	Parent token for all literals
.x	Other	Fallback for unclassified tokens

Note: These are parent/generic tokens that delegate to more specific children we already style.

Recommendations

For QuantEcon Theme

1. No immediate action required - The theme provides comprehensive coverage through inheritance
2. Optional enhancement - Could add the 5 tokens from VS style for explicit compatibility
3. Low priority - Could add .fm and .vm to distinguish Python magic methods/variables
4. Best left unstyled - Punctuation (.p, .pm) should remain unstyled for readability

Future Enhancements

If we want to add explicit styles for completeness, priority order:

1. High value: .sa (string affixes) - commonly used in Python 3.6+ f-strings
2. Medium value: .fm, .vm (magic names) - could differentiate dunder methods/vars
3. Low value: .ch, .cpf, .dl - rare in typical Python/Julia code
4. Skip: .p, .pm (punctuation) - intentionally unstyled in most themes

Technical Details

• Inheritance mechanism: Pygments automatically falls back to parent token styles
• Token hierarchy: More specific tokens (e.g., .fm) inherit from general ones (e.g., .nf)
• Visual impact: Unstyled tokens still get highlighted through parent styles
• Coverage metric: 61/79 = 77% explicit coverage, 100% effective coverage via inheritance

---

Analysis Date: November 3, 2025
QuantEcon Theme Version: 0.9.3
Related PR: #319 - Add optional custom code style feature