Skip to content

Fix _split_cells to handle non-unit width characters correctly - #4155

Open
nisha-muthurajan wants to merge 3 commits into
Textualize:mainfrom
nisha-muthurajan:fix/split-cells-non-unit-chars
Open

Fix _split_cells to handle non-unit width characters correctly#4155
nisha-muthurajan wants to merge 3 commits into
Textualize:mainfrom
nisha-muthurajan:fix/split-cells-non-unit-chars

Conversation

@nisha-muthurajan

Copy link
Copy Markdown

Fixes #3299

The previous implementation used a proportional heuristic to estimate the starting character position, which overshot for multi-cell characters like emoji. Replace with a linear scan that accumulates real cell widths.

Type of changes

-✅ Bug fix

  • New feature
  • Documentation / docstrings
  • ✅ Tests
  • Other

AI?

  • AI was used to generate this PR

AI generated PRs may be accepted, but only if @willmcgugan has responded on an issue or discussion.

Checklist

  • I've run the latest black with default args on new code.
  • I've updated CHANGELOG.md and CONTRIBUTORS.md where appropriate (see note about typos above).
  • ✅ I've added tests for new code.
  • ✅ I accept that @willmcgugan may be pedantic in the code review.

Description

Fixes #3299

Segment._split_cells used a proportional heuristic to guess the starting
character position:

pos = int((cut / cell_length) * len(text))

This overshot for multi-cell characters (emoji, CJK) because it assumed all
characters have equal width. The fallback loop then couldn't recover correctly,
producing wrong splits like ('🦊🦊 ', ' abcdef') instead of ('🦊 ', ' abcdef').

Fixed by replacing the heuristic + loop with a simple linear scan that
accumulates real cell widths character by character, stopping precisely at
the cut point.

Added a regression test test_split_cells_emoji covering the two examples
from the issue plus an exact-boundary case.

Fixes Textualize#3299

The previous implementation used a proportional heuristic to estimate
the starting character position, which overshot for multi-cell characters
like emoji. Replace with a linear scan that accumulates real cell widths.
Comment thread rich/segment.py Outdated
cell_size = get_character_cell_size
cell_pos = 0

for pos, char in enumerate(text):

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this drops the proportional-guess seek for a scan from index 0, so it is now O(pos) on every call where the old code usually converged in a step or two. on Segment("早"*400+"x") cut at 790 with the lru_cache cleared each time I get ~25x slower (0.074s vs 0.0029s for 2000 splits) — and since split_cells short-circuits single-width text, this path only ever sees the wide/mixed strings where that matters. also cell_size = get_character_cell_size is assigned twice, line 128 and 129.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch on both. Fixed the duplicate cell_size line. On the perf point — you're right that scanning from 0 unconditionally regresses the case you benchmarked. I've reworked it to keep a cheap initial guess (assuming ~1 cell per character) and do a bounded correction walk from that guess rather than from index 0, while still fixing the emoji overshoot bug from #3299.

Re-ran your exact benchmark (Segment("早"*400+"x"), cut at 790, 2000 iterations, cache cleared each time) locally: 0.0029s, matching the original baseline you cited. Also ran the full test suite (985 passed, 23 skipped) to confirm nothing else regressed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] Segment._split_cells doesn't handle non-unit characters well

2 participants