Skip to content

fix(cicd): chunk bot messages on character boundaries#5882

Open
anxkhn wants to merge 1 commit into
SQLMesh:mainfrom
anxkhn:fix/cicd-utf8-safe-chunking
Open

fix(cicd): chunk bot messages on character boundaries#5882
anxkhn wants to merge 1 commit into
SQLMesh:mainfrom
anxkhn:fix/cicd-utf8-safe-chunking

Conversation

@anxkhn

@anxkhn anxkhn commented Jul 5, 2026

Copy link
Copy Markdown
Contributor

GithubController._chunk_up_api_message splits a message into chunks that fit
GitHub's per-field byte limit (MAX_BYTE_LENGTH, 65535). It did this by UTF-8
encoding the whole message, slicing the resulting bytes at fixed
MAX_BYTE_LENGTH offsets, and decoding each slice with errors="ignore":

message_encoded = message.encode("utf-8")
return [
    message_encoded[i : i + self.MAX_BYTE_LENGTH].decode("utf-8", "ignore")
    for i in range(0, len(message_encoded), self.MAX_BYTE_LENGTH)
]

When a multibyte character (accented letter, CJK, emoji, etc.) straddles a chunk
boundary, its UTF-8 bytes are split across two slices. decode("utf-8", "ignore")
then discards the incomplete byte sequence on both sides, so that character is
silently dropped from the bot's PR comment or check-run output.

This change accumulates characters into a chunk while the running UTF-8 byte
length stays within MAX_BYTE_LENGTH, so a boundary is only ever placed between
whole characters and no character is bisected. Behavior for the common case is
unchanged: ASCII-only messages chunk into the exact same pieces as before, and an
empty message still yields no chunks. Both existing callers rely only on the
number of chunks (comment edit unpacks body, *truncated; check-run output
unpacks summary, text, *truncated), and that contract is preserved.

Test Plan

  • Added test_chunk_up_api_message_preserves_multibyte_chars, which builds a
    message just over MAX_BYTE_LENGTH with a multibyte character (é) placed on
    the byte boundary between two chunks, and asserts the chunks reassemble into the
    original message and each chunk stays within MAX_BYTE_LENGTH bytes. This test
    fails on the previous byte-slicing implementation (the boundary character is
    dropped) and passes with the fix.
  • pytest tests/integrations/github/cicd/ -m "not slow and not docker" -> 55
    passed, 28 deselected.
  • make style (ruff check + ruff format) clean on the changed files.

Checklist

  • I have run make style and fixed any issues
  • I have added tests for my changes (if applicable)
  • All existing tests pass (make fast-test)
  • My commits are signed off (git commit -s) per the DCO

_chunk_up_api_message sliced the UTF-8 encoded message at fixed byte
offsets and decoded each slice with errors="ignore". A multibyte
character that straddled a chunk boundary had its bytes split across two
slices, so the "ignore" handler silently dropped it from the bot's PR
comment or check-run output.

Accumulate characters into a chunk while the running UTF-8 byte length
stays within MAX_BYTE_LENGTH instead of slicing raw bytes, so no
character is ever bisected. ASCII-only messages chunk identically to
before, and an empty message still yields no chunks.

Signed-off-by: Anas Khan <83116240+anxkhn@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant