From 46d3f6de66b97642ba4f89c50944563c5d56af6c Mon Sep 17 00:00:00 2001 From: Min Huang <70873102+min0625@users.noreply.github.com> Date: Tue, 7 Jul 2026 10:06:04 +0000 Subject: [PATCH] feat: split long input into chunks for translation Input over 2000 runes is split at paragraph boundaries (falling back to line breaks, then whitespace) and translated chunk by chunk so a long document can't hit the model's output-token limit and come back truncated. Each chunk's original separator is reprinted verbatim so whitespace survives exactly; rotation-mode detection samples a single chunk instead of the whole document. Co-Authored-By: Claude Sonnet 5 --- AGENTS.md | 1 + README.ja.md | 2 +- README.ko.md | 2 +- README.md | 2 +- README.zh-CN.md | 2 +- README.zh-TW.md | 2 +- cmd/mint/chunk.go | 253 ++++++++++++++++++++++++ cmd/mint/chunk_test.go | 354 ++++++++++++++++++++++++++++++++++ cmd/mint/main.go | 233 +++++++++++++++++++--- cmd/mint/main_test.go | 428 +++++++++++++++++++++++++++++++++++++++++ docs/manual-testing.md | 29 +++ internal/llm/llm.go | 6 + 12 files changed, 1280 insertions(+), 34 deletions(-) create mode 100644 cmd/mint/chunk.go create mode 100644 cmd/mint/chunk_test.go diff --git a/AGENTS.md b/AGENTS.md index 5cb8c9f..6ae732f 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -107,6 +107,7 @@ bin/mint # compiled binary (gitignored) - Language detection: when no source language is given, the input language is inferred — by the LLM in rotation mode, or implicitly by the rewrite prompt for a single target. - Source language: optional `--source` / `-s` flag (BCP-47 tag); flag-only, no env var (a source is per-input, not a persistent preference). When set it skips detection and anchors the rewrite prompt to translate *from* that language, so cross-language homographs (e.g. French `chat` → English `cat`) and romanized input (e.g. `konnichiwa` → `hello`) are translated rather than treated as already-target text. Empty (the default) preserves the original auto-detect behavior. Pure language-neutral input still passes through unchanged regardless. - Language-neutral pass-through: if detected language is `neutral`, input is printed unchanged with no translation call. +- Long-input chunking: input over 2000 runes is split at paragraph boundaries (falling back to line breaks, then single whitespace runes — see `cmd/mint/chunk.go`), and each chunk is translated in its own request so no single response can hit the model's output-token cap. Paragraph/line boundaries recognize both `\n` and `\r\n`. Each chunk's original separator (paragraph gap, line break, single space, or nothing) is reprinted verbatim between chunk outputs — the model's own trailing newline is trimmed rather than assumed away, so chunk boundaries never inject or lose whitespace. Rotation-mode detection samples the first chunk that contains letters (falling back to the first chunk); if that sample comes back neutral, a single-chunk document still passes through unchanged, but a multi-chunk document falls back to the first configured target instead of assuming the whole document is neutral. Language-neutral chunks skip the LLM call entirely. - Same-language behavior: if detected input language matches the target language, the tool performs grammar and spelling correction instead of translation. - Target language priority: `--target` flag → `MINT_TARGET_LANG` env var → system locale (`$LC_ALL` / `$LC_MESSAGES` / `$LANG`) → `en`. - Language rotation: `MINT_TARGET_LANG` accepts a comma-separated list (e.g., `en,zh-TW,ja`); when the detected input language matches a tag in the list, the tool translates to the next tag (wraps around). BCP-47 variants sharing the same primary subtag (e.g., `zh-HK` and `zh-TW`) are treated as equivalent. diff --git a/README.ja.md b/README.ja.md index 6a71ba8..73f0940 100644 --- a/README.ja.md +++ b/README.ja.md @@ -250,7 +250,7 @@ mint "こんにちは" # ja → en: Hello - [x] `--source` / `-s` フラグによるソース言語の明示的指定 - [x] ストリーミング出力 - [x] GoReleaserによるマルチプラットフォームバイナリ配布（Linux / macOS / Windows） -- [ ] バッチ翻訳モード +- [x] バッチ翻訳モード — 長い入力を段落境界で分割し、チャンクごとに翻訳 - [ ] 用語集 / カスタム辞書サポート - [ ] 出力フォーマットオプション（プレーンテキスト、JSON、Markdown） - [ ] 翻訳結果のキャッシュ diff --git a/README.ko.md b/README.ko.md index 989d553..6e84483 100644 --- a/README.ko.md +++ b/README.ko.md @@ -250,7 +250,7 @@ mint "こんにちは" # ja → en: Hello - [x] `--source` / `-s` 플래그를 통한 명시적 소스 언어 지정 - [x] 스트리밍 출력 - [x] GoReleaser 기반 멀티 플랫폼 바이너리 릴리스 (Linux / macOS / Windows) -- [ ] 배치 번역 모드 +- [x] 배치 번역 모드 — 긴 입력을 단락 경계에서 분할해 청크 단위로 번역 - [ ] 용어집 / 사용자 지정 사전 지원 - [ ] 출력 형식 옵션 (일반 텍스트, JSON, Markdown) - [ ] 반복 번역 캐싱 diff --git a/README.md b/README.md index 7d643ad..53dbc3a 100644 --- a/README.md +++ b/README.md @@ -250,7 +250,7 @@ mint "こんにちは" # ja → en: Hello - [x] Explicit source language via `--source` / `-s` flag - [x] Streaming output - [x] GoReleaser multi-platform binary release (Linux / macOS / Windows) -- [ ] Batch translation mode +- [x] Batch translation mode — long input is split at paragraph boundaries and translated chunk by chunk - [ ] Glossary / custom dictionary support - [ ] Output format options (plain text, JSON, Markdown) - [ ] Caching for repeated translations diff --git a/README.zh-CN.md b/README.zh-CN.md index 856fbda..7942302 100644 --- a/README.zh-CN.md +++ b/README.zh-CN.md @@ -250,7 +250,7 @@ mint "こんにちは" # ja → en: Hello - [x] 通过 `--source` / `-s` 参数明确指定来源语言 - [x] 流式输出 - [x] GoReleaser 多平台二进制发布（Linux / macOS / Windows） -- [ ] 批量翻译模式 +- [x] 批量翻译模式 — 长输入按段落边界切块、逐块翻译 - [ ] 术语表 / 自定义词典支持 - [ ] 输出格式选项（纯文本、JSON、Markdown） - [ ] 翻译结果缓存 diff --git a/README.zh-TW.md b/README.zh-TW.md index f076114..11ce625 100644 --- a/README.zh-TW.md +++ b/README.zh-TW.md @@ -250,7 +250,7 @@ mint "こんにちは" # ja → en: Hello - [x] 透過 `--source` / `-s` 旗標明確指定來源語言 - [x] 串流輸出 - [x] GoReleaser 多平台二進位檔發布（Linux / macOS / Windows） -- [ ] 批次翻譯模式 +- [x] 批次翻譯模式 — 長輸入依段落邊界切塊、逐塊翻譯 - [ ] 術語表 / 自訂詞典支援 - [ ] 輸出格式選項（純文字、JSON、Markdown） - [ ] 翻譯結果快取 diff --git a/cmd/mint/chunk.go b/cmd/mint/chunk.go new file mode 100644 index 0000000..bd56673 --- /dev/null +++ b/cmd/mint/chunk.go @@ -0,0 +1,253 @@ +// Copyright 2026 The Mint Authors. +package main + +import ( + "strings" + "unicode" + "unicode/utf8" +) + +// maxChunkRunes caps how much text goes into a single translation request. +// Longer input is split into chunks and translated chunk by chunk, so a long +// document cannot hit the model's output-token limit and come back truncated. +// CJK text runs roughly one token per rune, so a chunk stays near 2k output +// tokens — inside the default context of even small local models. +const maxChunkRunes = 2000 + +// chunk is one translation unit plus the exact separator text (a paragraph +// gap, a line break, a single space, or nothing) that followed it in the +// original input, kept so the output can restore paragraph and word +// separation exactly as it appeared, regardless of how a chunk was split. +type chunk struct { + text string + sep string +} + +// splitChunks splits text into translation units of at most limit runes each. +// Splitting prefers paragraph boundaries (blank lines), then line breaks, +// then whitespace within a line; only a single word longer than the limit is +// ever cut mid-word. Adjacent small units are re-packed greedily so short +// paragraphs still share one request. Concatenating text+sep over the result +// reproduces the input exactly. +func splitChunks(text string, limit int) []chunk { + if limit < 1 { + limit = 1 + } + + if utf8.RuneCountInString(text) <= limit { + return []chunk{{text: text}} + } + + var units []chunk + for _, p := range splitOnNewlineRuns(text, 2) { + units = append(units, subdivideUnit(p, limit)...) + } + + return packUnits(units, limit) +} + +// newlineWidth returns the byte length of the newline token starting at +// s[i]: 2 for "\r\n", 1 for a bare "\n", or 0 if s[i] does not start a +// newline. Treating "\r\n" as a single token means Windows line endings are +// recognized as paragraph and line boundaries exactly like Unix ones. +func newlineWidth(s string, i int) int { + if i >= len(s) { + return 0 + } + + if s[i] == '\n' { + return 1 + } + + if s[i] == '\r' && i+1 < len(s) && s[i+1] == '\n' { + return 2 + } + + return 0 +} + +// indexNewlineRun returns the byte range [start, end) of the first run of at +// least minRun consecutive newline tokens ("\n" or "\r\n") in s, or (-1, -1) +// if there is none. The returned range extends through the whole run, not +// just its first minRun tokens, so e.g. three blank lines are captured +// together. +func indexNewlineRun(s string, minRun int) (start, end int) { + start, run := -1, 0 + + for i := 0; i < len(s); { + w := newlineWidth(s, i) + if w == 0 { + run, start = 0, -1 + i++ + + continue + } + + if run == 0 { + start = i + } + + run++ + i += w + + if run == minRun { + for { + w := newlineWidth(s, i) + if w == 0 { + return start, i + } + + i += w + } + } + } + + return -1, -1 +} + +// splitOnNewlineRuns splits text into units at every run of at least minRun +// consecutive newline tokens; each unit keeps the newline run that followed +// it as sep. A trailing empty unit is never produced when text ends exactly +// on a newline run — that run is captured as the last real unit's sep +// instead, matching the invariant that every unit carries translatable text. +func splitOnNewlineRuns(text string, minRun int) []chunk { + var units []chunk + + rest := text + + for { + i, j := indexNewlineRun(rest, minRun) + if i < 0 { + break + } + + units = append(units, chunk{text: rest[:i], sep: rest[i:j]}) + rest = rest[j:] + } + + if rest != "" || len(units) == 0 { + units = append(units, chunk{text: rest}) + } + + return units +} + +// subdivideUnit splits a paragraph unit that exceeds limit runes at single line +// breaks, then hard-splits any line that is still too long. The unit's own +// trailing separator is reattached to its last piece. +func subdivideUnit(u chunk, limit int) []chunk { + if utf8.RuneCountInString(u.text) <= limit { + return []chunk{u} + } + + var out []chunk + for _, line := range splitOnNewlineRuns(u.text, 1) { + out = append(out, splitLine(line, limit)...) + } + + // The paragraph's own trailing separator (u.sep) and the last piece's own + // sep are never both non-empty: u.sep is only non-empty when this unit is + // followed by more of the original document, in which case + // splitOnNewlineRuns(u.text, 1) already leaves the last piece's sep empty + // (u.text ends on non-newline content); conversely a non-empty last-piece + // sep only arises when u.text itself ends on a trailing newline token, + // which only happens for the document's final unit, where u.sep is "". + // Concatenating rather than overwriting keeps whichever one is real. + out[len(out)-1].sep += u.sep + + return out +} + +// splitLine splits a single line (no newline token) into pieces of at most +// limit runes, breaking at the last whitespace rune at or before the limit +// when one exists, so words stay intact. That whitespace rune becomes the +// piece's sep rather than staying in its text, so it is never sent to the +// LLM and reassembly rejoins the pieces exactly as the input had them. When +// no whitespace is found, the line is hard-split with sep = "". The line's +// own trailing separator is reattached to the last piece. +func splitLine(u chunk, limit int) []chunk { + runes := []rune(u.text) + if len(runes) <= limit { + return []chunk{u} + } + + var out []chunk + + for len(runes) > limit { + cut, sep := limit, "" + + for k := limit; k >= 0; k-- { + if unicode.IsSpace(runes[k]) { + cut = k + sep = string(runes[k]) + + break + } + } + + out = append(out, chunk{text: string(runes[:cut]), sep: sep}) + runes = runes[cut+utf8.RuneCountInString(sep):] + } + + return append(out, chunk{text: string(runes), sep: u.sep}) +} + +// packUnits greedily merges adjacent units into chunks of at most limit runes, +// so a long document of short paragraphs still needs only a few requests. +// +// A leading unit with empty text only occurs when the document starts with a +// paragraph-boundary newline run; unlike an empty unit produced between two +// real units (whose surrounding separators are meaningfully packed as literal +// text into one translation request), this one has no preceding real content +// to attach its separator to. Packing it in like any other unit would fold +// the document's leading whitespace into the first real chunk's text and +// send it to the LLM, which has no instruction to preserve it. Emitting it +// as its own chunk lets translateChunks reprint its separator verbatim +// before the first real chunk, exactly like an interior separator. +func packUnits(units []chunk, limit int) []chunk { + if len(units) == 0 { + return nil + } + + var out []chunk + + start := 0 + + if units[0].text == "" { + out = append(out, units[0]) + start = 1 + } + + if start >= len(units) { + return out + } + + var b strings.Builder + + b.WriteString(units[start].text) + + curLen := utf8.RuneCountInString(units[start].text) + curSep := units[start].sep + + for _, u := range units[start+1:] { + sepLen := utf8.RuneCountInString(curSep) + uLen := utf8.RuneCountInString(u.text) + + if curLen+sepLen+uLen <= limit { + b.WriteString(curSep) + b.WriteString(u.text) + curSep = u.sep + curLen += sepLen + uLen + + continue + } + + out = append(out, chunk{text: b.String(), sep: curSep}) + + b.Reset() + b.WriteString(u.text) + curLen, curSep = uLen, u.sep + } + + return append(out, chunk{text: b.String(), sep: curSep}) +} diff --git a/cmd/mint/chunk_test.go b/cmd/mint/chunk_test.go new file mode 100644 index 0000000..330f373 --- /dev/null +++ b/cmd/mint/chunk_test.go @@ -0,0 +1,354 @@ +// Copyright 2026 The Mint Authors. +package main + +import ( + "strings" + "testing" + "unicode/utf8" +) + +// reassemble concatenates text+sep over chunks; splitChunks guarantees this +// reproduces the original input exactly. +func reassemble(chunks []chunk) string { + var sb strings.Builder + for _, c := range chunks { + sb.WriteString(c.text) + sb.WriteString(c.sep) + } + + return sb.String() +} + +func TestSplitChunksShortInputSingleChunk(t *testing.T) { + text := "Hello world\n\nSecond paragraph" + + chunks := splitChunks(text, 100) + if len(chunks) != 1 { + t.Fatalf("expected 1 chunk, got %d", len(chunks)) + } + + if chunks[0].text != text || chunks[0].sep != "" { + t.Errorf("short input must pass through unchanged, got %+v", chunks[0]) + } +} + +func TestSplitChunksParagraphBoundaries(t *testing.T) { + text := "alpha one\n\nbravo two\n\ncandy six" + + chunks := splitChunks(text, 10) + if len(chunks) != 3 { + t.Fatalf("expected 3 chunks, got %d: %+v", len(chunks), chunks) + } + + want := []chunk{ + {text: "alpha one", sep: "\n\n"}, + {text: "bravo two", sep: "\n\n"}, + {text: "candy six", sep: ""}, + } + for i, c := range chunks { + if c != want[i] { + t.Errorf("chunk %d = %+v, want %+v", i, c, want[i]) + } + } +} + +func TestSplitChunksPacksSmallParagraphs(t *testing.T) { + text := "aa\n\nbb\n\ncc\n\ndddddddddd" + + chunks := splitChunks(text, 10) + if len(chunks) != 2 { + t.Fatalf("expected 2 chunks, got %d: %+v", len(chunks), chunks) + } + + // The three short paragraphs fit into one 10-rune chunk ("aa\n\nbb" is 6 + // runes; adding "\n\ncc" would make 10 — exactly at the limit). + if chunks[0].text != "aa\n\nbb\n\ncc" { + t.Errorf("chunk 0 = %q, want packed paragraphs", chunks[0].text) + } + + if reassemble(chunks) != text { + t.Errorf("reassembled = %q, want original", reassemble(chunks)) + } +} + +func TestSplitChunksOversizedParagraphSplitsOnLines(t *testing.T) { + text := "line one is here\nline two is here\n\nshort" + + chunks := splitChunks(text, 20) + if len(chunks) != 3 { + t.Fatalf("expected 3 chunks, got %d: %+v", len(chunks), chunks) + } + + want := []chunk{ + {text: "line one is here", sep: "\n"}, + {text: "line two is here", sep: "\n\n"}, + {text: "short", sep: ""}, + } + for i, c := range chunks { + if c != want[i] { + t.Errorf("chunk %d = %+v, want %+v", i, c, want[i]) + } + } +} + +func TestSplitChunksLongLineBreaksAtSpaces(t *testing.T) { + text := strings.TrimSuffix(strings.Repeat("word ", 20), " ") // 99 runes + + chunks := splitChunks(text, 30) + if len(chunks) < 4 { + t.Fatalf("expected at least 4 chunks, got %d", len(chunks)) + } + + for i, c := range chunks { + if n := utf8.RuneCountInString(c.text); n > 30 { + t.Errorf("chunk %d has %d runes, exceeds max 30", i, n) + } + + // Every piece except the last must break at a space, and that space + // must live in sep (not text), so it's never sent to the LLM. + if i < len(chunks)-1 { + if strings.HasSuffix(c.text, " ") { + t.Errorf("chunk %d text = %q retains the breaking space, want it moved to sep", i, c.text) + } + + if c.sep != " " { + t.Errorf("chunk %d sep = %q, want a single space", i, c.sep) + } + } + } + + if reassemble(chunks) != text { + t.Errorf("reassembled = %q, want original", reassemble(chunks)) + } +} + +func TestSplitChunksCJKHardSplit(t *testing.T) { + // No whitespace at all: must hard-split at the rune limit without + // corrupting any multi-byte rune. + text := strings.Repeat("翻譯測試", 25) // 100 runes + + chunks := splitChunks(text, 30) + if len(chunks) != 4 { + t.Fatalf("expected 4 chunks, got %d", len(chunks)) + } + + for i, c := range chunks { + if !utf8.ValidString(c.text) { + t.Errorf("chunk %d is not valid UTF-8", i) + } + + if n := utf8.RuneCountInString(c.text); n > 30 { + t.Errorf("chunk %d has %d runes, exceeds max 30", i, n) + } + } + + if reassemble(chunks) != text { + t.Error("reassembled text differs from original") + } +} + +func TestSplitChunksPreservesNewlineRuns(t *testing.T) { + // Three blank lines between paragraphs (a run of 4 newlines) must be + // preserved in the separator so output spacing can be restored. + text := "alpha one\n\n\n\nbravo two" + + chunks := splitChunks(text, 10) + if len(chunks) != 2 { + t.Fatalf("expected 2 chunks, got %d: %+v", len(chunks), chunks) + } + + if chunks[0].sep != "\n\n\n\n" { + t.Errorf("sep = %q, want the full newline run", chunks[0].sep) + } + + if reassemble(chunks) != text { + t.Error("reassembled text differs from original") + } +} + +func TestSplitChunksReassemblyExact(t *testing.T) { + inputs := []string{ + "plain short text", + "para one\n\npara two\n\n\npara three", + strings.Repeat("mixed 混合 text ", 40), + "leading\n\n\n" + strings.Repeat("x", 50) + "\ntrailing", + "\n\nstarts with blank lines", + "trailing newline run\n\n", + "trailing newline run\n\n\n\n", + "para one\r\n\r\npara two", + "line one\r\nline two\r\n\r\nshort", + // Oversized paragraph whose text itself ends in a single trailing + // newline (no blank-line run anywhere), the case subdivideUnit must + // not lose when reattaching the (empty) paragraph-level separator. + strings.Repeat("x", 15) + " " + strings.Repeat("y", 15) + "\n", + "line one is here\nline two is here\n", + } + for _, text := range inputs { + for _, max := range []int{10, 25, 80} { + if got := reassemble(splitChunks(text, max)); got != text { + t.Errorf("splitChunks(%q, %d): reassembled %q != original", text, max, got) + } + } + } +} + +func TestSplitChunksTrailingNewlineRunHasNoEmptyChunk(t *testing.T) { + // A document whose last paragraph is followed by a newline run must not + // produce a trailing empty chunk — that run belongs to the last real + // chunk's sep. + text := "alpha one\n\nbravo two\n\n" + + chunks := splitChunks(text, 10) + if len(chunks) != 2 { + t.Fatalf("expected 2 chunks, got %d: %+v", len(chunks), chunks) + } + + if chunks[1].text == "" { + t.Errorf("chunk 1 = %+v, want no trailing empty chunk", chunks[1]) + } + + if chunks[1].sep != "\n\n" { + t.Errorf("chunk 1 sep = %q, want the trailing newline run", chunks[1].sep) + } + + if reassemble(chunks) != text { + t.Errorf("reassembled = %q, want original", reassemble(chunks)) + } +} + +// TestSplitChunksLeadingBlankLineRunKeptAsSeparator verifies that a blank-line +// run before a long document's first paragraph becomes its own empty-text +// chunk carrying the run as sep, rather than being folded into the first +// real chunk's text — which would send the document's leading whitespace to +// the LLM with no instruction to preserve it. +func TestSplitChunksLeadingBlankLineRunKeptAsSeparator(t *testing.T) { + para := strings.TrimSuffix(strings.Repeat("hello world ", 200), " ") // > 2000 runes + text := "\n\n" + para + + chunks := splitChunks(text, 2000) + if len(chunks) < 2 { + t.Fatalf("expected at least 2 chunks, got %d", len(chunks)) + } + + if chunks[0].text != "" || chunks[0].sep != "\n\n" { + t.Errorf("chunk 0 = %+v, want the leading blank-line run isolated as its own empty-text chunk", chunks[0]) + } + + if strings.HasPrefix(chunks[1].text, "\n") { + t.Errorf("chunk 1 text = %q, must not carry the leading separator into real content", chunks[1].text) + } + + if reassemble(chunks) != text { + t.Errorf("reassembled = %q, want original", reassemble(chunks)) + } +} + +func TestSplitChunksCRLFParagraphBoundaries(t *testing.T) { + // CRLF blank lines must be recognized as paragraph boundaries just like + // bare "\n\n", with the "\r" bytes captured in sep rather than leaking + // into the text sent to the LLM. + text := "alpha one\r\n\r\nbravo two" + + chunks := splitChunks(text, 10) + if len(chunks) != 2 { + t.Fatalf("expected 2 chunks, got %d: %+v", len(chunks), chunks) + } + + want := []chunk{ + {text: "alpha one", sep: "\r\n\r\n"}, + {text: "bravo two", sep: ""}, + } + for i, c := range chunks { + if c != want[i] { + t.Errorf("chunk %d = %+v, want %+v", i, c, want[i]) + } + } + + if reassemble(chunks) != text { + t.Errorf("reassembled = %q, want original", reassemble(chunks)) + } +} + +// TestSplitChunksOversizedParagraphTrailingNewlinePreserved verifies that an +// oversized paragraph whose text ends in a single trailing newline (only +// possible for the document's final unit, where there is no paragraph-level +// separator to reattach) keeps that newline instead of losing it when +// subdivideUnit reattaches its own (empty) separator. +func TestSplitChunksOversizedParagraphTrailingNewlinePreserved(t *testing.T) { + text := "line one is here\nline two is here\n" + + chunks := splitChunks(text, 20) + if got := reassemble(chunks); got != text { + t.Errorf("reassemble(splitChunks(%q, 20)) = %q, want original (trailing newline lost)", text, got) + } +} + +// TestSplitChunksLineSplitDoesNotCorruptWordWhenSpaceExists verifies that +// splitLine finds a break point anywhere in the window (not just its second +// half), so a word shorter than the limit is never cut mid-word just because +// its only available space sits early in the window. +func TestSplitChunksLineSplitDoesNotCorruptWordWhenSpaceExists(t *testing.T) { + chunks := splitChunks("a bcde", 4) + for _, c := range chunks { + if c.text == "a bc" || c.text == "de" { + t.Errorf("chunks = %+v, want the word \"bcde\" (4 runes, within limit) kept intact", chunks) + } + } + + if got := reassemble(chunks); got != "a bcde" { + t.Errorf("reassembled = %q, want original", got) + } +} + +// TestSplitChunksLineSplitFindsSpaceAtLimitBoundary verifies that a +// whitespace rune sitting exactly at the limit boundary is used as the break +// point, rather than being missed and left as a leading space on the next +// piece (which would fuse translated pieces together, since the piece's own +// sep would then be empty). +func TestSplitChunksLineSplitFindsSpaceAtLimitBoundary(t *testing.T) { + chunks := splitChunks("abcd efg", 4) + + want := []chunk{ + {text: "abcd", sep: " "}, + {text: "efg", sep: ""}, + } + for i, c := range chunks { + if c != want[i] { + t.Errorf("chunk %d = %+v, want %+v", i, c, want[i]) + } + } + + if got := reassemble(chunks); got != "abcd efg" { + t.Errorf("reassembled = %q, want original", got) + } +} + +func TestIndexNewlineRun(t *testing.T) { + tests := []struct { + s string + minRun int + wantStart int + wantEnd int + }{ + {"abc", 1, -1, -1}, + {"a\nb", 1, 1, 2}, + {"a\nb", 2, -1, -1}, + {"a\n\nb", 2, 1, 3}, + {"ab\n\n\ncd", 2, 2, 5}, + {"\n\nab", 2, 0, 2}, + {"a\nb\n\nc", 2, 3, 5}, + // CRLF: "\r\n" is one newline token. + {"a\r\nb", 1, 1, 3}, + {"a\r\n\r\nb", 2, 1, 5}, + {"a\r\nb", 2, -1, -1}, + // A lone "\r" not followed by "\n" is not a newline token. + {"a\rb", 1, -1, -1}, + } + for _, tt := range tests { + start, end := indexNewlineRun(tt.s, tt.minRun) + if start != tt.wantStart || end != tt.wantEnd { + t.Errorf("indexNewlineRun(%q, %d) = (%d, %d), want (%d, %d)", + tt.s, tt.minRun, start, end, tt.wantStart, tt.wantEnd) + } + } +} diff --git a/cmd/mint/main.go b/cmd/mint/main.go index 9bf866f..aacae70 100644 --- a/cmd/mint/main.go +++ b/cmd/mint/main.go @@ -207,6 +207,14 @@ func newRootCmd() *cobra.Command { return nil } + // Long input is split into chunks so no single request can hit the + // model's output-token limit; each chunk is translated in its own + // request and streamed out in order. + chunks := splitChunks(text, maxChunkRunes) + if len(chunks) > 1 { + logv("long input — split into %d chunks (max %d runes each)", len(chunks), maxChunkRunes) + } + switch { case len(targetLangs) == 1: actualTargetLang = targetLangs[0] @@ -219,55 +227,50 @@ func newRootCmd() *cobra.Command { actualTargetLang = determineActualTargetLang(sourceLang, targetLangs) default: - inputLang, detectUsage, err := detectLanguage(ctx, t, text) + inputLang, detectUsage, err := detectLanguage(ctx, t, detectSample(chunks)) if err != nil { return fmt.Errorf("language detection failed: %w", err) } logv("detected input language: %q", canonicalLangTag(inputLang)) - // Language-neutral content (numbers, symbols): output unchanged, - // no rewrite call needed. - if inputLang == "" { + totalUsage.Add(detectUsage) + + switch { + case inputLang != "": + actualTargetLang = determineActualTargetLang(inputLang, targetLangs) + case len(chunks) == 1: + // The whole document is this one chunk and the LLM says + // it's language-neutral: output unchanged, no rewrite + // call needed. logv("language-neutral content — outputting unchanged") - logv("tokens: %d in / %d out", detectUsage.InputTokens, detectUsage.OutputTokens) + logv("tokens: %d in / %d out", totalUsage.InputTokens, totalUsage.OutputTokens) fmt.Println(text) return nil + default: + // The detection sample came back neutral, but it's only + // one chunk of a longer document — a later chunk may + // carry the real language, so the whole document can't be + // passed through on this alone. Fall back to the first + // configured target; each chunk's own neutral check still + // skips the LLM call for genuinely neutral chunks. + logv("neutral detection sample in a multi-chunk document — defaulting to first target") + + actualTargetLang = determineActualTargetLang(inputLang, targetLangs) } - - totalUsage.InputTokens += detectUsage.InputTokens - totalUsage.OutputTokens += detectUsage.OutputTokens - - actualTargetLang = determineActualTargetLang(inputLang, targetLangs) } logv("target language: %s", canonicalLangTag(actualTargetLang)) - // Rewrite the input in the target language, correcting grammar and - // spelling along the way. Anchoring on the target tag also pins the - // output script, so the model can't drift into the wrong variant - // (e.g. Simplified for zh-TW). - system, user, nonce := buildRewritePrompt(sourceLang, actualTargetLang, text) - - // Weaker models occasionally echo the nonce delimiter back into the - // output. Filter it out before it reaches the user — the nonce is - // our own injected format noise and must never be visible. - out := newNonceFilter(os.Stdout, nonce) + translateUsage, err := translateChunks(ctx, t, chunks, sourceLang, actualTargetLang) + totalUsage.Add(translateUsage) + logv("tokens: %d in / %d out", totalUsage.InputTokens, totalUsage.OutputTokens) - translateUsage, err := t.Complete(ctx, system, user, out) if err != nil { - return fmt.Errorf("translation failed: %w", err) - } - - if err := out.Flush(); err != nil { return err } - totalUsage.InputTokens += translateUsage.InputTokens - totalUsage.OutputTokens += translateUsage.OutputTokens - logv("tokens: %d in / %d out", totalUsage.InputTokens, totalUsage.OutputTokens) - return nil }, } @@ -469,6 +472,65 @@ func (f *nonceFilter) Flush() error { return err } +// newlineTrimWriter wraps an io.Writer and withholds any trailing run of +// complete newline tokens ("\n" or "\r\n"), releasing them only once further +// content arrives. Done discards whatever run remains withheld once the +// stream is complete. +// +// This lets translateChunk print a chunk's translation without committing to +// how many trailing newlines the model happened to emit (including a stray +// '\r' from a model that writes CRLF line endings) — the caller reprints the +// chunk's own original separator afterward, so the stream here must not carry +// a trailing newline of its own. A lone '\r' not paired with a following '\n' +// is not a newline token (matching chunk.go's own definition) and is never +// withheld, so it cannot be mistaken for part of a trimmable run. +type newlineTrimWriter struct { + w io.Writer + pending []byte +} + +func newNewlineTrimWriter(w io.Writer) *newlineTrimWriter { + return &newlineTrimWriter{w: w} +} + +// trimTrailingNewlineRun splits b into content and the maximal trailing run +// of newline tokens ("\n" or "\r\n") at its end. A lone trailing '\r' not +// paired with a following '\n' is not a newline token and stays in content. +func trimTrailingNewlineRun(b []byte) (content, trimmed []byte) { + end := len(b) + + for end > 0 && b[end-1] == '\n' { + if end >= 2 && b[end-2] == '\r' { + end -= 2 + } else { + end-- + } + } + + return b[:end], b[end:] +} + +func (t *newlineTrimWriter) Write(p []byte) (int, error) { + t.pending = append(t.pending, p...) + content, trimmed := trimTrailingNewlineRun(t.pending) + + if len(content) > 0 { + if _, err := t.w.Write(content); err != nil { + return 0, err + } + } + + t.pending = append([]byte(nil), trimmed...) + + return len(p), nil +} + +// Done discards any withheld trailing newlines. Must be called once, after +// the final Write. +func (t *newlineTrimWriter) Done() { + t.pending = nil +} + // buildRewritePrompt builds the system instruction and user message for the // rewrite/translation task. Instructions go to system (LLM role boundary); // only the nonce-wrapped user text goes to user (untrusted input). @@ -570,6 +632,119 @@ func isLangNeutral(text string) bool { return true } +// detectSample returns the first chunk that contains letters, falling back to +// the first chunk. Detection only needs a sample, not the whole document, so +// long input does not burn tokens on the detection call. +func detectSample(chunks []chunk) string { + for _, c := range chunks { + if !isLangNeutral(c.text) { + return c.text + } + } + + return chunks[0].text +} + +// translateChunks rewrites each chunk in the target language, correcting +// grammar and spelling along the way, and streams the results to stdout in +// order. Anchoring on the target tag also pins the output script, so the +// model can't drift into the wrong variant (e.g. Simplified for zh-TW). +// +// Each chunk's original separator (a paragraph gap, a line break, a single +// space, or nothing) is reprinted verbatim between chunks, so the document's +// exact internal whitespace survives translation regardless of how it was +// split. The whole document always ends with exactly one trailing newline — +// the last chunk's own separator (e.g. trailing blank lines in the original +// input) is not reprinted, and each chunk's own trailing newline run is +// trimmed before printing (see newlineTrimWriter) — regardless of how many +// newlines the model or the original input produced. +func translateChunks( + ctx context.Context, + t llm.Completer, + chunks []chunk, + sourceLang, targetLang string, +) (llm.Usage, error) { + var total llm.Usage + + chunked := len(chunks) > 1 + + // wrapErr names the chunk a failure happened in, for every error this + // function can return — not just translateChunk's own — so a write + // failure on a reprinted separator or the final newline is identified + // exactly like a translation failure. + wrapErr := func(err error, chunkNum int) error { + if chunked { + return fmt.Errorf("%w (chunk %d/%d)", err, chunkNum, len(chunks)) + } + + return err + } + + for i, c := range chunks { + usage, err := translateChunk(ctx, t, c, sourceLang, targetLang) + total.Add(usage) + + if err != nil { + return total, wrapErr(err, i+1) + } + + if i < len(chunks)-1 { + if _, err := fmt.Print(c.sep); err != nil { + return total, wrapErr(fmt.Errorf("write output: %w", err), i+1) + } + } + } + + if _, err := fmt.Println(); err != nil { + return total, wrapErr(fmt.Errorf("write output: %w", err), len(chunks)) + } + + return total, nil +} + +// translateChunk translates a single chunk to stdout, without a trailing +// newline of its own — translateChunks reprints the chunk's original +// separator afterward instead. A chunk with no letters (a block of numbers, a +// divider line) needs no translation and is printed unchanged. +func translateChunk(ctx context.Context, t llm.Completer, c chunk, sourceLang, targetLang string) (llm.Usage, error) { + if isLangNeutral(c.text) { + trim := newNewlineTrimWriter(os.Stdout) + if _, err := trim.Write([]byte(c.text)); err != nil { + return llm.Usage{}, fmt.Errorf("write output: %w", err) + } + + trim.Done() + + return llm.Usage{}, nil + } + + system, user, nonce := buildRewritePrompt(sourceLang, targetLang, c.text) + + // Every provider guarantees its stream ends with a newline, and a chatty + // model may add more of its own; trim that trailing run rather than + // assuming its size, since the chunk's real separator is reprinted by the + // caller instead. + trim := newNewlineTrimWriter(os.Stdout) + + // Weaker models occasionally echo the nonce delimiter back into the + // output. Filter it out before it reaches the user — the nonce is our own + // injected format noise and must never be visible. + out := newNonceFilter(trim, nonce) + + usage, err := t.Complete(ctx, system, user, out) + if err != nil { + return usage, fmt.Errorf("translation failed: %w", err) + } + + if err := out.Flush(); err != nil { + return usage, fmt.Errorf("write output: %w", err) + } + + trim.Done() + + return usage, nil +} + // detectLanguage detects the language of the input text. // Returns empty string if the input is language-neutral (e.g., numbers, symbols). func detectLanguage(ctx context.Context, t llm.Completer, text string) (string, llm.Usage, error) { diff --git a/cmd/mint/main_test.go b/cmd/mint/main_test.go index acc49d6..0048fd1 100644 --- a/cmd/mint/main_test.go +++ b/cmd/mint/main_test.go @@ -1068,3 +1068,431 @@ func TestNewRootCmdSourceRotation(t *testing.T) { t.Errorf("expected exactly 1 LLM call (no detection), got %d", calls) } } + +// TestNewRootCmdChunkedTranslation verifies that input longer than +// maxChunkRunes is split at paragraph boundaries, translated in one request +// per chunk, and reassembled with the original blank-line separation. +func TestNewRootCmdChunkedTranslation(t *testing.T) { + var calls int + + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + calls++ + + w.Header().Set("Content-Type", "text/event-stream") + _, _ = io.WriteString(w, "data: {\"choices\":[{\"delta\":{\"content\":\"OK\"}}]}\n\ndata: [DONE]\n") + })) + defer srv.Close() + + t.Setenv("MINT_PROVIDER", "openai") + t.Setenv("MINT_API_KEY", "test") + t.Setenv("MINT_BASE_URL", srv.URL) + t.Setenv("MINT_MODEL_NAME", "test-model") + + // Two paragraphs of ~1500 runes each: together they exceed maxChunkRunes, + // so each becomes its own chunk (and its own request). + para := strings.TrimSuffix(strings.Repeat("hello world ", 125), " ") + input := para + "\n\n" + para + + flush := captureStdout(t) + + cmd := newRootCmd() + cmd.SetArgs([]string{"--target", "fr", input}) + + if err := cmd.ExecuteContext(context.Background()); err != nil { + _ = flush() + + t.Fatalf("unexpected error: %v", err) + } + + if got, want := flush(), "OK\n\nOK\n"; got != want { + t.Errorf("output = %q, want %q", got, want) + } + + if calls != 2 { + t.Errorf("expected 2 LLM calls (one per chunk), got %d", calls) + } +} + +// TestNewRootCmdChunkedLeadingBlankLine verifies that a blank-line run before +// a long document's first paragraph survives in the output even though the +// mock LLM never echoes it — proving the leading separator is reprinted by +// the CLI itself rather than sent to the model as part of a chunk's text. +func TestNewRootCmdChunkedLeadingBlankLine(t *testing.T) { + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + w.Header().Set("Content-Type", "text/event-stream") + _, _ = io.WriteString(w, "data: {\"choices\":[{\"delta\":{\"content\":\"OK\"}}]}\n\ndata: [DONE]\n") + })) + defer srv.Close() + + t.Setenv("MINT_PROVIDER", "openai") + t.Setenv("MINT_API_KEY", "test") + t.Setenv("MINT_BASE_URL", srv.URL) + t.Setenv("MINT_MODEL_NAME", "test-model") + + // A leading blank-line run before a single paragraph long enough to force + // chunking; the mock always answers "OK", so if the leading "\n\n" showed + // up in the output it could only have come from the reprinted separator. + para := strings.TrimSuffix(strings.Repeat("hello world ", 200), " ") + input := "\n\n" + para + + flush := captureStdout(t) + + cmd := newRootCmd() + cmd.SetArgs([]string{"--target", "fr", input}) + + if err := cmd.ExecuteContext(context.Background()); err != nil { + _ = flush() + + t.Fatalf("unexpected error: %v", err) + } + + if got := flush(); !strings.HasPrefix(got, "\n\nOK") { + t.Errorf("output = %q, want it to start with the preserved leading blank-line run", got) + } +} + +// TestNewRootCmdChunkedNeutralChunk verifies that a language-neutral chunk +// (no letters) inside a long document is printed unchanged without an LLM +// call, while the surrounding chunks are still translated. +func TestNewRootCmdChunkedNeutralChunk(t *testing.T) { + var calls int + + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + calls++ + + w.Header().Set("Content-Type", "text/event-stream") + _, _ = io.WriteString(w, "data: {\"choices\":[{\"delta\":{\"content\":\"OK\"}}]}\n\ndata: [DONE]\n") + })) + defer srv.Close() + + t.Setenv("MINT_PROVIDER", "openai") + t.Setenv("MINT_API_KEY", "test") + t.Setenv("MINT_BASE_URL", srv.URL) + t.Setenv("MINT_MODEL_NAME", "test-model") + + // A 1900-rune digit paragraph between two 1500-rune text paragraphs: too + // big to pack with either neighbor, so it becomes its own (neutral) chunk. + para := strings.TrimSuffix(strings.Repeat("hello world ", 125), " ") + digits := strings.Repeat("1234567890", 190) + input := para + "\n\n" + digits + "\n\n" + para + + flush := captureStdout(t) + + cmd := newRootCmd() + cmd.SetArgs([]string{"--target", "fr", input}) + + if err := cmd.ExecuteContext(context.Background()); err != nil { + _ = flush() + + t.Fatalf("unexpected error: %v", err) + } + + if got, want := flush(), "OK\n\n"+digits+"\n\nOK\n"; got != want { + t.Errorf("output = %q, want %q", got, want) + } + + if calls != 2 { + t.Errorf("expected 2 LLM calls (neutral chunk skips the LLM), got %d", calls) + } +} + +// TestNewRootCmdChunkedNeutralFinalChunkTrailingNewline verifies that a +// language-neutral final chunk whose own text ends in a single trailing +// newline (not a paragraph-boundary run, so it stays inside the chunk's text +// rather than becoming a separator) does not produce a second trailing +// newline on top of the one translateChunks always appends. +func TestNewRootCmdChunkedNeutralFinalChunkTrailingNewline(t *testing.T) { + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + w.Header().Set("Content-Type", "text/event-stream") + _, _ = io.WriteString(w, "data: {\"choices\":[{\"delta\":{\"content\":\"OK\"}}]}\n\ndata: [DONE]\n") + })) + defer srv.Close() + + t.Setenv("MINT_PROVIDER", "openai") + t.Setenv("MINT_API_KEY", "test") + t.Setenv("MINT_BASE_URL", srv.URL) + t.Setenv("MINT_MODEL_NAME", "test-model") + + // The digit block ends in a single "\n" — below the 2-newline run needed + // to be captured as a separator — so it stays inside the final chunk's + // own text and that chunk is language-neutral (no letters). + para := strings.TrimSuffix(strings.Repeat("hello world ", 125), " ") + digits := strings.Repeat("1234567890", 190) + "\n" + input := para + "\n\n" + digits + + flush := captureStdout(t) + + cmd := newRootCmd() + cmd.SetArgs([]string{"--target", "fr", input}) + + if err := cmd.ExecuteContext(context.Background()); err != nil { + _ = flush() + + t.Fatalf("unexpected error: %v", err) + } + + if got, want := flush(), "OK\n\n"+digits; got != want { + t.Errorf("output = %q, want %q (exactly one trailing newline, not two)", got, want) + } +} + +// TestNewRootCmdChunkedErrorNamesChunk verifies that a failure mid-document +// reports which chunk failed. +func TestNewRootCmdChunkedErrorNamesChunk(t *testing.T) { + var calls int + + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + calls++ + if calls > 1 { + w.WriteHeader(http.StatusInternalServerError) + _, _ = io.WriteString(w, `{"error":"server error"}`) + + return + } + + w.Header().Set("Content-Type", "text/event-stream") + _, _ = io.WriteString(w, "data: {\"choices\":[{\"delta\":{\"content\":\"OK\"}}]}\n\ndata: [DONE]\n") + })) + defer srv.Close() + + t.Setenv("MINT_PROVIDER", "openai") + t.Setenv("MINT_API_KEY", "test") + t.Setenv("MINT_BASE_URL", srv.URL) + t.Setenv("MINT_MODEL_NAME", "test-model") + + para := strings.TrimSuffix(strings.Repeat("hello world ", 125), " ") + input := para + "\n\n" + para + + flush := captureStdout(t) + defer flush() + + cmd := newRootCmd() + cmd.SetArgs([]string{"--target", "fr", input}) + + err := cmd.ExecuteContext(context.Background()) + if err == nil { + t.Fatal("expected error, got nil") + } + + if !strings.Contains(err.Error(), "chunk 2/2") { + t.Errorf("error %q does not name the failing chunk", err.Error()) + } +} + +// TestNewRootCmdChunkedSpaceSplitJoinsWithSpace verifies that a single +// no-newline line long enough to be word-split keeps its pieces joined by +// the original space, not a hard-injected newline (every provider guarantees +// its own stream ends in a newline, which must not leak into the join). +func TestNewRootCmdChunkedSpaceSplitJoinsWithSpace(t *testing.T) { + var calls int + + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + calls++ + + w.Header().Set("Content-Type", "text/event-stream") + _, _ = io.WriteString(w, "data: {\"choices\":[{\"delta\":{\"content\":\"OK\"}}]}\n\ndata: [DONE]\n") + })) + defer srv.Close() + + t.Setenv("MINT_PROVIDER", "openai") + t.Setenv("MINT_API_KEY", "test") + t.Setenv("MINT_BASE_URL", srv.URL) + t.Setenv("MINT_MODEL_NAME", "test-model") + + // A single line with no newlines at all, long enough that it must be + // word-split into two chunks. + input := strings.TrimSuffix(strings.Repeat("hello world ", 210), " ") + + flush := captureStdout(t) + + cmd := newRootCmd() + cmd.SetArgs([]string{"--target", "fr", input}) + + if err := cmd.ExecuteContext(context.Background()); err != nil { + _ = flush() + + t.Fatalf("unexpected error: %v", err) + } + + if got, want := flush(), "OK OK\n"; got != want { + t.Errorf("output = %q, want %q (pieces joined by the original space, not a newline)", got, want) + } + + if calls != 2 { + t.Errorf("expected 2 LLM calls (one per word-split piece), got %d", calls) + } +} + +// TestNewRootCmdChunkedNeutralSampleFallsBackToTranslation verifies that when +// rotation-mode detection samples a chunk the model classifies as "neutral" +// in a multi-chunk document, the whole document is NOT passed through +// unchanged — a neutral sample doesn't mean the rest of a longer document is +// neutral too. Translation proceeds using the first configured target. +func TestNewRootCmdChunkedNeutralSampleFallsBackToTranslation(t *testing.T) { + var calls int + + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + calls++ + + w.Header().Set("Content-Type", "text/event-stream") + _, _ = io.WriteString(w, "data: {\"choices\":[{\"delta\":{\"content\":\"neutral\"}}]}\n\ndata: [DONE]\n") + })) + defer srv.Close() + + t.Setenv("MINT_PROVIDER", "openai") + t.Setenv("MINT_API_KEY", "test") + t.Setenv("MINT_BASE_URL", srv.URL) + t.Setenv("MINT_MODEL_NAME", "test-model") + t.Setenv("MINT_TARGET_LANG", "en,fr") + + para := strings.TrimSuffix(strings.Repeat("hello world ", 125), " ") + input := para + "\n\n" + para + + flush := captureStdout(t) + defer flush() + + cmd := newRootCmd() + cmd.SetArgs([]string{input}) + + if err := cmd.ExecuteContext(context.Background()); err != nil { + t.Fatalf("unexpected error: %v", err) + } + + // 1 detection call (sampling only the first chunk) + 1 translation call + // per chunk — not a single detection call followed by a verbatim + // pass-through of the whole document. + if calls != 3 { + t.Errorf("expected 3 LLM calls (1 detect + 2 translate), got %d", calls) + } +} + +// TestNewRootCmdChunkedUsageLoggedOnError verifies that token usage already +// spent on earlier, successful chunks is still logged in verbose mode even +// when a later chunk's request fails. +func TestNewRootCmdChunkedUsageLoggedOnError(t *testing.T) { + var calls int + + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + calls++ + if calls > 1 { + w.WriteHeader(http.StatusInternalServerError) + _, _ = io.WriteString(w, `{"error":"server error"}`) + + return + } + + w.Header().Set("Content-Type", "text/event-stream") + _, _ = io.WriteString(w, "data: {\"choices\":[{\"delta\":{\"content\":\"OK\"}}]}\n\ndata: [DONE]\n") + })) + defer srv.Close() + + t.Setenv("MINT_PROVIDER", "openai") + t.Setenv("MINT_API_KEY", "test") + t.Setenv("MINT_BASE_URL", srv.URL) + t.Setenv("MINT_MODEL_NAME", "test-model") + + para := strings.TrimSuffix(strings.Repeat("hello world ", 125), " ") + input := para + "\n\n" + para + + flushOut := captureStdout(t) + flushErr := captureStderr(t) + + cmd := newRootCmd() + cmd.SetArgs([]string{"--target", "fr", "--verbose", input}) + + err := cmd.ExecuteContext(context.Background()) + + _ = flushOut() + stderr := flushErr() + + if err == nil { + t.Fatal("expected error, got nil") + } + + if !strings.Contains(stderr, "tokens:") { + t.Errorf("stderr = %q, want a tokens: line despite the mid-document failure", stderr) + } +} + +// TestNewRootCmdChunkedTrailingNewlineRunNormalized verifies that a long, +// multi-chunk document whose original input ends in a run of blank lines +// still produces output with exactly one trailing newline — matching the +// single-chunk/unchunked behavior — instead of reprinting the last chunk's +// own separator on top of the terminating newline. +func TestNewRootCmdChunkedTrailingNewlineRunNormalized(t *testing.T) { + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + w.Header().Set("Content-Type", "text/event-stream") + _, _ = io.WriteString(w, "data: {\"choices\":[{\"delta\":{\"content\":\"OK\"}}]}\n\ndata: [DONE]\n") + })) + defer srv.Close() + + t.Setenv("MINT_PROVIDER", "openai") + t.Setenv("MINT_API_KEY", "test") + t.Setenv("MINT_BASE_URL", srv.URL) + t.Setenv("MINT_MODEL_NAME", "test-model") + + // A positional-arg document (unlike stdin, args are not newline-trimmed) + // whose two paragraphs are followed by a trailing blank-line run. + para := strings.TrimSuffix(strings.Repeat("hello world ", 125), " ") + input := para + "\n\n" + para + "\n\n" + + flush := captureStdout(t) + + cmd := newRootCmd() + cmd.SetArgs([]string{"--target", "fr", input}) + + if err := cmd.ExecuteContext(context.Background()); err != nil { + _ = flush() + + t.Fatalf("unexpected error: %v", err) + } + + if got, want := flush(), "OK\n\nOK\n"; got != want { + t.Errorf( + "output = %q, want %q (exactly one trailing newline, not the original blank-line run plus one)", + got, + want, + ) + } +} + +// TestNewlineTrimWriter verifies that newlineTrimWriter withholds a trailing +// run of '\n' bytes across Write calls and discards it entirely on Done, +// while newlines followed by further content pass through unchanged. +func TestNewlineTrimWriter(t *testing.T) { + tests := []struct { + name string + writes []string + want string + }{ + {"no trailing newline", []string{"foo"}, "foo"}, + {"single trailing newline", []string{"foo\n"}, "foo"}, + {"multiple trailing newlines", []string{"foo\n\n\n"}, "foo"}, + {"blank line mid-stream is preserved", []string{"foo\n\nbar"}, "foo\n\nbar"}, + {"trailing newlines split across writes", []string{"foo", "\n", "\n"}, "foo"}, + {"newline then more content flushes it", []string{"foo\n", "bar"}, "foo\nbar"}, + {"only newlines", []string{"\n\n\n"}, ""}, + {"no writes at all", []string{}, ""}, + {"trailing CRLF is fully withheld, no stray \\r", []string{"foo\r\n"}, "foo"}, + {"CRLF then more content flushes it verbatim", []string{"foo\r\n", "bar"}, "foo\r\nbar"}, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + var buf bytes.Buffer + + trim := newNewlineTrimWriter(&buf) + for _, w := range tt.writes { + if _, err := trim.Write([]byte(w)); err != nil { + t.Fatalf("Write(%q): %v", w, err) + } + } + + trim.Done() + + if got := buf.String(); got != tt.want { + t.Errorf("got %q, want %q", got, tt.want) + } + }) + } +} diff --git a/docs/manual-testing.md b/docs/manual-testing.md index 818fe77..9492ad7 100644 --- a/docs/manual-testing.md +++ b/docs/manual-testing.md @@ -278,3 +278,32 @@ Verbose stderr output (single-target mode): The `model:` line appears only when `MINT_MODEL_NAME` is set (likewise `base_url:` for `MINT_BASE_URL`); with provider defaults both are omitted, as in the abbreviated example at the top of this file. + +## 15. Long-input chunked translation + +Input longer than 2000 runes is split at paragraph boundaries (falling back to line +breaks, then spaces) and translated chunk by chunk — one LLM request per chunk — so a +long document cannot hit the model's output-token limit and come back truncated. Blank +lines between paragraphs are preserved in the output. Shorter input is unaffected: +one request, behavior identical to before. + +```sh +# Build a ~4500-rune three-paragraph document, then translate it +python3 -c 'p = ("hello world " * 125).strip(); print(p + "\n\n" + p + "\n\n" + p, end="")' > /tmp/long.txt +mint -t zh-TW -v < /tmp/long.txt +# [mint] long input — split into 3 chunks (max 2000 runes each) +# ... three translated paragraphs, separated by blank lines +``` + +In rotation mode, language detection samples the first chunk that contains letters +(falling back to the first chunk if none do), so detection cost does not grow with +document length. If that sample comes back language-neutral, a single-chunk document is +printed unchanged as usual, but a multi-chunk document falls back to the first +configured target instead — a neutral sample doesn't mean the rest of a longer document +is neutral too. A language-neutral chunk (e.g. a block of numbers) is printed unchanged +with no LLM call. If a request fails mid-document, the error names the chunk: +`Error: translation failed: ... (chunk 2/3)`, and any tokens already spent on earlier +chunks — including the failing chunk's own usage, if the provider returned any alongside +the error (e.g. hitting the output-token limit) — are still reported in verbose mode. A +local output-write failure (e.g. a closed pipe) is named the same way but is not +mislabeled as a translation failure: `Error: write output: ... (chunk 2/3)`. diff --git a/internal/llm/llm.go b/internal/llm/llm.go index 2f54670..aa61744 100644 --- a/internal/llm/llm.go +++ b/internal/llm/llm.go @@ -14,6 +14,12 @@ type Usage struct { OutputTokens int } +// Add accumulates token counts from another Usage. +func (u *Usage) Add(o Usage) { + u.InputTokens += o.InputTokens + u.OutputTokens += o.OutputTokens +} + // Completer sends a prompt to an LLM and streams the response. // system holds the task instructions; user holds the (possibly untrusted) input data. // Implementations write tokens directly to w as they arrive.