Skip to content

increase buffer size for stream to lines to accomodate oversize packets #3112

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
sawka merged 1 commit into from
Mar 24, 2026
Merged

increase buffer size for stream to lines to accomodate oversize packets #3112

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion pkg/util/utilfn/streamtolines.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,9 @@ type lineBuf struct {
inLongLine bool
}

const maxLineLength = 128 * 1024
// needs to be large enough to read the largest RPC packet
// there are some legacy file transfer packets that can send up to 32m (base64 encoded)
const maxLineLength = 64 * 1024 * 1024
Comment on lines +23 to +25

@coderabbitai coderabbitai Bot Mar 24, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

Scope the 64MB cap to RPC-specific usage instead of changing the global default.

This constant is shared by both StreamToLines and LineWriter, so Line 25 now allows ~64MB buffering per in-flight line for all callers. That can materially increase memory pressure under concurrent streams. Prefer a configurable limit (or an RPC-specific constructor/path with 64MB) while keeping a safer general default.

🤖 Prompt for AI Agents 
Verify each finding against the current code and only fix it if needed.

In `@pkg/util/utilfn/streamtolines.go` around lines 23 - 25, The global const
maxLineLength is too large for all callers and should be scoped to RPC usage;
change the shared constant into a smaller safe default (e.g.,
defaultMaxLineLength) and add an RPC-specific limit (e.g., rpcMaxLineLength) or
an explicit constructor/parameter for StreamToLines and LineWriter so callers
can opt into the 64MB cap only for RPC streams; update StreamToLines and
LineWriter to accept a configurable maxLineLength parameter (or provide
NewStreamToLinesWithLimit / NewLineWriterWithLimit) and use defaultMaxLineLength
when not provided, reserving the 64MB value only for RPC-specific construction
paths.


func ReadLineWithTimeout(ch chan LineOutput, timeout time.Duration) (string, error) {
select {
Expand Down
Loading