Skip to content

Reduce memory usage while converting files#3

Merged
rylanmoseley merged 10 commits intomainfrom
dont-eat-all-the-memory
Apr 14, 2026
Merged

Reduce memory usage while converting files#3
rylanmoseley merged 10 commits intomainfrom
dont-eat-all-the-memory

Conversation

@rylanmoseley
Copy link
Copy Markdown
Member

Instead of reading the whole REVLOG into a buffer, then writing the whole output WPILOG to a buffer, then writing the buffer to an output file or passing it on to whatever called it (very large buffers, lots of memory, potential for crashes with large enough files), this uses streams and an accumulator to parse chunks at a time.

This removes the options of passing in or receiving a Buffer, and users should either pass in/receive out streams or files.

Resolves #1

Draft for now until I can get more testing in, and test the change with Ascope. Works locally with files that made previous version crash. Supposedly the for await loop removes the need for the setImmediate call, but that still needs to be verified.

@rylanmoseley rylanmoseley marked this pull request as ready for review April 14, 2026 21:16
@rylanmoseley rylanmoseley merged commit 386c9d2 into main Apr 14, 2026
4 checks passed
@rylanmoseley rylanmoseley deleted the dont-eat-all-the-memory branch April 14, 2026 21:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Crash when opening large REVLOG files

2 participants