You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Lime currently sends all libraries handled by the same asset handler to a single haxelib run <handler> process invocation. For SWF projects with many libraries, this means one Neko process processes every SWF in sequence.
In large real-world projects with hundreds of SWFs, memory keeps growing throughout preprocessing until it reaches several GB and can eventually fail with a Neko GC/heap error. When that happens after some SWF caches have already been written, later rebuilds may reuse the partial cache state and fail with missing generated Haxe classes.
This change processes SWF libraries in separate handler runs, one library per invocation, while preserving the existing merge behavior for each handler output. This bounds memory usage to a single SWF processing run instead of accumulating across the entire SWF set.
swf-memory-repro.zip
It generates SWF files locally and demonstrates memory growth in the current batched mode.
This commit fixes a performance regression. Spawning a new neko process for each individual SWF library caused significant overhead, making repeat builds (where assets are already cached and up-to-date) excessively slow.
To resolve this, I introduced a "fast-check" pass before calling runLibraryHandler. The logic now checks if the generated cache files (.zip and .classes.txt) are newer than both the source .swf file and the swf tool itself.
If a valid cache is found, Lime now bypasses the neko process entirely and manually merges the cached assets, generated source paths (haxe/_generated), haxelibs (swf), and required haxeflags into the project. This restores near-instant performance for cached SWF assets while leaving the remaining stale ones to be processed correctly.
Since the memory-limit fix isolated each SWF into its own separate neko process, it unlocked the perfect opportunity to process them concurrently. This commit introduces a multi-threaded worker pool using sys.thread to take advantage of this new architecture.
How it works:
Uncached SWF libraries are pushed into an array of LibraryHandlerJob.
runLibraryHandlers processes these jobs in parallel, utilizing all available CPU cores (System.processorCores).
A Mutex coordinates the queue, and a Lock ensures the main thread waits for all workers to complete before merging the resulting HXProject fragments.
Result:
Because the SWF generation is heavily CPU-bound, parallelizing these independent neko instances drastically reduces cold-build times (e.g., dropping from ~9 minutes to under 2 minutes on my project), while still safely bypassing the memory limits of the original single-process architecture.
Tutez64
changed the title
Process SWF libraries in separate handler runs to avoid unbounded memory growth
Refactor SWF processing to fix memory exhaustion and improve compilation speed
May 12, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Lime currently sends all libraries handled by the same asset handler to a single
haxelib run <handler> processinvocation. For SWF projects with many libraries, this means one Neko process processes every SWF in sequence.In large real-world projects with hundreds of SWFs, memory keeps growing throughout preprocessing until it reaches several GB and can eventually fail with a Neko GC/heap error. When that happens after some SWF caches have already been written, later rebuilds may reuse the partial cache state and fail with missing generated Haxe classes.
This change processes SWF libraries in separate handler runs, one library per invocation, while preserving the existing merge behavior for each handler output. This bounds memory usage to a single SWF processing run instead of accumulating across the entire SWF set.
swf-memory-repro.zip
It generates SWF files locally and demonstrates memory growth in the current batched mode.