World fails to load when mod added to large pack

[nhansendev]

Versions:
Dissolver-Enhanced: 1.5.31
Minecraft: 1.21.1
NeoForge: 21.1.233

I'm trying to add the Dissolver Enhanced mod to FTB Skies 2, and while there are no outright errors in the logs it's clear that something is heavily loading the server in the backend since spark repeatedly times-out while trying to retrieve world statistics. I'm unable to load into the world, with the loading indicator hanging at "100%" indefinitely (30 min+).

[04:22:05] [spark-java-sampler-0-5/WARN] [spark/]: Timed out waiting for world statistics
[04:23:05] [spark-java-sampler-0-5/WARN] [spark/]: Timed out waiting for world statistics
[04:24:05] [spark-java-sampler-0-3/WARN] [spark/]: Timed out waiting for world statistics
[04:25:05] [spark-java-sampler-0-5/WARN] [spark/]: Timed out waiting for world statistics
[04:26:05] [spark-java-sampler-0-3/WARN] [spark/]: Timed out waiting for world statistics
...

The pack has a large number of items and recipes ("----- DissolverEnhanced initialized Startup - 35641 recipes -----") and I don't know if any infinite loops, or the sheer number of recipes could be causing the issue.

It also fails to retrieve the "defaults.yaml" file from the github even though the raw address seems to be correct:

[04:21:33] [dissolver-default-emc-values-updater/WARN] [dissolver_enhanced/]: Unable to check GitHub for Dissolver Enhanced default EMC value updates.
java.io.IOException: GitHub returned HTTP 404 for https://raw.githubusercontent.com/Exohayvan/Dissolver-Enhanced/refs/heads/main/minecraft/emc-values/defaults.yaml
	at TRANSFORMER/dissolver_enhanced@1.5.31/net.exohayvan.dissolver_enhanced.common.values.DefaultEmcValueUpdateMonitor.fetchRemoteYaml(DefaultEmcValueUpdateMonitor.java:95) ~[dissolver-enhanced-1.5.31-neoforge-mc1.21.x-c1.3.2.jar%23741!/:1.5.31] {re:classloading}
	at TRANSFORMER/dissolver_enhanced@1.5.31/net.exohayvan.dissolver_enhanced.common.values.DefaultEmcValueUpdateMonitor.checkForUpdate(DefaultEmcValueUpdateMonitor.java:65) ~[dissolver-enhanced-1.5.31-neoforge-mc1.21.x-c1.3.2.jar%23741!/:1.5.31] {re:classloading}
	at TRANSFORMER/dissolver_enhanced@1.5.31/net.exohayvan.dissolver_enhanced.common.values.DefaultEmcValueUpdateMonitor.lambda$start$1(DefaultEmcValueUpdateMonitor.java:52) ~[dissolver-enhanced-1.5.31-neoforge-mc1.21.x-c1.3.2.jar%23741!/:1.5.31] {re:classloading}
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) ~[?:?] {}
	at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:358) ~[?:?] {}
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) ~[?:?] {}
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?] {}
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?] {}
	at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] {re:mixin}

Let me know if any additional info would help.

[Exohayvan]

Possible future implementation direction

I traced the current NeoForge startup path. The report is consistent with the EMC recipe pipeline monopolizing the server lifecycle thread, although we should still reproduce it with the pack or a synthetic 35k-recipe fixture before treating that as the only cause.

Today, ForgeEmcValueLoader.onServerStarted(...) synchronously:

1. copies every recipe from RecipeManager;
2. expands each recipe result and ingredient alternatives into mutable maps; and
3. calls EMCValues.recipesLoaded(...), which immediately runs the EMC fixed-point query.

The query recursively rescans the full recipe map until no more recipes complete. Several hot-path lookups also use linear collections (COMPLETED, MISSING, and the stonecutter list), and recipeKeySearch(...) scans every recipe key. With 35,641 source recipes, replacement variants and repeated full passes can make the effective work much larger than the source count.

Further inspection of the released 1.5.31 path found a second important problem: two NeoForge recipe loaders are enabled at once. RecipeManagerMixin.applyMixin(...) already starts an unmanaged raw thread, waits 800 ms, decodes the prepared recipe map, and calls EMCValues.recipesLoaded(...). ForgeEmcValueLoader then repeats extraction and calculation from ServerStartedEvent. Both paths call beginStartup(...) and mutate the same unsynchronized static maps, lists, counters, and readiness flags. Depending on timing, the full calculation can run twice, one generation can reset another, or a stale result can publish over a newer reload.

Moving the current implementation onto another arbitrary raw thread is therefore not the fix. Parts of it are already asynchronous, but unsafely. Recipe-manager, registry, codec, ingredient, player, and networking access must remain on the appropriate Minecraft thread; only a detached plain-Java calculation snapshot should run in the background.

Safer design

1. Remove the duplicate/racy loading path first. Select one authoritative, generation-aware NeoForge coordinator for initial load and /reload; remove the arbitrary 800 ms delay and ensure each generation is extracted and calculated exactly once.
2. Add stage timing. Record bounded timings and counts for recipe snapshotting, ingredient expansion, graph solving, and final publication. Include generation ID, thread name, generated recipe-variant count, solver pass count, unresolved count, and cancellation/failure reason in the existing startup summary.
3. Fix algorithmic complexity before relying on concurrency. Use HashSet for completed/missing/stonecutter membership, pre-index outputs instead of calling recipeKeySearch(...), and use per-output sequence counters instead of probing recipe keys from index 1.
4. Capture Minecraft-owned data safely. On the server thread, convert recipes into immutable DTOs containing only IDs, counts, types, and ingredient alternatives. If capture itself is too expensive, process a bounded batch per server tick instead of doing all 35k recipes inside ServerStartedEvent.
5. Replace repeated global scans with a dependency work queue. Index recipes by unresolved ingredient/output, enqueue only recipes affected when an item gains an EMC value, and stop when that queue reaches a fixed point. Preserve deterministic ordering for averaging and debug/source output.
6. Compute off-thread only after capture. Submit the immutable recipe graph and copied base/tag values to one lifecycle-owned executor. The worker must not touch RecipeManager, registries, players, menus, networking, or the currently published static maps.
7. Publish atomically on the server thread. Build a complete generation result in isolation, then swap one immutable snapshot through server.execute(...). Guard publication with a monotonically increasing generation ID, cancel or supersede pending work on reload, and shut the executor down on server stop.
8. Define the not-ready behavior. While dynamic EMC is calculating, keep bundled/configured EMC available, disable only operations that require derived values, and show a clear “EMC calculation in progress” response rather than silently blocking startup or exposing partial maps.

Verification

• Synthetic fixtures at 1k, 10k, and 35k+ recipes.
• A duplicate-output-heavy fixture and a long dependency-chain fixture.
• A regression test proving world startup reaches ready state while calculation continues.
• Thread assertions proving Minecraft-owned state is captured/published only on the server thread.
• Reload/stop cancellation tests and a stale-generation publication test.
• Runtime validation in FTB Skies 2, ideally with a spark profile and the new stage timings.

The defaults.yaml 404 in 1.5.31 is separate from the hang. That build requests refs/heads/main, while the repository default branch is common. Current source already uses refs/heads/common via #18, so the next release should include that correction; a 404 should remain non-blocking regardless.

[Exohayvan]

Benchmark Update

Based on current benchmarking, I estimate the initial EMC generation for this modpack (~35,000+ recipes) will take approximately 45 minutes on my current system using the existing forward/backward propagation algorithm.

This confirms that the issue should remain locked to P1. Optimizing the calculation engine is the next objective before further expanding support for large modpacks, as it will have a significant impact on overall usability and development iteration speed.

[Exohayvan]

Architecture correction for this implementation

The calculation-performance correction and the code-ownership correction should be completed as the same implementation process.

The forward/backward EMC calculation is shared behavior, but EMCValues and its fixed-point solver are currently duplicated across the loader branches. The optimized dependency/work-queue solver, immutable calculation state, generation tracking, readiness lifecycle, and shared parity/performance tests should be moved into the Common branch/shared Minecraft module while this issue is addressed.

Each loader branch should then retain only a thin adapter responsible for:

• capturing loader-specific recipes and registry data on the correct game thread;
• converting them into Common's immutable recipe snapshot;
• submitting or coordinating the calculation; and
• publishing/synchronizing the completed Common result on the correct thread.

This should not be implemented as a NeoForge-only copy of a new solver and then ported independently later. Fixing both concerns together prevents the calculation logic from drifting across Fabric, Quilt, Forge, and NeoForge and gives every loader the same deterministic EMC results and performance behavior.