BEAM crashes with JS OOM error

[pejrich]

I had the following error which I don't think is the same as the other issue. I ran against v1.0.1 and build directly from the main branch just to double check.

When trying to convert a large Vega struct to PDF/PNG/JPG the beam crashes because of a crash down at the JS level. I don't think the root cause of the issue is in this library, but since the error is bubbling up and crashing the beam rather than just throwing an error, I thought it was worth raising the issue here in case it can at least be caught and the BEAM not being taken down(I know, NIFs can bite sometimes)

Here's a stacktrace. Happy to help if there's other logs or info you need. This was on macOS 15.3. OTP 27.2. Elixir 1.18.2-otp-27. and vega_lite_convert v1.0.1 and main

The Vega struct in question was ~42MB(this is byte_size after running it through :erlang.term_to_binary/1), so certainly not an average sized one. Also the code I was running did work as expected when taking smaller slices of the spectrogram, so it's not an issue of passing in invalid input, I think it's purely a size thing that triggers it.

iex(5)> NxAudio.Visualizations.Spectrogram.plot!(mel[0], bin_maxbins: 250) |> VegaLite.Convert.to_pdf()

<--- Last few GCs --->

[73653:0x118940000]    15842 ms: Scavenge (reduce) (interleaved) 1393.1 (1397.4) -> 1392.6 (1397.6) MB, pooled: 0 MB, 2.50 / 0.00 ms  (average mu = 0.239, current mu = 0.214) allocation failure;
[73653:0x118940000]    16013 ms: Mark-Compact (reduce) 1393.0 (1397.6) -> 1388.8 (1396.6) MB, pooled: 0 MB, 138.33 / 0.00 ms  (+ 431.3 ms in 0 steps since start of marking, biggest step 0.0 ms, walltime since start of marking 644 ms) (average mu = 0.222,

<--- JS stacktrace --->



#
# Fatal JavaScript out of memory: Ineffective mark-compacts near heap limit
#
==== C stack trace ===============================

0   libex_vl_convert.so                 0x000000016eee0f1c v8::base::debug::StackTrace::StackTrace() + 24
1   libex_vl_convert.so                 0x000000016eee6d74 v8::platform::(anonymous namespace)::PrintStackTrace() + 24
2   libex_vl_convert.so                 0x000000016eed7494 v8::base::FatalOOM(v8::base::OOMType, char const*) + 68
3   libex_vl_convert.so                 0x000000016ef3b1ac v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, v8::OOMDetails const&) + 616
4   libex_vl_convert.so                 0x000000016f0fac54 v8::internal::Heap::stack() + 0
5   libex_vl_convert.so                 0x000000016f10d5b8 _ZZN2v88internal4Heap14CollectGarbageENS0_15AllocationSpaceENS0_23GarbageCollectionReasonENS_15GCCallbackFlagsEENK3$_1clEv + 1836
6   libex_vl_convert.so                 0x000000016f10ce74 _ZN4heap4base5Stack24SetMarkerAndCallbackImplIZN2v88internal4Heap14CollectGarbageENS4_15AllocationSpaceENS4_23GarbageCollectionReasonENS3_15GCCallbackFlagsEE3$_1EEvPS1_PvPKv + 40
7   libex_vl_convert.so                 0x000000016f95546c PushAllRegistersAndIterateStack + 40
8   libex_vl_convert.so                 0x000000016f0f8dd0 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) + 412
9   libex_vl_convert.so                 0x000000016f0831f0 v8::internal::StackGuard::HandleInterrupts(v8::internal::StackGuard::InterruptLevel) + 504
10  libex_vl_convert.so                 0x000000016f4bc920 v8::internal::Runtime_StackGuard(int, unsigned long*, v8::internal::Isolate*) + 304
11  libex_vl_convert.so                 0x000000017024f9f4 Builtins_CEntry_Return1_ArgvOnStack_NoBuiltinExit + 84
12  libex_vl_convert.so                 0x000000017023cec8 Builtins_StringPrototypeReplace + 1192
13  ???                                 0x00000007a5248478 0x0 + 32835404920
14  ???                                 0x00000007a53fd4b4 0x0 + 32837194932
15  libex_vl_convert.so                 0x00000001701b8838 Builtins_InterpreterEntryTrampoline + 280
16  libex_vl_convert.so                 0x00000001701b8838 Builtins_InterpreterEntryTrampoline + 280
17  libex_vl_convert.so                 0x00000001701b8838 Builtins_InterpreterEntryTrampoline + 280
18  libex_vl_convert.so                 0x00000001701b8838 Builtins_InterpreterEntryTrampoline + 280
19  libex_vl_convert.so                 0x00000001701b8838 Builtins_InterpreterEntryTrampoline + 280
20  libex_vl_convert.so                 0x00000001701b8838 Builtins_InterpreterEntryTrampoline + 280
21  libex_vl_convert.so                 0x00000001701b8838 Builtins_InterpreterEntryTrampoline + 280
22  libex_vl_convert.so                 0x00000001701b8838 Builtins_InterpreterEntryTrampoline + 280
23  libex_vl_convert.so                 0x00000001701b8838 Builtins_InterpreterEntryTrampoline + 280
24  libex_vl_convert.so                 0x00000001701b8838 Builtins_InterpreterEntryTrampoline + 280
25  libex_vl_convert.so                 0x00000001701b8838 Builtins_InterpreterEntryTrampoline + 280
26  libex_vl_convert.so                 0x00000001701f5e20 Builtins_AsyncFunctionAwaitResolveClosure + 64
27  libex_vl_convert.so                 0x00000001702c3298 Builtins_PromiseFulfillReactionJob + 56
28  libex_vl_convert.so                 0x00000001701e5214 Builtins_RunMicrotasks + 564
29  libex_vl_convert.so                 0x00000001701b63f0 Builtins_JSRunMicrotasksEntry + 176
30  libex_vl_convert.so                 0x000000016f0580d4 v8::internal::(anonymous namespace)::Invoke(v8::internal::Isolate*, v8::internal::(anonymous namespace)::InvokeParams const&) + 1524
31  libex_vl_convert.so                 0x000000016f058994 v8::internal::(anonymous namespace)::InvokeWithTryCatch(v8::internal::Isolate*, v8::internal::(anonymous namespace)::InvokeParams const&) + 88
32  libex_vl_convert.so                 0x000000016f058acc v8::internal::Execution::TryRunMicrotasks(v8::internal::Isolate*, v8::internal::MicrotaskQueue*) + 60
33  libex_vl_convert.so                 0x000000016f0812a0 v8::internal::MicrotaskQueue::RunMicrotasks(v8::internal::Isolate*) + 356
34  libex_vl_convert.so                 0x000000016f08110c v8::internal::MicrotaskQueue::PerformCheckpointInternal(v8::Isolate*) + 88
35  libex_vl_convert.so                 0x000000016f073060 v8::internal::Isolate::FireCallCompletedCallbackInternal(v8::internal::MicrotaskQueue*) + 100
36  libex_vl_convert.so                 0x000000016ef3e6cc v8::Script::Run(v8::Local<v8::Context>, v8::Local<v8::Data>) + 844
37  libex_vl_convert.so                 0x000000016d2912bc deno_core::runtime::jsrealm::JsRealm::execute_script::h32295b04399ed647 + 496
38  libex_vl_convert.so                 0x000000016d29e1d0 deno_runtime::worker::MainWorker::execute_script::h74a232214471b631 + 92
39  libex_vl_convert.so                 0x000000016cce3d68 _ZN13vl_convert_rs9converter11VlConverter3new28_$u7b$$u7b$closure$u7d$$u7d$28_$u7b$$u7b$closure$u7d$$u7d$17h9c5e2c78adc346aaE + 22044
40  libex_vl_convert.so                 0x000000016ccd3dd0 tokio::runtime::scheduler::current_thread::CoreGuard::block_on::h23421b88fe22d5bf + 456
41  libex_vl_convert.so                 0x000000016cd045d8 tokio::runtime::context::runtime::enter_runtime::he1e560433fdac44c + 608
42  libex_vl_convert.so                 0x000000016ccfbc80 std::sys::backtrace::__rust_begin_short_backtrace::h8683c7b185ce53f2 + 244
43  libex_vl_convert.so                 0x000000016ccfc9ec _ZN4core3ops8function6FnOnce40call_once$u7b$$u7b$vtable.shim$u7d$$u7d$17h0e7a3fa082b67d11E + 152
44  libex_vl_convert.so                 0x00000001703aadd4 std::sys::pal::unix::thread::Thread::new::thread_start::hd88bc8e95f2ca709 + 52
45  libsystem_pthread.dylib             0x0000000193cb42e4 _pthread_start + 136
46  libsystem_pthread.dylib             0x0000000193caf0fc thread_start + 8
[1]    73653 trace trap  iex --erl "-kernel shell_history enabled +MIscs 16384" -S mix

[jonatanklosko]

Hey @pejrich! Can you provide a reproducible example? Since it's about data size, I expect zero-ed or random data should do.

[pejrich]

You bet. Here is a script that contains the exact struct that caused the crash. It's just the Vega struct serialized/compressed/b64 encoded, and the script just decodes the struct and runs it through VegaLite.Convert. Let me know if it works for you, as I imagine it could vary depending on whatever the JS runtime memory limit is. Though I don't believe I've ever manually adjusted mine(certainly not down if anything), so fingers crossed this crashes when you run it too(it's not too often I say that 😄).

vegalite_convert_crash_oom.exs.gz

[jonatanklosko]

@pejrich thanks, I could reproduced. I also managed to generate a very straightforward spec that OOMs given 1mln data points. I reported the issue upstream in vega/vl-convert#210.