check out these simd-json benchmarks using different memory allocators

[zooko]

Hi folks! I've been developing a new memory allocator -- smalloc -- and I've been using simd-json's benchmarks to compare smalloc's performance with the Rust default allocator, jemalloc, snmalloc, mimalloc, and rpmalloc. Thought you might be interested!

Here you can find my forks of the simd-json repo: https://github.com/zooko/simd-json . And here are the instructions for how to run the benchmarks: https://github.com/zooko/smalloc/blob/main/bench/README.md#benchmarking-user-code-with-different-allocators

Here are some benchmark results generated with this process:

test                                                                            default                jemalloc                snmalloc                mimalloc                rpmalloc                 smalloc
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
...
NORMALIZED (100s baseline work)                                      2100.0 s  (      )      2096.4 s  (      )      1944.6 s  (      )      1778.1 s  (      )      1756.2 s  (      )      1711.9 s  (      )
RELATIVE TO BASELINE                                                           ( +0.0%)                ( -0.2%)                ( -7.4%)                (-15.3%)                (-16.4%)                (-18.5%)

[Licenser]

That's super cool! Thank you for sharing!

The results look really sweet too, I'm curious how does it work on more multi threaded applications? I remember snmalloc for example optimized for allocation in one thread, deallocation in another.

[zooko]

Yes! smalloc is great at multithreading in general. And yes, snmalloc is indeed great at multithreading in general and at that case in particular, based on their publications and my benchmarks. Here are some benchmarks I wrote that just try to exercise the allocator operations, i.e. they are "synthetic" benchmarks instead of real-world apps like simd-json and rebar.

https://github.com/zooko/smalloc/blob/main/bench/results/cargo-bench.result.AppleM4Max.darwin25..txt

This one spawns 128 threads, and each thread just allocates another allocation and writes a few bytes into it and then iterates on that as fast as it can:

name:    de_mt_aww-128, threads:   128, iters:      2000, ns:      3,173,458, ns/i:     1,586.7
name:    je_mt_aww-128, threads:   128, iters:      2000, ns:     95,267,709, ns/i:    47,633.8
name:    sn_mt_aww-128, threads:   128, iters:      2000, ns:      6,787,750, ns/i:     3,393.8
name:    mi_mt_aww-128, threads:   128, iters:      2000, ns:      7,029,708, ns/i:     3,514.8
name:    rp_mt_aww-128, threads:   128, iters:      2000, ns:     11,132,417, ns/i:     5,566.2
name:    sm_mt_aww-128, threads:   128, iters:      2000, ns:      1,585,750, ns/i:       792.8
smalloc diff from  default:  -50%
smalloc diff from jemalloc:  -98%
smalloc diff from snmalloc:  -77%
smalloc diff from mimalloc:  -77%
smalloc diff from rpmalloc:  -86%

smalloc took 800 nanoseconds per allocation, snmalloc took 3400 nanoseconds per allocation.

There are several tests in smalloc/bench that exercise different kinds of multithreading behavior. I even have a test of the specific case that you mention: allocations allocated on one thread and then deallocated on another. However, those results are not included in that file I linked to above because they take too long to run. You can run them locally just by doing this:

git clone https://github.com/zooko/smalloc
cd smalloc
cargo build --package bench --features=snmalloc --release
./target/release/bench --compare --thorough

(You can also add mimalloc, jemalloc, and rpmalloc to features. mimalloc and rpmalloc perform—like snmalloc and smalloc—excellently in these benchmarks. jemalloc and the Rust default/system allocator not so much.)

The --thorough flag tells it to run the "extra" benchmarks that include the alloc-on-one-thread-free-on-another case. That case is named "fh" in the results file, short for "free hotspot". Here's the source code. In this test, one thread makes a bunch of allocations and hands them out to N different threads, and then we measure how fast those N threads can deallocate all of the allocations they were given. Here are the results on my Apple Silicon M4 Max:

name:         de_fh-10, threads:    10, iters:    100000, ns:        147,708, ns/i:         1.4
name:         sn_fh-10, threads:    10, iters:    100000, ns:         52,916, ns/i:         0.5
name:         sm_fh-10, threads:    10, iters:    100000, ns:     18,873,459, ns/i:       188.7
smalloc diff from  default: +12678%
smalloc diff from snmalloc: +35567%

name:         de_fh-20, threads:    20, iters:    100000, ns:        159,792, ns/i:         1.5
name:         sn_fh-20, threads:    20, iters:    100000, ns:         66,291, ns/i:         0.6
name:         sm_fh-20, threads:    20, iters:    100000, ns:     27,364,667, ns/i:       273.6
smalloc diff from  default: +17025%
smalloc diff from snmalloc: +41180%

name:         de_fh-40, threads:    40, iters:    100000, ns:        142,208, ns/i:         1.4
name:         sn_fh-40, threads:    40, iters:    100000, ns:        168,958, ns/i:         1.6
name:         sm_fh-40, threads:    40, iters:    100000, ns:     12,772,334, ns/i:       127.7
smalloc diff from  default: +8881%
smalloc diff from snmalloc: +7459%

name:        de_fh-100, threads:   100, iters:    100000, ns:        510,792, ns/i:         5.1
name:        sn_fh-100, threads:   100, iters:    100000, ns:        729,625, ns/i:         7.2
name:        sm_fh-100, threads:   100, iters:    100000, ns:      3,514,416, ns/i:        35.1
smalloc diff from  default: +588%
smalloc diff from snmalloc: +382%

This is actually the case that smalloc is worst at! (Its simple design allows gives it no room to maneuver in this case.) However, its performance is okayish even in this case. For example, in the "fh-20" case, each of 20 threads took about 30m nanoseconds to free all 100,000 of the allocations it had been given -- 300 nanoseconds per dealloc operation.

[Licenser]

Those are super impressive results!

[zooko]

Those are super impressive results!

Thank you!! I've been working on it for years and I'm super proud of it! :-)