Double free corruption with transactions in latest heed3

[rracariu]

I discovered an issue with heed3 v0.22.0 and latest from master where in a stress scenario I get a double free corruption error in my application:

#0  __pthread_kill_implementation (no_tid=0, signo=6, threadid=<optimized out>) at ./nptl/pthread_kill.c:44
#1  __pthread_kill_internal (signo=6, threadid=<optimized out>) at ./nptl/pthread_kill.c:78
#2  __GI___pthread_kill (threadid=<optimized out>, signo=signo@entry=6) at ./nptl/pthread_kill.c:89
#3  0x0000765c7a24527e in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#4  0x0000765c7a2288ff in __GI_abort () at ./stdlib/abort.c:79
#5  0x0000765c7a2297b6 in __libc_message_impl (fmt=fmt@entry=0x765c7a3ce8d7 "%s\n")
    at ../sysdeps/posix/libc_fatal.c:134
#6  0x0000765c7a2a8ff5 in malloc_printerr (str=str@entry=0x765c7a3d1aa0 "double free or corruption (top)")
    at ./malloc/malloc.c:5772
#7  0x0000765c7a2ab0fc in _int_free_merge_chunk (av=<optimized out>, p=<optimized out>, size=<optimized out>)
    at ./malloc/malloc.c:4671
#8  0x0000765c7a2ab43a in _int_free (av=0x765c7a403ac0 <main_arena>, p=<optimized out>, have_lock=<optimized out>)
    at ./malloc/malloc.c:4646
#9  0x0000765c7a2addae in __GI___libc_free (mem=0x64b790b30320) at ./malloc/malloc.c:3398
#10 0x0000765bf61d0d53 in mdb_midl_free (ids=0x64b790b30328)
    at /test/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/lmdb-master3-sys-0.2.5/lmdb/libraries/liblmdb/midl.c:117
#11 0x0000765bf61ba9b1 in mdb_txn_end (txn=0x64b78daf3670, mode=0)
    at /test/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/lmdb-master3-sys-0.2.5/lmdb/libraries/liblmdb/mdb.c:3677
#12 0x0000765bf61bd129 in _mdb_txn_commit (txn=0x64b78daf3670)
    at /test/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/lmdb-master3-sys-0.2.5/lmdb/libraries/liblmdb/mdb.c:4497
#13 0x0000765bf61bd18d in mdb_txn_commit (txn=0x64b78daf3670)
    at /test/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/lmdb-master3-sys-0.2.5/lmdb/libraries/liblmdb/mdb.c:4509
#14 0x0000765bf61b69a7 in heed3::txn::RwTxn::commit (self=...) at src/txn.rs:327

The code that triggers this looks like:

heed_db.put_with_flags(
                txn,
                PutFlags::NO_OVERWRITE,
                &some_key_bytes,
                &[],
            )

The env is configured to run with TLS on.

[Kerollmops]

Hey @rracariu 👋

I'm sorry for not getting back to you sooner. Thanks for the report. This must not happen, especially from a Rust library. I would have a couple of questions:

• Are you using features that are only available in heed3?
• If not, can you try with the heed crate instead?
• If yes, would you mind sharing the exact program you are using, please?

With your answer, I'll be able to determine whether the issue is in the head3 Rust wrapper or a bug in the experimental mdb.master3 LMDB branch.

Just so you better understand, heed3 is a wrapper around the mdb.master3 branch of LMDB, which is an experimental version. It features very cool stuff, like encryption-at-rest.

Do you think you could update the crate documentation, TOML description, and the README to specify that the heed3 crate is a wrapper for the experimental mdb.master3 LMDB branch, please? This way, fewer people will use this crate if they don't intend to use encryption-at-rest and other experimental features.

Have a nice day 🌵

[rracariu]

Hey @Kerollmops,

The issue only manifested in the heed3 using the experimental lmdb with encryption at rest.

I just wanted to report the issue, as maybe it points to the right direction for a possible fix.

Encryption at rest was the feature looking to try out, and looks that it still needs some more time to be usable, at least on my setup.

Thanks again for looking into this.

[Kerollmops]

Hey @rracariu 👋

Thanks for answering. In your setup, are you using multiple threads to operate the database? Like reading from different threads? Moving the write transaction to another thread or something similar?

By sharing your code, I can analyze it and determine whether the issue is on the heed3 or LMDB side. If the latter, you can report it to the LMDB maintainers (on their issue tracker).

Just so you know, I am working hard on updating LMDB to support a new feature, i.e., nested read-only transactions, and as such, I'm updating mdb.master and mdb.master3. I'll wait for my patch to land before updating heed and heed3 to use the latest commits and benefit from any possible fixes for your issue.

I am running plenty of tests on heed and heed3, and all of them are passing; however, tests don't catch everything, and you probably found something new. Would you mind testing heed3 from this branch, please? It is up to date, but with a small new feature.

Thank you for helping improve the situation,
Have a nice day 🌵