Skip to content

MDBX remap limitations#79

Open
shaleenji wants to merge 2 commits intomasterfrom
lmdb_mdbx
Open

MDBX remap limitations#79
shaleenji wants to merge 2 commits intomasterfrom
lmdb_mdbx

Conversation

@shaleenji
Copy link
Collaborator

@shaleenji shaleenji commented Mar 13, 2026

The issue we are trying to solve here is the following:
When creating many indexes

  1. The virtual address space depletes - mmap returns ENOMEM
  2. Moreover, the available memory fills up quickly.

@shaleenji
Copy link
Collaborator Author

shaleenji commented Mar 13, 2026

The reason virtual address space depletes is because mdbx reserves virtual address space based on the *_MAP_SIZE_MAX_BITS (utils/settings.hpp).

Memory depletes because:

  1. mdbx allocates *_MAP_SIZE_BITS in memory at the time of creation
  2. The evictIfNeeded() function is not getting triggered.

The above does two things:

  1. Makes a stop-gap dirty solution to force evictIfNeeded() - THIS IS TO BE CHANGED
  2. reduces the max file size in mdbx and adds a logic to manually resize the map when MDBX_MAP_FULL returned

Now the reason we used mdbx was for its automatic resizing capabilities. Since that is proving to not be scalable, we are going to check if lmdb works for this purpose too.
In that regard, we have also added a compile time capability to either use lmdb or mdbx.

Another observation while running operations on mdbx is that it does 4KB reads/writes. So we have also increased the page size to 64KB so as to use IOPs more efficiently on metered storage (eg. gp3). It is still unclear if that will help us in general upsert/search performance. The same support is added to lmdb aswell.

Currently, we are able to generate indexes upto the size of disk.

@shaleenji
Copy link
Collaborator Author

shaleenji commented Mar 13, 2026

To be sure this is the correct solution we have to do the following tests:

  1. Correctness tests using mdbx and lmdb - is the code correct.
  2. Stress test to check if manual resizing of the lmdb crashes the system in any way.
  3. Performance tests using mdbx and lmdb with different page sizes to profile it.

Add compile time flags to enable/disable changes made to lmdb for increasing page size.

@shaleenji shaleenji changed the title Fixing index creation limitations MDBX remap limitations Mar 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants