Conversation
|
The reason virtual address space depletes is because mdbx reserves virtual address space based on the *_MAP_SIZE_MAX_BITS (utils/settings.hpp). Memory depletes because:
The above does two things:
Now the reason we used mdbx was for its automatic resizing capabilities. Since that is proving to not be scalable, we are going to check if lmdb works for this purpose too. Another observation while running operations on mdbx is that it does 4KB reads/writes. So we have also increased the page size to 64KB so as to use IOPs more efficiently on metered storage (eg. gp3). It is still unclear if that will help us in general upsert/search performance. The same support is added to lmdb aswell. Currently, we are able to generate indexes upto the size of disk. |
|
To be sure this is the correct solution we have to do the following tests:
Add compile time flags to enable/disable changes made to lmdb for increasing page size. |
The issue we are trying to solve here is the following:
When creating many indexes