ES 3.1.0 : Inconsistent element count

[hrstoyanov]

Environment Details

• EclipseStore Version: 3.1.0
• JDK version: 25.0.2
• OS: MacOS

Describe the bug

Getting this out of nowhere:

org.eclipse.serializer.persistence.binary.exceptions.BinaryPersistenceException: Inconsistent element count: specified 107 vs. iterated 2651208047
        at org.eclipse.serializer.persistence.binary@3.1.0/org.eclipse.serializer.persistence.binary.types.Binary.validatePostIterationState(Binary.java:2099)
        at org.eclipse.serializer.persistence.binary@3.1.0/org.eclipse.serializer.persistence.binary.types.Binary.storeKeyValuesAsEntries(Binary.java:1911)
        at org.eclipse.serializer.persistence.binary@3.1.0/org.eclipse.serializer.persistence.binary.types.Binary.storeKeyValuesAsEntries(Binary.java:502)
        at org.eclipse.store.gigamap@3.1.0/org.eclipse.store.gigamap.types.AbstractBinaryHandlerAbstractBitmapIndexHashing.internalStore(AbstractBinaryHandlerAbstractBitmapIndexHashing.java:81)
        at org.eclipse.store.gigamap@3.1.0/org.eclipse.store.gigamap.types.AbstractBinaryHandlerAbstractBitmapIndexHashing.internalStore(AbstractBinaryHandlerAbstractBitmapIndexHashing.java:34)
        at org.eclipse.store.gigamap@3.1.0/org.eclipse.store.gigamap.types.AbstractBinaryHandlerStateChangeFlagged.store(AbstractBinaryHandlerStateChangeFlagged.java:71)
        at org.eclipse.store.gigamap@3.1.0/org.eclipse.store.gigamap.types.AbstractBinaryHandlerStateChangeFlagged.store(AbstractBinaryHandlerStateChangeFlagged.java:37)
        at org.eclipse.serializer.persistence.binary@3.1.0/org.eclipse.serializer.persistence.binary.types.BinaryStorer$Default.storeItem(BinaryStorer.java:491)
        at org.eclipse.serializer.persistence.binary@3.1.0/org.eclipse.serializer.persistence.binary.types.BinaryStorer$Default.processItems(BinaryStorer.java:476)
        at org.eclipse.serializer.persistence.binary@3.1.0/org.eclipse.serializer.persistence.binary.types.BinaryStorer$Default.internalStore(BinaryStorer.java:459)
        at org.eclipse.serializer.persistence.binary@3.1.0/org.eclipse.serializer.persistence.binary.types.BinaryStorer$Default.storeAll(BinaryStorer.java:517)
        at org.eclipse.serializer.persistence@3.1.0/org.eclipse.serializer.persistence.types.PersistenceManager$Default.storeAll(PersistenceManager.java:334)
        at org.eclipse.store.storage@3.1.0/org.eclipse.store.storage.types.StorageConnection.storeAll(StorageConnection.java:431)
        at peruncs.eclipsestore.core/peruncs.eclipsestore.core.EclipseStoreContext.lambda$commit$0(EclipseStoreContext.java:178)
        at org.eclipse.serializer.base@3.1.0/org.eclipse.serializer.concurrency.LockedExecutor$Default.write(LockedExecutor.java:198)
        at org.eclipse.serializer.base@3.1.0/org.eclipse.serializer.concurrency.LockScope.write(LockScope.java:87)
        at peruncs.eclipsestore.core/peruncs.eclipsestore.core.EclipseStoreContext.commit(EclipseStoreContext.java:172)
        at tourbiz.webapp/tourbiz.webapp.simulator.SimulationBehavior.run(SimulationBehavior.java:211)
        at tourbiz.webapp/tourbiz.webapp.simulator.Simulator.lambda$run$5(Simulator.java:412)
        at java.base/java.util.concurrent.StructuredTaskScopeImpl$SubtaskImpl.run(StructuredTaskScopeImpl.java:325)
        at java.base/java.lang.VirtualThread.run(VirtualThread.java:456)

[fh-ms]

We've investigated the stack trace and identified the root cause: a concurrent modification of the GigaMap's internal data structures during the store/commit operation.

The key detail in the stack trace is that the persistence is triggered through EclipseStoreContext.commit() rather than through GigaMap.store(). While GigaMap.store() is synchronized and guarantees that no other thread can modify the GigaMap while it's being stored, any external store path (like storageConnection.storeAll() or similar) does not hold the GigaMap's monitor lock. This allows another thread to modify the GigaMap's indices while they are being serialized, which causes the inconsistent element count.

The stack trace confirms this: the commit runs on a virtual thread inside a StructuredTaskScope, meaning multiple subtasks can run concurrently. When one thread is serializing a bitmap index's entries, another thread can modify them through the GigaMap's synchronized API, but that synchronization only protects individual method calls, not the entire persistence traversal.

This is also documented in the GigaMap.store() Javadoc:

Using this method guarantees concurrency safety since it locks the GigaMap instance internally. Storing the instance in any other way does NOT guarantee this safety.

To fix this, you have two options:

1. Use GigaMap.store() directly instead of going through a separate commit/storeAll path. This is the simplest and safest approach.
2. Synchronize on the GigaMap instance when committing externally:

synchronized (gigaMap) {
    eclipseStoreContext.commit();
}

I updated the documentation to describe this issue: #567

[hrstoyanov]

Thanks @fh-ms for looking deep into this!

I should have added the code for EclipseStoreContext (I wanted to offer it for inclusion in ES a while ago, but forgot - take a look, attached below)

• it offers familiar commit/rollback and batching of persistent operations for complex "all-or-nothing" cases where the persistence is deferred to last moment.
• improved disk i/o performance as writes are buffered
• "memory keeping" when objects are added during several nested function calls

So if you look at the "commit" method, it confirms your findings:

    /// Commits the changes made within this context to the underlying storage.
    /// This method stores all pending objects and clears the list of objects to be stored.
    ///
    /// @throws IllegalStateException If the transaction is marked as rollback-only.
    public EclipseStoreContext commit() {
        write(() -> {
            if (rollbackOnly) {
                throw new IllegalStateException("Cannot commit changes - roll back only");
            }

            if (!valuesToStore.isEmpty()) {
                persistenceStoring.storeAll(valuesToStore);
                valuesToStore.clear();
            }
        });

        return this;
    }

and the Eclipse Store documentation should be changed to warn about that.

[hrstoyanov]

@fh-ms please review the updated EclipseStoreContext - it does store the affected gigamaps in a transaction properly.

However, I am a bit concerned with your latest change #563 that immediately stores entities updated in GigaMap.update(...)

EclipseStoreContext.java

[fh-ms]

The change #563 doesn't automatically save updated entities. It marks them as dirty and stores them with the next store call on the GigaMap.

[ValentinBossi]

i understand the confusion cause the issue title is „Automatically persist entities modified via update or apply“ only checking the doc file makes it clear the need to call store() after update() and apply(). maybe a lifecycle diagram (tracked state/untracked state) would help in the docs.

[hrstoyanov]

Btw, with the guidance @fh-ms provided, I fixed up my EclispeStoreContext
and was able to run the simulator more than 8 hrs now with thousands of virtual threads waking at random times and performing complex behaviors (using Java 25 structured concurrency, and storing context in scoped values, etc.) No more zombies or crashes!

I am trying to subject my code and Eclipse Store to brutal and chaotic testing, following the approach of TigerBeetle DB

Thanks @fh-ms for the explanation on #563 .
So, it functions somewhat similarly to what my EclipseStoreContext was designed to do - keep track of all entities to be persisted, deferring storing to the last moment for efficiency?

But:

• it only does this for entities updated within GigaMaps (GigaMap.update(...))?
• how does Automatically persist entities modified via update or apply #563 deal with complex and nested entities? I assume the developers still need to traverse the entity's object tree of object and explicitly persist the deeper parts of it in the case of lazy persistence configuration?

[hrstoyanov]

@fh-ms @ValentinBossi - attached below is the fixed-up version of my EclipseContextStore, if anyone is interested - steal it!

Typical use in my code:

public class MyData {
     ...
    public final static ScopedValue<EclipseStoreContext> ECLIPSE_STORE_CONTEXT = ScopedValue.newInstance();

    public final static ScopedValue<MyData> MY_DATA_ROOT = ScopedValue.newInstance();

    public static EclipseStoreContext getEclipseStoreContext() {
        return ECLIPSE_STORE_CONTEXT.orElseThrow(() -> new IllegalStateException("No Eclipse store context not bound"));
    }

    public static MyData getDataRoot() {
        return MY_DATA_ROOT.orElseThrow(() -> new IllegalStateException("My data root context not bound"));
    }
}

class MyTest {

        @TempDir
        Path tempDir;

        //Prepare the context - this could involve several stores and data roots!
        //You would typically do this with http filter and set up the scoped values at the entry point of the the HTTP handler
        void run(Runnable r) {
                try (EmbeddedStorageManager manager = EmbeddedStorage.start(MyData.createEmpty(null), tempDir)) {
                       var root = (MyData) manager.root();
                       var context = new EclipseStoreContext(manager, Tsid::fast);

                       //Bind the data root and context into scoped values 
                       ScopedValue
                           .where(MyData.MY_DATA_ROOT, root)
                           .where(MyData.ECLIPSE_STORE_CONTEXT, context)
                           .run(r);

                     //Try to commit the collected changes efficiently, at last
                     context.attemptCommit();
                }
        }

        // This code can be deeply nested function calls and very complex, 
        // but you always have a clean way to get the context 
        // without passing it around as method parameter
        // and EcipseStoreContext provides a very nice way to buffer your changes!
        void testBookingLifecycle() {
          run(() -> {
              var data = MyData.getDataRoot();
              var context = MyData.getEclipseStoreContext();
             
             ... business logic ... other method calls ... 
             
              context.addToPersist(myGigaMap);
              // Duplicates are handled internally, no worries!
              context.addAllToPersist(myGigaMap, myOtherGigamap, myEntity)); 

             // You can also mark the transaction dead
             // any subsequent use of the context would be ignored!
             context.rollbackOnly();
              
         }
}

EclipseStoreContext.java

[fh-ms]

how does #563 deal with complex and nested entities? I assume the developers still need to traverse the entity's object tree of object and explicitly persist the deeper parts of it in the case of lazy persistence configuration?

In short, it doesn't.
It can only track changes made to the objects that are directly referenced in the GigaMap.