DgraphAsyncClient runs its async work on ForkJoinPool.commonPool() by default and blocks those threads for the full duration of each gRPC call. Under sustained or slow traffic this can exhaust the JVM-wide common pool and hang anything else in the process that depends on it (parallel streams, other CompletableFuture chains).
Originally reported previously:
Hi, we have noticed that dgraph4j DgraphAsyncClient is not closing connections properly.
It uses ForkJoinPool.commonPool which is JVM's fork join pool. Eventually, it would fill up and hang all execution.
Please investigate/fix, we are attempting to work around this for the moment by giving it a separate forkjoin pool and adding more keep alives and timeouts to requests but that's downstream of the client.
Root cause
Two things compound here:
- The default constructor sets
this.executor = ForkJoinPool.commonPool() (DgraphAsyncClient.java#L54).
CompletableFutures.runWithRetries wraps every operation in CompletableFuture.supplyAsync(...) on that executor and then calls a blocking .get() on the inner gRPC future (CompletableFutures.java#L46). The JWT-expiry retry path blocks the same thread twice more (retryLogin.get().get(), then a second ctxCallable.call().get()).
So every alter, query, mutation, checkVersion, runDQL, etc. parks a common-pool thread until the gRPC round trip completes. The blocking call is not wrapped in a ForkJoinPool.ManagedBlocker, so the pool never compensates with extra threads. Since commonPool parallelism is roughly CPU cores minus one and shared across the whole JVM, a modest burst of slow or hung requests (server under load, network stall, missing deadlines) starves the pool. The reporter's workaround (a dedicated executor via the DgraphAsyncClient(Executor, DgraphStub...) constructor) is consistent with this diagnosis.
Note the "futures are not properly closed" framing in the original report is likely a misread of the symptom; the futures complete fine, the threads they run on are just blocked.
Suggested fix
runWithRetries doesn't actually need to block at all. The stub already returns a CompletableFuture (via StreamObserverBridge), so the method can compose instead of parking a thread: thenCompose on the stub future and handle the JWT-expiry retry with an exceptional-completion stage. That removes the blocking .get() calls entirely and makes the executor a callback executor rather than a thread the request holds hostage.
Shorter-term mitigations, if a full rewrite isn't wanted:
- Default to a small dedicated executor instead of
ForkJoinPool.commonPool().
- Wrap the blocking section in a
ForkJoinPool.ManagedBlocker so the common pool can compensate.
- At minimum, document that the default constructor blocks common-pool threads and that production users should supply their own executor.
Affected versions
Present on main as of v25.0.0; the same pattern exists in earlier releases (previously inline in DgraphAsyncClient.runWithRetries before the extraction to CompletableFutures).
DgraphAsyncClientruns its async work onForkJoinPool.commonPool()by default and blocks those threads for the full duration of each gRPC call. Under sustained or slow traffic this can exhaust the JVM-wide common pool and hang anything else in the process that depends on it (parallel streams, otherCompletableFuturechains).Originally reported previously:
Root cause
Two things compound here:
this.executor = ForkJoinPool.commonPool()(DgraphAsyncClient.java#L54).CompletableFutures.runWithRetrieswraps every operation inCompletableFuture.supplyAsync(...)on that executor and then calls a blocking.get()on the inner gRPC future (CompletableFutures.java#L46). The JWT-expiry retry path blocks the same thread twice more (retryLogin.get().get(), then a secondctxCallable.call().get()).So every
alter, query, mutation,checkVersion,runDQL, etc. parks a common-pool thread until the gRPC round trip completes. The blocking call is not wrapped in aForkJoinPool.ManagedBlocker, so the pool never compensates with extra threads. Since commonPool parallelism is roughly CPU cores minus one and shared across the whole JVM, a modest burst of slow or hung requests (server under load, network stall, missing deadlines) starves the pool. The reporter's workaround (a dedicated executor via theDgraphAsyncClient(Executor, DgraphStub...)constructor) is consistent with this diagnosis.Note the "futures are not properly closed" framing in the original report is likely a misread of the symptom; the futures complete fine, the threads they run on are just blocked.
Suggested fix
runWithRetriesdoesn't actually need to block at all. The stub already returns aCompletableFuture(viaStreamObserverBridge), so the method can compose instead of parking a thread:thenComposeon the stub future and handle the JWT-expiry retry with an exceptional-completion stage. That removes the blocking.get()calls entirely and makes the executor a callback executor rather than a thread the request holds hostage.Shorter-term mitigations, if a full rewrite isn't wanted:
ForkJoinPool.commonPool().ForkJoinPool.ManagedBlockerso the common pool can compensate.Affected versions
Present on
mainas of v25.0.0; the same pattern exists in earlier releases (previously inline inDgraphAsyncClient.runWithRetriesbefore the extraction toCompletableFutures).