Continuous native memory growth in TDLib with multiple concurrent clients (Java JNI + SQLite backend)

[amartinezalto]

We are running TDLib in a long-lived production-like environment with multiple concurrent Telegram accounts in a single JVM process. Over time, we observe a continuous and linear increase in native memory usage that does not stabilize, eventually leading to system memory exhaustion.
We are trying to determine whether this is expected behavior under high multi-client load or indicates a memory retention issue in TDLib or its SQLite/native components.

Environment
TDLib version: 1.8.64
Integration: TDLib native library via Java JNI
Runtime: Java application (GlassFish)
OS: Linux x86_64
Setup: ~50 independent TDLib clients in a single JVM process
Each client represents a different Telegram account (SIM-based isolation)

Configuration
Each TDLib client is initialized with:
request.useMessageDatabase = true;
request.useFileDatabase = true;
request.useChatInfoDatabase = true;
~400+ groups per account
~20,000 total chats across all clients
Continuous message updates (high throughput)
Long-running process (several days uptime)

Issue
We observe a steady and linear increase in native memory usage over time, eventually exhausting system memory.
After ~5 days:
Total RSS: ~29–30 GB
Java heap: ~8 GB
Estimated native memory: ~21–22 GB
Memory growth is continuous and does not stabilize.

Observations
Growth correlates with message activity (higher load → faster growth)
No memory decrease during low activity periods
Native memory dominates the increase (not Java heap)
System-level inspection shows:
Large anonymous mappings (~7.2 GB + ~1.2 GB)
Many repeated ~64 MB memory regions (~130+ blocks)
These appear to be native allocations not attributable to Java heap or obvious DirectByteBuffers.

Expected behavior
After warm-up, native memory usage should stabilize under steady load.

Actual behavior
Native memory grows continuously over time
No observed plateau after several days
Eventually leads to system memory exhaustion

Question
Is this expected behavior under this workload, or could it indicate:

• unbounded internal caching
• SQLite-related memory retention
• native allocator behavior

Also, is there any recommended runtime or build configuration (TDLib compile flags, allocator settings, SQLite options, or TDLib parameters) that could influence or limit native memory usage in long-running multi-client deployments?

Thank you

[levlam]

The memory usage can grow over time, because TDLib has a cache which speeds up all requests and gives a big flexibility for the application. The cache contains data about all encountered users and chats and can contain some other data. Messages are removed from the cache if they aren't accessed for some short time. The size of the in-memory cache can grow over time, so if app wants to flush it, it needs to close and reopen TDLib instance. This will not release the memory to OS, but will allow to reuse the freed memory for new allocations.

[amartinezalto]

Hello again,

We have been working on a process to close and recreate the clients after some time, but we haven't seen any improvement. After recreating them, memory usage continues growing from the same point it was at before.

This is what we see when closing and recreating the client. The first (blue) line represents the total process memory, the orange line represents the Java heap, and the last (blue) line represents the number of threads.

The way we do it is by sending TdApi.Close() to the client, waiting for the AuthorizationStateClosed, setting client = null, and then creating it again with Client.create. Is that correct?

Is there any other way to free caches or memory, or is this something planned for future versions?

Thank you

[levlam]

You did this correctly. It should noticeably decrease memory growth speed, which you should be able to notice by comparing with a process which never closes TDLib instances. Closing all instances frees all allocated native memory, which can be reused later by the same process. You can also try to link with a non-default memory allocator like jemalloc or tcmalloc, which may be able to better reuse the freed memory.