RSS is not returned to the OS after bursts of large messages (jemalloc dirty pages retained; `background_thread` is off by default)

[IgorOhrimenko]

Summary

Under bursts of larger-than-message_buffer messages at high client concurrency, PgDog's RSS grows to the burst peak (multiple GB) and never comes back down while the process is idle — it stays elevated until the process/pod is restarted.

This is not live memory held by PgDog: SHOW SERVER MEMORY accounts for only a few MB of buffers while RSS sits at several GB. The retained memory is jemalloc dirty pages that are never purged, because PgDog ships jemalloc with background_thread:false (the default) and the per-request buffer reclaim (MessageBuffer::shrink_to_fit, #592) frees large chunks back to an allocator that then holds onto them.

Setting _RJEM_MALLOC_CONF=background_thread:true fully fixes it (RSS returns to baseline within ~10s of the burst ending). Filing this so it can be addressed in PgDog itself (default/flag/config) and to leave a quick-fix for anyone who hits it.

Environment

• PgDog 0.1.48.
• Linux, x86_64, Debian-based container (ghcr.io/pgdogdev/pgdog:0.1.48).
• Allocator: statically linked tikv-jemallocator (#[global_allocator]), jemalloc 5.3.0.
• jemalloc defaults in the shipped image (from _RJEM_MALLOC_CONF=stats_print:true):
  • opt.background_thread: false
  • opt.dirty_decay_ms: 10000
  • opt.muzzy_decay_ms: 0
  • opt.narenas: 32
• Transaction pooling, query_parser = "off".

Symptoms

• After a nightly traffic burst, one or more pods' RSS jumps to ~5–7 GB and stays there for hours (until restart), tripping memory alerts.
• The workload that triggers it is dominated by databases that move large messages (large result rows / large query payloads).
• SHOW SERVER MEMORY on a ballooned production pod (RSS 4949 MB, 576 server connections):
  • SUM(total_bytes) ≈ 7.1 MB
  • max buffer_bytes_alloc per connection ≈ 0.01 MB
  • prepared_statements_bytes ≈ 0.3 MB
  • → PgDog accounts for ~7 MB of live buffers while RSS is ~5 GB.

So the memory has been freed by PgDog but not returned to the OS.

Root cause (analysis)

1. MessageBuffer::ensure_capacity reserves enough space for the whole incoming message. A large message grows the per-connection buffer to that size.
2. After each request, MessageBuffer::shrink_to_fit (added in Reclaim space from buffer after every request #592, pgdog/src/net/messages/buffer.rs, called from frontend/client/mod.rs and backend/server.rs) re-allocates the buffer back down when it exceeded 2 * capacity, freeing the large chunk back to jemalloc.
3. With background_thread:false (default), jemalloc only advances its decay / purges dirty pages on allocation activity in that specific arena. After a burst, the arenas that grew go idle → decay never runs → dirty pages are retained → RSS stays at the peak.
4. With 32 arenas and a short, intense burst, a lot of dirty pages end up parked in arenas that subsequently go quiet, so a steady overall query rate does not clean them up (each thread's on-access decay only touches its own active arena).

muzzy_decay_ms:0 means there is no muzzy tier hiding the pages via MADV_FREE; they are genuinely retained dirty pages that a background purge would return via MADV_DONTNEED.

Minimal reproduction

The key ingredients are: high client concurrency, messages in jemalloc's arena size class (≈64 KB–4 MB, below the ~8 MB oversize/mmap threshold — huge single messages get munmap'd and released, so they do not reproduce it), and a small pool so many large client messages are buffered simultaneously while queued.

1. pgdog.toml

[general]
host = "0.0.0.0"
port = 6432
default_pool_size = 10          # small pool → client messages pile up buffered while queued
query_parser = "off"
passthrough_auth = "disabled"

[[databases]]
name = "lt"
host = "127.0.0.1"              # your postgres
port = 5432
database_name = "lt"

users.toml:

[[users]]
name = "lt"
database = "lt"
password = "ltpass"

2. Generate a load script with a ~3 MB client message (tiny result)

printf "SELECT octet_length('%s');\n" "$(head -c 3000000 /dev/zero | tr '\0' a)" > bigq.sql

3. Drive it: 500 concurrent clients for ~80s

PGPASSWORD=ltpass pgbench -n -c 500 -j 16 -T 80 -f bigq.sql \
  "host=127.0.0.1 port=6432 user=lt dbname=lt sslmode=disable"

4. Watch RSS of the pgdog process before / during / after

while true; do
  echo "$(date +%T) RSS=$(ps -o rss= -p "$(pgrep -x pgdog)") kB"
  sleep 3
done

Observed shape: RSS climbs to ~2 GB during load, then after pgbench stops it drops partway and plateaus well above baseline (retained dirty pages), staying there indefinitely while idle.

I ran this on Kubernetes (a throwaway postgres:18 + two PgDog deployments, default vs background_thread, driven by a pgbench pod). The full, self-contained manifest and the exact driver commands are here: https://gist.github.com/IgorOhrimenko/e0d77c8772cf6871b4851fd0afd80f2f — the commands above are the distilled, host-portable version.

Results

Same stand, same load (500 clients × 3 MB query, default_pool_size=10), baseline RSS ≈ 27–31 MB:

PgDog config	Peak RSS	RSS after ~60s idle
0.1.48 default	2018 MB	408 MB (stuck)
0.1.48 + _RJEM_MALLOC_CONF=background_thread:true	2017 MB	51 MB (returned)
0.1.48 + background_thread:true,dirty_decay_ms:5000,muzzy_decay_ms:5000	2009 MB	38 MB (returned)

• Same peak in all cases (the transient buffering is identical); the difference is purely whether jemalloc returns the freed pages.
• Throughput/behaviour unaffected: ~7.5k transactions, ~90 tps, 0 failed in both default and background_thread runs.

Workaround

Set the jemalloc env var on the PgDog process/container:

_RJEM_MALLOC_CONF=background_thread:true

(tikv-jemallocator reads _RJEM_MALLOC_CONF, not MALLOC_CONF.) background_thread:true alone is sufficient with the shipped defaults (dirty_decay_ms:10000, muzzy_decay_ms:0). Optionally add dirty_decay_ms:5000,muzzy_decay_ms:5000 to return memory a few seconds faster; the difference is marginal.

Kubernetes example:

env:
  - name: _RJEM_MALLOC_CONF
    value: "background_thread:true"

Questions / possible fixes for maintainers

Since PgDog owns the global allocator, this seems addressable inside PgDog rather than relying on an undocumented env var:

1. Enable background_thread by default (e.g. via tikv_jemallocator background_threads_runtime_support / a small mallctl call at startup, or a compile-time malloc_conf). This is the standard recommendation for long-running async servers and also smooths purge latency off the hot path.
2. Expose a config/flag for allocator tuning (e.g. [general] jemalloc_background_thread = true / decay knobs), so it doesn't depend on _RJEM_MALLOC_CONF.
3. Reconsider shrink_to_fit churn — reclaiming after every request frees large chunks back to an allocator that, by default, doesn't return them; a background purge (or a periodic mallctl arena.<i>.purge / .decay) would make the reclaim actually pay off in RSS.
4. Document the _RJEM_MALLOC_CONF knob and this behaviour for operators.

Happy to test any patch/config against the reproduction above and report RSS curves.

References

• pgdog/src/net/messages/buffer.rs — MessageBuffer::ensure_capacity / shrink_to_fit
• Reclaim space from buffer after every request #592 "Reclaim space from buffer after every request" (introduced the per-request realloc + SHOW CLIENT/SERVER MEMORY)
• jemalloc TUNING.md — background_thread, dirty_decay_ms, muzzy_decay_ms

[levkk]

Thanks for the report. Yes, I've seen this happen many times. Do you think you could submit a PR for the config-based option one? Not sure how straightforward it is to implement, let me know if I can help.

[IgorOhrimenko]

Thanks for confirming! Before I open a PR — which direction would you prefer?

1. Docs only — a short page describing the symptom (SHOW SERVER MEMORY ≪ RSS after a burst) and the _RJEM_MALLOC_CONF=background_thread:true workaround. Zero behaviour change, just makes the knob discoverable.
2. Code change in PgDog — enable jemalloc's background_thread from within PgDog so operators don't need the env var. Either:
   • a) turn it on by default at startup (mallctl background_thread=true via tikv-jemalloc-ctl), or
   • b) a config flag under [general] (e.g. jemalloc_background_thread = true, default off), optionally with dirty_decay_ms / muzzy_decay_ms passthrough.

I'm happy to do either (or docs + code). If you want the code path, do you prefer it on-by-default or opt-in via config? I can test any variant against the repro stand and report RSS curves.