Date: Wed, 11 Mar 2026 15:14:55 +0000 From: bugzilla-noreply@freebsd.org To: bugs@FreeBSD.org Subject: [Bug 293737] BIND 9.20 causes runaway RSS with non-matching memory statistics Message-ID: <bug-293737-227@https.bugs.freebsd.org/bugzilla/>
index | next in thread | raw e-mail
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=293737 Bug ID: 293737 Summary: BIND 9.20 causes runaway RSS with non-matching memory statistics Product: Base System Version: 15.0-STABLE Hardware: Any OS: Any Status: New Severity: Affects Only Me Priority: --- Component: kern Assignee: bugs@FreeBSD.org Reporter: ondrej@sury.org Attachment #268720 text/plain mime type: Created attachment 268720 --> https://bugs.freebsd.org/bugzilla/attachment.cgi?id=268720&action=edit Script to parse and compare two jemalloc stats print output in JSON Hi, this is as much bug report as it is plea for help. I don't know yet :). We've got multiple (2 external and 1 internal) people running FreeBSD (both 14.x and 15.x) where BIND 9.18 is fine, but BIND 9.20 has runaway RSS memory problem. I wrote this simple patch that prints out the jemalloc memory statistics once in a while: ```patch diff --git a/bin/named/server.c b/bin/named/server.c index 25eb888e7ad..9edd24f374a 100644 --- a/bin/named/server.c +++ b/bin/named/server.c @@ -6725,10 +6725,26 @@ pps_timer_tick(void *arg) { UNUSED(arg); +#ifdef JEMALLOC_API_SUPPORTED + /* + * Purge all jemalloc arenas before dumping stats so that RSS + * reflects actual usage rather than retained-but-free pages. + */ + { + char purge_mctl[64]; + snprintf(purge_mctl, sizeof(purge_mctl), "arena.%u.purge", + MALLCTL_ARENAS_ALL); + (void)mallctl(purge_mctl, NULL, NULL, NULL, 0); + } + + malloc_stats_print(NULL, NULL, "J"); + +#endif /* JEMALLOC_API_SUPPORTED */ + /* * Don't worry about wrapping as the overflow result will be right. */ - dns_pps = (requests - oldrequests) / 1200; + dns_pps = (requests - oldrequests) / 6000; oldrequests = requests; } @@ -8321,7 +8337,7 @@ apply_configuration(cfg_obj_t *effectiveconfig, cfg_obj_t *bindkeys, &interval); } - isc_interval_set(&interval, 1200, 0); + isc_interval_set(&interval, 6000, 0); isc_timer_start(server->pps_timer, isc_timertype_ticker, &interval); isc_interval_set(&interval, named_g_tat_interval, 0); ``` and asked LLM to code parser for the JSON (sorry, I am overloaded) - attached. Then the user who reported the issue ran this on his production system and basically, the output was like this (TL;DR, full output below): ``` Global counters: Allocated : 44.9 MiB (application-usable bytes) Active : 57.1 MiB (pages backing allocations) Metadata : 7.43 MiB (jemalloc bookkeeping) Resident : 64.2 MiB (RSS contribution) Mapped : 91.1 MiB (total mmap'd) Retained : 45.9 MiB (returned to OS, still mapped) ``` At the same time their RSS was: ``` SIZE = 495 MB RES = 412 MB ``` My running theory is that this is an artifact of jemalloc using `MADV_FREE` which doesn't immediately return the memory to the kernel, but they claim: > When it happened the first time it did not crash, but I saw it using swap space, which was crazy. So there was pressure. I will check again. I have a similar internal user running resolver on FreeBSD with following results: > my home production resolver has an RSS of 328M, but stats channel only reports 34M malloced I have so many questions :), but if I reduce them to just couple: - is this something that might happen (MADV_DONTUSE and MADV_FREE difference between Linux and FreeBSD) - is there any memory region that would manifest in RSS/RES, but not in jemalloc memory statistics (kernel memory?) - is there a reliable way how to test this? (rctl?) - is there a way how to enable jemalloc profiling? (although thinking about this now, this might not help as jemalloc shows 1/10 of the RES). Full jemalloc JSON summary... ``` ============================================================================== jemalloc statistics: primero.json ============================================================================== Version : 5.2.1-0-gea6b3e973b477b8061e0076bb257dbd7f3faa756 Configured arenas: 16 Dirty decay : 10000 ms Muzzy decay : 0 ms Global counters: Allocated : 44.7 MiB (application-usable bytes) Active : 55.0 MiB (pages backing allocations) Metadata : 7.34 MiB (jemalloc bookkeeping) Resident : 61.9 MiB (RSS contribution) Mapped : 89.0 MiB (total mmap'd) Retained : 41.0 MiB (returned to OS, still mapped) Fragmentation / overhead: Internal frag : 10.2 MiB (22.8% of allocated) External frag : 6.99 MiB (12.7% of active) Metadata ratio : 11.9% (metadata / resident) Map overhead : 27.0 MiB (mapped − resident) Allocation classes (merged): Small : 34.7 MiB (77.6%) live= 178,308 total_req= 524,094,282 Large : 10.0 MiB (22.4%) live= 90 total_req= 135,715 Metadata breakdown (merged): Base (arena metadata) : 6.96 MiB Internal (extent meta): 394 KiB Tcache bytes : 1.62 MiB Purge activity (merged): Dirty: purges= 26762 madvise calls= 50499 pages purged= 315777 Muzzy: purges= 0 madvise calls= 0 pages purged= 0 Per-arena summary: Arena Threads Active Resident Mapped Retained Tcache Uptime ───── ─────── ────────── ────────── ────────── ────────── ────────── ──────── 0 1 23.8 MiB 29.7 MiB 31.8 MiB 9.18 MiB 255 KiB 1.7h 1 1 0.00 B 40.0 KiB 2.00 MiB 0.00 B 60.3 KiB 1.7h 2 1 9.68 MiB 9.88 MiB 11.7 MiB 5.32 MiB 242 KiB 1.7h 3 1 9.36 MiB 9.55 MiB 11.4 MiB 5.64 MiB 225 KiB 1.7h 4 1 9.45 MiB 9.65 MiB 11.4 MiB 5.55 MiB 231 KiB 1.7h 5 1 280 KiB 324 KiB 2.27 MiB 1.73 MiB 18.5 KiB 1.7h 6 1 280 KiB 324 KiB 2.27 MiB 1.73 MiB 15.5 KiB 1.7h 7 1 276 KiB 324 KiB 2.27 MiB 1.73 MiB 19.7 KiB 1.7h 8 1 276 KiB 324 KiB 2.27 MiB 1.73 MiB 17.9 KiB 1.7h 9 1 368 KiB 420 KiB 2.36 MiB 1.64 MiB 124 KiB 1.7h 10 1 332 KiB 384 KiB 2.32 MiB 1.68 MiB 126 KiB 1.7h 11 1 332 KiB 384 KiB 2.32 MiB 1.68 MiB 126 KiB 1.7h 12 1 424 KiB 476 KiB 2.41 MiB 1.59 MiB 127 KiB 1.7h 13 1 140 KiB 184 KiB 2.14 MiB 1.86 MiB 71.3 KiB 1.7h Top 15 active small-object bins (merged): Size CurRegs CurBytes Slabs SlabBytes Util Requests ──────── ───────── ────────── ────── ────────── ────── ──────────── 640 B 10,612 6.48 MiB 392 7.66 MiB 84.6% 21,717,960 192 B 26,784 4.90 MiB 479 5.61 MiB 87.4% 8,087,561 128 B 21,725 2.65 MiB 801 3.13 MiB 84.8% 4,789,890 160 B 15,803 2.41 MiB 149 2.91 MiB 82.9% 43,602,308 256 B 9,674 2.36 MiB 771 3.01 MiB 78.4% 19,252,466 768 B 2,745 2.01 MiB 239 2.80 MiB 71.8% 28,845,970 112 B 18,403 1.97 MiB 86 2.35 MiB 83.6% 61,918,340 7.00 KiB 270 1.85 MiB 105 2.87 MiB 64.3% 6,604 448 B 3,079 1.32 MiB 56 1.53 MiB 85.9% 2,422,481 5.00 KiB 236 1.15 MiB 75 1.46 MiB 78.7% 5,150,995 320 B 3,680 1.12 MiB 82 1.60 MiB 70.1% 24,067,799 224 B 4,835 1.03 MiB 54 1.48 MiB 70.0% 11,736,777 1.50 KiB 656 984 KiB 145 1.70 MiB 56.6% 10,664,478 2.00 KiB 468 936 KiB 270 1.05 MiB 86.7% 102,880 1.25 KiB 669 836 KiB 54 1.05 MiB 77.4% 1,484,655 Top 10 active large extent classes (merged): Size Count Total ────────── ────── ──────────── 1.25 MiB 4 5.00 MiB 48.0 KiB 23 1.08 MiB 96.0 KiB 10 960 KiB 224 KiB 2 448 KiB 28.0 KiB 14 392 KiB 24.0 KiB 16 384 KiB 192 KiB 2 384 KiB 384 KiB 1 384 KiB 80.0 KiB 4 320 KiB 320 KiB 1 320 KiB ============================================================================== jemalloc statistics: segundo.json ============================================================================== Version : 5.2.1-0-gea6b3e973b477b8061e0076bb257dbd7f3faa756 Configured arenas: 16 Dirty decay : 10000 ms Muzzy decay : 0 ms Global counters: Allocated : 44.9 MiB (application-usable bytes) Active : 57.1 MiB (pages backing allocations) Metadata : 7.43 MiB (jemalloc bookkeeping) Resident : 64.2 MiB (RSS contribution) Mapped : 91.1 MiB (total mmap'd) Retained : 45.9 MiB (returned to OS, still mapped) Fragmentation / overhead: Internal frag : 12.2 MiB (27.0% of allocated) External frag : 7.07 MiB (12.4% of active) Metadata ratio : 11.6% (metadata / resident) Map overhead : 26.9 MiB (mapped − resident) Allocation classes (merged): Small : 34.6 MiB (77.1%) live= 177,786 total_req=2,590,735,106 Large : 10.3 MiB (22.9%) live= 92 total_req= 697,764 Metadata breakdown (merged): Base (arena metadata) : 7.04 MiB Internal (extent meta): 394 KiB Tcache bytes : 1.46 MiB Purge activity (merged): Dirty: purges=134784 madvise calls=250914 pages purged= 1618620 Muzzy: purges= 0 madvise calls= 0 pages purged= 0 Per-arena summary: Arena Threads Active Resident Mapped Retained Tcache Uptime ───── ─────── ────────── ────────── ────────── ────────── ────────── ──────── 0 1 24.8 MiB 30.8 MiB 32.8 MiB 8.21 MiB 124 KiB 10.0h 1 1 0.00 B 44.0 KiB 2.00 MiB 2.00 MiB 58.2 KiB 10.0h 2 1 10.2 MiB 10.4 MiB 12.2 MiB 9.83 MiB 247 KiB 10.0h 3 1 9.62 MiB 9.83 MiB 11.6 MiB 5.38 MiB 217 KiB 10.0h 4 1 9.81 MiB 10.0 MiB 11.8 MiB 5.19 MiB 193 KiB 10.0h 5 1 280 KiB 324 KiB 2.27 MiB 1.73 MiB 18.3 KiB 10.0h 6 1 272 KiB 320 KiB 2.27 MiB 1.73 MiB 24.2 KiB 10.0h 7 1 292 KiB 340 KiB 2.29 MiB 1.71 MiB 19.1 KiB 10.0h 8 1 288 KiB 336 KiB 2.28 MiB 1.72 MiB 17.9 KiB 10.0h 9 1 368 KiB 420 KiB 2.36 MiB 1.64 MiB 123 KiB 10.0h 10 1 424 KiB 480 KiB 2.41 MiB 1.59 MiB 126 KiB 10.0h 11 1 332 KiB 384 KiB 2.32 MiB 1.68 MiB 126 KiB 10.0h 12 1 384 KiB 436 KiB 2.38 MiB 1.62 MiB 126 KiB 10.0h 13 1 140 KiB 184 KiB 2.14 MiB 1.86 MiB 71.3 KiB 10.0h Top 15 active small-object bins (merged): Size CurRegs CurBytes Slabs SlabBytes Util Requests ──────── ───────── ────────── ────── ────────── ────── ──────────── 640 B 10,598 6.47 MiB 395 7.71 MiB 83.8% 108,408,887 192 B 26,897 4.92 MiB 488 5.72 MiB 86.1% 39,843,287 128 B 21,453 2.62 MiB 815 3.18 MiB 82.3% 23,242,382 256 B 10,625 2.59 MiB 784 3.06 MiB 84.7% 94,180,364 160 B 15,708 2.40 MiB 161 3.14 MiB 76.2% 216,237,523 112 B 18,248 1.95 MiB 88 2.41 MiB 81.0% 306,090,074 7.00 KiB 274 1.87 MiB 108 2.95 MiB 63.4% 8,907 768 B 2,439 1.79 MiB 248 2.91 MiB 61.5% 143,796,124 448 B 3,081 1.32 MiB 57 1.56 MiB 84.5% 11,783,239 320 B 3,732 1.14 MiB 87 1.70 MiB 67.0% 120,484,335 5.00 KiB 208 1.02 MiB 74 1.45 MiB 70.3% 25,663,705 224 B 4,620 1011 KiB 73 2.00 MiB 49.4% 58,447,087 2.00 KiB 482 964 KiB 283 1.11 MiB 85.2% 484,575 1.50 KiB 626 939 KiB 159 1.86 MiB 49.2% 53,205,906 1.25 KiB 673 841 KiB 55 1.07 MiB 76.5% 7,011,903 Top 10 active large extent classes (merged): Size Count Total ────────── ────── ──────────── 1.25 MiB 4 5.00 MiB 48.0 KiB 23 1.08 MiB 96.0 KiB 10 960 KiB 320 KiB 3 960 KiB 24.0 KiB 17 408 KiB 28.0 KiB 14 392 KiB 384 KiB 1 384 KiB 80.0 KiB 4 320 KiB 128 KiB 2 256 KiB 224 KiB 1 224 KiB ============================================================================== jemalloc comparison ============================================================================== [A] primero.json [B] segundo.json Version A: 5.2.1-0-gea6b3e973b477b8061e0076bb257dbd7f3faa756 Version B: 5.2.1-0-gea6b3e973b477b8061e0076bb257dbd7f3faa756 Global counters: [A] [B] Delta ────────────────────── ────────────── ────────────── ──────────────── Allocated 44.7 MiB 44.9 MiB +205 KiB Active 55.0 MiB 57.1 MiB +2.15 MiB Metadata 7.34 MiB 7.43 MiB +87.8 KiB Resident 61.9 MiB 64.2 MiB +2.23 MiB Mapped 89.0 MiB 91.1 MiB +2.15 MiB Retained 41.0 MiB 45.9 MiB +4.85 MiB Fragmentation: [A] [B] Delta ────────────────────── ────────────── ────────────── ──────────────── Internal frag 10.2 MiB 12.2 MiB +1.95 MiB External frag 6.99 MiB 7.07 MiB +88.0 KiB Metadata 7.34 MiB 7.43 MiB +87.8 KiB Internal frag % 22.8% 27.0% Allocation classes (merged): [A] [B] Delta ────────────────────── ────────────── ────────────── ──────────────── Small allocated 34.7 MiB 34.6 MiB -75.2 KiB Large allocated 10.0 MiB 10.3 MiB +280 KiB Small live objects 178,308 177,786 -522 Large live objects 90 92 +2 Tcache bytes 1.62 MiB 1.46 MiB -169 KiB Base metadata 6.96 MiB 7.04 MiB +87.8 KiB Internal metadata 394 KiB 394 KiB (same) Purge activity (merged): [A] [B] Delta ────────────────────── ────────────── ────────────── ──────────────── Dirty purges 26,762 134,784 +108,022 Dirty madvise 50,499 250,914 +200,415 Dirty pages purged 315,777 1,618,620 +1,302,843 Muzzy purges 0 0 (same) Top 15 small-bin changes by byte delta (merged): Size Regs[A] Regs[B] Bytes[A] Bytes[B] Delta Util[A] Util[B] ──────── ───────── ───────── ────────── ────────── ──────────── ─────── ─────── 256 B 9,674 10,625 2.36 MiB 2.59 MiB + 238 KiB 78.4% 84.7% 768 B 2,745 2,439 2.01 MiB 1.79 MiB -230 KiB 71.8% 61.5% 5.00 KiB 236 208 1.15 MiB 1.02 MiB -140 KiB 78.7% 70.3% 512 B 147 294 73.5 KiB 147 KiB + 73.5 KiB 51.0% 87.5% 224 B 4,835 4,620 1.03 MiB 1011 KiB -47.0 KiB 70.0% 49.4% 1.50 KiB 656 626 984 KiB 939 KiB -45.0 KiB 56.6% 49.2% 1.00 KiB 316 359 316 KiB 359 KiB + 43.0 KiB 69.3% 80.1% 128 B 21,725 21,453 2.65 MiB 2.62 MiB -34.0 KiB 84.8% 82.3% 8.00 KiB 7 11 56.0 KiB 88.0 KiB + 32.0 KiB 100.0% 100.0% 2.00 KiB 468 482 936 KiB 964 KiB + 28.0 KiB 86.7% 85.2% 7.00 KiB 270 274 1.85 MiB 1.87 MiB + 28.0 KiB 64.3% 63.4% 192 B 26,784 26,897 4.90 MiB 4.92 MiB + 21.2 KiB 87.4% 86.1% 112 B 18,403 18,248 1.97 MiB 1.95 MiB -17.0 KiB 83.6% 81.0% 48.0 B 7,600 7,246 356 KiB 340 KiB -16.6 KiB 63.2% 59.0% 320 B 3,680 3,732 1.12 MiB 1.14 MiB + 16.2 KiB 70.1% 67.0% Top 10 large-extent changes by byte delta (merged): Size Cnt[A] Cnt[B] Bytes[A] Bytes[B] Delta ────────── ─────── ─────── ──────────── ──────────── ──────────── 320 KiB 1 3 320 KiB 960 KiB + 640 KiB 224 KiB 2 1 448 KiB 224 KiB -224 KiB 192 KiB 2 1 384 KiB 192 KiB -192 KiB 32.0 KiB 2 3 64.0 KiB 96.0 KiB + 32.0 KiB 24.0 KiB 16 17 384 KiB 408 KiB + 24.0 KiB 16.0 KiB 9 9 144 KiB 144 KiB 0.00 B 28.0 KiB 14 14 392 KiB 392 KiB 0.00 B 48.0 KiB 23 23 1.08 MiB 1.08 MiB 0.00 B 80.0 KiB 4 4 320 KiB 320 KiB 0.00 B 96.0 KiB 10 10 960 KiB 960 KiB 0.00 B Per-arena resident memory: Arena Resident[A] Resident[B] Delta Thr[A] Thr[B] ───── ──────────── ──────────── ──────────── ────── ────── 0 29.7 MiB 30.8 MiB +1.01 MiB 1 1 1 40.0 KiB 44.0 KiB +4.00 KiB 1 1 2 9.88 MiB 10.4 MiB +512 KiB 1 1 3 9.55 MiB 9.83 MiB +284 KiB 1 1 4 9.65 MiB 10.0 MiB +372 KiB 1 1 5 324 KiB 324 KiB (same) 1 1 6 324 KiB 320 KiB -4.00 KiB 1 1 7 324 KiB 340 KiB +16.0 KiB 1 1 8 324 KiB 336 KiB +12.0 KiB 1 1 9 420 KiB 420 KiB (same) 1 1 10 384 KiB 480 KiB +96.0 KiB 1 1 11 384 KiB 384 KiB (same) 1 1 12 476 KiB 436 KiB -40.0 KiB 1 1 13 184 KiB 184 KiB (same) 1 1 ``` -- You are receiving this mail because: You are the assignee for the bug.home | help
Want to link to this message? Use this
URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?bug-293737-227>