Date: Mon, 16 Mar 2015 18:59:33 -0400 From: J David <j.david.lists@gmail.com> To: freebsd-stable <freebsd-stable@freebsd.org> Subject: Significant memory leak in 9.3p10? Message-ID: <CABXB=RRhynY5FWvw3tHrLFRyitTemavXYLBpev5Mjs_kPqimXA@mail.gmail.com>
next in thread | raw e-mail | index | archive | help
Recently we have seen a large-scale memory leak on amd64 machines running FreeBSD 9.3-RELEASE-p10. This was first observed on 9.3p2 but has since shown up all the way through p10. Here's what the header of top shows: last pid: 32329; load averages: 0.00, 0.01, 0.21 up 3+15:37:29 22:34:04 25 processes: 2 running, 22 sleeping, 1 waiting CPU: % user, % nice, % system, % interrupt, % idle Mem: 4072M Active, 895M Inact, 1284M Wired, 125M Cache, 826M Buf, 1521M Free Swap: 1024M Total, 874M Used, 149M Free, 85% Inuse About 4G actively being used, another 895M inactive, and another 874M in swap. So it seems like this is a user-space leak, rather than a kernel-space leak. At the time of measurement, this machine was not doing anything and every possible process had been killed trying to find a culprit. The entire output of "ps axlww" is: UID PID PPID CPU PRI NI VSZ RSS MWCHAN STAT TT TIME COMMAND 0 0 0 0 -52 0 0 224 - DLs ?? 0:00.82 [kernel] 0 1 0 0 20 0 6280 556 wait SLs ?? 0:00.57 /sbin/init -- 0 2 0 0 -16 0 0 16 pftm DL ?? 0:00.85 [pfpurge] 0 3 0 0 -16 0 0 16 waiting_ DL ?? 0:00.00 [sctp_iterator] 0 4 0 0 -16 0 0 16 - DL ?? 0:00.00 [xpt_thrd] 0 5 0 0 -16 0 0 16 psleep DL ?? 0:28.85 [pagedaemon] 0 6 0 0 -16 0 0 16 psleep DL ?? 0:45.03 [vmdaemon] 0 7 0 0 -16 0 0 16 pollid DL ?? 0:00.23 [idlepoll] 0 8 0 0 155 0 0 16 pgzero DL ?? 0:00.00 [pagezero] 0 9 0 0 -16 0 0 16 psleep DL ?? 0:00.83 [bufdaemon] 0 10 0 0 -16 0 0 16 audit_wo DL ?? 0:00.00 [audit] 0 11 0 0 155 0 0 32 - RL ?? 8317:13.37 [idle] 0 12 0 0 -76 0 0 240 - WL ?? 301:43.54 [intr] 0 13 0 0 -8 0 0 48 - DL ?? 0:09.89 [geom] 0 14 0 0 -16 0 0 16 - DL ?? 2:58.88 [yarrow] 0 15 0 0 -68 0 0 64 - DL ?? 0:02.32 [usb] 0 16 0 0 -16 0 0 16 vlruwt DL ?? 0:06.35 [vnlru] 0 17 0 0 16 0 0 16 syncer DL ?? 5:28.89 [syncer] 0 18 0 0 -16 0 0 16 sdflush DL ?? 0:10.27 [softdepflush] 0 19 0 0 -16 0 0 16 - DL ?? 0:55.09 [racctd] 0 830 1 0 20 0 45348 2396 wait Is u0 0:00.07 login [pam] (login) 500 32269 830 0 20 0 14556 2428 wait S u0 0:00.09 -sh (sh) 500 32340 32269 0 20 0 16296 1908 - R+ u0 0:00.00 ps axlww Since the issue doesn't seem related to kernel memory usage, vmstat -m and -z have been skipped, but nothing jumps out as using gigs of RAM; they do appear consistent with 1284M of wired memory, which is not unreasonable for the affected machines' tuning and workload. The only user-space processes running are login, sh, and ps. So where did 5.5G of userspace RAM go? The only other potentially useful information is that when this happens, shutting down the system will hang for about ten minutes. $ sudo halt -p Waiting (max 60 seconds) for system process `vnlru' to stop...done Waiting (max 60 seconds) for system process `bufdaemon' to stop...done Waiting (max 60 seconds) for system process `syncer' to stop... Syncing disks, vnodes remaining...0 0 0 0 0 0 0 0 0 done All buffers synced. <----- 10 MINUTE HANG AFTER PRINTING THIS Uptime: 3d15h56m32s usbus0: Controller shutdown uhub0: at usbus0, port 1, addr 1 (disconnected) usbus0: controller did not stop usbus0: Controller shutdown complete acpi0: Powering system off Connection closed by foreign host. So it seems like somewhere after "All buffers synced" and printing the uptime, it's very slowly unwinding whatever is using up all that RAM and swap. Does anyone have any idea what might be causing this or how to fix/prevent it? Thanks in advance for any advice!
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CABXB=RRhynY5FWvw3tHrLFRyitTemavXYLBpev5Mjs_kPqimXA>