Date: Tue, 22 Jan 2019 14:44:12 +0000 From: bugzilla-noreply@freebsd.org To: bugs@FreeBSD.org Subject: [Bug 235125] Process was killed: out of swap space on gmirror + zfs Message-ID: <bug-235125-227@https.bugs.freebsd.org/bugzilla/>
next in thread | raw e-mail | index | archive | help
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=235125 Bug ID: 235125 Summary: Process was killed: out of swap space on gmirror + zfs Product: Base System Version: 11.2-RELEASE Hardware: amd64 OS: Any Status: New Severity: Affects Some People Priority: --- Component: kern Assignee: bugs@FreeBSD.org Reporter: alaa.alassafin@card-1.com Hello, We've got FreeBSD 11.2p8 installed on 2 SSDs which are mirrored using gmirror. we also have a zpool which consists of 3x7 raidz1 drives, 2x Cache and 2x SLOG and 1x spare. dual Xeon CPUs and 128GB of Ram. The first time this Problem accrued right after the 11.2p8 upgrade from p4: ---- Jan 14 17:04:22 san2 zfsd: POLLHUP detected on devd socket. Jan 14 17:04:22 san2 kernel: pid 606 (devd), uid 0, was killed: out of swap space Jan 14 17:04:22 san2 kernel: Jan 14 17:04:22 san2 kernel: pid 606 (devd), uid 0, was killed: out of swap space Jan 14 17:04:22 san2 zfsd: Disconnecting from devd. Jan 14 17:04:22 san2 zfsd: ConnectToDevd: Connecting to devd. ---- we had to restart the machine. After 3 days we had the same Problem, but this time multiple processes were killed: ------ Jan 19 10:49:49 san2 kernel: pid 610 (devd), uid 0, was killed: out of swap space Jan 19 10:49:49 san2 kernel: Jan 19 10:49:49 san2 kernel: pid 610 (devd), uid 0, was killed: out of swap space Jan 19 11:09:49 san2 kernel: pid 835 (zabbix_agentd), uid 122, was killed: out of swap space Jan 19 11:09:49 san2 kernel: Jan 19 11:09:49 san2 kernel: pid 835 (zabbix_agentd), uid 122, was killed: out of swap space Jan 19 11:10:48 san2 kernel: pid 847 (bareos-fd), uid 0, was killed: out of swap space Jan 19 11:10:48 san2 kernel: Jan 19 11:10:48 san2 kernel: pid 847 (bareos-fd), uid 0, was killed: out of swap space Jan 19 11:11:15 san2 kernel: pid 838 (ntpd), uid 233, was killed: out of swap space Jan 19 11:11:15 san2 kernel: Jan 19 11:11:15 san2 kernel: pid 838 (ntpd), uid 233, was killed: out of swap space Jan 19 11:11:29 san2 kernel: pid 802 (ctld), uid 0, was killed: out of swap space Jan 19 11:11:29 san2 kernel: Jan 19 11:11:29 san2 kernel: pid 802 (ctld), uid 0, was killed: out of swap space Jan 19 11:11:45 san2 kernel: pid 116 (adjkerntz), uid 0, was killed: out of swap space Jan 19 11:11:45 san2 kernel: Jan 19 11:11:45 san2 kernel: pid 116 (adjkerntz), uid 0, was killed: out of swap space Jan 19 11:12:15 san2 kernel: pid 971 (getty), uid 0, was killed: out of swap space Jan 19 11:12:15 san2 kernel: Jan 19 11:12:15 san2 kernel: pid 971 (getty), uid 0, was killed: out of swap space Jan 19 11:12:29 san2 kernel: pid 32950 (getty), uid 0, was killed: out of swap space Jan 19 11:12:29 san2 kernel: Jan 19 11:12:29 san2 kernel: pid 32950 (getty), uid 0, was killed: out of swap space Jan 19 11:12:46 san2 kernel: pid 32951 (getty), uid 0, was killed: out of swap ----- Messages kept on repeating until we restarted the machine. We tried disabling zfsd, but that didn't help. This Machine is in production its really frustrating to have this behavior. I will gladly provide any more info/tests when needed. Thank you Here is some more info: ------ root@san2:~ # gmirror status Name Status Components mirror/boot COMPLETE gpt/boot0 (ACTIVE) gpt/boot1 (ACTIVE) mirror/swap COMPLETE gpt/swap0 (ACTIVE) gpt/swap1 (ACTIVE) mirror/root COMPLETE gpt/root0 (ACTIVE) gpt/root1 (ACTIVE) zpool list NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT san2pool 37.9T 17.4T 20.4T - - 2% 46% 1.00x ONLINE - root@san2:~ # vmstat procs memory page disks faults cpu r b w avm fre flt re pi po fr sr ad0 ad1 in sy cs us sy id 0 0 2 409M 3.2G 1170 1 5 4 1569 1355 0 0 3837 800 9140 0 7 93 Device 1K-blocks Used Avail Capacity /dev/mirror/swap 8388604 26528 8362076 0% -- You are receiving this mail because: You are the assignee for the bug.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?bug-235125-227>