Date: Tue, 22 Jan 2019 14:44:12 +0000 From: bugzilla-noreply@freebsd.org To: bugs@FreeBSD.org Subject: [Bug 235125] Process was killed: out of swap space on gmirror + zfs Message-ID: <bug-235125-227@https.bugs.freebsd.org/bugzilla/>
next in thread | raw e-mail | index | archive | help
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D235125 Bug ID: 235125 Summary: Process was killed: out of swap space on gmirror + zfs Product: Base System Version: 11.2-RELEASE Hardware: amd64 OS: Any Status: New Severity: Affects Some People Priority: --- Component: kern Assignee: bugs@FreeBSD.org Reporter: alaa.alassafin@card-1.com Hello, We've got FreeBSD 11.2p8 installed on 2 SSDs which are mirrored using gmirr= or. we also have a zpool which consists of 3x7 raidz1 drives, 2x Cache and 2x S= LOG and 1x spare. dual Xeon CPUs and 128GB of Ram. The first time this Problem accrued right after the 11.2p8 upgrade from p4: ---- Jan 14 17:04:22 san2 zfsd: POLLHUP detected on devd socket. Jan 14 17:04:22 san2 kernel: pid 606 (devd), uid 0, was killed: out of swap space Jan 14 17:04:22 san2 kernel: Jan 14 17:04:22 san2 kernel: pid 606 (devd), u= id 0, was killed: out of swap space Jan 14 17:04:22 san2 zfsd: Disconnecting from devd. Jan 14 17:04:22 san2 zfsd: ConnectToDevd: Connecting to devd. ---- we had to restart the machine. After 3 days we had the same Problem, but th= is time multiple processes were killed: ------ Jan 19 10:49:49 san2 kernel: pid 610 (devd), uid 0, was killed: out of swap space Jan 19 10:49:49 san2 kernel: Jan 19 10:49:49 san2 kernel: pid 610 (devd), u= id 0, was killed: out of swap space Jan 19 11:09:49 san2 kernel: pid 835 (zabbix_agentd), uid 122, was killed: = out of swap space Jan 19 11:09:49 san2 kernel: Jan 19 11:09:49 san2 kernel: pid 835 (zabbix_agentd), uid 122, was killed: out of swap space Jan 19 11:10:48 san2 kernel: pid 847 (bareos-fd), uid 0, was killed: out of swap space Jan 19 11:10:48 san2 kernel: Jan 19 11:10:48 san2 kernel: pid 847 (bareos-f= d), uid 0, was killed: out of swap space Jan 19 11:11:15 san2 kernel: pid 838 (ntpd), uid 233, was killed: out of sw= ap space Jan 19 11:11:15 san2 kernel: Jan 19 11:11:15 san2 kernel: pid 838 (ntpd), u= id 233, was killed: out of swap space Jan 19 11:11:29 san2 kernel: pid 802 (ctld), uid 0, was killed: out of swap space Jan 19 11:11:29 san2 kernel: Jan 19 11:11:29 san2 kernel: pid 802 (ctld), u= id 0, was killed: out of swap space Jan 19 11:11:45 san2 kernel: pid 116 (adjkerntz), uid 0, was killed: out of swap space Jan 19 11:11:45 san2 kernel: Jan 19 11:11:45 san2 kernel: pid 116 (adjkernt= z), uid 0, was killed: out of swap space Jan 19 11:12:15 san2 kernel: pid 971 (getty), uid 0, was killed: out of swap space Jan 19 11:12:15 san2 kernel: Jan 19 11:12:15 san2 kernel: pid 971 (getty), = uid 0, was killed: out of swap space Jan 19 11:12:29 san2 kernel: pid 32950 (getty), uid 0, was killed: out of s= wap space Jan 19 11:12:29 san2 kernel: Jan 19 11:12:29 san2 kernel: pid 32950 (getty), uid 0, was killed: out of swap space Jan 19 11:12:46 san2 kernel: pid 32951 (getty), uid 0, was killed: out of s= wap=20 ----- Messages kept on repeating until we restarted the machine. We tried disabling zfsd, but that didn't help. This Machine is in production its really frustrating to have this behavior.= I will gladly provide any more info/tests when needed. Thank you Here is some more info: ------ root@san2:~ # gmirror status Name Status Components mirror/boot COMPLETE gpt/boot0 (ACTIVE) gpt/boot1 (ACTIVE) mirror/swap COMPLETE gpt/swap0 (ACTIVE) gpt/swap1 (ACTIVE) mirror/root COMPLETE gpt/root0 (ACTIVE) gpt/root1 (ACTIVE) zpool list NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEAL= TH=20 ALTROOT san2pool 37.9T 17.4T 20.4T - - 2% 46% 1.00x ONLI= NE=20 - root@san2:~ # vmstat procs memory page disks faults cpu r b w avm fre flt re pi po fr sr ad0 ad1 in sy cs us s= y id 0 0 2 409M 3.2G 1170 1 5 4 1569 1355 0 0 3837 800 9140 0 = 7 93 Device 1K-blocks Used Avail Capacity /dev/mirror/swap 8388604 26528 8362076 0% --=20 You are receiving this mail because: You are the assignee for the bug.=
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?bug-235125-227>