From owner-freebsd-bugs@freebsd.org Tue Jan 22 14:44:14 2019 Return-Path: Delivered-To: freebsd-bugs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B8B0E149EF93 for ; Tue, 22 Jan 2019 14:44:14 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 51D268F328 for ; Tue, 22 Jan 2019 14:44:14 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id 15FF4149EF92; Tue, 22 Jan 2019 14:44:14 +0000 (UTC) Delivered-To: bugs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E7775149EF91 for ; Tue, 22 Jan 2019 14:44:13 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 7526A8F325 for ; Tue, 22 Jan 2019 14:44:13 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id 9B7BE16160 for ; Tue, 22 Jan 2019 14:44:12 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id x0MEiCXS076287 for ; Tue, 22 Jan 2019 14:44:12 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id x0MEiCXU076285 for bugs@FreeBSD.org; Tue, 22 Jan 2019 14:44:12 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: bugs@FreeBSD.org Subject: [Bug 235125] Process was killed: out of swap space on gmirror + zfs Date: Tue, 22 Jan 2019 14:44:12 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: new X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.2-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: alaa.alassafin@card-1.com X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: bugs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: bug_id short_desc product version rep_platform op_sys bug_status bug_severity priority component assigned_to reporter Message-ID: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-bugs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Bug reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 22 Jan 2019 14:44:14 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D235125 Bug ID: 235125 Summary: Process was killed: out of swap space on gmirror + zfs Product: Base System Version: 11.2-RELEASE Hardware: amd64 OS: Any Status: New Severity: Affects Some People Priority: --- Component: kern Assignee: bugs@FreeBSD.org Reporter: alaa.alassafin@card-1.com Hello, We've got FreeBSD 11.2p8 installed on 2 SSDs which are mirrored using gmirr= or. we also have a zpool which consists of 3x7 raidz1 drives, 2x Cache and 2x S= LOG and 1x spare. dual Xeon CPUs and 128GB of Ram. The first time this Problem accrued right after the 11.2p8 upgrade from p4: ---- Jan 14 17:04:22 san2 zfsd: POLLHUP detected on devd socket. Jan 14 17:04:22 san2 kernel: pid 606 (devd), uid 0, was killed: out of swap space Jan 14 17:04:22 san2 kernel: Jan 14 17:04:22 san2 kernel: pid 606 (devd), u= id 0, was killed: out of swap space Jan 14 17:04:22 san2 zfsd: Disconnecting from devd. Jan 14 17:04:22 san2 zfsd: ConnectToDevd: Connecting to devd. ---- we had to restart the machine. After 3 days we had the same Problem, but th= is time multiple processes were killed: ------ Jan 19 10:49:49 san2 kernel: pid 610 (devd), uid 0, was killed: out of swap space Jan 19 10:49:49 san2 kernel: Jan 19 10:49:49 san2 kernel: pid 610 (devd), u= id 0, was killed: out of swap space Jan 19 11:09:49 san2 kernel: pid 835 (zabbix_agentd), uid 122, was killed: = out of swap space Jan 19 11:09:49 san2 kernel: Jan 19 11:09:49 san2 kernel: pid 835 (zabbix_agentd), uid 122, was killed: out of swap space Jan 19 11:10:48 san2 kernel: pid 847 (bareos-fd), uid 0, was killed: out of swap space Jan 19 11:10:48 san2 kernel: Jan 19 11:10:48 san2 kernel: pid 847 (bareos-f= d), uid 0, was killed: out of swap space Jan 19 11:11:15 san2 kernel: pid 838 (ntpd), uid 233, was killed: out of sw= ap space Jan 19 11:11:15 san2 kernel: Jan 19 11:11:15 san2 kernel: pid 838 (ntpd), u= id 233, was killed: out of swap space Jan 19 11:11:29 san2 kernel: pid 802 (ctld), uid 0, was killed: out of swap space Jan 19 11:11:29 san2 kernel: Jan 19 11:11:29 san2 kernel: pid 802 (ctld), u= id 0, was killed: out of swap space Jan 19 11:11:45 san2 kernel: pid 116 (adjkerntz), uid 0, was killed: out of swap space Jan 19 11:11:45 san2 kernel: Jan 19 11:11:45 san2 kernel: pid 116 (adjkernt= z), uid 0, was killed: out of swap space Jan 19 11:12:15 san2 kernel: pid 971 (getty), uid 0, was killed: out of swap space Jan 19 11:12:15 san2 kernel: Jan 19 11:12:15 san2 kernel: pid 971 (getty), = uid 0, was killed: out of swap space Jan 19 11:12:29 san2 kernel: pid 32950 (getty), uid 0, was killed: out of s= wap space Jan 19 11:12:29 san2 kernel: Jan 19 11:12:29 san2 kernel: pid 32950 (getty), uid 0, was killed: out of swap space Jan 19 11:12:46 san2 kernel: pid 32951 (getty), uid 0, was killed: out of s= wap=20 ----- Messages kept on repeating until we restarted the machine. We tried disabling zfsd, but that didn't help. This Machine is in production its really frustrating to have this behavior.= I will gladly provide any more info/tests when needed. Thank you Here is some more info: ------ root@san2:~ # gmirror status Name Status Components mirror/boot COMPLETE gpt/boot0 (ACTIVE) gpt/boot1 (ACTIVE) mirror/swap COMPLETE gpt/swap0 (ACTIVE) gpt/swap1 (ACTIVE) mirror/root COMPLETE gpt/root0 (ACTIVE) gpt/root1 (ACTIVE) zpool list NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEAL= TH=20 ALTROOT san2pool 37.9T 17.4T 20.4T - - 2% 46% 1.00x ONLI= NE=20 - root@san2:~ # vmstat procs memory page disks faults cpu r b w avm fre flt re pi po fr sr ad0 ad1 in sy cs us s= y id 0 0 2 409M 3.2G 1170 1 5 4 1569 1355 0 0 3837 800 9140 0 = 7 93 Device 1K-blocks Used Avail Capacity /dev/mirror/swap 8388604 26528 8362076 0% --=20 You are receiving this mail because: You are the assignee for the bug.=