From owner-freebsd-fs@freebsd.org Sun Oct 14 18:24:26 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 41A5210DE35A for ; Sun, 14 Oct 2018 18:24:26 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id D3D2995C05 for ; Sun, 14 Oct 2018 18:24:25 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id 9620110DE359; Sun, 14 Oct 2018 18:24:25 +0000 (UTC) Delivered-To: fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 72B7C10DE358 for ; Sun, 14 Oct 2018 18:24:25 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id F414C95BFA for ; Sun, 14 Oct 2018 18:24:24 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id 4B66F15B96 for ; Sun, 14 Oct 2018 18:24:24 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id w9EIOOOm045840 for ; Sun, 14 Oct 2018 18:24:24 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id w9EIOOV6045839 for fs@FreeBSD.org; Sun, 14 Oct 2018 18:24:24 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 228750] panic on zfs mirror removal Date: Sun, 14 Oct 2018 18:24:22 +0000 X-Bugzilla-Reason: CC X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.1-STABLE X-Bugzilla-Keywords: panic X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: cheeky.m@live.com X-Bugzilla-Status: In Progress X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: mav@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 14 Oct 2018 18:24:26 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D228750 --- Comment #12 from Roger Hammerstein --- 11.2-stable also works (for one removal) with the second patch from 229007#= c12 FreeBSD freebsd11 11.2-STABLE FreeBSD 11.2-STABLE #0 r339346: Sat Oct 13 15:30:06 EDT 2018=20=20=20 without the second patch, the removal seemed to never finish, after leaving= it overnight: root@freebsd11:~ # zpool create test mirror /dev/da8 /dev/da7 mirror /dev/d= a6 /dev/da5 mirror /dev/da4 /dev/da3 mirror /dev/da2 /dev/da1 root@freebsd11:~ # zpool status pool: test state: ONLINE scan: none requested config: NAME STATE READ WRITE CKSUM test ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 da8 ONLINE 0 0 0 da7 ONLINE 0 0 0 mirror-1 ONLINE 0 0 0 da6 ONLINE 0 0 0 da5 ONLINE 0 0 0 mirror-2 ONLINE 0 0 0 da4 ONLINE 0 0 0 da3 ONLINE 0 0 0 mirror-3 ONLINE 0 0 0 da2 ONLINE 0 0 0 da1 ONLINE 0 0 0 errors: No known data errors root@freebsd11:~ # cp -a /usr/src /test/ root@freebsd11:~ # zpool remove test mirror-2 root@freebsd11:~ # root@freebsd11:~ # zpool status pool: test state: ONLINE scan: none requested remove: Evacuation of mirror-2 in progress since Sat Oct 13 15:44:31 2018 22.2M copied out of 22.2M at 2.46M/s, 100.00% done, 0h0m to go config: NAME STATE READ WRITE CKSUM test ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 da8 ONLINE 0 0 0 da7 ONLINE 0 0 0 mirror-1 ONLINE 0 0 0 da6 ONLINE 0 0 0 da5 ONLINE 0 0 0 mirror-2 ONLINE 0 0 0 da4 ONLINE 0 0 0 da3 ONLINE 0 0 0 mirror-3 ONLINE 0 0 0 da2 ONLINE 0 0 0 da1 ONLINE 0 0 0 errors: No known data errors root@freebsd11:~ #=20 root@freebsd11:~ # zpool status pool: test state: ONLINE scan: none requested remove: Evacuation of mirror-2 in progress since Sat Oct 13 15:44:31 2018 22.2M copied out of 22.2M at 169K/s, 100.00% done, 0h0m to go config: NAME STATE READ WRITE CKSUM test ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 da8 ONLINE 0 0 0 da7 ONLINE 0 0 0 mirror-1 ONLINE 0 0 0 da6 ONLINE 0 0 0 da5 ONLINE 0 0 0 mirror-2 ONLINE 0 0 0 da4 ONLINE 0 0 0 da3 ONLINE 0 0 0 mirror-3 ONLINE 0 0 0 da2 ONLINE 0 0 0 da1 ONLINE 0 0 0 errors: No known data errors root@freebsd11:~ # root@freebsd11:/usr/src # zpool status pool: test state: ONLINE scan: none requested remove: Evacuation of mirror-2 in progress since Sat Oct 13 15:44:31 2018 22.2M copied out of 22.2M at 6.68K/s, 100.00% done, 0h0m to go config: NAME STATE READ WRITE CKSUM test ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 da8 ONLINE 0 0 0 da7 ONLINE 0 0 0 mirror-1 ONLINE 0 0 0 da6 ONLINE 0 0 0 da5 ONLINE 0 0 0 mirror-2 ONLINE 0 0 0 da4 ONLINE 0 0 0 da3 ONLINE 0 0 0 mirror-3 ONLINE 0 0 0 da2 ONLINE 0 0 0 da1 ONLINE 0 0 0 errors: No known data errors root@freebsd11:/usr/src # after overnight: root@freebsd11:/usr/src # zpool status pool: test state: ONLINE scan: none requested remove: Evacuation of mirror-2 in progress since Sat Oct 13 15:44:31 2018 22.2M copied out of 22.2M at 355/s, 100.00% done, 0h0m to go config: NAME STATE READ WRITE CKSUM test ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 da8 ONLINE 0 0 0 da7 ONLINE 0 0 0 mirror-1 ONLINE 0 0 0 da6 ONLINE 0 0 0 da5 ONLINE 0 0 0 mirror-2 ONLINE 0 0 0 da4 ONLINE 0 0 0 da3 ONLINE 0 0 0 mirror-3 ONLINE 0 0 0 da2 ONLINE 0 0 0 da1 ONLINE 0 0 0 errors: No known data errors rebooting onto the new kernel with the patch from 229007 root@freebsd11:~ # zpool status pool: test state: ONLINE scan: none requested remove: Evacuation of mirror-2 in progress since Sat Oct 13 15:44:31 2018 1 copied out of 22.2M at 1/s, 0.00% done, (copy is slow, no estimated t= ime) config: NAME STATE READ WRITE CKSUM test ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 da8 ONLINE 0 0 0 da7 ONLINE 0 0 0 mirror-1 ONLINE 0 0 0 da6 ONLINE 0 0 0 da5 ONLINE 0 0 0 mirror-2 ONLINE 0 0 0 da4 ONLINE 0 0 0 da3 ONLINE 0 0 0 mirror-3 ONLINE 0 0 0 da2 ONLINE 0 0 0 da1 ONLINE 0 0 0 errors: No known data errors root@freebsd11:~ # zpool status pool: test state: ONLINE scan: none requested remove: Removal of vdev 2 copied 22.2M in 18h20m, completed on Sun Oct 14 10:04:46 2018 816 memory used for removed device mappings config: NAME STATE READ WRITE CKSUM test ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 da8 ONLINE 0 0 0 da7 ONLINE 0 0 0 mirror-1 ONLINE 0 0 0 da6 ONLINE 0 0 0 da5 ONLINE 0 0 0 mirror-3 ONLINE 0 0 0 da2 ONLINE 0 0 0 da1 ONLINE 0 0 0 errors: No known data errors but then a second remove of zpool remove test mirror-0=20 paniced KDB: stack backtrace: #0 0xffffffff80b40df7 at kdb_backtrace+0x67 #1 0xffffffff80afa337 at vpanic+0x177 #2 0xffffffff80afa1b3 at panic+0x43 #3 0xffffffff80f7c38f at trap_fatal+0x35f #4 0xffffffff80f7c3e9 at trap_pfault+0x49 #5 0xffffffff80f7ba8c at trap+0x29c #6 0xffffffff80f5bfcc at calltrap+0x8 #7 0xffffffff824bc89b at vdev_indirect_io_start+0x9b #8 0xffffffff824e9fa9 at zio_vdev_io_start+0x2a9 #9 0xffffffff824e68ec at zio_execute+0xbc #10 0xffffffff824e61fb at zio_nowait+0xcb #11 0xffffffff824c248f at vdev_mirror_io_start+0x41f #12 0xffffffff824e9e5c at zio_vdev_io_start+0x15c #13 0xffffffff824e68ec at zio_execute+0xbc #14 0xffffffff80b52694 at taskqueue_run_locked+0x154 #15 0xffffffff80b537f8 at taskqueue_thread_loop+0x98 #16 0xffffffff80abd963 at fork_exit+0x83 #17 0xffffffff80f5cf8e at fork_trampoline+0xe Uptime: 11m46s but i think not everything that is in head has been mfcd back to 11 yet --=20 You are receiving this mail because: You are on the CC list for the bug.=