From owner-freebsd-stable@freebsd.org Fri Jul 8 16:36:31 2016 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 878CAB83B12 for ; Fri, 8 Jul 2016 16:36:31 +0000 (UTC) (envelope-from guy.helmer@gmail.com) Received: from mail-it0-x242.google.com (mail-it0-x242.google.com [IPv6:2607:f8b0:4001:c0b::242]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 5164E1290 for ; Fri, 8 Jul 2016 16:36:31 +0000 (UTC) (envelope-from guy.helmer@gmail.com) Received: by mail-it0-x242.google.com with SMTP id y93so5616584ita.0 for ; Fri, 08 Jul 2016 09:36:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=from:content-transfer-encoding:subject:message-id:date:to :mime-version; bh=lmW0CR3imCn+C2XHTAXYQTCYNvcKr8g5+NYcehL92Hs=; b=gRJVqk+aFqC0A5N1XRW8Tv4wg+K0Aq2fge6T++e3MeBN0PGwFvVQL6SVku8bJa0ExR L6+SVvjl1C5p7VNEkJYv8Q6MaXoUpAUEyB8Qu1ec99zt2jaX4MxqnSIF5N8TBru3Mz3I 9xsa2yKgCWbGVJlWGshWyE+Ja5aNTQ3VwxN0iHoUma2LshzX2bVj29CHcsJWFeRNfwNq 69kGMMu06U+/BE8cqOEY5mKSMmUgtrS04lK0ZrjpNzZjS+JhhxuTNdtBsPc/y32ZhQPv bD6TV15di4GL7Igpq0viMVfRLo5JtQfUuKV0Ca3Tj1gLy8kusq8KuloGIMEj/9Iep5MI kCVw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:from:content-transfer-encoding:subject :message-id:date:to:mime-version; bh=lmW0CR3imCn+C2XHTAXYQTCYNvcKr8g5+NYcehL92Hs=; b=hhjd8TxeJhi1KbrX9qU4Dg3OYFTMnNzSrLIqVEu2B8pISEW6Rd0otaQPdxxVLBUsuD EZ6a+warIuYrsZ0QDIgtg2oxRDjqndc11vv8RaPK23MzY5nKQstS9wOSc1au2GJkOnh/ Nek4vh+d4rDuhIkWqVlMKDA+jfnrFFJkoYyNnMYHVtHnqV5mVEJtYELIw9UevcPeUB3Z Dj3oMKQb4Efnt1RUMjrIebpAJ9BzGo2+Bka5UNjK9MA6Tsv6NU3R+3rRSfL3fsv5MWI5 d7RdltcpvODxkiuqkuH99LAdPrQ1IfCcdsUZxDN+tz55zbybYt+fioelPxDDeQKSoSML qj/w== X-Gm-Message-State: ALyK8tKOeexMct7uxnYPDEwVFyYHZhFiaa5DLfwop4JpnYZ7awmc/Jho3J1lSKwOup5oYA== X-Received: by 10.36.91.80 with SMTP id g77mr4440108itb.46.1467995790375; Fri, 08 Jul 2016 09:36:30 -0700 (PDT) Received: from [172.22.132.55] ([192.119.231.58]) by smtp.gmail.com with ESMTPSA id d189sm3190939iod.3.2016.07.08.09.36.29 for (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Fri, 08 Jul 2016 09:36:29 -0700 (PDT) From: Guy Helmer Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Subject: 9-STABLE: Panic when destroying gmirror that is synchronizing Message-Id: Date: Fri, 8 Jul 2016 11:36:26 -0500 To: freebsd-stable@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) X-Mailer: Apple Mail (2.3124) X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 08 Jul 2016 16:36:31 -0000 Hi, I=E2=80=99m able to replicate a problem where destroying a gmirror that = is synchronizing causes a panic in 9-STABLE (^/stable/9 rev 302430) on = amd64. I=E2=80=99ve forced a gmirror into an inconsistent state by = forcing a reset, and then issued =E2=80=9Cgmirror destroy -f mirror1=E2=80= =9D after the reboot and the system is synchronizing the two disks in = the mirror. As a workaround, I can use =E2=80=9Cgeom mirror clear=E2=80=9D= on the providers before geom_mirror.ko is loaded. It=E2=80=99s a rare issue but I wanted to document it in case it can be = fixed. Kernel stack trace and code snippets follow. If anything other = information would be useful, please let me know. Guy Unread portion of the kernel message buffer: GEOM_MIRROR: Device mirror1: provider mirror/mirror1 destroyed. GEOM_MIRROR: Device mirror1: rebuilding provider = gptid/48c59e09-3c94-11e6-8928-000c29ce3c97 stopped. Fatal trap 12: page fault while in kernel mode cpuid =3D 0; apic id =3D 00 fault virtual address =3D 0x98 fault code =3D supervisor write data, page not present instruction pointer =3D 0x20:0xffffffff806df6ff stack pointer =3D 0x28:0xffffff800024eb30 frame pointer =3D 0x28:0xffffff800024eb40 code segment =3D base 0x0, limit 0xfffff, type 0x1b =3D DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags =3D interrupt enabled, resume, IOPL =3D 0 current process =3D 13 (g_up) trap number =3D 12 panic: page fault cpuid =3D 0 KDB: stack backtrace: #0 0xffffffff8072e336 at kdb_backtrace+0x66 #1 0xffffffff806f3d5e at panic+0x1ce #2 0xffffffff80910ae7 at trap_fatal+0x277 #3 0xffffffff80910e31 at trap_pfault+0x211 #4 0xffffffff809113f9 at trap+0x329 #5 0xffffffff808fa311 at calltrap+0x8 #6 0xffffffff81412593 at g_mirror_sync_done+0x53 #7 0xffffffff8077439e at biodone+0xae #8 0xffffffff806576ac at g_io_schedule_up+0xac #9 0xffffffff80657e0c at g_up_procbody+0x5c #10 0xffffffff806c0b4f at fork_exit+0x11f #11 0xffffffff808fa83e at fork_trampoline+0xe Uptime: 1m54s GEOM_MIRROR: Device mirror0: rebuilding provider = gptid/48ac38bf-3c94-11e6-8928-000c29ce3c97 stopped. Dumping 87 out of 238 MB:..19%..37%..55%..74%..92% Reading symbols from /boot/kernel/geom_mirror.ko...Reading symbols from = /boot/kernel/geom_mirror.ko.symbols...done. done. Loaded symbols for /boot/kernel/geom_mirror.ko #0 doadump (textdump=3D) at pcpu.h:235 235 pcpu.h: No such file or directory. in pcpu.h (kgdb) #0 doadump (textdump=3D) at pcpu.h:235 #1 0xffffffff806f3836 in kern_reboot (howto=3D260) at ../../../kern/kern_shutdown.c:454 #2 0xffffffff806f3d37 in panic (fmt=3D0x1
) at ../../../kern/kern_shutdown.c:642 #3 0xffffffff80910ae7 in trap_fatal (frame=3D0xc, eva=3D) at ../../../amd64/amd64/trap.c:876 #4 0xffffffff80910e31 in trap_pfault (frame=3D0xffffff800024ea80, = usermode=3D0) at ../../../amd64/amd64/trap.c:798 #5 0xffffffff809113f9 in trap (frame=3D0xffffff800024ea80) at ../../../amd64/amd64/trap.c:462 #6 0xffffffff808fa311 in calltrap () at = ../../../amd64/amd64/exception.S:238 #7 0xffffffff806df6ff in _mtx_lock_flags (m=3D0x80, opts=3D0,=20 file=3D0xffffffff8141d0d8 = "/usr/src/sys/modules/geom/geom_mirror/../../../geom/mirror/g_mirror.c", = line=3D990) at atomic.h:164 #8 0xffffffff81412593 in g_mirror_sync_done (bp=3D0xfffffe0004c963e0) at = /usr/src/sys/modules/geom/geom_mirror/../../../geom/mirror/g_mirror.c:990 #9 0xffffffff8077439e in biodone (bp=3D0xfffffe0004c963e0) at ../../../kern/vfs_bio.c:3667 #10 0xffffffff806576ac in g_io_schedule_up (tp=3D) at ../../../geom/geom_io.c:808 #11 0xffffffff80657e0c in g_up_procbody (arg=3D) at ../../../geom/geom_kern.c:97 #12 0xffffffff806c0b4f in fork_exit ( callout=3D0xffffffff80657db0 , arg=3D0x0,=20 frame=3D0xffffff800024ec40) at ../../../kern/kern_fork.c:1000 #13 0xffffffff808fa83e in fork_trampoline () at ../../../amd64/amd64/exception.S:613 #14 0x0000000000000000 in ?? () (kgdb)=20 geom/mirror/g_mirror.c:990: static void g_mirror_sync_done(struct bio *bp) { struct g_mirror_softc *sc; G_MIRROR_LOGREQ(3, bp, "Synchronization request delivered."); sc =3D bp->bio_from->geom->softc; bp->bio_cflags =3D G_MIRROR_BIO_FLAG_SYNC; mtx_lock(&sc->sc_queue_mtx); <--- bioq_insert_tail(&sc->sc_queue, bp); mtx_unlock(&sc->sc_queue_mtx); wakeup(sc); } kern/vfs_vio.c:3667: done =3D bp->bio_done; if (done =3D=3D NULL) wakeup(bp); mtx_unlock(mtxp); if (done !=3D NULL) done(bp); <--- if (transient) { pmap_qremove(start, OFF_TO_IDX(end - start)); vm_map_remove(bio_transient_map, start, end); atomic_add_int(&inflight_transient_maps, -1); } } geom/geom_io.c:808: bp =3D g_bioq_first(&g_bio_run_up); if (bp !=3D NULL) { g_bioq_unlock(&g_bio_run_up); THREAD_NO_SLEEPING(); CTR4(KTR_GEOM, "g_up biodone bp %p provider %s = off " "%jd len %ld", bp, bp->bio_to->name, bp->bio_offset, bp->bio_length); biodone(bp); <--- THREAD_SLEEPING_OK(); continue; } CTR0(KTR_GEOM, "g_up going to sleep"); geom/geom_kern.c:97: static void g_up_procbody(void *arg) { mtx_assert(&Giant, MA_NOTOWNED); thread_lock(g_up_td); sched_prio(g_up_td, PRIBIO); thread_unlock(g_up_td); for(;;) { g_io_schedule_up(g_up_td); <--- } }