From owner-freebsd-stable@freebsd.org Sun Jun 11 10:37:32 2017 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 41B6DC78CD3; Sun, 11 Jun 2017 10:37:32 +0000 (UTC) (envelope-from freebsd@omnilan.de) Received: from mx0.gentlemail.de (mx0.gentlemail.de [IPv6:2a00:e10:2800::a130]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id C5F4C9D2; Sun, 11 Jun 2017 10:37:31 +0000 (UTC) (envelope-from freebsd@omnilan.de) Received: from mh0.gentlemail.de (ezra.dcm1.omnilan.net [IPv6:2a00:e10:2800::a135]) by mx0.gentlemail.de (8.14.5/8.14.5) with ESMTP id v5BAbTel080220; Sun, 11 Jun 2017 12:37:30 +0200 (CEST) (envelope-from freebsd@omnilan.de) Received: from titan.inop.mo1.omnilan.net (s1.omnilan.de [217.91.127.234]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mh0.gentlemail.de (Postfix) with ESMTPSA id B8AD3F8F; Sun, 11 Jun 2017 12:37:29 +0200 (CEST) Message-ID: <593D1D5C.907@omnilan.de> Date: Sun, 11 Jun 2017 12:37:16 +0200 From: Harry Schmalzbauer Organization: OmniLAN User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; de-DE; rv:1.9.2.8) Gecko/20100906 Lightning/1.0b2 Thunderbird/3.1.2 MIME-Version: 1.0 To: freebsd-stable@freebsd.org, FreeBSD virtualization Subject: panic: Memory modified after free in zio_create, passthru in use [Was: 11.1-pre runtime Undefined symbol "xdr_accepted_reply" /lib/libc.so.7] References: <59369A15.2010901@omnilan.de> In-Reply-To: <59369A15.2010901@omnilan.de> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 8bit X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.7 (mx0.gentlemail.de [IPv6:2a00:e10:2800::a130]); Sun, 11 Jun 2017 12:37:30 +0200 (CEST) X-Milter: Spamilter (Reciever: mx0.gentlemail.de; Sender-ip: ; Sender-helo: mh0.gentlemail.de; ) X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 11 Jun 2017 10:37:32 -0000 Bezüglich Harry Schmalzbauer's Nachricht vom 06.06.2017 14:03 (localtime): > Hello, > > suddenly, I'm getting this error: > /lib/libc.so.7: Undefined symbol "xdr_accepted_reply" > > Very mysterious: It showed up on a running system, which worked > flawlessly for some hours. And that host has root-fs (/) mounted > readonly from a memorydisk. So to my understanding, it's completely > impossible that /lib/libc.so.7 is corrupted since last boot. > > I'm completely out of ideas what could cause this strange error during > "normal" operation. > > Normal operation in this case is serving as a bhyve test machine. > I first noticed that error after one guest - with passthru device > attached - was shut down. > > My suspicion is some undiscovered passthru interference... Since I > noticed one other _very_ strange passthru-effect: > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=215740 Hello, this time I caught a panic with a debuging kernel under 11.1-BETA1, which again occured after shuting down a VM which had ppt in use: cpuid = 5 KDB: stack backtrace: #0 0xffffffff805bf327 at kdb_backtrace+0x67 #1 0xffffffff8057f266 at vpanic+0x186 #2 0xffffffff8057f2e3 at panic+0x43 #3 0xffffffff8082eaeb at trash_ctor+0x4b #4 0xffffffff8082aaec at uma_zalloc_arg+0x52c #5 0xffffffff813b54a6 at zio_add_child+0x26 #6 0xffffffff813b5a05 at zio_create+0x385 #7 0xffffffff813b6de2 at zio_vdev_child_io+0x232 #8 0xffffffff81396be0 at vdev_mirror_io_start+0x370 #9 0xffffffff813bc629 at zio_vdev_io_start+0x4a9 #10 0xffffffff813b76bc at zio_execute+0x36c #11 0xffffffff813b6868 at zio_nowait+0xb8 #12 0xffffffff81396bec at vdev_mirror_io_start+0x37c #13 0xffffffff813bc383 at zio_vdev_io_start+0x203 #14 0xffffffff813b76bc at zio_execute+0x36c #15 0xffffffff805d10dd at taskqueue_run_locked+0x13d #16 0xffffffff805d1e78 at taskqueue_thread_loop+0x88 #17 0xffffffff80543844 at fork_exit+0x84 #0 doadump (textdump=) at pcpu.h:222 #1 0xffffffff8057ece0 in kern_reboot (howto=260) at /usr/local/share/deploy-tools/RELENG_11/src/sys/kern/kern_shutdown.c:366 #2 0xffffffff8057f2a0 in vpanic (fmt=, ap=) at /usr/local/share/deploy-tools/RELENG_11/src/sys/kern/kern_shutdown.c:759 #3 0xffffffff8057f2e3 in panic (fmt=) at /usr/local/share/deploy-tools/RELENG_11/src/sys/kern/kern_shutdown.c:690 #4 0xffffffff8082eaeb in trash_ctor (mem=, size=, arg=, flags=) at /usr/local/share/deploy-tools/RELENG_11/src/sys/vm/uma_dbg.c:80 #5 0xffffffff8082aaec in uma_zalloc_arg (zone=0xfffff8001febc680, udata=0xfffff8001ad5f340, flags=) at /usr/local/share/deploy-tools/RELENG_11/src/sys/vm/uma_core.c:2152 #6 0xffffffff813b54a6 in zio_add_child (pio=0xfffff8026f350b88, cio=0xfffff8002478b7b0) at /usr/local/share/deploy-tools/RELENG_11/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:460 #7 0xffffffff813b5a05 in zio_create (pio=0xfffff8026f350b88, spa=, txg=433989, bp=, data=0xfffffe0058afa000, size=1024, type=, priority=ZIO_PRIORITY_ASYNC_WRITE, flags=, vd=, offset=, zb=, pipeline=) at /usr/local/share/deploy-tools/RELENG_11/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:690 #8 0xffffffff813b6de2 in zio_vdev_child_io (pio=0xfffff8026f350b88, bp=, vd=, offset=325398016, data=, size=1024, type=, flags=1048704, done=) at /usr/local/share/deploy-tools/RELENG_11/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1141 #9 0xffffffff81396be0 in vdev_mirror_io_start (zio=0xfffff8026f350b88) at /usr/local/share/deploy-tools/RELENG_11/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_mirror.c:488 #10 0xffffffff813bc629 in zio_vdev_io_start (zio=0xfffff8026f350b88) at /usr/local/share/deploy-tools/RELENG_11/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3143 #11 0xffffffff813b76bc in zio_execute (zio=) at /usr/local/share/deploy-tools/RELENG_11/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1681 #12 0xffffffff813b6868 in zio_nowait (zio=0xfffff8026f350b88) at /usr/local/share/deploy-tools/RELENG_11/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1739 #13 0xffffffff81396bec in vdev_mirror_io_start (zio=0xfffff8026f7a7b88) at /usr/local/share/deploy-tools/RELENG_11/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_mirror.c:488 #14 0xffffffff813bc383 in zio_vdev_io_start (zio=0xfffff8026f7a7b88) at /usr/local/share/deploy-tools/RELENG_11/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3021 #15 0xffffffff813b76bc in zio_execute (zio=) at /usr/local/share/deploy-tools/RELENG_11/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1681 #16 0xffffffff805d10dd in taskqueue_run_locked (queue=0xfffff8001ab5a700) at /usr/local/share/deploy-tools/RELENG_11/src/sys/kern/subr_taskqueue.c:454 #17 0xffffffff805d1e78 in taskqueue_thread_loop (arg=) at /usr/local/share/deploy-tools/RELENG_11/src/sys/kern/subr_taskqueue.c:741 #18 0xffffffff80543844 in fork_exit (callout=0xffffffff805d1df0 , arg=0xfffff8001aa90720, frame=0xfffffe043f609ac0) at /usr/local/share/deploy-tools/RELENG_11/src/sys/kern/kern_fork.c:1042 #19 0xffffffff808598ae in fork_trampoline () at /usr/local/share/deploy-tools/RELENG_11/src/sys/amd64/amd64/exception.S:611 #20 0x0000000000000000 in ?? () This machine is usually rock solid, but shows most strange errors each time I utilize "passthru" with bhyve. Besides runngin a debug kernel, I don't know how to help tracking this down. I can imagine that above quoted PR and the unexplainable »lib/libc.so.7: Undefined symbol "xdr_accepted_reply"« error all have the same undiscovered cause, which shows up as soon as byhve and passtrhu are involved. Please, can anybody of the xperts add a comment? Thanks, -harry