From owner-freebsd-current@freebsd.org Fri Apr 28 11:56:26 2017 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id DDFC7D54DC8 for ; Fri, 28 Apr 2017 11:56:26 +0000 (UTC) (envelope-from mikej@mikej.com) Received: from mx2.paymentallianceintl.com (mx2.paymentallianceintl.com [216.26.158.171]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mx2.paymentallianceintl.com", Issuer "Go Daddy Secure Certificate Authority - G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id A428DFC3; Fri, 28 Apr 2017 11:56:26 +0000 (UTC) (envelope-from mikej@mikej.com) Received: from firewall.mikej.com (f [162.230.214.65]) by mx2.paymentallianceintl.com (8.15.2/8.15.2) with ESMTPS id v3SBuNXF052874 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL); Fri, 28 Apr 2017 07:56:24 -0400 (EDT) (envelope-from mikej@mikej.com) X-SenderID: Sendmail Sender-ID Filter v1.0.0 mx2.paymentallianceintl.com v3SBuNXF052874 Authentication-Results: mx2.paymentallianceintl.com; sender-id=pass header.from=mikej@mikej.com; spf=pass smtp.mfrom=mikej@mikej.com X-Authentication-Warning: mx2.paymentallianceintl.com: Host f [162.230.214.65] claimed to be firewall.mikej.com Received: from mail.mikej.com (firewall [192.168.6.63]) by firewall.mikej.com (8.15.2/8.15.2) with ESMTP id v3SBuMpb024495; Fri, 28 Apr 2017 07:56:22 -0400 (EDT) (envelope-from mikej@mikej.com) DMARC-Filter: OpenDMARC Filter v1.3.1 firewall.mikej.com v3SBuMpb024495 Authentication-Results: mail.mikej.com; dmarc=none header.from=mikej.com DKIM-Filter: OpenDKIM Filter v2.10.3 firewall.mikej.com v3SBuMpb024495 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=mikej.com; s=mail; t=1493380583; bh=UTkmIC2gbamUzy6QXl8dy38l8mxF6wQgg0PbowT3MAo=; h=Date:From:To:Cc:Subject:In-Reply-To:References; b=PRL/QdYJo0RCV2hcHKhjkJL04gzO2tKfNYOr2fJ0HwgSONKH21zOWOe4FvI3x5oI+ 5gUvhTPGhIoHt3M70dJ5ASmxfY2myaYIceP43mkGozjAajDqUfHRstgJoWJXnpD/zQ wLQm50ixYF9V6aYI0IxXAOvaFg1YZePCgfBDt4YBh4SQw9p7EzKuDK2RbYXIYMt47X wroJGzZm+YHuoD6BqfZdpcAagPSkLFKdBHZe9YoI4hKEJF7/aBn8yGqL+thO6f6gry 5qfatyGfTAB1uZQTqDMhulklwGInWKQL1dkLHCxLzjsG9DO/tJuk0pXjQ1NF0FnSlR mOit9yRMdpX6A== X-Authentication-Warning: firewall.mikej.com: Host firewall [192.168.6.63] claimed to be mail.mikej.com MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII; format=flowed Content-Transfer-Encoding: 7bit Date: Fri, 28 Apr 2017 07:56:21 -0400 From: Michael Jung To: Andriy Gapon Cc: freebsd-current@freebsd.org Subject: Re: Panic String: solaris assert: (lsize != psize) implies ((flags & ZIO_FLAG_RAW) != 0), file: /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c, line: 631 In-Reply-To: <32c84a1f-1377-e0a1-1c8b-d22eea80d871@FreeBSD.org> References: <32c84a1f-1377-e0a1-1c8b-d22eea80d871@FreeBSD.org> Message-ID: <19ac2524eba83333063822c063c6af3e@mikej.com> X-Sender: mikej@mikej.com User-Agent: Roundcube Webmail/1.2.4 X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Apr 2017 11:56:27 -0000 On 2017-04-27 17:57, Andriy Gapon wrote: > On 27/04/2017 18:52, Michael Jung wrote: >> Hi: >> >> Recently upgraded from r315905 to r317435 and during a poudriere run >> got this >> panic which I have not seen before. >> >> https://charon.gopai.com/core.txt.1 >> https://charon.gopai.com/info.1 >> >> Let me know what additional information I might supply. > > Mike, > > could you please edit function zio_free_sync() in > sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c so that the > zio_create() > call has "size, size" arguments instead of "size, BP_GET_PSIZE(bp)" and > see if > that helps? > (Your pool is probably low on space too.) > >> panic: solaris assert: (lsize != psize) implies ((flags & >> ZIO_FLAG_RAW) != 0), >> file: /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c, >> line: 631 >> cpuid = 6 >> time = 1493306220 >> KDB: stack backtrace: >> db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame >> 0xfffffe086140e850 >> vpanic() at vpanic+0x19c/frame 0xfffffe086140e8d0 >> panic() at panic+0x43/frame 0xfffffe086140e930 >> assfail() at assfail+0x1a/frame 0xfffffe086140e940 >> zio_create() at zio_create+0x11f/frame 0xfffffe086140e9a0 >> zio_free_sync() at zio_free_sync+0x197/frame 0xfffffe086140ea50 >> zio_gang_tree_issue() at zio_gang_tree_issue+0x13f/frame >> 0xfffffe086140eaa0 >> zio_gang_issue() at zio_gang_issue+0x152/frame 0xfffffe086140ead0 >> zio_execute() at zio_execute+0x36c/frame 0xfffffe086140eb20 >> taskqueue_run_locked() at taskqueue_run_locked+0x13d/frame >> 0xfffffe086140eb80 >> taskqueue_thread_loop() at taskqueue_thread_loop+0x88/frame >> 0xfffffe086140ebb0 >> fork_exit() at fork_exit+0x84/frame 0xfffffe086140ebf0 >> fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe086140ebf0 >> --- trap 0, rip = 0, rsp = 0, rbp = 0 --- >> KDB: enter: panic >> >> Reading symbols from /boot/kernel/vmm.ko...Reading symbols from >> /usr/lib/debug//boot/kernel/vmm.ko.debug...done. >> done. >> Loaded symbols for /boot/kernel/vmm.ko >> Reading symbols from /boot/kernel/filemon.ko...Reading symbols from >> /usr/lib/debug//boot/kernel/filemon.ko.debug...done. >> done. >> Loaded symbols for /boot/kernel/filemon.ko >> Reading symbols from /boot/kernel/zfs.ko...Reading symbols from >> /usr/lib/debug//boot/kernel/zfs.ko.debug...done. >> done. >> Loaded symbols for /boot/kernel/zfs.ko >> Reading symbols from /boot/kernel/opensolaris.ko...Reading symbols >> from >> /usr/lib/debug//boot/kernel/opensolaris.ko.debug...done. >> done. >> Loaded symbols for /boot/kernel/opensolaris.ko >> Reading symbols from >> /usr/local/lib/vmware-tools/modules/drivers/vmmemctl.ko...done. >> Loaded symbols for >> /usr/local/lib/vmware-tools/modules/drivers/vmmemctl.ko >> Reading symbols from >> /usr/local/lib/vmware-tools/modules/drivers/vmxnet.ko...done. >> Loaded symbols for >> /usr/local/lib/vmware-tools/modules/drivers/vmxnet.ko >> Reading symbols from >> /usr/local/lib/vmware-tools/modules/drivers/vmblock.ko...done. >> Loaded symbols for >> /usr/local/lib/vmware-tools/modules/drivers/vmblock.ko >> Reading symbols from >> /usr/local/lib/vmware-tools/modules/drivers/vmhgfs.ko...done. >> Loaded symbols for >> /usr/local/lib/vmware-tools/modules/drivers/vmhgfs.ko >> Reading symbols from /boot/kernel/linux.ko...Reading symbols from >> /usr/lib/debug//boot/kernel/linux.ko.debug...done. >> done. >> Loaded symbols for /boot/kernel/linux.ko >> Reading symbols from /boot/kernel/linux_common.ko...Reading symbols >> from >> /usr/lib/debug//boot/kernel/linux_common.ko.debug...done. >> done. >> Loaded symbols for /boot/kernel/linux_common.ko >> Reading symbols from /boot/kernel/linux64.ko...Reading symbols from >> /usr/lib/debug//boot/kernel/linux64.ko.debug...done. >> done. >> Loaded symbols for /boot/kernel/linux64.ko >> Reading symbols from /boot/kernel/nullfs.ko...Reading symbols from >> /usr/lib/debug//boot/kernel/nullfs.ko.debug...done. >> done. >> Loaded symbols for /boot/kernel/nullfs.ko >> Reading symbols from /boot/kernel/linprocfs.ko...Reading symbols from >> /usr/lib/debug//boot/kernel/linprocfs.ko.debug...done. >> done. >> Loaded symbols for /boot/kernel/linprocfs.ko >> Reading symbols from /boot/kernel/tmpfs.ko...Reading symbols from >> /usr/lib/debug//boot/kernel/tmpfs.ko.debug...done. >> done. >> Loaded symbols for /boot/kernel/tmpfs.ko >> Reading symbols from /boot/kernel/fdescfs.ko...Reading symbols from >> /usr/lib/debug//boot/kernel/fdescfs.ko.debug...done. >> done. >> Loaded symbols for /boot/kernel/fdescfs.ko >> #0 doadump (textdump=0) at pcpu.h:232 >> 232 pcpu.h: No such file or directory. >> in pcpu.h >> (kgdb) #0 doadump (textdump=0) at pcpu.h:232 >> #1 0xffffffff803a1f7b in db_dump (dummy=, >> dummy2=, dummy3=, >> dummy4=) at /usr/src/sys/ddb/db_command.c:546 >> #2 0xffffffff803a1d6f in db_command (cmd_table=) >> at /usr/src/sys/ddb/db_command.c:453 >> #3 0xffffffff803a1aa4 in db_command_loop () >> at /usr/src/sys/ddb/db_command.c:506 >> #4 0xffffffff803a4b6f in db_trap (type=, >> code=) at /usr/src/sys/ddb/db_main.c:248 >> #5 0xffffffff80a93333 in kdb_trap (type=3, code=-61456, >> tf=) at /usr/src/sys/kern/subr_kdb.c:654 >> #6 0xffffffff80ed2de6 in trap (frame=0xfffffe086140e780) >> at /usr/src/sys/amd64/amd64/trap.c:537 >> #7 0xffffffff80eb54e1 in calltrap () >> at /usr/src/sys/amd64/amd64/exception.S:236 >> #8 0xffffffff80a92a6b in kdb_enter (why=0xffffffff8143c265 "panic", >> msg=) at cpufunc.h:63 >> #9 0xffffffff80a513c9 in vpanic (fmt=, >> ap=0xfffffe086140e910) at /usr/src/sys/kern/kern_shutdown.c:772 >> #10 0xffffffff80a51433 in panic (fmt=) >> at /usr/src/sys/kern/kern_shutdown.c:710 >> #11 0xffffffff82a6623a in assfail (a=, >> f=, l=) >> at >> /usr/src/sys/cddl/compat/opensolaris/kern/opensolaris_cmn_err.c:81 >> #12 0xffffffff828eab3f in zio_create (pio=0xfffff807def8e810, >> spa=, txg=19514648, bp=0xfffff807def8e880, >> data=0x0, >> lsize=512, psize=1024, done=0, private=0x19, type=ZIO_TYPE_NULL, >> priority=512, offset=0, zb=0xffffffff80a98d40, pipeline=17301632) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:631 >> #13 0xffffffff828eb897 in zio_free_sync (pio=0xfffff807def8e810, >> spa=0xfffffe000289e000, txg=19514648, bp=0xfffff807def8e880, >> size=, flags=17301632) at time.h:67 >> #14 0xffffffff828f422f in zio_gang_tree_issue (pio=0xfffff807def8e810, >> gn=0xfffff805bd9162e0, bp=0xfffff807def8e880, data=0x0) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2114 >> #15 0xffffffff828f0992 in zio_gang_issue (zio=0xfffff807def8e810) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2162 >> #16 0xffffffff828ecb4c in zio_execute (zio=) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1694 >> #17 0xffffffff80aa56bd in taskqueue_run_locked >> (queue=0xfffff800096bf900) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2114 >> #15 0xffffffff828f0992 in zio_gang_issue (zio=0xfffff807def8e810) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2162 >> #16 0xffffffff828ecb4c in zio_execute (zio=) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1694 >> #17 0xffffffff80aa56bd in taskqueue_run_locked >> (queue=0xfffff800096bf900) >> at /usr/src/sys/kern/subr_taskqueue.c:454 >> #18 0xffffffff80aa6478 in taskqueue_thread_loop (arg=> out>) >> at /usr/src/sys/kern/subr_taskqueue.c:746 >> #19 0xffffffff80a13074 in fork_exit ( >> callout=0xffffffff80aa63f0 , >> arg=0xfffff80009350f00, frame=0xfffffe086140ec00) >> at /usr/src/sys/kern/kern_fork.c:1038 >> #20 0xffffffff80eb5a1e in fork_trampoline () >> at /usr/src/sys/amd64/amd64/exception.S:611 >> #21 0x0000000000000000 in ?? () >> Current language: auto; currently minimal >> (kgdb) >> _______________________________________________ >> freebsd-current@freebsd.org mailing list >> https://lists.freebsd.org/mailman/listinfo/freebsd-current >> To unsubscribe, send any mail to >> "freebsd-current-unsubscribe@freebsd.org" >> Andriy: I have mad the requested change.. [root@bsd11 /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs]# diff zio.c ~mikej/zio.c.orig 965c965 < size, NULL, NULL, ZIO_TYPE_FREE, ZIO_PRIORITY_NOW, --- > BP_GET_PSIZE(bp), NULL, NULL, ZIO_TYPE_FREE, ZIO_PRIORITY_NOW, [root@bsd11 /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs]# As to the pool size: [root@bsd11 /usr/home/mikej]# zpool list NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT tank 199G 143G 55.9G - 85% 71% 1.00x ONLINE - [root@bsd11 /usr/home/mikej]# I should have also mentioned that besides poudriere running a build, it was removing old logs - There was some 43G of old logs files that were in the process of being removed. I will hammer the box with and report back first of the week whether the panic re-occurs or not. Thank you. Michael Jung