Skip site navigation (1)Skip section navigation (2)
Date:      Fri, 28 Apr 2017 07:56:21 -0400
From:      Michael Jung <mikej@mikej.com>
To:        Andriy Gapon <avg@freebsd.org>
Cc:        freebsd-current@freebsd.org
Subject:   Re: Panic String: solaris assert: (lsize != psize) implies ((flags & ZIO_FLAG_RAW) != 0), file: /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c, line: 631
Message-ID:  <19ac2524eba83333063822c063c6af3e@mikej.com>
In-Reply-To: <32c84a1f-1377-e0a1-1c8b-d22eea80d871@FreeBSD.org>
References:  <b5bc4e3a078eb153e54a802f80161b12@mikej.com> <32c84a1f-1377-e0a1-1c8b-d22eea80d871@FreeBSD.org>

next in thread | previous in thread | raw e-mail | index | archive | help
On 2017-04-27 17:57, Andriy Gapon wrote:
> On 27/04/2017 18:52, Michael Jung wrote:
>> Hi:
>> 
>> Recently upgraded from r315905 to r317435 and during a poudriere run 
>> got this
>> panic which I have not seen before.
>> 
>> https://charon.gopai.com/core.txt.1
>> https://charon.gopai.com/info.1
>> 
>> Let me know what additional information I might supply.
> 
> Mike,
> 
> could you please edit function zio_free_sync() in
> sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c so that the 
> zio_create()
> call has "size, size" arguments instead of "size, BP_GET_PSIZE(bp)" and 
> see if
> that helps?
> (Your pool is probably low on space too.)
> 
>> panic: solaris assert: (lsize != psize) implies ((flags & 
>> ZIO_FLAG_RAW) != 0),
>> file: /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c, 
>> line: 631
>> cpuid = 6
>> time = 1493306220
>> KDB: stack backtrace:
>> db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 
>> 0xfffffe086140e850
>> vpanic() at vpanic+0x19c/frame 0xfffffe086140e8d0
>> panic() at panic+0x43/frame 0xfffffe086140e930
>> assfail() at assfail+0x1a/frame 0xfffffe086140e940
>> zio_create() at zio_create+0x11f/frame 0xfffffe086140e9a0
>> zio_free_sync() at zio_free_sync+0x197/frame 0xfffffe086140ea50
>> zio_gang_tree_issue() at zio_gang_tree_issue+0x13f/frame 
>> 0xfffffe086140eaa0
>> zio_gang_issue() at zio_gang_issue+0x152/frame 0xfffffe086140ead0
>> zio_execute() at zio_execute+0x36c/frame 0xfffffe086140eb20
>> taskqueue_run_locked() at taskqueue_run_locked+0x13d/frame 
>> 0xfffffe086140eb80
>> taskqueue_thread_loop() at taskqueue_thread_loop+0x88/frame 
>> 0xfffffe086140ebb0
>> fork_exit() at fork_exit+0x84/frame 0xfffffe086140ebf0
>> fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe086140ebf0
>> --- trap 0, rip = 0, rsp = 0, rbp = 0 ---
>> KDB: enter: panic
>> 
>> Reading symbols from /boot/kernel/vmm.ko...Reading symbols from
>> /usr/lib/debug//boot/kernel/vmm.ko.debug...done.
>> done.
>> Loaded symbols for /boot/kernel/vmm.ko
>> Reading symbols from /boot/kernel/filemon.ko...Reading symbols from
>> /usr/lib/debug//boot/kernel/filemon.ko.debug...done.
>> done.
>> Loaded symbols for /boot/kernel/filemon.ko
>> Reading symbols from /boot/kernel/zfs.ko...Reading symbols from
>> /usr/lib/debug//boot/kernel/zfs.ko.debug...done.
>> done.
>> Loaded symbols for /boot/kernel/zfs.ko
>> Reading symbols from /boot/kernel/opensolaris.ko...Reading symbols 
>> from
>> /usr/lib/debug//boot/kernel/opensolaris.ko.debug...done.
>> done.
>> Loaded symbols for /boot/kernel/opensolaris.ko
>> Reading symbols from
>> /usr/local/lib/vmware-tools/modules/drivers/vmmemctl.ko...done.
>> Loaded symbols for 
>> /usr/local/lib/vmware-tools/modules/drivers/vmmemctl.ko
>> Reading symbols from 
>> /usr/local/lib/vmware-tools/modules/drivers/vmxnet.ko...done.
>> Loaded symbols for 
>> /usr/local/lib/vmware-tools/modules/drivers/vmxnet.ko
>> Reading symbols from 
>> /usr/local/lib/vmware-tools/modules/drivers/vmblock.ko...done.
>> Loaded symbols for 
>> /usr/local/lib/vmware-tools/modules/drivers/vmblock.ko
>> Reading symbols from 
>> /usr/local/lib/vmware-tools/modules/drivers/vmhgfs.ko...done.
>> Loaded symbols for 
>> /usr/local/lib/vmware-tools/modules/drivers/vmhgfs.ko
>> Reading symbols from /boot/kernel/linux.ko...Reading symbols from
>> /usr/lib/debug//boot/kernel/linux.ko.debug...done.
>> done.
>> Loaded symbols for /boot/kernel/linux.ko
>> Reading symbols from /boot/kernel/linux_common.ko...Reading symbols 
>> from
>> /usr/lib/debug//boot/kernel/linux_common.ko.debug...done.
>> done.
>> Loaded symbols for /boot/kernel/linux_common.ko
>> Reading symbols from /boot/kernel/linux64.ko...Reading symbols from
>> /usr/lib/debug//boot/kernel/linux64.ko.debug...done.
>> done.
>> Loaded symbols for /boot/kernel/linux64.ko
>> Reading symbols from /boot/kernel/nullfs.ko...Reading symbols from
>> /usr/lib/debug//boot/kernel/nullfs.ko.debug...done.
>> done.
>> Loaded symbols for /boot/kernel/nullfs.ko
>> Reading symbols from /boot/kernel/linprocfs.ko...Reading symbols from
>> /usr/lib/debug//boot/kernel/linprocfs.ko.debug...done.
>> done.
>> Loaded symbols for /boot/kernel/linprocfs.ko
>> Reading symbols from /boot/kernel/tmpfs.ko...Reading symbols from
>> /usr/lib/debug//boot/kernel/tmpfs.ko.debug...done.
>> done.
>> Loaded symbols for /boot/kernel/tmpfs.ko
>> Reading symbols from /boot/kernel/fdescfs.ko...Reading symbols from
>> /usr/lib/debug//boot/kernel/fdescfs.ko.debug...done.
>> done.
>> Loaded symbols for /boot/kernel/fdescfs.ko
>> #0  doadump (textdump=0) at pcpu.h:232
>> 232     pcpu.h: No such file or directory.
>>         in pcpu.h
>> (kgdb) #0  doadump (textdump=0) at pcpu.h:232
>> #1  0xffffffff803a1f7b in db_dump (dummy=<value optimized out>,
>>     dummy2=<value optimized out>, dummy3=<value optimized out>,
>>     dummy4=<value optimized out>) at /usr/src/sys/ddb/db_command.c:546
>> #2  0xffffffff803a1d6f in db_command (cmd_table=<value optimized out>)
>>     at /usr/src/sys/ddb/db_command.c:453
>> #3  0xffffffff803a1aa4 in db_command_loop ()
>>     at /usr/src/sys/ddb/db_command.c:506
>> #4  0xffffffff803a4b6f in db_trap (type=<value optimized out>,
>>     code=<value optimized out>) at /usr/src/sys/ddb/db_main.c:248
>> #5  0xffffffff80a93333 in kdb_trap (type=3, code=-61456,
>>     tf=<value optimized out>) at /usr/src/sys/kern/subr_kdb.c:654
>> #6  0xffffffff80ed2de6 in trap (frame=0xfffffe086140e780)
>>     at /usr/src/sys/amd64/amd64/trap.c:537
>> #7  0xffffffff80eb54e1 in calltrap ()
>>     at /usr/src/sys/amd64/amd64/exception.S:236
>> #8  0xffffffff80a92a6b in kdb_enter (why=0xffffffff8143c265 "panic",
>>     msg=<value optimized out>) at cpufunc.h:63
>> #9  0xffffffff80a513c9 in vpanic (fmt=<value optimized out>,
>>     ap=0xfffffe086140e910) at /usr/src/sys/kern/kern_shutdown.c:772
>> #10 0xffffffff80a51433 in panic (fmt=<value optimized out>)
>>     at /usr/src/sys/kern/kern_shutdown.c:710
>> #11 0xffffffff82a6623a in assfail (a=<value optimized out>,
>>     f=<value optimized out>, l=<value optimized out>)
>>     at 
>> /usr/src/sys/cddl/compat/opensolaris/kern/opensolaris_cmn_err.c:81
>> #12 0xffffffff828eab3f in zio_create (pio=0xfffff807def8e810,
>>     spa=<value optimized out>, txg=19514648, bp=0xfffff807def8e880, 
>> data=0x0,
>>     lsize=512, psize=1024, done=0, private=0x19, type=ZIO_TYPE_NULL,
>>     priority=512, offset=0, zb=0xffffffff80a98d40, pipeline=17301632)
>>     at 
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:631
>> #13 0xffffffff828eb897 in zio_free_sync (pio=0xfffff807def8e810,
>>     spa=0xfffffe000289e000, txg=19514648, bp=0xfffff807def8e880,
>>     size=<value optimized out>, flags=17301632) at time.h:67
>> #14 0xffffffff828f422f in zio_gang_tree_issue (pio=0xfffff807def8e810,
>>     gn=0xfffff805bd9162e0, bp=0xfffff807def8e880, data=0x0)
>>     at 
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2114
>> #15 0xffffffff828f0992 in zio_gang_issue (zio=0xfffff807def8e810)
>>     at 
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2162
>> #16 0xffffffff828ecb4c in zio_execute (zio=<value optimized out>)
>>     at 
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1694
>> #17 0xffffffff80aa56bd in taskqueue_run_locked 
>> (queue=0xfffff800096bf900)
>>     at 
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2114
>> #15 0xffffffff828f0992 in zio_gang_issue (zio=0xfffff807def8e810)
>>     at 
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2162
>> #16 0xffffffff828ecb4c in zio_execute (zio=<value optimized out>)
>>     at 
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1694
>> #17 0xffffffff80aa56bd in taskqueue_run_locked 
>> (queue=0xfffff800096bf900)
>>     at /usr/src/sys/kern/subr_taskqueue.c:454
>> #18 0xffffffff80aa6478 in taskqueue_thread_loop (arg=<value optimized 
>> out>)
>>     at /usr/src/sys/kern/subr_taskqueue.c:746
>> #19 0xffffffff80a13074 in fork_exit (
>>     callout=0xffffffff80aa63f0 <taskqueue_thread_loop>,
>>     arg=0xfffff80009350f00, frame=0xfffffe086140ec00)
>>     at /usr/src/sys/kern/kern_fork.c:1038
>> #20 0xffffffff80eb5a1e in fork_trampoline ()
>>     at /usr/src/sys/amd64/amd64/exception.S:611
>> #21 0x0000000000000000 in ?? ()
>> Current language:  auto; currently minimal
>> (kgdb)
>> _______________________________________________
>> freebsd-current@freebsd.org mailing list
>> https://lists.freebsd.org/mailman/listinfo/freebsd-current
>> To unsubscribe, send any mail to 
>> "freebsd-current-unsubscribe@freebsd.org"
>> 

Andriy:

I have mad the requested change..

[root@bsd11 /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs]# 
diff zio.c ~mikej/zio.c.orig
965c965
< 	    size, NULL, NULL, ZIO_TYPE_FREE, ZIO_PRIORITY_NOW,
---
> 	    BP_GET_PSIZE(bp), NULL, NULL, ZIO_TYPE_FREE, ZIO_PRIORITY_NOW,
[root@bsd11 /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs]#

As to the pool size:

[root@bsd11 /usr/home/mikej]# zpool list
NAME   SIZE  ALLOC   FREE  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  
ALTROOT
tank   199G   143G  55.9G         -    85%    71%  1.00x  ONLINE  -
[root@bsd11 /usr/home/mikej]#

I should have also mentioned that besides poudriere running a build, it 
was
removing old logs - There was some 43G of old logs files that were in 
the process
of being removed.

I will hammer the box with and report back first of the week whether the 
panic
re-occurs or not.

Thank you.

Michael Jung






Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?19ac2524eba83333063822c063c6af3e>