From owner-freebsd-current@freebsd.org  Fri Jul 13 13:19:54 2018
Return-Path: <owner-freebsd-current@freebsd.org>
Delivered-To: freebsd-current@mailman.ysv.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
 by mailman.ysv.freebsd.org (Postfix) with ESMTP id E35B9103C373
 for <freebsd-current@mailman.ysv.freebsd.org>;
 Fri, 13 Jul 2018 13:19:53 +0000 (UTC) (envelope-from slw@zxy.spb.ru)
Received: from zxy.spb.ru (zxy.spb.ru [195.70.199.98])
 (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
 (Client did not present a certificate)
 by mx1.freebsd.org (Postfix) with ESMTPS id 5ECFE85A7D
 for <freebsd-current@freebsd.org>; Fri, 13 Jul 2018 13:19:53 +0000 (UTC)
 (envelope-from slw@zxy.spb.ru)
Received: from slw by zxy.spb.ru with local (Exim 4.86 (FreeBSD))
 (envelope-from <slw@zxy.spb.ru>)
 id 1fdxz4-0006XS-DU; Fri, 13 Jul 2018 16:19:42 +0300
Date: Fri, 13 Jul 2018 16:19:42 +0300
From: Slawa Olhovchenkov <slw@zxy.spb.ru>
To: Alexander Leidinger <Alexander@leidinger.net>
Cc: freebsd-current@freebsd.org
Subject: Re: Deadlocks / hangs in ZFS
Message-ID: <20180713131942.GA30893@zxy.spb.ru>
References: <20180522122924.GC1954@zxy.spb.ru>
 <20180522161632.Horde.ROSnBoZixBoE9ZBGp5VBQgZ@webmail.leidinger.net>
 <20180522144055.GD1954@zxy.spb.ru>
 <20180527194159.v54ox3vlthpuvx4q@jo>
 <20180527220612.GK1926@zxy.spb.ru>
 <20180528090201.Horde._E4JZcuEaZHfj_BNzWjci2O@webmail.leidinger.net>
 <20180603211450.Horde.pI-Fom6S1tUcaHvTF4MUjin@webmail.leidinger.net>
 <20180603192814.GP1926@zxy.spb.ru>
 <20180604223108.Horde.RcVquaVKWdNzNidD_5aJz7E@webmail.leidinger.net>
 <20180712144229.Horde.D_-hM4wiiKjbsuR-VROmvkZ@webmail.leidinger.net>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20180712144229.Horde.D_-hM4wiiKjbsuR-VROmvkZ@webmail.leidinger.net>
User-Agent: Mutt/1.5.24 (2015-08-30)
X-SA-Exim-Connect-IP: <locally generated>
X-SA-Exim-Mail-From: slw@zxy.spb.ru
X-SA-Exim-Scanned: No (on zxy.spb.ru); SAEximRunCond expanded to false
X-BeenThere: freebsd-current@freebsd.org
X-Mailman-Version: 2.1.27
Precedence: list
List-Id: Discussions about the use of FreeBSD-current
 <freebsd-current.freebsd.org>
List-Unsubscribe: <https://lists.freebsd.org/mailman/options/freebsd-current>, 
 <mailto:freebsd-current-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-current/>
List-Post: <mailto:freebsd-current@freebsd.org>
List-Help: <mailto:freebsd-current-request@freebsd.org?subject=help>
List-Subscribe: <https://lists.freebsd.org/mailman/listinfo/freebsd-current>, 
 <mailto:freebsd-current-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Fri, 13 Jul 2018 13:19:54 -0000

On Thu, Jul 12, 2018 at 02:42:29PM +0200, Alexander Leidinger wrote:

> __curthread () at ./machine/pcpu.h:230
> 230             __asm("movq %%gs:%1,%0" : "=r" (td)
> (kgdb) bt
> #0  __curthread () at ./machine/pcpu.h:230
> #1  doadump (textdump=1) at /usr/src/sys/kern/kern_shutdown.c:366
> #2  0xffffffff80485e11 in kern_reboot (howto=260) at  
> /usr/src/sys/kern/kern_shutdown.c:446
> #3  0xffffffff804863f3 in vpanic (fmt=<optimized out>, ap=0xfffffe0000457870)
>      at /usr/src/sys/kern/kern_shutdown.c:863
> #4  0xffffffff80486443 in panic (fmt=<unavailable>) at  
> /usr/src/sys/kern/kern_shutdown.c:790
> #5  0xffffffff8075279f in trap_fatal (frame=0xfffffe0000457a50,  
> eva=32) at /usr/src/sys/amd64/amd64/trap.c:892
> #6  0xffffffff80752812 in trap_pfault (frame=0xfffffe0000457a50,  
> usermode=<optimized out>)
>      at /usr/src/sys/amd64/amd64/trap.c:728
> #7  0xffffffff80751e1a in trap (frame=0xfffffe0000457a50) at  
> /usr/src/sys/amd64/amd64/trap.c:427
> #8  <signal handler called>
> #9  0xffffffff81391fbe in arc_check_uma_cache (lowest=-1011712)
>      at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c:4532
> #10 arc_reclaim_thread (unused=<optimized out>)
>      at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c:4657
> #11 0xffffffff8044ca74 in fork_exit (callout=0xffffffff81391b90  
> <arc_reclaim_thread>, arg=0x0,
>      frame=0xfffffe0000457c00) at /usr/src/sys/kern/kern_fork.c:1057
> #12 <signal handler called>
> (kgdb) up 9
> #9  0xffffffff81391fbe in arc_check_uma_cache (lowest=-1011712)
>      at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c:4532
> 4532                    lowest +=  
> uma_zone_get_free_size(zio_data_buf_cache[n]->kc_zone);
> (kgdb) list
> 4527            int                     iter = 4;
> 4528            int                     step = 1 << (SPA_MAXBLOCKSHIFT  
> - SPA_MINBLOCKSHIFT - 3);
> 4529            int                     n = (SPA_MAXBLOCKSIZE >>  
> SPA_MINBLOCKSHIFT) - 1;
> 4530
> 4531            while (n >= 0) {
> 4532                    lowest +=  
> uma_zone_get_free_size(zio_data_buf_cache[n]->kc_zone);
> 4533                    if (lowest >= 0)
> 4534                            return lowest;
> 4535                    n -= step;
> 4536                    if(--iter == 0) {
> (kgdb) print n
> $1 = 32767
> (kgdb) print zio_data_buf_cache[n]
> $2 = (kmem_cache_t *) 0x0
> (kgdb)

Very strange, zio_data_buf_cache[] can't be NULL, as asserted in
zio_init.