From owner-freebsd-fs@FreeBSD.ORG Fri Oct 17 04:37:18 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 35AF5778; Fri, 17 Oct 2014 04:37:18 +0000 (UTC) Received: from mail.jrv.org (adsl-70-243-84-11.dsl.austtx.swbell.net [70.243.84.11]) by mx1.freebsd.org (Postfix) with ESMTP id E1827F00; Fri, 17 Oct 2014 04:37:17 +0000 (UTC) Received: from localhost (localhost.localdomain [127.0.0.1]) by mail.jrv.org (Postfix) with ESMTP id 299E51A3624; Thu, 16 Oct 2014 23:37:17 -0500 (CDT) Received: from mail.jrv.org ([127.0.0.1]) by localhost (zimbra64.housenet.jrv [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id 0b9MMkB1wxKV; Thu, 16 Oct 2014 23:37:07 -0500 (CDT) Received: from localhost (localhost.localdomain [127.0.0.1]) by mail.jrv.org (Postfix) with ESMTP id 2D7251A361F; Thu, 16 Oct 2014 23:37:07 -0500 (CDT) X-Virus-Scanned: amavisd-new at zimbra64.housenet.jrv Received: from mail.jrv.org ([127.0.0.1]) by localhost (zimbra64.housenet.jrv [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id jhh0VrzWEfSt; Thu, 16 Oct 2014 23:37:07 -0500 (CDT) Received: from [192.168.138.128] (BMX.housenet.jrv [192.168.3.140]) by mail.jrv.org (Postfix) with ESMTPSA id 085DE1A361C; Thu, 16 Oct 2014 23:37:07 -0500 (CDT) Message-ID: <54409CFE.8070905@jrv.org> Date: Thu, 16 Oct 2014 23:37:18 -0500 From: "James R. Van Artsdalen" User-Agent: Mozilla/5.0 (Windows NT 5.0; rv:12.0) Gecko/20120428 Thunderbird/12.0.1 MIME-Version: 1.0 To: d@delphij.net Subject: Re: zfs recv hangs in kmem arena References: <54250AE9.6070609@jrv.org> <543FAB3C.4090503@jrv.org> <543FEE6F.5050007@delphij.net> <54409050.4070401@jrv.org> <544096B3.20306@delphij.net> In-Reply-To: <544096B3.20306@delphij.net> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org, "James R. Van Artsdalen" , current@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 17 Oct 2014 04:37:18 -0000 On 10/16/2014 11:10 PM, Xin Li wrote: > On 10/16/14 8:43 PM, James R. Van Artsdalen wrote: > > On 10/16/2014 11:12 AM, Xin Li wrote: > >>> On 9/26/2014 1:42 AM, James R. Van Artsdalen wrote: > >>>> FreeBSD BLACKIE.housenet.jrv 10.1-BETA2 FreeBSD 10.1-BETA2 > >>>> #2 r272070M: Wed Sep 24 17:36:56 CDT 2014 > >>>> james@BLACKIE.housenet.jrv:/usr/obj/usr/src/sys/GENERIC > >>>> amd64 > >>>> > >>>> With current STABLE10 I am unable to replicate a ZFS pool > >>>> using zfs send/recv without zfs hanging in state "kmem > >>>> arena", within the first 4TB or so (of a 23TB Pool). > > >> What does procstat -kk 1176 (or the PID of your 'zfs' process > >> that stuck in that state) say? > >> > >> Cheers, > >> > > SUPERTEX:/root# ps -lp 866 UID PID PPID CPU PRI NI VSZ RSS > > MWCHAN STAT TT TIME COMMAND 0 866 863 0 52 0 66800 > > 29716 kmem are D+ 1 57:40.82 zfs recv -duvF BIGTOX > > SUPERTEX:/root# procstat -kk 866 PID TID COMM TDNAME > > KSTACK 866 101573 zfs - mi_switch+0xe1 > > sleepq_wait+0x3a _cv_wait+0x16d vmem_xalloc+0x568 vmem_alloc+0x3d > > kmem_malloc+0x33 keg_alloc_slab+0xcd keg_fetch_slab+0x151 > > zone_fetch_slab+0x7e zone_import+0x40 uma_zalloc_arg+0x34e > > arc_get_data_buf+0x31a arc_buf_alloc+0xaa dmu_buf_will_fill+0x169 > > dmu_write+0xfc dmu_recv_stream+0xd40 zfs_ioc_recv+0x94e > > zfsdev_ioctl+0x5ca > > Do you have any special tuning in your /boot/loader.conf? > > Cheers, > Below. I had forgotten some of this was there. After sending the previous message I ran kgdb to see if I could get a backtrace with function args. I didn't see how to do it for this proc, but during all this the process un-blocked and started running again. The process blocked again in kmem arena after a few minutes. SUPERTEX:/root# cat /boot/loader.conf zfs_load="YES" # ZFS vfs.root.mountfrom="zfs:SUPERTEX/UNIX" # Specify root partition in a way the # kernel understands kern.maxfiles="32K" # Set the sys. wide open files limit kern.ktrace.request_pool="512" #vfs.zfs.debug=1 vfs.zfs.check_hostid=0 loader_logo="beastie" # Desired logo: fbsdbw, beastiebw, beastie, none boot_verbose="YES" # -v: Causes extra debugging information to be printed geom_mirror_load="YES" # RAID1 disk driver (see gmirror(8)) geom_label_load="YES" # File system labels (see glabel(8)) ahci_load="YES" siis_load="YES" mvs_load="YES" coretemp_load="YES" # Intel Core CPU temperature monitor #console="comconsole" kern.msgbufsize="131072" # Set size of kernel message buffer kern.geom.label.gpt.enable=0 kern.geom.label.gptid.enable=0 kern.geom.label.disk_ident.enable=0 SUPERTEX:/root#