From owner-freebsd-fs@FreeBSD.ORG Tue Oct 21 22:05:33 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 5D176454; Tue, 21 Oct 2014 22:05:33 +0000 (UTC) Received: from mail-ig0-x22a.google.com (mail-ig0-x22a.google.com [IPv6:2607:f8b0:4001:c05::22a]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 1E40B696; Tue, 21 Oct 2014 22:05:33 +0000 (UTC) Received: by mail-ig0-f170.google.com with SMTP id hn18so352950igb.5 for ; Tue, 21 Oct 2014 15:05:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:reply-to:in-reply-to:references:date:message-id :subject:from:to:cc:content-type; bh=HJdcIZnAuMAErzbYQdm/LSR4bxYIZDbj3joerIZoUsg=; b=O6v0OEiHc3g5dnvy9vFyiPSJXkb/emj3AO7ZDVgFU8LyZAbmD38w6lopvmjYOHEhbB d0AVoSeU9JFdZ7eA9rqEtbX3Anh2gzcKFRFyPSqDLBkiD5LXPqB8hcvCFmnnUnBurY/c 0Ld7D2c2ok4ZEfyWlQkp1CMuGiwoSIF2Wf2TAbo9F+HrTDX6xnzHGTXB3yi3aiGCyd0a zYBmdSoXDIeZThtQFWZ3ticwWYK6Rx4OZnq+8d1x+JZBKsJxZT2OZBRoSOIdt1cAIOLy JOipnJZLwAN2RpkWmWP6C9sQneEzd13AXf5CuCwivI+NwsSk5jufIWzDv8YT9xfG1iD8 Etrg== MIME-Version: 1.0 X-Received: by 10.107.165.76 with SMTP id o73mr40511143ioe.1.1413929132428; Tue, 21 Oct 2014 15:05:32 -0700 (PDT) Received: by 10.42.223.69 with HTTP; Tue, 21 Oct 2014 15:05:32 -0700 (PDT) Reply-To: alc@freebsd.org In-Reply-To: References: <54250AE9.6070609@jrv.org> <543FAB3C.4090503@jrv.org> <543FEE6F.5050007@delphij.net> <54409050.4070401@jrv.org> <544096B3.20306@delphij.net> <54409CFE.8070905@jrv.org> <5443D918.9090307@jrv.org> Date: Tue, 21 Oct 2014 17:05:32 -0500 Message-ID: Subject: Re: zfs recv hangs in kmem arena From: Alan Cox To: "James R. Van Artsdalen" Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 Cc: freebsd-fs@freebsd.org, d@delphij.net, "current@freebsd.org" X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 21 Oct 2014 22:05:33 -0000 On Tue, Oct 21, 2014 at 5:03 PM, Alan Cox wrote: > On Sun, Oct 19, 2014 at 10:30 AM, James R. Van Artsdalen < > james-freebsd-fs2@jrv.org> wrote: > >> Removing kern.maxfiles from loader.conf still hangs in "kmem arena". >> >> I tried using a memstick image of -CURRENT made from the release/ >> process and this also hangs in "kmem arena" >> >> An uninvolved server of mine hung Friday night in state"kmem arena" >> during periodic's "zpool history". After a reboot it did not hang >> Saturday night. >> >> > > > How up to date is your source tree? r2720221 is relevant. Without that > change, there are circumstances in which the code that is supposed to free > space from the kmem arena doesn't get called. > > That should be r272071. > > > >> On 10/16/2014 11:37 PM, James R. Van Artsdalen wrote: >> > On 10/16/2014 11:10 PM, Xin Li wrote: >> >> On 10/16/14 8:43 PM, James R. Van Artsdalen wrote: >> >>> On 10/16/2014 11:12 AM, Xin Li wrote: >> >>>>> On 9/26/2014 1:42 AM, James R. Van Artsdalen wrote: >> >>>>>> FreeBSD BLACKIE.housenet.jrv 10.1-BETA2 FreeBSD 10.1-BETA2 >> >>>>>> #2 r272070M: Wed Sep 24 17:36:56 CDT 2014 >> >>>>>> james@BLACKIE.housenet.jrv:/usr/obj/usr/src/sys/GENERIC >> >>>>>> amd64 >> >>>>>> >> >>>>>> With current STABLE10 I am unable to replicate a ZFS pool >> >>>>>> using zfs send/recv without zfs hanging in state "kmem >> >>>>>> arena", within the first 4TB or so (of a 23TB Pool). >> >>>> What does procstat -kk 1176 (or the PID of your 'zfs' process >> >>>> that stuck in that state) say? >> >>>> >> >>>> Cheers, >> >>>> >> >>> SUPERTEX:/root# ps -lp 866 UID PID PPID CPU PRI NI VSZ RSS >> >>> MWCHAN STAT TT TIME COMMAND 0 866 863 0 52 0 66800 >> >>> 29716 kmem are D+ 1 57:40.82 zfs recv -duvF BIGTOX >> >>> SUPERTEX:/root# procstat -kk 866 PID TID COMM TDNAME >> >>> KSTACK 866 101573 zfs - mi_switch+0xe1 >> >>> sleepq_wait+0x3a _cv_wait+0x16d vmem_xalloc+0x568 vmem_alloc+0x3d >> >>> kmem_malloc+0x33 keg_alloc_slab+0xcd keg_fetch_slab+0x151 >> >>> zone_fetch_slab+0x7e zone_import+0x40 uma_zalloc_arg+0x34e >> >>> arc_get_data_buf+0x31a arc_buf_alloc+0xaa dmu_buf_will_fill+0x169 >> >>> dmu_write+0xfc dmu_recv_stream+0xd40 zfs_ioc_recv+0x94e >> >>> zfsdev_ioctl+0x5ca >> >> Do you have any special tuning in your /boot/loader.conf? >> >> >> >> Cheers, >> >> >> > Below. I had forgotten some of this was there. >> > >> > After sending the previous message I ran kgdb to see if I could get a >> > backtrace with function args. I didn't see how to do it for this proc, >> > but during all this the process un-blocked and started running again. >> > >> > The process blocked again in kmem arena after a few minutes. >> > >> > >> > SUPERTEX:/root# cat /boot/loader.conf >> > zfs_load="YES" # ZFS >> > vfs.root.mountfrom="zfs:SUPERTEX/UNIX" # Specify root partition >> > in a way the >> > # kernel understands >> > kern.maxfiles="32K" # Set the sys. wide open files limit >> > kern.ktrace.request_pool="512" >> > #vfs.zfs.debug=1 >> > vfs.zfs.check_hostid=0 >> > >> > loader_logo="beastie" # Desired logo: fbsdbw, beastiebw, beastie, >> > none >> > boot_verbose="YES" # -v: Causes extra debugging information to be >> > printed >> > geom_mirror_load="YES" # RAID1 disk driver (see gmirror(8)) >> > geom_label_load="YES" # File system labels (see glabel(8)) >> > ahci_load="YES" >> > siis_load="YES" >> > mvs_load="YES" >> > coretemp_load="YES" # Intel Core CPU temperature monitor >> > #console="comconsole" >> > kern.msgbufsize="131072" # Set size of kernel message buffer >> > >> > kern.geom.label.gpt.enable=0 >> > kern.geom.label.gptid.enable=0 >> > kern.geom.label.disk_ident.enable=0 >> > SUPERTEX:/root# >> > >> >> _______________________________________________ >> freebsd-current@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-current >> To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org >> " >> > >