From owner-freebsd-fs@freebsd.org Mon Nov 30 12:49:40 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 57965A24824 for ; Mon, 30 Nov 2015 12:49:40 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: from mail-wm0-x236.google.com (mail-wm0-x236.google.com [IPv6:2a00:1450:400c:c09::236]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id E412B1C29 for ; Mon, 30 Nov 2015 12:49:39 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: by wmww144 with SMTP id w144so136090478wmw.0 for ; Mon, 30 Nov 2015 04:49:37 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=multiplay-co-uk.20150623.gappssmtp.com; s=20150623; h=subject:to:references:from:message-id:date:user-agent:mime-version :in-reply-to:content-type:content-transfer-encoding; bh=A+UV68Vs1i+v74eg3SCCGmFu7l9fxdlHXLcmUxztOvI=; b=R6r/9rrT6xEcogw5PVagS200FkZtvucZ+gfQoD3QT4yximKE388g0qUwV3eB5s3nBX ERKLApX1ryr91LBtpS2Z7cShTR8lEBOyPiJi3zKYsl2EqUuzGGdtwy5TOv4xJZrf+y32 wc/NMK4CxjRmLvMd+tGYSyX1di0P28OKG5D6Zoig0Obfe8sf8wpg33iNUABSSrr+4aTp 3cYUimnmHVaKx8YxZBj3MlrZXk1SbJtNLneUTV0aZia9RJTsGI2vg09/hs3C+jn6trXK BqDYWHEtwFmROTSD4dLhLHSgL2JfX6kgf1h2YKF53f+ID+GGAPsuKw7GoWLD1q1XNtTf 4sgg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:subject:to:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-type :content-transfer-encoding; bh=A+UV68Vs1i+v74eg3SCCGmFu7l9fxdlHXLcmUxztOvI=; b=YBHqgRZjlV4G33w6/ok3JyFf7U1pHtpWRzeRgYBYSLUogPxb9X9FIQj1gxGKHMP2Sb rYo8/Rpq9pCkeZ8pqagTRw0RfgGupcWaRXP6n9hi6Hle323MGDqfok4EySZY9wjuKJbZ 28NsrriTpscaxTUKEUsuJfi9qbURsPaL/fh4WmzJGHNbr2aQ7pKZDb9paMNkzHBXxYJa sz4HT44FyMqMbT31nSqHErGHMdtDgJGwCpI1xJyCY4IT5JIiXspqGUv2yYqVjXLkBqhF dVSBKn4NKOStiKr7XE8s8jckZxgZxKR7E34LYW65vj695MCoYiVcAXJZe+NPdZUq84mX xMsg== X-Gm-Message-State: ALoCoQmBGkHgtajWK9W3iJLejUpMSnB71HO//qPEinWj94c0fNCqHebkeHpxzhsWFoxqODaFLFSn X-Received: by 10.28.96.4 with SMTP id u4mr29748520wmb.52.1448887777283; Mon, 30 Nov 2015 04:49:37 -0800 (PST) Received: from [10.10.1.58] (liv3d.labs.multiplay.co.uk. [82.69.141.171]) by smtp.gmail.com with ESMTPSA id wh10sm46639852wjb.45.2015.11.30.04.49.36 for (version=TLSv1/SSLv3 cipher=OTHER); Mon, 30 Nov 2015 04:49:36 -0800 (PST) Subject: Re: ZFS on 10-STABLE r281159: programs, accessing ZFS pauses for minutes in state [*kmem arena] To: freebsd-fs@freebsd.org References: <201507312127.t6VLRAsE074782@gw.catspoiler.org> From: Steven Hartland Message-ID: <565C45DE.8060205@multiplay.co.uk> Date: Mon, 30 Nov 2015 12:49:34 +0000 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.4.0 MIME-Version: 1.0 In-Reply-To: <201507312127.t6VLRAsE074782@gw.catspoiler.org> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 30 Nov 2015 12:49:40 -0000 On 31/07/2015 22:27, Don Lewis wrote: > On 30 Jul, Konstantin Belousov wrote: >> On Thu, Jul 30, 2015 at 02:30:08PM +0300, Lev Serebryakov wrote: >>> Hello Freebsd-fs, >>> >>> >>> I'm migrating my NAS from geom_raid5 + UFS to ZFS raidz. My main storage >>> is 5x2Tb HDDs. Additionaly, I have 2x3Tb HDDs attached to hold my data when >>> I re-make my main storage. >>> >>> So, I have now two ZFS pools: >>> >>> ztemp mirror ada0 ada1 [both are 3Tb HDDS] >>> zstor raidz ada3 ada4 ada5 ada6 ada7 [all of them are 2Tb] >>> >>> ztemp contain one filesystem with 2.1Tb of my data. ztemp was populated >>> with my data from old geom_raid5 + UFS installation via "rsync" and it was >>> FAST (HDD-speed). >>> >>> zstor contains several empty file systems (one per user), like: >>> >>> zstor/home/lev >>> zstor/home/sveta >>> zstor/home/nsvn >>> zstor/home/torrents >>> zstor/home/storage >>> >>> Deduplication IS TURNED OFF. atime is turned off. Record size set to 1M as >>> I have a lot of big files (movies, RAW photo from DSLR, etc). Compression is >>> turned off. >>> >>> When I try to copy all my data from temporary HDDs (ztemp pool) to my new >>> shiny RIAD (zstor pool) with >>> >>> cd /ztemp/fs && rsync -avH lev sveta nsvn storage /usr/home/ >>> >>> rsync pauses for tens of minutes (!) after several hundreds of files. ^T >>> and top shows state "[*kmem arena]". When I stop rsync with ^C and try to do >>> "zfs list" it waits forever, in state "[*kmem arena]" again. >> Show the output of sysctl debug.vmem_check. >> >>> This server is equipped with 6GiB of RAM. >>> >>> It looks FreeBSD contains bug about year ago which leads to this behavior, >>> but mailing lists says, that it was fixed in r272221, 10 months ago. > I think I may have gotten bitten by this yesterday on a fairly recent > 10.2-PRERELEASE machine with 8 GB of RAM. It's nominally a zfs-only > machine, but I had some data on a couple of UFS drives that I needed to > copy over to a zfs filesystem. I connected one of the drives to a sata > to usb adapater and plugged it into the machine, then ran rsync to > transfer the contents of a ~100 GB filesystem. I had a number of active > programs running, including a rather bloated firefox process that had > gobbled lots of ram. In my case, arc stayed small (< 1 GB), inactive > memory was a couple of GB, and several GB of data got pushed to swap. > Free memory got very low, bouncing around in the 10's of MB for a while > before the machine locked. It wasn't totally dead because my X11 > desktop is configured in focus follow mouse mode and I could see the > window focus change when I moved the mouse around. Eventually I did > something to provoke the window manager and/or the Xorg server into > locking up as well. I wasn't able to switch to console mode. I > eventually gave up and hit the reset button. > > %sysctl debug.vmem_check > sysctl: unknown oid 'debug.vmem_check': No such file or directory > > With the same set of processes running, but no UFS, this is what top > says about memory usage: > > Mem: 1156M Active, 3403M Inact, 1682M Wired, 31M Cache, 1631M Free > ARC: 1129M Total, 588M MFU, 492M MRU, 54K Anon, 10M Header, 39M Other > Swap: 40G Total, 40G Free > This is a little late but just wanted to note for the record that the combination of r281026, r281108, r281109 (MFC10: r282361) and r282690 (MFC10: r283310) should address this issue, as noted on the PR: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=194513 If anyone still experiences this please let us know on the PR. Regards Steve