Date: Fri, 31 Jul 2015 14:27:10 -0700 (PDT) From: Don Lewis <truckman@FreeBSD.org> To: kostikbel@gmail.com Cc: lev@FreeBSD.org, freebsd-fs@FreeBSD.org Subject: Re: ZFS on 10-STABLE r281159: programs, accessing ZFS pauses for minutes in state [*kmem arena] Message-ID: <201507312127.t6VLRAsE074782@gw.catspoiler.org> In-Reply-To: <20150730121840.GS2072@kib.kiev.ua>
next in thread | previous in thread | raw e-mail | index | archive | help
On 30 Jul, Konstantin Belousov wrote: > On Thu, Jul 30, 2015 at 02:30:08PM +0300, Lev Serebryakov wrote: >> Hello Freebsd-fs, >> >> >> I'm migrating my NAS from geom_raid5 + UFS to ZFS raidz. My main storage >> is 5x2Tb HDDs. Additionaly, I have 2x3Tb HDDs attached to hold my data when >> I re-make my main storage. >> >> So, I have now two ZFS pools: >> >> ztemp mirror ada0 ada1 [both are 3Tb HDDS] >> zstor raidz ada3 ada4 ada5 ada6 ada7 [all of them are 2Tb] >> >> ztemp contain one filesystem with 2.1Tb of my data. ztemp was populated >> with my data from old geom_raid5 + UFS installation via "rsync" and it was >> FAST (HDD-speed). >> >> zstor contains several empty file systems (one per user), like: >> >> zstor/home/lev >> zstor/home/sveta >> zstor/home/nsvn >> zstor/home/torrents >> zstor/home/storage >> >> Deduplication IS TURNED OFF. atime is turned off. Record size set to 1M as >> I have a lot of big files (movies, RAW photo from DSLR, etc). Compression is >> turned off. >> >> When I try to copy all my data from temporary HDDs (ztemp pool) to my new >> shiny RIAD (zstor pool) with >> >> cd /ztemp/fs && rsync -avH lev sveta nsvn storage /usr/home/ >> >> rsync pauses for tens of minutes (!) after several hundreds of files. ^T >> and top shows state "[*kmem arena]". When I stop rsync with ^C and try to do >> "zfs list" it waits forever, in state "[*kmem arena]" again. > Show the output of sysctl debug.vmem_check. > >> >> This server is equipped with 6GiB of RAM. >> >> It looks FreeBSD contains bug about year ago which leads to this behavior, >> but mailing lists says, that it was fixed in r272221, 10 months ago. I think I may have gotten bitten by this yesterday on a fairly recent 10.2-PRERELEASE machine with 8 GB of RAM. It's nominally a zfs-only machine, but I had some data on a couple of UFS drives that I needed to copy over to a zfs filesystem. I connected one of the drives to a sata to usb adapater and plugged it into the machine, then ran rsync to transfer the contents of a ~100 GB filesystem. I had a number of active programs running, including a rather bloated firefox process that had gobbled lots of ram. In my case, arc stayed small (< 1 GB), inactive memory was a couple of GB, and several GB of data got pushed to swap. Free memory got very low, bouncing around in the 10's of MB for a while before the machine locked. It wasn't totally dead because my X11 desktop is configured in focus follow mouse mode and I could see the window focus change when I moved the mouse around. Eventually I did something to provoke the window manager and/or the Xorg server into locking up as well. I wasn't able to switch to console mode. I eventually gave up and hit the reset button. %sysctl debug.vmem_check sysctl: unknown oid 'debug.vmem_check': No such file or directory With the same set of processes running, but no UFS, this is what top says about memory usage: Mem: 1156M Active, 3403M Inact, 1682M Wired, 31M Cache, 1631M Free ARC: 1129M Total, 588M MFU, 492M MRU, 54K Anon, 10M Header, 39M Other Swap: 40G Total, 40G Free
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?201507312127.t6VLRAsE074782>