Date: Wed, 6 Mar 2013 01:08:28 +0200 From: Daniel Kalchev <daniel@digsys.bg> To: Freddie Cash <fjwcash@gmail.com> Cc: Garrett Wollman <wollman@hergotha.csail.mit.edu>, stable@freebsd.org, Steven Hartland <killing@multiplay.co.uk> Subject: Re: ZFS "stalls" -- and maybe we should be talking about defaults? Message-ID: <22720B5D-7BCA-41BC-B1E8-A2ACB2ADB795@digsys.bg> In-Reply-To: <CAOjFWZ7TDYgS=vvcuLRPr%2BfAv4bPk4ZU0CKbgf1BydxG1MeOYw@mail.gmail.com> References: <513524B2.6020600@denninger.net> <1362449266.92708.8.camel@btw.pki2.com> <51355F64.4040409@denninger.net> <201303050540.r255ecEC083742@hergotha.csail.mit.edu> <20130305152252.GA52706@in-addr.com> <CAOjFWZ7TDYgS=vvcuLRPr%2BfAv4bPk4ZU0CKbgf1BydxG1MeOYw@mail.gmail.com>
index | next in thread | previous in thread | raw e-mail
On Mar 5, 2013, at 8:17 PM, Freddie Cash <fjwcash@gmail.com> wrote: > > ZFS send/recv would eventually complete, but what used to take 15-20 > minutes would take 6-8 hours to complete. > > I've reduced the ARC to only 32 GB, with arc_meta set to 28 GB, and things > are running much smoother now (50-200 MB/s writes for 3-5 seconds every > 10s), and send/recv is back down to 10-15 minutes. > > Who would have thought "too much RAM" would be an issue? > > Will play with this over the next couple of days with different ARC max > settings to see where the problems start. All of our ZFS boxes until this > one had under 64 GB of RAM. (And we had issues with dedupe enabled on > boxes with too little RAM, as in under 32 GB.) I have an archive box running very similar setup as yours, but with 72GB of RAM. I have set both arc_max and arc_meta_limit to 64GB, with no issues. I am still doing a very complex snapshot reordering between two pools. One of the pools has dedup enabled (which prompted me to add RAM), with dedup ratio f over 10x and there are still no issues or any stalling. The other pool has both dedup and compression for some filesystems. My only issue is that replacing a drive in either pool takes few days (6-drive vdevs of 3TB drives). Perhaps the memory indexing/search algorithms are inefficient? Danielhome | help
Want to link to this message? Use this
URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?22720B5D-7BCA-41BC-B1E8-A2ACB2ADB795>