From owner-freebsd-stable@FreeBSD.ORG Mon Jul 12 09:38:26 2010 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 31B04106566C for ; Mon, 12 Jul 2010 09:38:26 +0000 (UTC) (envelope-from peterjeremy@acm.org) Received: from mail14.syd.optusnet.com.au (mail14.syd.optusnet.com.au [211.29.132.195]) by mx1.freebsd.org (Postfix) with ESMTP id B41A38FC21 for ; Mon, 12 Jul 2010 09:38:25 +0000 (UTC) Received: from server.vk2pj.dyndns.org (c211-30-160-13.belrs4.nsw.optusnet.com.au [211.30.160.13]) by mail14.syd.optusnet.com.au (8.13.1/8.13.1) with ESMTP id o6C9cKco027947 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Mon, 12 Jul 2010 19:38:21 +1000 X-Bogosity: Ham, spamicity=0.000000 Received: from server.vk2pj.dyndns.org (localhost.vk2pj.dyndns.org [127.0.0.1]) by server.vk2pj.dyndns.org (8.14.4/8.14.4) with ESMTP id o6C9cIjb027946; Mon, 12 Jul 2010 19:38:18 +1000 (EST) (envelope-from peter@server.vk2pj.dyndns.org) Received: (from peter@localhost) by server.vk2pj.dyndns.org (8.14.4/8.14.4/Submit) id o6C9cISh027945; Mon, 12 Jul 2010 19:38:18 +1000 (EST) (envelope-from peter) Date: Mon, 12 Jul 2010 19:38:18 +1000 From: Peter Jeremy To: Richard Lee Message-ID: <20100712093818.GA27693@server.vk2pj.dyndns.org> References: <20100711182511.GA21063@soda.CSUA.Berkeley.EDU> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="OgqxwSJOaUobr8KG" Content-Disposition: inline In-Reply-To: <20100711182511.GA21063@soda.CSUA.Berkeley.EDU> X-PGP-Key: http://members.optusnet.com.au/peterjeremy/pubkey.asc User-Agent: Mutt/1.5.20 (2009-06-14) Cc: freebsd-stable@freebsd.org Subject: Re: Serious zfs slowdown when mixed with another file system (ufs/msdosfs/etc.). X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 Jul 2010 09:38:26 -0000 --OgqxwSJOaUobr8KG Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On 2010-Jul-11 11:25:12 -0700, Richard Lee wrote: >But when almost all of the memory is taken by disk cache (of non-zfs >file system), ZFS disks start threshing like mad and the write >throughput goes down in 1-digit MB/second. It can go a lot lower than that... Yes, this is a known problem. The underlying problem is a disconnect between the ZFS cache (ARC) and the VM cache used by everything else, preventing ZFS reclaiming RAM from the VM cache. For several months, I was running a regular cron job that was a slightly fancier version of the perl one-liner. I have been using the attached arc.patch1 based on a patch written by Artem Belevich (see http://pastebin.com/ZCkzkWcs ) for about a month. I have had reasonable success with it (and junked my cronjob) but have managed to wedge my system a couple of times whilst doing zfs send|recv. Whilst looking at that diff, I just noticed a nasty signed/unsigned bug that could bite in low memory conditions and have revised it to arc.patch2 (untested as yet). Independently, Martin Matuska committed r209227 that corrects a number of ARC bugs reported on OpenSolaris. Whilst this patch doesn't add checks on "inactive" or "cache", some quick checks suggest it also helps (though I need to do further checks). See http://people.freebsd.org/~mm/patches/zfs/head-12636.patch --=20 Peter Jeremy --OgqxwSJOaUobr8KG Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.15 (FreeBSD) iEYEARECAAYFAkw64okACgkQ/opHv/APuIcWowCgiBc5ro8DGQfGRq4aafxxeUuu 6uUAn0IoSIvaaEfKUDncom1IrQ5NUakn =V4R+ -----END PGP SIGNATURE----- --OgqxwSJOaUobr8KG--