From owner-freebsd-fs@FreeBSD.ORG Wed Feb 20 16:07:16 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 678B69F8 for ; Wed, 20 Feb 2013 16:07:16 +0000 (UTC) (envelope-from toasty@dragondata.com) Received: from mail-ia0-x22e.google.com (ia-in-x022e.1e100.net [IPv6:2607:f8b0:4001:c02::22e]) by mx1.freebsd.org (Postfix) with ESMTP id E6C0CE75 for ; Wed, 20 Feb 2013 16:07:15 +0000 (UTC) Received: by mail-ia0-f174.google.com with SMTP id u20so3104499iag.5 for ; Wed, 20 Feb 2013 08:07:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=dragondata.com; s=google; h=x-received:content-type:mime-version:subject:from:in-reply-to:date :cc:content-transfer-encoding:message-id:references:to:x-mailer; bh=di9AS21CdIBzZAWj2A8TRmDTg3dF/VnKOSDwhlacPU8=; b=phfaBN8o5G261w5EbNaWYFV2Z4P5c0ol0meJbvSO17qrsNZIH/CfK+b9IvdLFcrFnE 14lYZucXNYbX0wBlLm5DdGSCra1fdh6HMxE5Y3gefix72VHORWYQf3wc/xYAGmIxb4bJ Dc8j1/xEXMNptMeRR7X7nYDcFVvzhkOs5cM+I= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=x-received:content-type:mime-version:subject:from:in-reply-to:date :cc:content-transfer-encoding:message-id:references:to:x-mailer :x-gm-message-state; bh=di9AS21CdIBzZAWj2A8TRmDTg3dF/VnKOSDwhlacPU8=; b=k3+bfI6lXqXAY/GFPBKP/FtYGS7hBg2PV2ncjW2aOapRvQ01IlF5kW5bm3XsX/nhuj jZ9mG0P6fO66tRKJp7Sf0SDO0WJFP/BrxtcW1E/AMvY3LKiLHSaJosuxCfAYG1kxgMq5 /xcRvh1BBaFI+ZMv3aufPYwsxgd+k/SCqI69AM1ipYE8PFSFEhj28Il5G/O4OeGpLM+5 EPlg7gjw1LI/GwTQxuAB7cu9JYiFszn+Q53drXgLS/c0z2RSXnm7OgrmQRcmsWoEpodS 8y6RsNcu9KGuL+eNufDnuXbR79xv9B0xUNJcu+R8HYBkoHkWRBc1aQPi1ppwIalm336f oa/Q== X-Received: by 10.42.95.146 with SMTP id f18mr9641819icn.9.1361376435436; Wed, 20 Feb 2013 08:07:15 -0800 (PST) Received: from vpn132.rw1.your.org (vpn132.rw1.your.org. [204.9.51.132]) by mx.google.com with ESMTPS id s8sm766074igs.0.2013.02.20.08.07.13 (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Wed, 20 Feb 2013 08:07:14 -0800 (PST) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 6.2 \(1499\)) Subject: Re: Improving ZFS performance for large directories From: Kevin Day In-Reply-To: <20130220082828.GA44920@server.rulingia.com> Date: Wed, 20 Feb 2013 10:07:11 -0600 Content-Transfer-Encoding: quoted-printable Message-Id: <2F90562A-7F98-49A5-8431-4313961EFA70@dragondata.com> References: <19DB8F4A-6788-44F6-9A2C-E01DEA01BED9@dragondata.com> <20130201192416.GA76461@server.rulingia.com> <19E0C908-79F1-43F8-899C-6B60F998D4A5@dragondata.com> <20130220082828.GA44920@server.rulingia.com> To: Peter Jeremy X-Mailer: Apple Mail (2.1499) X-Gm-Message-State: ALoCoQl1AObRP2c0/VpjQFroRFEvU3zm14dOacSgmElQHoY5m5tRgz5g59GyZkUDONvGkQ/Oxge2 Cc: FreeBSD Filesystems X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 20 Feb 2013 16:07:16 -0000 On Feb 20, 2013, at 2:28 AM, Peter Jeremy wrote: >> Thinking I'd make the primary cache metadata only, and the secondary >> cache "all" would improve things, >=20 > This won't work as expected. L2ARC only caches data coming out of ARC > so by setting ARC to cache metadata only, there's never any "data" in > ARC and hence never any evicted from ARC to L2ARC. >=20 That makes sense, I wasn't sure if it was smart enough to realize this = happening or not, but I guess it won't work. >> I wiped the device (SATA secure erase to make sure) >=20 > That's not necessary. L2ARC doesn't survive reboots because all teh > L2ARC "metadata" is in ARC only. This does mean that it takes quite > a while for L2ARC to warm up following a reboot. >=20 I was more concerned with the SSD's performance than ZFS caring what was = there. A few cases completely filled the SSD, which can slow things down = (there are no free blocks for it to use). Secure Erase will reset it so = the drive's controller knows EVERYTHING is really free. We have one = model of SSD here that will drop to about 5% of it's original = performance after every block on the drive has been written to once. = We're not using that model anymore, but I still like to be sure. :) >> There are roughly 29M files, growing at about 50k files/day. We >> recently upgraded, and are now at 96 3TB drives in the pool.=20 >=20 > That number of files isn't really excessive but it sounds like your > workload has very low locality. At this stage, my suggestions are: > 1) Disable atime if you don't need it & haven't already. > Otherwise file accesses are triggering metadata updates. > 2) Increase vfs.zfs.arc_meta_limit > You're still getting more metadata misses than data misses > 3) Increase your ARC size (more RAM) > Your pool is quite large compared to your RAM. >=20 Yeah, I think the locality is basically zero. It's multiple rsyncs = running across the entire filesystem repeatedly. Each directory is only = going to be touched once per pass through, so that isn't really going to = benefit much from cache unless we get lucky and two rsyncs come in = back-to-back where one is chasing another. Atime is already off globally - nothing we use needs it. We are at the = limit for RAM for this motherboard, so any further increases are going = to be quite expensive.=20 >=20 >> Is there any way to tell why more metadata isn't >> being pushed to the L2ARC? >=20 > ZFS treats writing to L2ARC very much as an afterthought. L2ARC = writes > are rate limited by vfs.zfs.l2arc_write_{boost,max} and will be = aborted > if they might interfere with a read. I'm not sure how to improve it. >=20 At this stage there are just zero writes being done, so perhaps the = problem is that with so much pressure on the arc metadata, nothing is = getting a chance to get pushed into the L2ARC. I'm going to try to = increase the meta limit on ARC, but there's not a great deal more I can = do. > Since this is all generic ZFS, you might like to try asking on > zfs@lists.illumos.org as well. Some of the experts there might have > some ideas. I will try that, thanks! -- Kevin