From owner-freebsd-fs@FreeBSD.ORG Wed Jan 30 02:24:58 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id E10623DF for ; Wed, 30 Jan 2013 02:24:58 +0000 (UTC) (envelope-from toasty@dragondata.com) Received: from mail-ie0-x235.google.com (mail-ie0-x235.google.com [IPv6:2607:f8b0:4001:c03::235]) by mx1.freebsd.org (Postfix) with ESMTP id 737257C3 for ; Wed, 30 Jan 2013 02:24:58 +0000 (UTC) Received: by mail-ie0-f181.google.com with SMTP id 17so919412iea.12 for ; Tue, 29 Jan 2013 18:24:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=dragondata.com; s=google; h=x-received:subject:mime-version:content-type:from:x-priority :in-reply-to:date:cc:content-transfer-encoding:message-id:references :to:x-mailer; bh=8Zzcq+JZK8DfDrybaLFcueH81JvX4Y2StmRgnRHs5C4=; b=YG6yuEdOTGr0886IDC2CRkM79p8JwMoNcX5wHp2qNdipOapVWvmCxaaSTXcQPbweIu 62CmrPOFr+riuPrmEA3btybecCpdWuHqLiF4U4BFD6krE+aHmAkpQZQOoHkuTpRKS1HK i3gnJxaBgNGagFa/uAuuuGIwB91GKzydBSJhE= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=x-received:subject:mime-version:content-type:from:x-priority :in-reply-to:date:cc:content-transfer-encoding:message-id:references :to:x-mailer:x-gm-message-state; bh=8Zzcq+JZK8DfDrybaLFcueH81JvX4Y2StmRgnRHs5C4=; b=EuCe7QedueA8qTP/b5u1UencayIhMgGavcLPECP6/3IY8azBUf9cpF9oUwwsYH2ump OWRJzQ1YdX7ufpY/kCjGiaQAyaxTBTvg5go6UyzeoFMXfB/WnbRJ0u6cyCXth84x85AD 6PqN8/LTwlh+x7fs+WyvxaMczHNMP3gysZPHcXRbnNSaon3vemaQcp0hI40DbVzxWqve zW/fHI10q8MrQdY3hCgrknbQoEUcxwZv8DaYCOGjFkJc7WR2Iy64YPHn6DINbUOAWhLj /+WgqPmI332EWEaN5TXWNBpQDnOomJnZi6m9/GNpiqCQD1vKAFHtajKR/pMKAlEnuhds PJmA== X-Received: by 10.42.27.74 with SMTP id i10mr2064818icc.47.1359512698139; Tue, 29 Jan 2013 18:24:58 -0800 (PST) Received: from vpn132.rw1.your.org (vpn132.rw1.your.org. [204.9.51.132]) by mx.google.com with ESMTPS id bg10sm3322632igc.6.2013.01.29.18.24.55 (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Tue, 29 Jan 2013 18:24:57 -0800 (PST) Subject: Re: Improving ZFS performance for large directories Mime-Version: 1.0 (Mac OS X Mail 6.2 \(1499\)) Content-Type: text/plain; charset=iso-8859-1 From: Kevin Day X-Priority: 3 In-Reply-To: <9792709BF58143EFBDAABE638F769775@multiplay.co.uk> Date: Tue, 29 Jan 2013 20:24:54 -0600 Content-Transfer-Encoding: quoted-printable Message-Id: References: <19DB8F4A-6788-44F6-9A2C-E01DEA01BED9@dragondata.com> <9792709BF58143EFBDAABE638F769775@multiplay.co.uk> To: "Steven Hartland" X-Mailer: Apple Mail (2.1499) X-Gm-Message-State: ALoCoQmoEPhlFq2Ko0MEwkfXrVoWjeJSF3zKexdL1H5iWwXSQ6a8mS6EPsPU9xyPK4jSzM1dVNW8 Cc: FreeBSD Filesystems , Matthew Ahrens X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 30 Jan 2013 02:24:58 -0000 On Jan 29, 2013, at 7:29 PM, "Steven Hartland" = wrote: >=20 > ----- Original Message ----- From: "Kevin Day" >=20 >> I think some of the issue is that nothing is being allowed to stay = cached long. We have several parallel rsyncs running at once that are = basically scanning every directory as fast as they can, combined with a = bunch of rsync, http and ftp clients. I'm guessing with all that = activity things are getting shoved out pretty quickly. >=20 > zfs send / recv a possible replacements for the rsyncs? Unfortunately not. We're pulling these files from a host that we do not = control, and isn't running ZFS. We're also serving these files up via a = public rsync daemon, and the vast majority of the clients receiving = files from it are not running ZFS either. Total data size is about 125TB now, growing to ~300TB in the near = future. It's just a ton of data that really isn't being stored in the = best manner for this kind of system, but we don't control the layout. -- Kevin