Date: Tue, 29 Jan 2013 20:24:54 -0600 From: Kevin Day <toasty@dragondata.com> To: "Steven Hartland" <killing@multiplay.co.uk> Cc: FreeBSD Filesystems <freebsd-fs@freebsd.org>, Matthew Ahrens <mahrens@delphix.com> Subject: Re: Improving ZFS performance for large directories Message-ID: <A8A14F21-1D9A-4B7A-93A0-266BD8099ECF@dragondata.com> In-Reply-To: <9792709BF58143EFBDAABE638F769775@multiplay.co.uk> References: <19DB8F4A-6788-44F6-9A2C-E01DEA01BED9@dragondata.com> <CAJjvXiE%2B8OMu_yvdRAsWugH7W=fhFW7bicOLLyjEn8YrgvCwiw@mail.gmail.com> <F4420A8C-FB92-4771-B261-6C47A736CF7F@dragondata.com> <9792709BF58143EFBDAABE638F769775@multiplay.co.uk>
next in thread | previous in thread | raw e-mail | index | archive | help
On Jan 29, 2013, at 7:29 PM, "Steven Hartland" <killing@multiplay.co.uk> = wrote: >=20 > ----- Original Message ----- From: "Kevin Day" <toasty@dragondata.com> >=20 >> I think some of the issue is that nothing is being allowed to stay = cached long. We have several parallel rsyncs running at once that are = basically scanning every directory as fast as they can, combined with a = bunch of rsync, http and ftp clients. I'm guessing with all that = activity things are getting shoved out pretty quickly. >=20 > zfs send / recv a possible replacements for the rsyncs? Unfortunately not. We're pulling these files from a host that we do not = control, and isn't running ZFS. We're also serving these files up via a = public rsync daemon, and the vast majority of the clients receiving = files from it are not running ZFS either. Total data size is about 125TB now, growing to ~300TB in the near = future. It's just a ton of data that really isn't being stored in the = best manner for this kind of system, but we don't control the layout. -- Kevin
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?A8A14F21-1D9A-4B7A-93A0-266BD8099ECF>