Date: Tue, 29 Jan 2013 20:24:54 -0600 From: Kevin Day <toasty@dragondata.com> To: "Steven Hartland" <killing@multiplay.co.uk> Cc: FreeBSD Filesystems <freebsd-fs@freebsd.org>, Matthew Ahrens <mahrens@delphix.com> Subject: Re: Improving ZFS performance for large directories Message-ID: <A8A14F21-1D9A-4B7A-93A0-266BD8099ECF@dragondata.com> In-Reply-To: <9792709BF58143EFBDAABE638F769775@multiplay.co.uk> References: <19DB8F4A-6788-44F6-9A2C-E01DEA01BED9@dragondata.com> <CAJjvXiE%2B8OMu_yvdRAsWugH7W=fhFW7bicOLLyjEn8YrgvCwiw@mail.gmail.com> <F4420A8C-FB92-4771-B261-6C47A736CF7F@dragondata.com> <9792709BF58143EFBDAABE638F769775@multiplay.co.uk>
next in thread | previous in thread | raw e-mail | index | archive | help
On Jan 29, 2013, at 7:29 PM, "Steven Hartland" <killing@multiplay.co.uk> wrote: > > ----- Original Message ----- From: "Kevin Day" <toasty@dragondata.com> > >> I think some of the issue is that nothing is being allowed to stay cached long. We have several parallel rsyncs running at once that are basically scanning every directory as fast as they can, combined with a bunch of rsync, http and ftp clients. I'm guessing with all that activity things are getting shoved out pretty quickly. > > zfs send / recv a possible replacements for the rsyncs? Unfortunately not. We're pulling these files from a host that we do not control, and isn't running ZFS. We're also serving these files up via a public rsync daemon, and the vast majority of the clients receiving files from it are not running ZFS either. Total data size is about 125TB now, growing to ~300TB in the near future. It's just a ton of data that really isn't being stored in the best manner for this kind of system, but we don't control the layout. -- Kevin
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?A8A14F21-1D9A-4B7A-93A0-266BD8099ECF>