Date: Wed, 15 Sep 2010 16:04:42 +0100 From: "Steven Hartland" <killing@multiplay.co.uk> To: "Andriy Gapon" <avg@freebsd.org> Cc: freebsd-fs@freebsd.org, jhell <jhell@DataIX.net>, Pawel Jakub Dawidek <pjd@freebsd.org> Subject: Re: zfs very poor performance compared to ufs due to lack of cache? Message-ID: <0B1A90A08DFE4ADA9540F9F3846FDF38@multiplay.co.uk> References: <5DB6E7C798E44D33A05673F4B773405E@multiplay.co.uk><AANLkTikNhsj5myhQCoPaNytUbpHtox1vg9AZm1N-OcMO@mail.gmail.com><4C85E91E.1010602@icyb.net.ua><4C873914.40404@freebsd.org><20100908084855.GF2465@deviant.kiev.zoral.com.ua><4C874F00.3050605@freebsd.org><A6D7E134B24F42E395C30A375A6B50AF@multiplay.co.uk><4C8D087B.5040404@freebsd.org><03537796FAB54E02959E2D64FC83004F@multiplay.co.uk><4C8D280F.3040803@freebsd.org><3FBF66BF11AA4CBBA6124CA435A4A31B@multiplay.co.uk><4C8E4212.30000@freebsd.org> <B98EBECBD399417CA5390C20627384B1@multiplay.co.uk> <D79F15FEB5794315BD8668E40B414BF0@multiplay.co.uk> <4C90B4C8.90203@freebsd.org> <6DFACB27CA8A4A22898BC81E55C4FD36@multiplay.co.uk> <4C90D3A1.7030008@freebsd.org>
next in thread | previous in thread | raw e-mail | index | archive | help
----- Original Message -----
From: "Andriy Gapon" <avg@freebsd.org>
>> Indeed. Where would this need to be addressed as ufs doesn't suffer from this?
>
> In ZFS. But I don't think that this is going to happen any time soon if at all.
> Authors of ZFS specifically chose to use a dedicated cache, which is ARC.
> Talk to them, or don't use ZFS, or get used to it.
> ARC has a price, but it supposedly has benefits too.
> Changing ZFS to use buffer cache is a lot of work and effectively means not using
> ARC, IMO.
Hmm, so taking a different track on the issue is the a way to make sendfile use data
directly from ARC instead of having to copy it first?
> Well, I thought that you hurried when you applied the patches and changed the
> settings at the same time. This made it impossible for you to judge properly what
> patches do and don't do for you.
No hurry just applying the patches that where suggested, retest, apply new retest etc but
in parrallel been reading up on the arc tunables.
>> Now we have a very simple setup so we can make sensible values for min / max but
>> it still means that for every file being sent when sendfile is enabled:
>> 1. There are two copies in memory which is still going to mean that only half the
>> amount files can be successfully cached and served without resorting to disk IO.
>
> Can't really say, depends on the size of the files.
> Though, it's approximately a half of what could have fit in memory with e.g. UFS, yes.
Out of interest if a copy of the data is being made from ARC whats ties those
two copies together, in order to prevent the next request for the same file having to
create a third copy etc...
>> 2. sendfile isn't achieving what it states it should be i.e. a zero-copy. Does
>> this explain
>> the other odd behaviour we noticed, high CPU usage from nginx?
>
> sendfile should achieve zero copy with all the patches applied once both copies of
> data are settled in memory. If you have insufficient memory to hold the workset,
> then that's a different issue of moving competing data in and out of memory. And
> that may explain the CPU load, but it's just a speculation.
Yes, more investigation needed ;-)
> At present I don't see any other way but brute force - throw even more RAM at the
> problem.
>
> Perhaps, a miracle would happen and someone would post patches that radically
> change ZFS behavior with respect to caches. But I don't expect it
> (pessimist/realist).
Or alternatively make sendfile work directly from ARC, would that be possible?
Thanks for all the info :)
Regards
Steve
================================================
This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it.
In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337
or return the E.mail to postmaster@multiplay.co.uk.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?0B1A90A08DFE4ADA9540F9F3846FDF38>