From owner-freebsd-fs@FreeBSD.ORG Wed Sep 15 15:15:56 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A69C8106564A; Wed, 15 Sep 2010 15:15:56 +0000 (UTC) (envelope-from avg@freebsd.org) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id 986AE8FC16; Wed, 15 Sep 2010 15:15:55 +0000 (UTC) Received: from odyssey.starpoint.kiev.ua (alpha-e.starpoint.kiev.ua [212.40.38.101]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id SAA11033; Wed, 15 Sep 2010 18:15:52 +0300 (EEST) (envelope-from avg@freebsd.org) Message-ID: <4C90E328.20606@freebsd.org> Date: Wed, 15 Sep 2010 18:15:52 +0300 From: Andriy Gapon User-Agent: Mozilla/5.0 (X11; U; FreeBSD amd64; en-US; rv:1.9.2.9) Gecko/20100909 Lightning/1.0b2 Thunderbird/3.1.3 MIME-Version: 1.0 To: Steven Hartland References: <5DB6E7C798E44D33A05673F4B773405E@multiplay.co.uk><4C85E91E.1010602@icyb.net.ua><4C873914.40404@freebsd.org><20100908084855.GF2465@deviant.kiev.zoral.com.ua><4C874F00.3050605@freebsd.org><4C8D087B.5040404@freebsd.org><03537796FAB54E02959E2D64FC83004F@multiplay.co.uk><4C8D280F.3040803@freebsd.org><3FBF66BF11AA4CBBA6124CA435A4A31B@multiplay.co.uk><4C8E4212.30000@freebsd.org> <4C90B4C8.90203@freebsd.org> <6DFACB27CA8A4A22898BC81E55C4FD36@multiplay.co.uk> <4C90D3A1.7030008@freebsd.org> <0B1A90A08DFE4ADA9540F9F3846FDF38@multiplay.co.uk> In-Reply-To: <0B1A90A08DFE4ADA9540F9F3846FDF38@multiplay.co.uk> X-Enigmail-Version: 1.1.2 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org, jhell , Pawel Jakub Dawidek Subject: Re: zfs very poor performance compared to ufs due to lack of cache? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 15 Sep 2010 15:15:56 -0000 on 15/09/2010 18:04 Steven Hartland said the following: > ----- Original Message ----- From: "Andriy Gapon" > >>> Indeed. Where would this need to be addressed as ufs doesn't suffer from this? >> >> In ZFS. But I don't think that this is going to happen any time soon if at all. >> Authors of ZFS specifically chose to use a dedicated cache, which is ARC. >> Talk to them, or don't use ZFS, or get used to it. >> ARC has a price, but it supposedly has benefits too. >> Changing ZFS to use buffer cache is a lot of work and effectively means not using >> ARC, IMO. > > Hmm, so taking a different track on the issue is the a way to make sendfile use data > directly from ARC instead of having to copy it first? Well, theoretically everything is possible, but I am not sure if it's feasible. It's a lot of work anyways, it should be a very specialized sendfile and a lot if inter-layer knowledge and dependencies. Don't hold your breath for it. >> Well, I thought that you hurried when you applied the patches and changed the >> settings at the same time. This made it impossible for you to judge properly what >> patches do and don't do for you. > > No hurry just applying the patches that where suggested, retest, apply new retest > etc but > in parrallel been reading up on the arc tunables. > >>> Now we have a very simple setup so we can make sensible values for min / max but >>> it still means that for every file being sent when sendfile is enabled: >>> 1. There are two copies in memory which is still going to mean that only half the >>> amount files can be successfully cached and served without resorting to disk IO. >> >> Can't really say, depends on the size of the files. >> Though, it's approximately a half of what could have fit in memory with e.g. >> UFS, yes. > > Out of interest if a copy of the data is being made from ARC whats ties those > two copies together, in order to prevent the next request for the same file having to > create a third copy etc... Read about FreeBSD VM, particularly about a notion of VM object. http://www.informit.com/store/product.aspx?isbn=0201702452 >>> 2. sendfile isn't achieving what it states it should be i.e. a zero-copy. Does >>> this explain >>> the other odd behaviour we noticed, high CPU usage from nginx? >> >> sendfile should achieve zero copy with all the patches applied once both copies of >> data are settled in memory. If you have insufficient memory to hold the workset, >> then that's a different issue of moving competing data in and out of memory. And >> that may explain the CPU load, but it's just a speculation. > > Yes, more investigation needed ;-) > >> At present I don't see any other way but brute force - throw even more RAM at the >> problem. >> >> Perhaps, a miracle would happen and someone would post patches that radically >> change ZFS behavior with respect to caches. But I don't expect it >> (pessimist/realist). > > Or alternatively make sendfile work directly from ARC, would that be possible? I'll be glad to review the patches :-) > Thanks for all the info :) You are welcome. You did a lot of testing and investigative work here and I hope this thread will be useful for other people researching the topic. -- Andriy Gapon