From owner-freebsd-fs@FreeBSD.ORG Mon Sep 6 19:07:46 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7C4B710656B6 for ; Mon, 6 Sep 2010 19:07:46 +0000 (UTC) (envelope-from bsd@vink.pl) Received: from mail-qw0-f54.google.com (mail-qw0-f54.google.com [209.85.216.54]) by mx1.freebsd.org (Postfix) with ESMTP id 325938FC15 for ; Mon, 6 Sep 2010 19:07:46 +0000 (UTC) Received: by qwg5 with SMTP id 5so4423531qwg.13 for ; Mon, 06 Sep 2010 12:07:45 -0700 (PDT) Received: by 10.224.11.20 with SMTP id r20mr200581qar.223.1283800065337; Mon, 06 Sep 2010 12:07:45 -0700 (PDT) Received: from mail-qy0-f182.google.com (mail-qy0-f182.google.com [209.85.216.182]) by mx.google.com with ESMTPS id t1sm5950642qcs.9.2010.09.06.12.07.44 (version=SSLv3 cipher=RC4-MD5); Mon, 06 Sep 2010 12:07:45 -0700 (PDT) Received: by qyk4 with SMTP id 4so4782445qyk.13 for ; Mon, 06 Sep 2010 12:07:44 -0700 (PDT) MIME-Version: 1.0 Received: by 10.229.248.84 with SMTP id mf20mr3828211qcb.16.1283800064508; Mon, 06 Sep 2010 12:07:44 -0700 (PDT) Received: by 10.229.100.131 with HTTP; Mon, 6 Sep 2010 12:07:44 -0700 (PDT) In-Reply-To: References: <5DB6E7C798E44D33A05673F4B773405E@multiplay.co.uk> Date: Mon, 6 Sep 2010 21:07:44 +0200 Message-ID: From: Wiktor Niesiobedzki To: Steven Hartland Content-Type: text/plain; charset=UTF-8 Cc: freebsd-fs@freebsd.org Subject: Re: zfs very poor performance compared to ufs due to lack of cache? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Sep 2010 19:07:46 -0000 2010/9/6 Wiktor Niesiobedzki : > Hi, > > As far as I have check recently, nginx is using sendfile by default. > There is already a reported bug against ZFS+sendfile > (http://www.freebsd.org/cgi/query-pr.cgi?pr=141305&cat=) which results > in bad performance. > > What I personally have observed, is that memory, that is used by > sendfile, once freed lands in Inact group. And ARC is not able to > force free of this memory. > > In my case, where I have 1G of ARC, then after sending 2G of file, my > ARC is barerly on minimum level, and my ARC hit ratio drops to ~50%. if did some further tests with nginx and sendfile setting. The setup is following: nginx have a 1GB file to be served to client ARC_MIN=256M, ARC_MAX=1G KMEM_SIZE=1.5G With sendfile off: 1. first download leads to read from disk (as suspected) 2. second download, mostly from ARC (some minor activity on disk, like 1-10% of actual transfer) measured by gstat With sendfile on (after doing previous tests): 1. first download - leads to read from disk (suspicous - this file should be already in ARC) 2. second download - leads to read from disk (suspicous - this file should be in both ARC and Inactive memory) After that the memory looks like: Mem: 58M Active, 1032M Inact, 723M Wired, 121M Free arc size: 670M With a 512MB file, with sendfile on, starting with memory like: Mem: 59M Active, 8948K Inact, 816M Wired, 1050M Free arc size: 780M ARC warmed with test file with: cat tfile512M > /dev/null 1. first fownload - no disk activity Mem: 51M Active, 517M Inact, 822M Wired, 545M Free arc size: 790M 2. second download - no disk activity Mem: 51M Active, 517M Inact, 822M Wired, 544M Free arc size: 790M The test is taking about 90 seconds, and kstat.zfs.misc.arcstats.hits goes up by ~2M In normal situation (no download activity) it goes up by: 200 During subsequent cat tfile512M > /dev/null it goes up by: 131k During nginx download (with sendfile off) it goes up by: 23k So my gut feelings about this situation are: 1. Sendfile on ZFS is not a "zero-copy" solution (we copy from ARC to some other memory before sending a file) 2. Whatever sendfile puts in some "cache", it's never used, as we will use ARC anyway (see big number of arc hits) Some other side observerations are: - nginx is faster (by 50%) with sendfile turned off (not benchmarked, just a feeling) - arcstat hits is growing extermely fast with sendfile (small request from sendfile to ARC?) - nginx is using quite small number of accesses to ARC, even compared to simple cat, to get the same file size (if I at all properly understand what does kstat.zfs.misc.arcstats.hits mean) Hope that helps, Cheers, Wiktor Niesiobedzki