From owner-freebsd-fs@FreeBSD.ORG Mon Sep 6 19:01:04 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E4B9210656BE for ; Mon, 6 Sep 2010 19:01:04 +0000 (UTC) (envelope-from bsd@vink.pl) Received: from mail-qy0-f182.google.com (mail-qy0-f182.google.com [209.85.216.182]) by mx1.freebsd.org (Postfix) with ESMTP id A34648FC21 for ; Mon, 6 Sep 2010 19:01:04 +0000 (UTC) Received: by qyk4 with SMTP id 4so4777121qyk.13 for ; Mon, 06 Sep 2010 12:01:04 -0700 (PDT) Received: by 10.224.89.11 with SMTP id c11mr797843qam.182.1283798104592; Mon, 06 Sep 2010 11:35:04 -0700 (PDT) Received: from mail-qw0-f54.google.com (mail-qw0-f54.google.com [209.85.216.54]) by mx.google.com with ESMTPS id r1sm5908272qcq.34.2010.09.06.11.35.04 (version=SSLv3 cipher=RC4-MD5); Mon, 06 Sep 2010 11:35:04 -0700 (PDT) Received: by qwg5 with SMTP id 5so4402448qwg.13 for ; Mon, 06 Sep 2010 11:35:02 -0700 (PDT) MIME-Version: 1.0 Received: by 10.224.11.18 with SMTP id r18mr394734qar.195.1283798099497; Mon, 06 Sep 2010 11:34:59 -0700 (PDT) Received: by 10.229.100.131 with HTTP; Mon, 6 Sep 2010 11:34:59 -0700 (PDT) In-Reply-To: <5DB6E7C798E44D33A05673F4B773405E@multiplay.co.uk> References: <5DB6E7C798E44D33A05673F4B773405E@multiplay.co.uk> Date: Mon, 6 Sep 2010 20:34:59 +0200 Message-ID: From: Wiktor Niesiobedzki To: Steven Hartland Content-Type: text/plain; charset=UTF-8 Cc: freebsd-fs@freebsd.org Subject: Re: zfs very poor performance compared to ufs due to lack of cache? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Sep 2010 19:01:05 -0000 2010/9/4 Steven Hartland : > When upgrading from 8.0 on our stream server to 8.1 we decided to go > for zfs to eliminate the costly fsck times should we experience > any unexpected reboots on the machine as it has a sizable RAID of > 1.6TB. > > After doing this all seemed good till after our latest event which > generated a significant amount of interest and hence the stream > server started to get quite a few requests. > > Basic install is 8.1 amd64 on a dual 2.8 Xeon with 4GB RAM and > areca controller with 6 disk in RAID 6. > > On that the machine runs nginx with the mp4 module to sudo stream > files. Hi, As far as I have check recently, nginx is using sendfile by default. There is already a reported bug against ZFS+sendfile (http://www.freebsd.org/cgi/query-pr.cgi?pr=141305&cat=) which results in bad performance. The quickest workaround is to set: sendfile off; In http {} sectio of nginx.conf. What I personally have observed, is that memory, that is used by sendfile, once freed lands in Inact group. And ARC is not able to force free of this memory. In my case, where I have 1G of ARC, then after sending 2G of file, my ARC is barerly on minimum level, and my ARC hit ratio drops to ~50%. If I remove the file that was sent through sendfile, memory is moved from Inact to free, from where ARC happly grabs what it wants, and ARC hit ratio comes back to normal (~99%). Cheers, Wiktor Niesiobedzki