From owner-freebsd-fs@FreeBSD.ORG Tue Sep 13 13:57:22 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5D6A2106564A for ; Tue, 13 Sep 2011 13:57:22 +0000 (UTC) (envelope-from bfriesen@simple.dallas.tx.us) Received: from blade.simplesystems.org (blade.simplesystems.org [65.66.246.74]) by mx1.freebsd.org (Postfix) with ESMTP id 258A98FC0C for ; Tue, 13 Sep 2011 13:57:21 +0000 (UTC) Received: from freddy.simplesystems.org (freddy.simplesystems.org [65.66.246.65]) by blade.simplesystems.org (8.14.4+Sun/8.14.4) with ESMTP id p8DDvKj6005259; Tue, 13 Sep 2011 08:57:20 -0500 (CDT) Date: Tue, 13 Sep 2011 08:57:20 -0500 (CDT) From: Bob Friesenhahn X-X-Sender: bfriesen@freddy.simplesystems.org To: Laszlo KAROLYI In-Reply-To: Message-ID: References: User-Agent: Alpine 2.01 (GSO 1266 2009-07-14) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.2 (blade.simplesystems.org [65.66.246.90]); Tue, 13 Sep 2011 08:57:21 -0500 (CDT) Cc: freebsd-fs Subject: Re: ZFS-lighttpd2-sendfile, too high IO X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Sep 2011 13:57:22 -0000 On Tue, 13 Sep 2011, Laszlo KAROLYI wrote: > Hello, > > it seems that turning off sendfile() in lighttpd2 completely solved my problem. > > The inactive memory usage ceased, wired memory raised for ARC cache, > and I see an enormous fall of IO load under the same (or even > higher) network load. The disk IO load now scales to the network > load. > > Something must be not right with the sendfile support in ZFS. It seems likely that sendfile is disabling read caching but is also not reading full block-aligned filesystem blocks from zfs (perhaps it does MMU pagesize sized reads). This would result in read amplification. The sendfile() API allows the application to do the wrong thing since it allows the application to specify the starting offset and number of bytes. This means that lighttpd2 may also be causing a problem if it does partial transfers which are not well aligned/sized for underlying filesystem blocks. A system call trace of lighttpd2 when it is configured to use sendfile may be illuminating. Bob -- Bob Friesenhahn bfriesen@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer, http://www.GraphicsMagick.org/