From owner-freebsd-stable@FreeBSD.ORG Mon Nov 2 11:06:55 2009 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 917401065676 for ; Mon, 2 Nov 2009 11:06:55 +0000 (UTC) (envelope-from 000.fbsd@quip.cz) Received: from elsa.codelab.cz (elsa.codelab.cz [94.124.105.4]) by mx1.freebsd.org (Postfix) with ESMTP id 40A518FC2B for ; Mon, 2 Nov 2009 11:06:55 +0000 (UTC) Received: from localhost (localhost.codelab.cz [127.0.0.1]) by elsa.codelab.cz (Postfix) with ESMTP id B7D2919E023; Mon, 2 Nov 2009 12:06:53 +0100 (CET) Received: from [192.168.1.2] (r5bb235.net.upc.cz [86.49.61.235]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by elsa.codelab.cz (Postfix) with ESMTPSA id 7E25E19E027; Mon, 2 Nov 2009 12:06:51 +0100 (CET) Message-ID: <4AEEBD4B.1050407@quip.cz> Date: Mon, 02 Nov 2009 12:06:51 +0100 From: Miroslav Lachman <000.fbsd@quip.cz> User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.9.1.4) Gecko/20091017 SeaMonkey/2.0 MIME-Version: 1.0 To: Ivan Voras References: <772532900-1257123963-cardhu_decombobulator_blackberry.rim.net-1402739480-@bda715.bisx.prod.on.blackberry> In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-stable@freebsd.org Subject: Re: Performance issues with 8.0 ZFS and sendfile/lighttpd X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 Nov 2009 11:06:55 -0000 Ivan Voras wrote: > gnukix@alltel.blackberry.com wrote: >> I can send in more documentation later but I am seeing severe zfs >> performance issues with lighttpd. Same machine using UFS will push >> 1gbit or more but same content and traffic load can not hit 200mbit. >> Ufs does around 3 megabytes/sec IO at 800mbit network but zfs pushes >> the disks into the ground with 50+ megabytes/sec dusk i/o. No >> compression no atime no checksums on zfs and still same IO levels. Ufs >> with soft updates and atime on. Orders of magnitude more disk IO... >> Like zfs isn't using cache or isn't coalescing disk reads or both. >> Has anyone else seen this or have any recommendations? Lighttpd config >> remains exactly the same as well FYI. Only difference is ufs vs zfs. > > AFAIK, ZFS is incompatible (currently) with some advanced VM operations > (like mmap, and I think sendfile relies on the same mechanism as mmap), > so that could be a cause of the slowdown. Though I'm surprised you can > only get 200 MBit/s - that's 25 MB/s and I think that even with multiple > memcpy-ing data around the kernel you should be able to get hundreds of > MB/s on newer hardware (which normally really can achieve tens of > gigabytes/s of sustained memory access). I have more strange issue with Lighttpd in jail on top of ZFS. Lighttpd is serving static content (mp3 downloads thru flash player). Is runs fine for relatively small number of parallel clients with bandwidth about 30 Mbps, but after some number of clients is reached (about 50-60 parallel clients) the throughput drops down to 6 Mbps. I can server hundereds of clients on same HW using Lighttpd not in jail and UFS2 with gjournal instead of ZFS reaching 100 Mbps (maybe more). I don't know if it is ZFS or Jail issue. Miroslav Lachman