From owner-freebsd-stable@FreeBSD.ORG Wed Nov 4 14:18:45 2009 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1569C1065672 for ; Wed, 4 Nov 2009 14:18:45 +0000 (UTC) (envelope-from urmas.lett@eenet.ee) Received: from muheleja.eenet.ee (muheleja.eenet.ee [193.40.0.132]) by mx1.freebsd.org (Postfix) with ESMTP id AA0318FC17 for ; Wed, 4 Nov 2009 14:18:44 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by localhost.eenet.ee (Postfix) with ESMTP id BF6861CC2A; Wed, 4 Nov 2009 16:00:59 +0200 (EET) X-Virus-Scanned: amavisd-new at eenet.ee Received: from muheleja.eenet.ee ([127.0.0.1]) by localhost (muheleja.eenet.ee [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id kPsAIiAQJNa8; Wed, 4 Nov 2009 16:00:57 +0200 (EET) Received: from [193.40.0.223] (poriseja.eenet.ee [193.40.0.223]) by muheleja.eenet.ee (Postfix) with ESMTP id 8687E1CC27; Wed, 4 Nov 2009 16:00:57 +0200 (EET) Message-ID: <4AF1891C.6010908@eenet.ee> Date: Wed, 04 Nov 2009 16:01:00 +0200 From: Urmas Lett Organization: EENet User-Agent: Thunderbird 2.0.0.23 (Windows/20090812) MIME-Version: 1.0 To: gnukix@alltel.blackberry.com References: <772532900-1257123963-cardhu_decombobulator_blackberry.rim.net-1402739480-@bda715.bisx.prod.on.blackberry> In-Reply-To: <772532900-1257123963-cardhu_decombobulator_blackberry.rim.net-1402739480-@bda715.bisx.prod.on.blackberry> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-stable@freebsd.org Subject: Re: Performance issues with 8.0 ZFS and sendfile/lighttpd X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 04 Nov 2009 14:18:45 -0000 gnukix@alltel.blackberry.com wrote: > I can send in more documentation later but I am seeing severe zfs performance issues with lighttpd. Same machine using UFS will push 1gbit or more but same content and traffic load can not hit 200mbit. Ufs does around 3 megabytes/sec IO at 800mbit network but zfs pushes the disks into the ground with 50+ megabytes/sec dusk i/o. No compression no atime no checksums on zfs and still same IO levels. Ufs with soft updates and atime on. Orders of magnitude more disk IO... Like zfs isn't using cache or isn't coalescing disk reads or both. > > Has anyone else seen this or have any recommendations? Lighttpd config remains exactly the same as well FYI. Only difference is ufs vs zfs. Hi, I can confirm this on FreeBSD 7.2-STABLE. first run: :~# wget -O /dev/null http://192.168.1.1/1000m.bin --2009-11-04 15:36:15-- http://192.168.1.1/1000m.bin Connecting to 192.168.1.1:80... connected. HTTP request sent, awaiting response... 200 OK Length: 1048576000 (1000M) [application/octet-stream] Saving to: `/dev/null' 100%[====================================>] 1,048,576,000 17.0M/s in 81s 2009-11-04 15:37:36 (12.3 MB/s) - `/dev/null' saved [1048576000/1048576000] second run is even slower, cannot wait till end: :~# wget -O /dev/null http://192.168.1.1/1000m.bin --2009-11-04 15:40:00-- http://192.168.1.1/1000m.bin Connecting to 192.168.1.1:80... connected. HTTP request sent, awaiting response... 200 OK Length: 1048576000 (1000M) [application/octet-stream] Saving to: `/dev/null' 71% [==========================> ] 752,173,056 2.10M/s eta 2m 0s ^C After changing in lighttpd.conf server.network-backend = "writev" first run: :~# wget -O /dev/null http://192.168.1.1/1000m.bin --2009-11-04 15:47:51-- http://192.168.1.1/1000m.bin Connecting to 192.168.1.1:80... connected. HTTP request sent, awaiting response... 200 OK Length: 1048576000 (1000M) [application/octet-stream] Saving to: `/dev/null' 100%[====================================>] 1,048,576,000 44.1M/s in 27s 2009-11-04 15:48:18 (37.2 MB/s) - `/dev/null' saved [1048576000/1048576000] second & third run: :~# wget -O /dev/null http://192.168.1.1/1000m.bin --2009-11-04 15:48:20-- http://192.168.1.1/1000m.bin Connecting to 192.168.1.1:80... connected. HTTP request sent, awaiting response... 200 OK Length: 1048576000 (1000M) [application/octet-stream] Saving to: `/dev/null' 100%[====================================>] 1,048,576,000 788M/s in 1.3s 2009-11-04 15:48:21 (788 MB/s) - `/dev/null' saved [1048576000/1048576000] :~# wget -O /dev/null http://192.168.1.1/1000m.bin --2009-11-04 15:48:24-- http://192.168.1.1/1000m.bin Connecting to 192.168.1.1:80... connected. HTTP request sent, awaiting response... 200 OK Length: 1048576000 (1000M) [application/octet-stream] Saving to: `/dev/null' 100%[====================================>] 1,048,576,000 910M/s in 1.1s 2009-11-04 15:48:25 (910 MB/s) - `/dev/null' saved [1048576000/1048576000] I have Intel 10GbE PCI-Express directly connected between two servers. -- Urmas Lett Tel: +(372) 7 302 110 Fax: +(372) 7 302 111 E-Mail: urmas.lett@eenet.ee