From owner-freebsd-fs@FreeBSD.ORG Sat Nov 7 20:18:31 2009 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 076B5106566B; Sat, 7 Nov 2009 20:18:31 +0000 (UTC) (envelope-from 000.fbsd@quip.cz) Received: from elsa.codelab.cz (elsa.codelab.cz [94.124.105.4]) by mx1.freebsd.org (Postfix) with ESMTP id B01AA8FC08; Sat, 7 Nov 2009 20:18:30 +0000 (UTC) Received: from localhost (localhost.codelab.cz [127.0.0.1]) by elsa.codelab.cz (Postfix) with ESMTP id 7179C19E023; Sat, 7 Nov 2009 21:18:28 +0100 (CET) Received: from [192.168.1.2] (r5bb235.net.upc.cz [86.49.61.235]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by elsa.codelab.cz (Postfix) with ESMTPSA id 17B8D19E019; Sat, 7 Nov 2009 21:18:25 +0100 (CET) Message-ID: <4AF5D611.7060408@quip.cz> Date: Sat, 07 Nov 2009 21:18:25 +0100 From: Miroslav Lachman <000.fbsd@quip.cz> User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.9.1.4) Gecko/20091017 SeaMonkey/2.0 MIME-Version: 1.0 To: Ivan Voras References: <772532900-1257123963-cardhu_decombobulator_blackberry.rim.net-1402739480-@bda715.bisx.prod.on.blackberry> <4AEEBD4B.1050407@quip.cz> <4AEEDB3B.5020600@quip.cz> <4AF46CA9.1040904@quip.cz> <9bbcef730911061101h5356d2acob2ac8791afe112@mail.gmail.com> In-Reply-To: <9bbcef730911061101h5356d2acob2ac8791afe112@mail.gmail.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org, freebsd-stable@freebsd.org Subject: Re: Performance issues with 8.0 ZFS and sendfile/lighttpd X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 07 Nov 2009 20:18:31 -0000 Ivan Voras wrote: > 2009/11/6 Miroslav Lachman<000.fbsd@quip.cz>: > >> I do not understand why there are 10MB/s read from disks when network >> traffic dropped to around 1MB/s (8Mbps) >> >> root@cage ~/# iostat -w 20 >> tty ad4 ad6 cpu >> tin tout KB/t tps MB/s KB/t tps MB/s us ni sy in id >> 0 14 41.66 53 2.17 41.82 53 2.18 0 0 2 0 97 >> 0 18 50.92 96 4.77 54.82 114 6.12 0 0 3 1 96 >> 0 6 53.52 101 5.29 54.98 108 5.81 1 0 4 1 94 >> 0 6 54.82 98 5.26 55.89 108 5.89 0 0 3 1 96 > > Yes, this could limit your IO if the requests are random enough. > Unfortunately I don't know how would you track down what is really > going on. Maybe some tracing with DTrace? > > I'd tell you to use "top -m io" to see if there is a process > responsible, but apparently these statistics are not updated for ZFS, > which in itself may be a bug (which is why I'm crossposting to > freebsd-fs). DTrace is totally out of my skills ;( There is otput of top -m io sorted by VCSW displaying JID. last pid: 17724; load averages: 0.01, 0.07, 0.08 up 74+20:49:49 21:03:40 195 processes: 1 running, 193 sleeping, 1 zombie CPU: 0.0% user, 0.0% nice, 3.6% system, 0.4% interrupt, 96.1% idle Mem: 462M Active, 2385M Inact, 977M Wired, 21M Cache, 399M Buf, 100M Free Swap: 6144M Total, 2024K Used, 6142M Free PID JID USERNAME VCSW IVCSW READ WRITE FAULT TOTAL PERCENT COMMAND 17681 8 www 657 64 0 0 0 0 0.00% lighttpd 17683 8 www 379 41 0 0 0 0 0.00% lighttpd 17680 8 www 136 5 0 0 0 0 0.00% lighttpd 17682 8 www 85 0 0 0 0 0 0.00% lighttpd 4689 1 90 10 0 0 0 0 0 0.00% fb_inet_server 3403 1 90 10 0 0 0 0 0 0.00% fb_inet_server 2632 1 90 10 0 0 0 0 0 0.00% fb_inet_server All four top consumers is Lighttpd workers. And as you noted, read, write, fault, total and percent are not updated on machine with ZFS, so I can't compare it with UFS2 based machine. Is this bug in top fixed in 8.x? Will you file a PR? (you know more about FS related things than me :]) Miroslav Lachman