From owner-freebsd-hackers@FreeBSD.ORG Sat Oct 25 08:02:01 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B3B011065672 for ; Sat, 25 Oct 2008 08:02:01 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from QMTA09.westchester.pa.mail.comcast.net (qmta09.westchester.pa.mail.comcast.net [76.96.62.96]) by mx1.freebsd.org (Postfix) with ESMTP id D9F398FC13 for ; Sat, 25 Oct 2008 08:02:00 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from OMTA03.westchester.pa.mail.comcast.net ([76.96.62.27]) by QMTA09.westchester.pa.mail.comcast.net with comcast id Ww0R1a0060bG4ec59w1zQo; Sat, 25 Oct 2008 08:01:59 +0000 Received: from koitsu.dyndns.org ([69.181.141.110]) by OMTA03.westchester.pa.mail.comcast.net with comcast id Ww1y1a01X2P6wsM3Pw1zPc; Sat, 25 Oct 2008 08:01:59 +0000 X-Authority-Analysis: v=1.0 c=1 a=QycZ5dHgAAAA:8 a=tQ9rm8fwbuQ4hyOsMbAA:9 a=p8GA1uGYW3cPqib4-f0A:7 a=2YpFDbIIgmTrhgGCxGe-fFlL-skA:4 a=EoioJ0NPDVgA:10 a=LY0hPdMaydYA:10 Received: by icarus.home.lan (Postfix, from userid 1000) id 31D90C9419; Sat, 25 Oct 2008 00:55:43 -0700 (PDT) Date: Sat, 25 Oct 2008 00:55:43 -0700 From: Jeremy Chadwick To: Danny Braniss Message-ID: <20081025075543.GA55339@icarus.home.lan> References: <20081024150916.GB41283@dan.emsphone.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.18 (2008-05-17) Cc: FreeBSD Hackers , Dan Nelson Subject: Re: zfs & waiting on zio->io_cv X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 25 Oct 2008 08:02:01 -0000 On Sat, Oct 25, 2008 at 09:48:15AM +0200, Danny Braniss wrote: > > In the last episode (Oct 24), Danny Braniss said: > > > there is a big delay (probably more than 1 sec.) when doing simple tasks > > > on this zfs, like ls(1), or 'zfs list', long enough to hit ^T > > > and get the same [zio->io_cv)], any hints? > > > > > > store-01# zfs list > > > (hitting ^T)load: 0.00 cmd: zfs 88376 [zio->io_cv)] 0.00u 0.00s 0% 1672k > > > (hitting ^T)load: 0.00 cmd: zfs 88376 [zio->io_cv)] 0.00u 0.00s 0% 1684k > > > NAME USED AVAIL REFER MOUNTPOINT > > > h 472G 11.2T 23K /h > > > h/home 466G 11.2T 466G /h/home > > > h/home@23-10-08 54K - 466G - > > > h/root 18K 11.2T 18K /h/root > > > h/src 18K 11.2T 18K /h/src > > > h/system 5.64G 11.2T 5.64G /h/system > > > > That's sort of the equivalent to waiting in "biord" on a UFS > > filesystem, I think. ZFS is just waiting for the disk to return a > > block. If you happen to do something during the window where ZFS is > > commiting its transaction group, it has to wait until the sync > > finishes. If some other process is doing a lot of writes, or you only > > have one disk in your zpool, or your pool is close to full, it may take > > a couple seconds to sync. > > > > There's a couple of things you can try to improve interactive > > performance. Raising zfs's arc_max is the easiest to do, and will let > > ZFS cache more stuff, increasing the likelyhood that an "ls" will be > > able to read from cache instead of having to go to disk. Setting it at > > 1/4 your physical RAM is probably as high as you can go without causing > > panics. > > > > Raising txg_time ( in /sys/cddl/.../zfs/txg.c ) from 5 to > > say 30 will tell zfs to sync less often, which can be a win if you > > don't actually do that much writing. With a single spindle, it may > > take a substantial fraction of a second just to sync a tiny txg due to > > the number of copies of metadata ZFS writes for redundancy. > > > > If you do a lot of writing, lowering zfs_vdev_max_pending ( in > > /sys/cddl/.../zfs/vdev_queue.c ) from 35 down to 16 or less will reduce > > the number of simultaneous I/Os ZFS will try to send to each disk, > > which will let your reads compete a little better with other I/O. On > > ATA or SATA disks, you might want to set it to 2. > > > ok, forgot to mention a small detail, the machine is a cuad core, with 8gb > of main memory, the disks are 14x1tb connected via a perc/raid5 > tests show that disk access is quiet fast, over 200Mg/s. > > the 'delays' are seen when the machine is totaly idle. (it's not production > yet) > and been up for some time. btw, I can't reproduce the 'delay', so I think > it has to do with caching. > > I guess this beast needs some tunning, are there any tools out there > to monitor/tune ZFS? Monitor ZFS: sysctl Tune ZFS: vi /boot/loader.conf or sysctl I'm not sure what you're looking for. :-) -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB |