From owner-freebsd-stable@FreeBSD.ORG Wed Mar 6 05:16:15 2013 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id B440077A for ; Wed, 6 Mar 2013 05:16:15 +0000 (UTC) (envelope-from jdc@koitsu.org) Received: from qmta07.emeryville.ca.mail.comcast.net (qmta07.emeryville.ca.mail.comcast.net [IPv6:2001:558:fe2d:43:76:96:30:64]) by mx1.freebsd.org (Postfix) with ESMTP id 2CA6B140 for ; Wed, 6 Mar 2013 05:16:14 +0000 (UTC) Received: from omta01.emeryville.ca.mail.comcast.net ([76.96.30.11]) by qmta07.emeryville.ca.mail.comcast.net with comcast id 856o1l0030EPchoA75GEVt; Wed, 06 Mar 2013 05:16:14 +0000 Received: from koitsu.strangled.net ([67.180.84.87]) by omta01.emeryville.ca.mail.comcast.net with comcast id 85GD1l00s1t3BNj8M5GEgm; Wed, 06 Mar 2013 05:16:14 +0000 Received: by icarus.home.lan (Postfix, from userid 1000) id E13C873A31; Tue, 5 Mar 2013 21:16:13 -0800 (PST) Date: Tue, 5 Mar 2013 21:16:13 -0800 From: Jeremy Chadwick To: Karl Denninger Subject: Re: ZFS "stalls" -- and maybe we should be talking about defaults? Message-ID: <20130306051613.GA62470@icarus.home.lan> References: <513524B2.6020600@denninger.net> <89680320E0FA4C0A99D522EA2037CE6E@multiplay.co.uk> <20130305050539.GA52821@anubis.morrow.me.uk> <20130305053249.GA38107@icarus.home.lan> <545CD2ABE3D146F2B91963ADF6090CDE@multiplay.co.uk> <20130305092700.GA43045@icarus.home.lan> <5135EB62.6060006@denninger.net> <20130306050809.GA61727@icarus.home.lan> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20130306050809.GA61727@icarus.home.lan> User-Agent: Mutt/1.5.21 (2010-09-15) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=comcast.net; s=q20121106; t=1362546974; bh=aPHThh7B1po5lOHmXLXALa4xc/FmB0Qyz5p+1agt+wo=; h=Received:Received:Received:Date:From:To:Subject:Message-ID: MIME-Version:Content-Type; b=luSFAec6lE1CVJE3XxutoT5zhd3t80LmjmMTosJF1ZXfAKxrQjbqF67tjI2PH4cdP F70C8aRhWs7pEgx45XZ7w41pRbuvfWbwEfd+WhP9VozV+mqoDaIAvz/c1N0YxBrQeQ vK0yN0fOMMF2LoF8JHxVFYRBfIjOPwXxEVEv7M7lmOqYHRbJjGwRsVI1m+Jv0TamGR hb33b1jXatYyaoLJa/GS1hwqeOHgV2xGJlHxVljzLDJ7I1dypEFihX2OFd1NEFD1iv jPnAP6A+9jooZeSTV+5udtY6b0ax5GRC4p3HI6o5YnZXBv8lpratbZDZIE6F7BcaMt ghnSEbSQcziJA== Cc: freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Mar 2013 05:16:15 -0000 On Tue, Mar 05, 2013 at 09:08:09PM -0800, Jeremy Chadwick wrote: > > > * How long the stall is in duration (ex. if there's some way to > > > roughly calculate this using "date" in a shell script) > > They're variable. Some last fractions of a second and are not really > > all that noticeable unless you happen to be paying CLOSE attention. > > Some last a few (5 or so) seconds. The really bad ones last long enough > > that the kernel throws the message "swap_pager: indefinite wait buffer". > > The message "swap_pager: indefinite wait buffer" indicates that some > part of the VM is trying to offload pages of memory to swap via standard > I/O write requests, and those writes have not come back within kern.hz*20 > seconds. That's a very, very long time. Two clarification points: 1. The timeout value is passed to msleep(9) and is literally kern.hz*20. Per sys/vm/swap_pager.c: 1216 if (msleep(mreq, VM_OBJECT_MTX(object), PSWP, "swread", hz*20)) { 1217 printf( 1218 "swap_pager: indefinite wait buffer: bufobj: %p, blkno: %jd, size: %ld\n", 1219 bp->b_bufobj, (intmax_t)bp->b_blkno, bp->b_bcount); How that's interpreted is documented in msleep(9): The parameter timo specifies a timeout for the sleep. If timo is not 0, then the thread will sleep for at most timo / hz seconds. If the timeout expires, then the sleep function will return EWOULDBLOCK. 2. The message appears to be for swap I/O *reads*, not writes; at least that's what the "swread" STATE string (you know, what you see in top(1)) implies. -- | Jeremy Chadwick jdc@koitsu.org | | UNIX Systems Administrator http://jdc.koitsu.org/ | | Mountain View, CA, US | | Making life hard for others since 1977. PGP 4BD6C0CB |