From owner-freebsd-stable@FreeBSD.ORG Mon Jan 2 19:57:35 2006 Return-Path: X-Original-To: freebsd-stable@freebsd.org Delivered-To: freebsd-stable@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id D68E716A41F for ; Mon, 2 Jan 2006 19:57:35 +0000 (GMT) (envelope-from lars+lister.freebsd@adventuras.no) Received: from mail.adventuras.no (mail.adventuras.no [194.63.250.215]) by mx1.FreeBSD.org (Postfix) with ESMTP id 4D84643D6B for ; Mon, 2 Jan 2006 19:57:31 +0000 (GMT) (envelope-from lars+lister.freebsd@adventuras.no) Received: from mail.adventuras.no (seven [127.0.0.1]) by mail.adventuras.no (8.12.10/8.12.10) with ESMTP id k02JvCjt013725 (version=TLSv1/SSLv3 cipher=EDH-RSA-DES-CBC3-SHA bits=168 verify=NO) for ; Mon, 2 Jan 2006 20:57:12 +0100 Received: (from apache@localhost) by mail.adventuras.no (8.12.10/8.12.10/Submit) id k02JvCxP013722; Mon, 2 Jan 2006 20:57:12 +0100 Received: from 80.111.250.79 (SquirrelMail authenticated user lars) by mail.adventuras.no with HTTP; Mon, 2 Jan 2006 20:57:12 +0100 (CET) Message-ID: <50410.80.111.250.79.1136231832.squirrel@mail.adventuras.no> In-Reply-To: <200601021114.k02BEgOQ004377@gw.catspoiler.org> References: <20051125032845.73799.qmail@web36208.mail.mud.yahoo.com> <200601021114.k02BEgOQ004377@gw.catspoiler.org> Date: Mon, 2 Jan 2006 20:57:12 +0100 (CET) From: "Lars Kristiansen" To: freebsd-stable@freebsd.org User-Agent: SquirrelMail/1.4.4-1 MIME-Version: 1.0 Content-Type: text/plain;charset=iso-8859-1 Content-Transfer-Encoding: 8bit X-Priority: 3 (Normal) Importance: Normal X-Adventuras-MailScanner-Information: Please contact the ISP for more information X-Adventuras: du kan filtrere etter AdvSpamScore over 5-10 X-Adventuras-SpamCheck: not spam, SpamAssassin (score=-4.399, required 6, autolearn=not spam, ALL_TRUSTED -1.80, BAYES_00 -2.60) X-MailScanner-From: lars+lister.freebsd@adventuras.no Subject: Re: Swapfile problem in 6? X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 Jan 2006 19:57:36 -0000 > Attempting to catch up with my backlog of unread email, only 12K unread > messages to go ... > > On 24 Nov, Rob wrote: > >> I have cvsup'ed the sources to STABLE as of Nov. 23rd >> 2005. >> After recompiling/installing world and debug-kernel, >> I again get a kernel deadlock when using swapfile: >> http://surfion.snu.ac.kr/~lahaye/swapfile2.txt >> >> Previous deadlocks are still documented here >> http://surfion.snu.ac.kr/~lahaye/swapfile.txt >> >> I hope this is of use for fixing this bug in 6. >> If further investigation is needed, then please let me >> know. > > This is a deadlock caused by memory exhaustion. The pagedaemon only has > a limited number of bufs that it uses for writing dirty pages to swap to > prevent it from saturating the I/O subsystem with large numbers of > writes. In this case, pagedaemon is trying to free up memory by writing > dirty pages, and it has used up all of its bufs and is waiting for the > write requests to complete and the bufs the bufs to be returned to it. > This isn't happening because md0 is stuck waiting for memory. This is a > little bit suprising to me because it looks like writes to vnode backed > devices are done synchronously by default. > > If you have a chance to test this again, a stack trace of md0 in the > deadlock state would be interesting. I'd like to know where md0 is > getting stuck. > > I wonder if pagedaemon should scan ahead and more agressively discard > clean pages when it has run out of bufs to write dirty pages, especially > in low memory situations. Preventing the creation of more dirty pages > would be nice, but I don't know how to do that ... Just in case it can help. Do not have this machine available for testing at the moment but this is the last debuginfo I did get from it. Here is a trace from a situation when a possible idle system got stuck during the night and db showed only one locked vnode: db> show lockedvnods Locked vnodes 0xc1309330: tag ufs, type VREG usecount 1, writecount 1, refcount 154 mountedhere 0 flags () v_object 0xc12cb39c ref 0 pages 606 lock type ufs: EXCL (count 1) by thread 0xc126b900 (pid 178) ino 8155, on dev ad0s1f db> trace 178 Tracing pid 178 tid 100058 td 0xc126b900 sched_switch(c126b900,0,1) at 0xc066a4db = sched_switch+0x17b mi_switch(1,0) at 0xc065f49e = mi_switch+0x27e sleepq_switch(c09b2a98,c484bacc,c065f0e3,c09b2a98,0) at 0xc0677f00 = sleepq_switch+0xe0 sleepq_wait(c09b2a98,0,0,c08ad92d,37b) at 0xc0678100 = sleepq_wait+0x30 msleep(c09b2a98,c09b2d00,244,c08adb6a,0) at 0xc065f0e3 = msleep+0x333 vm_wait(c12cb39c,0,c08990f3,ad7,c06512a4) at 0xc07c6a71 = vm_wait+0x91 allocbuf(c28fa9d8,4000,354000,0,354000) at 0xc06a2f89 = allocbuf+0x4e9 getblk(c1309330,d5,0,4000,0) at 0xc06a29cb = getblk+0x4eb cluster_read(c1309330,10000000,0,d5,0) at 0xc06a5d65 = cluster_read+0xe5 ffs_read(c484bc9c) at 0xc07a631f = ffs_read+0x28f VOP_READ_APV(c09309a0,c484bc9c) at 0xc0838aab = VOP_READ_APV+0x7b mdstart_vnode(c1310800,c1634294,c1310820,1,c0566e10) at 0xc056688c = mdstart_vnode+0xec md_kthread(c1310800,c484bd38,c1310800,c0566e10,0) at 0xc0566f7f = md_kthread+0x16f fork_exit(c0566e10,c1310800,c484bd38) at 0xc0645618 = fork_exit+0xa8 fork_trampoline() at 0xc0816f3c = fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xc484bd6c, ebp = 0 --- -- Lars