From owner-freebsd-stable@FreeBSD.ORG Sat Jan 8 19:29:17 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0667A1065767 for ; Sat, 8 Jan 2011 19:29:17 +0000 (UTC) (envelope-from lev@freebsd.org) Received: from ftp.translate.ru (ftp.translate.ru [80.249.188.42]) by mx1.freebsd.org (Postfix) with ESMTP id B76B28FC12 for ; Sat, 8 Jan 2011 19:29:16 +0000 (UTC) Received: from [192.168.134.2] (89.112.15.178.pppoe.eltel.net [89.112.15.178]) (Authenticated sender: lev@serebryakov.spb.ru) by ftp.translate.ru (Postfix) with ESMTPA id 9F65613DF61; Sat, 8 Jan 2011 22:29:15 +0300 (MSK) Date: Sat, 8 Jan 2011 22:29:09 +0300 From: Lev Serebryakov X-Priority: 3 (Normal) Message-ID: <1792026896.20110108222909@serebryakov.spb.ru> To: Kostik Belousov In-Reply-To: <20110108190232.GU12599@deviant.kiev.zoral.com.ua> References: <204344488.20110108214457@serebryakov.spb.ru> <20110108190232.GU12599@deviant.kiev.zoral.com.ua> MIME-Version: 1.0 Content-Type: text/plain; charset=windows-1251 Content-Transfer-Encoding: quoted-printable Cc: freebsd-stable@freebsd.org Subject: Re: 8.2-PRERELEASE: live deadlock, almost all processes in "pfault" state X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 08 Jan 2011 19:29:17 -0000 Hello, Kostik. You wrote 8 =FF=ED=E2=E0=F0=FF 2011 =E3., 22:02:32: > There is some weird backtrace at the pid 20, what is g_raid5 ? It is geom_raid5, with two threads -- working one and one for processing finished bios. > If I am guessing right, this creature has a classic deadlock when=20 > bio processing requires memory allocation. It seems that tid 100079 tid 100079 sleep in waiting for some data in queue. > is sleeping not even due to the free page shortage, but due to address > space exhaustion. As result, read/write requests are stalled. tid 100078 sleep in malloc(). But geom_raid5 never ever allocate more than 128MiB of memory and it is 64bit system with huge amount of kmem_size/kmem_size_max... How could I explore allocation (like vmstat -m) from kdb to be sure, it doesn't allocated more? And, if it is "classic deadlock" is here any "classical" solution to it? Really, I'm maintainer of geom_raid5 now, so I need fix this deadlock, but I don't really understand, why does it occur? I've hit panic with "kernel memory exhausted" symptoms when module allocate too much, but not deadlock :( > Then, syncer is blocked waiting for some physical buffer (look at tid > 100075), owning the vnode lock. Other processes also wait for the > locked buffers, etc. > So my belief is that this is plain driver (g_raid5, whatever is it) > i/o loss. Try the same load without it. I can not, because all data is on this GEOM :) --=20 // Black Lion AKA Lev Serebryakov