From owner-freebsd-stable@FreeBSD.ORG Sat Jan 8 19:56:18 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0244A106566C; Sat, 8 Jan 2011 19:56:18 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from mail.zoral.com.ua (mx0.zoral.com.ua [91.193.166.200]) by mx1.freebsd.org (Postfix) with ESMTP id 761DB8FC08; Sat, 8 Jan 2011 19:56:17 +0000 (UTC) Received: from deviant.kiev.zoral.com.ua (root@deviant.kiev.zoral.com.ua [10.1.1.148]) by mail.zoral.com.ua (8.14.2/8.14.2) with ESMTP id p08JuDQf010137 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Sat, 8 Jan 2011 21:56:13 +0200 (EET) (envelope-from kostikbel@gmail.com) Received: from deviant.kiev.zoral.com.ua (kostik@localhost [127.0.0.1]) by deviant.kiev.zoral.com.ua (8.14.4/8.14.4) with ESMTP id p08JuDod095571; Sat, 8 Jan 2011 21:56:13 +0200 (EET) (envelope-from kostikbel@gmail.com) Received: (from kostik@localhost) by deviant.kiev.zoral.com.ua (8.14.4/8.14.4/Submit) id p08JuDhH095570; Sat, 8 Jan 2011 21:56:13 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: deviant.kiev.zoral.com.ua: kostik set sender to kostikbel@gmail.com using -f Date: Sat, 8 Jan 2011 21:56:13 +0200 From: Kostik Belousov To: Lev Serebryakov Message-ID: <20110108195613.GW12599@deviant.kiev.zoral.com.ua> References: <204344488.20110108214457@serebryakov.spb.ru> <20110108190232.GU12599@deviant.kiev.zoral.com.ua> <1792026896.20110108222909@serebryakov.spb.ru> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="iAq/E3df7QH9m+R5" Content-Disposition: inline In-Reply-To: <1792026896.20110108222909@serebryakov.spb.ru> User-Agent: Mutt/1.4.2.3i X-Virus-Scanned: clamav-milter 0.95.2 at skuns.kiev.zoral.com.ua X-Virus-Status: Clean X-Spam-Status: No, score=-3.4 required=5.0 tests=ALL_TRUSTED,AWL,BAYES_00, DNS_FROM_OPENWHOIS autolearn=no version=3.2.5 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on skuns.kiev.zoral.com.ua Cc: freebsd-stable@freebsd.org Subject: Re: 8.2-PRERELEASE: live deadlock, almost all processes in "pfault" state X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 08 Jan 2011 19:56:18 -0000 --iAq/E3df7QH9m+R5 Content-Type: text/plain; charset=koi8-r Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sat, Jan 08, 2011 at 10:29:09PM +0300, Lev Serebryakov wrote: > Hello, Kostik. > You wrote 8 =D1=CE=D7=C1=D2=D1 2011 =C7., 22:02:32: >=20 >=20 > > There is some weird backtrace at the pid 20, what is g_raid5 ? > It is geom_raid5, with two threads -- working one and one for > processing finished bios. >=20 > > If I am guessing right, this creature has a classic deadlock when=20 > > bio processing requires memory allocation. It seems that tid 100079 > tid 100079 sleep in waiting for some data in queue. >=20 > > is sleeping not even due to the free page shortage, but due to address > > space exhaustion. As result, read/write requests are stalled. > tid 100078 sleep in malloc(). But geom_raid5 never ever allocate > more than 128MiB of memory and it is 64bit system with huge amount of > kmem_size/kmem_size_max... >=20 > How could I explore allocation (like vmstat -m) from kdb to be sure, > it doesn't allocated more? Use "show uma" and "show malloc" from ddb. >=20 > And, if it is "classic deadlock" is here any "classical" solution to > it? Do not allocate during bio processing. >=20 > Really, I'm maintainer of geom_raid5 now, so I need fix this > deadlock, but I don't really understand, why does it occur? I've > hit panic with "kernel memory exhausted" symptoms when module allocate > too much, but not deadlock :( Hm, I missed the kmem_back() in the stack. Yes, the thread is waiting for p= age allocation. >=20 > > Then, syncer is blocked waiting for some physical buffer (look at tid > > 100075), owning the vnode lock. Other processes also wait for the > > locked buffers, etc. >=20 > > So my belief is that this is plain driver (g_raid5, whatever is it) > > i/o loss. Try the same load without it. > I can not, because all data is on this GEOM :) >=20 > --=20 > // Black Lion AKA Lev Serebryakov >=20 --iAq/E3df7QH9m+R5 Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (FreeBSD) iEYEARECAAYFAk0owVwACgkQC3+MBN1Mb4gXngCgknenCiBYQm3Dnqcxk8QL4gcC L8cAoOD9mV9vUSfr71Nc2k2diqOUcauS =U4jp -----END PGP SIGNATURE----- --iAq/E3df7QH9m+R5--