Date: Sat, 8 Jan 2011 21:56:13 +0200 From: Kostik Belousov <kostikbel@gmail.com> To: Lev Serebryakov <lev@freebsd.org> Cc: freebsd-stable@freebsd.org Subject: Re: 8.2-PRERELEASE: live deadlock, almost all processes in "pfault" state Message-ID: <20110108195613.GW12599@deviant.kiev.zoral.com.ua> In-Reply-To: <1792026896.20110108222909@serebryakov.spb.ru> References: <204344488.20110108214457@serebryakov.spb.ru> <20110108190232.GU12599@deviant.kiev.zoral.com.ua> <1792026896.20110108222909@serebryakov.spb.ru>
next in thread | previous in thread | raw e-mail | index | archive | help
--iAq/E3df7QH9m+R5 Content-Type: text/plain; charset=koi8-r Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sat, Jan 08, 2011 at 10:29:09PM +0300, Lev Serebryakov wrote: > Hello, Kostik. > You wrote 8 =D1=CE=D7=C1=D2=D1 2011 =C7., 22:02:32: >=20 >=20 > > There is some weird backtrace at the pid 20, what is g_raid5 ? > It is geom_raid5, with two threads -- working one and one for > processing finished bios. >=20 > > If I am guessing right, this creature has a classic deadlock when=20 > > bio processing requires memory allocation. It seems that tid 100079 > tid 100079 sleep in waiting for some data in queue. >=20 > > is sleeping not even due to the free page shortage, but due to address > > space exhaustion. As result, read/write requests are stalled. > tid 100078 sleep in malloc(). But geom_raid5 never ever allocate > more than 128MiB of memory and it is 64bit system with huge amount of > kmem_size/kmem_size_max... >=20 > How could I explore allocation (like vmstat -m) from kdb to be sure, > it doesn't allocated more? Use "show uma" and "show malloc" from ddb. >=20 > And, if it is "classic deadlock" is here any "classical" solution to > it? Do not allocate during bio processing. >=20 > Really, I'm maintainer of geom_raid5 now, so I need fix this > deadlock, but I don't really understand, why does it occur? I've > hit panic with "kernel memory exhausted" symptoms when module allocate > too much, but not deadlock :( Hm, I missed the kmem_back() in the stack. Yes, the thread is waiting for p= age allocation. >=20 > > Then, syncer is blocked waiting for some physical buffer (look at tid > > 100075), owning the vnode lock. Other processes also wait for the > > locked buffers, etc. >=20 > > So my belief is that this is plain driver (g_raid5, whatever is it) > > i/o loss. Try the same load without it. > I can not, because all data is on this GEOM :) >=20 > --=20 > // Black Lion AKA Lev Serebryakov <lev@serebryakov.spb.ru> >=20 --iAq/E3df7QH9m+R5 Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (FreeBSD) iEYEARECAAYFAk0owVwACgkQC3+MBN1Mb4gXngCgknenCiBYQm3Dnqcxk8QL4gcC L8cAoOD9mV9vUSfr71Nc2k2diqOUcauS =U4jp -----END PGP SIGNATURE----- --iAq/E3df7QH9m+R5--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20110108195613.GW12599>