From owner-freebsd-stable@FreeBSD.ORG Sat Jan 8 19:02:36 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6A330106572C; Sat, 8 Jan 2011 19:02:36 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from mail.zoral.com.ua (mx0.zoral.com.ua [91.193.166.200]) by mx1.freebsd.org (Postfix) with ESMTP id 027468FC17; Sat, 8 Jan 2011 19:02:35 +0000 (UTC) Received: from deviant.kiev.zoral.com.ua (root@deviant.kiev.zoral.com.ua [10.1.1.148]) by mail.zoral.com.ua (8.14.2/8.14.2) with ESMTP id p08J2WU0006053 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Sat, 8 Jan 2011 21:02:32 +0200 (EET) (envelope-from kostikbel@gmail.com) Received: from deviant.kiev.zoral.com.ua (kostik@localhost [127.0.0.1]) by deviant.kiev.zoral.com.ua (8.14.4/8.14.4) with ESMTP id p08J2W3U095326; Sat, 8 Jan 2011 21:02:32 +0200 (EET) (envelope-from kostikbel@gmail.com) Received: (from kostik@localhost) by deviant.kiev.zoral.com.ua (8.14.4/8.14.4/Submit) id p08J2W4S095325; Sat, 8 Jan 2011 21:02:32 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: deviant.kiev.zoral.com.ua: kostik set sender to kostikbel@gmail.com using -f Date: Sat, 8 Jan 2011 21:02:32 +0200 From: Kostik Belousov To: Lev Serebryakov Message-ID: <20110108190232.GU12599@deviant.kiev.zoral.com.ua> References: <204344488.20110108214457@serebryakov.spb.ru> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="2JhMvOwMi9vhAgPY" Content-Disposition: inline In-Reply-To: <204344488.20110108214457@serebryakov.spb.ru> User-Agent: Mutt/1.4.2.3i X-Virus-Scanned: clamav-milter 0.95.2 at skuns.kiev.zoral.com.ua X-Virus-Status: Clean X-Spam-Status: No, score=-3.4 required=5.0 tests=ALL_TRUSTED,AWL,BAYES_00, DNS_FROM_OPENWHOIS autolearn=no version=3.2.5 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on skuns.kiev.zoral.com.ua Cc: freebsd-stable@freebsd.org Subject: Re: 8.2-PRERELEASE: live deadlock, almost all processes in "pfault" state X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 08 Jan 2011 19:02:36 -0000 --2JhMvOwMi9vhAgPY Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sat, Jan 08, 2011 at 09:44:57PM +0300, Lev Serebryakov wrote: > Hello, Freebsd-stable. >=20 > I've added `transmission' BitTorrent client to my home server and now > it deadlocks easily (after about 1 hour of intensive download and > seeding). This server is upgraded from 7.x and last time I've run > transmission on 7.x system without any problems. >=20 > I have home partition on geom_raid5 device, so I can not exclude this > third-party module from experiments. >=20 > My home filsystem has 32KiB block and all other filesystems (/, /var, > /tmp, /usr) has standard 16KiB block sizes. I know, that 7.x system > had (has?) deadlock when 16KiB and 64KiB file systems are mixed up on > one system, but I never experienced deadlocks with 16KiB and 32KiB > mixture. >=20 > All filesystems (Except root) is SU, but no gjournal so SU_J patch > are in use. >=20 > Same BitTorrent client on same filesystem, but accessed via NFS (from > other host), doesn't cause deadlock and works rock-stable for days. >=20 > I've built kernel with all debug options, waited for deadlock and > collect all information, mentioned in Developer's Handbook / Debugging > Deadlocks. >=20 > Capture from debug session is attached, together with kernel config > and dmesg from rebooting. >=20 > As I can easily reproduce this deadlock, I could provide any > additional information from kernel debugger, if needed. >=20 > System: FreeBSD 8.2-PRERELEASE > cvsup: 2011-01-08 00:41:24 MSK (GTM+3) time > Platform: amd64 There is some weird backtrace at the pid 20, what is g_raid5 ? If I am guessing right, this creature has a classic deadlock when=20 bio processing requires memory allocation. It seems that tid 100079 is sleeping not even due to the free page shortage, but due to address space exhaustion. As result, read/write requests are stalled. Then, syncer is blocked waiting for some physical buffer (look at tid 100075), owning the vnode lock. Other processes also wait for the locked buffers, etc. So my belief is that this is plain driver (g_raid5, whatever is it) i/o loss. Try the same load without it. --2JhMvOwMi9vhAgPY Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (FreeBSD) iEYEARECAAYFAk0otMcACgkQC3+MBN1Mb4je/QCgz7k3fEW5jQA3qC0CV1JTp8VL 2/4An0YNI0BSoc2zyzQBcYkwFQGXHOmn =TOTR -----END PGP SIGNATURE----- --2JhMvOwMi9vhAgPY--