From owner-freebsd-fs@FreeBSD.ORG Thu Jun 17 07:28:46 2010 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0F426106564A for ; Thu, 17 Jun 2010 07:28:46 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from mail.zoral.com.ua (mx0.zoral.com.ua [91.193.166.200]) by mx1.freebsd.org (Postfix) with ESMTP id 9D8528FC15 for ; Thu, 17 Jun 2010 07:28:45 +0000 (UTC) Received: from deviant.kiev.zoral.com.ua (root@deviant.kiev.zoral.com.ua [10.1.1.148]) by mail.zoral.com.ua (8.14.2/8.14.2) with ESMTP id o5H7Sfb5066122 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Thu, 17 Jun 2010 10:28:41 +0300 (EEST) (envelope-from kostikbel@gmail.com) Received: from deviant.kiev.zoral.com.ua (kostik@localhost [127.0.0.1]) by deviant.kiev.zoral.com.ua (8.14.4/8.14.4) with ESMTP id o5H7Sfod011220; Thu, 17 Jun 2010 10:28:41 +0300 (EEST) (envelope-from kostikbel@gmail.com) Received: (from kostik@localhost) by deviant.kiev.zoral.com.ua (8.14.4/8.14.4/Submit) id o5H7SfUf011219; Thu, 17 Jun 2010 10:28:41 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: deviant.kiev.zoral.com.ua: kostik set sender to kostikbel@gmail.com using -f Date: Thu, 17 Jun 2010 10:28:41 +0300 From: Kostik Belousov To: Dmitry Pryanishnikov Message-ID: <20100617072841.GY13238@deviant.kiev.zoral.com.ua> References: <201006162220.o5GMK3uM002900@freefall.freebsd.org> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="AQDb5/OyJPRJL1y9" Content-Disposition: inline In-Reply-To: <201006162220.o5GMK3uM002900@freefall.freebsd.org> User-Agent: Mutt/1.4.2.3i X-Virus-Scanned: clamav-milter 0.95.2 at skuns.kiev.zoral.com.ua X-Virus-Status: Clean X-Spam-Status: No, score=-2.3 required=5.0 tests=ALL_TRUSTED,AWL,BAYES_50, DNS_FROM_OPENWHOIS autolearn=no version=3.2.5 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on skuns.kiev.zoral.com.ua Cc: freebsd-fs@FreeBSD.org Subject: Re: kern/147890: [ufs] [regression] ufs-related lock problem in RELENG_8 (18.04.2010 -> 20.04.2010) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 17 Jun 2010 07:28:46 -0000 --AQDb5/OyJPRJL1y9 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Wed, Jun 16, 2010 at 10:20:03PM +0000, Dmitry Pryanishnikov wrote: > The following reply was made to PR kern/147890; it has been noted by GNAT= S. >=20 > From: Dmitry Pryanishnikov > To: Bruce Cran > Cc: bug-followup@freebsd.org > Subject: Re: kern/147890: [ufs] [regression] ufs-related lock problem in= =20 > RELENG_8 (18.04.2010 -> 20.04.2010) > Date: Thu, 17 Jun 2010 01:12:17 +0300 >=20 > Hello! > =20 > 2010/6/16 Bruce Cran : > > Could you run the commands shown in > > http://www.freebsd.org/doc/en_US.ISO8859-1/books/developers-handbook/k= erneldebug-deadlocks.html > > to get more information to help debug the problem please? > =20 > Done - I've rebuilt the kernel from the sources as of 20.04.2010 > with the missing debugging options, repeated the problem, turned I suspect you do not. In particular, DEBUG_VFS_LOCKS, DEBUG_LOCKS and WITNESS seems to be missed. > ddb(4) capture buffer on, issued recommended ddb commands, then called > 'doadump'; then rebooted and extracted capture buffer contents from > the crashdump using ddb(8). Due to missed options, we cannot trace the lock chains and see why the issue happens. > =20 > BTW, is there a way to get this information _without repeating the > problem_ - using only a crashdump from it's previous occurrence? All > data required for analysis should already be there. I suspect not > every deadlock is easy repeatable (like this one)... It is, but you have to understand how locks work, and what you are looking for. > =20 > db> ps > =20 > pid ppid pgrp uid state wmesg wchan cmd > 2143 1846 2143 0 S+ ufs 0xcd466dc8 sync > 2131 1 1911 1000 S ufs 0xcd466dc8 initial thread =2E.. > 18 0 0 0 SL ufs 0xcd466dc8 [syncer] =2E... > Tracing command syncer pid 18 tid 100067 td 0xc7383000 > sched_switch(c7383000,0,104,191,e660f504,...) at 0xc05292c0 =3D sched_sw= itch+0x330 > mi_switch(104,0,c0745097,1eb,50,...) at 0xc0510680 =3D mi_switch+0x200 > sleepq_switch(c7383000,0,c0745097,260,50,...) at 0xc054028f =3D > sleepq_switch+0x15f > sleepq_wait(cd466dc8,50,c07393ba,4,0,...) at 0xc0540f63 =3D sleepq_wait+= 0x63 > __lockmgr_args(cd466dc8,80100,cd466e30,0,0,...) at 0xc04f087f =3D > __lockmgr_args+0xb1f > ffs_lock(e732cb34,c054944b,c0763314,80100,cd466d70,...) at 0xc067e9f1 > =3D ffs_lock+0xa1 > VOP_LOCK1_APV(c07a47c0,e732cb34,c73830a4,c07b7a20,cd466d70,...) at > 0xc070c945 =3D VOP_LOCK1_APV+0xb5 > _vn_lock(cd466d70,80100,c074e6ce,82b,4,...) at 0xc05a2fd8 =3D _vn_lock+0= x78 > vget(cd466d70,80100,c7383000,39a,c763b870,...) at 0xc05969db =3D vget+0x= bb > qsync(c763b870,0,c0761f41,552,4,...) at 0xc0687db7 =3D qsync+0x197 > ffs_sync(c763b870,3,c074e6ce,d6b,c763b870,...) at 0xc067a139 =3D ffs_syn= c+0x349 > sync_fsync(e732cc7c,c0772c36,c76b8778,e732cc7c,c76b86b8,...) at > 0xc059812f =3D sync_fsync+0x18f > VOP_FSYNC_APV(c0799bc0,e732cc7c,c074e6ce,6a5,c7383000,...) at > 0xc070a545 =3D VOP_FSYNC_APV+0xc5 > sync_vnode(c093b618,c093b604,3e8,6d4,4e20,...) at 0xc059840b =3D sync_vn= ode+0x16b > sched_sync(0,e732cd38,c073bc0b,343,c759f7f8,...) at 0xc0598753 =3D > sched_sync+0x273 > fork_exit(c05984e0,0,e732cd38) at 0xc04dddf8 =3D fork_exit+0xb8 > fork_trampoline() at 0xc06d6b80 =3D fork_trampoline+0x8 > --- trap 0, eip =3D 0, esp =3D 0xe732cd70, ebp =3D 0 --- Do you have custoom kernel build, in particular, are quotas compiled in ? What are the drives ? Having to pull the generic information about the problematic system does not make debugging faster. --AQDb5/OyJPRJL1y9 Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (FreeBSD) iEYEARECAAYFAkwZzqkACgkQC3+MBN1Mb4h5BACeMbGnVuUNASRzA72BMrr00WVw BegAn3ZcY0q722VOo0G9xItKOttxiuVl =ryzd -----END PGP SIGNATURE----- --AQDb5/OyJPRJL1y9--