From owner-freebsd-stable@FreeBSD.ORG Thu Mar 19 16:02:56 2009 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 83F8C106566C; Thu, 19 Mar 2009 16:02:56 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from mail.terabit.net.ua (mail.terabit.net.ua [195.137.202.147]) by mx1.freebsd.org (Postfix) with ESMTP id 2591E8FC12; Thu, 19 Mar 2009 16:02:55 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from skuns.zoral.com.ua ([91.193.166.194] helo=mail.zoral.com.ua) by mail.terabit.net.ua with esmtps (TLSv1:AES256-SHA:256) (Exim 4.63 (FreeBSD)) (envelope-from ) id 1LkKhq-0003pS-9j; Thu, 19 Mar 2009 18:02:54 +0200 Received: from deviant.kiev.zoral.com.ua (root@deviant.kiev.zoral.com.ua [10.1.1.148]) by mail.zoral.com.ua (8.14.2/8.14.2) with ESMTP id n2JG2pj4017480 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Thu, 19 Mar 2009 18:02:51 +0200 (EET) (envelope-from kostikbel@gmail.com) Received: from deviant.kiev.zoral.com.ua (kostik@localhost [127.0.0.1]) by deviant.kiev.zoral.com.ua (8.14.3/8.14.3) with ESMTP id n2JG2pjd015512; Thu, 19 Mar 2009 18:02:51 +0200 (EET) (envelope-from kostikbel@gmail.com) Received: (from kostik@localhost) by deviant.kiev.zoral.com.ua (8.14.3/8.14.3/Submit) id n2JG2pog015493; Thu, 19 Mar 2009 18:02:51 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: deviant.kiev.zoral.com.ua: kostik set sender to kostikbel@gmail.com using -f Date: Thu, 19 Mar 2009 18:02:51 +0200 From: Kostik Belousov To: John Baldwin Message-ID: <20090319160251.GJ7716@deviant.kiev.zoral.com.ua> References: <200903191001.44491.jhb@freebsd.org> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="XigHxYirkHk2Kxsx" Content-Disposition: inline In-Reply-To: <200903191001.44491.jhb@freebsd.org> User-Agent: Mutt/1.4.2.3i X-Virus-Scanned: ClamAV version 0.94.2, clamav-milter version 0.94.2 on skuns.kiev.zoral.com.ua X-Virus-Status: Clean X-Spam-Status: No, score=-4.4 required=5.0 tests=ALL_TRUSTED,AWL,BAYES_00 autolearn=ham version=3.2.5 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on skuns.kiev.zoral.com.ua X-Virus-Scanned: mail.terabit.net.ua 1LkKhq-0003pS-9j ade5638aca6ad4f7c42d7993a7234cc5 X-Terabit: YES Cc: freebsd-stable@freebsd.org Subject: Re: in recent 7-STABLE: VOP_WRITE...is not exclusive locked but should be X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 19 Mar 2009 16:02:57 -0000 --XigHxYirkHk2Kxsx Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Thu, Mar 19, 2009 at 10:01:44AM -0400, John Baldwin wrote: > On Thursday 19 March 2009 8:05:34 am Tim Chase wrote: > > Hello, > >=20 > > I have a system that had been running quite well with an oldish 7-STABLE > > (from around August 7, 2008) but has started deadlocking within the past > > week or so. > >=20 > > I updated the kernel to a newer 7-STABLE (Mar 15, 2009) and enabled > > INVARIANTS, INVARIANT_SUPPORT, WITNESS, DEBUG_LOCKS DEBUG_VFS_LOCKS and > > DIAGNOSTIC and the message indicated in the subject line has now appear= ed > > 3 times as shown below. Is this something to be terribly concerned abo= ut? > > Is there anything I can to to further track down the cause? Since the > > system is a production mail server, I have it set to not drop into DDB > > when this happens. > >=20 > > The machine is a 4-core Xeon X5450 with 8G of RAM running FreeBSD > > amd64 and in userland it's pretty much just cyrus imapd and apache/php. > > The file systems are all ZFS on a bunch of SAS drives connected to a > > LSI Logic 1068 controller. > >=20 > > As to the deadlock that started this exercise, if the machine follows i= ts > > recent pattern, that should happen within the next 2-4 hours. >=20 > Err, the vn_write() routine should be using an exclusive vnode lock: >=20 > vn_write() > { >=20 > ... > vn_lock(vp, LK_EXCLUSIVE | LK_RETRY, td); > if ((flags & FOF_OFFSET) =3D=3D 0) > uio->uio_offset =3D fp->f_offset; > ioflag |=3D sequential_heuristic(uio, fp); > #ifdef MAC > error =3D mac_check_vnode_write(active_cred, fp->f_cred, vp); > if (error =3D=3D 0) > #endif > error =3D VOP_WRITE(vp, uio, ioflag, fp->f_cred); > ... > } >=20 > Can you check your /sys/kern/vfs_vnops.c and verify that LK_EXCLUSIVE is= =20 > present in your vn_write() routine? If so, then perhaps run memtest? Note that this happens on the ZFS. Might be, ZFS unlocks the vnode and then relocks it with some invalid flags ? --XigHxYirkHk2Kxsx Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (FreeBSD) iEYEARECAAYFAknCbKoACgkQC3+MBN1Mb4hjlgCaAp+sXu+hS5igKIyZar0Gl0oi YOMAoNne2paSwO+3Yb2ul98wSxAbu6+q =0vTG -----END PGP SIGNATURE----- --XigHxYirkHk2Kxsx--