From owner-freebsd-threads@FreeBSD.ORG Thu Jul 16 17:51:00 2009 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id CD4EA1065670; Thu, 16 Jul 2009 17:51:00 +0000 (UTC) (envelope-from nick@desert.net) Received: from tao.desert.net (tao.desert.net [209.104.5.188]) by mx1.freebsd.org (Postfix) with ESMTP id A308C8FC1C; Thu, 16 Jul 2009 17:51:00 +0000 (UTC) (envelope-from nick@desert.net) Received: from motoko.vpn.desert.net (motoko.vpn.desert.net [10.24.5.6]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) by tao.desert.net (Postfix) with ESMTP id 5CE7F5D16; Thu, 16 Jul 2009 10:33:03 -0700 (MST) Message-Id: <6E6A9516-6C69-4E41-803C-FE5F126F402C@desert.net> From: Nick Esborn To: Attilio Rao In-Reply-To: <3bbf2fe10907160548l74de896bka609de7a9a994899@mail.gmail.com> Content-Type: multipart/signed; protocol="application/pgp-signature"; micalg=pgp-sha1; boundary="Apple-Mail-24-1022926530" Content-Transfer-Encoding: 7bit Mime-Version: 1.0 (Apple Message framework v935.3) Date: Thu, 16 Jul 2009 10:33:02 -0700 References: <200907152140.n6FLe42l045879@freefall.freebsd.org> <3bbf2fe10907160526l2f066698qce8a5e77aee6366b@mail.gmail.com> <3bbf2fe10907160548l74de896bka609de7a9a994899@mail.gmail.com> X-Pgp-Agent: GPGMail 1.2.0 (v56) X-Mailer: Apple Mail (2.935.3) Cc: freebsd-threads@freebsd.org Subject: Re: threads/136345: Recursive read rwlocks in thread A cause deadlock with write lock in thread B X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 16 Jul 2009 17:51:01 -0000 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --Apple-Mail-24-1022926530 Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes Content-Transfer-Encoding: 7bit On Jul 16, 2009, at 5:48 AM, Attilio Rao wrote: > 2009/7/16 Attilio Rao : >> 2009/7/15 Nick Esborn : >>> The following reply was made to PR threads/136345; it has been >>> noted by GNATS. >>> >>> From: Nick Esborn >>> To: bug-followup@FreeBSD.org, >>> rink@FreeBSD.org >>> Cc: >>> Subject: Re: threads/136345: Recursive read rwlocks in thread A >>> cause deadlock with write lock in thread B >>> Date: Wed, 15 Jul 2009 14:32:38 -0700 >>> >>> This is an OpenPGP/MIME signed message (RFC 2440 and 3156) >>> --Apple-Mail-19-950902279 >>> Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes >>> Content-Transfer-Encoding: 7bit >>> >>> Even after the above patch, I still run into occasional MySQL thread >>> deadlocks, which I originally described in what is now threads/ >>> 135673. >>> >>> I also posted on freebsd-current a few days ago: >>> >>> http://lists.freebsd.org/pipermail/freebsd-current/2009-July/009328.html >>> >>> I'd be happy to collect whatever data would be helpful in tracking >>> down this deadlock. This only seems to happen under our production >>> workload, so that might make it harder to capture meaningful debug >>> data, but I'm certainly willing to try. I can also arrange for >>> developer access to the system in question, if that would help >>> significantly. >> >> So did you backport this to 7 and still experience deadlocks? >> I just committed the fix to HEAD not to STABLE branch. > > Ok, I got, you just upgraded. > Can you try the following things?: > - Upgrade to the -CURRENT of today > - Recompile the kernel with the following options: > KDB, DDB, SCHED_ULE, PREEMPTION, FULL_PREEMPTION, INVARIANT_SUPPORT, > INVARIANTS, WITNESS > - When the deadlock takes place break into DDB and please retrieve the > following info: > db> show allpcpu > db> ps > db> alltrace > db> show alllock > > - Save them with a serial console output or using the textdump(4) > format. > > (if necessary read the ddb(4) and textdump(4) before to set up the > whole system). > This would shade a light if the problem lives within the kernel or > not. > > Thanks, > Attilio > > > -- > Peace can only be achieved by understanding - A. Einstein I can definitely do the upgrade. KDB, DDB, SCHED_ULE, and PREEMPTION are already turned on. I will try FULL_PREEMPTION, INVARIANT_SUPPORT, INVARIANTS, and WITNESS, but when I first upgraded to 8.0, this server was unable to handle its workload with the INVARIANTS and WITNESS options turned on. Also, it can take a while for it to become clear that the deadlock has occurred -- usually our monitoring picks it up when replication falls behind. So it may be 15-20 minutes after the deadlock that I am able to run the above db commands. Of course the thread will still be deadlocked. Hopefully that doesn't reduce the value of the data obtained. Thanks, -nick -- nick@desert.net - all messages cryptographically signed --Apple-Mail-24-1022926530 content-type: application/pgp-signature; x-mac-type=70674453; name=PGP.sig content-description: This is a digitally signed message part content-disposition: inline; filename=PGP.sig content-transfer-encoding: 7bit -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (Darwin) iEYEARECAAYFAkpfZE4ACgkQw1bX5UNr2ACFZACdHIvw3MJUMXNBqE3oktQ5dfg0 AJwAnjYZJ0WpC1lix1ZceWa6hTfNg5YY =6IoP -----END PGP SIGNATURE----- --Apple-Mail-24-1022926530--