From owner-freebsd-threads@FreeBSD.ORG Mon Jul 13 11:07:09 2009 Return-Path: Delivered-To: freebsd-threads@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4C01F106566C for ; Mon, 13 Jul 2009 11:07:09 +0000 (UTC) (envelope-from owner-bugmaster@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 382118FC14 for ; Mon, 13 Jul 2009 11:07:09 +0000 (UTC) (envelope-from owner-bugmaster@FreeBSD.org) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.3/8.14.3) with ESMTP id n6DB79bQ040811 for ; Mon, 13 Jul 2009 11:07:09 GMT (envelope-from owner-bugmaster@FreeBSD.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.3/8.14.3/Submit) id n6DB78oB040807 for freebsd-threads@FreeBSD.org; Mon, 13 Jul 2009 11:07:08 GMT (envelope-from owner-bugmaster@FreeBSD.org) Date: Mon, 13 Jul 2009 11:07:08 GMT Message-Id: <200907131107.n6DB78oB040807@freefall.freebsd.org> X-Authentication-Warning: freefall.freebsd.org: gnats set sender to owner-bugmaster@FreeBSD.org using -f From: FreeBSD bugmaster To: freebsd-threads@FreeBSD.org Cc: Subject: Current problem reports assigned to freebsd-threads@FreeBSD.org X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 13 Jul 2009 11:07:09 -0000 Note: to view an individual PR, use: http://www.freebsd.org/cgi/query-pr.cgi?pr=(number). The following is a listing of current problems submitted by FreeBSD users. These represent problem reports covering all versions including experimental development code and obsolete releases. S Tracker Resp. Description -------------------------------------------------------------------------------- o threa/136345 threads Recursive read rwlocks in thread A cause deadlock with o threa/135462 threads [PATCH] _thread_cleanupspecific() doesn't handle delet o threa/133734 threads 32 bit libthr failing pthread_create() o threa/128922 threads threads hang with xorg running o threa/127225 threads bug in lib/libthr/thread/thr_init.c o threa/122923 threads 'nice' does not prevent background process from steali o threa/121336 threads lang/neko threading ok on UP, broken on SMP (FreeBSD 7 o threa/118715 threads kse problem o threa/116668 threads can no longer use jdk15 with libthr on -stable SMP o threa/116181 threads /dev/io-related io access permissions are not propagat o threa/115211 threads pthread_atfork misbehaves in initial thread o threa/110636 threads [request] gdb(1): using gdb with multi thread applicat o threa/110306 threads apache 2.0 segmentation violation when calling gethost o threa/103975 threads Implicit loading/unloading of libpthread.so may crash o threa/101323 threads [patch] fork(2) in threaded programs broken. s threa/100815 threads FBSD 5.5 broke nanosleep in libc_r s threa/94467 threads send(), sendto() and sendmsg() are not correct in libc s threa/84483 threads problems with devel/nspr and -lc_r on 4.x o threa/83914 threads [libc] popen() doesn't work in static threaded program o threa/80992 threads abort() sometimes not caught by gdb depending on threa o threa/80435 threads panic on high loads o threa/79887 threads [patch] freopen() isn't thread-safe o threa/79683 threads svctcp_create() fails if multiple threads call at the s threa/76694 threads fork cause hang in dup()/close() function in child (-l s threa/76690 threads fork hang in child for -lc_r o threa/75374 threads pthread_kill() ignores SA_SIGINFO flag o threa/75273 threads FBSD 5.3 libpthread (KSE) bug o threa/72953 threads fork() unblocks blocked signals w/o PTHREAD_SCOPE_SYST o threa/70975 threads [sysvipc] unexpected and unreliable behaviour when usi s threa/69020 threads pthreads library leaks _gc_mutex s threa/49087 threads Signals lost in programs linked with libc_r s threa/48856 threads Setting SIGCHLD to SIG_IGN still leaves zombies under s threa/40671 threads pthread_cancel doesn't remove thread from condition qu s threa/39922 threads [threads] [patch] Threaded applications executed with s threa/37676 threads libc_r: msgsnd(), msgrcv(), pread(), pwrite() need wra s threa/34536 threads accept() blocks other threads s threa/32295 threads [libc_r] [patch] pthread(3) dont dequeue signals s threa/30464 threads pthread mutex attributes -- pshared s threa/24632 threads libc_r delicate deviation from libc in handling SIGCHL s threa/24472 threads libc_r does not honor SO_SNDTIMEO/SO_RCVTIMEO socket o 40 problems total. From owner-freebsd-threads@FreeBSD.ORG Tue Jul 14 07:54:02 2009 Return-Path: Delivered-To: freebsd-threads@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A94E9106564A; Tue, 14 Jul 2009 07:54:02 +0000 (UTC) (envelope-from ale@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 7E3AF8FC1B; Tue, 14 Jul 2009 07:54:02 +0000 (UTC) (envelope-from ale@FreeBSD.org) Received: from freefall.freebsd.org (ale@localhost [127.0.0.1]) by freefall.freebsd.org (8.14.3/8.14.3) with ESMTP id n6E7s2oU058868; Tue, 14 Jul 2009 07:54:02 GMT (envelope-from ale@freefall.freebsd.org) Received: (from ale@localhost) by freefall.freebsd.org (8.14.3/8.14.3/Submit) id n6E7s22L058864; Tue, 14 Jul 2009 07:54:02 GMT (envelope-from ale) Date: Tue, 14 Jul 2009 07:54:02 GMT Message-Id: <200907140754.n6E7s22L058864@freefall.freebsd.org> To: ale@FreeBSD.org, ale@FreeBSD.org, freebsd-threads@FreeBSD.org From: ale@FreeBSD.org Cc: Subject: Re: threads/135673: databases/mysql50-server - MySQL query lock-ups on 7.2-RELEASE amd64 X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 Jul 2009 07:54:02 -0000 Synopsis: databases/mysql50-server - MySQL query lock-ups on 7.2-RELEASE amd64 Responsible-Changed-From-To: ale->freebsd-threads Responsible-Changed-By: ale Responsible-Changed-When: Tue Jul 14 07:52:43 UTC 2009 Responsible-Changed-Why: FreeBSD's threads problem. http://www.freebsd.org/cgi/query-pr.cgi?pr=135673 From owner-freebsd-threads@FreeBSD.ORG Wed Jul 15 21:40:04 2009 Return-Path: Delivered-To: freebsd-threads@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4122310656DC for ; Wed, 15 Jul 2009 21:40:04 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 2F2DD8FC28 for ; Wed, 15 Jul 2009 21:40:04 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (gnats@localhost [127.0.0.1]) by freefall.freebsd.org (8.14.3/8.14.3) with ESMTP id n6FLe4S9045880 for ; Wed, 15 Jul 2009 21:40:04 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.3/8.14.3/Submit) id n6FLe42l045879; Wed, 15 Jul 2009 21:40:04 GMT (envelope-from gnats) Date: Wed, 15 Jul 2009 21:40:04 GMT Message-Id: <200907152140.n6FLe42l045879@freefall.freebsd.org> To: freebsd-threads@FreeBSD.org From: Nick Esborn Cc: Subject: Re: threads/136345: Recursive read rwlocks in thread A cause deadlock with write lock in thread B X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Nick Esborn List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 15 Jul 2009 21:40:04 -0000 The following reply was made to PR threads/136345; it has been noted by GNATS. From: Nick Esborn To: bug-followup@FreeBSD.org, rink@FreeBSD.org Cc: Subject: Re: threads/136345: Recursive read rwlocks in thread A cause deadlock with write lock in thread B Date: Wed, 15 Jul 2009 14:32:38 -0700 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --Apple-Mail-19-950902279 Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes Content-Transfer-Encoding: 7bit Even after the above patch, I still run into occasional MySQL thread deadlocks, which I originally described in what is now threads/135673. I also posted on freebsd-current a few days ago: http://lists.freebsd.org/pipermail/freebsd-current/2009-July/009328.html I'd be happy to collect whatever data would be helpful in tracking down this deadlock. This only seems to happen under our production workload, so that might make it harder to capture meaningful debug data, but I'm certainly willing to try. I can also arrange for developer access to the system in question, if that would help significantly. -nick -- nick@desert.net - all messages cryptographically signed --Apple-Mail-19-950902279 content-type: application/pgp-signature; x-mac-type=70674453; name=PGP.sig content-description: This is a digitally signed message part content-disposition: inline; filename=PGP.sig content-transfer-encoding: 7bit -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (Darwin) iEYEARECAAYFAkpeSvYACgkQw1bX5UNr2ABajACeNpj/MW4X+4zfvlWNCXnqo6D9 EZkAoLpGHxs4RoHMd7yqyba4IPzKxsJh =5I4q -----END PGP SIGNATURE----- --Apple-Mail-19-950902279-- From owner-freebsd-threads@FreeBSD.ORG Thu Jul 16 12:26:50 2009 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B3DD51065678 for ; Thu, 16 Jul 2009 12:26:50 +0000 (UTC) (envelope-from asmrookie@gmail.com) Received: from mail-bw0-f208.google.com (mail-bw0-f208.google.com [209.85.218.208]) by mx1.freebsd.org (Postfix) with ESMTP id 3AD968FC19 for ; Thu, 16 Jul 2009 12:26:50 +0000 (UTC) (envelope-from asmrookie@gmail.com) Received: by bwz4 with SMTP id 4so67940bwz.43 for ; Thu, 16 Jul 2009 05:26:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:sender:received:in-reply-to :references:date:x-google-sender-auth:message-id:subject:from:to:cc :content-type:content-transfer-encoding; bh=NPkQaOCM6nbQyXaALr2+gBr1uypJCBa02tOcN8hiPGk=; b=HPuYYBL9QwQSZiDO5xYDmObARbgIJ/JBhXRLOyCYyBjw1iW5QqBzRvHHg7qilwpwHv Sdu9X8Lfi8+M7i+kJTf/nGVVL4BvxyFchSojDe52CTRs1lTGAkvIt4LzlDU9V9wONoY5 p47KAHs5PtlNro07bp3hEO9F4t5iyvbb14iQY= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type :content-transfer-encoding; b=RiCzQnwETTGHyMAaZ8UKaE/BNMG2/eH92ZDwEHulimrby2hgU1oR+L7yUjl/jKo4TO UCKj5o4Ne9ENGRW6uz+xQ9TVyQc2YDcr+CvxkCs5hGHsudg0l8EDPQ9uBICumBiSePmh j59xBbuQtddyYdY/jDKKhIMFLCYhs7U8R2DLU= MIME-Version: 1.0 Sender: asmrookie@gmail.com Received: by 10.223.114.74 with SMTP id d10mr4393861faq.87.1247747209207; Thu, 16 Jul 2009 05:26:49 -0700 (PDT) In-Reply-To: <200907152140.n6FLe42l045879@freefall.freebsd.org> References: <200907152140.n6FLe42l045879@freefall.freebsd.org> Date: Thu, 16 Jul 2009 14:26:49 +0200 X-Google-Sender-Auth: b03de7c3679a91c1 Message-ID: <3bbf2fe10907160526l2f066698qce8a5e77aee6366b@mail.gmail.com> From: Attilio Rao To: Nick Esborn Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Cc: freebsd-threads@freebsd.org Subject: Re: threads/136345: Recursive read rwlocks in thread A cause deadlock with write lock in thread B X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 16 Jul 2009 12:26:51 -0000 2009/7/15 Nick Esborn : > The following reply was made to PR threads/136345; it has been noted by GNATS. > > From: Nick Esborn > To: bug-followup@FreeBSD.org, > rink@FreeBSD.org > Cc: > Subject: Re: threads/136345: Recursive read rwlocks in thread A cause deadlock with write lock in thread B > Date: Wed, 15 Jul 2009 14:32:38 -0700 > > This is an OpenPGP/MIME signed message (RFC 2440 and 3156) > --Apple-Mail-19-950902279 > Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes > Content-Transfer-Encoding: 7bit > > Even after the above patch, I still run into occasional MySQL thread > deadlocks, which I originally described in what is now threads/135673. > > I also posted on freebsd-current a few days ago: > > http://lists.freebsd.org/pipermail/freebsd-current/2009-July/009328.html > > I'd be happy to collect whatever data would be helpful in tracking > down this deadlock. This only seems to happen under our production > workload, so that might make it harder to capture meaningful debug > data, but I'm certainly willing to try. I can also arrange for > developer access to the system in question, if that would help > significantly. So did you backport this to 7 and still experience deadlocks? I just committed the fix to HEAD not to STABLE branch. Thanks, Attilio -- Peace can only be achieved by understanding - A. Einstein From owner-freebsd-threads@FreeBSD.ORG Thu Jul 16 12:48:13 2009 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 20EDB106566C for ; Thu, 16 Jul 2009 12:48:13 +0000 (UTC) (envelope-from asmrookie@gmail.com) Received: from mail-bw0-f208.google.com (mail-bw0-f208.google.com [209.85.218.208]) by mx1.freebsd.org (Postfix) with ESMTP id 9568A8FC23 for ; Thu, 16 Jul 2009 12:48:12 +0000 (UTC) (envelope-from asmrookie@gmail.com) Received: by bwz4 with SMTP id 4so80292bwz.43 for ; Thu, 16 Jul 2009 05:48:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:sender:received:in-reply-to :references:date:x-google-sender-auth:message-id:subject:from:to:cc :content-type:content-transfer-encoding; bh=wY56AJBfyAiCqc0CQTx7Qp6BCJaTK8yyEniaDZRi1SM=; b=SggtN1H2LDDx/0pRerTX2MTgVZTHDrYy6U0CyWaNgmigmlwshGkIRAOJAYLQiGIUjA l8eLD7UiM8sMllwsV5Vx+VCOKfOLrNDG5JSMkW3JdTwet7o+ZwjoOR3E29IZ9C2zEqhC Pyg37WzvZlBDfhBzbN6roKVUGtUd2tktVTVXc= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type :content-transfer-encoding; b=GlfVm+UPZcNbQyf9JGvmKKLmGQ8xsdbLKwoodHURsmt8BRBLyIKZWyFALhlQVf0J3u hnrS4sNt7P7YVX0OYCZfqtK6eQssB/cVq9uR10JfCbW8kWcZnximirZLe7WHSwTPKr91 U9FWedYjynN7xXP0G0okyyWUqLS96Rqg9QT9M= MIME-Version: 1.0 Sender: asmrookie@gmail.com Received: by 10.223.107.9 with SMTP id z9mr4432003fao.1.1247748491333; Thu, 16 Jul 2009 05:48:11 -0700 (PDT) In-Reply-To: <3bbf2fe10907160526l2f066698qce8a5e77aee6366b@mail.gmail.com> References: <200907152140.n6FLe42l045879@freefall.freebsd.org> <3bbf2fe10907160526l2f066698qce8a5e77aee6366b@mail.gmail.com> Date: Thu, 16 Jul 2009 14:48:11 +0200 X-Google-Sender-Auth: fb69cdd58ecf0064 Message-ID: <3bbf2fe10907160548l74de896bka609de7a9a994899@mail.gmail.com> From: Attilio Rao To: Nick Esborn Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Cc: freebsd-threads@freebsd.org Subject: Re: threads/136345: Recursive read rwlocks in thread A cause deadlock with write lock in thread B X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 16 Jul 2009 12:48:13 -0000 2009/7/16 Attilio Rao : > 2009/7/15 Nick Esborn : >> The following reply was made to PR threads/136345; it has been noted by GNATS. >> >> From: Nick Esborn >> To: bug-followup@FreeBSD.org, >> rink@FreeBSD.org >> Cc: >> Subject: Re: threads/136345: Recursive read rwlocks in thread A cause deadlock with write lock in thread B >> Date: Wed, 15 Jul 2009 14:32:38 -0700 >> >> This is an OpenPGP/MIME signed message (RFC 2440 and 3156) >> --Apple-Mail-19-950902279 >> Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes >> Content-Transfer-Encoding: 7bit >> >> Even after the above patch, I still run into occasional MySQL thread >> deadlocks, which I originally described in what is now threads/135673. >> >> I also posted on freebsd-current a few days ago: >> >> http://lists.freebsd.org/pipermail/freebsd-current/2009-July/009328.html >> >> I'd be happy to collect whatever data would be helpful in tracking >> down this deadlock. This only seems to happen under our production >> workload, so that might make it harder to capture meaningful debug >> data, but I'm certainly willing to try. I can also arrange for >> developer access to the system in question, if that would help >> significantly. > > So did you backport this to 7 and still experience deadlocks? > I just committed the fix to HEAD not to STABLE branch. Ok, I got, you just upgraded. Can you try the following things?: - Upgrade to the -CURRENT of today - Recompile the kernel with the following options: KDB, DDB, SCHED_ULE, PREEMPTION, FULL_PREEMPTION, INVARIANT_SUPPORT, INVARIANTS, WITNESS - When the deadlock takes place break into DDB and please retrieve the following info: db> show allpcpu db> ps db> alltrace db> show alllock - Save them with a serial console output or using the textdump(4) format. (if necessary read the ddb(4) and textdump(4) before to set up the whole system). This would shade a light if the problem lives within the kernel or not. Thanks, Attilio -- Peace can only be achieved by understanding - A. Einstein From owner-freebsd-threads@FreeBSD.ORG Thu Jul 16 17:51:00 2009 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id CD4EA1065670; Thu, 16 Jul 2009 17:51:00 +0000 (UTC) (envelope-from nick@desert.net) Received: from tao.desert.net (tao.desert.net [209.104.5.188]) by mx1.freebsd.org (Postfix) with ESMTP id A308C8FC1C; Thu, 16 Jul 2009 17:51:00 +0000 (UTC) (envelope-from nick@desert.net) Received: from motoko.vpn.desert.net (motoko.vpn.desert.net [10.24.5.6]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) by tao.desert.net (Postfix) with ESMTP id 5CE7F5D16; Thu, 16 Jul 2009 10:33:03 -0700 (MST) Message-Id: <6E6A9516-6C69-4E41-803C-FE5F126F402C@desert.net> From: Nick Esborn To: Attilio Rao In-Reply-To: <3bbf2fe10907160548l74de896bka609de7a9a994899@mail.gmail.com> Content-Type: multipart/signed; protocol="application/pgp-signature"; micalg=pgp-sha1; boundary="Apple-Mail-24-1022926530" Content-Transfer-Encoding: 7bit Mime-Version: 1.0 (Apple Message framework v935.3) Date: Thu, 16 Jul 2009 10:33:02 -0700 References: <200907152140.n6FLe42l045879@freefall.freebsd.org> <3bbf2fe10907160526l2f066698qce8a5e77aee6366b@mail.gmail.com> <3bbf2fe10907160548l74de896bka609de7a9a994899@mail.gmail.com> X-Pgp-Agent: GPGMail 1.2.0 (v56) X-Mailer: Apple Mail (2.935.3) Cc: freebsd-threads@freebsd.org Subject: Re: threads/136345: Recursive read rwlocks in thread A cause deadlock with write lock in thread B X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 16 Jul 2009 17:51:01 -0000 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --Apple-Mail-24-1022926530 Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes Content-Transfer-Encoding: 7bit On Jul 16, 2009, at 5:48 AM, Attilio Rao wrote: > 2009/7/16 Attilio Rao : >> 2009/7/15 Nick Esborn : >>> The following reply was made to PR threads/136345; it has been >>> noted by GNATS. >>> >>> From: Nick Esborn >>> To: bug-followup@FreeBSD.org, >>> rink@FreeBSD.org >>> Cc: >>> Subject: Re: threads/136345: Recursive read rwlocks in thread A >>> cause deadlock with write lock in thread B >>> Date: Wed, 15 Jul 2009 14:32:38 -0700 >>> >>> This is an OpenPGP/MIME signed message (RFC 2440 and 3156) >>> --Apple-Mail-19-950902279 >>> Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes >>> Content-Transfer-Encoding: 7bit >>> >>> Even after the above patch, I still run into occasional MySQL thread >>> deadlocks, which I originally described in what is now threads/ >>> 135673. >>> >>> I also posted on freebsd-current a few days ago: >>> >>> http://lists.freebsd.org/pipermail/freebsd-current/2009-July/009328.html >>> >>> I'd be happy to collect whatever data would be helpful in tracking >>> down this deadlock. This only seems to happen under our production >>> workload, so that might make it harder to capture meaningful debug >>> data, but I'm certainly willing to try. I can also arrange for >>> developer access to the system in question, if that would help >>> significantly. >> >> So did you backport this to 7 and still experience deadlocks? >> I just committed the fix to HEAD not to STABLE branch. > > Ok, I got, you just upgraded. > Can you try the following things?: > - Upgrade to the -CURRENT of today > - Recompile the kernel with the following options: > KDB, DDB, SCHED_ULE, PREEMPTION, FULL_PREEMPTION, INVARIANT_SUPPORT, > INVARIANTS, WITNESS > - When the deadlock takes place break into DDB and please retrieve the > following info: > db> show allpcpu > db> ps > db> alltrace > db> show alllock > > - Save them with a serial console output or using the textdump(4) > format. > > (if necessary read the ddb(4) and textdump(4) before to set up the > whole system). > This would shade a light if the problem lives within the kernel or > not. > > Thanks, > Attilio > > > -- > Peace can only be achieved by understanding - A. Einstein I can definitely do the upgrade. KDB, DDB, SCHED_ULE, and PREEMPTION are already turned on. I will try FULL_PREEMPTION, INVARIANT_SUPPORT, INVARIANTS, and WITNESS, but when I first upgraded to 8.0, this server was unable to handle its workload with the INVARIANTS and WITNESS options turned on. Also, it can take a while for it to become clear that the deadlock has occurred -- usually our monitoring picks it up when replication falls behind. So it may be 15-20 minutes after the deadlock that I am able to run the above db commands. Of course the thread will still be deadlocked. Hopefully that doesn't reduce the value of the data obtained. Thanks, -nick -- nick@desert.net - all messages cryptographically signed --Apple-Mail-24-1022926530 content-type: application/pgp-signature; x-mac-type=70674453; name=PGP.sig content-description: This is a digitally signed message part content-disposition: inline; filename=PGP.sig content-transfer-encoding: 7bit -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (Darwin) iEYEARECAAYFAkpfZE4ACgkQw1bX5UNr2ACFZACdHIvw3MJUMXNBqE3oktQ5dfg0 AJwAnjYZJ0WpC1lix1ZceWa6hTfNg5YY =6IoP -----END PGP SIGNATURE----- --Apple-Mail-24-1022926530-- From owner-freebsd-threads@FreeBSD.ORG Thu Jul 16 17:53:29 2009 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 11BAD1065670 for ; Thu, 16 Jul 2009 17:53:29 +0000 (UTC) (envelope-from asmrookie@gmail.com) Received: from mail-bw0-f208.google.com (mail-bw0-f208.google.com [209.85.218.208]) by mx1.freebsd.org (Postfix) with ESMTP id 8B2B58FC1C for ; Thu, 16 Jul 2009 17:53:28 +0000 (UTC) (envelope-from asmrookie@gmail.com) Received: by bwz4 with SMTP id 4so254180bwz.43 for ; Thu, 16 Jul 2009 10:53:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:sender:received:in-reply-to :references:date:x-google-sender-auth:message-id:subject:from:to:cc :content-type:content-transfer-encoding; bh=n3WxjeGPOcSsQsrsnvnZgMS4J03kvQaQH6Va76XXRaw=; b=iDDy2Gb2dkbUMo9ZIGRDvONfFNvJBRQ9GSLDkPsO7bUqawPTJ0CpJnBGiFISpiWKQz xzBCGlusV0XZMuG/PLwwLZi+AGN2cwTZKc22uNfS65vYJoW8NDaZqWrJsmCLgHI7pv2q Fy/KvaStAPT8k/5vIIx0FwnkHOmTfaGHmPctY= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type :content-transfer-encoding; b=aCXFCd79gpc8pbzZo//jC8pEEv67hEpOSshQXc1lGGXy1IyXiUrRu/BJw8221z48wW aQT/qSzI8EkzGPvfCDVZUVm+LSMgbDaEfzloVyGpm47JStDnwTdzKGy7t9R5N80BYKZJ lBgijekzvLwa7kVkoetd4LOxDAur29TbuUYTg= MIME-Version: 1.0 Sender: asmrookie@gmail.com Received: by 10.223.123.141 with SMTP id p13mr2932far.84.1247766807583; Thu, 16 Jul 2009 10:53:27 -0700 (PDT) In-Reply-To: <6E6A9516-6C69-4E41-803C-FE5F126F402C@desert.net> References: <200907152140.n6FLe42l045879@freefall.freebsd.org> <3bbf2fe10907160526l2f066698qce8a5e77aee6366b@mail.gmail.com> <3bbf2fe10907160548l74de896bka609de7a9a994899@mail.gmail.com> <6E6A9516-6C69-4E41-803C-FE5F126F402C@desert.net> Date: Thu, 16 Jul 2009 19:53:27 +0200 X-Google-Sender-Auth: 35d45401fba492c7 Message-ID: <3bbf2fe10907161053x3b6aa60dneb8dbd5217b4cb03@mail.gmail.com> From: Attilio Rao To: Nick Esborn Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Cc: freebsd-threads@freebsd.org Subject: Re: threads/136345: Recursive read rwlocks in thread A cause deadlock with write lock in thread B X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 16 Jul 2009 17:53:29 -0000 2009/7/16 Nick Esborn : > > > KDB, DDB, SCHED_ULE, and PREEMPTION are already turned on. I will try > FULL_PREEMPTION, INVARIANT_SUPPORT, INVARIANTS, and WITNESS, but when I > first upgraded to 8.0, this server was unable to handle its workload with > the INVARIANTS and WITNESS options turned on. What do you mean with 'unable'? What was happening precisely? > Also, it can take a while for it to become clear that the deadlock has > occurred -- usually our monitoring picks it up when replication falls > behind. So it may be 15-20 minutes after the deadlock that I am able to run > the above db commands. Of course the thread will still be deadlocked. > Hopefully that doesn't reduce the value of the data obtained. It should be still fine. Thanks, Attilio -- Peace can only be achieved by understanding - A. Einstein From owner-freebsd-threads@FreeBSD.ORG Thu Jul 16 17:56:58 2009 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 62A18106566B; Thu, 16 Jul 2009 17:56:58 +0000 (UTC) (envelope-from nick@desert.net) Received: from tao.desert.net (tao.desert.net [209.104.5.188]) by mx1.freebsd.org (Postfix) with ESMTP id 37D4C8FC14; Thu, 16 Jul 2009 17:56:58 +0000 (UTC) (envelope-from nick@desert.net) Received: from motoko.vpn.desert.net (motoko.vpn.desert.net [10.24.5.6]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) by tao.desert.net (Postfix) with ESMTP id 98FA65D16; Thu, 16 Jul 2009 10:56:57 -0700 (MST) Message-Id: From: Nick Esborn To: Attilio Rao In-Reply-To: <3bbf2fe10907161053x3b6aa60dneb8dbd5217b4cb03@mail.gmail.com> Content-Type: multipart/signed; protocol="application/pgp-signature"; micalg=pgp-sha1; boundary="Apple-Mail-25-1024360403" Content-Transfer-Encoding: 7bit Mime-Version: 1.0 (Apple Message framework v935.3) Date: Thu, 16 Jul 2009 10:56:55 -0700 References: <200907152140.n6FLe42l045879@freefall.freebsd.org> <3bbf2fe10907160526l2f066698qce8a5e77aee6366b@mail.gmail.com> <3bbf2fe10907160548l74de896bka609de7a9a994899@mail.gmail.com> <6E6A9516-6C69-4E41-803C-FE5F126F402C@desert.net> <3bbf2fe10907161053x3b6aa60dneb8dbd5217b4cb03@mail.gmail.com> X-Pgp-Agent: GPGMail 1.2.0 (v56) X-Mailer: Apple Mail (2.935.3) Cc: freebsd-threads@freebsd.org Subject: Re: threads/136345: Recursive read rwlocks in thread A cause deadlock with write lock in thread B X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 16 Jul 2009 17:56:58 -0000 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --Apple-Mail-25-1024360403 Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes Content-Transfer-Encoding: 7bit On Jul 16, 2009, at 10:53 AM, Attilio Rao wrote: > 2009/7/16 Nick Esborn : >> >> >> KDB, DDB, SCHED_ULE, and PREEMPTION are already turned on. I will >> try >> FULL_PREEMPTION, INVARIANT_SUPPORT, INVARIANTS, and WITNESS, but >> when I >> first upgraded to 8.0, this server was unable to handle its >> workload with >> the INVARIANTS and WITNESS options turned on. > > What do you mean with 'unable'? What was happening precisely? System time would rise during periods of peak demand, and the system would quickly fall behind on its workload of queries. However, I have some hardware I can dedicate to this, and only run the one MySQL data set which exhibits this problem. That should be enough of a workload reduction to allow the server to keep up even with all the above options turned on. -nick > >> Also, it can take a while for it to become clear that the deadlock >> has >> occurred -- usually our monitoring picks it up when replication falls >> behind. So it may be 15-20 minutes after the deadlock that I am >> able to run >> the above db commands. Of course the thread will still be >> deadlocked. >> Hopefully that doesn't reduce the value of the data obtained. > > It should be still fine. > > Thanks, > Attilio > > > -- > Peace can only be achieved by understanding - A. Einstein -- nick@desert.net - all messages cryptographically signed --Apple-Mail-25-1024360403 content-type: application/pgp-signature; x-mac-type=70674453; name=PGP.sig content-description: This is a digitally signed message part content-disposition: inline; filename=PGP.sig content-transfer-encoding: 7bit -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (Darwin) iEYEARECAAYFAkpfaegACgkQw1bX5UNr2AB6qACgvzbhe6F7Csn7uHlsREMn4V/X YT0AniKg3hfVQr+ehpZ5+LUxAihxVGpm =zL/+ -----END PGP SIGNATURE----- --Apple-Mail-25-1024360403--