From owner-freebsd-threads@FreeBSD.ORG Mon Jul 2 11:08:58 2007 Return-Path: X-Original-To: freebsd-threads@FreeBSD.org Delivered-To: freebsd-threads@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 97F9116A488 for ; Mon, 2 Jul 2007 11:08:58 +0000 (UTC) (envelope-from owner-bugmaster@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [69.147.83.40]) by mx1.freebsd.org (Postfix) with ESMTP id 7A79513C480 for ; Mon, 2 Jul 2007 11:08:58 +0000 (UTC) (envelope-from owner-bugmaster@FreeBSD.org) Received: from freefall.freebsd.org (gnats@localhost [127.0.0.1]) by freefall.freebsd.org (8.13.4/8.13.4) with ESMTP id l62B8wXt082834 for ; Mon, 2 Jul 2007 11:08:58 GMT (envelope-from owner-bugmaster@FreeBSD.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.13.4/8.13.4/Submit) id l62B8vmk082830 for freebsd-threads@FreeBSD.org; Mon, 2 Jul 2007 11:08:57 GMT (envelope-from owner-bugmaster@FreeBSD.org) Date: Mon, 2 Jul 2007 11:08:57 GMT Message-Id: <200707021108.l62B8vmk082830@freefall.freebsd.org> X-Authentication-Warning: freefall.freebsd.org: gnats set sender to owner-bugmaster@FreeBSD.org using -f From: FreeBSD bugmaster To: freebsd-threads@FreeBSD.org Cc: Subject: Current problem reports assigned to you X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 Jul 2007 11:08:58 -0000 Current FreeBSD problem reports Critical problems S Tracker Resp. Description -------------------------------------------------------------------------------- s threa/76690 threads fork hang in child for -lc_r 1 problem total. Serious problems S Tracker Resp. Description -------------------------------------------------------------------------------- o kern/20016 threads pthreads: Cannot set scheduling timer/Cannot set virtu s threa/24472 threads libc_r does not honor SO_SNDTIMEO/SO_RCVTIMEO socket o s threa/24632 threads libc_r delicate deviation from libc in handling SIGCHL s bin/32295 threads pthread dont dequeue signals s threa/34536 threads accept() blocks other threads o kern/38549 threads the procces compiled whith pthread stopped in pthread_ s threa/39922 threads [threads] [patch] Threaded applications executed with s threa/48856 threads Setting SIGCHLD to SIG_IGN still leaves zombies under s threa/49087 threads Signals lost in programs linked with libc_r s kern/64313 threads FreeBSD (OpenBSD) pthread implicit set/unset O_NONBLOC o threa/70975 threads unexpected and unreliable behaviour when using SYSV se o threa/72429 threads threads blocked in stdio (fgets, etc) are not cancella o threa/72953 threads fork() unblocks blocked signals w/o PTHREAD_SCOPE_SYST o threa/75273 threads FBSD 5.3 libpthread (KSE) bug o threa/75374 threads pthread_kill() ignores SA_SIGINFO flag s threa/76694 threads fork cause hang in dup()/close() function in child (-l o threa/79683 threads svctcp_create() fails if multiple threads call at the o threa/80435 threads panic on high loads o threa/83914 threads [libc] popen() doesn't work in static threaded program s threa/84483 threads problems with devel/nspr and -lc_r on 4.x o threa/85160 threads [libthr] [patch] libobjc + libpthread/libthr crash pro o kern/91266 threads [threads] Trying sleep, but thread marked as sleeping s threa/94467 threads send(), sendto() and sendmsg() are not correct in libc s threa/100815 threads FBSD 5.5 broke nanosleep in libc_r o threa/101323 threads fork(2) in threaded programs broken. o threa/103975 threads Implicit loading/unloading of libpthread.so may crash o threa/110636 threads gdb(1): using gdb with multi thread application with l o threa/113666 threads misc/shared-mime-info doesn't install, can't find thre 28 problems total. Non-critical problems S Tracker Resp. Description -------------------------------------------------------------------------------- s kern/19247 threads uthread_sigaction.c does not do anything wrt SA_NOCLDW s kern/22190 threads A threaded read(2) from a socketpair(2) fd can sometim s threa/30464 threads pthread mutex attributes -- pshared s threa/37676 threads libc_r: msgsnd(), msgrcv(), pread(), pwrite() need wra s threa/40671 threads pthread_cancel doesn't remove thread from condition qu s threa/69020 threads pthreads library leaks _gc_mutex o threa/79887 threads [patch] freopen() isn't thread-safe o threa/80992 threads abort() sometimes not caught by gdb depending on threa o threa/81534 threads [libc_r] [patch] libc_r close() will fail on any fd ty o threa/110306 threads apache 2.0 segmentation violation when calling gethost 10 problems total. From owner-freebsd-threads@FreeBSD.ORG Fri Jul 6 06:27:47 2007 Return-Path: X-Original-To: freebsd-threads@freebsd.org Delivered-To: freebsd-threads@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 376EB16A41F; Fri, 6 Jul 2007 06:27:47 +0000 (UTC) (envelope-from prvs=moorthi=700a4fd99@ironport.com) Received: from a50.ironport.com (a50.ironport.com [63.251.108.112]) by mx1.freebsd.org (Postfix) with ESMTP id 1BF7813C44B; Fri, 6 Jul 2007 06:27:46 +0000 (UTC) (envelope-from prvs=moorthi=700a4fd99@ironport.com) DomainKey-Signature: s=key512; d=ironport.com; c=nofws; q=dns; b=IHioHQtOiVg0Gmzg5wrxix6xZ6QJ3+pQ+Sn1cVrHcOXB3BYTGUhSS92iIPcG6cWZ1daWslCGZEPANIV3Xq5uTA==; Received: from dooku.ironportsystems.com ([10.1.1.49]) by a50.ironport.com with ESMTP; 05 Jul 2007 22:58:36 -0700 Received: from anakin.ironportsystems.com ([10.1.1.166]) by dooku.ironportsystems.com with Microsoft SMTPSVC(6.0.3790.1830); Thu, 5 Jul 2007 22:58:36 -0700 X-MimeOLE: Produced By Microsoft Exchange V6.5 Content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Date: Thu, 5 Jul 2007 22:58:31 -0700 Message-ID: <1ADDEDB49C08A24C928673FA130ECA99D65617@anakin.ironportsystems.com> X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: python segfaulted, all threads in pthread_testcancel Thread-Index: Ace/kq6cm9NvngmsQvaJpbYimvFlPA== From: "Jay Moorthi" To: , X-OriginalArrivalTime: 06 Jul 2007 05:58:36.0359 (UTC) FILETIME=[B1455570:01C7BF92] Cc: Subject: python segfaulted, all threads in pthread_testcancel X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 06 Jul 2007 06:27:47 -0000 Hi Folks, I've been seeing python 2.4.2 processes crashing every now and then, running a handful of different scripts that use the threading module. I can't reliably reproduce the crash, and I have not yet noticed a pattern to the failures other than their similar stack traces. Every thread shows up in pthread_testcancel. The python binary I'm dealing with is stripped, so I don't have lots of debugging info, but I get the following when I open the latest example core in gdb: (gdb) info threads 5 process 100191 0x881351af in pthread_testcancel () from /usr/lib/libpthread.so.2 4 process 100227 0x881351af in pthread_testcancel () from /usr/lib/libpthread.so.2 3 process 100140 0x881351af in pthread_testcancel () from /usr/lib/libpthread.so.2 2 process 100187 0x8813520f in pthread_testcancel () from /usr/lib/libpthread.so.2 * 1 process 100126 0x881351ef in pthread_testcancel () from /usr/lib/libpthread.so.2 (gdb) thread apply all bt Thread 5 (process 100191): #0 0x881351af in pthread_testcancel () from /usr/lib/libpthread.so.2 #1 0x8812dbd0 in pthread_mutexattr_init () from /usr/lib/libpthread.so.2 #2 0x00000000 in ?? () Thread 4 (process 100227): #0 0x881351af in pthread_testcancel () from /usr/lib/libpthread.so.2 #1 0x8812e517 in pthread_mutexattr_init () from /usr/lib/libpthread.so.2 #2 0x08521000 in ?? () Thread 3 (process 100140): #0 0x881351af in pthread_testcancel () from /usr/lib/libpthread.so.2 #1 0x8812e517 in pthread_mutexattr_init () from /usr/lib/libpthread.so.2 #2 0x08521000 in ?? () Thread 2 (process 100187): #0 0x8813520f in pthread_testcancel () from /usr/lib/libpthread.so.2 #1 0x080bc702 in PyThread_release_lock () #2 0x0809b8e7 in PyEval_EvalFrame () #3 0x080a0100 in PyEval_EvalCodeEx () #4 0x080d54f6 in PyFunction_SetClosure () #5 0x0805a448 in PyObject_Call () #6 0x0805fce6 in PyMethod_New () #7 0x0805a448 in PyObject_Call () #8 0x0809a767 in PyEval_CallObjectWithKeywords () #9 0x0808339d in _PyObject_SlotCompare () #10 0x0807e22c in PyType_GenericNew () #11 0x080d3aa8 in PyGen_New () #12 0x080b6e52 in PyThreadState_Clear () #13 0x080b6e85 in PyInterpreterState_Clear () #14 0x080b9436 in Py_Finalize () #15 0x08055578 in Py_Main () #16 0x0805503d in main () Thread 1 (process 100126): #0 0x881351ef in pthread_testcancel () from /usr/lib/libpthread.so.2 #1 0x8812e2cd in pthread_mutexattr_init () from /usr/lib/libpthread.so.2 #2 0x8812e3ae in pthread_mutexattr_init () from /usr/lib/libpthread.so.2 #3 0x08521000 in ?? () (gdb) Is pthread_testcancel here a red herring? The ??s at the outermost frames make me think that gdb is getting the stack unwind slightly wrong and landing in pthread code by mistake, but thread 2's stack seems plausible. Is there anything I can do to determine whether these are sane backtraces? The system is running 6.0-STABLE: moorthi@wbrs2-app2$ uname -a FreeBSD wbrs2-app2.soma.ironport.com 6.0-STABLE FreeBSD 6.0-STABLE #0: Sat May 27 01:33:34 UTC 2006 root@wbrs2-app2.soma.ironport.com:/usr/src/sys/i386/compile/MESSAGING_GA TEWAY.i386_INSTALL i386 moorthi@wbrs2-app2$ moorthi@wbrs2-app2$ ldd /usr/local/bin/python /usr/local/bin/python: libpthread.so.2 =3D> /usr/lib/libpthread.so.2 (0x88118000) libutil.so.5 =3D> /lib/libutil.so.5 (0x8813d000) libstdc++.so.5 =3D> /usr/lib/libstdc++.so.5 (0x88149000) libm.so.4 =3D> /lib/libm.so.4 (0x88213000) libc.so.6 =3D> /lib/libc.so.6 (0x88229000) moorthi@wbrs2-app2$ more /etc/libmap.conf [mysqld] libpthread.so.2 libthr.so.2 libpthread.so libthr.so Another piece of information is it appears that these python processes do make significant progress in script executiong before they segfault. In general, they are all connected to one of many MySQL database servers using MySQLdb. Any advice you may have will be useful, especially suggestions on where I should look for more information. Thanks, Jay From owner-freebsd-threads@FreeBSD.ORG Fri Jul 6 07:01:08 2007 Return-Path: X-Original-To: freebsd-threads@freebsd.org Delivered-To: freebsd-threads@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 01F4716A46C for ; Fri, 6 Jul 2007 07:01:08 +0000 (UTC) (envelope-from julian@elischer.org) Received: from outF.internet-mail-service.net (outF.internet-mail-service.net [216.240.47.229]) by mx1.freebsd.org (Postfix) with ESMTP id CA1F713C44C for ; Fri, 6 Jul 2007 07:01:07 +0000 (UTC) (envelope-from julian@elischer.org) Received: from mx0.idiom.com (HELO idiom.com) (216.240.32.160) by out.internet-mail-service.net (qpsmtpd/0.32) with ESMTP; Thu, 05 Jul 2007 23:46:23 -0700 Received: from julian-mac.elischer.org (home.elischer.org [216.240.48.38]) by idiom.com (Postfix) with ESMTP id AE000125ADD; Thu, 5 Jul 2007 23:46:22 -0700 (PDT) Message-ID: <468DE54A.8090900@elischer.org> Date: Thu, 05 Jul 2007 23:46:34 -0700 From: Julian Elischer User-Agent: Thunderbird 2.0.0.4 (Macintosh/20070604) MIME-Version: 1.0 To: Jay Moorthi References: <1ADDEDB49C08A24C928673FA130ECA99D65617@anakin.ironportsystems.com> In-Reply-To: <1ADDEDB49C08A24C928673FA130ECA99D65617@anakin.ironportsystems.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-python@freebsd.org, freebsd-threads@freebsd.org Subject: Re: python segfaulted, all threads in pthread_testcancel X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 06 Jul 2007 07:01:08 -0000 I add that "this is fixed in 6.2" is an acceptable answer, but I suggested that Jay post here for assistance because I really don't know much about the userland side of the library. Jay Moorthi wrote: > Hi Folks, > > I've been seeing python 2.4.2 processes crashing every now and then, > running a handful of different scripts that use the threading module. I > can't reliably reproduce the crash, and I have not yet noticed a pattern > to the failures other than their similar stack traces. Every thread > shows up in pthread_testcancel. > > The python binary I'm dealing with is stripped, so I don't have lots of > debugging info, but I get the following when I open the latest example > core in gdb: > > (gdb) info threads > 5 process 100191 0x881351af in pthread_testcancel () from > /usr/lib/libpthread.so.2 > 4 process 100227 0x881351af in pthread_testcancel () from > /usr/lib/libpthread.so.2 > 3 process 100140 0x881351af in pthread_testcancel () from > /usr/lib/libpthread.so.2 > 2 process 100187 0x8813520f in pthread_testcancel () from > /usr/lib/libpthread.so.2 > * 1 process 100126 0x881351ef in pthread_testcancel () from > /usr/lib/libpthread.so.2 > (gdb) thread apply all bt > > Thread 5 (process 100191): > #0 0x881351af in pthread_testcancel () from /usr/lib/libpthread.so.2 > #1 0x8812dbd0 in pthread_mutexattr_init () from > /usr/lib/libpthread.so.2 > #2 0x00000000 in ?? () > > Thread 4 (process 100227): > #0 0x881351af in pthread_testcancel () from /usr/lib/libpthread.so.2 > #1 0x8812e517 in pthread_mutexattr_init () from > /usr/lib/libpthread.so.2 > #2 0x08521000 in ?? () > > Thread 3 (process 100140): > #0 0x881351af in pthread_testcancel () from /usr/lib/libpthread.so.2 > #1 0x8812e517 in pthread_mutexattr_init () from > /usr/lib/libpthread.so.2 > #2 0x08521000 in ?? () > > Thread 2 (process 100187): > #0 0x8813520f in pthread_testcancel () from /usr/lib/libpthread.so.2 > #1 0x080bc702 in PyThread_release_lock () > #2 0x0809b8e7 in PyEval_EvalFrame () > #3 0x080a0100 in PyEval_EvalCodeEx () > #4 0x080d54f6 in PyFunction_SetClosure () > #5 0x0805a448 in PyObject_Call () > #6 0x0805fce6 in PyMethod_New () > #7 0x0805a448 in PyObject_Call () > #8 0x0809a767 in PyEval_CallObjectWithKeywords () > #9 0x0808339d in _PyObject_SlotCompare () #10 0x0807e22c in > PyType_GenericNew () > #11 0x080d3aa8 in PyGen_New () > #12 0x080b6e52 in PyThreadState_Clear () > #13 0x080b6e85 in PyInterpreterState_Clear () > #14 0x080b9436 in Py_Finalize () > #15 0x08055578 in Py_Main () > #16 0x0805503d in main () > > Thread 1 (process 100126): > #0 0x881351ef in pthread_testcancel () from /usr/lib/libpthread.so.2 > #1 0x8812e2cd in pthread_mutexattr_init () from > /usr/lib/libpthread.so.2 > #2 0x8812e3ae in pthread_mutexattr_init () from > /usr/lib/libpthread.so.2 > #3 0x08521000 in ?? () > (gdb) > > Is pthread_testcancel here a red herring? The ??s at the outermost > frames make me think that gdb is getting the stack unwind slightly wrong > and landing in pthread code by mistake, but thread 2's stack seems > plausible. > > Is there anything I can do to determine whether these are sane > backtraces? > > The system is running 6.0-STABLE: > > moorthi@wbrs2-app2$ uname -a > FreeBSD wbrs2-app2.soma.ironport.com 6.0-STABLE FreeBSD 6.0-STABLE #0: > Sat May 27 01:33:34 UTC 2006 > root@wbrs2-app2.soma.ironport.com:/usr/src/sys/i386/compile/MESSAGING_GA > TEWAY.i386_INSTALL i386 > moorthi@wbrs2-app2$ > > moorthi@wbrs2-app2$ ldd /usr/local/bin/python > /usr/local/bin/python: > libpthread.so.2 => /usr/lib/libpthread.so.2 (0x88118000) > libutil.so.5 => /lib/libutil.so.5 (0x8813d000) > libstdc++.so.5 => /usr/lib/libstdc++.so.5 (0x88149000) > libm.so.4 => /lib/libm.so.4 (0x88213000) > libc.so.6 => /lib/libc.so.6 (0x88229000) > > moorthi@wbrs2-app2$ more /etc/libmap.conf [mysqld] > libpthread.so.2 libthr.so.2 > libpthread.so libthr.so > > Another piece of information is it appears that these python processes > do make significant progress in script executiong before they segfault. > In general, they are all connected to one of many MySQL database servers > using MySQLdb. > > Any advice you may have will be useful, especially suggestions on where > I should look for more information. > > Thanks, > > Jay > _______________________________________________ > freebsd-threads@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-threads > To unsubscribe, send any mail to "freebsd-threads-unsubscribe@freebsd.org"