From owner-freebsd-python@FreeBSD.ORG Fri Jul 6 06:27:47 2007 Return-Path: X-Original-To: freebsd-python@freebsd.org Delivered-To: freebsd-python@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 376EB16A41F; Fri, 6 Jul 2007 06:27:47 +0000 (UTC) (envelope-from prvs=moorthi=700a4fd99@ironport.com) Received: from a50.ironport.com (a50.ironport.com [63.251.108.112]) by mx1.freebsd.org (Postfix) with ESMTP id 1BF7813C44B; Fri, 6 Jul 2007 06:27:46 +0000 (UTC) (envelope-from prvs=moorthi=700a4fd99@ironport.com) DomainKey-Signature: s=key512; d=ironport.com; c=nofws; q=dns; b=IHioHQtOiVg0Gmzg5wrxix6xZ6QJ3+pQ+Sn1cVrHcOXB3BYTGUhSS92iIPcG6cWZ1daWslCGZEPANIV3Xq5uTA==; Received: from dooku.ironportsystems.com ([10.1.1.49]) by a50.ironport.com with ESMTP; 05 Jul 2007 22:58:36 -0700 Received: from anakin.ironportsystems.com ([10.1.1.166]) by dooku.ironportsystems.com with Microsoft SMTPSVC(6.0.3790.1830); Thu, 5 Jul 2007 22:58:36 -0700 X-MimeOLE: Produced By Microsoft Exchange V6.5 Content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Date: Thu, 5 Jul 2007 22:58:31 -0700 Message-ID: <1ADDEDB49C08A24C928673FA130ECA99D65617@anakin.ironportsystems.com> X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: python segfaulted, all threads in pthread_testcancel Thread-Index: Ace/kq6cm9NvngmsQvaJpbYimvFlPA== From: "Jay Moorthi" To: , X-OriginalArrivalTime: 06 Jul 2007 05:58:36.0359 (UTC) FILETIME=[B1455570:01C7BF92] Cc: Subject: python segfaulted, all threads in pthread_testcancel X-BeenThere: freebsd-python@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: FreeBSD-specific Python issues List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 06 Jul 2007 06:27:47 -0000 Hi Folks, I've been seeing python 2.4.2 processes crashing every now and then, running a handful of different scripts that use the threading module. I can't reliably reproduce the crash, and I have not yet noticed a pattern to the failures other than their similar stack traces. Every thread shows up in pthread_testcancel. The python binary I'm dealing with is stripped, so I don't have lots of debugging info, but I get the following when I open the latest example core in gdb: (gdb) info threads 5 process 100191 0x881351af in pthread_testcancel () from /usr/lib/libpthread.so.2 4 process 100227 0x881351af in pthread_testcancel () from /usr/lib/libpthread.so.2 3 process 100140 0x881351af in pthread_testcancel () from /usr/lib/libpthread.so.2 2 process 100187 0x8813520f in pthread_testcancel () from /usr/lib/libpthread.so.2 * 1 process 100126 0x881351ef in pthread_testcancel () from /usr/lib/libpthread.so.2 (gdb) thread apply all bt Thread 5 (process 100191): #0 0x881351af in pthread_testcancel () from /usr/lib/libpthread.so.2 #1 0x8812dbd0 in pthread_mutexattr_init () from /usr/lib/libpthread.so.2 #2 0x00000000 in ?? () Thread 4 (process 100227): #0 0x881351af in pthread_testcancel () from /usr/lib/libpthread.so.2 #1 0x8812e517 in pthread_mutexattr_init () from /usr/lib/libpthread.so.2 #2 0x08521000 in ?? () Thread 3 (process 100140): #0 0x881351af in pthread_testcancel () from /usr/lib/libpthread.so.2 #1 0x8812e517 in pthread_mutexattr_init () from /usr/lib/libpthread.so.2 #2 0x08521000 in ?? () Thread 2 (process 100187): #0 0x8813520f in pthread_testcancel () from /usr/lib/libpthread.so.2 #1 0x080bc702 in PyThread_release_lock () #2 0x0809b8e7 in PyEval_EvalFrame () #3 0x080a0100 in PyEval_EvalCodeEx () #4 0x080d54f6 in PyFunction_SetClosure () #5 0x0805a448 in PyObject_Call () #6 0x0805fce6 in PyMethod_New () #7 0x0805a448 in PyObject_Call () #8 0x0809a767 in PyEval_CallObjectWithKeywords () #9 0x0808339d in _PyObject_SlotCompare () #10 0x0807e22c in PyType_GenericNew () #11 0x080d3aa8 in PyGen_New () #12 0x080b6e52 in PyThreadState_Clear () #13 0x080b6e85 in PyInterpreterState_Clear () #14 0x080b9436 in Py_Finalize () #15 0x08055578 in Py_Main () #16 0x0805503d in main () Thread 1 (process 100126): #0 0x881351ef in pthread_testcancel () from /usr/lib/libpthread.so.2 #1 0x8812e2cd in pthread_mutexattr_init () from /usr/lib/libpthread.so.2 #2 0x8812e3ae in pthread_mutexattr_init () from /usr/lib/libpthread.so.2 #3 0x08521000 in ?? () (gdb) Is pthread_testcancel here a red herring? The ??s at the outermost frames make me think that gdb is getting the stack unwind slightly wrong and landing in pthread code by mistake, but thread 2's stack seems plausible. Is there anything I can do to determine whether these are sane backtraces? The system is running 6.0-STABLE: moorthi@wbrs2-app2$ uname -a FreeBSD wbrs2-app2.soma.ironport.com 6.0-STABLE FreeBSD 6.0-STABLE #0: Sat May 27 01:33:34 UTC 2006 root@wbrs2-app2.soma.ironport.com:/usr/src/sys/i386/compile/MESSAGING_GA TEWAY.i386_INSTALL i386 moorthi@wbrs2-app2$ moorthi@wbrs2-app2$ ldd /usr/local/bin/python /usr/local/bin/python: libpthread.so.2 =3D> /usr/lib/libpthread.so.2 (0x88118000) libutil.so.5 =3D> /lib/libutil.so.5 (0x8813d000) libstdc++.so.5 =3D> /usr/lib/libstdc++.so.5 (0x88149000) libm.so.4 =3D> /lib/libm.so.4 (0x88213000) libc.so.6 =3D> /lib/libc.so.6 (0x88229000) moorthi@wbrs2-app2$ more /etc/libmap.conf [mysqld] libpthread.so.2 libthr.so.2 libpthread.so libthr.so Another piece of information is it appears that these python processes do make significant progress in script executiong before they segfault. In general, they are all connected to one of many MySQL database servers using MySQLdb. Any advice you may have will be useful, especially suggestions on where I should look for more information. Thanks, Jay