From owner-freebsd-hackers@FreeBSD.ORG Thu Dec 17 15:45:42 2009 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0FDC3106568F; Thu, 17 Dec 2009 15:45:42 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from cyrus.watson.org (cyrus.watson.org [65.122.17.42]) by mx1.freebsd.org (Postfix) with ESMTP id D47FA8FC08; Thu, 17 Dec 2009 15:45:41 +0000 (UTC) Received: from bigwig.baldwin.cx (66.111.2.69.static.nyinternet.net [66.111.2.69]) by cyrus.watson.org (Postfix) with ESMTPSA id 7006A46B2A; Thu, 17 Dec 2009 10:45:41 -0500 (EST) Received: from jhbbsd.localnet (unknown [209.249.190.9]) by bigwig.baldwin.cx (Postfix) with ESMTPA id C967D8A01D; Thu, 17 Dec 2009 10:45:40 -0500 (EST) From: John Baldwin To: freebsd-hackers@freebsd.org Date: Thu, 17 Dec 2009 09:08:49 -0500 User-Agent: KMail/1.12.1 (FreeBSD/7.2-CBSD-20091103; KDE/4.3.1; amd64; ; ) References: In-Reply-To: MIME-Version: 1.0 Content-Type: Text/Plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Message-Id: <200912170908.49119.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.0.1 (bigwig.baldwin.cx); Thu, 17 Dec 2009 10:45:40 -0500 (EST) X-Virus-Scanned: clamav-milter 0.95.1 at bigwig.baldwin.cx X-Virus-Status: Clean X-Spam-Status: No, score=-2.5 required=4.2 tests=AWL,BAYES_00,RDNS_NONE autolearn=no version=3.2.5 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on bigwig.baldwin.cx Cc: freebsd-stable@freebsd.org, Steven Hartland Subject: Re: Passenger hangs on live and SEGV on tests possible threading / kernel bug? X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 17 Dec 2009 15:45:42 -0000 On Thursday 17 December 2009 6:12:07 am Steven Hartland wrote: > We're having an issue with Passenger on FreeBSD where it will hang > and stop processing any more requests the details are attach to > the following bug report: > http://code.google.com/p/phusion-passenger/issues/detail?id=318#c14 > > In addition the test suite crashes in what seems to be a very > basic test, which I'm at a loss with. > http://code.google.com/p/phusion-passenger/issues/detail?id=441 > > I'm thinking this may be a bugs in the FreeBSD either kernel or > thread library as the crashes don't make any sense from the > application side. > > Any advise on debugging or feedback on the stack traces would > be much appreciated. For the hang it seems you have a thread waiting in a blocking read(), a thread waiting in a blocking accept(), and lots of threads creating condition variables. However, the pthread_cond_init() in libpthread (libthr on FreeBSD) doesn't call pthread_cleanup_push(), so your stack trace doesn't make sense to me. However, that may be gdb getting confused. The pthread_cleanup_push() frame may be cond_init(). However, it doesn't call umtx_op() (the _thr_umutex_init() call it makes just initializes the structure, it doesn't make a _umtx_op() system call). You might try posting on threads@ to try to get more info on this, but your pthread_cond_init() stack traces don't really make sense. Can you rebuild libc and libthr with debug symbols? For example: # cd /usr/src/lib/libc # make clean # make DEBUG_FLAGS=-g # make DEBUG_FLAGS=-g install However, if you are hanging in read(), that usually means you have a socket that just doesn't have data. That might be an application bug of some sort. The segv trace doesn't include the first part of GDB messages which show which thread actually had a seg fault. It looks like it was the thread that was throwing an exception. However, nanosleep() doesn't throw exceptions, so that stack trace doesn't really make sense either. Perhaps that stack is hosed by the exception handling code? -- John Baldwin