From owner-freebsd-stable@FreeBSD.ORG Fri Dec 1 05:59:38 2006 Return-Path: X-Original-To: freebsd-stable@freebsd.org Delivered-To: freebsd-stable@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 4E46816A403 for ; Fri, 1 Dec 2006 05:59:38 +0000 (UTC) (envelope-from ganbold@micom.mng.net) Received: from publicd.ub.mng.net (publicd.ub.mng.net [202.179.0.88]) by mx1.FreeBSD.org (Postfix) with ESMTP id 8D3D143CA6 for ; Fri, 1 Dec 2006 05:59:25 +0000 (GMT) (envelope-from ganbold@micom.mng.net) Received: from [202.179.0.164] (helo=[192.168.0.18]) by publicd.ub.mng.net with esmtpa (Exim 4.63 (FreeBSD)) (envelope-from ) id 1Gq1Qs-0002hZ-96; Fri, 01 Dec 2006 13:59:34 +0800 Message-ID: <456FC4C6.30201@micom.mng.net> Date: Fri, 01 Dec 2006 13:59:34 +0800 From: Ganbold User-Agent: Thunderbird 1.5.0.4 (X11/20060612) MIME-Version: 1.0 To: Kris Kennaway References: <456652AF.7070808@micom.mng.net> <20061124030100.GA92039@xor.obsecurity.org> <456FB769.5070703@micom.mng.net> <20061201051343.GA97010@xor.obsecurity.org> In-Reply-To: <20061201051343.GA97010@xor.obsecurity.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: FreeBSD Stable Mailing List Subject: Re: application hangs in STABLE from time to time X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 01 Dec 2006 05:59:38 -0000 Kris, Kris Kennaway wrote: > On Fri, Dec 01, 2006 at 01:02:33PM +0800, Ganbold wrote: > >> Kris Kennaway wrote: >> >>> On Fri, Nov 24, 2006 at 10:02:23AM +0800, Ganbold wrote: >>> >>> >>> >>>> So do I have interrupt storms here and it is something related to bge? >>>> >>>> >>> No, your interrupts look fine. >>> >>> >>> >>>> What else should I check when application hangs again? >>>> >>>> >>> The most important thing to know is what is the application doing when >>> it hangs. Unfortunately none of the information you provided shows >>> this. Next time use the -o wchan argument to ps to find out what >>> state the process is blocked in. >>> >> Ok, Here it is: >> >> 573 ?? Is 0:00.02 /usr/sbin/inetd -wW -C 60 >> 78721 ?? I 0:00.01 /usr/local/Radiator-3.15/hooks/PSA >> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ >> 78744 ?? Is 0:00.05 sshd: tsgan [priv] (sshd) >> 78747 ?? S 0:00.02 sshd: tsgan@ttyp0 (sshd) >> 591 v0 Is+ 0:00.00 /usr/libexec/getty Pc ttyv0 >> 592 v1 Is+ 0:00.00 /usr/libexec/getty Pc ttyv1 >> 593 v2 Is+ 0:00.00 /usr/libexec/getty Pc ttyv2 >> 594 v3 Is+ 0:00.00 /usr/libexec/getty Pc ttyv3 >> 595 v4 Is+ 0:00.00 /usr/libexec/getty Pc ttyv4 >> 596 v5 Is+ 0:00.00 /usr/libexec/getty Pc ttyv5 >> 597 v6 Is+ 0:00.00 /usr/libexec/getty Pc ttyv6 >> 598 v7 Is+ 0:00.00 /usr/libexec/getty Pc ttyv7 >> 16099 p0- I 20:29.05 perl /usr/local/Radiator-3.15/radiusd >> -log_file /var/log/radius/logfile -config_file >> /usr/local/Radiator-3.15/voip.cfg -pid_file / >> 78748 p0 Is 0:00.01 -sh (sh) >> 78750 p0 I 0:00.01 su >> 78751 p0 S 0:00.04 _su (csh) >> 78761 p0 R+ 0:00.00 ps ax >> >> voiprad#ps axHlwww|grep PSA >> >> 0 78721 16099 0 4 0 1696 1184 sbwait I ?? 0:00.01 >> /usr/local/Radiator-3.15/hooks/PSA >> >> voiprad# >> voiprad# >> voiprad# ps -o wchan >> WCHAN >> ttyin >> ttyin >> ttyin >> ttyin >> ttyin >> ttyin >> ttyin >> ttyin >> piperd >> wait >> pause >> - >> > > Well, I meant a more complete command than that one ;-) Fortunately > it's also included in your previous output above ("sbwait"). This > means that the process is waiting for network traffic (usually waiting > for another local or remote process to send it data). So it's not > obviously pointing to a problem. > > Remind me again how you know this isn't an application bug (sorry, > I've forgotten context)? > This application connects to remote mysql-4.0.x server and sends some queries and does some calculations and returns. I tried to run this application from console, and it works fine. It runs from radius server and it works fine serving user access requests except sometimes it hangs. It used to work fine on FreeBSD 5.2-STABLE before upgrading to RELENG-6. So I guess there is something else. > >>> You can also use kgdb to find out >>> where it is waiting in the kernel: >>> >>> kgdb /dev/mem /boot/kernel/kernel.symbols >>> info threads >>> >>> thread >>> bt >>> >>> >> Oh, I don't have kernel.symbols file, how to enable it? >> > > It might be in your kernel compilation directory (possibly called > kernel.debug). Otherwise, you'll have to build a new kernel and > trigger the problem again. > I tried with with kernel.debug without success: voiprad# kgdb /dev/mem /usr/obj/usr/src/sys/VOIPRAD/kernel.debug kgdb: bad namelist thanks, Ganbold > kris >