Date: Fri, 01 Dec 2006 13:59:34 +0800 From: Ganbold <ganbold@micom.mng.net> To: Kris Kennaway <kris@obsecurity.org> Cc: FreeBSD Stable Mailing List <freebsd-stable@freebsd.org> Subject: Re: application hangs in STABLE from time to time Message-ID: <456FC4C6.30201@micom.mng.net> In-Reply-To: <20061201051343.GA97010@xor.obsecurity.org> References: <456652AF.7070808@micom.mng.net> <20061124030100.GA92039@xor.obsecurity.org> <456FB769.5070703@micom.mng.net> <20061201051343.GA97010@xor.obsecurity.org>
next in thread | previous in thread | raw e-mail | index | archive | help
Kris,
Kris Kennaway wrote:
> On Fri, Dec 01, 2006 at 01:02:33PM +0800, Ganbold wrote:
>
>> Kris Kennaway wrote:
>>
>>> On Fri, Nov 24, 2006 at 10:02:23AM +0800, Ganbold wrote:
>>>
>>>
>>>
>>>> So do I have interrupt storms here and it is something related to bge?
>>>>
>>>>
>>> No, your interrupts look fine.
>>>
>>>
>>>
>>>> What else should I check when application hangs again?
>>>>
>>>>
>>> The most important thing to know is what is the application doing when
>>> it hangs. Unfortunately none of the information you provided shows
>>> this. Next time use the -o wchan argument to ps to find out what
>>> state the process is blocked in.
>>>
>> Ok, Here it is:
>>
>> 573 ?? Is 0:00.02 /usr/sbin/inetd -wW -C 60
>> 78721 ?? I 0:00.01 /usr/local/Radiator-3.15/hooks/PSA
>> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>> 78744 ?? Is 0:00.05 sshd: tsgan [priv] (sshd)
>> 78747 ?? S 0:00.02 sshd: tsgan@ttyp0 (sshd)
>> 591 v0 Is+ 0:00.00 /usr/libexec/getty Pc ttyv0
>> 592 v1 Is+ 0:00.00 /usr/libexec/getty Pc ttyv1
>> 593 v2 Is+ 0:00.00 /usr/libexec/getty Pc ttyv2
>> 594 v3 Is+ 0:00.00 /usr/libexec/getty Pc ttyv3
>> 595 v4 Is+ 0:00.00 /usr/libexec/getty Pc ttyv4
>> 596 v5 Is+ 0:00.00 /usr/libexec/getty Pc ttyv5
>> 597 v6 Is+ 0:00.00 /usr/libexec/getty Pc ttyv6
>> 598 v7 Is+ 0:00.00 /usr/libexec/getty Pc ttyv7
>> 16099 p0- I 20:29.05 perl /usr/local/Radiator-3.15/radiusd
>> -log_file /var/log/radius/logfile -config_file
>> /usr/local/Radiator-3.15/voip.cfg -pid_file /
>> 78748 p0 Is 0:00.01 -sh (sh)
>> 78750 p0 I 0:00.01 su
>> 78751 p0 S 0:00.04 _su (csh)
>> 78761 p0 R+ 0:00.00 ps ax
>>
>> voiprad#ps axHlwww|grep PSA
>>
>> 0 78721 16099 0 4 0 1696 1184 sbwait I ?? 0:00.01
>> /usr/local/Radiator-3.15/hooks/PSA
>>
>> voiprad#
>> voiprad#
>> voiprad# ps -o wchan
>> WCHAN
>> ttyin
>> ttyin
>> ttyin
>> ttyin
>> ttyin
>> ttyin
>> ttyin
>> ttyin
>> piperd
>> wait
>> pause
>> -
>>
>
> Well, I meant a more complete command than that one ;-) Fortunately
> it's also included in your previous output above ("sbwait"). This
> means that the process is waiting for network traffic (usually waiting
> for another local or remote process to send it data). So it's not
> obviously pointing to a problem.
>
> Remind me again how you know this isn't an application bug (sorry,
> I've forgotten context)?
>
This application connects to remote mysql-4.0.x server
and sends some queries and does some calculations and returns.
I tried to run this application from console, and it works
fine. It runs from radius server and it works fine serving
user access requests except sometimes it hangs. It used to
work fine on FreeBSD 5.2-STABLE before upgrading to RELENG-6.
So I guess there is something else.
>
>>> You can also use kgdb to find out
>>> where it is waiting in the kernel:
>>>
>>> kgdb /dev/mem /boot/kernel/kernel.symbols
>>> info threads
>>> <find the thread corresponding to the process that is blocked>
>>> thread <tid>
>>> bt
>>>
>>>
>> Oh, I don't have kernel.symbols file, how to enable it?
>>
>
> It might be in your kernel compilation directory (possibly called
> kernel.debug). Otherwise, you'll have to build a new kernel and
> trigger the problem again.
>
I tried with with kernel.debug without success:
voiprad# kgdb /dev/mem /usr/obj/usr/src/sys/VOIPRAD/kernel.debug
kgdb: bad namelist
thanks,
Ganbold
> kris
>
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?456FC4C6.30201>