Date: Wed, 9 Feb 2005 22:00:29 -0700 From: Chad Leigh -- Shire.Net LLC <chad@shire.net> To: "Marc G. Fournier" <scrappy@hub.org> Cc: freebsd-questions@freebsd.org Subject: Re: 99% CPU usage in System (Was: Re: vinum in 4.x poor performer?) Message-ID: <AF1A08CE-7B20-11D9-B134-000D933E3CEC@shire.net> In-Reply-To: <20050209213253.O94338@ganymede.hub.org> References: <20050208231208.B94338@ganymede.hub.org> <20050209002232.B94338@ganymede.hub.org> <20050209104047.GN8619@alzatex.com> <20050209210602.X94338@ganymede.hub.org> <20050209213253.O94338@ganymede.hub.org>
next in thread | previous in thread | raw e-mail | index | archive | help
On Feb 9, 2005, at 6:34 PM, Marc G. Fournier wrote:
>
> Most odd, there definitely has to be a problem with the Dual-Xeon
> ysystem ... doing the same vmstat on my other vinum based system,
> running more, but on a Dual-PIII shows major idle time:
>
> # vmstat 5
> procs memory page disks faults
> cpu
> r b w avm fre flt re pi po fr sr da0 da1 in sy cs
> us sy id
> 20 1 0 4088636 219556 1664 1 2 1 3058 217 0 0 856 7937 2186
> 51 15 34
> 20 1 0 4115372 224220 472 0 0 0 2066 0 0 35 496 2915 745
> 7 7 86
> 10 1 0 4125252 221788 916 0 0 0 2513 0 2 71 798 4821 1538
> 6 11 83
> 9 1 0 36508 228452 534 0 0 2 2187 0 0 46 554 3384 1027
> 3 8 89
> 11 1 0 27672 218828 623 0 6 0 2337 0 0 61 583 2607 679
> 3 9 88
> 16 1 0 5776 220540 989 0 0 0 2393 0 9 32 514 3247 1115
> 3 8 90
>
> Which leads me further to believe this is a Dual-Xeon problem, and
> much further away from believing it has anything to do with software
> RAID :(
I only use AMD, so I cannot provide specifics, but look in the BIOS at
boot time and see if there is anything strange looking in the settings.
Chad
>
>
> On Wed, 9 Feb 2005, Marc G. Fournier wrote:
>
>>
>> still getting this:
>>
>> # vmstat 5
>> procs memory page disks faults
>> cpu
>> r b w avm fre flt re pi po fr sr da0 da1 in sy cs
>> us sy id
>> 11 2 0 3020036 267944 505 2 1 1 680 62 0 0 515 4005 918
>> 7 38 55
>> 19 2 0 3004568 268672 242 0 0 0 277 0 0 3 338 2767 690
>> 1 99 0
>> 21 2 0 2999152 271240 135 0 0 0 306 0 6 9 363 1749 525
>> 1 99 0
>> 13 2 0 3001508 269692 87 0 0 0 24 0 3 3 302 1524 285
>> 1 99 0
>> 17 2 0 3025892 268612 98 0 1 0 66 0 5 6 312 1523 479
>> 3 97 0
>>
>> Is there a way of determining what is sucking up so much Sys time?
>> stuff like pperl scripts running and such would use 'user time', no?
>> I've got some high CPU processes running, but would expect them to be
>> shooting up the 'user time' ...
>>
>> USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME
>> COMMAND
>> setiathome 21338 16.3 0.2 7888 7408 ?? RJ 9:05PM 0:11.35
>> /usr/bin/perl -wT /usr/local/majordomo/bin/mj_queuerun -v 0
>> setiathome 21380 15.1 0.1 2988 2484 ?? RsJ 9:06PM 0:02.42
>> /usr/bin/perl -wT /usr/local/majordomo/bin/mj_enqueue -r -d
>> postgresql.org -l pgsql-sql -P10 -p10
>> setiathome 21384 15.5 0.1 2988 2484 ?? RsJ 9:06PM 0:02.31
>> /usr/bin/perl -wT /usr/local/majordomo/bin/mj_enqueue -r -d
>> postgresql.org -l pgsql-docs -P10 -p10
>> setiathome 21389 15.0 0.1 2720 2216 ?? RsJ 9:06PM 0:02.06
>> /usr/bin/perl -wT /usr/local/majordomo/bin/mj_enqueue -r -d
>> postgresql.org -l pgsql-hackers -P10 -p10
>> setiathome 21386 13.7 0.1 2720 2216 ?? RsJ 9:06PM 0:02.03
>> /usr/bin/perl -wT /usr/local/majordomo/bin/mj_enqueue -r -d
>> postgresql.org -l pgsql-ports -P10 -p10
>> setiathome 21387 13.2 0.1 2724 2220 ?? RsJ 9:06PM 0:01.92
>> /usr/bin/perl -wT /usr/local/majordomo/bin/mj_enqueue -r -d
>> postgresql.org -l pgsql-interfaces -P10 -p10
>> setiathome 21390 14.6 0.1 2724 2216 ?? RsJ 9:06PM 0:01.93
>> /usr/bin/perl -wT /usr/local/majordomo/bin/mj_enqueue -o -d
>> postgresql.org -l pgsql-performance -P10 -p10
>> setiathome 21330 12.0 0.2 8492 7852 ?? RJ 9:05PM 0:15.55
>> /usr/bin/perl -wT /dev/fd/3//usr/local/www/mj/mj_wwwusr (perl5.8.5)
>> setiathome 7864 8.9 0.2 8912 8452 ?? RJ 7:20PM 29:54.88
>> /usr/bin/perl -wT /usr/local/majordomo/bin/mj_trigger -t hourly
>>
>> Is there some way of finding out where all the Sys Time is being
>> used? Something more fine grained them what vmstat/top shows?
>>
>>
>> On Wed, 9 Feb 2005, Loren M. Lang wrote:
>>
>>> On Wed, Feb 09, 2005 at 02:32:30AM -0400, Marc G. Fournier wrote:
>>>> Is there a command that I can run that provide me the syscall/sec
>>>> value,
>>>> that I could use in a script? I know vmstat reports it, but is
>>>> there an
>>>> easier way the having to parse the output? a perl module maybe, that
>>>> already does it?
>>> vmstat shouldn't be too hard to parse, try the following:
>>> vmstat|tail -1|awk '{print $15;}'
>>> To print out the 15th field of vmstat. Now if you want vmstat to
>>> keep
>>> running every five seconds or something, it's a little more
>>> complicated:
>>> vmstat 5|grep -v 'procs\|avm'|awk '{print $15;}'
>>>> Thanks ...
>>>> On Wed, 9 Feb 2005, Marc G. Fournier wrote:
>>>>> On Tue, 8 Feb 2005, Dan Nelson wrote:
>>>>>> Details on the array's performance, I think. Software RAID5 will
>>>>>> definitely have poor write performance (logging disks solve that
>>>>>> problem but vinum doesn't do that), but should have excellent read
>>>>>> rates. From this output, however:
>>>>>>> systat -v output help:
>>>>>>> 4 users Load 4.64 5.58 5.77
>>>>>>> Proc:r p d s w Csw Trp Sys Int Sof Flt
>>>>>>> 24 9282 949 8414***** 678 349 8198
>>>>>>> 54.6%Sys 0.2%Intr 45.2%User 0.0%Nice 0.0%Idl
>>>>>>> Disks da0 da1 da2 da3 da4 pass0 pass1
>>>>>>> KB/t 5.32 9.50 12.52 16.00 9.00 0.00 0.00
>>>>>>> tps 23 2 4 3 1 0 0
>>>>>>> MB/s 0.12 0.01 0.05 0.04 0.01 0.00 0.00
>>>>>>> % busy 3 1 1 1 0 0 0
>>>>>> , it looks like your disks aren't being touched at all. You are
>>>>>> doing
>>>>>> over 99999 syscalls/second, though, which is mighty high. The
>>>>>> 50% Sys
>>>>>> doesn't look good either. You may have a runaway process doing
>>>>>> some
>>>>>> syscall over and over. If this is not an MPSAFE syscall (see
>>>>>> /sys/kern/syscalls.master ), it will also prevent other processes
>>>>>> from
>>>>>> making non-MPSAFE syscalls, and in 4.x that's most of them.
>>>>> Wow, that actually pointed me in the right direction, I think ...
>>>>> I just
>>>>> killed an http process that was using alot of CPU, and syscalls
>>>>> drop'd
>>>>> down to a numeric value again ... I'm still curious as to why this
>>>>> only
>>>>> seem sto affect my Dual-Xeon box though :(
>>>>> Thanks ...
>>>>> ----
>>>>> Marc G. Fournier Hub.Org Networking Services
>>>>> (http://www.hub.org)
>>>>> Email: scrappy@hub.org Yahoo!: yscrappy
>>>>> ICQ: 7615664
>>>>> _______________________________________________
>>>>> freebsd-questions@freebsd.org mailing list
>>>>> http://lists.freebsd.org/mailman/listinfo/freebsd-questions
>>>>> To unsubscribe, send any mail to
>>>>> "freebsd-questions-unsubscribe@freebsd.org"
>>>> ----
>>>> Marc G. Fournier Hub.Org Networking Services
>>>> (http://www.hub.org)
>>>> Email: scrappy@hub.org Yahoo!: yscrappy ICQ:
>>>> 7615664
>>>> _______________________________________________
>>>> freebsd-questions@freebsd.org mailing list
>>>> http://lists.freebsd.org/mailman/listinfo/freebsd-questions
>>>> To unsubscribe, send any mail to
>>>> "freebsd-questions-unsubscribe@freebsd.org"
>>> --
>>> I sense much NT in you.
>>> NT leads to Bluescreen.
>>> Bluescreen leads to downtime.
>>> Downtime leads to suffering.
>>> NT is the path to the darkside.
>>> Powerful Unix is.
>>> Public Key: ftp://ftp.tallye.com/pub/lorenl_pubkey.asc
>>> Fingerprint: B3B9 D669 69C9 09EC 1BCD 835A FAF3 7A46 E4A3 280C
>>
>> ----
>> Marc G. Fournier Hub.Org Networking Services
>> (http://www.hub.org)
>> Email: scrappy@hub.org Yahoo!: yscrappy ICQ:
>> 7615664
>> _______________________________________________
>> freebsd-questions@freebsd.org mailing list
>> http://lists.freebsd.org/mailman/listinfo/freebsd-questions
>> To unsubscribe, send any mail to
>> "freebsd-questions-unsubscribe@freebsd.org"
>>
>
> ----
> Marc G. Fournier Hub.Org Networking Services
> (http://www.hub.org)
> Email: scrappy@hub.org Yahoo!: yscrappy ICQ:
> 7615664
> _______________________________________________
> freebsd-questions@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-questions
> To unsubscribe, send any mail to
> "freebsd-questions-unsubscribe@freebsd.org"
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?AF1A08CE-7B20-11D9-B134-000D933E3CEC>