Skip site navigation (1)Skip section navigation (2)
Date:      Thu, 10 Feb 2005 01:07:17 -0400 (AST)
From:      "Marc G. Fournier" <scrappy@hub.org>
To:        "Chad Leigh -- Shire.Net LLC" <chad@shire.net>
Cc:        freebsd-questions@freebsd.org
Subject:   Re: 99% CPU usage in System (Was: Re: vinum in 4.x poor performer?)
Message-ID:  <20050210010628.C94338@ganymede.hub.org>
In-Reply-To: <AF1A08CE-7B20-11D9-B134-000D933E3CEC@shire.net>
References:  <20050208231208.B94338@ganymede.hub.org> <20050209002232.B94338@ganymede.hub.org> <20050209210602.X94338@ganymede.hub.org> <AF1A08CE-7B20-11D9-B134-000D933E3CEC@shire.net>

next in thread | previous in thread | raw e-mail | index | archive | help
On Wed, 9 Feb 2005, Chad Leigh -- Shire.Net LLC wrote:

>
> On Feb 9, 2005, at 6:34 PM, Marc G. Fournier wrote:
>
>> 
>> Most odd, there definitely has to be a problem with the Dual-Xeon ysystem 
>> ... doing the same vmstat on my other vinum based system, running more, but 
>> on a Dual-PIII shows major idle time:
>> 
>> # vmstat 5
>>  procs      memory      page                    disks     faults      cpu
>>  r b w     avm    fre  flt  re  pi  po  fr  sr da0 da1   in   sy  cs us sy 
>> id
>> 20 1 0 4088636 219556 1664   1   2   1 3058 217   0   0  856 7937 2186 51 
>> 15 34
>> 20 1 0 4115372 224220  472   0   0   0 2066   0   0  35  496 2915 745  7  7 
>> 86
>> 10 1 0 4125252 221788  916   0   0   0 2513   0   2  71  798 4821 1538  6 
>> 11 83
>>  9 1 0   36508 228452  534   0   0   2 2187   0   0  46  554 3384 1027  3 
>> 8 89
>> 11 1 0   27672 218828  623   0   6   0 2337   0   0  61  583 2607 679  3  9 
>> 88
>> 16 1 0    5776 220540  989   0   0   0 2393   0   9  32  514 3247 1115  3 
>> 8 90
>> 
>> Which leads me further to believe this is a Dual-Xeon problem, and much 
>> further away from believing it has anything to do with software RAID :(
>
> I only use AMD, so I cannot provide specifics, but look in the BIOS at boot 
> time and see if there is anything strange looking in the settings.

Unfortunately, I'm dealing with remote servers, so without something 
specific to get a remote tech to check, BIOS related stuff will have to 
wait until I can visit the servers persoally :(

> Chad
>
>> 
>> 
>> On Wed, 9 Feb 2005, Marc G. Fournier wrote:
>> 
>>> 
>>> still getting this:
>>> 
>>> # vmstat 5
>>> procs      memory      page                    disks     faults      cpu
>>> r b w     avm    fre  flt  re  pi  po  fr  sr da0 da1   in   sy  cs us sy 
>>> id
>>> 11 2 0 3020036 267944  505   2   1   1 680  62   0   0  515 4005 918  7 38 
>>> 55
>>> 19 2 0 3004568 268672  242   0   0   0 277   0   0   3  338 2767 690  1 99 
>>> 0
>>> 21 2 0 2999152 271240  135   0   0   0 306   0   6   9  363 1749 525  1 99 
>>> 0
>>> 13 2 0 3001508 269692   87   0   0   0  24   0   3   3  302 1524 285  1 99 
>>> 0
>>> 17 2 0 3025892 268612   98   0   1   0  66   0   5   6  312 1523 479  3 97 
>>> 0
>>> 
>>> Is there a way of determining what is sucking up so much Sys time?  stuff 
>>> like pperl scripts running and such would use 'user time', no?  I've got 
>>> some high CPU processes running, but would expect them to be shooting up 
>>> the 'user time' ...
>>> 
>>> USER         PID %CPU %MEM   VSZ  RSS  TT  STAT STARTED      TIME COMMAND
>>> setiathome 21338 16.3  0.2  7888 7408  ??  RJ    9:05PM   0:11.35 
>>> /usr/bin/perl -wT /usr/local/majordomo/bin/mj_queuerun -v 0
>>> setiathome 21380 15.1  0.1  2988 2484  ??  RsJ   9:06PM   0:02.42 
>>> /usr/bin/perl -wT /usr/local/majordomo/bin/mj_enqueue -r -d postgresql.org 
>>> -l pgsql-sql -P10 -p10
>>> setiathome 21384 15.5  0.1  2988 2484  ??  RsJ   9:06PM   0:02.31 
>>> /usr/bin/perl -wT /usr/local/majordomo/bin/mj_enqueue -r -d postgresql.org 
>>> -l pgsql-docs -P10 -p10
>>> setiathome 21389 15.0  0.1  2720 2216  ??  RsJ   9:06PM   0:02.06 
>>> /usr/bin/perl -wT /usr/local/majordomo/bin/mj_enqueue -r -d postgresql.org 
>>> -l pgsql-hackers -P10 -p10
>>> setiathome 21386 13.7  0.1  2720 2216  ??  RsJ   9:06PM   0:02.03 
>>> /usr/bin/perl -wT /usr/local/majordomo/bin/mj_enqueue -r -d postgresql.org 
>>> -l pgsql-ports -P10 -p10
>>> setiathome 21387 13.2  0.1  2724 2220  ??  RsJ   9:06PM   0:01.92 
>>> /usr/bin/perl -wT /usr/local/majordomo/bin/mj_enqueue -r -d postgresql.org 
>>> -l pgsql-interfaces -P10 -p10
>>> setiathome 21390 14.6  0.1  2724 2216  ??  RsJ   9:06PM   0:01.93 
>>> /usr/bin/perl -wT /usr/local/majordomo/bin/mj_enqueue -o -d postgresql.org 
>>> -l pgsql-performance -P10 -p10
>>> setiathome 21330 12.0  0.2  8492 7852  ??  RJ    9:05PM   0:15.55 
>>> /usr/bin/perl -wT /dev/fd/3//usr/local/www/mj/mj_wwwusr (perl5.8.5)
>>> setiathome  7864  8.9  0.2  8912 8452  ??  RJ    7:20PM  29:54.88 
>>> /usr/bin/perl -wT /usr/local/majordomo/bin/mj_trigger -t hourly
>>> 
>>> Is there some way of finding out where all the Sys Time is being used? 
>>> Something more fine grained them what vmstat/top shows?
>>> 
>>> 
>>> On Wed, 9 Feb 2005, Loren M. Lang wrote:
>>> 
>>>> On Wed, Feb 09, 2005 at 02:32:30AM -0400, Marc G. Fournier wrote:
>>>>> Is there a command that I can run that provide me the syscall/sec value,
>>>>> that I could use in a script?  I know vmstat reports it, but is there an
>>>>> easier way the having to parse the output? a perl module maybe, that
>>>>> already does it?
>>>> vmstat shouldn't be too hard to parse, try the following:
>>>> vmstat|tail -1|awk '{print $15;}'
>>>> To print out the 15th field of vmstat.  Now if you want vmstat to keep
>>>> running every five seconds or something, it's a little more complicated:
>>>> vmstat 5|grep -v 'procs\|avm'|awk '{print $15;}'
>>>>> Thanks ...
>>>>> On Wed, 9 Feb 2005, Marc G. Fournier wrote:
>>>>>> On Tue, 8 Feb 2005, Dan Nelson wrote:
>>>>>>> Details on the array's performance, I think.  Software RAID5 will
>>>>>>> definitely have poor write performance (logging disks solve that
>>>>>>> problem but vinum doesn't do that), but should have excellent read
>>>>>>> rates.  From this output, however:
>>>>>>>> systat -v output help:
>>>>>>>>    4 users    Load  4.64  5.58  5.77
>>>>>>>> Proc:r  p  d  s  w    Csw  Trp  Sys  Int  Sof  Flt
>>>>>>>>    24     9282       949 8414*****  678  349 8198
>>>>>>>> 54.6%Sys   0.2%Intr 45.2%User  0.0%Nice  0.0%Idl
>>>>>>>> Disks   da0   da1   da2   da3   da4 pass0 pass1
>>>>>>>> KB/t   5.32  9.50 12.52 16.00  9.00  0.00  0.00
>>>>>>>> tps      23     2     4     3     1     0     0
>>>>>>>> MB/s   0.12  0.01  0.05  0.04  0.01  0.00  0.00
>>>>>>>> % busy    3     1     1     1     0     0     0
>>>>>>> , it looks like your disks aren't being touched at all.  You are doing
>>>>>>> over 99999 syscalls/second, though, which is mighty high.  The 50% Sys
>>>>>>> doesn't look good either.  You may have a runaway process doing some
>>>>>>> syscall over and over.  If this is not an MPSAFE syscall (see
>>>>>>> /sys/kern/syscalls.master ), it will also prevent other processes from
>>>>>>> making non-MPSAFE syscalls, and in 4.x that's most of them.
>>>>>> Wow, that actually pointed me in the right direction, I think ... I 
>>>>>> just
>>>>>> killed an http process that was using alot of CPU, and syscalls drop'd
>>>>>> down to a numeric value again ... I'm still curious as to why this only
>>>>>> seem sto affect my Dual-Xeon box though :(
>>>>>> Thanks ...
>>>>>> ----
>>>>>> Marc G. Fournier           Hub.Org Networking Services 
>>>>>> (http://www.hub.org)
>>>>>> Email: scrappy@hub.org           Yahoo!: yscrappy              ICQ: 
>>>>>> 7615664
>>>>>> _______________________________________________
>>>>>> freebsd-questions@freebsd.org mailing list
>>>>>> http://lists.freebsd.org/mailman/listinfo/freebsd-questions
>>>>>> To unsubscribe, send any mail to
>>>>>> "freebsd-questions-unsubscribe@freebsd.org"
>>>>> ----
>>>>> Marc G. Fournier           Hub.Org Networking Services 
>>>>> (http://www.hub.org)
>>>>> Email: scrappy@hub.org           Yahoo!: yscrappy              ICQ: 
>>>>> 7615664
>>>>> _______________________________________________
>>>>> freebsd-questions@freebsd.org mailing list
>>>>> http://lists.freebsd.org/mailman/listinfo/freebsd-questions
>>>>> To unsubscribe, send any mail to 
>>>>> "freebsd-questions-unsubscribe@freebsd.org"
>>>> -- 
>>>> I sense much NT in you.
>>>> NT leads to Bluescreen.
>>>> Bluescreen leads to downtime.
>>>> Downtime leads to suffering.
>>>> NT is the path to the darkside.
>>>> Powerful Unix is.
>>>> Public Key: ftp://ftp.tallye.com/pub/lorenl_pubkey.asc
>>>> Fingerprint: B3B9 D669 69C9 09EC 1BCD  835A FAF3 7A46 E4A3 280C
>>> 
>>> ----
>>> Marc G. Fournier           Hub.Org Networking Services 
>>> (http://www.hub.org)
>>> Email: scrappy@hub.org           Yahoo!: yscrappy              ICQ: 
>>> 7615664
>>> _______________________________________________
>>> freebsd-questions@freebsd.org mailing list
>>> http://lists.freebsd.org/mailman/listinfo/freebsd-questions
>>> To unsubscribe, send any mail to 
>>> "freebsd-questions-unsubscribe@freebsd.org"
>>> 
>> 
>> ----
>> Marc G. Fournier           Hub.Org Networking Services (http://www.hub.org)
>> Email: scrappy@hub.org           Yahoo!: yscrappy              ICQ: 7615664
>> _______________________________________________
>> freebsd-questions@freebsd.org mailing list
>> http://lists.freebsd.org/mailman/listinfo/freebsd-questions
>> To unsubscribe, send any mail to 
>> "freebsd-questions-unsubscribe@freebsd.org"
>
>

----
Marc G. Fournier           Hub.Org Networking Services (http://www.hub.org)
Email: scrappy@hub.org           Yahoo!: yscrappy              ICQ: 7615664



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20050210010628.C94338>