From owner-freebsd-questions@FreeBSD.ORG Thu Feb 10 05:07:22 2005 Return-Path: Delivered-To: freebsd-questions@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 8A58716A4CE for ; Thu, 10 Feb 2005 05:07:22 +0000 (GMT) Received: from hub.org (hub.org [200.46.204.220]) by mx1.FreeBSD.org (Postfix) with ESMTP id DBD5843D55 for ; Thu, 10 Feb 2005 05:07:21 +0000 (GMT) (envelope-from scrappy@hub.org) Received: from localhost (unknown [200.46.204.144]) by hub.org (Postfix) with ESMTP id 26C6912ADDD; Thu, 10 Feb 2005 01:07:20 -0400 (AST) Received: from hub.org ([200.46.204.220]) by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) with ESMTP id 78901-06; Thu, 10 Feb 2005 05:07:20 +0000 (GMT) Received: from ganymede.hub.org (blk-224-186-245.eastlink.ca [24.224.186.245]) by hub.org (Postfix) with ESMTP id BE0241291F2; Thu, 10 Feb 2005 01:07:18 -0400 (AST) Received: by ganymede.hub.org (Postfix, from userid 1000) id CE925341CE; Thu, 10 Feb 2005 01:07:17 -0400 (AST) Received: from localhost (localhost [127.0.0.1]) by ganymede.hub.org (Postfix) with ESMTP id CD1ED33FC8; Thu, 10 Feb 2005 01:07:17 -0400 (AST) Date: Thu, 10 Feb 2005 01:07:17 -0400 (AST) From: "Marc G. Fournier" To: "Chad Leigh -- Shire.Net LLC" In-Reply-To: Message-ID: <20050210010628.C94338@ganymede.hub.org> References: <20050208231208.B94338@ganymede.hub.org> <20050209002232.B94338@ganymede.hub.org> <20050209210602.X94338@ganymede.hub.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Virus-Scanned: by amavisd-new at hub.org cc: freebsd-questions@freebsd.org Subject: Re: 99% CPU usage in System (Was: Re: vinum in 4.x poor performer?) X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 Feb 2005 05:07:22 -0000 On Wed, 9 Feb 2005, Chad Leigh -- Shire.Net LLC wrote: > > On Feb 9, 2005, at 6:34 PM, Marc G. Fournier wrote: > >> >> Most odd, there definitely has to be a problem with the Dual-Xeon ysystem >> ... doing the same vmstat on my other vinum based system, running more, but >> on a Dual-PIII shows major idle time: >> >> # vmstat 5 >> procs memory page disks faults cpu >> r b w avm fre flt re pi po fr sr da0 da1 in sy cs us sy >> id >> 20 1 0 4088636 219556 1664 1 2 1 3058 217 0 0 856 7937 2186 51 >> 15 34 >> 20 1 0 4115372 224220 472 0 0 0 2066 0 0 35 496 2915 745 7 7 >> 86 >> 10 1 0 4125252 221788 916 0 0 0 2513 0 2 71 798 4821 1538 6 >> 11 83 >> 9 1 0 36508 228452 534 0 0 2 2187 0 0 46 554 3384 1027 3 >> 8 89 >> 11 1 0 27672 218828 623 0 6 0 2337 0 0 61 583 2607 679 3 9 >> 88 >> 16 1 0 5776 220540 989 0 0 0 2393 0 9 32 514 3247 1115 3 >> 8 90 >> >> Which leads me further to believe this is a Dual-Xeon problem, and much >> further away from believing it has anything to do with software RAID :( > > I only use AMD, so I cannot provide specifics, but look in the BIOS at boot > time and see if there is anything strange looking in the settings. Unfortunately, I'm dealing with remote servers, so without something specific to get a remote tech to check, BIOS related stuff will have to wait until I can visit the servers persoally :( > Chad > >> >> >> On Wed, 9 Feb 2005, Marc G. Fournier wrote: >> >>> >>> still getting this: >>> >>> # vmstat 5 >>> procs memory page disks faults cpu >>> r b w avm fre flt re pi po fr sr da0 da1 in sy cs us sy >>> id >>> 11 2 0 3020036 267944 505 2 1 1 680 62 0 0 515 4005 918 7 38 >>> 55 >>> 19 2 0 3004568 268672 242 0 0 0 277 0 0 3 338 2767 690 1 99 >>> 0 >>> 21 2 0 2999152 271240 135 0 0 0 306 0 6 9 363 1749 525 1 99 >>> 0 >>> 13 2 0 3001508 269692 87 0 0 0 24 0 3 3 302 1524 285 1 99 >>> 0 >>> 17 2 0 3025892 268612 98 0 1 0 66 0 5 6 312 1523 479 3 97 >>> 0 >>> >>> Is there a way of determining what is sucking up so much Sys time? stuff >>> like pperl scripts running and such would use 'user time', no? I've got >>> some high CPU processes running, but would expect them to be shooting up >>> the 'user time' ... >>> >>> USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND >>> setiathome 21338 16.3 0.2 7888 7408 ?? RJ 9:05PM 0:11.35 >>> /usr/bin/perl -wT /usr/local/majordomo/bin/mj_queuerun -v 0 >>> setiathome 21380 15.1 0.1 2988 2484 ?? RsJ 9:06PM 0:02.42 >>> /usr/bin/perl -wT /usr/local/majordomo/bin/mj_enqueue -r -d postgresql.org >>> -l pgsql-sql -P10 -p10 >>> setiathome 21384 15.5 0.1 2988 2484 ?? RsJ 9:06PM 0:02.31 >>> /usr/bin/perl -wT /usr/local/majordomo/bin/mj_enqueue -r -d postgresql.org >>> -l pgsql-docs -P10 -p10 >>> setiathome 21389 15.0 0.1 2720 2216 ?? RsJ 9:06PM 0:02.06 >>> /usr/bin/perl -wT /usr/local/majordomo/bin/mj_enqueue -r -d postgresql.org >>> -l pgsql-hackers -P10 -p10 >>> setiathome 21386 13.7 0.1 2720 2216 ?? RsJ 9:06PM 0:02.03 >>> /usr/bin/perl -wT /usr/local/majordomo/bin/mj_enqueue -r -d postgresql.org >>> -l pgsql-ports -P10 -p10 >>> setiathome 21387 13.2 0.1 2724 2220 ?? RsJ 9:06PM 0:01.92 >>> /usr/bin/perl -wT /usr/local/majordomo/bin/mj_enqueue -r -d postgresql.org >>> -l pgsql-interfaces -P10 -p10 >>> setiathome 21390 14.6 0.1 2724 2216 ?? RsJ 9:06PM 0:01.93 >>> /usr/bin/perl -wT /usr/local/majordomo/bin/mj_enqueue -o -d postgresql.org >>> -l pgsql-performance -P10 -p10 >>> setiathome 21330 12.0 0.2 8492 7852 ?? RJ 9:05PM 0:15.55 >>> /usr/bin/perl -wT /dev/fd/3//usr/local/www/mj/mj_wwwusr (perl5.8.5) >>> setiathome 7864 8.9 0.2 8912 8452 ?? RJ 7:20PM 29:54.88 >>> /usr/bin/perl -wT /usr/local/majordomo/bin/mj_trigger -t hourly >>> >>> Is there some way of finding out where all the Sys Time is being used? >>> Something more fine grained them what vmstat/top shows? >>> >>> >>> On Wed, 9 Feb 2005, Loren M. Lang wrote: >>> >>>> On Wed, Feb 09, 2005 at 02:32:30AM -0400, Marc G. Fournier wrote: >>>>> Is there a command that I can run that provide me the syscall/sec value, >>>>> that I could use in a script? I know vmstat reports it, but is there an >>>>> easier way the having to parse the output? a perl module maybe, that >>>>> already does it? >>>> vmstat shouldn't be too hard to parse, try the following: >>>> vmstat|tail -1|awk '{print $15;}' >>>> To print out the 15th field of vmstat. Now if you want vmstat to keep >>>> running every five seconds or something, it's a little more complicated: >>>> vmstat 5|grep -v 'procs\|avm'|awk '{print $15;}' >>>>> Thanks ... >>>>> On Wed, 9 Feb 2005, Marc G. Fournier wrote: >>>>>> On Tue, 8 Feb 2005, Dan Nelson wrote: >>>>>>> Details on the array's performance, I think. Software RAID5 will >>>>>>> definitely have poor write performance (logging disks solve that >>>>>>> problem but vinum doesn't do that), but should have excellent read >>>>>>> rates. From this output, however: >>>>>>>> systat -v output help: >>>>>>>> 4 users Load 4.64 5.58 5.77 >>>>>>>> Proc:r p d s w Csw Trp Sys Int Sof Flt >>>>>>>> 24 9282 949 8414***** 678 349 8198 >>>>>>>> 54.6%Sys 0.2%Intr 45.2%User 0.0%Nice 0.0%Idl >>>>>>>> Disks da0 da1 da2 da3 da4 pass0 pass1 >>>>>>>> KB/t 5.32 9.50 12.52 16.00 9.00 0.00 0.00 >>>>>>>> tps 23 2 4 3 1 0 0 >>>>>>>> MB/s 0.12 0.01 0.05 0.04 0.01 0.00 0.00 >>>>>>>> % busy 3 1 1 1 0 0 0 >>>>>>> , it looks like your disks aren't being touched at all. You are doing >>>>>>> over 99999 syscalls/second, though, which is mighty high. The 50% Sys >>>>>>> doesn't look good either. You may have a runaway process doing some >>>>>>> syscall over and over. If this is not an MPSAFE syscall (see >>>>>>> /sys/kern/syscalls.master ), it will also prevent other processes from >>>>>>> making non-MPSAFE syscalls, and in 4.x that's most of them. >>>>>> Wow, that actually pointed me in the right direction, I think ... I >>>>>> just >>>>>> killed an http process that was using alot of CPU, and syscalls drop'd >>>>>> down to a numeric value again ... I'm still curious as to why this only >>>>>> seem sto affect my Dual-Xeon box though :( >>>>>> Thanks ... >>>>>> ---- >>>>>> Marc G. Fournier Hub.Org Networking Services >>>>>> (http://www.hub.org) >>>>>> Email: scrappy@hub.org Yahoo!: yscrappy ICQ: >>>>>> 7615664 >>>>>> _______________________________________________ >>>>>> freebsd-questions@freebsd.org mailing list >>>>>> http://lists.freebsd.org/mailman/listinfo/freebsd-questions >>>>>> To unsubscribe, send any mail to >>>>>> "freebsd-questions-unsubscribe@freebsd.org" >>>>> ---- >>>>> Marc G. Fournier Hub.Org Networking Services >>>>> (http://www.hub.org) >>>>> Email: scrappy@hub.org Yahoo!: yscrappy ICQ: >>>>> 7615664 >>>>> _______________________________________________ >>>>> freebsd-questions@freebsd.org mailing list >>>>> http://lists.freebsd.org/mailman/listinfo/freebsd-questions >>>>> To unsubscribe, send any mail to >>>>> "freebsd-questions-unsubscribe@freebsd.org" >>>> -- >>>> I sense much NT in you. >>>> NT leads to Bluescreen. >>>> Bluescreen leads to downtime. >>>> Downtime leads to suffering. >>>> NT is the path to the darkside. >>>> Powerful Unix is. >>>> Public Key: ftp://ftp.tallye.com/pub/lorenl_pubkey.asc >>>> Fingerprint: B3B9 D669 69C9 09EC 1BCD 835A FAF3 7A46 E4A3 280C >>> >>> ---- >>> Marc G. Fournier Hub.Org Networking Services >>> (http://www.hub.org) >>> Email: scrappy@hub.org Yahoo!: yscrappy ICQ: >>> 7615664 >>> _______________________________________________ >>> freebsd-questions@freebsd.org mailing list >>> http://lists.freebsd.org/mailman/listinfo/freebsd-questions >>> To unsubscribe, send any mail to >>> "freebsd-questions-unsubscribe@freebsd.org" >>> >> >> ---- >> Marc G. Fournier Hub.Org Networking Services (http://www.hub.org) >> Email: scrappy@hub.org Yahoo!: yscrappy ICQ: 7615664 >> _______________________________________________ >> freebsd-questions@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-questions >> To unsubscribe, send any mail to >> "freebsd-questions-unsubscribe@freebsd.org" > > ---- Marc G. Fournier Hub.Org Networking Services (http://www.hub.org) Email: scrappy@hub.org Yahoo!: yscrappy ICQ: 7615664