Date: Mon, 26 May 2008 17:02:34 -0400 From: "Marc Spitzer" <mspitzer@gmail.com> To: "Mathieu Prevot" <mathieu.prevot@gmail.com> Cc: hackers@freebsd.org, Dan Nelson <dnelson@allantgroup.com> Subject: Re: network bitrate of a poll of processes Message-ID: <8c50a3c30805261402y596a37a2m61a8993b37d13d7b@mail.gmail.com> In-Reply-To: <3e473cc60805261327g38349857ga87dcfa0b855c1f6@mail.gmail.com> References: <3e473cc60805260752o2f573cf2h12910a45cf6849e6@mail.gmail.com> <20080526151614.GA5582@dan.emsphone.com> <3e473cc60805260914r537f113eh8d840a94c0a5d958@mail.gmail.com> <8c50a3c30805261237y6573a2c9ra4af56477d93af78@mail.gmail.com> <3e473cc60805261327g38349857ga87dcfa0b855c1f6@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Mon, May 26, 2008 at 4:27 PM, Mathieu Prevot <mathieu.prevot@gmail.com> wrote: > 2008/5/26 Marc Spitzer <mspitzer@gmail.com>: >> On Mon, May 26, 2008 at 12:14 PM, Mathieu Prevot >> <mathieu.prevot@gmail.com> wrote: >>> 2008/5/26 Dan Nelson <dnelson@allantgroup.com>: >>>> In the last episode (May 26), Mathieu Prevot said: >>>>> Hi, >>>>> >>>>> I would like to know the bitrate of a pool of child processes that use >>>>> a network connection, how can I have something like netstat -w1 >>>>> provide but at the process level ? >>>> >>>> If you can segregate them to their own UID, you can use an ipfw "count >>>> ip from any to any uid 6666" rule to count the packets. Another option >>>> would be to jail them to a dedicated IP address and count traffic on >>>> that IP. >>> >>> Both are nice... thank you. And don't you have something more portable >>> ie. that could run on "any" posix os - like a sh script that use >>> standard tools - and that doesn't need root user intervention ? >>> >>> Mathieu >> >> I do not think so, firewall for example is controlled by root. And >> users do not generally have detailed access to other users >> information. You can look into the BASM framework to track what your >> apps are doing and if you know the behavior to expect you can use >> netflow/argus/tcpdump to track that and post process the logs in each >> case into something useful. >> >> What are you trying to do, details do help? >> >> marc > > I have a script that generates many child processes that download > videos (~10MB) in this case, or do heavy data processing/analysing. I > can tune the number of workers as I want, but It's a shared cluster so > I would like to > > 1) measure 2) optimize/control (by number of workers ? at the script > level ! so I can update thing very dynamically) > a) CPU load b) IO on drive(s) c) network load > > and I prefer solutions that don't need root access/intervention. I can > measure indirectly network load using `date +%s` and `du -b |sed > 's/\t.*//' ` regularly... but I may have to deal with several 10GB > later - and actually the number of files is more critical than their > size. > Maybe writing a (very) quick and dirty python app that listen/attach > to a group of processes (for the network case) ? > > Mathieu > for the CPU stuff take a look at ps: bob% ps aO%mem,%cpu,ppid PID %MEM %CPU PPID TT STAT TIME COMMAND 768 0.2 0.0 1 v0 Is+ 0:00.01 /usr/libexec/getty Pc ttyv0 769 0.2 0.0 1 v1 Is+ 0:00.01 /usr/libexec/getty Pc ttyv1 770 0.2 0.0 1 v2 Is+ 0:00.01 /usr/libexec/getty Pc ttyv2 771 0.2 0.0 1 v3 Is+ 0:00.01 /usr/libexec/getty Pc ttyv3 772 0.2 0.0 1 v4 Is+ 0:00.01 /usr/libexec/getty Pc ttyv4 773 0.2 0.0 1 v5 Is+ 0:00.01 /usr/libexec/getty Pc ttyv5 774 0.2 0.0 1 v6 Is+ 0:00.01 /usr/libexec/getty Pc ttyv6 775 0.2 0.0 1 v7 Is+ 0:00.01 /usr/libexec/getty Pc ttyv7 272 0.2 0.0 1 con- I 0:00.04 dhclient: le0 [priv] (dhclient) 951 0.6 0.0 949 p0 Rs 0:00.28 -zsh (zsh) 1056 0.2 0.0 951 p0 R+ 0:00.01 ps aO%mem,%cpu,ppid add what you need and you have your cpu and such stats, it might do your IO as well also look for thing in (s)bin that end in stat for things that might be a better fit. Network I do not know how to get you the level of access you want without root/trusted access at least on collection. You will not need it on processing. as far as managing thing after you have the information I have nothing for you there. good luck, marc -- Freedom is nothing but a chance to be better. Albert Camus
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?8c50a3c30805261402y596a37a2m61a8993b37d13d7b>