From owner-freebsd-hackers@FreeBSD.ORG Mon May 26 22:45:21 2008 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8D4E7106564A for ; Mon, 26 May 2008 22:45:21 +0000 (UTC) (envelope-from mathieu.prevot@gmail.com) Received: from fk-out-0910.google.com (fk-out-0910.google.com [209.85.128.187]) by mx1.freebsd.org (Postfix) with ESMTP id 11B8C8FC20 for ; Mon, 26 May 2008 22:45:20 +0000 (UTC) (envelope-from mathieu.prevot@gmail.com) Received: by fk-out-0910.google.com with SMTP id k31so2105552fkk.11 for ; Mon, 26 May 2008 15:45:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; bh=nxooj2yjs0mnvCC6D9ZF+4N+D1x1SQMRQrm5UyKdSoE=; b=KkdORh61Pw0v0itdfEWnLR38qyqi7YUOYhzsp7i/WP33MskFJGMGsSgtEnan6W4jT9eXmzekZ4Fix4NPiMiyWjGcKntI1ldskN3IR00fAVYXMF3Fo1yJ3sTQADo+LOR0GnhfX7/lwETWgWbMej4+bQCNkP9aIQaqF8s3fJ9RgoM= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; b=WTj7FKP9zBo0inStFEZNaxc9kz+1y0tKfB/BTVDssTazl7O2oHuANdJEbCRV8jRJvXMgGTcrMeGrbaF0HayLDO+ce16kqixdAhoeKkdsy1dPI+ywR1YNYHMfDrl913YK4Yypj8W9RxWRpsnSt7H/2TFl+9aoxX2NuSw77iJlzIw= Received: by 10.82.154.5 with SMTP id b5mr188405bue.45.1211833657400; Mon, 26 May 2008 13:27:37 -0700 (PDT) Received: by 10.82.170.10 with HTTP; Mon, 26 May 2008 13:27:37 -0700 (PDT) Message-ID: <3e473cc60805261327g38349857ga87dcfa0b855c1f6@mail.gmail.com> Date: Mon, 26 May 2008 22:27:37 +0200 From: "Mathieu Prevot" To: "Marc Spitzer" In-Reply-To: <8c50a3c30805261237y6573a2c9ra4af56477d93af78@mail.gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <3e473cc60805260752o2f573cf2h12910a45cf6849e6@mail.gmail.com> <20080526151614.GA5582@dan.emsphone.com> <3e473cc60805260914r537f113eh8d840a94c0a5d958@mail.gmail.com> <8c50a3c30805261237y6573a2c9ra4af56477d93af78@mail.gmail.com> Cc: hackers@freebsd.org, Dan Nelson Subject: Re: network bitrate of a poll of processes X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 26 May 2008 22:45:21 -0000 2008/5/26 Marc Spitzer : > On Mon, May 26, 2008 at 12:14 PM, Mathieu Prevot > wrote: >> 2008/5/26 Dan Nelson : >>> In the last episode (May 26), Mathieu Prevot said: >>>> Hi, >>>> >>>> I would like to know the bitrate of a pool of child processes that use >>>> a network connection, how can I have something like netstat -w1 >>>> provide but at the process level ? >>> >>> If you can segregate them to their own UID, you can use an ipfw "count >>> ip from any to any uid 6666" rule to count the packets. Another option >>> would be to jail them to a dedicated IP address and count traffic on >>> that IP. >> >> Both are nice... thank you. And don't you have something more portable >> ie. that could run on "any" posix os - like a sh script that use >> standard tools - and that doesn't need root user intervention ? >> >> Mathieu > > I do not think so, firewall for example is controlled by root. And > users do not generally have detailed access to other users > information. You can look into the BASM framework to track what your > apps are doing and if you know the behavior to expect you can use > netflow/argus/tcpdump to track that and post process the logs in each > case into something useful. > > What are you trying to do, details do help? > > marc I have a script that generates many child processes that download videos (~10MB) in this case, or do heavy data processing/analysing. I can tune the number of workers as I want, but It's a shared cluster so I would like to 1) measure 2) optimize/control (by number of workers ? at the script level ! so I can update thing very dynamically) a) CPU load b) IO on drive(s) c) network load and I prefer solutions that don't need root access/intervention. I can measure indirectly network load using `date +%s` and `du -b |sed 's/\t.*//' ` regularly... but I may have to deal with several 10GB later - and actually the number of files is more critical than their size. Maybe writing a (very) quick and dirty python app that listen/attach to a group of processes (for the network case) ? Mathieu