From owner-freebsd-arch@FreeBSD.ORG Wed Apr 3 18:00:56 2013 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 79FEA9DF; Wed, 3 Apr 2013 18:00:56 +0000 (UTC) (envelope-from alfred@ixsystems.com) Received: from mail.iXsystems.com (newknight.ixsystems.com [206.40.55.70]) by mx1.freebsd.org (Postfix) with ESMTP id 476F6353; Wed, 3 Apr 2013 18:00:56 +0000 (UTC) Received: from localhost (mail.ixsystems.com [10.2.55.1]) by mail.iXsystems.com (Postfix) with ESMTP id C252E61EB5; Wed, 3 Apr 2013 11:00:54 -0700 (PDT) Received: from mail.iXsystems.com ([10.2.55.1]) by localhost (mail.ixsystems.com [10.2.55.1]) (maiad, port 10024) with ESMTP id 59753-07-4; Wed, 3 Apr 2013 11:00:54 -0700 (PDT) Received: from Alfreds-MacBook-Pro-9.local (unknown [10.8.0.26]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) by mail.iXsystems.com (Postfix) with ESMTPSA id 5031E61E8F; Wed, 3 Apr 2013 11:00:38 -0700 (PDT) Message-ID: <515C6E3A.9020300@ixsystems.com> Date: Wed, 03 Apr 2013 11:00:26 -0700 From: Alfred Perlstein User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:17.0) Gecko/20130328 Thunderbird/17.0.5 MIME-Version: 1.0 To: Adrian Chadd Subject: Re: [CFR][CFT] counter(9): new API for faster and raceless counters References: <20130401115128.GZ76816@FreeBSD.org> <20130402232606.GC1810@garage.freebsd.pl> <20130403002846.GB15334@onelab2.iet.unipi.it> <20130403100401.GA1349@garage.freebsd.pl> <515C68B5.2010006@ixsystems.com> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: arch@freebsd.org, Gleb Smirnoff X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 03 Apr 2013 18:00:56 -0000 On 4/3/13 10:45 AM, Adrian Chadd wrote: > On 3 April 2013 10:36, Alfred Perlstein wrote: >> Hey folks, sorry for the top post here, but I just came into this thread. >> >> Here at iXsystems we've just developed a set of scripts to scrape the >> various FreeBSD user land utilities (sysctl, netstat, nfsstat, vmstat, etc, >> etc) and put them into graphs based on time. > Cool! > >> The goal is to be able to line up all these metrics with whatever benchmark >> we are currently running and be able to see what may be causing issues. >> >> Potentially you should be able to scroll through the graphs and see things >> like "ran out of mbufs @time", "vm system began paging at @time", "buffer >> deaemon went nuts @time" >> >> Then we can take the information back and leverage it to make tuning >> decisions, or potentially change kernel algorithms. >> >> The only problem we have is that every user land tool has its own format, so >> along with my team we have written some shell to coerce the output from the >> various programs into pseudo-CSV (key/value pair) which can then be post >> processed by tools to convert to CSV which can then be put into something >> like open office, or put through an R program to graph it. >> >> I'm hoping to have something shortly. >> >> What I was hoping to do over the next few days was discuss with people how >> we can (or should we even) fix the user land statistics tools to output >> machine readable output that can be easily parsed. >> >> Example: netstat -m (hard to parse) versus 'vmstat -z | grep mbuf' easy to >> parse. >> >> The idea of outputting xml is good, CSV is OK, however CSV is problematic as >> in the case of sysctl, if new nodes appear, then we can't begin to emit >> them, we must either ignore them, or abort, or log them to auxiliary files. >> Anything that makes life easier is good. >> >> I should be able to share our scripts within the next couple of days. > that's quite shiny. > > It'd be really nice if we could come up with a consistent statistics > display, summary and export library. > > That way we could write tools that used a given data fetch/display API > and then we could have optional "things" that implement various export > methods. > > For example, I'm packing up Sam Leffler's "libstatfoo" for inclusion > into -HEAD, primarily so the tools that use it (wlanstats, mwlstats, > athstats and all the other ath stats programs I'm using) can use it. > > But once I've converted the stats tool over to that, I can do a few cute things: > > * the library has a generic way to list all of the supported > statistics fields - you register the statistic names with the library, > then you can create arbitrary format strings with the information; > * the library handles "now" versus "time series" data display itself - > you just need to populate the statistic structure with the relevant > stats and then call the right function to display things; > * I plan on extending it to spit out CSV output as a generic feature, > so I can start doing things like importing the output of those tools > direct into rrdtool/etc without any intermediary parsing scripts; > * .. and if I then add new stats fields, the (requested) output of the > script won't change, so tools won't break. > > Very cool. One thing to note, the CSV format: COLNAME1,COLNAME2,COLNAME3 DATA1,DATA2,DATA3 is vulnerable to problems where a new column will spring into being due to loading of a kernel module/driver/something. Imo it's better to look at XML or some other pseudo-CSV like: COLNAME1:DATA1,COLNAME2:DATA2,COLNAME3:DATA3 so that we are OK with columns springing into existence or leaving. -Alfred