From owner-freebsd-arch@FreeBSD.ORG Fri Apr 5 15:00:03 2013 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 57860EA7; Fri, 5 Apr 2013 15:00:03 +0000 (UTC) (envelope-from alfred@ixsystems.com) Received: from mail.iXsystems.com (newknight.ixsystems.com [206.40.55.70]) by mx1.freebsd.org (Postfix) with ESMTP id 2F3D2B5D; Fri, 5 Apr 2013 15:00:02 +0000 (UTC) Received: from localhost (mail.ixsystems.com [10.2.55.1]) by mail.iXsystems.com (Postfix) with ESMTP id 8CC1C625F9; Fri, 5 Apr 2013 08:00:02 -0700 (PDT) Received: from mail.iXsystems.com ([10.2.55.1]) by localhost (mail.ixsystems.com [10.2.55.1]) (maiad, port 10024) with ESMTP id 64721-06-5; Fri, 5 Apr 2013 08:00:02 -0700 (PDT) Received: from [10.226.170.14] (45.sub-174-234-67.myvzw.com [174.234.67.45]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) by mail.iXsystems.com (Postfix) with ESMTPSA id 2185162597; Fri, 5 Apr 2013 07:59:54 -0700 (PDT) References: <20130401115128.GZ76816@FreeBSD.org> <20130402232606.GC1810@garage.freebsd.pl> <20130403002846.GB15334@onelab2.iet.unipi.it> <20130403100401.GA1349@garage.freebsd.pl> <515C68B5.2010006@ixsystems.com> <515E8E6E.4030706@FreeBSD.org> Mime-Version: 1.0 (1.0) In-Reply-To: <515E8E6E.4030706@FreeBSD.org> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Message-Id: <471D2765-393A-473F-A17C-FE1B77D15A6B@ixsystems.com> X-Mailer: iPhone Mail (10B329) From: Alfred Perlstein Subject: Re: collecting statistics / metrics Date: Fri, 5 Apr 2013 07:59:50 -0700 To: Andriy Gapon Cc: "arch@FreeBSD.org" X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 05 Apr 2013 15:00:03 -0000 On Apr 5, 2013, at 1:42 AM, Andriy Gapon wrote: > on 03/04/2013 20:36 Alfred Perlstein said the following: >> Hey folks, sorry for the top post here, but I just came into this thread.= >>=20 >> Here at iXsystems we've just developed a set of scripts to scrape the var= ious >> FreeBSD user land utilities (sysctl, netstat, nfsstat, vmstat, etc, etc) a= nd put >> them into graphs based on time. >>=20 >> The goal is to be able to line up all these metrics with whatever benchma= rk we >> are currently running and be able to see what may be causing issues. >>=20 >> Potentially you should be able to scroll through the graphs and see thing= s like >> "ran out of mbufs @time", "vm system began paging at @time", "buffer deae= mon >> went nuts @time" >>=20 >> Then we can take the information back and leverage it to make tuning deci= sions, >> or potentially change kernel algorithms. >=20 > This is very very useful! >=20 >> The only problem we have is that every user land tool has its own format,= so >> along with my team we have written some shell to coerce the output from t= he >> various programs into pseudo-CSV (key/value pair) which can then be post >> processed by tools to convert to CSV which can then be put into something= like >> open office, or put through an R program to graph it. >>=20 >> I'm hoping to have something shortly. >>=20 >> What I was hoping to do over the next few days was discuss with people ho= w we >> can (or should we even) fix the user land statistics tools to output mach= ine >> readable output that can be easily parsed. >>=20 >> Example: netstat -m (hard to parse) versus 'vmstat -z | grep mbuf' easy t= o parse. >>=20 >> The idea of outputting xml is good, CSV is OK, however CSV is problematic= as in >> the case of sysctl, if new nodes appear, then we can't begin to emit them= , we >> must either ignore them, or abort, or log them to auxiliary files. Anyth= ing >> that makes life easier is good. >>=20 >> I should be able to share our scripts within the next couple of days. >=20 > Just an alternative idea... > I think gathering all this information via plugins to e.g. collectd could b= e > more flexible and less processing / parsing intensive. That would allow t= o > avoid unnecessary formatting and re-parsing and to store the data in a > convenient format. Ideally it would be great to have an umbrella library o= n top > of sysctl, devstat, etc that would expose various stats in a convenient fo= rm. > Another thing of convenience would be an ability to know which sysctls are= > actually stats. I think that you have already done work towards this goal= . > There are certain heuristics that may help to distinguish stats from knobs= , > constants, etc, but the explicit "this is a metric" should be used. Of co= urse, > it would take a lot of work to properly mark all the sysctls. >=20 > Just thinking out loud. I'm going to bring these suggestions to my team and I think we can incorpora= te some of these ideas for sure.=20 -Alfred=20=