From owner-freebsd-performance@FreeBSD.ORG Wed Sep 24 15:09:24 2014 Return-Path: Delivered-To: freebsd-performance@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 09896A0D for ; Wed, 24 Sep 2014 15:09:24 +0000 (UTC) Received: from albert.catwhisker.org (mx.catwhisker.org [198.144.209.73]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id BBAF8A34 for ; Wed, 24 Sep 2014 15:09:23 +0000 (UTC) Received: from albert.catwhisker.org (localhost [127.0.0.1]) by albert.catwhisker.org (8.14.9/8.14.9) with ESMTP id s8OF9FZD031741 for ; Wed, 24 Sep 2014 08:09:15 -0700 (PDT) (envelope-from david@albert.catwhisker.org) Received: (from david@localhost) by albert.catwhisker.org (8.14.9/8.14.9/Submit) id s8OF9Frm031740 for freebsd-performance@FreeBSD.org; Wed, 24 Sep 2014 08:09:15 -0700 (PDT) (envelope-from david) Date: Wed, 24 Sep 2014 08:09:15 -0700 From: David Wolfskill To: freebsd-performance@FreeBSD.org Subject: Re: I like iostat, but... Message-ID: <20140924150915.GC1221@albert.catwhisker.org> Reply-To: freebsd-performance@FreeBSD.org MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="b0bHyb0BmsSIM7d2" Content-Disposition: inline In-Reply-To: <5421355D.1020305@freebsd.org> <20140923121945.8975311d308a1ff088dd90b0@systemdatarecorder.org> <659899B2-1816-41FA-9DED-57416928A1EE@sarenet.es> <20140923113844.6f9e9584965dfd401f6943af@systemdatarecorder.org> <35F9FD7B-6847-4209-A2F9-B1F0ABD01E4C@sarenet.es> <542094B2.5040302@FreeBSD.org> User-Agent: Mutt/1.5.23 (2014-03-12) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 Sep 2014 15:09:24 -0000 --b0bHyb0BmsSIM7d2 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Tue, Sep 23, 2014 at 12:29:22AM +0300, Andriy Gapon wrote: > On 23/09/2014 00:22, David Wolfskill wrote: > > ... I rather wish I could get the same information via sysctl. (Well, > > something seems to be available via the "opaque" kern.devstat.all > > sysctl(8) variable, but sysctl(8) doesn't display all of it, and parsing > > it seems as if that would require knowledge about the internals of the > > system where the data were acquired.) >=20 > Perhaps sysutils/devstat could be of help? > ... On Tue, Sep 23, 2014 at 08:56:54AM +0200, Borja Marcos wrote: > ...=20 > Reading sysctl from a small C program is not hard at all. I did it for de= vilator (a data recollector for Orca). And > there's a lot of data available. An advantage is, you avoid launching sev= eral processes. > ... On Tue, Sep 23, 2014 at 10:45:14AM +0200, Borja Marcos wrote: > ...=20 > Anyway, for disk stats GEOM offers a nice API. You can get delays per GEO= M provider, bandwidths, etc. On Tue, Sep 23, 2014 at 11:38:44AM +0300, Stefan Parvu wrote: > ...=20 > I gave up parsing sysctl via Perl for disks and network devices. It would= be > nice to have devstat properly working via sysctl for disk devices. Simila= r way > kern.cp_times does. Currently there is no simple way to extract per disk = stats from > sysctl as a Perl or Sh consumer, unless we build a C module to do that.= =20 Folks, I appreciate the suggestions, but they address problems other than the one I am trying to solve. In particular: * I require that the tool must only depend on components of base FreeBSD; thus, I don't need to perturb the system I want to measure by installing otherwise unneeded software on it. * I am using a shell script (which uses date, sysctl, netstat, and awk) so I don't need to recompile my data acquisition tool. * The tasks I am trying to measure are software builds similar to a stable/10 "make universe" -- but about 2 - 3 times the duration (and reading and writing a significantly larger amount of data). Thus, the number of additional processes caused by my probe firing even as often as once/second is lost in the noise. > ... > > If iostat(8) could be taught to (optionally) provide a timestamp, that > > might suffice. >=20 > In fact all performance userland tools should be able to nicely produce t= imestamp > CSV records which can be used for capacity planning and feed them to an > analytic product. Something like: >=20 > 1411445589:4.01:16.03:383.97:1.93:0.29:1.68:0.11:95.99:0.00:0.00:0.00:0.0= 0:123.00:229516.00:570992.00 >=20 > or something like iostat: >=20 > 1411445733:ada0:0.00:0.00:0.00:0.00:0.00:0.00:0.00 >=20 > where the first field would be always the timestmp, unix time. It is not = that complicated > but it does not exist. The output of my shell script may be described as: # ::=3D newline # ::=3D | tab # ::=3D : # ::=3D [_0-9a-zA-Z][-_.0-9a-zA-Z]+ # ::=3D [^\t\n]* # Each record will have a field with a tag called "time"; the # associated value will be the number of seconds since the Epoch, # but will be coerced as necessary to ensure that it is monotonically # increasing. # Note that the colon (":") is a valid character in the value part # of a field (but not in the tag). Further, there is no whitespace # on either side of that delimiting colon: everything to the left of the # colon (up to, but not including, the previous tab, if any) is part of the # tag; everything to the right of the colon (up to, but not including, # the next tab or newline) is part of the value. A prior version of the script output CSV; for this version, I chose to use the above format because I had situations where some fields showed up on some lines, but not on others. That tends to make the use of CSV a bit problematic. (On a machine where I post-process the collected data, I have some Perl that reads the above format, creates new field names as necessary to cope with multivariate data (e.g., kern.cp_time or vm.loadavg), and generates CSV from the result.) > ... > My idea of having standard data recorders: sysrec, cpurec, nicrec diskrec= =20 > which can extract: overall system consumption, per device statistics. > http://www.systemdatarecorder.org/recording/agents.html=20 > ... That looks interesting and useful for a broad class of similar problems. However, as far as I can tell, it is not suitable for the problem(s) I am trying to solve in this case. Basically, I have something that works "well enough" for things like CPU counters, memory usage (at the rather coarse granularity that top(1) provides, vs. "vmstat -m" output), load avergaes, and NIC counters, and is readily extensible to any univariate (or simple list of multivariate) (non-opaque) sysctl OIDs. I'd like to be able to include information from the I/O subsystem -- in particular, data that is accessible from "iostat -x". Peace, david --=20 David H. Wolfskill david@catwhisker.org Taliban: Evil cowards with guns afraid of truth from a 14-year old girl. See http://www.catwhisker.org/~david/publickey.gpg for my public key. --b0bHyb0BmsSIM7d2 Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQJ8BAEBCgBmBQJUIt6ZXxSAAAAAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25zLm9w ZW5wZ3AuZmlmdGhob3JzZW1hbi5uZXQ4RThEMDY4QTIxMjc1MDZFRDIzODYzRTc4 QTY3RjlDOERFRjQxOTNCAAoJEIpn+cje9Bk7cuUP/0abmI3lcrpBNvTfnEyHkake Kxy475LBqjtGYBTq4jiagQC3wrFCCGLarrW2EW5cmNKOOjmyUl2Fa7KCpccFN2LP JeJDA7rTSH5XNKgt5Xe8902AgQz5FBgkXZeN11eQgLzNs9rn3rjyvJibcdWvO5rj +4eaKct4YZ3HF5BSSXkfMQM8GB6quroZPAQ4/2ZywwFraIUVd8k3akHMsvTOeJ31 XVTB+jGo4Jk0Y/wDjIJ0oTo5q+VyXMJMbqt54J7B81n+CVXzjO1WI2Y+hSDMa/qU Oy4bt7FQnSDND00yqJEa+I/IEuyWljLnvGvbi7/7TUjqPXW7jMNE9BtGZjzf2gY7 5QcZ87pbuLvU8Acsz0n4+VlPTczt7khQ7ibCn6V2aqj930Ow49QVo4MUinksJUQd hiPd2FfqVV+7gL/TVVk5uGdYOQNIaPXfuItZIPoIOz42iECcVk+lkeY1pLHmMiYf s8O9kPcHm2F9PzqcfOUEm0+EAp+whh7rzQPAH7D9+4MkSvJ0tmNuVrIjPU+dt3ic aZvGgFU1RZkeBiLGStaKVJjSlV5BWEq/ole8nSMLjaYbMdEZ+15a3trDTX9NSq4x GDXN0rVbS/WrtQwOFGXmBaBg57ma4LK/GBcXM8QBbCGXwPueVIk7gw0RwoIOrRTx +JtK3oM4x7gkaVGfNHQC =snLV -----END PGP SIGNATURE----- --b0bHyb0BmsSIM7d2--