From owner-freebsd-hackers@FreeBSD.ORG Wed May 21 04:56:13 2014 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id C139EEC0 for ; Wed, 21 May 2014 04:56:13 +0000 (UTC) Received: from udns.ultimateDNS.NET (ultimatedns.net [209.180.214.225]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 37C8F2832 for ; Wed, 21 May 2014 04:56:12 +0000 (UTC) Received: from udns.ultimateDNS.NET (localhost [127.0.0.1]) by udns.ultimateDNS.NET (8.14.5/8.14.5) with ESMTP id s4L4wA9R034051; Tue, 20 May 2014 21:58:16 -0700 (PDT) (envelope-from bsd-lists@bsdforge.com) Received: (from www@localhost) by udns.ultimateDNS.NET (8.14.5/8.14.5/Submit) id s4L4w4Vo034045; Tue, 20 May 2014 21:58:04 -0700 (PDT) (envelope-from bsd-lists@bsdforge.com) Received: from udns.ultimatedns.net ([209.180.214.225]) (UDNSMS authenticated user chrish) by ultimatedns.net with HTTP; Tue, 20 May 2014 21:58:04 -0700 (PDT) Message-ID: In-Reply-To: <537C2993.1060206@mu.org> References: <49E9736E-AD14-4647-8B15-30603D01360C@mail.bg> <91FE2526-F21C-42AB-BECB-058DBA975A9E@cederstrand.dk> <537C2993.1060206@mu.org> Date: Tue, 20 May 2014 21:58:04 -0700 (PDT) Subject: Re: [GSoC] Machine readable output from userland utilities From: "Chris H" To: "Alfred Perlstein" User-Agent: UDNSMS/2.0.3 MIME-Version: 1.0 Content-Type: text/plain;charset=utf-8 Content-Transfer-Encoding: 8bit X-Priority: 3 (Normal) Importance: Normal Cc: freebsd-hackers@freebsd.org, Erik Cederstrand X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 May 2014 04:56:13 -0000 > > On 5/20/14, 1:33 PM, Erik Cederstrand wrote: >> Den 20/05/2014 kl. 18.59 skrev Zaro Korchev : >> >>> I'm working on the project "Machine readable output from userland utilities" and I want >>> to share my ideas and thoughts. >>> >>> [...] >>> Do you have any suggestions or ideas? >> Microsoft PowerShell does something similar. They print human-readable output to the >> shell, but pass C# (I think) object instances along when piping data to other utilities. >> The idea of separating formatting from data is good, but it also leads to insanity because >> what you see is never what is passed on to the next utility. The utilities need to support >> the filtering and processing you would otherwise do with awk, grep, sed etc., and you need >> to look up two manual pages - one for the human output, and one for the C# models. Also, >> you can only combine utilities that were meant to be combined. This is a huge loss of >> flexibility compared to what I'm used to from UNIX. >> >> Has anyone given general thought to where the "Machine readable output from userland >> utilities" idea is headed? In particular, what should happen if I stick a pipe at the end >> of a command? Do we want utilities to understand XML/JSON for input, too? So for example >> 'rm' understands JSON from 'ls'. Unless this was really smart thought out, only tools that >> were prepared to work together would be able to do so, and you would need to look up which >> utilities work together. >> >> Even if we leave input alone and focus on output, we might want to let sed, grep, awk and >> friends understand JSON/XML input so we can e.g. use XPath syntax to grep for certain >> elements in XML output, or filter certain JSON elements from the output. At least I think >> FreeBSD should gain command-line tools that offer at a subset of the JSON and XML >> functionality that e.g. Python offers when working with these data formats. Using simple >> awk and grep with these formats would be regex madness. >> >> I've had long hours fighting with grep, awk and others to get simple scripts to behave >> properly due to the machine-unfriendly format of some tools, so I really like the idea of >> machine-readable output. sysctl offers some good examples of do's and dont's in this >> regard, and I feel that the concept of machine readability needs to be thought through >> very carefully to be exactly as awesome as when I first started grep'ing, awk'ing, sed'ing >> and piping my way through the UNIX shell. >> > > Basically the idea would be to write a simple tool that is able to > extract using an xpath or json selector. > > Example (very rough code): > > ifconfig --output xml | selector --format xml --path /name --path > /name/etheraddr | \ > while read name ether ; do > echo "Interface $name has hardware address $ether" ; > done > > In all seriousness though, the real target is people writing higher > level languages (than shell) on top of FreeBSD. Perhaps python or ruby > spawning a utility and then that utility making the output easy to read. > > One thing to note is that the output should not just be formatted but > normalized as well. The fact that "uptime" can emit 15 different > formats for the uptime string is terrible for people coding on top of > the base utils, the json/xml/other output should be decided on some form > of normalized data likely in seconds + microseconds or something, but > anything truly machine readable is better than the current output when > popen'd by a webapp. > > -Alfred Greetings, all. I may be getting into this thread a bit late in the game. But if I understand the gist of this correctly; isn't all this pretty much what Perl was intended for? All the best. --Chris > > > > > > > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" >