Date: Tue, 20 May 2014 21:20:35 -0700 From: Alfred Perlstein <bright@mu.org> To: Erik Cederstrand <erik+lists@cederstrand.dk> Cc: freebsd-hackers@freebsd.org Subject: Re: [GSoC] Machine readable output from userland utilities Message-ID: <537C2993.1060206@mu.org> In-Reply-To: <91FE2526-F21C-42AB-BECB-058DBA975A9E@cederstrand.dk> References: <49E9736E-AD14-4647-8B15-30603D01360C@mail.bg> <91FE2526-F21C-42AB-BECB-058DBA975A9E@cederstrand.dk>
next in thread | previous in thread | raw e-mail | index | archive | help
On 5/20/14, 1:33 PM, Erik Cederstrand wrote:
> Den 20/05/2014 kl. 18.59 skrev Zaro Korchev <zkorchev@mail.bg>:
>
>> I'm working on the project "Machine readable output from userland utilities" and I want to share my ideas and thoughts.
>>
>> [...]
>> Do you have any suggestions or ideas?
> Microsoft PowerShell does something similar. They print human-readable output to the shell, but pass C# (I think) object instances along when piping data to other utilities. The idea of separating formatting from data is good, but it also leads to insanity because what you see is never what is passed on to the next utility. The utilities need to support the filtering and processing you would otherwise do with awk, grep, sed etc., and you need to look up two manual pages - one for the human output, and one for the C# models. Also, you can only combine utilities that were meant to be combined. This is a huge loss of flexibility compared to what I'm used to from UNIX.
>
> Has anyone given general thought to where the "Machine readable output from userland utilities" idea is headed? In particular, what should happen if I stick a pipe at the end of a command? Do we want utilities to understand XML/JSON for input, too? So for example 'rm' understands JSON from 'ls'. Unless this was really smart thought out, only tools that were prepared to work together would be able to do so, and you would need to look up which utilities work together.
>
> Even if we leave input alone and focus on output, we might want to let sed, grep, awk and friends understand JSON/XML input so we can e.g. use XPath syntax to grep for certain elements in XML output, or filter certain JSON elements from the output. At least I think FreeBSD should gain command-line tools that offer at a subset of the JSON and XML functionality that e.g. Python offers when working with these data formats. Using simple awk and grep with these formats would be regex madness.
>
> I've had long hours fighting with grep, awk and others to get simple scripts to behave properly due to the machine-unfriendly format of some tools, so I really like the idea of machine-readable output. sysctl offers some good examples of do's and dont's in this regard, and I feel that the concept of machine readability needs to be thought through very carefully to be exactly as awesome as when I first started grep'ing, awk'ing, sed'ing and piping my way through the UNIX shell.
>
Basically the idea would be to write a simple tool that is able to
extract using an xpath or json selector.
Example (very rough code):
ifconfig --output xml | selector --format xml --path /name --path
/name/etheraddr | \
while read name ether ; do
echo "Interface $name has hardware address $ether" ;
done
In all seriousness though, the real target is people writing higher
level languages (than shell) on top of FreeBSD. Perhaps python or ruby
spawning a utility and then that utility making the output easy to read.
One thing to note is that the output should not just be formatted but
normalized as well. The fact that "uptime" can emit 15 different
formats for the uptime string is terrible for people coding on top of
the base utils, the json/xml/other output should be decided on some form
of normalized data likely in seconds + microseconds or something, but
anything truly machine readable is better than the current output when
popen'd by a webapp.
-Alfred
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?537C2993.1060206>