Machine readable output from userland utilities

• Student: ZarkoKorchev (zkorchev@)
• Mentor: EitanAdler (lists@eitanadler.com)

The output of many commands is intended to be read by humans. An unfortunate consequence is that machine parsing of these utilities is difficult and time consuming. The unfortunate effect is that retrieving most infomration requires either complex scripting and regular expressions, or it requires coding C directly to the system calls.

This proposal aims to fix that by providing an interface for userland utilities to emit machine readable output which can be easily parsed by other software components.

Commercial vendors have been modifying FreeBSD for many years to emit machine readable (XML/JSON/YAML) output for common commands to aid scripting against their implementations. A noteable example is JUNOS from Juniper Networking.

What we would do is leverage an existing library for emiting the most commmon scalar types (string, integer, float, array) and modify base FreeBSD utilities to call a library emit XML, JSON or YAML as opposed to directly calling printf.

Utilities we will tackle include:

• sysctl
• ifconfig
• netstat
• iostat
• vmstat

I am going to use the existing libraries (libnv and libucl). In the process I will make some changes to these libraries (like adding XML support to libucl). I will convert command line utilities like ls, sysctl, ifconfig, netstat, iostat, vmstat, procstat, fstat, etc. to use these libraries as an output layer.

• Support for JSON, XML and YAML output for ls, sysctl, ifconfig, netstat, iostat, vmstat, procstat, fstat.
• A library with plugable backends for formatted data output.
• Documentation and blog posts during the writing process.

After this project is ready, it will be easy to add support for machine-readable output to any command line tool. The converted tools will serve as examples for future contributors.

• Until 18 May: Get familiar with the source code. Analyze the output format of the command-line tools.
• 19 May - 25 May: Design a suitable output format for the command-line utilities.
• 26 May - 1 June: Design output abstractions that will allow to easily switch between output formats.
• 2 June - 15 June: Implement the underlying output layer in the form of library. The library must have pluggable back-ends for output.
• 16 June - 22 June: Implement library back-end that prints human-readable output.
• 23 June - 13 July: Adapt each of the command-line utilities to use the output layer.
• 14 July - 20 July: Test whether the output of the library matches the original output from the tools.
• 21 July - 3 August: Add library support for JSON, XML and YAML output.
• 4 August - 10 August: Test whether everything works and fix the bugs.
• 11 August - 17 August: Write documentation.

https://socsvn.freebsd.org/socsvn/soc2014/zkorchev