This project is about creating wrapper libraries to support monitoring and management applications to avoid direct use of the FreeBSD kernel memory interface. This approach would allow the kernel implementation to change and monitoring applications to be extended without breaking applications and requiring them to be recompiled. For this project, we propose to provide such a library for the network and process functions (i.e. libnetstat and libprocstat).

The goal of this project is to create wrapper libraries to support monitoring and management applicantions to avoid direct use of the kvm device in the FreeBSD kernel. This would allow the kernel implementation to change and monitoring to be extended without breaking applications and requiring them to be recompiled.

The initial idea is based on Robert Watson's libmemstat(3) library, where he created an abstract interface for collecting and providing information on memory allocation statistics. Its design and development was inspired by the following needs:

• Robustness for Application Binary Interface (ABI) over time. A library interface, rather than a system call, sysctl(8) variable, or kernel memory interface is the documented interface, and is designed to be robust against common changes to kernel data structures. For example, the struct inpcb (for Protocol Control Blocks) grows frequently as TCP gains new features, but the current sysctl(8) interface is not robust against that growth.

• Robustness for different word sizes. 32-bit monitoring tools should be able to work on a 64-bit kernel.

• Improved consistency when multiple sources are available. For example, there used to be a number of bugs involving netstat running on crash dumps, where it will once in a while show a value from the current running kernel instead because a new statistics for sysctl(8) has been added with no crash dump use in mind.

For this project, we propose to provide a library for the network and process functions (i.e. libnetstat and libprocstat). Hopefully, it is just a start, lessons learned during the design of this implementation could be used for creating similar libraries for other subsystems as time goes on.

The reason why libprocstat and libnetstat have been chosen, is that “the process” and “the network connection” are frequently monitored abstractions, with lot of FreeBSD and third-party tools wanting to show information about them.

For each case, there are three parts to be implemented:

• Add kernel support to export data in a less ABI-sensitive way.
• Write a library to present the information in an extensible way to applications.
• Update applications to use the library instead of reaching directly into the kernel internals.

For the implementation, we will need to find the most suitable abstractions. For example, the current libmemstat(3) API presents a set for memory types, and allows programs to query various statistics and properties on each type. In our opinion, the same could be implemented for process and network subsystems.

For netstat(1), possibly abstractions supporting the following functions will be needed (to “work around” the direct access to the kernel memory interface):

• Active sockets (protocol control blocks) for each network protocol, for particular protocol family, for a single protocol.
• Listen queues (unaccepted connections, unaccepted incomplete connections, maximum queued connections).
• Network interfaces (bytes in/out, dropped packets, watchdog timers, errors, and collisions), packet traffic.
• Statistics recorded by memory management routines (mbuf(9)).
• bpf(4) peers (packets matched, dropped, received, buffer size, device states).
• Routing statistics for routing tables, or routing tables for particular address family.
• IP multicast virtual-interface, routing tables, multicast group memberships.
• Multicast routing statistics.

For ps(1):

• Process environment.
• (Sub-)set of active processes in the kernel:
  • All processes and kernel visible threads.
  • All processes, without threads.
  • Processes with a given process identifier.
  • Processes with a given process group.
  • Processes with a given session.
  • Processes with a given TTY.
  • Processes with a given effective user identifier.
  • Processes with a given real user identifier.
• Command line arguments and environment variables passed to process (note that it is alterable by the process).
• procfs(5), accessing /proc/<pid>/mem (it is suggested to extend ptrace(2) to avoid procfs(5)).

For both cases, there are two options, namely -M and -N, which might require special attention (and might involve changes in the original software):

• Extract values associated with the name list from the specified core instead of the default /dev/kmem.
• Extract the name list from the specified system instead of the default, which is the kernel image the system has booted from.

Milestones are planned as follows:

• June 6, 2009. A more detailed analysis of utilities using statistical information (i.e. investigation of libmemstat(3), fstat(1), systat(1), etc.) to establish a common interface. An initial proposal for its design posted to the interested mailing lists to get opinions and feedback. It is possible that there will be case studies and prototypes on how to use the experimented interface.

• July 6, 2009. Based on the proposed design and the feedback received so far, provide an initial implementation for libnetstat, patches for the kernel, and a modified version of netstat(1) to use this library as much as possible. Start to implement tests.

• July 14, 2009. By following the experience gained from the implementation of libnetstat, an initial libprocstat library is also supplied. To make it useable and demonstrate its capabilities, some simple tools are included (similar to the ones were supplied for libmemstat(3)). Continue working on tests.

• July 21, 2009. Scrub and slush the implemented libraries, together with the accompanied kernel patches, make them more stable, profile performance, fix bugs, and catch up in case of slow progress. Create an initial documentation (i.e. manual pages, and a FreeBSD article, if time permits) for the interfaces and the tools.

• August 10, 2009. Final touches on the code, documentation. If there is time, start to extend the experimented interface to other statistics, build other libraries and tools, provide kernel patches for them.