PGJSoC2009 [FreeBSD WIKI]

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
===== Design and Implementation of Subsystem Support Libraries for Monitoring and Management =====

This project is about creating wrapper libraries to support monitoring and management applications to avoid direct use of the %%FreeBSD%% kernel memory and [[http://www.freebsd.org/cgi/man.cgi?query=sysctl&apropos=0&sektion=0&manpath=FreeBSD+8-current&format=html|sysctl(8)]] interface.  This approach would allow the kernel implementation to change and monitoring applications to be extended without breaking applications and requiring them to be recompiled.  For this project, we propose to provide such a library for the network statistics functions (i.e. //libnetstat//) as a proof of concept,
together with methods for building similar libraries for other subsystems.

==== Description ====

The goal of this project is to help creation of wrapper libraries to support monitoring and management applicantions to avoid direct use of the
[[http://www.freebsd.org/cgi/man.cgi?query=kvm&apropos=0&sektion=0&manpath=FreeBSD+8-current&format=html|kvm(3)]] device and the [[http://www.freebsd.org/cgi/man.cgi?query=sysctl&apropos=0&sektion=0&manpath=FreeBSD+8-current&format=html|sysctl(8)]] variables in the %%FreeBSD%% kernel.  This would allow the kernel implementation to //change// and monitoring to be extended //without breaking applications// and requiring them to be //recompiled//.

The [[http://unix.derkeiler.com/Mailing-Lists/FreeBSD/current/2005-07/0535.html|initial idea]] is based on Robert Watson's [[http://www.freebsd.org/cgi/man.cgi?query=libmemstat&apropos=0&sektion=0&manpath=FreeBSD+8-current&format=html|libmemstat(3)]] library, where he created an abstract interface for collecting and providing information on memory allocation statistics.  Its design and development was inspired by the following needs:

  * //Robustness for Application Binary Interface (ABI) over time.//  A library interface, rather than a system call, [[http://www.freebsd.org/cgi/man.cgi?query=sysctl&apropos=0&sektion=0&manpath=FreeBSD+8-current&format=html|sysctl(8)]] variable, or kernel memory interface is the documented interface, and is designed to be robust against common changes to kernel data structures.  For example, the ''%%struct inpcb%%'' (for Protocol Control Blocks) grows frequently as TCP gains new features, but the current [[http://www.freebsd.org/cgi/man.cgi?query=sysctl&apropos=0&sektion=0&manpath=FreeBSD+8-current&format=html|sysctl(8)]] interface is not robust against that growth.

  * //Robustness for different word sizes.//  32-bit monitoring tools should be able to work on a 64-bit kernel.

  * //Improved consistency when multiple sources are available.//  For example, there used to be a number of bugs involving [[http://www.freebsd.org/cgi/man.cgi?query=netstat&apropos=0&sektion=0&manpath=FreeBSD+8-current&format=html|netstat(1)]] running on crash dumps, where it will once in a while show a value from the current running kernel instead because a new statistics for [[http://www.freebsd.org/cgi/man.cgi?query=sysctl&apropos=0&sektion=0&manpath=FreeBSD+8-current&format=html|sysctl(8)]] has been added with no crash dump use in mind.

For this project, we propose to provide a library for the network statistics functions (i.e. //libnetstat//).   Hopefully, it is just a start, lessons learned during the design of this implementation could be used for creating similar libraries for other subsystems as time goes on.

The reason why libnetstat have been chosen, is that the network-related abstractions (e.g. "sockets", "socket buffers") are frequently monitored
abstractions, with lot of %%FreeBSD%% and third-party tools wanting to show information about them.

==== Components ====

There are three parts to be implemented:

  * Add kernel support for exporting data in a less ABI-sensitive way, with a focus on live monitoring via the [[http://www.freebsd.org/cgi/man.cgi?query=sysctl&apropos=0&sektion=0&manpath=FreeBSD+8-current&format=html|sysctl(8)]] interface.

  * Write a library to present the information in an extensible way to applications.  This is libnetstat.  It is derived from the sources of [[http://www.freebsd.org/cgi/man.cgi?query=netstat&apropos=0&sektion=0&manpath=FreeBSD+8-current&format=html|netstat(1)]] with the missing [[http://www.freebsd.org/cgi/man.cgi?query=kvm&apropos=0&sektion=0&manpath=FreeBSD+8-current&format=html|kvm(3)]] and [[http://www.freebsd.org/cgi/man.cgi?query=sysctl&apropos=0&sektion=0&manpath=FreeBSD+8-current&format=html|sysctl(8)]] interfaces added where needed.

  * Update applications to use this library instead of reaching directly into the kernel internals.  For now, two applications were selected: [[http://www.freebsd.org/cgi/man.cgi?query=netstat&apropos=0&sektion=0&manpath=FreeBSD+8-current&format=html|netstat(1)]] and [[http://www.freebsd.org/cgi/man.cgi?query=bsnmpd&apropos=0&sektion=0&manpath=FreeBSD+8-current&format=html|bsnmpd(1)]].

For the implementation, we will need to find the abstractions most suitable for the selected applications.  For example, the current [[http://www.freebsd.org/cgi/man.cgi?query=libmemstat&apropos=0&sektion=0&manpath=FreeBSD+8-current&format=html|libmemstat(3)]] API presents a set for memory types, and allows programs to query various statistics and properties on each type.  In our opinion, the same could be implemented for the network statistics.

Abstractions provided for the following functions:

  * Active sockets (protocol control blocks) and listen queues (unaccepted connections, unaccepted incomplete connections, maximum queued connections) for each network protocol, for particular protocol family, for a single protocol.

  * Network interfaces (bytes in/out, dropped packets, watchdog timers, errors, and collisions, packet traffic).

  * Statistics recorded by memory management routines ([[http://www.freebsd.org/cgi/man.cgi?query=netstat&apropos=0&sektion=0&manpath=FreeBSD+8-current&format=html|mbuf(9)]]).

  * [[http://www.freebsd.org/cgi/man.cgi?query=bpf&apropos=0&sektion=0&manpath=FreeBSD+8-current&format=html|bpf(4)]] peers (packets matched, dropped, received, buffer size, device states).

  * Routing statistics for routing tables, or routing tables for particular address family.

  * IP multicast virtual-interfaces, routing tables, multicast group memberships, multicast routing statistics.

==== Future Plans ====

  * Add SNMP support to //libnetstat//, so it could be used for remote hosts where [[http://www.freebsd.org/cgi/man.cgi?query=bsnmpd&apropos=0&sektion=0&manpath=FreeBSD+8-current&format=html|bsnmpd(1)]] is running.

  * Respect virtualized network stacks, so they can be monitored independently (for example, by a -J //jailname// flag).

  * Add support for information extraction from KVM images in an ABI-independent way.

==== Code ====

The current version of the implementation can be found in the project's [[http://p4db.freebsd.org/changeList.cgi?FSPC=%2F%2Fdepot%2Fprojects%2Fsoc2009%2Fpgj_libstat%2F...|Perforce depot]].