Network Stack KPI/KBI
This wiki page attempts to identify kernel data structures which, if changed after a .0 release on a branch, may prevent network device drivers, firewall modules, as well as userland monitoring tools, etc, from working properly.
Data structure status key
The following status options exist for each structure:
Immutable - This data structure must not be changed
The data structure is very sensitive to change, as it may be allocated statically in BSS or on the stack by modules, or directly accessed by userland programs that require it to remain the same length. Changing the size of the structure could lead to (possibly extensive) kernel memory corruption, broken userland monitoring tools, etc.
Padded - Spare fields may be changed as long as the size remains the same
The same as immutable except that spare fields have been added to the structure in order to allow new functionality to be added despite the structure layout being fixed. Care should be taken when using spare fields, especially relating to the lengths of types like long or void *, which may vary by architecture. Likewise, the initialized values of structures may vary based on how the are allocated, so old code may not be aware of new initialization requirements: if allocated in BSS, fields may simply be zeroed, but if on the stack they may contain arbitrary values.
Extendable - New fields may be added at the end of the data structure
These data structures are visible in APIs used by third-party code, but the data structures are always allocated and managed by the base kernel; existing fields can't be changed, but new fields may be added, with caution.
Private - May be changed with (relative) impunity
These data structures are used only inside of the kernel or specific modules, and the layout may be changed without (excessive) concern.
Key to usage
The following uses are of particular interest from a KPI/KBI perspective:
| Abbreviation | Description |
|---|---|
| MD | Module dereference: kernel modules encode the offset and type of structure fields in their implementation. |
| MA | Module allocation: kernel modules encode size of structure in their implementation, possibly in BSS, on the stack, or as an a direct argument to malloc(9) or uma(9). |
| SE | Sysctl export: the data structure is exposed to userspace via the sysctl(2) system call, and therefore part of the ABI. |
| SC | System call argument: data structure is an argument to a system call. |
| KD | KVM dereference: userspace monitoring or post-mortem analysis tools encode offset and type of structure fields in their implementation. |
| KS | KVM size dependency: userspace monitoring or post-mortem analysis tools encode the length of the structure in their implementation (i.e., due to the struct being allocated in an array in the kernel). |
sys
| Type | Status | MD | MA | SE | SC | KD | KS | Comments |
|---|---|---|---|---|---|---|---|---|
| struct mbuf | Immutable | Yes | ?? | No | No | No | No | Extreme caution required. |
| struct sockopt | Immutable | Yes | Yes | No | No | No | No | Allocated on the stack in kernel modules. |
| struct sockaddr | Immutable | Yes | Yes | Yes | Yes | ?? | ?? | Allocated statically, embedded in kernel structures, and part of the system call ABI. |
| struct unpcb | Immutable | Yes | No | Yes | No | Yes | ?? | Allocated only by uipc_usrreq.c, but accessed by portalfs, and, mysteriously, embedded in xunpcb, or would be Extendable. |
| struct xsocket | Immutable | No | No | Yes | No | No | No | Exposed explicitly via sysctl to userspace. |
| struct xunpcb | Immutable | No | No | No | Yes | No | No | Allocated only by uipc_usrreq.c, but exposed explicitly via sysctl to userspace. |
net
| Type | Status | Comments |
|---|---|---|
| struct bpf_d | Extendable | Allocated only in bpf.c, but fields accessed by MAC policy modules. |
| struct ifaddr | Immutable | Embedded in other data structures in protocols, such as in_ifaddr, etc. |
| struct ifnet | Padded | Allocated only in if.c, but used in all device drivers, and monitored from userspace via KVM. May now be safe to change this to Extendable. |
| struct netisr_handler | Padded | Allocated statically by protocol modules in BSS, but padded for future growth. |
| struct route | Immutable | Allocated statically in BSS, inside of data structures, and on the stack by protocols, device drivers, etc. |
| struct rtentry | Immutable | Generally only allocated by route.c, but used throughout the kernel. Would be Extendable if it weren't also allocated statically in one place by pf.c. |
netinet
| Type | Status | Comments |
|---|---|---|
| struct in_addr | Immutable | Embedded in countless user and kernel data structures and ABIs. |
| struct in_ifaddr | Extendable | Allocated only by in.c, but used in many places in net, netinet, netipsec. |
| struct inpcb | Padded | Allocated from a number of zones, depending on protocol type; exposed explicitly to userspace via xinpcb, xtcpcb via sysctl. Without exposure via sysctl, might be Extendable but would need to review all protocols to make sure that is safe. Is accessed from modules including firewalls, IPSEC, and TOE. |
| struct inpcbinfo | Extendable | Accessed by user monitoring tools via KVM. |
| struct ipq | Extendable | Allocated only by ip_input.c, but some fields read by MAC policy modules. |
| struct sackhint | Padded | Used only inside of TCP, but embedded in struct tcpcb so is exposed to userspace and other kernel modules with tcpcb visibility. |
| struct sockaddr_in | Immutable | Allocated statically and part of the system call ABI; embedded in other data structures. |
| struct syncache | Extendable | Currently allocated/consumed only in TCP, but user monitoring tools will access it via KVM in the future. |
| struct syncache_head | Private | Currently allocated and accessed only by TCP, but may be accessed via KVM by user monitoring tools in the future. |
| struct tcp_hostcache | Extendable | Currently allocated/consumed only in TCP, but user monitoring tools will access it via KVM in the future. |
| struct tcpcb | Immutable | Currently allocated only by TCP, but accessed in a number of places including NFS, and exported to userspace directly via xtcpcb in sysctl. Would probably be Extendable if it weren't exported directly. |
| struct tcpopt | Padded | Allocated and used only within TCP – except for TOE modules, also allocated it (on the stack) and pass it into syncache entry initialization. |
| struct tcptw | Private | Currently allocated and accessed only by TCP, but may be accessed via KVM by user monitoring tools in the future. |
| struct tcp_syncache | Extendable | User monitoring tools will access it via KVM in the future. |
| struct udpdb | Private | Currently allocated and accessed only by UDP/UDP6, but may be accessed via KVM by user monitoring tools in the future. |