This is an old revision of the document!
Note: This is a strawman WIP. Nothing is final.
Tasks / Roadmap
Kernel
This is an initial batch of ideas. As we get further down the road we will probably add more items to this list (including more APIs which need NUMA awareness).
| Description | Status | Owner | Commit / Branch / Patch |
| Parsing SRAT on x86 and adding domains to vm_phys | done | attilio / jeff / jhb | stable/10 |
| bus_get_domain() and dev.foo.N.%domain | committed | adrian / jhb | r272799 / r274976 |
CPU_WHICH_DOMAIN and cpuset -gd | needs testing | jeff / jhb | r276829 |
| Teach topology code about NUMA domains (for x86 hierarchy is package → domain → core → thread) | not started | ||
| bus_get_cpus() to query arbitrary CPU sets (including “local” CPUs and “best intr” CPUs) | needs testing | jhb | https://github.com/bsdjhb/freebsd/compare/bsdjhb:master...numa_bus_get_cpus |
| Assign interrupts to a local CPU in intr_cpus by default on x86 | not started | ||
| Design a NUMA allocation policy data type | in progress | jeff | projects/numa |
| Remove the “cache” page queue (makes subsequent vm_phys changes simpler) | in progress | alc / jeff | |
| Update the vm_phys layer to accept NUMA allocation policy | in progress | jeff | projects/numa |
| Update contigmalloc, kmem_* to accept NUMA allocation policy | in progress | jeff | projects/numa |
| Update static bus_dma allocations to allocate memory local to the device by default | not started | ||
| Add NUMA awareness to UMA | in progress | jeff | projects/numa |
| Per-domain page daemon improvements | not started | ||
| Per-domain free list locking | not started |
Userland
One open question here is what range of policies do we want to support? Linux supports a process-wide allocation policy that can be overridden for specific mappings. Do we want to support process-wide policies? Do we want per-thread policies as well? Per-object policies? Also, for the range of policies supported, what is the precedence ordering?
| Description | Status | Owner | Commit / Branch / Patch |
| Prototype process-wide policies | in progress | jeff | projects/numa |
| Implement mapping policies (vm_map) | not started | ||
| libnuma-like API? | not started | ||
| numactl-like functionality to adjust policy for new and/or existing processes | not started | ||
| A monitoring tool akin to numa-top | not started |
Reviews
| Link | Description |
| https://reviews.freebsd.org/D1897 | migrate taskqueue_start_threads_pinned() → taskqueue_start_threads_cpuset() |
| https://reviews.freebsd.org/D1676 | remove vm object cache |
| https://reviews.freebsd.org/D1675 | enable more aggressive contiguous memory reclamation |
| https://reviews.freebsd.org/D1674 | skip gratuitous inactive queueing |
| https://reviews.freebsd.org/D1672 | per-cpu page cache |