Requires PROC_LOCK for ucred to be reworked. Currently there is a lot of lock contention here. [Done in r284215]
This will allow us to assign actual memory limits per-jail. We currently use ulimit -m for per-process in jails.
Overrides for hw.ncpu, kern.smp.cpus and hw.physmem
Many ports will spy on these values and cause over-allocation of CPU and VM.
Java is one such port. It will see 96GB on the system and try to allocate much of that which can not only panic the system due to all of the other builds occurring on the system, but also prevents us from using ulimit -m since it will immediately break the limit.
Having a generic way to override any sysctl for a jail may make longterm sense for flexibility.
An alternative for all of these sysctls is to be able to configure the jail to limit its CPU and VM and have the values auto reflect the current values. options RACCT can already limit a jail's memory so should be considered if this feature is added.
Logging
Having all jail messages in dmesg(8) show the JID/JNAME would be nice for debugging package build issues. https://reviews.freebsd.org/D27 is a start to that by improving many places that currently matter for us.
Currently if a package build exceeds VM limits then the whole system goes into OOM state and could panic. Ideally we could control this better by marking the jails are first priority for OOM and stopping there. I.e., if OOM is invoked then only kill off the jails. Leave the host alone and do not kill off anything else. This is important not only for the package building cluster but for users as well. Many users with 1 or 2 CPU and ⇐8GB of RAM (think embedded even) will hit OOM while trying to use Poudriere to build packages. Poudriere is aggressive but it should not be risking panic/OOM so easily.