====== bhyve (and jails) manageability workstream ======

----

===== Documents =====

  * [[https://docs.google.com/document/d/1_0LEJqHT6v089YUI3PTRnNCLH8L14UZclvs2x_YWjtY/edit?usp=sharing|bhyve management PRD]]
  * [[https://docs.google.com/spreadsheets/d/1g_zuadGsCsAxS-Luxb53i2ZUjU2tai-G16uN97cza88/edit?usp=sharing|Desired bhyve features]]
  * [[https://docs.google.com/spreadsheets/d/1IJ5MylIZzHhakblAG0dM4mZkFKIXSpY3HskYYwuGG8w/edit#gid=0|Desired jails features]]

===== bhyve =====

There is a collection of desired features for [[https://docs.google.com/spreadsheets/d/1g_zuadGsCsAxS-Luxb53i2ZUjU2tai-G16uN97cza88/edit#gid=0|bhyve]].

As of 2023/11, priorities appear to be on

  - state management daemon: possibly including a kernel component; Jan B is working on a draft for this, but code is not ready to be shared
  - redesign of snapshot and restore functionality as precursor to live migration capability; Vitaly G is looking into options for redesign.

There is also some additional tooling to help construct better overall bhyve management tooling

  * [[https://gist.github.com/Crest/3c89deed4e9d8e9f5160ccdbee4c4745|mk-epair.sh]] helper script for managing epairs (useful for vnet jails)

==== Snapshot Redesign ====

There is a [[https://lists.freebsd.org/archives/freebsd-virtualization/2023-June/001341.html|mail thread]] about redesigning the snapshot file format and there have been numerous approaches to implementing live migration; the current consensus as of 2023/11 appears to be that a redesign of the snapshot file format is required first before moving into migration.

Unless file level issues are not fixed, adding migration functionality would only complicate the matter overall. As of 13.2, snapshot functionality is not enabled by default but requires activating an option in kernel configs, adding a ''%%make.conf%%'' option and then recompiling world and kernel.

| ID | Title || Status || Created || Last Update || Reviewers ||
| [[https://reviews.freebsd.org/D26387|D26387]] | bhyve - Snapshot Save and Restore multiple devices || Needs Review || Sep 10 2020, 10:21 AM || Jun 21 2023, 6:15 AM || jhb ||
| [[https://reviews.freebsd.org/D28270|D28270]] | Warm Migration feature for bhyve || Needs Review || Jan 21 2021, 2:19 PM || Jun 30 2021, 9:30 AM || jhb ||
| [[https://reviews.freebsd.org/D29262|D29262]] | Bhyve - Using JSON format for saving and restoring the state || Needs Review || Mar 14 2021, 5:37 PM || Jun 21 2023, 6:15 AM || jhb ||
| [[https://reviews.freebsd.org/D29538|D29538]] | bhyve: Move the gdb_active check to gdb_cpu_suspend(). || Closed/Accepted || Apr 1 2021, 4:47 PM || Apr 4 2021, 8:53 AM || - ||
| [[https://reviews.freebsd.org/D30471|D30471]] | Bhyve - Capsicum integration || Landed in different review || May 26 2021, 3:48 PM || Jun 21 2023, 6:15 AM || jhb ||
| [[https://reviews.freebsd.org/D30954|D30954]] | Live Migration feature for bhyve || Needs Review || Jun 30 2021, 10:31 AM || Jul 26 2021, 5:45 PM || jhb ||
| [[https://reviews.freebsd.org/D33431|D33431]] | bhyve snapshot fix for AMD CPUs || Closed/Denied || Dec 14 2021, 9:22 AM || Jun 30 2022, 11:15 PM || - ||
| [[https://reviews.freebsd.org/D34547|D34547]] | bhyve - snapshot capsicum integration[Part 1] || Needs Review || Mar 14 2022, 1:41 PM || Jun 21 2023, 6:15 AM || rew ||
| [[https://reviews.freebsd.org/D34718|D34718]] | Warm Migration feature for bhyve [Part 2] || Needs Revision || Mar 31 2022, 6:47 AM || Tue, Nov 7, 7:59 PM || (no:)afedorov, markj,  jhb, rew ||