Vimage Coding - beginners guide / FAQ

• Q. What are all the V_ prefixes on variable names?
• A. These are virtualized variables. Note: do not confuse the v or VOP prefixes from vnode(9) oder VFS(9) with these.
• A. The V_ prefix is/was needed to distinguish the the (former) globals from their virtualized counterparts so that a macro could be used to switch between variants. Moreover, while the majority (but not all) of the former globals could have been in fact directly converted to macros without the V_ prefix prepended, the V_ prefix helps to instantly differentiate between true globals and virtualized variables when reading the code.

• Q. What do those V_ macros do exactly?
• A. Depending on the variant compiled, the V_ macros will resolve the variable to either a global, a member of a global container struct or a member of a container struct hanging off of a virtual image.

• Q. What variants are there?
• A. During the transition there are three, in the end there will be two variants:
  1. (during transition only) There are the global variables as they had been forever (option VIMAGE_GLOBALS).
  2. There are a few globals of container structs holding the former globals if no virtualization is compiled in (nooptions VIMAGE).
  3. There is full (network stack) virtualization (options VIMAGE).
• A. See also KernelOptions for a more detailed description.

• Q. What is a container struct?
• A. A container struct is a large C structure holding the former globals of a specific subsystem or module like INET, INET6, IPSEC, gif(4), … .

• Q. Where do I find the container structs?
• A. Some of the structs got their own header file, along with the V_ macros, etc., like:
  • sys/net/vnet.h
  • sys/netinet/vinet.h
  • sys/netinet6/vinet6.h
  • .. while some others were just added to the header files of a subsystem or module with more limited scope like:
  • sys/net/if_gif.h
  • sys/netgraph/netgraph.h
  • sys/netipsec/ipsec.h
  • sys/netinet/ip_fw.h
  • ..

• Q. Why do you still keep the globals if you have the container structs?
• A. The globals will eventually go away. During the transition period they were needed to be able to bring in the code within smaller chunks. They are kept until the full virtualization is in to be able to benchmark the same code with all three variants. Once we are confident that everything works and there is no huge regression, the globals (under #ifdef VIMAGE_GLOBALS) will be removed from the source tree and only the container structs will be kept. We will try to get rid of the globals as soon as possible as for now it means that developers need to mainain all virtualized variables in two places.

• Q. You were talking about a few new globals, why do you need those?
• A. In its final version one will be able to compile virtualization in or out the kernel. In case of the of virtualization things will be hung off a vimage like:

struct vimage *vimage        /* The vimage */
    struct vnet *v_net       /* The virtualize network stack. */
        void *mod_data[n]    /* The different subsystem types like INET, INET6, GIF, ... */

Obviously lower levels might be cached somewhere else to not always need the full indirections from *vimage. In case virtualization is compiled out, there will always be only a single instance of a container struct for each subsystem type. Those are globals then, like:

#ifndef VIMAGE
struct vnet_inet vnet_inet_0;
struct vnet_ipfw vnet_ipfw_0;
struct vnet_netgraph vnet_netgraph_0;
struct vnet_gif vnet_gif_0;
...
#endif

(obviously not under a single #ifndef but each of the globals in the file close(st) to the subsystem they represent).

• Q. Why is there more than a single global?
• Q. Why is there more than a single container struct for the network stack?
• A. There are multiple reasons:
  • With a single global only one change to the struct would require complete recompiliation.
  • Third party vendors would not be able to define their things for their own private modules or self-reliant subsystems easily.
  • Compiling out of parts like IPsec would be harder.
  • We would need even more #include spamming on unrelated parts of the network stack and modules.
  • ..

• Q. How do you initialize the members of the container structs?
• A. There is no static initialization as before. Initialization functions, that had been there partly before, are now initializing the variables, either globals or in the container structs depending on how things were compiled. The former initialization values were kept so that the defaults did not change. The developers should be careful not to reintroduce static initializers (initializers at variable instantiation) for globals corresponding to virtualized variables, until the globals currently enclosed in #ifdef VIMAGE_GLOBALS block eventually get completely removed.

• Q. Why are there INIT_VNET_foo() macros at the beginning of a lot of functions?
• A. Unless full virtualization is on, those macros are NOPs and resolve to an empty line. If virtualization is compiled in the macro provides a base pointer for the foo type or module. This would be the equivalent to the global vnet_ipfw_0 for example, just being a pointer *vnet_ipfw from one of the running virtualized instances. The base pointers are usually taken from a cached copy of the struct vnet *v_net pointer. Else we would possibly have to go through various stages of indirection for each variable access.

• Q. Why is TCP or UDP or … not an own type?
• A. The problem in the network stack is that there are places that already need to access variables from multiple (3+-n) types like INET, INET6 and IPSEC for example. Adding too many types will mean lots of local base variables and it might no longer be able to have them all in registers, etc. which would hurt performance. That said we may want to rethink some of the distribution of variables into different types in the future.

• Q. I want to know more about virtualizing my own code. Where can I find the real beef description?
• A. There is a document in Perforce. You can always find the latest version here: http://p4web.freebsd.org/@md=d&cd=//&c=N0M@//depot/projects/vimage/porting_to_vimage.txt?ac=22. You should ignore all the comments about V_CPU or V_PROCG for now. There is experimental code in Perforce but it will not be comitted to SVN (for now).
• A. There is the mailto:freebsd-virtualization@freebsd.org mailing list where you can ask for help. Here is the archives and subscription page: http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization.