====== Xen bus_dma Grant Table Handler ====== **Student:** Vaibhav Kumar Gautam vaibhav@%%FreeBSD%%.org **Mentor:** Roger Pau Monné royger@%%FreeBSD%%.org ===== Project Description ===== The %%OpenBSD%% has integrated grant references into its bus_dma(9) subsystem, allowing Xen drivers to use a more transparent integration with the rest of the OS. The aim of this project is to bring the %%OpenBSD%% bus_dma(9) grant table handlers from %%OpenBSD%% into %%FreeBSD%%, expand them in order to support all the grant operations (share and map), and modify the existing frontends and backend in %%FreeBSD%% in order to make use of them. Grant tables represent a mechanism of passing references to pages of memory allocated by the guest across domains. This allows drivers for paravirtualized devices to take advantage of a standard approach to DMA memory management. ===== Approach to Solving the Problem ===== 1. Study the existing implementation of bus_dma(9) in %%OpenBSD%% and %%FreeBSD%% and find out the similarities in data structures, functions, and APIs that could be directly reused. 2. Study the grant table implementation in %%OpenBSD%% and %%FreeBSD%%, and map the APIs as per their functionalities. 3. Implement the prototype and test it on the Kernel. 4. Replicate the implementation to other areas using bus_dma (9). ===== Deliverables ===== **Mid-term deliverable:** Have a Xen-specific implementation of bus_dma(9) in order to map and share grants with domains. **Final deliverable:** Have the interface fully integrated with all the frontends and backends present in %%FreeBSD%%. ===== Milestones ===== ^ Starts ^ Ends ^ Description ^ Comments ^ Status ^ | **May 15** | **May 30** | Preparation work, community bonding, create wiki page, setup accounts and working environment, read %%FreeBSD%% coding guidelines, checked-out the %%FreeBSD%% code | This activity took time because I couldn’t setup suitable coding environment for myself. Since I did not download the GUI for %%FreeBSD%%, so I maintained two working environments, one for coding which is Ubuntu (preferred) and another for testing which is %%FreeBSD%% so, had to run OS on virtual machine. Therefore, it took time to setup right environment for myself suiting my needs. | ✔ | | **May 30** | **June 20** | %%FreeBSD%% code walkthrough (bus_dma, grant table), %%OpenBSD%% code walkthrough (xen.c), understanding virtualization, Xen, device drivers, debugging kernel, understanding code APIs - https://www.freebsd.org/cgi/man.cgi?query=busdma&sektion=9 | During this period, the biggest challenge was to understand the technical terms used in the literature [1], [2] and their relevance implementation wise [1]. For example, [1] directly talks about functions and data structures, which I had no idea. Before beginning the project, I tried to enhance my understanding on key concepts like virtualization [6], hypervisor [6], Xen [6], Xenstore [6]. After the above exercise, I could comprehend the terms used in the codebase and be able to relate the code’s functionality through comments. The [1] helped me in understanding the sys/dev/pv/xen.c in %%OpenBSD%%. When I looked at %%FreeBSD%% code the issue was what files to look at. I started with few random files with xen and bus_dma in its name or code. [5] and xen.c of %%OpenBSD%% helped me structuring my understanding of code flow, some trivial APIs and data structures common across BSD OS. After confirming from my mentor, now, I had identified the area I need to work on. I mapped the common functions like _bus_dmamap_*, referred [1] and started discovering the code. For grant table, I used the predefined %%FreeBSD%% APIs and didn’t create my own as in %%OpenBSD%%. **Note:** Numbers in the bracket represents the order number of the like in the useful link section below. | ✔ | | **June 20** | **June 26** | First prototype of integration of grant table into bus_dma(9) and seek improvements to make it a part of the first deliverable | In this period, I also designed several prototypes and tried to solve the problem in numerous ways. I crashed Kernel several times, even stuck in between because the idea required more work or the required information is missing and need to store it in some data structure. I debugged [7] the code only when it crashed to find its cause. By this time, I knew every small thing about the code I was working on and what was talked in [1] and [2]. | ✔ | | **June 26** | **June 30** | First deliverable to be delivered | I have delivered a small implementation of the project. My changes are limited to the busdma_bounce.c. I am able to store grant table references, assign them to the DMA maps, loading them and then destroying them. My implementation is similar to what it is done in %%OpenBSD%% but I have more ideas around implementation. I would be planning to extend it to other areas of code using bus_dma(9). | {o} | | **June 30** | **July 20** | Resume coding and integrate the functionality to all frontends and backends | | {o} | | **July 20** | **July 24** | Stress testing, bugs removal, further enhancements, and code review | | {o} | | **July 24** | **July 28** | Second deliverable to be delivered | | {o} | | **July 28** | **August 21** | Documentation, code review, comments incorporation, performance improvements, testing, and bugs removal | | {o} | ===== The Code ===== https://svnweb.freebsd.org/socsvn/soc2017/vaibhav/ ===== Useful links ===== **Xen bus_dma grant table handler** https://www.openbsd.org/papers/asiabsdcon2016-xen-paper.pdf http://www.openbsd.org/papers/asiabsdcon2016-xen-slides.pdf http://cvsweb.openbsd.org/cgi-bin/cvsweb/~checkout~/src/sys/dev/pv/xen.c?content-type=text/plain https://xenbits.xen.org/docs/unstable/misc/grant-tables.txt The above papers explained everything on integration of grant table in %%OpenBSD%%. The papers discussed data structures and APIs in detail which was helpful as some of the APIs could be reused. It also talked about correct way of handling the data (also how it is supposed to be done in %%FreeBSD%%) like maintaining a cookie, saving grant table references and updating addresses on ds_addr. Which again helped me in understanding the grant table implementation in %%OpenBSD%%. **Device Drivers** http://www.unixag-kl.fh-kl.de/~jkunz/projekte/NetBSD-driver_writing-1.0.1e.pdf.gz The above paper talked about writing device drivers. Though, it talked about %%NetBSD%%, it was helpful in debugging the code as some of the file structures and data structures are like %%FreeBSD%% and %%OpenBSD%%. For example, how device is attached and the use of _SOFTC structure for future reference is well explained in the paper and could be help in debugging. Also, since very less literature is available elsewhere, it is a good thing to start with. **Xen** https://www.netbsd.org/gallery/presentations/jym/eurobsdcon2011/BSDs_and_Xen.pdf Really helpful in getting an overview on Xen and virtualization. It also helped in understanding Xenstore, device drivers and domains which are widely used in the current project. **Kernel Debugging** https://people.freebsd.org/~syrinx/presentations/FreeBSD-debugging-draft.pdf