AFreeBSD/bhyve version of the netmap virtual passthrough (ptnetmap) for VMs. Student: Stefano Garzarella Mentor: Luigi Rizzo ============================ Project description =============================== To avoid VMs networking bottlenecks, I recently worked with my advisors on ptnetmap: a netmap virtual passthrough for VMs (ptnetmap). ptnetmap allows userspace applications running in a guest VM to safely use any netmap port with near-native performance (physical devices [14.88 Mpps], software switches [20 Mpps], shared memory channels [75 Mpps]). I developed a preliminary version for linux/KVM hosts. For this GSoC project I'll develop host ptnetmap support for bhyve/FreeBSD. The code will be completely BSD licensed. (We wrote ourselves the entire ptnemtap code for linux, so we own the copyright on it). The implementation of ptnetmap will be based on our previous prototype above, which requires work in the following areas: - netmap framework modifications both for the guest part and the host part, to implement the virtio-like communication and the shared netmap memory; (existing code ~ 1900 loc in netmap framework + 300 loc OS specific; estimated new code ~ 300 + 300 loc) - guest device driver modifications for the setup of the notification mechanism and to share netmap memory; (estimated new code ~800 loc for virtio, assuming the existing netmap support for virtio developed by my advisor) - bhyve extensions to expose netmap host memory to the guest and to exchange notifications. We need a mechanism like linux's eventfd to exchange notifications and a way to expose the host memory to the guest through PCI device. (estimated new code unknown) =============================== Deliverables =================================== D1 (due by week 3): modify virtio device driver to allow the FreeBSD guest to use ptnetmap. (We can test it on linux-KVM where ptnetmap is already done.) Since the device driver is used for the setup, we expect to have the same performance of e1000. D2 (due by mid-term): map netmap host memory into the guest. We need to expose the memory of the netmap instance running in the host to the netmap instance running in the guest. Risk factor: In linux we used PCI BAR to expose the host memory to the guest, but we don't know if it is possible in easy way also in bhyve. D3 (due by week 9): kernel thread in netmap host adapter to implement ptnetmap on FreeBSD host. We expect to have the same performance of KVM (over 20 Mpps on VALE port; over 50 Mpps on netmap-pipe; line rate [14.88 Mpps] on Physical [10Gbps] NICs). The code that the kthread runs, is already done. We need to implement only the functions to create/delete kernel thread and to handle notifications. (~300 loc on linux). D4 (due by the end of project): implement notification mechanism in bhyve/FreeBSD. We want to have a similar mechanism to ioeventfd and irqfd available in linux, to exchange notifications between guest and host threads. Risk factor: We don't know how long it takes because in linux we had eventfd already implemented. ================================ Milestones ==================================== Start date: 2015/05/25 Estimated end dates: 2015/08/17-21 Timetable: Week 1-2: --- DONE --- add ptnetmap support to virtio device driver and QEMU frontend [D1]. Week 3: --- DONE --- test, bugfix, and performance evaluation [D1]. Week 4-5: --- DONE --- write code to map netmap host memory into the guest [D2]. Week 6: --- DONE --- test and bugfix [D2]. prepare documents to mid-term evaluation. Week 7-8: --- DONE --- implement kernel thread in netmap host adapter [D3]. Week 9: --- DONE --- test and bugfix [D3]. Week 10: --- DONE --- implement notification mechanism in bhyve/freebsd [D4]. Week 11: --- DONE --- test and performance evaluation [D4]. Week 12: --- DONE --- clean code and prepare documents to final evaluation. =========================== Final Documentation =============================== My code is available here: https://svnweb.freebsd.org/socsvn/soc2015/stefano/ptnetmap/stable/10/ I used the 10-STABLE branch for my work. To get the patches of modified modules you can follow the following steps: - bhyve svn diff -r 287649 usr.sbin/bhyve - virtio-net svn diff -r 287649 sys/dev/virtio/network - vmm.ko svn diff -r 287649 lib/libvmmapi sys/modules/vmm sys/amd64 - netmap the changes will be shortly committed in HEAD and R10 svn diff -r 287649 sys/conf/files sys/modules/netmap sys/dev/cxgbe \ sys/dev/netmap sys/net The ptnetmap support for linux-KVM and QEMU is available here: https://github.com/stefano-garzarella/ptnetmap I implemented ptnetmap on bhyve working on the following steps: - bhyve network backends I reused the code developed by Luigi Rizzo (my mentor) and Vincenzo Maffione to support multiple backend in bhyve and to interface them with a fronteds. The backends availabale are: - tap - netmap (netmap, vale) - ptnetmap (ptnetmap, ptvale) - ptnetmap support on virtio-net device for FreeBSD I modified the virtio-net guest device driver and the virtio-net fronted of thh host hypervisor (bhyve) to support ptnetmap. guest: - new PTNETMAP config flag in virtio-net device driver to check if ptnetmap is supported. - ptnetmap device-specific code (netmap - if_vtnet_netmap.h) host (hypervisor): - ptnetmap support on virtio-net fronted (bhyve - pci_virtio_ptnetmap.h) - ptnetmap backend [name: ptnetmap, ptvale] (bhyve - net_backends.c) - map netmap host memory into the guest I added a new IOCTL to vmm.ko to map an userspace guest buffer in the guest VM. Then I implemented a new PCI device (ptnetmap-memdev) to map the netmap host memory in the guest through PCI MEMORY-BAR. kernel host (vmm.ko): - new VM_MAP_USER_BUF ioctl to map buffer in the guest userspace host (bhyve): - vm_map_user_buf() in libvmmapi - ptnetmap‐memdev device emulation (bhyve - pci_ptnetmap_memdev.c) kernel guest (netmap): - device driver for ptnetmap-memdev inluded in netmap module - ptnetmap support for FreeBSD host: I implemented kernel thread in netmap module to support ptnetmap on FreeBSD host. kernel host (netmap): - nm_os_kthread_*() functions to handle netmap kthreads. (netmap - netmap_freebsd.c ptnetmap.c) - netmap guest/host notification mechanisms. I needed two mechanisms: 1) notification from ptnetmap kthread to guest VM (interrupt/irq) vmm.ko already has IOCTL to send interrupt to the guest and I used it in the ptnetmap kernel threads. 2) notification from guest VM to ptnetmap kthread (write into the specific register) I added new IOCTL on vmm.ko (VM_IO_REG_HANDLER) to catch write/read on specific I/O address and send notification. For now I've implemented only one type of handler (VM_IO_REGH_KWEVENTS) to use these events in the kernel through wakeup() and tsleep()/msleep(), but I wrote the code to be easily extended to support other type of handler (cond_signal, write/ioctl on fd, etc). kernel host (vmm.ko): - new VM_IO_REG_HANDLER ioctl to catch write/read on specific I/O address and to choose an handler. (eg. wakeup(event_id) - vm_io_reg_handler() in libvmmapi kernel host (netmap) - msleep() on event_id. userspace host (bhyve) - vm_io_reg_handler(VTCFG_R_QNOTIFY, event_id) send guest notifications to ptnetmap kthreads when the guest writes on this virtio register (bhyve - pci_virtio_ptnetmap.h) - tell to netmap kthreads the event_id where they can wait for events Example: Run a 2GB single‐CPU virtual machine with three network ports which use netmap and ptnetmap backends: bhyve ‐s 0,hostbridge ‐s 1,lpc \ ‐s 2:1,virtio‐net,vale0:1 \ /* normal vale backend */ ‐s 2:2,ptnetmap‐memdev \ /* ptnetmap-memdev is needed for each * ptnetmap port * (If two or more ptnetmap ports * share the same netmap memory allocator, * only one ptnetmap‐memdev is required) */ ‐s 2:3,virtio‐net,ptvale1:1 \ /* vale port in ptnetmap mode * If "pt" prefix is used, the port * is opened in passthrough mode (ptnetmap) */ ‐s 2:2,ptnetmap‐memdev \ ‐s 2:3,virtio‐net,ptvale2{1 \ /* netmap-pipe in ptnetmap mode */ ‐s 3,ptnetmap‐memdev \ ‐s 4,virtio‐net,ptnetmap:ix0 \ /* NIC in ptnetmap mode */ ‐l com1,stdio ‐A ‐H ‐P ‐m 2G netmapvm