Date: Mon, 17 Aug 2015 11:09:45 +0200 From: Stefano Garzarella <stefanogarzarella@gmail.com> To: soc-status@freebsd.org Cc: Luigi Rizzo <rizzo@iet.unipi.it>, Peter Grehan <grehan@freebsd.org>, Neel Natu <neel@freebsd.org> Subject: ptnetmap on bhyve - final status report Message-ID: <CAO0mX5ZmRp7HoAJVmHC39_QYSuBHe_zpZ8V0YOasBeS=X3PnpQ@mail.gmail.com>
next in thread | raw e-mail | index | archive | help
Dear All,
I finished the last step on my GSoC 2015 project ("A FreeBSD/bhyve version
of
the netmap virtual passthrough (ptnetmap) for VMs").
In the last week I tested my code and I added more comments in bhyve,
vmm.ko,
and netmap, to describe my work.
I also added some lines in bhyve.8 man page to explain how use ptnetmap
backend.
My code is available here:
https://svnweb.freebsd.org/socsvn/soc2015/stefano/ptnetmap/stable/10/
I used the 10-STABLE branch for my work.
To get the patches of modified modules, you can follow the following steps
on my
stable/10 branch:
- bhyve
svn diff -r 287649 usr.sbin/bhyve
- virtio-net
svn diff -r 287649 sys/dev/virtio/network
- vmm.ko
svn diff -r 287649 lib/libvmmapi sys/modules/vmm sys/amd64
- netmap
the changes will be shortly committed in HEAD and R10 by my mentor
svn diff -r 287649 sys/conf/files sys/modules/netmap sys/dev/cxgbe
\
sys/dev/netmap sys/net
The ptnetmap support for linux-KVM and QEMU is available here:
https://github.com/stefano-garzarella/ptnetmap
I implemented ptnetmap on bhyve working on the following steps:
- bhyve network backends
I reused the code developed by Luigi Rizzo (my mentor) and Vincenz=
o
Maffione to support multiple backend in bhyve and to interface the=
m
with a fronteds.
The backends availabale are:
- tap
- netmap (netmap, vale)
- ptnetmap (ptnetmap, ptvale)
- ptnetmap support on virtio-net device for FreeBSD
I modified the virtio-net guest device driver and the virtio-net
fronted
of thh host hypervisor (bhyve) to support ptnetmap.
guest:
- new PTNETMAP config flag in virtio-net device driver to
check
if ptnetmap is supported.
- ptnetmap device-specific code (netmap -
if_vtnet_netmap.h)
host (hypervisor):
- ptnetmap support on virtio-net fronted (bhyve -
pci_virtio_ptnetmap.h)
- ptnetmap backend [name: ptnetmap, ptvale] (bhyve -
net_backends.c)
- map netmap host memory into the guest
I added a new IOCTL to vmm.ko to map an userspace guest buffer in
the
guest VM. Then I implemented a new PCI device (ptnetmap-memdev) to
map
the netmap host memory in the guest through PCI MEMORY-BAR.
kernel host (vmm.ko):
- new VM_MAP_USER_BUF ioctl to map buffer in the guest
userspace host (bhyve):
- vm_map_user_buf() in libvmmapi
- ptnetmap=E2=80=90memdev device emulation (bhyve -
pci_ptnetmap_memdev.c)
kernel guest (netmap):
- device driver for ptnetmap-memdev inluded in netmap
module
- ptnetmap support for FreeBSD host:
I implemented kernel thread in netmap module to support ptnetmap o=
n
FreeBSD host.
kernel host (netmap):
- nm_os_kthread_*() functions to handle netmap kthreads.
(netmap - netmap_freebsd.c ptnetmap.c)
- netmap guest/host notification mechanisms.
I needed two mechanisms:
1) notification from ptnetmap kthread to guest VM (interrupt/irq)
vmm.ko already has IOCTL to send interrupt to the guest and I
used it
in the ptnetmap kernel threads.
2) notification from guest VM to ptnetmap kthread (write into the
specific register)
I added new IOCTL on vmm.ko (VM_IO_REG_HANDLER) to catch
write/read
on specific I/O address and send notification.
For now I've implemented only one type of handler
(VM_IO_REGH_KWEVENTS)
to use these events in the kernel through wakeup() and
tsleep()/msleep(),
but I wrote the code to be easily extended to support other
type of
handler (cond_signal, write/ioctl on fd, etc).
kernel host (vmm.ko):
- new VM_IO_REG_HANDLER ioctl to catch write/read on
specific
I/O address and to choose an handler. (eg.
wakeup(event_id)
- vm_io_reg_handler() in libvmmapi
kernel host (netmap)
- msleep() on event_id.
userspace host (bhyve)
- vm_io_reg_handler(VTCFG_R_QNOTIFY, event_id)
send guest notifications to ptnetmap kthreads when the
guest
writes on this virtio register (bhyve -
pci_virtio_ptnetmap.h)
- tell to netmap kthreads the event_id where they can wait
for
events
Example:
Run a 2GB single=E2=80=90CPU virtual machine with three network ports whic=
h use
netmap and ptnetmap backends:
bhyve =E2=80=90s 0,hostbridge =E2=80=90s 1,lpc \
=E2=80=90s 2:1,virtio=E2=80=90net,vale0:1 \ /* normal v=
ale backend */
=E2=80=90s 2:2,ptnetmap=E2=80=90memdev \ /* ptnetmap-memdev=
is needed for
each
* ptnetmap port
* (If two or
more ptnetmap ports
* share the
same netmap memory allocator,
* only one
ptnetmap=E2=80=90memdev is required)
*/
=E2=80=90s 2:3,virtio=E2=80=90net,ptvale1:1 \ /* vale port=
in ptnetmap mode
* If "pt"
prefix is used, the port
* is opened in
passthrough mode (ptnetmap)
*/
=E2=80=90s 2:2,ptnetmap=E2=80=90memdev \
=E2=80=90s 2:3,virtio=E2=80=90net,ptvale2{1 \ /* netmap-pi=
pe in ptnetmap
mode */
=E2=80=90s 3,ptnetmap=E2=80=90memdev \
=E2=80=90s 4,virtio=E2=80=90net,ptnetmap:ix0 \ /* NIC in ptne=
tmap mode */
=E2=80=90l com1,stdio =E2=80=90A =E2=80=90H =E2=80=90P =E2=80=90m =
2G netmapvm
I have results (obtained with pkt-gen) very close to linux/KVM and netmap
host
(both linux and FreeBSD) experiments (physical devices [14.88 Mpps],
software switches
[25 Mpps], shared memory channels [50 Mpps]).
I used one instance of pkt-gen in the guest and the other one in the host.
Thanks for your help!
It was a pleasure working with you.
Cheers,
Stefano
--=20
*Stefano Garzarella*
Software Engineer
e-mail: stefano.garzarella@gmail.com
github: http://github.com/stefano-garzarella
linkedin: http://it.linkedin.com/pub/stefano-garzarella
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAO0mX5ZmRp7HoAJVmHC39_QYSuBHe_zpZ8V0YOasBeS=X3PnpQ>