Date: Tue, 4 Aug 2015 15:22:53 GMT From: stefano@FreeBSD.org To: svn-soc-all@FreeBSD.org Subject: socsvn commit: r289212 - soc2015/stefano/ptnetmap/stable/10/usr.sbin/bhyve Message-ID: <201508041522.t74FMrVY062555@socsvn.freebsd.org>
next in thread | raw e-mail | index | archive | help
Author: stefano Date: Tue Aug 4 15:22:52 2015 New Revision: 289212 URL: http://svnweb.FreeBSD.org/socsvn/?view=rev&rev=289212 Log: [bhyve/ptnetmap] add guest/host(kthread) notification and cleanup 1) notification from ptnetmap kthread to guest VM (interrupt/irq) The bhyve user-space, during netmap configuration, passes the ioctl parameters (fd, com, data) to ptnetmap kthreads. The ptnetmap kthread (attached to bhyve process) uses kern_ioctl() with these parameters to notify the guest VM. 2) notification from guest VM to ptnetmap kthread (write into the specific register) Use new IOCTL on vmm.ko (VM_IO_REG_HANDLER) to catch write/read on specific I/O address and send notification. With this new IOCTL, when the guest writes on specific register, the vmm.ko invokes wakeup() on the parameter (chan) specified from bhyve user-space application. The same parameter is passed to the ptnetmap kthreads that call tsleep(chan, ...) to wait the event. Added: soc2015/stefano/ptnetmap/stable/10/usr.sbin/bhyve/README.network Modified: soc2015/stefano/ptnetmap/stable/10/usr.sbin/bhyve/net_backends.c soc2015/stefano/ptnetmap/stable/10/usr.sbin/bhyve/pci_ptnetmap_memdev.c soc2015/stefano/ptnetmap/stable/10/usr.sbin/bhyve/pci_virtio_ptnetmap.h soc2015/stefano/ptnetmap/stable/10/usr.sbin/bhyve/ptnetmap.h Added: soc2015/stefano/ptnetmap/stable/10/usr.sbin/bhyve/README.network ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ soc2015/stefano/ptnetmap/stable/10/usr.sbin/bhyve/README.network Tue Aug 4 15:22:52 2015 (r289212) @@ -0,0 +1,122 @@ +Network frontends and backends in bhyve + +20140804 + +The following is a proposal on how to implement network frontends (FE) +and backends (BE) in bhyve, and how to interface them. + +We need to address the following requirements: +1. Depending on the type of FE and BE, one may need zero or more + additional threads (iothreads) in charge of sending or receiving + packets, implementing one side of paravirtualized devices, etc. + + The requirements for an intermediate thread are the following: + + --- transmit --- + Frontend Backend Operations + non-PV non block* (1) all in VCPU + other cases (2) notify_tx() in VCPU, I/O in thread + + case 1: VCPU executes (autonomously or on kick_tx) + fe->fe2be(); // fill the ring + be->transmit(); + fe->fe2be(); // reclaim buffers + + case 2: VCPU executes (autonomously) + be->notify_tx(); + + iothread executes + for (;;) { + <loop or wait notify_tx> + fe->fe2be(); // fill the ring + be->transmit(); // e.g. txsync, write + fe->fe2be(); // reclaim buffers + } + + non blocking means that either the BE tx function is non + blocking, or that we have an independent notification mechanism + (e.g. an event loop, kevent, epoll...) to notify the FE when + transmit will be possible and issue a kick_tx() to the VCPU + + --- receive --- + Backend Operations + non block* (1) all in VCPU + other cases (2) all in thread + + case 1: VCPU executes (autonomously or on kick_rx) + if (needed) fe->be2fe(); // release buffers from FE + be->receive(); // read or rxsync + if (needed) fe->be2fe(); // pass packets to FE + main thread does fe->kick_rx() when data available + + case 2: i/o thread executes (autonomously) + for (;;) { + <loop or wait notify_rx> + fe->be2fe(); // reclaim buffers from FE + be->receive(); // read or rxsync + fe->be2fe(); // pass packets to FE and kick_rx + } + /* note, in netmap be->receive() may be empty because poll() + * will already fill the ring with packets. + */ + + same as above, non blocking means that the BE will use an + existing mechanism to notify the FE when receive will be possible + and issue a kick_rx() to the VCPU + +2. for performance reasons, it is important that the interface between + FE and BE support batched transfers and asynchronous completion + notifications. + + Given that the netmap API has the features required by #2, and that + netmap backends (and possibly even frontends) are going to be used + with bhyve, we suggest to standardize netmap as the FE-BE API + +In practice, the FE will need to call some BE-supplied function +during its operation, and the same goes for the BE. +To implement this, both entities export some information +through descriptors visible to the other entity. + +Frontend descriptor + + struct fe_desc { + ... + int (*be2fe)(struct fe_desc *, struct netmap_ring *); + int (*fe2be)(struct fe_desc *, struct netmap_ring *); + ... + } + + fe->be2fe() + is invoked by the BE (when receiving from the network) + to move packets from the netmap ring into the FE and to release + completed buffers from the FE back into the netmap ring. + The amount of data moved can be determined by the BE by comparing + the ring pointers before and after the call. + + Before returning, be2fe() will likely issue a kick to the VCPU + (e.g. an interrupt) to notify the availability of new data. + + The return value may be used to indicate events that cannot + be represented by the ring pointers. + + The function runs in the context of a thread "owned" by the BE, + and should implement protection against concurrent + activities on the FE's data structures, whereas the netmap_ring + is only used by this function during its execution. + + fe->fe2be() + is it needed ? + +Backend descriptors + + struct be_desc { + ... + int (*notify_tx)(struct re_desc *, struct netmap_ring *); + int (*notify_rx)(struct re_desc *, struct netmap_ring *); + ... + } + + be->notify_tx() + is invoked by the ... + + Modified: soc2015/stefano/ptnetmap/stable/10/usr.sbin/bhyve/net_backends.c ============================================================================== --- soc2015/stefano/ptnetmap/stable/10/usr.sbin/bhyve/net_backends.c Tue Aug 4 15:11:36 2015 (r289211) +++ soc2015/stefano/ptnetmap/stable/10/usr.sbin/bhyve/net_backends.c Tue Aug 4 15:22:52 2015 (r289212) @@ -46,7 +46,7 @@ #include <assert.h> #include "mevent.h" -#include "dev/virtio/network/virtio_net.h" +#include <dev/virtio/network/virtio_net.h> #include "net_backends.h" #include <sys/linker_set.h> @@ -443,30 +443,38 @@ return 0; } +/* used by netmap and ptnetmap */ static int -netmap_init(struct net_backend *be, const char *devname, - net_backend_cb_t cb, void *param) +netmap_commom_init(struct net_backend *be, struct netmap_priv *priv, uint32_t nr_flags, + const char *devname, net_backend_cb_t cb, void *param) { const char *ndname = "/dev/netmap"; - struct netmap_priv *priv = NULL; + struct nmreq req; char tname[40]; - priv = calloc(1, sizeof(struct netmap_priv)); - if (priv == NULL) { - WPRINTF(("Unable alloc netmap private data\n")); - return -1; - } - strncpy(priv->ifname, devname, sizeof(priv->ifname)); priv->ifname[sizeof(priv->ifname) - 1] = '\0'; - priv->nmd = nm_open(priv->ifname, NULL, NETMAP_NO_TX_POLL, NULL); + memset(&req, 0, sizeof(req)); + req.nr_flags |= nr_flags; + + priv->nmd = nm_open(priv->ifname, &req, NETMAP_NO_TX_POLL, NULL); if (priv->nmd == NULL) { WPRINTF(("Unable to nm_open(): device '%s', " "interface '%s', errno (%s)\n", ndname, devname, strerror(errno))); goto err_open; } +#if 0 + /* check parent (nm_desc with the same allocator already mapped) */ + parent_nmd = netmap_find_parent(nmd); + /* mmap or inherit from parent */ + if (nm_mmap(nmd, parent_nmd)) { + error_report("failed to mmap %s: %s", netmap_opts->ifname, strerror(errno)); + nm_close(nmd); + return -1; + } +#endif priv->tx = NETMAP_TXRING(priv->nmd->nifp, 0); priv->rx = NETMAP_RXRING(priv->nmd->nifp, 0); @@ -476,7 +484,6 @@ priv->rx_continue = 0; be->fd = priv->nmd->fd; - be->priv = priv; /* Create a thread for netmap poll. */ pthread_create(&priv->evloop_tid, NULL, netmap_evloop_thread, (void *)be); @@ -486,6 +493,32 @@ return 0; err_open: + return -1; +} + +static int +netmap_init(struct net_backend *be, const char *devname, + net_backend_cb_t cb, void *param) +{ + const char *ndname = "/dev/netmap"; + struct netmap_priv *priv = NULL; + char tname[40]; + + priv = calloc(1, sizeof(struct netmap_priv)); + if (priv == NULL) { + WPRINTF(("Unable alloc netmap private data\n")); + return -1; + } + + if (netmap_commom_init(be, priv, 0, devname, cb, param)) { + goto err; + } + + be->priv = priv; + + return 0; + +err: free(priv); return -1; @@ -708,9 +741,9 @@ /* * The ptnetmap backend */ -#include <stddef.h> -#include "net/netmap.h" -#include "dev/netmap/netmap_virt.h" +#include <stddef.h> /* IFNAMSIZ */ +#include <net/netmap.h> +#include <dev/netmap/netmap_virt.h> #include "ptnetmap.h" struct ptnbe_priv { @@ -721,10 +754,8 @@ unsigned long acked_features; /* ptnetmap acked features */ }; - - /* The virtio-net features supported by ptnetmap. */ -#define VIRTIO_NET_F_PTNETMAP 0x2000000 /* ptnetmap available */ +#define VIRTIO_NET_F_PTNETMAP 0x2000000 /* (25) ptnetmap available */ #define PTNETMAP_FEATURES VIRTIO_NET_F_PTNETMAP @@ -737,10 +768,6 @@ static uint64_t ptnbe_set_features(struct net_backend *be, uint64_t features) { - if (features & PTNETMAP_FEATURES) { - WPRINTF(("ptnbe_set_features\n")); - } - return 0; } @@ -763,48 +790,20 @@ WPRINTF(("Unable alloc netmap private data\n")); return -1; } - WPRINTF(("ptnbe_init(): devname '%s'\n", devname)); npriv = &priv->up; - strncpy(npriv->ifname, devname + PTNETMAP_NAME_HDR, sizeof(npriv->ifname)); - npriv->ifname[sizeof(npriv->ifname) - 1] = '\0'; - memset(&req, 0, sizeof(req)); - req.nr_flags |= NR_PASSTHROUGH_HOST; - - npriv->nmd = nm_open(npriv->ifname, &req, NETMAP_NO_TX_POLL, NULL); - if (npriv->nmd == NULL) { - WPRINTF(("Unable to nm_open(): device '%s', " - "interface '%s', errno (%s)\n", - ndname, devname, strerror(errno))); - goto err_open; - } -#if 0 - /* check parent (nm_desc with the same allocator already mapped) */ - parent_nmd = netmap_find_parent(nmd); - /* mmap or inherit from parent */ - if (nm_mmap(nmd, parent_nmd)) { - error_report("failed to mmap %s: %s", netmap_opts->ifname, strerror(errno)); - nm_close(nmd); - return -1; + if (netmap_commom_init(be, npriv, NR_PTNETMAP_HOST, devname + PTNETMAP_NAME_HDR, cb, param)) { + goto err; } -#endif - - - npriv->tx = NETMAP_TXRING(npriv->nmd->nifp, 0); - npriv->rx = NETMAP_RXRING(npriv->nmd->nifp, 0); - npriv->cb = cb; - npriv->cb_param = param; - - be->fd = npriv->nmd->fd; be->priv = priv; ptnbe_init_ptnetmap(be); return 0; -err_open: +err: free(priv); return -1; @@ -816,6 +815,7 @@ struct ptnbe_priv *priv = be->priv; if (priv) { + ptnetmap_delete(&priv->ptns); nm_close(priv->up.nmd); } be->fd = -1; @@ -903,8 +903,7 @@ if (priv->created) return 0; - /* TODO: ioctl to start kthreads */ -#if 1 + /* ioctl to start ptnetmap kthreads */ memset(&req, 0, sizeof(req)); strncpy(req.nr_name, priv->up.ifname, sizeof(req.nr_name)); req.nr_version = NETMAP_API; @@ -916,8 +915,6 @@ priv->up.ifname, strerror(errno)); } else priv->created = 1; -#endif - return err; } @@ -929,15 +926,15 @@ struct nmreq req; int err; + if (!priv->created) + return 0; + if (!(priv->acked_features & NET_PTN_FEATURES_BASE)) { printf("ptnetmap features not acked\n"); return EFAULT; } - if (!priv->created) - return 0; - - /* TODO: ioctl to stop kthreads */ + /* ioctl to stop ptnetmap kthreads */ memset(&req, 0, sizeof(req)); strncpy(req.nr_name, priv->up.ifname, sizeof(req.nr_name)); req.nr_version = NETMAP_API; @@ -956,6 +953,8 @@ .name = "ptnetmap|ptvale", .init = ptnbe_init, .cleanup = ptnbe_cleanup, + .send = netmap_send, + .recv = netmap_receive, .get_features = ptnbe_get_features, .set_features = ptnbe_set_features, .get_ptnetmap = ptnbe_get_ptnetmap, Modified: soc2015/stefano/ptnetmap/stable/10/usr.sbin/bhyve/pci_ptnetmap_memdev.c ============================================================================== --- soc2015/stefano/ptnetmap/stable/10/usr.sbin/bhyve/pci_ptnetmap_memdev.c Tue Aug 4 15:11:36 2015 (r289211) +++ soc2015/stefano/ptnetmap/stable/10/usr.sbin/bhyve/pci_ptnetmap_memdev.c Tue Aug 4 15:22:52 2015 (r289212) @@ -1,3 +1,29 @@ +/* + * Copyright (C) 2015 Stefano Garzarella (stefano.garzarella@gmail.com) + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + */ + #include <sys/cdefs.h> __FBSDID("$FreeBSD$"); @@ -17,22 +43,9 @@ #include "pci_emul.h" #include "ptnetmap.h" - -/* ptnetmap memdev PCI-ID and PCI-BARS XXX-ste: remove*/ -#define PTN_MEMDEV_NAME "ptnetmap-memdev" -#define PTNETMAP_PCI_VENDOR_ID 0x3333 /* XXX-ste: set vendor_id */ -#define PTNETMAP_PCI_DEVICE_ID 0x0001 -#define PTNETMAP_IO_PCI_BAR 0 -#define PTNETMAP_MEM_PCI_BAR 1 - -/* ptnetmap memdev register */ -/* 32 bit r/o */ -#define PTNETMAP_IO_PCI_FEATURES 0 -/* 32 bit r/o */ -#define PTNETMAP_IO_PCI_MEMSIZE 4 -/* 16 bit r/o */ -#define PTNETMAP_IO_PCI_HOSTID 8 -#define PTNEMTAP_IO_SIZE 10 +#include <net/if.h> /* IFNAMSIZ */ +#include <net/netmap.h> +#include <dev/netmap/netmap_virt.h> struct ptn_memdev_softc { struct pci_devinst *pi; /* PCI device instance */ @@ -129,8 +142,8 @@ return 0; if (baridx == PTNETMAP_MEM_PCI_BAR) { - printf("ptnetmap_memdev: MEM read\n"); - printf("ptnentmap_memdev: mem_read - offset: %lx size: %d ret: %lx\n", offset, size, ret); + printf("ptnetmap_memdev: unexpected MEM read - offset: %lx size: %d ret: %lx\n", + offset, size, ret); return 0; /* XXX */ } @@ -149,7 +162,6 @@ break; } - printf("ptnentmap_memdev: io_read - offset: %lx size: %d ret: %llu\n", offset, size,(unsigned long long)ret); return ret; } @@ -164,22 +176,16 @@ return; if (baridx == PTNETMAP_MEM_PCI_BAR) { - printf("ptnetmap_memdev: MEM write\n"); - return; /* XXX */ + printf("ptnetmap_memdev: unexpected MEM write - offset: %lx size: %d value: %lx\n", + offset, size, value); + return; } - /* XXX probably should do something better than just assert() */ - assert(baridx == PTNETMAP_IO_PCI_BAR); - switch (offset) { - default: printf("ptnentmap_memdev: write io reg unexpected\n"); break; } - - - printf("ptnentmap_memdev: io_write - offset: %lx size: %d val: %lx\n", offset, size, value); } static int @@ -187,12 +193,9 @@ { int ret; - printf("ptnetmap_memdev: configuring\n"); - if (sc->pi == NULL || sc->mem_ptr == NULL) return 0; - /* init iobar */ ret = pci_emul_alloc_bar(sc->pi, PTNETMAP_IO_PCI_BAR, PCIBAR_IO, PTNEMTAP_IO_SIZE); if (ret) { @@ -209,10 +212,6 @@ return ret; } - printf("ptnetmap_memdev: pci_addr: %llx, mem_size: %llu, mem_ptr: %p\n", - (unsigned long long) sc->pi->pi_bar[PTNETMAP_MEM_PCI_BAR].addr, - (unsigned long long) sc->mem_size, sc->mem_ptr); - ret = vm_map_user_buf(sc->pi->pi_vmctx, sc->pi->pi_bar[PTNETMAP_MEM_PCI_BAR].addr, sc->mem_size, sc->mem_ptr); if (ret) { @@ -220,8 +219,6 @@ return ret; } - printf("ptnetmap_memdev: configured\n"); - return 0; } @@ -232,8 +229,6 @@ uint64_t size; int ret; - printf("ptnetmap_memdev: loading\n"); - sc = ptn_memdev_find_empty_pi(); if (sc == NULL) { sc = ptn_memdev_create(); @@ -260,8 +255,6 @@ goto err; } - printf("ptnetmap_memdev: loaded\n"); - return (0); err: ptn_memdev_delete(sc); @@ -274,7 +267,6 @@ { struct ptn_memdev_softc *sc; int ret; - printf("ptnetmap_memdev: attaching\n"); /* if a device with the same mem_id is already attached, we are done */ if (ptn_memdev_find_memid(mem_id)) { @@ -295,17 +287,13 @@ sc->mem_size = mem_size; sc->mem_id = mem_id; - printf("ptnetmap_memdev_attach: mem_id: %u, mem_size: %lu, mem_ptr: %p\n", mem_id, - (unsigned long) mem_size, mem_ptr); - - /* TODO: configure device BARs */ + /* configure device BARs */ ret = ptn_memdev_configure(sc); if (ret) { printf("ptnetmap_memdev: configure error\n"); goto err; } - printf("ptnetmap_memdev: attached\n"); return 0; err: Modified: soc2015/stefano/ptnetmap/stable/10/usr.sbin/bhyve/pci_virtio_ptnetmap.h ============================================================================== --- soc2015/stefano/ptnetmap/stable/10/usr.sbin/bhyve/pci_virtio_ptnetmap.h Tue Aug 4 15:11:36 2015 (r289211) +++ soc2015/stefano/ptnetmap/stable/10/usr.sbin/bhyve/pci_virtio_ptnetmap.h Tue Aug 4 15:22:52 2015 (r289212) @@ -1,12 +1,41 @@ +/* + * Copyright (C) 2015 Stefano Garzarella (stefano.garzarella@gmail.com) + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + */ + #ifndef __PCI_VIRTIO_PTNETMAP_H__ #define __PCI_VIRTIO_PTNETMAP_H__ -/* XXX-ste: move in other file and split in .c .h? */ #ifdef BHYVE_VIRTIO_PTNETMAP +#include <machine/vmm.h> +#include <machine/vmm_dev.h> /* VM_LAPIC_MSI */ +#include <vmmapi.h> + +#include "ptnetmap.h" /* ptnetmap virtio register BASE */ #define PTNETMAP_VIRTIO_IO_BASE sizeof(struct virtio_net_config) -#include "ptnetmap.h" static void ptnetmap_configure_csb(struct vmctx *ctx, struct paravirt_csb** csb, uint32_t csbbal, @@ -23,7 +52,6 @@ * The CSB is then remapped if the new pointer is != 0 */ if (*csb) { - /* TODO: unmap */ *csb = NULL; } if (base) { @@ -56,8 +84,6 @@ /* extend cfgsize. virtio creates PCIBAR for us */ vc->vc_cfgsize += PTNEMTAP_VIRTIO_IO_SIZE; - - printf("ptnetmap-virtio init END\n"); } static int @@ -75,17 +101,12 @@ printf("ERROR ptnetmap: csb not initialized\n"); return ret; } + /* share netmap_if info to the guest through CSB */ csb->nifp_offset = ptns->offset; csb->num_tx_rings = ptns->num_tx_rings; csb->num_rx_rings = ptns->num_rx_rings; csb->num_tx_slots = ptns->num_tx_slots; csb->num_rx_slots = ptns->num_rx_slots; - printf("txr %u rxr %u txd %u rxd %u nifp_offset %u\n", - csb->num_tx_rings, - csb->num_rx_rings, - csb->num_tx_slots, - csb->num_rx_slots, - csb->nifp_offset); return ret; } @@ -95,6 +116,12 @@ { struct ptnetmap_state *ptns = sc->ptn.state; struct paravirt_csb *csb = sc->ptn.csb; + struct pci_devinst *pi; + struct vmctx *vmctx; + struct vqueue_info *vq; + struct msix_table_entry *mte; + struct iovec iov[1]; + uint16_t idx; int ret; if (sc->ptn.up) { @@ -107,22 +134,59 @@ return -1; } - /* TODO-ste: add support for multiqueue */ + /* TODO: add support for multiqueue */ + pi = sc->vsc_vs.vs_pi; + vmctx = pi->pi_vmctx; - /* TODO: Stop processing guest/host IO notifications in qemu. + /* Configure the RX ring */ + sc->ptn.cfg.rx_ring.irqfd = vm_get_fd(vmctx); + sc->ptn.cfg.rx_ioctl.com = VM_LAPIC_MSI; + vq = &sc->vsc_queues[VTNET_RXQ]; + mte = &pi->pi_msix.table[vq->vq_msix_idx]; + sc->ptn.cfg.rx_ioctl.data.msg = mte->msg_data; + sc->ptn.cfg.rx_ioctl.data.addr = mte->addr; + /* push fake-elem in the rx queue to enable interrupts */ + if (vq_getchain(vq, &idx, iov, 1, NULL) > 0) { + vq_relchain(vq, idx, 0); + } + /* enable rx notification from guest */ + vq->vq_used->vu_flags &= ~VRING_USED_F_NO_NOTIFY; + /* + * Stop processing guest/host IO notifications in bhyve. * Start processing them in ptnetmap. */ - - - /* Configure the RX ring */ - sc->ptn.cfg.rx_ring.ioeventfd = -1; - sc->ptn.cfg.rx_ring.irqfd = -1; + ret = vm_io_reg_handler(vmctx, pi->pi_bar[0].addr + VTCFG_R_QNOTIFY, 0, + 0xFFFFFFFF, VTNET_RXQ, VM_IO_REGH_KWEVENTS, (void *) vq); + if (ret != 0) { + printf("ERROR ptnetmap: vm_io_reg_handler %d\n", ret); + goto err_reg_rx; + } + sc->ptn.cfg.rx_ring.ioeventfd = (uint64_t) vq; /* Configure the TX ring */ - sc->ptn.cfg.tx_ring.ioeventfd = -1; - sc->ptn.cfg.tx_ring.irqfd = -1; - - /* TODO: push fake-elem in the tx/rx queue to enable interrupts */ + sc->ptn.cfg.tx_ring.irqfd = vm_get_fd(vmctx); + sc->ptn.cfg.tx_ioctl.com = VM_LAPIC_MSI; + vq = &sc->vsc_queues[VTNET_TXQ]; + mte = &pi->pi_msix.table[vq->vq_msix_idx]; + sc->ptn.cfg.tx_ioctl.data.msg = mte->msg_data; + sc->ptn.cfg.tx_ioctl.data.addr = mte->addr; + /* push fake-elem in the tx queue to enable interrupts */ + if (vq_getchain(vq, &idx, iov, 1, NULL) > 0) { + vq_relchain(vq, idx, 0); + } + /* enable tx notification from guest */ + vq->vq_used->vu_flags &= ~VRING_USED_F_NO_NOTIFY; + /* + * Stop processing guest/host IO notifications in bhyve. + * Start processing them in ptnetmap. + */ + ret = vm_io_reg_handler(vmctx, pi->pi_bar[0].addr + VTCFG_R_QNOTIFY, 0, + 0xFFFFFFFF, VTNET_TXQ, VM_IO_REGH_KWEVENTS, (void *) vq); + if (ret != 0) { + printf("ERROR ptnetmap: vm_io_reg_handler %d\n", ret); + goto err_reg_tx; + } + sc->ptn.cfg.tx_ring.ioeventfd = (uint64_t) vq; /* Initialize CSB */ sc->ptn.cfg.csb = sc->ptn.csb; @@ -131,7 +195,8 @@ sc->ptn.csb->guest_need_rxkick = 1; sc->ptn.csb->host_need_rxkick = 1; - sc->ptn.cfg.features = PTNETMAP_CFG_FEAT_CSB | PTNETMAP_CFG_FEAT_EVENTFD; + sc->ptn.cfg.features = PTNETMAP_CFG_FEAT_CSB | PTNETMAP_CFG_FEAT_EVENTFD | + PTNETMAP_CFG_FEAT_IOCTL; /* Configure the net backend. */ ret = ptnetmap_create(sc->ptn.state, &sc->ptn.cfg); @@ -143,22 +208,38 @@ return (0); err_ptn_create: + vm_io_reg_handler(vmctx, pi->pi_bar[0].addr + VTCFG_R_QNOTIFY, 0, + 0xFFFFFFFF, VTNET_TXQ, VM_IO_REGH_DELETE, 0); +err_reg_tx: + vm_io_reg_handler(vmctx, pi->pi_bar[0].addr + VTCFG_R_QNOTIFY, 0, + 0xFFFFFFFF, VTNET_RXQ, VM_IO_REGH_DELETE, 0); +err_reg_rx: return (ret); } static int pci_vtnet_ptnetmap_down(struct pci_vtnet_softc *sc) { + struct pci_devinst *pi; + struct vmctx *vmctx; int ret; if (!sc->ptn.state || !sc->ptn.up) { return (0); } - sc->ptn.up = 0; + pi = sc->vsc_vs.vs_pi; + vmctx = pi->pi_vmctx; + /* - * TODO: Start processing guest/host IO notifications in qemu. + * Start processing guest/host IO notifications in bhyve. */ + vm_io_reg_handler(vmctx, pi->pi_bar[0].addr + VTCFG_R_QNOTIFY, 0, + 0xFFFFFFFF, VTNET_RXQ, VM_IO_REGH_DELETE, 0); + vm_io_reg_handler(vmctx, pi->pi_bar[0].addr + VTCFG_R_QNOTIFY, 0, + 0xFFFFFFFF, VTNET_TXQ, VM_IO_REGH_DELETE, 0); + + sc->ptn.up = 0; return (ptnetmap_delete(sc->ptn.state)); } @@ -177,53 +258,50 @@ memcpy(&sc->ptn.reg[offset], &value, size); switch (offset) { - case PTNETMAP_VIRTIO_IO_PTFEAT: - val = (uint32_t *)(sc->ptn.reg + offset); - ret = (sc->ptn.features &= *val); - ptnetmap_ack_features(sc->ptn.state, sc->ptn.features); - printf("ptnetmap acked features: %x\n", sc->ptn.features); - - sc->ptn.reg[PTNETMAP_VIRTIO_IO_PTFEAT] = ret; + case PTNETMAP_VIRTIO_IO_PTFEAT: + val = (uint32_t *)(sc->ptn.reg + offset); + ret = (sc->ptn.features &= *val); + ptnetmap_ack_features(sc->ptn.state, sc->ptn.features); + + sc->ptn.reg[PTNETMAP_VIRTIO_IO_PTFEAT] = ret; + break; + case PTNETMAP_VIRTIO_IO_PTCTL: + val = (uint32_t *)(sc->ptn.reg + offset); + + ret = EINVAL; + switch(*val) { + case NET_PARAVIRT_PTCTL_CONFIG: + ret = pci_vtnet_ptnetmap_get_mem(sc); break; - case PTNETMAP_VIRTIO_IO_PTCTL: - val = (uint32_t *)(sc->ptn.reg + offset); - - ret = EINVAL; - switch(*val) { - case NET_PARAVIRT_PTCTL_CONFIG: - ret = pci_vtnet_ptnetmap_get_mem(sc); - break; - case NET_PARAVIRT_PTCTL_REGIF: - ret = pci_vtnet_ptnetmap_up(sc); - break; - case NET_PARAVIRT_PTCTL_UNREGIF: - ret = pci_vtnet_ptnetmap_down(sc); - break; - case NET_PARAVIRT_PTCTL_HOSTMEMID: - ret = ptnetmap_get_hostmemid(sc->ptn.state); - break; - case NET_PARAVIRT_PTCTL_IFNEW: - case NET_PARAVIRT_PTCTL_IFDELETE: - case NET_PARAVIRT_PTCTL_FINALIZE: - case NET_PARAVIRT_PTCTL_DEREF: - ret = 0; - break; - } - printf("PTSTS - ret %d\n", ret); - sc->ptn.reg[PTNETMAP_VIRTIO_IO_PTSTS] = ret; + case NET_PARAVIRT_PTCTL_REGIF: + ret = pci_vtnet_ptnetmap_up(sc); break; - case PTNETMAP_VIRTIO_IO_CSBBAH: + case NET_PARAVIRT_PTCTL_UNREGIF: + ret = pci_vtnet_ptnetmap_down(sc); break; - case PTNETMAP_VIRTIO_IO_CSBBAL: - ptnetmap_configure_csb(sc->vsc_vs.vs_pi->pi_vmctx, &sc->ptn.csb, *((uint32_t *)(sc->ptn.reg + PTNETMAP_VIRTIO_IO_CSBBAL)), - *((uint32_t *)(sc->ptn.reg + PTNETMAP_VIRTIO_IO_CSBBAH))); + case NET_PARAVIRT_PTCTL_HOSTMEMID: + ret = ptnetmap_get_hostmemid(sc->ptn.state); break; - default: + case NET_PARAVIRT_PTCTL_IFNEW: + case NET_PARAVIRT_PTCTL_IFDELETE: + case NET_PARAVIRT_PTCTL_FINALIZE: + case NET_PARAVIRT_PTCTL_DEREF: + ret = 0; break; + } + sc->ptn.reg[PTNETMAP_VIRTIO_IO_PTSTS] = ret; + break; + case PTNETMAP_VIRTIO_IO_CSBBAH: + break; + case PTNETMAP_VIRTIO_IO_CSBBAL: + ptnetmap_configure_csb(sc->vsc_vs.vs_pi->pi_vmctx, &sc->ptn.csb, + *((uint32_t *)(sc->ptn.reg + PTNETMAP_VIRTIO_IO_CSBBAL)), + *((uint32_t *)(sc->ptn.reg + PTNETMAP_VIRTIO_IO_CSBBAH))); + break; + default: + break; } - printf("ptnentmap_vtnet: io_write - offset: %d size: %d val: %u\n", offset, size, value); - return (0); } @@ -240,14 +318,14 @@ memcpy(value, &sc->ptn.reg[offset], size); #if 0 switch (offset) { - case PTNETMAP_VIRTIO_IO_PTFEAT: - case PTNETMAP_VIRTIO_IO_PTSTS: - break; - default: - break; + case PTNETMAP_VIRTIO_IO_PTFEAT: + case PTNETMAP_VIRTIO_IO_PTSTS: + break; + default: + printf("pci_vtnet_ptnentmap: write io reg unexpected\n"); + break; } #endif - printf("ptnentmap_vtnet: io_read - offset: %d size: %d ret: %u\n", offset, size, *value); return (0); } Modified: soc2015/stefano/ptnetmap/stable/10/usr.sbin/bhyve/ptnetmap.h ============================================================================== --- soc2015/stefano/ptnetmap/stable/10/usr.sbin/bhyve/ptnetmap.h Tue Aug 4 15:11:36 2015 (r289211) +++ soc2015/stefano/ptnetmap/stable/10/usr.sbin/bhyve/ptnetmap.h Tue Aug 4 15:22:52 2015 (r289212) @@ -1,3 +1,29 @@ +/* + * Copyright (C) 2015 Stefano Garzarella (stefano.garzarella@gmail.com) + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + */ + #ifndef __PTNETMAP_H__ #define __PTNETMAP_H__
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?201508041522.t74FMrVY062555>