Date: Tue, 6 Jan 2015 08:31:07 -0800 From: Avinash Sridharan <avinash.sridharan@gmail.com> To: Vincenzo Maffione <v.maffione@gmail.com> Cc: Luigi Rizzo <rizzo@iet.unipi.it>, "freebsd-net@freebsd.org" <net@freebsd.org> Subject: Re: netmap over virtio giving packets with extra 12 bytes Message-ID: <CADJyuDjo%2BVDvAt6zKD2A0iZbU__HUSr-AfWsMdc5Bu1tARf9vw@mail.gmail.com> In-Reply-To: <CA%2B_eA9jsAYPm%2B7f6Lt6oXuOuDRbQOMnE=n2arQiUAGLeXg7UvQ@mail.gmail.com> References: <CADJyuDg-PujV%2BtknoSBi3fDd3%2BK%2BOwvjgwh1%2B=Z-eoBUkP2gPg@mail.gmail.com> <CA%2BhQ2%2BgRz0Q-f5N-C_CrC27qE1i-zSvb3rjWDH0JPCq4Q1%2BA8A@mail.gmail.com> <CA%2B_eA9jsAYPm%2B7f6Lt6oXuOuDRbQOMnE=n2arQiUAGLeXg7UvQ@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Hi Vincenzo, Thanks for the explanation. From your explanation it seems like the netmap in "native" mode over virtio-net should be giving some indication of how many extra bytes have been added by the virtio-net driver (or for that matter any other driver that provides this type of rx-descriptor). Otherwise, the application will have to store knowledge about the specifics of the underlying devices which dosen't seem that clean. (I think Adrian was referring to the same issue) That said, how do we handle TX in this case? Since the underlying driver (netmap + virtio-net) expects an extra 12 bytes of header that the application should know when to add. Or is this optional? On Tue, Jan 6, 2015 at 8:17 AM, Vincenzo Maffione <v.maffione@gmail.com> wrote: > Hello, > > From what I can guess you are dealing with a QEMU-KVM guest that > uses virtio-net device(s) and runs netmap over that device(s). > Then, you connect the guest to the host (gentoo) network stack using a > standard linux bridge: a TAP device is used by QEMU to forward guest > traffic from/to the host network stack. > > Is that correct? > > Following Luigi's explanations, the virtio-net header is part of the > virtio standard, and its purpose is to carry offloading info > (checksum, TSO) across the guests and host kernels. For instance, your > guest kernel can offload the TCP checksum to the virtio-net device, > which in turn uses the virtio-net header (that requires TAP driver > support) to postpone the checksum to the host kernel. If packets > arrive to a physical NIC that supports checksum offloading (e.g. a > r8169 NIC attached to the same bridge to which the TAP is attached), > you have effectively offloaded the checksum computation from the guest > kernel straight to the physical NIC in the physical host. > > If you see the virtio-net header with "pkt-gen -f rx", it means that > you are using netmap in "native" mode, that is you use the specific > virtio netmap adapter to send/receive packets from the (virtual) NIC. > If you used netmap over virtio-net in "emulated" mode you wouldn't see > the virtio-net header, because netmap would be using the standard > driver (slow) datapath under the hood: In the rx datapath, the driver > converts the virtio-net header into skbuffs/mbufs metadata, so you > don't see it. > > I don't remember having tried to make QEMU use a TAP with no > virtio-net-header extension, but I see that it is possible to disable > it invoking qemu from command line > > $ x86_64-softmmu/qemu-system-x86_64 --help | grep tap > > -net > tap[,vlan=n][,name=str][,fd=h][,fds=x:y:...:z][,ifname=name][,script=file][,downscript=dfile][,helper=helper][,sndbuf=nbytes][,vnet_hdr=on|off][,vhost=on|off][,vhostfd=h][,vhostfds=x:y:...:z][,vhostforce=on|off][,queues=n] > use vnet_hdr=off to avoid enabling the IFF_VNET_HDR tap > flag > -netdev > [user|tap|bridge|vde|netmap|vhost-user|socket|hubport],id=str[,option][,option][,...] > > where you see that you can specify "vnet_hdr=off" when declaring the > qemu "backend" associated to the virtio-net guest device. > Never tried, but it should work. In the worst case you can recompile > the tap driver without IFF_VNET_HDR extension, so that QEMU does not > find it. > > > Cheers, > Vincenzo > > 2015-01-05 13:19 GMT+01:00 Luigi Rizzo <rizzo@iet.unipi.it>: > > What you see is a virtio issue. > > > > virtio prepends a 10 or 12-byte "virtio header" > > to all packets, which is used to define what sort > > of NIC accelerations (checksum, tso etc.) are > > expected on the link. > > > > I do not remember if there is a way in qemu-kvm to > > remove the header. Maybe Vincenzo (in Cc) remembers. > > > > cheers > > luigi > > > > On Mon, Jan 5, 2015 at 6:54 AM, Avinash Sridharan > > <avinash.sridharan@gmail.com> wrote: > >> > >> I am using netmap with the click modular router, running the > click-modular > >> router in user space. A while back I was using this combination with the > >> e1000 device driver, with a slightly older netmap code-base. > >> > >> Recently I updated my netmap code base and am trying to use the > >> click-modular router with netmap over a virtio-net device driver (over > KVM). > >> With this combination, though I was able to receive packets I was > unable to > >> interpret any packets coming from the FromDevice element. > >> > >> To debug this issue (and to negate any changes I made to the > click-modular > >> router), I ran the pkt-gen application with the "dump payload" option: > >> > >> sudo ~/pkt-gen -i eth1 -f rx -X > >> > >> This showed that packets are being received correctly from the > >> netmap-enabled interface, but there are an extra "12" bytes appended to > the > >> packet. > >> > >> 381.088570 main_thread [1446] 1 pps (1 pkts in 1001088 usec) > >> > >> ring 0x7f133bca6000 cur 1 [buf 516 flags 0x0000 len 72] > >> > >> 0: 00 00 00 00 00 00 00 00 00 00 01 00 01 80 c2 00 ................ > << > >> extra 12 bytes > >> > >> 16: 00 00 40 16 7e 5b 50 f0 00 26 42 42 03 00 00 00 ..@.~[P..&BB.... > >> > >> 32: 00 00 80 00 40 16 7e 5b 50 f0 00 00 00 00 80 00 ....@.~[P....... > >> > >> 48: 40 16 7e 5b 50 f0 80 01 00 00 14 00 02 00 00 00 @.~[P........... > >> > >> 64: 00 00 00 00 bc 9b f6 74 > >> > >> > >> As we can see, the above is an STP BPDU, and there are 12 leading bytes > in > >> the payload. > >> > >> > >> The extra leading bytes screw up the packet interpretation. > >> > >> > >> So is this is an artifact of the virtio-net driver or has something > >> changed in the netmap device driver? > >> > >> > >> Thanks, > >> > >> Avinash > > > > > > > > > > -- > > -----------------------------------------+------------------------------- > > Prof. Luigi RIZZO, rizzo@iet.unipi.it . Dip. di Ing. dell'Informazione > > http://www.iet.unipi.it/~luigi/ . Universita` di Pisa > > TEL +39-050-2211611 . via Diotisalvi 2 > > Mobile +39-338-6809875 . 56122 PISA (Italy) > > -----------------------------------------+------------------------------- > > > > -- > Vincenzo Maffione >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CADJyuDjo%2BVDvAt6zKD2A0iZbU__HUSr-AfWsMdc5Bu1tARf9vw>