Date: Tue, 6 Jan 2015 08:33:19 -0800 From: Avinash Sridharan <avinash.sridharan@gmail.com> To: Vincenzo Maffione <v.maffione@gmail.com> Cc: Luigi Rizzo <rizzo@iet.unipi.it>, "freebsd-net@freebsd.org" <net@freebsd.org> Subject: Re: netmap over virtio giving packets with extra 12 bytes Message-ID: <CADJyuDhwWhtUFaPXOACZpzMeZFds8E7tFpXtKwguf4mFTZep2w@mail.gmail.com> In-Reply-To: <CADJyuDjo%2BVDvAt6zKD2A0iZbU__HUSr-AfWsMdc5Bu1tARf9vw@mail.gmail.com> References: <CADJyuDg-PujV%2BtknoSBi3fDd3%2BK%2BOwvjgwh1%2B=Z-eoBUkP2gPg@mail.gmail.com> <CA%2BhQ2%2BgRz0Q-f5N-C_CrC27qE1i-zSvb3rjWDH0JPCq4Q1%2BA8A@mail.gmail.com> <CA%2B_eA9jsAYPm%2B7f6Lt6oXuOuDRbQOMnE=n2arQiUAGLeXg7UvQ@mail.gmail.com> <CADJyuDjo%2BVDvAt6zKD2A0iZbU__HUSr-AfWsMdc5Bu1tARf9vw@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
By the way, Vincenzo, your assumptions about my system setup: "From what I can guess you are dealing with a QEMU-KVM guest that uses virtio-net device(s) and runs netmap over that device(s). Then, you connect the guest to the host (gentoo) network stack using a standard linux bridge: a TAP device is used by QEMU to forward guest traffic from/to the host network stack." are correct. On Tue, Jan 6, 2015 at 8:31 AM, Avinash Sridharan < avinash.sridharan@gmail.com> wrote: > Hi Vincenzo, > Thanks for the explanation. From your explanation it seems like the > netmap in "native" mode over virtio-net should be giving some indication of > how many extra bytes have been added by the virtio-net driver (or for that > matter any other driver that provides this type of rx-descriptor). > Otherwise, the application will have to store knowledge about the specifics > of the underlying devices which dosen't seem that clean. (I think Adrian > was referring to the same issue) > > That said, how do we handle TX in this case? Since the underlying driver > (netmap + virtio-net) expects an extra 12 bytes of header that the > application should know when to add. Or is this optional? > > > > > > On Tue, Jan 6, 2015 at 8:17 AM, Vincenzo Maffione <v.maffione@gmail.com> > wrote: > >> Hello, >> >> From what I can guess you are dealing with a QEMU-KVM guest that >> uses virtio-net device(s) and runs netmap over that device(s). >> Then, you connect the guest to the host (gentoo) network stack using a >> standard linux bridge: a TAP device is used by QEMU to forward guest >> traffic from/to the host network stack. >> >> Is that correct? >> >> Following Luigi's explanations, the virtio-net header is part of the >> virtio standard, and its purpose is to carry offloading info >> (checksum, TSO) across the guests and host kernels. For instance, your >> guest kernel can offload the TCP checksum to the virtio-net device, >> which in turn uses the virtio-net header (that requires TAP driver >> support) to postpone the checksum to the host kernel. If packets >> arrive to a physical NIC that supports checksum offloading (e.g. a >> r8169 NIC attached to the same bridge to which the TAP is attached), >> you have effectively offloaded the checksum computation from the guest >> kernel straight to the physical NIC in the physical host. >> >> If you see the virtio-net header with "pkt-gen -f rx", it means that >> you are using netmap in "native" mode, that is you use the specific >> virtio netmap adapter to send/receive packets from the (virtual) NIC. >> If you used netmap over virtio-net in "emulated" mode you wouldn't see >> the virtio-net header, because netmap would be using the standard >> driver (slow) datapath under the hood: In the rx datapath, the driver >> converts the virtio-net header into skbuffs/mbufs metadata, so you >> don't see it. >> >> I don't remember having tried to make QEMU use a TAP with no >> virtio-net-header extension, but I see that it is possible to disable >> it invoking qemu from command line >> >> $ x86_64-softmmu/qemu-system-x86_64 --help | grep tap >> >> -net >> tap[,vlan=n][,name=str][,fd=h][,fds=x:y:...:z][,ifname=name][,script=file][,downscript=dfile][,helper=helper][,sndbuf=nbytes][,vnet_hdr=on|off][,vhost=on|off][,vhostfd=h][,vhostfds=x:y:...:z][,vhostforce=on|off][,queues=n] >> use vnet_hdr=off to avoid enabling the IFF_VNET_HDR tap >> flag >> -netdev >> [user|tap|bridge|vde|netmap|vhost-user|socket|hubport],id=str[,option][,option][,...] >> >> where you see that you can specify "vnet_hdr=off" when declaring the >> qemu "backend" associated to the virtio-net guest device. >> Never tried, but it should work. In the worst case you can recompile >> the tap driver without IFF_VNET_HDR extension, so that QEMU does not >> find it. >> >> >> Cheers, >> Vincenzo >> >> 2015-01-05 13:19 GMT+01:00 Luigi Rizzo <rizzo@iet.unipi.it>: >> > What you see is a virtio issue. >> > >> > virtio prepends a 10 or 12-byte "virtio header" >> > to all packets, which is used to define what sort >> > of NIC accelerations (checksum, tso etc.) are >> > expected on the link. >> > >> > I do not remember if there is a way in qemu-kvm to >> > remove the header. Maybe Vincenzo (in Cc) remembers. >> > >> > cheers >> > luigi >> > >> > On Mon, Jan 5, 2015 at 6:54 AM, Avinash Sridharan >> > <avinash.sridharan@gmail.com> wrote: >> >> >> >> I am using netmap with the click modular router, running the >> click-modular >> >> router in user space. A while back I was using this combination with >> the >> >> e1000 device driver, with a slightly older netmap code-base. >> >> >> >> Recently I updated my netmap code base and am trying to use the >> >> click-modular router with netmap over a virtio-net device driver (over >> KVM). >> >> With this combination, though I was able to receive packets I was >> unable to >> >> interpret any packets coming from the FromDevice element. >> >> >> >> To debug this issue (and to negate any changes I made to the >> click-modular >> >> router), I ran the pkt-gen application with the "dump payload" option: >> >> >> >> sudo ~/pkt-gen -i eth1 -f rx -X >> >> >> >> This showed that packets are being received correctly from the >> >> netmap-enabled interface, but there are an extra "12" bytes appended >> to the >> >> packet. >> >> >> >> 381.088570 main_thread [1446] 1 pps (1 pkts in 1001088 usec) >> >> >> >> ring 0x7f133bca6000 cur 1 [buf 516 flags 0x0000 len 72] >> >> >> >> 0: 00 00 00 00 00 00 00 00 00 00 01 00 01 80 c2 00 >> ................ << >> >> extra 12 bytes >> >> >> >> 16: 00 00 40 16 7e 5b 50 f0 00 26 42 42 03 00 00 00 ..@.~[P..&BB.... >> >> >> >> 32: 00 00 80 00 40 16 7e 5b 50 f0 00 00 00 00 80 00 ....@.~[P....... >> >> >> >> 48: 40 16 7e 5b 50 f0 80 01 00 00 14 00 02 00 00 00 @.~[P........... >> >> >> >> 64: 00 00 00 00 bc 9b f6 74 >> >> >> >> >> >> As we can see, the above is an STP BPDU, and there are 12 leading >> bytes in >> >> the payload. >> >> >> >> >> >> The extra leading bytes screw up the packet interpretation. >> >> >> >> >> >> So is this is an artifact of the virtio-net driver or has something >> >> changed in the netmap device driver? >> >> >> >> >> >> Thanks, >> >> >> >> Avinash >> > >> > >> > >> > >> > -- >> > >> -----------------------------------------+------------------------------- >> > Prof. Luigi RIZZO, rizzo@iet.unipi.it . Dip. di Ing. >> dell'Informazione >> > http://www.iet.unipi.it/~luigi/ . Universita` di Pisa >> > TEL +39-050-2211611 . via Diotisalvi 2 >> > Mobile +39-338-6809875 . 56122 PISA (Italy) >> > >> -----------------------------------------+------------------------------- >> >> >> >> -- >> Vincenzo Maffione >> > >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CADJyuDhwWhtUFaPXOACZpzMeZFds8E7tFpXtKwguf4mFTZep2w>