Date: Tue, 6 Jan 2015 17:17:53 +0100 From: Vincenzo Maffione <v.maffione@gmail.com> To: Luigi Rizzo <rizzo@iet.unipi.it> Cc: Avinash Sridharan <avinash.sridharan@gmail.com>, "freebsd-net@freebsd.org" <net@freebsd.org> Subject: Re: netmap over virtio giving packets with extra 12 bytes Message-ID: <CA%2B_eA9jsAYPm%2B7f6Lt6oXuOuDRbQOMnE=n2arQiUAGLeXg7UvQ@mail.gmail.com> In-Reply-To: <CA%2BhQ2%2BgRz0Q-f5N-C_CrC27qE1i-zSvb3rjWDH0JPCq4Q1%2BA8A@mail.gmail.com> References: <CADJyuDg-PujV%2BtknoSBi3fDd3%2BK%2BOwvjgwh1%2B=Z-eoBUkP2gPg@mail.gmail.com> <CA%2BhQ2%2BgRz0Q-f5N-C_CrC27qE1i-zSvb3rjWDH0JPCq4Q1%2BA8A@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Hello,
From what I can guess you are dealing with a QEMU-KVM guest that
uses virtio-net device(s) and runs netmap over that device(s).
Then, you connect the guest to the host (gentoo) network stack using a
standard linux bridge: a TAP device is used by QEMU to forward guest
traffic from/to the host network stack.
Is that correct?
Following Luigi's explanations, the virtio-net header is part of the
virtio standard, and its purpose is to carry offloading info
(checksum, TSO) across the guests and host kernels. For instance, your
guest kernel can offload the TCP checksum to the virtio-net device,
which in turn uses the virtio-net header (that requires TAP driver
support) to postpone the checksum to the host kernel. If packets
arrive to a physical NIC that supports checksum offloading (e.g. a
r8169 NIC attached to the same bridge to which the TAP is attached),
you have effectively offloaded the checksum computation from the guest
kernel straight to the physical NIC in the physical host.
If you see the virtio-net header with "pkt-gen -f rx", it means that
you are using netmap in "native" mode, that is you use the specific
virtio netmap adapter to send/receive packets from the (virtual) NIC.
If you used netmap over virtio-net in "emulated" mode you wouldn't see
the virtio-net header, because netmap would be using the standard
driver (slow) datapath under the hood: In the rx datapath, the driver
converts the virtio-net header into skbuffs/mbufs metadata, so you
don't see it.
I don't remember having tried to make QEMU use a TAP with no
virtio-net-header extension, but I see that it is possible to disable
it invoking qemu from command line
$ x86_64-softmmu/qemu-system-x86_64 --help | grep tap
-net tap[,vlan=n][,name=str][,fd=h][,fds=x:y:...:z][,ifname=name][,script=file][,downscript=dfile][,helper=helper][,sndbuf=nbytes][,vnet_hdr=on|off][,vhost=on|off][,vhostfd=h][,vhostfds=x:y:...:z][,vhostforce=on|off][,queues=n]
use vnet_hdr=off to avoid enabling the IFF_VNET_HDR tap flag
-netdev [user|tap|bridge|vde|netmap|vhost-user|socket|hubport],id=str[,option][,option][,...]
where you see that you can specify "vnet_hdr=off" when declaring the
qemu "backend" associated to the virtio-net guest device.
Never tried, but it should work. In the worst case you can recompile
the tap driver without IFF_VNET_HDR extension, so that QEMU does not
find it.
Cheers,
Vincenzo
2015-01-05 13:19 GMT+01:00 Luigi Rizzo <rizzo@iet.unipi.it>:
> What you see is a virtio issue.
>
> virtio prepends a 10 or 12-byte "virtio header"
> to all packets, which is used to define what sort
> of NIC accelerations (checksum, tso etc.) are
> expected on the link.
>
> I do not remember if there is a way in qemu-kvm to
> remove the header. Maybe Vincenzo (in Cc) remembers.
>
> cheers
> luigi
>
> On Mon, Jan 5, 2015 at 6:54 AM, Avinash Sridharan
> <avinash.sridharan@gmail.com> wrote:
>>
>> I am using netmap with the click modular router, running the click-modular
>> router in user space. A while back I was using this combination with the
>> e1000 device driver, with a slightly older netmap code-base.
>>
>> Recently I updated my netmap code base and am trying to use the
>> click-modular router with netmap over a virtio-net device driver (over KVM).
>> With this combination, though I was able to receive packets I was unable to
>> interpret any packets coming from the FromDevice element.
>>
>> To debug this issue (and to negate any changes I made to the click-modular
>> router), I ran the pkt-gen application with the "dump payload" option:
>>
>> sudo ~/pkt-gen -i eth1 -f rx -X
>>
>> This showed that packets are being received correctly from the
>> netmap-enabled interface, but there are an extra "12" bytes appended to the
>> packet.
>>
>> 381.088570 main_thread [1446] 1 pps (1 pkts in 1001088 usec)
>>
>> ring 0x7f133bca6000 cur 1 [buf 516 flags 0x0000 len 72]
>>
>> 0: 00 00 00 00 00 00 00 00 00 00 01 00 01 80 c2 00 ................ <<
>> extra 12 bytes
>>
>> 16: 00 00 40 16 7e 5b 50 f0 00 26 42 42 03 00 00 00 ..@.~[P..&BB....
>>
>> 32: 00 00 80 00 40 16 7e 5b 50 f0 00 00 00 00 80 00 ....@.~[P.......
>>
>> 48: 40 16 7e 5b 50 f0 80 01 00 00 14 00 02 00 00 00 @.~[P...........
>>
>> 64: 00 00 00 00 bc 9b f6 74
>>
>>
>> As we can see, the above is an STP BPDU, and there are 12 leading bytes in
>> the payload.
>>
>>
>> The extra leading bytes screw up the packet interpretation.
>>
>>
>> So is this is an artifact of the virtio-net driver or has something
>> changed in the netmap device driver?
>>
>>
>> Thanks,
>>
>> Avinash
>
>
>
>
> --
> -----------------------------------------+-------------------------------
> Prof. Luigi RIZZO, rizzo@iet.unipi.it . Dip. di Ing. dell'Informazione
> http://www.iet.unipi.it/~luigi/ . Universita` di Pisa
> TEL +39-050-2211611 . via Diotisalvi 2
> Mobile +39-338-6809875 . 56122 PISA (Italy)
> -----------------------------------------+-------------------------------
--
Vincenzo Maffione
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CA%2B_eA9jsAYPm%2B7f6Lt6oXuOuDRbQOMnE=n2arQiUAGLeXg7UvQ>