Date: Fri, 27 Oct 2017 08:58:57 +0200 From: Vincenzo Maffione <v.maffione@gmail.com> To: Joe Buehler <aspam@cox.net> Cc: FreeBSD Net <freebsd-net@freebsd.org> Subject: Re: virtio_net / netmap RX dropping frames Message-ID: <CA%2B_eA9iRkyNFBwvSok7Jt6xbsafP75cBGigkr_TbwVsq1mm0QQ@mail.gmail.com> In-Reply-To: <59F223C3.5040907@cox.net> References: <59F0FBEE.6030008@cox.net> <59F1F1BA.3020608@cox.net> <59F214F6.9060905@cox.net> <59F223C3.5040907@cox.net>
next in thread | previous in thread | raw e-mail | index | archive | help
2017-10-26 20:04 GMT+02:00 Joe Buehler <aspam@cox.net>: > Vincenzo Maffione wrote: > > > You can have how many threads and processes you want. The constraint is > > that there must not be two threads accessing the same ring at the same > > time. In this case each pktgen is accessing different rings. > > Thanks that was very useful info. I had run this before and got large > frame drop so assumed it was a violation of threading constraints. So > now I can remove a mutex from my app, which has an RX and TX thread. > > Running the two pkt-gen instances, I am getting a lot of RX frame drop. > Based on counters, the TX frames are making it to the external loopback > device, about 1 million frames/sec, which is looping them all back, but > the macvtap interface on the host that feeds into the > ixgbe/vhost/virtio_net/netmap interface in the VM shows about 80-90% of > them as dropped. CPU usage is low in the VM, very roughly 25% for the > TX thread and 5% for the RX thread. The frame rate displayed by pkt-gen > and the CPU displayed by top is bouncing around. > > Yes, mutexes are only needed for concurrent access to the same ring, which is not your case. >From this description it seems that your problem is not netmap or pkt-gen. Your TX pkt-gen transmits 1Mpps with 0.25 CPUs, which is ok giving the limitations of virtio-net. Your RX pkt-gen is not really doing that much work (0.05 CPU), which means that the virtio-net RX ring is almost always empty. You need to figure out who is dropping the RX packets and why. This seems to happen before the packets really make their way to the virtio-net RX ring. So it must be some queue overflow in the macvtap or your ixgbe device. I think the received packets are handled by this function https://elixir.free-electrons.com/linux/latest/source/drivers/net/tap.c#L317 which is registered here https://elixir.free-electrons.com/linux/latest/source/drivers/net/macvtap.c#L101 so if you see the macvtap interface drop counters increasing, it must be tap_handle_frame() dropping. If this is true, it means that your problem is macvtap. Cheers, Vincenzo > Joe Buehler > -- Vincenzo Maffione
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CA%2B_eA9iRkyNFBwvSok7Jt6xbsafP75cBGigkr_TbwVsq1mm0QQ>