Date: Wed, 22 Nov 2017 14:39:55 +0100 From: Vincenzo Maffione <v.maffione@gmail.com> To: Xiaoye Sun <Xiaoye.Sun@rice.edu> Cc: Luigi Rizzo <rizzo@iet.unipi.it>, "freebsd-net@freebsd.org" <freebsd-net@freebsd.org>, Victor Detoni <victordetoni@gmail.com>, Pavel Odintsov <pavel.odintsov@gmail.com> Subject: Re: swaping ring slots between NIC ring and Host ring does not always success Message-ID: <CA%2B_eA9h3NL6Ga4=HRjEiDtNaANkR8m7oZ9%2BCCpoWVB%2BaYW3%2Btw@mail.gmail.com> In-Reply-To: <CAJnByzipicNHhmw0kGq0cXnfgXqq1xP0aJ4dqr12SryBV1Ny3w@mail.gmail.com> References: <CAJnByzj6Dj3vouZ2NbxqvCV-2-7TVtTR4FaWKuCFaaRN2X%2ByAA@mail.gmail.com> <CALgsdbd3XuE3wMYp4ey%2B1aer%2BHSVNojLYoVqwqTBPAXXdf9i%2BQ@mail.gmail.com> <CAJnByzirLXdCe-kwHV2s_E6ytGJG0Dth=0Ms12RrEk7FK_%2B8Og@mail.gmail.com> <CA%2BhQ2%2BgMWY0eabjHGw0=PJCAkS-wO=RBrN5brSbaqWc3_AOYoQ@mail.gmail.com> <CAJnByziBS8o6LtmpUrUu5xtRUd008Z2hnCsp=WVFv35r2J0rHw@mail.gmail.com> <CA%2BhQ2%2Bim9nFfYnqDS2HgRbAzdf5D0iaLCmCYhfXQVVRMouUFuw@mail.gmail.com> <CAJnByzht-qfDcm8oEg1aSRyVBZ1ygPvc2eMuoyJcq4geueTZ0Q@mail.gmail.com> <CA%2BhQ2%2BiERgWJ=cdFB-cByfT3r11T1kKr-5HiuCYZY-rxbjf=XA@mail.gmail.com> <CAJnByziDzdR2C6DcSRNPtrWACLq0XFpe4X1Ek9yXtFP9ivqWQw@mail.gmail.com> <CA%2BhQ2%2BhjnuGo1xKgc8CQ7gP35tiaZG7%2BroZBmX8aBgb8qWnLgg@mail.gmail.com> <CAJnByzh-VrRZeYdpkRFtCUGEN_arFBkemcN7byb51XV6UPswyg@mail.gmail.com> <CA%2BhQ2%2BiMw3kxjpcZy77vgOEsfk2UY0-farh9C8RKXZHMU7D8kw@mail.gmail.com> <CAJnByzgsuNBhdfPJsGrrHcU79xjK%2Bdq2RENgUkbZcehFm8MUxg@mail.gmail.com> <CAJnByzgNZ9YsYd7tBgYxiQPvuS_VZbhZNGvsPS-0apCDga7XFA@mail.gmail.com> <CANpwN=uHk-VwOoFz7NaPE9A-0B=MAapqxJ-uyCBtn=oMdacYnw@mail.gmail.com> <CAJnByzgjEEAzmWZu7BsSWHXmpjUtZcqXFGN8umCqmvgME1Jv%2BA@mail.gmail.com> <CANpwN=tfqitQW0BTXA7bU%2BTfmP8=wr7gE8wAP=hjAamjD7ny9Q@mail.gmail.com> <CAJnByzi5WHHvwcrmEOkJOHf5SJekbTtQoUgLmPbMtwTotc8mzA@mail.gmail.com> <CAJnByzhJnrmiwiLEEQV0meg7%2BDnLJ6Jq_J=6L=35Z9Lgw1GcyA@mail.gmail.com> <CA%2BhQ2%2BjTXyYdN4N4aWOkDdkBr%2BD3quocHn%2Bc8MA%2Bxc9yLs9V4w@mail.gmail.com> <CAJnByzg0PCXCyjnypS_2%2BRhKB0mf_4s8X1niFiyfedaCLUku7Q@mail.gmail.com> <CAJnByzipicNHhmw0kGq0cXnfgXqq1xP0aJ4dqr12SryBV1Ny3w@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Hi, 2017-11-21 7:51 GMT+01:00 Xiaoye Sun <Xiaoye.Sun@rice.edu>: > Hi, > > Recently I found another problem with netmap. I think this new problem > could be related to the problems in this threads so I just post the new > problem here. > > In my setup, I have a sender program having a netmap ring (a pair of > RX/TX ring) for the NIC and a ring for the host stack. The sender program > puts customized packets (each packet has a unique sequence number and the > sender sends the packet in a sequence number increasing order) to the NIC > TX ring directly and also forwards the packets from the host RX ring to the > NIC TX ring using "zerocopy" by swapping the buffer indices. > However, the receiver sees duplicated customized packets. For example, in > the case where the ring size is 32 (32 slots in a ring) the order of the > sequence numbers the receiver see is 1,2,3,4,5,...,68,69,*70* > ,71,72,73,...,99,100,*70*,101,102,103,... . An interesting thing I found > is > that the "gaps" between these two duplicated packets (70 in the example) > are always a number very close to the ring size, 32 in this example. In my > experiment, I use a ring with 4096 slots and the gap is always more than > 4090 and close to 4096. I verified that this duplication happens due to the > sender, not the receiver. Assuming my sender's implementation is correct, > then this duplication may happen in netmap and the NIC driver (ixgbe). > Netmap itself doesn't do any duplication nor takes a look at the packets. It just passes down ring->cur/ring->head to the ixgbe driver (after validation). The ixgbe driver datapath is bypassed and replaced with a netmap-enabled datapath (see https://github.com/luigirizzo/netmap/blob/master/LINUX/ixgbe_netmap_linux.h#L294-L461 ); no duplication should happen there as each netmap slot (1 TX packet) is used only once. > > > Thinking back to the original problem in this post, I think these problems > may be related. It seems to me that there could be multiple threads pulling > the packets from the NIC TX ring (or the thread moved to other CPUs when > the problem occurs) and these threads may run on different cores so that > the outdated content in the buffer may be sent out when new content is > written to the buffer. > > There are no such threads pulling from the NIC TX ring. Your application directly puts new packets to be transmitted in the netmap buffers referenced in the netmap TX ring. When then you call NIOCTXSYNC or poll(), all the new TX buffers (e.g. all the ones from the previous value ring->head (included) to the new value of ring->head (excluded)) are moved to the NIC TX ring. This happens in the context of your application thread, no worker threads are used. Then the NIC hardware starts the transmission. > I am wondering if there is a way to pin the NIC driver of the netmap module > to a specific core. or is there a way to know the root of such problem? > The only threads are the ones of your application. Maybe your problem comes from concurrent accesses to the netmap TX ring from different threads? Only one thread at a given time should update a netmap TX/RX ring. Otherwise the behaviour is unspecified. Cheers, Vincenzo > > Best, > Xiaoye > > -- Vincenzo Maffione
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CA%2B_eA9h3NL6Ga4=HRjEiDtNaANkR8m7oZ9%2BCCpoWVB%2BaYW3%2Btw>