Date: Fri, 24 Nov 2017 14:49:57 -0600 From: Xiaoye Sun <Xiaoye.Sun@rice.edu> To: Vincenzo Maffione <v.maffione@gmail.com> Cc: Luigi Rizzo <rizzo@iet.unipi.it>, "freebsd-net@freebsd.org" <freebsd-net@freebsd.org>, Victor Detoni <victordetoni@gmail.com>, Pavel Odintsov <pavel.odintsov@gmail.com> Subject: Re: swaping ring slots between NIC ring and Host ring does not always success Message-ID: <CAJnByzjozV4nKHN%2BQ=tgYXWwxg-5=inKNL_nWdtKvHXx5MTZpQ@mail.gmail.com> In-Reply-To: <CA%2B_eA9h3NL6Ga4=HRjEiDtNaANkR8m7oZ9%2BCCpoWVB%2BaYW3%2Btw@mail.gmail.com> References: <CAJnByzj6Dj3vouZ2NbxqvCV-2-7TVtTR4FaWKuCFaaRN2X%2ByAA@mail.gmail.com> <CALgsdbd3XuE3wMYp4ey%2B1aer%2BHSVNojLYoVqwqTBPAXXdf9i%2BQ@mail.gmail.com> <CAJnByzirLXdCe-kwHV2s_E6ytGJG0Dth=0Ms12RrEk7FK_%2B8Og@mail.gmail.com> <CA%2BhQ2%2BgMWY0eabjHGw0=PJCAkS-wO=RBrN5brSbaqWc3_AOYoQ@mail.gmail.com> <CAJnByziBS8o6LtmpUrUu5xtRUd008Z2hnCsp=WVFv35r2J0rHw@mail.gmail.com> <CA%2BhQ2%2Bim9nFfYnqDS2HgRbAzdf5D0iaLCmCYhfXQVVRMouUFuw@mail.gmail.com> <CAJnByzht-qfDcm8oEg1aSRyVBZ1ygPvc2eMuoyJcq4geueTZ0Q@mail.gmail.com> <CA%2BhQ2%2BiERgWJ=cdFB-cByfT3r11T1kKr-5HiuCYZY-rxbjf=XA@mail.gmail.com> <CAJnByziDzdR2C6DcSRNPtrWACLq0XFpe4X1Ek9yXtFP9ivqWQw@mail.gmail.com> <CA%2BhQ2%2BhjnuGo1xKgc8CQ7gP35tiaZG7%2BroZBmX8aBgb8qWnLgg@mail.gmail.com> <CAJnByzh-VrRZeYdpkRFtCUGEN_arFBkemcN7byb51XV6UPswyg@mail.gmail.com> <CA%2BhQ2%2BiMw3kxjpcZy77vgOEsfk2UY0-farh9C8RKXZHMU7D8kw@mail.gmail.com> <CAJnByzgsuNBhdfPJsGrrHcU79xjK%2Bdq2RENgUkbZcehFm8MUxg@mail.gmail.com> <CAJnByzgNZ9YsYd7tBgYxiQPvuS_VZbhZNGvsPS-0apCDga7XFA@mail.gmail.com> <CANpwN=uHk-VwOoFz7NaPE9A-0B=MAapqxJ-uyCBtn=oMdacYnw@mail.gmail.com> <CAJnByzgjEEAzmWZu7BsSWHXmpjUtZcqXFGN8umCqmvgME1Jv%2BA@mail.gmail.com> <CANpwN=tfqitQW0BTXA7bU%2BTfmP8=wr7gE8wAP=hjAamjD7ny9Q@mail.gmail.com> <CAJnByzi5WHHvwcrmEOkJOHf5SJekbTtQoUgLmPbMtwTotc8mzA@mail.gmail.com> <CAJnByzhJnrmiwiLEEQV0meg7%2BDnLJ6Jq_J=6L=35Z9Lgw1GcyA@mail.gmail.com> <CA%2BhQ2%2BjTXyYdN4N4aWOkDdkBr%2BD3quocHn%2Bc8MA%2Bxc9yLs9V4w@mail.gmail.com> <CAJnByzg0PCXCyjnypS_2%2BRhKB0mf_4s8X1niFiyfedaCLUku7Q@mail.gmail.com> <CAJnByzipicNHhmw0kGq0cXnfgXqq1xP0aJ4dqr12SryBV1Ny3w@mail.gmail.com> <CA%2B_eA9h3NL6Ga4=HRjEiDtNaANkR8m7oZ9%2BCCpoWVB%2BaYW3%2Btw@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Hi Vincenzo, Thanks for your reply. Let me clarify my problem. I have a program, which is an extension of bridge.c https://github.com/luigirizzo/netmap/blob/788f25dcc48dfec2e481573277b662968f690042/LINUX/ixgbe_netmap_linux.h#L377 On Wed, Nov 22, 2017 at 7:39 AM, Vincenzo Maffione <v.maffione@gmail.com> wrote: > Hi, > > 2017-11-21 7:51 GMT+01:00 Xiaoye Sun <Xiaoye.Sun@rice.edu>: > >> Hi, >> >> Recently I found another problem with netmap. I think this new problem >> could be related to the problems in this threads so I just post the new >> problem here. >> >> In my setup, I have a sender program having a netmap ring (a pair of >> RX/TX ring) for the NIC and a ring for the host stack. The sender program >> puts customized packets (each packet has a unique sequence number and the >> sender sends the packet in a sequence number increasing order) to the NIC >> TX ring directly and also forwards the packets from the host RX ring to >> the >> NIC TX ring using "zerocopy" by swapping the buffer indices. >> However, the receiver sees duplicated customized packets. For example, in >> the case where the ring size is 32 (32 slots in a ring) the order of the >> sequence numbers the receiver see is 1,2,3,4,5,...,68,69,*70* >> ,71,72,73,...,99,100,*70*,101,102,103,... . An interesting thing I found >> is >> that the "gaps" between these two duplicated packets (70 in the example) >> are always a number very close to the ring size, 32 in this example. In my >> experiment, I use a ring with 4096 slots and the gap is always more than >> 4090 and close to 4096. I verified that this duplication happens due to >> the >> sender, not the receiver. Assuming my sender's implementation is correct, >> then this duplication may happen in netmap and the NIC driver (ixgbe). >> > > Netmap itself doesn't do any duplication nor takes a look at the packets. > It just passes > down ring->cur/ring->head to the ixgbe driver (after validation). > The ixgbe driver datapath is bypassed and replaced with a netmap-enabled > datapath (see https://github.com/luigirizzo/netmap/blob/master/LINUX/ > ixgbe_netmap_linux.h#L294-L461); > no duplication should happen there as each netmap slot (1 TX packet) is > used > only once. > >> >> >> Thinking back to the original problem in this post, I think these problems >> may be related. It seems to me that there could be multiple threads >> pulling >> the packets from the NIC TX ring (or the thread moved to other CPUs when >> the problem occurs) and these threads may run on different cores so that >> the outdated content in the buffer may be sent out when new content is >> written to the buffer. >> >> > There are no such threads pulling from the NIC TX ring. Your application > directly > puts new packets to be transmitted in the netmap buffers referenced in the > netmap TX > ring. When then you call NIOCTXSYNC or poll(), all the new TX buffers > (e.g. all > the ones from the previous value ring->head (included) to the new value of > ring->head (excluded)) > are moved to the NIC TX ring. This happens in the context of your > application thread, > no worker threads are used. Then the NIC hardware starts the transmission. > > >> I am wondering if there is a way to pin the NIC driver of the netmap >> module >> to a specific core. or is there a way to know the root of such problem? >> > > The only threads are the ones of your application. > Maybe your problem comes from concurrent accesses to the netmap TX ring > from different threads? Only one thread at a given time should update a > netmap > TX/RX ring. Otherwise the behaviour is unspecified. > > Cheers, > Vincenzo > > >> >> Best, >> Xiaoye >> >> > -- > Vincenzo Maffione >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAJnByzjozV4nKHN%2BQ=tgYXWwxg-5=inKNL_nWdtKvHXx5MTZpQ>