Date: Fri, 24 Nov 2017 15:11:22 -0600 From: Xiaoye Sun <Xiaoye.Sun@rice.edu> To: Vincenzo Maffione <v.maffione@gmail.com> Cc: Luigi Rizzo <rizzo@iet.unipi.it>, "freebsd-net@freebsd.org" <freebsd-net@freebsd.org>, Victor Detoni <victordetoni@gmail.com>, Pavel Odintsov <pavel.odintsov@gmail.com> Subject: Re: swaping ring slots between NIC ring and Host ring does not always success Message-ID: <CAJnByziVfVJ6gX_aL%2BwFvmW9ZsfwPV1RRCkxgEbZMoe1kSo35A@mail.gmail.com> In-Reply-To: <CA%2B_eA9h3NL6Ga4=HRjEiDtNaANkR8m7oZ9%2BCCpoWVB%2BaYW3%2Btw@mail.gmail.com> References: <CAJnByzj6Dj3vouZ2NbxqvCV-2-7TVtTR4FaWKuCFaaRN2X%2ByAA@mail.gmail.com> <CALgsdbd3XuE3wMYp4ey%2B1aer%2BHSVNojLYoVqwqTBPAXXdf9i%2BQ@mail.gmail.com> <CAJnByzirLXdCe-kwHV2s_E6ytGJG0Dth=0Ms12RrEk7FK_%2B8Og@mail.gmail.com> <CA%2BhQ2%2BgMWY0eabjHGw0=PJCAkS-wO=RBrN5brSbaqWc3_AOYoQ@mail.gmail.com> <CAJnByziBS8o6LtmpUrUu5xtRUd008Z2hnCsp=WVFv35r2J0rHw@mail.gmail.com> <CA%2BhQ2%2Bim9nFfYnqDS2HgRbAzdf5D0iaLCmCYhfXQVVRMouUFuw@mail.gmail.com> <CAJnByzht-qfDcm8oEg1aSRyVBZ1ygPvc2eMuoyJcq4geueTZ0Q@mail.gmail.com> <CA%2BhQ2%2BiERgWJ=cdFB-cByfT3r11T1kKr-5HiuCYZY-rxbjf=XA@mail.gmail.com> <CAJnByziDzdR2C6DcSRNPtrWACLq0XFpe4X1Ek9yXtFP9ivqWQw@mail.gmail.com> <CA%2BhQ2%2BhjnuGo1xKgc8CQ7gP35tiaZG7%2BroZBmX8aBgb8qWnLgg@mail.gmail.com> <CAJnByzh-VrRZeYdpkRFtCUGEN_arFBkemcN7byb51XV6UPswyg@mail.gmail.com> <CA%2BhQ2%2BiMw3kxjpcZy77vgOEsfk2UY0-farh9C8RKXZHMU7D8kw@mail.gmail.com> <CAJnByzgsuNBhdfPJsGrrHcU79xjK%2Bdq2RENgUkbZcehFm8MUxg@mail.gmail.com> <CAJnByzgNZ9YsYd7tBgYxiQPvuS_VZbhZNGvsPS-0apCDga7XFA@mail.gmail.com> <CANpwN=uHk-VwOoFz7NaPE9A-0B=MAapqxJ-uyCBtn=oMdacYnw@mail.gmail.com> <CAJnByzgjEEAzmWZu7BsSWHXmpjUtZcqXFGN8umCqmvgME1Jv%2BA@mail.gmail.com> <CANpwN=tfqitQW0BTXA7bU%2BTfmP8=wr7gE8wAP=hjAamjD7ny9Q@mail.gmail.com> <CAJnByzi5WHHvwcrmEOkJOHf5SJekbTtQoUgLmPbMtwTotc8mzA@mail.gmail.com> <CAJnByzhJnrmiwiLEEQV0meg7%2BDnLJ6Jq_J=6L=35Z9Lgw1GcyA@mail.gmail.com> <CA%2BhQ2%2BjTXyYdN4N4aWOkDdkBr%2BD3quocHn%2Bc8MA%2Bxc9yLs9V4w@mail.gmail.com> <CAJnByzg0PCXCyjnypS_2%2BRhKB0mf_4s8X1niFiyfedaCLUku7Q@mail.gmail.com> <CAJnByzipicNHhmw0kGq0cXnfgXqq1xP0aJ4dqr12SryBV1Ny3w@mail.gmail.com> <CA%2B_eA9h3NL6Ga4=HRjEiDtNaANkR8m7oZ9%2BCCpoWVB%2BaYW3%2Btw@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Hi Vincenzo, Let me clarify my problem. (please ignore the previous incompleted email) I have a program, which is an extension of bridge.c https://github.com/luigirizzo/netmap/blob/master/apps/bridge/bridge.c The only difference is that my program also generates customized packets sent to the NIC directly. These customized packets have increasing sequence numbers. So, this program not only sends these customized packets but also forwards packets between NIC and host stack using zerocopy. The program only takes one NIC queue and there is only one thread. I think the problem is that there is a chance where netmap does not update the pointer to the buffer even when NS_BUF_CHANGED is set (buf_idx is changed). Let's say the NIC tx ring has 4096 slots. The customized packet sequence 16 is filled in the buffer of slot 2057. The customized packets keep filling the slots until the next available slot is 2056. Now the customised packet sequence 4111 is filled to 2056. Then the netmap program is notified that there is a packet from the host stack sent to the NIC. The netmap program swaps the buf_idx between slot 2057 and the corresponding slot in the host rx ring and set the NS_BUF_CHANGED flag of both slots. Then the netmap program fills sequence 4112 to slot 2058. However, the buffer swap seems not succeed so that the original content of slot 2057 (sequence 16) is sent out. So that at the receiver side, the receiver sees two sequence 16s.(16,17...4110,4111,16,4112,4113). So think the root of the problem is that the buffer pointer is not always successfully/timely updated even after the NS_BUF_CHANGED flag is set and the buf_idx is updated. Best, Xiaoye On Wed, Nov 22, 2017 at 7:39 AM, Vincenzo Maffione <v.maffione@gmail.com> wrote: > Hi, > > 2017-11-21 7:51 GMT+01:00 Xiaoye Sun <Xiaoye.Sun@rice.edu>: > >> Hi, >> >> Recently I found another problem with netmap. I think this new problem >> could be related to the problems in this threads so I just post the new >> problem here. >> >> In my setup, I have a sender program having a netmap ring (a pair of >> RX/TX ring) for the NIC and a ring for the host stack. The sender program >> puts customized packets (each packet has a unique sequence number and the >> sender sends the packet in a sequence number increasing order) to the NIC >> TX ring directly and also forwards the packets from the host RX ring to >> the >> NIC TX ring using "zerocopy" by swapping the buffer indices. >> However, the receiver sees duplicated customized packets. For example, in >> the case where the ring size is 32 (32 slots in a ring) the order of the >> sequence numbers the receiver see is 1,2,3,4,5,...,68,69,*70* >> ,71,72,73,...,99,100,*70*,101,102,103,... . An interesting thing I found >> is >> that the "gaps" between these two duplicated packets (70 in the example) >> are always a number very close to the ring size, 32 in this example. In my >> experiment, I use a ring with 4096 slots and the gap is always more than >> 4090 and close to 4096. I verified that this duplication happens due to >> the >> sender, not the receiver. Assuming my sender's implementation is correct, >> then this duplication may happen in netmap and the NIC driver (ixgbe). >> > > Netmap itself doesn't do any duplication nor takes a look at the packets. > It just passes > down ring->cur/ring->head to the ixgbe driver (after validation). > The ixgbe driver datapath is bypassed and replaced with a netmap-enabled > datapath (see https://github.com/luigirizzo/netmap/blob/master/LINUX/ > ixgbe_netmap_linux.h#L294-L461); > no duplication should happen there as each netmap slot (1 TX packet) is > used > only once. > >> >> >> Thinking back to the original problem in this post, I think these problems >> may be related. It seems to me that there could be multiple threads >> pulling >> the packets from the NIC TX ring (or the thread moved to other CPUs when >> the problem occurs) and these threads may run on different cores so that >> the outdated content in the buffer may be sent out when new content is >> written to the buffer. >> >> > There are no such threads pulling from the NIC TX ring. Your application > directly > puts new packets to be transmitted in the netmap buffers referenced in the > netmap TX > ring. When then you call NIOCTXSYNC or poll(), all the new TX buffers > (e.g. all > the ones from the previous value ring->head (included) to the new value of > ring->head (excluded)) > are moved to the NIC TX ring. This happens in the context of your > application thread, > no worker threads are used. Then the NIC hardware starts the transmission. > > >> I am wondering if there is a way to pin the NIC driver of the netmap >> module >> to a specific core. or is there a way to know the root of such problem? >> > > The only threads are the ones of your application. > Maybe your problem comes from concurrent accesses to the netmap TX ring > from different threads? Only one thread at a given time should update a > netmap > TX/RX ring. Otherwise the behaviour is unspecified. > > Cheers, > Vincenzo > > >> >> Best, >> Xiaoye >> >> > -- > Vincenzo Maffione >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAJnByziVfVJ6gX_aL%2BwFvmW9ZsfwPV1RRCkxgEbZMoe1kSo35A>