From owner-freebsd-net@freebsd.org Tue Feb 2 04:48:55 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2A684A98733 for ; Tue, 2 Feb 2016 04:48:55 +0000 (UTC) (envelope-from rizzo.unipi@gmail.com) Received: from mail-lf0-x234.google.com (mail-lf0-x234.google.com [IPv6:2a00:1450:4010:c07::234]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 87E91F99 for ; Tue, 2 Feb 2016 04:48:54 +0000 (UTC) (envelope-from rizzo.unipi@gmail.com) Received: by mail-lf0-x234.google.com with SMTP id m1so35664579lfg.0 for ; Mon, 01 Feb 2016 20:48:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=dDuBUbj8fiuD9SoHi8K+UO2Yg7IpvaZKoRfQ5ulQNZo=; b=RUtNa+yJ8K/L1dV2CU9yNe00b3JCHOiJMqWNq3Saq7tLW4vCc8GteZiUUTeltIZkLU 1Ynf3cR1u6XX5Lb69kkaGtnGFBeIa20MpZ8n3vmyE9gahPjjq747OXC20kpiUHk9Dpsc tVS9t9/f/UE5VMeMDGkkX7pOiL5O/FtseNAzwNbSV/wkzzPpxBHsU4iqFhaJC4WEMLNw 79OyZG6vFjxsXXIo2sy/f0IR5g4VYfUmx9lcgZ3/WNNOvTpTnKN+y8W097Uk0XUQt5uI I9Jm2Kuu4uXFuqmCWsO8TCgyBzjJAu+kE5vULuTQvUH34U3rCbVLddTY0Cb1AAHNWbXh o3IQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:sender:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=dDuBUbj8fiuD9SoHi8K+UO2Yg7IpvaZKoRfQ5ulQNZo=; b=Z60rT+/Fi8e+nR1uN1SyLOAq1jlckTt1FT9r/A7SjiwFuIY2zoa/M7laV+GP6Up++y ICdzOQxXTDLDZp4XSSxz8oTtIr8l8uvO+D9RnCl7AJbRkfILcnnUdMFcPNCFnIg5QKRV 4xldhrI9Y+1NQeuBgJBStW7ZSc2bXIuPS0JcecoIeX+ojOZDDe2eaeNSbpfTyvfhksJ+ JyvxQaaxlkkAGaBcwX3nwUZCScPMTda6zpvqcnBFGfeoFI5nH/cqHtbsHnZpajagMeMz shuQqj05yqECwBQ4ZrRS8aUSMpS0z+k35OPlMA0fUrnY2evFuaMKXBFoBPkmyz5pFOKJ YGqA== X-Gm-Message-State: AG10YOTAtNYfTOpnvs0IW4XXZ6ATDN4N2WA6shB6Xo4Bb9U4qAw9eL5uqvxmDPfq84WmMFxHm02amNNYnI8raw== MIME-Version: 1.0 X-Received: by 10.25.84.20 with SMTP id i20mr9918899lfb.7.1454388532328; Mon, 01 Feb 2016 20:48:52 -0800 (PST) Sender: rizzo.unipi@gmail.com Received: by 10.114.4.232 with HTTP; Mon, 1 Feb 2016 20:48:52 -0800 (PST) In-Reply-To: References: Date: Tue, 2 Feb 2016 05:48:52 +0100 X-Google-Sender-Auth: 5jHSepP2oOzwyLQAnaeW3KN_ONk Message-ID: Subject: Re: swaping ring slots between NIC ring and Host ring does not always success From: Luigi Rizzo To: Xiaoye Sun Cc: Pavel Odintsov , "freebsd-net@freebsd.org" Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 02 Feb 2016 04:48:55 -0000 Hi, there must be some wrong with your setting because slot indexes must be sequential and in your case they are not (see the jump from 295 to 474 and then back from 485 to 296, and the numerous interleavings that you are seeing later). I have no idea of the cause but typically this pattern is what you see when there are multiple input rings and not just one. Cheers Luigi On Tue, Feb 2, 2016 at 12:24 AM, Xiaoye Sun wrote: > Hi Luigi, > > Thanks for the detailed advice. > > With more detailed experiments, actually I found that the udp > sender/receiver packet reorder issue *might* be irrelevant to the original > issue I posted. However, I think we should solve the udp sender/receiver > issue first. > I run the experiment with more detailed log. Here is my findings. > > 1. I am running a netmap version available since about Oct 13rd from github > (https://github.com/luigirizzo/netmap). So I think this is not the one > related to the buffer allocation issue. I tried to running the newest > version, however, that version causes problem when I exit the bridge program > (something like kernel error which make the os crash). > > 2 & 3. I changed the receiver.c & bridge.c so that I can get more > information (more detailed log). > The reorder happens multiple times (about 10 times) within a second. Here is > one example trace collected from the above two programs. (remembering that > we have udp sender running on one machine; netmap bridge and udp receiver > are running on another machine). > There is only one pair of rings each with 512 slots (511 slot usable) on the > receiver machine. > > =================== packet trace collected from receiver.c > =================== > ===== together with the slot and buf_idx of the corresponding netmap ring > slots ====== > [seq] [slot] [buf_idx] > 8208 294 1833 > 8209 295 1834 > 8388 474 2013 > ... (packet received in order) > 8398 484 2023 > 8399 485 2024 > 8210 296 1835 > 8211 297 1836 > ... (packet received in order) > ... > 8222 308 1847 > 8400 486 2025 > 8223 309 1848 > 8401 487 2026 > 8224 310 1849 > 8402 488 2027 > 8225 311 1850 > 8403 489 2028 > 8226 312 1851 > 8404 450 2029 > 8227 313 1852 > 8228 314 1853 > =================================================================== > As we can see that the udp receiver got packet 8210 after it got 8399, which > is the first reorder. Then, the receiver got 8211 to 8222 sequentially. Then > it got packet from 8223-8227 and 8400-8404 interleaved. > > > ==================== event order seen by netmap bridge ================== > get 8209 > poll called > get 8210 > ... > ... > get 8228 > poll called > get 8229 > ... > ... > get 8383 > poll called > get 8384 > ... > get 8387 > poll called > get 8388 > ... > get 8393 > poll called > get 8394 > ... > get 8399 > poll called > get 8400 > ... > get 8404 > poll called > get 8405 > =================================================================== > As we can see, from the event ordering see by the bridge.c, all the packets > are receiver in order, which means the the reorder happens when the bridge > code swap the buf_idx between the nic ring(slot) and the host ring(slot). > The reordered seq usually right before or after the poll function call. > > Best, > Xiaoye > > > > > > > > > On Fri, Jan 29, 2016 at 4:27 PM, Luigi Rizzo wrote: >> >> On Fri, Jan 29, 2016 at 2:12 PM, Xiaoye Sun wrote: >> > Hi Luigi, >> > >> > Thanks for your advice. >> > I forgot to mention that I use the command "ethtool -L eth1 combined 1" >> > to >> > set the number of rings of the nic to 1. The host also only has one >> > ring. >> > I understand the situation where the first tx ring is full so the bridge >> > will swap the packets to the second tx ring and then the host/nic might >> > drain either rings. But this is not the case in the experiment. >> >> ok good to know that. >> >> So if we have ruled out multiqueue and iommu, let's look at >> the internal allocator and at bridge.c >> >> 1. are you running the most recent version of netmap ? >> Some older version (probably 1-2 years ago) had a bug >> in the buffer allocator and some buffers were allocated >> twice. >> >> 2. can you tweak your receiver.c to report some more info >> on how often you get out of sequence packets, how much >> out of sequence they are ? >> Also it would be useful to report gaps on the increasing side >> (i.e. new_seq != old_seq +1 ) >> >> 3. can you tweak bridge.c so that it writes into the packet >> the netmap buffer indexes and slots on the rx and tx side, >> so when you detect a sequence error we can figure out >> where it is happening. >> Ideally you could also add the sequence number detection >> code in bridge.c so we can check whether the errors appear >> on the input or output sides. >> >> cheers >> luigi >> > -- -----------------------------------------+------------------------------- Prof. Luigi RIZZO, rizzo@iet.unipi.it . Dip. di Ing. dell'Informazione http://www.iet.unipi.it/~luigi/ . Universita` di Pisa TEL +39-050-2217533 . via Diotisalvi 2 Mobile +39-338-6809875 . 56122 PISA (Italy) -----------------------------------------+-------------------------------