From owner-freebsd-net@freebsd.org Fri Jan 29 22:27:05 2016 Return-Path: <owner-freebsd-net@freebsd.org> Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 30208A7269B for <freebsd-net@mailman.ysv.freebsd.org>; Fri, 29 Jan 2016 22:27:05 +0000 (UTC) (envelope-from rizzo.unipi@gmail.com) Received: from mail-lf0-x22d.google.com (mail-lf0-x22d.google.com [IPv6:2a00:1450:4010:c07::22d]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id A526910A7 for <freebsd-net@freebsd.org>; Fri, 29 Jan 2016 22:27:04 +0000 (UTC) (envelope-from rizzo.unipi@gmail.com) Received: by mail-lf0-x22d.google.com with SMTP id c192so55674572lfe.2 for <freebsd-net@freebsd.org>; Fri, 29 Jan 2016 14:27:04 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=wKUZL3wnQNMJpU7HBSDSOEiKP1xGIakCc0akYWjQEaw=; b=rrIea+EY9yd4+Vm5cMJcrmtCKg+GGqLOc7WyZixeSDABWYhSn9Fg74gAVJci8kaDD5 tOkIMKePHAU3oYKm4EtmvDX23akrZUI6XcANHDC318FfONgMZEHYAmgar0ieT9sFyXvD 7FAFKbfI4Bh/MFajegw3kq9coiWshmeC7ZV0C3JyrMGy1qTcKE7NOXMgIRcNLCf+Z0b4 VER7oOmvhbOED6BdNYdQX0sE6bgO2EwND8cTfvY8V3yGRWcnLju4wPr/JNDLJnNZdo9i /qn6P1kaREsuBIFR/gUSqMkZol7/HYVLE0vR7PkktYzAqFZNGc+DgY7+NAgbXmsqmH/m zr2A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:sender:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=wKUZL3wnQNMJpU7HBSDSOEiKP1xGIakCc0akYWjQEaw=; b=MUKcMhBi3F0uGOtuLrrRDebuiLnXc04T/KiFzcG3+UFSyDYp1UdVrNwzh61QPcB3l4 YB4rmLO73tZS1dTlpiCEVfC7/LEX3/tiHDuuzTy9ANT1mhOQJVD+wot5RhwBVozvyxos voLqAypUTGb26tLfNx5Nc7CCQVt51qaKJzLAWBcTvQZSFD2cqS753JqCy3TqgMncXyhY PXKBYnZ8gZURlIoukNE0zBAWNg1ldPAGgGCuA19hCLUecOY/qHOu6ArJ01BPYs4/l4bv npUHWtiClDmkbqXtofTIHFQTw8tUq6I8sk4ncY9IgTuzjbg8zhwhDwwwY2kTcIUApooQ R1RQ== X-Gm-Message-State: AG10YOSsadOg7Q2ylQgR4s8lrKwIS6Gw98YYV09LbtFDV5lDyiH18k5Rt5TRyE86vpvpYakiPbr7uL3XNxZffw== MIME-Version: 1.0 X-Received: by 10.25.25.142 with SMTP id 136mr4364134lfz.42.1454106422853; Fri, 29 Jan 2016 14:27:02 -0800 (PST) Sender: rizzo.unipi@gmail.com Received: by 10.114.4.232 with HTTP; Fri, 29 Jan 2016 14:27:02 -0800 (PST) In-Reply-To: <CAJnByziBS8o6LtmpUrUu5xtRUd008Z2hnCsp=WVFv35r2J0rHw@mail.gmail.com> References: <CAJnByzj6Dj3vouZ2NbxqvCV-2-7TVtTR4FaWKuCFaaRN2X+yAA@mail.gmail.com> <CALgsdbd3XuE3wMYp4ey+1aer+HSVNojLYoVqwqTBPAXXdf9i+Q@mail.gmail.com> <CAJnByzirLXdCe-kwHV2s_E6ytGJG0Dth=0Ms12RrEk7FK_+8Og@mail.gmail.com> <CA+hQ2+gMWY0eabjHGw0=PJCAkS-wO=RBrN5brSbaqWc3_AOYoQ@mail.gmail.com> <CAJnByziBS8o6LtmpUrUu5xtRUd008Z2hnCsp=WVFv35r2J0rHw@mail.gmail.com> Date: Fri, 29 Jan 2016 14:27:02 -0800 X-Google-Sender-Auth: Mbbv4oFOgF8Ft65L0xfWzDLwGTY Message-ID: <CA+hQ2+im9nFfYnqDS2HgRbAzdf5D0iaLCmCYhfXQVVRMouUFuw@mail.gmail.com> Subject: Re: swaping ring slots between NIC ring and Host ring does not always success From: Luigi Rizzo <rizzo@iet.unipi.it> To: Xiaoye Sun <Xiaoye.Sun@rice.edu> Cc: Pavel Odintsov <pavel.odintsov@gmail.com>, "freebsd-net@freebsd.org" <freebsd-net@freebsd.org> Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD <freebsd-net.freebsd.org> List-Unsubscribe: <https://lists.freebsd.org/mailman/options/freebsd-net>, <mailto:freebsd-net-request@freebsd.org?subject=unsubscribe> List-Archive: <http://lists.freebsd.org/pipermail/freebsd-net/> List-Post: <mailto:freebsd-net@freebsd.org> List-Help: <mailto:freebsd-net-request@freebsd.org?subject=help> List-Subscribe: <https://lists.freebsd.org/mailman/listinfo/freebsd-net>, <mailto:freebsd-net-request@freebsd.org?subject=subscribe> X-List-Received-Date: Fri, 29 Jan 2016 22:27:05 -0000 On Fri, Jan 29, 2016 at 2:12 PM, Xiaoye Sun <Xiaoye.Sun@rice.edu> wrote: > Hi Luigi, > > Thanks for your advice. > I forgot to mention that I use the command "ethtool -L eth1 combined 1" to > set the number of rings of the nic to 1. The host also only has one ring. > I understand the situation where the first tx ring is full so the bridge > will swap the packets to the second tx ring and then the host/nic might > drain either rings. But this is not the case in the experiment. ok good to know that. So if we have ruled out multiqueue and iommu, let's look at the internal allocator and at bridge.c 1. are you running the most recent version of netmap ? Some older version (probably 1-2 years ago) had a bug in the buffer allocator and some buffers were allocated twice. 2. can you tweak your receiver.c to report some more info on how often you get out of sequence packets, how much out of sequence they are ? Also it would be useful to report gaps on the increasing side (i.e. new_seq != old_seq +1 ) 3. can you tweak bridge.c so that it writes into the packet the netmap buffer indexes and slots on the rx and tx side, so when you detect a sequence error we can figure out where it is happening. Ideally you could also add the sequence number detection code in bridge.c so we can check whether the errors appear on the input or output sides. cheers luigi