From owner-freebsd-net@freebsd.org Tue Jan 26 07:53:12 2016 Return-Path: <owner-freebsd-net@freebsd.org> Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4122EA6E62C for <freebsd-net@mailman.ysv.freebsd.org>; Tue, 26 Jan 2016 07:53:12 +0000 (UTC) (envelope-from pavel.odintsov@gmail.com) Received: from mail-ig0-x229.google.com (mail-ig0-x229.google.com [IPv6:2607:f8b0:4001:c05::229]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 0E85CEF3 for <freebsd-net@freebsd.org>; Tue, 26 Jan 2016 07:53:12 +0000 (UTC) (envelope-from pavel.odintsov@gmail.com) Received: by mail-ig0-x229.google.com with SMTP id t15so53401676igr.0 for <freebsd-net@freebsd.org>; Mon, 25 Jan 2016 23:53:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=QB1rYhq0tepcKbRIMRJBcSZ8aQtFq6r8HQj6C2sEJYA=; b=0OtmEDTHe0wa7X8sZO2i5SFi6Aoz7hjNg95CnNB4H3+7pvVVATz928oxO6jx8+d65n TqyLHqw0w1MWj32xh4td9pZ8lxhLsCp6+DHIFeulScGYX/fOqW7lzjo+tb6WMP+mTdLs bDaZUNlJWFb1AvX1wVfSR7Hpda6F+B7LuNhp07bWLrgFsPuwhaYfMAQJYkI0nZFhrtjw zmSvxcatK2ekxvAR1a1Zpq5bIR+uNcFVUWHOgmBk1hH839GD2agmexbiYHpcduH0/W3W sJC0euFOj7Gdh5LD/kIoFmvO8IHh18jEZVBWe+0O7U3EfYcgodQ+LxrmMEqINfjMsByI yAmw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=QB1rYhq0tepcKbRIMRJBcSZ8aQtFq6r8HQj6C2sEJYA=; b=TCqQcbw8I6Gvw/wqD7pb3f1mu6a7Ql5JP5UTqZq2ELtdWeaDvo25SziBEbX/TrTI/Z y4PSxal/Oc4ditCmM9qlmjwRQwsXnlJ9uSo9+h40RRYaoUE9AxxKy64Q1PRHeDi+BSKZ 5g9COCe9xlflvVujIofosU8F3POmUZV+IRQ+nxCHtmvDkrjGNxJJHzf5eJZAWzw4h2QH oxpq/TSOTdgHrP63kOpt50h50G0Q42L/ZKtNxQVhVczh0XPlj60wGBMR93FELVviWH5q lI6NiYWr5MmTuDctEetOtdGgMUn77fTTgEhrcABemV77wCQsmbnZaG4wggMakoLTrQOf 1rdA== X-Gm-Message-State: AG10YOTVhoUKff8ytEHuYDWLidndFeRJcABNpzdA13xPKX+sSYtIdPwBIYnjAGXiWYJnIDLKcwDi+c9APifF3g== MIME-Version: 1.0 X-Received: by 10.50.143.10 with SMTP id sa10mr22556453igb.91.1453794791419; Mon, 25 Jan 2016 23:53:11 -0800 (PST) Received: by 10.79.36.20 with HTTP; Mon, 25 Jan 2016 23:53:11 -0800 (PST) In-Reply-To: <CAJnByzj6Dj3vouZ2NbxqvCV-2-7TVtTR4FaWKuCFaaRN2X+yAA@mail.gmail.com> References: <CAJnByzj6Dj3vouZ2NbxqvCV-2-7TVtTR4FaWKuCFaaRN2X+yAA@mail.gmail.com> Date: Tue, 26 Jan 2016 10:53:11 +0300 Message-ID: <CALgsdbd3XuE3wMYp4ey+1aer+HSVNojLYoVqwqTBPAXXdf9i+Q@mail.gmail.com> Subject: Re: swaping ring slots between NIC ring and Host ring does not always success From: Pavel Odintsov <pavel.odintsov@gmail.com> To: Xiaoye Sun <Xiaoye.Sun@rice.edu> Cc: freebsd-net@freebsd.org Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD <freebsd-net.freebsd.org> List-Unsubscribe: <https://lists.freebsd.org/mailman/options/freebsd-net>, <mailto:freebsd-net-request@freebsd.org?subject=unsubscribe> List-Archive: <http://lists.freebsd.org/pipermail/freebsd-net/> List-Post: <mailto:freebsd-net@freebsd.org> List-Help: <mailto:freebsd-net-request@freebsd.org?subject=help> List-Subscribe: <https://lists.freebsd.org/mailman/listinfo/freebsd-net>, <mailto:freebsd-net-request@freebsd.org?subject=subscribe> X-List-Received-Date: Tue, 26 Jan 2016 07:53:12 -0000 Hello! Really useful and detailed research. Thanks a lot! Could you share your validation code? It could be useful for consistency tests for netmap. So I could not help with your original question. Let's wait answer from developers of awesome Netmap :) On Tue, Jan 26, 2016 at 2:55 AM, Xiaoye Sun <Xiaoye.Sun@rice.edu> wrote: > Hi all, > > I wrote a netmap application that is very similar to bridge.c in master/example > directory of the netmap code. The major difference between my program and > bridge.c is that my application also creates custom packets that are > directly put into the tx ring of the NIC (like a packet generator with > NIC-Host packet forwarding). Each generated packet has an unique sequence > number so that the receiver can tell if a packet is lost. > > Like bridge.c, the packet forwarding between the nic and the host uses > 'zerocopy', so that the program only swaps the buf_idx of the slots to > virtually forward the packet to the other end. NS_BUF_CHANGED is set for > the slots (both rx and tx) whose buf_idx has been changed due to zerocopy. > > I set the number of rings on nic to 1 using the ethtool command, i.e., one > pair of netmap rings for the nic (one for host tx and one for host rx) and > another pair of rings for the host (one for host tx and one for host rx). > > However, at the receiver side, I found that there is a chance (very little > chance, less than 1% of the swaps) where swapping the buf_idx does not > success. The receiver might get the packet in the buffer swapped out. For > example, the netmap program wants to swap host slot *SH* with NIC slot *SN* > in order to send the packet in *SH* from host to the receiver; the receiver > might get the packet in *SN* instead. This usually happens to the very end > of the available slots. > > Our experiment is done on *Linux* machine (Linux 3.16.0-4-amd64 #1 SMP > Debian 3.16.7-ckt11-1 (2015-05-24) x86_64 GNU/Linux) with* intel NIC* > (Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection) using > the *ixgbe* driver. > > To understand this problem, I look into the patch code to the ixgbe driver. > I found that the flag NS_BUF_CHANGED is not used in the linux driver for > ixgbe. please look the header file /LINUX/ixgbe_netmap_linux.h. In funtions > ixgbe_netmap_txsync and ixgbe_netmap_rxsync, the function call > netmap_reload_map is commented out. However, the ixgbe's driver patch for > FreeBSD calls the reload function. > > So, I am wondering if this is a known problem when using netmap on LINUX. > Has anybody found the same problem? > Is netmap_reload_map required in Linux? why it is not called in the driver > patch? > What is the recommended solution for this problem? > > Thanks! > > Best, > Xiaoye > _______________________________________________ > freebsd-net@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" -- Sincerely yours, Pavel Odintsov