From owner-freebsd-net@freebsd.org Fri Jan 29 22:12:56 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 321A6A72263 for ; Fri, 29 Jan 2016 22:12:56 +0000 (UTC) (envelope-from sunxiaoye07@gmail.com) Received: from mail-ob0-x236.google.com (mail-ob0-x236.google.com [IPv6:2607:f8b0:4003:c01::236]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id E0FB41AEE for ; Fri, 29 Jan 2016 22:12:55 +0000 (UTC) (envelope-from sunxiaoye07@gmail.com) Received: by mail-ob0-x236.google.com with SMTP id is5so74944295obc.0 for ; Fri, 29 Jan 2016 14:12:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=A1DLkYfr9PsO4TsrwSddp+eUHG2NqzDwhmREI4Kijok=; b=ofn0AVoeTjNA4DFR8xc2gt1YcISVUOJwC6WMaWaAJ5vQCdbhdjYIqzYLDlcoIY/09c qbY1Tfl20BzmyvlNR1GUw3Vv9B+vWKcWzD+SRvff7mZE/Izg0R+oebcctgoKCXd8gT+H pHz3Y4tDD6+i+VyfcFom+aohgVN+hKuKp+ZmoSjybfQa3cNDP5BIM8zjh9P44U3d2KSS LB7ZgqFTpNNC2EvhnCGgpgR98ZfdkhzBvB6Jto0gyVAoLBheJtPd7WPODn1+IYsWQWz2 2po0XFtVEjPD75BKuRGeSaUKd/WwBbXmcJCj3dZDE5wx1rH6qYjgldOZEZj+tcKve8GD Dswg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rice-edu.20150623.gappssmtp.com; s=20150623; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=A1DLkYfr9PsO4TsrwSddp+eUHG2NqzDwhmREI4Kijok=; b=wauoMS0lIyYd7aydA6TxPtZ5MO2G4BcqHIkPvlKOZhGGgenEubcYArg/0G+Uxayz5q PGtoqdeAQDfQZM34Q6rUld6MyRQFyBnepYDkamgKBr4cnjcxLDWiQZy3ERnn9OvljRVx HMxa5/o7BWqrwAICkZI0UpGLwrynsy05J+jOnNCRGGiHjS1O6MLAKMZ6cSuxdI11cuOg PNl9iG//ir2YlD0BL+bEFW1LuU3fIqu96gTjnK/7C+youjgjh9/PanD2wjyrwdHVh9Fi Hvf8ZD0Fd78a2cBhSC7fSeJibaeVnu3Abk1CEDgcSuJeratu5/k/CRLpU73KrujUWZAr 411A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:sender:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=A1DLkYfr9PsO4TsrwSddp+eUHG2NqzDwhmREI4Kijok=; b=BJLTY8p41JALIUw81KowheZC14VGn9M9RINMK4BQv2TQGs7qEfQiFIW6odfYdCpTwP Qi+SYLFbtk8TEe8EMr2hBwj6wOfBm4EUH8gX+YvLE4jVf2mBiokaxWr5AJj0RtRbL70E caWyZPBCVU4uapFM2Y61u+LPIW8z4EHLZ1gobr21ybyvk9kHeqBrHkC4Gman3gj7ADGH XWuPiHuUQSWKVQxCv+S9Vvy+UF6mwZe4yJKkYIMZ/LqBYpwBjwI/lUVIE2B6tfDovDS4 9xNBPrDrfpkUddti5sFCgX/7rNmsEpbJ2akD0JXftbHUO3YHtsR9utpVNEBWch5g7YMh wkpA== X-Gm-Message-State: AG10YOQXyWL/CPQcBQwzuNLCSECQxgomjwmmAfk1CaRjIZv5tSDX1KI5No0bCfW5snZ+wBjbxYM+nNKFKm/qvg== MIME-Version: 1.0 X-Received: by 10.60.178.180 with SMTP id cz20mr8471271oec.15.1454105575106; Fri, 29 Jan 2016 14:12:55 -0800 (PST) Sender: sunxiaoye07@gmail.com Received: by 10.202.223.5 with HTTP; Fri, 29 Jan 2016 14:12:55 -0800 (PST) In-Reply-To: References: Date: Fri, 29 Jan 2016 16:12:55 -0600 X-Google-Sender-Auth: DfAS3uCa70BRA5MnD37j17ZdJEY Message-ID: Subject: Re: swaping ring slots between NIC ring and Host ring does not always success From: Xiaoye Sun To: Luigi Rizzo Cc: Pavel Odintsov , "freebsd-net@freebsd.org" Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 29 Jan 2016 22:12:56 -0000 Hi Luigi, Thanks for your advice. I forgot to mention that I use the command "ethtool -L eth1 combined 1" to set the number of rings of the nic to 1. The host also only has one ring. I understand the situation where the first tx ring is full so the bridge will swap the packets to the second tx ring and then the host/nic might drain either rings. But this is not the case in the experiment. Thanks a lot! Xiaoye On Fri, Jan 29, 2016 at 3:45 PM, Luigi Rizzo wrote: > On Fri, Jan 29, 2016 at 1:39 PM, Xiaoye Sun wrote: > > Hi Pavel, > > > > Our code is somewhat complicate. > > So I did a very simple experiment having the same problem so that you > could > > regenerate the problem. > > Xiaoye, > > before doing more experiments can you please clarify > the operating conditions, specifically make sure that > you are only using a single receive and transmit queue > on the NICs in the bridge machine ? > I suspect the problem lies here (it was not clear > from your previous description). > netmap cannot make any guarantee on the transmit > order if you are using multiple transmit queues > (same problem on receive, though it is not relevant > here because almost surely all input traffic goes to > the same queue). > > cheers > luigi > > > In the new experiment, I used a very simple udp sender and udp receiver > > that sends and receives udp packet with seq num. > > > > I put the code here. > > http://www.owlnet.rice.edu/~xs6/receiver.c > > http://www.owlnet.rice.edu/~xs6/sender.c > > > > To repeat the experiment, on machine A, I run the sender program > > (command: sender > > 10.10.10.1 (your receiver IP)). > > On machien B, I run a netmap bridge that swap the slot printer between > the > > host ring and the nic ring (the bridge example in netmap package > > /examples/bridge.c using the command ./bridge netmap:eth1 netmap:eth1) > and > > also run the receiver program. > > > > I am able to see that the receiver print message saying that the new seq > > number is less than the seq number received in the previous round. > > > > Please let me know if you successfully generates the same problem. > > > > Thank! > > Xiaoye > > > > > > > > On Tue, Jan 26, 2016 at 1:53 AM, Pavel Odintsov < > pavel.odintsov@gmail.com> > > wrote: > > > >> Hello! > >> > >> Really useful and detailed research. Thanks a lot! Could you share > >> your validation code? It could be useful for consistency tests for > >> netmap. > >> > >> So I could not help with your original question. Let's wait answer > >> from developers of awesome Netmap :) > >> > >> On Tue, Jan 26, 2016 at 2:55 AM, Xiaoye Sun > wrote: > >> > Hi all, > >> > > >> > I wrote a netmap application that is very similar to bridge.c in > >> master/example > >> > directory of the netmap code. The major difference between my program > and > >> > bridge.c is that my application also creates custom packets that are > >> > directly put into the tx ring of the NIC (like a packet generator with > >> > NIC-Host packet forwarding). Each generated packet has an unique > sequence > >> > number so that the receiver can tell if a packet is lost. > >> > > >> > Like bridge.c, the packet forwarding between the nic and the host uses > >> > 'zerocopy', so that the program only swaps the buf_idx of the slots to > >> > virtually forward the packet to the other end. NS_BUF_CHANGED is set > for > >> > the slots (both rx and tx) whose buf_idx has been changed due to > >> zerocopy. > >> > > >> > I set the number of rings on nic to 1 using the ethtool command, i.e., > >> one > >> > pair of netmap rings for the nic (one for host tx and one for host rx) > >> and > >> > another pair of rings for the host (one for host tx and one for host > rx). > >> > > >> > However, at the receiver side, I found that there is a chance (very > >> little > >> > chance, less than 1% of the swaps) where swapping the buf_idx does not > >> > success. The receiver might get the packet in the buffer swapped out. > For > >> > example, the netmap program wants to swap host slot *SH* with NIC slot > >> *SN* > >> > in order to send the packet in *SH* from host to the receiver; the > >> receiver > >> > might get the packet in *SN* instead. This usually happens to the very > >> end > >> > of the available slots. > >> > > >> > Our experiment is done on *Linux* machine (Linux 3.16.0-4-amd64 #1 SMP > >> > Debian 3.16.7-ckt11-1 (2015-05-24) x86_64 GNU/Linux) with* intel NIC* > >> > (Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection) > using > >> > the *ixgbe* driver. > >> > > >> > To understand this problem, I look into the patch code to the ixgbe > >> driver. > >> > I found that the flag NS_BUF_CHANGED is not used in the linux driver > for > >> > ixgbe. please look the header file /LINUX/ixgbe_netmap_linux.h. In > >> funtions > >> > ixgbe_netmap_txsync and ixgbe_netmap_rxsync, the function call > >> > netmap_reload_map is commented out. However, the ixgbe's driver patch > for > >> > FreeBSD calls the reload function. > >> > > >> > So, I am wondering if this is a known problem when using netmap on > LINUX. > >> > Has anybody found the same problem? > >> > Is netmap_reload_map required in Linux? why it is not called in the > >> driver > >> > patch? > >> > What is the recommended solution for this problem? > >> > > >> > Thanks! > >> > > >> > Best, > >> > Xiaoye > >> > _______________________________________________ > >> > freebsd-net@freebsd.org mailing list > >> > https://lists.freebsd.org/mailman/listinfo/freebsd-net > >> > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org > " > >> > >> > >> > >> -- > >> Sincerely yours, Pavel Odintsov > >> > >> > > _______________________________________________ > > freebsd-net@freebsd.org mailing list > > https://lists.freebsd.org/mailman/listinfo/freebsd-net > > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > > > > -- > -----------------------------------------+------------------------------- > Prof. Luigi RIZZO, rizzo@iet.unipi.it . Dip. di Ing. dell'Informazione > http://www.iet.unipi.it/~luigi/ . Universita` di Pisa > TEL +39-050-2217533 . via Diotisalvi 2 > Mobile +39-338-6809875 . 56122 PISA (Italy) > -----------------------------------------+------------------------------- > >