From owner-freebsd-net@freebsd.org Tue Jan 26 08:01:13 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id AC4C1A6EADE for ; Tue, 26 Jan 2016 08:01:13 +0000 (UTC) (envelope-from rizzo.unipi@gmail.com) Received: from mail-lb0-x230.google.com (mail-lb0-x230.google.com [IPv6:2a00:1450:4010:c04::230]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 35BA93F4 for ; Tue, 26 Jan 2016 08:01:13 +0000 (UTC) (envelope-from rizzo.unipi@gmail.com) Received: by mail-lb0-x230.google.com with SMTP id x4so88052890lbm.0 for ; Tue, 26 Jan 2016 00:01:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=VNP0KfEF+F6XWJS/jdQVSMllKGxOZFqAZ03M5IFpmoY=; b=Vp8V7p4f7BTTrknOsp3WptjTZbtzQbhqaZvr6uhQZ/CR60fwH5x1YHTdPVe9QgwXb5 py0IGWhjZcbsJCCgTz/5Tv6fJrSmtIaMYnhqutoFY8yYmknWfBu7LbV3WKc5Bf4Datx5 MUMST9E64YHbVy1BjLHdtxhlF/88KG3RKzCRXDtSbDoJ4oepHEFWmsHnaj2O7/TDxPk2 fq5jOXfVSfEr7H2LYelg47/oDyW6U1algOSl9HtXdgIu1kSxS+7o+FZy1VwFPhUR+9fh nkoF0wQn/1Ddd6PNvpthXfJvsMsHFnF1pgXNI7lws/FudUutlSYQpuGKUorX9on68W1r zezw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:sender:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=VNP0KfEF+F6XWJS/jdQVSMllKGxOZFqAZ03M5IFpmoY=; b=KaqTQU1TyTt/Y61C5OvBGXBjEUd+Ranr00dLZy1jORN4Nos51R+9NQyB5aBR1ED3M7 WZ6EfTanuCnVJyO2VJyrSWTD4ty0lXElBp7mkkmPuB9Ds7DaN9HhGQslQXHSwhKbsGrN 1c6qZAlel6liYcCwBo1Nqp7FaH2/LQP2gXtUgoWMxy+++GBTvxd9x9Y8T8pJj+JMWwLA vnaAobw/f9YjRrG3I6R81ORJdbQcCgEC20ZO7UGV1ojzMg5vrMS67adKZk2Ng/xCp5L9 NzzqBRGborp0AW7+seCOjt09gTObvdFbubxKU+5lebx0dEjuTAtjNwIf127t0tEdOOOV UXfA== X-Gm-Message-State: AG10YOQIr4JuTBOY6J9Xt5jTjAvQuHFWCligs9QG5ONx/BnXQbJfrVbWh3qh9KPmTWvQdZZ0cL6fXFEv8Btgyw== MIME-Version: 1.0 X-Received: by 10.112.170.101 with SMTP id al5mr7994605lbc.92.1453795271127; Tue, 26 Jan 2016 00:01:11 -0800 (PST) Sender: rizzo.unipi@gmail.com Received: by 10.114.4.232 with HTTP; Tue, 26 Jan 2016 00:01:11 -0800 (PST) In-Reply-To: References: Date: Tue, 26 Jan 2016 00:01:11 -0800 X-Google-Sender-Auth: sxghxxZuvlXf3UTYJFdN9dgl6vM Message-ID: Subject: Re: swaping ring slots between NIC ring and Host ring does not always success From: Luigi Rizzo To: Xiaoye Sun Cc: "freebsd-net@freebsd.org" Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 26 Jan 2016 08:01:13 -0000 Hi, in FreeBSD, netmap_reload_map() is basically a NOP unless the NIC uses bounce buffers (in which case it is pointless to use netmap, and I am not even sure the code works), or there is an iommu which may need to be reprogrammed (but in that case you don't want to do it dynamically on individual buffers, but once for all when the device is opened.) This said, can you check that the iommu is disabled in the bios and re-run the test ? This said I am not sure that the problem is the iommu -- a wrong entry should result in bogus content, certainly not the one before the swap. I suspect some kind of race, possibly missing memory barriers. cheers luigi On Mon, Jan 25, 2016 at 3:55 PM, Xiaoye Sun wrote: > Hi all, > > I wrote a netmap application that is very similar to bridge.c in master/example > directory of the netmap code. The major difference between my program and > bridge.c is that my application also creates custom packets that are > directly put into the tx ring of the NIC (like a packet generator with > NIC-Host packet forwarding). Each generated packet has an unique sequence > number so that the receiver can tell if a packet is lost. > > Like bridge.c, the packet forwarding between the nic and the host uses > 'zerocopy', so that the program only swaps the buf_idx of the slots to > virtually forward the packet to the other end. NS_BUF_CHANGED is set for > the slots (both rx and tx) whose buf_idx has been changed due to zerocopy. > > I set the number of rings on nic to 1 using the ethtool command, i.e., one > pair of netmap rings for the nic (one for host tx and one for host rx) and > another pair of rings for the host (one for host tx and one for host rx). > > However, at the receiver side, I found that there is a chance (very little > chance, less than 1% of the swaps) where swapping the buf_idx does not > success. The receiver might get the packet in the buffer swapped out. For > example, the netmap program wants to swap host slot *SH* with NIC slot *SN* > in order to send the packet in *SH* from host to the receiver; the receiver > might get the packet in *SN* instead. This usually happens to the very end > of the available slots. > > Our experiment is done on *Linux* machine (Linux 3.16.0-4-amd64 #1 SMP > Debian 3.16.7-ckt11-1 (2015-05-24) x86_64 GNU/Linux) with* intel NIC* > (Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection) using > the *ixgbe* driver. > > To understand this problem, I look into the patch code to the ixgbe driver. > I found that the flag NS_BUF_CHANGED is not used in the linux driver for > ixgbe. please look the header file /LINUX/ixgbe_netmap_linux.h. In funtions > ixgbe_netmap_txsync and ixgbe_netmap_rxsync, the function call > netmap_reload_map is commented out. However, the ixgbe's driver patch for > FreeBSD calls the reload function. > > So, I am wondering if this is a known problem when using netmap on LINUX. > Has anybody found the same problem? > Is netmap_reload_map required in Linux? why it is not called in the driver > patch? > What is the recommended solution for this problem? > > Thanks! > > Best, > Xiaoye > _______________________________________________ > freebsd-net@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" -- -----------------------------------------+------------------------------- Prof. Luigi RIZZO, rizzo@iet.unipi.it . Dip. di Ing. dell'Informazione http://www.iet.unipi.it/~luigi/ . Universita` di Pisa TEL +39-050-2217533 . via Diotisalvi 2 Mobile +39-338-6809875 . 56122 PISA (Italy) -----------------------------------------+-------------------------------