From owner-freebsd-net@freebsd.org Mon Jan 25 23:55:21 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A7C2DA4554A for ; Mon, 25 Jan 2016 23:55:21 +0000 (UTC) (envelope-from sunxiaoye07@gmail.com) Received: from mail-oi0-x22c.google.com (mail-oi0-x22c.google.com [IPv6:2607:f8b0:4003:c06::22c]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 66894D14 for ; Mon, 25 Jan 2016 23:55:21 +0000 (UTC) (envelope-from sunxiaoye07@gmail.com) Received: by mail-oi0-x22c.google.com with SMTP id w75so98239371oie.0 for ; Mon, 25 Jan 2016 15:55:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:date:message-id:subject:from:to:content-type; bh=V7HfN4hscS7ZglDSy7yyJCoLEmVj3yHTDTWBo/jdXVk=; b=l9dFmuQh37GkmRzFIb3f4FHFICyz/2P3qAIB40IoQimy9uYrK0HXdjNg0MCkJt4pXk PM2RjgUCGuFvv98wgdqHlII24iYsi9MNKGTUW/y7PyIhqqQ0fJku3r8pipGmH3lTbXKR pafqs7mjdC4wmL2JNVpq0mv/a/0KANmHxiEIjZCJVnJDshMNOtxKb0vzwpDfZayvkld0 s91UFZ3s3F26lNTf9uoIMQhFMLnLvlxP6cjciyrDufKf0vi6kOdZSQsX3QvXhH7IcivU 0LAhLJ4DqEdw7mUN/0whK4hyYc5rH3q2wIvw/hTRU+Q8aKkPxB4waBlvM3E50O1Xwcei vSbw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rice-edu.20150623.gappssmtp.com; s=20150623; h=mime-version:sender:date:message-id:subject:from:to:content-type; bh=V7HfN4hscS7ZglDSy7yyJCoLEmVj3yHTDTWBo/jdXVk=; b=DCzW4+1NbAbLTI/PvTcoJItEreeM8tv8amwbDfSFnLnHS+I7UNFv+d4NHh6tfkr4SK GhTLPazQ4a1csTqJg4yENhbc/Vx0ZryCwmF5WxLc+L34G6jugE9DWYvXjecixCK60Irc Pf15FwxHTZTMZCDfNIbQDGanzjLyX89Vieyb3x6PSiv/sz72dfksQElx7FgPvHegnhUn XG/oSeGIX5vBItFLOFlJPArD1/tN2eHuzpFyUNcMbNntPPHduaBZMqUTIV2L7NqCNDYa KJvudEGDznSIbjeLX4GrMf5ZgEr5hF64xjszUYZOtBg525ZJ2DOffat3iQXrysXIIJyB /qgg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:sender:date:message-id:subject:from :to:content-type; bh=V7HfN4hscS7ZglDSy7yyJCoLEmVj3yHTDTWBo/jdXVk=; b=JvGWvECzri0xbBDoDidph/Eoyr29apcKGYOz78fWWJKBxdWE0U8X8S/9eixfqYRMUu K1Qz3fJn4nEBLuSp1VQdoDoXOSq84dxYZ2gufJt/EVAOxPL9bTxBFYvdORaXJhR2SYrB 0eNoutV63iNHLVNSrkuNjwPpvQgIWm4EcW3FOK+sCycDQqs1OIovbUXe/Qmax8s+zcgr O6vwpvwbIDT70JgU3XQBBSWrnrfuGsh3NxHdt4j4oMezKgRUsVcHYWPrg2N+x/B+2MOm dvXhf6B9SJ6OR+Mq3zcieL2i63x/SY4PU6LtlI9wm3IvV54RFSFFd0PZWdvW9dtAm9cB HNvg== X-Gm-Message-State: AG10YOSXnl/XavSj4BAzWjPJW927BFTqiNtFNXym4QT57k0Wf7t3gXcJMa7mxVnmxkLdEk6+3WIfHdh042iMJA== MIME-Version: 1.0 X-Received: by 10.202.4.82 with SMTP id 79mr15231360oie.118.1453766120698; Mon, 25 Jan 2016 15:55:20 -0800 (PST) Sender: sunxiaoye07@gmail.com Received: by 10.202.223.5 with HTTP; Mon, 25 Jan 2016 15:55:20 -0800 (PST) Date: Mon, 25 Jan 2016 17:55:20 -0600 X-Google-Sender-Auth: t2Eh3CMV7kIVtE07rFXhp-ruok8 Message-ID: Subject: swaping ring slots between NIC ring and Host ring does not always success From: Xiaoye Sun To: freebsd-net@freebsd.org Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 25 Jan 2016 23:55:21 -0000 Hi all, I wrote a netmap application that is very similar to bridge.c in master/example directory of the netmap code. The major difference between my program and bridge.c is that my application also creates custom packets that are directly put into the tx ring of the NIC (like a packet generator with NIC-Host packet forwarding). Each generated packet has an unique sequence number so that the receiver can tell if a packet is lost. Like bridge.c, the packet forwarding between the nic and the host uses 'zerocopy', so that the program only swaps the buf_idx of the slots to virtually forward the packet to the other end. NS_BUF_CHANGED is set for the slots (both rx and tx) whose buf_idx has been changed due to zerocopy. I set the number of rings on nic to 1 using the ethtool command, i.e., one pair of netmap rings for the nic (one for host tx and one for host rx) and another pair of rings for the host (one for host tx and one for host rx). However, at the receiver side, I found that there is a chance (very little chance, less than 1% of the swaps) where swapping the buf_idx does not success. The receiver might get the packet in the buffer swapped out. For example, the netmap program wants to swap host slot *SH* with NIC slot *SN* in order to send the packet in *SH* from host to the receiver; the receiver might get the packet in *SN* instead. This usually happens to the very end of the available slots. Our experiment is done on *Linux* machine (Linux 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt11-1 (2015-05-24) x86_64 GNU/Linux) with* intel NIC* (Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection) using the *ixgbe* driver. To understand this problem, I look into the patch code to the ixgbe driver. I found that the flag NS_BUF_CHANGED is not used in the linux driver for ixgbe. please look the header file /LINUX/ixgbe_netmap_linux.h. In funtions ixgbe_netmap_txsync and ixgbe_netmap_rxsync, the function call netmap_reload_map is commented out. However, the ixgbe's driver patch for FreeBSD calls the reload function. So, I am wondering if this is a known problem when using netmap on LINUX. Has anybody found the same problem? Is netmap_reload_map required in Linux? why it is not called in the driver patch? What is the recommended solution for this problem? Thanks! Best, Xiaoye