From owner-freebsd-net@FreeBSD.ORG Mon Mar 26 14:48:13 2012 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 062EB106564A; Mon, 26 Mar 2012 14:48:13 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from cyrus.watson.org (cyrus.watson.org [65.122.17.42]) by mx1.freebsd.org (Postfix) with ESMTP id CEF248FC1E; Mon, 26 Mar 2012 14:48:12 +0000 (UTC) Received: from bigwig.baldwin.cx (bigwig.baldwin.cx [96.47.65.170]) by cyrus.watson.org (Postfix) with ESMTPSA id 862BA46B0D; Mon, 26 Mar 2012 10:48:12 -0400 (EDT) Received: from jhbbsd.localnet (unknown [209.249.190.124]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id C3546B94E; Mon, 26 Mar 2012 10:48:11 -0400 (EDT) From: John Baldwin To: freebsd-net@freebsd.org Date: Mon, 26 Mar 2012 10:42:15 -0400 User-Agent: KMail/1.13.5 (FreeBSD/8.2-CBSD-20110714-p10; KDE/4.5.5; amd64; ; ) References: <4F6DB568.1060604@lexa.ru> In-Reply-To: <4F6DB568.1060604@lexa.ru> MIME-Version: 1.0 Content-Type: Text/Plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Message-Id: <201203261042.15948.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7 (bigwig.baldwin.cx); Mon, 26 Mar 2012 10:48:11 -0400 (EDT) Cc: Alex Tutubalin , Jeff Roberson , freebsd-stable@freebsd.org Subject: Re: 9-STABLE + Infiniband - incorrect interface counters X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 26 Mar 2012 14:48:13 -0000 On Saturday, March 24, 2012 7:52:08 am Alex Tutubalin wrote: > Hi, > > I'm playing with two FreeBSD 9-STABLE boxes connected via 10Gbps > Infiniband (more details below) in Infiniband connected mode. > > I see incorrect interface statistics (e.g. in netstat output), output > counters are 2x more than expected. > > EXAMPLE, ftp transfer of 1 GiB file: > > ftp> put file /dev/null > local: file remote: /dev/null > 229 Entering Extended Passive Mode (|||57978|) > 150 Opening BINARY mode data connection for '/dev/null'. > 100% |***********************************| 953 MiB 390.43 MiB/s > 00:00 ETA > 226 Transfer complete. > 1000000000 bytes sent in 00:02 (390.13 MiB/s) > > Netstat on receiving side, counters are correct (for input): > > lexa@home-gw:/home/lexa# netstat -I ib1 5 > input (ib1) output > packets errs idrops bytes packets errs bytes colls > 0 0 0 0 0 0 0 0 > 13955 0 0 222688126 9027 0 1192796 0 > 48921 0 0 780832960 32129 0 4240596 0 > 0 0 0 0 0 0 80 0 > > Sum of bytes (input) is 1003521086, as expected. > > Netstat on sending size, output is 2x more: > > lexa@new-gw:/home/lexa# netstat -I ib0 5 > input (ib0) output > packets errs idrops bytes packets errs bytes colls > 1 0 0 100 0 0 0 0 > 41162 0 0 2305210 62878 0 2008325984 0 > 1 0 0 100 0 0 0 0 > > It looks like packet count is correct (13955+48921=62876, two packets > missed somewhere), while byte count is exact 2x more. Yes, this is a bug. if_obytes already gets incremented in IFQ_HANDOFF(), so the IB code doesn't need to do it again. Try the patch at www.freebsd.org/~jhb/patches/ipoib_obytes.patch I can't speak to the MTU issue though. -- John Baldwin