From owner-freebsd-net@FreeBSD.ORG Sat Mar 24 12:04:08 2012 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 6E994106564A for ; Sat, 24 Mar 2012 12:04:08 +0000 (UTC) (envelope-from lexa@lexa.ru) Received: from www.lexa.ru (www.lexa.ru [213.59.0.94]) by mx1.freebsd.org (Postfix) with ESMTP id 221018FC14 for ; Sat, 24 Mar 2012 12:04:07 +0000 (UTC) Received: by www.lexa.ru (Postfix, from userid 66) id 82B741149A; Sat, 24 Mar 2012 15:56:06 +0400 (MSK) Received: from [127.0.0.1] (unknown [193.124.130.130]) by home-gw.lexa.ru (Postfix) with ESMTP id EB97161F9A; Sat, 24 Mar 2012 15:52:11 +0400 (MSK) Message-ID: <4F6DB568.1060604@lexa.ru> Date: Sat, 24 Mar 2012 15:52:08 +0400 From: Alex Tutubalin User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:11.0) Gecko/20120312 Thunderbird/11.0 MIME-Version: 1.0 To: freebsd-stable@freebsd.org, freebsd-net@freebsd.org Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: Subject: 9-STABLE + Infiniband - incorrect interface counters X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 24 Mar 2012 12:04:08 -0000 Hi, I'm playing with two FreeBSD 9-STABLE boxes connected via 10Gbps Infiniband (more details below) in Infiniband connected mode. I see incorrect interface statistics (e.g. in netstat output), output counters are 2x more than expected. EXAMPLE, ftp transfer of 1 GiB file: ftp> put file /dev/null local: file remote: /dev/null 229 Entering Extended Passive Mode (|||57978|) 150 Opening BINARY mode data connection for '/dev/null'. 100% |***********************************| 953 MiB 390.43 MiB/s 00:00 ETA 226 Transfer complete. 1000000000 bytes sent in 00:02 (390.13 MiB/s) Netstat on receiving side, counters are correct (for input): lexa@home-gw:/home/lexa# netstat -I ib1 5 input (ib1) output packets errs idrops bytes packets errs bytes colls 0 0 0 0 0 0 0 0 13955 0 0 222688126 9027 0 1192796 0 48921 0 0 780832960 32129 0 4240596 0 0 0 0 0 0 0 80 0 Sum of bytes (input) is 1003521086, as expected. Netstat on sending size, output is 2x more: lexa@new-gw:/home/lexa# netstat -I ib0 5 input (ib0) output packets errs idrops bytes packets errs bytes colls 1 0 0 100 0 0 0 0 41162 0 0 2305210 62878 0 2008325984 0 1 0 0 100 0 0 0 0 It looks like packet count is correct (13955+48921=62876, two packets missed somewhere), while byte count is exact 2x more. ==== More details on my setup ==== FreeBSD 9-STABLE, cvsuped today. One box is Core 2 Quad (Q9300), second one Core i7-920 1) Device MELLANOX MHEA28-XTC 10GBPS INFINIBAND HCA CARD (two port) Boot message: ib_mthca0: mem 0xfe900000-0xfe9fffff,0xfd000000-0xfd7fffff irq 16 at device 0.0 on pci1 ib_mthca: Mellanox InfiniBand HCA driver v1.0-ofed1.5.2 (August 4, 2010) Two cards connected via cable, no Infiniband switch 2) Kernel config: include GENERIC options OFED options SDP device ipoib options IPOIB_CM device mthca 3) Regardles of MTU settings (tried 16000, 32000, 48000), actual packed size in tcp flow is about 16000. Have not investigated it in details 4) There is no packet loss: lexa@new-gw:/home/lexa# ping -s 32000 -c 10000 -f 10.1.1.1 PING 10.1.1.1 (10.1.1.1): 32000 data bytes . --- 10.1.1.1 ping statistics --- 10000 packets transmitted, 10000 packets received, 0.0% packet loss round-trip min/avg/max/stddev = 0.157/0.225/1.758/0.156 ms -- Alex Tutubalin Web: http://blog.lexa.ru mailto:lexa@lexa.ru