Date: Sat, 24 Mar 2012 15:52:08 +0400 From: Alex Tutubalin <lexa@lexa.ru> To: freebsd-stable@freebsd.org, freebsd-net@freebsd.org Subject: 9-STABLE + Infiniband - incorrect interface counters Message-ID: <4F6DB568.1060604@lexa.ru>
next in thread | raw e-mail | index | archive | help
Hi,
I'm playing with two FreeBSD 9-STABLE boxes connected via 10Gbps
Infiniband (more details below) in Infiniband connected mode.
I see incorrect interface statistics (e.g. in netstat output), output
counters are 2x more than expected.
EXAMPLE, ftp transfer of 1 GiB file:
ftp> put file /dev/null
local: file remote: /dev/null
229 Entering Extended Passive Mode (|||57978|)
150 Opening BINARY mode data connection for '/dev/null'.
100% |***********************************| 953 MiB 390.43 MiB/s
00:00 ETA
226 Transfer complete.
1000000000 bytes sent in 00:02 (390.13 MiB/s)
Netstat on receiving side, counters are correct (for input):
lexa@home-gw:/home/lexa# netstat -I ib1 5
input (ib1) output
packets errs idrops bytes packets errs bytes colls
0 0 0 0 0 0 0 0
13955 0 0 222688126 9027 0 1192796 0
48921 0 0 780832960 32129 0 4240596 0
0 0 0 0 0 0 80 0
Sum of bytes (input) is 1003521086, as expected.
Netstat on sending size, output is 2x more:
lexa@new-gw:/home/lexa# netstat -I ib0 5
input (ib0) output
packets errs idrops bytes packets errs bytes colls
1 0 0 100 0 0 0 0
41162 0 0 2305210 62878 0 2008325984 0
1 0 0 100 0 0 0 0
It looks like packet count is correct (13955+48921=62876, two packets
missed somewhere), while byte count is exact 2x more.
==== More details on my setup ====
FreeBSD 9-STABLE, cvsuped today. One box is Core 2 Quad (Q9300), second
one Core i7-920
1) Device MELLANOX MHEA28-XTC 10GBPS INFINIBAND HCA CARD (two port)
Boot message:
ib_mthca0: <ib_mthca> mem 0xfe900000-0xfe9fffff,0xfd000000-0xfd7fffff
irq 16 at device 0.0 on pci1
ib_mthca: Mellanox InfiniBand HCA driver v1.0-ofed1.5.2 (August 4, 2010)
Two cards connected via cable, no Infiniband switch
2) Kernel config:
include GENERIC
options OFED
options SDP
device ipoib
options IPOIB_CM
device mthca
3) Regardles of MTU settings (tried 16000, 32000, 48000), actual packed
size in tcp flow is about 16000. Have not investigated it in details
4) There is no packet loss:
lexa@new-gw:/home/lexa# ping -s 32000 -c 10000 -f 10.1.1.1
PING 10.1.1.1 (10.1.1.1): 32000 data bytes
.
--- 10.1.1.1 ping statistics ---
10000 packets transmitted, 10000 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.157/0.225/1.758/0.156 ms
--
Alex Tutubalin
Web: http://blog.lexa.ru
mailto:lexa@lexa.ru
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?4F6DB568.1060604>