From owner-freebsd-infiniband@FreeBSD.ORG Thu Dec 6 10:49:27 2012 Return-Path: Delivered-To: infiniband@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 266F7BAE for ; Thu, 6 Dec 2012 10:49:27 +0000 (UTC) (envelope-from daichi@bsd-consulting.jp) Received: from natial.ongs.co.jp (natial.ongs.co.jp [202.216.246.90]) by mx1.freebsd.org (Postfix) with ESMTP id 41AB68FC0C for ; Thu, 6 Dec 2012 10:49:25 +0000 (UTC) Received: from parancell.ongs.co.jp (dullmdaler.ongs.co.jp [202.216.246.94]) by natial.ongs.co.jp (Postfix) with ESMTPSA id 874F5125424; Thu, 6 Dec 2012 19:49:24 +0900 (JST) Date: Thu, 6 Dec 2012 19:49:21 +0900 From: Daichi GOTO To: infiniband@freebsd.org Subject: IB test report : 20121206 Message-Id: <20121206194921.12f07e3d889c9d8cf5a44967@bsd-consulting.jp> Organization: BSD Consulting Inc. X-Mailer: Sylpheed 3.3.0 (GTK+ 2.24.6; amd64-portbld-freebsd9.1) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Cc: Masanori OZAWA , hrs@allbsd.org X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 06 Dec 2012 10:49:27 -0000 Hi guys, ================================================================ IB test report. OS and IB cards: FreeBSD9-stable r243935 MHQH19B-XTR (ConnectX2) MCX353A-QCBT (ConnectX2) ================================================================ TEST1 IBoIP UD (Unreliable Datagram) Kernel configuration include GENERIC ident INFINIBAND options OFED # Infiniband protocol options SDP # Sockets Direct Protocol for infiniband device mlx4ib # ConnectX Infiniband support device mlxen # ConnectX Ethernet support device mthca # Infinihost cards device ipoib # IP over IB devices make.conf WITH_OFED= YES rc.conf (server side) opensm_enable="YES" IP setting ifconfig ib0 mtu 2044 ifconfig ib0 inet 192.168.2.x netmask 0xffffff00 server: iperf bench # iperf -s ------------------------------------------------------------ Server listening on TCP port 5001 TCP window size: 64.0 KByte (default) ------------------------------------------------------------ [ 4] local 192.168.2.1 port 5001 connected with 192.168.2.2 port 12551 [ ID] Interval Transfer Bandwidth [ 4] 0.0-10.0 sec 6.00 GBytes 5.15 Gbits/sec client: iperf bench # iperf -c 192.168.2.1 -i 1 ------------------------------------------------------------ Client connecting to 192.168.2.1, TCP port 5001 TCP window size: 33.1 KByte (default) ------------------------------------------------------------ [ 3] local 192.168.2.2 port 12551 connected with 192.168.2.1 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0- 1.0 sec 47.2 MBytes 396 Mbits/sec [ 3] 1.0- 2.0 sec 49.0 MBytes 411 Mbits/sec [ 3] 2.0- 3.0 sec 260 MBytes 2.18 Gbits/sec [ 3] 3.0- 4.0 sec 880 MBytes 7.39 Gbits/sec [ 3] 4.0- 5.0 sec 926 MBytes 7.77 Gbits/sec [ 3] 5.0- 6.0 sec 879 MBytes 7.38 Gbits/sec [ 3] 6.0- 7.0 sec 464 MBytes 3.90 Gbits/sec [ 3] 7.0- 8.0 sec 947 MBytes 7.95 Gbits/sec [ 3] 8.0- 9.0 sec 945 MBytes 7.93 Gbits/sec [ 3] 9.0-10.0 sec 742 MBytes 6.22 Gbits/sec [ 3] 0.0-10.0 sec 6.00 GBytes 5.15 Gbits/sec result: stable transmission. average 800MB/sec-1000MB/sec mtu 2044 is max. ================================================================ TEST2 IBoIP CM Kernel configuration include GENERIC ident INFINIBAND options OFED # Infiniband protocol options SDP # Sockets Direct Protocol for infiniband options IPOIB_CM#Use connect mode ipoib device mlx4ib # ConnectX Infiniband support device mlxen # ConnectX Ethernet support device mthca # Infinihost cards device ipoib # IP over IB devices others the same result unstable. terrible. average 0MB/sec-1000MB/sec ================================================================ Consideration from this first experimetns For OFED1.5.x of FreeBSD and ConnectX2 cards, Unreliable Datagram mode IB over IP looks the only way to use. CM is very unstable. It's odd. Needs improvement. (We must improve around mtu code? 2044 is dinky) I'm wondering that how many developers are using ConnectX2 cards with CM OFED kernel. ================================================================ -- Daichi GOTO Director | BSD Consulting Inc. 3F SS bldg. 3-4-2, Nishishinbashi, Minato, Tokyo 81-80-4174-6161 | daichi@bsd-consulting.jp