From owner-freebsd-infiniband@FreeBSD.ORG Mon Dec 3 16:23:49 2012 Return-Path: Delivered-To: infiniband@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id BDA904FE for ; Mon, 3 Dec 2012 16:23:49 +0000 (UTC) (envelope-from gnn@freebsd.org) Received: from vps.hungerhost.com (vps.hungerhost.com [216.38.53.176]) by mx1.freebsd.org (Postfix) with ESMTP id 43FC98FC14 for ; Mon, 3 Dec 2012 16:23:45 +0000 (UTC) Received: from [38.105.238.108] (port=54846 helo=[10.7.1.235]) by vps.hungerhost.com with esmtpsa (TLSv1:AES128-SHA:128) (Exim 4.80) (envelope-from ) id 1TfYo5-0004YN-Be for infiniband@freebsd.org; Mon, 03 Dec 2012 11:23:45 -0500 From: George Neville-Neil Subject: Old panic report... Date: Mon, 3 Dec 2012 11:23:46 -0500 References: <201205301517.25647.jhb@freebsd.org> To: infiniband@freebsd.org Message-Id: Mime-Version: 1.0 (Mac OS X Mail 6.2 \(1499\)) X-Mailer: Apple Mail (2.1499) X-AntiAbuse: This header was added to track abuse, please include it with any abuse report X-AntiAbuse: Primary Hostname - vps.hungerhost.com X-AntiAbuse: Original Domain - freebsd.org X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12] X-AntiAbuse: Sender Address Domain - freebsd.org X-Get-Message-Sender-Via: vps.hungerhost.com: authenticated_id: gnn@neville-neil.com Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.14 X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Dec 2012 16:23:49 -0000 Howdy, Was cleaning out my inbox and found this. Whoever has infiniband on = their plate atm should take a look at this report. Best, George Begin forwarded message: > From: John Baldwin > Subject: Fwd: Re: Kernel panic caused by OFED mlx4 driver > Date: May 30, 2012 3:17:25 EDT > To: "George Neville-Neil" >=20 > FYI... >=20 > --=20 > John Baldwin >=20 > From: Olivier Cinquin > Subject: Re: Kernel panic caused by OFED mlx4 driver > Date: May 26, 2012 12:20:30 EDT > To: John Baldwin >=20 >=20 > Hi John, > I thought I'd let you know I have things working now. Thanks for your = fix. >=20 > I also wanted to mention that I've identified another problem. This = problem is unlikely to affect me in practice, and I don't know if it's = closely related to your areas of expertise and interest, but I just = thought I'd mention it. When running performance tests of the IP over = Infiniband connection, I found that iperf reported dismal numbers = (around ~1Mb/s). I did further testing and found much higher rates using = the following: > cat /dev/zero | ssh other_machines_ip "cat > /dev/null" > and monitoring traffic with systat -ifs. The throughput of the latter = test is limited by CPU usage of ssh. Using multiple instances of the = above test running in parallel, I could get total throughput up to = ~10Gb/s. However, if after reaching that throughput I launched another = instance of the test, total throughput suddenly dropped back down to = very low levels. >=20 > My guess is that there's a congestion management problem, which I have = no idea how to solve (just to play around, I tried loading kernel = modules cc_cubic.ko or cc_htcp.ko but that didn't address the problem). = It doesn't matter that much to me because my usage is unlikely to = produce rates above 10Gb/s, but other people might run into the problem = (and the iperf results are misleading for all users). >=20 > Best wishes, > Olivier >=20 >=20 >=20 >=20 >=20 > On May 23, 2012, at 6:35 AM, John Baldwin wrote: >=20 >> On Tuesday, May 22, 2012 4:52:52 pm Olivier Cinquin wrote: >>> Here you go... >>> Olivier >>>=20 >>>=20 >>> interrupt total rate >>> irq275: mlx4_core0 0 0 >>> irq276: mlx4_core0 0 0 >>> irq277: mlx4_core0 0 0 >>> irq278: mlx4_core0 0 0 >>> irq279: mlx4_core0 0 0 >>> irq280: mlx4_core0 0 0 >>> irq281: mlx4_core0 0 0 >>> irq282: mlx4_core0 0 0 >>> irq283: mlx4_core0 0 0 >>> irq284: mlx4_core0 0 0 >>> irq285: mlx4_core0 0 0 >>> irq286: mlx4_core0 0 0 >>> irq287: mlx4_core0 0 0 >>> irq288: mlx4_core0 0 0 >>> irq289: mlx4_core0 0 0 >>> irq290: mlx4_core0 0 0 >>> irq291: mlx4_core0 0 0 >>> irq292: mlx4_core0 0 0 >>> irq293: mlx4_core0 0 0 >>> irq294: mlx4_core0 0 0 >>> irq295: mlx4_core0 0 0 >>> irq296: mlx4_core0 0 0 >>> irq297: mlx4_core0 0 0 >>> irq298: mlx4_core0 0 0 >>> irq299: mlx4_core0 0 0 >>> irq300: mlx4_core0 0 0 >>> irq301: mlx4_core0 0 0 >>> irq302: mlx4_core0 0 0 >>> irq303: mlx4_core0 0 0 >>> irq304: mlx4_core0 0 0 >>> irq305: mlx4_core0 0 0 >>> irq306: mlx4_core0 0 0 >>> irq307: mlx4_core0 0 0 >>> irq308: mlx4_core0 0 0 >>> irq309: mlx4_core0 0 0 >>> irq310: mlx4_core0 0 0 >>> irq311: mlx4_core0 0 0 >>> irq312: mlx4_core0 0 0 >>> irq313: mlx4_core0 0 0 >>> irq314: mlx4_core0 0 0 >>> irq315: mlx4_core0 0 0 >>> irq316: mlx4_core0 0 0 >>> irq317: mlx4_core0 0 0 >>> irq318: mlx4_core0 0 0 >>> irq319: mlx4_core0 0 0 >>> irq320: mlx4_core0 0 0 >>> irq321: mlx4_core0 0 0 >>> irq322: mlx4_core0 0 0 >>> irq323: mlx4_core0 0 0 >>> irq324: mlx4_core0 0 0 >>> irq325: mlx4_core0 0 0 >>> irq326: mlx4_core0 0 0 >>> irq327: mlx4_core0 0 0 >>> irq328: mlx4_core0 0 0 >>> irq329: mlx4_core0 0 0 >>> irq330: mlx4_core0 0 0 >>> irq331: mlx4_core0 0 0 >>> irq332: mlx4_core0 0 0 >>> irq333: mlx4_core0 0 0 >>> irq334: mlx4_core0 0 0 >>> irq335: mlx4_core0 0 0 >>> irq336: mlx4_core0 0 0 >>> irq337: mlx4_core0 0 0 >>> irq338: mlx4_core0 0 0 >>> irq339: mlx4_core0 426 0 >>> Total 3076439 341 >>=20 >> 64 interrupts, wow! Well, that explains why you hit this bug then. = I'll=20 >> commit the fix. >>=20 >>>=20 >>> On May 22, 2012, at 1:42 PM, John Baldwin wrote: >>>=20 >>>> On Tuesday, May 22, 2012 2:48:52 pm Olivier Cinquin wrote: >>>>> Thanks, that seems to have fixed the problem! Will this patch make = it=20 >> into=20 >>>> the next release? >>>>> I have no idea how many interrupts my card has. I'm happy to find = out if=20 >> you=20 >>>> let me know how, if that can help you in any way. >>>>> Should I expect everything to work fine now? I take it the card is=20= >>>> recognized since ib0 is attached to mlx4_0 port 1 >>>>>=20 >>>>> mlx4_core0: mem = 0xdfe00000-0xdfefffff,0xdc800000-0xdcffffff=20 >> irq=20 >>>> 36 at device 0.0 on pci3 >>>>> mlx4_core: Mellanox ConnectX core driver v1.0-ofed1.5.2 (August 4, = 2010) >>>>> mlx4_core: Initializing mlx4_core >>>>> vboxdrv: fAsync=3D0 offMin=3D0x123c offMax=3D0xec01 >>>>> vboxnet0: Ethernet address: 0a:00:27:00:00:00 >>>>> mlx4_en: Mellanox ConnectX HCA Ethernet driver v1.5.2 (July 2010) >>>>> mlx4_ib: Mellanox ConnectX InfiniBand driver v1.0-ofed1.5.2 = (August 4,=20 >> 2010) >>>>> ib0: max_srq_sge=3D31 >>>>> ib0: max_cm_mtu =3D 0x10000, num_frags=3D16 >>>>> ib0: Attached to mlx4_0 port 1 >>>>>=20 >>>>>=20 >>>>> (I need to get cables before I can test connectivity between = different=20 >>>> machines). >>>>>=20 >>>>> Thanks again for your help! >>>>=20 >>>> Very interesting! Can you get the output of 'vmstat -ai | grep -v = stray'? >>>>=20 >>>> --=20 >>>> John Baldwin >>>>=20 >>>=20 >>>=20 >>=20 >> --=20 >> John Baldwin >>=20 >=20 >=20 >=20 From owner-freebsd-infiniband@FreeBSD.ORG Tue Dec 4 01:21:49 2012 Return-Path: Delivered-To: freebsd-infiniband@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id C87C7A12 for ; Tue, 4 Dec 2012 01:21:49 +0000 (UTC) (envelope-from daichi@ongs.co.jp) Received: from natial.ongs.co.jp (natial.ongs.co.jp [202.216.246.90]) by mx1.freebsd.org (Postfix) with ESMTP id 5CD7A8FC16 for ; Tue, 4 Dec 2012 01:21:48 +0000 (UTC) Received: from [172.20.10.3] (pw126159005169.97.tss.panda-world.ne.jp [126.159.5.169]) by natial.ongs.co.jp (Postfix) with ESMTPSA id 94BEA125422; Tue, 4 Dec 2012 10:21:41 +0900 (JST) Mime-Version: 1.0 (Mac OS X Mail 6.2 \(1499\)) Subject: Re: Infiniband experiment environment setting up From: Daichi GOTO In-Reply-To: Date: Tue, 4 Dec 2012 10:21:39 +0900 Message-Id: <55A90DB4-4417-4581-9AFB-2BF65787DF80@ongs.co.jp> References: <20121119115350.3456c580209197b4aea33d75@bsdconsulting.co.jp> <2A385CE3-72E8-4D40-8413-69E00841536D@neville-neil.com> To: Garrett Cooper X-Mailer: Apple Mail (2.1499) Content-Type: text/plain; charset=iso-2022-jp Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: freebsd-infiniband@freebsd.org, George Neville-Neil , Daichi GOTO X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 04 Dec 2012 01:21:50 -0000 IB with OFED test bed setting up is done. I set up with FreeBSD 9.1-RELEASE and it's OFED.=20 Should I use 10-current instead of? On 2012/11/21, at 4:09, Garrett Cooper wrote: > On Tue, Nov 20, 2012 at 10:59 AM, George Neville-Neil > wrote: >>=20 >> On Nov 18, 2012, at 21:53 , Daichi GOTO = wrote: >>=20 >>> Hi, >>>=20 >>> Currently, we are getting ready to test InfiniBands on FreeBSD >>> using follow devices: >>>=20 >>> CPU: Intel Xeon E3-1240v2 >>> M/B: Super Micro X9SCM >>> RAM: Kingston KVR16E11/8 (8GBx4) >>>=20 >>> IB Card: Mellanox Technologies ConnectX-2 VPI = MHQH19B-XTR >>> IB Card: Mellanox Technologies ConnectX-3 VPI = MCX353A-QCBT >>> IB Switch: Mellanox Technologies EXW-IS5022 >>> QSFP cable: MC22096-130-001 >>>=20 >>> Is there some information about above InfiniBand devices? >>=20 >> Well it's only the cards you have to worry about. I know the = ConnectX-2 works >> but have not tested the ConnectX-3. >=20 > We're getting access to ConnectX-3 cards at Isilon soon, but I'm > not sure what sourcebase we'll be testing against (our local > sourcebase, CURRENT, or CURRENT + new OFED). My team can best speak to > what we'll be doing here. >=20 >> Also, note that it depends on what you are testing. That is, = Infiniband has low >> latency at low payloads. The numbers quoted by Mellanox about low = latency depend >> on you sending one byte (yes, 1 byte). At real packet sizes = Infiniband is no faster than >> 10GbE. >=20 > I heard similar. > Thanks, > -Garrett > _______________________________________________ > freebsd-infiniband@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-infiniband > To unsubscribe, send any mail to = "freebsd-infiniband-unsubscribe@freebsd.org" -- =1B$B8eF#BgCO=1B(B =1B$BBeI=3D Delivered-To: freebsd-infiniband@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id EC26AD24; Tue, 4 Dec 2012 01:28:38 +0000 (UTC) (envelope-from daichi@ongs.co.jp) Received: from natial.ongs.co.jp (natial.ongs.co.jp [202.216.246.90]) by mx1.freebsd.org (Postfix) with ESMTP id 848868FC0C; Tue, 4 Dec 2012 01:28:38 +0000 (UTC) Received: from [172.20.10.3] (pw126159005169.97.tss.panda-world.ne.jp [126.159.5.169]) by natial.ongs.co.jp (Postfix) with ESMTPSA id 23865125425; Tue, 4 Dec 2012 10:28:37 +0900 (JST) Mime-Version: 1.0 (Mac OS X Mail 6.2 \(1499\)) Subject: Re: Infiniband device information From: Daichi GOTO In-Reply-To: Date: Tue, 4 Dec 2012 10:28:31 +0900 Message-Id: <5DC4979A-F815-4911-B470-AFA3B3E86AA4@ongs.co.jp> References: <20121119110945.ff40669f8feefc2e996df959@bsd-consulting.jp> To: Garrett Cooper X-Mailer: Apple Mail (2.1499) Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: freebsd-infiniband@freebsd.org, daichi@freebsd.org X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 04 Dec 2012 01:28:39 -0000 On 2012/11/21, at 5:09, Garrett Cooper wrote: > On Tue, Nov 20, 2012 at 11:39 AM, Garrett Cooper = wrote: >> On Sun, Nov 18, 2012 at 6:09 PM, Daichi GOTO = wrote: >>> Hi infiniband guys, thanks for IB ml :-) >>>=20 >>> InfiniBand devices are becoming more and more important >>> in Japanese HPC business environments (In Japan, word "BigData" >>> is used instead of HPC in these days, a business buzz word). >>>=20 >>> To promoto FreeBSD as the best product to use in BigData >>> businesses, we need InfiniBand devices lists, devices working >>> well on FreeBSD. >>>=20 >>> http://wiki.freebsd.org/InfiniBand >>=20 >> That should have been updated. I'll go and polish it up a bit later = on >> today with more build instructions; the instructions only mention how >> to build some of the userland pieces -- not the kernel pieces, and = the >> wiki page could be reformatted so the instructions are a bit more >> straightforward. >=20 > The wiki page has been updated a bit. > Thanks, Thanks Garrett, I think that we should say "what version is better for OFED test". FreeBSD 9 or 10-current? > -Garrett > _______________________________________________ > freebsd-infiniband@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-infiniband > To unsubscribe, send any mail to = "freebsd-infiniband-unsubscribe@freebsd.org" -- Daichi GOTO 81-42-316-7945 | daichi@ongs.co.jp | http://www.ongs.co.jp LinkedIn: http://linkedin.com/in/daichigoto From owner-freebsd-infiniband@FreeBSD.ORG Tue Dec 4 07:00:21 2012 Return-Path: Delivered-To: freebsd-infiniband@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 2722817A for ; Tue, 4 Dec 2012 07:00:21 +0000 (UTC) (envelope-from yanegomi@gmail.com) Received: from mail-da0-f54.google.com (mail-da0-f54.google.com [209.85.210.54]) by mx1.freebsd.org (Postfix) with ESMTP id E19928FC0C for ; Tue, 4 Dec 2012 07:00:20 +0000 (UTC) Received: by mail-da0-f54.google.com with SMTP id n2so1644110dad.13 for ; Mon, 03 Dec 2012 23:00:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=references:mime-version:in-reply-to:content-type :content-transfer-encoding:message-id:cc:x-mailer:from:subject:date :to; bh=aaJ4FAuACImZxgIuPUDcuEY9KF1mmEzaNpHOolvo6UY=; b=blM+ozVJcqIMa4gOhu1TfXcDJlNA+0+0znEiT5G4A80ihp60SzqBZzUKDeFpNLfTq6 zfvU2M3M7zSVegLTuXfiHffvFnvPjb/sy4y5gNWqM44gbY1XXAYyLXjvEBSTvwvXbx/u 9QpIL7quwfSbJioBG8eleZWYvy4CsKKWQqp1Kwc5ulAe7ZdzZXkr+kjKGdwctWr1L4zn Id7+t5Plo9OtJqroVsyUoirbQXwY4sOve2StszEgp4gpK///ROdFiT3Bc3RgY/aXyVqC 1gbNI8BACFFvH8RB/vDeKId7jooS9YeCH3IwXmIUNTllY6C+OsIY4s/M19t7XTrQVXIR hR4w== Received: by 10.68.191.200 with SMTP id ha8mr36027847pbc.51.1354604420520; Mon, 03 Dec 2012 23:00:20 -0800 (PST) Received: from [10.32.150.107] (mobile-166-147-095-108.mycingular.net. [166.147.95.108]) by mx.google.com with ESMTPS id o1sm304196paw.0.2012.12.03.23.00.17 (version=SSLv3 cipher=OTHER); Mon, 03 Dec 2012 23:00:18 -0800 (PST) References: <20121119115350.3456c580209197b4aea33d75@bsdconsulting.co.jp> <2A385CE3-72E8-4D40-8413-69E00841536D@neville-neil.com> <55A90DB4-4417-4581-9AFB-2BF65787DF80@ongs.co.jp> Mime-Version: 1.0 (1.0) In-Reply-To: <55A90DB4-4417-4581-9AFB-2BF65787DF80@ongs.co.jp> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Message-Id: <5746E73D-DF8B-4637-8819-E283C1933B2E@gmail.com> X-Mailer: iPhone Mail (10A523) From: Garrett Cooper Subject: Re: Infiniband experiment environment setting up Date: Mon, 3 Dec 2012 23:00:14 -0800 To: Daichi GOTO Cc: "freebsd-infiniband@freebsd.org" , George Neville-Neil , Daichi GOTO X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 04 Dec 2012 07:00:21 -0000 On Dec 3, 2012, at 5:21 PM, Daichi GOTO wrote: > IB with OFED test bed setting up is done. > I set up with FreeBSD 9.1-RELEASE and it's OFED.=20 >=20 > Should I use 10-current instead of? Don't think it would matter because ofed is the same in 9.1 and 10, minus so= me bugfixes that dim committed. I got some connectx2 cards, so if I can conjure up some cables and have time= I can run some tests to compare results. Thanks! -Garrett= From owner-freebsd-infiniband@FreeBSD.ORG Tue Dec 4 07:14:54 2012 Return-Path: Delivered-To: freebsd-infiniband@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 4B398487 for ; Tue, 4 Dec 2012 07:14:54 +0000 (UTC) (envelope-from daichi@ongs.co.jp) Received: from natial.ongs.co.jp (natial.ongs.co.jp [202.216.246.90]) by mx1.freebsd.org (Postfix) with ESMTP id DE4548FC0C for ; Tue, 4 Dec 2012 07:14:53 +0000 (UTC) Received: from [172.20.10.3] (pw126159005169.97.tss.panda-world.ne.jp [126.159.5.169]) by natial.ongs.co.jp (Postfix) with ESMTPSA id 0E12C125424; Tue, 4 Dec 2012 16:14:51 +0900 (JST) Mime-Version: 1.0 (Mac OS X Mail 6.2 \(1499\)) Subject: Re: Infiniband experiment environment setting up From: Daichi GOTO In-Reply-To: <5746E73D-DF8B-4637-8819-E283C1933B2E@gmail.com> Date: Tue, 4 Dec 2012 16:14:52 +0900 Message-Id: <538FF79F-8236-4513-9EFC-A38F02E55A17@ongs.co.jp> References: <20121119115350.3456c580209197b4aea33d75@bsdconsulting.co.jp> <2A385CE3-72E8-4D40-8413-69E00841536D@neville-neil.com> <55A90DB4-4417-4581-9AFB-2BF65787DF80@ongs.co.jp> <5746E73D-DF8B-4637-8819-E283C1933B2E@gmail.com> To: Garrett Cooper X-Mailer: Apple Mail (2.1499) Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: "freebsd-infiniband@freebsd.org" , George Neville-Neil , Daichi GOTO X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 04 Dec 2012 07:14:54 -0000 On 2012/12/04, at 16:00, Garrett Cooper wrote: > On Dec 3, 2012, at 5:21 PM, Daichi GOTO wrote: >=20 >> IB with OFED test bed setting up is done. >> I set up with FreeBSD 9.1-RELEASE and it's OFED.=20 >>=20 >> Should I use 10-current instead of? >=20 > Don't think it would matter because ofed is the same in 9.1 and 10, = minus some bugfixes that dim committed. Roger. > I got some connectx2 cards, so if I can conjure up some cables and = have time I can run some tests to compare results. Great :) > Thanks! > -Garrett -- Daichi GOTO Board and CTO | BSD Consulting Inc. 3F SS build., 3-4-2 Nishishinbashi, Minato-ku, Tokyo 81-3-6721-5777 | daichi@bsdconsulting.co.jp From owner-freebsd-infiniband@FreeBSD.ORG Thu Dec 6 10:49:27 2012 Return-Path: Delivered-To: infiniband@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 266F7BAE for ; Thu, 6 Dec 2012 10:49:27 +0000 (UTC) (envelope-from daichi@bsd-consulting.jp) Received: from natial.ongs.co.jp (natial.ongs.co.jp [202.216.246.90]) by mx1.freebsd.org (Postfix) with ESMTP id 41AB68FC0C for ; Thu, 6 Dec 2012 10:49:25 +0000 (UTC) Received: from parancell.ongs.co.jp (dullmdaler.ongs.co.jp [202.216.246.94]) by natial.ongs.co.jp (Postfix) with ESMTPSA id 874F5125424; Thu, 6 Dec 2012 19:49:24 +0900 (JST) Date: Thu, 6 Dec 2012 19:49:21 +0900 From: Daichi GOTO To: infiniband@freebsd.org Subject: IB test report : 20121206 Message-Id: <20121206194921.12f07e3d889c9d8cf5a44967@bsd-consulting.jp> Organization: BSD Consulting Inc. X-Mailer: Sylpheed 3.3.0 (GTK+ 2.24.6; amd64-portbld-freebsd9.1) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Cc: Masanori OZAWA , hrs@allbsd.org X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 06 Dec 2012 10:49:27 -0000 Hi guys, ================================================================ IB test report. OS and IB cards: FreeBSD9-stable r243935 MHQH19B-XTR (ConnectX2) MCX353A-QCBT (ConnectX2) ================================================================ TEST1 IBoIP UD (Unreliable Datagram) Kernel configuration include GENERIC ident INFINIBAND options OFED # Infiniband protocol options SDP # Sockets Direct Protocol for infiniband device mlx4ib # ConnectX Infiniband support device mlxen # ConnectX Ethernet support device mthca # Infinihost cards device ipoib # IP over IB devices make.conf WITH_OFED= YES rc.conf (server side) opensm_enable="YES" IP setting ifconfig ib0 mtu 2044 ifconfig ib0 inet 192.168.2.x netmask 0xffffff00 server: iperf bench # iperf -s ------------------------------------------------------------ Server listening on TCP port 5001 TCP window size: 64.0 KByte (default) ------------------------------------------------------------ [ 4] local 192.168.2.1 port 5001 connected with 192.168.2.2 port 12551 [ ID] Interval Transfer Bandwidth [ 4] 0.0-10.0 sec 6.00 GBytes 5.15 Gbits/sec client: iperf bench # iperf -c 192.168.2.1 -i 1 ------------------------------------------------------------ Client connecting to 192.168.2.1, TCP port 5001 TCP window size: 33.1 KByte (default) ------------------------------------------------------------ [ 3] local 192.168.2.2 port 12551 connected with 192.168.2.1 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0- 1.0 sec 47.2 MBytes 396 Mbits/sec [ 3] 1.0- 2.0 sec 49.0 MBytes 411 Mbits/sec [ 3] 2.0- 3.0 sec 260 MBytes 2.18 Gbits/sec [ 3] 3.0- 4.0 sec 880 MBytes 7.39 Gbits/sec [ 3] 4.0- 5.0 sec 926 MBytes 7.77 Gbits/sec [ 3] 5.0- 6.0 sec 879 MBytes 7.38 Gbits/sec [ 3] 6.0- 7.0 sec 464 MBytes 3.90 Gbits/sec [ 3] 7.0- 8.0 sec 947 MBytes 7.95 Gbits/sec [ 3] 8.0- 9.0 sec 945 MBytes 7.93 Gbits/sec [ 3] 9.0-10.0 sec 742 MBytes 6.22 Gbits/sec [ 3] 0.0-10.0 sec 6.00 GBytes 5.15 Gbits/sec result: stable transmission. average 800MB/sec-1000MB/sec mtu 2044 is max. ================================================================ TEST2 IBoIP CM Kernel configuration include GENERIC ident INFINIBAND options OFED # Infiniband protocol options SDP # Sockets Direct Protocol for infiniband options IPOIB_CM#Use connect mode ipoib device mlx4ib # ConnectX Infiniband support device mlxen # ConnectX Ethernet support device mthca # Infinihost cards device ipoib # IP over IB devices others the same result unstable. terrible. average 0MB/sec-1000MB/sec ================================================================ Consideration from this first experimetns For OFED1.5.x of FreeBSD and ConnectX2 cards, Unreliable Datagram mode IB over IP looks the only way to use. CM is very unstable. It's odd. Needs improvement. (We must improve around mtu code? 2044 is dinky) I'm wondering that how many developers are using ConnectX2 cards with CM OFED kernel. ================================================================ -- Daichi GOTO Director | BSD Consulting Inc. 3F SS bldg. 3-4-2, Nishishinbashi, Minato, Tokyo 81-80-4174-6161 | daichi@bsd-consulting.jp From owner-freebsd-infiniband@FreeBSD.ORG Thu Dec 6 10:56:37 2012 Return-Path: Delivered-To: infiniband@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 13F90C91 for ; Thu, 6 Dec 2012 10:56:37 +0000 (UTC) (envelope-from daichi@bsd-consulting.jp) Received: from natial.ongs.co.jp (natial.ongs.co.jp [202.216.246.90]) by mx1.freebsd.org (Postfix) with ESMTP id D2B208FC08 for ; Thu, 6 Dec 2012 10:56:35 +0000 (UTC) Received: from parancell.ongs.co.jp (dullmdaler.ongs.co.jp [202.216.246.94]) by natial.ongs.co.jp (Postfix) with ESMTPSA id 3B428125422 for ; Thu, 6 Dec 2012 19:56:35 +0900 (JST) Date: Thu, 6 Dec 2012 19:56:32 +0900 From: Daichi GOTO To: infiniband@freebsd.org Subject: IB not-over IP transmission Message-Id: <20121206195632.5b9cd732e5cef5ba43c1e836@bsd-consulting.jp> Organization: BSD Consulting Inc. X-Mailer: Sylpheed 3.3.0 (GTK+ 2.24.6; amd64-portbld-freebsd9.1) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 06 Dec 2012 10:56:37 -0000 Hi guys, Ultimately, at long last, to achieve fastest data transmission, we should use IB not-over IP. There is no public information around FreeBSD IB and IB not-over-IP transmission. Anyone has any ideas? -- Daichi GOTO Director | BSD Consulting Inc. 3F SS bldg. 3-4-2, Nishishinbashi, Minato, Tokyo 81-80-4174-6161 | daichi@bsd-consulting.jp From owner-freebsd-infiniband@FreeBSD.ORG Thu Dec 6 11:06:39 2012 Return-Path: Delivered-To: infiniband@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 97CA8EED for ; Thu, 6 Dec 2012 11:06:39 +0000 (UTC) (envelope-from daichi@bsdconsulting.co.jp) Received: from natial.ongs.co.jp (natial.ongs.co.jp [202.216.246.90]) by mx1.freebsd.org (Postfix) with ESMTP id 5C2BD8FC12 for ; Thu, 6 Dec 2012 11:06:39 +0000 (UTC) Received: from parancell.ongs.co.jp (dullmdaler.ongs.co.jp [202.216.246.94]) by natial.ongs.co.jp (Postfix) with ESMTPSA id 52EEB125422 for ; Thu, 6 Dec 2012 20:06:38 +0900 (JST) Date: Thu, 6 Dec 2012 20:06:35 +0900 From: Daichi GOTO To: infiniband@freebsd.org Subject: IP not-over IB (Re: IB not-over IP transmission) Message-Id: <20121206200635.0991056a9a6f379b448f8177@bsdconsulting.co.jp> In-Reply-To: <20121206195632.5b9cd732e5cef5ba43c1e836@bsd-consulting.jp> References: <20121206195632.5b9cd732e5cef5ba43c1e836@bsd-consulting.jp> Organization: BSD Consulting Inc. X-Mailer: Sylpheed 3.3.0 (GTK+ 2.24.6; amd64-portbld-freebsd9.1) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 06 Dec 2012 11:06:39 -0000 ya, s/IBoIP/IPoIB/g On Thu, 6 Dec 2012 19:56:32 +0900 Daichi GOTO wrote: > Hi guys, > > Ultimately, at long last, to achieve fastest data transmission, > we should use IB not-over IP. There is no public information > around FreeBSD IB and IB not-over-IP transmission. > > Anyone has any ideas? > > -- > Daichi GOTO > Director | BSD Consulting Inc. > 3F SS bldg. 3-4-2, Nishishinbashi, Minato, Tokyo > 81-80-4174-6161 | daichi@bsd-consulting.jp > _______________________________________________ > freebsd-infiniband@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-infiniband > To unsubscribe, send any mail to "freebsd-infiniband-unsubscribe@freebsd.org" -- Daichi GOTO Director | BSD Consulting Inc. 3F SS bldg. 3-4-2, Nishishinbashi, Minato, Tokyo 81-80-4174-6161 | daichi@bsdconsulting.co.jp From owner-freebsd-infiniband@FreeBSD.ORG Fri Dec 7 04:07:40 2012 Return-Path: Delivered-To: freebsd-infiniband@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 8604E14A for ; Fri, 7 Dec 2012 04:07:40 +0000 (UTC) (envelope-from accornehl@gmail.com) Received: from mail-la0-f54.google.com (mail-la0-f54.google.com [209.85.215.54]) by mx1.freebsd.org (Postfix) with ESMTP id EE30A8FC13 for ; Fri, 7 Dec 2012 04:07:39 +0000 (UTC) Received: by mail-la0-f54.google.com with SMTP id j13so56391lah.13 for ; Thu, 06 Dec 2012 20:07:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=DKF89rMPqAexF/LZ/hXRrEb7HbB+FzYO0JgUc6jDSos=; b=Oej1Qmi+/cyEhgp2cTjdo+WR9AUK5sj8WxMR+L2DTXlpL+IPy0ee+54k2VIDEFmD2f 7LW72jXrNRUFvxEHfo7IBF0OgZZ2j3v/PuvQpnNG77IGpTu5DXHUWZuXdlCwVI4VZong vMokTMO1MOyFuUqwcEwuzhQCvS43hyW02dJrwxj3OduXM1R9q1UgZXAjeGbKMMxHlKZC G2OEtXWL+ognFaBgWhCrq/BEHgN7KMUeYt3DKYewzFWLG+fyzQ3IDGwIujG+xyts/lLl PRK3sPPPA8b565HkBDtAn0aJprwwK8hSYJOR91ad37gjEv0/oZO+sPB7wx7UGakLmlNi q6DA== MIME-Version: 1.0 Received: by 10.112.86.232 with SMTP id s8mr1944418lbz.86.1354853253084; Thu, 06 Dec 2012 20:07:33 -0800 (PST) Received: by 10.114.9.73 with HTTP; Thu, 6 Dec 2012 20:07:32 -0800 (PST) Received: by 10.114.9.73 with HTTP; Thu, 6 Dec 2012 20:07:32 -0800 (PST) In-Reply-To: References: <3A359B33-380C-4230-A62C-623765E9376A@jnielsen.net> <53A5EDFD-B2C5-4E24-9AB9-5504AA693602@jnielsen.net> Date: Thu, 6 Dec 2012 20:07:32 -0800 Message-ID: Subject: Re: using ConnectX card as Ethernet (mlxen) From: Anthony Cornehl To: yanegomi@gmail.com Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: freebsd-infiniband@freebsd.org X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 07 Dec 2012 04:07:40 -0000 There is an issue with the sysfs/sysctl abstraction for storing values. Whups. http://www.freebsd.org/cgi/query-pr.cgi?pr=174213 On Nov 14, 2012 3:53 PM, "Garrett Cooper" wrote: > On Wed, Nov 14, 2012 at 3:32 PM, John Nielsen wrote: > > > On Nov 9, 2012, at 9:51 PM, Garrett Cooper wrote: > > > > > On Fri, Nov 9, 2012 at 7:44 PM, John Nielsen > wrote: > > > On the off chance there's someone on this list already that didn't see > > my earlier (unanswered) question, I'm reposting it below. I don't have > > FreeBSD on this hardware any more but I would like to know for future > > reference if using these cards with an Ethernet switch is an option. > > > > > > I know this might seem dumb, but what if you did > > sys.device.mlx4_core0.mlx4_port1="`printf 'en\n'`" (the newline's > > important). > > > > I'll try it next time I have an opportunity. Can you explain why the > > newline might make a difference? Also, I don't think I ever tried 'en', > > just 'eth'. Can someone say what the right value is or should be? > > > > It was a WAG based on this section of code (from > sys/ofed/drivers/net/mlx4/main.c)... > > 469 static ssize_t set_port_type(struct device *dev, > 470 struct device_attribute *attr, > 471 const char *buf, size_t count) > 472 { > 473 struct mlx4_port_info *info = container_of(attr, struct > mlx4_port_i nfo, > 474 port_attr); > 475 struct mlx4_dev *mdev = info->dev; > 476 struct mlx4_priv *priv = mlx4_priv(mdev); > 477 enum mlx4_port_type types[MLX4_MAX_PORTS]; > 478 enum mlx4_port_type new_types[MLX4_MAX_PORTS]; > 479 int i; > 480 int err = 0; > 481 > 482 if (!strcmp(buf, "ib\n")) > 483 info->tmp_type = MLX4_PORT_TYPE_IB; > 484 else if (!strcmp(buf, "eth\n")) > 485 info->tmp_type = MLX4_PORT_TYPE_ETH; > 486 else if (!strcmp(buf, "auto\n")) > 487 info->tmp_type = MLX4_PORT_TYPE_AUTO; > 488 else { > 489 mlx4_err(mdev, "%s is not supported port type\n", > buf); > 490 return -EINVAL; > 491 } > > I don't have one of these cards so I can't verify whether or not that's the > issue today, but it seemed like a likely cause. > > This being there is probably the remnant of a pseudo-valid hacky assumption > on Linux (you're echo'ing to a sysfs node and it's tacking on a newline), > but an invalid assumption on FreeBSD (the sysctl<->sysfs compat layer > doesn't make any translation assumptions w.r.t. newlines, etc). > > Thanks! > -Garrett > _______________________________________________ > freebsd-infiniband@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-infiniband > To unsubscribe, send any mail to " > freebsd-infiniband-unsubscribe@freebsd.org" >