From owner-freebsd-infiniband@freebsd.org Fri Dec 9 13:32:06 2016 Return-Path: Delivered-To: freebsd-infiniband@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 45C07C6E488 for ; Fri, 9 Dec 2016 13:32:06 +0000 (UTC) (envelope-from supportadmin@mellanox.com) Received: from smtp14-phx-sp1.mta.salesforce.com (smtp14-phx-sp1.mta.salesforce.com [136.147.46.237]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 27373133 for ; Fri, 9 Dec 2016 13:32:05 +0000 (UTC) (envelope-from supportadmin@mellanox.com) Received: from [10.246.216.26] ([10.246.216.26:35574] helo=na3-app2-8-phx.ops.sfdc.net) by mx4-phx-sp1.mta.salesforce.com (envelope-from ) (ecelerity 3.6.25.56547 r(Core:3.6.25.0)) with ESMTPS (cipher=DES-CBC3-SHA) id 76/F7-07904-321BA485; Fri, 09 Dec 2016 13:26:59 +0000 Date: Fri, 9 Dec 2016 13:26:59 +0000 (GMT) From: Mellanox Support Admin To: "freebsd-infiniband@freebsd.org" Message-ID: Subject: Mellanox Case # 00320058 : RE: Slow speed with Mellanox IS5023 Switch and Connect-X3 cards was updated [ref:_00D507heg._50050q9Npg:ref ] MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-SFDC-LK: 00D500000007heg X-SFDC-User: 005500000013ipV X-Sender: postmaster@salesforce.com X-mail_abuse_inquiries: http://www.salesforce.com/company/abuse.jsp X-SFDC-TLS-NoRelay: 1 X-SFDC-Binding: 1WrIRBV94myi25uB X-SFDC-EmailCategory: workflowActionAlert X-SFDC-EntityId: 01W500000004xvX X-SFDC-Interface: internal X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 09 Dec 2016 13:32:06 -0000 Dear Vin=C3=ADcius Ferr=C3=A3o,=20 Your Mellanox support case # 00320058/ has been updated By , with the fol= lowing comment:=20 /------------------------------------------------------------- Comment By Customer: denis@versatushpc.com.br it's FreeBSD 10.3/Linux CentOS 7.2 Environment. D. 2016-12-08 22:22 GMT-02:00 Mellanox Support Admin : > Dear Vin=C3=ADcius Ferr=C3=A3o, > > Mellanox support case # 00320058 has been updated By , with the following > comment: > /------------------------------------------------------------- > Hi Vinicius, > > Our Mellanox OFED Driver currently only supports FreeBSD 10.1 and we don'= t > have any indication of future release at this time supporting later versi= on > of this OS. > Keep in mind that you are using the software stack released as OFED > (OpenFabrics Enterprise Distribution) driver. > > Does this environment has servers running FreeBSD only or do you have > servers with different Linux flavors? > > Regards, > Sophie. > -------------------------------------------------------------/ > > Thank You, > > > > You can access all your closed/Open cases, just go to > http://mellanox.com/content/pages.php?pg=3Dcustomer_support > > Click on the link to access the case: > > https://mymellanox.force.com/support/5005000000q9Npg > > > Don=E2=80=99t forget to check out our Online Academy, register now to lea= rn about > the products and technology, http://academy.mellanox.com/login/index.php > /------ Your Case Comments History ------ > ------------------------------------------------- > Created by: Sophie Naudin Pruitt 2016-12-09 00:22:26 > Hi Vinicius, > > Our Mellanox OFED Driver currently only supports FreeBSD 10.1 and we don'= t > have any indication of future release at this time supporting later versi= on > of this OS. > Keep in mind that you are using the software stack released as OFED > (OpenFabrics Enterprise Distribution) driver. > > Does this environment has servers running FreeBSD only or do you have > servers with different Linux flavors? > > Regards, > Sophie. > > ------------------------------------------------- > 2016-12-08 00:33:13 > Comment By Customer: ferrao@vmio.com.br > Hello Sophie, > > Recompiling the software isn=E2=80=99t an viable option. > > I was able to mitigate the issue on this specific case raising the MTU to > 65520 on all machines. It turns out that the old machines were configured > on Connected Mode using only 2044 MTU and an MTU mismatch between the > server and the clients was the real issue. > > But I=E2=80=99m worried about the viability of Infiniband on FreeBSD. Cha= nging > from Connected Mode to Unreliable Datagram should be easier than > recompiling the kernel module. > > There=E2=80=99s a way to ask Mellanox to change the drivers to make the s= witch > from UD to CM viable on FreeBSD? A feature request perhaps? > > Thanks in advance, > V. > > > On 7 Dec 2016, at 19:05, Mellanox Support Admin < > supportadmin@mellanox.com> > > ------------------------------------------------- > Created by: Sophie Naudin Pruitt 2016-12-07 21:05:15 > Hi Vinicius, > > Have you seen my last statement posted to this case and what is the > current status? > Please let me know. > > Thank you, > Sophie. > > ------------------------------------------------- > Created by: Sophie Naudin Pruitt 2016-12-05 23:02:35 > Hi Vinicius, > > Searching some forums, I found the following information: > > Building -------------------------------------------------------------/ Thank You,=20 =20 Click on the link to access the case:=20 https://mymellanox.force.com/support/5005000000q9Npg Don=E2=80=99t forget to check out our Online Academy, register now to learn= about the products and technology, http://academy.mellanox.com/login/index= .php=20 /------ Your Case Comments History ------=20 ------------------------------------------------- 2016-12-09 13:26:56 Comment By Customer: denis@versatushpc.com.br it's FreeBSD 10.3/Linux CentOS 7.2 Environment. D. 2016-12-08 22:22 GMT-02:00 Mellanox Support Admin : > Dear Vin=C3=ADcius Ferr=C3=A3o, > > Mellanox support case # 00320058 has been updated By , with the following > comment: > /------------------------------------------------------------- > Hi Vinicius, > > Our Mellanox OFED Driver currently only supports FreeBSD 10.1 and we don'= t > have any indication of future release at this time supporting later versi= on > of this OS. > Keep in mind that you are using the software stack released as OFED > (OpenFabrics Enterprise Distribution) driver. > > Does this environment has servers running FreeBSD only or do you have > servers with different Linux flavors? > > Regards, > Sophie. > -------------------------------------------------------------/ > > Thank You, > > > > You can access all your closed/Open cases, just go to > http://mellanox.com/content/pages.php?pg=3Dcustomer_support > > Click on the link to access the case: > > https://mymellanox.force.com/support/5005000000q9Npg > > > Don=E2=80=99t forget to check out our Online Academy, register now to lea= rn about > the products and technology, http://academy.mellanox.com/login/index.php > /------ Your Case Comments History ------ > ------------------------------------------------- > Created by: Sophie Naudin Pruitt 2016-12-09 00:22:26 > Hi Vinicius, > > Our Mellanox OFED Driver currently only supports FreeBSD 10.1 and we don'= t > have any indication of future release at this time supporting later versi= on > of this OS. > Keep in mind that you are using the software stack released as OFED > (OpenFabrics Enterprise Distribution) driver. > > Does this environment has servers running FreeBSD only or do you have > servers with different Linux flavors? > > Regards, > Sophie. > > ------------------------------------------------- > 2016-12-08 00:33:13 > Comment By Customer: ferrao@vmio.com.br > Hello Sophie, > > Recompiling the software isn=E2=80=99t an viable option. > > I was able to mitigate the issue on this specific case raising the MTU to > 65520 on all machines. It turns out that the old machines were configured > on Connected Mode using only 2044 MTU and an MTU mismatch between the > server and the clients was the real issue. > > But I=E2=80=99m worried about the viability of Infiniband on FreeBSD. Cha= nging > from Connected Mode to Unreliable Datagram should be easier than > recompiling the kernel module. > > There=E2=80=99s a way to ask Mellanox to change the drivers to make the s= witch > from UD to CM viable on FreeBSD? A feature request perhaps? > > Thanks in advance, > V. > > > On 7 Dec 2016, at 19:05, Mellanox Support Admin < > supportadmin@mellanox.com> > > ------------------------------------------------- > Created by: Sophie Naudin Pruitt 2016-12-07 21:05:15 > Hi Vinicius, > > Have you seen my last statement posted to this case and what is the > current status? > Please let me know. > > Thank you, > Sophie. > > ------------------------------------------------- > Created by: Sophie Naudin Pruitt 2016-12-05 23:02:35 > Hi Vinicius, > > Searching some forums, I found the following information: > > Building ------------------------------------------------- Created by: Sophie Naudin Pruitt 2016-12-09 00:22:26 Hi Vinicius, Our Mellanox OFED Driver currently only supports FreeBSD 10.1 and we don't = have any indication of future release at this time supporting later version= of this OS. Keep in mind that you are using the software stack released as OFED (OpenFa= brics Enterprise Distribution) driver. Does this environment has servers running FreeBSD only or do you have serve= rs with different Linux flavors? Regards, Sophie. ------------------------------------------------- 2016-12-08 00:33:13 Comment By Customer: ferrao@vmio.com.br Hello Sophie, Recompiling the software isn=E2=80=99t an viable option. I was able to mitigate the issue on this specific case raising the MTU to 6= 5520 on all machines. It turns out that the old machines were configured on= Connected Mode using only 2044 MTU and an MTU mismatch between the server = and the clients was the real issue. But I=E2=80=99m worried about the viability of Infiniband on FreeBSD. Chang= ing from Connected Mode to Unreliable Datagram should be easier than recomp= iling the kernel module. There=E2=80=99s a way to ask Mellanox to change the drivers to make the swi= tch from UD to CM viable on FreeBSD? A feature request perhaps? Thanks in advance, V. > On 7 Dec 2016, at 19:05, Mellanox Support Admin ------------------------------------------------- Created by: Sophie Naudin Pruitt 2016-12-07 21:05:15 Hi Vinicius, Have you seen my last statement posted to this case and what is the current= status? Please let me know. Thank you, Sophie. ------------------------------------------------- Created by: Sophie Naudin Pruitt 2016-12-05 23:02:35 Hi Vinicius, Searching some forums, I found the following information: Building OFED in FreeBSD Kernel =E2=80=A2Add the following options to your KERNCONF to compile the necessar= y protocols/protocol options into your kernel:=20 options OFED # Infiniband protocol stack and support options SDP # Sockets Direct Protocol for infiniband options IPOIB_CM # Use connect mode ipoib >>>recompile withou= t that option and the mode should be datagram. Regards, Sophie. ------------------------------------------------- Created by: Sophie Naudin Pruitt 2016-12-05 22:53:06 Hi Vin=C3=ADcius, You are not using Mellanox OFED Driver if you are running OFED (OpenFabrics= Enterprise Distribution), what is the version and did you consult the UM? With our driver, you can change the mode with: echo datagram > /sys/class/net/ib0/mode sets the mode of ib0 to UD echo connected > /sys/class/net/ib0/mode sets the mode ib0 to CM Though FreeBSD does not have a /sys. Regards, Sophie. ------------------------------------------------- Created by: Anderson Dias Silva 2016-12-05 20:29:35 Hi Benny, I am Anderson Dias Silva (LATAM SE). Please, see below an important conside= ration of our customer. ########################################### After some testing we discovered that was a mismatch between the client and= the server on operating modes of IPoIB. One side was using Connected Mode = (the FreeBSD one) and the Linux client was on Datagram mode). So I=E2=80=99m considering running FreeBSD in UD (Datagram) mode to see if = I finally can get rid of the problems. But, how can I change between CM and= UD in FreeBSD? We don't have a /sys folder on FreeBSD so I cant just set f= rom 0 to 1 and change modes. ########################################### So, how can we change between CM and UD in FreeBSD? Ragards! ------------------------------------------------- Created by: Benny Filin 2016-12-05 01:03:55 Hello Vin=C3=ADcius Ferr=C3=A3o, Thank you for contacting Mellanox Technical Support. 1) DId you try changing switch ports when testing performance between two F= reeBSD machines through switch? 2) What happens when you try a back-to-back connection between two FreeBSD = machines? Do you get good results? 3) Did you test the performance between FreeBSD and a Linux machine through= switch? Please, follow the provided procedure in order to query your unmanaged swit= ch:=20 =20 1) Download and install MFT (Mellanox Firmware Tool): http://www.mellanox.com/content/pages.php?pg=3Dmanagement_tools&menu_sectio= n=3D34 Refer to the User Manual for the installation instructions: http://www.mellanox.com/pdf/MFT/MFT_user_manual.pdf =20 2) From one of the nodes in the fabric, invoke: ibswitches. Match the relevant switch GUID from the Mellanox switch label to the LID. =20 3) From one of the nodes in the fabric , invoke: mst start -> mst ib add ->= mst status =20 4) Identify the relevant unmanaged switch mst representation based on the s= witch LID (from previous step 2). * Matching the remote switch mst device representation to the relevant swit= ch procedure: Locate the switch in the "ibswitches" output and retrieve the switch LID.= =20 Convert the LID from Decimal to Hexadecimal base. Locate the device with this LID (in hex base) from the "mst status" output.= =20 5) Invoke the following command, and provide the output: flint -d q We're looking forward to hearing from you. Best regards, Mellanox Technical Support ------------------------------------------------- Created by: Jubran Jubran 2016-12-01 16:41:35 Hello Vin=C3=ADcius Ferr=C3=A3o, Thank you for contacting Mellanox Technical Support. 1) DId you try changing switch ports when testing performance between two F= reeBSD machines through switch? 2) What happens when you try a back-to-back connection between two FreeBSD = machines? Do you get good results? 3) Did you test the performance between FreeBSD and a Linux machine through= switch? Please, follow the provided procedure in order to query your unmanaged swit= ch:=20 =C2=A0 1) Download and install MFT (Mellanox Firmware Tool): http://www.mellanox.com/content/pages.php?pg=3Dmanagement_tools&menu_sectio= n=3D34 Refer to the User Manual for the installation instructions: http://www.mellanox.com/pdf/MFT/MFT_user_manual.pdf =C2=A0 2) From one of the nodes in the fabric, invoke: ibswitches. Match the relevant switch GUID from the Mellanox switch label to the LID. =C2=A0 3) From one of the nodes in the fabric , invoke: mst start -> mst ib add ->= mst status =C2=A0 4) Identify the relevant unmanaged switch mst representation based on the s= witch LID (from previous step 2). * Matching the remote switch mst device representation to the relevant swit= ch procedure: Locate the switch in the "ibswitches" output and retrieve the switch LID.= =20 Convert the LID from Decimal to Hexadecimal base. Locate the device with this LID (in hex base) from the "mst status" output.= =C2=A0 5) Invoke the following command, and provide the output: flint -d q We're looking forward to hearing from you. Best regards, Mellanox Technical Support=20 -------------------------------------------------------/