Date: Thu, 8 Dec 2016 00:33:16 +0000 (GMT) From: Mellanox Support Admin <supportadmin@mellanox.com> To: "freebsd-infiniband@freebsd.org" <freebsd-infiniband@freebsd.org> Subject: Mellanox Case # 00320058 : RE: Slow speed with Mellanox IS5023 Switch and Connect-X3 cards was updated [ref:_00D507heg._50050q9Npg:ref ] Message-ID: <Lb7qJ000000000000000000000000000000000000000000000OHUC7D00oXC5fjVBQaK8BW_j4tGt8A@sfdc.net>
next in thread | raw e-mail | index | archive | help
Dear Vin=C3=ADcius Ferr=C3=A3o,=20 Your Mellanox support case # 00320058/ has been updated By , with the fol= lowing comment:=20 /------------------------------------------------------------- Comment By Customer: ferrao@vmio.com.br Hello Sophie, Recompiling the software isn=E2=80=99t an viable option. I was able to mitigate the issue on this specific case raising the MTU to 6= 5520 on all machines. It turns out that the old machines were configured on= Connected Mode using only 2044 MTU and an MTU mismatch between the server = and the clients was the real issue. But I=E2=80=99m worried about the viability of Infiniband on FreeBSD. Chang= ing from Connected Mode to Unreliable Datagram should be easier than recomp= iling the kernel module. There=E2=80=99s a way to ask Mellanox to change the drivers to make the swi= tch from UD to CM viable on FreeBSD? A feature request perhaps? Thanks in advance, V. > On 7 Dec 2016, at 19:05, Mellanox Support Admin <supportadmin@mellanox.co= m> -------------------------------------------------------------/ Thank You,=20 =20 Click on the link to access the case:=20 https://mymellanox.force.com/support/5005000000q9Npg Don=E2=80=99t forget to check out our Online Academy, register now to learn= about the products and technology, http://academy.mellanox.com/login/index= .php=20 /------ Your Case Comments History ------=20 ------------------------------------------------- 2016-12-08 00:33:13 Comment By Customer: ferrao@vmio.com.br Hello Sophie, Recompiling the software isn=E2=80=99t an viable option. I was able to mitigate the issue on this specific case raising the MTU to 6= 5520 on all machines. It turns out that the old machines were configured on= Connected Mode using only 2044 MTU and an MTU mismatch between the server = and the clients was the real issue. But I=E2=80=99m worried about the viability of Infiniband on FreeBSD. Chang= ing from Connected Mode to Unreliable Datagram should be easier than recomp= iling the kernel module. There=E2=80=99s a way to ask Mellanox to change the drivers to make the swi= tch from UD to CM viable on FreeBSD? A feature request perhaps? Thanks in advance, V. > On 7 Dec 2016, at 19:05, Mellanox Support Admin <supportadmin@mellanox.co= m> ------------------------------------------------- Created by: Sophie Naudin Pruitt 2016-12-07 21:05:15 Hi Vinicius, Have you seen my last statement posted to this case and what is the current= status? Please let me know. Thank you, Sophie. ------------------------------------------------- Created by: Sophie Naudin Pruitt 2016-12-05 23:02:35 Hi Vinicius, Searching some forums, I found the following information: Building OFED in FreeBSD Kernel =E2=80=A2Add the following options to your KERNCONF to compile the necessar= y protocols/protocol options into your kernel:=20 options OFED # Infiniband protocol stack and support options SDP # Sockets Direct Protocol for infiniband options IPOIB_CM # Use connect mode ipoib >>>recompile withou= t that option and the mode should be datagram. Regards, Sophie. ------------------------------------------------- Created by: Sophie Naudin Pruitt 2016-12-05 22:53:06 Hi Vin=C3=ADcius, You are not using Mellanox OFED Driver if you are running OFED (OpenFabrics= Enterprise Distribution), what is the version and did you consult the UM? With our driver, you can change the mode with: echo datagram > /sys/class/net/ib0/mode sets the mode of ib0 to UD echo connected > /sys/class/net/ib0/mode sets the mode ib0 to CM Though FreeBSD does not have a /sys. Regards, Sophie. ------------------------------------------------- Created by: Anderson Dias Silva 2016-12-05 20:29:35 Hi Benny, I am Anderson Dias Silva (LATAM SE). Please, see below an important conside= ration of our customer. ########################################### After some testing we discovered that was a mismatch between the client and= the server on operating modes of IPoIB. One side was using Connected Mode = (the FreeBSD one) and the Linux client was on Datagram mode). So I=E2=80=99m considering running FreeBSD in UD (Datagram) mode to see if = I finally can get rid of the problems. But, how can I change between CM and= UD in FreeBSD? We don't have a /sys folder on FreeBSD so I cant just set f= rom 0 to 1 and change modes. ########################################### So, how can we change between CM and UD in FreeBSD? Ragards! ------------------------------------------------- Created by: Benny Filin 2016-12-05 01:03:55 Hello Vin=C3=ADcius Ferr=C3=A3o, Thank you for contacting Mellanox Technical Support. 1) DId you try changing switch ports when testing performance between two F= reeBSD machines through switch? 2) What happens when you try a back-to-back connection between two FreeBSD = machines? Do you get good results? 3) Did you test the performance between FreeBSD and a Linux machine through= switch? Please, follow the provided procedure in order to query your unmanaged swit= ch:=20 =20 1) Download and install MFT (Mellanox Firmware Tool): http://www.mellanox.com/content/pages.php?pg=3Dmanagement_tools&menu_sectio= n=3D34 Refer to the User Manual for the installation instructions: http://www.mellanox.com/pdf/MFT/MFT_user_manual.pdf =20 2) From one of the nodes in the fabric, invoke: ibswitches. Match the relevant switch GUID from the Mellanox switch label to the LID. =20 3) From one of the nodes in the fabric , invoke: mst start -> mst ib add ->= mst status =20 4) Identify the relevant unmanaged switch mst representation based on the s= witch LID (from previous step 2). * Matching the remote switch mst device representation to the relevant swit= ch procedure: Locate the switch in the "ibswitches" output and retrieve the switch LID.= =20 Convert the LID from Decimal to Hexadecimal base. Locate the device with this LID (in hex base) from the "mst status" output.= =20 5) Invoke the following command, and provide the output: flint -d <remote_switch_mst_device> q We're looking forward to hearing from you. Best regards, Mellanox Technical Support ------------------------------------------------- Created by: Jubran Jubran 2016-12-01 16:41:35 Hello Vin=C3=ADcius Ferr=C3=A3o, Thank you for contacting Mellanox Technical Support. 1) DId you try changing switch ports when testing performance between two F= reeBSD machines through switch? 2) What happens when you try a back-to-back connection between two FreeBSD = machines? Do you get good results? 3) Did you test the performance between FreeBSD and a Linux machine through= switch? Please, follow the provided procedure in order to query your unmanaged swit= ch:=20 =C2=A0 1) Download and install MFT (Mellanox Firmware Tool): http://www.mellanox.com/content/pages.php?pg=3Dmanagement_tools&menu_sectio= n=3D34 Refer to the User Manual for the installation instructions: http://www.mellanox.com/pdf/MFT/MFT_user_manual.pdf =C2=A0 2) From one of the nodes in the fabric, invoke: ibswitches. Match the relevant switch GUID from the Mellanox switch label to the LID. =C2=A0 3) From one of the nodes in the fabric , invoke: mst start -> mst ib add ->= mst status =C2=A0 4) Identify the relevant unmanaged switch mst representation based on the s= witch LID (from previous step 2). * Matching the remote switch mst device representation to the relevant swit= ch procedure: Locate the switch in the "ibswitches" output and retrieve the switch LID.= =20 Convert the LID from Decimal to Hexadecimal base. Locate the device with this LID (in hex base) from the "mst status" output.= =C2=A0 5) Invoke the following command, and provide the output: flint -d <remote_switch_mst_device> q We're looking forward to hearing from you. Best regards, Mellanox Technical Support=20 -------------------------------------------------------/
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?Lb7qJ000000000000000000000000000000000000000000000OHUC7D00oXC5fjVBQaK8BW_j4tGt8A>