From owner-freebsd-infiniband@freebsd.org Mon Dec 12 21:41:48 2016 Return-Path: Delivered-To: freebsd-infiniband@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 34383C74559 for ; Mon, 12 Dec 2016 21:41:48 +0000 (UTC) (envelope-from supportadmin@mellanox.com) Received: from smtp07-phx-sp1.mta.salesforce.com (smtp07-phx-sp1.mta.salesforce.com [136.147.46.230]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 1A8221A3E for ; Mon, 12 Dec 2016 21:41:47 +0000 (UTC) (envelope-from supportadmin@mellanox.com) Received: from [10.246.216.27] ([10.246.216.27:38364] helo=na3-app2-10-phx.ops.sfdc.net) by mx4-phx-sp1.mta.salesforce.com (envelope-from ) (ecelerity 3.6.25.56547 r(Core:3.6.25.0)) with ESMTPS (cipher=DES-CBC3-SHA) id 72/12-60673-5991F485; Mon, 12 Dec 2016 21:41:41 +0000 Date: Mon, 12 Dec 2016 21:41:41 +0000 (GMT) From: Mellanox Support Admin To: "freebsd-infiniband@freebsd.org" Message-ID: Subject: Mellanox Case # 00320058 : RE: Slow speed with Mellanox IS5023 Switch and Connect-X3 cards was updated [ref:_00D507heg._50050q9Npg:ref ] MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-SFDC-LK: 00D500000007heg X-SFDC-User: 005500000035Kpb X-Sender: postmaster@salesforce.com X-mail_abuse_inquiries: http://www.salesforce.com/company/abuse.jsp X-SFDC-TLS-NoRelay: 1 X-SFDC-Binding: 1WrIRBV94myi25uB X-SFDC-EmailCategory: workflowActionAlert X-SFDC-EntityId: 01W500000004xvX X-SFDC-Interface: internal X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 Dec 2016 21:41:48 -0000 Dear Vin=C3=ADcius Ferr=C3=A3o,=20 Your Mellanox support case # 00320058/ has been updated By , with the fol= lowing comment:=20 /------------------------------------------------------------- Hi Vinicius, As a side note, for your CentOS servers running Mellanox OFED Driver, if th= ey are configured as UD and attempt to talk to nodes configured as RC, the = default mode on both side will automatically be as UD. Regards, Sophie. -------------------------------------------------------------/ Thank You,=20 =20 Click on the link to access the case:=20 https://mymellanox.force.com/support/5005000000q9Npg Don=E2=80=99t forget to check out our Online Academy, register now to learn= about the products and technology, http://academy.mellanox.com/login/index= .php=20 /------ Your Case Comments History ------=20 ------------------------------------------------- Created by: Sophie Naudin Pruitt 2016-12-12 21:41:39 Hi Vinicius, As a side note, for your CentOS servers running Mellanox OFED Driver, if th= ey are configured as UD and attempt to talk to nodes configured as RC, the = default mode on both side will automatically be as UD. Regards, Sophie. ------------------------------------------------- 2016-12-09 22:04:18 Comment By Customer: ferrao@vmio.com.br Sophie, I think we can safely close this issue. The main problem was with the UD mo= de of IPoIB. Since we=E2=80=99ve already changed all the machines on the cl= uster for IPoIB Connected Mode and with the properly MTU settings everythin= g worked as expected. It=E2=80=99s an annoyance being unable to change from CM to UD, but we will= stay with this since we cannot have a better solution, and moving the serv= er from FreeBSD to Linux is unpractical for us. But just for the sake of completeness everything was working correctly on t= he CentOS side using the Mellanox bundled OFED. Thanks for helping us out. V. > On 9 Dec 2016, at 17:22, Mellanox Support Admin ------------------------------------------------- Created by: Sophie Naudin Pruitt 2016-12-09 19:22:48 Thank you. What is the driver in use for the RHEL environment? Mellanox OFED Driver or= OFED? What is the mode configured on the RHEL environment? Is this fabric will communicate between servers with FreeBSD & RHEL? Regards, Sophie. ------------------------------------------------- 2016-12-09 13:26:56 Comment By Customer: denis@versatushpc.com.br it's FreeBSD 10.3/Linux CentOS 7.2 Environment. D. 2016-12-08 22:22 GMT-02:00 Mellanox Support Admin : > Dear Vin=C3=ADcius Ferr=C3=A3o, > > Mellanox support case # 00320058 has been updated By , with the following > comment: > /------------------------------------------------------------- > Hi Vinicius, > > Our Mellanox OFED Driver currently only supports FreeBSD 10.1 and we don'= t > have any indication of future release at this time supporting later versi= on > of this OS. > Keep in mind that you are using the software stack released as OFED > (OpenFabrics Enterprise Distribution) driver. > > Does this environment has servers running FreeBSD only or do you have > servers with different Linux flavors? > > Regards, > Sophie. > -------------------------------------------------------------/ > > Thank You, > > > > You can access all your closed/Open cases, just go to > http://mellanox.com/content/pages.php?pg=3Dcustomer_support > > Click on the link to access the case: > > https://mymellanox.force.com/support/5005000000q9Npg > > > Don=E2=80=99t forget to check out our Online Academy, register now to lea= rn about > the products and technology, http://academy.mellanox.com/login/index.php > /------ Your Case Comments History ------ > ------------------------------------------------- > Created by: Sophie Naudin Pruitt 2016-12-09 00:22:26 > Hi Vinicius, > > Our Mellanox OFED Driver currently only supports FreeBSD 10.1 and we don'= t > have any indication of future release at this time supporting later versi= on > of this OS. > Keep in mind that you are using the software stack released as OFED > (OpenFabrics Enterprise Distribution) driver. > > Does this environment has servers running FreeBSD only or do you have > servers with different Linux flavors? > > Regards, > Sophie. > > ------------------------------------------------- > 2016-12-08 00:33:13 > Comment By Customer: ferrao@vmio.com.br > Hello Sophie, > > Recompiling the software isn=E2=80=99t an viable option. > > I was able to mitigate the issue on this specific case raising the MTU to > 65520 on all machines. It turns out that the old machines were configured > on Connected Mode using only 2044 MTU and an MTU mismatch between the > server and the clients was the real issue. > > But I=E2=80=99m worried about the viability of Infiniband on FreeBSD. Cha= nging > from Connected Mode to Unreliable Datagram should be easier than > recompiling the kernel module. > > There=E2=80=99s a way to ask Mellanox to change the drivers to make the s= witch > from UD to CM viable on FreeBSD? A feature request perhaps? > > Thanks in advance, > V. > > > On 7 Dec 2016, at 19:05, Mellanox Support Admin < > supportadmin@mellanox.com> > > ------------------------------------------------- > Created by: Sophie Naudin Pruitt 2016-12-07 21:05:15 > Hi Vinicius, > > Have you seen my last statement posted to this case and what is the > current status? > Please let me know. > > Thank you, > Sophie. > > ------------------------------------------------- > Created by: Sophie Naudin Pruitt 2016-12-05 23:02:35 > Hi Vinicius, > > Searching some forums, I found the following information: > > Building ------------------------------------------------- Created by: Sophie Naudin Pruitt 2016-12-09 00:22:26 Hi Vinicius, Our Mellanox OFED Driver currently only supports FreeBSD 10.1 and we don't = have any indication of future release at this time supporting later version= of this OS. Keep in mind that you are using the software stack released as OFED (OpenFa= brics Enterprise Distribution) driver. Does this environment has servers running FreeBSD only or do you have serve= rs with different Linux flavors? Regards, Sophie. ------------------------------------------------- 2016-12-08 00:33:13 Comment By Customer: ferrao@vmio.com.br Hello Sophie, Recompiling the software isn=E2=80=99t an viable option. I was able to mitigate the issue on this specific case raising the MTU to 6= 5520 on all machines. It turns out that the old machines were configured on= Connected Mode using only 2044 MTU and an MTU mismatch between the server = and the clients was the real issue. But I=E2=80=99m worried about the viability of Infiniband on FreeBSD. Chang= ing from Connected Mode to Unreliable Datagram should be easier than recomp= iling the kernel module. There=E2=80=99s a way to ask Mellanox to change the drivers to make the swi= tch from UD to CM viable on FreeBSD? A feature request perhaps? Thanks in advance, V. > On 7 Dec 2016, at 19:05, Mellanox Support Admin ------------------------------------------------- Created by: Sophie Naudin Pruitt 2016-12-07 21:05:15 Hi Vinicius, Have you seen my last statement posted to this case and what is the current= status? Please let me know. Thank you, Sophie. ------------------------------------------------- Created by: Sophie Naudin Pruitt 2016-12-05 23:02:35 Hi Vinicius, Searching some forums, I found the following information: Building OFED in FreeBSD Kernel =E2=80=A2Add the following options to your KERNCONF to compile the necessar= y protocols/protocol options into your kernel:=20 options OFED # Infiniband protocol stack and support options SDP # Sockets Direct Protocol for infiniband options IPOIB_CM # Use connect mode ipoib >>>recompile withou= t that option and the mode should be datagram. Regards, Sophie. ------------------------------------------------- Created by: Sophie Naudin Pruitt 2016-12-05 22:53:06 Hi Vin=C3=ADcius, You are not using Mellanox OFED Driver if you are running OFED (OpenFabrics= Enterprise Distribution), what is the version and did you consult the UM? With our driver, you can change the mode with: echo datagram > /sys/class/net/ib0/mode sets the mode of ib0 to UD echo connected > /sys/class/net/ib0/mode sets the mode ib0 to CM Though FreeBSD does not have a /sys. Regards, Sophie. ------------------------------------------------- Created by: Anderson Dias Silva 2016-12-05 20:29:35 Hi Benny, I am Anderson Dias Silva (LATAM SE). Please, see below an important conside= ration of our customer. ########################################### After some testing we discovered that was a mismatch between the client and= the server on operating modes of IPoIB. One side was using Connected Mode = (the FreeBSD one) and the Linux client was on Datagram mode). So I=E2=80=99m considering running FreeBSD in UD (Datagram) mode to see if = I finally can get rid of the problems. But, how can I change between CM and= UD in FreeBSD? We don't have a /sys folder on FreeBSD so I cant just set f= rom 0 to 1 and change modes. ########################################### So, how can we change between CM and UD in FreeBSD? Ragards!=20 -------------------------------------------------------/ From owner-freebsd-infiniband@freebsd.org Mon Dec 12 21:45:13 2016 Return-Path: Delivered-To: freebsd-infiniband@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 32398C7463F for ; Mon, 12 Dec 2016 21:45:13 +0000 (UTC) (envelope-from supportadmin@mellanox.com) Received: from smtp06-phx-sp1.mta.salesforce.com (smtp06-phx-sp1.mta.salesforce.com [136.147.46.229]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 129651CE3 for ; Mon, 12 Dec 2016 21:45:12 +0000 (UTC) (envelope-from supportadmin@mellanox.com) Received: from [10.246.216.26] ([10.246.216.26:41338] helo=na3-app2-8-phx.ops.sfdc.net) by mx2-phx-sp1.mta.salesforce.com (envelope-from ) (ecelerity 3.6.25.56547 r(Core:3.6.25.0)) with ESMTPS (cipher=DES-CBC3-SHA) id DE/2E-26051-26A1F485; Mon, 12 Dec 2016 21:45:06 +0000 Date: Mon, 12 Dec 2016 21:45:06 +0000 (GMT) From: Mellanox Support Admin To: "freebsd-infiniband@freebsd.org" Message-ID: Subject: Mellanox Support Package Invite to better serve your needs MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable X-SFDC-LK: 00D500000007heg X-SFDC-User: 005500000035Kpb X-Sender: postmaster@salesforce.com X-mail_abuse_inquiries: http://www.salesforce.com/company/abuse.jsp X-SFDC-TLS-NoRelay: 1 X-SFDC-Binding: 1WrIRBV94myi25uB X-SFDC-EmailCategory: workflowActionAlert X-SFDC-EntityId: 01W500000004oDA X-SFDC-Interface: internal X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 Dec 2016 21:45:13 -0000 Dear Valued Customer, We at Mellanox are happy to assist you with your current needs and feel it = is important to continue providing our customers with World Class Support i= n a professional and prompt manner. Mellanox is dedicated to our customers and strongly feels it is important t= hat at this time we assist you even though you have opted not to take advan= tage of our many Support contract offerings available to you for all your M= ellanox hardware and Software products. . Our World Wide Technical Support = Staff=92s primary objectives are to provide a personalized level of Support= that is unsurpassed in the industry. =20 If you feel you have been incorrectly identified as a non contracted Premiu= m Support customer please assist us in correcting this error by providing u= s with as many of the purchase details as possible. This will assist us gre= atly in expediting the required research.=20 Please consider that this support operation is funded by our contract custo= mers and partners. Moving forward we recommend that you consider contacti= ng Mellanox Sales to find out more about a Support package that best fits = your needs so that we can continue to provide a level of Support that has a= llowed for us to be recognized as a leader in the industry.=20 Please send all inquiries to the following address: contracts@mellanox.com You can also contact any of the regional offices listed below: North America * Western Region E-mail: western_usa_sales@mellanox.com * Eastern Region E-mail: eastern_usa_sales@mellanox.com * Latin America Region E-mail: latin_america@mellanox.com =09 * Central Region E-mail: central_usa_sales@mellanox.com * Canada Region E-mail: canada_sales@mellanox.com =20 Europe/Middle East * European Region E-mail: europe_sales@mellanox.com * Middle East Region E-mail: me_sales@mellanox.com Asia * Japan Region E-mail: japan_sales@mellanox.com * China Region E-mail: china_sales@mellanox.com * Asia Region E-mail: asia_sales@mellanox.com India * India Region E-mail: india_sales@mellanox.com Best Regards=20 Mellanox Support From owner-freebsd-infiniband@freebsd.org Tue Dec 13 05:02:28 2016 Return-Path: Delivered-To: freebsd-infiniband@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8B1EAC74CD5 for ; Tue, 13 Dec 2016 05:02:28 +0000 (UTC) (envelope-from Krishna2@chelsio.com) Received: from NAM01-SN1-obe.outbound.protection.outlook.com (mail-sn1nam01on0126.outbound.protection.outlook.com [104.47.32.126]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "Microsoft IT SSL SHA2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4704366E for ; Tue, 13 Dec 2016 05:02:27 +0000 (UTC) (envelope-from Krishna2@chelsio.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chelsious.onmicrosoft.com; s=selector1-chelsio-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=XjPIUB6hHMs5ma+gdEsEV/yjMRLI9FJ+0r5kSys96hg=; b=n423g7b+FGD6IfrUk3xg/mszAQIRPv5ZzgaHKObbTZ48GwB90iWZl8+T6/iGRrXD5FWvI1wLYy0PoOC83FHeRDIb++yIXJqMp5k54cXpgaGqsOkarka15XGomqBB4UR+hdx4Wq8MkRolad96qCaYM6ZCNghqf634xlZt5nQPSog= Received: from BN4PR12MB0818.namprd12.prod.outlook.com (10.164.58.152) by BN4PR12MB0820.namprd12.prod.outlook.com (10.164.58.154) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.771.8; Tue, 13 Dec 2016 04:38:20 +0000 Received: from BN4PR12MB0818.namprd12.prod.outlook.com ([10.164.58.152]) by BN4PR12MB0818.namprd12.prod.outlook.com ([10.164.58.152]) with mapi id 15.01.0771.014; Tue, 13 Dec 2016 04:38:20 +0000 From: KrishnamRaju ErapaRaju To: 'Hans Petter Selasky' , "'freebsd-infiniband@freebsd.org'" Subject: Warning message "libibverbs: Warning: couldn't open config directory '/etc/ibverbs/'" seen while running any rdma app Thread-Topic: Warning message "libibverbs: Warning: couldn't open config directory '/etc/ibverbs/'" seen while running any rdma app Thread-Index: AdJU+ozRZqp87EEtT+ODaOYs3EM7RQ== Date: Tue, 13 Dec 2016 04:38:20 +0000 Message-ID: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=Krishna2@chelsio.com; x-originating-ip: [103.252.25.47] x-ms-office365-filtering-correlation-id: 5c67d405-5257-4cf4-7244-08d42311ddc0 x-microsoft-antispam: UriScan:;BCL:0;PCL:0;RULEID:(22001);SRVR:BN4PR12MB0820; x-microsoft-exchange-diagnostics: 1; BN4PR12MB0820; 7:XFCRFFP72WU4cg9BBMaqhJBs1XfBUHUbcN0MLT1gMmwni62CwSE/UpVwGYXnswlz0fhucFqSEdm4N77+AyQNBake2jwckhAG79KHIQjFnj2HmKv0U5HVlq2YMAlZRrg8o/nw7h3QeFLjIPRjogHH7GV0OlBl0IkLaVmSljRU6SD2mzm+OVkp/6o/UXtajWaKt8PcQPqMRAzauGLyVy/doJ6vnc4tCwCQngKMH+gFLhPHqWRvEm5eTnJLlMmYxa9yb6l09zhj+0BJ9mvhcHYL4o4oJgROuqugBuYc8BOMDZNUzA3g4i+KuzuleJVOUfWNyc7OI1bgrd4e0Dt2fO8sQZ1aa221SEbnJsQ5kZumA65fGvckeNZErQLlIWzg/saxxbDYGx/1Q187ofDc/pCrJfK78cgxK+uQ4xB8Er+RxkhRyQfRMVN9pk2zxJdENisFI9c0uVdQFiu6PD24qoL+aQ==; 23:RwuJ4AumNDIMXZDAwsUGxC8M1AB7Fb2WF6/6VgqXT06mMGTkxWfWIkiCmcPOX/IGyjt2UlbW4bGzvkDTWNqE7JsDIg30FtZYwpy4gk+IkA8rXNLRjI3aZJP75xXn2ToMtA3ssuqdRCBe2AighR3STQ== x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:; x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(6040375)(601004)(2401047)(8121501046)(5005006)(3002001)(10201501046)(6041248)(20161123555025)(2016111802025)(20161123560025)(20161123564025)(20161123562025)(6043046)(6072148); SRVR:BN4PR12MB0820; BCL:0; PCL:0; RULEID:; SRVR:BN4PR12MB0820; x-forefront-prvs: 01559F388D x-forefront-antispam-report: SFV:NSPM; SFS:(10019020)(6009001)(7916002)(39450400003)(39840400002)(39410400002)(189002)(199003)(66066001)(102836003)(5001770100001)(86362001)(50986999)(2900100001)(92566002)(122556002)(6116002)(3846002)(74316002)(33656002)(7736002)(305945005)(9686002)(2906002)(68736007)(77096006)(54356999)(3280700002)(101416001)(6506006)(5660300001)(106356001)(3660700001)(7696004)(6436002)(99286002)(81166006)(105586002)(8936002)(189998001)(97736004)(81156014)(8676002)(107886002)(76576001)(38730400001)(491001); DIR:OUT; SFP:1102; SCL:1; SRVR:BN4PR12MB0820; H:BN4PR12MB0818.namprd12.prod.outlook.com; FPR:; SPF:None; PTR:InfoNoRecords; MX:1; A:1; LANG:en; received-spf: None (protection.outlook.com: chelsio.com does not designate permitted sender hosts) spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Exchange-CrossTenant-originalarrivaltime: 13 Dec 2016 04:38:20.6184 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 065db76d-a7ae-4c60-b78a-501e8fc17095 X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN4PR12MB0820 X-OriginatorOrg: chelsio.com X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Dec 2016 05:02:28 -0000 Hi Hans, [root@taj ~]# rping -s -a 102.7.7.180 -Vvd libibverbs: Warning: couldn't open config directory '/etc/ibverbs/'. created cm_id 0x801c2d000 rdma_bind_addr successful rdma_listen Regarding the above Warning, I am seeing this warning while running any rdm= a application. Am I missing something? I have built the TOT FreeBSD HEAD and done the below configuration: Added WITH_OFED=3D'yes' to /etc/src.conf and also /etc/make.conf How to fix the above warning, all apps are getting this same warning, thoug= h apps works without any issue. I am using Chelsio IWARP adapter. Thanks, Krishna.