From owner-freebsd-infiniband@freebsd.org Thu Mar 4 02:49:55 2021 Return-Path: Delivered-To: freebsd-infiniband@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 69FDC55F01A for ; Thu, 4 Mar 2021 02:49:55 +0000 (UTC) (envelope-from gfoster@panasas.com) Received: from NAM11-DM6-obe.outbound.protection.outlook.com (mail-dm6nam11on2063.outbound.protection.outlook.com [40.107.223.63]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "DigiCert Cloud Services CA-1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Drb1t1WcBz4Xgc for ; Thu, 4 Mar 2021 02:49:53 +0000 (UTC) (envelope-from gfoster@panasas.com) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=PYjkAOcd2EQUmeb2GhIXqw5lgQ7Xve6jZYxdrP+0mu1jm91JgEqKSuB5b+gYEElHnSDAq1gnScFrxuOJe9ve0tx11AinKm99siybqcgM+tKpMUH4Y/H1p0mf9FI21Sf414WSUIEhSnPpmm8ZjFJkOA+YU85f7jM5neHUT41Q1rovmJ954iIJwC3PuBYqAFebD6oY0ypHQPZ7lFQ2qfkxRKTLL7msBC0f0LHxzA9f+oUmvPpruXRPt3Zh1JN+QfwkLxt/m1WwJN2iR078zk0FtKpOpjMAWzlK20XUazUobQfIrLk2D7BkNGc9xwf5MxM39vAIqMZ4Nf3FRAKLO6y7ZA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=AUc1FvvWJNhET39uE6FT8KM2ig1uALLzxvmNx3hO3A8=; b=SoSiz1OoY/+i6EkYf5X5RUJ04r/AM/LVkxQkxM2/Xh12GEo8FI3hJ6I+QutShzuKBVB9z0qnm9R5Eyf6cJxLFtgyQ6THvlYLVoocrwG1MAVOhU9UAdc7g6KhaSbLTwvSU6NziAkecRoknQ69qL67ivjTlpc2gHbhgh1h/yCdD/s7eVta4hPooUfFs7KLlJXNEj384zLqdnThj04yNrtAcxxNwQkZxyjYQndRKH1y9lIpw18cpYNLhw4hHFf8yY8lbWK9vlT7CdeZGWLsz7TPj38NSimJrTyNQDSmW8WEqDCP2UrkPjgqMLtX2Mvfjr5zvDU40O5uYSOlvZWGFLNm1A== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=panasas.com; dmarc=pass action=none header.from=panasas.com; dkim=pass header.d=panasas.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=panasas.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=AUc1FvvWJNhET39uE6FT8KM2ig1uALLzxvmNx3hO3A8=; b=RRZAoITkraKwpKkccTHnu41cggHOY2iHmFiolQ/2mMixuit/Er3Yk5Rc7MfycnqYzFQBMk1625oGvASrcdf1tOsl277aovtYthYIY7rtvK2PcBt2jaokdZJCOyNJhymRowMBX4lB/n7mBC0w7h9HrzQWUvCS4/jSNB4IICjMnXo= Received: from BYAPR08MB6038.namprd08.prod.outlook.com (2603:10b6:a03:c6::31) by SJ0PR08MB6541.namprd08.prod.outlook.com (2603:10b6:a03:2d4::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3912.17; Thu, 4 Mar 2021 02:49:49 +0000 Received: from BYAPR08MB6038.namprd08.prod.outlook.com ([fe80::c02f:f2e7:ab04:a93]) by BYAPR08MB6038.namprd08.prod.outlook.com ([fe80::c02f:f2e7:ab04:a93%4]) with mapi id 15.20.3912.017; Thu, 4 Mar 2021 02:49:49 +0000 From: "Foster, Greg" To: "freebsd-infiniband@freebsd.org" CC: "Pokala, Ravi" Subject: Changing Between Infiniband Connected Mode and Datagram Mode at Runtime on FreeBSD 12.2 Thread-Topic: Changing Between Infiniband Connected Mode and Datagram Mode at Runtime on FreeBSD 12.2 Thread-Index: AdcQoQgyjYdLaWsTTuaF0E2P358oGw== Date: Thu, 4 Mar 2021 02:49:49 +0000 Message-ID: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [71.92.224.94] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: c9ec7736-8dbd-40a4-8efc-08d8deb82dcd x-ms-traffictypediagnostic: SJ0PR08MB6541: x-ms-exchange-transport-forked: True x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:9508; x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: a4Yg0VZhEbIUGDig52h0xbzVRaObQPdUddCPGFLJKbIPNWp+2u8j80Pljo9tuFpfA1poASKIYf5eqEj+MKrQCULAcqKjSm8CfoMh+HrVnno3aobP/ROJ+8mdKR6qsj+2o8txgeuYsZI7QswMZgL3CjvxPLvBhPU8MfsaI3Jsrv9FoJYm6RUV5XLo7T1hzktT6OdOgYoA9o3DGbwiAovE9lswWhToeXDdkFrNdEj1KzZt/GkpLOGF2zLWjcDhMvW9XsEmGd1wn13kyMmEXy3hmFlTDqEFqPABQG546/LzBI2Az1mBeWuui9eCcIfSQLZ2oKJihaYFchOy9HxE2zYrQOCA+RzGObvqLoCkNyDB4HufVIcy8AOV8H9LrqjBowb2NQeOfDp+x4doQnWIV8l8Hxfp8O+fCqPSy389hzHNowlPgTaO+IZ3U1YHSuoFBiVDIuMCe5OYv8xTUYwFddPPUHLo71ggj/LrpQelJ4IEv3PeVX/SaHhAFQiR4cYnhTWimVKOiryqMy853VaCgt5Yp2RToPYEh0yIgiNDIXEp4apAtuN9x3ZEnHU3MBwVb4Sfjw053q138Vee3smn/xLu6Gb4baKWblZ47O/khwJ2KNM= x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BYAPR08MB6038.namprd08.prod.outlook.com; PTR:; CAT:NONE; SFS:(346002)(136003)(39840400004)(376002)(366004)(396003)(66556008)(5660300002)(66476007)(66946007)(66446008)(64756008)(966005)(4744005)(33656002)(478600001)(107886003)(2906002)(186003)(4326008)(8936002)(26005)(6506007)(6916009)(9686003)(316002)(8676002)(7696005)(86362001)(52536014)(55016002)(76116006)(21615005)(166002)(71200400001); DIR:OUT; SFP:1101; x-ms-exchange-antispam-messagedata: =?us-ascii?Q?pXhy4qXCxc2/z86G+7O2VbUbTcYwn5408oSvqIU2pBoVWQxNbNMQ/FgPhoRH?= =?us-ascii?Q?aXhmcJbU6hjLrAN7ITyVisHV0ZCQl5uHmXDPM/8oBkmx82uA2bmtRhc1qE7T?= =?us-ascii?Q?dvD9J+jbu4gQjdAGDslrJzwD2cB/OmMIW+j48TSKPR4OjY7ruMqTP1d/NNQr?= =?us-ascii?Q?X7EXh3qCa69w9ILoqk5CTPr3DJTz7Zlqy+JP6Mpp3fO+QJeYE1Hz0jOrWGMP?= =?us-ascii?Q?Z5Dvs5S85sNJpnCBV6mY320JDRw9whhjXPvEOkzhK32WGs0rnkQUIiEEc7uH?= =?us-ascii?Q?QQmPtxPfpdeWYdVHSRZN20ATEqbsTcAnpX/bGyMIpUVkZk+2oNbsfzVy/2uZ?= =?us-ascii?Q?cnRg34+s7Bx7t8abe/oJpdqE5TuEu+D2tUl9swUP5TzDJPSnj8EmhgUbXzmH?= =?us-ascii?Q?nZjk5oHtWosal2oW6N0THR0xLeEa4lQIeB07E1ycbZjeAB2lKKjEQHyTTVRH?= =?us-ascii?Q?eLiUq+i+yAkfyUyBIeOOS9XsE4GsdezHch2f62EVSxePy2y2nXFeNDqJwfxr?= =?us-ascii?Q?jIk7DdrZ0q4RELq9eehCajZ7xpp0FVk0+0jVwaBJ7GIRLqnM3AHD+1pLujw4?= =?us-ascii?Q?T7YDuMUex1K+mJ0Kdz7yrEfbOZFtYMrOBsYRvMqDzGVC2cH1yJX4ucxUAbkj?= =?us-ascii?Q?uDWY5X7BO7gF2nwuejfYMXVIdkGOzYqBePu3wn6P4l6yRdSht/OgqutBH8UC?= =?us-ascii?Q?qGk2h2+Ie6QEPwpyDenbFLTj/uycvg/0jEAX4e4rASdBM5Rb+PEcu/kFOzYK?= =?us-ascii?Q?Oqel/Iu7+L0mKcqyiDMwvb9mfSTc733QCFua9oakO49ezR/bM8wQI0trpyPR?= =?us-ascii?Q?qOXCNpuex/hNQtkyWdEUGudlU1Kb3JjLolSeNPMBpB3Qw+gzHavgjyxhqD4X?= =?us-ascii?Q?ou/xIAwQmLaTicLWNUHfRYHrjHqNocyvyNt9fK5LjwrbqoAdaB98k/7QqO9i?= =?us-ascii?Q?Qqw2xIYFNi49WQLN93exTVUS7pEgZZEUUjJI0lq/ZwGlIql43XgdolybraUP?= =?us-ascii?Q?dX989ursJdApcvuPcxfd6L0ITGG+2WOlcjhJO0nUry/SehDEFGw4i1fpsN+i?= =?us-ascii?Q?pDIqDdRV4rYvRVz1cAMln+7hkRbt4zU7caHubwbNW1WVVKWk22qvDtO/VCcg?= =?us-ascii?Q?ZLbXb79ryw+lF7/9bq3oXZaqW4I63kksthmPVaQCbpLUbGAmtA57xzIYc07C?= =?us-ascii?Q?RrAdFGgva/WSD0p5UNrnzBEiRucLBsg/NMagk+rwjB8fPnDyeJJyH5ZOA10C?= =?us-ascii?Q?9UUcnKAy+8ukkBqpYxbQ2f70r1CDOJbdnVMIkrH8FtHE1ABo73vvRQZTa+VS?= =?us-ascii?Q?eWs=3D?= MIME-Version: 1.0 X-OriginatorOrg: panasas.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: BYAPR08MB6038.namprd08.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: c9ec7736-8dbd-40a4-8efc-08d8deb82dcd X-MS-Exchange-CrossTenant-originalarrivaltime: 04 Mar 2021 02:49:49.5360 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: acf01c9d-c699-42af-bdbb-44bf582e60b0 X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: d7yoqLkSSKD42DtjMYT/4F1YwOd5eKjWgsuTbIYq2KSjihcPv5+8phug2ut0A01kuITmtug8Qp/Ktnl3MpPxAA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR08MB6541 X-Rspamd-Queue-Id: 4Drb1t1WcBz4Xgc X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=panasas.com header.s=selector1 header.b=RRZAoITk; arc=pass (microsoft.com:s=arcselector9901:i=1); dmarc=none; spf=pass (mx1.freebsd.org: domain of gfoster@panasas.com designates 40.107.223.63 as permitted sender) smtp.mailfrom=gfoster@panasas.com X-Spamd-Result: default: False [-4.49 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; RBL_DBL_DONT_QUERY_IPS(0.00)[40.107.223.63:from]; R_DKIM_ALLOW(-0.20)[panasas.com:s=selector1]; HAS_XOIP(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:40.107.0.0/16]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; DMARC_NA(0.00)[panasas.com]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; SPAMHAUS_ZRD(0.00)[40.107.223.63:from:127.0.2.255]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[panasas.com:+]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[40.107.223.63:from]; NEURAL_HAM_SHORT(-0.99)[-0.988]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:8075, ipnet:40.104.0.0/14, country:US]; ARC_ALLOW(-1.00)[microsoft.com:s=arcselector9901:i=1]; MAILMAN_DEST(0.00)[freebsd-infiniband]; RWL_MAILSPIKE_POSSIBLE(0.00)[40.107.223.63:from] X-Mailman-Approved-At: Thu, 04 Mar 2021 06:14:51 +0000 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.34 X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 04 Mar 2021 02:49:55 -0000 Hello FreeBSD IB, What is the procedure to change between Infiniband Connected Mode and Datag= ram Mode at runtime on FreeBSD 12.2? I've seen different references on the WEB, but the procedure still seems un= clear. In one reference https://lists.freebsd.org/pipermail/freebsd-infiniband/201= 3-May/000032.html refers to the Linux procedure, which is not applicable on FreeBSD. Another reference, https://wiki.freebsd.org/InfiniBand, implies the driver = must be recompiled to change modes. Yet another, https://lists.freebsd.org/pipermail/freebsd-infiniband/2013-Ju= ne/000047.html Suggests it can be done at runtime to add a tuneable/sysctl but doesn't spe= cify any details... Is the solution to maybe compile the drivers twice, once for CM and once fo= r DM, keeping both around and loading the desired one? Thanks Greg Foster From owner-freebsd-infiniband@freebsd.org Thu Mar 4 10:14:02 2021 Return-Path: Delivered-To: freebsd-infiniband@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 5FC5456AF40 for ; Thu, 4 Mar 2021 10:14:02 +0000 (UTC) (envelope-from hps@selasky.org) Received: from mail.turbocat.net (turbocat.net [88.99.82.50]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4DrmtK32qQz3FqJ for ; Thu, 4 Mar 2021 10:14:01 +0000 (UTC) (envelope-from hps@selasky.org) Received: from hps2020.home.selasky.org (unknown [178.17.145.105]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mail.turbocat.net (Postfix) with ESMTPSA id D4F1A260243; Thu, 4 Mar 2021 11:13:52 +0100 (CET) Subject: Re: Changing Between Infiniband Connected Mode and Datagram Mode at Runtime on FreeBSD 12.2 To: "Foster, Greg" , "freebsd-infiniband@freebsd.org" Cc: "Pokala, Ravi" References: From: Hans Petter Selasky Message-ID: Date: Thu, 4 Mar 2021 11:13:36 +0100 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:78.0) Gecko/20100101 Thunderbird/78.5.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 4DrmtK32qQz3FqJ X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of hps@selasky.org designates 88.99.82.50 as permitted sender) smtp.mailfrom=hps@selasky.org X-Spamd-Result: default: False [-3.30 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCVD_TLS_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; TO_DN_SOME(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; MIME_GOOD(-0.10)[text/plain]; R_SPF_ALLOW(-0.20)[+a:mail.turbocat.net]; ARC_NA(0.00)[]; DMARC_NA(0.00)[selasky.org]; SPAMHAUS_ZRD(0.00)[88.99.82.50:from:127.0.2.255]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RBL_DBL_DONT_QUERY_IPS(0.00)[88.99.82.50:from]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:24940, ipnet:88.99.0.0/16, country:DE]; RCVD_COUNT_TWO(0.00)[2]; MAILMAN_DEST(0.00)[freebsd-infiniband] X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 04 Mar 2021 10:14:02 -0000 On 3/4/21 3:49 AM, Foster, Greg wrote: > Hello FreeBSD IB, > > What is the procedure to change between Infiniband Connected Mode and Datagram Mode > at runtime on FreeBSD 12.2? > > I've seen different references on the WEB, but the procedure still seems unclear. > > In one reference https://lists.freebsd.org/pipermail/freebsd-infiniband/2013-May/000032.html > refers to the Linux procedure, which is not applicable on FreeBSD. > > Another reference, https://wiki.freebsd.org/InfiniBand, implies the driver must be recompiled > to change modes. > > Yet another, https://lists.freebsd.org/pipermail/freebsd-infiniband/2013-June/000047.html > Suggests it can be done at runtime to add a tuneable/sysctl but doesn't specify any details... > > Is the solution to maybe compile the drivers twice, once for CM and once for DM, keeping both around > and loading the desired one? Hi, For IPoIB there is an option you need to set at compile time for this feature to be activated: sys/conf/options:IPOIB_CM opt_ofed.h --HPS From owner-freebsd-infiniband@freebsd.org Thu Mar 4 19:11:20 2021 Return-Path: Delivered-To: freebsd-infiniband@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 897A0555FDA for ; Thu, 4 Mar 2021 19:11:20 +0000 (UTC) (envelope-from hps@selasky.org) Received: from mail.turbocat.net (turbocat.net [88.99.82.50]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Ds0pH3hd8z4jsc for ; Thu, 4 Mar 2021 19:11:19 +0000 (UTC) (envelope-from hps@selasky.org) Received: from hps2020.home.selasky.org (unknown [178.17.145.105]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mail.turbocat.net (Postfix) with ESMTPSA id 49448260100; Thu, 4 Mar 2021 20:11:17 +0100 (CET) Subject: Re: Changing Between Infiniband Connected Mode and Datagram Mode at Runtime on FreeBSD 12.2 To: "Pokala, Ravi" , "Foster, Greg" , "freebsd-infiniband@freebsd.org" References: <08557FD0-003C-4802-A588-92298A54D4CD@panasas.com> From: Hans Petter Selasky Message-ID: <71b1d84d-917e-3f6a-cad6-794f06fefaee@selasky.org> Date: Thu, 4 Mar 2021 20:10:59 +0100 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:78.0) Gecko/20100101 Thunderbird/78.5.0 MIME-Version: 1.0 In-Reply-To: <08557FD0-003C-4802-A588-92298A54D4CD@panasas.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 4Ds0pH3hd8z4jsc X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of hps@selasky.org designates 88.99.82.50 as permitted sender) smtp.mailfrom=hps@selasky.org X-Spamd-Result: default: False [-3.30 / 15.00]; ARC_NA(0.00)[]; TO_DN_EQ_ADDR_SOME(0.00)[]; RCVD_TLS_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; R_SPF_ALLOW(-0.20)[+a:mail.turbocat.net:c]; MID_RHS_MATCH_FROM(0.00)[]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[selasky.org]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_DN_SOME(0.00)[]; SPAMHAUS_ZRD(0.00)[88.99.82.50:from:127.0.2.255]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RBL_DBL_DONT_QUERY_IPS(0.00)[88.99.82.50:from]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:24940, ipnet:88.99.0.0/16, country:DE]; RCVD_COUNT_TWO(0.00)[2]; MAILMAN_DEST(0.00)[freebsd-infiniband] X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 04 Mar 2021 19:11:20 -0000 On 3/4/21 8:00 PM, Pokala, Ravi wrote: > -----Original Message----- > From: Hans Petter Selasky > Date: 2021-03-04, Thursday at 02:13 > To: "Foster, Greg" , "freebsd-infiniband@freebsd.org" > Cc: Ravi Pokala > Subject: Re: Changing Between Infiniband Connected Mode and Datagram Mode at Runtime on FreeBSD 12.2 > > On 3/4/21 3:49 AM, Foster, Greg wrote: > > Hello FreeBSD IB, > > > > What is the procedure to change between Infiniband Connected Mode and Datagram Mode > > at runtime on FreeBSD 12.2? > > > > I've seen different references on the WEB, but the procedure still seems unclear. > > > > In one reference https://lists.freebsd.org/pipermail/freebsd-infiniband/2013-May/000032.html > > refers to the Linux procedure, which is not applicable on FreeBSD. > > > > Another reference, https://wiki.freebsd.org/InfiniBand, implies the driver must be recompiled > > to change modes. > > > > Yet another, https://lists.freebsd.org/pipermail/freebsd-infiniband/2013-June/000047.html > > Suggests it can be done at runtime to add a tuneable/sysctl but doesn't specify any details... > > > > Is the solution to maybe compile the drivers twice, once for CM and once for DM, keeping both around > > and loading the desired one? > > Hi, > > For IPoIB there is an option you need to set at compile time for this > feature to be activated: > > sys/conf/options:IPOIB_CM opt_ofed.h > > --HPS > > Hi Hans, > > It looks like IPOIB_CM causes CONFIG_INFINIBAND_IPOIB_CM to be defined. I see that CONFIG_INFINIBAND_IPOIB_CM adds fields to structures, which is why it can't be a run-time or boot-time tunable. :-( > > But it looks like the changes associated with CONFIG_INFINIBAND_IPOIB_CM are all additions, not replacing datagram code with connected-mode code. Other than having unnecessary fields in structures, would there be any harm in building with IPOIB_CM, but running in datagram mode? Or are you saying that we first need to build with IPOIB_CM, and then we can use a tunable to switch modes? > Hi, At the moment there is no tunable for this feature, as I'm aware of. The easiest would be to build ipoib two times, one time with CM enabled and one time without CM enabled. --HPS From owner-freebsd-infiniband@freebsd.org Thu Mar 4 19:00:47 2021 Return-Path: Delivered-To: freebsd-infiniband@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 6FCB9555C11 for ; Thu, 4 Mar 2021 19:00:47 +0000 (UTC) (envelope-from rpokala@panasas.com) Received: from NAM12-DM6-obe.outbound.protection.outlook.com (mail-dm6nam12on2046.outbound.protection.outlook.com [40.107.243.46]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "DigiCert Cloud Services CA-1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Ds0Z621vsz4jWB for ; Thu, 4 Mar 2021 19:00:45 +0000 (UTC) (envelope-from rpokala@panasas.com) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=ZXBGu17W/S8NL9Dl33ntK+lKX2l5NwANl7CmTsB6MpVKQX7TA0AqnbDDlN+fXp/Yq0wV9iHqaefJFoSYRKezqFiam2WbICHcY0d5nveJaHiQMgrtEPF0wsRfCSEIZFhX0i5s5tVF0Fhs4QIcIlo/eAoORNXqPKJElb3YkuHb50xYtM/YZczyYXvyVfd0hxtbflOoto3LZRt4XugNDedl3muZjm704ZOkN+sE2JyQjR0H3D82GdKTpb7sYVfWfS2MIZ/75/p3ZR7Eh2XhF3QcCudokrJR0PFrkRnkHkL9k8CPuDW72Om8TfgciOZY7+so76t8B+Nu6ROgU7FI3/uHEg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=EQr9IG0vHY769O9zrXc93qF04cg7GQyjugAIRZXgm3g=; b=ClfydWQjRhZxJPhpGzIt7ZWxK0Dtbi1NweBe/CKpyXv51+c4Tt0oA3Q5jFG3QpqSiiA8ixxU5nCBk1SE/3lY9wiAQpz6bcwmaTQZRz93s+guTXsUgHvkSbFvfVvBP8/Fi1wii2STz8scOKjo7SUeoiCqLJxNE7eXI3hrADI6zwtTI4IDERa8+CCD4mNPR2CbDv70y0ehvkLkQFyabalXY/UrBvE+S4FiPZMpWC7oB8nP6kxBhxSEn1lFl9PhzUh5yEmUSzeyxM4oBMz3vTPFwkJRJcSW0jigx+q8PWM2clLvHSnzFtbRrDTVPAAw39z6axsJgUiAvdzwdLUT/6Cghg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=panasas.com; dmarc=pass action=none header.from=panasas.com; dkim=pass header.d=panasas.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=panasas.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=EQr9IG0vHY769O9zrXc93qF04cg7GQyjugAIRZXgm3g=; b=WA55X2efqpgvwA34N8vqdwYZ9YGp37UNO8hY0aHlgaiFg4XCgAIHfIxobCNY/jEFWt++SYBDdDGVdUSqc14KWy09NiMaFapnYQO3oMXKhkMBJhXZe0lUqktWDn6S0nVqeMIFXhmHRpzuGf9iAseinqxgXYOqDsdi0wIVGmRTKOY= Received: from BYAPR08MB4966.namprd08.prod.outlook.com (2603:10b6:a03:66::32) by BYAPR08MB5781.namprd08.prod.outlook.com (2603:10b6:a03:126::24) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3912.17; Thu, 4 Mar 2021 19:00:43 +0000 Received: from BYAPR08MB4966.namprd08.prod.outlook.com ([fe80::fd82:ae89:c5dd:a3a4]) by BYAPR08MB4966.namprd08.prod.outlook.com ([fe80::fd82:ae89:c5dd:a3a4%7]) with mapi id 15.20.3890.028; Thu, 4 Mar 2021 19:00:43 +0000 From: "Pokala, Ravi" To: Hans Petter Selasky , "Foster, Greg" , "freebsd-infiniband@freebsd.org" Subject: Re: Changing Between Infiniband Connected Mode and Datagram Mode at Runtime on FreeBSD 12.2 Thread-Topic: Changing Between Infiniband Connected Mode and Datagram Mode at Runtime on FreeBSD 12.2 Thread-Index: AdcQoQgyjYdLaWsTTuaF0E2P358oGwAPgGoAAAGk9oA= Date: Thu, 4 Mar 2021 19:00:42 +0000 Message-ID: <08557FD0-003C-4802-A588-92298A54D4CD@panasas.com> References: In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: user-agent: Microsoft-MacOutlook/16.45.21011103 x-originating-ip: [98.42.164.217] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 1679f6b3-43d0-4c90-73a0-08d8df3fcf82 x-ms-traffictypediagnostic: BYAPR08MB5781: x-ms-exchange-transport-forked: True x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:9508; x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: l6QA5zMxf5Rdgc1YqHVFMD5T2B/9DKnTZ/9kIfL3LoBsd7O87iElRvVJbdMZ/+WQxbAzO1XQxcPlVbp7dgBALYkC65j4+ZyVG+1G775Y397nnLpgg9DSmLH8VQlvpDtCXTEYU3P80plvUdk94pbOM89ENYtKnj9MlDrRzOrScc1rNfxT260b21qM8NVgBKDv/BzMN8c43Etsuqb76bpWitQP+3k7hMDgi/rSbqpHtlF8b980rDqYI+FQEfIapNh+Auo2hU/l+xZ9pTx2X97qCAXMaetA5VsMkT21b2g0GmyQ8oEYJdYmUqN4Z8gn9NSDmXd6Qw0qqFHzH6niCkmlc+/R9AmGNjH0UaAQ2+WgEkct0SxTVtv0Xj5IpqpaZzfHA/oxhIu1mK5krTDKwSOlUcGnRZZJg+sh7AD+6q1SCcz+Vef5BmvCQoqnboplZdqoegcR6Z1ivrJQVuSgPKBBD8MgHRf3grGE17uxMLjH2226gPz3kgGSeb4+NClLQwLUJZskZLfUEATijVkd/9MPdlaGBoPnY3UFzGngzGNaMwx0M0gN9hwutE/H6azWrrg6L4UjCJzVJ2fjlvlBj8JccrDYz/yAnjCiGuesP2UzeeQiFaW5hx3x575xj1VqXfhvFiEgA/b/28qqrqy0x9NLiFY+JPBrY9DY4bfeOorbIco= x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BYAPR08MB4966.namprd08.prod.outlook.com; PTR:; CAT:NONE; SFS:(136003)(366004)(346002)(396003)(376002)(39840400004)(36756003)(66476007)(2616005)(186003)(76116006)(33656002)(316002)(6486002)(6506007)(6512007)(83380400001)(8676002)(966005)(110136005)(478600001)(26005)(71200400001)(53546011)(8936002)(66446008)(5660300002)(66946007)(64756008)(66556008)(2906002)(86362001)(45980500001); DIR:OUT; SFP:1101; x-ms-exchange-antispam-messagedata: =?utf-8?B?K0g0VG9zYUU3K3BId2MzTWFBTjcyUVVtZ0NmZU1PSXFmWnlkblVDNjZUa0R3?= =?utf-8?B?SkFSUTZkbnJlcW9DRWJkaWhKcDJ0ZHM5VDRZTzhRSWdTb1pxQ3dxb2Qxd0Er?= =?utf-8?B?VGpsOEZGMmxXdk1td1g0cTdnZ1pUWlZpVm9rT1g2WGtDYW5TL2luOWl2R1Fm?= =?utf-8?B?Y1MwZHpqRVU0UUc3czJveEhGZ3c1cmc5dFBhNUZsRkZmd0c1QVd1UTB1eURr?= =?utf-8?B?ck1mQVY0NjhNTzIySlZMQUZpMHBqVnFyRVVFSUFNei90KzBtRGJIVnBOUzR1?= =?utf-8?B?S3BLK3k0N3FlaWtkR1ZhZUljdkl4WmcwdmdhZFFoVEgxN3hzaldsNFgzbXpO?= =?utf-8?B?Qm04VkN3Zk5MRDQ0c211MktkbERCRSt4SWFBc1M1UVJXM2JPM2p2Mi8yUVIw?= =?utf-8?B?MVhFZFhkM3paNm1lZnAvQ0dpQTFIQVp6OFJOK3ZqbVdIUWxVclVFVGVOTk5N?= =?utf-8?B?RkMrRmduNnBqMkcrTWc2c01QbGxtbEs5bTBYcUZlMzBvWFFmMzF0MVI5TjZk?= =?utf-8?B?L0lYMGM5NkhmQXRoR002OWJ6YXowRVBaZllhVThVMTRZMG00UGdaQTl5aTN0?= =?utf-8?B?WG5WN2JOQS9MQjJydCsxMHVLYURsaytsek1NK2pVbnJiR25HNFdrUHF0Tld5?= =?utf-8?B?Q25EMzA1ZW41cGlaRmVBbjdJLyt3SWxUQlBydkl5cEYwNVVJcEU1b1o3VERy?= =?utf-8?B?V3J0MFlVWFMrYlM3NzJmWXR0SkVGSkkyQi9CM00wd2RST3FTekpIOUUrWVFX?= =?utf-8?B?WVlqRmpZSDJLN3Z5NEwwdFVCaXp6SkYrdDEvSHpEZ1UveGE0ZmZkeXZsa0Iy?= =?utf-8?B?ZjFWZGxaa1ZpY2dYUHRZTnJGa29DMUF5NDViYjhvNDFlTUZTSERXQmc3SkZW?= =?utf-8?B?RktHVTNoamFmNzdYQ1o5TmVqOUc0SHdsNmVndnVVc0I3MzJ1ZXNOM0c3ZnhB?= =?utf-8?B?czRBdE9MbE1PV2VIRFUzSlpDRkV0OFNOdXEvcVRDUlcwZUJSbFdia1RoWUZ2?= =?utf-8?B?K2kvZFpOUVowZmpVUUlXL2tOUEVPNXdRdnE3ZVI5QWh3b293UlFNR2hWRGt4?= =?utf-8?B?dURZS0d5cnJFTW5mRlIyc0s1RXNMWHBBSGNZS2lYV0RjV1Aya3k5ajkyak1K?= =?utf-8?B?a3pnemtHR3dlaXdQLy9qRUV5bWwwOGZ3OWZVa042RVlIV3NzOEJuNjV1T1Vt?= =?utf-8?B?Mm5OTVBGSC9hUnYvUDRhVmsxK2kzVUdGclA1VHNlWUhPSUtBZy9jYSszTHV2?= =?utf-8?B?WEVWRW1sZWVLUE9zSW12VGJ0M3lxV0ZnVVhDdUtvOHorK3RVLzZ2TURRN2Jk?= =?utf-8?B?RGd0YXIwemp6Y2JydGJqd1hZSDVGUXpEOTJCeksvOEMrM0VWdndITncyZjFN?= =?utf-8?B?MXphNmRxS1BXTzdKbVB5WkNJVlpxQk9RYUU0eFRvazFsZUwwUkhTdWZnYW5q?= =?utf-8?B?UTVRUmYvVmhocUtEMXYvaU5kUDlhQkhaUUtzMlMxc0NqYzRKMVY3WXZCTWRz?= =?utf-8?B?TzVpSXRwdzdiRHFINy9OdXBQYnFOUnhFdzA1bXUvR2tRekVrb1hLNEJvTnc4?= =?utf-8?B?VE1tTkVSVHFlMWI5K25iRDZkV3hFOG94TnlaQ255ZEd4cTQvMjlrbXZMaVdE?= =?utf-8?B?QUFDQmJPeGtoby9OM1ZVZ0hxYzRyUG1FM1piNVM5bXU3QVdXa256ZUJERlNZ?= =?utf-8?B?MXRDMTdHR2NRTjlJVks3NE4ycG03Y3FrcDVXN0JXTFdqd05JTHBJSGJKcWpB?= =?utf-8?Q?gVgHESoI39ZgcyYsDAxdKJEojEScB/Jpz7awBsO?= Content-Type: text/plain; charset="utf-8" Content-ID: <0F443BA0E4FFFF4A953096A6B760C137@namprd08.prod.outlook.com> Content-Transfer-Encoding: base64 MIME-Version: 1.0 X-OriginatorOrg: panasas.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: BYAPR08MB4966.namprd08.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: 1679f6b3-43d0-4c90-73a0-08d8df3fcf82 X-MS-Exchange-CrossTenant-originalarrivaltime: 04 Mar 2021 19:00:42.8992 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: acf01c9d-c699-42af-bdbb-44bf582e60b0 X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: ECBpZZPxzNeMNhYFKAYd2G6/ySWMn0zL8w9A6qa0ireqSC7vsxUhwn2mNwA1C4zFxdsktitCWtBiGsimrYnXiQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR08MB5781 X-Rspamd-Queue-Id: 4Ds0Z621vsz4jWB X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=panasas.com header.s=selector1 header.b=WA55X2ef; arc=pass (microsoft.com:s=arcselector9901:i=1); dmarc=none; spf=pass (mx1.freebsd.org: domain of rpokala@panasas.com designates 40.107.243.46 as permitted sender) smtp.mailfrom=rpokala@panasas.com X-Spamd-Result: default: False [-3.40 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; HAS_XOIP(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:40.107.0.0/16]; MIME_BASE64_TEXT_BOGUS(1.00)[]; RCVD_COUNT_THREE(0.00)[3]; DKIM_TRACE(0.00)[panasas.com:+]; MIME_BASE64_TEXT(0.10)[]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; RBL_DBL_DONT_QUERY_IPS(0.00)[40.107.243.46:from]; ARC_ALLOW(-1.00)[microsoft.com:s=arcselector9901:i=1]; MIME_TRACE(0.00)[0:+]; MID_RHS_MATCH_FROM(0.00)[]; ASN(0.00)[asn:8075, ipnet:40.104.0.0/14, country:US]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[panasas.com:s=selector1]; FREEFALL_USER(0.00)[rpokala]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[panasas.com]; SPAMHAUS_ZRD(0.00)[40.107.243.46:from:127.0.2.255]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[40.107.243.46:from]; RWL_MAILSPIKE_POSSIBLE(0.00)[40.107.243.46:from]; MAILMAN_DEST(0.00)[freebsd-infiniband] X-Mailman-Approved-At: Fri, 05 Mar 2021 05:35:16 +0000 X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 04 Mar 2021 19:00:47 -0000 LS0tLS1PcmlnaW5hbCBNZXNzYWdlLS0tLS0NCkZyb206IEhhbnMgUGV0dGVyIFNlbGFza3kgPGhw c0BzZWxhc2t5Lm9yZz4NCkRhdGU6IDIwMjEtMDMtMDQsIFRodXJzZGF5IGF0IDAyOjEzDQpUbzog IkZvc3RlciwgR3JlZyIgPGdmb3N0ZXJAcGFuYXNhcy5jb20+LCAiZnJlZWJzZC1pbmZpbmliYW5k QGZyZWVic2Qub3JnIiA8ZnJlZWJzZC1pbmZpbmliYW5kQGZyZWVic2Qub3JnPg0KQ2M6IFJhdmkg UG9rYWxhIDxycG9rYWxhQHBhbmFzYXMuY29tPg0KU3ViamVjdDogUmU6IENoYW5naW5nIEJldHdl ZW4gSW5maW5pYmFuZCBDb25uZWN0ZWQgTW9kZSBhbmQgRGF0YWdyYW0gTW9kZSBhdCBSdW50aW1l IG9uIEZyZWVCU0QgMTIuMg0KDQogICAgT24gMy80LzIxIDM6NDkgQU0sIEZvc3RlciwgR3JlZyB3 cm90ZToNCiAgICA+IEhlbGxvIEZyZWVCU0QgSUIsDQogICAgPiANCiAgICA+IFdoYXQgaXMgdGhl IHByb2NlZHVyZSB0byBjaGFuZ2UgYmV0d2VlbiBJbmZpbmliYW5kIENvbm5lY3RlZCBNb2RlIGFu ZCBEYXRhZ3JhbSBNb2RlDQogICAgPiBhdCBydW50aW1lIG9uIEZyZWVCU0QgMTIuMj8NCiAgICA+ IA0KICAgID4gSSd2ZSBzZWVuIGRpZmZlcmVudCByZWZlcmVuY2VzIG9uIHRoZSBXRUIsIGJ1dCB0 aGUgcHJvY2VkdXJlIHN0aWxsIHNlZW1zIHVuY2xlYXIuDQogICAgPiANCiAgICA+IEluIG9uZSBy ZWZlcmVuY2UgaHR0cHM6Ly9saXN0cy5mcmVlYnNkLm9yZy9waXBlcm1haWwvZnJlZWJzZC1pbmZp bmliYW5kLzIwMTMtTWF5LzAwMDAzMi5odG1sDQogICAgPiByZWZlcnMgdG8gdGhlIExpbnV4IHBy b2NlZHVyZSwgd2hpY2ggaXMgbm90IGFwcGxpY2FibGUgb24gRnJlZUJTRC4NCiAgICA+IA0KICAg ID4gQW5vdGhlciByZWZlcmVuY2UsIGh0dHBzOi8vd2lraS5mcmVlYnNkLm9yZy9JbmZpbmlCYW5k LCBpbXBsaWVzIHRoZSBkcml2ZXIgbXVzdCBiZSByZWNvbXBpbGVkDQogICAgPiB0byBjaGFuZ2Ug bW9kZXMuDQogICAgPiANCiAgICA+IFlldCBhbm90aGVyLCBodHRwczovL2xpc3RzLmZyZWVic2Qu b3JnL3BpcGVybWFpbC9mcmVlYnNkLWluZmluaWJhbmQvMjAxMy1KdW5lLzAwMDA0Ny5odG1sDQog ICAgPiBTdWdnZXN0cyBpdCBjYW4gYmUgZG9uZSBhdCBydW50aW1lIHRvIGFkZCBhIHR1bmVhYmxl L3N5c2N0bCBidXQgZG9lc24ndCBzcGVjaWZ5IGFueSBkZXRhaWxzLi4uDQogICAgPiANCiAgICA+ IElzIHRoZSBzb2x1dGlvbiB0byBtYXliZSBjb21waWxlIHRoZSBkcml2ZXJzIHR3aWNlLCBvbmNl IGZvciBDTSBhbmQgb25jZSBmb3IgRE0sIGtlZXBpbmcgYm90aCBhcm91bmQNCiAgICA+IGFuZCBs b2FkaW5nIHRoZSBkZXNpcmVkIG9uZT8NCg0KICAgIEhpLA0KDQogICAgRm9yIElQb0lCIHRoZXJl IGlzIGFuIG9wdGlvbiB5b3UgbmVlZCB0byBzZXQgYXQgY29tcGlsZSB0aW1lIGZvciB0aGlzIA0K ICAgIGZlYXR1cmUgdG8gYmUgYWN0aXZhdGVkOg0KDQogICAgc3lzL2NvbmYvb3B0aW9uczpJUE9J Ql9DTQlvcHRfb2ZlZC5oDQoNCiAgICAtLUhQUw0KDQpIaSBIYW5zLA0KDQpJdCBsb29rcyBsaWtl IElQT0lCX0NNIGNhdXNlcyBDT05GSUdfSU5GSU5JQkFORF9JUE9JQl9DTSB0byBiZSBkZWZpbmVk LiBJIHNlZSB0aGF0IENPTkZJR19JTkZJTklCQU5EX0lQT0lCX0NNIGFkZHMgZmllbGRzIHRvIHN0 cnVjdHVyZXMsIHdoaWNoIGlzIHdoeSBpdCBjYW4ndCBiZSBhIHJ1bi10aW1lIG9yIGJvb3QtdGlt ZSB0dW5hYmxlLiA6LSgNCg0KQnV0IGl0IGxvb2tzIGxpa2UgdGhlIGNoYW5nZXMgYXNzb2NpYXRl ZCB3aXRoIENPTkZJR19JTkZJTklCQU5EX0lQT0lCX0NNIGFyZSBhbGwgYWRkaXRpb25zLCBub3Qg cmVwbGFjaW5nIGRhdGFncmFtIGNvZGUgd2l0aCBjb25uZWN0ZWQtbW9kZSBjb2RlLiBPdGhlciB0 aGFuIGhhdmluZyB1bm5lY2Vzc2FyeSBmaWVsZHMgaW4gc3RydWN0dXJlcywgd291bGQgdGhlcmUg YmUgYW55IGhhcm0gaW4gYnVpbGRpbmcgd2l0aCBJUE9JQl9DTSwgYnV0IHJ1bm5pbmcgaW4gZGF0 YWdyYW0gbW9kZT8gT3IgYXJlIHlvdSBzYXlpbmcgdGhhdCB3ZSBmaXJzdCBuZWVkIHRvIGJ1aWxk IHdpdGggSVBPSUJfQ00sIGFuZCB0aGVuIHdlIGNhbiB1c2UgYSB0dW5hYmxlIHRvIHN3aXRjaCBt b2Rlcz8NCg0KVGhhbmtzLA0KDQpSYXZpDQoNCg== From owner-freebsd-infiniband@freebsd.org Thu Mar 4 19:37:28 2021 Return-Path: Delivered-To: freebsd-infiniband@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 18667557065 for ; Thu, 4 Mar 2021 19:37:28 +0000 (UTC) (envelope-from rpokala@panasas.com) Received: from NAM12-DM6-obe.outbound.protection.outlook.com (mail-dm6nam12on2054.outbound.protection.outlook.com [40.107.243.54]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "DigiCert Cloud Services CA-1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Ds1NR070sz4mcN for ; Thu, 4 Mar 2021 19:37:26 +0000 (UTC) (envelope-from rpokala@panasas.com) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=J2alrE/AKJJk8YNG9gahFotePrhDls4F+Itgt9cOUxZZHmoH+55c3aXdovP3Pwy59UUqjGub/SVg55YonD7U51VVNu02LaohDt/wbSrpQhawjdEGM06+Qb+jyiezsWqlDkQFPj4mpBeZ7wFruZeY7/lYplu8T53BORU7qvsRAuDj8DW+qjgxCpOOC7YX8Gw0C2muIsWBm7dYLVxGEGZgnasBQ3OONbBgUNR8UQYiPlzzgv/Hind7HAY23WmsqOeyNMGGLmRi02z32K3ukleQrdSC0DWWrFtgSziv6MAVrHn3T9b/aLj+Ncerw52SAW83rSwsVq842ojY/8qoKwZG+g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=hkCQpYoOcPlvKPsFAkc5UwIiZEAaFBf3w4hZk1naPNk=; b=ZSSr36G2qhElvVq74AjPj241qkZg6qo7YnlD/1XB/Fx+TbwCvUilI58aifzzToXKPYVVw0Y8/cNgzajVmCxuoxRnSWT7b+eqKo4Giv9KYclETOgHOEGuJbpZiHMDy0Z6+0vt/860dSjsFpgbmUTXh+AZL38onJ8C2ZV3UOR/lH6qernGsbQlzQtnCKBMs69D2TAWMHL3rwA8SHO0G+ZmEcbB1v9KcXN6rbndYLc3+YwBeS6V3WA7bJCQb5mvxH9d7lJb/9WzCZGUTg2XBc3qlpCillzdx/xwH+0TZ/9leqD4t30TDRaPacj3KgFedeJqQ4YOdM0xW2vy/Jw8BbI/iA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=panasas.com; dmarc=pass action=none header.from=panasas.com; dkim=pass header.d=panasas.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=panasas.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=hkCQpYoOcPlvKPsFAkc5UwIiZEAaFBf3w4hZk1naPNk=; b=iPBGI9DbOfofyckU0laL/yFoHEZVum/4hkPOMdB7iPh9Bga8Z1CNRtDAZZ3j271uGQ/ZEfoMNVxVQqjtMODSN9YRPW3YgW6LGVMbLwJt5bWcw2gYx+o2/ynoGuc1FPcDeTsPeCqSMI7qg7+0ZWkf5i9Gcmn1vjFtLZGXmYPdu9w= Received: from BYAPR08MB4966.namprd08.prod.outlook.com (2603:10b6:a03:66::32) by BY3PR08MB7091.namprd08.prod.outlook.com (2603:10b6:a03:365::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3912.17; Thu, 4 Mar 2021 19:37:24 +0000 Received: from BYAPR08MB4966.namprd08.prod.outlook.com ([fe80::fd82:ae89:c5dd:a3a4]) by BYAPR08MB4966.namprd08.prod.outlook.com ([fe80::fd82:ae89:c5dd:a3a4%7]) with mapi id 15.20.3890.028; Thu, 4 Mar 2021 19:37:23 +0000 From: "Pokala, Ravi" To: Hans Petter Selasky , "Foster, Greg" , "freebsd-infiniband@freebsd.org" Subject: Re: Changing Between Infiniband Connected Mode and Datagram Mode at Runtime on FreeBSD 12.2 Thread-Topic: Changing Between Infiniband Connected Mode and Datagram Mode at Runtime on FreeBSD 12.2 Thread-Index: AdcQoQgyjYdLaWsTTuaF0E2P358oGwAPgGoAAAGk9oAAER+fgP//gUSA Date: Thu, 4 Mar 2021 19:37:23 +0000 Message-ID: <55947DF3-4697-4F44-BAAD-5A70136036D7@panasas.com> References: <08557FD0-003C-4802-A588-92298A54D4CD@panasas.com> <71b1d84d-917e-3f6a-cad6-794f06fefaee@selasky.org> In-Reply-To: <71b1d84d-917e-3f6a-cad6-794f06fefaee@selasky.org> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: user-agent: Microsoft-MacOutlook/16.45.21011103 x-originating-ip: [98.42.164.217] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 19566a98-2786-4bd0-bff0-08d8df44ef5e x-ms-traffictypediagnostic: BY3PR08MB7091: x-ms-exchange-transport-forked: True x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:7219; x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: 41a8x+YFht82ZTGLfiRTMHbzWAZ+6d12B9q5FSzwF3cg6W0cBQSwFhM2ZtUZkrFy3NkHbXSR25N2RY+BBFqgrctObQYrTBOtYDw6re0Z/YtzlOvpt5w+Ndbefs3f6Yfec9pUcLfs7h7kNx5qkDfDRgAr3ct8w2/BubWKWKVFMyQYwPCuMgrFLWS2gePCpT0cupaKxUz7ZsNg15TYmZXHhR1FT406EXlUvhvwOgBu3onvSm7v2lWRhRecC6LUS9kXgL/D+mG0KjQBBoLqnMXNd3CNQf/kUg+n0gwabjZ24BSDMqIWaKLkqSBatx+YWUJBSgr31HJ2iRjYQ6yZIU6IF69ItHmUMA6kZW4E4QT7RQKc816TcrY1FxdVLDXjwu9EBPYTKTM6+vO8TfcFL1FtdSAGppazolMUfEu9kOZ+HcQlVGq9voMkPMYVlR6x2h0hI7HMlmi1mb+9Qc05JaiLTQBd3jt063HWvwTitRUGhn+xCkOHR8oH/dY3hx/W5DFssDcAEh4ka1wTr/mNiMv4ufcfiKRFRUzmJxJYnD8RtHH/XAjJu3qpO4o3wFODkik/FAigwHMldaHZogVccRQ9UulLh61RaYGqrYzH6O2JY5A8jw/DCp768As+FVZcc0QonY0oz28uycdWH4/gM3C6QewC59YZR8SJMlJw+uHQdoQ= x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BYAPR08MB4966.namprd08.prod.outlook.com; PTR:; CAT:NONE; SFS:(396003)(366004)(376002)(346002)(136003)(39840400004)(66446008)(33656002)(6506007)(966005)(478600001)(26005)(8936002)(71200400001)(2616005)(186003)(8676002)(110136005)(6512007)(53546011)(86362001)(5660300002)(83380400001)(66946007)(316002)(64756008)(2906002)(76116006)(36756003)(66476007)(66556008)(6486002)(45980500001); DIR:OUT; SFP:1101; x-ms-exchange-antispam-messagedata: =?utf-8?B?dzVFbU80dkJvaHMrcThObWordWFDOUhjUng3ME14RlRkZHoxYkNBVlkxNHB6?= =?utf-8?B?blRLUjZvQ01WRkIrbk9DZGl4SWpUZHFpdjdhOTI4KzkzMnhkdndxanNkeUln?= =?utf-8?B?QlJNaVhaajcwZGE3STNKL05LUE94Tm84Rnh1bHB2elhzQkg1NVNBVGdtdXJE?= =?utf-8?B?WjE5enNXcTRZeXZ2VmplUUp6UUhYMmZoRlZJVnhlejd4MWVCWU1lUEdTODhs?= =?utf-8?B?emJoZHhLUk9ITGNjQ2JzaWJ6eHV6VXVqQ05oMFUxMXNqRjRXWkNMdmRDZWJM?= =?utf-8?B?Y2JuOXhpS2R3Y2hsVkhPaXNNV1VqM3NDWnhpeVVvZ3c4eG5HM1o2dFMzeHFC?= =?utf-8?B?WUNGRWlEYzFPc3VreWUwVFlpenk5cmxkSUpvd0g3Ymx5Nk81SmdkRENKU2NJ?= =?utf-8?B?d2ZDZzJncmpncnJNTnRNVWcvQTU0TUNCTU81UTd3R1p3aURaeVFwWGVoNUVT?= =?utf-8?B?SCtuNXE2cDhGSDh3d2VORVZQSmlFOTN3cEFDQTdlZU1vSG1uUWVQdktuOWlx?= =?utf-8?B?YmNVdkMvcWgrQjU4YlhjbUZtd0lMa2YvV1BnbXd5ZjBtMFFiTW4rZ0ZjaEVa?= =?utf-8?B?UHFkOXJWY3ZiWU5rZ3AzNzU3SmllTWdwYk5xd0VHZWs4UDJJd1ZXZjA4YW82?= =?utf-8?B?SVkzUHNQUGNnQzEvSzNGMEZUSjhGTTJGV0dwQk93d3hRUStDVnI0QXplL3l1?= =?utf-8?B?QUVjbjYvVzBJZWtINUxsZlUrZEFtTys1bnJ5ZjZrRmRKRnZ3NGp1b0hDUXRu?= =?utf-8?B?UHRVL0ZjNE93anpxeUI1UWEyRjllbXRTZEg2WFlYaUhQZTFFOXhGTEVKelcz?= =?utf-8?B?VGhtTHJaN3lmREN5Ky9xeGpXVDRoQ2R2MXBFcnlOUnJnNyswZURwNDNLN0Zv?= =?utf-8?B?enlJU2I5QWZnbDczU3dlaG1CY2txMVJDOXNOZThkYkFYL0g5ZnVITmJMQlZo?= =?utf-8?B?RysyYzNjdml4SjloMDZKNjQ3TXQ0OG9hUnRldk1OWHJIUEZyUXg0YS9RNW8x?= =?utf-8?B?aEt0TkxQRVAyRERXRFE3SFZRVm1TeTVRRmJkM2U1WS9nUGUxOEtpYzZNVmZo?= =?utf-8?B?RXpZaUV6UElwWUdHVyt3YWRLUFVHOWtjTjJ6d3d0VmhBb2cweVFRc1g2Rmxs?= =?utf-8?B?NFZkRE0ycTJRbDBJcCtlU0YwZnVTZWtDYm9FWnh3aXVWekFhZzV4azhJblAw?= =?utf-8?B?S2E0eDB1UGhNQUhpUmlFN0dYZUFDTTVJNDNlZHFXRGI2L0RDUHFxcHZVNkVj?= =?utf-8?B?ZklRL3JGWE8zYXdTcVZXUmN2WWZkOFhmRS8yOEdjMUp2Y2lUbkxoVUpSSkZk?= =?utf-8?B?REJLSWpCdThqVFZFVUJQSTc4QnNrSXByTHNLVHdOdHBSeHkzRm1QMmZDS0tZ?= =?utf-8?B?ME54SlYzVXZVS3B3NWFpL1MyV3ZGNWk0ZHNGQWhqS3llTkp6ZHptSjBObTRX?= =?utf-8?B?WjNTOGdSOXVlU3NZbnFIcHJCbDQ3TENWRm5lbVp4NUpUNVd3YW4yRXJjaWM4?= =?utf-8?B?Wm1VQytJZkVheG5BYmRuMWU5UkZBTWE1eTc1QmJoWnNKUXh6cjQ1THA1U3gz?= =?utf-8?B?WW0yN3ZLK2tBNGtmK2lNbG9RVDN6SWlxaUFMVmxmVmtVR1loQzJuY3JqRUNG?= =?utf-8?B?THZPOE5oYyt6aitJTXIxeDBEYVZxRDlmbXRDWEJSNFl5L2l5NW9BV3pWbG9v?= =?utf-8?B?L3dweVA0NWxmTDRzRzVYVzNiTFlSaFNPVzN5M3AvU0ExaENXU3VVRFBLckpw?= =?utf-8?Q?Dkebl80KE7q5Pf6pLAXtsXEDsXeM0buWxGaXDpd?= Content-Type: text/plain; charset="utf-8" Content-ID: <964F25BB8B3F554E9874565F9A3896DE@namprd08.prod.outlook.com> Content-Transfer-Encoding: base64 MIME-Version: 1.0 X-OriginatorOrg: panasas.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: BYAPR08MB4966.namprd08.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: 19566a98-2786-4bd0-bff0-08d8df44ef5e X-MS-Exchange-CrossTenant-originalarrivaltime: 04 Mar 2021 19:37:23.7860 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: acf01c9d-c699-42af-bdbb-44bf582e60b0 X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: ghxLlkUGZm+nMpOK8IhN/aUHQI3tQsmQmkonK4t6R/ENPn77LVO2GiO/eRIBncHd91foICwGnvvKbNkkW1FFzQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: BY3PR08MB7091 X-Rspamd-Queue-Id: 4Ds1NR070sz4mcN X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=panasas.com header.s=selector1 header.b=iPBGI9Db; arc=pass (microsoft.com:s=arcselector9901:i=1); dmarc=none; spf=pass (mx1.freebsd.org: domain of rpokala@panasas.com designates 40.107.243.54 as permitted sender) smtp.mailfrom=rpokala@panasas.com X-Spamd-Result: default: False [-3.30 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; HAS_XOIP(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:40.107.0.0/16]; MIME_BASE64_TEXT_BOGUS(1.00)[]; RCVD_COUNT_THREE(0.00)[3]; DKIM_TRACE(0.00)[panasas.com:+]; MIME_BASE64_TEXT(0.10)[]; NEURAL_HAM_SHORT(-0.90)[-0.897]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; RBL_DBL_DONT_QUERY_IPS(0.00)[40.107.243.54:from]; ARC_ALLOW(-1.00)[microsoft.com:s=arcselector9901:i=1]; MIME_TRACE(0.00)[0:+]; MID_RHS_MATCH_FROM(0.00)[]; ASN(0.00)[asn:8075, ipnet:40.104.0.0/14, country:US]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[panasas.com:s=selector1]; FREEFALL_USER(0.00)[rpokala]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[panasas.com]; SPAMHAUS_ZRD(0.00)[40.107.243.54:from:127.0.2.255]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[40.107.243.54:from]; RWL_MAILSPIKE_POSSIBLE(0.00)[40.107.243.54:from]; MAILMAN_DEST(0.00)[freebsd-infiniband] X-Mailman-Approved-At: Fri, 05 Mar 2021 05:35:42 +0000 X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 04 Mar 2021 19:37:28 -0000 LS0tLS1PcmlnaW5hbCBNZXNzYWdlLS0tLS0NCkZyb206IEhhbnMgUGV0dGVyIFNlbGFza3kgPGhw c0BzZWxhc2t5Lm9yZz4NCkRhdGU6IDIwMjEtMDMtMDQsIFRodXJzZGF5IGF0IDExOjExDQpUbzog UmF2aSBQb2thbGEgPHJwb2thbGFAcGFuYXNhcy5jb20+LCAiRm9zdGVyLCBHcmVnIiA8Z2Zvc3Rl ckBwYW5hc2FzLmNvbT4sICJmcmVlYnNkLWluZmluaWJhbmRAZnJlZWJzZC5vcmciIDxmcmVlYnNk LWluZmluaWJhbmRAZnJlZWJzZC5vcmc+DQpTdWJqZWN0OiBSZTogQ2hhbmdpbmcgQmV0d2VlbiBJ bmZpbmliYW5kIENvbm5lY3RlZCBNb2RlIGFuZCBEYXRhZ3JhbSBNb2RlIGF0IFJ1bnRpbWUgb24g RnJlZUJTRCAxMi4yDQoNCiAgICBPbiAzLzQvMjEgODowMCBQTSwgUG9rYWxhLCBSYXZpIHdyb3Rl Og0KICAgID4gLS0tLS1PcmlnaW5hbCBNZXNzYWdlLS0tLS0NCiAgICA+IEZyb206IEhhbnMgUGV0 dGVyIFNlbGFza3kgPGhwc0BzZWxhc2t5Lm9yZz4NCiAgICA+IERhdGU6IDIwMjEtMDMtMDQsIFRo dXJzZGF5IGF0IDAyOjEzDQogICAgPiBUbzogIkZvc3RlciwgR3JlZyIgPGdmb3N0ZXJAcGFuYXNh cy5jb20+LCAiZnJlZWJzZC1pbmZpbmliYW5kQGZyZWVic2Qub3JnIiA8ZnJlZWJzZC1pbmZpbmli YW5kQGZyZWVic2Qub3JnPg0KICAgID4gQ2M6IFJhdmkgUG9rYWxhIDxycG9rYWxhQHBhbmFzYXMu Y29tPg0KICAgID4gU3ViamVjdDogUmU6IENoYW5naW5nIEJldHdlZW4gSW5maW5pYmFuZCBDb25u ZWN0ZWQgTW9kZSBhbmQgRGF0YWdyYW0gTW9kZSBhdCBSdW50aW1lIG9uIEZyZWVCU0QgMTIuMg0K ICAgID4gDQogICAgPiAgICAgIE9uIDMvNC8yMSAzOjQ5IEFNLCBGb3N0ZXIsIEdyZWcgd3JvdGU6 DQogICAgPiAgICAgID4gSGVsbG8gRnJlZUJTRCBJQiwNCiAgICA+ICAgICAgPg0KICAgID4gICAg ICA+IFdoYXQgaXMgdGhlIHByb2NlZHVyZSB0byBjaGFuZ2UgYmV0d2VlbiBJbmZpbmliYW5kIENv bm5lY3RlZCBNb2RlIGFuZCBEYXRhZ3JhbSBNb2RlDQogICAgPiAgICAgID4gYXQgcnVudGltZSBv biBGcmVlQlNEIDEyLjI/DQogICAgPiAgICAgID4NCiAgICA+ICAgICAgPiBJJ3ZlIHNlZW4gZGlm ZmVyZW50IHJlZmVyZW5jZXMgb24gdGhlIFdFQiwgYnV0IHRoZSBwcm9jZWR1cmUgc3RpbGwgc2Vl bXMgdW5jbGVhci4NCiAgICA+ICAgICAgPg0KICAgID4gICAgICA+IEluIG9uZSByZWZlcmVuY2Ug aHR0cHM6Ly9saXN0cy5mcmVlYnNkLm9yZy9waXBlcm1haWwvZnJlZWJzZC1pbmZpbmliYW5kLzIw MTMtTWF5LzAwMDAzMi5odG1sDQogICAgPiAgICAgID4gcmVmZXJzIHRvIHRoZSBMaW51eCBwcm9j ZWR1cmUsIHdoaWNoIGlzIG5vdCBhcHBsaWNhYmxlIG9uIEZyZWVCU0QuDQogICAgPiAgICAgID4N CiAgICA+ICAgICAgPiBBbm90aGVyIHJlZmVyZW5jZSwgaHR0cHM6Ly93aWtpLmZyZWVic2Qub3Jn L0luZmluaUJhbmQsIGltcGxpZXMgdGhlIGRyaXZlciBtdXN0IGJlIHJlY29tcGlsZWQNCiAgICA+ ICAgICAgPiB0byBjaGFuZ2UgbW9kZXMuDQogICAgPiAgICAgID4NCiAgICA+ICAgICAgPiBZZXQg YW5vdGhlciwgaHR0cHM6Ly9saXN0cy5mcmVlYnNkLm9yZy9waXBlcm1haWwvZnJlZWJzZC1pbmZp bmliYW5kLzIwMTMtSnVuZS8wMDAwNDcuaHRtbA0KICAgID4gICAgICA+IFN1Z2dlc3RzIGl0IGNh biBiZSBkb25lIGF0IHJ1bnRpbWUgdG8gYWRkIGEgdHVuZWFibGUvc3lzY3RsIGJ1dCBkb2Vzbid0 IHNwZWNpZnkgYW55IGRldGFpbHMuLi4NCiAgICA+ICAgICAgPg0KICAgID4gICAgICA+IElzIHRo ZSBzb2x1dGlvbiB0byBtYXliZSBjb21waWxlIHRoZSBkcml2ZXJzIHR3aWNlLCBvbmNlIGZvciBD TSBhbmQgb25jZSBmb3IgRE0sIGtlZXBpbmcgYm90aCBhcm91bmQNCiAgICA+ICAgICAgPiBhbmQg bG9hZGluZyB0aGUgZGVzaXJlZCBvbmU/DQogICAgPiANCiAgICA+ICAgICAgSGksDQogICAgPiAN CiAgICA+ICAgICAgRm9yIElQb0lCIHRoZXJlIGlzIGFuIG9wdGlvbiB5b3UgbmVlZCB0byBzZXQg YXQgY29tcGlsZSB0aW1lIGZvciB0aGlzDQogICAgPiAgICAgIGZlYXR1cmUgdG8gYmUgYWN0aXZh dGVkOg0KICAgID4gDQogICAgPiAgICAgIHN5cy9jb25mL29wdGlvbnM6SVBPSUJfQ00Jb3B0X29m ZWQuaA0KICAgID4gDQogICAgPiAgICAgIC0tSFBTDQogICAgPiANCiAgICA+IEhpIEhhbnMsDQog ICAgPiANCiAgICA+IEl0IGxvb2tzIGxpa2UgSVBPSUJfQ00gY2F1c2VzIENPTkZJR19JTkZJTklC QU5EX0lQT0lCX0NNIHRvIGJlIGRlZmluZWQuIEkgc2VlIHRoYXQgQ09ORklHX0lORklOSUJBTkRf SVBPSUJfQ00gYWRkcyBmaWVsZHMgdG8gc3RydWN0dXJlcywgd2hpY2ggaXMgd2h5IGl0IGNhbid0 IGJlIGEgcnVuLXRpbWUgb3IgYm9vdC10aW1lIHR1bmFibGUuIDotKA0KICAgID4gDQogICAgPiBC dXQgaXQgbG9va3MgbGlrZSB0aGUgY2hhbmdlcyBhc3NvY2lhdGVkIHdpdGggQ09ORklHX0lORklO SUJBTkRfSVBPSUJfQ00gYXJlIGFsbCBhZGRpdGlvbnMsIG5vdCByZXBsYWNpbmcgZGF0YWdyYW0g Y29kZSB3aXRoIGNvbm5lY3RlZC1tb2RlIGNvZGUuIE90aGVyIHRoYW4gaGF2aW5nIHVubmVjZXNz YXJ5IGZpZWxkcyBpbiBzdHJ1Y3R1cmVzLCB3b3VsZCB0aGVyZSBiZSBhbnkgaGFybSBpbiBidWls ZGluZyB3aXRoIElQT0lCX0NNLCBidXQgcnVubmluZyBpbiBkYXRhZ3JhbSBtb2RlPyBPciBhcmUg eW91IHNheWluZyB0aGF0IHdlIGZpcnN0IG5lZWQgdG8gYnVpbGQgd2l0aCBJUE9JQl9DTSwgYW5k IHRoZW4gd2UgY2FuIHVzZSBhIHR1bmFibGUgdG8gc3dpdGNoIG1vZGVzPw0KICAgID4gDQoNCiAg ICBIaSwNCg0KICAgIEF0IHRoZSBtb21lbnQgdGhlcmUgaXMgbm8gdHVuYWJsZSBmb3IgdGhpcyBm ZWF0dXJlLCBhcyBJJ20gYXdhcmUgb2YuIFRoZSANCiAgICBlYXNpZXN0IHdvdWxkIGJlIHRvIGJ1 aWxkIGlwb2liIHR3byB0aW1lcywgb25lIHRpbWUgd2l0aCBDTSBlbmFibGVkIGFuZCANCiAgICBv bmUgdGltZSB3aXRob3V0IENNIGVuYWJsZWQuDQoNCiAgICAtLUhQUw0KDQpIaSBIYW5zLA0KDQpD b3VsZG4ndCB3ZSBkbyB0aGUgZm9sbG93aW5nOg0KDQooYSkgR2V0IHJpZCBvZiBJUE9JQl9DTSBh cyBhIGNvbXBpbGUtdGltZSBvcHRpb24NCihiKSBSZW1vdmUgQ09ORklHX0lORklOSUJBTkRfSVBP SUJfQ00gZ3VhcmRzIHdoZW4gZGVmaW5pbmcgc3RydWN0dXJlcw0KKGMpIENoYW5nZSBDT05GSUdf SU5GSU5JQkFORF9JUE9JQl9DTSBndWFyZHMgb24gdGhlIGNvZGUsIHRvIGJlaW5nIGNvbmRpdGlv bmFsIG9uIGEgbmV3IHR1bmFibGUuDQoNCkl0IHNlZW1zIGxpa2UgdGhhdCB3b3VsZCBkbyB3aGF0 IEdyZWcgYW5kIEkgYXJlIGxvb2tpbmcgZm9yLg0KDQpBbHRlcm5hdGl2ZWx5LCB3ZSBjb3VsZCAq bm90KiBkbyAoYSkgb3IgKGIpLCBqdXN0IGFkZCB0aGUgY2hlY2sgZm9yIHRoZSB0dW5hYmxlIGRl c2NyaWJlZCBpbiAoYyksIGFuZCB0aGVuIGJ1aWxkIHdpdGggSVBPSUJfQ00uIFRoYXQgd291bGQg bWluaW1pemUgdGhlIGNoYW5nZSwgYW5kIHN0aWxsIGdpdmUgdXMgY29udHJvbCBvZiB0aGUgY29k ZSB2aWEgdGhlIHR1bmFibGUuDQoNCklmIGVpdGhlciBvZiB0aG9zZSBzb3VuZCBva2F5LCBJJ20g aGFwcHkgdG8gbWFrZSB0aGUgY2hhbmdlIG15c2VsZiBhbmQgcHV0IHlvdSBkb3duIGF0IHRoZSBy ZXZpZXdlciBpbiBQaGFicmljYXRvci4NCg0KVGhhbmtzLA0KDQpSYXZpDQoNCg== From owner-freebsd-infiniband@freebsd.org Fri Mar 5 08:25:35 2021 Return-Path: Delivered-To: freebsd-infiniband@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id D434E552A0D for ; Fri, 5 Mar 2021 08:25:35 +0000 (UTC) (envelope-from hps@selasky.org) Received: from mail.turbocat.net (turbocat.net [88.99.82.50]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4DsLQk31w7z3pQl for ; Fri, 5 Mar 2021 08:25:34 +0000 (UTC) (envelope-from hps@selasky.org) Received: from hps2020.home.selasky.org (unknown [178.17.145.105]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mail.turbocat.net (Postfix) with ESMTPSA id E124826044A; Fri, 5 Mar 2021 09:25:31 +0100 (CET) Subject: Re: Changing Between Infiniband Connected Mode and Datagram Mode at Runtime on FreeBSD 12.2 To: "Pokala, Ravi" , "Foster, Greg" , "freebsd-infiniband@freebsd.org" References: <08557FD0-003C-4802-A588-92298A54D4CD@panasas.com> <71b1d84d-917e-3f6a-cad6-794f06fefaee@selasky.org> <55947DF3-4697-4F44-BAAD-5A70136036D7@panasas.com> From: Hans Petter Selasky Message-ID: Date: Fri, 5 Mar 2021 09:25:15 +0100 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:78.0) Gecko/20100101 Thunderbird/78.5.0 MIME-Version: 1.0 In-Reply-To: <55947DF3-4697-4F44-BAAD-5A70136036D7@panasas.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 4DsLQk31w7z3pQl X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of hps@selasky.org designates 88.99.82.50 as permitted sender) smtp.mailfrom=hps@selasky.org X-Spamd-Result: default: False [-3.30 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCVD_TLS_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; TO_DN_SOME(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; MIME_GOOD(-0.10)[text/plain]; R_SPF_ALLOW(-0.20)[+a:mail.turbocat.net]; ARC_NA(0.00)[]; DMARC_NA(0.00)[selasky.org]; SPAMHAUS_ZRD(0.00)[88.99.82.50:from:127.0.2.255]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RBL_DBL_DONT_QUERY_IPS(0.00)[88.99.82.50:from]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.995]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:24940, ipnet:88.99.0.0/16, country:DE]; RCVD_COUNT_TWO(0.00)[2]; MAILMAN_DEST(0.00)[freebsd-infiniband] X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 05 Mar 2021 08:25:35 -0000 On 3/4/21 8:37 PM, Pokala, Ravi wrote: > -----Original Message----- > From: Hans Petter Selasky > Date: 2021-03-04, Thursday at 11:11 > To: Ravi Pokala , "Foster, Greg" , "freebsd-infiniband@freebsd.org" > Subject: Re: Changing Between Infiniband Connected Mode and Datagram Mode at Runtime on FreeBSD 12.2 > > On 3/4/21 8:00 PM, Pokala, Ravi wrote: > > -----Original Message----- > > From: Hans Petter Selasky > > Date: 2021-03-04, Thursday at 02:13 > > To: "Foster, Greg" , "freebsd-infiniband@freebsd.org" > > Cc: Ravi Pokala > > Subject: Re: Changing Between Infiniband Connected Mode and Datagram Mode at Runtime on FreeBSD 12.2 > > > > On 3/4/21 3:49 AM, Foster, Greg wrote: > > > Hello FreeBSD IB, > > > > > > What is the procedure to change between Infiniband Connected Mode and Datagram Mode > > > at runtime on FreeBSD 12.2? > > > > > > I've seen different references on the WEB, but the procedure still seems unclear. > > > > > > In one reference https://lists.freebsd.org/pipermail/freebsd-infiniband/2013-May/000032.html > > > refers to the Linux procedure, which is not applicable on FreeBSD. > > > > > > Another reference, https://wiki.freebsd.org/InfiniBand, implies the driver must be recompiled > > > to change modes. > > > > > > Yet another, https://lists.freebsd.org/pipermail/freebsd-infiniband/2013-June/000047.html > > > Suggests it can be done at runtime to add a tuneable/sysctl but doesn't specify any details... > > > > > > Is the solution to maybe compile the drivers twice, once for CM and once for DM, keeping both around > > > and loading the desired one? > > > > Hi, > > > > For IPoIB there is an option you need to set at compile time for this > > feature to be activated: > > > > sys/conf/options:IPOIB_CM opt_ofed.h > > > > --HPS > > > > Hi Hans, > > > > It looks like IPOIB_CM causes CONFIG_INFINIBAND_IPOIB_CM to be defined. I see that CONFIG_INFINIBAND_IPOIB_CM adds fields to structures, which is why it can't be a run-time or boot-time tunable. :-( > > > > But it looks like the changes associated with CONFIG_INFINIBAND_IPOIB_CM are all additions, not replacing datagram code with connected-mode code. Other than having unnecessary fields in structures, would there be any harm in building with IPOIB_CM, but running in datagram mode? Or are you saying that we first need to build with IPOIB_CM, and then we can use a tunable to switch modes? > > > > Hi, > > At the moment there is no tunable for this feature, as I'm aware of. The > easiest would be to build ipoib two times, one time with CM enabled and > one time without CM enabled. > > --HPS > > Hi Hans, > > Couldn't we do the following: > > (a) Get rid of IPOIB_CM as a compile-time option > (b) Remove CONFIG_INFINIBAND_IPOIB_CM guards when defining structures > (c) Change CONFIG_INFINIBAND_IPOIB_CM guards on the code, to being conditional on a new tunable. > > It seems like that would do what Greg and I are looking for. Hi, Sounds good to me. If you want to work on it, please go ahead and make a review. --HPS > > Alternatively, we could *not* do (a) or (b), just add the check for the tunable described in (c), and then build with IPOIB_CM. That would minimize the change, and still give us control of the code via the tunable. > > If either of those sound okay, I'm happy to make the change myself and put you down at the reviewer in Phabricator. > > Thanks, > > Ravi >