From owner-freebsd-infiniband@freebsd.org Wed Nov 9 11:55:19 2016 Return-Path: Delivered-To: freebsd-infiniband@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A6E2EC38D0E for ; Wed, 9 Nov 2016 11:55:19 +0000 (UTC) (envelope-from ferrao@vmio.com.br) Received: from na01-bn1-obe.outbound.protection.outlook.com (mail-bn1bn0109.outbound.protection.outlook.com [157.56.110.109]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "Microsoft IT SSL SHA2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 20E63310 for ; Wed, 9 Nov 2016 11:55:18 +0000 (UTC) (envelope-from ferrao@vmio.com.br) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=vmio.onmicrosoft.com; s=selector1-vmio-com-br; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=pPqeAZeucOmEbIgeC8F7iPljTX3vEvYkznubp7hSB6k=; b=SjkGrIzXoOhpJZMqPtAGow8ICZO3acWv/bCSDL2JcOuT0owiMa7EtgujHTIt1qS1w2UToPPAXNed4Ey60UGNMPWR8fQhgQ5X3eXtmPoTDzezH/MrOefZtCOPcUk6rvJqihdwkMDZws1JMnOqbNx1m8BKl+FOoF2SW4xtAL8QkS0= Received: from CP1PR80MB296.lamprd80.prod.outlook.com (10.242.254.150) by CP1PR80MB295.lamprd80.prod.outlook.com (10.242.254.149) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.693.12; Tue, 8 Nov 2016 16:27:13 +0000 Received: from CP1PR80MB296.lamprd80.prod.outlook.com ([10.242.254.150]) by CP1PR80MB296.lamprd80.prod.outlook.com ([10.242.254.150]) with mapi id 15.01.0707.006; Tue, 8 Nov 2016 16:27:13 +0000 From: =?utf-8?B?Vmluw61jaXVzIEZlcnLDo28=?= To: "freebsd-infiniband@freebsd.org" Subject: Device hints to put mlx4 on IB mode Thread-Topic: Device hints to put mlx4 on IB mode Thread-Index: AQHSOdz2Pcf6O3BBuEK/HEBs92tmQw== Date: Tue, 8 Nov 2016 16:27:13 +0000 Message-ID: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=ferrao@vmio.com.br; x-ms-exchange-messagesentrepresentingtype: 1 x-originating-ip: [200.204.163.103] x-ms-office365-filtering-correlation-id: b4a1a045-2f3a-4fd4-6652-08d407f418e2 x-microsoft-exchange-diagnostics: 1; CP1PR80MB295; 7:3Mg1KFAmuhLmKlH2FdA5EQXsF852f5KmrA5UHXPtYwM06tH68yNk2W6eXCB4BhLNJ+QVh8kI+WSDKq6HrbANsXl/4g9PFK2EYs+49nUQOu5P80Jv1LLmR2WDmGlWxTfvOYW1TEIItsywje1juEuBmZ/J/abJd/x/R5pSANCkc35uDJkLCVZ3XT/AozQHgkcSk2gw74tHNgoycjRsCidnRvgEsXmKB9hE1U+hZvsSQgTkX4nPWUjmgh5U7jH5bJ2Ugb2hTYif4ltCWisrA2krD1rx4/fYb9ZcmQ8Mv/wWm5w7aVFICa05s8bLFTH0Qo8q42zxCOgbvkZqFDVvGgDIIOjBkZuOrXhOu1U7KXyr/C4=; 23:7gWv+6OF5waig82w829tu1nuYgkJF2m3bH7B8s2DUIiDTDDsaetDCXPE2I4LKPVmOf+zYozLTRstmvQJaDUujZKNTC9EYb3IwYbwo58bsBV93EeNDYOl6u13rLEzDjZFEAGOGSDp5/TfiYtP5jDKPs4+pledvVinFzB1CDnzsDmjGjkw+dtpCwVcXeAFV+Ry x-microsoft-antispam: UriScan:;BCL:0;PCL:0;RULEID:;SRVR:CP1PR80MB295; x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:; x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(6040176)(601004)(2401047)(5005006)(8121501046)(3002001)(10201501046)(6042046)(6043046); SRVR:CP1PR80MB295; BCL:0; PCL:0; RULEID:; SRVR:CP1PR80MB295; x-forefront-prvs: 01208B1E18 x-forefront-antispam-report: SFV:NSPM; SFS:(10019020)(6009001)(7916002)(51874003)(199003)(189002)(101416001)(54356999)(229853001)(2501003)(2351001)(50986999)(305945005)(6916009)(42882006)(5660300001)(92566002)(105586002)(110136003)(74482002)(7846002)(106356001)(2900100001)(106116001)(85202003)(7736002)(36756003)(77096005)(97736004)(122556002)(107886002)(81156014)(83716003)(450100001)(189998001)(81166006)(33656002)(66066001)(8936002)(3846002)(3660700001)(5640700001)(6116002)(2906002)(68736007)(102836003)(85182001)(87936001)(82746002)(8676002)(586003)(86362001)(3280700002)(104396002); DIR:OUT; SFP:1102; SCL:1; SRVR:CP1PR80MB295; H:CP1PR80MB296.lamprd80.prod.outlook.com; FPR:; SPF:None; PTR:InfoNoRecords; A:0; MX:1; LANG:en; received-spf: None (protection.outlook.com: vmio.com.br does not designate permitted sender hosts) spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="utf-8" Content-ID: Content-Transfer-Encoding: base64 MIME-Version: 1.0 X-MS-Exchange-CrossTenant-originalarrivaltime: 08 Nov 2016 16:27:13.5556 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 3da02919-4b9d-42af-a7f7-a5975700d7c4 X-MS-Exchange-Transport-CrossTenantHeadersStamped: CP1PR80MB295 X-OriginatorOrg: vmio.com.br X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Nov 2016 11:55:19 -0000 SGVsbG8gZ3V5cywNCg0KSSB3b3VsZCBsaWtlIHRvIHB1dCBteSBDb25uZWN0LVgzIGNhcmQgaW4g SUIgbW9kZSBkdXJpbmcgYm9vdCB0aW1lIHVzaW5nIGRldmljZSBoaW50cyBpbnN0ZWFkIG9mIHN5 c2N0bCdzLg0KDQpUb2RheSB3ZSBoYXZlIHRoaXMgZGlydHkgd29ya2Fycm91bmQgb24gL2V0Yy9z eXNjdGwuY29uZjoNCnN5cy5kZXZpY2UubWx4NF9jb3JlMC5tbHg0X3BvcnQxPWliDQpzeXMuZGV2 aWNlLm1seDRfY29yZTAubWx4NF9wb3J0Mj1pYg0KDQpJdOKAmXMgZmluZSB3aGVuIHdlIHVzZSBh IENvbm5lY3QtWDMgY2FyZCB3aXRoIHR3byBJbmZpbmliYW5kIGludGVyZmFjZXMsIGJ1dCBpZiBh IENvbm5lY3RYLTMgY2FyZCB3aXRoIG9ubHkgb25lIGludGVyZmFjZSBpcyB1c2VkIGFuIGVycm9y IG1lc3NhZ2UgaXMgZHVtcGVkIG9uIHRoZSBjb25zb2xlIGFib3V0IGZhaWxpbmcgdG8gc2V0IHRo ZSAic3lzLmRldmljZS5tbHg0X2NvcmUwLm1seDRfcG9ydDI9aWLigJ0gc3lzY3RsLg0KDQpUbyBz b2x2ZSB0aGlzIEkgd291bGQgbGlrZSB0byB1c2UgZGV2aWNlIGhpbnRzIHBhc3NlZCBmcm9tIHRo ZSBHUlVCMiBib290bG9hZGVyLCBidXQgSSB3YXMgdW5hYmxlIHRvIGZpbmQgd2hpY2ggaGludHMg SSBzaG91bGQgdXNlIGZvciB0aGUgbWx4NCBkcml2ZXJzLiBJIGRvbuKAmXQgZXZlbiBrbm93IGlm IHRoaXMgaXMgc3VwcG9ydGVkLg0KDQpUaGUgcHJvYmxlbSBpcyBwdXJlbHkgY29zbWV0aWMsIEkg anVzdCB3YW50IHRvIGdldCByaWQgb2YgdGhlIGVycm9yIG1lc3NhZ2UgZHVyaW5nIHRoZSBib290 IHByb2Nlc3MuDQoNClRoYW5rcyBpbiBhZHZhbmNlLA0KVi4NCg0KDQoNCg0K From owner-freebsd-infiniband@freebsd.org Thu Nov 10 12:36:49 2016 Return-Path: Delivered-To: freebsd-infiniband@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B57D5C38678 for ; Thu, 10 Nov 2016 12:36:49 +0000 (UTC) (envelope-from menyy@mellanox.com) Received: from EUR01-VE1-obe.outbound.protection.outlook.com (mail-ve1eur01on0076.outbound.protection.outlook.com [104.47.1.76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "Microsoft IT SSL SHA2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 2C65390B for ; Thu, 10 Nov 2016 12:36:48 +0000 (UTC) (envelope-from menyy@mellanox.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Mellanox.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=wC5/7OvRUZSDzXFokcCEe8Sdufrkdq1TsS6WR/Eli5M=; b=XFHrVkB0rM3TALewwKQ5AQ8cQYUbi3gBVw61eGK/gC6fbQzGnFhwkQs3UjVrK5Zn+ajxRg5K8rCsSayguNHaTRvfk2d+MTKgYN9I6UeqsDzprTKPO67c72yyWRr66tdQcKC8LxRogDGYqV8ZO+MT0fE+00nxsLtB1s67ajCIQUU= Received: from DB5PR05MB1765.eurprd05.prod.outlook.com (10.165.7.23) by DB5PR05MB1383.eurprd05.prod.outlook.com (10.162.157.29) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.707.6; Thu, 10 Nov 2016 12:21:00 +0000 Received: from DB5PR05MB1765.eurprd05.prod.outlook.com ([10.165.7.23]) by DB5PR05MB1765.eurprd05.prod.outlook.com ([10.165.7.23]) with mapi id 15.01.0721.010; Thu, 10 Nov 2016 12:21:00 +0000 From: Meny Yossefi To: "'freebsd-infiniband@freebsd.org'" CC: Mark Bloch , Yuval Bason , "Hans Petter Selasky" Subject: RE: Device hints to put mlx4 on IB mode Thread-Topic: Device hints to put mlx4 on IB mode Thread-Index: AQHSOdz2Pcf6O3BBuEK/HEBs92tmQ6DQjJuWgAGWlvA= Date: Thu, 10 Nov 2016 12:21:00 +0000 Message-ID: References: In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=menyy@mellanox.com; x-originating-ip: [193.47.165.251] x-microsoft-exchange-diagnostics: 1; DB5PR05MB1383; 7:aa1qcNEHSwQe++Q+wASHK33YIeOh8Hmhlhe45648HNqVxoH5n0MtBmdgcZ/2sFcx5tWRCUEOEihKLnYLUytBogK5I25MMVYNTiqP/8FajHoTDv6x744p+kuQD+ldFXPvq7sXruEciApsuhFzFoGaGciuiR38wgDNLH/ga4jac6xGEz3nEPT+57dYbbJ+g6byup7J7/qwheTYdrLkKIlCj93sEAs00DSrZqETV+qBdvCfU97VTjOrFLUCPXdMWPO0rlZtQqD2liCttNXlwt9P2fo36e0rtHCDdvzB2Teu5worD3R84VfS0q/GMwdMr6k2+Nr/TOOH2XB0IuwlR4NPefpliMovJeSGEutW5WWbd6A= x-ms-office365-filtering-correlation-id: 33724786-3836-4865-7519-08d409640850 x-microsoft-antispam: UriScan:;BCL:0;PCL:0;RULEID:(22001);SRVR:DB5PR05MB1383; x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:(75325880899374); x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(6040176)(601004)(2401047)(5005006)(8121501046)(3002001)(10201501046)(6055026); SRVR:DB5PR05MB1383; BCL:0; PCL:0; RULEID:; SRVR:DB5PR05MB1383; x-forefront-prvs: 01221E3973 x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(6009001)(7916002)(199003)(189002)(13464003)(51874003)(377454003)(2906002)(81166006)(3846002)(102836003)(106356001)(81156014)(66066001)(9686002)(106116001)(6116002)(8676002)(77096005)(2900100001)(122556002)(97736004)(105586002)(450100001)(7736002)(4326007)(68736007)(305945005)(86362001)(8936002)(76576001)(7846002)(74316002)(54356999)(50986999)(76176999)(586003)(101416001)(107886002)(110136003)(189998001)(4001430100002)(2950100002)(92566002)(87936001)(6916009)(3900700001)(5660300001)(33656002)(229853002)(7696004)(3660700001)(3280700002)(491001); DIR:OUT; SFP:1101; SCL:1; SRVR:DB5PR05MB1383; H:DB5PR05MB1765.eurprd05.prod.outlook.com; FPR:; SPF:None; PTR:InfoNoRecords; A:1; MX:1; LANG:en; received-spf: None (protection.outlook.com: mellanox.com does not designate permitted sender hosts) spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-originalarrivaltime: 10 Nov 2016 12:21:00.3590 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: a652971c-7d2e-4d9b-a6a4-d149256f461b X-MS-Exchange-Transport-CrossTenantHeadersStamped: DB5PR05MB1383 X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 Nov 2016 12:36:49 -0000 Hi,=20 I'm not familiar of any hints passed from the bootloader. We do have some FW tools to identify number of NIC ports, but I guess that = won't serve your purpose. The only option I can think of at the moment is to force all ports to come = up in IB mode. Try using this as reference:=20 sys/ofed/drivers/net/mlx4/main.c: static struct param_data port_type_array =3D { .dbdf2val =3D { .name =3D "port_type_array param", .num_vals =3D 2, - .def_val =3D {MLX4_PORT_TYPE_NONE, MLX4_PORT_TYPE_NONE}, + .def_val =3D {MLX4_PORT_TYPE_IB, MLX4_PORT_TYPE_IB}, .range =3D {MLX4_PORT_TYPE_IB, MLX4_PORT_TYPE_NA} } }; @@ -759,6 +759,9 @@ static int mlx4_dev_cap(struct mlx4_dev *dev, struct ml= x4_dev_cap *dev_cap) (dev->caps.flags & MLX4_DEV_CAP_FLAG_DPDP) && (dev->caps.flags & MLX4_DEV_CAP_FLAG_SENSE_SUPPORT= )); + /* MY: disablling auto sense for default IB/Eth ports support */ + mlx4_priv(dev)->sense.sense_allowed[i] =3D 0; + /* * If "default_sense" bit is set, we move the port to "AUTO= " mode * and perform sense_port FW command to try and set the cor= rect -Meny -----Original Message----- From: freebsd-commits-tracker=20 Sent: Wednesday, November 09, 2016 1:55 PM To: Mark Bloch ; Meny Yossefi ; Yuv= al Bason Subject: FW: Device hints to put mlx4 on IB mode ________________________________________ From: owner-freebsd-infiniband@freebsd.orgOn Behalf OfVin=EDcius Ferr=E3o Sent: Tuesday, November 8, 2016 4:27:13 PM (UTC+00:00) Monrovia, Reykjavik To: freebsd-infiniband@freebsd.org Subject: Device hints to put mlx4 on IB mode Hello guys, I would like to put my Connect-X3 card in IB mode during boot time using de= vice hints instead of sysctl's. Today we have this dirty workarround on /etc/sysctl.conf: sys.device.mlx4_core0.mlx4_port1=3Dib sys.device.mlx4_core0.mlx4_port2=3Dib It's fine when we use a Connect-X3 card with two Infiniband interfaces, but= if a ConnectX-3 card with only one interface is used an error message is d= umped on the console about failing to set the "sys.device.mlx4_core0.mlx4_p= ort2=3Dib" sysctl. To solve this I would like to use device hints passed from the GRUB2 bootlo= ader, but I was unable to find which hints I should use for the mlx4 driver= s. I don't even know if this is supported. The problem is purely cosmetic, I just want to get rid of the error message= during the boot process. Thanks in advance, V. _______________________________________________ freebsd-infiniband@freebsd.org mailing list https://lists.freebsd.org/mailm= an/listinfo/freebsd-infiniband To unsubscribe, send any mail to "freebsd-infiniband-unsubscribe@freebsd.or= g" From owner-freebsd-infiniband@freebsd.org Thu Nov 10 16:11:59 2016 Return-Path: Delivered-To: freebsd-infiniband@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7C1CCC3AECE for ; Thu, 10 Nov 2016 16:11:59 +0000 (UTC) (envelope-from ferrao@vmio.com.br) Received: from na01-by2-obe.outbound.protection.outlook.com (mail-by2on0135.outbound.protection.outlook.com [207.46.100.135]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "Microsoft IT SSL SHA2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id C6093799 for ; Thu, 10 Nov 2016 16:11:57 +0000 (UTC) (envelope-from ferrao@vmio.com.br) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=vmio.onmicrosoft.com; s=selector1-vmio-com-br; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=tHkN2le1a4yvKzYovM2TcsOLiC4AnZNvZ3BV5SRi3wg=; b=Gpvmjww9e9pQxUcVi9csQsaJ87bYPeQ+fWpxyqk/7hDpRgyPBhAQCs4e1hY5LFdEz/VRsmUfj9GjNsLTR7SG8s1Y7g48jsY/JqC7dU/8CkcqYM1nQQrKKd7R44DVYIyA6Gpe5l3pVIfVaAwUtioPvxyWdVZchF6wMkwoepm3ZE8= Received: from CP1PR80MB296.lamprd80.prod.outlook.com (10.242.254.150) by CP1PR80MB296.lamprd80.prod.outlook.com (10.242.254.150) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.707.6; Thu, 10 Nov 2016 15:39:47 +0000 Received: from CP1PR80MB296.lamprd80.prod.outlook.com ([10.242.254.150]) by CP1PR80MB296.lamprd80.prod.outlook.com ([10.242.254.150]) with mapi id 15.01.0707.006; Thu, 10 Nov 2016 15:39:47 +0000 From: =?utf-8?B?Vmluw61jaXVzIEZlcnLDo28=?= To: Meny Yossefi CC: "freebsd-infiniband@freebsd.org" , "Mark Bloch" , Yuval Bason , "Hans Petter Selasky" Subject: Re: Device hints to put mlx4 on IB mode Thread-Topic: Device hints to put mlx4 on IB mode Thread-Index: AQHSOdz2HTqzbjwZbEKpjBOuZLYEPaDQjJuWgAGWlvCAADplgA== Date: Thu, 10 Nov 2016 15:39:46 +0000 Message-ID: References: In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=ferrao@vmio.com.br; x-ms-exchange-messagesentrepresentingtype: 1 x-originating-ip: [146.164.112.10] x-microsoft-exchange-diagnostics: 1; CP1PR80MB296; 7:Wtk/5hyYvuG2eB6Z50koNWu/o6RNiX/SzyYqeORfRWh66rd4YzYO5C/a/UNVrKE1Nm227Ny5KwdkiId+17SuLwVmrX7LLFJS/tqSpkfh7bcVMt49KFjXZd04gGTMYRaBG2BWH/cr3eHd+LRiHdV0NKQwoT/H5HHOYhXqbQoPSVT4Z06fi074LOSoJ2Qz63kd8hMlVnlxT/ApENDh0qS/h6rFKD4hRqs6UYHExFuBflU48q3Sbmle0RCA51OAVLlI9smKN8GtD9xf9ppH+i5dv7UFBg5yy/WJDKSvTbPBBAQE6p0+TpQtnfyLpp+4MOJosk1Kb9YFx4+xfZVSN601kQL+99kcgGwOIrPCBkB3kMA= x-ms-office365-filtering-correlation-id: 9f0c6fb0-eac5-42f7-b4b7-08d4097fcd08 x-microsoft-antispam: UriScan:;BCL:0;PCL:0;RULEID:(22001);SRVR:CP1PR80MB296; x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:(75325880899374); x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(6040176)(601004)(2401047)(8121501046)(5005006)(3002001)(10201501046)(6042046)(6043046); SRVR:CP1PR80MB296; BCL:0; PCL:0; RULEID:; SRVR:CP1PR80MB296; x-forefront-prvs: 01221E3973 x-forefront-antispam-report: SFV:NSPM; SFS:(10019020)(6009001)(7916002)(189002)(199003)(377454003)(51874003)(13464003)(24454002)(51914003)(2906002)(36756003)(122556002)(97736004)(110136003)(2900100001)(5660300001)(77096005)(92566002)(229853002)(66066001)(87936001)(83716003)(85182001)(3280700002)(101416001)(50986999)(54356999)(82746002)(74482002)(76176999)(8936002)(8676002)(33656002)(6916009)(42882006)(86362001)(2950100002)(305945005)(68736007)(6116002)(586003)(102836003)(81166006)(81156014)(3846002)(4326007)(189998001)(106356001)(106116001)(85202003)(3660700001)(7846002)(7736002)(105586002)(3900700001)(104396002); DIR:OUT; SFP:1102; SCL:1; SRVR:CP1PR80MB296; H:CP1PR80MB296.lamprd80.prod.outlook.com; FPR:; SPF:None; PTR:InfoNoRecords; MX:1; A:0; LANG:en; received-spf: None (protection.outlook.com: vmio.com.br does not designate permitted sender hosts) spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="utf-8" Content-ID: Content-Transfer-Encoding: base64 MIME-Version: 1.0 X-OriginatorOrg: vmio.com.br X-MS-Exchange-CrossTenant-originalarrivaltime: 10 Nov 2016 15:39:46.8146 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 3da02919-4b9d-42af-a7f7-a5975700d7c4 X-MS-Exchange-Transport-CrossTenantHeadersStamped: CP1PR80MB296 X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 Nov 2016 16:11:59 -0000 SGVsbG8gTWVueSwNCg0KVGhhbmtzIGZvciB0aGUgaW5wdXQuIFlvdeKAmXJlIHJlY29tbWVuZGlu ZyBhIGRyaXZlciBjb2RlIGNoYW5nZSB0byBzdGFydCBhbHdheXMgb24gSUIgbW9kZS4gVGhpcyBh cHBlYXJzIHRvIGJlIE9LLCBidXQgaXQncyBsaXR0bGUgZGFuZ2Vyb3VzLiBJIHdpbGwgbmVlZCB0 byBrZWVwIHRoZSBwYXRjaGVzIGZvciB0aGUgZHJpdmVycy9tb2R1bGVzIGlmIGFueXRoaW5nIGNo YW5nZXMsIHJpZ2h0Pw0KDQpBbm90aGVyIHF1ZXN0aW9uLCBjaGFuZ2luZyB0aGUgc3RhcnRpbmcg bW9kZSBjYW5ub3QgYmUgZG9uZSBpbiBvdGhlciBwbGFjZT8gTGlrZSB0aGUgQ29ubmVjdC1YMyBm aXJtd2FyZT8gSWYgbm90LCBzaW5jZSB5b3XigJlyZSBvbiBNZWxsYW5veCBjYW4gd2UgcmVjb21t ZW5kIGEgYmV0dGVyIGFwcHJvYWNoIGZyb20gdGhlIHVwc3RyZWFtIChtbHgpIG9yIHRoaXMgd291 bGQgYmUgaW52aWFibGU/DQoNClRoYW5rcyBmb3IgeW91ciBhdHRlbnRpb24gYW5kIHRpbWUsIGl0 IHdhcyByZWFsbHkgaGVscGZ1bC4NCg0KVi4NCg0KPiBPbiAxMCBOb3YgMjAxNiwgYXQgMTA6MjEs IE1lbnkgWW9zc2VmaSA8bWVueXlAbWVsbGFub3guY29tPiB3cm90ZToNCj4gDQo+IEhpLCANCj4g DQo+IEknbSBub3QgZmFtaWxpYXIgb2YgYW55IGhpbnRzIHBhc3NlZCBmcm9tIHRoZSBib290bG9h ZGVyLg0KPiANCj4gV2UgZG8gaGF2ZSBzb21lIEZXIHRvb2xzIHRvIGlkZW50aWZ5IG51bWJlciBv ZiBOSUMgcG9ydHMsIGJ1dCBJIGd1ZXNzIHRoYXQgd29uJ3Qgc2VydmUgeW91ciBwdXJwb3NlLg0K PiANCj4gVGhlIG9ubHkgb3B0aW9uIEkgY2FuIHRoaW5rIG9mIGF0IHRoZSBtb21lbnQgaXMgdG8g Zm9yY2UgYWxsIHBvcnRzIHRvIGNvbWUgdXAgaW4gSUIgbW9kZS4NCj4gDQo+IFRyeSB1c2luZyB0 aGlzIGFzIHJlZmVyZW5jZTogDQo+IA0KPiBzeXMvb2ZlZC9kcml2ZXJzL25ldC9tbHg0L21haW4u YzoNCj4gDQo+IA0KPiBzdGF0aWMgc3RydWN0IHBhcmFtX2RhdGEgcG9ydF90eXBlX2FycmF5ID0g ew0KPiAgICAgICAgLmRiZGYydmFsID0gew0KPiAgICAgICAgICAgICAgICAubmFtZSAgICAgICAg ICAgPSAicG9ydF90eXBlX2FycmF5IHBhcmFtIiwNCj4gICAgICAgICAgICAgICAgLm51bV92YWxz ICAgICAgID0gMiwNCj4gLSAgICAgICAgICAgLmRlZl92YWwgICAgICAgID0ge01MWDRfUE9SVF9U WVBFX05PTkUsIE1MWDRfUE9SVF9UWVBFX05PTkV9LA0KPiArICAgICAgICAgLmRlZl92YWwgICAg PSB7TUxYNF9QT1JUX1RZUEVfSUIsIE1MWDRfUE9SVF9UWVBFX0lCfSwNCj4gICAgICAgICAgICAg ICAgLnJhbmdlICAgICAgICAgID0ge01MWDRfUE9SVF9UWVBFX0lCLCBNTFg0X1BPUlRfVFlQRV9O QX0NCj4gICAgICAgIH0NCj4gfTsNCj4gQEAgLTc1OSw2ICs3NTksOSBAQCBzdGF0aWMgaW50IG1s eDRfZGV2X2NhcChzdHJ1Y3QgbWx4NF9kZXYgKmRldiwgc3RydWN0IG1seDRfZGV2X2NhcCAqZGV2 X2NhcCkNCj4gICAgICAgICAgICAgICAgICAgICAgICAgKGRldi0+Y2Fwcy5mbGFncyAmIE1MWDRf REVWX0NBUF9GTEFHX0RQRFApICYmDQo+ICAgICAgICAgICAgICAgICAgICAgICAgIChkZXYtPmNh cHMuZmxhZ3MgJiBNTFg0X0RFVl9DQVBfRkxBR19TRU5TRV9TVVBQT1JUKSk7DQo+IA0KPiArICAg ICAgICAgLyogTVk6IGRpc2FibGxpbmcgYXV0byBzZW5zZSBmb3IgZGVmYXVsdCBJQi9FdGggcG9y dHMgc3VwcG9ydCAqLw0KPiArICAgICAgICAgbWx4NF9wcml2KGRldiktPnNlbnNlLnNlbnNlX2Fs bG93ZWRbaV0gPSAwOw0KPiArDQo+ICAgICAgICAgICAgICAgIC8qDQo+ICAgICAgICAgICAgICAg ICAqIElmICJkZWZhdWx0X3NlbnNlIiBiaXQgaXMgc2V0LCB3ZSBtb3ZlIHRoZSBwb3J0IHRvICJB VVRPIiBtb2RlDQo+ICAgICAgICAgICAgICAgICAqIGFuZCBwZXJmb3JtIHNlbnNlX3BvcnQgRlcg Y29tbWFuZCB0byB0cnkgYW5kIHNldCB0aGUgY29ycmVjdA0KPiANCj4gLU1lbnkNCj4gDQo+IA0K PiAtLS0tLU9yaWdpbmFsIE1lc3NhZ2UtLS0tLQ0KPiBGcm9tOiBmcmVlYnNkLWNvbW1pdHMtdHJh Y2tlciANCj4gU2VudDogV2VkbmVzZGF5LCBOb3ZlbWJlciAwOSwgMjAxNiAxOjU1IFBNDQo+IFRv OiBNYXJrIEJsb2NoIDxtYXJrYkBtZWxsYW5veC5jb20+OyBNZW55IFlvc3NlZmkgPG1lbnl5QG1l bGxhbm94LmNvbT47IFl1dmFsIEJhc29uIDx5dXZhbGJhQG1lbGxhbm94LmNvbT4NCj4gU3ViamVj dDogRlc6IERldmljZSBoaW50cyB0byBwdXQgbWx4NCBvbiBJQiBtb2RlDQo+IA0KPiANCj4gDQo+ IA0KPiBfX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fDQo+IEZyb206IG93 bmVyLWZyZWVic2QtaW5maW5pYmFuZEBmcmVlYnNkLm9yZ09uIEJlaGFsZiBPZlZpbsOtY2l1cyBG ZXJyw6NvDQo+IFNlbnQ6IFR1ZXNkYXksIE5vdmVtYmVyIDgsIDIwMTYgNDoyNzoxMyBQTSAoVVRD KzAwOjAwKSBNb25yb3ZpYSwgUmV5a2phdmlrDQo+IFRvOiBmcmVlYnNkLWluZmluaWJhbmRAZnJl ZWJzZC5vcmcNCj4gU3ViamVjdDogRGV2aWNlIGhpbnRzIHRvIHB1dCBtbHg0IG9uIElCIG1vZGUN Cj4gDQo+IEhlbGxvIGd1eXMsDQo+IA0KPiBJIHdvdWxkIGxpa2UgdG8gcHV0IG15IENvbm5lY3Qt WDMgY2FyZCBpbiBJQiBtb2RlIGR1cmluZyBib290IHRpbWUgdXNpbmcgZGV2aWNlIGhpbnRzIGlu c3RlYWQgb2Ygc3lzY3RsJ3MuDQo+IA0KPiBUb2RheSB3ZSBoYXZlIHRoaXMgZGlydHkgd29ya2Fy cm91bmQgb24gL2V0Yy9zeXNjdGwuY29uZjoNCj4gc3lzLmRldmljZS5tbHg0X2NvcmUwLm1seDRf cG9ydDE9aWINCj4gc3lzLmRldmljZS5tbHg0X2NvcmUwLm1seDRfcG9ydDI9aWINCj4gDQo+IEl0 J3MgZmluZSB3aGVuIHdlIHVzZSBhIENvbm5lY3QtWDMgY2FyZCB3aXRoIHR3byBJbmZpbmliYW5k IGludGVyZmFjZXMsIGJ1dCBpZiBhIENvbm5lY3RYLTMgY2FyZCB3aXRoIG9ubHkgb25lIGludGVy ZmFjZSBpcyB1c2VkIGFuIGVycm9yIG1lc3NhZ2UgaXMgZHVtcGVkIG9uIHRoZSBjb25zb2xlIGFi b3V0IGZhaWxpbmcgdG8gc2V0IHRoZSAic3lzLmRldmljZS5tbHg0X2NvcmUwLm1seDRfcG9ydDI9 aWIiIHN5c2N0bC4NCj4gDQo+IFRvIHNvbHZlIHRoaXMgSSB3b3VsZCBsaWtlIHRvIHVzZSBkZXZp Y2UgaGludHMgcGFzc2VkIGZyb20gdGhlIEdSVUIyIGJvb3Rsb2FkZXIsIGJ1dCBJIHdhcyB1bmFi bGUgdG8gZmluZCB3aGljaCBoaW50cyBJIHNob3VsZCB1c2UgZm9yIHRoZSBtbHg0IGRyaXZlcnMu IEkgZG9uJ3QgZXZlbiBrbm93IGlmIHRoaXMgaXMgc3VwcG9ydGVkLg0KPiANCj4gVGhlIHByb2Js ZW0gaXMgcHVyZWx5IGNvc21ldGljLCBJIGp1c3Qgd2FudCB0byBnZXQgcmlkIG9mIHRoZSBlcnJv ciBtZXNzYWdlIGR1cmluZyB0aGUgYm9vdCBwcm9jZXNzLg0KPiANCj4gVGhhbmtzIGluIGFkdmFu Y2UsDQo+IFYuDQo+IA0KPiANCj4gDQo+IA0KPiBfX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fXw0KPiBmcmVlYnNkLWluZmluaWJhbmRAZnJlZWJzZC5vcmcgbWFp bGluZyBsaXN0IGh0dHBzOi8vbGlzdHMuZnJlZWJzZC5vcmcvbWFpbG1hbi9saXN0aW5mby9mcmVl YnNkLWluZmluaWJhbmQNCj4gVG8gdW5zdWJzY3JpYmUsIHNlbmQgYW55IG1haWwgdG8gImZyZWVi c2QtaW5maW5pYmFuZC11bnN1YnNjcmliZUBmcmVlYnNkLm9yZyINCj4gX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18NCj4gZnJlZWJzZC1pbmZpbmliYW5kQGZy ZWVic2Qub3JnIG1haWxpbmcgbGlzdA0KPiBodHRwczovL2xpc3RzLmZyZWVic2Qub3JnL21haWxt YW4vbGlzdGluZm8vZnJlZWJzZC1pbmZpbmliYW5kDQo+IFRvIHVuc3Vic2NyaWJlLCBzZW5kIGFu eSBtYWlsIHRvICJmcmVlYnNkLWluZmluaWJhbmQtdW5zdWJzY3JpYmVAZnJlZWJzZC5vcmciDQoN Cg== From owner-freebsd-infiniband@freebsd.org Thu Nov 10 16:20:28 2016 Return-Path: Delivered-To: freebsd-infiniband@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D4AC2C3A231; Thu, 10 Nov 2016 16:20:28 +0000 (UTC) (envelope-from justin@postgresql.org) Received: from meldrar.postgresql.org (meldrar.postgresql.org [IPv6:2a02:c0:301:0:ffff::31]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "*.postgresql.org", Issuer "Gandi Standard SSL CA 2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 99521CF2; Thu, 10 Nov 2016 16:20:28 +0000 (UTC) (envelope-from justin@postgresql.org) Received: from meldrar.postgresql.org ([87.238.57.231] helo=webmail.postgresql.org) by meldrar.postgresql.org with esmtpsa (TLS1.0:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.84_2) (envelope-from ) id 1c4s5N-0002Qs-IF; Thu, 10 Nov 2016 16:20:23 +0000 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit Date: Thu, 10 Nov 2016 16:20:20 +0000 From: Justin Clift To: =?UTF-8?Q?Vin=C3=ADcius_Ferr=C3=A3o?= Cc: Meny Yossefi , Mark Bloch , freebsd-infiniband@freebsd.org, Yuval Bason , Hans Petter Selasky , owner-freebsd-infiniband@freebsd.org Subject: Re: Device hints to put mlx4 on IB mode In-Reply-To: References: Message-ID: <20872ef5058b19aa9793734d82ef3396@postgresql.org> X-Sender: justin@postgresql.org User-Agent: Roundcube Webmail/1.1.5 X-Pg-Spam-Score: -5.8 (-----) X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 Nov 2016 16:20:28 -0000 On 2016-11-10 15:39, Vinícius Ferrão wrote: > Hello Meny, > > Thanks for the input. You’re recommending a driver code change to > start always on IB mode. This appears to be OK, but it's little > dangerous. I will need to keep the patches for the drivers/modules if > anything changes, right? > > Another question, changing the starting mode cannot be done in other > place? Like the Connect-X3 firmware? If not, since you’re on Mellanox > can we recommend a better approach from the upstream (mlx) or this > would be inviable? > > Thanks for your attention and time, it was really helpful. As a thought, are you using a config mgmt system? (eg Ansible, Salt, Puppet, etc) If so, that would be able to set this correctly (via sysctl) for all the nodes. :) + Justin From owner-freebsd-infiniband@freebsd.org Thu Nov 10 21:55:22 2016 Return-Path: Delivered-To: freebsd-infiniband@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C105AC3ADEA; Thu, 10 Nov 2016 21:55:22 +0000 (UTC) (envelope-from ferrao@vmio.com.br) Received: from na01-bn1-obe.outbound.protection.outlook.com (mail-bn1on0141.outbound.protection.outlook.com [157.56.110.141]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "Microsoft IT SSL SHA2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 1305D866; Thu, 10 Nov 2016 21:55:21 +0000 (UTC) (envelope-from ferrao@vmio.com.br) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=vmio.onmicrosoft.com; s=selector1-vmio-com-br; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=wG0MqXu6+/Z13L7JYy5eNs9oBOs1II7l4DTv9cSDGdQ=; b=FCicpyFJNt5yKoPTCzBXUkirigiUjY7uttbc5Vx2j8BBOxfRQf4ntm0NPyPdReq83xhtcKFcPRdGQ44gAtWEbafEsWaoJ4hEpGnLR80OiO/lH2/Qb3fMGCNSrwMVUN8PSkz0Gw8I6v8VAOzALNIwvJX0DK74jiTeHaPYJlw35qQ= Received: from CP1PR80MB296.lamprd80.prod.outlook.com (10.242.254.150) by CP1PR80MB294.lamprd80.prod.outlook.com (10.242.254.148) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.693.12; Thu, 10 Nov 2016 18:22:08 +0000 Received: from CP1PR80MB296.lamprd80.prod.outlook.com ([10.242.254.150]) by CP1PR80MB296.lamprd80.prod.outlook.com ([10.242.254.150]) with mapi id 15.01.0707.006; Thu, 10 Nov 2016 18:22:08 +0000 From: =?utf-8?B?Vmluw61jaXVzIEZlcnLDo28=?= To: Justin Clift CC: Meny Yossefi , Mark Bloch , "freebsd-infiniband@freebsd.org" , "Yuval Bason" , Hans Petter Selasky , "owner-freebsd-infiniband@freebsd.org" Subject: Re: Device hints to put mlx4 on IB mode Thread-Topic: Device hints to put mlx4 on IB mode Thread-Index: AQHSOdz2HTqzbjwZbEKpjBOuZLYEPaDQjJuWgAGWlvCAADplgIAAC1kAgAAiBQA= Date: Thu, 10 Nov 2016 18:22:07 +0000 Message-ID: References: <20872ef5058b19aa9793734d82ef3396@postgresql.org> In-Reply-To: <20872ef5058b19aa9793734d82ef3396@postgresql.org> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=ferrao@vmio.com.br; x-ms-exchange-messagesentrepresentingtype: 1 x-originating-ip: [146.164.112.10] x-microsoft-exchange-diagnostics: 1; CP1PR80MB294; 7:af/JbkWVW/ZB1EYaQW4pLsx9ttpeOXumyo0g2oLjSSctgCojUf6zG+A3BNAqSrJPjluC0MB6GreUJAwMYxW+UKvOqdiDQMBpSME1CtVC1gl6WGcFSYLNLcCx2NTmhDpzyYWaq/kpgkCTcMqAeoVkjNeKOyfrgXBcr6SZdmJtKB6I/o0IaeUg9FW+tI+Y0f8wqfDaOw9Djlm7yUQvr63V+yPSkLVOFFIk5cXWvlnewJsYHjZBdwErNE47hIFELAmfoObB5lvKEJ1weriP1YkAYEdZBCzvP//rsRqNW6XLnTjqHc5zItzWJlGX9XpNJDk36Z6O/Z2dAboClcJ01DbmoQJGb8sGZgiCQxsavpC5Dqs= x-ms-office365-filtering-correlation-id: e5600391-f641-47b5-df4c-08d409967b2e x-microsoft-antispam: UriScan:;BCL:0;PCL:0;RULEID:(22001);SRVR:CP1PR80MB294; x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:; x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(6040176)(601004)(2401047)(5005006)(8121501046)(10201501046)(3002001)(6043046)(6042046); SRVR:CP1PR80MB294; BCL:0; PCL:0; RULEID:; SRVR:CP1PR80MB294; x-forefront-prvs: 01221E3973 x-forefront-antispam-report: SFV:NSPM; SFS:(10019020)(6009001)(7916002)(199003)(24454002)(51914003)(377424004)(189002)(81156014)(8676002)(81166006)(97736004)(33656002)(92566002)(54356999)(76176999)(50986999)(105586002)(2900100001)(106356001)(4001150100001)(77096005)(7736002)(36756003)(85202003)(3660700001)(7846002)(101416001)(3280700002)(106116001)(5660300001)(305945005)(8936002)(2950100002)(4326007)(110136003)(42882006)(66066001)(6916009)(82746002)(2906002)(86362001)(68736007)(85182001)(102836003)(122556002)(6116002)(189998001)(74482002)(93886004)(586003)(229853002)(83716003)(87936001)(3846002)(104396002); DIR:OUT; SFP:1102; SCL:1; SRVR:CP1PR80MB294; H:CP1PR80MB296.lamprd80.prod.outlook.com; FPR:; SPF:None; PTR:InfoNoRecords; A:0; MX:1; LANG:en; received-spf: None (protection.outlook.com: vmio.com.br does not designate permitted sender hosts) spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="utf-8" Content-ID: <5CD4AA4E4E9A1246AA6DA0FCC8A1A1F7@lamprd80.prod.outlook.com> Content-Transfer-Encoding: base64 MIME-Version: 1.0 X-OriginatorOrg: vmio.com.br X-MS-Exchange-CrossTenant-originalarrivaltime: 10 Nov 2016 18:22:07.7382 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 3da02919-4b9d-42af-a7f7-a5975700d7c4 X-MS-Exchange-Transport-CrossTenantHeadersStamped: CP1PR80MB294 X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 Nov 2016 21:55:22 -0000 SGVsbG8gSnVzdGluLA0KDQpJ4oCZbSBhY3R1YWxseSBidWlsZGluZyBhIHNlcnZlci1vbmx5IGFw cGxpYW5jZSBiYXNlZCBvbiBJbmZpbmliYW5kIG9uIElCIE1vZGUgZm9yIHN0b3JhZ2UgbmV0d29y ay4gU28gdGhlIGNvbXB1dGUgbm9kZXMgaXNu4oCZdCBwYXJ0IG9mIHRoZSBwcm9ibGVtLg0KDQpT byB0aGUgcHJvYmxlbSBoZXJlIGlzIGNvc21ldGljYWwsIGFzIEkgc2FpZC4gRHVtcGluZyBlcnJv ciBtZXNzYWdlcyBldmVyeSBib290IGlzIGp1c3QgdWdseS4NCg0KVi4NCg0KPiBPbiAxMCBOb3Yg MjAxNiwgYXQgMTQ6MjAsIEp1c3RpbiBDbGlmdCA8anVzdGluQHBvc3RncmVzcWwub3JnPiB3cm90 ZToNCj4gDQo+IE9uIDIwMTYtMTEtMTAgMTU6MzksIFZpbsOtY2l1cyBGZXJyw6NvIHdyb3RlOg0K Pj4gSGVsbG8gTWVueSwNCj4+IFRoYW5rcyBmb3IgdGhlIGlucHV0LiBZb3XigJlyZSByZWNvbW1l bmRpbmcgYSBkcml2ZXIgY29kZSBjaGFuZ2UgdG8NCj4+IHN0YXJ0IGFsd2F5cyBvbiBJQiBtb2Rl LiBUaGlzIGFwcGVhcnMgdG8gYmUgT0ssIGJ1dCBpdCdzIGxpdHRsZQ0KPj4gZGFuZ2Vyb3VzLiBJ IHdpbGwgbmVlZCB0byBrZWVwIHRoZSBwYXRjaGVzIGZvciB0aGUgZHJpdmVycy9tb2R1bGVzIGlm DQo+PiBhbnl0aGluZyBjaGFuZ2VzLCByaWdodD8NCj4+IEFub3RoZXIgcXVlc3Rpb24sIGNoYW5n aW5nIHRoZSBzdGFydGluZyBtb2RlIGNhbm5vdCBiZSBkb25lIGluIG90aGVyDQo+PiBwbGFjZT8g TGlrZSB0aGUgQ29ubmVjdC1YMyBmaXJtd2FyZT8gSWYgbm90LCBzaW5jZSB5b3XigJlyZSBvbiBN ZWxsYW5veA0KPj4gY2FuIHdlIHJlY29tbWVuZCBhIGJldHRlciBhcHByb2FjaCBmcm9tIHRoZSB1 cHN0cmVhbSAobWx4KSBvciB0aGlzDQo+PiB3b3VsZCBiZSBpbnZpYWJsZT8NCj4+IFRoYW5rcyBm b3IgeW91ciBhdHRlbnRpb24gYW5kIHRpbWUsIGl0IHdhcyByZWFsbHkgaGVscGZ1bC4NCj4gDQo+ IEFzIGEgdGhvdWdodCwgYXJlIHlvdSB1c2luZyBhIGNvbmZpZyBtZ210IHN5c3RlbT8gKGVnIEFu c2libGUsIFNhbHQsIFB1cHBldCwgZXRjKQ0KPiANCj4gSWYgc28sIHRoYXQgd291bGQgYmUgYWJs ZSB0byBzZXQgdGhpcyBjb3JyZWN0bHkgKHZpYSBzeXNjdGwpIGZvciBhbGwgdGhlIG5vZGVzLiA6 KQ0KPiANCj4gKyBKdXN0aW4NCg0K