From owner-freebsd-drivers@freebsd.org Fri Aug 25 03:16:51 2017 Return-Path: Delivered-To: freebsd-drivers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id CD49CDEE07A for ; Fri, 25 Aug 2017 03:16:51 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-it0-x22c.google.com (mail-it0-x22c.google.com [IPv6:2607:f8b0:4001:c0b::22c]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 957917099E for ; Fri, 25 Aug 2017 03:16:51 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: by mail-it0-x22c.google.com with SMTP id b76so2073171itb.0 for ; Thu, 24 Aug 2017 20:16:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20150623.gappssmtp.com; s=20150623; h=mime-version:sender:in-reply-to:references:from:date:message-id :subject:to:cc; bh=Qg4b8HSke2XCoOhG+thuQ+3yB/Kft6eVumAzSO5sBfw=; b=tRPEKSVB/11BuEn+UYSf5kkbUb0HE6z4TuCIPCkBksdGVvHmJ9voVbTgWxOGqa8Omp fA9RrMA8nsHklLW5GmAgkCdmndcjvsZ8jnJao/xia77Sdj6ha0KlkYLElG7plua2qju9 h7nSxNoMvFBaPahXIi/RAXZu4s2E0mIARBIm2uggpEKjHcTdkNNd+VO02hLQfBT55NTM qvTQn949Y71nyKRPt4DadmIPccsjbrY0dKo+rjPlbIvy8FdhRWhOXFLpfUEOCxL0Zl/K wl+1cpx0x8B4Nbq1gQvBCPDHR0zbF6GY/F+5eXp4oMFe0harFkLddFyrHzvD6JhpWH+z IT0A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:sender:in-reply-to:references:from :date:message-id:subject:to:cc; bh=Qg4b8HSke2XCoOhG+thuQ+3yB/Kft6eVumAzSO5sBfw=; b=IivO7fldSnZDhPO3CrIoyYnTM6wAhksIQw9NXbuuY4EdavAhCgVn6AHY3qBB8cHdms iJGdofkVZ998ObIcH23MU59rzcqoZT2hkv8AMsN86SpcIxxTbOQRaeKzmTnLfkFCvTt2 6O8M6IBKp9HgZRfZ14y1rJSccF4TxQieS7L31knJcFflnrdHiLkLl0igcVOG/pdnH5Wa eSl+VcNniGsm9rDA12f+ioIeyD0tZ2ydN6VhOp6sR610VImbUK2vkIB1Hx12fGgKugLK TmKJVVNFU/mTCyk9r3BFFvT0IEddQX+QCJywcLoVpTWeXXXI1E75R0vdzZNzDTsefQtI k0Fg== X-Gm-Message-State: AHYfb5hUQP9wVESO2UofahymFqr96koYCcOJoaSaluRvb3GPf5tRO6Su poTtBoeJZmJ0+6XmiRSiTSBbg4NcWS40 X-Received: by 10.36.65.21 with SMTP id x21mr780898ita.24.1503631010879; Thu, 24 Aug 2017 20:16:50 -0700 (PDT) MIME-Version: 1.0 Sender: wlosh@bsdimp.com Received: by 10.79.10.71 with HTTP; Thu, 24 Aug 2017 20:16:50 -0700 (PDT) X-Originating-IP: [2603:300b:6:5100:9c7e:5904:e863:717e] In-Reply-To: References: From: Warner Losh Date: Thu, 24 Aug 2017 21:16:50 -0600 X-Google-Sender-Auth: XWAVkV4nDQVfFVqFgPTaH-Pp_Bs Message-ID: Subject: Re: NVMe driver init sequence To: Jim Harris Cc: Kinjal Patel , "freebsd-drivers@freebsd.org" Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-drivers@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Writing device drivers for FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 25 Aug 2017 03:16:51 -0000 Greetings, I've just committed r322872 which implements this, with a slightly tweaked comment. Warner On Fri, Aug 18, 2017 at 4:09 PM, Jim Harris wrote: > On Fri, Aug 18, 2017 at 2:04 PM, Kinjal Patel > wrote: > > Created a bug #22166 (https://bugs.freebsd.org/bugzilla/show_bug.cgi?id= > 221616) > > Updated the description and patch to it. > > > > Regards, > > Kinjal Patel > > > > From: Kinjal Patel > > Sent: Friday, August 18, 2017 1:51 PM > > To: 'Warner Losh' > > Cc: freebsd-drivers@freebsd.org > > Subject: RE: NVMe driver init sequence > > > > Here is a the change I propose. This will make the init sequence right. > > Looks good to me. > > > > > Index: sys/dev/nvme/nvme.c > > =================================================================== > > --- sys/dev/nvme/nvme.c (revision 322672) > > +++ sys/dev/nvme/nvme.c (working copy) > > @@ -253,6 +253,9 @@ > > return (status); > > } > > > > + /* make device bus-master before attempting to enable controller */ > > + pci_enable_busmaster(dev); > > + > > /* > > * Reset controller twice to ensure we do a transition from > cc.en==1 > > * to cc.en==0. This is because we don't really know what > status > > @@ -270,8 +273,6 @@ > > return (status); > > } > > > > - pci_enable_busmaster(dev); > > - > > ctrlr->config_hook.ich_func = nvme_ctrlr_start_config_hook; > > ctrlr->config_hook.ich_arg = ctrlr; > > > > > > > > regards, > > Kinjal Patel > > > > From: wlosh@bsdimp.com [mailto:wlosh@bsdimp.com] > On Behalf Of Warner Losh > > Sent: Thursday, August 17, 2017 3:10 PM > > To: Kinjal Patel > > Cc: freebsd-drivers@freebsd.org > > Subject: Re: NVMe driver init sequence > > > > This sounds right to me.... Most BIOSes seem to enable BME=1. > > > > Warner > > > > On Thu, Aug 17, 2017 at 2:45 PM, Kinjal Patel < > Kinjal.Patel@taec.toshiba.com> > wrote: > > Hi, > > > > I have a query on NVMe driver init sequence in FreeBSD 10.3(stable). > > > > The driver init sequence is, > > > > 1) Enable controller (CC.EN=1) > > > > 2) Wait for controller ready (CSTS.RDY=1) > > > > 3) Set PCI bus master enable (BME=1) > > > > As per NVMe spec, when NVMe controller becomes ready it has to process > command. > > > > "Enable (EN): When set to '1', then the controller shall process > commands based on Submission Queue Tail doorbell writes" > > > > And per PCI Express spec when BME is not set, the PCI Express Function > is not allowed to issue any Memory or I/O requests. > > "Bus Master Enable - Controls the ability of a PCI Express Endpoint to > issue Memory95 and I/O Read/Write Requests, and > > the ability of a Root or Switch Port to forward Memory and I/O > Read/Write Requests in the Upstream direction" > > > > Enabling controller before setting BME=1 is violation of spec, as > controller has to accept commands but BME is prerequisite for that. > > > > The Linux device driver init sequence is, > > > > 1) Set PCI bus master enable (BME=1) > > > > 2) Enable Controller (CC.EN=1) > > > > 3) Wait for controller ready (CSTS.RDY=1) > > > > The FreeBSD NVMe driver sequence should be changed to set BME=1 before > attempting to enable controller. > > > > > > Regards, > > Kinjal Patel > > _______________________________________________ > > freebsd-drivers@freebsd.org mailing > list > > https://lists.freebsd.org/mailman/listinfo/freebsd-drivers > > To unsubscribe, send any mail to "freebsd-drivers-unsubscribe@ > freebsd.org" > > > > _______________________________________________ > > freebsd-drivers@freebsd.org mailing list > > https://lists.freebsd.org/mailman/listinfo/freebsd-drivers > > To unsubscribe, send any mail to "freebsd-drivers-unsubscribe@ > freebsd.org" > From owner-freebsd-drivers@freebsd.org Fri Aug 25 15:54:48 2017 Return-Path: Delivered-To: freebsd-drivers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 65FB5DDBA2D for ; Fri, 25 Aug 2017 15:54:48 +0000 (UTC) (envelope-from Kinjal.Patel@taec.toshiba.com) Received: from mailhost.taec.toshiba.com (mailhost.taec.toshiba.com [209.243.128.33]) by mx1.freebsd.org (Postfix) with ESMTP id 472EE65B00 for ; Fri, 25 Aug 2017 15:54:47 +0000 (UTC) (envelope-from Kinjal.Patel@taec.toshiba.com) Received: from EXMAIL1.taec.toshiba.com (unknown [209.243.133.250]) by mailhost.taec.toshiba.com (Postfix) with ESMTP id A0103A082B; Fri, 25 Aug 2017 08:54:41 -0700 (PDT) Received: from EXMAIL2.taec.toshiba.com (209.243.166.31) by EXMAIL1.taec.toshiba.com (209.243.172.198) with Microsoft SMTP Server (TLS) id 14.3.123.3; Fri, 25 Aug 2017 08:54:41 -0700 Received: from EXMAIL1.taec.toshiba.com ([169.254.1.230]) by EXMAIL2.taec.toshiba.com ([169.254.2.30]) with mapi id 14.02.0347.000; Fri, 25 Aug 2017 08:54:41 -0700 From: Kinjal Patel To: Warner Losh , Jim Harris CC: "freebsd-drivers@freebsd.org" Subject: RE: NVMe driver init sequence Thread-Topic: NVMe driver init sequence Thread-Index: AdMXlSp3BhHvaAmDQQG9NyRAc27kigASxKyAACB3OKAAAM0YcAARBICAATh43QAAC8RSsA== Date: Fri, 25 Aug 2017 15:54:40 +0000 Message-ID: References: In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.93.33.1] MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-drivers@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Writing device drivers for FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 25 Aug 2017 15:54:48 -0000 R3JlYXQhDQpUaGFua3MgV2FybmVyLg0KDQoNClJlZ2FyZHMsDQpLaW5qYWwgUGF0ZWwNCg0KRnJv bTogd2xvc2hAYnNkaW1wLmNvbSBbbWFpbHRvOndsb3NoQGJzZGltcC5jb21dIE9uIEJlaGFsZiBP ZiBXYXJuZXIgTG9zaA0KU2VudDogVGh1cnNkYXksIEF1Z3VzdCAyNCwgMjAxNyA4OjE3IFBNDQpU bzogSmltIEhhcnJpcw0KQ2M6IEtpbmphbCBQYXRlbDsgZnJlZWJzZC1kcml2ZXJzQGZyZWVic2Qu b3JnDQpTdWJqZWN0OiBSZTogTlZNZSBkcml2ZXIgaW5pdCBzZXF1ZW5jZQ0KDQpHcmVldGluZ3Ms DQoNCkkndmUganVzdCBjb21taXR0ZWQgcjMyMjg3MiB3aGljaCBpbXBsZW1lbnRzIHRoaXMsIHdp dGggYSBzbGlnaHRseSB0d2Vha2VkIGNvbW1lbnQuDQoNCldhcm5lcg0KDQpPbiBGcmksIEF1ZyAx OCwgMjAxNyBhdCA0OjA5IFBNLCBKaW0gSGFycmlzIDxqaW0uaGFycmlzQGdtYWlsLmNvbTxtYWls dG86amltLmhhcnJpc0BnbWFpbC5jb20+PiB3cm90ZToNCk9uIEZyaSwgQXVnIDE4LCAyMDE3IGF0 IDI6MDQgUE0sIEtpbmphbCBQYXRlbA0KPEtpbmphbC5QYXRlbEB0YWVjLnRvc2hpYmEuY29tPG1h aWx0bzpLaW5qYWwuUGF0ZWxAdGFlYy50b3NoaWJhLmNvbT4+IHdyb3RlOg0KPiBDcmVhdGVkIGEg YnVnICMyMjE2NiAoaHR0cHM6Ly9idWdzLmZyZWVic2Qub3JnL2J1Z3ppbGxhL3Nob3dfYnVnLmNn aT9pZD0yMjE2MTYpDQo+IFVwZGF0ZWQgdGhlIGRlc2NyaXB0aW9uIGFuZCBwYXRjaCB0byBpdC4N Cj4NCj4gUmVnYXJkcywNCj4gS2luamFsIFBhdGVsDQo+DQo+IEZyb206IEtpbmphbCBQYXRlbA0K PiBTZW50OiBGcmlkYXksIEF1Z3VzdCAxOCwgMjAxNyAxOjUxIFBNDQo+IFRvOiAnV2FybmVyIExv c2gnDQo+IENjOiBmcmVlYnNkLWRyaXZlcnNAZnJlZWJzZC5vcmc8bWFpbHRvOmZyZWVic2QtZHJp dmVyc0BmcmVlYnNkLm9yZz4NCj4gU3ViamVjdDogUkU6IE5WTWUgZHJpdmVyIGluaXQgc2VxdWVu Y2UNCj4NCj4gSGVyZSBpcyBhIHRoZSBjaGFuZ2UgSSBwcm9wb3NlLiBUaGlzIHdpbGwgbWFrZSB0 aGUgaW5pdCBzZXF1ZW5jZSByaWdodC4NCg0KTG9va3MgZ29vZCB0byBtZS4NCg0KPg0KPiBJbmRl eDogc3lzL2Rldi9udm1lL252bWUuYw0KPiA9PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09DQo+IC0tLSBzeXMvZGV2L252bWUv bnZtZS5jIChyZXZpc2lvbiAzMjI2NzIpDQo+ICsrKyBzeXMvZGV2L252bWUvbnZtZS5jICh3b3Jr aW5nIGNvcHkpDQo+IEBAIC0yNTMsNiArMjUzLDkgQEANCj4gICAgICAgICAgICAgICAgIHJldHVy biAoc3RhdHVzKTsNCj4gICAgICAgICB9DQo+DQo+ICsgICAgLyogbWFrZSBkZXZpY2UgYnVzLW1h c3RlciBiZWZvcmUgYXR0ZW1wdGluZyB0byBlbmFibGUgY29udHJvbGxlciAqLw0KPiArICAgIHBj aV9lbmFibGVfYnVzbWFzdGVyKGRldik7DQo+ICsNCj4gICAgICAgICAvKg0KPiAgICAgICAgICAq IFJlc2V0IGNvbnRyb2xsZXIgdHdpY2UgdG8gZW5zdXJlIHdlIGRvIGEgdHJhbnNpdGlvbiBmcm9t IGNjLmVuPT0xDQo+ICAgICAgICAgICogIHRvIGNjLmVuPT0wLiAgVGhpcyBpcyBiZWNhdXNlIHdl IGRvbid0IHJlYWxseSBrbm93IHdoYXQgc3RhdHVzDQo+IEBAIC0yNzAsOCArMjczLDYgQEANCj4g ICAgICAgICAgICAgICAgIHJldHVybiAoc3RhdHVzKTsNCj4gICAgICAgICB9DQo+DQo+IC0gICAg ICAgcGNpX2VuYWJsZV9idXNtYXN0ZXIoZGV2KTsNCj4gLQ0KPiAgICAgICAgIGN0cmxyLT5jb25m aWdfaG9vay5pY2hfZnVuYyA9IG52bWVfY3RybHJfc3RhcnRfY29uZmlnX2hvb2s7DQo+ICAgICAg ICAgY3RybHItPmNvbmZpZ19ob29rLmljaF9hcmcgPSBjdHJscjsNCj4NCj4NCj4NCj4gcmVnYXJk cywNCj4gS2luamFsIFBhdGVsDQo+DQo+IEZyb206IHdsb3NoQGJzZGltcC5jb208bWFpbHRvOnds b3NoQGJzZGltcC5jb20+PG1haWx0bzp3bG9zaEBic2RpbXAuY29tPG1haWx0bzp3bG9zaEBic2Rp bXAuY29tPj4gW21haWx0bzp3bG9zaEBic2RpbXAuY29tPG1haWx0bzp3bG9zaEBic2RpbXAuY29t Pl0gT24gQmVoYWxmIE9mIFdhcm5lciBMb3NoDQo+IFNlbnQ6IFRodXJzZGF5LCBBdWd1c3QgMTcs IDIwMTcgMzoxMCBQTQ0KPiBUbzogS2luamFsIFBhdGVsDQo+IENjOiBmcmVlYnNkLWRyaXZlcnNA ZnJlZWJzZC5vcmc8bWFpbHRvOmZyZWVic2QtZHJpdmVyc0BmcmVlYnNkLm9yZz48bWFpbHRvOmZy ZWVic2QtZHJpdmVyc0BmcmVlYnNkLm9yZzxtYWlsdG86ZnJlZWJzZC1kcml2ZXJzQGZyZWVic2Qu b3JnPj4NCj4gU3ViamVjdDogUmU6IE5WTWUgZHJpdmVyIGluaXQgc2VxdWVuY2UNCj4NCj4gVGhp cyBzb3VuZHMgcmlnaHQgdG8gbWUuLi4uICBNb3N0IEJJT1NlcyBzZWVtIHRvIGVuYWJsZSBCTUU9 MS4NCj4NCj4gV2FybmVyDQo+DQo+IE9uIFRodSwgQXVnIDE3LCAyMDE3IGF0IDI6NDUgUE0sIEtp bmphbCBQYXRlbCA8S2luamFsLlBhdGVsQHRhZWMudG9zaGliYS5jb208bWFpbHRvOktpbmphbC5Q YXRlbEB0YWVjLnRvc2hpYmEuY29tPjxtYWlsdG86S2luamFsLlBhdGVsQHRhZWMudG9zaGliYS5j b208bWFpbHRvOktpbmphbC5QYXRlbEB0YWVjLnRvc2hpYmEuY29tPj4+IHdyb3RlOg0KPiBIaSwN Cj4NCj4gSSBoYXZlIGEgcXVlcnkgb24gTlZNZSBkcml2ZXIgaW5pdCBzZXF1ZW5jZSBpbiBGcmVl QlNEIDEwLjMoc3RhYmxlKS4NCj4NCj4gVGhlIGRyaXZlciBpbml0IHNlcXVlbmNlIGlzLA0KPg0K PiAxKSAgICAgIEVuYWJsZSBjb250cm9sbGVyIChDQy5FTj0xKQ0KPg0KPiAyKSAgICAgIFdhaXQg Zm9yIGNvbnRyb2xsZXIgcmVhZHkgKENTVFMuUkRZPTEpDQo+DQo+IDMpICAgICAgU2V0IFBDSSBi dXMgbWFzdGVyIGVuYWJsZSAoQk1FPTEpDQo+DQo+IEFzIHBlciBOVk1lIHNwZWMsIHdoZW4gTlZN ZSBjb250cm9sbGVyIGJlY29tZXMgcmVhZHkgaXQgaGFzIHRvIHByb2Nlc3MgY29tbWFuZC4NCj4N Cj4gIkVuYWJsZSAoRU4pOiBXaGVuIHNldCB0byAnMScsIHRoZW4gdGhlIGNvbnRyb2xsZXIgc2hh bGwgcHJvY2VzcyBjb21tYW5kcyBiYXNlZCBvbiBTdWJtaXNzaW9uIFF1ZXVlIFRhaWwgZG9vcmJl bGwgd3JpdGVzIg0KPg0KPiBBbmQgcGVyIFBDSSBFeHByZXNzIHNwZWMgd2hlbiBCTUUgaXMgbm90 IHNldCwgdGhlIFBDSSBFeHByZXNzIEZ1bmN0aW9uIGlzIG5vdCBhbGxvd2VkIHRvIGlzc3VlIGFu eSBNZW1vcnkgb3IgSS9PIHJlcXVlc3RzLg0KPiAiQnVzIE1hc3RlciBFbmFibGUgLSBDb250cm9s cyB0aGUgYWJpbGl0eSBvZiBhIFBDSSBFeHByZXNzIEVuZHBvaW50IHRvIGlzc3VlIE1lbW9yeTk1 IGFuZCBJL08gUmVhZC9Xcml0ZSBSZXF1ZXN0cywgYW5kDQo+IHRoZSBhYmlsaXR5IG9mIGEgUm9v dCBvciBTd2l0Y2ggUG9ydCB0byBmb3J3YXJkIE1lbW9yeSBhbmQgSS9PIFJlYWQvV3JpdGUgUmVx dWVzdHMgaW4gdGhlIFVwc3RyZWFtIGRpcmVjdGlvbiINCj4NCj4gRW5hYmxpbmcgY29udHJvbGxl ciBiZWZvcmUgc2V0dGluZyBCTUU9MSBpcyB2aW9sYXRpb24gb2Ygc3BlYywgYXMgY29udHJvbGxl ciBoYXMgdG8gYWNjZXB0IGNvbW1hbmRzIGJ1dCBCTUUgaXMgcHJlcmVxdWlzaXRlIGZvciB0aGF0 Lg0KPg0KPiBUaGUgTGludXggZGV2aWNlIGRyaXZlciBpbml0IHNlcXVlbmNlIGlzLA0KPg0KPiAx KSAgICAgU2V0IFBDSSBidXMgbWFzdGVyIGVuYWJsZSAoQk1FPTEpDQo+DQo+IDIpICAgICBFbmFi bGUgQ29udHJvbGxlciAoQ0MuRU49MSkNCj4NCj4gMykgICAgIFdhaXQgZm9yIGNvbnRyb2xsZXIg cmVhZHkgKENTVFMuUkRZPTEpDQo+DQo+IFRoZSBGcmVlQlNEIE5WTWUgZHJpdmVyIHNlcXVlbmNl IHNob3VsZCBiZSBjaGFuZ2VkIHRvIHNldCBCTUU9MSBiZWZvcmUgYXR0ZW1wdGluZyB0byBlbmFi bGUgY29udHJvbGxlci4NCj4NCj4NCj4gUmVnYXJkcywNCj4gS2luamFsIFBhdGVsDQo+IF9fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fDQo+IGZyZWVic2QtZHJp dmVyc0BmcmVlYnNkLm9yZzxtYWlsdG86ZnJlZWJzZC1kcml2ZXJzQGZyZWVic2Qub3JnPjxtYWls dG86ZnJlZWJzZC1kcml2ZXJzQGZyZWVic2Qub3JnPG1haWx0bzpmcmVlYnNkLWRyaXZlcnNAZnJl ZWJzZC5vcmc+PiBtYWlsaW5nIGxpc3QNCj4gaHR0cHM6Ly9saXN0cy5mcmVlYnNkLm9yZy9tYWls bWFuL2xpc3RpbmZvL2ZyZWVic2QtZHJpdmVycw0KPiBUbyB1bnN1YnNjcmliZSwgc2VuZCBhbnkg bWFpbCB0byAiZnJlZWJzZC1kcml2ZXJzLXVuc3Vic2NyaWJlQGZyZWVic2Qub3JnPG1haWx0bzpm cmVlYnNkLWRyaXZlcnMtdW5zdWJzY3JpYmVAZnJlZWJzZC5vcmc+PG1haWx0bzpmcmVlYnNkLWRy aXZlcnMtdW5zdWJzY3JpYmVAZnJlZWJzZC5vcmc8bWFpbHRvOmZyZWVic2QtZHJpdmVycy11bnN1 YnNjcmliZUBmcmVlYnNkLm9yZz4+Ig0KPg0KPiBfX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fXw0KPiBmcmVlYnNkLWRyaXZlcnNAZnJlZWJzZC5vcmc8bWFpbHRv OmZyZWVic2QtZHJpdmVyc0BmcmVlYnNkLm9yZz4gbWFpbGluZyBsaXN0DQo+IGh0dHBzOi8vbGlz dHMuZnJlZWJzZC5vcmcvbWFpbG1hbi9saXN0aW5mby9mcmVlYnNkLWRyaXZlcnMNCj4gVG8gdW5z dWJzY3JpYmUsIHNlbmQgYW55IG1haWwgdG8gImZyZWVic2QtZHJpdmVycy11bnN1YnNjcmliZUBm cmVlYnNkLm9yZzxtYWlsdG86ZnJlZWJzZC1kcml2ZXJzLXVuc3Vic2NyaWJlQGZyZWVic2Qub3Jn PiINCg0K From owner-freebsd-drivers@freebsd.org Fri Aug 25 16:00:22 2017 Return-Path: Delivered-To: freebsd-drivers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id AC19ADDBAEA for ; Fri, 25 Aug 2017 16:00:22 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-it0-x22f.google.com (mail-it0-x22f.google.com [IPv6:2607:f8b0:4001:c0b::22f]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 6DE5C65BCD for ; Fri, 25 Aug 2017 16:00:22 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: by mail-it0-x22f.google.com with SMTP id p10so1175281ite.1 for ; Fri, 25 Aug 2017 09:00:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20150623.gappssmtp.com; s=20150623; h=mime-version:sender:in-reply-to:references:from:date:message-id :subject:to:cc; bh=1Dllcmpvgn+dCywNwXbXsNKkjOHYHjUZaKnot7F5Apg=; b=oBnKnZ9osUe1lxqu2FDGDjHcuy3LA8jRaGxLgYU1xYXF8r2gdnKdBhZODlUtPF9xgQ PONiQWCh8yjeRe/EKu/0lY5EfLGF+pNrRiQ3vRZYuiy3QYcw/+WbWJb9vbA2klq4TJzS bopdA5Ozeol/KiU4fBJqpuh/cfUASd8mWeZTxWewmVcQ9bKUkrQd83Sm6r/QpTCB238S i2TkiMHaylKmRd/41GMylNAQdgFffWhcffNS3yqPq6hVeohxx1h2cV9ykibXzqbH3CyS hyk/ar9L2WfjQ+D4vPLkQwhlV8KwzvqNUMwMBhgFVSnQOXDUPDGot9drMKIb+n0v9xQc griQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:sender:in-reply-to:references:from :date:message-id:subject:to:cc; bh=1Dllcmpvgn+dCywNwXbXsNKkjOHYHjUZaKnot7F5Apg=; b=CCtiuZvM7+6t9Ab8axAnv5tjPruF8q3HQBQw1TvuUmYQWpVNYQ5pqrp1YDFhi2x+lm U26XkFN7HOdE2I6Gqe18Ky3dPnGPp9Zy9ubIeKwkPWp0jz5Gs3j/Z16P3663+a4KJt8t paRE6OGhhCL6hNZWQRFFZD8FUK9K4Dl034HTDo/ZjJkkzSfEUgT6GHrwXejyYx1qxsWC JDA72Zi9g6NYy1SV4kL3pmZYxPFNaund6cvxfLduIZaiqGzePmU+Pf/9EhsIBp3heVrg n0R4boBrOOhQXhLprMAigrIfEt2PpJTJDdhG8qUnrxTTuBgKRIESJouribW7Xw+aE7Fx JPEg== X-Gm-Message-State: AHYfb5hjpUU4zAK7uo4uu/ajmDon3kUhbU4YXfDqY/hkzhFwk6T5bwcm kCUqcNon1zvDPVa2cFEBRuXD8TsB5KVE X-Received: by 10.36.237.8 with SMTP id r8mr2754116ith.125.1503676821389; Fri, 25 Aug 2017 09:00:21 -0700 (PDT) MIME-Version: 1.0 Sender: wlosh@bsdimp.com Received: by 10.79.10.71 with HTTP; Fri, 25 Aug 2017 09:00:20 -0700 (PDT) X-Originating-IP: [2603:300b:6:5100:c525:dd46:ae7:666a] In-Reply-To: References: From: Warner Losh Date: Fri, 25 Aug 2017 10:00:20 -0600 X-Google-Sender-Auth: mkVF7INCQ0g0xoa_kNneu_k5tWg Message-ID: Subject: Re: NVMe driver init sequence To: Kinjal Patel Cc: Jim Harris , "freebsd-drivers@freebsd.org" Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-drivers@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Writing device drivers for FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 25 Aug 2017 16:00:22 -0000 BTW, I just noticed you reported this against 10.3R. Is there some reason (perhaps a customer with an issue) that you reported it there? I'm trying to decide if I need to MFC this change to 11.x and 10.x (the latter has some time urgency due to upcoming 10.4R release). Warner On Fri, Aug 25, 2017 at 9:54 AM, Kinjal Patel wrote: > Great! > > Thanks Warner. > > > > > > Regards, > > Kinjal Patel > > > > *From:* wlosh@bsdimp.com [mailto:wlosh@bsdimp.com] *On Behalf Of *Warner > Losh > *Sent:* Thursday, August 24, 2017 8:17 PM > *To:* Jim Harris > *Cc:* Kinjal Patel; freebsd-drivers@freebsd.org > > *Subject:* Re: NVMe driver init sequence > > > > Greetings, > > > > I've just committed r322872 which implements this, with a slightly tweaked > comment. > > > > Warner > > > > On Fri, Aug 18, 2017 at 4:09 PM, Jim Harris wrote: > > On Fri, Aug 18, 2017 at 2:04 PM, Kinjal Patel > wrote: > > Created a bug #22166 (https://bugs.freebsd.org/bugzilla/show_bug.cgi?id= > 221616) > > Updated the description and patch to it. > > > > Regards, > > Kinjal Patel > > > > From: Kinjal Patel > > Sent: Friday, August 18, 2017 1:51 PM > > To: 'Warner Losh' > > Cc: freebsd-drivers@freebsd.org > > Subject: RE: NVMe driver init sequence > > > > Here is a the change I propose. This will make the init sequence right. > > Looks good to me. > > > > > Index: sys/dev/nvme/nvme.c > > =================================================================== > > --- sys/dev/nvme/nvme.c (revision 322672) > > +++ sys/dev/nvme/nvme.c (working copy) > > @@ -253,6 +253,9 @@ > > return (status); > > } > > > > + /* make device bus-master before attempting to enable controller */ > > + pci_enable_busmaster(dev); > > + > > /* > > * Reset controller twice to ensure we do a transition from > cc.en==1 > > * to cc.en==0. This is because we don't really know what > status > > @@ -270,8 +273,6 @@ > > return (status); > > } > > > > - pci_enable_busmaster(dev); > > - > > ctrlr->config_hook.ich_func = nvme_ctrlr_start_config_hook; > > ctrlr->config_hook.ich_arg = ctrlr; > > > > > > > > regards, > > Kinjal Patel > > > > From: wlosh@bsdimp.com [mailto:wlosh@bsdimp.com] > On Behalf Of Warner Losh > > Sent: Thursday, August 17, 2017 3:10 PM > > To: Kinjal Patel > > Cc: freebsd-drivers@freebsd.org > > Subject: Re: NVMe driver init sequence > > > > This sounds right to me.... Most BIOSes seem to enable BME=1. > > > > Warner > > > > On Thu, Aug 17, 2017 at 2:45 PM, Kinjal Patel < > Kinjal.Patel@taec.toshiba.com> > wrote: > > Hi, > > > > I have a query on NVMe driver init sequence in FreeBSD 10.3(stable). > > > > The driver init sequence is, > > > > 1) Enable controller (CC.EN=1) > > > > 2) Wait for controller ready (CSTS.RDY=1) > > > > 3) Set PCI bus master enable (BME=1) > > > > As per NVMe spec, when NVMe controller becomes ready it has to process > command. > > > > "Enable (EN): When set to '1', then the controller shall process > commands based on Submission Queue Tail doorbell writes" > > > > And per PCI Express spec when BME is not set, the PCI Express Function > is not allowed to issue any Memory or I/O requests. > > "Bus Master Enable - Controls the ability of a PCI Express Endpoint to > issue Memory95 and I/O Read/Write Requests, and > > the ability of a Root or Switch Port to forward Memory and I/O > Read/Write Requests in the Upstream direction" > > > > Enabling controller before setting BME=1 is violation of spec, as > controller has to accept commands but BME is prerequisite for that. > > > > The Linux device driver init sequence is, > > > > 1) Set PCI bus master enable (BME=1) > > > > 2) Enable Controller (CC.EN=1) > > > > 3) Wait for controller ready (CSTS.RDY=1) > > > > The FreeBSD NVMe driver sequence should be changed to set BME=1 before > attempting to enable controller. > > > > > > Regards, > > Kinjal Patel > > _______________________________________________ > > freebsd-drivers@freebsd.org mailing > list > > https://lists.freebsd.org/mailman/listinfo/freebsd-drivers > > To unsubscribe, send any mail to "freebsd-drivers-unsubscribe@ > freebsd.org" > > > > > _______________________________________________ > > freebsd-drivers@freebsd.org mailing list > > https://lists.freebsd.org/mailman/listinfo/freebsd-drivers > > To unsubscribe, send any mail to "freebsd-drivers-unsubscribe@ > freebsd.org" > > > From owner-freebsd-drivers@freebsd.org Fri Aug 25 16:11:38 2017 Return-Path: Delivered-To: freebsd-drivers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E3CF0DDBEEB for ; Fri, 25 Aug 2017 16:11:38 +0000 (UTC) (envelope-from Kinjal.Patel@taec.toshiba.com) Received: from mailhost.taec.toshiba.com (mailhost.taec.toshiba.com [209.243.128.33]) by mx1.freebsd.org (Postfix) with ESMTP id CBE5C66229 for ; Fri, 25 Aug 2017 16:11:38 +0000 (UTC) (envelope-from Kinjal.Patel@taec.toshiba.com) Received: from EXMAIL1.taec.toshiba.com (unknown [209.243.133.250]) by mailhost.taec.toshiba.com (Postfix) with ESMTP id 8DE55A08F0; Fri, 25 Aug 2017 09:11:38 -0700 (PDT) Received: from EXMAIL2.taec.toshiba.com (209.243.166.31) by EXMAIL1.taec.toshiba.com (209.243.172.198) with Microsoft SMTP Server (TLS) id 14.3.123.3; Fri, 25 Aug 2017 09:11:38 -0700 Received: from EXMAIL1.taec.toshiba.com ([169.254.1.230]) by EXMAIL2.taec.toshiba.com ([169.254.2.30]) with mapi id 14.02.0347.000; Fri, 25 Aug 2017 09:11:38 -0700 From: Kinjal Patel To: Warner Losh CC: Jim Harris , "freebsd-drivers@freebsd.org" Subject: RE: NVMe driver init sequence Thread-Topic: NVMe driver init sequence Thread-Index: AdMXlSp3BhHvaAmDQQG9NyRAc27kigASxKyAACB3OKAAAM0YcAARBICAATh43QAAC8RSsAAO5eYAAA6FQOA= Date: Fri, 25 Aug 2017 16:11:37 +0000 Message-ID: References: In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.93.33.1] MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-drivers@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Writing device drivers for FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 25 Aug 2017 16:11:39 -0000 SSBoYXZlIG9ic2VydmVkIHRoaXMgYmVoYXZpb3IgaW4gb3VyIGxhYiBvbiAxMC4zLVNUQUJMRS4g U28gSSBoYXZlIHJlcG9ydGVkIGFnYWluc3QgaXQuDQpJZiB5b3UgY2FuIGJyaW5nIGluIHRoZSBj aGFuZ2UgZm9yIDEwLjRSIHRoYXQgd2lsbCBiZSBncmVhdCAobG90IG9mIHVzZXJzIHN0aWNrIHRv IDEwIGJhc2VkIHJlbGVhc2VzIGZvciB0aGVpciBkZXZlbG9wbWVudCkuIEFuZCBpbiBjYXNlLCBp ZiBjdXN0b21lciBmYWNlcyB0aGlzIGlzc3VlLCBJIGNhbiBwb2ludCB0byB0aGlzIGNvbW1pdCDi mLouDQoNClJlZ2FyZHMsDQpLaW5qYWwgUGF0ZWwNCg0KRnJvbTogd2xvc2hAYnNkaW1wLmNvbSBb bWFpbHRvOndsb3NoQGJzZGltcC5jb21dIE9uIEJlaGFsZiBPZiBXYXJuZXIgTG9zaA0KU2VudDog RnJpZGF5LCBBdWd1c3QgMjUsIDIwMTcgOTowMCBBTQ0KVG86IEtpbmphbCBQYXRlbA0KQ2M6IEpp bSBIYXJyaXM7IGZyZWVic2QtZHJpdmVyc0BmcmVlYnNkLm9yZw0KU3ViamVjdDogUmU6IE5WTWUg ZHJpdmVyIGluaXQgc2VxdWVuY2UNCg0KQlRXLCBJIGp1c3Qgbm90aWNlZCB5b3UgcmVwb3J0ZWQg dGhpcyBhZ2FpbnN0IDEwLjNSLiBJcyB0aGVyZSBzb21lIHJlYXNvbiAocGVyaGFwcyBhIGN1c3Rv bWVyIHdpdGggYW4gaXNzdWUpIHRoYXQgeW91IHJlcG9ydGVkIGl0IHRoZXJlPyBJJ20gdHJ5aW5n IHRvIGRlY2lkZSBpZiBJIG5lZWQgdG8gTUZDIHRoaXMgY2hhbmdlIHRvIDExLnggYW5kIDEwLngg KHRoZSBsYXR0ZXIgaGFzIHNvbWUgdGltZSB1cmdlbmN5IGR1ZSB0byB1cGNvbWluZyAxMC40UiBy ZWxlYXNlKS4NCg0KV2FybmVyDQoNCk9uIEZyaSwgQXVnIDI1LCAyMDE3IGF0IDk6NTQgQU0sIEtp bmphbCBQYXRlbCA8S2luamFsLlBhdGVsQHRhZWMudG9zaGliYS5jb208bWFpbHRvOktpbmphbC5Q YXRlbEB0YWVjLnRvc2hpYmEuY29tPj4gd3JvdGU6DQpHcmVhdCENClRoYW5rcyBXYXJuZXIuDQoN Cg0KUmVnYXJkcywNCktpbmphbCBQYXRlbA0KDQpGcm9tOiB3bG9zaEBic2RpbXAuY29tPG1haWx0 bzp3bG9zaEBic2RpbXAuY29tPiBbbWFpbHRvOndsb3NoQGJzZGltcC5jb208bWFpbHRvOndsb3No QGJzZGltcC5jb20+XSBPbiBCZWhhbGYgT2YgV2FybmVyIExvc2gNClNlbnQ6IFRodXJzZGF5LCBB dWd1c3QgMjQsIDIwMTcgODoxNyBQTQ0KVG86IEppbSBIYXJyaXMNCkNjOiBLaW5qYWwgUGF0ZWw7 IGZyZWVic2QtZHJpdmVyc0BmcmVlYnNkLm9yZzxtYWlsdG86ZnJlZWJzZC1kcml2ZXJzQGZyZWVi c2Qub3JnPg0KDQpTdWJqZWN0OiBSZTogTlZNZSBkcml2ZXIgaW5pdCBzZXF1ZW5jZQ0KDQpHcmVl dGluZ3MsDQoNCkkndmUganVzdCBjb21taXR0ZWQgcjMyMjg3MiB3aGljaCBpbXBsZW1lbnRzIHRo aXMsIHdpdGggYSBzbGlnaHRseSB0d2Vha2VkIGNvbW1lbnQuDQoNCldhcm5lcg0KDQpPbiBGcmks IEF1ZyAxOCwgMjAxNyBhdCA0OjA5IFBNLCBKaW0gSGFycmlzIDxqaW0uaGFycmlzQGdtYWlsLmNv bTxtYWlsdG86amltLmhhcnJpc0BnbWFpbC5jb20+PiB3cm90ZToNCk9uIEZyaSwgQXVnIDE4LCAy MDE3IGF0IDI6MDQgUE0sIEtpbmphbCBQYXRlbA0KPEtpbmphbC5QYXRlbEB0YWVjLnRvc2hpYmEu Y29tPG1haWx0bzpLaW5qYWwuUGF0ZWxAdGFlYy50b3NoaWJhLmNvbT4+IHdyb3RlOg0KPiBDcmVh dGVkIGEgYnVnICMyMjE2NiAoaHR0cHM6Ly9idWdzLmZyZWVic2Qub3JnL2J1Z3ppbGxhL3Nob3df YnVnLmNnaT9pZD0yMjE2MTYpDQo+IFVwZGF0ZWQgdGhlIGRlc2NyaXB0aW9uIGFuZCBwYXRjaCB0 byBpdC4NCj4NCj4gUmVnYXJkcywNCj4gS2luamFsIFBhdGVsDQo+DQo+IEZyb206IEtpbmphbCBQ YXRlbA0KPiBTZW50OiBGcmlkYXksIEF1Z3VzdCAxOCwgMjAxNyAxOjUxIFBNDQo+IFRvOiAnV2Fy bmVyIExvc2gnDQo+IENjOiBmcmVlYnNkLWRyaXZlcnNAZnJlZWJzZC5vcmc8bWFpbHRvOmZyZWVi c2QtZHJpdmVyc0BmcmVlYnNkLm9yZz4NCj4gU3ViamVjdDogUkU6IE5WTWUgZHJpdmVyIGluaXQg c2VxdWVuY2UNCj4NCj4gSGVyZSBpcyBhIHRoZSBjaGFuZ2UgSSBwcm9wb3NlLiBUaGlzIHdpbGwg bWFrZSB0aGUgaW5pdCBzZXF1ZW5jZSByaWdodC4NCg0KTG9va3MgZ29vZCB0byBtZS4NCg0KPg0K PiBJbmRleDogc3lzL2Rldi9udm1lL252bWUuYw0KPiA9PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09DQo+IC0tLSBzeXMvZGV2 L252bWUvbnZtZS5jIChyZXZpc2lvbiAzMjI2NzIpDQo+ICsrKyBzeXMvZGV2L252bWUvbnZtZS5j ICh3b3JraW5nIGNvcHkpDQo+IEBAIC0yNTMsNiArMjUzLDkgQEANCj4gICAgICAgICAgICAgICAg IHJldHVybiAoc3RhdHVzKTsNCj4gICAgICAgICB9DQo+DQo+ICsgICAgLyogbWFrZSBkZXZpY2Ug YnVzLW1hc3RlciBiZWZvcmUgYXR0ZW1wdGluZyB0byBlbmFibGUgY29udHJvbGxlciAqLw0KPiAr ICAgIHBjaV9lbmFibGVfYnVzbWFzdGVyKGRldik7DQo+ICsNCj4gICAgICAgICAvKg0KPiAgICAg ICAgICAqIFJlc2V0IGNvbnRyb2xsZXIgdHdpY2UgdG8gZW5zdXJlIHdlIGRvIGEgdHJhbnNpdGlv biBmcm9tIGNjLmVuPT0xDQo+ICAgICAgICAgICogIHRvIGNjLmVuPT0wLiAgVGhpcyBpcyBiZWNh dXNlIHdlIGRvbid0IHJlYWxseSBrbm93IHdoYXQgc3RhdHVzDQo+IEBAIC0yNzAsOCArMjczLDYg QEANCj4gICAgICAgICAgICAgICAgIHJldHVybiAoc3RhdHVzKTsNCj4gICAgICAgICB9DQo+DQo+ IC0gICAgICAgcGNpX2VuYWJsZV9idXNtYXN0ZXIoZGV2KTsNCj4gLQ0KPiAgICAgICAgIGN0cmxy LT5jb25maWdfaG9vay5pY2hfZnVuYyA9IG52bWVfY3RybHJfc3RhcnRfY29uZmlnX2hvb2s7DQo+ ICAgICAgICAgY3RybHItPmNvbmZpZ19ob29rLmljaF9hcmcgPSBjdHJscjsNCj4NCj4NCj4NCj4g cmVnYXJkcywNCj4gS2luamFsIFBhdGVsDQo+DQo+IEZyb206IHdsb3NoQGJzZGltcC5jb208bWFp bHRvOndsb3NoQGJzZGltcC5jb20+PG1haWx0bzp3bG9zaEBic2RpbXAuY29tPG1haWx0bzp3bG9z aEBic2RpbXAuY29tPj4gW21haWx0bzp3bG9zaEBic2RpbXAuY29tPG1haWx0bzp3bG9zaEBic2Rp bXAuY29tPl0gT24gQmVoYWxmIE9mIFdhcm5lciBMb3NoDQo+IFNlbnQ6IFRodXJzZGF5LCBBdWd1 c3QgMTcsIDIwMTcgMzoxMCBQTQ0KPiBUbzogS2luamFsIFBhdGVsDQo+IENjOiBmcmVlYnNkLWRy aXZlcnNAZnJlZWJzZC5vcmc8bWFpbHRvOmZyZWVic2QtZHJpdmVyc0BmcmVlYnNkLm9yZz48bWFp bHRvOmZyZWVic2QtZHJpdmVyc0BmcmVlYnNkLm9yZzxtYWlsdG86ZnJlZWJzZC1kcml2ZXJzQGZy ZWVic2Qub3JnPj4NCj4gU3ViamVjdDogUmU6IE5WTWUgZHJpdmVyIGluaXQgc2VxdWVuY2UNCj4N Cj4gVGhpcyBzb3VuZHMgcmlnaHQgdG8gbWUuLi4uICBNb3N0IEJJT1NlcyBzZWVtIHRvIGVuYWJs ZSBCTUU9MS4NCj4NCj4gV2FybmVyDQo+DQo+IE9uIFRodSwgQXVnIDE3LCAyMDE3IGF0IDI6NDUg UE0sIEtpbmphbCBQYXRlbCA8S2luamFsLlBhdGVsQHRhZWMudG9zaGliYS5jb208bWFpbHRvOktp bmphbC5QYXRlbEB0YWVjLnRvc2hpYmEuY29tPjxtYWlsdG86S2luamFsLlBhdGVsQHRhZWMudG9z aGliYS5jb208bWFpbHRvOktpbmphbC5QYXRlbEB0YWVjLnRvc2hpYmEuY29tPj4+IHdyb3RlOg0K PiBIaSwNCj4NCj4gSSBoYXZlIGEgcXVlcnkgb24gTlZNZSBkcml2ZXIgaW5pdCBzZXF1ZW5jZSBp biBGcmVlQlNEIDEwLjMoc3RhYmxlKS4NCj4NCj4gVGhlIGRyaXZlciBpbml0IHNlcXVlbmNlIGlz LA0KPg0KPiAxKSAgICAgIEVuYWJsZSBjb250cm9sbGVyIChDQy5FTj0xKQ0KPg0KPiAyKSAgICAg IFdhaXQgZm9yIGNvbnRyb2xsZXIgcmVhZHkgKENTVFMuUkRZPTEpDQo+DQo+IDMpICAgICAgU2V0 IFBDSSBidXMgbWFzdGVyIGVuYWJsZSAoQk1FPTEpDQo+DQo+IEFzIHBlciBOVk1lIHNwZWMsIHdo ZW4gTlZNZSBjb250cm9sbGVyIGJlY29tZXMgcmVhZHkgaXQgaGFzIHRvIHByb2Nlc3MgY29tbWFu ZC4NCj4NCj4gIkVuYWJsZSAoRU4pOiBXaGVuIHNldCB0byAnMScsIHRoZW4gdGhlIGNvbnRyb2xs ZXIgc2hhbGwgcHJvY2VzcyBjb21tYW5kcyBiYXNlZCBvbiBTdWJtaXNzaW9uIFF1ZXVlIFRhaWwg ZG9vcmJlbGwgd3JpdGVzIg0KPg0KPiBBbmQgcGVyIFBDSSBFeHByZXNzIHNwZWMgd2hlbiBCTUUg aXMgbm90IHNldCwgdGhlIFBDSSBFeHByZXNzIEZ1bmN0aW9uIGlzIG5vdCBhbGxvd2VkIHRvIGlz c3VlIGFueSBNZW1vcnkgb3IgSS9PIHJlcXVlc3RzLg0KPiAiQnVzIE1hc3RlciBFbmFibGUgLSBD b250cm9scyB0aGUgYWJpbGl0eSBvZiBhIFBDSSBFeHByZXNzIEVuZHBvaW50IHRvIGlzc3VlIE1l bW9yeTk1IGFuZCBJL08gUmVhZC9Xcml0ZSBSZXF1ZXN0cywgYW5kDQo+IHRoZSBhYmlsaXR5IG9m IGEgUm9vdCBvciBTd2l0Y2ggUG9ydCB0byBmb3J3YXJkIE1lbW9yeSBhbmQgSS9PIFJlYWQvV3Jp dGUgUmVxdWVzdHMgaW4gdGhlIFVwc3RyZWFtIGRpcmVjdGlvbiINCj4NCj4gRW5hYmxpbmcgY29u dHJvbGxlciBiZWZvcmUgc2V0dGluZyBCTUU9MSBpcyB2aW9sYXRpb24gb2Ygc3BlYywgYXMgY29u dHJvbGxlciBoYXMgdG8gYWNjZXB0IGNvbW1hbmRzIGJ1dCBCTUUgaXMgcHJlcmVxdWlzaXRlIGZv ciB0aGF0Lg0KPg0KPiBUaGUgTGludXggZGV2aWNlIGRyaXZlciBpbml0IHNlcXVlbmNlIGlzLA0K Pg0KPiAxKSAgICAgU2V0IFBDSSBidXMgbWFzdGVyIGVuYWJsZSAoQk1FPTEpDQo+DQo+IDIpICAg ICBFbmFibGUgQ29udHJvbGxlciAoQ0MuRU49MSkNCj4NCj4gMykgICAgIFdhaXQgZm9yIGNvbnRy b2xsZXIgcmVhZHkgKENTVFMuUkRZPTEpDQo+DQo+IFRoZSBGcmVlQlNEIE5WTWUgZHJpdmVyIHNl cXVlbmNlIHNob3VsZCBiZSBjaGFuZ2VkIHRvIHNldCBCTUU9MSBiZWZvcmUgYXR0ZW1wdGluZyB0 byBlbmFibGUgY29udHJvbGxlci4NCj4NCj4NCj4gUmVnYXJkcywNCj4gS2luamFsIFBhdGVsDQo+ IF9fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fDQo+IGZyZWVi c2QtZHJpdmVyc0BmcmVlYnNkLm9yZzxtYWlsdG86ZnJlZWJzZC1kcml2ZXJzQGZyZWVic2Qub3Jn PjxtYWlsdG86ZnJlZWJzZC1kcml2ZXJzQGZyZWVic2Qub3JnPG1haWx0bzpmcmVlYnNkLWRyaXZl cnNAZnJlZWJzZC5vcmc+PiBtYWlsaW5nIGxpc3QNCj4gaHR0cHM6Ly9saXN0cy5mcmVlYnNkLm9y Zy9tYWlsbWFuL2xpc3RpbmZvL2ZyZWVic2QtZHJpdmVycw0KPiBUbyB1bnN1YnNjcmliZSwgc2Vu ZCBhbnkgbWFpbCB0byAiZnJlZWJzZC1kcml2ZXJzLXVuc3Vic2NyaWJlQGZyZWVic2Qub3JnPG1h aWx0bzpmcmVlYnNkLWRyaXZlcnMtdW5zdWJzY3JpYmVAZnJlZWJzZC5vcmc+PG1haWx0bzpmcmVl YnNkLWRyaXZlcnMtdW5zdWJzY3JpYmVAZnJlZWJzZC5vcmc8bWFpbHRvOmZyZWVic2QtZHJpdmVy cy11bnN1YnNjcmliZUBmcmVlYnNkLm9yZz4+Ig0KPg0KPiBfX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fXw0KPiBmcmVlYnNkLWRyaXZlcnNAZnJlZWJzZC5vcmc8 bWFpbHRvOmZyZWVic2QtZHJpdmVyc0BmcmVlYnNkLm9yZz4gbWFpbGluZyBsaXN0DQo+IGh0dHBz Oi8vbGlzdHMuZnJlZWJzZC5vcmcvbWFpbG1hbi9saXN0aW5mby9mcmVlYnNkLWRyaXZlcnMNCj4g VG8gdW5zdWJzY3JpYmUsIHNlbmQgYW55IG1haWwgdG8gImZyZWVic2QtZHJpdmVycy11bnN1YnNj cmliZUBmcmVlYnNkLm9yZzxtYWlsdG86ZnJlZWJzZC1kcml2ZXJzLXVuc3Vic2NyaWJlQGZyZWVi c2Qub3JnPiINCg0KDQo=