From owner-freebsd-drivers@freebsd.org Tue Aug 15 14:59:09 2017 Return-Path: Delivered-To: freebsd-drivers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 61B55DD90EB for ; Tue, 15 Aug 2017 14:59:09 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from mail.baldwin.cx (bigwig.baldwin.cx [IPv6:2001:470:1f11:75::1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 40DCA72D6F for ; Tue, 15 Aug 2017 14:59:09 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from ralph.baldwin.cx (c-73-231-226-104.hsd1.ca.comcast.net [73.231.226.104]) by mail.baldwin.cx (Postfix) with ESMTPSA id 437F510AF07; Tue, 15 Aug 2017 10:59:07 -0400 (EDT) From: John Baldwin To: freebsd-drivers@freebsd.org Subject: Re: Driver crash detaches entire PCI bus? Date: Tue, 15 Aug 2017 07:59:03 -0700 Message-ID: <3743762.5XtXr8DZrt@ralph.baldwin.cx> User-Agent: KMail/4.14.10 (FreeBSD/11.1-STABLE; KDE/4.14.30; amd64; ; ) In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: 7Bit Content-Type: text/plain; charset="us-ascii" X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.4.3 (mail.baldwin.cx); Tue, 15 Aug 2017 10:59:07 -0400 (EDT) X-Virus-Scanned: clamav-milter 0.99.2 at mail.baldwin.cx X-Virus-Status: Clean X-BeenThere: freebsd-drivers@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Writing device drivers for FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Aug 2017 14:59:09 -0000 On Thursday, August 03, 2017 09:52:54 PM Farhan Khan wrote: > Hi all, I am trying to write the Realtek 8188e (pci) driver for FreeBSD. I > am > making slow incremental progress, and having the driver safely exit out (as > opposed to a segfault). I noticed that when the device detaches, due to an > error > in the driver, the entire pci bus will also detach! For example, here is my > kernel output: > > rtwn0: rtwn_load_firmware: failed to upload firmware rtwn-rtl8188eefw > (error 60) > rtwn0: detached > pci2: detached > > The first line is where the code crashes: sys/dev/rtwn/if_rtwn_fw.c line 129 > https://github.com/khanzf/freebsd/blob/103c05369b1ced770a2cadc9468e0134c8d9421b/sys/dev/rtwn/if_rtwn_fw.c#L129 > > Why would this result in the entire pci2 bus detatching? Strangely enough, > I did > not have this problem a few months ago, then it suddenly began. This is > significantly slowing down my workflow, and would be nice to have it > resolved. > Otherwise, I have to reboot to bring pci2 back up. You should be able to 'devctl attach pci2' I think without having to reboot. I would perhaps use dtrace to figure out why the PCI bus is detaching by just doing something like 'dtrace -n 'fbt::pci_detach:entry { stack(); }' to get a stack trace when pci_detach is called. We might be able to figure out the bug from there. -- John Baldwin From owner-freebsd-drivers@freebsd.org Thu Aug 17 20:54:17 2017 Return-Path: Delivered-To: freebsd-drivers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A3C91DD67AA for ; Thu, 17 Aug 2017 20:54:17 +0000 (UTC) (envelope-from Kinjal.Patel@taec.toshiba.com) Received: from mailhost.taec.toshiba.com (mailhost.taec.com [209.243.128.33]) by mx1.freebsd.org (Postfix) with ESMTP id 902CD6443B for ; Thu, 17 Aug 2017 20:54:17 +0000 (UTC) (envelope-from Kinjal.Patel@taec.toshiba.com) Received: from EXMAIL1.taec.toshiba.com (unknown [209.243.133.250]) by mailhost.taec.toshiba.com (Postfix) with ESMTP id 6C32CA0756 for ; Thu, 17 Aug 2017 13:45:39 -0700 (PDT) Received: from EXMAIL2.taec.toshiba.com (209.243.166.31) by EXMAIL1.taec.toshiba.com (209.243.172.198) with Microsoft SMTP Server (TLS) id 14.3.123.3; Thu, 17 Aug 2017 13:45:38 -0700 Received: from EXMAIL1.taec.toshiba.com ([169.254.1.230]) by EXMAIL2.taec.toshiba.com ([169.254.2.30]) with mapi id 14.02.0347.000; Thu, 17 Aug 2017 13:45:39 -0700 From: Kinjal Patel To: "freebsd-drivers@freebsd.org" Subject: NVMe driver init sequence Thread-Topic: NVMe driver init sequence Thread-Index: AdMXlSp3BhHvaAmDQQG9NyRAc27kig== Date: Thu, 17 Aug 2017 20:45:38 +0000 Message-ID: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.93.33.62] MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-drivers@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Writing device drivers for FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 17 Aug 2017 20:54:17 -0000 Hi, I have a query on NVMe driver init sequence in FreeBSD 10.3(stable). The driver init sequence is, 1) Enable controller (CC.EN=3D1) 2) Wait for controller ready (CSTS.RDY=3D1) 3) Set PCI bus master enable (BME=3D1) As per NVMe spec, when NVMe controller becomes ready it has to process comm= and. "Enable (EN): When set to '1', then the controller shall process commands b= ased on Submission Queue Tail doorbell writes" And per PCI Express spec when BME is not set, the PCI Express Function is n= ot allowed to issue any Memory or I/O requests. "Bus Master Enable - Controls the ability of a PCI Express Endpoint to issu= e Memory95 and I/O Read/Write Requests, and the ability of a Root or Switch Port to forward Memory and I/O Read/Write R= equests in the Upstream direction" Enabling controller before setting BME=3D1 is violation of spec, as control= ler has to accept commands but BME is prerequisite for that. The Linux device driver init sequence is, 1) Set PCI bus master enable (BME=3D1) 2) Enable Controller (CC.EN=3D1) 3) Wait for controller ready (CSTS.RDY=3D1) The FreeBSD NVMe driver sequence should be changed to set BME=3D1 before at= tempting to enable controller. Regards, Kinjal Patel From owner-freebsd-drivers@freebsd.org Thu Aug 17 22:10:00 2017 Return-Path: Delivered-To: freebsd-drivers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D21F9DDABEA for ; Thu, 17 Aug 2017 22:10:00 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-io0-x234.google.com (mail-io0-x234.google.com [IPv6:2607:f8b0:4001:c06::234]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 9D1C66710A for ; Thu, 17 Aug 2017 22:10:00 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: by mail-io0-x234.google.com with SMTP id c74so27652360iod.4 for ; Thu, 17 Aug 2017 15:10:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20150623.gappssmtp.com; s=20150623; h=mime-version:sender:in-reply-to:references:from:date:message-id :subject:to:cc; bh=IQxKxNHeCoAOpiWSkwxqvbZuX//HQJKiKC7HCiy8/co=; b=jhsSk2M6qBQ9u4OPMUdCDW9ynNKoFDP6LHLzfAFofBV96VrVlw3uaW2NC0QxLbS5KJ X3UaimS3/WxaUjVxLVFkxKZ0tSec6HqFxB/MFXVrlG4fTC1bFq+FVdiUdUAdk1V56kNF saU7exny0+GWVEsAWuy60DcDgdCbtvqELvdRqjgDsP8IGEaUAVtRqO1OwzPbs8dzgpga sdKhN1AD6cqmYEyS35RBghGwj8WrkQTD8dRe1qYwJWSjkTkiCjTU9UrApmb0a4zD6fxw 0x2ZURVza2SSo3bY1URS7AaEudvAFPqYQ+TQ5Ui7hJhOxM7sxzobu7c6zAq5jOvGwSBb yLeg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:sender:in-reply-to:references:from :date:message-id:subject:to:cc; bh=IQxKxNHeCoAOpiWSkwxqvbZuX//HQJKiKC7HCiy8/co=; b=HRQeCx7fhN2/rkDsKNF1pkzLhj8d6T6ivNLuBw2yZL6v0CTa4SXX7xO51BDDZsV8t+ qDteV4RtOYwnz49Bu/jOcthVYLPI4zHtclDUftk/E57jYLI98xjghU3ajk384ddR5Vmv pqQ3tCBQofrMwktiiDJKoOYiFITy4AoJqvi12bHWLZO6c7hf48xl3nUaa3s3bb4FiWG8 ED/QssCbJNzhQbnVJ0vFOXMIBObPBBdve5g0HkfOw+OcU0mQR2qEXiPtwxeEqTv3iO8U jfdSd2etU7JbTrmzHsDItdcX6LPmSEAacoSTw2JcVcwA6p+faLccCqyEQKmbStgIOtLx 9UOQ== X-Gm-Message-State: AHYfb5gk6KmS+UuP2KtkXMuatlIGpWWk2eKOVm/82jNs0Ew8yCvrlN/r FcEBaE6qqakSl7dXguKgxQt04CV9vF2I X-Received: by 10.107.3.234 with SMTP id e103mr6403556ioi.57.1503007799934; Thu, 17 Aug 2017 15:09:59 -0700 (PDT) MIME-Version: 1.0 Sender: wlosh@bsdimp.com Received: by 10.79.10.71 with HTTP; Thu, 17 Aug 2017 15:09:59 -0700 (PDT) X-Originating-IP: [69.27.219.53] In-Reply-To: References: From: Warner Losh Date: Thu, 17 Aug 2017 16:09:59 -0600 X-Google-Sender-Auth: sxqxDkiuusAb8i0GX5HgomvO2sc Message-ID: Subject: Re: NVMe driver init sequence To: Kinjal Patel Cc: "freebsd-drivers@freebsd.org" Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-drivers@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Writing device drivers for FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 17 Aug 2017 22:10:00 -0000 This sounds right to me.... Most BIOSes seem to enable BME=1. Warner On Thu, Aug 17, 2017 at 2:45 PM, Kinjal Patel wrote: > Hi, > > I have a query on NVMe driver init sequence in FreeBSD 10.3(stable). > > The driver init sequence is, > > 1) Enable controller (CC.EN=1) > > 2) Wait for controller ready (CSTS.RDY=1) > > 3) Set PCI bus master enable (BME=1) > > As per NVMe spec, when NVMe controller becomes ready it has to process > command. > > "Enable (EN): When set to '1', then the controller shall process commands > based on Submission Queue Tail doorbell writes" > > And per PCI Express spec when BME is not set, the PCI Express Function is > not allowed to issue any Memory or I/O requests. > "Bus Master Enable - Controls the ability of a PCI Express Endpoint to > issue Memory95 and I/O Read/Write Requests, and > the ability of a Root or Switch Port to forward Memory and I/O Read/Write > Requests in the Upstream direction" > > Enabling controller before setting BME=1 is violation of spec, as > controller has to accept commands but BME is prerequisite for that. > > The Linux device driver init sequence is, > > 1) Set PCI bus master enable (BME=1) > > 2) Enable Controller (CC.EN=1) > > 3) Wait for controller ready (CSTS.RDY=1) > > The FreeBSD NVMe driver sequence should be changed to set BME=1 before > attempting to enable controller. > > > Regards, > Kinjal Patel > _______________________________________________ > freebsd-drivers@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-drivers > To unsubscribe, send any mail to "freebsd-drivers-unsubscribe@freebsd.org" > From owner-freebsd-drivers@freebsd.org Fri Aug 18 20:51:02 2017 Return-Path: Delivered-To: freebsd-drivers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 68583DD6564 for ; Fri, 18 Aug 2017 20:51:02 +0000 (UTC) (envelope-from Kinjal.Patel@taec.toshiba.com) Received: from mailhost.taec.toshiba.com (mailhost.taec.com [209.243.128.33]) by mx1.freebsd.org (Postfix) with ESMTP id 52DB67507A for ; Fri, 18 Aug 2017 20:51:01 +0000 (UTC) (envelope-from Kinjal.Patel@taec.toshiba.com) Received: from EXMAIL1.taec.toshiba.com (unknown [209.243.133.250]) by mailhost.taec.toshiba.com (Postfix) with ESMTP id 55394A089B; Fri, 18 Aug 2017 13:50:55 -0700 (PDT) Received: from EXMAIL2.taec.toshiba.com (209.243.166.31) by EXMAIL1.taec.toshiba.com (209.243.172.198) with Microsoft SMTP Server (TLS) id 14.3.123.3; Fri, 18 Aug 2017 13:50:55 -0700 Received: from EXMAIL1.taec.toshiba.com ([169.254.1.230]) by EXMAIL2.taec.toshiba.com ([169.254.2.30]) with mapi id 14.02.0347.000; Fri, 18 Aug 2017 13:50:55 -0700 From: Kinjal Patel To: Warner Losh CC: "freebsd-drivers@freebsd.org" Subject: RE: NVMe driver init sequence Thread-Topic: NVMe driver init sequence Thread-Index: AdMXlSp3BhHvaAmDQQG9NyRAc27kigASxKyAACB3OKA= Date: Fri, 18 Aug 2017 20:50:54 +0000 Message-ID: References: In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.93.34.111] MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-drivers@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Writing device drivers for FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 18 Aug 2017 20:51:02 -0000 SGVyZSBpcyBhIHRoZSBjaGFuZ2UgSSBwcm9wb3NlLiBUaGlzIHdpbGwgbWFrZSB0aGUgaW5pdCBz ZXF1ZW5jZSByaWdodC4NCg0KSW5kZXg6IHN5cy9kZXYvbnZtZS9udm1lLmMNCj09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT0N Ci0tLSBzeXMvZGV2L252bWUvbnZtZS5jIChyZXZpc2lvbiAzMjI2NzIpDQorKysgc3lzL2Rldi9u dm1lL252bWUuYyAod29ya2luZyBjb3B5KQ0KQEAgLTI1Myw2ICsyNTMsOSBAQA0KICAgICAgICAg ICAgICAgIHJldHVybiAoc3RhdHVzKTsNCiAgICAgICAgfQ0KDQorICAgIC8qIG1ha2UgZGV2aWNl IGJ1cy1tYXN0ZXIgYmVmb3JlIGF0dGVtcHRpbmcgdG8gZW5hYmxlIGNvbnRyb2xsZXIgKi8NCisg ICAgcGNpX2VuYWJsZV9idXNtYXN0ZXIoZGV2KTsNCisNCiAgICAgICAgLyoNCiAgICAgICAgICog UmVzZXQgY29udHJvbGxlciB0d2ljZSB0byBlbnN1cmUgd2UgZG8gYSB0cmFuc2l0aW9uIGZyb20g Y2MuZW49PTENCiAgICAgICAgICogIHRvIGNjLmVuPT0wLiAgVGhpcyBpcyBiZWNhdXNlIHdlIGRv bid0IHJlYWxseSBrbm93IHdoYXQgc3RhdHVzDQpAQCAtMjcwLDggKzI3Myw2IEBADQogICAgICAg ICAgICAgICAgcmV0dXJuIChzdGF0dXMpOw0KICAgICAgICB9DQoNCi0gICAgICAgcGNpX2VuYWJs ZV9idXNtYXN0ZXIoZGV2KTsNCi0NCiAgICAgICAgY3RybHItPmNvbmZpZ19ob29rLmljaF9mdW5j ID0gbnZtZV9jdHJscl9zdGFydF9jb25maWdfaG9vazsNCiAgICAgICAgY3RybHItPmNvbmZpZ19o b29rLmljaF9hcmcgPSBjdHJscjsNCg0KDQoNCnJlZ2FyZHMsDQpLaW5qYWwgUGF0ZWwNCg0KRnJv bTogd2xvc2hAYnNkaW1wLmNvbSBbbWFpbHRvOndsb3NoQGJzZGltcC5jb21dIE9uIEJlaGFsZiBP ZiBXYXJuZXIgTG9zaA0KU2VudDogVGh1cnNkYXksIEF1Z3VzdCAxNywgMjAxNyAzOjEwIFBNDQpU bzogS2luamFsIFBhdGVsDQpDYzogZnJlZWJzZC1kcml2ZXJzQGZyZWVic2Qub3JnDQpTdWJqZWN0 OiBSZTogTlZNZSBkcml2ZXIgaW5pdCBzZXF1ZW5jZQ0KDQpUaGlzIHNvdW5kcyByaWdodCB0byBt ZS4uLi4gIE1vc3QgQklPU2VzIHNlZW0gdG8gZW5hYmxlIEJNRT0xLg0KDQpXYXJuZXINCg0KT24g VGh1LCBBdWcgMTcsIDIwMTcgYXQgMjo0NSBQTSwgS2luamFsIFBhdGVsIDxLaW5qYWwuUGF0ZWxA dGFlYy50b3NoaWJhLmNvbTxtYWlsdG86S2luamFsLlBhdGVsQHRhZWMudG9zaGliYS5jb20+PiB3 cm90ZToNCkhpLA0KDQpJIGhhdmUgYSBxdWVyeSBvbiBOVk1lIGRyaXZlciBpbml0IHNlcXVlbmNl IGluIEZyZWVCU0QgMTAuMyhzdGFibGUpLg0KDQpUaGUgZHJpdmVyIGluaXQgc2VxdWVuY2UgaXMs DQoNCjEpICAgICAgRW5hYmxlIGNvbnRyb2xsZXIgKENDLkVOPTEpDQoNCjIpICAgICAgV2FpdCBm b3IgY29udHJvbGxlciByZWFkeSAoQ1NUUy5SRFk9MSkNCg0KMykgICAgICBTZXQgUENJIGJ1cyBt YXN0ZXIgZW5hYmxlIChCTUU9MSkNCg0KQXMgcGVyIE5WTWUgc3BlYywgd2hlbiBOVk1lIGNvbnRy b2xsZXIgYmVjb21lcyByZWFkeSBpdCBoYXMgdG8gcHJvY2VzcyBjb21tYW5kLg0KDQoiRW5hYmxl IChFTik6IFdoZW4gc2V0IHRvICcxJywgdGhlbiB0aGUgY29udHJvbGxlciBzaGFsbCBwcm9jZXNz IGNvbW1hbmRzIGJhc2VkIG9uIFN1Ym1pc3Npb24gUXVldWUgVGFpbCBkb29yYmVsbCB3cml0ZXMi DQoNCkFuZCBwZXIgUENJIEV4cHJlc3Mgc3BlYyB3aGVuIEJNRSBpcyBub3Qgc2V0LCB0aGUgUENJ IEV4cHJlc3MgRnVuY3Rpb24gaXMgbm90IGFsbG93ZWQgdG8gaXNzdWUgYW55IE1lbW9yeSBvciBJ L08gcmVxdWVzdHMuDQoiQnVzIE1hc3RlciBFbmFibGUgLSBDb250cm9scyB0aGUgYWJpbGl0eSBv ZiBhIFBDSSBFeHByZXNzIEVuZHBvaW50IHRvIGlzc3VlIE1lbW9yeTk1IGFuZCBJL08gUmVhZC9X cml0ZSBSZXF1ZXN0cywgYW5kDQp0aGUgYWJpbGl0eSBvZiBhIFJvb3Qgb3IgU3dpdGNoIFBvcnQg dG8gZm9yd2FyZCBNZW1vcnkgYW5kIEkvTyBSZWFkL1dyaXRlIFJlcXVlc3RzIGluIHRoZSBVcHN0 cmVhbSBkaXJlY3Rpb24iDQoNCkVuYWJsaW5nIGNvbnRyb2xsZXIgYmVmb3JlIHNldHRpbmcgQk1F PTEgaXMgdmlvbGF0aW9uIG9mIHNwZWMsIGFzIGNvbnRyb2xsZXIgaGFzIHRvIGFjY2VwdCBjb21t YW5kcyBidXQgQk1FIGlzIHByZXJlcXVpc2l0ZSBmb3IgdGhhdC4NCg0KVGhlIExpbnV4IGRldmlj ZSBkcml2ZXIgaW5pdCBzZXF1ZW5jZSBpcywNCg0KMSkgICAgIFNldCBQQ0kgYnVzIG1hc3RlciBl bmFibGUgKEJNRT0xKQ0KDQoyKSAgICAgRW5hYmxlIENvbnRyb2xsZXIgKENDLkVOPTEpDQoNCjMp ICAgICBXYWl0IGZvciBjb250cm9sbGVyIHJlYWR5IChDU1RTLlJEWT0xKQ0KDQpUaGUgRnJlZUJT RCBOVk1lIGRyaXZlciBzZXF1ZW5jZSBzaG91bGQgYmUgY2hhbmdlZCB0byBzZXQgQk1FPTEgYmVm b3JlIGF0dGVtcHRpbmcgdG8gZW5hYmxlIGNvbnRyb2xsZXIuDQoNCg0KUmVnYXJkcywNCktpbmph bCBQYXRlbA0KX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18N CmZyZWVic2QtZHJpdmVyc0BmcmVlYnNkLm9yZzxtYWlsdG86ZnJlZWJzZC1kcml2ZXJzQGZyZWVi c2Qub3JnPiBtYWlsaW5nIGxpc3QNCmh0dHBzOi8vbGlzdHMuZnJlZWJzZC5vcmcvbWFpbG1hbi9s aXN0aW5mby9mcmVlYnNkLWRyaXZlcnMNClRvIHVuc3Vic2NyaWJlLCBzZW5kIGFueSBtYWlsIHRv ICJmcmVlYnNkLWRyaXZlcnMtdW5zdWJzY3JpYmVAZnJlZWJzZC5vcmc8bWFpbHRvOmZyZWVic2Qt ZHJpdmVycy11bnN1YnNjcmliZUBmcmVlYnNkLm9yZz4iDQoNCg== From owner-freebsd-drivers@freebsd.org Fri Aug 18 21:04:03 2017 Return-Path: Delivered-To: freebsd-drivers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id CADFADD726D for ; Fri, 18 Aug 2017 21:04:03 +0000 (UTC) (envelope-from Kinjal.Patel@taec.toshiba.com) Received: from mailhost.taec.toshiba.com (mailhost.taec.toshiba.com [209.243.128.33]) by mx1.freebsd.org (Postfix) with ESMTP id B7A4D75A10 for ; Fri, 18 Aug 2017 21:04:03 +0000 (UTC) (envelope-from Kinjal.Patel@taec.toshiba.com) Received: from EXMAIL1.taec.toshiba.com (unknown [209.243.133.250]) by mailhost.taec.toshiba.com (Postfix) with ESMTP id B2431A0742; Fri, 18 Aug 2017 14:04:02 -0700 (PDT) Received: from EXMAIL2.taec.toshiba.com (209.243.166.31) by EXMAIL1.taec.toshiba.com (209.243.172.198) with Microsoft SMTP Server (TLS) id 14.3.123.3; Fri, 18 Aug 2017 14:04:02 -0700 Received: from EXMAIL1.taec.toshiba.com ([169.254.1.230]) by EXMAIL2.taec.toshiba.com ([169.254.2.30]) with mapi id 14.02.0347.000; Fri, 18 Aug 2017 14:04:02 -0700 From: Kinjal Patel To: Warner Losh CC: "freebsd-drivers@freebsd.org" Subject: RE: NVMe driver init sequence Thread-Topic: NVMe driver init sequence Thread-Index: AdMXlSp3BhHvaAmDQQG9NyRAc27kigASxKyAACB3OKAAAM0YcA== Date: Fri, 18 Aug 2017 21:04:01 +0000 Message-ID: References: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.93.34.111] MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-drivers@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Writing device drivers for FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 18 Aug 2017 21:04:03 -0000 Q3JlYXRlZCBhIGJ1ZyAjMjIxNjYgKGh0dHBzOi8vYnVncy5mcmVlYnNkLm9yZy9idWd6aWxsYS9z aG93X2J1Zy5jZ2k/aWQ9MjIxNjE2KQ0KVXBkYXRlZCB0aGUgZGVzY3JpcHRpb24gYW5kIHBhdGNo IHRvIGl0Lg0KDQpSZWdhcmRzLA0KS2luamFsIFBhdGVsDQoNCkZyb206IEtpbmphbCBQYXRlbA0K U2VudDogRnJpZGF5LCBBdWd1c3QgMTgsIDIwMTcgMTo1MSBQTQ0KVG86ICdXYXJuZXIgTG9zaCcN CkNjOiBmcmVlYnNkLWRyaXZlcnNAZnJlZWJzZC5vcmcNClN1YmplY3Q6IFJFOiBOVk1lIGRyaXZl ciBpbml0IHNlcXVlbmNlDQoNCkhlcmUgaXMgYSB0aGUgY2hhbmdlIEkgcHJvcG9zZS4gVGhpcyB3 aWxsIG1ha2UgdGhlIGluaXQgc2VxdWVuY2UgcmlnaHQuDQoNCkluZGV4OiBzeXMvZGV2L252bWUv bnZtZS5jDQo9PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09DQotLS0gc3lzL2Rldi9udm1lL252bWUuYyAocmV2aXNpb24gMzIy NjcyKQ0KKysrIHN5cy9kZXYvbnZtZS9udm1lLmMgKHdvcmtpbmcgY29weSkNCkBAIC0yNTMsNiAr MjUzLDkgQEANCiAgICAgICAgICAgICAgICByZXR1cm4gKHN0YXR1cyk7DQogICAgICAgIH0NCg0K KyAgICAvKiBtYWtlIGRldmljZSBidXMtbWFzdGVyIGJlZm9yZSBhdHRlbXB0aW5nIHRvIGVuYWJs ZSBjb250cm9sbGVyICovDQorICAgIHBjaV9lbmFibGVfYnVzbWFzdGVyKGRldik7DQorDQogICAg ICAgIC8qDQogICAgICAgICAqIFJlc2V0IGNvbnRyb2xsZXIgdHdpY2UgdG8gZW5zdXJlIHdlIGRv IGEgdHJhbnNpdGlvbiBmcm9tIGNjLmVuPT0xDQogICAgICAgICAqICB0byBjYy5lbj09MC4gIFRo aXMgaXMgYmVjYXVzZSB3ZSBkb24ndCByZWFsbHkga25vdyB3aGF0IHN0YXR1cw0KQEAgLTI3MCw4 ICsyNzMsNiBAQA0KICAgICAgICAgICAgICAgIHJldHVybiAoc3RhdHVzKTsNCiAgICAgICAgfQ0K DQotICAgICAgIHBjaV9lbmFibGVfYnVzbWFzdGVyKGRldik7DQotDQogICAgICAgIGN0cmxyLT5j b25maWdfaG9vay5pY2hfZnVuYyA9IG52bWVfY3RybHJfc3RhcnRfY29uZmlnX2hvb2s7DQogICAg ICAgIGN0cmxyLT5jb25maWdfaG9vay5pY2hfYXJnID0gY3RybHI7DQoNCg0KDQpyZWdhcmRzLA0K S2luamFsIFBhdGVsDQoNCkZyb206IHdsb3NoQGJzZGltcC5jb208bWFpbHRvOndsb3NoQGJzZGlt cC5jb20+IFttYWlsdG86d2xvc2hAYnNkaW1wLmNvbV0gT24gQmVoYWxmIE9mIFdhcm5lciBMb3No DQpTZW50OiBUaHVyc2RheSwgQXVndXN0IDE3LCAyMDE3IDM6MTAgUE0NClRvOiBLaW5qYWwgUGF0 ZWwNCkNjOiBmcmVlYnNkLWRyaXZlcnNAZnJlZWJzZC5vcmc8bWFpbHRvOmZyZWVic2QtZHJpdmVy c0BmcmVlYnNkLm9yZz4NClN1YmplY3Q6IFJlOiBOVk1lIGRyaXZlciBpbml0IHNlcXVlbmNlDQoN ClRoaXMgc291bmRzIHJpZ2h0IHRvIG1lLi4uLiAgTW9zdCBCSU9TZXMgc2VlbSB0byBlbmFibGUg Qk1FPTEuDQoNCldhcm5lcg0KDQpPbiBUaHUsIEF1ZyAxNywgMjAxNyBhdCAyOjQ1IFBNLCBLaW5q YWwgUGF0ZWwgPEtpbmphbC5QYXRlbEB0YWVjLnRvc2hpYmEuY29tPG1haWx0bzpLaW5qYWwuUGF0 ZWxAdGFlYy50b3NoaWJhLmNvbT4+IHdyb3RlOg0KSGksDQoNCkkgaGF2ZSBhIHF1ZXJ5IG9uIE5W TWUgZHJpdmVyIGluaXQgc2VxdWVuY2UgaW4gRnJlZUJTRCAxMC4zKHN0YWJsZSkuDQoNClRoZSBk cml2ZXIgaW5pdCBzZXF1ZW5jZSBpcywNCg0KMSkgICAgICBFbmFibGUgY29udHJvbGxlciAoQ0Mu RU49MSkNCg0KMikgICAgICBXYWl0IGZvciBjb250cm9sbGVyIHJlYWR5IChDU1RTLlJEWT0xKQ0K DQozKSAgICAgIFNldCBQQ0kgYnVzIG1hc3RlciBlbmFibGUgKEJNRT0xKQ0KDQpBcyBwZXIgTlZN ZSBzcGVjLCB3aGVuIE5WTWUgY29udHJvbGxlciBiZWNvbWVzIHJlYWR5IGl0IGhhcyB0byBwcm9j ZXNzIGNvbW1hbmQuDQoNCiJFbmFibGUgKEVOKTogV2hlbiBzZXQgdG8gJzEnLCB0aGVuIHRoZSBj b250cm9sbGVyIHNoYWxsIHByb2Nlc3MgY29tbWFuZHMgYmFzZWQgb24gU3VibWlzc2lvbiBRdWV1 ZSBUYWlsIGRvb3JiZWxsIHdyaXRlcyINCg0KQW5kIHBlciBQQ0kgRXhwcmVzcyBzcGVjIHdoZW4g Qk1FIGlzIG5vdCBzZXQsIHRoZSBQQ0kgRXhwcmVzcyBGdW5jdGlvbiBpcyBub3QgYWxsb3dlZCB0 byBpc3N1ZSBhbnkgTWVtb3J5IG9yIEkvTyByZXF1ZXN0cy4NCiJCdXMgTWFzdGVyIEVuYWJsZSAt IENvbnRyb2xzIHRoZSBhYmlsaXR5IG9mIGEgUENJIEV4cHJlc3MgRW5kcG9pbnQgdG8gaXNzdWUg TWVtb3J5OTUgYW5kIEkvTyBSZWFkL1dyaXRlIFJlcXVlc3RzLCBhbmQNCnRoZSBhYmlsaXR5IG9m IGEgUm9vdCBvciBTd2l0Y2ggUG9ydCB0byBmb3J3YXJkIE1lbW9yeSBhbmQgSS9PIFJlYWQvV3Jp dGUgUmVxdWVzdHMgaW4gdGhlIFVwc3RyZWFtIGRpcmVjdGlvbiINCg0KRW5hYmxpbmcgY29udHJv bGxlciBiZWZvcmUgc2V0dGluZyBCTUU9MSBpcyB2aW9sYXRpb24gb2Ygc3BlYywgYXMgY29udHJv bGxlciBoYXMgdG8gYWNjZXB0IGNvbW1hbmRzIGJ1dCBCTUUgaXMgcHJlcmVxdWlzaXRlIGZvciB0 aGF0Lg0KDQpUaGUgTGludXggZGV2aWNlIGRyaXZlciBpbml0IHNlcXVlbmNlIGlzLA0KDQoxKSAg ICAgU2V0IFBDSSBidXMgbWFzdGVyIGVuYWJsZSAoQk1FPTEpDQoNCjIpICAgICBFbmFibGUgQ29u dHJvbGxlciAoQ0MuRU49MSkNCg0KMykgICAgIFdhaXQgZm9yIGNvbnRyb2xsZXIgcmVhZHkgKENT VFMuUkRZPTEpDQoNClRoZSBGcmVlQlNEIE5WTWUgZHJpdmVyIHNlcXVlbmNlIHNob3VsZCBiZSBj aGFuZ2VkIHRvIHNldCBCTUU9MSBiZWZvcmUgYXR0ZW1wdGluZyB0byBlbmFibGUgY29udHJvbGxl ci4NCg0KDQpSZWdhcmRzLA0KS2luamFsIFBhdGVsDQpfX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fXw0KZnJlZWJzZC1kcml2ZXJzQGZyZWVic2Qub3JnPG1haWx0 bzpmcmVlYnNkLWRyaXZlcnNAZnJlZWJzZC5vcmc+IG1haWxpbmcgbGlzdA0KaHR0cHM6Ly9saXN0 cy5mcmVlYnNkLm9yZy9tYWlsbWFuL2xpc3RpbmZvL2ZyZWVic2QtZHJpdmVycw0KVG8gdW5zdWJz Y3JpYmUsIHNlbmQgYW55IG1haWwgdG8gImZyZWVic2QtZHJpdmVycy11bnN1YnNjcmliZUBmcmVl YnNkLm9yZzxtYWlsdG86ZnJlZWJzZC1kcml2ZXJzLXVuc3Vic2NyaWJlQGZyZWVic2Qub3JnPiIN Cg0K From owner-freebsd-drivers@freebsd.org Fri Aug 18 22:09:49 2017 Return-Path: Delivered-To: freebsd-drivers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5624ADDB162 for ; Fri, 18 Aug 2017 22:09:49 +0000 (UTC) (envelope-from jim.harris@gmail.com) Received: from mail-qk0-x244.google.com (mail-qk0-x244.google.com [IPv6:2607:f8b0:400d:c09::244]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 1614277BC9 for ; Fri, 18 Aug 2017 22:09:49 +0000 (UTC) (envelope-from jim.harris@gmail.com) Received: by mail-qk0-x244.google.com with SMTP id 130so2098955qkg.5 for ; Fri, 18 Aug 2017 15:09:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=BDk32loU3z97zLdArU+rbIy5fevuig7O+hqzE3fgIyM=; b=hS/Ln4L17QUW8asUfSl+zBn+URHbJxmp+FYNuveZa6NnhVywnmJWscKg6cvL7cMrIi BuURKuh9MzarmUn3TpGTrpr7PYZupiC9y0kNXoMqHEgwGoq161V10oPgVPZpKOE0GVpL WMMmSicm4p4Fb6tzmIhL27n9zQWYH+uykPOJq8xnjPdyWD70U5ryOMAK3day/oeHB9qo mymIAh4EeRa9t0rBNkr0GxS5ghgs13oULmWZuJ/ULazXXE6P07len+tqvXOzNCoo/nkM 6wpQR/kuOkIoNp5BApm5vWMhc1U2h00Ar83ZSt41+8HwnClMYDCsIzIJPrDqcNrcUVnF 8rNw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=BDk32loU3z97zLdArU+rbIy5fevuig7O+hqzE3fgIyM=; b=ULX2O/8Wjr9TZdV1cJ8qPtIKdA1FPrKxW1rWM4KzODj7UV4DT1p8fw/GPN82gsyKO1 bwassy7WmQIfnlb5Mu5vaxS1a4Ewi6sV1lujr/Zgi6+7g9XKDK/vbMr+vAbvSGnsu1AD wspUE7wgsBZjWhUZXvuzhBJ0bQ+19Gh5FnbBK6TKK2RWvSRtr20MpAyXbphzuB2dUD4F gYVCmX7tJD5wM8NvK2buNLZYqyLUdxh68WPP1HhvtS9vD3b34cndbYy6yoyYNPhKbSyS vSQj+qnAVwTqNQKIb/2Irx/E6VDBNNCjGvXTchrcx7JLDmwmFYeH6DYibOZF+67H45Vm gofA== X-Gm-Message-State: AHYfb5gx5brkql4kL2tz6Sq2suvf6UB02Ch+Ue1FzgjRWPVLLGJgEC/O /Ggrt06f+RJBBXpsvaI5RFhtIn/wbA== X-Received: by 10.55.210.133 with SMTP id f127mr14992183qkj.109.1503094188224; Fri, 18 Aug 2017 15:09:48 -0700 (PDT) MIME-Version: 1.0 Received: by 10.140.38.139 with HTTP; Fri, 18 Aug 2017 15:09:47 -0700 (PDT) In-Reply-To: References: From: Jim Harris Date: Fri, 18 Aug 2017 15:09:47 -0700 Message-ID: Subject: Re: NVMe driver init sequence To: Kinjal Patel Cc: Warner Losh , "freebsd-drivers@freebsd.org" Content-Type: text/plain; charset="UTF-8" X-BeenThere: freebsd-drivers@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Writing device drivers for FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 18 Aug 2017 22:09:49 -0000 On Fri, Aug 18, 2017 at 2:04 PM, Kinjal Patel wrote: > Created a bug #22166 (https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=221616) > Updated the description and patch to it. > > Regards, > Kinjal Patel > > From: Kinjal Patel > Sent: Friday, August 18, 2017 1:51 PM > To: 'Warner Losh' > Cc: freebsd-drivers@freebsd.org > Subject: RE: NVMe driver init sequence > > Here is a the change I propose. This will make the init sequence right. Looks good to me. > > Index: sys/dev/nvme/nvme.c > =================================================================== > --- sys/dev/nvme/nvme.c (revision 322672) > +++ sys/dev/nvme/nvme.c (working copy) > @@ -253,6 +253,9 @@ > return (status); > } > > + /* make device bus-master before attempting to enable controller */ > + pci_enable_busmaster(dev); > + > /* > * Reset controller twice to ensure we do a transition from cc.en==1 > * to cc.en==0. This is because we don't really know what status > @@ -270,8 +273,6 @@ > return (status); > } > > - pci_enable_busmaster(dev); > - > ctrlr->config_hook.ich_func = nvme_ctrlr_start_config_hook; > ctrlr->config_hook.ich_arg = ctrlr; > > > > regards, > Kinjal Patel > > From: wlosh@bsdimp.com [mailto:wlosh@bsdimp.com] On Behalf Of Warner Losh > Sent: Thursday, August 17, 2017 3:10 PM > To: Kinjal Patel > Cc: freebsd-drivers@freebsd.org > Subject: Re: NVMe driver init sequence > > This sounds right to me.... Most BIOSes seem to enable BME=1. > > Warner > > On Thu, Aug 17, 2017 at 2:45 PM, Kinjal Patel > wrote: > Hi, > > I have a query on NVMe driver init sequence in FreeBSD 10.3(stable). > > The driver init sequence is, > > 1) Enable controller (CC.EN=1) > > 2) Wait for controller ready (CSTS.RDY=1) > > 3) Set PCI bus master enable (BME=1) > > As per NVMe spec, when NVMe controller becomes ready it has to process command. > > "Enable (EN): When set to '1', then the controller shall process commands based on Submission Queue Tail doorbell writes" > > And per PCI Express spec when BME is not set, the PCI Express Function is not allowed to issue any Memory or I/O requests. > "Bus Master Enable - Controls the ability of a PCI Express Endpoint to issue Memory95 and I/O Read/Write Requests, and > the ability of a Root or Switch Port to forward Memory and I/O Read/Write Requests in the Upstream direction" > > Enabling controller before setting BME=1 is violation of spec, as controller has to accept commands but BME is prerequisite for that. > > The Linux device driver init sequence is, > > 1) Set PCI bus master enable (BME=1) > > 2) Enable Controller (CC.EN=1) > > 3) Wait for controller ready (CSTS.RDY=1) > > The FreeBSD NVMe driver sequence should be changed to set BME=1 before attempting to enable controller. > > > Regards, > Kinjal Patel > _______________________________________________ > freebsd-drivers@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-drivers > To unsubscribe, send any mail to "freebsd-drivers-unsubscribe@freebsd.org" > > _______________________________________________ > freebsd-drivers@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-drivers > To unsubscribe, send any mail to "freebsd-drivers-unsubscribe@freebsd.org"