From owner-freebsd-drivers@freebsd.org Fri Aug 25 03:16:51 2017 Return-Path: Delivered-To: freebsd-drivers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id CD49CDEE07A for ; Fri, 25 Aug 2017 03:16:51 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-it0-x22c.google.com (mail-it0-x22c.google.com [IPv6:2607:f8b0:4001:c0b::22c]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 957917099E for ; Fri, 25 Aug 2017 03:16:51 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: by mail-it0-x22c.google.com with SMTP id b76so2073171itb.0 for ; Thu, 24 Aug 2017 20:16:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20150623.gappssmtp.com; s=20150623; h=mime-version:sender:in-reply-to:references:from:date:message-id :subject:to:cc; bh=Qg4b8HSke2XCoOhG+thuQ+3yB/Kft6eVumAzSO5sBfw=; b=tRPEKSVB/11BuEn+UYSf5kkbUb0HE6z4TuCIPCkBksdGVvHmJ9voVbTgWxOGqa8Omp fA9RrMA8nsHklLW5GmAgkCdmndcjvsZ8jnJao/xia77Sdj6ha0KlkYLElG7plua2qju9 h7nSxNoMvFBaPahXIi/RAXZu4s2E0mIARBIm2uggpEKjHcTdkNNd+VO02hLQfBT55NTM qvTQn949Y71nyKRPt4DadmIPccsjbrY0dKo+rjPlbIvy8FdhRWhOXFLpfUEOCxL0Zl/K wl+1cpx0x8B4Nbq1gQvBCPDHR0zbF6GY/F+5eXp4oMFe0harFkLddFyrHzvD6JhpWH+z IT0A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:sender:in-reply-to:references:from :date:message-id:subject:to:cc; bh=Qg4b8HSke2XCoOhG+thuQ+3yB/Kft6eVumAzSO5sBfw=; b=IivO7fldSnZDhPO3CrIoyYnTM6wAhksIQw9NXbuuY4EdavAhCgVn6AHY3qBB8cHdms iJGdofkVZ998ObIcH23MU59rzcqoZT2hkv8AMsN86SpcIxxTbOQRaeKzmTnLfkFCvTt2 6O8M6IBKp9HgZRfZ14y1rJSccF4TxQieS7L31knJcFflnrdHiLkLl0igcVOG/pdnH5Wa eSl+VcNniGsm9rDA12f+ioIeyD0tZ2ydN6VhOp6sR610VImbUK2vkIB1Hx12fGgKugLK TmKJVVNFU/mTCyk9r3BFFvT0IEddQX+QCJywcLoVpTWeXXXI1E75R0vdzZNzDTsefQtI k0Fg== X-Gm-Message-State: AHYfb5hUQP9wVESO2UofahymFqr96koYCcOJoaSaluRvb3GPf5tRO6Su poTtBoeJZmJ0+6XmiRSiTSBbg4NcWS40 X-Received: by 10.36.65.21 with SMTP id x21mr780898ita.24.1503631010879; Thu, 24 Aug 2017 20:16:50 -0700 (PDT) MIME-Version: 1.0 Sender: wlosh@bsdimp.com Received: by 10.79.10.71 with HTTP; Thu, 24 Aug 2017 20:16:50 -0700 (PDT) X-Originating-IP: [2603:300b:6:5100:9c7e:5904:e863:717e] In-Reply-To: References: From: Warner Losh Date: Thu, 24 Aug 2017 21:16:50 -0600 X-Google-Sender-Auth: XWAVkV4nDQVfFVqFgPTaH-Pp_Bs Message-ID: Subject: Re: NVMe driver init sequence To: Jim Harris Cc: Kinjal Patel , "freebsd-drivers@freebsd.org" Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-drivers@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Writing device drivers for FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 25 Aug 2017 03:16:51 -0000 Greetings, I've just committed r322872 which implements this, with a slightly tweaked comment. Warner On Fri, Aug 18, 2017 at 4:09 PM, Jim Harris wrote: > On Fri, Aug 18, 2017 at 2:04 PM, Kinjal Patel > wrote: > > Created a bug #22166 (https://bugs.freebsd.org/bugzilla/show_bug.cgi?id= > 221616) > > Updated the description and patch to it. > > > > Regards, > > Kinjal Patel > > > > From: Kinjal Patel > > Sent: Friday, August 18, 2017 1:51 PM > > To: 'Warner Losh' > > Cc: freebsd-drivers@freebsd.org > > Subject: RE: NVMe driver init sequence > > > > Here is a the change I propose. This will make the init sequence right. > > Looks good to me. > > > > > Index: sys/dev/nvme/nvme.c > > =================================================================== > > --- sys/dev/nvme/nvme.c (revision 322672) > > +++ sys/dev/nvme/nvme.c (working copy) > > @@ -253,6 +253,9 @@ > > return (status); > > } > > > > + /* make device bus-master before attempting to enable controller */ > > + pci_enable_busmaster(dev); > > + > > /* > > * Reset controller twice to ensure we do a transition from > cc.en==1 > > * to cc.en==0. This is because we don't really know what > status > > @@ -270,8 +273,6 @@ > > return (status); > > } > > > > - pci_enable_busmaster(dev); > > - > > ctrlr->config_hook.ich_func = nvme_ctrlr_start_config_hook; > > ctrlr->config_hook.ich_arg = ctrlr; > > > > > > > > regards, > > Kinjal Patel > > > > From: wlosh@bsdimp.com [mailto:wlosh@bsdimp.com] > On Behalf Of Warner Losh > > Sent: Thursday, August 17, 2017 3:10 PM > > To: Kinjal Patel > > Cc: freebsd-drivers@freebsd.org > > Subject: Re: NVMe driver init sequence > > > > This sounds right to me.... Most BIOSes seem to enable BME=1. > > > > Warner > > > > On Thu, Aug 17, 2017 at 2:45 PM, Kinjal Patel < > Kinjal.Patel@taec.toshiba.com> > wrote: > > Hi, > > > > I have a query on NVMe driver init sequence in FreeBSD 10.3(stable). > > > > The driver init sequence is, > > > > 1) Enable controller (CC.EN=1) > > > > 2) Wait for controller ready (CSTS.RDY=1) > > > > 3) Set PCI bus master enable (BME=1) > > > > As per NVMe spec, when NVMe controller becomes ready it has to process > command. > > > > "Enable (EN): When set to '1', then the controller shall process > commands based on Submission Queue Tail doorbell writes" > > > > And per PCI Express spec when BME is not set, the PCI Express Function > is not allowed to issue any Memory or I/O requests. > > "Bus Master Enable - Controls the ability of a PCI Express Endpoint to > issue Memory95 and I/O Read/Write Requests, and > > the ability of a Root or Switch Port to forward Memory and I/O > Read/Write Requests in the Upstream direction" > > > > Enabling controller before setting BME=1 is violation of spec, as > controller has to accept commands but BME is prerequisite for that. > > > > The Linux device driver init sequence is, > > > > 1) Set PCI bus master enable (BME=1) > > > > 2) Enable Controller (CC.EN=1) > > > > 3) Wait for controller ready (CSTS.RDY=1) > > > > The FreeBSD NVMe driver sequence should be changed to set BME=1 before > attempting to enable controller. > > > > > > Regards, > > Kinjal Patel > > _______________________________________________ > > freebsd-drivers@freebsd.org mailing > list > > https://lists.freebsd.org/mailman/listinfo/freebsd-drivers > > To unsubscribe, send any mail to "freebsd-drivers-unsubscribe@ > freebsd.org" > > > > _______________________________________________ > > freebsd-drivers@freebsd.org mailing list > > https://lists.freebsd.org/mailman/listinfo/freebsd-drivers > > To unsubscribe, send any mail to "freebsd-drivers-unsubscribe@ > freebsd.org" >