From owner-freebsd-drivers@freebsd.org Fri Aug 25 16:00:22 2017 Return-Path: Delivered-To: freebsd-drivers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id AC19ADDBAEA for ; Fri, 25 Aug 2017 16:00:22 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-it0-x22f.google.com (mail-it0-x22f.google.com [IPv6:2607:f8b0:4001:c0b::22f]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 6DE5C65BCD for ; Fri, 25 Aug 2017 16:00:22 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: by mail-it0-x22f.google.com with SMTP id p10so1175281ite.1 for ; Fri, 25 Aug 2017 09:00:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20150623.gappssmtp.com; s=20150623; h=mime-version:sender:in-reply-to:references:from:date:message-id :subject:to:cc; bh=1Dllcmpvgn+dCywNwXbXsNKkjOHYHjUZaKnot7F5Apg=; b=oBnKnZ9osUe1lxqu2FDGDjHcuy3LA8jRaGxLgYU1xYXF8r2gdnKdBhZODlUtPF9xgQ PONiQWCh8yjeRe/EKu/0lY5EfLGF+pNrRiQ3vRZYuiy3QYcw/+WbWJb9vbA2klq4TJzS bopdA5Ozeol/KiU4fBJqpuh/cfUASd8mWeZTxWewmVcQ9bKUkrQd83Sm6r/QpTCB238S i2TkiMHaylKmRd/41GMylNAQdgFffWhcffNS3yqPq6hVeohxx1h2cV9ykibXzqbH3CyS hyk/ar9L2WfjQ+D4vPLkQwhlV8KwzvqNUMwMBhgFVSnQOXDUPDGot9drMKIb+n0v9xQc griQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:sender:in-reply-to:references:from :date:message-id:subject:to:cc; bh=1Dllcmpvgn+dCywNwXbXsNKkjOHYHjUZaKnot7F5Apg=; b=CCtiuZvM7+6t9Ab8axAnv5tjPruF8q3HQBQw1TvuUmYQWpVNYQ5pqrp1YDFhi2x+lm U26XkFN7HOdE2I6Gqe18Ky3dPnGPp9Zy9ubIeKwkPWp0jz5Gs3j/Z16P3663+a4KJt8t paRE6OGhhCL6hNZWQRFFZD8FUK9K4Dl034HTDo/ZjJkkzSfEUgT6GHrwXejyYx1qxsWC JDA72Zi9g6NYy1SV4kL3pmZYxPFNaund6cvxfLduIZaiqGzePmU+Pf/9EhsIBp3heVrg n0R4boBrOOhQXhLprMAigrIfEt2PpJTJDdhG8qUnrxTTuBgKRIESJouribW7Xw+aE7Fx JPEg== X-Gm-Message-State: AHYfb5hjpUU4zAK7uo4uu/ajmDon3kUhbU4YXfDqY/hkzhFwk6T5bwcm kCUqcNon1zvDPVa2cFEBRuXD8TsB5KVE X-Received: by 10.36.237.8 with SMTP id r8mr2754116ith.125.1503676821389; Fri, 25 Aug 2017 09:00:21 -0700 (PDT) MIME-Version: 1.0 Sender: wlosh@bsdimp.com Received: by 10.79.10.71 with HTTP; Fri, 25 Aug 2017 09:00:20 -0700 (PDT) X-Originating-IP: [2603:300b:6:5100:c525:dd46:ae7:666a] In-Reply-To: References: From: Warner Losh Date: Fri, 25 Aug 2017 10:00:20 -0600 X-Google-Sender-Auth: mkVF7INCQ0g0xoa_kNneu_k5tWg Message-ID: Subject: Re: NVMe driver init sequence To: Kinjal Patel Cc: Jim Harris , "freebsd-drivers@freebsd.org" Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-drivers@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Writing device drivers for FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 25 Aug 2017 16:00:22 -0000 BTW, I just noticed you reported this against 10.3R. Is there some reason (perhaps a customer with an issue) that you reported it there? I'm trying to decide if I need to MFC this change to 11.x and 10.x (the latter has some time urgency due to upcoming 10.4R release). Warner On Fri, Aug 25, 2017 at 9:54 AM, Kinjal Patel wrote: > Great! > > Thanks Warner. > > > > > > Regards, > > Kinjal Patel > > > > *From:* wlosh@bsdimp.com [mailto:wlosh@bsdimp.com] *On Behalf Of *Warner > Losh > *Sent:* Thursday, August 24, 2017 8:17 PM > *To:* Jim Harris > *Cc:* Kinjal Patel; freebsd-drivers@freebsd.org > > *Subject:* Re: NVMe driver init sequence > > > > Greetings, > > > > I've just committed r322872 which implements this, with a slightly tweaked > comment. > > > > Warner > > > > On Fri, Aug 18, 2017 at 4:09 PM, Jim Harris wrote: > > On Fri, Aug 18, 2017 at 2:04 PM, Kinjal Patel > wrote: > > Created a bug #22166 (https://bugs.freebsd.org/bugzilla/show_bug.cgi?id= > 221616) > > Updated the description and patch to it. > > > > Regards, > > Kinjal Patel > > > > From: Kinjal Patel > > Sent: Friday, August 18, 2017 1:51 PM > > To: 'Warner Losh' > > Cc: freebsd-drivers@freebsd.org > > Subject: RE: NVMe driver init sequence > > > > Here is a the change I propose. This will make the init sequence right. > > Looks good to me. > > > > > Index: sys/dev/nvme/nvme.c > > =================================================================== > > --- sys/dev/nvme/nvme.c (revision 322672) > > +++ sys/dev/nvme/nvme.c (working copy) > > @@ -253,6 +253,9 @@ > > return (status); > > } > > > > + /* make device bus-master before attempting to enable controller */ > > + pci_enable_busmaster(dev); > > + > > /* > > * Reset controller twice to ensure we do a transition from > cc.en==1 > > * to cc.en==0. This is because we don't really know what > status > > @@ -270,8 +273,6 @@ > > return (status); > > } > > > > - pci_enable_busmaster(dev); > > - > > ctrlr->config_hook.ich_func = nvme_ctrlr_start_config_hook; > > ctrlr->config_hook.ich_arg = ctrlr; > > > > > > > > regards, > > Kinjal Patel > > > > From: wlosh@bsdimp.com [mailto:wlosh@bsdimp.com] > On Behalf Of Warner Losh > > Sent: Thursday, August 17, 2017 3:10 PM > > To: Kinjal Patel > > Cc: freebsd-drivers@freebsd.org > > Subject: Re: NVMe driver init sequence > > > > This sounds right to me.... Most BIOSes seem to enable BME=1. > > > > Warner > > > > On Thu, Aug 17, 2017 at 2:45 PM, Kinjal Patel < > Kinjal.Patel@taec.toshiba.com> > wrote: > > Hi, > > > > I have a query on NVMe driver init sequence in FreeBSD 10.3(stable). > > > > The driver init sequence is, > > > > 1) Enable controller (CC.EN=1) > > > > 2) Wait for controller ready (CSTS.RDY=1) > > > > 3) Set PCI bus master enable (BME=1) > > > > As per NVMe spec, when NVMe controller becomes ready it has to process > command. > > > > "Enable (EN): When set to '1', then the controller shall process > commands based on Submission Queue Tail doorbell writes" > > > > And per PCI Express spec when BME is not set, the PCI Express Function > is not allowed to issue any Memory or I/O requests. > > "Bus Master Enable - Controls the ability of a PCI Express Endpoint to > issue Memory95 and I/O Read/Write Requests, and > > the ability of a Root or Switch Port to forward Memory and I/O > Read/Write Requests in the Upstream direction" > > > > Enabling controller before setting BME=1 is violation of spec, as > controller has to accept commands but BME is prerequisite for that. > > > > The Linux device driver init sequence is, > > > > 1) Set PCI bus master enable (BME=1) > > > > 2) Enable Controller (CC.EN=1) > > > > 3) Wait for controller ready (CSTS.RDY=1) > > > > The FreeBSD NVMe driver sequence should be changed to set BME=1 before > attempting to enable controller. > > > > > > Regards, > > Kinjal Patel > > _______________________________________________ > > freebsd-drivers@freebsd.org mailing > list > > https://lists.freebsd.org/mailman/listinfo/freebsd-drivers > > To unsubscribe, send any mail to "freebsd-drivers-unsubscribe@ > freebsd.org" > > > > > _______________________________________________ > > freebsd-drivers@freebsd.org mailing list > > https://lists.freebsd.org/mailman/listinfo/freebsd-drivers > > To unsubscribe, send any mail to "freebsd-drivers-unsubscribe@ > freebsd.org" > > >