From owner-svn-src-all@FreeBSD.ORG Wed Apr 8 22:51:38 2015 Return-Path: Delivered-To: svn-src-all@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id CAB13912; Wed, 8 Apr 2015 22:51:38 +0000 (UTC) Received: from mail-qk0-x232.google.com (mail-qk0-x232.google.com [IPv6:2607:f8b0:400d:c09::232]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 80A5276D; Wed, 8 Apr 2015 22:51:38 +0000 (UTC) Received: by qkx62 with SMTP id 62so99835330qkx.0; Wed, 08 Apr 2015 15:51:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=mjdXcRVH6bwfV7VHoGokH4n3osMDyOS9VcHM0v2R5cI=; b=E7CEj7NGUn4y99OTDpY+IWhyvruESgkBCMIEfhhRqvJa282b3cbsBg58L6H3E3sygb sLmNLUVEXKK6BlXRrq5RwajHc68g2h69pJZZWIa3S4DFSFcAL+/Rpi4g5gndxs1wQXYF OoiBNfAlaT3nLBs27oPtKm0su+0uErJWkZP04dRDiTTsNsxmlpYfyCGvE0jYXkO8ZPGl 0DqmmZ+pR9xcAJD8X6ye6YUi7NPzlrpru83KGnLIFiHbEhhikksEMXbvjZTwFCXg1Hc7 304FwPhmAT2Hsokkj3Cj7A7r2sKcHu3pUeR5em0jVZ1lb0iiVlbca7RzihQwBkrhK1iB e1gg== MIME-Version: 1.0 X-Received: by 10.55.50.201 with SMTP id y192mr36535125qky.10.1428533497595; Wed, 08 Apr 2015 15:51:37 -0700 (PDT) Received: by 10.140.38.73 with HTTP; Wed, 8 Apr 2015 15:51:37 -0700 (PDT) In-Reply-To: <20150408182150.03c2b832@kan> References: <201504082146.t38LkJ9s058300@svn.freebsd.org> <20150408182150.03c2b832@kan> Date: Wed, 8 Apr 2015 15:51:37 -0700 Message-ID: Subject: Re: svn commit: r281280 - head/sys/dev/nvme From: Jim Harris To: Alexander Kabaev Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 Cc: svn-src-head@freebsd.org, svn-src-all@freebsd.org, src-committers@freebsd.org X-BeenThere: svn-src-all@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: "SVN commit messages for the entire src tree \(except for " user" and " projects" \)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Apr 2015 22:51:38 -0000 On Wed, Apr 8, 2015 at 3:21 PM, Alexander Kabaev wrote: > On Wed, 8 Apr 2015 21:46:19 +0000 (UTC) > Jim Harris wrote: > > > Author: jimharris > > Date: Wed Apr 8 21:46:18 2015 > > New Revision: 281280 > > URL: https://svnweb.freebsd.org/changeset/base/281280 > > > > Log: > > nvme: fall back to a smaller MSI-X vector allocation if necessary > > > > Previously, if per-CPU MSI-X vectors could not be allocated, > > nvme(4) would fall back to INTx with a single I/O queue pair. > > This change will still fall back to a single I/O queue pair, but > > allocate MSI-X vectors instead of reverting to INTx. > > > > MFC after: 1 week > > Sponsored by: Intel > > > > Modified: > > head/sys/dev/nvme/nvme_ctrlr.c > > > > Modified: head/sys/dev/nvme/nvme_ctrlr.c > > > ============================================================================== > > --- head/sys/dev/nvme/nvme_ctrlr.c Wed Apr 8 21:10:13 > > 2015 (r281279) +++ head/sys/dev/nvme/nvme_ctrlr.c Wed > > Apr 8 21:46:18 2015 (r281280) @@ -1144,9 +1144,17 @@ > > nvme_ctrlr_construct(struct nvme_control /* One vector per IO queue, > > plus one vector for admin queue. */ num_vectors = > > ctrlr->num_io_queues + 1; > > - if (pci_msix_count(dev) < num_vectors) { > > + /* > > + * If we cannot even allocate 2 vectors (one for admin, one > > for > > + * I/O), then revert to INTx. > > + */ > > + if (pci_msix_count(dev) < 2) { > > ctrlr->msix_enabled = 0; > > goto intx; > > + } else if (pci_msix_count(dev) < num_vectors) { > > + ctrlr->per_cpu_io_queues = FALSE; > > + ctrlr->num_io_queues = 1; > > + num_vectors = 2; /* one for admin, one for I/O */ > > } > > > > if (pci_alloc_msix(dev, &num_vectors) != 0) { > > Huh, Linux just falls back to as many vectors as it can and just > allocates them between per-cpu queues in a round-robin manner. I think > is is a better approach than what we have here, would you consider it? > This has been on my todo list for a while but have not had time to tackle it. I'm hoping to spend some time on it in the next couple of weeks though. I would prefer it to be smarter than just round-robin. For example, if multiple cores are sharing a queue pair, we probably want those cores to be on the same CPU socket. Or if hyper-threading is enabled, we likely want to assign those logical cores to the same queue pair. But short-term, yes - simple round-robin would be better than the current fallback scheme. -Jim > -- > Alexander Kabaev >