From owner-freebsd-arch@FreeBSD.ORG Thu Feb 19 20:22:13 2015 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 8504B540; Thu, 19 Feb 2015 20:22:13 +0000 (UTC) Received: from bigwig.baldwin.cx (bigwig.baldwin.cx [IPv6:2001:470:1f11:75::1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 5CB45D08; Thu, 19 Feb 2015 20:22:13 +0000 (UTC) Received: from ralph.baldwin.cx (pool-173-54-116-245.nwrknj.fios.verizon.net [173.54.116.245]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id D05B1B94E; Thu, 19 Feb 2015 15:22:11 -0500 (EST) From: John Baldwin To: Warner Losh Subject: Re: RFC: bus_get_cpus(9) Date: Thu, 19 Feb 2015 12:33:38 -0500 Message-ID: <6632720.8QN4idWR9d@ralph.baldwin.cx> User-Agent: KMail/4.14.2 (FreeBSD/10.1-STABLE; KDE/4.14.2; amd64; ; ) In-Reply-To: References: <1848011.eGOHhpCEMm@ralph.baldwin.cx> <2650364.MV3AvSBuVe@ralph.baldwin.cx> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7 (bigwig.baldwin.cx); Thu, 19 Feb 2015 15:22:11 -0500 (EST) Cc: "freebsd-arch@freebsd.org" , Adrian Chadd X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 19 Feb 2015 20:22:13 -0000 On Thursday, February 19, 2015 10:15:56 AM Warner Losh wrote: > > On Feb 19, 2015, at 10:03 AM, John Baldwin wrote:= > >=20 > > On Thursday, February 19, 2015 08:28:34 AM Adrian Chadd wrote: > >> On 19 February 2015 at 07:37, John Baldwin wrote= : > >>> There's nothing preventing the RSS code from calling bus_get_cpus= () > >>> internally to populate the info it returns in its APIs. > >>>=20 > >>> That is, I imagine something like: > >>>=20 > >>> #ifdef RSS > >>>=20 > >>> queue_info =3D fetch_rss_info(dev); > >>> for (queue in queue_info) { > >>> =20 > >>> create queue for CPU queue->cpu > >>> =20 > >>> } > >>>=20 > >>> #else > >>>=20 > >>> /* Use bus_get_cpus directly and do 1:1 */ > >>>=20 > >>> #endif > >>>=20 > >>> That is, I think RSS should provide a layer on top of new-bus, no= t be a > >>> bus_foo API. At some point all drivers might only have the #ifde= f RSS > >>> case > >>> and not use bus_get_cpus() directly at all, but it doesn't seem l= ike the > >>> RSS API is quite there yet. > >>=20 > >> I wasn't suggesting we have RSS as a newbus method, just that driv= ers > >> don't necessarily need to call the bus method and iterate themselv= es. > >>=20 > >> I was suggesting that we do what i've done for rss, but as a gener= ic > >> "how should devices create queues and map them to cpusets / interr= upt > >> locality" and that calls the bus method(s) to discover topology an= d > >> query local-interrupt and local-memory sets to do things > >> appropriately. > >>=20 > >> Then RSS is just a flavour of that API call - network drivers coul= d > >> either be RSS aware and call it to get the mapping, or call some > >> higher level bus API call to do the "generic" hints or whatever ba= sed > >> mapping. > >=20 > > Can you provide a sample API (function prototype, etc.)? Aside fro= m RSS > > (which will have its own API for other reasons), I don't know of an= other > > use case that is well understood enough to let us build an abstract= ion on > > yet (we all know the perils of abstracting from one use case), so I= 'm > > hesitant to go much further than "these are the best place to do > > interrupts=E2=80=9D. >=20 > Newer LSI cards could benefit from that, but the rest of the storage = stack > may need tweaks to allow for true multi-queue implementations. Interr= upts > would be part of that. Right, storage in particular I think we don't really know what model we= want=20 yet, so aren't really ready for an API that tries to be abstract across= both=20 network and storage. --=20 John Baldwin