From owner-freebsd-arch@FreeBSD.ORG Wed Oct 8 23:12:59 2014 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id F3744719 for ; Wed, 8 Oct 2014 23:12:58 +0000 (UTC) Received: from mail-ig0-x22c.google.com (mail-ig0-x22c.google.com [IPv6:2607:f8b0:4001:c05::22c]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id C0CA6CBF for ; Wed, 8 Oct 2014 23:12:58 +0000 (UTC) Received: by mail-ig0-f172.google.com with SMTP id r2so9342521igi.11 for ; Wed, 08 Oct 2014 16:12:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type:content-transfer-encoding; bh=XOBlh9Ij4rzIWjOaAeve8hXbA3F81PxT8cHk5dzV1JE=; b=A84yHkpzF+/vy++CGa5SgVkxgL246BqOL1eRBl1DG9zAA6eRwAnxvsI02q8ybg8uP/ ABWmnJBNoMyx7rShus/kTmES8lgHGzWYyGutFwwflMlfT0g0LXdmZJUWtAYgP3nKZaJ7 c0ssLDSO+pXavpKyB8RqQyIN6HNF0KWHCsfaDiK+iCxYWBnK6fmEfUXeyP+0aPk9fWdk +OmqXG51+uQon+DdzmI2ucxJRu7St98Em6wW8q+T5oBS/b/8UPmJpxPkgAhjpbMI/VDK lClRWELme7UUl1a9grSR0s+m7w9ZhKQPbn0O2gk3c2pSn9gVpvBENgh4nYeIwCTnNUZG QLSA== MIME-Version: 1.0 X-Received: by 10.50.43.137 with SMTP id w9mr48466603igl.36.1412809978057; Wed, 08 Oct 2014 16:12:58 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.50.78.4 with HTTP; Wed, 8 Oct 2014 16:12:57 -0700 (PDT) In-Reply-To: <2975E3D3-0335-4739-9242-5733CCEE726C@bsdimp.com> References: <2975E3D3-0335-4739-9242-5733CCEE726C@bsdimp.com> Date: Wed, 8 Oct 2014 16:12:57 -0700 X-Google-Sender-Auth: oZzYm2rnxcxTvJotnBECjb46t_k Message-ID: Subject: Re: [rfc] enumerating device / bus domain information From: Adrian Chadd To: Warner Losh Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Cc: "freebsd-arch@freebsd.org" X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Oct 2014 23:12:59 -0000 On 8 October 2014 12:07, Warner Losh wrote: > > On Oct 7, 2014, at 7:37 PM, Adrian Chadd wrote: > >> Hi, >> >> Right now we're not enumerating any NUMA domain information about device= s. >> >> The more recent intel NUMA stuff has some extra affinity information >> for devices that (eventually) will allow us to bind kernel/user >> threads and/or memory allocation to devices to keep access local. >> There's a penalty for DMAing in/out of remote memory, so we'll want to >> figure out what counts as "Local" for memory allocation and perhaps >> constrain the CPU set that worker threads for a device run on. >> >> This patch adds a few things: >> >> * it adds a bus_if.m method for fetching the VM domain ID of a given >> device; or ENOENT if it's not in a VM domain; > > Maybe a default VM domain. All devices are in VM domains :) By default > today, we have only one VM domain, and that=E2=80=99s the model that most= of the > code expects=E2=80=A6 Right, and that doesn't change until you compile in with num domains > 1. Then, CPUs and memory have VM domains, but devices may or may not have a VM domain. There's no "default" VM domain defined if num domains > 1. The devices themselves don't know about VM domains right now, so there's nothing constraining things like IRQ routing, CPU set, memory allocation, etc. The isilon team is working on extending the cpuset and allocators to "know" about numa and I'm sure this stuff will fall out of whatever they're working on. So when I go to add sysctl and other tree knowledge for device -> vm domain mapping I'm going to make them return -1 for "no domain." (Things will get pretty hilarious later on if we have devices that are "local" to two or more VM domains ..) -a