From owner-freebsd-arch@FreeBSD.ORG Fri Oct 10 08:23:53 2014 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 1FBF6D4A for ; Fri, 10 Oct 2014 08:23:53 +0000 (UTC) Received: from mail-wi0-x22b.google.com (mail-wi0-x22b.google.com [IPv6:2a00:1450:400c:c05::22b]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id AB20F762 for ; Fri, 10 Oct 2014 08:23:52 +0000 (UTC) Received: by mail-wi0-f171.google.com with SMTP id em10so1248860wid.10 for ; Fri, 10 Oct 2014 01:23:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=YcXgowkSFkuwvyBzRon5ltCNkeCoAj0d3ZGvKLm52y0=; b=emhZpuKw9rRh3zRb4My4qDn+ZEHQFfobCzTBKHIFNiUEuZ7xMMHL3l/G+htOW9CP4U vXclC4HqFJzviHMN7bCDr5fBWRJA6xF11JeBTJsVADwpGeYDNEH01AdEIai0jQ+Xr2y7 zwMyaljEq8o0z4nvhshELLbd1G6CcHZCgd7LsFg5/h0z4SP/37i0t2a+adwQBWCbQ8Pf PjWnlouH3+gdQq+AiZWMvCgyeU3pZ9bf+7RC3+ThIfi0ns23oWsU0XbYPRWUc5NuH27F 5q/iRU5t6NRjV/9CgA/z4bKF2dN+F/Cc9aGVoAimYm7GLs4RbAMlM3SwCvGaVEPK762K td6A== MIME-Version: 1.0 X-Received: by 10.180.187.83 with SMTP id fq19mr2969270wic.59.1412929430771; Fri, 10 Oct 2014 01:23:50 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.216.106.136 with HTTP; Fri, 10 Oct 2014 01:23:50 -0700 (PDT) In-Reply-To: <838B58B2-22D6-4AA4-97D5-62E87101F234@bsdimp.com> References: <2975E3D3-0335-4739-9242-5733CCEE726C@bsdimp.com> <838B58B2-22D6-4AA4-97D5-62E87101F234@bsdimp.com> Date: Fri, 10 Oct 2014 01:23:50 -0700 X-Google-Sender-Auth: uEvNJY8-2PFTEnFwhtqRQh4xebE Message-ID: Subject: Re: [rfc] enumerating device / bus domain information From: Adrian Chadd To: Warner Losh Content-Type: text/plain; charset=UTF-8 Cc: "freebsd-arch@freebsd.org" X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 10 Oct 2014 08:23:53 -0000 ... because for some topologies, as you've said, the devices are equal cost to all CPUs/memories. In the current parlance they don't have a VM domain and they're not assigned a proximity domain at all. We currently don't have a concept of "all domains" when compiling a kernel with a VM domain count of > 1. Now, should there be one? Yes. Should it be -1? Who knows. Let's have that discussion. Maybe we can just label -1 as the "default whole machine VM domain" and the behaviour won't change. And yes, I know about device costs. I was thinking about implementing this using the device/CPU/memory _PXM and for platforms with SLIT tables mapping out the actual costs to various other proximity domains. I'll eventually write a SLIT parser anyway as we may want the memory allocation to be "local" vs varying levels of "less local" rather than "local" vs "non-local". Now, whether the bus code can completely enumerate all of the requirements for allocating device local memory is another discussion. But drivers right now don't consistently at all say "please route stuff locally"; nor do they consistently say "please give me the cpuset of things that are local so I can allocate how many default queues and where to constrain them to run" - they just arbitrarily pick how much of what to run where and how. One simple version of the device/bus locality for this stuff is now in -HEAD. There's a little missing glue to do. Let's have the conversation of "how should drivers do this stuff and have the defaults behave consistently but be easily changed" so we can start experimenting with different ways to do all of this and move towards something that'll appear in -11. -a -a