From owner-freebsd-arch@FreeBSD.ORG Sat Mar 21 22:26:26 2015 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 8344553E; Sat, 21 Mar 2015 22:26:26 +0000 (UTC) Received: from mail-yk0-x233.google.com (mail-yk0-x233.google.com [IPv6:2607:f8b0:4002:c07::233]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 3EE5E273; Sat, 21 Mar 2015 22:26:26 +0000 (UTC) Received: by ykfs63 with SMTP id s63so57353080ykf.2; Sat, 21 Mar 2015 15:26:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=ScRJX2EjEYkxcdXp3vitmPLMsVkDIEik4iyHINp6fc4=; b=P8NjzzuVd3Vp0A4z9ROVfuNllJT3VEmx6x57BY/RNpVrYdWO7uS5S+KpxeRoouWknl +5RIvpa32oylBfeRYMZMWR1xnzYRQhrutD/5KmSVkz70xVAulI8S2qd0XLZL2ZV4Lq0A vYYV21zLmSDOvPq6bYL8S7fZMOcRMIJif8QCXF3/aDFKQh5pf3Vfg0/2robr1efam5ht YwJPEqNvyvuTuU+OLm5nvRuD24h6od3gwDDf88G+m/IQQY8yCBJvC1LdZH8in0dtoBOw 0oW52GdvzHQxtTUUDJeEvC0XJfKcIgw/uQMNe5aBINEC5MnWnhQx2akiUoU7UWB4q1Ht kKMQ== MIME-Version: 1.0 X-Received: by 10.170.98.133 with SMTP id p127mr53401872yka.115.1426976785325; Sat, 21 Mar 2015 15:26:25 -0700 (PDT) Sender: kmacybsd@gmail.com Received: by 10.170.205.193 with HTTP; Sat, 21 Mar 2015 15:26:25 -0700 (PDT) In-Reply-To: <550D92C6.1080002@FreeBSD.org> References: <550D92C6.1080002@FreeBSD.org> Date: Sat, 21 Mar 2015 15:26:25 -0700 X-Google-Sender-Auth: kKyJfirJKfKZyd5GDl-OMpz3DoA Message-ID: Subject: Re: LOCAL_CPUS vs INTR_CPUS From: "K. Macy" To: John Baldwin Content-Type: text/plain; charset=UTF-8 Cc: "freebsd-arch@freebsd.org" X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 21 Mar 2015 22:26:26 -0000 >> In reference to: >> https://github.com/freebsd/freebsd/compare/master...bsdjhb:numa_bus_get_cpus >> >> >> The LOCAL_CPUS option will return the cpuset corresponding to the NUMA >> (_pxm) domain id of a given device. INTR_CPUS can narrow that further. >> However, in this particular branch INTR_CPUS will only ever return the >> value given by LOCAL_CPUS. When and where would this come in to >> effect? Do the new Haswells have cores in the same memory domain but >> with non-uniform access to the PCIe bus? > > On x86 this means that INTR_CPUS only includes one thread per core in a > the NUMA domain a device belongs to whereas LOCAL_CPUS will include all > threads in the domain. So if you have a dual-socket 8-core SB/IB system > for example, a device on the IO hub for the first processor would use > CPUs 0-15 for LOCAL_CPUS and 0,2,4,6,8,10,12,14 for INTR_CPUS if HTT is > enabled. If it is disabled, both requests would return CPUs 0-7. > I was confused by the fact that it was propagating it up the chain only to have the nexus return intr_cpus and was in turn overlooking intr_cpus being initialized in set_interrupt_apic_ids . The appearance being that it was leaving the door open for some intermediate node to narrow the cpuset - and nexus was just the default. As it stands now just directly OR'ing LOCAL_CPUS with intr_cpus now that you've made it a global would make more sense. But perhaps the added flexibility will prove useful in the future. Thanks.