From owner-svn-src-head@FreeBSD.ORG Thu Apr 23 11:55:53 2009 Return-Path: Delivered-To: svn-src-head@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D4CF9106564A; Thu, 23 Apr 2009 11:55:53 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from cyrus.watson.org (cyrus.watson.org [65.122.17.42]) by mx1.freebsd.org (Postfix) with ESMTP id 977D28FC08; Thu, 23 Apr 2009 11:55:53 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from fledge.watson.org (fledge.watson.org [65.122.17.41]) by cyrus.watson.org (Postfix) with ESMTPS id 3B32C46B2D; Thu, 23 Apr 2009 07:55:53 -0400 (EDT) Date: Thu, 23 Apr 2009 12:55:53 +0100 (BST) From: Robert Watson X-X-Sender: robert@fledge.watson.org To: John Baldwin In-Reply-To: <200904222140.n3MLebn3068260@svn.freebsd.org> Message-ID: References: <200904222140.n3MLebn3068260@svn.freebsd.org> User-Agent: Alpine 2.00 (BSF 1167 2008-08-23) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: svn-src-head@freebsd.org, svn-src-all@freebsd.org, src-committers@freebsd.org Subject: Re: svn commit: r191405 - in head/sys: amd64/amd64 i386/i386 X-BeenThere: svn-src-head@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: SVN commit messages for the src tree for head/-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 23 Apr 2009 11:55:55 -0000 On Wed, 22 Apr 2009, John Baldwin wrote: > Adjust the way we number CPUs on x86 so that we attempt to "group" all > logical CPUs in a package. We do this by numbering the non-boot CPUs > by starting with the first CPU whose APIC ID is after the boot CPU and > wrapping back around to APIC ID 0 if needed rather than always starting > at APIC ID 0. While here, adjust the cpu_mp_announce() routine to list > CPUs based on the mapping established by assign_cpu_ids() rather than > making assumptions about the algorithm assign_cpu_ids() uses. Something we'd like to be able to do in the fairly near future is assign work from one thread/core to another thread/core that is cache-wise "close" to the current one. For example, if we take a cacheline miss on an mbuf head and the first line of a packet's data on one CPU, and want to hand it off to another CPU for further processing, trying to send it somewhere where it's already quite close, cache-wise, might well be beneficial. Likewise, sending it to another thread or core that is cache-wise close on the lock protecting the queue used to pass it between CPUs, would also be nice. Do you have any ideas about ways to usefully represent and manage concepts like "pick a close CPU" or "the set of CPUs that are close"? For example, if I have available a flow identifier, hashing to one of a set of available CPUs is easy, but what would you suggest as an efficient representation to hash from a set of close available CPUs rather than the entire pool? Robert N M Watson Computer Laboratory University of Cambridge > > MFC after: 1 month > > Modified: > head/sys/amd64/amd64/mp_machdep.c > head/sys/i386/i386/mp_machdep.c > > Modified: head/sys/amd64/amd64/mp_machdep.c > ============================================================================== > --- head/sys/amd64/amd64/mp_machdep.c Wed Apr 22 20:58:23 2009 (r191404) > +++ head/sys/amd64/amd64/mp_machdep.c Wed Apr 22 21:40:37 2009 (r191405) > @@ -354,8 +354,6 @@ cpu_mp_start(void) > } else > KASSERT(boot_cpu_id == PCPU_GET(apic_id), > ("BSP's APIC ID doesn't match boot_cpu_id")); > - cpu_apic_ids[0] = boot_cpu_id; > - apic_cpuids[boot_cpu_id] = 0; > > /* Setup the initial logical CPUs info. */ > logical_cpus = logical_cpus_mask = 0; > @@ -418,28 +416,30 @@ cpu_mp_start(void) > void > cpu_mp_announce(void) > { > - int i, x; > const char *hyperthread; > + int i; > > - /* List CPUs */ > + /* List active CPUs first. */ > printf(" cpu0 (BSP): APIC ID: %2d\n", boot_cpu_id); > - for (i = 1, x = 0; x <= MAX_APIC_ID; x++) { > - if (!cpu_info[x].cpu_present || cpu_info[x].cpu_bsp) > + for (i = 1; i < mp_ncpus; i++) { > + if (cpu_info[cpu_apic_ids[i]].cpu_hyperthread) > + hyperthread = "/HT"; > + else > + hyperthread = ""; > + printf(" cpu%d (AP%s): APIC ID: %2d\n", i, hyperthread, > + cpu_apic_ids[i]); > + } > + > + /* List disabled CPUs last. */ > + for (i = 0; i <= MAX_APIC_ID; i++) { > + if (!cpu_info[i].cpu_present || !cpu_info[i].cpu_disabled) > continue; > - if (cpu_info[x].cpu_hyperthread) { > + if (cpu_info[i].cpu_hyperthread) > hyperthread = "/HT"; > - } else { > + else > hyperthread = ""; > - } > - if (cpu_info[x].cpu_disabled) > - printf(" cpu (AP%s): APIC ID: %2d (disabled)\n", > - hyperthread, x); > - else { > - KASSERT(i < mp_ncpus, > - ("mp_ncpus and actual cpus are out of whack")); > - printf(" cpu%d (AP%s): APIC ID: %2d\n", i++, > - hyperthread, x); > - } > + printf(" cpu (AP%s): APIC ID: %2d (disabled)\n", hyperthread, > + i); > } > } > > @@ -693,11 +693,19 @@ assign_cpu_ids(void) > > /* > * Assign CPU IDs to local APIC IDs and disable any CPUs > - * beyond MAXCPU. CPU 0 has already been assigned to the BSP, > - * so we only have to assign IDs for APs. > + * beyond MAXCPU. CPU 0 is always assigned to the BSP. > + * > + * To minimize confusion for userland, we attempt to number > + * CPUs such that all threads and cores in a package are > + * grouped together. For now we assume that the BSP is always > + * the first thread in a package and just start adding APs > + * starting with the BSP's APIC ID. > */ > mp_ncpus = 1; > - for (i = 0; i <= MAX_APIC_ID; i++) { > + cpu_apic_ids[0] = boot_cpu_id; > + apic_cpuids[boot_cpu_id] = 0; > + for (i = boot_cpu_id + 1; i != boot_cpu_id; > + i == MAX_APIC_ID ? i = 0 : i++) { > if (!cpu_info[i].cpu_present || cpu_info[i].cpu_bsp || > cpu_info[i].cpu_disabled) > continue; > > Modified: head/sys/i386/i386/mp_machdep.c > ============================================================================== > --- head/sys/i386/i386/mp_machdep.c Wed Apr 22 20:58:23 2009 (r191404) > +++ head/sys/i386/i386/mp_machdep.c Wed Apr 22 21:40:37 2009 (r191405) > @@ -398,8 +398,6 @@ cpu_mp_start(void) > } else > KASSERT(boot_cpu_id == PCPU_GET(apic_id), > ("BSP's APIC ID doesn't match boot_cpu_id")); > - cpu_apic_ids[0] = boot_cpu_id; > - apic_cpuids[boot_cpu_id] = 0; > > /* Setup the initial logical CPUs info. */ > logical_cpus = logical_cpus_mask = 0; > @@ -462,28 +460,30 @@ cpu_mp_start(void) > void > cpu_mp_announce(void) > { > - int i, x; > const char *hyperthread; > + int i; > > - /* List CPUs */ > + /* List active CPUs first. */ > printf(" cpu0 (BSP): APIC ID: %2d\n", boot_cpu_id); > - for (i = 1, x = 0; x <= MAX_APIC_ID; x++) { > - if (!cpu_info[x].cpu_present || cpu_info[x].cpu_bsp) > + for (i = 1; i < mp_ncpus; i++) { > + if (cpu_info[cpu_apic_ids[i]].cpu_hyperthread) > + hyperthread = "/HT"; > + else > + hyperthread = ""; > + printf(" cpu%d (AP%s): APIC ID: %2d\n", i, hyperthread, > + cpu_apic_ids[i]); > + } > + > + /* List disabled CPUs last. */ > + for (i = 0; i <= MAX_APIC_ID; i++) { > + if (!cpu_info[i].cpu_present || !cpu_info[i].cpu_disabled) > continue; > - if (cpu_info[x].cpu_hyperthread) { > + if (cpu_info[i].cpu_hyperthread) > hyperthread = "/HT"; > - } else { > + else > hyperthread = ""; > - } > - if (cpu_info[x].cpu_disabled) > - printf(" cpu (AP%s): APIC ID: %2d (disabled)\n", > - hyperthread, x); > - else { > - KASSERT(i < mp_ncpus, > - ("mp_ncpus and actual cpus are out of whack")); > - printf(" cpu%d (AP%s): APIC ID: %2d\n", i++, > - hyperthread, x); > - } > + printf(" cpu (AP%s): APIC ID: %2d (disabled)\n", hyperthread, > + i); > } > } > > @@ -722,11 +722,19 @@ assign_cpu_ids(void) > > /* > * Assign CPU IDs to local APIC IDs and disable any CPUs > - * beyond MAXCPU. CPU 0 has already been assigned to the BSP, > - * so we only have to assign IDs for APs. > + * beyond MAXCPU. CPU 0 is always assigned to the BSP. > + * > + * To minimize confusion for userland, we attempt to number > + * CPUs such that all threads and cores in a package are > + * grouped together. For now we assume that the BSP is always > + * the first thread in a package and just start adding APs > + * starting with the BSP's APIC ID. > */ > mp_ncpus = 1; > - for (i = 0; i <= MAX_APIC_ID; i++) { > + cpu_apic_ids[0] = boot_cpu_id; > + apic_cpuids[boot_cpu_id] = 0; > + for (i = boot_cpu_id + 1; i != boot_cpu_id; > + i == MAX_APIC_ID ? i = 0 : i++) { > if (!cpu_info[i].cpu_present || cpu_info[i].cpu_bsp || > cpu_info[i].cpu_disabled) > continue; >