Date: Mon, 17 Mar 2008 09:47:25 -0400 From: John Baldwin <jhb@freebsd.org> To: "Joseph Koshy" <joseph.koshy@gmail.com> Cc: freebsd-arch@freebsd.org Subject: Re: [PATCH] hwpmc(4) changes to use 'mp_maxid' instead of 'mp_ncpus'. Message-ID: <200803170947.25205.jhb@freebsd.org> In-Reply-To: <84dead720803142243r6c8cc68dm325e7fb925189fd@mail.gmail.com> References: <20080313180805.GA83406@dragon.NUXI.org> <200803141431.53846.jhb@freebsd.org> <84dead720803142243r6c8cc68dm325e7fb925189fd@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Saturday 15 March 2008 01:43:00 am Joseph Koshy wrote:
> > FreeBSD has been trying to not be quite as i386-centric as it used to
> > be. If you look at other code in the kernel that handles per-cpu data
> > such as UMA you will see that it uses mp_maxid and CPU_ABSENT(). There
> > are other places in the kernel that are broken though (such as ndis(4)).
>
> HWPMC is very x86 centric, for obvious reasons.
Considering other CPU archictectures support various performance counters it
really shouldn't be designed to be x86-centric even if it is currently only
implemented for x86 CPUs.
> > > - Will sysctl hw.ncpus represent the count of present CPUs or will it
> > > represent the maximum CPU id?
> >
> > hw.ncpus is always mp_ncpus
> > kern.smp.cpus is also mp_ncpus
> > kern.smp.maxcpus is MAX_CPUS.
> >
> > Userland can just iterate from 0 to kern.smp.maxcpus while handling
> > absent CPUs. (For example, the kern.cp_time[] sysctl just writes out all
> > 0's for absent CPUs so that is how userland can determine an absent CPU
> > in that case.)
>
> I thought of that. For PMCTools use, using the proposed 'online_cpus'
> mask would be a better option. MAX_CPUS is a compile time value and could
> be large, whereas most machines will have far fewer CPUs than that limit.
> Why waste cycles needlessly?
Userland cycles are "cheaper". :) I think having both is fine and userland
can choose which to use (maxcpus is probably easier to impl but perhaps less
efficient).
> Now it appears to me that in the scheme of things described
> above one of mp_maxid and mp_ncpus is superfluous.
>
> Here is the reasoning:
>
> 0) We need a compile time limit for the kernel; this is kern.smp.maxcpus.
>
> 1) A given machine has a maximum number of CPUs that can fit in it.
> This is usually <<= MAXCPUS. Let us call this {MACHINE-MAX}.
> We need to scale kernel data structures based on {MACHINE-MAX}
> since using {MAXCPUS} is probably wasteful. We cannot just count the
> current number of CPUS, as we do today, because more could be
> hotplugged in later.
>
> 2) At any given instant a subset of CPUs 0..{MACHINE_MAX} will be
> online. This would be tracked by the kern.smp.online_cpus/all_cpus
> bitmask.
>
> Therefore we can use either a count (mp_ncpus) or a maximum id
> (mp_maxid) to represent {MACHINE-MAX}, but either one would do.
>
> However, x86 MD code uses both, with newer code seeming to prefer
> mp_maxid. So I am puzzled. There are far more uses of mp_ncpus
> there though.
The mp_ncpus uses are mostly bugs (e.g. ndis). I think mp_ncpus's primary use
is for userland so people can do:
make -j $(sysctl hw.ncpus)
or the like. That is, if you need a simple count of CPUs in the system, that
is what mp_ncpus is for. If you need to address invididual CPUs by ID, then
mp_ncpus is not appropriate and you need to iterate from 0 to mp_maxid
suitable to some bitmask (e.g. all_cpus via CPU_ABSENT, or a
not-yet-implemented onlines_cpus wth CPU_ONLINE/CPU_OFFLINE wrappers).
> jk> Changing HWPMC and its userland before the base kernel itself
> jk> changes does not seem to be the right thing to do.
>
> jb> While the userland intIerface is somewhat lacking, all of the
> in-kernel jb> infrastructure has been in place for at least the past 4
> years, and there is
> jb> no excuse for any in-kernel code not properly handling sparse CPU IDs.
>
> I try keep userland, kernel and documentation associated with PmcTools
> in sync.
>
> Looking around, there appear to be lots of nits that need correction.
> For one, the kern.smp sysctl hierarchy is undocumented.
Not entirely:
> sysctl -d kern.smp
kern.smp: Kernel SMP
kern.smp.maxcpus: Max number of CPUs that the system was compiled for.
kern.smp.active: Number of Auxillary Processors (APs) that were successfully
started
kern.smp.disabled: SMP has been disabled from the loader
kern.smp.cpus: Number of CPUs online
(On a UP 6.3 box)
--
John Baldwin
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?200803170947.25205.jhb>