Date: Mon, 6 Apr 2009 06:41:00 -0700 (PDT) From: Barney Cordoba <barney_cordoba@yahoo.com> To: Robert Watson <rwatson@FreeBSD.org> Cc: freebsd-net@freebsd.org, Ivan Voras <ivoras@freebsd.org> Subject: Re: Advice on a multithreaded netisr patch? Message-ID: <812958.41771.qm@web63906.mail.re1.yahoo.com> In-Reply-To: <alpine.BSF.2.00.0904061300160.34905@fledge.watson.org>
next in thread | previous in thread | raw e-mail | index | archive | help
--- On Mon, 4/6/09, Robert Watson <rwatson@FreeBSD.org> wrote:
> From: Robert Watson <rwatson@FreeBSD.org>
> Subject: Re: Advice on a multithreaded netisr patch?
> To: "Barney Cordoba" <barney_cordoba@yahoo.com>
> Cc: freebsd-net@freebsd.org, "Ivan Voras" <ivoras@freebsd.org>
> Date: Monday, April 6, 2009, 8:09 AM
> On Mon, 6 Apr 2009, Barney Cordoba wrote:
>
> > Is there a way to give a kernel thread exclusive use
> of a core? I know you can pin a kernel thread with
> sched_bind(), but is there a way to keep other threads from
> using the core? On an 8 core system it almost seems that the
> randomness of more cores is a negative in some situations.
> >
> > Also, I've noticed that calling sched_bind()
> during bootup is a bad thing in that it locks the system.
> I'm not certain but I suspect its the thread_lock that
> is the culprit. Is there a clean way to determine that its
> safe to lock curthread and do a cpu bind?
>
> There isn't an interface to cleanly express "Use
> CPUs 4-7 for only network processing". You can
> configure the system this way using the cpuset command
> (including directing the low-level interrupts to specific
> CPUs in 8.x), but if we think this is going to be a
> frequently desired policy, a bit more abstraction will be
> required.
>
> I'm not familiar with the problem you're seeing
> with sched_bind() -- I'm using it from within some of my
> code without a problem, and that's fairly early in the
> boot. A number of deadlocks are possible if one isn't
> very careful early in the boot though, so I might look
> specifically for some of those: if you migrate a thread to a
> CPU that isn't yet started, it won't be able to run
> until the CPU has started. This means it's important
> not to migrate threads that might lead to priority
> version-like deadlocks:
>
> - Be careful not to migrate threads that hold locks the
> system requires to get
> to the point where multiple CPUs run.
> - Be careful not to migrate threads that will signal a
> resource being
> available, such as a device driver, required to get to
> the point where
> multiple CPUs run.
> - Be careful not to migrate the main boot thread.
>
> Could you be running into one of those cases? Usually
> they're fairly easy to diagnose using DDB, if you can
> get into it, because you can see what the main boot thread
> is waiting for, and reason about what's holding it. Are
> you able to get into DDB when this occurs? (Perhaps using
> an NMI?)
Yes, the cpus are launched quite late, so that must be it. I guess
the mp_ncpus is set before they are launched. Is there a way to determine
that a specific core has been lauched?
Regarding using cpuset, John B indicated that you couldn't allocate
"sets" for kernel threads; and that sched_bind() was the only function
available. So that brings 2 questions:
1) How do you get the thread ID for a process from user space to use with
cpuset? I don't see that ps displays it.
2) Can cpu sets be manipulated / setup from within the kernel?
Barney
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?812958.41771.qm>