Date: Tue, 19 Mar 2019 22:10:21 +0100 From: =?UTF-8?Q?Mi=C5=82osz_Kaniewski?= <m.kaniewski@fudosecurity.com> To: freebsd-hackers@freebsd.org Subject: sched_pickcpu() and a pinned thread. Message-ID: <CABQUGZzxrnS%2B9xkrbufc9cx1jyG%2B1V553AurYGeSV-WtJX0A2w@mail.gmail.com>
next in thread | raw e-mail | index | archive | help
Hi,
I have an application which exchanges packets between two NIC ports.
Most of a CPU time is therefore consumed by the userspace process and
two IRQ threads. All of them are pinned to the same CPU using
cpu_setaffinity() function so I expect that the scheduler usage should
be minimal even on a high network load when a lot of interrupts are
generated.
I recently find out that my application spends a lot of time in
cpu_search_lowest() function. A backtrace shown me that
cpu_search_lowest() is called by sched_pickcpu(). I tried to analyze
this second function but I don't know what exactly should I expect
from it if my thread is pinned to a specific CPU. I assume that the
function should get to a point where ts->ts_cpu is returned. However
it doesn't do it and sched_pickcpu() always ends looking for the
lowest busy CPU using cpu_search_lowest(). In result it finds the CPU
that the thread is pinned to but it is very time consuming process.
Therefore I suspect that sched_pickcpu() should return before that
happens.
For interrupt threads I see that there is a special case at the
beginning of sched_pickcpu():
/*
* Prefer to run interrupt threads on the processors that generate
* the interrupt.
*/
pri =3D td->td_priority;
if (td->td_priority <=3D PRI_MAX_ITHD && THREAD_CAN_SCHED(td, self) &&
curthread->td_intr_nesting_level && ts->ts_cpu !=3D self) {
SCHED_STAT_INC(pickcpu_intrbind);
ts->ts_cpu =3D self;
if (TDQ_CPU(self)->tdq_lowpri > pri) {
SCHED_STAT_INC(pickcpu_affinity);
return (ts->ts_cpu);
}
}
However IRQ threads from the NIC doesn't fall into this case because
ts->ts_cpu =3D=3D self (these variables are equal to the CPU to which my
threads are pinned). Is this check required?
For non-interrupt threads there are some other possible options where
ts->ts_cpu is returned however I am not sure which should handle a
pinned thread.
I would be grateful if someone would give me some clue how should I
understand the sched_pickcpu() function and when it should return in
case of a pinned thread.
Thanks
Mi=C5=82osz Kaniewski
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CABQUGZzxrnS%2B9xkrbufc9cx1jyG%2B1V553AurYGeSV-WtJX0A2w>