Date: Fri, 19 Jun 2015 09:12:05 -0400 From: Chris Fitch <cmfitch1@gmail.com> To: freebsd-hackers@freebsd.org Subject: Realtime process CPU starvation Message-ID: <CABDHHOVninq73Bm5e9uNkMz=cA9SWAniDf93WiKqG_Hk97jyjg@mail.gmail.com>
next in thread | raw e-mail | index | archive | help
Hello all,
I have a problem that appears to be scheduler/filesystem related and could
use some expert advice. I have a process running at realtime priority 0
under FreeBSD 10.0. The main thread needs to run every 10 ms, and when it
has completed its work, it yields the processor via a nanosleep() call. The
sleep time is computed to be the remainder of the 10 ms period that is not
needed. This mostly works, but there are occurrences where the thread
doesn't run again for an extended period of time. The thread is monitoring
how far behind the realtime target it is and reporting whenever it falls
more than 250 ms behind. I have seen it report anywhere from 270 ms to
almost 2 seconds behind. This was confirmed using an off-cpu dtrace script
with the following results when the thread reported that it fell 500 ms
behind:
Off cpu times (milliseconds):
process
value ------------- Distribution ------------- count
-1 | 0
0 | 18
1 | 1
2 | 5
4 | 12
8 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 42229
16 | 7
32 | 7
64 | 0
128 | 3
256 | 1
512 | 0
How can a thread with the highest realtime priority not run for such a
lengthy period? I dove into the ULE scheduler, and although I was in a bit
over my head, it seems to me that the only way this is possible is for
kernel priority threads to be starving it of CPU. I tried decreasing
kern.sched.slice to 3 hoping that smaller scheduler timeslices would
decrease the wait time if several kernel threads were ahead of it in the
run queue, but it didn't help.
I was suspicious of the ZFS filesystem, as the delayed running of the
thread seemed to coincide with short periods of moderate disk activity. I
obtained some evidence to confirm my suspicions using KTR_SCHED and
schedgraph.py. The trace shows that 'solthread 0xfffffff' was running for
almost 500 ms when the problem occurred. During this 500 ms, two of the
four CPUs were idle. Based on the graph, it appears that an OpenSolaris
thread can starve a realtime process of CPU when idle CPUs are available.
Any thoughts?
Thanks,
Chris
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CABDHHOVninq73Bm5e9uNkMz=cA9SWAniDf93WiKqG_Hk97jyjg>