Date: Sat, 23 Jun 2018 21:03:02 +0000 From: Rick Macklem <rmacklem@uoguelph.ca> To: "freebsd-current@freebsd.org" <freebsd-current@freebsd.org> Cc: "kib@FreeBSD.org" <kib@FreeBSD.org>, Alexander Motin <mav@FreeBSD.org> Subject: nfsd kernel threads won't die via SIGKILL Message-ID: <YTXPR0101MB0959B4E960B85ACAC07B819DDD740@YTXPR0101MB0959.CANPRD01.PROD.OUTLOOK.COM>
next in thread | raw e-mail | index | archive | help
During testing of the pNFS server I have been frequently killing/restarting=
the nfsd.
Once in a while, the "slave" nfsd process doesn't terminate and a "ps axHl"=
shows:
0 48889 1 0 20 0 5884 812 svcexit D - 0:00.01 nfsd: ser=
ver=20
0 48889 1 0 40 0 5884 812 rpcsvc I - 0:00.00 nfsd: ser=
ver=20
... more of the same
0 48889 1 0 40 0 5884 812 rpcsvc I - 0:00.00 nfsd: ser=
ver=20
0 48889 1 0 -8 0 5884 812 rpcsvc I - 1:51.78 nfsd: ser=
ver=20
0 48889 1 0 -8 0 5884 812 rpcsvc I - 2:27.75 nfsd: ser=
ver=20
You can see that the top thread (the one that was created with the process)=
is
stuck in "D" on "svcexit".
The rest of the threads are still servicing NFS RPCs. If you still have an =
NFS mount on
the server, the mount continues to work and the CPU time for the last two t=
hreads
slowly climbs, due to NFS RPC activity. A SIGKILL was posted for the proces=
s and
these threads (created by kthread_add) are here, but the
cv_wait_sig/cv_timedwait_sig never seems to return EINTR for these other th=
reads.
if (ismaster || (!ismaster &&
1207 grp->sg_threadcount > grp->sg_minthreads)=
)
1208 error =3D cv_timedwait_sig(&st->st_co=
nd,
1209 &grp->sg_lock, 5 * hz);
1210 else
1211 error =3D cv_wait_sig(&st->st_cond,
1212 &grp->sg_lock);
The top thread (referred to in svc.c as "ismaster" did return from here wit=
h EINTR
and has now done an msleep() here, waiting for the other threads to termina=
te.
/* Waiting for threads to stop. */
1387 for (g =3D 0; g < pool->sp_groupcount; g++) {
1388 grp =3D &pool->sp_groups[g];
1389 mtx_lock(&grp->sg_lock);
1390 while (grp->sg_threadcount > 0)
1391 msleep(grp, &grp->sg_lock, 0, "svcexit", 0);
1392 mtx_unlock(&grp->sg_lock);
1393 }
Although I can't be sure if this patch has fixed the problem because it hap=
pens
intermittently, I have not seen the problem since applying this patch:
--- rpc/svc.c.sav 2018-06-21 22:52:11.623955000 -0400
+++ rpc/svc.c 2018-06-22 09:01:40.271803000 -0400
@@ -1388,7 +1388,7 @@ svc_run(SVCPOOL *pool)
grp =3D &pool->sp_groups[g];
mtx_lock(&grp->sg_lock);
while (grp->sg_threadcount > 0)
- msleep(grp, &grp->sg_lock, 0, "svcexit", 0);
+ msleep(grp, &grp->sg_lock, 0, "svcexit", 1);
mtx_unlock(&grp->sg_lock);
}
}
As you can see, all it does is add a timeout to the msleep().=20
I am not familiar with the signal delivery code in sleepqeue, so it probabl=
y
isn't correct, but my theory is alonge the lines of...
Since the msleep() doesn't have PCATCH, it does not set TDF_SINTR
and if that happens before the other threads return EINTR from cv_wait_sig(=
),
they no longer do so?
And I thought that waking up from the msleep() via timeouts would maybe all=
ow
the other threads to return EINTR from cv_wait_sig()?
Does this make sense? rick
ps: I'll post if I see the problem again with the patch applied.
pss: This is a single core i386 system, just in case that might affect this=
.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?YTXPR0101MB0959B4E960B85ACAC07B819DDD740>