Date: Tue, 21 Mar 2006 17:37:06 +0200 From: Kostik Belousov <kostikbel@gmail.com> To: Rong-En Fan <grafan@gmail.com> Cc: Mohan Srinivasan <mohan_srinivasan@yahoo.com>, current@freebsd.org Subject: [patches] [more problems] Re: unkillable process stuck in "nfsaio" Message-ID: <20060321153706.GA63595@deviant.kiev.zoral.com.ua> In-Reply-To: <6eb82e0603171026h4bc49a08kb0bda30e171feaec@mail.gmail.com> References: <200603161516.27081.mi%2Bmx@aldan.algebra.com> <20060317094103.GD1203@deviant.kiev.zoral.com.ua> <6eb82e0603171026h4bc49a08kb0bda30e171feaec@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
--ZPt4rx8FFjLCG7dd
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
On Fri, Mar 17, 2006 at 01:26:13PM -0500, Rong-En Fan wrote:
> On 3/17/06, Kostik Belousov <kostikbel@gmail.com> wrote:
> > Just out of curiosity:
> >
> > could you, please, test this little patch:
> >
> > Index: sys/nfsclient/nfs_bio.c
>=20
> I have been played around with intr/nointr with this patch + some in
> nfs_bio.c, nfs_vnops.c.
> Here are the results (note that, the revision # below means the
> changes made in that
> revision), tested on RELENG_6 as of yesterday, i386, SMP. kernel is built=
with
> INVARIANT on.
>=20
> * nfs_vnops.c 1.262, nfs_bio.c 1.154
>=20
> - intr
>=20
> $ dd if=3D/dev/zero of=3Db bs=3D1m count=3D50
> 50+0 records in
> 50+0 records out
> 52428800 bytes transferred in 4.410424 secs (11887474 bytes/sec)
> $ dd if=3D/dev/zero of=3Db bs=3D1m count=3D50
> ^C^C^C^C^C^C^C^C^C^C7+0 records in
> 6+0 records out
> 6291456 bytes transferred in 291.017236 secs (21619 bytes/sec)
> (stuck in nfsaio)
>=20
> - nointr
>=20
> $ dd if=3D/dev/zero of=3Db bs=3D1m count=3D50
> 50+0 records in
> 50+0 records out
> 52428800 bytes transferred in 4.264193 secs (12295128 bytes/sec)
> $ dd if=3D/dev/zero of=3Db bs=3D1m count=3D50
> ^C12+0 records in
> 11+0 records out
> 11534336 bytes transferred in 0.990210 secs (11648373 bytes/sec)
>=20
> * nfs_vnops.c 1.262
>=20
> - intr
>=20
> $dd if=3D/dev/zero of=3Db bs=3D1m count=3D50
> 50+0 records in
> 50+0 records out
> 52428800 bytes transferred in 4.238704 secs (12369064 bytes/sec)
> $ dd if=3D/dev/zero of=3Db bs=3D1m count=3D50
> ^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C
> ^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^Ccc^C^C^C^C15+0 records =
in
> 14+0 records out
> 14680064 bytes transferred in 677.578696 secs (21665 bytes/sec)
> (stuck in nfsaio)
>=20
> - nointr
>=20
> $ dd if=3D/dev/zero of=3Db bs=3D1m count=3D50
> 50+0 records in
> 50+0 records out
> 52428800 bytes transferred in 4.255155 secs (12321244 bytes/sec)
> $ dd if=3D/dev/zero of=3Db bs=3D1m count=3D50
> ^C11+0 records in
> 10+0 records out
> 10485760 bytes transferred in 0.899381 secs (11658864 bytes/sec)
>=20
> * nfs_vnops.c 1.262, nfs_bio.c (remove slpflag =3D 0)
>=20
> - intr
>=20
> $ dd if=3D/dev/zero of=3Db bs=3D1m count=3D50
> 50+0 records in
> 50+0 records out
> 52428800 bytes transferred in 4.245185 secs (12350181 bytes/sec)
> $ dd if=3D/dev/zero of=3Db bs=3D1m count=3D50
> ^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C
> (top's state is changing between CPU0, CPU1, RUN, *Giant)
> (11 minutes passed, I reboot this box)
>=20
> - nointr
> $ dd if=3D/dev/zero of=3Db bs=3D1m count=3D50
> 50+0 records in
> 50+0 records out
> 52428800 bytes transferred in 4.454680 secs (11769375 bytes/sec)
> $ dd if=3D/dev/zero of=3Db bs=3D1m count=3D50
> ^C17+0 records in
> 16+0 records out
> 16777216 bytes transferred in 1.458180 secs (11505587 bytes/sec)
>=20
> * nfs_vnops.c 1.262, nfs_bio.c 1.154 (remove slpflags =3D 0)
>=20
> - intr
> $ dd if=3D/dev/zero of=3Db bs=3D1m count=3D50
> 50+0 records in
> 50+0 records out
> 52428800 bytes transferred in 4.386083 secs (11953445 bytes/sec)
> $ dd if=3D/dev/zero of=3Db bs=3D1m count=3D50
> ^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C
> (top's state is changing between CPU0, CPU1, RUN, *Giant)
> (44 minutes passwd, I rebooted)
>=20
> - noitntr
> $ dd if=3D/dev/zero of=3Db bs=3D1m count=3D50
> 50+0 records in
> 50+0 records out
> 52428800 bytes transferred in 4.370959 secs (11994805 bytes/sec)
> $ dd if=3D/dev/zero of=3Db bs=3D1m count=3D50
> ^C25+0 records in
> 24+0 records out
> 25165824 bytes transferred in 2.122789 secs (11855076 bytes/sec)
>=20
> Looks like that the changes to nfs_vnops.c last Nov by ps@, ^C can really=
stop
> the process, but it take too much time :(
>=20
> Hope this helps,
> Rong-En Fan
Sure, removal of setting slpflag to 0 was wrong. It seems I found the reaso=
n(s)
for the process to stuck in nfsaio state:
1. It is wrong to call nfs_asyncio wih NULL td. This function assumes
it running in the context of the real process and do the checks
for signals delivered to it. Giving NULL td effectively
ignored any signals. Patch:
Index: sys/nfsclient/nfs_vnops.c
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
RCS file: /usr/local/arch/ncvs/src/sys/nfsclient/nfs_vnops.c,v
retrieving revision 1.264
diff -u -r1.264 nfs_vnops.c
--- sys/nfsclient/nfs_vnops.c 8 Mar 2006 01:43:01 -0000 1.264
+++ sys/nfsclient/nfs_vnops.c 21 Mar 2006 15:23:29 -0000
@@ -2588,7 +2588,7 @@
* otherwise just do it ourselves.
*/
if ((bp->b_flags & B_ASYNC) =3D=3D 0 ||
- nfs_asyncio(VFSTONFS(ap->a_vp->v_mount), bp, NOCRED, td))
+ nfs_asyncio(VFSTONFS(ap->a_vp->v_mount), bp, NOCRED, curthread))
(void)nfs_doio(ap->a_vp, bp, cr, td);
return (0);
}
2. Signals delivered to the process may be actually put into the thread
signal list. nfs_sigintr checked global process signal list only.
Patch:
Index: sys/nfsclient/nfs_socket.c
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
RCS file: /usr/local/arch/ncvs/src/sys/nfsclient/nfs_socket.c,v
retrieving revision 1.135
diff -u -r1.135 nfs_socket.c
--- sys/nfsclient/nfs_socket.c 20 Jan 2006 15:07:18 -0000 1.135
+++ sys/nfsclient/nfs_socket.c 21 Mar 2006 15:23:29 -0000
@@ -1513,11 +1513,13 @@
p =3D td->td_proc;
PROC_LOCK(p);
tmpset =3D p->p_siglist;
+ SIGSETOR(tmpset, td->td_siglist);
SIGSETNAND(tmpset, td->td_sigmask);
mtx_lock(&p->p_sigacts->ps_mtx);
SIGSETNAND(tmpset, p->p_sigacts->ps_sigignore);
mtx_unlock(&p->p_sigacts->ps_mtx);
- if (SIGNOTEMPTY(p->p_siglist) && nfs_sig_pending(tmpset)) {
+ if ((SIGNOTEMPTY(p->p_siglist) || SIGNOTEMPTY(td->td_siglist))
+ && nfs_sig_pending(tmpset)) {
PROC_UNLOCK(p);
return (EINTR);
}
But, this just reveals (at least) two another problems,
for which I do not have patches. I will
continue numeration.
[Situations below where obtained by running dd as specified in quoted
text, and, after some time, making nfs server unavailable by firewall
rules.]
3. Sometimes, the system goes into livelock with processes hanged in the
flswai state (AKA bwillwrite() function).
4. Sometimes, the dd process could not exit. It stucks in the exit()
code, trying to close filedescriptor for nfs-located file. This
nfs vnode is exclusively locked by bufdaemon, and close() path
tries to get exclusive lock on it.
So, it seems, that combination of intr mounting and large writes
currently causes only problems. :(
--ZPt4rx8FFjLCG7dd
Content-Type: application/pgp-signature
Content-Disposition: inline
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2.2 (FreeBSD)
iD8DBQFEIB2hC3+MBN1Mb4gRAn2EAJ4n/aK15gicB03ALjY8eGVZe2u/RwCg2hYI
XHgW2y/ZUIHQvFeD6U9Tlig=
=aAO6
-----END PGP SIGNATURE-----
--ZPt4rx8FFjLCG7dd--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20060321153706.GA63595>