Date: Sun, 23 Dec 2012 21:37:26 +0200 From: Konstantin Belousov <kostikbel@gmail.com> To: Ian Lepore <freebsd@damnhippie.dyndns.org> Cc: freebsd-current@freebsd.org Subject: Re: Why does sleep(1) end up blocked in bwillwrite()? Message-ID: <20121223193726.GX53644@kib.kiev.ua> In-Reply-To: <1356288915.1129.68.camel@revolution.hippie.lan> References: <1356288915.1129.68.camel@revolution.hippie.lan>
next in thread | previous in thread | raw e-mail | index | archive | help
--BtNCO5h84ExDStzY
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
On Sun, Dec 23, 2012 at 11:55:15AM -0700, Ian Lepore wrote:
> Background: I'm trying to get nandfs working on a low-end small-memory
> embedded system. I'm debugging performance problems that manifest as
> the system (or large portions of it) becoming unresponsive for many
> seconds at a time. It appears that sometimes the nandfs background
> garbage collector does things that lead to dirtying lots of buffers way
> faster than they can be written. When that happens it seems to take too
> long (many seconds) for the problem to clear. That's the basic
> situation I'm investigating, but NOT what this mail is about, that's
> just the background.
>=20
> When this situation happens, some of the threads in my application keep
> running fine. Others get blocked unexpectedly even though they do no
> disk IO at all, they're working with sockets and serial (uart) devices.
> I discovered by accident that I can see a form of the problem happening
> just using sleep(1) and hitting ^T while the buffer starvation is in
> progress...
>=20
> guava# sleep 999999
> [ hit ^T]
> load: 1.03 cmd: sleep 472 [nanslp] 2.03r 0.01u 0.02s 0% 1372k
> sleep: about 999997 second(s) left out of the original 999999
> [ hit ^T]
> load: 1.27 cmd: sleep 472 [nanslp] 9.32r 0.01u 0.02s 0% 1376k
> sleep: about 999989 second(s) left out of the original 999999
> [ hit ^T]
> load: 1.49 cmd: sleep 472 [nanslp] 11.53r 0.01u 0.02s 0% 1376k
> [ note no output from sleep(1) here, repeated ^T now gives...]
> load: 1.49 cmd: sleep 472 [flswai] 12.01r 0.01u 0.03s 0% 1376k
> load: 1.49 cmd: sleep 472 [flswai] 12.27r 0.01u 0.03s 0% 1376k
> load: 1.49 cmd: sleep 472 [flswai] 12.76r 0.01u 0.03s 0% 1376k
> load: 1.49 cmd: sleep 472 [flswai] 13.06r 0.01u 0.03s 0% 1376k
> load: 1.49 cmd: sleep 472 [flswai] 13.26r 0.01u 0.03s 0% 1376k
> load: 1.61 cmd: sleep 472 [flswai] 20.03r 0.02u 0.07s 0% 1376k
> load: 1.64 cmd: sleep 472 [flswai] 20.49r 0.02u 0.07s 0% 1376k
> load: 1.64 cmd: sleep 472 [flswai] 20.68r 0.02u 0.08s 0% 1376k
> sleep: about 999987 second(s) left out of the original 999999
>=20
> So here sleep(1) was blocked in bwillwrite() for about 9 seconds on a
> write to stderr (which is an ssh xterm connection).
>=20
> The call to bwillwrite() is in kern/sys_generic.c in dofilewrite():
>=20
> if (fp->f_type =3D=3D DTYPE_VNODE)
> bwillwrite();
>=20
> I just noticed the checkin message that added the DTYPE_VNODE check
> specifically mentions not penalizing devices and pipes and such. I
> think maybe things have evolved since then (Dec 2000) and this check is
> no longer sufficient. Maybe it needs to be something more like
>=20
> if (fp->f_type =3D=3D DTYPE_VNODE && fp->f_vnode->v_type =3D=3D VREG)
>=20
> but I have a gut feeling it needs to be more complex than that (can
> f_vnode be NULL, what sort of locking is required to peek into f_vnode
> at this point, etc), so I can't really propose a patch for this. In
> fact, I can't even say for sure it's a bug, but it sure feels like one
> to the application-developer part of me.
The patch below would do what you want. But my opinion is that it is more
bug in the filesystem than in the VFS. Anyway, try this and report how
it works for you.
diff --git a/sys/fs/devfs/devfs_vnops.c b/sys/fs/devfs/devfs_vnops.c
index 97a1bcf..9851229 100644
--- a/sys/fs/devfs/devfs_vnops.c
+++ b/sys/fs/devfs/devfs_vnops.c
@@ -1049,6 +1049,7 @@ devfs_open(struct vop_open_args *ap)
int error, ref, vlocked;
struct cdevsw *dsw;
struct file *fpop;
+ struct mtx *mtxp;
=20
if (vp->v_type =3D=3D VBLK)
return (ENXIO);
@@ -1099,6 +1100,16 @@ devfs_open(struct vop_open_args *ap)
#endif
if (fp->f_ops =3D=3D &badfileops)
finit(fp, fp->f_flag, DTYPE_VNODE, dev, &devfs_ops_f);
+ mtxp =3D mtx_pool_find(mtxpool_sleep, fp);
+
+ /*
+ * Hint to the dofilewrite() to not force the buffer draining
+ * on the writer to the file. Most likely, the write would
+ * not need normal buffers.
+ */
+ mtx_lock(mtxp);
+ fp->f_vnread_flags |=3D FDEVFS_VNODE;
+ mtx_unlock(mtxp);
return (error);
}
=20
diff --git a/sys/kern/sys_generic.c b/sys/kern/sys_generic.c
index f47cb03..b97ff7f 100644
--- a/sys/kern/sys_generic.c
+++ b/sys/kern/sys_generic.c
@@ -536,7 +536,8 @@ dofilewrite(td, fd, fp, auio, offset, flags)
ktruio =3D cloneuio(auio);
#endif
cnt =3D auio->uio_resid;
- if (fp->f_type =3D=3D DTYPE_VNODE)
+ if (fp->f_type =3D=3D DTYPE_VNODE &&
+ (fp->f_vnread_flags & FDEVFS_VNODE) =3D=3D 0)
bwillwrite();
if ((error =3D fo_write(fp, auio, td->td_ucred, flags, td))) {
if (auio->uio_resid !=3D cnt && (error =3D=3D ERESTART ||
diff --git a/sys/sys/file.h b/sys/sys/file.h
index dc49895..cf5f1ea 100644
--- a/sys/sys/file.h
+++ b/sys/sys/file.h
@@ -178,7 +178,8 @@ struct file {
#define f_advice f_vnun.fvn_advice
=20
#define FOFFSET_LOCKED 0x1
-#define FOFFSET_LOCK_WAITING 0x2 =20
+#define FOFFSET_LOCK_WAITING 0x2
+#define FDEVFS_VNODE 0x4
=20
#endif /* _KERNEL || _WANT_FILE */
=20
--BtNCO5h84ExDStzY
Content-Type: application/pgp-signature
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.19 (FreeBSD)
iQIcBAEBAgAGBQJQ1112AAoJEJDCuSvBvK1BUUEP/2B8DGgz6+B3R+m+kw6kxbRS
0ykn+DhUnrfCqB8RvcVKh75w4OE4J+JXrLf++je/EME3HopaUroGSydDEfHgXsQE
FKr9qSxlDIf06TKyVf8LeQwSS0+9+Otmp24pFPuLzLuRP5JXO+X/sB2TFsLiWN1/
N3rc8y/K5X5NjTi3uxlHSwM5GpDKOwxNDcZ5rLjO8lCq/iWAuyDGvJ7R2G7gvN7t
yveUcd8ryJBAdMHkD3SjxAJNwdm/wLEo7Df2YizJTiuscj3nkXozNhyzhkEkEflQ
AYzXV0/jvmrm0wBT1WfREMTL6clkPL+7Bd1PnWi8yvvydkEp2OBFQbH+nC4FSn54
jJ7l3y4TT/K0rSxrJpJmjI/ZMR1aifXzkLQdSKUH22EFXl0hHdwJuosBL/yu6org
DfCGhMbETP3naJOzsqVjRHzjTDW5y7y21q+/pAtgMLtzOspDvpN8SWsnPhPvytiw
hn/cBOD0LdAqOEibyVzDmGYbSPVjl6vnKY6q8x7F2g6K3+P4BJyQ7vnUas4UAjka
mPAh6hC1sHGHQL87WlqNjT+3fiNcB9L0a+tEThZiYDlUAGdzxr+apcMPBKvpdeDA
JCzr/VWGdhlrgcSmZ9/UKKtkZLl8yU+vY55hlihmFgkJ67Dvw3sjAuAl6Pf+YDg2
GvUSF7hrsyPIsQZ6kruz
=iGgs
-----END PGP SIGNATURE-----
--BtNCO5h84ExDStzY--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20121223193726.GX53644>