Date: Fri, 16 Aug 2019 10:16:05 -0700 From: Shrikanth Kamath <shrikanth07@gmail.com> To: freebsd-hackers@freebsd.org Subject: Reclaiming "dirty buffers" after seeing "fsync: giving up on dirty..." / Unplugging USB while copy in progress Message-ID: <CAEOAkMUxmTM5OK8hKE1Fyp3w5yxCyQZfmnK=LaMit8YJ%2BQw=JQ@mail.gmail.com>
next in thread | raw e-mail | index | archive | help
How do =E2=80=9Clingering=E2=80=9D dirty buffers get reclaimed? In the func=
tion
vop_stdfsync there is logic to retry but eventually fail after =E2=80=9Cmax=
retry=E2=80=9D
and print =E2=80=9Cfsync: giving up on dirty (error =E2=80=9C while returni=
ng the error. In
a scenario where a USB stick is plugged in and a large file (> 1.5G) is
being copied to it from the host filesystem when the USB device is abruptly
removed. I see the fsync function retrying for a number of times before
returning with the below error
fsync: giving up on dirty 0xfffff8058091d1d8: tag devfs, type VCHR
usecount 1, writecount 0, refcount 1070 mountedhere 0xfffff805808af800
flags (VI_DOOMED|VI_ACTIVE)
v_object 0xfffff807a6efe948 ref 0 pages 1069 cleanbuf 893 dirtybuf 174
lock type devfs: EXCL by thread 0xfffff8009aebb560 (pid 6463, chassisd,
tid 100270)
What is eventually happening is there are other processes that start
appearing to be stuck waiting in =E2=80=9Cflswai=E2=80=9D state (including =
the copy
operation to the USB stick).
# ps jaux -o mwchan -o command | grep flswai
6423 1 6423 6423 0 Ds - 0:02.35 /usr/sbin/eventd
0.0 0.0 744768 12916 06:22 flswai /usr/sbin/eventd -r -s -A
6463 6428 6427 6427 0 D - 8:25.69 /usr/sbin/chassi 0.0
0.1 862940 56472 06:22 flswai /usr/sbin/chassisd -N
19753 19195 19753 6453 1 D+ u0 0:01.08 cp junos-vmhost- 0.0
0.0 8164 2968 12:13 flswai cp
junos-vmhost-install-mx-x86-64-19.3I-14062-TB-130172-_cd-builder.tgz /mnt/
Looking at the code, this seems to be coming from the =E2=80=9Cbwillwrite=
=E2=80=9D function
(sys/kern/vfs_bio.c) where it explains it will block prior to =E2=80=9C=E2=
=80=A6locking of
any vnodes we attempt to avoid the situation where a locked vnode prevents
the various system daemons from flushing related buffers=E2=80=A6=E2=80=9D =
How does the
dirty buffers in this scenario get reclaimed?
The dmesg log is from a Juniper device running stable/11 (closer to
11.1ish) based Junos.
Jul 23 12:06:31.740 da0 at umass-sim0 bus 0 scbus3 target 0 lun 0
Jul 23 12:06:31.740 da0: <USBFlash USBFlashDrive 0100> s/n
AA04012700046751 detached
Jul 23 12:06:31.740 g_vfs_done():da0p1[WRITE(offset=3D272711680,
length=3D65536)]error =3D 6
...
Jul 23 12:06:31.943 g_vfs_done():da0p1[WRITE(offset=3D277626880,
length=3D65536)]error =3D 6
...
Jul 23 12:06:31.992 g_vfs_done():da0p1[WRITE(offset=3D281624576,
length=3D65536)]error =3D 6
...
Jul 23 12:06:32.144 g_vfs_done():da0p1[WRITE(offset=3D285687808,
length=3D65536)]error =3D 6
Jul 23 12:06:32.144 (da0:umass-sim0:0:0:0): Periph destroyed
Jul 23 12:06:32.144 umass0: detached
Jul 23 12:06:36.672 fsync: giving up on dirty 0xfffff8058091d1d8: tag
devfs, type VCHR
Jul 23 12:06:36.672 usecount 1, writecount 0, refcount 1070
mountedhere 0xfffff805808af800
Jul 23 12:06:36.672 flags (VI_DOOMED|VI_ACTIVE)
Jul 23 12:06:36.672 v_object 0xfffff807a6efe948 ref 0 pages 1069
cleanbuf 893 dirtybuf 174
--
Shrikanth R K
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAEOAkMUxmTM5OK8hKE1Fyp3w5yxCyQZfmnK=LaMit8YJ%2BQw=JQ>