From owner-freebsd-hackers@freebsd.org Fri Aug 16 17:16:23 2019 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 297D5A91C1 for ; Fri, 16 Aug 2019 17:16:23 +0000 (UTC) (envelope-from shrikanth07@gmail.com) Received: from mail-wr1-x436.google.com (mail-wr1-x436.google.com [IPv6:2a00:1450:4864:20::436]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 46992t2Lpvz4Tn9 for ; Fri, 16 Aug 2019 17:16:22 +0000 (UTC) (envelope-from shrikanth07@gmail.com) Received: by mail-wr1-x436.google.com with SMTP id t16so2223692wra.6 for ; Fri, 16 Aug 2019 10:16:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:from:date:message-id:subject:to; bh=No9X/PuMlvorZAFvrBTy5W0y28W0CxmL1jsvlqyoJVA=; b=mP79yCAktarv84oSZw1Hsr7XzE/tIdQYkKUJbQt8D3hkZWPhdjHcHKlr33U2dhPj+h 7FIjwyXpXRllzrQlF7OjowHDJ2bFXqcoCVvOhhN/xqiKL9xz2Er6UVFYUNIdayu5zQcM PsSMdGOo1aaOvLrm4SdFfik+l9zZPcVGHLJz9nehVoHo/Y6Pawp3VB+jZLR7bCQzwQ3v t+WwfE5WI7GqjFpkNrYX++nyjxXqH5Gy6BFMmUmzJLhXtazI5jOHQJv34AZ8kFV88R+/ wEX+Y6czLUbUkhyQaQ1bcf48CFHhIPvgr3pdWtfGW8JSW3kki3SjN6Mgz1oH25SGKssE HgCA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=No9X/PuMlvorZAFvrBTy5W0y28W0CxmL1jsvlqyoJVA=; b=r4/KzLQ0FVBuhQeI61HvoEYRdbwDK6poKSBcP8nrYrDyoO8BpJhqsIVMNyv40vAxvv xk1Nn/Ih/SbgvRmS0l4lHZGvIyx/8hUM4JWMGycy4D65pSSVD65BIXVlEl6lHh0SOmyG w/d44c3CHnavifknj2tNXO7wE2RVC5+iWP51PkEgLK4tOYwOybgt9Nul3MHkcEllpvAA wLQyKd/mjcmhgrNaMs9iu7DGYPy/vqBsCtxOiy57K2BkoGVv17Y7nvht70B8kw2mUiYx VR58M8B4MSGH/2OpKFbWHTNuXLw0kBD43aaEWmepf5ja+jewaoAipGaAkrb7wM4GqWQn xb5g== X-Gm-Message-State: APjAAAWV67Cpo5GbNGH5A6bU8EU5W05AuIEwZBGuetrg9/AxMYxgbwkP wS5fLVMtBekljx8SlMOys2CE4fcowirHSZM2IsdTSVY4 X-Google-Smtp-Source: APXvYqzfSgGmlQzdJ5bonoytPZufcr1A004nj9fUYk6unJa2QU+Uh6eVWmX+eRi0YJG4T/gokXh/7toGnXRcr+GvBwM= X-Received: by 2002:a05:6000:1085:: with SMTP id y5mr12178969wrw.285.1565975776708; Fri, 16 Aug 2019 10:16:16 -0700 (PDT) MIME-Version: 1.0 From: Shrikanth Kamath Date: Fri, 16 Aug 2019 10:16:05 -0700 Message-ID: Subject: Reclaiming "dirty buffers" after seeing "fsync: giving up on dirty..." / Unplugging USB while copy in progress To: freebsd-hackers@freebsd.org X-Rspamd-Queue-Id: 46992t2Lpvz4Tn9 X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=mP79yCAk; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of shrikanth07@gmail.com designates 2a00:1450:4864:20::436 as permitted sender) smtp.mailfrom=shrikanth07@gmail.com X-Spamd-Result: default: False [-4.00 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36:c]; FREEMAIL_FROM(0.00)[gmail.com]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; TO_DN_NONE(0.00)[]; IP_SCORE_FREEMAIL(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; IP_SCORE(0.00)[ip: (-9.21), ipnet: 2a00:1450::/32(-3.03), asn: 15169(-2.38), country: US(-0.05)]; TO_MATCH_ENVRCPT_ALL(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; RCVD_IN_DNSWL_NONE(0.00)[6.3.4.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.5.4.1.0.0.a.2.list.dnswl.org : 127.0.5.0]; NEURAL_HAM_SHORT(-1.00)[-0.998,0]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com.dwl.dnswl.org : 127.0.5.0] Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 Aug 2019 17:16:23 -0000 How do =E2=80=9Clingering=E2=80=9D dirty buffers get reclaimed? In the func= tion vop_stdfsync there is logic to retry but eventually fail after =E2=80=9Cmax= retry=E2=80=9D and print =E2=80=9Cfsync: giving up on dirty (error =E2=80=9C while returni= ng the error. In a scenario where a USB stick is plugged in and a large file (> 1.5G) is being copied to it from the host filesystem when the USB device is abruptly removed. I see the fsync function retrying for a number of times before returning with the below error fsync: giving up on dirty 0xfffff8058091d1d8: tag devfs, type VCHR usecount 1, writecount 0, refcount 1070 mountedhere 0xfffff805808af800 flags (VI_DOOMED|VI_ACTIVE) v_object 0xfffff807a6efe948 ref 0 pages 1069 cleanbuf 893 dirtybuf 174 lock type devfs: EXCL by thread 0xfffff8009aebb560 (pid 6463, chassisd, tid 100270) What is eventually happening is there are other processes that start appearing to be stuck waiting in =E2=80=9Cflswai=E2=80=9D state (including = the copy operation to the USB stick). # ps jaux -o mwchan -o command | grep flswai 6423 1 6423 6423 0 Ds - 0:02.35 /usr/sbin/eventd 0.0 0.0 744768 12916 06:22 flswai /usr/sbin/eventd -r -s -A 6463 6428 6427 6427 0 D - 8:25.69 /usr/sbin/chassi 0.0 0.1 862940 56472 06:22 flswai /usr/sbin/chassisd -N 19753 19195 19753 6453 1 D+ u0 0:01.08 cp junos-vmhost- 0.0 0.0 8164 2968 12:13 flswai cp junos-vmhost-install-mx-x86-64-19.3I-14062-TB-130172-_cd-builder.tgz /mnt/ Looking at the code, this seems to be coming from the =E2=80=9Cbwillwrite= =E2=80=9D function (sys/kern/vfs_bio.c) where it explains it will block prior to =E2=80=9C=E2= =80=A6locking of any vnodes we attempt to avoid the situation where a locked vnode prevents the various system daemons from flushing related buffers=E2=80=A6=E2=80=9D = How does the dirty buffers in this scenario get reclaimed? The dmesg log is from a Juniper device running stable/11 (closer to 11.1ish) based Junos. Jul 23 12:06:31.740 da0 at umass-sim0 bus 0 scbus3 target 0 lun 0 Jul 23 12:06:31.740 da0: s/n AA04012700046751 detached Jul 23 12:06:31.740 g_vfs_done():da0p1[WRITE(offset=3D272711680, length=3D65536)]error =3D 6 ... Jul 23 12:06:31.943 g_vfs_done():da0p1[WRITE(offset=3D277626880, length=3D65536)]error =3D 6 ... Jul 23 12:06:31.992 g_vfs_done():da0p1[WRITE(offset=3D281624576, length=3D65536)]error =3D 6 ... Jul 23 12:06:32.144 g_vfs_done():da0p1[WRITE(offset=3D285687808, length=3D65536)]error =3D 6 Jul 23 12:06:32.144 (da0:umass-sim0:0:0:0): Periph destroyed Jul 23 12:06:32.144 umass0: detached Jul 23 12:06:36.672 fsync: giving up on dirty 0xfffff8058091d1d8: tag devfs, type VCHR Jul 23 12:06:36.672 usecount 1, writecount 0, refcount 1070 mountedhere 0xfffff805808af800 Jul 23 12:06:36.672 flags (VI_DOOMED|VI_ACTIVE) Jul 23 12:06:36.672 v_object 0xfffff807a6efe948 ref 0 pages 1069 cleanbuf 893 dirtybuf 174 -- Shrikanth R K