From owner-freebsd-fs@freebsd.org Thu Oct 3 08:51:14 2019 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 8218E12E4AD for ; Thu, 3 Oct 2019 08:51:14 +0000 (UTC) (envelope-from pen@lysator.liu.se) Received: from mail.lysator.liu.se (mail.lysator.liu.se [IPv6:2001:6b0:17:f0a0::3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 46kRYs19Jpz3KjM for ; Thu, 3 Oct 2019 08:51:12 +0000 (UTC) (envelope-from pen@lysator.liu.se) Received: from mail.lysator.liu.se (localhost [127.0.0.1]) by mail.lysator.liu.se (Postfix) with ESMTP id A24DB4000D for ; Thu, 3 Oct 2019 10:51:09 +0200 (CEST) Received: from [IPv6:2001:6b0:17:f002:4d34:eff1:ee71:9078] (unknown [IPv6:2001:6b0:17:f002:4d34:eff1:ee71:9078]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.lysator.liu.se (Postfix) with ESMTPSA id 845064000C for ; Thu, 3 Oct 2019 10:51:09 +0200 (CEST) From: Peter Eriksson Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Mime-Version: 1.0 (Mac OS X Mail 12.4 \(3445.104.11\)) Subject: Re: zfs_unlinked_drain "forever"? Date: Thu, 3 Oct 2019 10:51:09 +0200 References: To: =?utf-8?Q?Karli_Sj=C3=B6berg_via_freebsd-fs?= In-Reply-To: Message-Id: X-Mailer: Apple Mail (2.3445.104.11) X-Virus-Scanned: ClamAV using ClamSMTP X-Rspamd-Queue-Id: 46kRYs19Jpz3KjM X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=pass (policy=none) header.from=liu.se; spf=pass (mx1.freebsd.org: domain of pen@lysator.liu.se designates 2001:6b0:17:f0a0::3 as permitted sender) smtp.mailfrom=pen@lysator.liu.se X-Spamd-Result: default: False [-2.28 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-0.999,0]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+a:mail.lysator.liu.se]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; MIME_TRACE(0.00)[0:+]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; RCVD_COUNT_THREE(0.00)[3]; RCVD_TLS_LAST(0.00)[]; TO_DN_ALL(0.00)[]; RCVD_IN_DNSWL_MED(-0.20)[3.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.a.0.f.7.1.0.0.0.b.6.0.1.0.0.2.list.dnswl.org : 127.0.11.2]; DMARC_POLICY_ALLOW(-0.50)[liu.se,none]; MV_CASE(0.50)[]; IP_SCORE(-0.78)[ipnet: 2001:6b0::/32(-2.20), asn: 1653(-1.69), country: EU(-0.01)]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; SUBJECT_ENDS_QUESTION(1.00)[]; ASN(0.00)[asn:1653, ipnet:2001:6b0::/32, country:EU]; MID_RHS_MATCH_FROM(0.00)[] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 03 Oct 2019 08:51:14 -0000 Weee.. I can report that _that_ =E2=80=9Czfs mount=E2=80=9D of that = filesystem took ~18 hours. Now it is continuing with the rest=E2=80=A6 # df -h | wc -l 11563 (It=E2=80=99s currently mounting about 1 filesystem per second so at = that pace it=E2=80=99ll be done in=E2=80=A6 12 hours). - Peter > On 3 Oct 2019, at 09:51, Peter Eriksson wrote: >=20 > Just upgraded and rebooted one of our servers from 11.2 to = 11.3-RELEASE-p3 and now it seems =E2=80=9Cstuck=E2=80=9D at mounting the = filesystems=E2=80=A6=20 >=20 > =E2=80=9Cstuck=E2=80=9D as in it _is_ doing something: >=20 >> # zpool iostat 10 >> capacity operations bandwidth >> pool alloc free read write read write >> ---------- ----- ----- ----- ----- ----- ----- >> DATA2 351T 11.3T 5 517 25.3K 4.97M >> DATA3 71.4T 73.6T 0 62 1.84K 863K >> DATA4 115T 12.2T 0 0 1.38K 152 >> zroot 34.2G 78.8G 0 61 5.68K 461K >> ---------- ----- ----- ----- ----- ----- ----- >> DATA2 351T 11.3T 0 272 0 2.46M >> DATA3 71.4T 73.6T 0 0 0 0 >> DATA4 115T 12.2T 0 0 0 0 >> zroot 34.2G 78.8G 0 47 0 200K >> ---------- ----- ----- ----- ----- ----- ----- >=20 > It=E2=80=99s been doing these 272-300 Write IOPS on pool DATA2 since = around 15:00 yesterday now and has mounted 3781 filesystems out of = 58295... >=20 >=20 > A =E2=80=9Cprocstat -kka=E2=80=9D shows one =E2=80=9Czfs mount=E2=80=9D = process currently doing: >=20 >> 26508 102901 zfs - mi_switch+0xeb = sleepq_wait+0x2c _cv_wait+0x16e txg_wait_synced+0xa5=20 >> dmu_tx_assign+0x48 zfs_rmnode+0x122 zfs_freebsd_reclaim+0x4e = VOP_RECLAIM_APV+0x80 vgonel+0x213=20 >> vrecycle+0x46 zfs_freebsd_inactive+0xd VOP_INACTIVE_APV+0x80 = vinactive+0xf0 vputx+0x2c3=20 >> zfs_unlinked_drain+0x1b8 zfsvfs_setup+0x5e zfs_mount+0x623 = vfs_domount+0x573=20 >=20 >=20 >> # ps auxwww|egrep zfs >> root 17 0.0 0.0 0 2864 - DL 14:53 7:03.54 = [zfskern] >> root 960 0.0 0.0 104716 31900 - Is 14:55 0:00.05 = /usr/sbin/mountd -r -S /etc/exports /etc/zfs/exports >> root 4390 0.0 0.0 9040 5872 - Is 14:57 0:00.02 = /usr/sbin/zfsd >> root 20330 0.0 0.0 22652 18388 - S 15:07 0:48.90 = perl /usr/local/bin/parallel --will-cite -j 40 zfs mount {} >> root 26508 0.0 0.0 7804 5316 - D 15:09 0:08.58 = /sbin/zfs mount DATA2/filur04.it.liu.se/DATA/staff/nikca89 >> root 101 0.0 0.0 20148 14860 u1- I 14:55 0:00.88 = /bin/bash /sbin/zfs-speedmount >> root 770 0.0 0.0 6732 2700 0 S+ 09:45 0:00.00 = egrep zfs >=20 > (=E2=80=9Czfs-speedmount=E2=80=9D is a locally developed script that = runs multiple =E2=80=9Czfs mount=E2=80=9D commands in parallel - speeds = up mounting of filesystems a lot on this server _a lot_ (normally), = since =E2=80=9Czfs mount=E2=80=9D didn=E2=80=99t use to mount stuff in = parallell before, took multiple hours when rebooting servers). >=20 >=20 > A google-search found stuff about zfs_unlinked_drain improvements in = Nexenta and Linux ZFS: >=20 >> https://github.com/zfsonlinux/zfs/pull/8142/commits = >=20 > Anyone know if this (or similar) fixes are in FreeBSD ZFS = (11.3-RELEASE-p3)? >=20 >=20 > - Peter >=20 > _______________________________________________ > freebsd-fs@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org"