From owner-freebsd-fs@freebsd.org Sat Apr 11 17:36:41 2020 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 0CE0E2BE41F for ; Sat, 11 Apr 2020 17:36:41 +0000 (UTC) (envelope-from bounces+13739864-7f7e-freebsd-fs=freebsd.org@em848.distal.com) Received: from xtrwsxqr.outbound-mail.sendgrid.net (xtrwsxqr.outbound-mail.sendgrid.net [167.89.106.69]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 49029z4gyKz3Ph7 for ; Sat, 11 Apr 2020 17:36:39 +0000 (UTC) (envelope-from bounces+13739864-7f7e-freebsd-fs=freebsd.org@em848.distal.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=distal.com; h=from:content-type:content-transfer-encoding:mime-version:subject:to; s=s1; bh=50yhssOYidXTlUBY1vxW4pLi02F5jOulGkJoUUhZfvw=; b=kEn7g27av/sCjioqzW7wWZf0puHyf+t4z2Tvn7bjvdFlKUV9pe6rVp3t5844xOnRtWcq vQTZH/LJNxHbO6XdD5KB8M60JkrtiMsUGvyxvmrU6Qz7n7BdosFsBh/7QL7ikNsKDDZxaU qdJpE376QcSqlpQp/oSbZ02kST7sDZprs= Received: by filter0072p3las1.sendgrid.net with SMTP id filter0072p3las1-25929-5E920025-7E 2020-04-11 17:36:37.467380761 +0000 UTC m=+849917.549179951 Received: from mail.distal.com (unknown) by ismtpd0088p1mdw1.sendgrid.net (SG) with ESMTP id f2ecHczMQSClfGzLEGgvdQ Sat, 11 Apr 2020 17:36:37.357 +0000 (UTC) Received: from magrathea.distal.com (magrathea.distal.com [2001:470:e24c:200:14a4:6888:ae02:b3bb]) by tristain.distal.com (OpenSMTPD) with ESMTPSA id 4d32cc50 (TLSv1.2:ECDHE-RSA-AES256-GCM-SHA384:256:NO); Sat, 11 Apr 2020 13:36:35 -0400 (EDT) From: Chris Ross Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Mime-Version: 1.0 (Mac OS X Mail 13.4 \(3608.80.23.2.2\)) Subject: ZFS server has gone crazy slow Message-Id: <2182C27C-A5D3-41BF-9CE9-7C6883E43074@distal.com> Date: Sat, 11 Apr 2020 17:36:37 +0000 (UTC) X-Mailer: Apple Mail (2.3608.80.23.2.2) X-SG-EID: =?us-ascii?Q?gDj=2Futz1vvM0Gg5Dx3C984MHE5rSknXbUiMX+7YYKy8H35YTA3nJpmg1IQ5mw9?= =?us-ascii?Q?RS2bD=2FH4Uluvnmedtzs9ncXZBmcg47PFtWNkjuk?= =?us-ascii?Q?OHD3k9B4AgosUkZt9jcDqCvebQRVsgzS=2FGHvMXh?= =?us-ascii?Q?KhjfCD5FlkPfv79=2FlaPX5rxUjImWBhLjCAG8azS?= =?us-ascii?Q?I6nPaQEBGxHl9vsoyKxChT6nafpJeWUHojxXJJb?= =?us-ascii?Q?uG0XKYghJ2U7Lyz8Cz16G+jdmpiEIVnETriqQd?= To: freebsd-fs , freebsd-stable@freebsd.org X-Rspamd-Queue-Id: 49029z4gyKz3Ph7 X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=distal.com header.s=s1 header.b=kEn7g27a; dmarc=pass (policy=none) header.from=distal.com; spf=pass (mx1.freebsd.org: domain of bounces@em848.distal.com designates 167.89.106.69 as permitted sender) smtp.mailfrom=bounces@em848.distal.com X-Spamd-Result: default: False [-3.87 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_DKIM_ALLOW(-0.20)[distal.com:s=s1]; FROM_NEQ_ENVFROM(0.00)[cross@distal.com,bounces@em848.distal.com]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:167.89.0.0/17]; MV_CASE(0.50)[]; MIME_GOOD(-0.10)[text/plain]; IP_SCORE(-1.67)[ipnet: 167.89.96.0/20(-4.61), asn: 11377(-3.67), country: US(-0.05)]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[distal.com:+]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[69.106.89.167.list.dnswl.org : 127.0.15.0]; DMARC_POLICY_ALLOW(-0.50)[distal.com,none]; FORGED_SENDER(0.30)[cross@distal.com,bounces@em848.distal.com]; RWL_MAILSPIKE_POSSIBLE(0.00)[69.106.89.167.rep.mailspike.net : 127.0.0.17]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:11377, ipnet:167.89.96.0/20, country:US]; MID_RHS_MATCH_FROM(0.00)[]; TAGGED_FROM(0.00)[13739864-7f7e-freebsd-fs=freebsd.org,freebsd] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 11 Apr 2020 17:36:41 -0000 I have a FreeBSD 11.3-STABLE server that is my router, using a ZFS mirror (= of two GPT disks) as it=E2=80=99s disk. It=E2=80=99s many years old, and h= as only been misbehaving like this for a day or so. I=E2=80=99m trying to = figure out what=E2=80=99s wrong. I confirmed that internet connectivity isn=E2=80=99t the problem, and a reb= oot didn=E2=80=99t fix it. (The reboot took 10-15 minutes to finish going = multi-user, starting daemons, due to the underlying problem described below= .) Truss=E2=80=99ing a very basic command (date), I can see that close() and e= xit() calls are taking 1-2 seconds. All of the files being opened are on Z= FS, but I don=E2=80=99t know if that=E2=80=99s for sure related. Similarly= , using shell builtin =E2=80=9Cecho foo=E2=80=9D always is immediate, but = =E2=80=9C/bin/echo=E2=80=9D sometimes works quickly, but sometimes the clos= e() on /var/run/ld-elf.so.hints takes 3-5 seconds. I _think_ this is a filesystem problem. It=E2=80=99s very hard to diagnose= because logging in, and doing anything, takes many seconds per command. z= pool status shows my mirror as online, so I=E2=80=99m not sure where I shou= ld check. I=E2=80=99d appreciate any help! Thanks much=E2=80=A6 - Chris