From owner-freebsd-stable@freebsd.org Sat Apr 11 19:25:59 2020 Return-Path: Delivered-To: freebsd-stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id A891B278A04 for ; Sat, 11 Apr 2020 19:25:59 +0000 (UTC) (envelope-from bounces+13739864-b44a-freebsd-stable=freebsd.org@em848.distal.com) Received: from xtrwsxqr.outbound-mail.sendgrid.net (xtrwsxqr.outbound-mail.sendgrid.net [167.89.106.69]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4904c65fqPz429R for ; Sat, 11 Apr 2020 19:25:58 +0000 (UTC) (envelope-from bounces+13739864-b44a-freebsd-stable=freebsd.org@em848.distal.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=distal.com; h=content-type:mime-version:subject:from:in-reply-to: content-transfer-encoding:references:to:cc; s=s1; bh=L3QU1aqrr/dCTxcLfEoi1w/gjK/CJSvAUy84I6UkH9g=; b=GpiM4OdcHB6D16ruVkTJJVwFTAwYh89AYZ2u7ColCSHmycYNHP0/s0Tu1POCN6c2H7cp cXionFHzJ/oFJjoGD3GtXJG/YIklS+L+LnL6LhGJXlTSH38ttbx/nGW33dEfQvP1wbNfVZ Bdf7hJF2d7X9EkbROFbnaiEv82NmC6yYI= Received: by filter0081p3las1.sendgrid.net with SMTP id filter0081p3las1-23682-5E9219C5-20 2020-04-11 19:25:57.242616472 +0000 UTC m=+856498.432381552 Received: from mail.distal.com (unknown) by ismtpd0039p1iad1.sendgrid.net (SG) with ESMTP id vR56VFraQ86V-X3Z0BRvVw Sat, 11 Apr 2020 19:25:57.023 +0000 (UTC) Received: from magrathea.distal.com (magrathea.distal.com [206.138.151.12]) by tristain.distal.com (OpenSMTPD) with ESMTPSA id 99ccdff1 (TLSv1.2:ECDHE-RSA-AES256-GCM-SHA384:256:NO); Sat, 11 Apr 2020 14:59:16 -0400 (EDT) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 13.4 \(3608.80.23.2.2\)) Subject: Re: ZFS server has gone crazy slow From: Chris Ross In-Reply-To: <68328a40-0e3d-f9cf-510b-9cbfd7cb8acd@grosbein.net> Date: Sat, 11 Apr 2020 19:25:57 +0000 (UTC) Content-Transfer-Encoding: quoted-printable Message-Id: <654D00F8-DBEC-49BD-B871-7EB830F49D50@distal.com> References: <2182C27C-A5D3-41BF-9CE9-7C6883E43074@distal.com> <68328a40-0e3d-f9cf-510b-9cbfd7cb8acd@grosbein.net> X-Mailer: Apple Mail (2.3608.80.23.2.2) X-SG-EID: =?us-ascii?Q?gDj=2Futz1vvM0Gg5Dx3C984MHE5rSknXbUiMX+7YYKy8H35YTA3nJpmg1IQ5mw9?= =?us-ascii?Q?RS2bD=2FH4Uluvnmedtzs9ncXcFUA0lhjELtoV3Mz?= =?us-ascii?Q?d0JINwqw+ym612NsdzaAma+ZokkaT5h2jF+7492?= =?us-ascii?Q?xJJOarFaPq3ghR47aSEsoKnFBF8YX8=2F+fMRJNWQ?= =?us-ascii?Q?d9q81nStGV=2FzhpyrXJzYNsmea5plqZKfVnAT6s8?= =?us-ascii?Q?mWJ8PCUxU6hP5j5fXA+cOszn5oql=2FUVxXqk2as?= To: Eugene Grosbein Cc: freebsd-fs , freebsd-stable@freebsd.org X-Rspamd-Queue-Id: 4904c65fqPz429R X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=distal.com header.s=s1 header.b=GpiM4Odc; dmarc=pass (policy=none) header.from=distal.com; spf=pass (mx1.freebsd.org: domain of bounces@em848.distal.com designates 167.89.106.69 as permitted sender) smtp.mailfrom=bounces@em848.distal.com X-Spamd-Result: default: False [-3.87 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_DKIM_ALLOW(-0.20)[distal.com:s=s1]; FROM_NEQ_ENVFROM(0.00)[cross@distal.com,bounces@em848.distal.com]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; R_SPF_ALLOW(-0.20)[+ip4:167.89.0.0/17:c]; MV_CASE(0.50)[]; MIME_GOOD(-0.10)[text/plain]; IP_SCORE(-1.67)[ipnet: 167.89.96.0/20(-4.61), asn: 11377(-3.67), country: US(-0.05)]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; TO_DN_SOME(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[distal.com:+]; DMARC_POLICY_ALLOW(-0.50)[distal.com,none]; RCVD_IN_DNSWL_NONE(0.00)[69.106.89.167.list.dnswl.org : 127.0.15.0]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FORGED_SENDER(0.30)[cross@distal.com,bounces@em848.distal.com]; RWL_MAILSPIKE_POSSIBLE(0.00)[69.106.89.167.rep.mailspike.net : 127.0.0.17]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:11377, ipnet:167.89.96.0/20, country:US]; MID_RHS_MATCH_FROM(0.00)[]; TAGGED_FROM(0.00)[13739864-b44a-freebsd-stable=freebsd.org,freebsd] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 11 Apr 2020 19:25:59 -0000 > On Apr 11, 2020, at 14:33, Eugene Grosbein wrote: >=20 > 12.04.2020 0:36, Chris Ross wrote: >=20 >> I have a FreeBSD 11.3-STABLE server that is my router, using a ZFS mirro= r (of two GPT disks) as it=E2=80=99s disk. It=E2=80=99s many years old, an= d has only been misbehaving like this for a day or so. I=E2=80=99m trying = to figure out what=E2=80=99s wrong. >>=20 >> [=E2=80=A6] >>=20 >> I _think_ this is a filesystem problem. It=E2=80=99s very hard to diagn= ose because logging in, and doing anything, takes many seconds per command.= zpool status shows my mirror as online, so I=E2=80=99m not sure where I s= hould check. >>=20 >> I=E2=80=99d appreciate any help! Thanks much=E2=80=A6 >=20 > First of all you should check if any of your ZFS pools is low on space. Wow. I=E2=80=99m so embarrassed that I didn=E2=80=99t notice that myself. = You mentioned it, and now I look back at df output and see that the filesy= stems are all very nearly full! It=E2=80=99s very slowly booting now, but assumedly after it comes online, = I=E2=80=99ll be able to rectify that situation and hopefully that will be t= he issue. Thanks, and sorry that I hadn=E2=80=99t seen that myself! - Chris