From owner-freebsd-fs@freebsd.org Sat Apr 11 18:41:42 2020 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 01A322BFDBF for ; Sat, 11 Apr 2020 18:41:42 +0000 (UTC) (envelope-from bounces+13739864-7f7e-freebsd-fs=freebsd.org@em848.distal.com) Received: from xtrwsqdf.outbound-mail.sendgrid.net (xtrwsqdf.outbound-mail.sendgrid.net [167.89.100.223]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4903d0366mz406Q for ; Sat, 11 Apr 2020 18:41:40 +0000 (UTC) (envelope-from bounces+13739864-7f7e-freebsd-fs=freebsd.org@em848.distal.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=distal.com; h=content-type:mime-version:subject:from:in-reply-to: content-transfer-encoding:references:to:cc; s=s1; bh=QPl02ssWqCfpd8Fci0bpDopQtSTwy1rMFoAVCZLAw1k=; b=Fmhif1UhQG7mkaiMQ9DYF6m6ldlmqPOEwtbUNODd2IUDte05NDxyhRurfqZPyxK+/Ae0 sQmkUlDtTGyn93xDaOyWmhJ9cYu0jCHbBUMt8IpHGNFNLl6VeX9pW5hF4RzbqFlHQbkqvj 97qbkigtvbaSxqzomJFdmBgjTzkiSDWaA= Received: by filterdrecv-p3iad2-8ddf98858-f4h4l with SMTP id filterdrecv-p3iad2-8ddf98858-f4h4l-19-5E920F61-31 2020-04-11 18:41:37.505406232 +0000 UTC m=+1445650.824638391 Received: from mail.distal.com (unknown) by ismtpd0117p1mdw1.sendgrid.net (SG) with ESMTP id U7BAIbKwQVyx6GagocLS4w Sat, 11 Apr 2020 18:41:37.377 +0000 (UTC) Received: from magrathea.distal.com (magrathea.distal.com [2001:470:e24c:200:14a4:6888:ae02:b3bb]) by tristain.distal.com (OpenSMTPD) with ESMTPSA id 2e44148d (TLSv1.2:ECDHE-RSA-AES256-GCM-SHA384:256:NO); Sat, 11 Apr 2020 14:41:36 -0400 (EDT) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 13.4 \(3608.80.23.2.2\)) Subject: Re: ZFS server has gone crazy slow From: Chris Ross In-Reply-To: <6190573D-BCA7-44F9-86BD-0DCBB1F69D1D@distal.com> Date: Sat, 11 Apr 2020 18:41:37 +0000 (UTC) Content-Transfer-Encoding: quoted-printable Message-Id: References: <2182C27C-A5D3-41BF-9CE9-7C6883E43074@distal.com> <20200411174831.GA54397@fuz.su> <6190573D-BCA7-44F9-86BD-0DCBB1F69D1D@distal.com> X-Mailer: Apple Mail (2.3608.80.23.2.2) X-SG-EID: =?us-ascii?Q?gDj=2Futz1vvM0Gg5Dx3C984MHE5rSknXbUiMX+7YYKy8H35YTA3nJpmg1IQ5mw9?= =?us-ascii?Q?RS2bD=2FH4Uluvnmedtzs9ncXdF33uVZzQoN2Gb9j?= =?us-ascii?Q?+9vBOXXzj7CDtgHY4gWTcwh0HnVeoqIacxK4gmX?= =?us-ascii?Q?ZVwomrgxP7=2FJkBeY9CzGuVKQBme7KtznJ=2FcaBS7?= =?us-ascii?Q?j7CkPJeVNgUt36YMcIk3XbydZdAHZeDW6Ck3YZd?= =?us-ascii?Q?A0VxRFrl+j5aOCCIprfydXI1EXANsaTzMzuaBU8?= =?us-ascii?Q?63Ceu=2Fh6V7NBTN+usymvw=3D=3D?= To: Robert Clausecker Cc: freebsd-fs , Chris Ross X-Rspamd-Queue-Id: 4903d0366mz406Q X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=distal.com header.s=s1 header.b=Fmhif1Uh; dmarc=pass (policy=none) header.from=distal.com; spf=pass (mx1.freebsd.org: domain of bounces@em848.distal.com designates 167.89.100.223 as permitted sender) smtp.mailfrom=bounces@em848.distal.com X-Spamd-Result: default: False [-4.95 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_DKIM_ALLOW(-0.20)[distal.com:s=s1]; FROM_NEQ_ENVFROM(0.00)[cross@distal.com,bounces@em848.distal.com]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; R_SPF_ALLOW(-0.20)[+ip4:167.89.0.0/17]; MV_CASE(0.50)[]; TAGGED_RCPT(0.00)[freebsd]; MIME_GOOD(-0.10)[text/plain]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[distal.com:+]; DMARC_POLICY_ALLOW(-0.50)[distal.com,none]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; IP_SCORE(-2.75)[ip: (-5.41), ipnet: 167.89.96.0/20(-4.61), asn: 11377(-3.67), country: US(-0.05)]; FORGED_SENDER(0.30)[cross@distal.com,bounces@em848.distal.com]; RWL_MAILSPIKE_POSSIBLE(0.00)[223.100.89.167.rep.mailspike.net : 127.0.0.17]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:11377, ipnet:167.89.96.0/20, country:US]; MID_RHS_MATCH_FROM(0.00)[]; TAGGED_FROM(0.00)[13739864-7f7e-freebsd-fs=freebsd.org,freebsd] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 11 Apr 2020 18:41:42 -0000 > On Apr 11, 2020, at 14:02, Chris Ross wrote: > I _think_ this is a filesystem problem. [=E2=80=A6] Okay, this is pretty certain now. I=E2=80=99m trying to disable some daemo= ns (squid specifically), and to move a large log file out of the way, I did= : % mv access.log access.log.old && touch access.log ...as the squid user. For 10 minutes or more, that mv was in disk wait: squid 2365 0.0 0.0 4240 1892 3 D+ 14:22 0:00.02 mv access= .log access.log.old I didn=E2=80=99t truss that mv to know why, but that should never happen. = A rename in the same directory is about the most simple action to perform= =E2=80=A6 I=E2=80=99m going to reboot now that I=E2=80=99ve disabled squid and moved = it=E2=80=99s logs (it did eventually complete, somewhere in the 11-14 minut= e range). I asked the daemon to stop 30+ minutes ago, it still hasn=E2=80= =99t exited=E2=80=A6. I doubt that will fix the issues I=E2=80=99m having, = but it=E2=80=99s a start. - Chris