From owner-freebsd-stable@freebsd.org Sat Apr 11 18:39:49 2020 Return-Path: Delivered-To: freebsd-stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 298672BF963; Sat, 11 Apr 2020 18:39:49 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from hz.grosbein.net (hz.grosbein.net [IPv6:2a01:4f8:c2c:26d8::2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "hz.grosbein.net", Issuer "hz.grosbein.net" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4903Zr1X1Bz3ymL; Sat, 11 Apr 2020 18:39:47 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from eg.sd.rdtc.ru (eg.sd.rdtc.ru [IPv6:2a03:3100:c:13:0:0:0:5]) by hz.grosbein.net (8.15.2/8.15.2) with ESMTPS id 03BIX8Nt031880 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Sat, 11 Apr 2020 18:33:08 GMT (envelope-from eugen@grosbein.net) X-Envelope-From: eugen@grosbein.net X-Envelope-To: cross+freebsd@distal.com Received: from [10.58.0.10] (dadv@dadvw [10.58.0.10]) by eg.sd.rdtc.ru (8.15.2/8.15.2) with ESMTPS id 03BIX5at095068 (version=TLSv1.2 cipher=DHE-RSA-AES128-SHA bits=128 verify=NOT); Sun, 12 Apr 2020 01:33:05 +0700 (+07) (envelope-from eugen@grosbein.net) Subject: Re: ZFS server has gone crazy slow To: Chris Ross , freebsd-fs , freebsd-stable@freebsd.org References: <2182C27C-A5D3-41BF-9CE9-7C6883E43074@distal.com> From: Eugene Grosbein Message-ID: <68328a40-0e3d-f9cf-510b-9cbfd7cb8acd@grosbein.net> Date: Sun, 12 Apr 2020 01:33:04 +0700 User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.8.0 MIME-Version: 1.0 In-Reply-To: <2182C27C-A5D3-41BF-9CE9-7C6883E43074@distal.com> Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Spam-Status: No, score=0.3 required=5.0 tests=BAYES_00,LOCAL_FROM, SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.2 X-Spam-Report: * -2.3 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] * -0.0 SPF_PASS SPF: sender matches SPF record * 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record * 2.6 LOCAL_FROM From my domains X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on hz.grosbein.net X-Rspamd-Queue-Id: 4903Zr1X1Bz3ymL X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=permerror (mx1.freebsd.org: domain of eugen@grosbein.net uses mechanism not recognized by this client) smtp.mailfrom=eugen@grosbein.net X-Spamd-Result: default: False [-3.98 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; TO_DN_SOME(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; TAGGED_RCPT(0.00)[freebsd]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[grosbein.net]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; R_SPF_PERMFAIL(0.00)[]; IP_SCORE(-1.89)[ip: (-5.21), ipnet: 2a01:4f8::/29(-2.61), asn: 24940(-1.58), country: DE(-0.02)]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:24940, ipnet:2a01:4f8::/29, country:DE]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_TLS_ALL(0.00)[] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 11 Apr 2020 18:39:49 -0000 12.04.2020 0:36, Chris Ross wrote: > I have a FreeBSD 11.3-STABLE server that is my router, using a ZFS mirror (of two GPT disks) as it’s disk. It’s many years old, and has only been misbehaving like this for a day or so. I’m trying to figure out what’s wrong. > > I confirmed that internet connectivity isn’t the problem, and a reboot didn’t fix it. (The reboot took 10-15 minutes to finish going multi-user, starting daemons, due to the underlying problem described below.) > > Truss’ing a very basic command (date), I can see that close() and exit() calls are taking 1-2 seconds. All of the files being opened are on ZFS, but I don’t know if that’s for sure related. Similarly, using shell builtin “echo foo” always is immediate, but “/bin/echo” sometimes works quickly, but sometimes the close() on /var/run/ld-elf.so.hints takes 3-5 seconds. > > I _think_ this is a filesystem problem. It’s very hard to diagnose because logging in, and doing anything, takes many seconds per command. zpool status shows my mirror as online, so I’m not sure where I should check. > > I’d appreciate any help! Thanks much… First of all you should check if any of your ZFS pools is low on space.