From owner-freebsd-stable@freebsd.org Sun Apr 12 05:53:45 2020 Return-Path: Delivered-To: freebsd-stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id EC6962B3374 for ; Sun, 12 Apr 2020 05:53:45 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from hz.grosbein.net (hz.grosbein.net [IPv6:2a01:4f8:c2c:26d8::2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "hz.grosbein.net", Issuer "hz.grosbein.net" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 490LXS48W8z4fhK for ; Sun, 12 Apr 2020 05:53:43 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from eg.sd.rdtc.ru (eg.sd.rdtc.ru [IPv6:2a03:3100:c:13:0:0:0:5]) by hz.grosbein.net (8.15.2/8.15.2) with ESMTPS id 03C5mI7F046514 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Sun, 12 Apr 2020 05:48:21 GMT (envelope-from eugen@grosbein.net) X-Envelope-From: eugen@grosbein.net X-Envelope-To: cross+freebsd@distal.com Received: from [10.58.0.10] (dadvw [10.58.0.10]) by eg.sd.rdtc.ru (8.15.2/8.15.2) with ESMTPS id 03C5mCCL000847 (version=TLSv1.2 cipher=DHE-RSA-AES128-SHA bits=128 verify=NOT); Sun, 12 Apr 2020 12:48:12 +0700 (+07) (envelope-from eugen@grosbein.net) Subject: Re: ZFS server has gone crazy slow To: Chris Ross References: <2182C27C-A5D3-41BF-9CE9-7C6883E43074@distal.com> <68328a40-0e3d-f9cf-510b-9cbfd7cb8acd@grosbein.net> <654D00F8-DBEC-49BD-B871-7EB830F49D50@distal.com> Cc: freebsd-stable@freebsd.org From: Eugene Grosbein Message-ID: <89932f19-6aac-5b45-646a-04ceecaaa19d@grosbein.net> Date: Sun, 12 Apr 2020 12:48:06 +0700 User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.8.0 MIME-Version: 1.0 In-Reply-To: <654D00F8-DBEC-49BD-B871-7EB830F49D50@distal.com> Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Spam-Status: No, score=0.3 required=5.0 tests=BAYES_00,LOCAL_FROM, SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.2 X-Spam-Report: * -2.3 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] * 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record * -0.0 SPF_PASS SPF: sender matches SPF record * 2.6 LOCAL_FROM From my domains X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on hz.grosbein.net X-Rspamd-Queue-Id: 490LXS48W8z4fhK X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=permerror (mx1.freebsd.org: domain of eugen@grosbein.net uses mechanism not recognized by this client) smtp.mailfrom=eugen@grosbein.net X-Spamd-Result: default: False [-3.99 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; TAGGED_RCPT(0.00)[freebsd]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[grosbein.net]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; R_SPF_PERMFAIL(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; IP_SCORE(-1.89)[ip: (-5.22), ipnet: 2a01:4f8::/29(-2.61), asn: 24940(-1.58), country: DE(-0.02)]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:24940, ipnet:2a01:4f8::/29, country:DE]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_TLS_ALL(0.00)[] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 12 Apr 2020 05:53:46 -0000 12.04.2020 2:25, Chris Ross пишет: > >> On Apr 11, 2020, at 14:33, Eugene Grosbein wrote: >> >> 12.04.2020 0:36, Chris Ross wrote: >> >>> I have a FreeBSD 11.3-STABLE server that is my router, using a ZFS mirror (of two GPT disks) as it’s disk. It’s many years old, and has only been misbehaving like this for a day or so. I’m trying to figure out what’s wrong. >>> >>> […] >>> >>> I _think_ this is a filesystem problem. It’s very hard to diagnose because logging in, and doing anything, takes many seconds per command. zpool status shows my mirror as online, so I’m not sure where I should check. >>> >>> I’d appreciate any help! Thanks much… >> >> First of all you should check if any of your ZFS pools is low on space. > > Wow. I’m so embarrassed that I didn’t notice that myself. You mentioned it, and now I look back at df output and see that the filesystems are all very nearly full! > > It’s very slowly booting now, but assumedly after it comes online, I’ll be able to rectify that situation and hopefully that will be the issue. Thanks, and sorry that I hadn’t seen that myself! There is very simple way to prevent such problem, use: zfs set reservation=1G for single "root" file system of the pool. This way ZFS won't allow applications to fill the pool to the point it starts crawling. Instead, writing applications would obtain ENOSPC error when pool's free space hits the limit.