From owner-freebsd-fs@freebsd.org Sat Apr 11 21:24:35 2020 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 8565D27BBDE for ; Sat, 11 Apr 2020 21:24:35 +0000 (UTC) (envelope-from pen@lysator.liu.se) Received: from mail.lysator.liu.se (mail.lysator.liu.se [IPv6:2001:6b0:17:f0a0::3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4907Dy2ph0z48JP for ; Sat, 11 Apr 2020 21:24:34 +0000 (UTC) (envelope-from pen@lysator.liu.se) Received: from mail.lysator.liu.se (localhost [127.0.0.1]) by mail.lysator.liu.se (Postfix) with ESMTP id 2251D40004 for ; Sat, 11 Apr 2020 23:24:29 +0200 (CEST) Received: by mail.lysator.liu.se (Postfix, from userid 1004) id 107C440003; Sat, 11 Apr 2020 23:24:28 +0200 (CEST) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on bernadotte.lysator.liu.se X-Spam-Level: X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED,AWL autolearn=disabled version=3.4.2 X-Spam-Score: -1.0 Received: from [IPv6:2001:9b1:28ff:d901:f4ee:5e2b:488:2842] (unknown [IPv6:2001:9b1:28ff:d901:f4ee:5e2b:488:2842]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.lysator.liu.se (Postfix) with ESMTPSA id 7BA6940003; Sat, 11 Apr 2020 23:24:18 +0200 (CEST) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 13.0 \(3608.60.0.2.5\)) Subject: Re: ZFS server has gone crazy slow From: Peter Eriksson In-Reply-To: <7AA1EA07-6041-464A-A39A-158ACD1DC11C@distal.com> Date: Sat, 11 Apr 2020 23:24:18 +0200 Content-Transfer-Encoding: quoted-printable Message-Id: References: <2182C27C-A5D3-41BF-9CE9-7C6883E43074@distal.com> <20200411174831.GA54397@fuz.su> <6190573D-BCA7-44F9-86BD-0DCBB1F69D1D@distal.com> <6fd7a561-462e-242d-5057-51c52d716d68@wp.pl> <7AA1EA07-6041-464A-A39A-158ACD1DC11C@distal.com> To: freebsd-fs X-Mailer: Apple Mail (2.3608.60.0.2.5) X-Virus-Scanned: ClamAV using ClamSMTP X-Rspamd-Queue-Id: 4907Dy2ph0z48JP X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=pass (policy=none) header.from=liu.se; spf=pass (mx1.freebsd.org: domain of pen@lysator.liu.se designates 2001:6b0:17:f0a0::3 as permitted sender) smtp.mailfrom=pen@lysator.liu.se X-Spamd-Result: default: False [-3.29 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; R_SPF_ALLOW(-0.20)[+a:mail.lysator.liu.se]; MV_CASE(0.50)[]; TAGGED_RCPT(0.00)[freebsd]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; RCVD_COUNT_THREE(0.00)[4]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; IP_SCORE(-0.99)[ip: (-3.05), ipnet: 2001:6b0::/32(-1.03), asn: 1653(-0.85), country: EU(-0.01)]; DMARC_POLICY_ALLOW(-0.50)[liu.se,none]; RCVD_IN_DNSWL_NONE(0.00)[3.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.a.0.f.7.1.0.0.0.b.6.0.1.0.0.2.list.dnswl.org : 127.0.11.0]; RCVD_TLS_LAST(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:1653, ipnet:2001:6b0::/32, country:EU]; FREEMAIL_CC(0.00)[wp.pl]; MID_RHS_MATCH_FROM(0.00)[] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 11 Apr 2020 21:24:35 -0000 Yes, this is a know =E2=80=9Cfeature=E2=80=9D of ZFS. When filesystems = are nearly full (or near quota limits) it will tune down the write = transaction sizes which causes I/O (in order to make sure it doesn=E2=80=99= t write more than the quota, and try to opimize for storage instead of = latency/speed) to inflate and things basically grinds to a halt=E2=80=A6=20= Another fun thing that might happen is if you reboot your server and = happen to have a lot of queued up writes in the ZIL (for example if you = did a =E2=80=9Czfs destroy -d -r POOL@snapshots=E2=80=9D = (deferred(background) destroys of snapshots) and do a hard reboot while = it=E2=80=99s busy it will =E2=80=9Cwrite out=E2=80=9D those queued = transactions at filesystem mount time during the boot sequence - which = may take a long time (many hours if your have big pools and some of them = have filesystems (datasets) that are nearly full - ours pools are = 100-400TB). Taking snapshots will also grind to a crawl if you are near = quota limits on filesystems. Fun fun :-) (We have modified our rc.d startup scripts so we do the =E2=80=9Czfs = mount -a=E2=80=9D part in the background so the rest of the system = doesn=E2=80=99t have to wait for it (so we can get a ssh login prompt = and login and check the progress of the filesystem mounts. (And then = =E2=80=9Cnfs exports=E2=80=9D, nfsd and samba also waits in the = background for that to finish before starting up. A bit of a hack but it = works (a real parallel server manager with dependencies would have been = better :-) - Peter > On 11 Apr 2020, at 21:26, Chris Ross wrote: >=20 >=20 >=20 >> On Apr 11, 2020, at 14:10, Ireneusz Pluta/wp.pl wrote: >> W dniu 2020-04-11 o 20:02, Chris Ross pisze: >>> And, I=E2=80=99m not sure how to check SMART values. This is a Dell = server, and I=E2=80=99m 95% sure I have it=E2=80=99s RAID controller = (Dell PERC 6, mfi0) just set to provide JBOD for the two disks. Will = the smartmontools package be useful with disks on an mfi controller? >> Does the controller expose these jbods as /dev/da? devices to the = system ? >=20 > mfid devices. And, googling showed that I might want to use mfip to = allow me to access the drives for smartctl, and I=E2=80=99ll work on = that more in a bit.. >=20 > But, Eugene was kind enough to point out something that should=E2=80=99v= e been obvious, that I=E2=80=99d filled my ZFS volumes. I had a = snapshot of everything from a while ago, and one of the filesystems = contains large data files. Deleting the snapshot on that volume freed = things back to functional. >=20 > Thanks all. >=20 > - Chris > _______________________________________________ > freebsd-fs@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org"