Date: Sat, 11 Apr 2020 23:24:18 +0200 From: Peter Eriksson <pen@lysator.liu.se> To: freebsd-fs <freebsd-fs@freebsd.org> Subject: Re: ZFS server has gone crazy slow Message-ID: <FE84C045-89B1-4772-AF1F-35F78B9877D8@lysator.liu.se> In-Reply-To: <7AA1EA07-6041-464A-A39A-158ACD1DC11C@distal.com> References: <2182C27C-A5D3-41BF-9CE9-7C6883E43074@distal.com> <20200411174831.GA54397@fuz.su> <6190573D-BCA7-44F9-86BD-0DCBB1F69D1D@distal.com> <6fd7a561-462e-242d-5057-51c52d716d68@wp.pl> <7AA1EA07-6041-464A-A39A-158ACD1DC11C@distal.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Yes, this is a know =E2=80=9Cfeature=E2=80=9D of ZFS. When filesystems = are nearly full (or near quota limits) it will tune down the write = transaction sizes which causes I/O (in order to make sure it doesn=E2=80=99= t write more than the quota, and try to opimize for storage instead of = latency/speed) to inflate and things basically grinds to a halt=E2=80=A6=20= Another fun thing that might happen is if you reboot your server and = happen to have a lot of queued up writes in the ZIL (for example if you = did a =E2=80=9Czfs destroy -d -r POOL@snapshots=E2=80=9D = (deferred(background) destroys of snapshots) and do a hard reboot while = it=E2=80=99s busy it will =E2=80=9Cwrite out=E2=80=9D those queued = transactions at filesystem mount time during the boot sequence - which = may take a long time (many hours if your have big pools and some of them = have filesystems (datasets) that are nearly full - ours pools are = 100-400TB). Taking snapshots will also grind to a crawl if you are near = quota limits on filesystems. Fun fun :-) (We have modified our rc.d startup scripts so we do the =E2=80=9Czfs = mount -a=E2=80=9D part in the background so the rest of the system = doesn=E2=80=99t have to wait for it (so we can get a ssh login prompt = and login and check the progress of the filesystem mounts. (And then = =E2=80=9Cnfs exports=E2=80=9D, nfsd and samba also waits in the = background for that to finish before starting up. A bit of a hack but it = works (a real parallel server manager with dependencies would have been = better :-) - Peter > On 11 Apr 2020, at 21:26, Chris Ross <cross+freebsd@distal.com> wrote: >=20 >=20 >=20 >> On Apr 11, 2020, at 14:10, Ireneusz Pluta/wp.pl <ipluta@wp.pl> wrote: >> W dniu 2020-04-11 o 20:02, Chris Ross pisze: >>> And, I=E2=80=99m not sure how to check SMART values. This is a Dell = server, and I=E2=80=99m 95% sure I have it=E2=80=99s RAID controller = (Dell PERC 6, mfi0) just set to provide JBOD for the two disks. Will = the smartmontools package be useful with disks on an mfi controller? >> Does the controller expose these jbods as /dev/da? devices to the = system ? >=20 > mfid devices. And, googling showed that I might want to use mfip to = allow me to access the drives for smartctl, and I=E2=80=99ll work on = that more in a bit.. >=20 > But, Eugene was kind enough to point out something that should=E2=80=99v= e been obvious, that I=E2=80=99d filled my ZFS volumes. I had a = snapshot of everything from a while ago, and one of the filesystems = contains large data files. Deleting the snapshot on that volume freed = things back to functional. >=20 > Thanks all. >=20 > - Chris > _______________________________________________ > freebsd-fs@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org"
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?FE84C045-89B1-4772-AF1F-35F78B9877D8>