Date: Fri, 8 Nov 2019 08:03:52 +0100 From: Peter Eriksson <pen@lysator.liu.se> To: Chris Watson <bsdunix44@gmail.com>, Jan Behrens <jbe-mlist@magnetkern.de>, =?utf-8?Q?Karli_Sj=C3=B6berg_via_freebsd-fs?= <freebsd-fs@freebsd.org> Subject: Re: ZFS snapdir readability (Crosspost) Message-ID: <C5B9E02E-BC08-4077-81CD-5CABB67F422B@lysator.liu.se> In-Reply-To: <65AE896D-A32E-451A-B9D0-EC40D438BB03@gmail.com> References: <FBB088B0-CE5C-45DC-8F2F-0D0AA2703846@lysator.liu.se> <65AE896D-A32E-451A-B9D0-EC40D438BB03@gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Yes, and no. We use ‘refquotas’ on the user filesystems to prevent a user from using up all the free space in the zpool. However, what happens is that ZFS will decrease the “transaction size” of writes when a filesystem is nearly at quota in order to prevent a transaction to write more than the assigned quota (since it might write more due to compression/snapshots etc (or whatever)). Now this is (apparently) by design and it works. The effect is that a user trying to write to a nearly full (at quota) filesystem will “see” an extremely slow fileserver (from MB/s to KB/s (or B/s if _really_ full) which makes for interesting bug reports and problem diagnosing (since other users at the same time will see no such slow-downs). _However_ - other effects are that creating snapshots, or destroying snapshots _also_ can take longer than usual. Or if you delete a lot of snapshots at the same time as a user has deleted a lot of files and you then reboot the server - then when it starts up again and attempts to mount all filesystems you will notice that it might take “forever” and it might look like “zfs mount -a” is “stuck”. Luckily for us when this happened we had previously modified the system startup scripts so all zfs mounting is done in the background and in parallell - so we got a login prompt and could login and run some commands (normally all filesystems are mounted first before allowing system logins….) So we logged in and noticed that it was doing a lot of writes (zpool iostat) and was “stuck” at attempting to mount a single specific filesystem. A filesystem that happened to have 0 bytes free refquota. When I added some 10G more ‘refquota’ to that filesystem things speeded up a lot :-). (First time this happened we let the server finish mounting by itself - which took about 2.5 days…) - Peter > On 8 Nov 2019, at 01:31, Chris Watson <bsdunix44@gmail.com> wrote: > > Peter, on your last point about 100% utilization, don’t you use quotas/user quotas to prevent that? > > Chris > > Sent from my iPhone > >> On Nov 7, 2019, at 4:06 PM, Peter Eriksson <pen@lysator.liu.se> wrote: >> >> The “easy” solution is to give each user (or group / project) their own ZFS filesystem. Then the “.zfs” directory would be inside the users own $HOME and you can set $HOME to 0700…. >> >> That is what we are doing. Granted it generates a “few” filesystems (like some 20000 per server (we have around 120k users), and then add hourly snapshots to each as “icing” on the cake). Mounting all those takes a bit of time - but luckily with the latest FreeBSD release things are much faster these days :-) >> >> There are some other issues with that - like 100% full filesystems causing severe system slowdown during writes… So you really wanna have some monitoring system that warns for that. >> >> - Peter >> >> >>> >>> I recently noticed that all ZFS filesystems in FreeBSD allow access to >>> the .zfs directory (snapdir) for all users of the system. It is >>> possible to hide that directory using the snapdir option: >> >> >> _______________________________________________ >> freebsd-fs@freebsd.org mailing list >> https://lists.freebsd.org/mailman/listinfo/freebsd-fs >> To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org"
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?C5B9E02E-BC08-4077-81CD-5CABB67F422B>