Date: Sat, 24 Aug 2013 15:35:01 -0500 (CDT) From: "Valeri Galtsev" <galtsev@kicp.uchicago.edu> To: "Konstantin Belousov" <kostikbel@gmail.com> Cc: freebsd-jail@freebsd.org Subject: Re: per user quotas inside jail? Message-ID: <55726.68.255.103.36.1377376501.squirrel@cosmo.uchicago.edu> In-Reply-To: <20130824150831.GO4972@kib.kiev.ua> References: <19176.128.135.70.2.1377267872.squirrel@cosmo.uchicago.edu> <20130823160549.GD4972@kib.kiev.ua> <17536.128.135.70.2.1377281124.squirrel@cosmo.uchicago.edu> <20130823182356.GH4972@kib.kiev.ua> <37112.128.135.70.2.1377283759.squirrel@cosmo.uchicago.edu> <20130824150831.GO4972@kib.kiev.ua>
next in thread | previous in thread | raw e-mail | index | archive | help
On Sat, August 24, 2013 10:08 am, Konstantin Belousov wrote: > On Fri, Aug 23, 2013 at 01:49:19PM -0500, Valeri Galtsev wrote: >> >> On Fri, August 23, 2013 1:23 pm, Konstantin Belousov wrote: >> > On Fri, Aug 23, 2013 at 01:05:24PM -0500, Valeri Galtsev wrote: >> >> On Fri, August 23, 2013 11:05 am, Konstantin Belousov wrote: >> >> > On Fri, Aug 23, 2013 at 09:24:32AM -0500, Valeri Galtsev wrote: >> >> >> Dear Experts, >> >> >> After searching the web, reading FreeBSD Docs, trying some hacks >> >> found on >> >> >> some discussion boards... I feel it is not easily possible. Yet, >> as >> >> always >> >> >> there may be some expert who knows how to do it: >> >> >> How can one have per user quotas inside jail? >> >> >> Basically, I would like to give users shell access to some server, >> >> but >> >> that I prefer to have in jail, where I will mount all filesystems >> they >> >> need access to... and the only question is: how do I restrict them so >> >> one >> >> >> (or few) user doesn't fill up the whole filesystem. My mind is not >> >> married >> >> >> to any particular filesystem, UFS2, XFS, ZFS... - the only thing I >> >> would >> >> >> stay away from is NFS exporting on host and then NFS mounting in >> jail >> >> (which may be easiest if not the only way quota wise). >> >> > >> >> > UFS quotas work regardless of jailed/non-jailed user. The only >> >> confusing >> >> > issue is that quotas are per host uid. In other words, if host and >> >> jail >> >> user, or two users from different jails has the same uid, you get one >> >> quota setting applied and accounted for them. >> >> > >> >> > Usual mitigation is to ensure that user uids are globally unique. >> >> > >> >> >> >> Thanks, Konstantin. >> >> >> >> Still it doesn't work for me. My system is: >> >> >> >> 9.1-RELEASE-p5 amd64 >> >> >> >> Kernel: the same as GENERIC, with one option added: >> >> >> >> options QUOTA # Add disk quota support >> >> >> >> filesystem with quota enabled is directly mounted (UFS; rw,userquota) >> >> into >> >> directory inside jail. User (with the same username and UID) exists >> on >> >> the >> >> host system and in jail. Quotas work on the host system. Quotas don't >> >> work >> >> inside jail, so this user can fill up the whole filesystem when >> logged >> >> into jail (jail accepts ssh connections with different hostname...) >> >> >> >> Apart from that I tried a hack which I lifted from someone's FreeBSD >> 7 >> >> hack (only the variable name changed since then), namely: >> >> >> >> in kernel, in: >> >> >> >> /usr/src/sys/kern/vfs_syscalls.c >> >> >> >> I kicked out two lines: >> >> >> >> if (!prison_allow(td->td_ucred, PR_ALLOW_QUOTAS)) >> >> return (EPERM); >> >> >> >> (which basically obliterate that if done from inside jail as far as I >> >> understand), >> >> >> >> rebuilt and installed this kernel; in file >> >> >> >> /etc/rc.d/quota >> >> >> >> removed line >> >> >> >> # KEYWORD: nojail >> >> >> >> Yet, I'm still where I was: quotas work outside jail, not inside >> jail... >> >> >> >> So, I'm at loss. I guess I will have to dive into zfs following Aaron >> >> Kaufman's suggestion... Sigh. >> > >> > UFS quotas work per mount. So if jail root is on a filesystem which >> > has no quotas configured, obviously the thing cannot work. >> > >> > You did not provided any details of your configuration, which makes >> > a diagnostic impossible. >> > >> >> Hi Konstantin, >> >> Thanks a lot for helping me! Sorry, my usual fault, not sufficient >> details... >> >> Jails are set up pretty much as in: >> >> http://www.freebsd.org/doc/handbook/jails-application.html >> >> (directory names and locations are slightly different). Someone >> mentioned, >> ezjail does virtually the same too - if that helps. >> >> In /jail/mroot there is the structure resembling real system (binaries, >> libraries,...) except for a few things that have to be writable inside >> jail; those are replaced with symlinks pointing to these inside >> subdirectory s: >> >> ls -l /jail/mroot >> total 48 >> drwxr-xr-x 2 root wheel 1024 Aug 19 13:02 bin >> drwxr-xr-x 7 root wheel 1024 Aug 19 13:03 boot >> dr-xr-xr-x 2 root wheel 512 Aug 19 13:02 dev >> lrwxr-xr-x 1 root wheel 5 Aug 19 13:11 etc -> s/etc >> lrwxr-xr-x 1 root wheel 6 Aug 19 13:11 home -> s/home >> drwxr-xr-x 3 root wheel 1536 Aug 19 13:03 lib >> drwxr-xr-x 3 root wheel 512 Aug 19 13:03 libexec >> drwxr-xr-x 2 root wheel 512 Aug 19 13:02 media >> drwxr-xr-x 2 root wheel 512 Aug 19 13:02 mnt >> dr-xr-xr-x 2 root wheel 512 Aug 19 13:02 proc >> drwxr-xr-x 2 root wheel 2560 Aug 19 13:03 rescue >> lrwxr-xr-x 1 root wheel 6 Aug 19 13:11 root -> s/root >> drwxr-xr-x 2 root wheel 512 Aug 19 13:11 s >> drwxr-xr-x 2 root wheel 2560 Aug 19 13:03 sbin >> lrwxr-xr-x 1 root wheel 11 Aug 19 13:02 sys -> usr/src/sys >> lrwxr-xr-x 1 root wheel 5 Aug 19 13:11 tmp -> s/tmp >> drwxr-xr-x 14 root wheel 512 Aug 19 13:11 usr >> lrwxr-xr-x 1 root wheel 5 Aug 19 13:11 var -> s/var >> >> particular jail lives in its root directory: >> >> /jail/shell >> >> /jail/mroot is nullfs readonly mounted onto /jail/shell, rw unique for >> each shell filesystem is mounted into /jail/shell/s (and populated with >> appropriate /etc, /var ....), filesystem that has to be with quotas is >> mounted (UFS; rw,userquota) into >> >> /jail/shell/s/home >> >> This last one is the one in question: quotas on this work when user will >> ssh to host system and will write to /jail/shell/s/home; quota does not >> work if user will ssh into jail (which is accessible from network with >> different hostname). When the user writes into /home in jail (into >> /s/home >> actually, symlink points there which on host system is our >> /jail/shell/s/home), the quotas do not work. >> >> I don't quite understand what quota on jail root filesystem (enabled or >> not enabled) has to do with quota on different filesystem that is >> mounted >> inside that filesystem. Outside jail / has no quotas, different >> filesystem >> mounted somewhere inside (/jail/shell/s/home or just /home or /var) with >> quotas and it does honor quotas. Am I missing something trivial or >> fundamental? >> >> Thanks again for helping me! What other details could help? > > I decided that I have no desire to try to understand all the layers of > indirections which are only relevant to you anyway. Instead, I demostrate > you what I mean by working quotas. Below is the transcript of the simple > test. > > sandy% mount -v /mnt > ~ > mount: /dev/ada1p4: Operation not permitted > /dev/ada1p4 on /mnt (ufs, local, with quotas, soft-updates, writes: sync 2 > async 37, reads: sync 7 async 0) > sandy% sudo repquota -uah | grep kostik > ~ > kostik -- 14G 0 0 - 461057 > 0 0 - > sandy% sudo jail -u kostik / test1 127.0.0.1 /bin/sh > ~ > $ dd if=/dev/zero bs=1m of=/mnt/1/dddd count=1024 > 1024+0 records in > 1024+0 records out > 1073741824 bytes transferred in 10.765265 secs (99741328 bytes/sec) > $ ^D% > sandy% sudo repquota -uah | grep kostik > ~ > kostik -- 15G 0 0 - 461058 > 0 0 - > > You could see that the accounted space and inodes are properly increased > after the dd. > > IMO, you should make sure that the users operate on the filesystem which > has quotas enabled. Or, you should provide a simple to reproduce test > case, among the lines of the script I pasted above, for me to recreate > the issue locally. > Thanks again for helping me! I guess, I understand now what the difference is. Apparently, you are much better expert, so correct me if I'm wrong. You run your jail with root of jail filesystems (/) the same as root filesystem of host (/). Therefore, inside your jail you have access to all host's /etc/fstab; /dev, ... I'll try to run jail the same way and will see if in that case quotas will work for me. If yes, then I at least I will know that my problem is not on the kernel level, but in the environment accessible inside jail. I have all jails set up so that one when in jail is not able to access filesystem outside jail's own root, which is something like /jail/{$jailname}... therefore host's /etc /dev are not visible for one inside jail; what they see inside jail as / is /jail/{$jailname} on host. Thanks again for all your efforts in helping me!! Sincerely yours, Valeri PS I like _very_much_ your username on that machine: kostik ;-) !! ++++++++++++++++++++++++++++++++++++++++ Valeri Galtsev Sr System Administrator Department of Astronomy and Astrophysics Kavli Institute for Cosmological Physics University of Chicago Phone: 773-702-4247 ++++++++++++++++++++++++++++++++++++++++
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?55726.68.255.103.36.1377376501.squirrel>