Date: Tue, 3 Dec 2002 20:33:02 -0500 (EST) From: Simon1 <simon1@server.simon1.net> To: Greg Goodman <admin@fastserve.net> Cc: <freebsd-questions@FreeBSD.ORG> Subject: Re: Virtual Private Servers/Jails Message-ID: <20021203193834.X471-100000@server.simon1.net> In-Reply-To: <1531119979925.20021203160149@fastserve.net>
next in thread | previous in thread | raw e-mail | index | archive | help
> Does anyone have any information on setting up virtual private > servers with Freebsd? There are a lot of people doing it out there > but I can't seem to find any documentation supporting it. FreeBSD doesn't offer true virtual machines. FreeBSD does come with jails. Which can provide similar functionality in many cases. Jails aren't nearly as polished as the virtual servers I've seen in the linux world. In most cases, the problems I've dealt with either aren't jail specific, or else are caused by the various utilities inability to adapt to jailed environments. I've run jails on production webservers, in most cases to consolidate older legacy systems, so I've got some experience with them. I also took on the fun task of porting a lot of the custom utilities we used over to them. Now, I haven't worked with jails since the end of July, but I doubt much has changed in the past 4 or 5 months. If I'm wrong, please correct me. I hammer quotas a lot, because of how important they and other limits are in the web hosting environments I was using them in. What I've found: 1) Connecting (aka telnet, ftp, ssh) from one jail to another or even to the physical host is supposed to work, but I was never able to make it happen. If anyone knows why, please chip in... The jails could access the internet, but not its host or sister jails. /stand/sysinstall also didn't like to download ports, which I'm guessing is for a related reason. 2) Quotas work, but its painful. The FreeBSD quota system/utilities hasn't really been modified for jails. It works, but it isn't fun getting it to. I've seen one, maybe two sites out there that actually give some good information on this. Essentially, FreeBSD tracks quotas by UID, host side. They're still effective in jails, but you have to make sure that you don't have two UID's on the same filesystem. Also, to make the jails work happily with quotas and, almost as important, quota utilities, you've got to hack things up a bit with the init scripts. Make sure the jails /etc/fstab actually does list the *real* devices if you want to do anything quota related from them. The jail may not need to know about the devices in /etc/fstab, but almost all of the quota utilities do. The perl quota module can't cope with the jail environment, edquota will work, but I think I had to use the /etc/fstab workaround to get it to function. To make the quotas work, and to make them editable /from within the jail/ you have to mount the root of the filesystem to a point *within the jail.* 3) Virtual Filesystems are a no-no. In theory, you'd just make a new vfs for each jail. That sidesteps the problem of duplicate quotas on a single fs and/or migrating. To migrate, you just move the file to a new host and issue the mount/jail start commands. That's the theory anyhow. Do they work, yes. Do they work well? No. I had so many problems with the things deadlocking. Once that happens, you're screwed. If you issue a reboot command FreeBSD hangs while waiting for the fs to unmount -- which it can't do sence a write/read is pending/hung. Very nasty when you're working on a server you don't have physical access too. Also, when mounting/unmounting these things, espically during the creation process, I found that unmounts wouldn't always work cleanly. Probably related to the above problems. You'd issue an unmount command, and it would work.. sort of. Except you couldn't remount the vfs to a new device, nor could you use the old vfs device either.. It just wouldn't show up in df anymore. A couple of points: DO use a separate filesystem for each jail DO NOT use the virtual filesystems for it. DO make sure to use separate UID's for /each/ jail if at all possible. I would use ranges for each jail (ie: 5,000-10,000 for the firt one, 10,001-12,000 for the second and so on.. ) DO mount proc for process commands to show up. The linux virtual servers I played with a bit were more like true virtual machines. Which is probably what you're after. The jail is more or less a locked down subset of the main, physical host. Aside from the quotas, it also wasn't possible to set jail-specific limits. IE: 10% of processor, 2gb total disk space for the jail, etc. Also, while someone with root in a jail can't trash the main system, they can still do a lot of damage. If you're thinking of using shared filesystems for multiple jails, and use quotas, you're going to have problems if two users share the same UID. --Quotas are checked against UID /on the physical host side/, and then checked by the filesystem (again, on the physical host side). Since root wasn't given out, except to people with root access on the physical machine, I didn't have problems with quota's for the root partition. I grouped 5-10 jails on a single disk and didn't worry about it. For the users, since their quota's *did* count, the jails /home was an actual filesystem mounted directly into the jail environment. (ie, if /disk2/jail was the jails root, I might have mounted another disk/partition on as /disk2/jail/home for the users. Since the root point of the filesystem was within the jails scope, quotas were accessable jail-side, as well as host-side.) 4) Needed utilities and commands. (Call it my wishlist) 1) A way to list jails. 2) A way to list processes BY jail, and a way to show (host side) which jail a process belongs to. 3) jail halt, jail restart commands to close out the jail, and possibly restart it. 4) The ability to limit resource usage /by jail/. 5) A more polished jail-side quota system. 6) More reliable vfs systems, and/or a way to forcefully dismount a vfs device. I believe the state my vfs filesystems kept getting hung in was the (to me) dreaded biowait state. I doubt this is possible, but it would be nice to have. Making the reboot command force completion would also be nice. I realize this is probably a lot more than you were looking for, but hopefully this will help others that may have questions. I was very happy with how wells the jails worked, but I would have preferred more of a virtual machine for what I was doing. If you need to set limits on a per-jail basis (other than by doing the filesystem limitations above) you'll want to look elswhere. If CPU/memory isn't as big an issue, and/or if you're not giving the jail root out, jails could provide added levels of security for programs and applications you're running. -Michael Wolfe President, Simon1, Ltd. Custom Computing Solutions To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-questions" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20021203193834.X471-100000>