From owner-freebsd-questions@FreeBSD.ORG Fri Sep 26 07:54:49 2008 Return-Path: Delivered-To: freebsd-questions@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4A538106568D for ; Fri, 26 Sep 2008 07:54:49 +0000 (UTC) (envelope-from m.seaman@infracaninophile.co.uk) Received: from smtp.infracaninophile.co.uk (gate6.infracaninophile.co.uk [IPv6:2001:8b0:151:1::1]) by mx1.freebsd.org (Postfix) with ESMTP id 9F70E8FC1B for ; Fri, 26 Sep 2008 07:54:48 +0000 (UTC) (envelope-from m.seaman@infracaninophile.co.uk) Received: from lack-of-gravitas.thebunker.net (gateway.ash.thebunker.net [213.129.64.4]) (authenticated bits=0) by smtp.infracaninophile.co.uk (8.14.3/8.14.3) with ESMTP id m8Q7sJ4f074275 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Fri, 26 Sep 2008 08:54:26 +0100 (BST) (envelope-from m.seaman@infracaninophile.co.uk) X-DKIM: Sendmail DKIM Filter v2.7.2 smtp.infracaninophile.co.uk m8Q7sJ4f074275 DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=infracaninophile.co.uk; s=200708; t=1222415667; bh=oqhCimvICR6ZSQ om/BvOVo79urzhuTCCJdx4esSGO7I=; h=Message-ID:Date:From:MIME-Version: To:CC:Subject:References:In-Reply-To:Content-Type: Content-Transfer-Encoding:Cc:Content-Type:Date:From:In-Reply-To: Message-ID:Mime-Version:References:To; z=Message-ID:=20<48DC952B.5 070308@infracaninophile.co.uk>|Date:=20Fri,=2026=20Sep=202008=2008: 54:19=20+0100|From:=20Matthew=20Seaman=20|Organization:=20Infracaninophile|User-Agent:=20Thunderbird= 202.0.0.16=20(X11/20080811)|MIME-Version:=201.0|To:=20freebsd-quest ions@freebsd.org|CC:=20David=20Polak=20|Subject: =20Re:=20mount_unionfs=20for=20jails|References:=20<005401c91f35$3c f09fa0$b6d1dee0$@com>=20<48DBF589.3030906@skoberne.net>|In-Reply-To :=20<48DBF589.3030906@skoberne.net>|X-Enigmail-Version:=200.95.6|Co ntent-Type:=20text/plain=3B=20charset=3DUTF-8=3B=20format=3Dflowed| Content-Transfer-Encoding:=207bit; b=HFbrrCCzRTioN58JVeTyNpzmz0tIVS kGivxTOLKADzKR4v0AVYEd4fpLwexshQh2pJjvNxlxG1+XNYnAcNlSKtVt1BIfhkWLm ku/7Lp9dJLd+gMJrYPoXnE+wkvrxpeRe7HFCuPe1Yt60GiF8IbiWO3YxLbOw6YuRDsb f/tuhNc= Message-ID: <48DC952B.5070308@infracaninophile.co.uk> Date: Fri, 26 Sep 2008 08:54:19 +0100 From: Matthew Seaman Organization: Infracaninophile User-Agent: Thunderbird 2.0.0.16 (X11/20080811) MIME-Version: 1.0 To: freebsd-questions@freebsd.org References: <005401c91f35$3cf09fa0$b6d1dee0$@com> <48DBF589.3030906@skoberne.net> In-Reply-To: <48DBF589.3030906@skoberne.net> X-Enigmail-Version: 0.95.6 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.0 (smtp.infracaninophile.co.uk [81.187.76.162]); Fri, 26 Sep 2008 08:54:27 +0100 (BST) X-Virus-Scanned: ClamAV 0.94/8341/Fri Sep 26 00:00:43 2008 on happy-idiot-talk.infracaninophile.co.uk X-Virus-Status: Clean X-Spam-Status: No, score=-2.6 required=5.0 tests=AWL,BAYES_00,DKIM_SIGNED, DKIM_VERIFIED,SPF_FAIL autolearn=no version=3.2.5 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on happy-idiot-talk.infracaninophile.co.uk Cc: David Polak Subject: Re: mount_unionfs for jails X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 26 Sep 2008 07:54:49 -0000 -----BEGIN PGP SIGNED MESSAGE----- Hash: RIPEMD160 Nejc S wrote: | Hello, | |> Here is what I am trying to do: |> |> mount_unionfs -o below /usr/jails/basejail /usr/jail/jail1 |> |> after I do that I edit /usr/jail/jail1/etc/rc.conf and add the appropriate |> entries to the host system rc.conf, but when I start the jail it starts |> using the settings from /usr/jails/basejail. | | I have the same setup and it works for me. | |> Is my mount_unionfs syntax wrong, is this a bug in unionfs (man page says |> unionfs is broken, but doesn't specify how its broken) or is this expected |> behavior? | | It shouldn't be wrong. I have this in my fstab: | | /jail/base /jail/spl/nejcspl unionfs rw,noatime,below 0 0 | | (noatime option is completely optional, of course.) | | But, if I were you, I would update the RELEASE to STABLE. This will also fix | some bugs in unionfs. However at least some other bugs still aren't fixed in 7-STABLE | to this day (most notably the socket bug, which prevents mysql from running in a jail | and writing socket file to /tmp/mysql.sock), so we had to MFC the patch from | HEAD manually. If you need the patch, let me know. | | However, I don't suggest running jails on top of unionfs where you need | decent stability (i.e. in production). I am writing thesis at the moment which | also covers this topic. We also stumbled upon these issues: | | - socket file bug, mentioned before, still present in 7-STABLE, no ideas | when it will be MFCed; | | - "mv" bug (see freebsd-fs archives for August 2008, me and my friend posted | a few posts there) which causes troubles when moving directories (files | would appear as gone and then reappear again) which exist or don't exist | on lower and upper levels; | | - another "mv" bug which I discovered yesterday and seems to be very strange | and hard to replay - I didn't even mess with the lower level, it seems that | also just the upper layer can behave strangely sometimes (erros like | "mv: invalid argument" when simply trying to move a big (>10 GB) directory - | the error was gone after I restarted the jail (i.e. also remounting the | unionfs); | | - strange behaviour of some applications (apache in my case) not "seeing" the | lower layer (/etc/hosts most notably) - we had to do "touch" (and then copy | to all jails on change) on files we _really_ need to be visible. However, | after we "fscked" our partition with unionfs directories, we weren't able | to reproduce this error; | | - UFS filesystem would get to inconsistent state (we don't know exactly when) | so some commands would behave strangely and fsck (see above) is needed in | single user mode; | | - _most notably_: there hasn't been a single reply to our unionfs related | problem reports and posts to freebsd-fs list. So I guess that people who | are in charge for unionfs in FreeBSD aren't really responsive and that | the future of unionfs in FreeBSD isn't really bright. It's a pity, though, | since this is a very useful feature, especially for jailed systems. However, | hope remains, that things will be fixed at least in 8.0 if not in 7.1. I think the problem is not so much lack of interest amongst available people, as lack of available people interested in work on that bit of filesystem code. Problems with unionfs and generally with VFS related stuff have been around for quite some time. | So, you can see that there are (still) many issues with unionfs on FreeBSD. | Please let me know if you are able to solve your problem. Or else we can make | this list a little longer. :) Having just gone through an attempt to set up a series of jails using unionfs layering, here's another annoyance. What I wanted to do was have a 'basejail' + unionfs overlay setup, but with various directories (/home, /usr/ports, /usr/src, and so forth) shared (ie. nullfs mounted) between all the jails and the base system. However an fstab.jails.jail0 like this: /jails/basejail /jails/jail0 unionfs rw,noatime,below,copymode=transparent,whiteout=whenneeded 0 0 /usr/ports /jails/jail0/usr/ports nulls rw 0 0 fails to work saying 'no such file or directory: /jails/jail0/usr/ports' irrespective of the existence of that directory in either layer of the unionfs. Seems it's impossible to have a nullfs mountpoint on top of a unionfs filesystem. At least, that's as far as I got when playing with this: press of time led me to implement a work around. If anyone knows how to get such a setup working as I originally intended I'd be very glad of a pointer to any documentation. Cheers, Matthew - -- Dr Matthew J Seaman MA, D.Phil. Flat 3 ~ 7 Priory Courtyard PGP: http://www.infracaninophile.co.uk/pgpkey Ramsgate ~ Kent, CT11 9PW, UK -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.9 (FreeBSD) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEAREDAAYFAkjclSoACgkQ3jDkPpsZ+VbP5QCfaGxXTnWIv2075a7yCseBgJQo 6IAAn3dCnTM9953sh9M54tMKA5w0p8Va =br6C -----END PGP SIGNATURE-----