From owner-freebsd-stable@FreeBSD.ORG Sat Jan 1 13:56:30 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1A0ED106564A; Sat, 1 Jan 2011 13:56:30 +0000 (UTC) (envelope-from 000.fbsd@quip.cz) Received: from elsa.codelab.cz (elsa.codelab.cz [94.124.105.4]) by mx1.freebsd.org (Postfix) with ESMTP id 8DBE58FC15; Sat, 1 Jan 2011 13:56:29 +0000 (UTC) Received: from elsa.codelab.cz (localhost.codelab.cz [127.0.0.1]) by elsa.codelab.cz (Postfix) with ESMTP id 5793519E030; Sat, 1 Jan 2011 14:56:27 +0100 (CET) Received: from [192.168.1.2] (ip-86-49-61-235.net.upcbroadband.cz [86.49.61.235]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by elsa.codelab.cz (Postfix) with ESMTPSA id 76C0519E02F; Sat, 1 Jan 2011 14:56:24 +0100 (CET) Message-ID: <4D1F3287.6080708@quip.cz> Date: Sat, 01 Jan 2011 14:56:23 +0100 From: Miroslav Lachman <000.fbsd@quip.cz> User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.9.1.16) Gecko/20101123 SeaMonkey/2.0.11 MIME-Version: 1.0 To: John Baldwin References: <4BCE4D0F.2020807@quip.cz> <201012130824.33968.jhb@freebsd.org> <4D15D8DD.9010900@quip.cz> <201012281123.53669.jhb@freebsd.org> In-Reply-To: <201012281123.53669.jhb@freebsd.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-stable@freebsd.org Subject: Re: /libexec/ld-elf.so.1: Cannot execute objects on / X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 01 Jan 2011 13:56:30 -0000 John Baldwin wrote: > On Saturday, December 25, 2010 6:43:25 am Miroslav Lachman wrote: >> John Baldwin wrote: >>> On Saturday, December 11, 2010 11:51:41 am Miroslav Lachman wrote: >>>> Miroslav Lachman wrote: >>>>> Garrett Cooper wrote: >>>>>> 2010/4/20 Miroslav Lachman<000.fbsd@quip.cz>: >>>>>>> I have large storage partition (/vol0) mounted as noexec and nosuid. >>>>>>> Then >>>>>>> one directory from this partition is mounted by nullfs as "exec and >>>>>>> suid" so >>>>>>> anything on it can be executed. >>>>>>> >>>>>>> The directory contains full installation of jail. Jail is running >>>>>>> fine, but >>>>>>> some ports (PHP for example) cannot be compiled inside the jail with >>>>>>> message: >>>>>>> >>>>>>> /libexec/ld-elf.so.1: Cannot execute objects on / >>>>>>> >>>>>>> The same apply to executing of apxs >>>>>>> >>>>>>> root@rainnew ~/# /usr/local/sbin/apxs -q MPM_NAME >>>>>>> /libexec/ld-elf.so.1: Cannot execute objects on / >>>>>>> >>>>>>> apxs:Error: Sorry, no shared object support for Apache. >>>>>>> apxs:Error: available under your platform. Make sure. >>>>>>> apxs:Error: the Apache module mod_so is compiled into. >>>>>>> apxs:Error: your server binary '/usr/local/sbin/httpd'.. >>>>>>> >>>>>>> (it should return "prefork") >>>>>>> >>>>>>> So I think there is some bug in checking the mountpoint options, >>>>>>> where the >>>>>>> check is made on "parent" of the nullfs instead of the nullfs target >>>>>>> mountpoint. >>>>>>> >>>>>>> It is on 6.4-RELEASE i386 GENERIC. I did not test it on another release. >>>>>>> >>>>>>> This is list of related mount points: >>>>>>> >>>>>>> /dev/mirror/gm0s2d on /vol0 (ufs, local, noexec, nosuid, soft-updates) >>>>>>> /vol0/jail/.nullfs/rain on /vol0/jail/rain_new (nullfs, local) >>>>>>> /usr/ports on /vol0/jail/rain_new/usr/ports (nullfs, local) >>>>>>> devfs on /vol0/jail/rain_new/dev (devfs, local) >>>>>>> >>>>>>> If I changed /vol0 options to (ufs, local, soft-updates) the above >>>>>>> error is >>>>>>> gone and apxs / compilation works fine. >>>>>>> >>>>>>> Can somebody look at this problem? >>>>>> >>>>>> Can you please provide output from ktrace / truss for the issue? >>>>> >>>>> I did >>>>> # ktrace /usr/local/sbin/apxs -q MPM_NAME >>>>> >>>>> The output is here http://freebsd.quip.cz/ld-elf/ktrace.out >>>>> >>>>> Let me know if you need something else. >>>>> >>>>> Thank you for your interest! >>>> >>>> The problem is still there in FreeBSD 8.1-RELEASE amd64 GENERIC (and in >>>> 7.x). >>>> >>>> Can somebody say if this is a bug or an expected "feature"? >>> >>> I think this is the expected behavior as nullfs is simply re-exposing /vol0 >>> and it shouldn't be able to create a more privileged mount than the underlying >>> mount I think (e.g. a read/write nullfs mount of a read-only filesystem would >>> not make the underlying files read/write). It can be used to provide less >>> privilege (e.g. a readonly nullfs mount of a read/write filesystem does not >>> allow writes via the nullfs layer). >>> >>> That said, I'm not sure exactly where the permission check is failing. >>> execve() only checks MNT_NOEXEC on the "upper" vnode's mountpoint (i.e. the >>> nullfs mountpoint) and the VOP_ACCESS(.., V_EXEC) check does not look at >>> MNT_NOEXEC either. >>> >>> I do think there might be bugs in that a nullfs mount that specifies noexec or >>> nosuid might not enforce the noexec or nosuid bits if the underlying mount >>> point does not have them set (from what I can see). >> >> Thank you for your explanation. Then it is strange, that there is bug, >> that allows execution on originally non executable mountpoint. >> It should be mentioned in the bugs section of the mount_nullfs man page. >> >> It would be useful, if 'mount' output shows inherited options for nullfs. >> >> If parent is: >> /dev/mirror/gm0s2d on /vol0 (ufs, local, noexec, nosuid, soft-updates) >> >> Then nullfs line will be: >> /vol0/jail/.nullfs/rain on /vol0/jail/rain_new (nullfs, local, noexec, >> nosuid) >> >> instead of just >> /vol0/jail/.nullfs/rain on /vol0/jail/rain_new (nullfs, local) >> >> >> Then I can understand what is expected behavior, but our current state >> is half working, if I can execute scripts and binaries, run jail on it, >> but can't execute "apxs -q MPM_NAME" and few others. > > Hmm, so I was a bit mistaken. The kernel is not failing to exec the binary. > Instead, rtld is reporting the error here: > > static Obj_Entry * > do_load_object(int fd, const char *name, char *path, struct stat *sbp, > int flags) > { > Obj_Entry *obj; > struct statfs fs; > > /* > * but first, make sure that environment variables haven't been > * used to circumvent the noexec flag on a filesystem. > */ > if (dangerous_ld_env) { > if (fstatfs(fd,&fs) != 0) { > _rtld_error("Cannot fstatfs \"%s\"", path); > return NULL; > } > if (fs.f_flags& MNT_NOEXEC) { > _rtld_error("Cannot execute objects on %s\n", fs.f_mntonname); > return NULL; > } > } > > I wonder if the fstatfs is falling down to the original mount rather than > being caught by nullfs. > > Hmm, nullfs' statfs method returns the flags for the underlying mount, not > the flags for the nullfs mount. This is possibly broken, but it is the > behavior nullfs has always had and the behavior it still has on other BSDs. I am sorry, I am not a programmer, so the code doesn't tell me much. Does it mean "we must leave it in current state" (for compatibility with other BSDs) or can it be fixed in the future? I can't tell if it will be better to disable all exec operations if parental mount is noexec, or to allow all exec operations. I just think that current state is broken if something can be executed ant something can't. And again, thank you for your time, explanation and interest in this problem! Miroslav Lachman