From owner-freebsd-jail@FreeBSD.ORG Sun Aug 7 10:38:19 2011 Return-Path: Delivered-To: freebsd-jail@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 55027106567B; Sun, 7 Aug 2011 10:38:19 +0000 (UTC) (envelope-from simon@nitro.dk) Received: from emx.nitro.dk (emx.nitro.dk [IPv6:2a01:4f8:120:7384::102]) by mx1.freebsd.org (Postfix) with ESMTP id 77F668FC9F; Sun, 7 Aug 2011 10:37:54 +0000 (UTC) Received: from mailscan.leto.nitro.dk (mailscan.leto.nitro.dk [127.0.1.4]) by emx.nitro.dk (Postfix) with ESMTP id A1449BCB2A; Sun, 7 Aug 2011 10:37:53 +0000 (UTC) Received: from emx.nitro.dk ([127.0.1.2]) by mailscan.leto.nitro.dk (mailscan.leto.nitro.dk [127.0.1.4]) (amavisd-new, port 10024) with LMTP id yl3IAzCrpp+K; Sun, 7 Aug 2011 10:37:51 +0000 (UTC) Received: from [192.168.4.21] (4304ds2-vlb.1.fullrate.dk [90.184.171.166]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) by emx.nitro.dk (Postfix) with ESMTPSA id D928DBCB20; Sun, 7 Aug 2011 10:37:50 +0000 (UTC) Mime-Version: 1.0 (Apple Message framework v1244.3) Content-Type: text/plain; charset=iso-8859-1 From: "Simon L. B. Nielsen" In-Reply-To: <4E25BB7C.4090106@FreeBSD.org> Date: Sun, 7 Aug 2011 12:37:49 +0200 Content-Transfer-Encoding: quoted-printable Message-Id: <005878F6-3CF5-482E-98E8-E5E4B8CA6C99@nitro.dk> References: <4E114EA9.4000605@FreeBSD.org> <20110718190839.GA81421@psconsult.nl> <4E25BB7C.4090106@FreeBSD.org> To: Jamie Gritton X-Mailer: Apple Mail (2.1244.3) Cc: freebsd-jail@FreeBSD.org, freebsd-arch@FreeBSD.org Subject: Re: New jail(8) with configuration files, not yet in head X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 07 Aug 2011 10:38:20 -0000 On 19 Jul 2011, at 19:14, Jamie Gritton wrote: > On 07/18/11 13:08, Paul Schenkeveld wrote: >=20 >> Although I really like this new functionality, there is one issue = that >> I am concerned about. Should all this functionality be integrated = into >> the jail(8) command? >=20 > This project came from a desire to improve the jail startup procedure = in rc.d/jail, which remains stuck handling the old fixed-parameter = jails. Rather that continue to extend an already unwieldy number of = rc.conf shell variables, I opted to add a configuration file like other = subsystems use (e.g. apmd, devd). The new jail pseudo-parameters added = to the config file exist mostly to match the existing rc.d/jail = functionality - the mount.* and exec.* parameters are direct analogs to = rc.conf shell variables. Some other parameters match the command-line = options of the existing jail(8). [This is less a mail to Jamie and more me getting around to publicly = supporting they way it's done] A thing to note is also that when starting a jail you have to be really = careful to do all of the related operations in the right order and in a = safe manner. E.g. mounting file systems are only safe in some = circumstances (ref: symlink attacks) so that's one reason I think the = new approach is the right one. Also try reading the current rc.d/jail = code for checking for those symlinks etc... not pretty. There are also some other quirks which means a slightly more = comprehensive program is better. E.g. current rc.d jails have a bug = where they can actually fill /tmp if they produce a lot of console = output due to redirection to temp file (this is rarely a real problem so = I never gotten around to trying to fix it). Bloat is of course a concern, but I don't think that risk outweigh the = benefits of Jamie's new work. There is still room and need for a wrapper management framework (ezjail = or something close to it) which handles the actual creation, update etc. = which makes sense as a separate utility not part of jail(8). ezjail = already uses rc.d/jail heavily so I think it can nicely integrate with = the new jail(8). > I wouldn't want to do away with a config file, as that's a much = cleaner way to define multiple jails than depending on the rc system or = requiring a "roll your own" approach that is currently the only way to = use the newer features. Just reading /etc/rc.d/jail is IMO good proof of this... > It's clear now that this won't be happening in 9.0. So none of this = is in danger of getting pushed through in a hurry. I really hope that this can go into head shortly after the branch so it = can hopefully make it into 9.1. --=20 Simon L. B. Nielsen From owner-freebsd-jail@FreeBSD.ORG Sun Aug 7 19:05:26 2011 Return-Path: Delivered-To: freebsd-jail@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 25FE41065670; Sun, 7 Aug 2011 19:05:26 +0000 (UTC) (envelope-from joris.dedieu@gmail.com) Received: from mail-iy0-f172.google.com (mail-iy0-f172.google.com [209.85.210.172]) by mx1.freebsd.org (Postfix) with ESMTP id CC4988FC0A; Sun, 7 Aug 2011 19:05:25 +0000 (UTC) Received: by iye7 with SMTP id 7so7657795iye.17 for ; Sun, 07 Aug 2011 12:05:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; bh=+eH8/nP8YMduuzAdUMe185OCjTBX1rtFjKhu4Q1m54w=; b=IeGP6+qVqDXOkUhGEDlP6mqHoO+dLfbmZyEeIE/SImVF8hkt+h8BRp7xtFFFpxFzmT nJTr32t5MxM41Wa4he30EBpRBn0kDyWdXk66oKwr6kQaNR4D1BLKHXxbu/cMrY2fHoSF Vtad1xvFV3VSi6Jta7oqBP2qk/EhcSKG8fdBU= MIME-Version: 1.0 Received: by 10.231.91.69 with SMTP id l5mr3996155ibm.47.1312742529326; Sun, 07 Aug 2011 11:42:09 -0700 (PDT) Received: by 10.231.13.204 with HTTP; Sun, 7 Aug 2011 11:42:09 -0700 (PDT) In-Reply-To: <005878F6-3CF5-482E-98E8-E5E4B8CA6C99@nitro.dk> References: <4E114EA9.4000605@FreeBSD.org> <20110718190839.GA81421@psconsult.nl> <4E25BB7C.4090106@FreeBSD.org> <005878F6-3CF5-482E-98E8-E5E4B8CA6C99@nitro.dk> Date: Sun, 7 Aug 2011 20:42:09 +0200 Message-ID: From: joris dedieu To: "Simon L. B. Nielsen" Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: freebsd-jail@freebsd.org, Jamie Gritton , freebsd-arch@freebsd.org Subject: Re: New jail(8) with configuration files, not yet in head X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 07 Aug 2011 19:05:26 -0000 2011/8/7 Simon L. B. Nielsen : > > On 19 Jul 2011, at 19:14, Jamie Gritton wrote: > >> On 07/18/11 13:08, Paul Schenkeveld wrote: >> >>> Although I really like this new functionality, there is one issue that >>> I am concerned about. =A0Should all this functionality be integrated in= to >>> the jail(8) command? >> >> This project came from a desire to improve the jail startup procedure in= rc.d/jail, which remains stuck handling the old fixed-parameter jails. Rat= her that continue to extend an already unwieldy number of rc.conf shell var= iables, I opted to add a configuration file like other subsystems use (e.g.= apmd, devd). =A0The new jail pseudo-parameters added to the config file ex= ist mostly to match the existing rc.d/jail functionality - the mount.* and = exec.* parameters are direct analogs to rc.conf shell variables. =A0Some ot= her parameters match the command-line options of the existing jail(8). > > [This is less a mail to Jamie and more me getting around to publicly supp= orting they way it's done] > > A thing to note is also that when starting a jail you have to be really c= areful to do all of the related operations in the right order and in a safe= manner. E.g. mounting file systems are only safe in some circumstances (re= f: symlink attacks) so that's one reason I think the new approach is the ri= ght one. Also try reading the current rc.d/jail code for checking for those= symlinks etc... not pretty. > > There are also some other quirks which means a slightly more comprehensiv= e program is better. =A0E.g. current rc.d jails have a bug where they can a= ctually fill /tmp if they produce a lot of console output due to redirectio= n to temp file (this is rarely a real problem so I never gotten around to t= rying to fix it). > > Bloat is of course a concern, but I don't think that risk outweigh the be= nefits of Jamie's new work. > > There is still room and need for a wrapper management framework (ezjail o= r something close to it) which handles the actual creation, update etc. whi= ch makes sense as a separate utility not part of jail(8). ezjail already us= es rc.d/jail heavily so I think it can nicely integrate with the new jail(8= ). > >> I wouldn't want to do away with a config file, as that's a much cleaner = way to define multiple jails than depending on the rc system or requiring a= "roll your own" approach that is currently the only way to use the newer f= eatures. > > Just reading /etc/rc.d/jail is IMO good proof of this... > >> It's clear now that this won't be happening in 9.0. =A0So none of this i= s in danger of getting pushed through in a hurry. > > I really hope that this can go into head shortly after the branch so it c= an hopefully make it into 9.1. IMHO It's a need. Jails v2 effort began in 7.2 with multiple ip support. /etc/rc.d/jail is clearly unpatchable (see comments in conf/142972). It's now reasonable to think that a way to cleanly start jails v2 at boot time can be hoped form OS primitives. Joris > > -- > Simon L. B. Nielsen > > _______________________________________________ > freebsd-arch@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-arch > To unsubscribe, send any mail to "freebsd-arch-unsubscribe@freebsd.org" > From owner-freebsd-jail@FreeBSD.ORG Mon Aug 8 11:07:07 2011 Return-Path: Delivered-To: freebsd-jail@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id EF867106566C for ; Mon, 8 Aug 2011 11:07:07 +0000 (UTC) (envelope-from owner-bugmaster@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id DDE8D8FC0A for ; Mon, 8 Aug 2011 11:07:07 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.4/8.14.4) with ESMTP id p78B778o078589 for ; Mon, 8 Aug 2011 11:07:07 GMT (envelope-from owner-bugmaster@FreeBSD.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.4/8.14.4/Submit) id p78B770S078587 for freebsd-jail@FreeBSD.org; Mon, 8 Aug 2011 11:07:07 GMT (envelope-from owner-bugmaster@FreeBSD.org) Date: Mon, 8 Aug 2011 11:07:07 GMT Message-Id: <201108081107.p78B770S078587@freefall.freebsd.org> X-Authentication-Warning: freefall.freebsd.org: gnats set sender to owner-bugmaster@FreeBSD.org using -f From: FreeBSD bugmaster To: freebsd-jail@FreeBSD.org Cc: Subject: Current problem reports assigned to freebsd-jail@FreeBSD.org X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 08 Aug 2011 11:07:08 -0000 Note: to view an individual PR, use: http://www.freebsd.org/cgi/query-pr.cgi?pr=(number). The following is a listing of current problems submitted by FreeBSD users. These represent problem reports covering all versions including experimental development code and obsolete releases. S Tracker Resp. Description -------------------------------------------------------------------------------- o kern/156111 jail [jail] procstat -b not supported in jail o misc/155765 jail [patch] `buildworld' does not honors WITHOUT_JAIL o conf/154246 jail [jail] [patch] Bad symlink created if devfs mount poin o conf/149050 jail [jail] rcorder ``nojail'' too coarse for Jail+VNET s conf/142972 jail [jail] [patch] Support JAILv2 and vnet in rc.d/jail o conf/141317 jail [patch] uncorrect jail stop in /etc/rc.d/jail o kern/133265 jail [jail] is there a solution how to run nfs client in ja o kern/119842 jail [smbfs] [jail] "Bad address" with smbfs inside a jail o bin/99566 jail [jail] [patch] fstat(1) according to specified jid o bin/32828 jail [jail] w(1) incorrectly handles stale utmp slots with 10 problems total. From owner-freebsd-jail@FreeBSD.ORG Tue Aug 9 12:45:50 2011 Return-Path: Delivered-To: freebsd-jail@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 801D21065673 for ; Tue, 9 Aug 2011 12:45:50 +0000 (UTC) (envelope-from freebsd@psconsult.nl) Received: from mx1.psconsult.nl (mx1.psconsult.nl [80.89.238.138]) by mx1.freebsd.org (Postfix) with ESMTP id DBEFC8FC13 for ; Tue, 9 Aug 2011 12:45:49 +0000 (UTC) Received: from mx1.psconsult.nl ([80.89.238.138]) by mx1.psconsult.nl (8.14.4/8.14.4) with ESMTP id p79Cjh39023102 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Tue, 9 Aug 2011 14:45:48 +0200 (CEST) (envelope-from freebsd@psconsult.nl) Received: (from paul@localhost) by mx1.psconsult.nl (8.14.4/8.14.4/Submit) id p79Cjhs3023101 for freebsd-jail@freebsd.org; Tue, 9 Aug 2011 14:45:43 +0200 (CEST) (envelope-from freebsd@psconsult.nl) X-Authentication-Warning: mx1.psconsult.nl: paul set sender to freebsd@psconsult.nl using -f Date: Tue, 9 Aug 2011 14:45:43 +0200 From: Paul Schenkeveld To: freebsd-jail@freebsd.org Message-ID: <20110809124543.GA22077@psconsult.nl> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.5.21 (2010-09-15) Subject: Jexec and access to tty X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 09 Aug 2011 12:45:50 -0000 Hi, There have been several threads about this issue, some people have come up with work arounds but I think that the issue is more fundamental, that's why I wanted to start this new thread. When using jexec to do interactive work inside an existing jail, people find out that they no longer have access to their tty device. As a result, programs requiring input of passwords or passphrases behave unexpectedly in one of several ways. Ssh says "Host key verification failed." and refuses to log in to another system (unless pubkey authentication is user in combination with an agent of course). Some programs fall back to using stdin/stdout and echo the password as it is typed (the mysql clients are popular examples). Work-arounds that have been suggested are 1. Run a sshd inside the jail and log in using ssh 2. Start tmux inside the jail so you get a new pseudo tty slave inside the jail. People trying screen find that it won't work unlike tmux. 3. I tried using 'script -q /dev/null' inside the jail because it is part op the base system and it doesn't change your terminal type and interpret keyboard input and screen output. I found out that I failed when I resized my window :-( I don't like 1 on a machine with many jails, especially if some of them share the same IP address (e.g. sometimes I have to run a mail server on the same IP adress as a webserver but in a distinct jail). 2 is not ideal either because tmux emulates a different terminal on the inside than the terminal on the outside that it runs on. 3 is really a kludge and causes problems when you resize your window. I thought that I found a solution by rewriting jexec such that it will open a pseudo tty and does the passing of data between the jailed pts and the tty from where jexec was started but that's not going to work as the pseudo tty most be opened by the child process inside the jail but the parent outside the jail must have access to the master side of the pseudo tty. So far we are still talking about work-arounds. Why not look at the root cause. Unfortunately I'm not familiar with kernel sources so if I'm wrong, please forgive me, I write this with the best intentions. The root cause of th problem appears to be that pseudo ttys opened outside a jail are not visible nor accessible inside a jail, pseudo ttys created inside a jail are visible and accessible though. Would it be conceivable that by using jexec the controlling tty of jexec magically becomes visible and accessible inside the jail? Preferrable only until jexec dies. I understand that this is not trivial but given the number of threads about this problem, it's a real issue to many people. To me it's worth some $ or EUR to solve this in a clean way. Kind regards, Paul Schenkeveld From owner-freebsd-jail@FreeBSD.ORG Wed Aug 10 02:36:34 2011 Return-Path: Delivered-To: freebsd-jail@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id AD09E1065675 for ; Wed, 10 Aug 2011 02:36:34 +0000 (UTC) (envelope-from ian@weta.stanford.edu) Received: from smtp.stanford.edu (smtp2.Stanford.EDU [171.67.219.82]) by mx1.freebsd.org (Postfix) with ESMTP id 96C518FC17 for ; Wed, 10 Aug 2011 02:36:34 +0000 (UTC) Received: from smtp.stanford.edu (localhost.localdomain [127.0.0.1]) by localhost (Postfix) with SMTP id 8307D8161 for ; Tue, 9 Aug 2011 19:17:51 -0700 (PDT) Received: from weta.stanford.edu (weta.Stanford.EDU [128.12.181.231]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by smtp.stanford.edu (Postfix) with ESMTPS id 1A8C78076 for ; Tue, 9 Aug 2011 19:17:51 -0700 (PDT) Received: from ian by weta.stanford.edu with local (Exim 4.74 (FreeBSD)) (envelope-from ) id 1QqyMh-000MQ6-04 for freebsd-jail@freebsd.org; Tue, 09 Aug 2011 19:17:51 -0700 Date: Tue, 9 Aug 2011 19:17:50 -0700 From: Ian Downes To: freebsd-jail@freebsd.org Message-ID: <20110810021750.GA83262@weta.stanford.edu> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.5.20 (2009-06-14) Sender: Ian Downes Subject: umounting md backed jail filesystems - busy X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 Aug 2011 02:36:34 -0000 Hi everyone, I'm trying to cleanup after shutting down some jails but I'm getting device busy errors when trying to umount some of the filesystems. More specifically, I've got an ephemeral zfs filesystem that serves as the root of the jail. On '/etc/rc.d/jail stop' the jail stops cleanly but when I try to destroy the zfs filesystem the initial umount fails, claiming it's busy. This happens everytime. I can't for the life of me work out who's tying it up. I've tried fstat, lsof and fuser but nothing is reported as active! No processes, no active files. Details: $ uname -a FreeBSD XXX.com 8.2-RELEASE FreeBSD 8.2-RELEASE #0: Tue May 31 19:05:32 UTC 2011 root@XXX.com:/usr/obj/usr/src/sys/XENHVM amd64 'data' is a md backed zpool $ mount | grep data/path/to/jail/root on /path/to/jail/root (zfs, local) $ fstat -f /path/to/jail/root USER CMD PID FD MOUNT INUM MODE SZ|DV R/W $ unmount /path/to/jail/root cannot unmount '/path/to/jail/root': Device busy Some time later, measured in minutes, something frees up and I can umount/destroy the filesystem ok. Can anyone offer some suggestions on what it could be or other ways to determine what's going on? Thanks, Ian From owner-freebsd-jail@FreeBSD.ORG Wed Aug 10 12:49:30 2011 Return-Path: Delivered-To: freebsd-jail@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1B3DA106566B for ; Wed, 10 Aug 2011 12:49:30 +0000 (UTC) (envelope-from joris.dedieu@gmail.com) Received: from mail-iy0-f172.google.com (mail-iy0-f172.google.com [209.85.210.172]) by mx1.freebsd.org (Postfix) with ESMTP id DD9B88FC15 for ; Wed, 10 Aug 2011 12:49:29 +0000 (UTC) Received: by iye7 with SMTP id 7so1157450iye.17 for ; Wed, 10 Aug 2011 05:49:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; bh=rfzgFabvSg02+d7XYUd9HwnFcCVN4P/rxr7ejnbiX8M=; b=pAtD/FeiPovvC4+iauVfBblhFqgf1nlUFK1i0xpXmHUEJQW4q95vQQnuPCD9FqIZgf yUtjZ2yQiCpessF59A1s0uAWXSR656/zCusSs+Tk+IZfTkmCnconiAzHGZFflMay/QGF JsTs7nOOxJxVbcuH46W1IF2KdLOlIpdPKB/S4= MIME-Version: 1.0 Received: by 10.231.42.133 with SMTP id s5mr2664029ibe.34.1312980569219; Wed, 10 Aug 2011 05:49:29 -0700 (PDT) Received: by 10.231.13.204 with HTTP; Wed, 10 Aug 2011 05:49:29 -0700 (PDT) In-Reply-To: <20110809124543.GA22077@psconsult.nl> References: <20110809124543.GA22077@psconsult.nl> Date: Wed, 10 Aug 2011 14:49:29 +0200 Message-ID: From: joris dedieu To: Paul Schenkeveld Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: freebsd-jail@freebsd.org Subject: Re: Jexec and access to tty X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 Aug 2011 12:49:30 -0000 2011/8/9 Paul Schenkeveld : > Hi, > > There have been several threads about this issue, some people have come > up with work arounds but I think that the issue is more fundamental, > that's why I wanted to start this new thread. > > When using jexec to do interactive work inside an existing jail, people > find out that they no longer have access to their tty device. =A0As a > result, programs requiring input of passwords or passphrases behave > unexpectedly in one of several ways. > > Ssh says "Host key verification failed." and refuses to log in to > another system (unless pubkey authentication is user in combination with > an agent of course). =A0Some programs fall back to using stdin/stdout > and echo the password as it is typed (the mysql clients are popular > examples). > > Work-arounds that have been suggested are > =A01. Run a sshd inside the jail and log in using ssh > =A02. Start tmux inside the jail so you get a new pseudo tty slave inside > =A0 =A0the jail. =A0People trying screen find that it won't work unlike t= mux. > =A03. I tried using 'script -q /dev/null' inside the jail because it is > =A0 =A0part op the base system and it doesn't change your terminal type > =A0 =A0and interpret keyboard input and screen output. =A0I found out tha= t I > =A0 =A0failed when I resized my window :-( An other way is to use chroot(5) to enter the jail. Maybe chroot /jail/root login -f $USER should be acceptable in some situati= ons. > > I don't like 1 on a machine with many jails, especially if some of them > share the same IP address (e.g. sometimes I have to run a mail server on > the same IP adress as a webserver but in a distinct jail). > > 2 is not ideal either because tmux emulates a different terminal on > the inside than the terminal on the outside that it runs on. > > 3 is really a kludge and causes problems when you resize your window. > > I thought that I found a solution by rewriting jexec such that it will > open a pseudo tty and does the passing of data between the jailed pts > and the tty from where jexec was started but that's not going to work as > the pseudo tty most be opened by the child process inside the jail but > the parent outside the jail must have access to the master side of the > pseudo tty. > > So far we are still talking about work-arounds. =A0Why not look at the > root cause. =A0Unfortunately I'm not familiar with kernel sources so if > I'm wrong, please forgive me, I write this with the best intentions. > > The root cause of th problem appears to be that pseudo ttys opened > outside a jail are not visible nor accessible inside a jail, pseudo ttys > created inside a jail are visible and accessible though. As far as I understand, sys/fs/devfs/devfs_vnops.c uses prison_check(9) too see if an item as been build in the same jail or in a child. The tty exists inside the jail but you can't use it (and so you can't escape the jail). > > Would it be conceivable that by using jexec the controlling tty of jexec > magically becomes visible and accessible inside the jail? =A0Preferrable > only until jexec dies. I'm not sure. The only way should be to temporary disable the check with a variable. But it will also brake jail security during jexec excution. Using chroot(2) instead of jail(2) should be an option (but it's non trivial to affect jail context for all other subsystems). I think the only right way is to open a new tty while entering the jail (this is what tmux or a jailed ssh does). But it should be difficult to make things like echo ps | jexec 1 sh works. regards Joris > > I understand that this is not trivial but given the number of threads > about this problem, it's a real issue to many people. =A0To me it's worth > some $ or EUR to solve this in a clean way. > > Kind regards, > > Paul Schenkeveld > _______________________________________________ > freebsd-jail@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-jail > To unsubscribe, send any mail to "freebsd-jail-unsubscribe@freebsd.org" > From owner-freebsd-jail@FreeBSD.ORG Wed Aug 10 13:16:54 2011 Return-Path: Delivered-To: freebsd-jail@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A4509106564A for ; Wed, 10 Aug 2011 13:16:54 +0000 (UTC) (envelope-from joris.dedieu@gmail.com) Received: from mail-iy0-f172.google.com (mail-iy0-f172.google.com [209.85.210.172]) by mx1.freebsd.org (Postfix) with ESMTP id 722728FC18 for ; Wed, 10 Aug 2011 13:16:54 +0000 (UTC) Received: by iye7 with SMTP id 7so1196151iye.17 for ; Wed, 10 Aug 2011 06:16:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; bh=4PmRZeImvvnE2jNI9oivbjSYpKXG5SnozqnnpfWxfXo=; b=BC73fMnDddvYKhBBODtazK54NEDzfaRWRxF+vi+afvhe6JvwklnljUvOt2dY5AUlUN yOlcbsO0TwOOvSYztq1DR9PNEd7HDa+ldKmNWEGS4uvpHnlIvcwhOqSEWpVPqx+zYm3o dkkcXRPX7kCPn/OVEIxF6ax9JluY3u0II4/Ao= MIME-Version: 1.0 Received: by 10.42.150.68 with SMTP id z4mr7951387icv.23.1312982213620; Wed, 10 Aug 2011 06:16:53 -0700 (PDT) Received: by 10.231.13.204 with HTTP; Wed, 10 Aug 2011 06:16:53 -0700 (PDT) In-Reply-To: <20110810021750.GA83262@weta.stanford.edu> References: <20110810021750.GA83262@weta.stanford.edu> Date: Wed, 10 Aug 2011 15:16:53 +0200 Message-ID: From: joris dedieu To: Ian Downes Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: freebsd-jail@freebsd.org Subject: Re: umounting md backed jail filesystems - busy X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 Aug 2011 13:16:54 -0000 2011/8/10 Ian Downes : > Hi everyone, > > I'm trying to cleanup after shutting down some jails but I'm getting devi= ce > busy errors when trying to umount some of the filesystems. > > More specifically, I've got an ephemeral zfs filesystem that serves as th= e root > of the jail. On '/etc/rc.d/jail stop' the jail stops cleanly but when I t= ry to > destroy the zfs filesystem the initial umount fails, claiming it's busy. = This > happens everytime. > > I can't for the life of me work out who's tying it up. I've tried fstat, = lsof > and fuser but nothing is reported as active! No processes, no active file= s. What gives top -S -n 10000 |grep -i zfs while system is not unmountable (could you see some processes in zfs state) ? What gives jls during this time (does it report anythink still alive) ? Regards Joris > > Details: > > $ uname -a > FreeBSD XXX.com 8.2-RELEASE FreeBSD 8.2-RELEASE #0: Tue May 31 19:05:32 U= TC 2011 =A0 =A0 root@XXX.com:/usr/obj/usr/src/sys/XENHVM =A0amd64 > > 'data' is a md backed zpool > > $ mount | grep > data/path/to/jail/root on /path/to/jail/root (zfs, local) > > $ fstat -f /path/to/jail/root > USER =A0 =A0 CMD =A0 =A0 =A0 =A0 =A0PID =A0 FD MOUNT =A0 =A0 =A0INUM MODE= =A0 =A0 =A0 =A0 SZ|DV R/W > > $ unmount /path/to/jail/root > cannot unmount '/path/to/jail/root': Device busy > > Some time later, measured in minutes, something frees up and I can > umount/destroy the filesystem ok. > > Can anyone offer some suggestions on what it could be or other ways to > determine what's going on? > > Thanks, > > Ian > > _______________________________________________ > freebsd-jail@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-jail > To unsubscribe, send any mail to "freebsd-jail-unsubscribe@freebsd.org" > From owner-freebsd-jail@FreeBSD.ORG Wed Aug 10 16:00:49 2011 Return-Path: Delivered-To: freebsd-jail@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8B654106564A for ; Wed, 10 Aug 2011 16:00:49 +0000 (UTC) (envelope-from ian@weta.stanford.edu) Received: from smtp.stanford.edu (smtp2.Stanford.EDU [171.67.219.82]) by mx1.freebsd.org (Postfix) with ESMTP id 7318B8FC08 for ; Wed, 10 Aug 2011 16:00:49 +0000 (UTC) Received: from smtp.stanford.edu (localhost.localdomain [127.0.0.1]) by localhost (Postfix) with SMTP id 074C782E8; Wed, 10 Aug 2011 09:00:49 -0700 (PDT) Received: from weta.stanford.edu (weta.Stanford.EDU [128.12.181.231]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by smtp.stanford.edu (Postfix) with ESMTPS id 36FFC81AC; Wed, 10 Aug 2011 09:00:48 -0700 (PDT) Received: from ian by weta.stanford.edu with local (Exim 4.74 (FreeBSD)) (envelope-from ) id 1QrBD6-000Jkx-3e; Wed, 10 Aug 2011 09:00:48 -0700 Date: Wed, 10 Aug 2011 09:00:48 -0700 From: Ian Downes To: joris dedieu Message-ID: <20110810160047.GA74133@weta.stanford.edu> References: <20110810021750.GA83262@weta.stanford.edu> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.20 (2009-06-14) Sender: Ian Downes Cc: freebsd-jail@freebsd.org Subject: Re: umounting md backed jail filesystems - busy X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 Aug 2011 16:00:49 -0000 Joris, thanks for the suggestions but they don't appear to indicate anything unusual. > > I'm trying to cleanup after shutting down some jails but I'm getting device > > busy errors when trying to umount some of the filesystems. > > > > More specifically, I've got an ephemeral zfs filesystem that serves as the root > > of the jail. On '/etc/rc.d/jail stop' the jail stops cleanly but when I try to > > destroy the zfs filesystem the initial umount fails, claiming it's busy. This > > happens everytime. > > > > I can't for the life of me work out who's tying it up. I've tried fstat, lsof > > and fuser but nothing is reported as active! No processes, no active files. > > What gives top -S -n 10000 |grep -i zfs while system is not > unmountable (could you see some processes in zfs state) ? > What gives jls during this time (does it report anythink still alive) ? $ jls JID IP Address Hostname Path $ top -S -n 10000 |grep -i zfs 5 root 6 -8 - 0K 88K tx->tx 2 0:01 0.00% zfskern It's entirely possible that this isn't due to a process related to the jail but it is zfs tying things up for a while. I'd like to understand what's going on though as I'd rather not just hack something together to schedule purges at some later time. > > > > Details: > > > > $ uname -a > > FreeBSD XXX.com 8.2-RELEASE FreeBSD 8.2-RELEASE #0: Tue May 31 19:05:32 UTC 2011 ? ? root@XXX.com:/usr/obj/usr/src/sys/XENHVM ?amd64 > > > > 'data' is a md backed zpool > > > > $ mount | grep > > data/path/to/jail/root on /path/to/jail/root (zfs, local) > > > > $ fstat -f /path/to/jail/root > > USER ? ? CMD ? ? ? ? ?PID ? FD MOUNT ? ? ?INUM MODE ? ? ? ? SZ|DV R/W > > > > $ unmount /path/to/jail/root > > cannot unmount '/path/to/jail/root': Device busy > > > > Some time later, measured in minutes, something frees up and I can > > umount/destroy the filesystem ok. > > > > Can anyone offer some suggestions on what it could be or other ways to > > determine what's going on? From owner-freebsd-jail@FreeBSD.ORG Wed Aug 10 16:36:01 2011 Return-Path: Delivered-To: freebsd-jail@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A65D7106566B for ; Wed, 10 Aug 2011 16:36:01 +0000 (UTC) (envelope-from ian@weta.stanford.edu) Received: from smtp.stanford.edu (smtp3.Stanford.EDU [171.67.219.83]) by mx1.freebsd.org (Postfix) with ESMTP id 8D41C8FC1A for ; Wed, 10 Aug 2011 16:36:01 +0000 (UTC) Received: from smtp.stanford.edu (localhost.localdomain [127.0.0.1]) by localhost (Postfix) with SMTP id 3DBA4D8118; Wed, 10 Aug 2011 09:36:01 -0700 (PDT) Received: from weta.stanford.edu (weta.Stanford.EDU [128.12.181.231]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by smtp.stanford.edu (Postfix) with ESMTPS id C34BAD8057; Wed, 10 Aug 2011 09:36:00 -0700 (PDT) Received: from ian by weta.stanford.edu with local (Exim 4.74 (FreeBSD)) (envelope-from ) id 1QrBlA-000KlN-MU; Wed, 10 Aug 2011 09:36:00 -0700 Date: Wed, 10 Aug 2011 09:36:00 -0700 From: Ian Downes To: Steven Hartland Message-ID: <20110810163600.GA77840@weta.stanford.edu> References: <20110810021750.GA83262@weta.stanford.edu> <20110810160047.GA74133@weta.stanford.edu> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.20 (2009-06-14) Sender: Ian Downes Cc: freebsd-jail@freebsd.org Subject: Re: umounting md backed jail filesystems - busy X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 Aug 2011 16:36:01 -0000 > > $ jls > > JID IP Address Hostname Path > > What does jls -d say? i.e. is the jail really shutdown or is it still dieing? Thanks, indeed jls -d does show the jail as in the process of dying. I watched jls -d and (unscientifically) as soon as jls -d reported the jail was completely dead I was able to umount and destroy the filesystem. I hadn't expected it to take a minute or more for a jail to die. Is this usual? Can anyone suggest what may be causing this? Perhaps a network related timeout? thanks, Ian From owner-freebsd-jail@FreeBSD.ORG Wed Aug 10 16:38:55 2011 Return-Path: Delivered-To: freebsd-jail@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A184B106566B for ; Wed, 10 Aug 2011 16:38:55 +0000 (UTC) (envelope-from prvs=1203777b38=killing@multiplay.co.uk) Received: from mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) by mx1.freebsd.org (Postfix) with ESMTP id 29AC28FC18 for ; Wed, 10 Aug 2011 16:38:54 +0000 (UTC) X-MDAV-Processed: mail1.multiplay.co.uk, Wed, 10 Aug 2011 17:27:32 +0100 X-Spam-Processed: mail1.multiplay.co.uk, Wed, 10 Aug 2011 17:27:32 +0100 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on mail1.multiplay.co.uk X-Spam-Level: X-Spam-Status: No, score=-5.0 required=6.0 tests=USER_IN_WHITELIST shortcircuit=ham autolearn=disabled version=3.2.5 Received: from r2d2 ([188.220.16.49]) by mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) (MDaemon PRO v10.0.4) with ESMTP id md50014546045.msg for ; Wed, 10 Aug 2011 17:27:32 +0100 X-MDRemoteIP: 188.220.16.49 X-Return-Path: prvs=1203777b38=killing@multiplay.co.uk X-Envelope-From: killing@multiplay.co.uk X-MDaemon-Deliver-To: freebsd-jail@freebsd.org Message-ID: From: "Steven Hartland" To: "Ian Downes" , "joris dedieu" References: <20110810021750.GA83262@weta.stanford.edu> <20110810160047.GA74133@weta.stanford.edu> Date: Wed, 10 Aug 2011 17:28:03 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6109 Cc: freebsd-jail@freebsd.org Subject: Re: umounting md backed jail filesystems - busy X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 Aug 2011 16:38:55 -0000 ----- Original Message ----- From: "Ian Downes" > $ jls > JID IP Address Hostname Path What does jls -d say? i.e. is the jail really shutdown or is it still dieing? Regards Steve ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmaster@multiplay.co.uk. From owner-freebsd-jail@FreeBSD.ORG Wed Aug 10 16:43:17 2011 Return-Path: Delivered-To: freebsd-jail@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2ABCD1065670 for ; Wed, 10 Aug 2011 16:43:17 +0000 (UTC) (envelope-from prvs=1203777b38=killing@multiplay.co.uk) Received: from mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) by mx1.freebsd.org (Postfix) with ESMTP id A99608FC12 for ; Wed, 10 Aug 2011 16:43:16 +0000 (UTC) X-MDAV-Processed: mail1.multiplay.co.uk, Wed, 10 Aug 2011 17:42:40 +0100 X-Spam-Processed: mail1.multiplay.co.uk, Wed, 10 Aug 2011 17:42:40 +0100 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on mail1.multiplay.co.uk X-Spam-Level: X-Spam-Status: No, score=-5.0 required=6.0 tests=USER_IN_WHITELIST shortcircuit=ham autolearn=disabled version=3.2.5 Received: from r2d2 ([188.220.16.49]) by mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) (MDaemon PRO v10.0.4) with ESMTP id md50014546209.msg for ; Wed, 10 Aug 2011 17:42:38 +0100 X-MDRemoteIP: 188.220.16.49 X-Return-Path: prvs=1203777b38=killing@multiplay.co.uk X-Envelope-From: killing@multiplay.co.uk X-MDaemon-Deliver-To: freebsd-jail@freebsd.org Message-ID: <629ADCE73AB740D1A24CF9327449234C@multiplay.co.uk> From: "Steven Hartland" To: "Ian Downes" References: <20110810021750.GA83262@weta.stanford.edu> <20110810160047.GA74133@weta.stanford.edu> <20110810163600.GA77840@weta.stanford.edu> Date: Wed, 10 Aug 2011 17:43:09 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6109 Cc: freebsd-jail@freebsd.org Subject: Re: umounting md backed jail filesystems - busy X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 Aug 2011 16:43:17 -0000 ----- Original Message ----- From: "Ian Downes" > Thanks, indeed jls -d does show the jail as in the process of dying. I > watched jls -d and (unscientifically) as soon as jls -d reported the jail was > completely dead I was able to umount and destroy the filesystem. > > I hadn't expected it to take a minute or more for a jail to die. Is this usual? > Can anyone suggest what may be causing this? Perhaps a network related timeout? Yes its very likely a network timeout. You can either track down the app which is not closing properly or you could try reducing the tcp timeout via the sysctl:- net.inet.tcp.msl Be aware that changing this does technically break the RFC. Regards Steve ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmaster@multiplay.co.uk. From owner-freebsd-jail@FreeBSD.ORG Wed Aug 10 22:14:54 2011 Return-Path: Delivered-To: freebsd-jail@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1ED4C106564A for ; Wed, 10 Aug 2011 22:14:54 +0000 (UTC) (envelope-from joris.dedieu@gmail.com) Received: from mail-iy0-f172.google.com (mail-iy0-f172.google.com [209.85.210.172]) by mx1.freebsd.org (Postfix) with ESMTP id DFB648FC0A for ; Wed, 10 Aug 2011 22:14:53 +0000 (UTC) Received: by iye7 with SMTP id 7so1964960iye.17 for ; Wed, 10 Aug 2011 15:14:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type:content-transfer-encoding; bh=JJ1fQ/CufHH19gcYx6QuhkDNtZ5ETHecsdQaN+RSxYw=; b=QY/VaJxxKZhUPfecif8JKBEIhVPywOFhCsLQ5n1CzT9cWtQdFJEJdzJ7cox/htTwC6 l+0LLCZzpXt4A/lj4dMBNDzbBuT6DD+GVfvB5VZSiNEJPYDcO0QGheCiESoW/ENgdlpk UWYhnZBeKrgVrjB/asme4VcbzGy2eqrMjN+5c= MIME-Version: 1.0 Received: by 10.231.65.73 with SMTP id h9mr10921205ibi.21.1313014492993; Wed, 10 Aug 2011 15:14:52 -0700 (PDT) Received: by 10.231.13.204 with HTTP; Wed, 10 Aug 2011 15:14:52 -0700 (PDT) In-Reply-To: References: <20110809124543.GA22077@psconsult.nl> Date: Thu, 11 Aug 2011 00:14:52 +0200 Message-ID: From: joris dedieu To: freebsd-jail@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Subject: Re: Jexec and access to tty X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 Aug 2011 22:14:54 -0000 2011/8/10 joris dedieu : > 2011/8/9 Paul Schenkeveld : >> Hi, >> >> There have been several threads about this issue, some people have come >> up with work arounds but I think that the issue is more fundamental, >> that's why I wanted to start this new thread. >> >> When using jexec to do interactive work inside an existing jail, people >> find out that they no longer have access to their tty device. =A0As a >> result, programs requiring input of passwords or passphrases behave >> unexpectedly in one of several ways. >> >> Ssh says "Host key verification failed." and refuses to log in to >> another system (unless pubkey authentication is user in combination with >> an agent of course). =A0Some programs fall back to using stdin/stdout >> and echo the password as it is typed (the mysql clients are popular >> examples). >> >> Work-arounds that have been suggested are >> =A01. Run a sshd inside the jail and log in using ssh >> =A02. Start tmux inside the jail so you get a new pseudo tty slave insid= e >> =A0 =A0the jail. =A0People trying screen find that it won't work unlike = tmux. >> =A03. I tried using 'script -q /dev/null' inside the jail because it is >> =A0 =A0part op the base system and it doesn't change your terminal type >> =A0 =A0and interpret keyboard input and screen output. =A0I found out th= at I >> =A0 =A0failed when I resized my window :-( > > An other way is to use chroot(5) to enter the jail. > Maybe chroot /jail/root login -f $USER should be acceptable in some situa= tions. > >> >> I don't like 1 on a machine with many jails, especially if some of them >> share the same IP address (e.g. sometimes I have to run a mail server on >> the same IP adress as a webserver but in a distinct jail). >> >> 2 is not ideal either because tmux emulates a different terminal on >> the inside than the terminal on the outside that it runs on. >> >> 3 is really a kludge and causes problems when you resize your window. >> >> I thought that I found a solution by rewriting jexec such that it will >> open a pseudo tty and does the passing of data between the jailed pts >> and the tty from where jexec was started but that's not going to work as >> the pseudo tty most be opened by the child process inside the jail but >> the parent outside the jail must have access to the master side of the >> pseudo tty. >> >> So far we are still talking about work-arounds. =A0Why not look at the >> root cause. =A0Unfortunately I'm not familiar with kernel sources so if >> I'm wrong, please forgive me, I write this with the best intentions. >> >> The root cause of th problem appears to be that pseudo ttys opened >> outside a jail are not visible nor accessible inside a jail, pseudo ttys >> created inside a jail are visible and accessible though. > > As far as I understand, sys/fs/devfs/devfs_vnops.c uses > prison_check(9) too see if an item as been build in the same jail or > in a child. > The tty exists inside the jail but you can't use it (and so you can't > escape the jail). > >> >> Would it be conceivable that by using jexec the controlling tty of jexec >> magically becomes visible and accessible inside the jail? =A0Preferrable >> only until jexec dies. > > I'm not sure. The only way should be =A0to temporary disable the check > with a =A0variable. But it will also brake jail security during jexec > excution. > Using chroot(2) instead of jail(2) should be an option (but it's =A0non > trivial to affect jail context for all other subsystems). > > I think the only right way is to open a new tty while entering the > jail (this is what tmux or a jailed ssh does). > But it should be difficult to make things like echo ps | jexec 1 sh works= . I gave this way a try and quickly produced a dirty patch. I did not yet investigate on the right way, but I already know that I'm wrong on several items so sorry for your eyes :) Basically this patch introduced -t option that forks jexec inside the jail, open a new pts and tries to establish a communication with the old one. It mostly works has expected but breaks some console inputs like control commands, completion and so on. Before spending more time trying to produce something acceptable, please let me know if it should be the expected feature or sounds like a nasty workaround. --- usr.sbin/jexec/jexec.c.orig 2009-08-03 10:13:06.000000000 +0200 +++ usr.sbin/jexec/jexec.c 2011-08-10 23:33:44.000000000 +0200 @@ -37,16 +37,19 @@ #include #include +#include #include #include #include #include #include #include +#include #include #include static void usage(void); +static int ntty_exec(const char *file, char *const argv[]); #define GET_USER_INFO do { \ pwd =3D getpwnam(username); \ @@ -64,6 +67,8 @@ err(1, "getgrouplist: %s", username); \ } while (0) +#define MAXREAD 256 + int main(int argc, char *argv[]) { @@ -71,17 +76,17 @@ login_cap_t *lcap =3D NULL; struct passwd *pwd =3D NULL; gid_t *groups =3D NULL; - int ch, ngroups, uflag, Uflag; + int ch, ngroups, uflag, Uflag, tflag; long ngroups_max; char *username; - ch =3D uflag =3D Uflag =3D 0; + ch =3D uflag =3D Uflag =3D tflag =3D 0; username =3D NULL; ngroups_max =3D sysconf(_SC_NGROUPS_MAX) + 1; if ((groups =3D malloc(sizeof(gid_t) * ngroups_max)) =3D=3D NULL) err(1, "malloc"); - while ((ch =3D getopt(argc, argv, "nu:U:")) !=3D -1) { + while ((ch =3D getopt(argc, argv, "nu:U:t")) !=3D -1) { switch (ch) { case 'n': /* Specified name, now unused */ @@ -94,6 +99,9 @@ username =3D optarg; Uflag =3D 1; break; + case 't': + tflag =3D 1; + break; default: usage(); } @@ -125,8 +133,14 @@ err(1, "setusercontext"); login_close(lcap); } - if (execvp(argv[1], argv + 1) =3D=3D -1) - err(1, "execvp(): %s", argv[1]); + if (tflag) { + if(ntty_exec(argv[1], argv + 1) =3D=3D -1) + err(1, "ntty_exec(): %s", argv[1]); + } + else { + if (execvp(argv[1], argv + 1) =3D=3D -1) + err(1, "execvp(): %s", argv[1]); + } exit(0); } @@ -138,3 +152,77 @@ "usage: jexec [-u username | -U username] jail command ..."= ); exit(1); } + +static int +ntty_exec(const char *file, char *const argv[]) +{ + int pseudoterm, pseudoterm_fd, res; + struct fd_set in_fd; + struct termios termset; + char input[MAXREAD]; + + if ((pseudoterm =3D posix_openpt(O_RDWR)) =3D=3D -1) + err(1, "posix_openpt"); + if (grantpt(pseudoterm) !=3D 0) + err(1, "grantpt"); + if (unlockpt(pseudoterm) !=3D 0) + err(1, "grantpt"); + if ((pseudoterm_fd =3D open(ptsname(pseudoterm), O_RDWR)) =3D=3D -1= ) + err(1, "open"); + + switch (fork()) { + case -1 : + err(1, "fork"); + case 0 : + close(pseudoterm); + if (tcgetattr(pseudoterm_fd, &termset) =3D=3D -1) + err(1, "tcgetattr"); + cfmakeraw(&termset); + tcsetattr(pseudoterm_fd, TCSANOW, &termset); + fclose(stdin); + fclose(stdout); + fclose(stderr); + dup(pseudoterm_fd); + dup(pseudoterm_fd); + dup(pseudoterm_fd); + close(pseudoterm_fd); + setsid(); + ioctl(0, TIOCSCTTY, 1); + return execvp(file, argv); + default : + close(pseudoterm_fd); + while(1) { + FD_ZERO(&in_fd); + FD_SET(0, &in_fd); + FD_SET(pseudoterm, &in_fd); + if (select(pseudoterm + 1, &in_fd, NULL, NULL, NULL) =3D=3D -1) + err(1, "select"); + if (FD_ISSET(0, &in_fd)) { + /* stdin */ + if ((res =3D read(0, input, sizeof(input)))= > 0 ) + write(pseudoterm, input, res); + else { + if (errno =3D=3D ENOENT) + return 0; + else + err(1, "read"); + } + } + if (FD_ISSET(pseudoterm, &in_fd)) { + if ((res =3D read(pseudoterm, input, + sizeof(input))) > 0 ) + /* stdout */ + write(1, input, res); + else { + if (errno =3D=3D ENOENT) + return 0; + else + err(1, "write"); + } + } + } + } + + return 0; +} + regards Joris > > regards > Joris > >> >> I understand that this is not trivial but given the number of threads >> about this problem, it's a real issue to many people. =A0To me it's wort= h >> some $ or EUR to solve this in a clean way. >> >> Kind regards, >> >> Paul Schenkeveld >> _______________________________________________ >> freebsd-jail@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-jail >> To unsubscribe, send any mail to "freebsd-jail-unsubscribe@freebsd.org" >> > From owner-freebsd-jail@FreeBSD.ORG Thu Aug 11 12:34:03 2011 Return-Path: Delivered-To: freebsd-jail@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 482FA106566B for ; Thu, 11 Aug 2011 12:34:03 +0000 (UTC) (envelope-from joris.dedieu@gmail.com) Received: from mail-iy0-f172.google.com (mail-iy0-f172.google.com [209.85.210.172]) by mx1.freebsd.org (Postfix) with ESMTP id C9E298FC1E for ; Thu, 11 Aug 2011 12:34:02 +0000 (UTC) Received: by iye7 with SMTP id 7so3348374iye.17 for ; Thu, 11 Aug 2011 05:34:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type:content-transfer-encoding; bh=xF1FnG0HluQSAtc1hhIZXPeXjzUjp4LvNo1N15csWZg=; b=jo2tJPGKykY2w9pS2PZMvawY4o6Bc/653hSQMTch2SlOFB4Ub16Fk0qCSQpAAey9gt NyDAMqBxWyneW0cgFQN5wsF+eUWBrUM8nZYudOGwQ/ve4TafHPuf2pnmo/HkiEltGYt1 P2zApZOY+F+zgYcRke114IpUQoqH9O2pcmuh4= MIME-Version: 1.0 Received: by 10.231.91.69 with SMTP id l5mr13061909ibm.47.1313066041970; Thu, 11 Aug 2011 05:34:01 -0700 (PDT) Received: by 10.231.13.204 with HTTP; Thu, 11 Aug 2011 05:34:01 -0700 (PDT) In-Reply-To: References: <20110809124543.GA22077@psconsult.nl> Date: Thu, 11 Aug 2011 14:34:01 +0200 Message-ID: From: joris dedieu To: freebsd-jail@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Subject: Re: Jexec and access to tty X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 11 Aug 2011 12:34:03 -0000 2011/8/11 joris dedieu : > 2011/8/10 joris dedieu : >> 2011/8/9 Paul Schenkeveld : >>> Hi, >>> >>> There have been several threads about this issue, some people have come >>> up with work arounds but I think that the issue is more fundamental, >>> that's why I wanted to start this new thread. >>> >>> When using jexec to do interactive work inside an existing jail, people >>> find out that they no longer have access to their tty device. =A0As a >>> result, programs requiring input of passwords or passphrases behave >>> unexpectedly in one of several ways. >>> >>> Ssh says "Host key verification failed." and refuses to log in to >>> another system (unless pubkey authentication is user in combination wit= h >>> an agent of course). =A0Some programs fall back to using stdin/stdout >>> and echo the password as it is typed (the mysql clients are popular >>> examples). >>> >>> Work-arounds that have been suggested are >>> =A01. Run a sshd inside the jail and log in using ssh >>> =A02. Start tmux inside the jail so you get a new pseudo tty slave insi= de >>> =A0 =A0the jail. =A0People trying screen find that it won't work unlike= tmux. >>> =A03. I tried using 'script -q /dev/null' inside the jail because it is >>> =A0 =A0part op the base system and it doesn't change your terminal type >>> =A0 =A0and interpret keyboard input and screen output. =A0I found out t= hat I >>> =A0 =A0failed when I resized my window :-( >> >> An other way is to use chroot(5) to enter the jail. >> Maybe chroot /jail/root login -f $USER should be acceptable in some situ= ations. >> >>> >>> I don't like 1 on a machine with many jails, especially if some of them >>> share the same IP address (e.g. sometimes I have to run a mail server o= n >>> the same IP adress as a webserver but in a distinct jail). >>> >>> 2 is not ideal either because tmux emulates a different terminal on >>> the inside than the terminal on the outside that it runs on. >>> >>> 3 is really a kludge and causes problems when you resize your window. >>> >>> I thought that I found a solution by rewriting jexec such that it will >>> open a pseudo tty and does the passing of data between the jailed pts >>> and the tty from where jexec was started but that's not going to work a= s >>> the pseudo tty most be opened by the child process inside the jail but >>> the parent outside the jail must have access to the master side of the >>> pseudo tty. >>> >>> So far we are still talking about work-arounds. =A0Why not look at the >>> root cause. =A0Unfortunately I'm not familiar with kernel sources so if >>> I'm wrong, please forgive me, I write this with the best intentions. >>> >>> The root cause of th problem appears to be that pseudo ttys opened >>> outside a jail are not visible nor accessible inside a jail, pseudo tty= s >>> created inside a jail are visible and accessible though. >> >> As far as I understand, sys/fs/devfs/devfs_vnops.c uses >> prison_check(9) too see if an item as been build in the same jail or >> in a child. >> The tty exists inside the jail but you can't use it (and so you can't >> escape the jail). >> >>> >>> Would it be conceivable that by using jexec the controlling tty of jexe= c >>> magically becomes visible and accessible inside the jail? =A0Preferrabl= e >>> only until jexec dies. >> >> I'm not sure. The only way should be =A0to temporary disable the check >> with a =A0variable. But it will also brake jail security during jexec >> excution. >> Using chroot(2) instead of jail(2) should be an option (but it's =A0non >> trivial to affect jail context for all other subsystems). >> >> I think the only right way is to open a new tty while entering the >> jail (this is what tmux or a jailed ssh does). >> But it should be difficult to make things like echo ps | jexec 1 sh work= s. > > I gave this way a try and quickly produced a dirty patch. I did not > yet investigate on the right way, but I already know that I'm wrong on > several items so sorry for your eyes :) > > Basically this patch =A0introduced =A0-t option that forks jexec inside > the jail, open a new pts and tries to establish a communication with > the old one. > It mostly works has expected but breaks some console inputs like > control commands, completion and so on. > Before spending more time =A0trying to produce something acceptable, > please let =A0me know if it should be the expected feature or sounds > like a nasty workaround. > > --- usr.sbin/jexec/jexec.c.orig 2009-08-03 10:13:06.000000000 +0200 > +++ usr.sbin/jexec/jexec.c =A0 =A0 =A02011-08-10 23:33:44.000000000 +0200 > @@ -37,16 +37,19 @@ > > =A0#include > =A0#include > +#include > =A0#include > =A0#include > =A0#include > =A0#include > =A0#include > =A0#include > +#include > =A0#include > =A0#include > > =A0static void =A0 =A0usage(void); > +static int =A0 =A0 ntty_exec(const char *file, char *const argv[]); > > =A0#define GET_USER_INFO do { =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0= =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 \ > =A0 =A0 =A0 =A0pwd =3D getpwnam(username); =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 \ > @@ -64,6 +67,8 @@ > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0err(1, "getgrouplist: %s", username); =A0 = =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 \ > =A0} while (0) > > +#define MAXREAD 256 > + > =A0int > =A0main(int argc, char *argv[]) > =A0{ > @@ -71,17 +76,17 @@ > =A0 =A0 =A0 =A0login_cap_t *lcap =3D NULL; > =A0 =A0 =A0 =A0struct passwd *pwd =3D NULL; > =A0 =A0 =A0 =A0gid_t *groups =3D NULL; > - =A0 =A0 =A0 int ch, ngroups, uflag, Uflag; > + =A0 =A0 =A0 int ch, ngroups, uflag, Uflag, tflag; > =A0 =A0 =A0 =A0long ngroups_max; > =A0 =A0 =A0 =A0char *username; > > - =A0 =A0 =A0 ch =3D uflag =3D Uflag =3D 0; > + =A0 =A0 =A0 ch =3D uflag =3D Uflag =3D tflag =3D 0; > =A0 =A0 =A0 =A0username =3D NULL; > =A0 =A0 =A0 =A0ngroups_max =3D sysconf(_SC_NGROUPS_MAX) + 1; > =A0 =A0 =A0 =A0if ((groups =3D malloc(sizeof(gid_t) * ngroups_max)) =3D= =3D NULL) > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0err(1, "malloc"); > > - =A0 =A0 =A0 while ((ch =3D getopt(argc, argv, "nu:U:")) !=3D -1) { > + =A0 =A0 =A0 while ((ch =3D getopt(argc, argv, "nu:U:t")) !=3D -1) { > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0switch (ch) { > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0case 'n': > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0/* Specified name, now unu= sed */ > @@ -94,6 +99,9 @@ > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0username =3D optarg; > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0Uflag =3D 1; > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0break; > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 case 't': > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 tflag =3D 1; > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 break; > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0default: > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0usage(); > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0} > @@ -125,8 +133,14 @@ > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0err(1, "setusercontext"); > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0login_close(lcap); > =A0 =A0 =A0 =A0} > - =A0 =A0 =A0 if (execvp(argv[1], argv + 1) =3D=3D -1) > - =A0 =A0 =A0 =A0 =A0 =A0 =A0 err(1, "execvp(): %s", argv[1]); > + =A0 =A0 =A0 if (tflag) { > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 if(ntty_exec(argv[1], argv + 1) =3D=3D -1) > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 err(1, "ntty_exec(): %s", a= rgv[1]); > + =A0 =A0 =A0 } > + =A0 =A0 =A0 else { > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 if (execvp(argv[1], argv + 1) =3D=3D -1) > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 err(1, "execvp(): %s", argv= [1]); > + =A0 =A0 =A0 } > =A0 =A0 =A0 =A0exit(0); > =A0} > > @@ -138,3 +152,77 @@ > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0"usage: jexec [-u username | -U username] = jail command ..."); > =A0 =A0 =A0 =A0exit(1); > =A0} > + > +static int > +ntty_exec(const char *file, char *const argv[]) > +{ > + =A0 =A0 =A0 int pseudoterm, pseudoterm_fd, res; > + =A0 =A0 =A0 struct fd_set in_fd; > + =A0 =A0 =A0 struct termios termset; > + =A0 =A0 =A0 char input[MAXREAD]; > + > + =A0 =A0 =A0 if ((pseudoterm =3D posix_openpt(O_RDWR)) =3D=3D -1) > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 err(1, "posix_openpt"); > + =A0 =A0 =A0 if (grantpt(pseudoterm) !=3D 0) > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 err(1, "grantpt"); > + =A0 =A0 =A0 if (unlockpt(pseudoterm) !=3D 0) > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 err(1, "grantpt"); > + =A0 =A0 =A0 if ((pseudoterm_fd =3D open(ptsname(pseudoterm), O_RDWR)) = =3D=3D -1) > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 err(1, "open"); > + > + =A0 =A0 =A0 switch (fork()) { > + =A0 =A0 =A0 case -1 : > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 err(1, "fork"); > + =A0 =A0 =A0 case 0 =A0: > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 close(pseudoterm); > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 if (tcgetattr(pseudoterm_fd, &termset) =3D= =3D -1) > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 err(1, "tcgetattr"); > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 cfmakeraw(&termset); > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 tcsetattr(pseudoterm_fd, TCSANOW, &termset)= ; > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 fclose(stdin); > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 fclose(stdout); > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 fclose(stderr); > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 dup(pseudoterm_fd); > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 dup(pseudoterm_fd); > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 dup(pseudoterm_fd); > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 close(pseudoterm_fd); > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 setsid(); > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 ioctl(0, TIOCSCTTY, 1); > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 return execvp(file, argv); > + =A0 =A0 =A0 default : > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 close(pseudoterm_fd); > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 while(1) { > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 FD_ZERO(&in_fd); > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 FD_SET(0, &in_fd); > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 FD_SET(pseudoterm, &in_fd); > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 if (select(pseudoterm + 1, = &in_fd, NULL, NULL, > NULL) =3D=3D -1) > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 err(1, "sel= ect"); > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 if (FD_ISSET(0, &in_fd)) { > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 /* stdin */ > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 if ((res = =3D read(0, input, sizeof(input))) > 0 ) > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0= =A0 write(pseudoterm, input, res); > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 else { > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0= =A0 if (errno =3D=3D ENOENT) > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0= =A0 =A0 =A0 =A0 =A0 return 0; > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0= =A0 else > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0= =A0 =A0 =A0 =A0 =A0 err(1, "read"); > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 } > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 } > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 if (FD_ISSET(pseudoterm, &i= n_fd)) { > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 if ((res = =3D read(pseudoterm, input, > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 siz= eof(input))) > 0 ) > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0= =A0 /* stdout */ > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0= =A0 write(1, input, res); > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 else { > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0= =A0 if (errno =3D=3D ENOENT) > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0= =A0 =A0 =A0 =A0 =A0 return 0; > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0= =A0 else > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0= =A0 =A0 =A0 =A0 =A0 err(1, "write"); > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0= =A0 } > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 } > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 } > + =A0 =A0 =A0 } > + > + =A0 =A0 =A0 return 0; > +} > + > > regards > Joris Finally I produced a most acceptable patch as almost everything exists in script(1) sources. I don't experienced any problems using it. --- usr.sbin/jexec/jexec.c.orig 2009-08-03 10:13:06.000000000 +0200 +++ usr.sbin/jexec/jexec.c 2011-08-11 14:25:11.000000000 +0200 @@ -37,16 +37,20 @@ #include #include +#include #include #include +#include #include #include #include #include +#include #include #include static void usage(void); +static void newpty(void); #define GET_USER_INFO do { \ pwd =3D getpwnam(username); \ @@ -71,17 +75,17 @@ login_cap_t *lcap =3D NULL; struct passwd *pwd =3D NULL; gid_t *groups =3D NULL; - int ch, ngroups, uflag, Uflag; + int ch, ngroups, uflag, Uflag, tflag; long ngroups_max; char *username; - ch =3D uflag =3D Uflag =3D 0; + ch =3D uflag =3D Uflag =3D tflag =3D 0; username =3D NULL; ngroups_max =3D sysconf(_SC_NGROUPS_MAX) + 1; if ((groups =3D malloc(sizeof(gid_t) * ngroups_max)) =3D=3D NULL) err(1, "malloc"); - while ((ch =3D getopt(argc, argv, "nu:U:")) !=3D -1) { + while ((ch =3D getopt(argc, argv, "nu:U:t")) !=3D -1) { switch (ch) { case 'n': /* Specified name, now unused */ @@ -94,6 +98,9 @@ username =3D optarg; Uflag =3D 1; break; + case 't': + tflag =3D 1; + break; default: usage(); } @@ -125,6 +132,8 @@ err(1, "setusercontext"); login_close(lcap); } + if (tflag) + newpty(); if (execvp(argv[1], argv + 1) =3D=3D -1) err(1, "execvp(): %s", argv[1]); exit(0); @@ -135,6 +144,68 @@ { fprintf(stderr, "%s\n", - "usage: jexec [-u username | -U username] jail command ..."= ); + "usage: jexec [-u username | -U username] [-t] jail command ..."); exit(1); } + +static void +newpty(void) +{ + int master, slave, child, n, cc; + fd_set rfd; + struct termios tt, rtt; + struct winsize win; + char obuf[BUFSIZ]; + char ibuf[BUFSIZ]; + + if (tcgetattr(STDIN_FILENO, &tt) =3D=3D -1) + err(1, "tcgetattr"); + if (ioctl(STDIN_FILENO, TIOCGWINSZ, &win) =3D=3D -1) + err(1, "ioctl"); + if (openpty(&master, &slave, NULL, &tt, &win) =3D=3D -1) + err(1, "openpty"); + rtt =3D tt; + cfmakeraw(&rtt); + rtt.c_lflag &=3D ~ECHO; + tcsetattr(STDIN_FILENO, TCSAFLUSH, &rtt); + + child =3D fork(); + if (child < 0) + err(1, "fork"); + if (child =3D=3D 0) { + close(master); + login_tty(slave); + dup2(slave, STDIN_FILENO); + dup2(slave, STDOUT_FILENO); + dup2(slave, STDERR_FILENO); + close(slave); + return; + } + FD_ZERO(&rfd); + for (;;) { + FD_SET(master, &rfd); + FD_SET(STDIN_FILENO, &rfd); + n =3D select(master + 1, &rfd, 0, 0, NULL); + if (n < 0 && errno !=3D EINTR) + break; + if (n > 0 && FD_ISSET(STDIN_FILENO, &rfd)) { + cc =3D read(STDIN_FILENO, ibuf, BUFSIZ); + if (cc < 0) + break; + if (cc =3D=3D 0) + write(master, ibuf, 0); + if (cc > 0) { + write(master, ibuf, cc); + } + } + if (n > 0 && FD_ISSET(master, &rfd)) { + cc =3D read(master, obuf, sizeof (obuf)); + if (cc <=3D 0) + break; + write(STDOUT_FILENO, obuf, cc); + } + } + tcsetattr(STDIN_FILENO, TCSAFLUSH, &tt); + exit(0); +} + > >> >> regards >> Joris >> >>> >>> I understand that this is not trivial but given the number of threads >>> about this problem, it's a real issue to many people. =A0To me it's wor= th >>> some $ or EUR to solve this in a clean way. >>> >>> Kind regards, >>> >>> Paul Schenkeveld >>> _______________________________________________ >>> freebsd-jail@freebsd.org mailing list >>> http://lists.freebsd.org/mailman/listinfo/freebsd-jail >>> To unsubscribe, send any mail to "freebsd-jail-unsubscribe@freebsd.org" >>> >> > From owner-freebsd-jail@FreeBSD.ORG Thu Aug 11 14:19:14 2011 Return-Path: Delivered-To: freebsd-jail@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7760B106564A for ; Thu, 11 Aug 2011 14:19:14 +0000 (UTC) (envelope-from freebsd@psconsult.nl) Received: from mx1.psconsult.nl (mx1.psconsult.nl [80.89.238.138]) by mx1.freebsd.org (Postfix) with ESMTP id 046888FC0C for ; Thu, 11 Aug 2011 14:19:13 +0000 (UTC) Received: from mx1.psconsult.nl ([80.89.238.138]) by mx1.psconsult.nl (8.14.4/8.14.4) with ESMTP id p7BEJ7Tg083608 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Thu, 11 Aug 2011 16:19:12 +0200 (CEST) (envelope-from freebsd@psconsult.nl) Received: (from paul@localhost) by mx1.psconsult.nl (8.14.4/8.14.4/Submit) id p7BEJ7j2083607; Thu, 11 Aug 2011 16:19:07 +0200 (CEST) (envelope-from freebsd@psconsult.nl) X-Authentication-Warning: mx1.psconsult.nl: paul set sender to freebsd@psconsult.nl using -f Date: Thu, 11 Aug 2011 16:19:07 +0200 From: Paul Schenkeveld To: joris dedieu Message-ID: <20110811141907.GA62023@psconsult.nl> References: <20110809124543.GA22077@psconsult.nl> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-jail@freebsd.org Subject: Re: Jexec and access to tty X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 11 Aug 2011 14:19:14 -0000 On Thu, Aug 11, 2011 at 02:34:01PM +0200, joris dedieu wrote: > 2011/8/11 joris dedieu : > > 2011/8/10 joris dedieu : > >> 2011/8/9 Paul Schenkeveld : > >>> Hi, > >>> > >>> There have been several threads about this issue, some people have come > >>> up with work arounds but I think that the issue is more fundamental, > >>> that's why I wanted to start this new thread. > >>> > >>> When using jexec to do interactive work inside an existing jail, people > >>> find out that they no longer have access to their tty device.  As a > >>> result, programs requiring input of passwords or passphrases behave > >>> unexpectedly in one of several ways. > >>> > >>> Ssh says "Host key verification failed." and refuses to log in to > >>> another system (unless pubkey authentication is user in combination with > >>> an agent of course).  Some programs fall back to using stdin/stdout > >>> and echo the password as it is typed (the mysql clients are popular > >>> examples). > >>> > >>> Work-arounds that have been suggested are > >>>  1. Run a sshd inside the jail and log in using ssh > >>>  2. Start tmux inside the jail so you get a new pseudo tty slave inside > >>>    the jail.  People trying screen find that it won't work unlike tmux. > >>>  3. I tried using 'script -q /dev/null' inside the jail because it is > >>>    part op the base system and it doesn't change your terminal type > >>>    and interpret keyboard input and screen output.  I found out that I > >>>    failed when I resized my window :-( > >> > >> An other way is to use chroot(5) to enter the jail. > >> Maybe chroot /jail/root login -f $USER should be acceptable in some situations. > >> > >>> > >>> I don't like 1 on a machine with many jails, especially if some of them > >>> share the same IP address (e.g. sometimes I have to run a mail server on > >>> the same IP adress as a webserver but in a distinct jail). > >>> > >>> 2 is not ideal either because tmux emulates a different terminal on > >>> the inside than the terminal on the outside that it runs on. > >>> > >>> 3 is really a kludge and causes problems when you resize your window. > >>> > >>> I thought that I found a solution by rewriting jexec such that it will > >>> open a pseudo tty and does the passing of data between the jailed pts > >>> and the tty from where jexec was started but that's not going to work as > >>> the pseudo tty most be opened by the child process inside the jail but > >>> the parent outside the jail must have access to the master side of the > >>> pseudo tty. > >>> > >>> So far we are still talking about work-arounds.  Why not look at the > >>> root cause.  Unfortunately I'm not familiar with kernel sources so if > >>> I'm wrong, please forgive me, I write this with the best intentions. > >>> > >>> The root cause of th problem appears to be that pseudo ttys opened > >>> outside a jail are not visible nor accessible inside a jail, pseudo ttys > >>> created inside a jail are visible and accessible though. > >> > >> As far as I understand, sys/fs/devfs/devfs_vnops.c uses > >> prison_check(9) too see if an item as been build in the same jail or > >> in a child. > >> The tty exists inside the jail but you can't use it (and so you can't > >> escape the jail). > >> > >>> > >>> Would it be conceivable that by using jexec the controlling tty of jexec > >>> magically becomes visible and accessible inside the jail?  Preferrable > >>> only until jexec dies. > >> > >> I'm not sure. The only way should be  to temporary disable the check > >> with a  variable. But it will also brake jail security during jexec > >> excution. > >> Using chroot(2) instead of jail(2) should be an option (but it's  non > >> trivial to affect jail context for all other subsystems). > >> > >> I think the only right way is to open a new tty while entering the > >> jail (this is what tmux or a jailed ssh does). > >> But it should be difficult to make things like echo ps | jexec 1 sh works. > > > > I gave this way a try and quickly produced a dirty patch. I did not > > yet investigate on the right way, but I already know that I'm wrong on > > several items so sorry for your eyes :) > > > > Basically this patch  introduced  -t option that forks jexec inside > > the jail, open a new pts and tries to establish a communication with > > the old one. > > It mostly works has expected but breaks some console inputs like > > control commands, completion and so on. > > Before spending more time  trying to produce something acceptable, > > please let  me know if it should be the expected feature or sounds > > like a nasty workaround. In fact this is what I thought of writing some time ago but you open the pseudo tty before jailing the child so the problem remains that the slave side of the pseudo tty cannot be reopened from within the jail. In other words, if tty(1) returns "/dev/pts/1" while inside the jail, the command "echo foo > /dev/pts/1" replies "cannot create /dev/pts/1: No such file or directory". If you open the pseudo tty after the child process is jailed, the parent process has no access to the master side of the pseudo tty. Perhaps having a UNIX domain socket between parent and child and send the file desciptor of the master side of the pseudo tty back from child to parent could work but we're still looking at a work-around and I believe it would be fixable in devfs because the child of jexec which is in the jail has the slave side of the pseudo tty as controlling terminal so that could be checked in devfs to decide whether /dev/pts/X should be visible and accessible to a process inside the jail. Wish I were familiar with FreeBSD kernel code :-( Kind regards, Paul Schenkeveld