From owner-freebsd-jail@freebsd.org Sat Feb 18 17:38:05 2017 Return-Path: Delivered-To: freebsd-jail@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 50CE3CE4BD6 for ; Sat, 18 Feb 2017 17:38:05 +0000 (UTC) (envelope-from luzar722@gmail.com) Received: from mail-it0-x22c.google.com (mail-it0-x22c.google.com [IPv6:2607:f8b0:4001:c0b::22c]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 17AE21889 for ; Sat, 18 Feb 2017 17:38:05 +0000 (UTC) (envelope-from luzar722@gmail.com) Received: by mail-it0-x22c.google.com with SMTP id h10so44416847ith.1 for ; Sat, 18 Feb 2017 09:38:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:date:from:user-agent:mime-version:to:cc:subject :references:in-reply-to:content-transfer-encoding; bh=yh1kdmFEWE42UQdwJfSWa45a/blaxRmdg99Lbu4QQbU=; b=nEpowGYW/J/AbbPevg/z1h4MeSbdRDnYFsrbJkCR+A4Ct0yXknScbd6wJEcAU9UrTY mpngkq22ZG2xdHGgcBarNt3/BeSGwyHIzn70C89nAATk6dco5ojZQRztzc8oGuVzSIue Kfq3LeKXL15FJ9Ctaa8ZMW8OraQ3Yker3+Us4+J1UkBGrFBc4WkjgMlY6zkIb9jIP7Tr 5oUN4C1rt2uOydLdSGbBc9b5UcffpPm9FHEdtUtBXTyslIRN6h9n4Ad9lvZAkCkAB3hF nM+0JcZCzqHgSpK8SC4rQRzjj/PmYyi2AqihoUMVIRmB0RtOz95/+jGlyM5hvaq8pm0U EjwQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:date:from:user-agent:mime-version:to :cc:subject:references:in-reply-to:content-transfer-encoding; bh=yh1kdmFEWE42UQdwJfSWa45a/blaxRmdg99Lbu4QQbU=; b=dI8dDSARhr4ryURmEWBp96Wfung4oZGY1hVsaTEHVT394C+xqb2gmdMj1MQ0udo2Ea Ex7gi+/+5zyT4/JR2/XdjtC3v4kABgJjOcfwY5OS5xSCEeeHSosdvSv7Rhfr7aSfMdhz wVDPqAYapBw3WZPgITjEq5com9de13PczgN6b9jzY++5J00f0beBOvmXJQlo9SCWMgIM oZx2ddaCnJo8EvDiFxVqjFo8+PgVxPlNIGcvyHn1ojJNz7XdqzKZM49HzbZXhWuV8LtN GB7nhOsRMVD2VSEZS+VhQihpn/0jI2ltJSmslFEPfG/t9HML03FD2KUSaxczLh7G88us +liA== X-Gm-Message-State: AMke39kg/pZYF+Enn+tdtAuaJAskOtsIFeK9jZ0rmsnYzNiniVX1pktEufavtMAFuJkcxg== X-Received: by 10.36.81.142 with SMTP id s136mr4749103ita.119.1487439484458; Sat, 18 Feb 2017 09:38:04 -0800 (PST) Received: from [10.0.10.3] (cpe-24-165-207-226.neo.res.rr.com. [24.165.207.226]) by smtp.googlemail.com with ESMTPSA id x197sm2586071ite.17.2017.02.18.09.38.03 (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Sat, 18 Feb 2017 09:38:04 -0800 (PST) Message-ID: <58A88689.9030607@gmail.com> Date: Sat, 18 Feb 2017 12:38:17 -0500 From: Ernie Luzar User-Agent: Thunderbird 2.0.0.24 (Windows/20100228) MIME-Version: 1.0 To: Jeff Kletsky CC: freebsd-jail@freebsd.org Subject: Re: Using jail.conf array parameters in exec.* commands References: <58A42DC7.5040702@gmail.com> <5c11e326-cd4b-73e1-a681-9d116a0c1cd3@wagsky.com> <58A780C4.6030503@gmail.com> <052f45c5-a808-724c-90d5-1b7464e9a585@wagsky.com> In-Reply-To: <052f45c5-a808-724c-90d5-1b7464e9a585@wagsky.com> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 18 Feb 2017 17:38:05 -0000 On release 10.3 or maybe 10.2 there were Devin Teske's examples of vnet/netgraph setups in these directories. /usr/share/examples/netgraph /usr/share/examples/jails 11.0 does not have them any more. If I recall correctly, some of the netgraph commands need the jid number, which means the vnet jail has to be started. But here "started" does not have the normal meaning. A vnet jail can be created, jail -c, and as long as there is no "exec.start sh /etc/rc" command in the jail.conf definition there is nothing running in the jail. Now your script has a jid to use and can setup the netghaph world that will allow the vnet jail to commutate. After all that stuff is complete, a jexec command can be issued to start services in the jail. "sh /etc/rc" is all you need to fire up services in the vnet jail. Any userland application like apache have to be installed the normal way and have its apache24_enable="YES" statement id the vnet's jail rc.conf file. Here's a general observation. During the 10.x releases I had a netgraph vnet script that Devin Teske posted to questions or jail list that worked. I think it was the same one that was in 10.x /usr/share/examples/jails directory. But when I tried it on 11.0 it stopped functioning giving a error on some netctl command. For me netgraph is just to hard. That is why I use the bridge/epair method. Jeff Kletsky wrote: > Thanks again for your thoughts on this. > > I *am* using "pure" jail(8) and jail.conf(5) techniques and have been > for many years now. What I'd like to get to is a robust way to start > jails the way I have been > > # jail -c some_jail > > and just have it work reliably, especially when there is setup that > needs to be done on the host to enable the smooth running of the jail. > > Preferably, *all* configuration of the jail and its connectivity is > done in jail.conf, be it /etc/jail.conf or a jail-specific one. I'm > trying to avoid going back to having to define another service that > wraps a call to jail(8) which would just start the jail with persist > set, then have to set up networking in the wrapper, jexec /etc/rc, > and then deal with shutdown of a jail created with persist set. > > I agree that *usually* the reason a jail won't start is > misconfiguration. However, there are other conditions that can occur, > such as lack of resources. If you want to more gracefully respond to > this, the wrapper script would need either to parse the appropriate > jail.conf, or to have all the pertinent information available in > another form. Having individual jail.conf files for each jail at > least makes easier to parse, at the expense of not being able to > define global and regional jail properties that are inherited across > the appropriate jails. > > > The network has to be up and connected *before* /etc/rc runs, > especially where services in the jail need network interfaces present > to bind to specific addresses, to mount network file systems, or have > access to critical services, such as DNS (for example, nginx will fail > to start if it can't resolve proxy host names). > > Since there is no jail vnet or jail ID (number) available, you can't > have jail(8) run needed operations in the jail.conf-declared > exec.prestart command, including, for example: > > * ifconfig interface vnet jail > * ipfw add action proto from src to dst jail prisonID > * ipfw add lookup jail table_name > > By the time jail(8) will run the jail.conf declared exec.poststart, > exec.start has already run to completion inside the jail. > > > How do you handle getting the network up *before* /etc/rc or the > specific service is started in the jail? > > > I unfortunately suspect you're right that I can't use the existing > jail(8) and jail.conf(5) approach without wrapping the whole thing in > a script. The hooks, even for networking, don't seem to be there. > > > Jeff > > > > On 2/17/17 3:01 PM, Ernie Luzar wrote: > > >Lets make this simple. Do not use the "service jail jailname start" > >command to start / stop your jails. > > >Your mixing legacy rc.conf jail method with jail.conf method. All > >ways use the jail(8) command itself to start/stop your jails. If you > >do this in a script then you can check the jail resulting return code > >to determine if the jail start/stop failed. But there is no > >information to tell you why it failed. In all most all cases it's > >caused by jail.conf parameters syntax coding error or invalid value > >content. Really pretty simple to determine cause by looking at the > >jail.conf content for the offending vnet jail. > > >Change your mind set from thinking you have to use the exec.* hooks > >to configure the vnet jails netgraph network setup. > > >Just have individual jail.conf files for each vnet jail with no vnet > >interface defined. > > >Now you can start the jail with just the standard exec.start line and > >standard exec.stop line. Once your script has issued the jail(8) > >command to start the jail then follow it with all the netgraph > >commands to enable its network. The vnet jail it self has no > >knowledge of any network connectivity at start up, you can wrap > >either bridge/epair or netgraph around it and it don't care. > > >This was learned the hard way. > > > Jeff Kletsky wrote: > >> Thanks for the suggestion of trying to use 'ifconfig interface vnet > jail' > >> in the scripts themselves. > >> > >> I'll get my scripts up once I've got them running again confidently > >> and can get proper licensing on them. > >> > >> TL;DR > >> > >> * Is there a clean way to "catch" failures in jail(8) creation after > >> exec.prestart completes, such as vnet.interface failing? > >> > >> * Is there a good way to execute commands in the host environment once > >> jail(8) brings up the jail, but before exec.start runs? > >> [...] >