From owner-freebsd-jail@freebsd.org Mon Feb 13 20:14:17 2017 Return-Path: Delivered-To: freebsd-jail@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 15AD9CDDA69 for ; Mon, 13 Feb 2017 20:14:17 +0000 (UTC) (envelope-from prvs=3217cfba6c=jocelynh@fb.com) Received: from mx0a-00082601.pphosted.com (mx0b-00082601.pphosted.com [67.231.153.30]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id C0A5B10FC for ; Mon, 13 Feb 2017 20:14:15 +0000 (UTC) (envelope-from prvs=3217cfba6c=jocelynh@fb.com) Received: from pps.filterd (m0089730.ppops.net [127.0.0.1]) by m0089730.ppops.net (8.16.0.20/8.16.0.20) with SMTP id v1DK9I6n010923 for ; Mon, 13 Feb 2017 12:14:09 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : subject : date : message-id : content-type : mime-version; s=facebook; bh=LW0vHACpv1NjjzM6jv4GOjuOBUhhM65Df1YcORe7XlQ=; b=SNdFSyga2IVtmr6xuXQqihyl26FqBm26dVs+JmRAADCW7K1Qg3blbV5rBzh724CU5TjY 5JvsCq+8cwpDW51cdawut4IS+CXWpJ+de0AElwIRrZwDQCohZqxIQTpJYDRjX/PR8Szi /IIRLp3639/WJIU75q1SDq6UZucahMUSzUE= Received: from maileast.thefacebook.com ([199.201.65.23]) by m0089730.ppops.net with ESMTP id 28kemqs4pj-1 (version=TLSv1 cipher=ECDHE-RSA-AES256-SHA bits=256 verify=NOT) for ; Mon, 13 Feb 2017 12:14:09 -0800 Received: from NAM03-CO1-obe.outbound.protection.outlook.com (192.168.183.28) by o365-in.thefacebook.com (192.168.177.29) with Microsoft SMTP Server (TLS) id 14.3.294.0; Mon, 13 Feb 2017 15:14:08 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.onmicrosoft.com; s=selector1-fb-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=LW0vHACpv1NjjzM6jv4GOjuOBUhhM65Df1YcORe7XlQ=; b=VdhNY31NUROYrTJLLmlubpyPPGshDIgXnqFnN/eeHEx5hbiZ64/92HJvBaKp0MD25rwZwORyqSF0RNJFRWHBejMOm+ZL/QTKSZgueoRXVhqlCx8w0K77NJ70+xu3FnrrGARaV2n1Av/atq/TDu3KMOIlWCLQdKBWrc7GakFEw8A= Received: from SN1PR15MB0335.namprd15.prod.outlook.com (10.162.107.18) by SN1PR15MB0336.namprd15.prod.outlook.com (10.162.107.19) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.888.16; Mon, 13 Feb 2017 20:13:39 +0000 Received: from SN1PR15MB0335.namprd15.prod.outlook.com ([10.162.107.18]) by SN1PR15MB0335.namprd15.prod.outlook.com ([10.162.107.18]) with mapi id 15.01.0888.030; Mon, 13 Feb 2017 20:13:38 +0000 From: Jocelyn Hoffman To: "freebsd-jail@freebsd.org" Subject: $ perl -e 'print "Hello from Facebook!"' Thread-Topic: $ perl -e 'print "Hello from Facebook!"' Thread-Index: AdKGNaFkoivbYNGAQMy16O4OkufuhA== Date: Mon, 13 Feb 2017 20:13:38 +0000 Message-ID: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: yes X-MS-TNEF-Correlator: x-originating-ip: [2620:10d:c090:200::a:878d] x-ms-office365-filtering-correlation-id: b5c95c25-b817-497a-1e18-08d4544ccc5a x-microsoft-antispam: UriScan:;BCL:0;PCL:0;RULEID:(22001);SRVR:SN1PR15MB0336; x-microsoft-exchange-diagnostics: 1; SN1PR15MB0336; 7:Eff3dg11qXwAbb89dDZEztLxijTiQq5BDvU+pWtZbNs26C84fHumFlhUypCtwnD0t55WC8Btjltx9cyZaIThLw/OvYq2A5/XWvdybKZlcMo4EU6RhZWY1NTvfbX6ey0JazWXJNC7cvOeiRpzMI9cBXrhvJTNPvLdrLF3MvvzfzXdJymLCeBBZORT78OVl57ewjQXirBKjnZf2YOqMnq/ludrrLQ3ub+V/HIwNNR1Rka2mfJ7dC7Tj2myoxEI3iZTxWMAMj1zx3YLdaTIAAgEAVlrcs48/AEm7gbd6zTkECyrJIOvnYbSK9SS9u0HttJ+xrPvkdgzFm9U5hsMk16TLqFOM9HUCAR0txmaQLY/gFzLkqZbvoYxy7nB1fki+clSDnbRIeGK8h5H0ArevSPJ/gKEkJw27w4MTWvw+iBsRerVj4Mi2FSaPx/m+BvQhvD9bV7tSEGTqXoXQtIOApM1F9O3n7JSlmTzA54Wu/D6XqRiai+vc5gssp0MG2iz/1+6V6G2nSl8qXrhY7KFEAxtag==; 20:4ix+t0SuMlT8oXeUozNRgTH+6ATgcFKjfk0djTJNhXyq86dYWqUr0i9Z0alHkvQEBTrGNtkAUDuBdQTmifElPU45T8DshJrlcbmAUnep4vLpGlNVUHmcL7UBernBnN8Spww46/Skwcf2bCcA4K5eNzrHC47wpyfNU5s1nkMdMX0= x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:(209352067349851)(86561027422486)(81227570615382)(21748063052155)(64217206974132); x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(102415395)(6040375)(601004)(2401047)(8121501046)(5005006)(10201501046)(3002001)(6041248)(20161123564025)(20161123560025)(20161123562025)(20161123555025)(20161123558025)(6072148); SRVR:SN1PR15MB0336; BCL:0; PCL:0; RULEID:; SRVR:SN1PR15MB0336; x-forefront-prvs: 02176E2458 x-forefront-antispam-report: SFV:SPM; SFS:(10019020)(7916002)(39450400003)(199003)(189002)(51414003)(54556002)(110136004)(6916009)(450100001)(236005)(25786008)(55016002)(6116002)(102836003)(5640700003)(790700001)(33656002)(2906002)(77096006)(6506006)(733005)(606005)(9686003)(38730400002)(54896002)(2501003)(68736007)(8936002)(122556002)(86362001)(99286003)(6436002)(8676002)(9326002)(7696004)(81166006)(7906003)(861006)(81156014)(3280700002)(6306002)(97736004)(2351001)(74316002)(7736002)(3660700001)(106356001)(105586002)(101416001)(5660300001)(50986999)(92566002)(189998001)(99936001)(5630700001)(53936002)(54356999)(2900100001)(7099028)(351534003)(430054002); DIR:OUT; SFP:1501; SCL:5; SRVR:SN1PR15MB0336; H:SN1PR15MB0335.namprd15.prod.outlook.com; FPR:; SPF:None; PTR:InfoNoRecords; A:1; MX:1; received-spf: None (protection.outlook.com: fb.com does not designate permitted sender hosts) spamdiagnosticoutput: 1:22 MIME-Version: 1.0 X-MS-Exchange-CrossTenant-originalarrivaltime: 13 Feb 2017 20:13:38.7667 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 8ae927fe-1255-47a7-a2af-5f3a069daaa2 X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN1PR15MB0336 X-OriginatorOrg: fb.com X-Proofpoint-Spam-Reason: safe X-FB-Internal: Safe X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:, , definitions=2017-02-13_10:, , signatures=0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 13 Feb 2017 20:14:17 -0000 Hi Jamie, I came across your profile online and I wanted to check and see if you migh= t be interested in learning more about the Production Engineering team at F= acebook. We are currently hiring full time for our Menlo Park CA, Seattle W= A, and New York City, NY office locations. We're seeking a hybrid Systems / Software Engineer to help develop our syst= ems infrastructure. These Engineers are typically embedded with different d= evelopment teams and work to build the infrastructure that supports our ser= vices and ensure they are highly scalable within our production environment= . It's all on a massive scale and there is a huge emphasis on scalability a= nd reliability, as you can imagine. I thought your experience in Linux/Unix= systems and Perl development would make you a great fit for the role! I've pasted a link to the job description below if you'd like to take a loo= k, and I'd be happy to set up some time to speak to discuss the role furthe= r. I look forward to hearing back. Best, Jocelyn Job Description: https://www.facebook.com/careers/jobs/a2KA0000000LknwMAC/ Video: https://www.facebook.com/facebookcareers/videos/1747855735501113/ Cool P.E Development: https://code.facebook.com/posts/156810174519680/making-facebook-self-healin= g/?__mref=3Dmessage_bubble ________________________________ Jocelyn Hoffman Technical Sourcer | Production Engineering M: (650) 485-9154 [cid:image001.jpg@01D285F2.9B141490] facebook.com/careers From owner-freebsd-jail@freebsd.org Wed Feb 15 01:48:32 2017 Return-Path: Delivered-To: freebsd-jail@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D80A3CDFCA2 for ; Wed, 15 Feb 2017 01:48:32 +0000 (UTC) (envelope-from jmk@wagsky.com) Received: from bmx.allycomm.com (bmx.allycomm.com [198.199.108.230]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id CB9959F8 for ; Wed, 15 Feb 2017 01:48:32 +0000 (UTC) (envelope-from jmk@wagsky.com) Received: from jkletsky1-mbp15.guidewire.com (inet.guidewire.com [199.91.42.30]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by bmx.allycomm.com (Postfix) with ESMTPSA id D5FC8D7C9B; Tue, 14 Feb 2017 17:48:01 -0800 (PST) To: freebsd-jail@freebsd.org From: Jeff Kletsky Subject: Using jail.conf array parameters in exec.* commands Message-ID: Date: Tue, 14 Feb 2017 17:48:01 -0800 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:45.0) Gecko/20100101 Thunderbird/45.7.1 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 15 Feb 2017 01:48:32 -0000 TL;DR Is there a patch available to allow substitution of "array" parameters into the strings used for exec.prestart, exec.poststop, and the like? Longer: I'd like to be able to use array parameters in exec.* commands, but trying to do so results in jail: test-two: exec.prestart: array cannot be substituted inline A quick check of the source didn't suggest it would work, but the bash-isms ${host.interface[*]} and ${host.interface[@]} failed too. The immediate application is in creating what will be the vnet.interface(s) using netgraph on the appropriate host interfaces vnet.interface = ng0, ng1; Works well, once the interfaces are created, but I'd rather not have to define that list twice in each jail. If I could write something like: vnet.interface = ng0, ng1; host.interface = re0, re0.100; and then iterate over them in the prestart and poststop scripts to: * Create ng0 connected to re0 * Create ng1 connected to re0.100 it would be easier to maintain than having to write something like: vnet.interface = ng0, ng1; parent.interfaces = "re0 re0.100"; cloned.interfaces = "ng0 ng1"; I'm open to ideas here, including if I can somehow "write back" to vnet.interface based on exec.prestart parsing the "custom" variables. TIA, Jeff From owner-freebsd-jail@freebsd.org Wed Feb 15 10:30:30 2017 Return-Path: Delivered-To: freebsd-jail@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 70ED5CDF667 for ; Wed, 15 Feb 2017 10:30:30 +0000 (UTC) (envelope-from luzar722@gmail.com) Received: from mail-it0-x22d.google.com (mail-it0-x22d.google.com [IPv6:2607:f8b0:4001:c0b::22d]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 2D9971029 for ; Wed, 15 Feb 2017 10:30:30 +0000 (UTC) (envelope-from luzar722@gmail.com) Received: by mail-it0-x22d.google.com with SMTP id 203so62205326ith.0 for ; Wed, 15 Feb 2017 02:30:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:date:from:user-agent:mime-version:to:cc:subject :references:in-reply-to:content-transfer-encoding; bh=hEgS+MjT3PzWT+GM5duUhAvMK2MQ+v+75erez3yOcfk=; b=r2AhQ2Po+eueBeIq+5x9Ch9LcCvpk9bjgki2KeZjItz6vhl3VQSt67pQoUXoqob2XN 5E1P2LMFy1tY9wj2VBaQZp/P8HxbUhU305uLG5yP3SG6FaYwm+3ZVfwbwX0lnCFBZT42 U2lq9u8tiiVwU7UWijAbRWY5pZMkyyQ3/CwCswRC4T7QZ5Q2a6aUy1lxveM7lQ+hw9pf lph3Ux1BqF00C6W/piAE75rWvf27bKHMq6bjwsUGwR4pSXNrGRmq37iLnAauav2I81aM NJi8i+yrNJhoVoPHm0w1mQJ6huay05FHLyFbYw9ajs2AEtDcNjwlcSJIAOR8lLcrXViw SD6A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:date:from:user-agent:mime-version:to :cc:subject:references:in-reply-to:content-transfer-encoding; bh=hEgS+MjT3PzWT+GM5duUhAvMK2MQ+v+75erez3yOcfk=; b=Ct6b/stXUz8ptnR5NoUOwuIhjlv9mc50U5iyrZZhnLda8UyneCwDCxhtADJJlmhBs9 JimmiTFs8fhNz+/39u0Fj5cOc9NNxN/HzgCtzc/Z2UtsZggqwHLebYPbRabjxGIx7Q9Z sIuXQe/dWQNG6NGFngjo+VaaCVHvG6t4ok+mBodltv9eJKOk1YDXY/q5/H68mB6gaQj0 N4X3MdNBiDEHLFl5vplZ2bfgM3mfp8Zwhn4u1C3GUWfKC0aErOZt+8nTcr13ovWqMLIC lafwQyKlp/VT1MAGN+Q+MCjNAkoKMyYaCGf1qqP0TopZzZEEo829EJ3nMIyp9MP02ysC 916w== X-Gm-Message-State: AMke39lu1e04ZjmhUjvpMnctkAPXyjdqYemCD3OG6QzkN6PbIv4o/JKRvCvBQSW6tHvL6Q== X-Received: by 10.36.252.65 with SMTP id b62mr7816664ith.38.1487154629468; Wed, 15 Feb 2017 02:30:29 -0800 (PST) Received: from [10.0.10.3] (cpe-24-165-207-226.neo.res.rr.com. [24.165.207.226]) by smtp.googlemail.com with ESMTPSA id a4sm1561491ioa.43.2017.02.15.02.30.28 (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Wed, 15 Feb 2017 02:30:28 -0800 (PST) Message-ID: <58A42DC7.5040702@gmail.com> Date: Wed, 15 Feb 2017 05:30:31 -0500 From: Ernie Luzar User-Agent: Thunderbird 2.0.0.24 (Windows/20100228) MIME-Version: 1.0 To: Jeff Kletsky CC: freebsd-jail@freebsd.org Subject: Re: Using jail.conf array parameters in exec.* commands References: In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 15 Feb 2017 10:30:30 -0000 Jeff Kletsky wrote: > TL;DR > > Is there a patch available to allow substitution of "array" parameters > into the strings used for exec.prestart, exec.poststop, and the like? > > > Longer: > > I'd like to be able to use array parameters in exec.* commands, > but trying to do so results in > > jail: test-two: exec.prestart: array cannot be substituted inline > > A quick check of the source didn't suggest it would work, but the > bash-isms ${host.interface[*]} and ${host.interface[@]} failed too. > > > The immediate application is in creating what will be the vnet.interface(s) > using netgraph on the appropriate host interfaces > > vnet.interface = ng0, ng1; > > Works well, once the interfaces are created, but I'd rather not have > to define that list twice in each jail. > > > If I could write something like: > > vnet.interface = ng0, ng1; > host.interface = re0, re0.100; > > and then iterate over them in the prestart and poststop scripts to: > > * Create ng0 connected to re0 > * Create ng1 connected to re0.100 > > it would be easier to maintain than having to write something like: > > vnet.interface = ng0, ng1; > parent.interfaces = "re0 re0.100"; > cloned.interfaces = "ng0 ng1"; > > > > I'm open to ideas here, including if I can somehow "write back" to > vnet.interface based on exec.prestart parsing the "custom" variables. > > > TIA, > > Jeff > An alternate method to coding the jail.conf vnet.interface parameter is to use the "ifconfig vnet" command to enable it and "ifconfig -vnet" command to disable it in your netgraph script that starts and stops the vnet jail. Doing so would eliminate your current desire for array processing in the jail.conf definition all together. I use the bridge/epair method my self because its so much easier to understand. If you don't mind sharing, I sure would like to see your netgraph script for vnet jail control once you get it working. From owner-freebsd-jail@freebsd.org Fri Feb 17 17:58:57 2017 Return-Path: Delivered-To: freebsd-jail@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4850ECE3599 for ; Fri, 17 Feb 2017 17:58:57 +0000 (UTC) (envelope-from jmk@wagsky.com) Received: from bmx.allycomm.com (bmx.allycomm.com [198.199.108.230]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 3A5521288 for ; Fri, 17 Feb 2017 17:58:56 +0000 (UTC) (envelope-from jmk@wagsky.com) Received: from JKLETSKY1-MBP15.local (184-23-191-243.vpn.dynamic.sonic.net [184.23.191.243]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by bmx.allycomm.com (Postfix) with ESMTPSA id 8F59DD7A3A; Fri, 17 Feb 2017 09:58:49 -0800 (PST) Subject: Re: Using jail.conf array parameters in exec.* commands To: Ernie Luzar , freebsd-jail@freebsd.org References: <58A42DC7.5040702@gmail.com> From: Jeff Kletsky Message-ID: <5c11e326-cd4b-73e1-a681-9d116a0c1cd3@wagsky.com> Date: Fri, 17 Feb 2017 09:58:48 -0800 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:45.0) Gecko/20100101 Thunderbird/45.7.1 MIME-Version: 1.0 In-Reply-To: <58A42DC7.5040702@gmail.com> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 17 Feb 2017 17:58:57 -0000 Thanks for the suggestion of trying to use 'ifconfig interface vnet jail' in the scripts themselves. I'll get my scripts up once I've got them running again confidently and can get proper licensing on them. TL;DR * Is there a clean way to "catch" failures in jail(8) creation after exec.prestart completes, such as vnet.interface failing? * Is there a good way to execute commands in the host environment once jail(8) brings up the jail, but before exec.start runs? The rest: I've been thinking about that for a while, especially as there isn't a way to "catch" an execution error in jail(8) itself, such as the vnet transition failing. (Yes, I'll open an issue on that once I'm convinced I can't do it with the current jail functionality.) To be able to call 'ifconfig interface vnet jail' the jail needs to exist already: # ifconfig ngeth3 vnet t2 ifconfig: jail "t2" not found Further, the network needs to be up and running when services are started. ntpd, anything that binds to a specific interface (rather than *), anything that needs DNS (such as nginx providing proxy services), ... jail(8) tells me I have the following hooks available exec.prestart -- jail isn't created yet exec.start -- runs *in* the jail; typically starts execution exec.poststart -- runs in the host, after exec.start completes There isn't a "jail up, but not executing yet" hook in the host environment that I am aware of. There is a somewhat ugly approach along the lines of: exec.prestart -- do the setup on the host side exec.start -- '/bin/true' or 'return 0'-- don't do anything exec.poststart -- 'ifconfig interface vnet jail'-like things 'jexec jail sh /etc/rc > ${exec.consolelog}' Is there a better approach that someone out there knows of? Thanks! Jeff On 2/15/17 2:30 AM, Ernie Luzar wrote: > Jeff Kletsky wrote: >> TL;DR >> >> Is there a patch available to allow substitution of "array" parameters >> into the strings used for exec.prestart, exec.poststop, and the like? >> >> [...] > An alternate method to coding the jail.conf vnet.interface parameter is to use the "ifconfig vnet" command to enable it and "ifconfig -vnet" command to disable it in your netgraph script that starts and stops the vnet jail. Doing so would eliminate your current desire for array processing in the jail.conf definition all together. > > I use the bridge/epair method my self because its so much easier to understand. If you don't mind sharing, I sure would like to see your netgraph script for vnet jail control once you get it working From owner-freebsd-jail@freebsd.org Fri Feb 17 23:01:02 2017 Return-Path: Delivered-To: freebsd-jail@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3EE32CE2E0C for ; Fri, 17 Feb 2017 23:01:02 +0000 (UTC) (envelope-from luzar722@gmail.com) Received: from mail-io0-x22a.google.com (mail-io0-x22a.google.com [IPv6:2607:f8b0:4001:c06::22a]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 064C516C7 for ; Fri, 17 Feb 2017 23:01:02 +0000 (UTC) (envelope-from luzar722@gmail.com) Received: by mail-io0-x22a.google.com with SMTP id j18so2968269ioe.2 for ; Fri, 17 Feb 2017 15:01:02 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:date:from:user-agent:mime-version:to:cc:subject :references:in-reply-to:content-transfer-encoding; bh=NYE4rD+179kMY6hqGUOsIxFNrzy/ujJMSC2JrjgFRLs=; b=NP8eK6sKBcv0GxmAZGmO50mUBLCleTP2V5kwGeaqGtwrkKYsRzF7J7Zs96CmucYkso psKFIE4w2bM8ftnbhiRaz9yGQxNdLyZlsmuzt6pR+FAodpj2Tqvs5d1f9opQeirKKtiA +5Ck9r9N0arL2njrIVnV5Yhwb+bvAeyo0EvZG12rC142Xvk+d2f5xvFlr5MLIHrBLZiK xz0h8U36QYMHiqPCUrf2EahP8cCsyLGfLUb4/jKn0Qd9lrRv31h1vSAA9ELXZ4WD/b7i kwf9plFuhPYWrC+MizO7AntayhUwBkMYe2tq9spnVa9uDhm9OGv2X9VvSTUM/b41nmCU f8sQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:date:from:user-agent:mime-version:to :cc:subject:references:in-reply-to:content-transfer-encoding; bh=NYE4rD+179kMY6hqGUOsIxFNrzy/ujJMSC2JrjgFRLs=; b=gvYE1bWHvcRhet4e7JZE2uuibV82mFCVys2bliY+UZEZLp1wS0vqVmGhU/Z2vo19F6 auv6A+tALZwhhTDXnI2w5Cjkbzz5oqMEX6s+Mt7pMo47D9CWLwspZM1LnTjWj26mxH3b mAIN87tOYxX8CK+PAeKKDBaR1XNTZgIkLK6kPHo4s6NV4EocNQ833qUTPF+BKwIyM3he PdsqSe/7wJwwDM93OPRveOLzp6827JDhP5TZfZhhsR8U0Yw1vbtV8Fwu77z+55W4ZHu+ HkWO6OyH/jQdxv4pCKTDePeLfupB56EOitpvkg/6Wa4HZ9INgxAk2rerkTvkRju4grVF YNiQ== X-Gm-Message-State: AMke39nq0jIomcFct5pLQ2V8g6VuWvUH9ZUWYHOPpTFhQve40DYqLKGywJ2LuinFZ8HKYw== X-Received: by 10.107.19.9 with SMTP id b9mr8961966ioj.48.1487372461328; Fri, 17 Feb 2017 15:01:01 -0800 (PST) Received: from [10.0.10.3] (cpe-24-165-207-226.neo.res.rr.com. [24.165.207.226]) by smtp.googlemail.com with ESMTPSA id d5sm1382187itd.3.2017.02.17.15.01.00 (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Fri, 17 Feb 2017 15:01:00 -0800 (PST) Message-ID: <58A780C4.6030503@gmail.com> Date: Fri, 17 Feb 2017 18:01:24 -0500 From: Ernie Luzar User-Agent: Thunderbird 2.0.0.24 (Windows/20100228) MIME-Version: 1.0 To: Jeff Kletsky CC: freebsd-jail@freebsd.org Subject: Re: Using jail.conf array parameters in exec.* commands References: <58A42DC7.5040702@gmail.com> <5c11e326-cd4b-73e1-a681-9d116a0c1cd3@wagsky.com> In-Reply-To: <5c11e326-cd4b-73e1-a681-9d116a0c1cd3@wagsky.com> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 17 Feb 2017 23:01:02 -0000 Jeff Kletsky wrote: > Thanks for the suggestion of trying to use 'ifconfig interface vnet jail' > in the scripts themselves. > > I'll get my scripts up once I've got them running again confidently > and can get proper licensing on them. > > TL;DR > > * Is there a clean way to "catch" failures in jail(8) creation after > exec.prestart completes, such as vnet.interface failing? > > * Is there a good way to execute commands in the host environment once > jail(8) brings up the jail, but before exec.start runs? > > > > The rest: > > I've been thinking about that for a while, especially as there isn't a > way to "catch" an execution error in jail(8) itself, such as the vnet > transition failing. (Yes, I'll open an issue on that once I'm convinced > I can't do it with the current jail functionality.) > > To be able to call 'ifconfig interface vnet jail' the jail needs to > exist already: > > # ifconfig ngeth3 vnet t2 > ifconfig: jail "t2" not found > > Further, the network needs to be up and running when services are > started. ntpd, anything that binds to a specific interface (rather > than *), anything that needs DNS (such as nginx providing proxy > services), ... > > > jail(8) tells me I have the following hooks available > > exec.prestart -- jail isn't created yet > exec.start -- runs *in* the jail; typically starts execution > exec.poststart -- runs in the host, after exec.start completes > > There isn't a "jail up, but not executing yet" hook in the host > environment that I am aware of. > > There is a somewhat ugly approach along the lines of: > > exec.prestart -- do the setup on the host side > exec.start -- '/bin/true' or 'return 0'-- don't do anything > exec.poststart -- 'ifconfig interface vnet jail'-like things > 'jexec jail sh /etc/rc > ${exec.consolelog}' > > > Is there a better approach that someone out there knows of? > > > Thanks! > > Jeff > Lets make this simple. Do not use the "service jail jailname start" command to start / stop your jails. Your mixing legacy rc.conf jail method with jail.conf method. All ways use the jail(8) command itself to start/stop your jails. If you do this in a script then you can check the jail resulting return code to determine if the jail start/stop failed. But there is no information to tell you why it failed. In all most all cases it's caused by jail.conf parameters syntax coding error or invalid value content. Really pretty simple to determine cause by looking at the jail.conf content for the offending vnet jail. Change your mind set from thinking you have to use the exec.* hooks to configure the vnet jails netgraph network setup. Just have individual jail.conf files for each vnet jail with no vnet interface defined. Now you can start the jail with just the standard exec.start line and standard exec.stop line. Once your script has issued the jail(8) command to start the jail then follow it with all the netgraph commands to enable its network. The vnet jail it self has no knowledge of any network connectivity at start up, you can wrap either bridge/epair or netgraph around it and it don't care. This was learned the hard way. From owner-freebsd-jail@freebsd.org Fri Feb 17 23:57:14 2017 Return-Path: Delivered-To: freebsd-jail@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id DE7BDCE3D5B for ; Fri, 17 Feb 2017 23:57:14 +0000 (UTC) (envelope-from jmk@wagsky.com) Received: from bmx.allycomm.com (bmx.allycomm.com [198.199.108.230]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id C2B441706 for ; Fri, 17 Feb 2017 23:57:14 +0000 (UTC) (envelope-from jmk@wagsky.com) Received: from JKLETSKY1-MBP15.local (184-23-191-243.vpn.dynamic.sonic.net [184.23.191.243]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by bmx.allycomm.com (Postfix) with ESMTPSA id 6D810D7A6C; Fri, 17 Feb 2017 15:57:11 -0800 (PST) Subject: Re: Using jail.conf array parameters in exec.* commands To: Ernie Luzar , freebsd-jail@freebsd.org References: <58A42DC7.5040702@gmail.com> <5c11e326-cd4b-73e1-a681-9d116a0c1cd3@wagsky.com> <58A780C4.6030503@gmail.com> From: Jeff Kletsky Message-ID: <052f45c5-a808-724c-90d5-1b7464e9a585@wagsky.com> Date: Fri, 17 Feb 2017 15:57:10 -0800 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:45.0) Gecko/20100101 Thunderbird/45.7.1 MIME-Version: 1.0 In-Reply-To: <58A780C4.6030503@gmail.com> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 17 Feb 2017 23:57:15 -0000 Thanks again for your thoughts on this. I *am* using "pure" jail(8) and jail.conf(5) techniques and have been for many years now. What I'd like to get to is a robust way to start jails the way I have been # jail -c some_jail and just have it work reliably, especially when there is setup that needs to be done on the host to enable the smooth running of the jail. Preferably, *all* configuration of the jail and its connectivity is done in jail.conf, be it /etc/jail.conf or a jail-specific one. I'm trying to avoid going back to having to define another service that wraps a call to jail(8) which would just start the jail with persist set, then have to set up networking in the wrapper, jexec /etc/rc, and then deal with shutdown of a jail created with persist set. I agree that *usually* the reason a jail won't start is misconfiguration. However, there are other conditions that can occur, such as lack of resources. If you want to more gracefully respond to this, the wrapper script would need either to parse the appropriate jail.conf, or to have all the pertinent information available in another form. Having individual jail.conf files for each jail at least makes easier to parse, at the expense of not being able to define global and regional jail properties that are inherited across the appropriate jails. The network has to be up and connected *before* /etc/rc runs, especially where services in the jail need network interfaces present to bind to specific addresses, to mount network file systems, or have access to critical services, such as DNS (for example, nginx will fail to start if it can't resolve proxy host names). Since there is no jail vnet or jail ID (number) available, you can't have jail(8) run needed operations in the jail.conf-declared exec.prestart command, including, for example: * ifconfig interface vnet jail * ipfw add action proto from src to dst jail prisonID * ipfw add lookup jail table_name By the time jail(8) will run the jail.conf declared exec.poststart, exec.start has already run to completion inside the jail. How do you handle getting the network up *before* /etc/rc or the specific service is started in the jail? I unfortunately suspect you're right that I can't use the existing jail(8) and jail.conf(5) approach without wrapping the whole thing in a script. The hooks, even for networking, don't seem to be there. Jeff On 2/17/17 3:01 PM, Ernie Luzar wrote: >Lets make this simple. Do not use the "service jail jailname start" >command to start / stop your jails. >Your mixing legacy rc.conf jail method with jail.conf method. All >ways use the jail(8) command itself to start/stop your jails. If you >do this in a script then you can check the jail resulting return code >to determine if the jail start/stop failed. But there is no >information to tell you why it failed. In all most all cases it's >caused by jail.conf parameters syntax coding error or invalid value >content. Really pretty simple to determine cause by looking at the >jail.conf content for the offending vnet jail. >Change your mind set from thinking you have to use the exec.* hooks >to configure the vnet jails netgraph network setup. >Just have individual jail.conf files for each vnet jail with no vnet >interface defined. >Now you can start the jail with just the standard exec.start line and >standard exec.stop line. Once your script has issued the jail(8) >command to start the jail then follow it with all the netgraph >commands to enable its network. The vnet jail it self has no >knowledge of any network connectivity at start up, you can wrap >either bridge/epair or netgraph around it and it don't care. >This was learned the hard way. > Jeff Kletsky wrote: >> Thanks for the suggestion of trying to use 'ifconfig interface vnet jail' >> in the scripts themselves. >> >> I'll get my scripts up once I've got them running again confidently >> and can get proper licensing on them. >> >> TL;DR >> >> * Is there a clean way to "catch" failures in jail(8) creation after >> exec.prestart completes, such as vnet.interface failing? >> >> * Is there a good way to execute commands in the host environment once >> jail(8) brings up the jail, but before exec.start runs? >> [...] From owner-freebsd-jail@freebsd.org Sat Feb 18 17:18:48 2017 Return-Path: Delivered-To: freebsd-jail@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 44D0FCE46FA for ; Sat, 18 Feb 2017 17:18:48 +0000 (UTC) (envelope-from jamie@freebsd.org) Received: from gritton.org (gritton.org [199.192.165.131]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "gritton.org", Issuer "Let's Encrypt Authority X3" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 0F1101119 for ; Sat, 18 Feb 2017 17:18:47 +0000 (UTC) (envelope-from jamie@freebsd.org) Received: from gritton.org (gritton.org [199.192.165.131]) by gritton.org (8.15.2/8.15.2) with ESMTP id v1IGqHYG043476; Sat, 18 Feb 2017 09:52:17 -0700 (MST) (envelope-from jamie@freebsd.org) Received: (from www@localhost) by gritton.org (8.15.2/8.15.2/Submit) id v1IGqHE8043475; Sat, 18 Feb 2017 09:52:17 -0700 (MST) (envelope-from jamie@freebsd.org) X-Authentication-Warning: gritton.org: www set sender to jamie@freebsd.org using -f To: Jeff Kletsky Subject: Re: Using jail.conf array parameters in exec.* commands X-PHP-Originating-Script: 0:rcube.php MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII; format=flowed Content-Transfer-Encoding: 7bit Date: Sat, 18 Feb 2017 09:52:17 -0700 From: James Gritton Cc: freebsd-jail@freebsd.org In-Reply-To: <5c11e326-cd4b-73e1-a681-9d116a0c1cd3@wagsky.com> References: <58A42DC7.5040702@gmail.com> <5c11e326-cd4b-73e1-a681-9d116a0c1cd3@wagsky.com> Message-ID: <804a21e85046d5c9f536c7faefc0fb05@freebsd.org> X-Sender: jamie@freebsd.org User-Agent: Roundcube Webmail/1.2.3 X-Greylist: inspected by milter-greylist-4.6.2 (gritton.org [199.192.165.131]); Sat, 18 Feb 2017 09:52:17 -0700 (MST) for IP:'199.192.165.131' DOMAIN:'gritton.org' HELO:'gritton.org' FROM:'jamie@freebsd.org' RCPT:'' X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.6.2 (gritton.org [199.192.165.131]); Sat, 18 Feb 2017 09:52:17 -0700 (MST) X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 18 Feb 2017 17:18:48 -0000 On 2017-02-17 10:58, Jeff Kletsky wrote: > ... > I've been thinking about that for a while, especially as there isn't a > way to "catch" an execution error in jail(8) itself, such as the vnet > transition failing. (Yes, I'll open an issue on that once I'm convinced > I can't do it with the current jail functionality.) > > To be able to call 'ifconfig interface vnet jail' the jail needs to > exist already: > > # ifconfig ngeth3 vnet t2 > ifconfig: jail "t2" not found > > Further, the network needs to be up and running when services are > started. ntpd, anything that binds to a specific interface (rather > than *), anything that needs DNS (such as nginx providing proxy > services), ... > > > jail(8) tells me I have the following hooks available > > exec.prestart -- jail isn't created yet > exec.start -- runs *in* the jail; typically starts execution > exec.poststart -- runs in the host, after exec.start completes > > There isn't a "jail up, but not executing yet" hook in the host > environment that I am aware of. > > There is a somewhat ugly approach along the lines of: > > exec.prestart -- do the setup on the host side > exec.start -- '/bin/true' or 'return 0'-- don't do anything > exec.poststart -- 'ifconfig interface vnet jail'-like things > 'jexec jail sh /etc/rc > ${exec.consolelog}' > > > Is there a better approach that someone out there knows of? There's nothing better at this point - the ugly solution you mention is the current best way. The exec.* options come from analogs of the rc-script days, which precede vnet. The specific "ifconfig interface vnet jail" thing was handled by the vnet.interface parameter, but it would be good to have a more general set of exec scripts to run in the create side post-create but pre-start. But I'm not sure such a thing will appear. Aside from the cumbersome naming of something between prestart and start, I can see this blowing up: there could well be a situation where you want something run in the host, something in the jail, something else in the host, something else in the jail. I considered vnet.interface to be the common case, but there will always be more specific work where the best solution is to just run a script on the host side. - Jamie From owner-freebsd-jail@freebsd.org Sat Feb 18 17:38:05 2017 Return-Path: Delivered-To: freebsd-jail@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 50CE3CE4BD6 for ; Sat, 18 Feb 2017 17:38:05 +0000 (UTC) (envelope-from luzar722@gmail.com) Received: from mail-it0-x22c.google.com (mail-it0-x22c.google.com [IPv6:2607:f8b0:4001:c0b::22c]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 17AE21889 for ; Sat, 18 Feb 2017 17:38:05 +0000 (UTC) (envelope-from luzar722@gmail.com) Received: by mail-it0-x22c.google.com with SMTP id h10so44416847ith.1 for ; Sat, 18 Feb 2017 09:38:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:date:from:user-agent:mime-version:to:cc:subject :references:in-reply-to:content-transfer-encoding; bh=yh1kdmFEWE42UQdwJfSWa45a/blaxRmdg99Lbu4QQbU=; b=nEpowGYW/J/AbbPevg/z1h4MeSbdRDnYFsrbJkCR+A4Ct0yXknScbd6wJEcAU9UrTY mpngkq22ZG2xdHGgcBarNt3/BeSGwyHIzn70C89nAATk6dco5ojZQRztzc8oGuVzSIue Kfq3LeKXL15FJ9Ctaa8ZMW8OraQ3Yker3+Us4+J1UkBGrFBc4WkjgMlY6zkIb9jIP7Tr 5oUN4C1rt2uOydLdSGbBc9b5UcffpPm9FHEdtUtBXTyslIRN6h9n4Ad9lvZAkCkAB3hF nM+0JcZCzqHgSpK8SC4rQRzjj/PmYyi2AqihoUMVIRmB0RtOz95/+jGlyM5hvaq8pm0U EjwQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:date:from:user-agent:mime-version:to :cc:subject:references:in-reply-to:content-transfer-encoding; bh=yh1kdmFEWE42UQdwJfSWa45a/blaxRmdg99Lbu4QQbU=; b=dI8dDSARhr4ryURmEWBp96Wfung4oZGY1hVsaTEHVT394C+xqb2gmdMj1MQ0udo2Ea Ex7gi+/+5zyT4/JR2/XdjtC3v4kABgJjOcfwY5OS5xSCEeeHSosdvSv7Rhfr7aSfMdhz wVDPqAYapBw3WZPgITjEq5com9de13PczgN6b9jzY++5J00f0beBOvmXJQlo9SCWMgIM oZx2ddaCnJo8EvDiFxVqjFo8+PgVxPlNIGcvyHn1ojJNz7XdqzKZM49HzbZXhWuV8LtN GB7nhOsRMVD2VSEZS+VhQihpn/0jI2ltJSmslFEPfG/t9HML03FD2KUSaxczLh7G88us +liA== X-Gm-Message-State: AMke39kg/pZYF+Enn+tdtAuaJAskOtsIFeK9jZ0rmsnYzNiniVX1pktEufavtMAFuJkcxg== X-Received: by 10.36.81.142 with SMTP id s136mr4749103ita.119.1487439484458; Sat, 18 Feb 2017 09:38:04 -0800 (PST) Received: from [10.0.10.3] (cpe-24-165-207-226.neo.res.rr.com. [24.165.207.226]) by smtp.googlemail.com with ESMTPSA id x197sm2586071ite.17.2017.02.18.09.38.03 (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Sat, 18 Feb 2017 09:38:04 -0800 (PST) Message-ID: <58A88689.9030607@gmail.com> Date: Sat, 18 Feb 2017 12:38:17 -0500 From: Ernie Luzar User-Agent: Thunderbird 2.0.0.24 (Windows/20100228) MIME-Version: 1.0 To: Jeff Kletsky CC: freebsd-jail@freebsd.org Subject: Re: Using jail.conf array parameters in exec.* commands References: <58A42DC7.5040702@gmail.com> <5c11e326-cd4b-73e1-a681-9d116a0c1cd3@wagsky.com> <58A780C4.6030503@gmail.com> <052f45c5-a808-724c-90d5-1b7464e9a585@wagsky.com> In-Reply-To: <052f45c5-a808-724c-90d5-1b7464e9a585@wagsky.com> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 18 Feb 2017 17:38:05 -0000 On release 10.3 or maybe 10.2 there were Devin Teske's examples of vnet/netgraph setups in these directories. /usr/share/examples/netgraph /usr/share/examples/jails 11.0 does not have them any more. If I recall correctly, some of the netgraph commands need the jid number, which means the vnet jail has to be started. But here "started" does not have the normal meaning. A vnet jail can be created, jail -c, and as long as there is no "exec.start sh /etc/rc" command in the jail.conf definition there is nothing running in the jail. Now your script has a jid to use and can setup the netghaph world that will allow the vnet jail to commutate. After all that stuff is complete, a jexec command can be issued to start services in the jail. "sh /etc/rc" is all you need to fire up services in the vnet jail. Any userland application like apache have to be installed the normal way and have its apache24_enable="YES" statement id the vnet's jail rc.conf file. Here's a general observation. During the 10.x releases I had a netgraph vnet script that Devin Teske posted to questions or jail list that worked. I think it was the same one that was in 10.x /usr/share/examples/jails directory. But when I tried it on 11.0 it stopped functioning giving a error on some netctl command. For me netgraph is just to hard. That is why I use the bridge/epair method. Jeff Kletsky wrote: > Thanks again for your thoughts on this. > > I *am* using "pure" jail(8) and jail.conf(5) techniques and have been > for many years now. What I'd like to get to is a robust way to start > jails the way I have been > > # jail -c some_jail > > and just have it work reliably, especially when there is setup that > needs to be done on the host to enable the smooth running of the jail. > > Preferably, *all* configuration of the jail and its connectivity is > done in jail.conf, be it /etc/jail.conf or a jail-specific one. I'm > trying to avoid going back to having to define another service that > wraps a call to jail(8) which would just start the jail with persist > set, then have to set up networking in the wrapper, jexec /etc/rc, > and then deal with shutdown of a jail created with persist set. > > I agree that *usually* the reason a jail won't start is > misconfiguration. However, there are other conditions that can occur, > such as lack of resources. If you want to more gracefully respond to > this, the wrapper script would need either to parse the appropriate > jail.conf, or to have all the pertinent information available in > another form. Having individual jail.conf files for each jail at > least makes easier to parse, at the expense of not being able to > define global and regional jail properties that are inherited across > the appropriate jails. > > > The network has to be up and connected *before* /etc/rc runs, > especially where services in the jail need network interfaces present > to bind to specific addresses, to mount network file systems, or have > access to critical services, such as DNS (for example, nginx will fail > to start if it can't resolve proxy host names). > > Since there is no jail vnet or jail ID (number) available, you can't > have jail(8) run needed operations in the jail.conf-declared > exec.prestart command, including, for example: > > * ifconfig interface vnet jail > * ipfw add action proto from src to dst jail prisonID > * ipfw add lookup jail table_name > > By the time jail(8) will run the jail.conf declared exec.poststart, > exec.start has already run to completion inside the jail. > > > How do you handle getting the network up *before* /etc/rc or the > specific service is started in the jail? > > > I unfortunately suspect you're right that I can't use the existing > jail(8) and jail.conf(5) approach without wrapping the whole thing in > a script. The hooks, even for networking, don't seem to be there. > > > Jeff > > > > On 2/17/17 3:01 PM, Ernie Luzar wrote: > > >Lets make this simple. Do not use the "service jail jailname start" > >command to start / stop your jails. > > >Your mixing legacy rc.conf jail method with jail.conf method. All > >ways use the jail(8) command itself to start/stop your jails. If you > >do this in a script then you can check the jail resulting return code > >to determine if the jail start/stop failed. But there is no > >information to tell you why it failed. In all most all cases it's > >caused by jail.conf parameters syntax coding error or invalid value > >content. Really pretty simple to determine cause by looking at the > >jail.conf content for the offending vnet jail. > > >Change your mind set from thinking you have to use the exec.* hooks > >to configure the vnet jails netgraph network setup. > > >Just have individual jail.conf files for each vnet jail with no vnet > >interface defined. > > >Now you can start the jail with just the standard exec.start line and > >standard exec.stop line. Once your script has issued the jail(8) > >command to start the jail then follow it with all the netgraph > >commands to enable its network. The vnet jail it self has no > >knowledge of any network connectivity at start up, you can wrap > >either bridge/epair or netgraph around it and it don't care. > > >This was learned the hard way. > > > Jeff Kletsky wrote: > >> Thanks for the suggestion of trying to use 'ifconfig interface vnet > jail' > >> in the scripts themselves. > >> > >> I'll get my scripts up once I've got them running again confidently > >> and can get proper licensing on them. > >> > >> TL;DR > >> > >> * Is there a clean way to "catch" failures in jail(8) creation after > >> exec.prestart completes, such as vnet.interface failing? > >> > >> * Is there a good way to execute commands in the host environment once > >> jail(8) brings up the jail, but before exec.start runs? > >> [...] >