From owner-freebsd-jail@freebsd.org Sun May 29 00:16:24 2016 Return-Path: Delivered-To: freebsd-jail@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D1067B450C1 for ; Sun, 29 May 2016 00:16:24 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id C19411C8F for ; Sun, 29 May 2016 00:16:24 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id u4T0GOm7096403 for ; Sun, 29 May 2016 00:16:24 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-jail@FreeBSD.org Subject: [Bug 208001] After turning off the jail does not remove network routes Date: Sun, 29 May 2016 00:16:24 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bin X-Bugzilla-Version: 10.3-BETA2 X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Many People X-Bugzilla-Who: admin@support.od.ua X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-jail@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: bug_status resolution Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 29 May 2016 00:16:24 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D208001 Vladislav V. Prodan changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Closed |Open Resolution|Works As Intended |--- --- Comment #4 from Vladislav V. Prodan --- The problem is not the creation of an IP address, network mask, other than = /32. I can give some examples of network topology, when the IP jail need to assi= gn a subnet mask for the correct operation of the network protocols. For example= , a DHCP-server or Samba services (nmbd and winbindd) The problem is in the removal of a route to this IP when you turn off the j= ail. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-jail@freebsd.org Sun May 29 00:30:46 2016 Return-Path: Delivered-To: freebsd-jail@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6C55FB45673 for ; Sun, 29 May 2016 00:30:46 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 5D14E1779 for ; Sun, 29 May 2016 00:30:46 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id u4T0UkSF028696 for ; Sun, 29 May 2016 00:30:46 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-jail@FreeBSD.org Subject: [Bug 208001] After turning off the jail does not remove network routes Date: Sun, 29 May 2016 00:30:46 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bin X-Bugzilla-Version: 10.3-BETA2 X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Many People X-Bugzilla-Who: jamie@FreeBSD.org X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-jail@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 29 May 2016 00:30:46 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D208001 --- Comment #5 from Jamie Gritton --- Yes, of course there are cases where something besides a /32 is appropriate= - that is why jail(8) allows that. However, as I mentioned it did appear that you had violated the specification that an alias should be on a non-conflic= ting netmask. The fact remains that I am unable to reproduce your problem. Perhaps I cou= ld if I had your entire configuration - all jails, all other network setup. jail(8) simply calls ifconfig(8) with "alias" to add IP addresses, and with "-alias" to remove them - see the output of "jail -vc" and "jail -vr". The jail will not be removed if the "ifconfig ... -alias" command fails, which implies that the command is succeeding. Unless of course there actually is= a bug in the way jail(8) is running this program. My guess is the command is succeeding, but isn't removing some arp entry because the alias when incorrectly specified when it was created. If it's clear (from "jail -v") that the correct ifconfig commands are being run, then this might be considered an ifconfig bug. If the correct commands aren't being run, then it could be a jail bug. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-jail@freebsd.org Sun May 29 01:56:27 2016 Return-Path: Delivered-To: freebsd-jail@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id F2698B52F8D for ; Sun, 29 May 2016 01:56:27 +0000 (UTC) (envelope-from allanjude@freebsd.org) Received: from mx1.scaleengine.net (mx1.scaleengine.net [209.51.186.6]) by mx1.freebsd.org (Postfix) with ESMTP id D7DC016B6 for ; Sun, 29 May 2016 01:56:27 +0000 (UTC) (envelope-from allanjude@freebsd.org) Received: from [10.1.1.2] (unknown [10.1.1.2]) (Authenticated sender: allanjude.freebsd@scaleengine.com) by mx1.scaleengine.net (Postfix) with ESMTPSA id E4BEADA3F for ; Sun, 29 May 2016 01:56:20 +0000 (UTC) Subject: Re: [Bug 208001] After turning off the jail does not remove network routes To: freebsd-jail@freebsd.org References: From: Allan Jude Message-ID: <8a575b8b-e9e9-d79c-0b31-708e7bbd35fd@freebsd.org> Date: Sat, 28 May 2016 21:56:20 -0400 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.1.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 29 May 2016 01:56:28 -0000 On 2016-05-28 20:30, bugzilla-noreply@freebsd.org wrote: > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=208001 > > --- Comment #5 from Jamie Gritton --- > Yes, of course there are cases where something besides a /32 is appropriate - > that is why jail(8) allows that. However, as I mentioned it did appear that > you had violated the specification that an alias should be on a non-conflicting > netmask. > > The fact remains that I am unable to reproduce your problem. Perhaps I could > if I had your entire configuration - all jails, all other network setup. > > jail(8) simply calls ifconfig(8) with "alias" to add IP addresses, and with > "-alias" to remove them - see the output of "jail -vc" and "jail -vr". The > jail will not be removed if the "ifconfig ... -alias" command fails, which > implies that the command is succeeding. Unless of course there actually is a > bug in the way jail(8) is running this program. My guess is the command is > succeeding, but isn't removing some arp entry because the alias when > incorrectly specified when it was created. > > If it's clear (from "jail -v") that the correct ifconfig commands are being > run, then this might be considered an ifconfig bug. If the correct commands > aren't being run, then it could be a jail bug. > I think that is actually the problem ifconfig -alias only accepts the IP address, not with the CIDR. #ifconfig lo0 alias 10.0.0.1/24 #ifconfig lo0 -alias 10.0.0.1/24 ifconfig: 10.0.0.1/24: bad value you want to do just: #ifconfig lo0 -alias 10.0.0.1 So jail(8) needs to strip the /24 off when passing it to ifconfig -alias -- Allan Jude From owner-freebsd-jail@freebsd.org Sun May 29 14:51:50 2016 Return-Path: Delivered-To: freebsd-jail@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9A85EB532F2 for ; Sun, 29 May 2016 14:51:50 +0000 (UTC) (envelope-from jamie@freebsd.org) Received: from gritton.org (gritton.org [162.220.209.3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "www.gritton.org", Issuer "StartCom Class 1 Primary Intermediate Server CA" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 79B2D1DBC; Sun, 29 May 2016 14:51:50 +0000 (UTC) (envelope-from jamie@freebsd.org) Received: from gritton.org (gritton.org [162.220.209.3]) by gritton.org (8.15.2/8.15.2) with ESMTPS id u4TETk9E018006 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Sun, 29 May 2016 08:29:46 -0600 (MDT) (envelope-from jamie@freebsd.org) Received: (from www@localhost) by gritton.org (8.15.2/8.15.2/Submit) id u4TETjR1018005; Sun, 29 May 2016 08:29:45 -0600 (MDT) (envelope-from jamie@freebsd.org) X-Authentication-Warning: gritton.org: www set sender to jamie@freebsd.org using -f To: freebsd-jail@freebsd.org Subject: Re: [Bug 208001] After turning off the jail does not remove network routes X-PHP-Originating-Script: 0:rcube.php MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII; format=flowed Content-Transfer-Encoding: 7bit Date: Sun, 29 May 2016 08:29:45 -0600 From: James Gritton In-Reply-To: <8a575b8b-e9e9-d79c-0b31-708e7bbd35fd@freebsd.org> References: <8a575b8b-e9e9-d79c-0b31-708e7bbd35fd@freebsd.org> Message-ID: <22f599502bd9a932ae41ddb5e70164fa@gritton.org> X-Sender: jamie@freebsd.org User-Agent: Roundcube Webmail/1.1.2 X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 29 May 2016 14:51:50 -0000 On 2016-05-28 19:56, Allan Jude wrote: > On 2016-05-28 20:30, bugzilla-noreply@freebsd.org wrote: >> https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=208001 >> >> --- Comment #5 from Jamie Gritton --- >> Yes, of course there are cases where something besides a /32 is >> appropriate - >> that is why jail(8) allows that. However, as I mentioned it did >> appear that >> you had violated the specification that an alias should be on a >> non-conflicting >> netmask. >> >> The fact remains that I am unable to reproduce your problem. Perhaps >> I could >> if I had your entire configuration - all jails, all other network >> setup. >> >> jail(8) simply calls ifconfig(8) with "alias" to add IP addresses, and >> with >> "-alias" to remove them - see the output of "jail -vc" and "jail -vr". >> The >> jail will not be removed if the "ifconfig ... -alias" command fails, >> which >> implies that the command is succeeding. Unless of course there >> actually is a >> bug in the way jail(8) is running this program. My guess is the >> command is >> succeeding, but isn't removing some arp entry because the alias when >> incorrectly specified when it was created. >> >> If it's clear (from "jail -v") that the correct ifconfig commands are >> being >> run, then this might be considered an ifconfig bug. If the correct >> commands >> aren't being run, then it could be a jail bug. >> > > > I think that is actually the problem > > ifconfig -alias > only accepts the IP address, not with the CIDR. > > #ifconfig lo0 alias 10.0.0.1/24 > #ifconfig lo0 -alias 10.0.0.1/24 > ifconfig: 10.0.0.1/24: bad value > > you want to do just: > #ifconfig lo0 -alias 10.0.0.1 > > So jail(8) needs to strip the /24 off when passing it to ifconfig > -alias Actually is doesn't. While your "-alias" command doesn't work, the one that jail uses does: #ifconfig lo0 inet 10.0.0.1/24 -alias At first I thought it was the "inet" that did it. But further exploration suggests there's something magic about moving the "-alias" to the end. It doesn't make sense, and if I had first tried it with the "[-]alias" tag earlier on the command line I probably would have ended up working out the netmask myself. Serendipity. - Jamie From owner-freebsd-jail@freebsd.org Sun May 29 15:18:28 2016 Return-Path: Delivered-To: freebsd-jail@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 969A1B53925 for ; Sun, 29 May 2016 15:18:28 +0000 (UTC) (envelope-from juanperiz@yahoo.com.ar) Received: from nm50-vm1.bullet.mail.gq1.yahoo.com (nm50-vm1.bullet.mail.gq1.yahoo.com [67.195.87.241]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 713C31D8F for ; Sun, 29 May 2016 15:18:27 +0000 (UTC) (envelope-from juanperiz@yahoo.com.ar) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com.ar; s=s2048; t=1464535101; bh=ana10InN59pWiUlA8ZYy48PLl3kWmlMOHdgLbmfWFVw=; h=Date:From:Reply-To:To:Subject:References:From:Subject; b=KG8Ng/ZG+8R3owVyAmkF0NwwY9+AzHdP5p18gGpwBOq9AIRNYEwrTj9NQ0qvom5A3+rG/88wBFtOid3+SPeXXMII4b/tJadvptrVtP77RBRvW2XxQdwXjKiSGQf4MEpaVLPHMb8qPdZjudGTgyfGPRPW2HXimPqpiqPqpcjqz9oC8Pq0QmjKr7t/+F0YdWUHeW5ayGBkTMqCVQz+PW5O6OBD9vl8LBsCDNbd0ZvC07iv727oQoOzhzlKGGw/DkVFf28LJ/TRBTdqpo2wfdPtCUxInKtM3ccmmhzs4hoamP2jdShEv4RvAJQciYp6XZ4Yi5nFJ/DjKbE3si2vBBUeHw== Received: from [127.0.0.1] by nm50.bullet.mail.gq1.yahoo.com with NNFMP; 29 May 2016 15:18:21 -0000 Received: from [98.137.12.191] by nm50.bullet.mail.gq1.yahoo.com with NNFMP; 29 May 2016 15:15:34 -0000 Received: from [98.137.12.214] by tm12.bullet.mail.gq1.yahoo.com with NNFMP; 29 May 2016 15:15:34 -0000 Received: from [127.0.0.1] by omp1022.mail.gq1.yahoo.com with NNFMP; 29 May 2016 15:15:34 -0000 X-Yahoo-Newman-Property: ymail-4 X-Yahoo-Newman-Id: 697821.47183.bm@omp1022.mail.gq1.yahoo.com X-YMail-OSG: 0CG6cj0VM1n8ai7F8PqzTWJd3Zy_b4KLJy3qmCo6yDnIoik_P.o7ENfqyN4oUOi i79eZdR9y.3qhfDqtzoQGXjuglla7Dw_MOJy6mY8yHYk.Ug.JNIl13dweeJNTew_fR6y42u4.4wa mnk9rY7Io4jqMX8praNFxg2gytwX97dssZfYBCSxRgYV81hlPzwlkEHqGJ7lsqatBBmMfeJ9z2zY TtVAW4LeyDyf.b0zIEZjNs0f.5uNlMtKyt8W8QXHVWp3pHb9SXYntLwCg8nj6.IH.J5I54GLPo1t xDbN3LmFS0SywJftTZYgHb_HiDp8.dc9_.KF73ILPgBJ0axsdOISpBO.XSHZpqXadoWCmNlddCON uJzz1DKsBcp3ZcG9RPUmb3e8XSxck8ajKsB5B_XiNtfE2KYhnCws0GZG5jEG6h5ktayHoAh78pXM WNFOw.JPlR3eKluK080bFO2.XxD6ai9VF_xuZLu1rfWXRY8sjlSYnR12o4wWpRVsv7i08Q2_DuJF j6Qfvg2tjVPAb3qW4yVzkoOk- Received: from jws10714.mail.gq1.yahoo.com by sendmailws140.mail.gq1.yahoo.com; Sun, 29 May 2016 15:15:34 +0000; 1464534934.216 Date: Sun, 29 May 2016 15:15:33 +0000 (UTC) From: =?UTF-8?Q?Sebasti=C3=A1n_Maruca?= Reply-To: =?UTF-8?Q?Sebasti=C3=A1n_Maruca?= To: Message-ID: <366569840.1294540.1464534933908.JavaMail.yahoo@mail.yahoo.com> Subject: deploy multiple vnets with VIMAGE/VNET + Production Ready? MIME-Version: 1.0 References: <366569840.1294540.1464534933908.JavaMail.yahoo.ref@mail.yahoo.com> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.22 X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 29 May 2016 15:18:28 -0000 Hi to everyone! I want to deploy several "jailed" firewalls, where each one of them would c= ontain at least three multiple virtual interfaces (associated with virtual = internal nets) like "WAN", "LAN" and "DMZ" for example... First *innocent* question (I beg you pardon for my ignorance dealing with j= ails!) Can vnet/vimage help me deploy such a complex jailed environment??? Secod *innocent* question, so far so good, reading at =C2=A0jail manpage (c= irca=C2=A0July 6, 2015/FreeBSD 10.3) it seems VNET/VIMAGE is fully integrat= ed to the FreeBSD kernel, is VNET/VIMAGE ready for production level??? As a side note, at the host level would a be some kind of API/service that = would deal with pfctl in order to rule flows between all of them... Best regards,Seba From owner-freebsd-jail@freebsd.org Sun May 29 22:46:19 2016 Return-Path: Delivered-To: freebsd-jail@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 15CB9B54955 for ; Sun, 29 May 2016 22:46:19 +0000 (UTC) (envelope-from jamie@freebsd.org) Received: from gritton.org (gritton.org [162.220.209.3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "www.gritton.org", Issuer "StartCom Class 1 Primary Intermediate Server CA" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id DD9C812DF for ; Sun, 29 May 2016 22:46:18 +0000 (UTC) (envelope-from jamie@freebsd.org) Received: from gritton.org (gritton.org [162.220.209.3]) by gritton.org (8.15.2/8.15.2) with ESMTPS id u4TMjnoB024266 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Sun, 29 May 2016 16:45:49 -0600 (MDT) (envelope-from jamie@freebsd.org) Received: (from www@localhost) by gritton.org (8.15.2/8.15.2/Submit) id u4TMjnrf024265; Sun, 29 May 2016 16:45:49 -0600 (MDT) (envelope-from jamie@freebsd.org) X-Authentication-Warning: gritton.org: www set sender to jamie@freebsd.org using -f To: freebsd-jail@freebsd.org Subject: Re: deploy multiple vnets with VIMAGE/VNET + Production Ready? X-PHP-Originating-Script: 0:rcube.php MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit Date: Sun, 29 May 2016 16:45:48 -0600 From: James Gritton Cc: =?UTF-8?Q?Sebasti=C3=A1n_Maruca?= In-Reply-To: <366569840.1294540.1464534933908.JavaMail.yahoo@mail.yahoo.com> References: <366569840.1294540.1464534933908.JavaMail.yahoo.ref@mail.yahoo.com> <366569840.1294540.1464534933908.JavaMail.yahoo@mail.yahoo.com> Message-ID: <9796987a0c51b8449065f895c8f00cf8@gritton.org> X-Sender: jamie@freebsd.org User-Agent: Roundcube Webmail/1.1.2 X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 29 May 2016 22:46:19 -0000 On 2016-05-29 09:15, Sebastián Maruca via freebsd-jail wrote: > Hi to everyone! > I want to deploy several "jailed" firewalls, where each one of them > would contain at least three multiple virtual interfaces (associated > with virtual internal nets) like "WAN", "LAN" and "DMZ" for example... > First *innocent* question (I beg you pardon for my ignorance dealing > with jails!) Can vnet/vimage help me deploy such a complex jailed > environment??? Yes, I think that sounds like just the sort of complicated mess that vnet jails are best with. It's all about per-jail virtual interfaces. > Secod *innocent* question, so far so good, reading at  jail manpage > (circa July 6, 2015/FreeBSD 10.3) it seems VNET/VIMAGE is fully > integrated to the FreeBSD kernel, is VNET/VIMAGE ready for production > level??? > As a side note, at the host level would a be some kind of API/service > that would deal with pfctl in order to rule flows between all of > them... That's more of a maybe. There are definitely still outstanding issues in the vimage world, especially regarding pf. I don't use either one myself, so I'm just going by what I see on bug reports and the like. - Jamie From owner-freebsd-jail@freebsd.org Mon May 30 13:40:37 2016 Return-Path: Delivered-To: freebsd-jail@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 775CCB539CE for ; Mon, 30 May 2016 13:40:37 +0000 (UTC) (envelope-from luzar722@gmail.com) Received: from mail-it0-x243.google.com (mail-it0-x243.google.com [IPv6:2607:f8b0:4001:c0b::243]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 3F81E13AD for ; Mon, 30 May 2016 13:40:37 +0000 (UTC) (envelope-from luzar722@gmail.com) Received: by mail-it0-x243.google.com with SMTP id i127so4519898ita.3 for ; Mon, 30 May 2016 06:40:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=message-id:date:from:user-agent:mime-version:to:cc:subject :references:in-reply-to:content-transfer-encoding; bh=3NTkbfX/0F3h41UCkqGHL6eBvliweiWxTVbRFjwRru4=; b=p+oRHP1tWi2tSIDh/bXsBteWdXlYGvsjrCFpJ+XFVn0gmLdHa3jKWweyd/Rven+J9f 2hzErYO25zXPNU4/GuYX54G4FMbmBahbQu95Y8LsbrSG62lMEHDcHyync42hEn7MyErA XgSYlerZoAfWQpZeCF+NlyQyZ4S/tA+sB/0j6GpbPoZdO6cviOY9/XVWow8EoIblfIvl fJUmfvAmXVxXNcI90gRkrqFe7RSDprXsAZjy4JJYl2DjT7XZcHscnLJ80stlxfYUIUL9 AA4eGVYBDtjHrroWCFacHpw9TdC4cIKZPIQS1jQSaZInKQg1z5dtUCMwRx86sPF6dgVW LOdQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:message-id:date:from:user-agent:mime-version:to :cc:subject:references:in-reply-to:content-transfer-encoding; bh=3NTkbfX/0F3h41UCkqGHL6eBvliweiWxTVbRFjwRru4=; b=ZBMmiI3xtMaDBdw6JUV8q6MR/ftoDisAY3GRPC32c93H9sNWO+mNy1k/8PB9yz0z3o uVYhAaUjoCcN170w0wXkuuRbZ7yxcLk57+7gEHiL/ioKn7gOmSFlGRhC2Fa5g0HowqfM u96jbsMo5VuxO8ZaoAo1hu4BHbjXdmuwKqmnYxcjvGct+rmxnGoHqvseeOFIp8FiGa+0 roRhV2J0G+PMqytID4o5v/Tt39PJKGjNxTxmx8VujDKsORWbk3jPKehLE0GBio56vKN4 Y4YSXYpnCF++XPk3svfdFD3OOLlaeq+0IoPv4cOKzSoMcRGMpcfOGsONqESTig4PjuIJ FU7w== X-Gm-Message-State: ALyK8tKOsHLzFXhOEkQzJInS6HYiR7sBtFlrKcDKDY32LbU8arX/nVyl2mS211+VdCGb2w== X-Received: by 10.36.92.199 with SMTP id q190mr7509999itb.25.1464615636580; Mon, 30 May 2016 06:40:36 -0700 (PDT) Received: from [10.0.10.3] (cpe-184-56-210-236.neo.res.rr.com. [184.56.210.236]) by smtp.googlemail.com with ESMTPSA id z138sm7753478itc.2.2016.05.30.06.40.35 (version=TLSv1/SSLv3 cipher=OTHER); Mon, 30 May 2016 06:40:35 -0700 (PDT) Message-ID: <574C42DA.6030101@gmail.com> Date: Mon, 30 May 2016 09:40:42 -0400 From: Ernie Luzar User-Agent: Thunderbird 2.0.0.24 (Windows/20100228) MIME-Version: 1.0 To: =?UTF-8?B?U2ViYXN0acOhbiBNYXJ1Y2E=?= CC: freebsd-jail@freebsd.org, =?UTF-8?B?U2ViYXN0acOhbiBNYXJ1Y2E=?= Subject: Re: deploy multiple vnets with VIMAGE/VNET + Production Ready? References: <366569840.1294540.1464534933908.JavaMail.yahoo.ref@mail.yahoo.com> <366569840.1294540.1464534933908.JavaMail.yahoo@mail.yahoo.com> In-Reply-To: <366569840.1294540.1464534933908.JavaMail.yahoo@mail.yahoo.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 30 May 2016 13:40:37 -0000 Here are the bare truths without any sugar coating. Vimage is officially described as experimental. You have to recompile the kernel to included vimage. Enabling pf or ipf firewalls cause the host to crash. ipfw firewall does not cause a crash but has next to no real life usage on vimage. When stopping vimage jails there is a problem with memory loss. You need a high proficiency in coding netgraph which is used to tie the hosts network to each vimage jail. Needs a public network with multiple static ip address & registered domain names even to test it. A few brave soles have accepted these short comings and have deployed vimage in a production environment with good results so they say, or at best they have not reported any problems. I guess it all depends of what your shop defines "production ready" as. At my shop vimage is NOT considered something management is willing to base the business on. Maybe your shop is different. There are a few write ups about how to configure vet/vimage jails, but their out of date. IE: 8.x & 9.x releases which are at EOL [end of life, unsupported]. The current production version of Freebsd is at 10.3 with 11.0 due out in August. Only know of one utility jail tool that has vnet/vimage function. Try the qjail port, it will shorten your learning curve. Now there is a guy who is patching vimage trying to get it so it can be incorporated into the base kernel. His goal was to get it into release 11.0, but updates to 11.0 source are now suspended until 11.0 is published so thats not going to happen. They sure would not incorporate viamge without a general announcement calling for users to test drive it first. This has not happened yet that I know of. vnet/vimage is like a stand alone computer. You have to login to it to manage any firewall or other system function or port application. This can be done from the host console or over the network. Going down this road will make the shop totally dependent on you and your ability. A mega size pay bump is in your future. The shop will be fubar-ed if you die or get hurt requiring a hospital stay and long recovery. User beware. From owner-freebsd-jail@freebsd.org Mon May 30 18:47:00 2016 Return-Path: Delivered-To: freebsd-jail@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4D174B5450E for ; Mon, 30 May 2016 18:47:00 +0000 (UTC) (envelope-from artemrts@ukr.net) Received: from frv191.fwdcdn.com (frv191.fwdcdn.com [212.42.77.191]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 018F511F1 for ; Mon, 30 May 2016 18:46:59 +0000 (UTC) (envelope-from artemrts@ukr.net) Received: from [10.10.1.23] (helo=frv199.fwdcdn.com) by frv191.fwdcdn.com with esmtp ID 1b7RyU-000Pb9-Lt for freebsd-jail@freebsd.org; Mon, 30 May 2016 21:31:38 +0300 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=ukr.net; s=ffe; h=Content-Transfer-Encoding:Content-Type:MIME-Version:References:In-Reply-To:Message-Id:Cc:To:Subject:From:Date; bh=dt+XmoYZSSky8QgWA2lxqRU8t80Njd1S7gptiOFSv5M=; b=oUhLb3bK0QNolM7Ot0NVIIrnJGblwVhJlFEpUskMQO8kWIN2ob7wr4dE0je+4wS9Ntmw7RuNnMTChV5i6M32Fr9zMzECs19kx+QxXtvmqmdlnuWlu8fR4yn0ewagrtreihTSD47oS+FYL/TB0d0qswszpbmdB7M8X/RMNtuRhkU=; Received: from [10.10.10.34] (helo=frv34.fwdcdn.com) by frv199.fwdcdn.com with smtp ID 1b7RyJ-000PO8-EP for freebsd-jail@freebsd.org; Mon, 30 May 2016 21:31:27 +0300 Date: Mon, 30 May 2016 21:31:27 +0300 From: wishmaster Subject: Re: deploy multiple vnets with VIMAGE/VNET + Production Ready? To: =?iso-8859-1?q?Sebasti=E1n?= Maruca Cc: freebsd-jail@freebsd.org X-Mailer: mail.ukr.net 5.0 Message-Id: <1464632136.649261509.wqj3p1n9@frv34.fwdcdn.com> In-Reply-To: <366569840.1294540.1464534933908.JavaMail.yahoo@mail.yahoo.com> References: <366569840.1294540.1464534933908.JavaMail.yahoo.ref@mail.yahoo.com> <366569840.1294540.1464534933908.JavaMail.yahoo@mail.yahoo.com> X-Reply-Action: reply Received: from artemrts@ukr.net by frv34.fwdcdn.com; Mon, 30 May 2016 21:31:27 +0300 MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: binary Content-Disposition: inline X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 30 May 2016 18:47:00 -0000 Hi, > Hi to everyone! > I want to deploy several "jailed" firewalls, where each one of them would contain at least three multiple virtual interfaces (associated with virtual internal nets) like "WAN", "LAN" and "DMZ" for example... > First *innocent* question (I beg you pardon for my ignorance dealing with jails!) Can vnet/vimage help me deploy such a complex jailed environment??? Yes. If you need help you can email me privately. > Secod *innocent* question, so far so good, reading at jail manpage (circa July 6, 2015/FreeBSD 10.3) it seems VNET/VIMAGE is fully integrated to the FreeBSD kernel, is VNET/VIMAGE ready for production level??? Yes. I have been using vneted Jail from 10.0 in quite complex scenarios. Yes, there are some open issues with vnet (pf, memory leak on stopping jail and so on), but I think in 11-RELEASE this bugs will be fixed. Currently Bjorn Zeeb works on this problems. See https://svnweb.freebsd.org/base/projects/vnet/ But for now, you can safely use vnet. Just use IPFW and do not start/stop jails needlessly. > As a side note, at the host level would a be some kind of API/service that would deal with pfctl in order to rule flows between all of them... > Best regards,Seba -- Vitaliy From owner-freebsd-jail@freebsd.org Tue May 31 02:45:23 2016 Return-Path: Delivered-To: freebsd-jail@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 451C2B55D27 for ; Tue, 31 May 2016 02:45:23 +0000 (UTC) (envelope-from juanperiz@yahoo.com.ar) Received: from nm50-vm3.bullet.mail.gq1.yahoo.com (nm50-vm3.bullet.mail.gq1.yahoo.com [67.195.87.243]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 1550B1A49 for ; Tue, 31 May 2016 02:45:22 +0000 (UTC) (envelope-from juanperiz@yahoo.com.ar) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com.ar; s=s2048; t=1464662721; bh=VGjEwt5TchMtuwZgeRTcll7NAT0wgxmbO9xIvRQ3Tvs=; h=Date:From:Reply-To:To:Cc:In-Reply-To:References:Subject:From:Subject; b=FgqSQ4CsdNwiVaJtvWdg8ssngPRxVdp5lLndBTgNuDTeIT9+TWLKIXXUCt+qf51e0Fn6+Is96hDTkxS0JMLltxlWzYsYuNYMXZ+enB//VdFVF26e6SWhpvpdtnLsdJWDBq6uG+gpb+1hI5VYj4MDF78FEndr8trOJK9CYyo6pAQd+kK1s/bwIag+gBbhxHf49C0JYKHBRujhQzZLS2hJCI01ohCbizHSQcFC3dd3uqND5l15961yaiRIe9zN0DyYvIhK1Cc1nSDEnRz5YefH82HUDwLDlgm3v4XzrT7aDP+3e1OrwkpCnkNweYAWI3+3eC65yCjqTNEN64rZP5Or4Q== Received: from [127.0.0.1] by nm50.bullet.mail.gq1.yahoo.com with NNFMP; 31 May 2016 02:45:21 -0000 Received: from [216.39.60.181] by nm50.bullet.mail.gq1.yahoo.com with NNFMP; 31 May 2016 02:42:38 -0000 Received: from [98.137.12.224] by tm17.bullet.mail.gq1.yahoo.com with NNFMP; 31 May 2016 02:42:38 -0000 Received: from [127.0.0.1] by omp1032.mail.gq1.yahoo.com with NNFMP; 31 May 2016 02:42:38 -0000 X-Yahoo-Newman-Property: ymail-4 X-Yahoo-Newman-Id: 231123.59687.bm@omp1032.mail.gq1.yahoo.com X-YMail-OSG: w3VcaOMVM1mQiR531BAGiJVBT775YdPJzb8QdZbj7kR362FEf5jvur8C4kYdzmF b_qG0eKP0LbICezmBC1VsItxXK1plI64GB68l7CcSm_k7rq6liflcSMEaBrTjRe3M9A2J2FfXYx7 MXR9EtJtkr0qaUiCofYsfXxaeO7lvUbhU4_1Uo78zLJ6U8Tw308Y27EWSNnoDMUvY7Sf9aUHjSRz Uwk0HtCxRgmfK6R59Bgjzd4Dv8JS3ntfNA5Uh8RZW45Z2jIFZu5OUq6pSUpiTEcspg791odRim2_ vSzrrB_p2pxNJBKeH.HhcpZjyOhvAK13YlYP5X2RmijuMHriJPjo3v.t84yLQ8qXVijf.v2n1ABi DxQgVa4Npd_EuU8m.MjENuB0NIwF7TpCvq4qBQpjfERu0zK9KvUTtAukwb8fp3.DdCi6sCGxElC7 e.0I9sKHNYi5V.EBovwCPnXer3UIHTgEzh98VaCwcOLr7eLNRSYqaZwHR0Seshf7VMKb3eUt.cuB kwgBhY2Itc8h6QkgXrC_6 Received: from jws10702.mail.gq1.yahoo.com by sendmailws136.mail.gq1.yahoo.com; Tue, 31 May 2016 02:42:37 +0000; 1464662557.787 Date: Tue, 31 May 2016 02:42:37 +0000 (UTC) From: =?UTF-8?Q?Sebasti=C3=A1n_Maruca?= Reply-To: =?UTF-8?Q?Sebasti=C3=A1n_Maruca?= To: wishmaster , =?UTF-8?Q?Sebasti=C3=A1n_Maruca?= Cc: "freebsd-jail@freebsd.org" Message-ID: <1681543684.2075452.1464662557505.JavaMail.yahoo@mail.yahoo.com> In-Reply-To: <1464632136.649261509.wqj3p1n9@frv34.fwdcdn.com> References: <366569840.1294540.1464534933908.JavaMail.yahoo.ref@mail.yahoo.com> <366569840.1294540.1464534933908.JavaMail.yahoo@mail.yahoo.com> <1464632136.649261509.wqj3p1n9@frv34.fwdcdn.com> Subject: Re: deploy multiple vnets with VIMAGE/VNET + Production Ready? MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.22 X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 31 May 2016 02:45:23 -0000 I thank you all for your fast and kind reply! I was in spite of building some kind of API above pf(4) to let each jail ac= t as a tenant firewall... Maybe I should wait to 11-RELEASE birth to go for= it...=C2=A0 Meanwhile I think I'll get over it with an API/framework that can handle pf= with its anchor files doing basic VLAN acls as a virtual way of achieving = this, aside security concerns... De: wishmaster Para: Sebasti=C3=A1n Maruca =20 CC: freebsd-jail@freebsd.org Enviado: Lunes, 30 de mayo, 2016 15:31:27 Asunto: Re: deploy multiple vnets with VIMAGE/VNET + Production Ready? =20 Hi, > Hi to everyone! > I want to deploy several "jailed" firewalls, where each one of them would= contain at least three multiple virtual interfaces (associated with virtua= l internal nets) like "WAN", "LAN" and "DMZ" for example... > First *innocent* question (I beg you pardon for my ignorance dealing with= jails!) Can vnet/vimage help me deploy such a complex jailed environment??= ? =C2=A0 Yes. If you need help you can email me privately. > Secod *innocent* question, so far so good, reading at jail manpage (circa= July 6, 2015/FreeBSD 10.3) it seems VNET/VIMAGE is fully integrated to the= FreeBSD kernel, is VNET/VIMAGE ready for production level??? =20 Yes. I have been using vneted Jail from 10.0 in quite complex scenarios. Ye= s, there are some open issues with vnet (pf, memory leak on stopping jail a= nd so on), but I think in 11-RELEASE this bugs will be fixed. Currently Bjo= rn Zeeb works on this problems. See https://svnweb.freebsd.org/base/project= s/vnet/ But for now, you can safely use vnet. Just use IPFW and do not start/stop j= ails needlessly. > As a side note, at the host level would a be some kind of API/service tha= t would deal with pfctl in order to rule flows between all of them... > Best regards,Seba -- Vitaliy =20 =20 _______________________________________________ freebsd-jail@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-jail To unsubscribe, send any mail to "freebsd-jail-unsubscribe@freebsd.org" From owner-freebsd-jail@freebsd.org Tue May 31 06:39:37 2016 Return-Path: Delivered-To: freebsd-jail@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id BAFC7B55B46 for ; Tue, 31 May 2016 06:39:37 +0000 (UTC) (envelope-from lars@e-new.0x20.net) Received: from mail.0x20.net (mail.0x20.net [217.69.76.211]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.0x20.net", Issuer "mail.0x20.net" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 46B251772 for ; Tue, 31 May 2016 06:39:36 +0000 (UTC) (envelope-from lars@e-new.0x20.net) Received: from e-new.0x20.net (mail.0x20.net [IPv6:2001:aa8:fffb:1::3]) (using TLSv1 with cipher DHE-RSA-AES128-SHA (128/128 bits)) (No client certificate requested) by mail.0x20.net (Postfix) with ESMTPS id 83B4F6E0081; Tue, 31 May 2016 08:39:33 +0200 (CEST) Received: from e-new.0x20.net (localhost [127.0.0.1]) by e-new.0x20.net (8.14.7/8.14.7) with ESMTP id u4V6dXFn014545; Tue, 31 May 2016 08:39:33 +0200 (CEST) (envelope-from lars@e-new.0x20.net) Received: (from lars@localhost) by e-new.0x20.net (8.14.7/8.14.7/Submit) id u4V6dUMs011238; Tue, 31 May 2016 08:39:30 +0200 (CEST) (envelope-from lars) Date: Tue, 31 May 2016 08:39:30 +0200 From: Lars Engels To: Ernie Luzar Cc: =?utf-8?Q?Sebasti=C3=A1n?= Maruca , freebsd-jail@freebsd.org Subject: Re: deploy multiple vnets with VIMAGE/VNET + Production Ready? Message-ID: <20160531063930.GE15808@e-new.0x20.net> References: <366569840.1294540.1464534933908.JavaMail.yahoo.ref@mail.yahoo.com> <366569840.1294540.1464534933908.JavaMail.yahoo@mail.yahoo.com> <574C42DA.6030101@gmail.com> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="Vx/N56bIaYnO6ICL" Content-Disposition: inline In-Reply-To: <574C42DA.6030101@gmail.com> X-Editor: VIM - Vi IMproved 7.4 X-Operation-System: FreeBSD 8.4-RELEASE-p23 User-Agent: Mutt/1.5.23 (2014-03-12) X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 31 May 2016 06:39:37 -0000 --Vx/N56bIaYnO6ICL Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Mon, May 30, 2016 at 09:40:42AM -0400, Ernie Luzar wrote: > Here are the bare truths without any sugar coating. > Vimage is officially described as experimental. You have to recompile=20 > the kernel to included vimage. Enabling pf or ipf firewalls cause the=20 > host to crash. ipfw firewall does not cause a crash but has next to no=20 > real life usage on vimage. When stopping vimage jails there is a problem= =20 > with memory loss. You need a high proficiency in coding netgraph which=20 > is used to tie the hosts network to each vimage jail. Needs a public=20 > network with multiple static ip address & registered domain names even=20 > to test it. >=20 > A few brave soles have accepted these short comings and have deployed=20 > vimage in a production environment with good results so they say, or at= =20 > best they have not reported any problems. I guess it all depends of what= =20 > your shop defines "production ready" as. At my shop vimage is NOT=20 > considered something management is willing to base the business on.=20 > Maybe your shop is different. >=20 > There are a few write ups about how to configure vet/vimage jails, but=20 > their out of date. IE: 8.x & 9.x releases which are at EOL [end of life,= =20 > unsupported]. The current production version of Freebsd is at 10.3 with= =20 > 11.0 due out in August. Only know of one utility jail tool that has=20 > vnet/vimage function. Try the qjail port, it will shorten your learning= =20 > curve. sysutils/iocage also supports VIMAGE >=20 > Now there is a guy who is patching vimage trying to get it so it can be= =20 > incorporated into the base kernel. His goal was to get it into release=20 > 11.0, but updates to 11.0 source are now suspended until 11.0 is=20 > published so thats not going to happen. They sure would not incorporate= =20 > viamge without a general announcement calling for users to test drive it= =20 > first. This has not happened yet that I know of. You seem to forget that there have been fixes already in HEAD:=20 http://freshbsd.org/search?branch=3DHEAD&project=3Dfreebsd&q=3Dvimage+OR+vn= et --Vx/N56bIaYnO6ICL Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQF8BAEBCgBmBQJXTTGiXxSAAAAAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25zLm9w ZW5wZ3AuZmlmdGhob3JzZW1hbi5uZXQ4RjQwMDE3RTRERjUzMTI1N0FGRTUxNDlF NTRDQjM3RDNBMDg5RDZEAAoJEOVMs306CJ1t7DgIAJEju0efxuggs+whRmfPSRDj nRjtMgZPmMhgtzMaAX2WndlEvgWEXeyrKzCUYDzVRbcPHqYTGRI3NVUTXRd4PGaf /4fwBusT0Uc1X0A5A7opO9Yx/aYkh1cKYYy/yvb/wRBpe81W69h/zAkfv1q0+UxZ FaJWerInmxX9xEg9AVdbtvXLchlpMMG8dgyj6O2HI2YA72ijq//8PT2CdP3HVKu1 sYakwJQOi73wztbgDqtW/uDW34xJDNpIOOiLCM6Fhg4t7RK4sMQ50JeCds1qu+LB CZdH/YVT+VFal9ub5f+0w7to1ORWqO6kqfLCH31o336S17JGLEb6/IytiiaH+uQ= =DxQm -----END PGP SIGNATURE----- --Vx/N56bIaYnO6ICL-- From owner-freebsd-jail@freebsd.org Tue May 31 14:56:57 2016 Return-Path: Delivered-To: freebsd-jail@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C89E5B55D98 for ; Tue, 31 May 2016 14:56:57 +0000 (UTC) (envelope-from juanperiz@yahoo.com.ar) Received: from nm34-vm9.bullet.mail.gq1.yahoo.com (nm34-vm9.bullet.mail.gq1.yahoo.com [98.136.216.138]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 986021F3A for ; Tue, 31 May 2016 14:56:57 +0000 (UTC) (envelope-from juanperiz@yahoo.com.ar) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com.ar; s=s2048; t=1464706611; bh=ROvrW432Q7g5nAvgAok+vsAdpVQVAu9OOimlvCUj9I8=; h=Date:From:Reply-To:To:Cc:In-Reply-To:References:Subject:From:Subject; b=TPzebd/5Mx/4e4aovvmb7/AztntrlAzvYSv3NEQUuWKlM/N1oEGd1eaUpPf8cmL8fblCM+VwoPPzanbS3cH5pXz5W9bDjq7gzGQjcjxFxXLuiV2vgkkfafrZda2VRaxwr2KzwknrUeQ/qnzHaSqFV6Z3zAbxC3qmHv/fzyEwJ4+YO+0alUwdiNrIN4HDjN+f3+6hRPJHKCXbpSKLHGmpWgcsjvXBX/DxZboFLIR4NpmbB4IRkUu72xZ4ixm2T2sKR6hkjraU561Ao0dMIHaZXZLTXlYs4Ps02lu3f3r7rrAFL8umABvI0J//W31qFRunjI4pWK4v6oZDZvcbtR+eNQ== Received: from [127.0.0.1] by nm34.bullet.mail.gq1.yahoo.com with NNFMP; 31 May 2016 14:56:51 -0000 Received: from [98.137.12.188] by nm34.bullet.mail.gq1.yahoo.com with NNFMP; 31 May 2016 14:53:56 -0000 Received: from [98.137.12.195] by tm9.bullet.mail.gq1.yahoo.com with NNFMP; 31 May 2016 14:53:56 -0000 Received: from [127.0.0.1] by omp1003.mail.gq1.yahoo.com with NNFMP; 31 May 2016 14:53:56 -0000 X-Yahoo-Newman-Property: ymail-4 X-Yahoo-Newman-Id: 561416.29980.bm@omp1003.mail.gq1.yahoo.com X-YMail-OSG: niFhgUgVM1mSo8w1tFubQBEq3EGh5sros5_UQYS4zZdzHOSpb3BRCrl2gNmnWQ1 TXxseA3Qp.xxWQ5oQwWwvtqpkBeDj4yGJ3yvFSV9nKDhG.Mj4FNPVMfrL6m0c7k.YyPmBuIhSM.s kuvD6ZKT.VXLhBlAbcICr.ybMx_vgX1MwG5iyI2thWFWbYgSt1CSiKoR_hrVfwYtEGrmqDPPfJ0y FiIqnbIVsWRMqHrVUuEtRUTYtQI4XrenclM7yCgkDaU70HGt9Q0YkwlTUI.PDgUo2j4_9C79YIiR PRJRXbsltKMQZ3DxI7DGEERqanedJKCsPn4NyKrZxe_eYqdts3wOUFBmiirLSCI4NQZfjR2cVfPg sCh7acW4Rlieq0vgIlsthdV9596ohz_QOcvZnkq3VF2nME8ve.bdiXWYSAvhPsirfxd0N1HpemTV yiFZ06.dNaV6H7U0AUieLhvl0ff1kiendWsco_IsRplGhRMtIYXxjqFTCnIDV54ij6MdsaVf7ehI hgB5ByRHsEaISs6IHWPyPJp2ojw-- Received: from jws10738.mail.gq1.yahoo.com by sendmailws139.mail.gq1.yahoo.com; Tue, 31 May 2016 14:53:56 +0000; 1464706436.059 Date: Tue, 31 May 2016 14:53:54 +0000 (UTC) From: =?UTF-8?Q?Sebasti=C3=A1n_Maruca?= Reply-To: =?UTF-8?Q?Sebasti=C3=A1n_Maruca?= To: Lars Engels , Ernie Luzar Cc: "freebsd-jail@freebsd.org" , =?UTF-8?Q?Sebasti=C3=A1n_Maruca?= Message-ID: <1044161792.2386277.1464706434032.JavaMail.yahoo@mail.yahoo.com> In-Reply-To: <20160531063930.GE15808@e-new.0x20.net> References: <366569840.1294540.1464534933908.JavaMail.yahoo.ref@mail.yahoo.com> <366569840.1294540.1464534933908.JavaMail.yahoo@mail.yahoo.com> <574C42DA.6030101@gmail.com> <20160531063930.GE15808@e-new.0x20.net> Subject: Re: deploy multiple vnets with VIMAGE/VNET + Production Ready? MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.22 X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 31 May 2016 14:56:57 -0000 Yeah, guess I'll have to wait till they release the brand-new iocage [iocage is being rewritten in a different language] >>> sysutils/iocage also supports VIMAGE Well, I can give a try to 10.3 Current and see if iocage do the trick... >>> You seem to forget that there have been fixes already in HEAD: http://freshbsd.org/search?branch=HEAD&project=freebsd&q=vimage+OR+vnet But as I said, some kind of API/framework to deal with "virtual isolated" PF(4) anchor files as a way of getting multi-tenant feature of OPNSense... From owner-freebsd-jail@freebsd.org Wed Jun 1 16:07:39 2016 Return-Path: Delivered-To: freebsd-jail@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E95E9B6132D for ; Wed, 1 Jun 2016 16:07:39 +0000 (UTC) (envelope-from marquis@roble.com) Received: from mx5.roble.com (mx5.roble.com [206.40.34.5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mx5.roble.com", Issuer "mx5.roble.com" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id E19481703 for ; Wed, 1 Jun 2016 16:07:39 +0000 (UTC) (envelope-from marquis@roble.com) Date: Wed, 1 Jun 2016 09:07:33 -0700 (PDT) From: Roger Marquis To: freebsd-jail@freebsd.org Subject: Re: deploy multiple vnets with VIMAGE/VNET + Production Ready? MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 01 Jun 2016 16:07:40 -0000 Ernie Luzar wrote: > the kernel to included vimage. Enabling pf or ipf firewalls cause the > host to crash. ipfw firewall does not cause a crash but has next to no > real life usage on vimage. Considering we have had ipfw/vimage/netgraph jails for several years I'd be interested in your data sources. > When stopping vimage jails there is a problem with memory loss. Have you tested this, on a recent release? > You need a high proficiency in coding netgraph which > is used to tie the hosts network to each vimage jail. This certainly used to be true and IMO has been a significant barrier to netgraph usage but the scripts in head/share/examples/jails/ are at least helpful. > Needs a public network with multiple static ip address & registered domain > names even to test it. How are you implementing vimage that needs a registered domain name? > There are a few write ups about how to configure vet/vimage jails, but > their out of date. IE: 8.x & 9.x releases which are at EOL [end of life, > unsupported]. Vimage gets little attention. Unfortunately the mapping of non-vimage localhost interfaces to the primary external interface isn't noted nearly enough either. These are weaknesses in bsd jails, the latter a non-trivial security issue on many non-vimage systems considering daemons like sendmail are installed and listening on "localhost" by default. > Going down this road will make the shop totally dependent on you and your > ability. A mega size pay bump is in your future. The shop will be fubar-ed > if you die or get hurt requiring a hospital stay and long recovery. Potentially true of any Unix or Linux application in my experience. Have you tried vimage with epair/if_bridge instead of netgraph? It's considerably simpler though the documentation is almost as conflicting and insufficient. Roger From owner-freebsd-jail@freebsd.org Wed Jun 1 19:25:41 2016 Return-Path: Delivered-To: freebsd-jail@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7FEE3B61EC4 for ; Wed, 1 Jun 2016 19:25:41 +0000 (UTC) (envelope-from juanperiz@yahoo.com.ar) Received: from nm35-vm7.bullet.mail.gq1.yahoo.com (nm35-vm7.bullet.mail.gq1.yahoo.com [98.136.216.190]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 591EC10E1 for ; Wed, 1 Jun 2016 19:25:40 +0000 (UTC) (envelope-from juanperiz@yahoo.com.ar) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com.ar; s=s2048; t=1464809134; bh=/e3/OOoZUqFd8tCDfmT/KdRr9b3IueTR/uzjRpXdK/c=; h=Date:From:Reply-To:To:Subject:References:From:Subject; b=Uj0STSp8E6S7GIGtmwO5an0C8nwybZ7XPbRY67+C1Q/n0Vrspt6OGN/G/VBdtdsqsVNHCTglxkDMDF62/yBasLg5xdny3MpzgXVC5CVDKvO/5duZSf2n1OaRfxsemDMBJta6Vm0+SHhPox7uO2GC5v1vLrq+SHb+JLuuqxAIkq2tDevvrwxkqJ/9DL7p3bHYK7MkZrXjezO1C0Mhqnp+1sa0vS3NCtMVgRtJJGxYrj549XggypVmHaDtvvSzkpMgX1Fat2qhptXekmTP9iMWEoDVVSfQtyazw6MPdfHd3XiZfZhVt9VIF0i10UHnFGJtj0EUYx481HZRotcAgIktRg== Received: from [127.0.0.1] by nm35.bullet.mail.gq1.yahoo.com with NNFMP; 01 Jun 2016 19:25:34 -0000 Received: from [98.137.12.56] by nm35.bullet.mail.gq1.yahoo.com with NNFMP; 01 Jun 2016 19:22:42 -0000 Received: from [98.137.12.220] by tm1.bullet.mail.gq1.yahoo.com with NNFMP; 01 Jun 2016 19:22:42 -0000 Received: from [127.0.0.1] by omp1028.mail.gq1.yahoo.com with NNFMP; 01 Jun 2016 19:22:42 -0000 X-Yahoo-Newman-Property: ymail-4 X-Yahoo-Newman-Id: 320418.70713.bm@omp1028.mail.gq1.yahoo.com X-YMail-OSG: Cgxl.zIVM1kfDSiD4MnYFd1Tfj.GkTl5q87aW4fP.TcptMQt38uNZWb2OSFpghz UVeLPEfhUJRUBTzqXowJngIJEJihcBmiLAuwkxZAE8KlV_NpgcRujAV8_HUNomRqjaI6VpPWD6hV MFkcEEZTQU_zxNriHGLVp2rArXoQMCkWo88Mj..pqOBUpkP2HBcA9q5WRQvnpA0s7.dHRc0ZsAMR lae2K2WPlrJcVE4Emenh38zLt4UT8cQ2av9fy6NS2eXdSe8UP0OEPr.eyb5loZgPx5ZoS9RYoJJg VQy5Oy311FTR.u55TsOQ.l9Gp0Z2Ta5pR33RJ7wSH4pz_OJeDoJ6t5h_1KzS6PxOP_fLtxuRzIcL v9_AiWF7JIdlbcyBwIfsM4mKM1jAcV8LDZuKFuOcrJzNgRgMJBtpQPCU.1_kih_ik3n87dO7PiRw 8vuTRtFUnfTX7Wrt3ivUfSOAc3fHCpTKQwHbupPl4ngoDXaBpVNQuYd.zVaWqjNXlDyFoKf0T4rO gyRjupcokF8YBFCnZUOVs Received: from jws10726.mail.gq1.yahoo.com by sendmailws125.mail.gq1.yahoo.com; Wed, 01 Jun 2016 19:22:41 +0000; 1464808961.881 Date: Wed, 1 Jun 2016 19:22:41 +0000 (UTC) From: =?UTF-8?Q?Sebasti=C3=A1n_Maruca?= Reply-To: =?UTF-8?Q?Sebasti=C3=A1n_Maruca?= To: Roger Marquis , "freebsd-jail@freebsd.org" Message-ID: <140851342.3380283.1464808961455.JavaMail.yahoo@mail.yahoo.com> Subject: Re: deploy multiple vnets with VIMAGE/VNET + Production Ready? MIME-Version: 1.0 References: <140851342.3380283.1464808961455.JavaMail.yahoo.ref@mail.yahoo.com> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.22 X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 01 Jun 2016 19:25:41 -0000 Well... The spirit of this post inspires me the good way! Now we're talking about 10.3-HEAD wiht Jails+vnet... but then again, has an= yone tried it? Roger, it seems you are thumbing up my challenge... But I guess i'll have to stick with netgraph instead epair/if_bridge becaus= e the later is not so documented as the first one... Best regards,again... De: Roger Marquis Para: freebsd-jail@freebsd.org=20 Enviado: Mi=C3=A9rcoles, 1 de junio, 2016 13:07:33 Asunto: Re: deploy multiple vnets with VIMAGE/VNET + Production Ready? =20 Ernie Luzar wrote: > the kernel to included vimage. Enabling pf or ipf firewalls cause the > host to crash. ipfw firewall does not cause a crash but has next to no > real life usage on vimage. Considering we have had ipfw/vimage/netgraph jails for several years I'd be interested in your data sources. > When stopping vimage jails there is a problem with memory loss. Have you tested this, on a recent release? > You need a high proficiency in coding netgraph which > is used to tie the hosts network to each vimage jail. This certainly used to be true and IMO has been a significant barrier to netgraph usage but the scripts in head/share/examples/jails/ are at least helpful. > Needs a public network with multiple static ip address & registered domai= n > names even to test it. How are you implementing vimage that needs a registered domain name? > There are a few write ups about how to configure vet/vimage jails, but > their out of date. IE: 8.x & 9.x releases which are at EOL [end of life, > unsupported]. Vimage gets little attention.=C2=A0 Unfortunately the mapping of non-vimage localhost interfaces to the primary external interface isn't noted=20 nearly enough either.=C2=A0 These are weaknesses in bsd jails, the latter a non-trivial security issue on many non-vimage systems considering daemons like sendmail are installed and listening on "localhost" by default. > Going down this road will make the shop totally dependent on you and your > ability. A mega size pay bump is in your future. The shop will be fubar-e= d > if you die or get hurt requiring a hospital stay and long recovery. Potentially true of any Unix or Linux application in my experience. Have you tried vimage with epair/if_bridge instead of netgraph?=C2=A0 It's considerably simpler though the documentation is almost as conflicting and insufficient. Roger _______________________________________________ freebsd-jail@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-jail To unsubscribe, send any mail to "freebsd-jail-unsubscribe@freebsd.org" From owner-freebsd-jail@freebsd.org Wed Jun 1 21:34:37 2016 Return-Path: Delivered-To: freebsd-jail@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C3C66B65C64 for ; Wed, 1 Jun 2016 21:34:37 +0000 (UTC) (envelope-from luzar722@gmail.com) Received: from mail-it0-x22e.google.com (mail-it0-x22e.google.com [IPv6:2607:f8b0:4001:c0b::22e]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 8D95C1B29 for ; Wed, 1 Jun 2016 21:34:37 +0000 (UTC) (envelope-from luzar722@gmail.com) Received: by mail-it0-x22e.google.com with SMTP id z123so35364239itg.0 for ; Wed, 01 Jun 2016 14:34:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=message-id:date:from:user-agent:mime-version:to:cc:subject :references:in-reply-to:content-transfer-encoding; bh=ag4U0GtqbZUWOcn4BLd9q1z11e9vVnM+b/JsefO6rXs=; b=NoMb/no6USsK4t4SkSxCsY0OkkR0ILicVs8rQ/g17Wwye2JJ/UnqIiM2QliTCbmoTt 7GTZbWoKOj+uvyIzkGD4aRPKcqX3QJgDJTHWl4AuFc9hr3XHM0d7qjPnrvtPAt5n0jzt tSgOdtVMA1VirVZNFbhT9sauA8Z1l6KCLLn/Cah18aJVn8j2ejVSn/HdupjcBKcAFn2b 1Q/fDBvxqLWa6ei1CjcBN0A3FF6mAohMpkR6gNT0rUPj983lcZYhDOYEXgerRWvgjaG4 Cl5DNBcowkMC64wAGs8k5V8QS3/+jZorE/U4hglplghpQM7U4c+uDudYlFFbo0lWTXmn gPnw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:message-id:date:from:user-agent:mime-version:to :cc:subject:references:in-reply-to:content-transfer-encoding; bh=ag4U0GtqbZUWOcn4BLd9q1z11e9vVnM+b/JsefO6rXs=; b=mJUOZNEIkdidzFZKUOucsX9eWnZiv0viQMCm64ZgMldxXIqxpfHkQFekzpaEk7kZT6 vMgohMdZPy71XBjB5UUgBv9HRm4gMnTCzAIVWS+qPHrgmxGGh3JZjhZfUYXgNSRx8Yye pWu0tIyUXRa0Lr7GHpCP3JBlO9E3NLyZ0eNCEfaYPB7v7U1mb0oeoKT7bHlz5dL5sT9U u1UQ+DaHXH1Su0W8Q8dF6vfNQNkHaDJ4y0ahW4cpX599XHvaJ/Hsit0sluSLqIXDzNrz 5Zin3D5QJ6uNYZ6hM4v1ZpdU40U9ZEol2dUeKKOa2W27O0oKetc7IShhiW6nBoZV22ib WmmQ== X-Gm-Message-State: ALyK8tLav0I+VS5IdC8j1ifFQgmzvNY3v3n4DC5xG1uDcSzv+d8CdcLx/xJBTt+yK89inA== X-Received: by 10.36.160.5 with SMTP id o5mr133612ite.31.1464816876831; Wed, 01 Jun 2016 14:34:36 -0700 (PDT) Received: from [10.0.10.3] (cpe-184-56-210-236.neo.res.rr.com. [184.56.210.236]) by smtp.googlemail.com with ESMTPSA id r65sm16101510iod.7.2016.06.01.14.34.35 (version=TLSv1/SSLv3 cipher=OTHER); Wed, 01 Jun 2016 14:34:35 -0700 (PDT) Message-ID: <574F54FC.3040203@gmail.com> Date: Wed, 01 Jun 2016 17:34:52 -0400 From: Ernie Luzar User-Agent: Thunderbird 2.0.0.24 (Windows/20100228) MIME-Version: 1.0 To: Roger Marquis CC: freebsd-jail@freebsd.org Subject: Re: deploy multiple vnets with VIMAGE/VNET + Production Ready? References: <574f0851.ca0b620a.c7073.5becSMTPIN_ADDED_MISSING@mx.google.com> In-Reply-To: <574f0851.ca0b620a.c7073.5becSMTPIN_ADDED_MISSING@mx.google.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 01 Jun 2016 21:34:37 -0000 Roger Marquis wrote: > Ernie Luzar wrote: >> the kernel to included vimage. Enabling pf or ipf firewalls cause the >> host to crash. ipfw firewall does not cause a crash but has next to no >> real life usage on vimage. > > Considering we have had ipfw/vimage/netgraph jails for several years I'd > be interested in your data sources. The source is personal experience. Tested 9.3 & 10.0 with ipfw running in vnet/vimage jails. At that time ipfw was logging to the host and not to the vimage jail. Definitely a security violation. You know I give you a lot of credit for risking things on vnet/vimage jails in your shop. Most management just wouldn't take that risk. > >> When stopping vimage jails there is a problem with memory loss. > > Have you tested this, on a recent release? NO why would I when release notes didn't say anything about vimage changes or pf, ipf firewall becoming vimage aware. > >> You need a high proficiency in coding netgraph which >> is used to tie the hosts network to each vimage jail. > > This certainly used to be true and IMO has been a significant barrier to > netgraph usage but the scripts in head/share/examples/jails/ are > at least helpful. > I checked out those examples. Hardly any comments about what is happening or why their being done. All they are is a starting point to experiment doing trial and error testing >> Needs a public network with multiple static ip address & registered >> domain names even to test it. > > How are you implementing vimage that needs a registered domain name? > Maybe the real question is how do you drive un-solicited public traffic to your vnet/vimage jail without them. The real point here is, are you talking about a production config or some home play ground? There is no need for a vnet/vimage jail setup just for some server on the lan restricted to local usage only. The power of vnet/image comes to shine when used by a ISP or hosting company. There you have customers with static ip address and domain names. They have what looks like a real FreeBSd system to use when in reality its just one jail of many. >> There are a few write ups about how to configure vet/vimage jails, but >> their out of date. IE: 8.x & 9.x releases which are at EOL [end of life, >> unsupported]. > > Vimage gets little attention. Unfortunately the mapping of non-vimage > localhost interfaces to the primary external interface isn't noted > nearly enough either. These are weaknesses in bsd jails, the latter a > non-trivial security issue on many non-vimage systems considering > daemons like sendmail are installed and listening on "localhost" by > default. > After learning the usage of the jail(8) command doing testing the manual way, I found it to be so tedious keeping all the many different jail config options and command formats in my head, mistakes were common. qjail changed all that. Its so user friendly. In qjail sendmail is disabled by default and the cron status reports run faster because all the sendmail status checks are turned off. I disagree with you about the security issue of using localhost. Running sendmail in a non-vimage jail using its default config listening on localhost is still contained in the jail. Localhost is internally converted to the jails assigned ip address by jail(8). Why do you think this is a non-trivial security issue? >> Going down this road will make the shop totally dependent on you and your >> ability. A mega size pay bump is in your future. The shop will be >> fubar-ed >> if you die or get hurt requiring a hospital stay and long recovery. > > Potentially true of any Unix or Linux application in my experience. > Have you tried vimage with epair/if_bridge instead of netgraph? It's > considerably simpler though the documentation is almost as conflicting > and insufficient. > Yes epair/if_bridge is way simpler, but far less flexible when you want to re-point your public network ip address to different jails as circumstances change. Yep netgraph documentation sucks big time. My time for playing around is very limited. I'll wait for 11.0 to be published and see what the "release notes" say about vimage and the firewalls becoming vimage aware. Also will be checking the closed bugs for vimage to see what has been fixed. Then I will make up my mind about giving vimage another ride. But qjail will be the tool I use to perform the test ride. http://freshbsd.org/search?branch=HEAD&project=freebsd&q=vimage+OR+vnet shows 286 commits for vnet/vimage. This worries me that there has not been a call for vnet/vimage testers of -current. Just have to wait and see what happens. Maybe letting other vnet/vimage users lead the way with what is a bleeding edge version of vimage is the conservative way to approach this. I just think about zfs and how many releases containing zfs bug fixes before it became reliable. Its been many years and FreeBSD releases since vimage first became available as a kernel compile option. There is no way to know if vimage development will continue or even if bugs will be addresses. Vimage is not enjoying paid support. I do hope vnet/vimage has finally become of age and reliable for production like the non-vimage jails have become. From owner-freebsd-jail@freebsd.org Wed Jun 1 22:31:33 2016 Return-Path: Delivered-To: freebsd-jail@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id DA196B667E9 for ; Wed, 1 Jun 2016 22:31:33 +0000 (UTC) (envelope-from marquis@roble.com) Received: from mx5.roble.com (mx5.roble.com [206.40.34.5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mx5.roble.com", Issuer "mx5.roble.com" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id CE6561AD0 for ; Wed, 1 Jun 2016 22:31:33 +0000 (UTC) (envelope-from marquis@roble.com) Received: from alba.roble.net (mx5.roble.com [206.40.34.5]) by mx5.roble.com (Postfix) with ESMTP id 05FEF69529; Wed, 1 Jun 2016 15:31:32 -0700 (PDT) In-Reply-To: <574F54FC.3040203@gmail.com> References: <574f0851.ca0b620a.c7073.5becSMTPIN_ADDED_MISSING@mx.google.com> <574F54FC.3040203@gmail.com> Date: Wed, 1 Jun 2016 15:31:32 -0700 Subject: Re: deploy multiple vnets with VIMAGE/VNET + Production Ready? From: "Roger Marquis" To: "Ernie Luzar" Cc: freebsd-jail@freebsd.org Reply-To: marquis@roble.com MIME-Version: 1.0 Content-Type: text/plain;charset=iso-8859-1 Content-Transfer-Encoding: 8bit X-Priority: 3 (Normal) Importance: Normal X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 01 Jun 2016 22:31:34 -0000 >> Ernie Luzar wrote: >> Considering we have had ipfw/vimage/netgraph jails for several years I'd >> be interested in your data sources. > > The source is personal experience. Tested 9.3 & 10.0 with ipfw running > in vnet/vimage jails. At that time ipfw was logging to the host and not > to the vimage jail. Definitely a security violation. Kernel logging in general, not just for ipfw, is something that really should not propagate to jails but does. > You know I give you a lot of credit for risking things on vnet/vimage > jails in your shop. Most management just wouldn't take that risk. Wasn't me but the engineers here before me. My personal preference is for non-vimage jails, at least where the networking makes sense, Prefs aside we do have many vimage/netgraph/ipfw systems working well in the lab and field (of production high-volume financial applications). >> the scripts in head/share/examples/jails/ are at least helpful. > > I checked out those examples. Hardly any comments about what is > happening or why their being done. All they are is a starting point to > experiment doing trial and error testing The j?? scripts aren't meant as documentation but for easy of setup, to be called from /etc/jail.conf with a straightforward set of parameters. Agreed documentation here is still wholly insufficient. > I disagree with you about the security issue of using localhost. Running > sendmail in a non-vimage jail using its default config listening on > localhost is still contained in the jail. Localhost is internally > converted to the jails assigned ip address by jail(8). How is anything listening on localhost internally converted yet still contained in the jail? I mean what is the mechanism and why sendmail but not other daemons? > Why do you think this is a non-trivial security issue? telnet $jail 25 ehlo ... mail from: <...> rcpt to: <...> data Sendmail has never been a relatively secure app and DOS/DDOS and spam are vulnerabilities but point taken. Problem is the localhost to external mapping impacts not just sendmail but named, postfix and anything else listing on 127.0.0.1. > My time for playing around is very limited. I'll wait for 11.0 to be > published and see what the "release notes" say about vimage and the > firewalls becoming vimage aware. Also will be checking the closed bugs > for vimage to see what has been fixed. I have tested 11-CURRENT non-vimage, netgraph and if_bridge jails using iperf3 and not yet been able to trigger a crash. YMMV of course as the two bridging technologies do need far more substantial QA if we don't want to continue leaving this point strictly to Linux advocates. > I do hope vnet/vimage has finally become of age and reliable for > production like the non-vimage jails have become. More reliable, better documented AND simpler would be ideal. I believe the crux is A) in the code's complexity and readability, B) inherit difficulties of testing and of course C) funding. Roger From owner-freebsd-jail@freebsd.org Thu Jun 2 16:22:23 2016 Return-Path: Delivered-To: freebsd-jail@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E9C53B65CA2 for ; Thu, 2 Jun 2016 16:22:23 +0000 (UTC) (envelope-from marquis@roble.com) Received: from mx5.roble.com (mx5.roble.com [206.40.34.5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mx5.roble.com", Issuer "mx5.roble.com" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id D6C831B7C for ; Thu, 2 Jun 2016 16:22:23 +0000 (UTC) (envelope-from marquis@roble.com) Date: Thu, 2 Jun 2016 09:22:23 -0700 (PDT) From: Roger Marquis To: =?ISO-8859-15?Q?Sebasti=E1n_Maruca?= cc: "freebsd-jail@freebsd.org" Subject: Re: deploy multiple vnets with VIMAGE/VNET + Production Ready? In-Reply-To: <140851342.3380283.1464808961455.JavaMail.yahoo@mail.yahoo.com> References: <140851342.3380283.1464808961455.JavaMail.yahoo.ref@mail.yahoo.com> <140851342.3380283.1464808961455.JavaMail.yahoo@mail.yahoo.com> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 02 Jun 2016 16:22:24 -0000 > Well... The spirit of this post inspires me the good way! > Now we're talking about 10.3-HEAD wiht Jails+vnet... but then again, has anyone tried it? Roger, it seems you are thumbing up my challenge... All I would add is "test, test, test". If that goes well, and I'd expect it would, implement incrementally and keep a backout plan up-to-date. we haven't seen the memory leak issue but also don't shutdown jails very often. If you do then a cron job might be indicated, something to check rss and memfree and email an alert if needed. That plus an NMS like Zabbix should address the concerns expressed here. Roger From owner-freebsd-jail@freebsd.org Thu Jun 2 17:46:14 2016 Return-Path: Delivered-To: freebsd-jail@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 59E48B65219 for ; Thu, 2 Jun 2016 17:46:14 +0000 (UTC) (envelope-from markham@ssimicro.com) Received: from mail.ssimicro.com (mail.ssimicro.com [64.247.129.10]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client CN "*.ssimicro.com", Issuer "RapidSSL SHA256 CA - G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 0F5531A78 for ; Thu, 2 Jun 2016 17:46:13 +0000 (UTC) (envelope-from markham@ssimicro.com) Received: from markham.ssimicro.com (markham.ssimicro.com [64.247.130.99]) (authenticated bits=0) by mail.ssimicro.com (8.14.7/8.14.7) with ESMTP id u52HjWTZ087979 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES128-SHA bits=128 verify=NOT) for ; Thu, 2 Jun 2016 11:45:32 -0600 (MDT) Subject: Re: deploy multiple vnets with VIMAGE/VNET + Production Ready? To: freebsd-jail@freebsd.org References: <574f0851.ca0b620a.c7073.5becSMTPIN_ADDED_MISSING@mx.google.com> <574F54FC.3040203@gmail.com> <20160602162137.7A719E8E382@barracuda.ssimicro.com> From: markham breitbach Message-ID: <3392c787-6df5-5238-7de6-ad07d70442ae@ssimicro.com> Date: Thu, 2 Jun 2016 11:45:57 -0600 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:45.0) Gecko/20100101 Thunderbird/45.1.0 MIME-Version: 1.0 In-Reply-To: <20160602162137.7A719E8E382@barracuda.ssimicro.com> Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.22 X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 02 Jun 2016 17:46:14 -0000 >> I disagree with you about the security issue of using localhost. Runni= ng >> sendmail in a non-vimage jail using its default config listening on >> localhost is still contained in the jail. Localhost is internally >> converted to the jails assigned ip address by jail(8). > How is anything listening on localhost internally converted yet still > contained in the jail? I mean what is the mechanism and why sendmail b= ut not > other daemons? > >> Why do you think this is a non-trivial security issue? > telnet $jail 25 > ehlo ... > mail from: <...> > rcpt to: <...> > data > > Sendmail has never been a relatively secure app and DOS/DDOS and spam a= re > vulnerabilities but point taken. > > Problem is the localhost to external mapping impacts not just sendmail = but > named, postfix and anything else listing on 127.0.0.1. > > I think, perhaps, you misunderstand this issue. Inside a jail, lo0[127.0.0.1] is mapped directly to the primary IP address of a jail. For example, if you are building a typical public-facing web-hosting stack and you bind mysql to 127.0.0.1 expecting it to be only accessible to the localhost, you will be horribly surprised when you find you have just exposed your mysql server to the whole world. That is a terrible security issue because someone working as the system administrator for that web-host, should not need to be aware that 127.0.0.1 isn't really localhost for this particular host. localhost should behave according to spec, and should not just magically map loopback packets onto the public network. FTFRFC: 127.0.0.0/8 - This block is assigned for use as the Internet host loopback address. A datagram sent by a higher level protocol to an address anywhere within this block should loop back inside the host. This is ordinarily implemented using only 127.0.0.1/32 for loopback, but no addresses within this block should ever appear on any network anywhere [RFC1700, page 5]. -M From owner-freebsd-jail@freebsd.org Thu Jun 2 18:24:46 2016 Return-Path: Delivered-To: freebsd-jail@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9FD9EB65E2E for ; Thu, 2 Jun 2016 18:24:46 +0000 (UTC) (envelope-from trashcan@ellael.org) Received: from mx1.enfer-du-nord.net (mx1.enfer-du-nord.net [IPv6:2001:41d0:1008:bcb:1:1:0:1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 3927C1FC2 for ; Thu, 2 Jun 2016 18:24:45 +0000 (UTC) (envelope-from trashcan@ellael.org) Received: from [IPv6:2003:8c:2e03:e201:1cf:2a54:f28f:3bb1] (p2003008C2E03E20101CF2A54F28F3BB1.dip0.t-ipconnect.de [IPv6:2003:8c:2e03:e201:1cf:2a54:f28f:3bb1]) by mx1.enfer-du-nord.net (Postfix) with ESMTPSA id 3rLFxb70D7zL6J for ; Thu, 2 Jun 2016 20:24:35 +0200 (CEST) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) Subject: Re: deploy multiple vnets with VIMAGE/VNET + Production Ready? From: Michael Grimm In-Reply-To: <140851342.3380283.1464808961455.JavaMail.yahoo@mail.yahoo.com> Date: Thu, 2 Jun 2016 20:24:34 +0200 Content-Transfer-Encoding: quoted-printable Message-Id: <2CD81649-9D95-44B8-B0E3-DA38B8C3F31B@ellael.org> References: <140851342.3380283.1464808961455.JavaMail.yahoo.ref@mail.yahoo.com> <140851342.3380283.1464808961455.JavaMail.yahoo@mail.yahoo.com> To: "freebsd-jail@freebsd.org" X-Virus-Scanned: clamav-milter 0.99.2 at mail X-Virus-Status: Clean X-Mailer: Apple Mail (2.3124) X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 02 Jun 2016 18:24:46 -0000 Sebasti=C3=A1n Maruca via freebsd-jail wrote: > Now we're talking about 10.3-HEAD wiht Jails+vnet... but then again, = has anyone tried it? Roger, it seems you are thumbing up my challenge... > But I guess i'll have to stick with netgraph instead epair/if_bridge = because the later is not so documented as the first one=E2=80=A6 Preamble: I switched to VNET+epair/if_bridge jails starting 10.2-STABLE, = now 10.3-STABLE, and haven't seen any issues, sofar. Currently I do have = 10 jails running, firewall is pf at the host, only. My servers are not = big scaled ISP like, more small business-like, though. I am considering = myself a hobby admin.=20 Here's my configuration that may show you one way to get that running, = but I am sure your will have to tweak it to your needs: 1) Jails have been created by ezjail in the past, thus they are still at = ezjail's infrastructure. But I do no longer use ezjail for starting or = stopping my jails due to ezjail's lack of dealing with VNET jails (yet). = So I do still have fstab definitions in /etc for all jails, e.g.: /etc/fstab.www /path-to-your/jails/basejail = /path-to-your/jails/www/basejail nullfs ro 0 0=20 2) All external IPv4 or IPv6 addresses are NAT'ed or NAT66'ed to = 10.1.1.x or fd00:dead:dead:beef::x 3) Networking regarding VNET jails defined in /etc/rc.conf: # set up one bridge interface cloned_interfaces=3D"bridge0" # needed for default routes within jails ifconfig_bridge0=3D"inet 10.1.1.254 netmask 255.255.255.0" ifconfig_bridge0_ipv6=3D"inet6 fd00:dead:dead:beef::254 = prefixlen 64" 4) Thus, jails are controlled by jail(8) (shown for 3 example jails): /etc/rc.conf = =E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2= =80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94BEGIN---------= --------------- jail_enable=3D"YES" jail_reverse_stop=3D"YES" jail_list=3D"dns www mail" = =E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2= =80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94-END=E2=80=94=E2= =80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80= =94=E2=80=94=E2=80=94 /etc/jail.conf: # # host dependent global settings # $ip6prefixLOCAL =3D "fd00:dead:dead:beef"; =09 # # global jail settings # host.hostname =3D "${name}"; path =3D = "/path-to-your/jails/${name}"; mount.fstab =3D "/etc/fstab.${name}"; exec.consolelog =3D = "/var/log/jail_${name}_console.log"; vnet =3D "new"; vnet.interface =3D "epair${jailID}b"; exec.clean; mount.devfs; persist; =09 # # network settings to apply/destroy during start/stop of = every jail # exec.prestart =3D "sleep 2"; exec.prestart +=3D "ifconfig epair${jailID} = create up"; exec.prestart +=3D "ifconfig bridge0 addm = epair${jailID}a"; exec.start =3D "/sbin/ifconfig lo0 = 127.0.0.1 up"; exec.start +=3D "/sbin/ifconfig = epair${jailID}b inet ${ip4_addr}"; exec.start +=3D "/sbin/ifconfig = epair${jailID}b inet6 ${ip6_addr}"; exec.start +=3D "/sbin/route add default = -gateway 10.1.1.254"; exec.start +=3D "/sbin/route add -inet6 = default -gateway ${ip6prefixLOCAL}::254"; #exec.stop =3D "/sbin/route del default"; #exec.stop +=3D "/sbin/route del -inet6 = default"; exec.stop +=3D "/bin/sh /etc/rc.shutdown"; exec.poststop =3D "ifconfig epair${jailID}a = destroy"; =09 # # individual jail settings # mail { $jailID =3D 1; $ip4_addr =3D 10.1.1.1; $ip6_addr =3D ${ip6prefixLOCAL}::1/64; exec.start +=3D "/bin/sh /etc/rc"; } =09 www { $jailID =3D 2; $ip4_addr =3D 10.1.1.2; $ip6_addr =3D ${ip6prefixLOCAL}::2/64; exec.start +=3D "/bin/sh /etc/rc"; } =09 dns { $jailID =3D 3; $ip4_addr =3D 10.1.1.3; $ip4_addr_2 =3D 10.1.1.4; $ip6_addr =3D ${ip6prefixLOCAL}::3/64; $ip6_addr_2 =3D ${ip6prefixLOCAL}::4/64; exec.start +=3D "/sbin/ifconfig = epair${jailID}b inet ${ip4_addr_2} alias"; exec.start +=3D "/sbin/ifconfig = epair${jailID}b inet6 ${ip6_addr_2} alias"; exec.start +=3D "/bin/sh /etc/rc"; } =09 Now you can use "service jail" to start/stop your jails, e.g.: service jail stop=20 service jail restart dns service jail start dns mail 5) NOTE: I am refraining from restarting VNET jails the hard way as = shown above, and I am using a similar approach as iocage, namely "soft = restarts". As this functionality isn't available in 10.3-STABLE (IIRC) I = am using a homemade shell script instead. This script has to be run = *inside* a jail which can be triggered from the outside (still using = ezjail-admin) by e.g.: "sudo ezjail-admin console -e = '/usr/local/etc/_JAIL_SOFT_RESTART' www" #!/bin/csh =09 # # restart jail services without removing jail and its network # =09 # # global definitions # set LOGGER =3D "/usr/bin/logger -p user.info -t _JAIL_SOFT_RC" set RCDIR =3D "/usr/local/etc/rc.d" set TAB =3D " " =09 # # evaluate list of rc files in /usr/local/etc/rc.d # set RCFILES =3D `rcorder ${RCDIR}/* |& grep -v ^rcorder:` =09 # # evaluate reverse order of RCFILES # set RCFILES_REVERSE =3D "" foreach rcname ( ${RCFILES} ) set RCFILES_REVERSE =3D "${rcname} ${RCFILES_REVERSE}" end =09 # # stop rc services # echo "stopping:" foreach rcname ( ${RCFILES_REVERSE} ) ${LOGGER} stopping ${rcname} ${rcname} stop >& /dev/null echo "${TAB}" ${rcname} end =09 # # start rc services # echo "starting:" foreach rcname ( ${RCFILES} ) ${LOGGER} starting ${rcname} ${rcname} start >& /dev/null echo "${TAB}" ${rcname} end =09 exit 0 This script isn't perfect, and if you start or stop a jail you need to = separate the relevant part. This can easily be coded into that script, I = know. But I was lazy ;-) I hope that helps for a start. Again, I am sure you may need some = tweaking at your site. Regards, Michael From owner-freebsd-jail@freebsd.org Thu Jun 2 21:15:34 2016 Return-Path: Delivered-To: freebsd-jail@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4BE49B67190 for ; Thu, 2 Jun 2016 21:15:34 +0000 (UTC) (envelope-from k@free.de) Received: from smtp.free.de (smtp.free.de [91.204.6.103]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id DEFAC1E2E for ; Thu, 2 Jun 2016 21:15:33 +0000 (UTC) (envelope-from k@free.de) Received: (qmail 39509 invoked from network); 2 Jun 2016 23:15:31 +0200 Received: from smtp.free.de (HELO [91.204.5.142]) (k@free.de@[91.204.6.103]) (envelope-sender ) by smtp.free.de (qmail-ldap-1.03) with AES128-SHA encrypted SMTP for ; 2 Jun 2016 23:15:31 +0200 Subject: Re: deploy multiple vnets with VIMAGE/VNET + Production Ready? To: freebsd-jail@freebsd.org From: Kai Gallasch X-Enigmail-Draft-Status: N1110 Organization: FREE! Cc: marquis@roble.com Message-ID: <5750A1F3.8010000@free.de> Date: Thu, 2 Jun 2016 23:15:31 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101 Thunderbird/38.8.0 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 02 Jun 2016 21:15:34 -0000 On 01.06.2016 18:07 Roger Marquis wrote: > Ernie Luzar wrote: >> the kernel to included vimage. Enabling pf or ipf firewalls cause the >> host to crash. ipfw firewall does not cause a crash but has next to no >> real life usage on vimage. > Vimage gets little attention. Unfortunately the mapping of non-vimage > localhost interfaces to the primary external interface isn't noted > nearly enough either. These are weaknesses in bsd jails, the latter a > non-trivial security issue on many non-vimage systems considering > daemons like sendmail are installed and listening on "localhost" by > default.T Yes. If you startup a standard jail the jailed processes inside the jail that would normally bind to the loopback address 127.0.0.1 will bind to the ip address of the jail, thus being reachable from the outside of the jail.. But there is a workaround. On the jailhost you can clone an additional loopack interface (e.g. lo1) for exclusive jail usage. Assign to each jail an individual loopback address on this interface in addition to the jails ip address, like this: jailhost: igb0:10.10.10.100/24 | | jail1: ip4_addr:lo1|127.0.1.121,igb0|10.10.10.121/24 jail2: ip4_addr:lo1|127.0.1.122,igb0|10.10.10.121/24 ... .. If you start a jail configured on jail1, a sendmail process will happily bind to 127.0.1.121 as loopback and not to 10.10.10.121! not being exposed to the internet or lan. If you telnet - inside the jail1 - to localhost:25 you will receive the sendmail greeting. The way this workaround works jail2 has access to services bound to 127.0.1.121 on jail1, but this can be firewalled, if need be. (pf) block return in inet from 127.0.1.121 to !127.0.1.121 block return in inet from 127.0.1.122 to !127.0.1.122 Works for me. K. -- PGP-KeyID = 0x70654D7C4FB1F588 Internet of Things roadmap: 1) Put Linux on everything 2) Never update it From owner-freebsd-jail@freebsd.org Thu Jun 2 21:29:34 2016 Return-Path: Delivered-To: freebsd-jail@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1DD89B67523 for ; Thu, 2 Jun 2016 21:29:34 +0000 (UTC) (envelope-from juanperiz@yahoo.com.ar) Received: from nm39-vm8.bullet.mail.gq1.yahoo.com (nm39-vm8.bullet.mail.gq1.yahoo.com [98.136.217.111]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id E13D81330 for ; Thu, 2 Jun 2016 21:29:33 +0000 (UTC) (envelope-from juanperiz@yahoo.com.ar) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com.ar; s=s2048; t=1464902967; bh=KtSsZJcyOA8MSGsJpiTIast853JgFeDurd5sIwI3+So=; h=Date:From:Reply-To:To:In-Reply-To:References:Subject:From:Subject; b=Wo4EXu3gA+mAqe6RfRuqAcl/95/Nvi80WvYS/j8huiIknjdrG0eEFRczPCkfEaMhnLEWDk2kVE/7Y4djOz2aUBc1hFbRpALqWxJ/AS1DM+K8kazaQDvo5uaFMuRTH/q/FzgbFcweabCNDfBQURuP96P+MjZrJvbmrIRGG3IYTjaziH4lM5R4N6lq2BXPHHbDNm4uM57XfJ/UKcOcZVBkAQpITQzVF/vwcthCYBglAT7qv0ar652oRMGEtjfiHxpX/1/t7gCq2kdOu0XUApXWCTo5icpHQyiM7c4QySIctUexWXauSSrBlFuTjZmUcZLbOsuy3dR/95pu89V65FxS5g== Received: from [127.0.0.1] by nm39.bullet.mail.gq1.yahoo.com with NNFMP; 02 Jun 2016 21:29:27 -0000 Received: from [98.137.12.190] by nm39.bullet.mail.gq1.yahoo.com with NNFMP; 02 Jun 2016 21:26:42 -0000 Received: from [98.137.12.212] by tm11.bullet.mail.gq1.yahoo.com with NNFMP; 02 Jun 2016 21:26:42 -0000 Received: from [127.0.0.1] by omp1020.mail.gq1.yahoo.com with NNFMP; 02 Jun 2016 21:26:42 -0000 X-Yahoo-Newman-Property: ymail-4 X-Yahoo-Newman-Id: 64175.6436.bm@omp1020.mail.gq1.yahoo.com X-YMail-OSG: EYqV43MVM1kzETPCbJQlvk8bVUbD32nJ1ecVnjdpn53GwJMEIRCZxaEO97bs.Ng cnzkcbhi_HKhpBLQMJlEJpS_3wWFLOBunXsDs9_eZO24pHdxMm9ECRVWP44e5U8Va75aAFH9hcJp HsGAntnxZL1zfWK8jhiJOASQ5NZ4ePARcciUhM6Jdsk6ISSSxC4qeu8YCqg42ay_mjiUWzdpSjLA vSvXahE0hS5xrS96odEi2aIffNQSvO7CuRonofv6RutTgi7Xgt7BZOKUKMTQEfmNxAviuJEjbZjx Blr58vAsjuj7hPidVmCrWDJ.zloNxyQUaebReC7HMEx.Lnf2TFpJFhvlyztpucE_5.XrZyBeaIy3 CLlyZPC7M6B3Fjs5WwAWzwqyoBwovvoIvYDinE5pGOZEYnuRojiMi1WJN750RhBj649DOps.r.ub Ftp6ARG2H7QZcPWmcPnIFfX9Z5ZX6Ez.hyw0LudU_33zDoPdEAUZRqGv4k8KgZyTppMtTMGxv4eW 9R41VaWACCki3g7LtsxAwz9h.7HbYZH8lChk- Received: from jws10746.mail.gq1.yahoo.com by sendmailws128.mail.gq1.yahoo.com; Thu, 02 Jun 2016 21:26:41 +0000; 1464902801.582 Date: Thu, 2 Jun 2016 21:26:41 +0000 (UTC) From: =?UTF-8?Q?Sebasti=C3=A1n_Maruca?= Reply-To: =?UTF-8?Q?Sebasti=C3=A1n_Maruca?= To: Michael Grimm , "freebsd-jail@freebsd.org" Message-ID: <377963018.4245125.1464902801251.JavaMail.yahoo@mail.yahoo.com> In-Reply-To: <2CD81649-9D95-44B8-B0E3-DA38B8C3F31B@ellael.org> References: <140851342.3380283.1464808961455.JavaMail.yahoo.ref@mail.yahoo.com> <140851342.3380283.1464808961455.JavaMail.yahoo@mail.yahoo.com> <2CD81649-9D95-44B8-B0E3-DA38B8C3F31B@ellael.org> Subject: Re: deploy multiple vnets with VIMAGE/VNET + Production Ready? MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.22 X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 02 Jun 2016 21:29:34 -0000 Michael... even though you consider yourself as a admin hobbier, I can tell= you have the "lend hander" top grade you're honored ;) I'll start from this big step you're posting (and all the other which repli= ed too) and carry on dancing 'til I got my jails running DMZ, VLAN and WAN = like a pro... Best Regards,Seba De: Michael Grimm Para: "freebsd-jail@freebsd.org" =20 Enviado: Jueves, 2 de junio, 2016 15:24:34 Asunto: Re: deploy multiple vnets with VIMAGE/VNET + Production Ready? =20 Sebasti=C3=A1n Maruca via freebsd-jail wrote: > Now we're talking about 10.3-HEAD wiht Jails+vnet... but then again, has = anyone tried it? Roger, it seems you are thumbing up my challenge... > But I guess i'll have to stick with netgraph instead epair/if_bridge beca= use the later is not so documented as the first one=E2=80=A6 Preamble: I switched to VNET+epair/if_bridge jails starting 10.2-STABLE, no= w 10.3-STABLE, and haven't seen any issues, sofar. Currently I do have 10 j= ails running, firewall is pf at the host, only. My servers are not big scal= ed ISP like, more small business-like, though. I am considering myself a ho= bby admin.=20 Here's my configuration that may show you one way to get that running, but = I am sure your will have to tweak it to your needs: 1) Jails have been created by ezjail in the past, thus they are still at ez= jail's infrastructure. But I do no longer use ezjail for starting or stoppi= ng my jails due to ezjail's lack of dealing with VNET jails (yet). So I do = still have fstab definitions in /etc for all jails, e.g.: =C2=A0=C2=A0=C2=A0 /etc/fstab.www =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 /path-to-your/jails/basejail /path-to= -your/jails/www/basejail nullfs ro 0 0=20 2) All external IPv4 or IPv6 addresses are NAT'ed or NAT66'ed to 10.1.1.x o= r fd00:dead:dead:beef::x 3) Networking regarding VNET jails defined in /etc/rc.conf: =C2=A0=C2=A0=C2=A0 # set up one bridge interface =C2=A0=C2=A0=C2=A0 cloned_interfaces=3D"bridge0" =C2=A0=C2=A0=C2=A0 # needed for default routes within jails =C2=A0=C2=A0=C2=A0 ifconfig_bridge0=3D"inet 10.1.1.254 netmask 255.255.255.= 0" =C2=A0=C2=A0=C2=A0 ifconfig_bridge0_ipv6=3D"inet6 fd00:dead:dead:beef::254 = prefixlen 64" 4) Thus, jails are controlled by jail(8) (shown for 3 example jails): =C2=A0=C2=A0=C2=A0 /etc/rc.conf =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =E2=80=94=E2=80=94=E2=80=94=E2=80=94= =E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2= =80=94=E2=80=94=E2=80=94BEGIN------------------------ =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 jail_enable=3D"YES" =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 jail_reverse_stop=3D"YES" =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 jail_list=3D"dns www mail" =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =E2=80=94=E2=80=94=E2=80=94=E2=80=94= =E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2= =80=94=E2=80=94=E2=80=94-END=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94= =E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94 =C2=A0=C2=A0=C2=A0 /etc/jail.conf: =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 # =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 # host dependent global settings =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 # =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 $ip6prefixLOCAL=C2=A0=C2=A0=C2=A0 =C2= =A0=C2=A0=C2=A0 =3D "fd00:dead:dead:beef"; =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0=20 =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 # =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 # global jail settings =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 # =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 host.hostname=C2=A0=C2=A0=C2=A0 =C2= =A0=C2=A0=C2=A0 =3D "${name}"; =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 path=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0= =C2=A0 =C2=A0=C2=A0=C2=A0 =3D "/path-to-your/jails/${name}"; =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 mount.fstab=C2=A0=C2=A0=C2=A0 =C2=A0= =C2=A0=C2=A0 =3D "/etc/fstab.${name}"; =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 exec.consolelog =C2=A0=C2=A0=C2=A0 = =3D "/var/log/jail_${name}_console.log"; =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 vnet=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0= =C2=A0 =C2=A0=C2=A0=C2=A0 =3D "new"; =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 vnet.interface=C2=A0=C2=A0=C2=A0 =C2= =A0=C2=A0=C2=A0 =3D "epair${jailID}b"; =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 exec.clean; =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 mount.devfs; =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 persist; =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0=20 =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 # =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 # network settings to apply/destroy d= uring start/stop of every jail =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 # =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 exec.prestart=C2=A0=C2=A0=C2=A0 =C2= =A0=C2=A0=C2=A0 =3D "sleep 2"; =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 exec.prestart=C2=A0=C2=A0=C2=A0 =C2= =A0=C2=A0=C2=A0 +=3D "ifconfig epair${jailID} create up"; =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 exec.prestart=C2=A0=C2=A0=C2=A0 =C2= =A0=C2=A0=C2=A0 +=3D "ifconfig bridge0 addm epair${jailID}a"; =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 exec.start=C2=A0=C2=A0=C2=A0 =C2=A0= =C2=A0=C2=A0 =3D "/sbin/ifconfig lo0 127.0.0.1 up"; =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 exec.start=C2=A0=C2=A0=C2=A0 =C2=A0= =C2=A0=C2=A0 +=3D "/sbin/ifconfig epair${jailID}b inet ${ip4_addr}"; =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 exec.start=C2=A0=C2=A0=C2=A0 =C2=A0= =C2=A0=C2=A0 +=3D "/sbin/ifconfig epair${jailID}b inet6 ${ip6_addr}"; =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 exec.start=C2=A0=C2=A0=C2=A0 =C2=A0= =C2=A0=C2=A0 +=3D "/sbin/route add default -gateway 10.1.1.254"; =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 exec.start=C2=A0=C2=A0=C2=A0 =C2=A0= =C2=A0=C2=A0 +=3D "/sbin/route add -inet6 default -gateway ${ip6prefixLOCAL= }::254"; =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 #exec.stop=C2=A0=C2=A0=C2=A0 =C2=A0= =C2=A0=C2=A0 =3D "/sbin/route del default"; =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 #exec.stop=C2=A0=C2=A0=C2=A0 =C2=A0= =C2=A0=C2=A0 +=3D "/sbin/route del -inet6 default"; =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 exec.stop=C2=A0=C2=A0=C2=A0 =C2=A0=C2= =A0=C2=A0 +=3D "/bin/sh /etc/rc.shutdown"; =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 exec.poststop =C2=A0=C2=A0=C2=A0 =C2= =A0=C2=A0=C2=A0 =3D "ifconfig epair${jailID}a destroy"; =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0=20 =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 # =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 # individual jail settings =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 # =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 mail { =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 $jailID=C2=A0=C2= =A0=C2=A0 =C2=A0=C2=A0=C2=A0 =3D 1; =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 $ip4_addr=C2=A0=C2= =A0=C2=A0 =3D 10.1.1.1; =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 $ip6_addr=C2=A0=C2= =A0=C2=A0 =3D ${ip6prefixLOCAL}::1/64; =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 exec.start=C2=A0= =C2=A0=C2=A0 +=3D "/bin/sh /etc/rc"; =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 } =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0=20 =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 www { =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 $jailID=C2=A0=C2= =A0=C2=A0 =C2=A0=C2=A0=C2=A0 =3D 2; =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 $ip4_addr=C2=A0=C2= =A0=C2=A0 =3D 10.1.1.2; =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 $ip6_addr=C2=A0=C2= =A0=C2=A0 =3D ${ip6prefixLOCAL}::2/64; =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 exec.start=C2=A0= =C2=A0=C2=A0 +=3D "/bin/sh /etc/rc"; =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 } =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0=20 =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 dns { =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 $jailID=C2=A0=C2= =A0=C2=A0 =C2=A0=C2=A0=C2=A0 =3D 3; =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 $ip4_addr=C2=A0=C2= =A0=C2=A0 =3D 10.1.1.3; =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 $ip4_addr_2=C2=A0= =C2=A0=C2=A0 =3D 10.1.1.4; =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 $ip6_addr=C2=A0=C2= =A0=C2=A0 =3D ${ip6prefixLOCAL}::3/64; =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 $ip6_addr_2=C2=A0= =C2=A0=C2=A0 =3D ${ip6prefixLOCAL}::4/64; =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 exec.start=C2=A0= =C2=A0=C2=A0 +=3D "/sbin/ifconfig epair${jailID}b inet=C2=A0 ${ip4_addr_2} = alias"; =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 exec.start=C2=A0= =C2=A0=C2=A0 +=3D "/sbin/ifconfig epair${jailID}b inet6 ${ip6_addr_2} alias= "; =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 exec.start=C2=A0= =C2=A0=C2=A0 +=3D "/bin/sh /etc/rc"; =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 } =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0=20 =C2=A0=C2=A0=C2=A0 Now you can use "service jail" to start/stop your jails,= e.g.: =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 service jail stop=20 =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 service jail restart dns =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 service jail start dns mail 5) NOTE: I am refraining from restarting VNET jails the hard way as shown a= bove, and I am using a similar approach as iocage, namely "soft restarts". = As this functionality isn't available in 10.3-STABLE (IIRC) I am using a ho= memade shell script instead. This script has to be run *inside* a jail whic= h can be triggered from the outside (still using ezjail-admin) by e.g.: "su= do ezjail-admin console -e '/usr/local/etc/_JAIL_SOFT_RESTART' www" =C2=A0=C2=A0=C2=A0 #!/bin/csh =C2=A0=C2=A0=C2=A0=20 =C2=A0=C2=A0=C2=A0 # =C2=A0=C2=A0=C2=A0 # restart jail services without removing jail and its ne= twork =C2=A0=C2=A0=C2=A0 # =C2=A0=C2=A0=C2=A0=20 =C2=A0=C2=A0=C2=A0 # =C2=A0=C2=A0=C2=A0 # global definitions =C2=A0=C2=A0=C2=A0 # =C2=A0=C2=A0=C2=A0 set LOGGER =3D "/usr/bin/logger -p user.info -t _JAIL_SO= FT_RC" =C2=A0=C2=A0=C2=A0 set RCDIR =3D "/usr/local/etc/rc.d" =C2=A0=C2=A0=C2=A0 set TAB =3D "=C2=A0 =C2=A0 =C2=A0 =C2=A0 " =C2=A0=C2=A0=C2=A0=20 =C2=A0=C2=A0=C2=A0 # =C2=A0=C2=A0=C2=A0 # evaluate list of rc files in /usr/local/etc/rc.d =C2=A0=C2=A0=C2=A0 # =C2=A0=C2=A0=C2=A0 set RCFILES =3D `rcorder ${RCDIR}/* |& grep -v ^rcorder:= ` =C2=A0=C2=A0=C2=A0=20 =C2=A0=C2=A0=C2=A0 # =C2=A0=C2=A0=C2=A0 # evaluate reverse order of RCFILES =C2=A0=C2=A0=C2=A0 # =C2=A0=C2=A0=C2=A0 set RCFILES_REVERSE =3D "" =C2=A0=C2=A0=C2=A0 foreach rcname ( ${RCFILES} ) =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 set RCFILES_REVERSE =3D "${rcname} ${= RCFILES_REVERSE}" =C2=A0=C2=A0=C2=A0 end =C2=A0=C2=A0=C2=A0=20 =C2=A0=C2=A0=C2=A0 # =C2=A0=C2=A0=C2=A0 # stop rc services =C2=A0=C2=A0=C2=A0 # =C2=A0=C2=A0=C2=A0 echo "stopping:" =C2=A0=C2=A0=C2=A0 foreach rcname ( ${RCFILES_REVERSE} ) =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 ${LOGGER} stopping ${rcname} =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 ${rcname} stop >& /dev/null =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 echo "${TAB}" ${rcname} =C2=A0=C2=A0=C2=A0 end =C2=A0=C2=A0=C2=A0=20 =C2=A0=C2=A0=C2=A0 # =C2=A0=C2=A0=C2=A0 # start rc services =C2=A0=C2=A0=C2=A0 # =C2=A0=C2=A0=C2=A0 echo "starting:" =C2=A0=C2=A0=C2=A0 foreach rcname ( ${RCFILES} ) =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 ${LOGGER} starting ${rcname} =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 ${rcname} start >& /dev/null =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 echo "${TAB}" ${rcname} =C2=A0=C2=A0=C2=A0 end =C2=A0=C2=A0=C2=A0=20 =C2=A0=C2=A0=C2=A0 exit 0 This script isn't perfect, and if you start or stop a jail you need to sepa= rate the relevant part. This can easily be coded into that script, I know. = But I was lazy ;-) I hope that helps for a start. Again, I am sure you may need some tweaking = at your site. Regards, Michael _______________________________________________ freebsd-jail@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-jail To unsubscribe, send any mail to "freebsd-jail-unsubscribe@freebsd.org" From owner-freebsd-jail@freebsd.org Thu Jun 2 22:05:25 2016 Return-Path: Delivered-To: freebsd-jail@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D6EB8B67FF8 for ; Thu, 2 Jun 2016 22:05:25 +0000 (UTC) (envelope-from luzar722@gmail.com) Received: from mail-it0-x22c.google.com (mail-it0-x22c.google.com [IPv6:2607:f8b0:4001:c0b::22c]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 9F8661796 for ; Thu, 2 Jun 2016 22:05:25 +0000 (UTC) (envelope-from luzar722@gmail.com) Received: by mail-it0-x22c.google.com with SMTP id i127so63881716ita.1 for ; Thu, 02 Jun 2016 15:05:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=message-id:date:from:user-agent:mime-version:to:cc:subject :references:in-reply-to:content-transfer-encoding; bh=e/rPc/1dbW5I8qmQan0CHYWFwykS7g60OSrNYbmOLuw=; b=ez18eA1R9mIHJy2OcyJ7nnJ1sAMy8GI2FI9ztcN7jQ8Ouo45XzHAyO2pNKgZYIaU4s 5Z0BfVTWsXIhxFQYTYUTV3o9elsBYVWTskoO1iCYp+Lsw3Rgb/AVbIfw+xXmWBPeAfNO +SY/Va+Gq8DUbd9Zm07yGsMPrTjdve/JtT2h6tKkgwNnLoRCq3GFC/JBV9h89EucdX8J hmWh2TsJOGSE06YrubZSH50OG3OwpIq+dTKF5wxLo2xyqa5orivIxXezXXRyKNDFkasl QdBAdUAh4swLyLt24VctsuZzMVH21F1oAORvIS7F58JMoYHJ/5s65kbSMCJWHXm3gTCC I8uw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:message-id:date:from:user-agent:mime-version:to :cc:subject:references:in-reply-to:content-transfer-encoding; bh=e/rPc/1dbW5I8qmQan0CHYWFwykS7g60OSrNYbmOLuw=; b=aIfBAlq0U2QmtEzQFDWe7gPxDqO03IMyx0qEtsTWPyvWAoENyGjIjegRoDQczvY3kL G8Nn2MrzJTE5bSkW4jlcUhzghZz5z5nWAVh630EP+rl7QA2qufkGtFLLOgd/hduWTC2s 4ty3RxP1BXr0XSCt5w8Ts2bmyXOCbZlpj6MTvho2GW/mgtTPQgSzjjC9u0XheTfT0AZ4 SFTAva3jRTzggo6xRwzbAnBwx4e3/KkLNJE/M4AOyTY+npRdJK7V77emVxD3Lrnbg+VG RnM6pSC8flmn5HNd5C3Z1MWFU517HMD+kOyE62F38cPwF9ZZnXmH39hzHFZ9Smai7zjE myRQ== X-Gm-Message-State: ALyK8tLbG2iVwFeT1cpKvTB6KQAoNbDIZQA5nP50mC9quw4oOROaSAyonuB4IVdd9OOaIQ== X-Received: by 10.36.19.16 with SMTP id 16mr1257845itz.76.1464905123864; Thu, 02 Jun 2016 15:05:23 -0700 (PDT) Received: from [10.0.10.3] (cpe-184-56-210-236.neo.res.rr.com. [184.56.210.236]) by smtp.googlemail.com with ESMTPSA id l131sm1407098iol.13.2016.06.02.15.05.23 (version=TLSv1/SSLv3 cipher=OTHER); Thu, 02 Jun 2016 15:05:23 -0700 (PDT) Message-ID: <5750ADB7.8010409@gmail.com> Date: Thu, 02 Jun 2016 18:05:43 -0400 From: Ernie Luzar User-Agent: Thunderbird 2.0.0.24 (Windows/20100228) MIME-Version: 1.0 To: Michael Grimm CC: "freebsd-jail@freebsd.org" Subject: Re: deploy multiple vnets with VIMAGE/VNET + Production Ready? References: <140851342.3380283.1464808961455.JavaMail.yahoo.ref@mail.yahoo.com> <140851342.3380283.1464808961455.JavaMail.yahoo@mail.yahoo.com> <2CD81649-9D95-44B8-B0E3-DA38B8C3F31B@ellael.org> In-Reply-To: <2CD81649-9D95-44B8-B0E3-DA38B8C3F31B@ellael.org> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 02 Jun 2016 22:05:25 -0000 Michael Grimm wrote: > Sebastián Maruca via freebsd-jail wrote: > >> Now we're talking about 10.3-HEAD wiht Jails+vnet... but then again, has anyone tried it? Roger, it seems you are thumbing up my challenge... >> But I guess i'll have to stick with netgraph instead epair/if_bridge because the later is not so documented as the first one… > > Preamble: I switched to VNET+epair/if_bridge jails starting 10.2-STABLE, now 10.3-STABLE, and haven't seen any issues, sofar. Currently I do have 10 jails running, firewall is pf at the host, only. My servers are not big scaled ISP like, more small business-like, though. I am considering myself a hobby admin. > > > Here's my configuration that may show you one way to get that running, but I am sure your will have to tweak it to your needs: > > 1) Jails have been created by ezjail in the past, thus they are still at ezjail's infrastructure. But I do no longer use ezjail for starting or stopping my jails due to ezjail's lack of dealing with VNET jails (yet). So I do still have fstab definitions in /etc for all jails, e.g.: > > /etc/fstab.www > /path-to-your/jails/basejail /path-to-your/jails/www/basejail nullfs ro 0 0 > > 2) All external IPv4 or IPv6 addresses are NAT'ed or NAT66'ed to 10.1.1.x or fd00:dead:dead:beef::x > > 3) Networking regarding VNET jails defined in /etc/rc.conf: > > # set up one bridge interface > cloned_interfaces="bridge0" > > # needed for default routes within jails > ifconfig_bridge0="inet 10.1.1.254 netmask 255.255.255.0" > ifconfig_bridge0_ipv6="inet6 fd00:dead:dead:beef::254 prefixlen 64" > > 4) Thus, jails are controlled by jail(8) (shown for 3 example jails): > > /etc/rc.conf > ———————————————BEGIN------------------------ > jail_enable="YES" > jail_reverse_stop="YES" > jail_list="dns www mail" > ———————————————-END———————————— > > /etc/jail.conf: > # > # host dependent global settings > # > $ip6prefixLOCAL = "fd00:dead:dead:beef"; > > # > # global jail settings > # > host.hostname = "${name}"; > path = "/path-to-your/jails/${name}"; > mount.fstab = "/etc/fstab.${name}"; > exec.consolelog = "/var/log/jail_${name}_console.log"; > vnet = "new"; > vnet.interface = "epair${jailID}b"; > exec.clean; > mount.devfs; > persist; > > # > # network settings to apply/destroy during start/stop of every jail > # > exec.prestart = "sleep 2"; > exec.prestart += "ifconfig epair${jailID} create up"; > exec.prestart += "ifconfig bridge0 addm epair${jailID}a"; > exec.start = "/sbin/ifconfig lo0 127.0.0.1 up"; > exec.start += "/sbin/ifconfig epair${jailID}b inet ${ip4_addr}"; > exec.start += "/sbin/ifconfig epair${jailID}b inet6 ${ip6_addr}"; > exec.start += "/sbin/route add default -gateway 10.1.1.254"; > exec.start += "/sbin/route add -inet6 default -gateway ${ip6prefixLOCAL}::254"; > #exec.stop = "/sbin/route del default"; > #exec.stop += "/sbin/route del -inet6 default"; > exec.stop += "/bin/sh /etc/rc.shutdown"; > exec.poststop = "ifconfig epair${jailID}a destroy"; > > # > # individual jail settings > # > mail { > $jailID = 1; > $ip4_addr = 10.1.1.1; > $ip6_addr = ${ip6prefixLOCAL}::1/64; > exec.start += "/bin/sh /etc/rc"; > } > > www { > $jailID = 2; > $ip4_addr = 10.1.1.2; > $ip6_addr = ${ip6prefixLOCAL}::2/64; > exec.start += "/bin/sh /etc/rc"; > } > > dns { > $jailID = 3; > $ip4_addr = 10.1.1.3; > $ip4_addr_2 = 10.1.1.4; > $ip6_addr = ${ip6prefixLOCAL}::3/64; > $ip6_addr_2 = ${ip6prefixLOCAL}::4/64; > exec.start += "/sbin/ifconfig epair${jailID}b inet ${ip4_addr_2} alias"; > exec.start += "/sbin/ifconfig epair${jailID}b inet6 ${ip6_addr_2} alias"; > exec.start += "/bin/sh /etc/rc"; > } > > Now you can use "service jail" to start/stop your jails, e.g.: > > service jail stop > service jail restart dns > service jail start dns mail > > 5) NOTE: I am refraining from restarting VNET jails the hard way as shown above, and I am using a similar approach as iocage, namely "soft restarts". As this functionality isn't available in 10.3-STABLE (IIRC) I am using a homemade shell script instead. This script has to be run *inside* a jail which can be triggered from the outside (still using ezjail-admin) by e.g.: "sudo ezjail-admin console -e '/usr/local/etc/_JAIL_SOFT_RESTART' www" > > #!/bin/csh > > # > # restart jail services without removing jail and its network > # > > # > # global definitions > # > set LOGGER = "/usr/bin/logger -p user.info -t _JAIL_SOFT_RC" > set RCDIR = "/usr/local/etc/rc.d" > set TAB = " " > > # > # evaluate list of rc files in /usr/local/etc/rc.d > # > set RCFILES = `rcorder ${RCDIR}/* |& grep -v ^rcorder:` > > # > # evaluate reverse order of RCFILES > # > set RCFILES_REVERSE = "" > foreach rcname ( ${RCFILES} ) > set RCFILES_REVERSE = "${rcname} ${RCFILES_REVERSE}" > end > > # > # stop rc services > # > echo "stopping:" > foreach rcname ( ${RCFILES_REVERSE} ) > ${LOGGER} stopping ${rcname} > ${rcname} stop >& /dev/null > echo "${TAB}" ${rcname} > end > > # > # start rc services > # > echo "starting:" > foreach rcname ( ${RCFILES} ) > ${LOGGER} starting ${rcname} > ${rcname} start >& /dev/null > echo "${TAB}" ${rcname} > end > > exit 0 > > This script isn't perfect, and if you start or stop a jail you need to separate the relevant part. This can easily be coded into that script, I know. But I was lazy ;-) > > I hope that helps for a start. Again, I am sure you may need some tweaking at your site. > > Regards, > Michael > > Michael, You left out whether you had to compile the kernel with the vimage option or whether vimage was already included in the kernel? From owner-freebsd-jail@freebsd.org Fri Jun 3 09:23:40 2016 Return-Path: Delivered-To: freebsd-jail@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 796C5B68D4C for ; Fri, 3 Jun 2016 09:23:40 +0000 (UTC) (envelope-from trashcan@ellael.org) Received: from mx1.enfer-du-nord.net (mx1.enfer-du-nord.net [IPv6:2001:41d0:1008:bcb:1:1:0:1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4331311EC for ; Fri, 3 Jun 2016 09:23:40 +0000 (UTC) (envelope-from trashcan@ellael.org) Received: from mx1.enfer-du-nord.net (www.kaan-bock.invalid [10.10.10.2]) by mx1.enfer-du-nord.net (Postfix) with ESMTPSA id 3rLdty3lj9zKMX for ; Fri, 3 Jun 2016 11:23:38 +0200 (CEST) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII; format=flowed Content-Transfer-Encoding: 7bit Date: Fri, 03 Jun 2016 11:23:38 +0200 From: Michael Grimm To: freebsd-jail@freebsd.org Subject: Re: deploy multiple vnets with VIMAGE/VNET + Production Ready? In-Reply-To: <5750ADB7.8010409@gmail.com> References: <140851342.3380283.1464808961455.JavaMail.yahoo.ref@mail.yahoo.com> <140851342.3380283.1464808961455.JavaMail.yahoo@mail.yahoo.com> <2CD81649-9D95-44B8-B0E3-DA38B8C3F31B@ellael.org> <5750ADB7.8010409@gmail.com> Message-ID: <76a532eb68fc1ec4239be3e80ce519cc@mx1.enfer-du-nord.net> X-Sender: trashcan@ellael.org X-Virus-Scanned: clamav-milter 0.99.2 at mail X-Virus-Status: Clean User-Agent: Roundcube Webmail/1.1.5 X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 03 Jun 2016 09:23:40 -0000 On 2016-06-03 0:05, Ernie Luzar wrote: > Michael Grimm wrote: >> I switched to VNET+epair/if_bridge jails starting 10.2-STABLE, >> now 10.3-STABLE, and haven't seen any issues, sofar. > You left out whether you had to compile the kernel with the vimage > option or whether vimage was already included in the kernel? Obviously, I had had to compile a custom kernel because I am running 10.x-STABLE. And, because I am using IPSec, I have to do so anyway: include GENERIC # include GENERIC definitions, first ident CUSTOM # custom kernel name cpu HAMMER # amd64 and intel 64bit cpus device crypto # IPsec depends on this options IPSEC # IP security options IPSEC_NAT_T # IP security NAT-T in transport mode options VIMAGE # network virtualization Regards, Michael From owner-freebsd-jail@freebsd.org Sat Jun 4 14:30:29 2016 Return-Path: Delivered-To: freebsd-jail@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7F198B6A317 for ; Sat, 4 Jun 2016 14:30:29 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 6FACC19EC for ; Sat, 4 Jun 2016 14:30:29 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id u54EUTKq097307 for ; Sat, 4 Jun 2016 14:30:29 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-jail@FreeBSD.org Subject: [Bug 206012] jail(8): Cannot assign link-local IPv6 address to a jail Date: Sat, 04 Jun 2016 14:30:29 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.0-CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: s.biberhofer@spherical-elephant.com X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-jail@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 04 Jun 2016 14:30:29 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D206012 --- Comment #4 from Sascha Biberhofer = --- Hi Jamie, That would be a great and at least avoid the issue of having to specify the (slightly volatile) interface number explicitly in jail.conf. However, since the interface that an address is meant for is always specifi= ed in jail.conf anyway (either via the interface parameter or the if|address format in ip4.addr and ip6.addr) it would be even nicer if we could generate the interface index directly from the interface information in jail.conf, i= .e. ip6.addr =3D 'lo1|fe80::dead:beef'=20 or interface =3D 'lo1' ip6.addr =3D 'fe80::dead:beef' should suffice for the sake of convenience. I guess one could map these for= ms to fe80::dead:beef%if and then map this to the index form like you suggeste= d. --=20 You are receiving this mail because: You are the assignee for the bug.=