From owner-freebsd-jail@FreeBSD.ORG Fri Jan 3 08:17:02 2014 Return-Path: Delivered-To: freebsd-jail@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 5BFE3705 for ; Fri, 3 Jan 2014 08:17:02 +0000 (UTC) Received: from mail.monkeybrains.net (mail.monkeybrains.net [208.69.40.19]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 3EBFE14E1 for ; Fri, 3 Jan 2014 08:17:01 +0000 (UTC) Received: from invalid-dns.rfc1918.monkeybrains.net (208-90-212-98.PUBLIC.monkeybrains.net [208.90.212.98]) (authenticated bits=0) by mail.monkeybrains.net (8.14.7/8.14.7) with ESMTP id s038096W079124 (version=TLSv1/SSLv3 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO) for ; Fri, 3 Jan 2014 00:00:10 -0800 (PST) (envelope-from crapsh@monkeybrains.net) X-Authentication-Warning: mail.monkeybrains.net: Host 208-90-212-98.PUBLIC.monkeybrains.net [208.90.212.98] claimed to be invalid-dns.rfc1918.monkeybrains.net Message-ID: <52C66E09.80307@monkeybrains.net> Date: Fri, 03 Jan 2014 00:00:09 -0800 From: "Rudy (bulk)" User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:24.0) Gecko/20100101 Thunderbird/24.2.0 MIME-Version: 1.0 To: freebsd-jail@freebsd.org Subject: Allowing routing table visibility in jails to make multiple IPs work properly References: <201311301000.rAUA00eG045983@freefall.freebsd.org> In-Reply-To: <201311301000.rAUA00eG045983@freefall.freebsd.org> X-Forwarded-Message-Id: <201311301000.rAUA00eG045983@freefall.freebsd.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: clamav-milter 0.98 at mail.monkeybrains.net X-Virus-Status: Clean X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 03 Jan 2014 08:17:02 -0000 I'm having issues when putting multiple IPs on a jail... one external, one internal (on a different vlan). The source IP from the jail is always the first IP, so a solution is to use ipfw_nat to nat when using the internal vlan to the 'second ip'. Ugly hack. and it doesn't work when there is an MTU difference between the vlans: http://www.freebsd.org/cgi/query-pr.cgi?pr=184389 Re: kern/184389: libalias fails to adjust MTU from jails The other solution is to let the jail 'see' the routing table: devfs -m /data/example.monkeybrains.net/dev rule apply path kmem unhide devfs -m /data/example.monkeybrains.net/dev rule apply path mem unhide Is there anyway (or plans for) a method to reveal the routing table but not all of mem and kmem to the jail? Rudy From owner-freebsd-jail@FreeBSD.ORG Fri Jan 3 13:06:02 2014 Return-Path: Delivered-To: freebsd-jail@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 85820D5E for ; Fri, 3 Jan 2014 13:06:02 +0000 (UTC) Received: from mail-pa0-f44.google.com (mail-pa0-f44.google.com [209.85.220.44]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 5C75C199C for ; Fri, 3 Jan 2014 13:06:02 +0000 (UTC) Received: by mail-pa0-f44.google.com with SMTP id fa1so15751819pad.31 for ; Fri, 03 Jan 2014 05:05:55 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=G+u8nn9AaZHKEFOSYKSmCClmjfoDo/5MJwWvKuIrWII=; b=FATqbz/b3D7O6MyQ8WTUrzDUxvcsd4L4Aa+NM/2s0MQF/qIc0l2kVuOnxm7TlQOk4J eaz7lILc7nTO3MwhOudz8eaejPSJSktJjMNM4yN7OZERGhvSs41be/fjn3X1j+UHJanc D5N6c7bT0YqD6IL/8uqRg1PYPTlRypNOPMt1+U2oKGj+F6mB1yg/uLDinjAubchZMal5 TPseNSxk9S/Ov0+xAtJ65+4eDfM8X7AJjBDMKLe4cLgsIHiJksYZiYB2QjFiao+VyBg5 bz5ViHBOjW/qcDu6OwS/FEvHDPh3w2HlRL5UX6aF9uuV471Ub3Y1RKhoHwbNK4XpdzV2 wzmQ== X-Gm-Message-State: ALoCoQlv9C5vED9IbcvMjSIsgc+QfuSjRduFXMaWy+4CFcE4Mdx8QF6B6oWBGDsmsOjc0gdmoJ8A MIME-Version: 1.0 X-Received: by 10.68.66.1 with SMTP id b1mr94929602pbt.43.1388754355645; Fri, 03 Jan 2014 05:05:55 -0800 (PST) Received: by 10.66.249.232 with HTTP; Fri, 3 Jan 2014 05:05:55 -0800 (PST) In-Reply-To: <52C66E09.80307@monkeybrains.net> References: <201311301000.rAUA00eG045983@freefall.freebsd.org> <52C66E09.80307@monkeybrains.net> Date: Fri, 3 Jan 2014 08:05:55 -0500 Message-ID: Subject: Re: Allowing routing table visibility in jails to make multiple IPs work properly From: Alejandro Imass To: "Rudy (bulk)" Content-Type: text/plain; charset=ISO-8859-1 Cc: freebsd-jail@freebsd.org X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 03 Jan 2014 13:06:02 -0000 On Fri, Jan 3, 2014 at 3:00 AM, Rudy (bulk) wrote: > > I'm having issues when putting multiple IPs on a jail... one external, one > internal (on a different vlan). The source IP from the jail is always the > first IP, so a solution is to use ipfw_nat to nat when using the internal > vlan to the 'second ip'. Ugly hack. and it doesn't work when there is an > MTU difference between the vlans: > Greetings Rudy, I had the same exact problem and found that the problem is natd. Actually it is mentioned in natd's documentation. If you want to get rid of this problem you need to get rid of natd and nat your jail traffic with some other means. Kernel nat should be a solution but I've never gotten around to test if it actually solves the problem. Please share if you find a way to fix this. Best, Alejandro Imass From owner-freebsd-jail@FreeBSD.ORG Fri Jan 3 13:10:31 2014 Return-Path: Delivered-To: freebsd-jail@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id AD47EE2F for ; Fri, 3 Jan 2014 13:10:31 +0000 (UTC) Received: from out3-smtp.messagingengine.com (out3-smtp.messagingengine.com [66.111.4.27]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 7E8051A13 for ; Fri, 3 Jan 2014 13:10:31 +0000 (UTC) Received: from compute5.internal (compute5.nyi.mail.srv.osa [10.202.2.45]) by gateway1.nyi.mail.srv.osa (Postfix) with ESMTP id DEA0720E49 for ; Fri, 3 Jan 2014 08:10:29 -0500 (EST) Received: from web3 ([10.202.2.213]) by compute5.internal (MEProxy); Fri, 03 Jan 2014 08:10:29 -0500 DKIM-Signature: v=1; a=rsa-sha1; c=relaxed/relaxed; d= messagingengine.com; h=message-id:from:to:mime-version :content-transfer-encoding:content-type:in-reply-to:references :subject:date; s=smtpout; bh=lOa+7W3U2pYTvlRCq2M8w58IJcU=; b=Ll1 DiLTVCFOsrvIYzQp8lpQU+1n5x2LPwGHxrVrD2zuHWyaNNnTA8gW9a5OkuXcfOwO QzGi0UrGwvIDzjPJVIpQiZYedDtcdIZ77Ef0NafY4jX5lRCbVX5tP35UlG6rjZPG zqOW0MT0MsoCP9dEu2RzWm53pIxKFckUfR17g0bk= Received: by web3.nyi.mail.srv.osa (Postfix, from userid 99) id B29CA109C08; Fri, 3 Jan 2014 08:10:29 -0500 (EST) Message-Id: <1388754629.28024.66145985.72ADDF43@webmail.messagingengine.com> X-Sasl-Enc: gYDrEmYIv8GSIBZwojZUesIjiht4aUv7x5e6SdbaGfbh 1388754629 From: Mark Felder To: freebsd-jail@freebsd.org MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Type: text/plain X-Mailer: MessagingEngine.com Webmail Interface - ajax-885bfc1c In-Reply-To: <52C66E09.80307@monkeybrains.net> References: <201311301000.rAUA00eG045983@freefall.freebsd.org> <52C66E09.80307@monkeybrains.net> Subject: Re: Allowing routing table visibility in jails to make multiple IPs work properly Date: Fri, 03 Jan 2014 07:10:29 -0600 X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 03 Jan 2014 13:10:31 -0000 On Fri, Jan 3, 2014, at 2:00, Rudy (bulk) wrote: > > I'm having issues when putting multiple IPs on a jail... one external, > one internal (on a different vlan). The source IP from the jail is > always the first IP, so a solution is to use ipfw_nat to nat when using > the internal vlan to the 'second ip'. Ugly hack. and it doesn't work > when there is an MTU difference between the vlans: > > > http://www.freebsd.org/cgi/query-pr.cgi?pr=184389 > Re: kern/184389: libalias fails to adjust MTU from jails > > > The other solution is to let the jail 'see' the routing table: > devfs -m /data/example.monkeybrains.net/dev rule apply path kmem unhide > devfs -m /data/example.monkeybrains.net/dev rule apply path mem unhide > > Is there anyway (or plans for) a method to reveal the routing table but > not all of mem and kmem to the jail? > > Hi! You've hit a bug I found a while back. Can you reconfirm the findings that myself and bz had? The issue is not that the first IP is used for *all* traffic, but only for traffic that uses raw sockets (like ICMP). I actually have patches bz@ provided me for ping and fping which work around this issue, but the fix should be done in the kernel instead. Here's my PR, please take a look. http://www.freebsd.org/cgi/query-pr.cgi?pr=168678 Your solution with the kmem/mem unhide is interesting. I do not have a system that I could try that on at this time; my needs were temporary/transitional (moving a monitoring server from 32bit to 64bit... architecture dependent RRDs, etc... ) Thanks! From owner-freebsd-jail@FreeBSD.ORG Fri Jan 3 14:10:25 2014 Return-Path: Delivered-To: freebsd-jail@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 8B7B1F08 for ; Fri, 3 Jan 2014 14:10:25 +0000 (UTC) Received: from sola.nimnet.asn.au (paqi.nimnet.asn.au [115.70.110.159]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 06D7A10A3 for ; Fri, 3 Jan 2014 14:10:24 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by sola.nimnet.asn.au (8.14.2/8.14.2) with ESMTP id s03EAEl0097289; Sat, 4 Jan 2014 01:10:15 +1100 (EST) (envelope-from smithi@nimnet.asn.au) Date: Sat, 4 Jan 2014 01:10:14 +1100 (EST) From: Ian Smith To: Alejandro Imass Subject: Re: Allowing routing table visibility in jails to make multiple IPs work properly In-Reply-To: Message-ID: <20140104005845.V35277@sola.nimnet.asn.au> References: <201311301000.rAUA00eG045983@freefall.freebsd.org> <52C66E09.80307@monkeybrains.net> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Cc: freebsd-jail@freebsd.org X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 03 Jan 2014 14:10:25 -0000 On Fri, 3 Jan 2014 08:05:55 -0500, Alejandro Imass wrote: > On Fri, Jan 3, 2014 at 3:00 AM, Rudy (bulk) wrote: > > > > I'm having issues when putting multiple IPs on a jail... one external, one > > internal (on a different vlan). The source IP from the jail is always the > > first IP, so a solution is to use ipfw_nat to nat when using the internal > > vlan to the 'second ip'. Ugly hack. and it doesn't work when there is an > > MTU difference between the vlans: > > > > Greetings Rudy, > > I had the same exact problem and found that the problem is natd. > Actually it is mentioned in natd's documentation. Alejandro, hi, can you point out where in natd(8) it indicates .. what exactly? > If you want to get rid of this problem you need to get rid of natd and > nat your jail traffic with some other means. Kernel nat should be a > solution but I've never gotten around to test if it actually solves > the problem. Please share if you find a way to fix this. I may have missed it, but I've yet to see anyone report any functional differences between natd and ipfw_nat, ie of something working in one but not the other. Both use the underlying libalias(3) after all. cheers, Ian From owner-freebsd-jail@FreeBSD.ORG Fri Jan 3 15:57:15 2014 Return-Path: Delivered-To: freebsd-jail@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 5DCB5A28 for ; Fri, 3 Jan 2014 15:57:15 +0000 (UTC) Received: from mail-pa0-f44.google.com (mail-pa0-f44.google.com [209.85.220.44]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 3288618FB for ; Fri, 3 Jan 2014 15:57:14 +0000 (UTC) Received: by mail-pa0-f44.google.com with SMTP id fa1so15892753pad.17 for ; Fri, 03 Jan 2014 07:57:14 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=YD4x5CymToiaQwqTkuU3Idtk3Z2JZF1RKgmZnUxmx7g=; b=j0m7AdQByxRnMHp3LXXNHeC/pfavWcUOz0pMHEWsBATbHAz6kb/ztcHpUl/QL2YmGP dFt9PzpXyRX6RXmn4boJoyTNB+22+kqCOFOQ6PLFnrYY14reZ+tHM0XS4wfPZoAVba0v Hk+okzjE+0eyfpgRz9L2LgoEwc1BDLRdf3baZuWOcsqO7yLnRyNtc6D8rwkVj8Ju6oKx yEVet8WZHC2TtP41Ry79wglX0yz2A26dPn/ogCanzrTq3tAICbSdF3XuHE+9ZxskI7Bv XWMyF1Tv/qSRCTNgOL/Fdwlv6oyArTSTfNOyMdldpXXi+BYXlkNyAmMYp5kUCNpL6/e2 I08g== X-Gm-Message-State: ALoCoQkQPeltA/m/SWk8sDjSpAYGXdOjpUO6zD2s1kzbpcXI2N4Qaa2xzb5Ck4uusouiV+rjZjZR MIME-Version: 1.0 X-Received: by 10.66.155.102 with SMTP id vv6mr96263175pab.89.1388764634403; Fri, 03 Jan 2014 07:57:14 -0800 (PST) Received: by 10.66.249.232 with HTTP; Fri, 3 Jan 2014 07:57:14 -0800 (PST) In-Reply-To: <20140104005845.V35277@sola.nimnet.asn.au> References: <201311301000.rAUA00eG045983@freefall.freebsd.org> <52C66E09.80307@monkeybrains.net> <20140104005845.V35277@sola.nimnet.asn.au> Date: Fri, 3 Jan 2014 10:57:14 -0500 Message-ID: Subject: Re: Allowing routing table visibility in jails to make multiple IPs work properly From: Alejandro Imass To: Ian Smith Content-Type: text/plain; charset=ISO-8859-1 Cc: freebsd-jail@freebsd.org X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 03 Jan 2014 15:57:15 -0000 On Fri, Jan 3, 2014 at 9:10 AM, Ian Smith wrote: > On Fri, 3 Jan 2014 08:05:55 -0500, Alejandro Imass wrote: > > On Fri, Jan 3, 2014 at 3:00 AM, Rudy (bulk) wrote: > > > > > > I'm having issues when putting multiple IPs on a jail... one external, one > > > internal (on a different vlan). The source IP from the jail is always the > > > first IP, so a solution is to use ipfw_nat to nat when using the internal > > > vlan to the 'second ip'. Ugly hack. and it doesn't work when there is an > > > MTU difference between the vlans: > > > > > > > Greetings Rudy, > > > > I had the same exact problem and found that the problem is natd. > > Actually it is mentioned in natd's documentation. > > Alejandro, hi, > > can you point out where in natd(8) it indicates .. what exactly? > It's what natd does "It changes all packets destined for another host so that their source IP address is that of the current machine." The problem is that it chooses the first IP assigned to the interface so for example if you have several public IP's assigned to the same physical interface and assign one to each jail, any outbound connection from either jail will show the first IP regardless of what IP is assigned to what jail. In fact outbound connections from the base host will also show the first IP even if using the -b switch which make FBSD behave like Linux when natd is running. When natd is in operation all source address will always be the first IP address assigned to that interface. You can test this with outbound ssh even by forcing with the -b switch in an outbound ssh from a jail and you will see it uses the first IP always. Turn off natd and you will see it uses the correct IP. I had a long discussion a while back, check the archives. > > If you want to get rid of this problem you need to get rid of natd and > > nat your jail traffic with some other means. Kernel nat should be a > > solution but I've never gotten around to test if it actually solves > > the problem. Please share if you find a way to fix this. > > I may have missed it, but I've yet to see anyone report any functional > differences between natd and ipfw_nat, ie of something working in one > but not the other. Both use the underlying libalias(3) after all. > I have never been able to solve this but thought I read somewhere that by using specific ipfw nat it could be solved. I still have the problem and is not my expertise obviously and I haven't had the time to investigate the problem further. I just know that using natd causes any outbound connection from a jail to always show the first IP assigned to that interface. Best, Alejandro Imass