From owner-freebsd-stable@FreeBSD.ORG Wed Jan 31 12:29:09 2007 Return-Path: X-Original-To: freebsd-stable@freebsd.org Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 1440B16A400 for ; Wed, 31 Jan 2007 12:29:09 +0000 (UTC) (envelope-from jeff@sailorfej.net) Received: from mail.sailorfej.net (mail.sailorfej.net [66.93.72.123]) by mx1.freebsd.org (Postfix) with ESMTP id DF59E13C441 for ; Wed, 31 Jan 2007 12:29:08 +0000 (UTC) (envelope-from jeff@sailorfej.net) Received: from [192.168.150.100] (c-24-20-239-104.hsd1.mn.comcast.net [24.20.239.104]) (authenticated bits=0) by mail.sailorfej.net (8.13.4/8.13.4) with ESMTP id l0VCClgQ068560 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Wed, 31 Jan 2007 04:12:48 -0800 (PST) (envelope-from jeff@sailorfej.net) Message-ID: <45C08B7E.7090303@sailorfej.net> Date: Wed, 31 Jan 2007 04:28:46 -0800 From: Jeffrey Williams User-Agent: Thunderbird 1.5.0.9 (Windows/20061207) MIME-Version: 1.0 To: freebsd-stable@freebsd.org References: <200701311153.l0VBrn4D068132@lurza.secnetix.de> In-Reply-To: <200701311153.l0VBrn4D068132@lurza.secnetix.de> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Status: No, score=-2.6 required=6.0 tests=BAYES_00 autolearn=failed version=3.1.1 X-Spam-Checker-Version: SpamAssassin 3.1.1 (2006-03-10) on mail.sailorfej.net Subject: Re: jails and multple interfaces X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 31 Jan 2007 12:29:09 -0000 Oliver Fromme wrote: > Jeffrey Williams wrote: > > I am trying to set a jail hosting server to support multiple jails for > > development testing. > > > > The server has two network interfaces, I am configuring one for host > > server to use, and the other with several aliased IPs, one for each of > > the jail servers. > > > > All the services running on the host are configured to bind to the host > > IP on the first interface. > > > > The crux is both interfaces on the same network, I am seeing the > > expected arp errors (e.g. kernel: arp: x.x.x.x is on int0 but got reply > > on int1), now I know I set the sysctl variable > > net.link.ether.inet.log_arp_wrong_iface=0 to get rid of these messages, > > but what I want to know if there are any other problems I am going to > > have having both interfaces live on the same network. > > What exactly are your inetrface configurations and netmasks > (ifconfig output might be useful)? > > You say that both NICs are on teh same network. Does that > mean they're connected to teh same switch? That's generally > not a good idea. It doesn't buy you anything (unless you > use VLAN technology or other additional measures). > > > Also even though > > I have the jail host's services all binding to the first interfaces ip, > > there is not guarantee that network traffic originating from the jail > > host will only use its primary interface/IP, is their anyway to ensure > > that the jail host does not try to talk through the interface being used > > by the jails? > > Any network traffic originating from a jail is guaranteed > to use the jail's IP address. The interface that will be > used is the one according to your routing table entry for > that IP address. (Unless you use things like IPFW FWD > or similar to redirect the packets somewhere else.) > > Best regards > Oliver > > PS: Be very careful when binding services to localhost > (127.0.0.1) within the jail. They will listen on the jail's > official IP address instead! For that reason I often > configure an addition address on lo0 (e.g. 127.0.0.2) > and use that one for internal-only traffic such as DNS > and mail between host and jails. > Yes, both NIC's are on the same network, connected to the same switch, I not just using the jail's as a sandbox for a couple of services, rather I am using a virtual servers, ideally I would like to have a separate NIC for eash jail, just like I would do if I were setting these up in vmware. It is currently setup and running, at the moment, as near as I can tell I am not having any IP routing issues, all the appropriate configs are below. I don't believe (I could be wrong, definitely tell me if I am) I am going to have and IP issues (i.e. layer 3), what I am concerned about are the potential ethernet issues ARP/RARP (i.e. layer 2). I was getting the expected arp errors, but I did set the net.link.ether.inet.log_arp_wrong_iface to 0, so I am not logging the errors, of course that doesn't mean the errors went away, they just aren't clogging the log files now. What I want to know is, if there are any deleterious effects from having two ethernet interfaces on the same network segment/switch. Frankly I would be very concerned if there wasn't a way to pull this off, while I realize jails were not originally intended to be used as a mechanism to provide virtual servers, it is such an obvious application of the jail functionality, and it is perfectly reasonable to want to have a dedicated NIC for each virtual server, with out having to place each one on a separate network segment. Configurations: ############rc.conf # Please make all changes to this file, not to /etc/defaults/rc.conf. # This file now contains just the overrides from /etc/defaults/rc.conf. defaultrouter="192.168.10.1" hostname="dev.inside.mydomain.com" ifconfig_em0="inet 192.168.10.41 netmask 255.255.255.0" ifconfig_vr0="inet 192.168.10.70 netmask 255.255.255.0" ifconfig_vr0_alias0="192.168.10.71 netmask 255.255.255.0" ifconfig_vr0_alias1="192.168.10.72 netmask 255.255.255.0" ifconfig_vr0_alias2="192.168.10.73 netmask 255.255.255.0" ifconfig_vr0_alias3="192.168.10.74 netmask 255.255.255.0" syslogd_flags="-s -b 192.168.10.41" sendmail_enable="NO" moused_enable="YES" sshd_enable="YES" usbd_enable="YES" ntpd_enable="YES" rpcbind_enable="NO" inetd_flags="-wW -a 192.168.10.41" jail_enable="YES" jail_list="test1 test2 test3 test4 test5" jail_set_hostname_allow="NO" jail_exec_start="/bin/sh /etc/rc" jail_exec_stop="/bin/sh /etc/rc.shutdown" jail_devfs_enable="YES" jail_test1_rootdir="/jails/test1" jail_test1_hostname="test1.inside.mydomain.com" jail_test1_ip="192.168.10.70" jail_test2_rootdir="/jails/test2" jail_test2_hostname="test2.inside.mydomain.com" jail_test2_ip="192.168.10.71" jail_test3_rootdir="/jails/test3" jail_test3_hostname="test3.inside.mydomain.com" jail_test3_ip="192.168.10.72" jail_test4_rootdir="/jails/test4" jail_test4_hostname="test4.inside.mydomain.com" jail_test4_ip="192.168.10.73" jail_test5_rootdir="/jails/test5" jail_test5_hostname="test5.inside.mydomain.com" jail_test5_ip="192.168.10.74" ################# netstat -r Routing tables Internet: Destination Gateway Flags Refs Use Netif Expire default gateway UGS 0 25149 em0 localhost localhost UH 0 1230 lo0 192.168.10 link#1 UC 0 0 em0 gateway link#1 UHLW 2 0 em0 scope 00:13:d3:95:44:80 UHLW 1 22959 em0 559 ns1 00:13:d3:95:43:6c UHLW 1 43 em0 980 ns2 00:13:d3:95:43:90 UHLW 1 25 em0 562 dev 00:11:25:57:e8:a1 UHLW 1 0 lo0 test1 00:13:46:e7:04:bc UHLW 1 24 em0 980 test2 00:13:46:e7:04:bc UHLW 1 24 em0 978 test3 00:13:46:e7:04:bc UHLW 1 24 em0 981 test4 00:13:46:e7:04:bc UHLW 1 24 em0 981 test5 00:13:46:e7:04:bc UHLW 1 24 em0 981 Internet6: Destination Gateway Flags Netif Expire localhost.inside localhost.inside UHL lo0 fe80::%lo0 fe80::1%lo0 U lo0 fe80::1%lo0 link#4 UHL lo0 ff01:4:: fe80::1%lo0 UC lo0 ff02::%lo0 fe80::1%lo0 UC lo0 ######################## ifconfig em0: flags=8843 mtu 1500 options=b inet 192.168.10.41 netmask 0xffffff00 broadcast 192.168.10.255 ether 00:11:25:57:e8:a1 media: Ethernet autoselect (100baseTX ) status: active vr0: flags=8843 mtu 1500 inet 192.168.10.70 netmask 0xffffff00 broadcast 192.168.10.255 inet 192.168.10.71 netmask 0xffffff00 broadcast 192.168.10.255 inet 192.168.10.72 netmask 0xffffff00 broadcast 192.168.10.255 inet 192.168.10.73 netmask 0xffffff00 broadcast 192.168.10.255 inet 192.168.10.74 netmask 0xffffff00 broadcast 192.168.10.255 ether 00:13:46:e7:04:bc media: Ethernet autoselect (100baseTX ) status: active plip0: flags=108810 mtu 1500 lo0: flags=8049 mtu 16384 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x4 inet6 ::1 prefixlen 128 inet 127.0.0.1 netmask 0xff000000 Thanks Jeff