From owner-freebsd-jail@FreeBSD.ORG Fri May 3 00:54:36 2013 Return-Path: Delivered-To: freebsd-jail@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id EB7B1C55 for ; Fri, 3 May 2013 00:54:36 +0000 (UTC) (envelope-from fbsd8@a1poweruser.com) Received: from mail-03.name-services.com (mail-03.name-services.com [69.64.155.195]) by mx1.freebsd.org (Postfix) with ESMTP id D528D1EE5 for ; Fri, 3 May 2013 00:54:36 +0000 (UTC) Received: from [10.0.10.1] ([173.88.202.176]) by mail-03.name-services.com with Microsoft SMTPSVC(6.0.3790.4675); Thu, 2 May 2013 17:54:37 -0700 Message-ID: <51830AC9.9080708@a1poweruser.com> Date: Thu, 02 May 2013 20:54:33 -0400 From: Joe User-Agent: Thunderbird 2.0.0.17 (Windows/20080914) MIME-Version: 1.0 To: Anders Hagman Subject: Re: vnet jail with ipfw having logging problem References: <44AC45947DA14449AEDFB13B9F6C5F7DAF3E1FA5@ltcfiswmsgmb25> <517A7BCB.8060604@a1poweruser.com> <13CA24D6AB415D428143D44749F57D7201F22068@ltcfiswmsgmb21> <517D3426.1090703@a1poweruser.com> <51805EFB.6050806@a1poweruser.com> <20130502021830.O30818@sola.nimnet.asn.au> <51818C67.7070708@a1poweruser.com> <20130502142443.V30818@sola.nimnet.asn.au> <20130503010007.C30818@sola.nimnet.asn.au> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-OriginalArrivalTime: 03 May 2013 00:54:37.0370 (UTC) FILETIME=[C90879A0:01CE4798] X-Sender: fbsd8@a1poweruser.com X-Authenticated-Sender: fbsd8@a1poweruser.com X-EchoSenderHash: [fbsd8]-[a1poweruser*com] Cc: freebsd-jail , Ian Smith X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 03 May 2013 00:54:37 -0000 I am posting 2 console logs created using the script command. The main differences between the 2 is, log 1 is a 9.1 kernel with modules and vimage compiled in. This shows the first problem being that dynamically loaded ipfw with a vimage kernel don't work. Log 2 is a 9.1 kernel with modules and vimage plus ipfw compiled in. This shows the second problem with vnet jails running ipfw log to host security file and don't log any ipfw log messages to the hosts message file. Secondly the vnet jails security and messages files never get populated with ipfw log messages. Console log 1. 9.1-RELEASE ipfw dynamically loaded by firewall statements in hosts rc.conf with modules and only vimage compiled into kernel. logger cmd on host did not work until after vnet jail was started and stopped. vnet jail pings passed through vnet jail but was not handed to host ipfw. vnet jail pings got logged to hosts security file but not messages. After vnet jail stopped, host logger cmd works and host pings work and logged correctly to security and messages. # /root >sysctl net.inet.ip.fw.verbose net.inet.ip.fw.verbose: 1 # /root >sysctl net.inet.ip.fw.verbose_limit net.inet.ip.fw.verbose_limit: 0 # /root >cat /etc/rc.comf # snip firewall_enable="YES" firewall_logging="YES" firewall_script="/etc/ipfw.rules" # /root >logger security.notice this msg is from logger cmd on host # /root >cat /var/log/security empty file # /root >cat /var/log/messages empty file # /root >ping -c 4 freebsd.org PING freebsd.org (8.8.178.135): 56 data bytes 64 bytes from 8.8.178.135: icmp_seq=0 ttl=51 time=102.814 ms 64 bytes from 8.8.178.135: icmp_seq=1 ttl=51 time=84.625 ms 64 bytes from 8.8.178.135: icmp_seq=2 ttl=51 time=101.332 ms 64 bytes from 8.8.178.135: icmp_seq=3 ttl=51 time=120.662 ms --- freebsd.org ping statistics --- 4 packets transmitted, 4 packets received, 0.0% packet loss round-trip min/avg/max/stddev = 84.625/102.358/120.662/12.755 ms # /root >cat /var/log/messages empty file # /root >cat /var/log/security May 2 19:05:50 fbsdjones kernel: ipfw: 11 Accept UDP 10.0.10.5:42524 209.18.47.61:53 out via rl0 May 2 19:05:50 fbsdjones kernel: ipfw: 11 Accept UDP 209.18.47.61:53 10.0.10.5:42524 in via rl0 May 2 19:05:50 fbsdjones kernel: ipfw: 11 Accept ICMP:8.0 10.0.10.5 8.8.178.135 out via rl0 May 2 19:05:50 fbsdjones kernel: ipfw: 11 Accept ICMP:0.0 8.8.178.135 10.0.10.5 in via rl0 May 2 19:05:51 fbsdjones kernel: ipfw: 11 Accept ICMP:8.0 10.0.10.5 8.8.178.135 out via rl0 May 2 19:05:51 fbsdjones kernel: ipfw: 11 Accept ICMP:0.0 8.8.178.135 10.0.10.5 in via rl0 May 2 19:05:52 fbsdjones kernel: ipfw: 11 Accept ICMP:8.0 10.0.10.5 8.8.178.135 out via rl0 May 2 19:05:52 fbsdjones kernel: ipfw: 11 Accept ICMP:0.0 8.8.178.135 10.0.10.5 in via rl0 May 2 19:05:53 fbsdjones kernel: ipfw: 11 Accept ICMP:8.0 10.0.10.5 8.8.178.135 out via rl0 May 2 19:05:53 fbsdjones kernel: ipfw: 11 Accept ICMP:0.0 8.8.178.135 10.0.10.5 in via rl0 # /root >logger security.notice this msg is from logger cmd on host # /root >cat /var/log/security May 2 19:05:50 fbsdjones kernel: ipfw: 11 Accept UDP 10.0.10.5:42524 209.18.47.61:53 out via rl0 May 2 19:05:50 fbsdjones kernel: ipfw: 11 Accept UDP 209.18.47.61:53 10.0.10.5:42524 in via rl0 May 2 19:05:50 fbsdjones kernel: ipfw: 11 Accept ICMP:8.0 10.0.10.5 8.8.178.135 out via rl0 May 2 19:05:50 fbsdjones kernel: ipfw: 11 Accept ICMP:0.0 8.8.178.135 10.0.10.5 in via rl0 May 2 19:05:51 fbsdjones kernel: ipfw: 11 Accept ICMP:8.0 10.0.10.5 8.8.178.135 out via rl0 May 2 19:05:51 fbsdjones kernel: ipfw: 11 Accept ICMP:0.0 8.8.178.135 10.0.10.5 in via rl0 May 2 19:05:52 fbsdjones kernel: ipfw: 11 Accept ICMP:8.0 10.0.10.5 8.8.178.135 out via rl0 May 2 19:05:52 fbsdjones kernel: ipfw: 11 Accept ICMP:0.0 8.8.178.135 10.0.10.5 in via rl0 May 2 19:05:53 fbsdjones kernel: ipfw: 11 Accept ICMP:8.0 10.0.10.5 8.8.178.135 out via rl0 May 2 19:05:53 fbsdjones kernel: ipfw: 11 Accept ICMP:0.0 8.8.178.135 10.0.10.5 in via rl0 vnet jail gets started # /root >jls JID IP Address Hostname Path 2 - vdir2 /usr/jails/vdir2 # /root >jexec vdir2 tcsh vdir2 / >logger -p security.notice logger cmd msg from within the host vdir2 / >ipfw -a list 00010 0 0 allow ip from any to any via lo0 00011 0 0 allow log ip from any to any via epair2b 65535 5 368 deny ip from any to any vdir2 / >ping -c 4 freebsd.org ping: cannot resolve freebsd.org: Host name lookup failure vdir2 / >ipfw -a list 00010 0 0 allow ip from any to any via lo0 00011 8 480 allow log ip from any to any via epair2b 65535 5 368 deny ip from any to any vdir2 / >exit exit # back on the host # /root >cat /var/log/security May 2 19:05:50 fbsdjones kernel: ipfw: 11 Accept UDP 10.0.10.5:42524 209.18.47.61:53 out via rl0 May 2 19:05:50 fbsdjones kernel: ipfw: 11 Accept UDP 209.18.47.61:53 10.0.10.5:42524 in via rl0 May 2 19:05:50 fbsdjones kernel: ipfw: 11 Accept ICMP:8.0 10.0.10.5 8.8.178.135 out via rl0 May 2 19:05:50 fbsdjones kernel: ipfw: 11 Accept ICMP:0.0 8.8.178.135 10.0.10.5 in via rl0 May 2 19:05:51 fbsdjones kernel: ipfw: 11 Accept ICMP:8.0 10.0.10.5 8.8.178.135 out via rl0 May 2 19:05:51 fbsdjones kernel: ipfw: 11 Accept ICMP:0.0 8.8.178.135 10.0.10.5 in via rl0 May 2 19:05:52 fbsdjones kernel: ipfw: 11 Accept ICMP:8.0 10.0.10.5 8.8.178.135 out via rl0 May 2 19:05:52 fbsdjones kernel: ipfw: 11 Accept ICMP:0.0 8.8.178.135 10.0.10.5 in via rl0 May 2 19:05:53 fbsdjones kernel: ipfw: 11 Accept ICMP:8.0 10.0.10.5 8.8.178.135 out via rl0 May 2 19:05:53 fbsdjones kernel: ipfw: 11 Accept ICMP:0.0 8.8.178.135 10.0.10.5 in via rl0 May 2 19:10:50 fbsdjones kernel: ipfw: 11 Accept UDP 10.2.0.2:32606 209.18.47.61:53 out via epair2b May 2 19:10:55 fbsdjones kernel: ipfw: 11 Accept UDP 10.2.0.2:29810 209.18.47.62:53 out via epair2b May 2 19:10:57 fbsdjones kernel: ipfw: 11 Accept UDP 10.2.0.2:32606 209.18.47.61:53 out via epair2b May 2 19:11:00 fbsdjones kernel: ipfw: 11 Accept UDP 10.2.0.2:35933 209.18.47.61:53 out via epair2b May 2 19:11:05 fbsdjones kernel: ipfw: 11 Accept UDP 10.2.0.2:56823 209.18.47.62:53 out via epair2b May 2 19:11:07 fbsdjones kernel: ipfw: 11 Accept UDP 10.2.0.2:35933 209.18.47.61:53 out via epair2b May 2 19:11:07 fbsdjones kernel: ipfw: 11 Accept UDP 10.2.0.2:29810 209.18.47.62:53 out via epair2b May 2 19:11:17 fbsdjones kernel: ipfw: 11 Accept UDP 10.2.0.2:56823 209.18.47.62:53 out via epair2b May 2 19:11:22 fbsdjones kernel: ipfw: 11 Accept UDP 10.2.0.2:37981 209.18.47.61:53 out via epair2b May 2 19:11:27 fbsdjones kernel: ipfw: 11 Accept UDP 10.2.0.2:24567 209.18.47.62:53 out via epair2b May 2 19:11:29 fbsdjones kernel: ipfw: 11 Accept UDP 10.2.0.2:37981 209.18.47.61:53 out via epair2b May 2 19:11:39 fbsdjones kernel: ipfw: 11 Accept UDP 10.2.0.2:24567 209.18.47.62:53 out via epair2b May 2 19:11:44 fbsdjones kernel: ipfw: 11 Accept UDP 10.2.0.2:54854 209.18.47.61:53 out via epair2b May 2 19:11:49 fbsdjones kernel: ipfw: 11 Accept UDP 10.2.0.2:33964 209.18.47.62:53 out via epair2b May 2 19:11:51 fbsdjones kernel: ipfw: 11 Accept UDP 10.2.0.2:54854 209.18.47.61:53 out via epair2b # /root >logger -p security.notice host logger msg # /root >cat /var/log/security May 2 19:11:39 fbsdjones kernel: ipfw: 11 Accept UDP 10.2.0.2:24567 209.18.47.62:53 out via epair2b May 2 19:11:44 fbsdjones kernel: ipfw: 11 Accept UDP 10.2.0.2:54854 209.18.47.61:53 out via epair2b May 2 19:11:49 fbsdjones kernel: ipfw: 11 Accept UDP 10.2.0.2:33964 209.18.47.62:53 out via epair2b May 2 19:11:51 fbsdjones kernel: ipfw: 11 Accept UDP 10.2.0.2:54854 209.18.47.61:53 out via epair2b May 2 19:12:01 fbsdjones kernel: ipfw: 11 Accept UDP 10.2.0.2:33964 209.18.47.62:53 out via epair2b May 2 19:12:50 fbsdjones root: host logger msg # /root >cat /var/log/messages May 2 19:08:10 fbsdjones kernel: bridge0: Ethernet address: 02:8f:94:84:0c:00 May 2 19:08:10 fbsdjones kernel: bridge0: link state changed to UP May 2 19:08:10 fbsdjones kernel: epair2a: Ethernet address: 02:c0:a4:00:0a:0a May 2 19:08:10 fbsdjones kernel: epair2b: Ethernet address: 02:c0:a4:00:0b:0b May 2 19:08:10 fbsdjones kernel: epair2a: link state changed to UP May 2 19:08:10 fbsdjones kernel: epair2b: link state changed to UP May 2 19:12:50 fbsdjones root: host logger msg Console log 2. This test run is using 9.1-RELEASE with modules plus vimage and ipfw compiled in. options IPFIREWALL options IPFIREWALL_VERBOSE options IPFIREWALL_DEFAULT_TO_ACCEPT logger command works. logged msg in both security and messages on host vnet jail can ping the public internet. Hosts security file has log messages from both jail and host. ipfw log messages are not being put into the hosts messages file. # ran on the host # /root >sysctl net.inet.ip.fw.verbose net.inet.ip.fw.verbose: 1 # /root >sysctl net.inet.ip.fw.verbose_limit net.inet.ip.fw.verbose_limit: 0 # /root >ipfw -a list 00010 0 0 allow ip from any to any via lo0 00011 0 0 allow log ip from any to any via rl0 65535 1 328 allow ip from any to any # /root >/var/log/security empty file # /root >cat /var/log/messages empty file # /root >logger -p security.notice host logger cmd 1 # /root >cat /var/log/security May 2 19:45:51 fbsdjones root: host logger cmd 1 # /root >cat /var/log/messages May 2 19:45:51 fbsdjones root: host logger cmd 1 # /root >ipfw -a list 00010 0 0 allow ip from any to any via lo0 00011 0 0 allow log ip from any to any via rl0 65535 1 328 allow ip from any to any # /root >ping -c 3 freebsd.org PING freebsd.org (8.8.178.135): 56 data bytes 64 bytes from 8.8.178.135: icmp_seq=0 ttl=51 time=85.032 ms 64 bytes from 8.8.178.135: icmp_seq=1 ttl=51 time=84.381 ms 64 bytes from 8.8.178.135: icmp_seq=2 ttl=51 time=84.647 ms --- freebsd.org ping statistics --- 3 packets transmitted, 3 packets received, 0.0% packet loss round-trip min/avg/max/stddev = 84.381/84.687/85.032/0.267 ms # /root >ipfw -a list 00010 0 0 allow ip from any to any via lo0 00011 9 869 allow log ip from any to any via rl0 65535 1 328 allow ip from any to any vnet jail started # /root >jls JID IP Address Hostname Path 1 - vdir2 /usr/jails/vdir2 # /root >jexec vdir2 tcsh vdir2 / >cat /etc/ipfw.rules # Flush out the list before we begin. ipfw -q -f flush cmd="ipfw -q add" if [ -e /etc/epair ]; then pif=`cat "/etc/epair"` rm /etc/epair else pif="lo0" fi $cmd 010 allow all from any to any via lo0 $cmd 011 allow log all from any to any via $pif vdir2 / >ipfw -a list 00010 0 0 allow ip from any to any via lo0 00011 0 0 allow log ip from any to any via epair1b 65535 8 624 allow ip from any to any vdir2 / >ping -c 3 freebsd.org PING freebsd.org (8.8.178.135): 56 data bytes 64 bytes from 8.8.178.135: icmp_seq=0 ttl=51 time=84.342 ms 64 bytes from 8.8.178.135: icmp_seq=1 ttl=51 time=84.195 ms 64 bytes from 8.8.178.135: icmp_seq=2 ttl=51 time=84.015 ms --- freebsd.org ping statistics --- 3 packets transmitted, 3 packets received, 0.0% packet loss round-trip min/avg/max/stddev = 84.015/84.184/84.342/0.134 ms vdir2 / >ipfw -a list 00010 0 0 allow ip from any to any via lo0 00011 8 634 allow log ip from any to any via epair1b 65535 8 624 allow ip from any to any vdir2 / >cat /var/log/security May 1 21:56:27 vdir2 newsyslog[5202]: logfile first created vdir2 / >cat /var/log/messages May 1 21:56:27 vdir2 newsyslog[5202]: logfile first created vdir2 / >exit exit Back on the host # /root >cat /var/log/security May 2 19:45:51 fbsdjones root: host logger cmd 1 May 2 19:46:53 fbsdjones kernel: ipfw: 11 Accept UDP 10.0.10.1:138 10.0.10.7:138 in via rl0 May 2 19:46:58 fbsdjones kernel: ipfw: 11 Accept UDP 10.0.10.5:64721 209.18.47.61:53 out via rl0 May 2 19:46:58 fbsdjones kernel: ipfw: 11 Accept UDP 209.18.47.61:53 10.0.10.5:64721 in via rl0 May 2 19:46:58 fbsdjones kernel: ipfw: 11 Accept ICMP:8.0 10.0.10.5 8.8.178.135 out via rl0 May 2 19:46:58 fbsdjones kernel: ipfw: 11 Accept ICMP:0.0 8.8.178.135 10.0.10.5 in via rl0 May 2 19:46:59 fbsdjones kernel: ipfw: 11 Accept ICMP:8.0 10.0.10.5 8.8.178.135 out via rl0 May 2 19:46:59 fbsdjones kernel: ipfw: 11 Accept ICMP:0.0 8.8.178.135 10.0.10.5 in via rl0 May 2 19:47:00 fbsdjones kernel: ipfw: 11 Accept ICMP:8.0 10.0.10.5 8.8.178.135 out via rl0 May 2 19:47:00 fbsdjones kernel: ipfw: 11 Accept ICMP:0.0 8.8.178.135 10.0.10.5 in via rl0 May 2 19:47:38 fbsdjones kernel: ipfw: 11 Accept ICMPv6:143.0 [::] [ff02::16] out via rl0 May 2 19:47:38 fbsdjones kernel: ipfw: 11 Accept ICMPv6:143.0 [::] [ff02::16] out via rl0 May 2 19:47:39 fbsdjones kernel: ipfw: 11 Accept ICMPv6:135.0 [::] [ff02::1:ff00:b0b] out via rl0 May 2 19:47:39 fbsdjones kernel: ipfw: 11 Accept ICMPv6:143.0 [::] [ff02::16] out via rl0 May 2 19:49:21 fbsdjones kernel: ipfw: 11 Accept UDP 10.1.0.2:13101 209.18.47.61:53 out via epair1b May 2 19:49:21 fbsdjones kernel: ipfw: 11 Accept UDP 10.1.0.2:13101 209.18.47.61:53 out via rl0 May 2 19:49:21 fbsdjones kernel: ipfw: 11 Accept UDP 209.18.47.61:53 10.1.0.2:13101 in via rl0 May 2 19:49:21 fbsdjones kernel: ipfw: 11 Accept UDP 209.18.47.61:53 10.1.0.2:13101 in via rl0 May 2 19:49:21 fbsdjones kernel: ipfw: 11 Accept UDP 209.18.47.61:53 10.1.0.2:13101 in via epair1b May 2 19:49:21 fbsdjones kernel: ipfw: 11 Accept ICMP:8.0 10.1.0.2 8.8.178.135 out via epair1b May 2 19:49:21 fbsdjones kernel: ipfw: 11 Accept ICMP:8.0 10.1.0.2 8.8.178.135 out via rl0 May 2 19:49:21 fbsdjones kernel: ipfw: 11 Accept ICMP:0.0 8.8.178.135 10.1.0.2 in via rl0 May 2 19:49:21 fbsdjones kernel: ipfw: 11 Accept ICMP:0.0 8.8.178.135 10.1.0.2 in via rl0 May 2 19:49:21 fbsdjones kernel: ipfw: 11 Accept ICMP:0.0 8.8.178.135 10.1.0.2 in via epair1b May 2 19:49:22 fbsdjones kernel: ipfw: 11 Accept ICMP:8.0 10.1.0.2 8.8.178.135 out via epair1b May 2 19:49:22 fbsdjones kernel: ipfw: 11 Accept ICMP:8.0 10.1.0.2 8.8.178.135 out via rl0 May 2 19:49:22 fbsdjones kernel: ipfw: 11 Accept ICMP:0.0 8.8.178.135 10.1.0.2 in via rl0 May 2 19:49:22 fbsdjones kernel: ipfw: 11 Accept ICMP:0.0 8.8.178.135 10.1.0.2 in via rl0 May 2 19:49:22 fbsdjones kernel: ipfw: 11 Accept ICMP:0.0 8.8.178.135 10.1.0.2 in via epair1b May 2 19:49:23 fbsdjones kernel: ipfw: 11 Accept ICMP:8.0 10.1.0.2 8.8.178.135 out via epair1b May 2 19:49:23 fbsdjones kernel: ipfw: 11 Accept ICMP:8.0 10.1.0.2 8.8.178.135 out via rl0 May 2 19:49:23 fbsdjones kernel: ipfw: 11 Accept ICMP:0.0 8.8.178.135 10.1.0.2 in via rl0 May 2 19:49:23 fbsdjones kernel: ipfw: 11 Accept ICMP:0.0 8.8.178.135 10.1.0.2 in via rl0 May 2 19:49:23 fbsdjones kernel: ipfw: 11 Accept ICMP:0.0 8.8.178.135 10.1.0.2 in via epair1b # /root >cat /var/log/messages May 2 19:45:51 fbsdjones root: host logger cmd 1 May 2 19:47:38 fbsdjones kernel: bridge0: Ethernet address: 02:8f:94:84:0c:00 May 2 19:47:38 fbsdjones kernel: bridge0: link state changed to UP May 2 19:47:38 fbsdjones kernel: epair1a: Ethernet address: 02:c0:24:00:0a:0a May 2 19:47:38 fbsdjones kernel: epair1b: Ethernet address: 02:c0:24:00:0b:0b May 2 19:47:38 fbsdjones kernel: epair1a: link state changed to UP May 2 19:47:38 fbsdjones kernel: epair1b: link state changed to UP May 2 19:50:59 fbsdjones kernel: epair1a: link state changed to DOWN May 2 19:50:59 fbsdjones kernel: epair1b: link state changed to DOWN May 2 19:50:59 fbsdjones kernel: bridge0: link state changed to DOWN May 2 19:51:02 fbsdjones kernel: Freed UMA keg was not empty (30 items). Lost 2 pages of memory. May 2 19:51:02 fbsdjones kernel: Freed UMA keg was not empty (203 items). Lost 1 pages of memory. May 2 19:51:02 fbsdjones kernel: Freed UMA keg was not empty (30 items). Lost 2 pages of memory. May 2 19:51:02 fbsdjones kernel: hhook_vnet_uninit: hhook_head type=1, id=1 cleanup required May 2 19:51:02 fbsdjones kernel: hhook_vnet_uninit: hhook_head type=1, id=0 cleanup required # /root >exit exit