Date: Thu, 2 May 2013 22:05:49 +0200 From: Anders Hagman <anders.hagman@netplex.se> To: Ian Smith <smithi@nimnet.asn.au> Cc: freebsd-jail <freebsd-jail@freebsd.org> Subject: Re: vnet jail with ipfw having logging problem Message-ID: <E0B1754F-2A61-43A3-BA2F-479915A10AB3@netplex.se> In-Reply-To: <20130503010007.C30818@sola.nimnet.asn.au> References: <44AC45947DA14449AEDFB13B9F6C5F7DAF3E1FA5@ltcfiswmsgmb25> <517A7BCB.8060604@a1poweruser.com> <13CA24D6AB415D428143D44749F57D7201F22068@ltcfiswmsgmb21> <517D3426.1090703@a1poweruser.com> <51805EFB.6050806@a1poweruser.com> <20130502021830.O30818@sola.nimnet.asn.au> <51818C67.7070708@a1poweruser.com> <20130502142443.V30818@sola.nimnet.asn.au> <FC2D92AC-207F-4AC3-8B57-E837EF8DD779@netplex.se> <20130503010007.C30818@sola.nimnet.asn.au>
next in thread | previous in thread | raw e-mail | index | archive | help
2 maj 2013 kl. 18:46 skrev Ian Smith <smithi@nimnet.asn.au>: > On Thu, 2 May 2013 12:09:08 +0200, Anders Hagman wrote: >> Hi > Yo >> 2 maj 2013 kl. 07:42 skrev Ian Smith <smithi@nimnet.asn.au>: >>=20 >>> On Wed, 1 May 2013 17:43:03 -0400, Joe wrote: >>>>>> I have ipfw running inside of a vnet jail on a 9.1-RELEASE host = using >>>>> the >>>>>> jail(8) definition statements for starting and stopping the vnet = jail. >>>>> As a >>>>>> side note non-vnet jails are working as expected. >>>>>>> The host is running a custom kernel with modules and with >>>>>> options VIMAGE >>>>>> nooptions SCTP >>>>>> options IPFIREWALL >>>>>> options IPFIREWALL_VERBOSE >>>>>> options IPFIREWALL_VERBOSE_LIMIT=3D10 >>>=20 >>> Please maintain attributions for the archives. I wrote: >>>=20 >>>>> What steps have you taken during testing to override this = ridiculously low >>>>> limit on logging? Otherwise, after e.g. just 5 pings and 5 ping = responses >>>>> are logged, all logging ceases until issuing 'ipfw resetlog'. >>>>=20 >>>> /usr/src/sys/conf/NOTES says IPFIREWALL_VERBOSE_LIMIT; limits the = number of >>>> times a matching entry can be logged. Says nothing about this limit = being the >>>> maximum number of log records allowed after which the log file is = closed for >>>> business. Are you saying the /usr/src/sys/conf/NOTES info is no = longer true? >>>=20 >>> You showed one (1) 'log' rule for each of the host's and jail's = ruleset.=20 >>> Once that one rule has been logged 'logamount' times (default as per=20= >>> NOTES is 100, but in your case is 10) then logging for THAT rule = stops,=20 >>> therefore with only one 'log' rule, ALL logging stops. Understand? >>>=20 >>> If you take the time to properly study the correct reference, = ipfw(8),=20 >>> all of this will become clear. See especially section SYSCTL = VARIABLES,=20 >>> and read thoroughly 'log [logamount number]', at the very least. = Ignore=20 >>> the Handbook section on ipfw, it's full of errors and = misunderstandings. >>>=20 >>>> Without IPFIREWALL_VERBOSE and IPFIREWALL_VERBOSE_LIMIT where does = the logged >>>> packets get written to? /var/log/security >>>=20 >>> See above. Both of these options merely set defaults for the = sysctls. >>>=20 >>>> I have not used ipfw since it's ipfw2 rewrite so my knowledge is = dated. >>>=20 >>> Indeed it is; that's a very long time ago. >>>=20 >>>>>> options IPFIREWALL_DEFAULT_TO_ACCEPT >>>>>> options IPFIREWALL_IPDIVERT >>>>>=20 >>>>> You'd likely do better using in-kernel NAT; natd doesn't get much = love. >>>>>=20 >>>>=20 >>>> I kept getting kernel compile errors using "options = IPFIREWALL_NAT". I >>>> thought the error was caused by vimage. Now I know "options = LIBALIAS" is >>>> required. Could not find info on internet search for IPFIREWALL_NAT = with >>>> vimage kernel. >>>=20 >>> Apart from FIREWALL_FORWARD (not even that in 10.x), none of that = needs=20 >>> to be in the kernel, it's all loadable as modules; see = /etc/rc.d/ipfw. >>>=20 >>> If you're doing NAT in the vimage jail, you must have at least two=20= >>> interfaces assigned to the jail. Care to show your config for that? >>>=20 >>>> Do you have first hand experience getting "ipfw kernel nat" to work = in a >>>> vimage jail or having logging work on the host and within the vnet = jail? >>>=20 >>> No, but I have just on 15 years experience managing ipfw firewalls = :) >>=20 >> When you are new at things you do mistakes, remember. >=20 > I still do mistakes. Trying to teach fishing rather than just tossing=20= > another fish is often one of mine :) I'm glad you had some to spare. I know the game. ;-> >=20 >> To try to answer Joes question: >>=20 >> You don't need to compile anything into the kernel regarding ipfw. >>=20 >> Just load the ipfw module in the host system with: >>=20 >> kldload ipfw >>=20 >> By default a deny all rule is added, so add a allow rule to the host = system. >>=20 >> ipfw add 10 allow ip from any to any >>=20 >> To log things you change the sysctl value net.inet.ip.fw.verbose to 1 >>=20 >> sysctl net.inet.ip.fw.verbose=3D1 >>=20 >> If you keep net.inet.ip.fw.verbose_limit=3D0 you don't have a log = limit, and for tests thats fine. >=20 > Sure, though the default of 100 is plenty for such tests; it's=20 > surprisingly easy to DoS syslogd with e.g. a logged flood ping .. >=20 >> log in to the jail system. Change the sysctl value = net.inet.ip.fw.verbose to 1 >>=20 >> sysctl net.inet.ip.fw.verbose=3D1 >>=20 >> Add a logging firewall rule >>=20 >> ipfw add 10 allow log ip from any to any >>=20 >> Do a ping to an external system. >> Look inside /var/log/security in the jail system and its empty. >=20 > But it does exist, rw for root, with 0 or more bytes, right? And does=20= > the vimage jail's /etc/syslog.conf contain: > security.* /var/log/security >=20 Yes > That is, I'm checking that the jail's syslogd should be handling = these. =20 > What happens if you run in the jail, say: > # logger -p security.info Syslog, wherefore art thou, Syslog? > Does that go to the jail's /var/log/security? or the host's? In jail system webben: logger -p security.info Syslog, wherefore art thou, Syslog? tail /var/log/security May 2 21:24:48 webben root: Syslog, wherefore art thou, Syslog? >=20 >> Go to the main host and look at the /var/log/security file and you = will find log entries. >=20 > Showing the host's hostname, or the jail's? Can you post some = examples? In host system dator5: tail /var/log/security May 2 21:29:15 dator5 kernel: ipfw: 10 Accept TCP 10.2.0.101:80 = 94.153.64.32:3085 out via vlan101 May 2 21:29:15 dator5 kernel: ipfw: 10 Accept TCP 94.153.64.32:3085 = 10.2.0.101:80 in via vlan101 >=20 >> I can confirm Joes bug. I don't have a log rule in the main host but = still get log messages. >> All log messages are from the log rule in the jail system. >>=20 >> System used: 9.1-RELEASE-p2 >>=20 >> BR >> /Anders >=20 > Ok, before determining that this is an ipfw-only issue - in which case=20= > we need to move it over to freebsd-ipfw@ - can you confirm that normal=20= > syslogging in the jail to /var/log/messages and such is working? >=20 In jail system login anders password ***** tail /var/log/messages May 2 21:41:57 webben login: login_getclass: unknown class 'svensk' May 2 21:42:00 webben last message repeated 3 times > In particular I'm wondering what happens when you do set (say)=20 > net.inet.ip.fw.verbose_limit=3D10 and then ping from the jail until=20 > logging stops .. you should then see a message such as >=20 > Apr 23 23:42:05 sola kernel: ipfw: limit 500 reached on entry 26400 >=20 > both in /var/log/security and in /var/log/messages since it's logged=20= > as security.notice and default syslog.conf is for *.notice to log to > /var/log/messages .. see the tail of /sys/netpfil/ipfw/ip_fw_log.c >=20 > Yes sure, I'm flying blind, don't have a system with jails here yet, = and=20 > am making assumptions about how syslogd(8) should work in jails that I=20= > really don't have time to properly research currently, nor am I = properly > across all the security implications of (particularly vimage) jails. >=20 On jail system: =20 sysctl net.inet.ip.fw.verbose_limit=3D10 Pinging repeatedly. Just continue to log to host system. Add new ipfw log role will use the new limit: ipfw add 5 allow log ip from any to any 00005 allow log logamount 10 ip from any to any New ping test. /var/log/security in host system : May 2 21:52:28 dator5 kernel: ipfw: 5 Accept ICMP:8.0 10.2.0.101 = 195.49.241.132 out via vlan101 May 2 21:52:28 dator5 kernel: ipfw: 5 Accept ICMP:0.0 195.49.241.132 = 10.2.0.101 in via vlan101 May 2 21:52:28 dator5 kernel: ipfw: limit 10 reached on entry 5 /var/log/messages in host system : May 2 21:52:28 dator5 kernel: ipfw: limit 10 reached on entry 5 Nothing at all is logged to the jail syslog. BR /Anders
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?E0B1754F-2A61-43A3-BA2F-479915A10AB3>