Date: Sun, 23 Dec 2007 01:49:39 +1100 (EST) From: Ian Smith <smithi@nimnet.asn.au> To: "W. D." <WD@US-Webmasters.com> Cc: Tuomo Latto <djv@iki.fi>, freebsd-questions@freebsd.org Subject: Re: IPFW: Blocking me out. How to debug? Message-ID: <Pine.BSF.3.96.1071222150144.22888A-100000@gaia.nimnet.asn.au> In-Reply-To: <20071221073023.70F3113C447@mx1.freebsd.org>
next in thread | previous in thread | raw e-mail | index | archive | help
Warning: overlong message. I'm moving this to questions@ from security@ as it's a usage issue. Anyone wishing to follow the up to here can read from: http://lists.freebsd.org/pipermail/freebsd-security/2007-December/004541.html On Fri, 21 Dec 2007, W. D. wrote: > Date: Fri, 21 Dec 2007 01:30:11 -0600 > From: "W. D." <WD@US-Webmasters.com> > To: FreeBSD-Security@freebsd.org > Cc: Ian Smith <info@plot.uz> Never been to Uzbekistan and don't know this bloke :) > At 05:45 12/20/2007, Ian Smith, wrote: > > Thanks for your reply Ian. This is the kind of > information I am looking for. > > >Firstly, this really belongs over on freebsd-net@ if not > >freebsd-questions@, but anyway .. > > I'll be glad to move it there if you would like. I > figured that since IPFW/Firewalls are security > related, that FreeBSD-Security would be the most > appropriate place. > > >On Thu, 20 Dec 2007, W. D. wrote: > > > > > At 03:49 12/17/2007, Tuomo Latto wrote: > > > >W. D. wrote: > > > >> How do I tell which rule is blocking me out? SSH *is* working, > > > >> but others are not. > > > > > > > >It all depends on what you mean by "blocking you out" and "others". > > > >True; it's not really clear what you're trying to do, whether this is a > >single server with a single net interface with no NAT or what, but based > >on your present rules I'll have to make that assumption. > > OK, sorry. I guess I just assumed that it would be obvious > that this is a Web server. ("Never assume anything, my good > fellow" - Sherlock Holmes). Ok, and sorry I needed a days' sleep + $life before getting back to you. Here are many people who can and likely will offer opinions and advice. > By the way, it is/will be running Plesk server management > software, if it matters: > http://www.swsoft.com/en/products/plesk/reqs/ I know nothing of Plesk, but doubt it's relevant to this now. > Also, this server is on an internal LAN before I subject > it to the wild, untamed, InterWeb, with its dangerous > internets darting back and forth inside all of the tubes. Really good idea :) > > > >> add allow all from any to any via lo0 > > > >> add deny ip from any to 127.0.0.0/8 > > > >> add deny ip from 127.0.0.0/8 to any Ok. > > > >> # Allow established connections: > > > >> add allow tcp from any to any established > > > >That's ok. It may help you in debugging what's happening to use: > > > > allow [log] tcp from any to any in established > > allow [log] tcp from any to any out established > > I assume here that "[log]" means to insert "log" for > debugging like this: > > allow log tcp from any to any in established > allow log tcp from any to any out established > > rather than including the square brackets, "[" & "]", > correct? Yeah sorry, I meant [log] as in optionally just for debugging. > I have done that and have included my latest ruleset > below. > > >and really, using 'any to any' without specifying on which interfaces or > >whether 'any' is your box or the outside world is a bit too general, but > >moving on .. > > OK. What should I do? I only plan on having one > Ethernet interface. What would be more secure? In that case 'me to any' or 'any to me' provides unambiguous direction where appropriate. As shown in your ipfw show below, direction can help make things clear, and clarity means safety when it comes to firewalls, even if it means a slightly larger ruleset. > > > >> # Deny fragmented packets: > > > >> add deny ip from any to any frag > > > > >> # Show pings: > > > >> add count icmp from any to any icmptypes 8 in > > > > > > > >That's inbound ping requests. Don't forget that 'inbound' means coming > >into the firewall, not necessarily from the outside world. Your own > >ping requests _from_ this box also have to both come in, and go out. > > Hmmm. OK. Outbound Ping will be rarely used, but should > be allowed. Isn't that included in the next rule? Yes it is, so here ambiguous directionality works ok, as long as you're well aware of it. > > > >> # Allow pings, ping replies, and host unreach: > > > >> add allow icmp from any to any icmptypes 0,8,3 > > > >Add icmptype 11 as well if you want traceroutes to work .. Traceroutes want to see 'TTL exceeded in transit' icmp messages. > > > >> # Allow UDP traceroutes: > > > >> add allow udp from any to any 33434-34458 in > > > >> add allow udp from any 33434-34458 to any out > > > > > > > >Ok, though udp rules are often better done statefully. See below. > > > > > >> # Allow DNS with name server > > > >> add allow udp from any to any domain out > > > >> add allow udp from any domain to any in > > > >Nope. > > > >You want to watch out here. This allows udp packets from any address > >with source port 53 to connect with any open udp port on your system, > >and allows the responses as well. It's a simple matter using such as > >netcat to source packets from port 53. > > Should I restrict it by specifically stating the service? > How can I be safe? What would the rule look like? > > >I gather from this that you're not running a DNS server yourself, but > >using upstream server/s? In that case a stateful rule is safer: > > Again, I apologize for not being clear. I will be running > DNS on this box for the domains being hosted. So, it will > be polled whenever a request for a hosted domain is needed. Ok, so your nameserver will be making upstream requests too, and you'll need to do TCP 53 traffic with your secondary nameserver/s as well as UDP 53 traffic with upstream nameservers, up to the root unless you're only using specified upstream forwarders. Given that you're checking TCP setup, allowing established, then maybe: allow udp from me to any 53 out keep-state # my requests allow udp from any to me 53 in keep-state # serve outside requests allow tcp from me to $secondaries 53 setup # zone transfers out allow tcp from $secondaries to me 53 setup # zone transfers in though you'll want to protect named with ACLs for xfers as well. > > > >> # SSH > > > >> # Note that /etc/hosts.allow has restrictions > > > >> # on which IP addresses are allowed. > > > >> # > > > >> # Allow SSH: > > > >> add allow tcp from any to any ssh in setup > > > >By 'ssh working', I guess you mean ssh connections to this box from > >elsewhere, rather than ssh connections from this box? Not clear. > > Sorry! I am using SSH into this box, since it is easier to > cut and paste for editing and configuration. > > I can't really see a situation where I would normally need to SSH > outbound, can you? I use the Windoze boxes for that. You never know; you may want to use ssh or scp to other boxes, for backups and such, but you can always add rules whenever required. > > > >> # HTTP & HTTPS: > > > >> add allow tcp from any to any https in setup > > > >> add allow tcp from any to any http in setup > > > > > > > >So, you have a webserver running on this box, listening on ports 80 and > >443? You've verified with 'netstat -finet -a' that this is the case? > > Yes: > > # netstat -finet -a > Active Internet connections (including servers) > Proto Recv-Q Send-Q Local Address Foreign Address (state) > tcp4 0 0 192.168.1.109.ssh 192.168.1.107.3502 ESTABLISHED > tcp4 0 52 192.168.1.109.ssh 192.168.1.107.2266 ESTABLISHED > tcp4 0 0 *.poppassd *.* LISTEN > tcp4 0 0 *.ftp *.* LISTEN > tcp4 0 0 *.smtps *.* LISTEN > tcp4 0 0 *.smtp *.* LISTEN > tcp4 0 0 localhost.locald.3000 *.* LISTEN > tcp4 0 0 *.pop3s *.* LISTEN > tcp4 0 0 *.pop3 *.* LISTEN > tcp4 0 0 *.imaps *.* LISTEN > tcp4 0 0 *.imap *.* LISTEN > tcp4 0 0 *.8443 *.* LISTEN > tcp4 0 0 *.8880 *.* LISTEN > tcp4 0 0 *.3306 *.* LISTEN > tcp4 0 0 localhost.locald.8005 *.* LISTEN > tcp4 0 0 *.9008 *.* LISTEN > tcp4 0 0 *.8009 *.* LISTEN > tcp46 0 0 *.https *.* LISTEN > tcp46 0 0 *.http *.* LISTEN > tcp4 0 0 *.9080 *.* LISTEN > tcp4 0 0 *.8180 *.* LISTEN > tcp4 0 0 localhost.locald.postg *.* LISTEN > tcp4 0 0 localhost.locald.rndc *.* LISTEN > tcp4 0 0 localhost.locald.domai *.* LISTEN > tcp4 0 0 192.168.1.109.domain *.* LISTEN > tcp4 0 0 *.ssh *.* LISTEN > udp4 0 0 192.168.1.109.24889 ns1.ournameserver.net.53 > udp4 0 0 *.51750 *.* > udp4 0 0 localhost.locald.domai *.* > udp4 0 0 192.168.1.109.domain *.* > udp4 0 0 *.syslog *.* That's a bunch of services. Any being offered outside of your internal network want enabling and protecting by the firewall, those that are only internal to your LAN (or this box) likewise, by rules limiting access to only your LAN. Will this webserver later have a public IP address, or run behind NAT with port forwarding? > > > >> # Mail: SMTP & IMAP: > > > >> add allow tcp from any to any smtp in setup > > > >> add allow tcp from any to any imap in setup > > > > > > > >You're running SMTP and IMAP servers, verified as above? > > > >You see, this also allows you (as 'any') to connect to any outside SMTP > >server too. It really helps to differentiate connections into your box > >from those you're making to outside boxes, which these don't do. > > > >Have a close look at the 'simple' section in rc.firewall. > > I have scanned various versions of "rc.firewall". I kinda understand > what is going on, but there are so many places that seem anti-intuitive > to me. > > Also, what are the differences between running a script and loading > these rules on bootup? Well the script just loads rules too, by a series of ipfw add commands, some possibly generated programatically and/or conditionally. > >There are > >advantages to running a script such as that rather than rules in a file, > >like variable substitution, at least while getting it all working right. > > I have mixed feelings about variables. I guess they make > it easier if you change a network card or IP addresses--you > only have to do it in one place. > > However, a search and replace command doesn't take much > time at all. Also, they add a level of complexity to > a situation that (to me) is complex enough already. > And, I wonder if by not using variables, I can save a > few microseconds when processing traffic. ;^) No, it only saves a few milliseconds (using -q) while loading the rules; once the script is done then it's just the same. I find it easier to be able to specify which interfaces rules apply to, lists of IP addresses to deny or allow for particular services, port lists and ranges, targets for skipto rules, generating accounting rules and such. In your case of a single interface you can probably keep track of things ok. > > > >> # FTP: > > > >> add allow tcp from any to any ftp in setup > > > >> add allow tcp from any to any ftp\-data in setup > > > >> add allow tcp from any ftp\-data to any setup out > >Mmm, I prefer using and enforcing FTP passive mode, but YMMV. > > How would I do that? This guy doesn't think it's even > possible: > http://tinyurl.com/2z6ynr Mmm, ok. Passive mode needs allowing connections to this port range net.inet.ip.portrange.hifirst: 49152 net.inet.ip.portrange.hilast: 65535 which is adjustable, but I'm unsure of my ground regarding ftp - pass. > > > >> # Allow NTP in and out > > > >> add allow udp from any ntp to 128.252.19.1 ntp out > > > >> add allow udp from 128.252.19.1 ntp to any ntp in > > > > > > > >Unless running a time service for other boxes, something like: > > > > allow udp from me to any ntp keep-state # or to a specific server > > Well, I think that since NTP is such a minimal user > of resources, that I would like to rely on this > box for the correct time. That way, I don't have > to bug the stratum 1 boxes. Shall I use my original? Sure. I use a more local server pool. I'd still suggest getting in the habit of using 'me' rather than 'any' where it may be ambiguous .. some firewalls tend to be on routers or bridges where specificity matters. > > > >> # Deny and log everything else: > > > >> add deny log all from any to any > > > >Bingo! > > > >Ok, so you got rid of interface 'all', great. [..] > "Fine Manual" says: [..] > ip | all > Matches any packet. > > According to this, "ip" and "all" are synonymous. Criminy! > What am I missing here? My confusion; someone else used 'deny log ip from any to any recv all' where 'all' wasn't an interface. Ignore me too :) > > > I've been banging my head against this for the past few > > > days. I don't get it. > > > > > > My understanding of the way this is supposed to work is > > > that: > > > > > > # HTTP & HTTPS: > > > add allow tcp from any to any https in setup > > > add allow tcp from any to any http in setup > > > > > > should let initial HTTP & HTTPS requests through, > > > and that: > > > > > > # Allow established connections: > > > add allow tcp from any to any established > > > >Not quite. Looks like you're allowing http/https setup packets in (ie, > >into the firewall) but not letting them out (of the firewall, to the > >webserver). For example in the 'simple' ruleset mentioned, we have: Forget I said that .. I was thinking about routed packets. Which still leaves as a mystery why you're not (apparently) getting connections to your webserver when you are with ssh, like you said. Something else .. > > # Allow access to our WWW > > ${fwcmd} add pass tcp from any to ${oip} 80 setup > > > > # Reject&Log all setup of incoming connections from the outside > > ${fwcmd} add deny log tcp from any to any in via ${oif} setup > > > > # Allow setup of any other TCP connection > > ${fwcmd} add pass tcp from any to any setup > > I really don't get the above rule. Isn't it saying that > *any* kind of TCP connection can come in or go out initially? Yes, but only AFTER denying & logging any unaccounted for inbound setup requests on the outside interface. The last rule therefore allows setup of a) TCP requests from this box to anywhere, and b) setup requests from any boxes 'behind' this box on the LAN. You don't have any (b) but may need (a) to access external services (anything: c{,v}sup comes to mind); you may rather limit these to specific services or from uid root etc, and having no other interfaces, can use from 'me' rather than 'any'. > In my set, should I include some "out" rules like this: > > add allow tcp from any to any https out setup > add allow tcp from any to any http out setup Again, 'any to any' is too broad a brush. Once you allow everything you want to in from outside addresses (which currently includes your LAN on your outside interface!) and then deny the rest there, then 'pass tcp from me to any setup' is safe, and covers the above, and anything else. > > > should allow connections that are "setup" to > > > continue. Do I need a "check-state" or "keep-state" > > > statement somewhere? > > > >No, though you can use stateful TCP rules if you want to, in which case > >you'll want to DENY established connections. Personally I find relying > >on the TCP state established by using 'setup' and 'established' fine for > >TCP, but tend to use keep-state for UDP and some ICMP rules. > > That sounds reasonable. Others will disagree for sure :) > > > I don't understand what is wrong with the last rule: > > > > > > # Deny and log everything else: > > > add deny log all from any to any > > > > > > My understanding is that anything that doesn't match > > > the previous rules will match this one and hence > > > be logged and denied. Is this not correct? > > > >That's correct. Aren't you seeing any? Try show rather than tell. > > Showing: > > # ipfw -a -S -N -t list > 00100 688 173384 Thu Dec 20 15:32:17 2007 set 0 allow log logamount 10 ip from any to any via lo0 > 00200 0 0 set 0 deny log logamount 10 ip from any to 127.0.0.0/8 > 00300 0 0 set 0 deny log logamount 10 ip from 127.0.0.0/8 to any > 00400 4344 1712050 Fri Dec 21 00:23:37 2007 set 0 allow log logamount 10 tcp from any to any established > 00500 0 0 set 0 deny log logamount 10 ip from any to any frag > 00600 4 240 Wed Dec 19 23:05:31 2007 set 0 count icmp from any to any icmptypes 8 in > 00700 8 480 Wed Dec 19 23:05:31 2007 set 0 allow log logamount 10 icmp from any to any icmptypes 0,3,8 > 00800 0 0 set 0 allow log logamount 10 udp from any to any dst-port 33434-34458 in > 00900 0 0 set 0 allow log logamount 10 udp from any 33434-34458 to any out > 01000 366 24038 Fri Dec 21 00:02:00 2007 set 0 allow log logamount 10 udp from any to any dst-port domain out > 01100 364 59582 Fri Dec 21 00:02:00 2007 set 0 allow log logamount 10 udp from any domain to any in > 01200 1 48 Thu Dec 20 16:49:47 2007 set 0 allow log logamount 10 tcp from any to any dst-port ssh in setup > 01300 0 0 set 0 allow log logamount 10 tcp from any to any dst-port https in setup > 01400 6 288 Thu Dec 20 14:43:38 2007 set 0 allow log logamount 10 tcp from any to any dst-port http in setup Maybe 6 connections were made to your webserver, OR maybe you made 6 connections out to some other webserver. With 'any to any' there's no sense of direction, where 'any to me' would show inbound connections. Sorry if I keep stressing this, but it does help see what's going on. However searching for rule 1400 in your ipfw log will tell you. > 01500 98 6272 Fri Dec 21 00:02:00 2007 set 0 allow log logamount 10 tcp from any to any dst-port http It's not clear what these are either. They're not inbound setup packets, 1400 would catch those, and they couldn't be established packets since they're allowed above, in or out, so must be outbound setup packets from this box to another webserver, right? > 01600 1 64 Thu Dec 20 15:25:01 2007 set 0 allow log logamount 10 tcp from any to any dst-port https > 01700 0 0 set 0 allow log logamount 10 tcp from any to any dst-port smtp in setup > 01800 0 0 set 0 allow log logamount 10 tcp from any to any dst-port imap in setup > 01900 43 2064 Wed Dec 19 23:16:18 2007 set 0 allow log logamount 10 tcp from any to any dst-port ftp in setup So is ftp in from another box working ok? > 02000 0 0 set 0 allow log logamount 10 tcp from any to any dst-port ftp-data in setup > 02100 0 0 set 0 allow log logamount 10 tcp from any ftp-data to any setup out > 02200 100 7600 Thu Dec 20 23:47:00 2007 set 0 allow log logamount 10 udp from any ntp to navobs1.wustl.edu dst-port ntp out > 02300 100 7600 Thu Dec 20 23:47:00 2007 set 0 allow log logamount 10 udp from navobs1.wustl.edu ntp to any dst-port ntp in > 02400 2058 226123 Fri Dec 21 00:17:20 2007 set 0 deny log logamount 10 ip from any to any > 65535 7 909 Wed Dec 19 22:58:29 2007 set 31 deny ip from any to any > > Lot of stuff being denied. I think some of that > is my HTTP and HTTPS initial requests. What to do? Since you've logged some denied packets, searching denied packets in /var/log/security should show you exactly what they are to debug this, no need to speculate. However for this rule try maybe 'logamount 500' along with running tcpdump in another console till you spot the problem. Whenever you like, without reloading others, you can do eg: # ipfw delete 2400; ipfw add 2400 deny log logamount 500 ip from any to any and of course run 'ipfw resetlog 2400' anytime you want some more. Are these unsuccessful HTTP/S sessions only coming from inside your LAN presently? From the same box your successful ssh session is from? If not, this could be a routing or some other network problem than ipfw, or even an httpd.conf issue, despite appearing to be listening on *.80 > > > Again, I am having a great deal of difficulty > > > understanding why these rules don't work as expected. > > > I've scoured the 'Net and printed out just about > > > every coherent ruleset out there. > > > > > > Besides adding the "log" keyword on all of the rules, > > > these are the debugging tools I have been using: > > > > > > ipfw disable firewall > > > ipfw -f flush > > > ipfw enable firewall > > > /etc/rc.d/ipfw start > > > ipfw -a -S -N -t list > > > ipfw list > > > tail -f /var/log/ipfw/ipfw.log Ah right, you're not using /var/log/security. Is ipfw.log working ok? > > > tcpdump -i nve0 'proto \tcp && port http' Doesn't that show incoming http setup requests, and responses (or not)? I'd use 'tcpdump -pn -i nve0 tcp port 80' to keep it simple and numeric. > netstat -finet -a > > > > > > Could anyone please throw this tired dog a bone? [.. way too long already .. hoping somebody else spots something ..] > Love the "Fine Manual" --not! Just not enough examples for > me to understand everything. Too much abstraction--AAArrrrgh! That's why rc.firewall suggests some heavy reading material .. > Am using this link, since "man ipfw" doesn't work on 6.2. (I dare > someone to explain to me how to get it to work): > http://www.freebsd.org/cgi/man.cgi?query=ipfw&sektion=8 That's weird. Does man work for others in section 8, eg man mount ? > Thanks for your help, Ian. Would appreciate it if you would > kick my butt in the proper direction again. > > Any other takers/kickers? Yes please .. and please slash content ruthlessly on any replies (ie do as I plead, and not as I've just done :) cheers, Ian > Latest grope in the dark: > =============================================================== > # ipfw.rules > # ipfw firewall ruleset > # 2007 Dec 20 > > # By default, everything is denied access. You > # need to specifically allow something for it > # to work. > > # Loopback: > # Allow anything on the local loopback: > add allow log all from any to any via lo0 > > # Disallow spoofed access to local: > add deny log ip from any to 127.0.0.0/8 > add deny log ip from 127.0.0.0/8 to any > > # Allow established connections: > add allow log tcp from any to any established > > # Deny fragmented packets: > add deny log ip from any to any frag > > # Show pings: > add count icmp from any to any icmptypes 8 in > > # Allow pings, ping replies, and host unreach: > add allow log icmp from any to any icmptypes 0,8,3 > > # Allow UDP traceroutes: > add allow log udp from any to any 33434-34458 in > add allow log udp from any 33434-34458 to any out > > # Allow DNS with name server > add allow log udp from any to any domain out > add allow log udp from any domain to any in > > # SSH > # Note that /etc/hosts.allow has restrictions > # on which IP addresses are allowed. > # > # Allow SSH: > add allow log tcp from any to any ssh in setup > > # HTTP & HTTPS: > add allow log tcp from any to any https in setup > add allow log tcp from any to any http in setup > > add allow log tcp from any to any dst-port 80 > add allow log tcp from any to any dst-port 443 > > # Mail: SMTP & IMAP: > add allow log tcp from any to any smtp in setup > add allow log tcp from any to any imap in setup > > # FTP: > add allow log tcp from any to any ftp in setup > add allow log tcp from any to any ftp\-data in setup > add allow log tcp from any ftp\-data to any setup out > > # Allow NTP in and out > add allow log udp from any ntp to 128.252.19.1 ntp out > add allow log udp from 128.252.19.1 ntp to any ntp in > > > # Deny and log everything else: > # add deny log all from any to any > add deny log ip from any to any > ===============================================================
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?Pine.BSF.3.96.1071222150144.22888A-100000>