Date: Fri, 5 Dec 2008 01:36:11 +1100 (EST) From: Ian Smith <smithi@nimnet.asn.au> To: Brett Davidson <brett@net24.co.nz> Cc: freebsd-questions@freebsd.org Subject: Re: Is there anything weird I should know about using ipfw on alias addresses? Message-ID: <20081205003414.U60430@sola.nimnet.asn.au> In-Reply-To: <4937288A.5070106@net24.co.nz> References: <20081201120023.9E1821065688@hub.freebsd.org> <20081201233222.L34249@sola.nimnet.asn.au> <4934534D.1060100@net24.co.nz> <20081204012026.O60430@sola.nimnet.asn.au> <4937288A.5070106@net24.co.nz>
next in thread | previous in thread | raw e-mail | index | archive | help
On Thu, 4 Dec 2008, Brett Davidson wrote: > Ian Smith wrote: > > On Tue, 2 Dec 2008, Brett Davidson wrote: > > > Ian Smith wrote: > > > > On Mon, 01 Dec 2008 16:52:12 +1300 Brett Davidson <brett@net24.co.nz> > > > > wrote: > > > > > > > ifconfig shows the alias addresses correctly bound. > > > > > Creating an ipfw rule and testing it from the command line works > > > > > (connects out from master address, not alias) > > > > > > From website on alias address, the firewall blocks the packets. > > > > > > > > > > The weird thing is that it tags them (in the security log) as > > coming > > > > > from the master address (not the alias) out the correct interface. In > > a > > > > > normal world that would mean the packet would match!!!!! > > > > > > What's goin' on here Willis? > > > > > > Difficult to tell without seeing a) ifconfig b) netstat -rn c) at > > least the > > > > relevant firewall rule/s and d) log entries that illustrate your > > problem. > > > > Obscure sensitive information by all means, but otherwise pretend we > > > > haven't the slightest clue how your system is configured :) > > > > > > Fair enough. > > > > ifconfig below: > > > > bce1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500 > > > options=3b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU> > > > inet 210.5.50.5 netmask 0xffffffe0 broadcast 210.5.50.31 > > NB .. > > > inet 210.5.51.32 netmask 0xffffffff broadcast 210.5.51.32 > > > inet 210.5.51.27 netmask 0xffffffff broadcast 210.5.51.27 > > > inet 210.5.51.33 netmask 0xffffffff broadcast 210.5.51.33 > > > inet 210.5.51.34 netmask 0xffffffff broadcast 210.5.51.34 > > > inet 210.5.51.42 netmask 0xffffffff broadcast 210.5.51.42 > > > inet 210.5.51.4 netmask 0xffffffff broadcast 210.5.51.4 > > > ether 00:1c:c4:c0:56:94 > > > media: Ethernet autoselect (1000baseSX <full-duplex>) > > > status: active > > > > Relevant /etc/rc.conf entries : > > > ifconfig_bce1="inet 210.5.50.5 netmask 255.255.255.224" > > > ifconfig_bce1_alias0="inet 210.5.50.5 netmask 255.255.255.224" > > > > Your first alias here is a repeat of the 'primary' address. ifonfig seems > > to have resolved/merged that above, but it's not an alias. > > > > > True. Blame that on the piece of software (Plesk) that manages the IP > addresses for the websites we host. Ok in this instance. Please copy the list on replies, for archives. > > > ifconfig_bce1_alias1="inet 210.5.51.4 netmask 255.255.255.255" > > > ifconfig_bce1_alias2="inet 210.5.51.27 netmask 255.255.255.255" > > > ifconfig_bce1_alias3="inet 210.5.51.32 netmask 255.255.255.255" > > > ifconfig_bce1_alias4="inet 210.5.51.33 netmask 255.255.255.255" > > > ifconfig_bce1_alias5="inet 210.5.51.34 netmask 255.255.255.255" > > > ifconfig_bce1_alias6="inet 210.5.51.42 netmask 255.255.255.255" > > > > I didn't spot on first reading this that the first address is in a > > different subnet than all the others. I'm not entirely sure whether that's > > relevant, or how, just pointing it out as being non-obvious, and suspecting > > one of the 210.5.51 subnet should show a broader netmask. > > > I've wondered that as well but it all works EXCEPT for when ipfw is involved. Looks like we may need to see more, if not all, of your ipfw ruleset. 'ipfw -ted show' is pretty good for seeing everything. try adding 'log' to some more rules, until you can SEE where packets are getting blocked. Doesn't 'tcpdump -pn -i bce1 host 210.5.51.42 and host 208.69.123.164' provide any good clues to these flows? Or in this case maybe better: tcpdump -pn -i bce1 host \(210.5.51.42 or 210.5.50.5\) and host 208.69.123.164 > > > Relevant ipfw rules : > > > ipfw -q add 02012 allow tcp from any to 208.69.123.164 80 out via bce1 > > setup > > > keep-state > > > ipfw -q add 02012 allow tcp from any to 208.69.123.164 443 out via bce1 > > setup > > > keep-state Do you have a check-state rule? Where? Are there any skiptos that might miss anything? Do you have rules affecting established traffic? Sorry, but I find this too like a guessing game, or pulling teeth :) > > netstat -finet -rn (or -rna) please? unclear where your default route > > goes, or how the 210.5.51 subnet is routed or its netmask, but assume that > > 208.69.123.164 is probably accessed via the default route .. > > > > > Routing tables > > Internet: > Destination Gateway Flags Refs Use Netif Expire > default 210.5.50.1 UGS 0 296628406 bce1 > 127.0.0.1 127.0.0.1 UH 0 4339898 lo0 > 172.16.1/24 link#1 UC 0 0 bce0 > 172.16.1.1 00:04:28:ad:10:00 UHLW 1 0 bce0 1035 > 172.16.1.4 00:04:23:08:28:30 UHLW 1 167202525 bce0 1189 > 172.16.1.8 00:04:23:b2:f7:17 UHLW 1 0 bce0 1021 > 172.16.1.9 00:04:23:c7:79:0d UHLW 1 1 bce0 1190 > 172.16.1.12 00:07:e9:f4:cc:51 UHLW 1 0 bce0 1021 > 172.16.1.23 00:04:23:c7:78:cd UHLW 1 0 bce0 1199 > 172.16.1.30 00:04:23:d1:b9:c7 UHLW 1 0 bce0 1104 > 172.16.1.255 ff:ff:ff:ff:ff:ff UHLWb 1 156242 bce0 > 210.5.50/27 link#2 UC 0 0 bce1 > 210.5.50.1 00:04:28:ad:10:00 UHLW 2 0 bce1 1060 > 210.5.50.5 00:1c:c4:c0:56:94 UHLW 1 5010367 lo0 > 210.5.50.31 ff:ff:ff:ff:ff:ff UHLWb 1 350707 bce1 > 210.5.51.4/32 link#2 UC 0 0 bce1 > 210.5.51.27/32 link#2 UC 0 0 bce1 > 210.5.51.32/32 link#2 UC 0 0 bce1 > 210.5.51.33/32 link#2 UC 0 0 bce1 > 210.5.51.34/32 link#2 UC 0 0 bce1 > 210.5.51.42/32 link#2 UC 0 0 bce1 > > As you can see,. it's pretty straightforward and works except where ipfw is > involved with the website on 210.5.51.42. Ok, so if it works without ipfw running, then some ipfw rule is blocking traffic, it's safe enough to say. See above about showing timestamps to see what happened when (ie what just happened on the latest test). > > > Interesting entries in /var/log/security : > > > Dec 1 16:42:25 <servername> kernel: ipfw: 9999 Deny TCP > > 210.5.50.5:49708 > > > 208.69.123.164:80 out via bce1 > > > > Did that occur =after= the above rules were installed? Just the one? > > Seems odd on face value, but without knowing what your other rules do. > > > Yes - it's our last default catch-all rule that blocks and logs all traffic > not handled by any previous rules. Mmm. And these keep on occurring all the while? Or just on startup? > > > What makes this interesting is that I can connect to that port via the > > > command line. > > > > You mean like with 'telnet 208.69.123.164 80' ? With 210.5.50.5 as source > > address? tcpdump output may help understand or explain this. > > > Indeed! That's exactly what I mean which is why the security log output is a > little weird. tcpdump output may help .. and some timestamped logging. Dig deeper. > > > It's the website that lives on 210.5.51.42 that is having problems. Why, > > if > > > the rule is valid enough for the command line is it having problems from > > an > > > aliased address? > > > > Hang on; do you mean you're having a webserver on 210.5.51.42 trying to > > connect out to another webserver on 208.69.123.164 ? If not, what? > > > > > Bingo! Ah well. So what are you using / how are you getting this webserver's alias to establish outbound sockets using that address as the source address, rather than the 210.5.50.5 the system will normally assume? Some things - named's one example - allow specifying the source query address, so show how are you doing that from a webserver, where you can specify listening address/es inbound, but not (usually) outbound? If these packets were sourced from 210.5.50.5 and not 210.5.51.42 as you're expecting, then replies will never go back to 210.5.51.42, eh? > > I guess you have rules allowing inbound port 80 access to 210.5.51.42 ? > > > Yes, as well as 443 (the reason for having a seperate address in the first > place). > > > And that your upstream is routing 210.5.51.42/something to 210.5.50.5 ? > > > Correct. > > > > This MUST have something to do with the way ipfw is working with aliased > > > addresses but I'm blowed if I know what is wrong. > > > > ipfw doesn't do anything different with any address in particular except > > when using the forward action. ipfw certainly has no concept of primary or > > alias addresses, it just applies the addresses/masks you specify. > > > > Nor does ipfw know or care (even when forwarding) whence the stack is next > > going to route outbound packets .. but netstat -rn will tell us. Sorry, still way too little evidence to convict this masked culprit .. for this non-psychic bear of little brain anyway. Did I mention tcpdump? cheers, Ian
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20081205003414.U60430>