Date: Fri, 14 May 2010 12:22:39 -0700 From: Pyun YongHyeon <pyunyh@gmail.com> To: list@cykotix.com Cc: freebsd-net@freebsd.org Subject: Re: Packet Loss on FW1 but not FW2 (CARP + PF on FBSD8) Message-ID: <20100514192239.GC24686@michelle.cdnetworks.com> In-Reply-To: <20100514145612.14566x4tj40yhyos@webmail.lahni.com> References: <20100514145612.14566x4tj40yhyos@webmail.lahni.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Fri, May 14, 2010 at 02:56:12PM -0400, list@cykotix.com wrote:
> Hello,
>
> I recently just purchased 2 Soekris5501 with identical 120gb 2.5" WD
> Scorpio HDDs. I'm using them for network failover, using CARP, PF and
> pfSync on FreeBSD 8-STABLE.
>
> The short version of my problem:
>
> I setup FW2 first, imaged its hard drive to FW1. I changed the
> necessary configs to update the IPs and ensure FW1 was carp MASTER.
> Using a known working port on the switch, I continue to get 70% packet
> loss on FW1 on vr0 (vr0 - extif, vr1 - intif, vr2 - pfsync). If I
> flip FW1 and FW2, the packet loss follows FW1. I took FW1 home,
> plugged it into my home network on vr0 and it works fine with 0%
> packet loss so the interface seems fine. I also took the IP bound to
> vr0 on FW1 and bound it to vr0 on FW2 and the ISP isn't the problem.
>
Show me the output of "sysctl dev.vr.0.stats=1" and "netstat -ndI vr0".
> The long version:
>
> Both Soekris5501's use vr0 (ext), vr1 (int) and vr2 (pfsync). I was
> given 98.xxx.xxx.58 - .62 with .57 being the gateway IP. FW1 was
> assigned .59. FW2 was assigned .60 and I was going to use .58 to NAT
> the office traffic over CARP. If I take carp0 and carp1 down off FW1,
> it moves all traffic to FW2 appropriately. If I bring carp0 and carp1
> back up on FW1, it assumes MASTER again as it should.
>
> FW1 /etc/rc.conf:
> -----------------
> cloned_interfaces="carp0 carp1"
> ifconfig_vr0="inet 98.xxx.xxx.59 netmask 255.255.255.248"
> ifconfig_vr1="inet 192.168.1.10 netmask 255.255.255.0"
> ifconfig_vr2="inet 10.0.10.12 netmask 255.255.255.0"
> ifconfig_carp0="inet 98.xxx.xxx.58 netmask 255.255.255.248 pass
> pabsoekris1959 vhid 1"
> ifconfig_carp0_alias0="inet 98.xxx.xxx.61 netmask 255.255.255.248"
> ifconfig_carp0_alias1="inet 98.xxx.xxx.62 netmask 255.255.255.248"
> ifconfig_carp1="inet 192.168.1.1 netmask 255.255.255.0 pass
> pabsoekris1959 vhid 2"
> ifconfig_pfsync0="syncpeer 10.0.10.13 syncdev vr2"
> defaultrouter="98.xxx.xxx.57"
> gateway_enable="YES"
>
> FW2 /etc/rc.conf:
> -----------------
> cloned_interfaces="carp0 carp1"
> ifconfig_vr0="inet 98.xxx.xxx.60 netmask 255.255.255.248"
> ifconfig_vr1="inet 192.168.1.11 netmask 255.255.255.0"
> ifconfig_vr2="inet 10.0.10.13 netmask 255.255.255.0"
> ifconfig_carp0="inet 98.xxx.xxx.58 netmask 255.255.255.248 pass
> pabsoekris1959 advskew 100 vhid 1"
> ifconfig_carp0_alias0="inet 98.xxx.xxx.61 netmask 255.255.255.248"
> ifconfig_carp0_alias1="inet 98.xxx.xxx.62 netmask 255.255.255.248"
> ifconfig_carp1="inet 192.168.1.1 netmask 255.255.255.0 pass
> pabsoekris1959 vhid 2"
> ifconfig_pfsync0="syncpeer 10.0.10.12 syncdev vr2"
> defaultrouter="98.xxx.xxx.57"
> gateway_enable="YES"
>
> FW1 /etc/pf.conf:
> ------------------------------------------------
> ext_if = vr0 # External WAN interface
> int_if = vr1 # Internal LAN interface
> pfs_if = vr2 # Pfsync interface
> carp_extif = carp0 # External CARP interface
> carp_intif = carp1
>
> ### hosts
> office = "192.168.1.0/24"
> office_ext = "98.xxx.xxx.58"
> soekris1 = "98.xxx.xxx.59"
> soekris2 = "98.xxx.xxx.60"
> pab = "192.168.1.2"
>
> ### icmp
> icmp_types = "{ echoreq, unreach }"
>
> ### tables
> table <bruteforce-ssh> persist
> table <badguys> persist file "/etc/badguys"
> table <goodguys> { $office }
>
> set block-policy drop
> set loginterface $ext_if
> set skip on lo
>
> scrub on $ext_if reassemble tcp no-df random-id
>
> ### NAT outgoing connections
> nat on $ext_if inet from $int_if:network to any -> $office_ext
>
>
> ### port forwards
> rdr on $ext_if proto tcp from any to $office_ext port XXXXX -> $pab port 22
> rdr on $ext_if proto tcp from any to $office_ext port XXXXX -> $pab port
> 3389
>
> ### ruleset
> block in log all # default deny
> block in log quick from urpf-failed # spoofed address protection
> block in log quick from { <bruteforce-ssh>, <badguys> }
>
> pass log from { lo0, $int_if:network, $ext_if, $carp_extif,
> $carp_intif } to any keep state
> pass in quick from <goodguys> keep state
> pass log inet proto icmp all icmp-type $icmp_types
> pass quick on $pfs_if proto pfsync keep state (no-sync) #
> enable pfsync
> pass on { $int_if, $ext_if } proto carp keep state (no-sync) # enable
> CARP
>
>
> FW2 /etc/pf.conf:
> -----------------
> ext_if = vr0 # External WAN interface
> int_if = vr1 # Internal LAN interface
> pfs_if = vr2 # Pfsync interface
> carp_extif = carp0 # External CARP interface
> carp_intif = carp1
>
> ### hosts
> office = "192.168.1.0/24"
> office_ext = "98.xxx.xxx.58"
> soekris1 = "98.xxx.xxx.59"
> soekris2 = "98.xxx.xxx.60"
> pab = "192.168.1.2"
>
> ### icmp
> icmp_types = "{ echoreq, unreach }"
>
>
> ### tables
> table <bruteforce-ssh> persist
> table <badguys> persist file "/etc/badguys"
> table <goodguys> { $office }
>
>
> set block-policy drop
> set loginterface $ext_if
> set skip on lo
>
> scrub on $ext_if reassemble tcp no-df random-id
>
> ### NAT outgoing connections
> nat on $ext_if inet from $int_if:network to any -> $office_ext
>
>
> ### port forwards
> rdr on $ext_if proto tcp from any to $office_ext port XXXXX -> $pab port 22
> rdr on $ext_if proto tcp from any to $office_ext port XXXXX -> $pab port
> 3389
>
> ### ruleset
> block in log all # default deny
> block in log quick from urpf-failed # spoofed address protection
> block in log quick from { <bruteforce-ssh>, <badguys> }
>
> pass log from { lo0, $int_if:network, $ext_if, $carp_extif,
> $carp_intif } to any keep state
> pass in quick from <goodguys> keep state
> pass log inet proto icmp all icmp-type $icmp_types
> pass quick on $pfs_if proto pfsync keep state (no-sync) #
> enable pfsync
> pass on { $int_if, $ext_if } proto carp keep state (no-sync) # enable
> CARP
>
>
> FW1 ifconfig (carp0 and carp1 are down, packet loss happens regardless):
> ------------------------------------------------------------------------
> soekris1# ifconfig
> vr0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric
> 0 mtu 1500
> options=280b<RXCSUM,TXCSUM,VLAN_MTU,WOL_UCAST,WOL_MAGIC>
> ether 00:00:24:cc:cb:94
> inet 98.xxx.xxx.59 netmask 0xfffffff8 broadcast 98.xxx.xxx.63
> media: Ethernet autoselect (100baseTX <full-duplex>)
> status: active
> vr1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric
> 0 mtu 1500
> options=280b<RXCSUM,TXCSUM,VLAN_MTU,WOL_UCAST,WOL_MAGIC>
> ether 00:00:24:cc:cb:95
> inet 192.168.1.10 netmask 0xffffff00 broadcast 192.168.1.255
> media: Ethernet autoselect (100baseTX <full-duplex>)
> status: active
> vr2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
> options=280b<RXCSUM,TXCSUM,VLAN_MTU,WOL_UCAST,WOL_MAGIC>
> ether 00:00:24:cc:cb:96
> inet 10.0.10.12 netmask 0xffffff00 broadcast 10.0.10.255
> media: Ethernet autoselect (100baseTX <full-duplex>)
> status: active
> pfsync0: flags=41<UP,RUNNING> metric 0 mtu 1460
> pfsync: syncdev: vr2 syncpeer: 10.0.10.13 maxupd: 128
> carp0: flags=8<LOOPBACK> metric 0 mtu 1500
> inet 98.xxx.xxx.61 netmask 0xfffffff8
> inet 98.xxx.xxx.62 netmask 0xfffffff8
> inet 98.xxx.xxx.58 netmask 0xfffffff8
> carp: INIT vhid 1 advbase 1 advskew 0
> carp1: flags=8<LOOPBACK> metric 0 mtu 1500
> inet 192.168.1.1 netmask 0xffffff00
> carp: INIT vhid 2 advbase 1 advskew 0
>
>
> FW2 ifconfig (carp0 and carp1 are up and in failover mode):
> -----------------------------------------------------------
> soekris2# ifconfig
> vr0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric
> 0 mtu 1500
> options=280b<RXCSUM,TXCSUM,VLAN_MTU,WOL_UCAST,WOL_MAGIC>
> ether 00:00:24:ca:40:60
> inet 98.xxx.xxx.60 netmask 0xfffffff8 broadcast 98.xxx.xxx.63
> media: Ethernet autoselect (100baseTX <full-duplex>)
> status: active
> vr1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric
> 0 mtu 1500
> options=280b<RXCSUM,TXCSUM,VLAN_MTU,WOL_UCAST,WOL_MAGIC>
> ether 00:00:24:ca:40:61
> inet 192.168.1.11 netmask 0xffffff00 broadcast 192.168.1.255
> media: Ethernet autoselect (100baseTX <full-duplex>)
> status: active
> vr2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
> options=280b<RXCSUM,TXCSUM,VLAN_MTU,WOL_UCAST,WOL_MAGIC>
> ether 00:00:24:ca:40:62
> inet 10.0.10.13 netmask 0xffffff00 broadcast 10.0.10.255
> media: Ethernet autoselect (100baseTX <full-duplex>)
> status: active
> pfsync0: flags=41<UP,RUNNING> metric 0 mtu 1460
> pfsync: syncdev: vr2 syncpeer: 10.0.10.12 maxupd: 128
> carp0: flags=49<UP,LOOPBACK,RUNNING> metric 0 mtu 1500
> inet 98.xxx.xxx.61 netmask 0xfffffff8
> inet 98.xxx.xxx.62 netmask 0xfffffff8
> inet 98.xxx.xxx.58 netmask 0xfffffff8
> carp: MASTER vhid 1 advbase 1 advskew 100
> carp1: flags=49<UP,LOOPBACK,RUNNING> metric 0 mtu 1500
> inet 192.168.1.1 netmask 0xffffff00
> carp: MASTER vhid 2 advbase 1 advskew 100
>
> Regardless if I flip IPs, flip ports on the switch, anything plugged
> into vr0 on FW1 at the office causes 70% packet loss, yet it's fine on
> FW2. FW1 vr0 works fine at my house using one of my localnet IPs.
>
> Any suggestions on how to track down where this packet loss is coming
> from? I appreciate any input!
>
> Thanks!
>
> Patrick
>
> ----------------------------------------------------------------
> This message was sent using IMP, the Internet Messaging Program.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20100514192239.GC24686>