Skip site navigation (1)Skip section navigation (2)
Date:      Thu, 14 Sep 2017 16:21:08 +0200
From:      Dave Cottlehuber <dch@skunkwerks.at>
To:        freebsd-pf@freebsd.org, freebsd-net@freebsd.org
Subject:   NATted outbound traffic sometimes uses backup CARP IP on LACP/LAGG interface
Message-ID:  <1505398868.955393.1106053824.42CA3E40@webmail.messagingengine.com>

next in thread | raw e-mail | index | archive | help
Hi,

Outgoing traffic (from a jail) via PF NAT over a LAGG/LACP sometimes has
the *backup* CARP IP address assigned to it.

Obivously as this IP is only active on the "other" server, the return
TCP connection traffic never actually gets back to our CARP master, and
the other server sees spurious TCP connections. This is very
reproducible and appears to be deterministic, like a round robin IP
allocation. In practice, inside a jail, `curl $URL` will fail
repeatedly.

Hopefully this is some misconfiguration on my part - what am I doing
wrong?

BTW I wrote this up a while back on the forums where the config files
are easier to read: https://forums.freebsd.org/threads/61552

###############################
# /etc/rc.conf network
ifconfig_igb0="up"
ifconfig_igb1="up"
cloned_interfaces="${cloned_interfaces} lagg0"

defaultrouter="1.2.3.81"
ipv6_defaultrouter="1:2:3:4::1"

ifconfig_lagg0="inet 1.2.3.83/28 laggproto lacp laggport igb0 laggport
igb1"
ifconfig_lagg0_ipv6="inet6 1:2:3:4::83/64"

# carp on
kld_list="${kld_list} carp"
ifconfig_lagg0_aliases="\
  inet  vhid 1 advskew 100 pass pw1 1.2.3.84/32 \
  inet6 vhid 2 advskew 100 pass pw2 1:2:3:4::84/64 \
  inet  vhid 3 advskew   0 pass pw3 1.2.3.85/32 \
  inet6 vhid 4 advskew   0 pass pw4 1:2:3:4::85/64 \
"

# jail networks use their own separate cloned if
cloned_interfaces="${cloned_interfaces} lo1"
ifconfig_lo1_aliases="inet 10.241.0.0-15/16"


###############################
# /etc/pf.conf
# macros
protocols = "{ tcp, udp, icmp }"

# interfaces
extl_if="lagg0"
intl_if="lo0"
jail_if="lo1"

# networks
intl_net = $intl_if:network
jail_net = $jail_if:network
internet = $extl_if:network

# limits
set limit { states 200000, frags 80000, src-nodes 80000 }
set timeout { adaptive.start 180000, adaptive.end 200000 }

# clean packets are happy packets
scrub in all

# jails are allowed outbound connections but not inbound
nat on $extl_if proto $protocols from   $jail_net to any -> ($extl_if)

# o ye of little faith
pass in all
pass out all

###############################
######## running configs ######

pfctl indeed shows its a round-robin

###############################
#  pfctl -vnf /etc/pf.conf
protocols = "{ tcp, udp, icmp }"
extl_if = "lagg0"
intl_if = "lo0"
jail_if = "lo1"
intl_net = "lo0:network"
jail_net = "lo1:network"
internet = "lagg0:network"
set limit states 200000
set limit frags 80000
set limit src-nodes 80000
set timeout adaptive.start 180000
set timeout adaptive.end 200000
scrub in all fragment reassemble
nat on lagg0 inet proto tcp from 10.241.0.0/16 to any -> (lagg0)
round-robin
nat on lagg0 inet proto tcp from 10.241.0.1 to any -> (lagg0)
round-robin
... repeated for each IP


###############################
# ifconfig
lagg0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric
0 mtu 1500
 options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
   ether 78:45:c4:fa:d2:99
   inet 1.2.3.82 netmask 0xfffffff0 broadcast 1.2.3.95
*   inet 1.2.3.84 netmask 0xffffffff broadcast 1.2.3.84 vhid 1
*   inet 1.2.3.85 netmask 0xffffffff broadcast 1.2.3.85 vhid 3
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
   inet6 fe80::7a45:c4ff:fefa:d299%lagg0 prefixlen 64 scopeid 0x4
   inet6 1:2:3:4::82 prefixlen 64
*   inet6 1:2:3:4::84 prefixlen 64 vhid 2
*   inet6 1:2:3:4::85 prefixlen 64 vhid 4
   nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
   media: Ethernet autoselect
   status: active
*   carp: MASTER vhid 1 advbase 1 advskew 0
*   carp: BACKUP vhid 3 advbase 1 advskew 100
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
*   carp: MASTER vhid 2 advbase 1 advskew 0
*   carp: BACKUP vhid 4 advbase 1 advskew 100
   groups: lagg
   laggproto lacp lagghash l2,l3,l4
*   laggport: igb0 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
*   laggport: igb1 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>

# I removed the lines appended with !!!!!!!!!!!.. so that the system
actually works atm



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?1505398868.955393.1106053824.42CA3E40>