From owner-freebsd-net@freebsd.org  Thu Sep 14 21:32:24 2017
Return-Path: <owner-freebsd-net@freebsd.org>
Delivered-To: freebsd-net@mailman.ysv.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org
 [IPv6:2001:1900:2254:206a::19:1])
 by mailman.ysv.freebsd.org (Postfix) with ESMTP id BD84EE0D71F;
 Thu, 14 Sep 2017 21:32:24 +0000 (UTC)
 (envelope-from srs0=woc2=ap=sigsegv.be=kristof@codepro.be)
Received: from venus.codepro.be (venus.codepro.be [IPv6:2a01:4f8:162:1127::2])
 (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256
 bits))
 (Client CN "*.codepro.be", Issuer "Gandi Standard SSL CA 2" (verified OK))
 by mx1.freebsd.org (Postfix) with ESMTPS id 86C006AA01;
 Thu, 14 Sep 2017 21:32:24 +0000 (UTC)
 (envelope-from srs0=woc2=ap=sigsegv.be=kristof@codepro.be)
Received: from [192.168.228.1]
 (ptr-8ripyyf09rrerk0utqq.18120a2.ip6.access.telenet.be
 [IPv6:2a02:1811:2419:4e02:2812:4e88:3eaf:f212])
 (Authenticated sender: kp)
 by venus.codepro.be (Postfix) with ESMTPSA id E71D146074;
 Thu, 14 Sep 2017 23:32:20 +0200 (CEST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sigsegv.be; s=mail;
 t=1505424741; bh=a/9B4TIF1FknxE80ydLk9S19/O1NRT2F9pBzOBTjfzQ=;
 h=From:To:Cc:Subject:Date:In-Reply-To:References;
 b=2rqxWSegRGhej1M6wSurDXY/xehdQ8UEf6RgKrdxPMj+2W+tJRWgxdGqytNdHoxM9
 7A9ANPjQBWZDAnaMMiGdCmNmFz+z4/+FicAu7SUHBfgZurOdbSLgB/tg/wOI3TvpkS
 Vs4xlU2jsbTI+WZMaWPB7aycM+d6c2sNkiir9wZQ=
From: "Kristof Provost" <kristof@sigsegv.be>
To: "Dave Cottlehuber" <dch@skunkwerks.at>
Cc: freebsd-pf@freebsd.org, freebsd-net@freebsd.org
Subject: Re: NATted outbound traffic sometimes uses backup CARP IP on
 LACP/LAGG interface
Date: Thu, 14 Sep 2017 23:32:21 +0200
Message-ID: <4250D983-3CE2-4156-85FB-E580F6C592D2@sigsegv.be>
In-Reply-To: <1505398868.955393.1106053824.42CA3E40@webmail.messagingengine.com>
References: <1505398868.955393.1106053824.42CA3E40@webmail.messagingengine.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 8bit
X-Mailer: MailMate (2.0BETAr6090)
X-BeenThere: freebsd-net@freebsd.org
X-Mailman-Version: 2.1.23
Precedence: list
List-Id: Networking and TCP/IP with FreeBSD <freebsd-net.freebsd.org>
List-Unsubscribe: <https://lists.freebsd.org/mailman/options/freebsd-net>,
 <mailto:freebsd-net-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-net/>
List-Post: <mailto:freebsd-net@freebsd.org>
List-Help: <mailto:freebsd-net-request@freebsd.org?subject=help>
List-Subscribe: <https://lists.freebsd.org/mailman/listinfo/freebsd-net>,
 <mailto:freebsd-net-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Thu, 14 Sep 2017 21:32:24 -0000

On 14 Sep 2017, at 16:21, Dave Cottlehuber wrote:
> Outgoing traffic (from a jail) via PF NAT over a LAGG/LACP sometimes 
> has
> the *backup* CARP IP address assigned to it.
>

>
> ###############################
> ######## running configs ######
>
> pfctl indeed shows its a round-robin
>
> ###############################
> #  pfctl -vnf /etc/pf.conf
> protocols = "{ tcp, udp, icmp }"
> extl_if = "lagg0"
> intl_if = "lo0"
> jail_if = "lo1"
> intl_net = "lo0:network"
> jail_net = "lo1:network"
> internet = "lagg0:network"
> set limit states 200000
> set limit frags 80000
> set limit src-nodes 80000
> set timeout adaptive.start 180000
> set timeout adaptive.end 200000
> scrub in all fragment reassemble
> nat on lagg0 inet proto tcp from 10.241.0.0/16 to any -> (lagg0)
> round-robin
> nat on lagg0 inet proto tcp from 10.241.0.1 to any -> (lagg0)
> round-robin
>
I think this is your problem. You’re telling pf to nat to the IP 
address of lagg0, but lagg0 has multiple addresses assigned.

‘(lagg0:0)’ should work, or just use the IP address.

Regards,
Kristof