From owner-freebsd-bugs@freebsd.org Fri Dec 21 10:20:13 2018 Return-Path: Delivered-To: freebsd-bugs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C5BAF1344326 for ; Fri, 21 Dec 2018 10:20:13 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 619098FC40 for ; Fri, 21 Dec 2018 10:20:13 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id 24E5D1344323; Fri, 21 Dec 2018 10:20:13 +0000 (UTC) Delivered-To: bugs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id F3E6E1344321 for ; Fri, 21 Dec 2018 10:20:12 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 8FAB68FC3E for ; Fri, 21 Dec 2018 10:20:12 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id CBB1512B79 for ; Fri, 21 Dec 2018 10:20:11 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id wBLAKBvr019091 for ; Fri, 21 Dec 2018 10:20:11 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id wBLAKBu6019090 for bugs@FreeBSD.org; Fri, 21 Dec 2018 10:20:11 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: bugs@FreeBSD.org Subject: [Bug 234242] LACP l2,l3,l4 load sharing only respects dst port Date: Fri, 21 Dec 2018 10:20:11 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: new X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.2-STABLE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Many People X-Bugzilla-Who: m.muenz@gmail.com X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: bugs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: bug_id short_desc product version rep_platform op_sys bug_status bug_severity priority component assigned_to reporter Message-ID: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-bugs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Bug reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 21 Dec 2018 10:20:14 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D234242 Bug ID: 234242 Summary: LACP l2,l3,l4 load sharing only respects dst port Product: Base System Version: 11.2-STABLE Hardware: amd64 OS: Any Status: New Severity: Affects Many People Priority: --- Component: kern Assignee: bugs@FreeBSD.org Reporter: m.muenz@gmail.com Hi, I'm chasing a performance problem when using FreeBSD as a router.=20 The system has 4 Mellanox ConnectX-3 cards. Bonded to 2 LAGGs with LACP. mlxen0 and 1 are lagg0 and mlxen2 and 3 are lagg1. Directly on these interface are linux boxes also with ConnectX-3 cards. This is the ifconfig from the router: mlxen0: flags=3D8843 metric 0 mtu 1= 500 =20=20=20=20=20=20=20 options=3D8d00b8 ether 24:8a:07:f7:5b:30 hwaddr 24:8a:07:f7:5b:30 nd6 options=3D21 media: Ethernet autoselect (10Gbase-CX4 ) status: active mlxen1: flags=3D8843 metric 0 mtu 1= 500 =20=20=20=20=20=20=20 options=3D8d00b8 ether 24:8a:07:f7:5b:30 hwaddr 24:8a:07:f7:5b:31 nd6 options=3D21 media: Ethernet autoselect (10Gbase-CX4 ) status: active mlxen2: flags=3D8843 metric 0 mtu 1= 500 =20=20=20=20=20=20=20 options=3D8d00b8 ether 24:8a:07:f7:5f:10 hwaddr 24:8a:07:f7:5f:10 nd6 options=3D21 media: Ethernet autoselect (10Gbase-CX4 ) status: active mlxen3: flags=3D8843 metric 0 mtu 1= 500 =20=20=20=20=20=20=20 options=3D8d00b8 ether 24:8a:07:f7:5f:10 hwaddr 24:8a:07:f7:5f:11 nd6 options=3D21 media: Ethernet autoselect (10Gbase-CX4 ) status: active lagg0: flags=3D8843 metric 0 mtu 15= 00 =20=20=20=20=20=20=20 options=3D8d00b8 ether 24:8a:07:f7:5b:30 inet6 fe80::268a:7ff:fef7:5b30%lagg0 prefixlen 64 scopeid 0xb inet 10.22.1.1 netmask 0xffffff00 broadcast 10.22.1.255 nd6 options=3D21 media: Ethernet autoselect status: active groups: lagg laggproto lacp lagghash l2,l3,l4 laggport: mlxen0 flags=3D1c laggport: mlxen1 flags=3D1c lagg1: flags=3D8843 metric 0 mtu 15= 00 =20=20=20=20=20=20=20 options=3D8d00b8 ether 24:8a:07:f7:5f:10 inet6 fe80::268a:7ff:fef7:5f10%lagg1 prefixlen 64 scopeid 0xc inet 10.22.2.1 netmask 0xffffff00 broadcast 10.22.2.255 nd6 options=3D21 media: Ethernet autoselect status: active groups: lagg laggproto lacp lagghash l2,l3,l4 laggport: mlxen2 flags=3D1c laggport: mlxen3 flags=3D1c When I put the 2 linux boxes directly together I can easily achive 20GBits. iperf server: iperf3 -V -p 5000 -f m -s iperf client: iperf3 -p 5000 -f m -V -c 10.22.2.10 -t 30 -P 10 When I put the BSD router between I can only achieve 10Gbit.=20 >From the man page of ifconfig there states: lagghash option[,option] Set the packet layers to hash for aggregation protocols which load balance. The default is "l2,l3,l4". The options can be combined using commas. l2 src/dst mac address and optional vlan number. l3 src/dst address for IPv4 or IPv6. l4 src/dst port for TCP/UDP/SCTP. The problem is that l4 is not really true, because iperf on multistream (-P= 10) uses multiple source ports, but the load comes from lagg0 with 5Gbit divide= d on mlxen0 and mlxen1 and goes via lagg1 through one of the interfaces with 10G= bit.=20 If I start a second iperf instance, listening on a different port and start both, the traffic flows with 20Gbit correctly shared.=20 Searching bugtracker, forums and asking IRC doesn't gave any good answer, already played with lacp strict mode and enable/disable flowid does not hel= p.=20 Right now I'm not sure if this is a bug in kernel or documentation, but it would be cool if we can include src and dst ports in hashing calculation. Thanks Michael --=20 You are receiving this mail because: You are the assignee for the bug.=