Date: Fri, 21 Dec 2018 10:20:11 +0000 From: bugzilla-noreply@freebsd.org To: bugs@FreeBSD.org Subject: [Bug 234242] LACP l2,l3,l4 load sharing only respects dst port Message-ID: <bug-234242-227@https.bugs.freebsd.org/bugzilla/>
next in thread | raw e-mail | index | archive | help
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D234242 Bug ID: 234242 Summary: LACP l2,l3,l4 load sharing only respects dst port Product: Base System Version: 11.2-STABLE Hardware: amd64 OS: Any Status: New Severity: Affects Many People Priority: --- Component: kern Assignee: bugs@FreeBSD.org Reporter: m.muenz@gmail.com Hi, I'm chasing a performance problem when using FreeBSD as a router.=20 The system has 4 Mellanox ConnectX-3 cards. Bonded to 2 LAGGs with LACP. mlxen0 and 1 are lagg0 and mlxen2 and 3 are lagg1. Directly on these interface are linux boxes also with ConnectX-3 cards. This is the ifconfig from the router: mlxen0: flags=3D8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1= 500 =20=20=20=20=20=20=20 options=3D8d00b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTE= R,VLAN_HWTSO,LINKSTATE> ether 24:8a:07:f7:5b:30 hwaddr 24:8a:07:f7:5b:30 nd6 options=3D21<PERFORMNUD,AUTO_LINKLOCAL> media: Ethernet autoselect (10Gbase-CX4 <full-duplex,rxpause,txpaus= e>) status: active mlxen1: flags=3D8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1= 500 =20=20=20=20=20=20=20 options=3D8d00b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTE= R,VLAN_HWTSO,LINKSTATE> ether 24:8a:07:f7:5b:30 hwaddr 24:8a:07:f7:5b:31 nd6 options=3D21<PERFORMNUD,AUTO_LINKLOCAL> media: Ethernet autoselect (10Gbase-CX4 <full-duplex,rxpause,txpaus= e>) status: active mlxen2: flags=3D8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1= 500 =20=20=20=20=20=20=20 options=3D8d00b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTE= R,VLAN_HWTSO,LINKSTATE> ether 24:8a:07:f7:5f:10 hwaddr 24:8a:07:f7:5f:10 nd6 options=3D21<PERFORMNUD,AUTO_LINKLOCAL> media: Ethernet autoselect (10Gbase-CX4 <full-duplex,rxpause,txpaus= e>) status: active mlxen3: flags=3D8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1= 500 =20=20=20=20=20=20=20 options=3D8d00b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTE= R,VLAN_HWTSO,LINKSTATE> ether 24:8a:07:f7:5f:10 hwaddr 24:8a:07:f7:5f:11 nd6 options=3D21<PERFORMNUD,AUTO_LINKLOCAL> media: Ethernet autoselect (10Gbase-CX4 <full-duplex,rxpause,txpaus= e>) status: active lagg0: flags=3D8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 15= 00 =20=20=20=20=20=20=20 options=3D8d00b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTE= R,VLAN_HWTSO,LINKSTATE> ether 24:8a:07:f7:5b:30 inet6 fe80::268a:7ff:fef7:5b30%lagg0 prefixlen 64 scopeid 0xb inet 10.22.1.1 netmask 0xffffff00 broadcast 10.22.1.255 nd6 options=3D21<PERFORMNUD,AUTO_LINKLOCAL> media: Ethernet autoselect status: active groups: lagg laggproto lacp lagghash l2,l3,l4 laggport: mlxen0 flags=3D1c<ACTIVE,COLLECTING,DISTRIBUTING> laggport: mlxen1 flags=3D1c<ACTIVE,COLLECTING,DISTRIBUTING> lagg1: flags=3D8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 15= 00 =20=20=20=20=20=20=20 options=3D8d00b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTE= R,VLAN_HWTSO,LINKSTATE> ether 24:8a:07:f7:5f:10 inet6 fe80::268a:7ff:fef7:5f10%lagg1 prefixlen 64 scopeid 0xc inet 10.22.2.1 netmask 0xffffff00 broadcast 10.22.2.255 nd6 options=3D21<PERFORMNUD,AUTO_LINKLOCAL> media: Ethernet autoselect status: active groups: lagg laggproto lacp lagghash l2,l3,l4 laggport: mlxen2 flags=3D1c<ACTIVE,COLLECTING,DISTRIBUTING> laggport: mlxen3 flags=3D1c<ACTIVE,COLLECTING,DISTRIBUTING> When I put the 2 linux boxes directly together I can easily achive 20GBits. iperf server: iperf3 -V -p 5000 -f m -s iperf client: iperf3 -p 5000 -f m -V -c 10.22.2.10 -t 30 -P 10 When I put the BSD router between I can only achieve 10Gbit.=20 >From the man page of ifconfig there states: lagghash option[,option] Set the packet layers to hash for aggregation protocols which load balance. The default is "l2,l3,l4". The options can be combined using commas. l2 src/dst mac address and optional vlan number. l3 src/dst address for IPv4 or IPv6. l4 src/dst port for TCP/UDP/SCTP. The problem is that l4 is not really true, because iperf on multistream (-P= 10) uses multiple source ports, but the load comes from lagg0 with 5Gbit divide= d on mlxen0 and mlxen1 and goes via lagg1 through one of the interfaces with 10G= bit.=20 If I start a second iperf instance, listening on a different port and start both, the traffic flows with 20Gbit correctly shared.=20 Searching bugtracker, forums and asking IRC doesn't gave any good answer, already played with lacp strict mode and enable/disable flowid does not hel= p.=20 Right now I'm not sure if this is a bug in kernel or documentation, but it would be cool if we can include src and dst ports in hashing calculation. Thanks Michael --=20 You are receiving this mail because: You are the assignee for the bug.=
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?bug-234242-227>