Date: Thu, 12 Feb 2026 11:58:56 +0000 From: bugzilla-noreply@freebsd.org To: bugs@FreeBSD.org Subject: [Bug 293136] net.route.hash_outbound not automatically enabled for locally originated traffic over multipath ECMP routes Message-ID: <bug-293136-227@https.bugs.freebsd.org/bugzilla/>
index | next in thread | raw e-mail
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=293136 Bug ID: 293136 Summary: net.route.hash_outbound not automatically enabled for locally originated traffic over multipath ECMP routes Product: Base System Version: 15.0-RELEASE Hardware: amd64 OS: Any Status: New Severity: Affects Some People Priority: --- Component: kern Assignee: bugs@FreeBSD.org Reporter: przemyslaw@mika.pro Created attachment 268002 --> https://bugs.freebsd.org/bugzilla/attachment.cgi?id=268002&action=edit iperf test after /usr/local/etc/rc.d/bird forcestart ##TL;DR net.route.hash_outbound defaults to 0 and the kernel never flips it to 1 when a routing daemon (such as Bird3) installs multipath ECMP routes into the FIB. This means locally originated traffic on a FreeBSD box always goes out through a single next hop instead of being distributed. Forwarded traffic is fine. The only workaround is a hack involving adding and removing a dummy multipath route at boot to trigger an internal code path that flips the flag. The kernel should automatically enable hash_outbound when multipath routes are present. ## Description We run a FreeBSD router that aggregates bandwidth over multiple WireGuard tunnels using ECMP (Equal Cost Multi Path). The setup has four primary fiber tunnels (wg0 through wg3), two LTE backup tunnels (wg5 and wg6), and one mobile VPN tunnel (wg4). Bird3 installs IPv6 multipath routes into the kernel FIB via its "merge paths" mechanism. For example, the primary /48 prefix gets four equal cost next hops (one per fiber tunnel), and the LTE backup group adds another two. All tunnel endpoints are monitored with BFD. The multipath routes are correctly present in the routing table, and forwarded traffic (packets transiting the box) is hashed and distributed across the available next hops as expected. However, locally originated traffic (connections initiated by the FreeBSD host itself, such as DNS queries, NTP, package fetches, BGP sessions sourced from the box, and so on) is never distributed. It always leaves through the first next hop in the multipath route. After some digging we found that the sysctl `net.route.hash_outbound` controls this behavior and it defaults to `0`. The problem is that this sysctl is not writable at runtime through the normal `sysctl(8)` interface. The only way we found to flip it to `1` is a hack: we add a dummy multipath route and immediately remove it at boot time, which triggers an internal code path that sets `hash_outbound` to `1` as a side effect. Once flipped, locally originated traffic is correctly hashed across all ECMP next hops and everything works perfectly. It would be great to see this fixed so that anyone running ECMP on FreeBSD does not have to resort to the dummy route workaround. ## How to Reproduce **Environment:** FreeBSD 15.0-RELEASE-p2, Bird3 (any recent version), multiple WireGuard tunnels configured as point to point links (our reproducer uses seven: wg0 through wg3 for primary fiber ECMP, wg5 and wg6 for LTE backup ECMP, and wg4 for mobile VPN). Set up the WireGuard tunnels on the FreeBSD box, each with its own /126 IPv6 transport subnet. Install Bird3 and use a configuration similar to the one appended below. The key parts are the kernel protocol with `merge paths on` and several static protocols that define the same /48 prefix with multiple next hops at different preferences (200 for fiber, 100 for LTE, 10 for a blackhole fallback). After Bird3 starts, verify that the multipath route is in the FIB: # netstat -rn6 | grep 2001:db8:1:: You should see the prefix with multiple next hops listed (four when all fiber tunnels are up). To demonstrate the problem clearly, run iperf3 in reverse mode. Start an iperf3 server on a host reachable through the tunnels (for example on the remote tunnel endpoint) and then initiate multiple parallel streams from the FreeBSD box: # iperf3 -6 -c 2001:db8:1:ffff::2 -R -P8 -t 30 The -R flag makes the remote side send traffic back to the FreeBSD host, so the locally originated control connection (and the resulting reverse data flows) should in theory be distributed across the ECMP next hops. While the test is running, watch each tunnel interface with tcpdump or netstat -I: # tcpdump -n -i wg0 & # tcpdump -n -i wg1 & # tcpdump -n -i wg2 & # tcpdump -n -i wg3 & You will find that all iperf3 traffic leaves through only one of the tunnels. The other three tunnels carry no iperf3 traffic at all, even though the multipath route has four equal cost next hops. Now check the sysctl: # sysctl net.route.hash_outbound net.route.hash_outbound: 0 Attempting to set it directly fails: # sysctl net.route.hash_outbound=1 The workaround is to add and immediately delete a dummy multipath route, which triggers the kernel to flip hash_outbound to 1 internally: # route add -6 2001:db8:dead::/48 2001:db8:1:ffff::2 2001:db8:1:ffff::6 # route delete -6 2001:db8:dead::/48 # sysctl net.route.hash_outbound net.route.hash_outbound: 1 After this, locally originated traffic is correctly distributed across all ECMP next hops. ### Expected Behavior The kernel should automatically enable net.route.hash_outbound whenever a route utilizing a nexthop group (ECMP) is successfully installed in the FIB, regardless of the method used to install it. Currently, it appears that the logic to flip this flag might be tied to specific route modification paths (e.g., incrementally adding next hops to an existing route, which route add likely triggers). However, when a routing daemon like Bird3 installs a complete multipath route in a single operation (or replaces an existing one), this check seems to be bypassed. The kernel should generically verify if the new route is a multipath group after any route addition and enable the hashing of locally originated traffic if so. ### Actual Behavior `net.route.hash_outbound` defaults to `0` and is not writable at runtime. Locally originated traffic always uses a single next hop. The only way to enable proper ECMP hashing for local traffic is the dummy route side effect described above. ### Reproducer Bird3 Config (anonymized) See the attached `bird.conf` bellow. All ASNs, prefixes and addresses have been replaced with documentation ranges (RFC 3849 / RFC 5737). The original setup uses real routable prefixes, seven WireGuard tunnels (four fiber primary, two LTE backup, one mobile VPN) and two BGP sessions to an upstream provider. The structure and behavior are identical. ``` log "/var/log/bird.log" { debug, trace, info, remote, warning, error, auth, fatal, bug }; router id 198.51.100.1; filter as54321_v6 prefix set allnet6; { allnet6 = [ 2001:db8:1::/48 ]; if ! (net ~ allnet6) then reject; accept; } filter export_to_kernel_v6 prefix set peering6; { peering6 = [ 2001:db8:ff::18/126, 2001:db8:ff::1c/126 ]; if (net ~ peering6) then reject; accept; } filter export_to_kernel_v4 { accept; } protocol device { scan time 5; interface "vtnet0" { preferred 2001:db8:1:abc::1; }; }; protocol direct { interface "vtnet0", "wg0", "wg1", "wg2", "wg3", "wg4", "wg5", "wg6"; ipv4 { table master4; }; ipv6 { table master6; }; } protocol kernel kipv6 { learn; merge paths on; merge paths limit 4; ipv6 { table master6; import all; export filter export_to_kernel_v6; }; } protocol kernel kipv4 { ipv4 { export none; import none; }; learn; persist; } protocol bfd { interface "wg0", "wg1", "wg2", "wg3" { min rx interval 300 ms; min tx interval 300 ms; multiplier 3; }; interface "wg5", "wg6" { min rx interval 1000 ms; min tx interval 1000 ms; multiplier 3; }; } protocol static ecmp_v6_fiber { ipv6 { table master6; preference 200; }; route 2001:db8:1::/48 via 2001:db8:1:ffff::2 bfd via 2001:db8:1:ffff::6 bfd via 2001:db8:1:ffff::a bfd via 2001:db8:1:ffff::e bfd; } protocol static ecmp_v6_lte { ipv6 { table master6; preference 100; }; route 2001:db8:1::/48 via 2001:db8:1:ffff::12 bfd via 2001:db8:1:ffff::16 bfd; } protocol static mobile_v6 { ipv6 { table master6; preference 250; }; route 2001:db8:1:abc::/64 via "wg4"; } protocol static always_announce_v6 { ipv6 { table master6; preference 10; }; route 2001:db8:1::/48 blackhole; } template bgp Upstream1_v6 { local 2001:db8:ff::1a as 12345; source address 2001:db8:ff::1a; ipv6 { table master6; import all; export filter as54321_v6; }; } protocol bgp Upstream1_IPv6 from Upstream1_v6 { description "Upstream IPv6 session 1"; neighbor 2001:db8:ff::19 as 54321; } template bgp Upstream2_v6 { local 2001:db8:ff::1e as 12345; source address 2001:db8:ff::1e; ipv6 { table master6; import all; export filter as54321_v6; }; } protocol bgp Upstream2_IPv6 from Upstream2_v6 { description "Upstream IPv6 session 2"; neighbor 2001:db8:ff::1d as 54321; } ``` This report was created using AI but was ultimately approved by a real human. -- You are receiving this mail because: You are the assignee for the bug.home | help
Want to link to this message? Use this
URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?bug-293136-227>