From owner-freebsd-net@FreeBSD.ORG Tue Dec 21 10:39:26 2010 Return-Path: Delivered-To: net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3D47B106566B for ; Tue, 21 Dec 2010 10:39:26 +0000 (UTC) (envelope-from egrosbein@rdtc.ru) Received: from eg.sd.rdtc.ru (eg.sd.rdtc.ru [62.231.161.221]) by mx1.freebsd.org (Postfix) with ESMTP id 81DA88FC0C for ; Tue, 21 Dec 2010 10:39:25 +0000 (UTC) Received: from eg.sd.rdtc.ru (localhost [127.0.0.1]) by eg.sd.rdtc.ru (8.14.4/8.14.4) with ESMTP id oBLAdN13031561; Tue, 21 Dec 2010 16:39:23 +0600 (NOVT) (envelope-from egrosbein@rdtc.ru) Message-ID: <4D1083D6.6010707@rdtc.ru> Date: Tue, 21 Dec 2010 16:39:18 +0600 From: Eugene Grosbein User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; ru-RU; rv:1.9.1.10) Gecko/20100712 Thunderbird/3.0.5 MIME-Version: 1.0 To: Shtorm References: <4D0CFEFF.3000902@rdtc.ru> <1292844095.1917.136.camel@stormi> In-Reply-To: <1292844095.1917.136.camel@stormi> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Cc: net@freebsd.org Subject: Re: lagg/lacp poor traffic distribution X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 21 Dec 2010 10:39:26 -0000 On 20.12.2010 17:21, Shtorm wrote: > On Sun, 2010-12-19 at 00:35 +0600, Eugene Grosbein wrote: >> Hi! >> >> I've loaded router using two lagg interfaces in LACP mode. >> lagg0 has IP address and two ports (em0 and em1) and carry untagged frames. >> lagg1 has no IP address and has two ports (igb0 and igb1) and carry >> about 1000 dot-q vlans with lots of hosts in each vlan. >> >> For lagg1, lagg distributes outgoing traffic over two ports just fine. >> For lagg0 (untagged ethernet segment with only 2 MAC addresses) >> less than 0.07% (54Mbit/s max) of traffic goes to em0 >> and over 99.92% goes to em1, that's bad. >> >> That's general traffic of several thousands of customers surfing the web, >> using torrents etc. I've glanced over lagg/lacp sources if src/sys/net/ >> and found nothing suspicious, it should extract and use srcIP/dstIP for hash. >> >> How do I debug this problem? >> >> Eugene Grosbein >> _______________________________________________ >> freebsd-net@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-net >> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > > I had this problem with igb driver, and I found, that lagg selects > outgoing interface based on packet header flowid field if M_FLOWID field > is set. And in the igb driver code flowid is set as > > #if __FreeBSD_version >= 800000 > <------><------><------>rxr->fmp->m_pkthdr.flowid = que->msix; > <------><------><------>rxr->fmp->m_flags |= M_FLOWID; > #endif > > The same thing in em driver with MULTIQUEUE > > That does not give enough number of flows to balance traffic well, so I > commented check in if_lagg.c > > lagg_lb_start(struct lagg_softc *sc, struct mbuf *m) > { > <------>struct lagg_lb *lb = (struct lagg_lb *)sc->sc_psc; > <------>struct lagg_port *lp = NULL; > <------>uint32_t p = 0; > > //<---->if (m->m_flags & M_FLOWID) > //<----><------>p = m->m_pkthdr.flowid; > //<---->else > > and with this change I have much better load distribution across interfaces. > > Hope it helps. You are perfectly right. By disabling flow usage I've obtained load sharing close to even (final patch follows). Two questions: 1. Is it a bug or design problem? 2. Will I get problems like packet reordering by permanently disabling usage of these flows in lagg(4)? --- if_lagg.c.orig 2010-12-20 22:53:21.000000000 +0600 +++ if_lagg.c 2010-12-21 13:37:20.000000000 +0600 @@ -168,6 +168,11 @@ &lagg_failover_rx_all, 0, "Accept input from any interface in a failover lagg"); +int lagg_use_flows = 1; +SYSCTL_INT(_net_link_lagg, OID_AUTO, use_flows, CTLFLAG_RW, + &lagg_use_flows, 1, + "Use flows for load sharing"); + static int lagg_modevent(module_t mod, int type, void *data) { @@ -1666,7 +1671,7 @@ struct lagg_port *lp = NULL; uint32_t p = 0; - if (m->m_flags & M_FLOWID) + if (lagg_use_flows && (m->m_flags & M_FLOWID)) p = m->m_pkthdr.flowid; else p = lagg_hashmbuf(m, lb->lb_key); --- if_lagg.h.orig 2010-12-21 16:34:35.000000000 +0600 +++ if_lagg.h 2010-12-21 16:35:27.000000000 +0600 @@ -242,6 +242,8 @@ int lagg_enqueue(struct ifnet *, struct mbuf *); uint32_t lagg_hashmbuf(struct mbuf *, uint32_t); +extern int lagg_use_flows; + #endif /* _KERNEL */ #endif /* _NET_LAGG_H */ --- ieee8023ad_lacp.c.orig 2010-12-21 16:36:09.000000000 +0600 +++ ieee8023ad_lacp.c 2010-12-21 16:35:58.000000000 +0600 @@ -812,7 +812,7 @@ return (NULL); } - if (m->m_flags & M_FLOWID) + if (lagg_use_flows && (m->m_flags & M_FLOWID)) hash = m->m_pkthdr.flowid; else hash = lagg_hashmbuf(m, lsc->lsc_hashkey); Eugene Grosbein