From owner-freebsd-net@FreeBSD.ORG  Mon Dec 20 11:41:33 2010
Return-Path: <owner-freebsd-net@FreeBSD.ORG>
Delivered-To: net@freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id 203A21065674
	for <net@freebsd.org>; Mon, 20 Dec 2010 11:41:33 +0000 (UTC)
	(envelope-from admin@shtorm.com)
Received: from ns.shtorm.com (ns.shtorm.com [195.62.14.3])
	by mx1.freebsd.org (Postfix) with ESMTP id D21A68FC0C
	for <net@freebsd.org>; Mon, 20 Dec 2010 11:41:32 +0000 (UTC)
Received: from [10.66.6.77] (unknown [10.66.6.77])
	by ns.shtorm.com (Postfix) with ESMTP id CB8201A0AD1;
	Mon, 20 Dec 2010 13:23:07 +0200 (EET)
From: Shtorm <admin@shtorm.com>
To: Eugene Grosbein <egrosbein@rdtc.ru>
In-Reply-To: <4D0CFEFF.3000902@rdtc.ru>
References: <4D0CFEFF.3000902@rdtc.ru>
Content-Type: text/plain; charset="UTF-8"
Date: Mon, 20 Dec 2010 13:21:35 +0200
Message-ID: <1292844095.1917.136.camel@stormi>
Mime-Version: 1.0
X-Mailer: Evolution 2.30.3 
Content-Transfer-Encoding: 7bit
Cc: net@freebsd.org
Subject: Re: lagg/lacp poor traffic distribution
X-BeenThere: freebsd-net@freebsd.org
X-Mailman-Version: 2.1.5
Precedence: list
List-Id: Networking and TCP/IP with FreeBSD <freebsd-net.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-net>,
	<mailto:freebsd-net-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-net>
List-Post: <mailto:freebsd-net@freebsd.org>
List-Help: <mailto:freebsd-net-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-net>,
	<mailto:freebsd-net-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Mon, 20 Dec 2010 11:41:33 -0000

On Sun, 2010-12-19 at 00:35 +0600, Eugene Grosbein wrote:
> Hi!
> 
> I've loaded router using two lagg interfaces in LACP mode.
> lagg0 has IP address and two ports (em0 and em1) and carry untagged frames.
> lagg1 has no IP address and has two ports (igb0 and igb1) and carry
> about 1000 dot-q vlans with lots of hosts in each vlan.
> 
> For lagg1, lagg distributes outgoing traffic over two ports just fine.
> For lagg0 (untagged ethernet segment with only 2 MAC addresses)
> less than 0.07% (54Mbit/s max) of traffic goes to em0
> and over 99.92% goes to em1, that's bad.
> 
> That's general traffic of several thousands of customers surfing the web,
> using torrents etc.  I've glanced over lagg/lacp sources if src/sys/net/
> and found nothing suspicious, it should extract and use srcIP/dstIP for hash.
> 
> How do I debug this problem?
> 
> Eugene Grosbein
> _______________________________________________
> freebsd-net@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-net
> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org"

I had this problem with igb driver, and I found, that lagg selects
outgoing interface based on packet header flowid field if M_FLOWID field
is set. And in the igb driver code flowid is set as 

#if __FreeBSD_version >= 800000
<------><------><------>rxr->fmp->m_pkthdr.flowid = que->msix;
<------><------><------>rxr->fmp->m_flags |= M_FLOWID;
#endif

The same thing in em driver with MULTIQUEUE 

That does not give enough number of flows to balance traffic well, so I
commented check in if_lagg.c

lagg_lb_start(struct lagg_softc *sc, struct mbuf *m)
{
<------>struct lagg_lb *lb = (struct lagg_lb *)sc->sc_psc;
<------>struct lagg_port *lp = NULL;
<------>uint32_t p = 0;

//<---->if (m->m_flags & M_FLOWID)
//<----><------>p = m->m_pkthdr.flowid;
//<---->else

and with this change I have much better load distribution across interfaces.

Hope it helps.