From owner-freebsd-net@FreeBSD.ORG  Tue Dec 21 14:11:30 2010
Return-Path: <owner-freebsd-net@FreeBSD.ORG>
Delivered-To: net@freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id 90A91106566C
	for <net@freebsd.org>; Tue, 21 Dec 2010 14:11:30 +0000 (UTC)
	(envelope-from egrosbein@rdtc.ru)
Received: from grosbein.pp.ru (grosbein.pp.ru [89.189.172.146])
	by mx1.freebsd.org (Postfix) with ESMTP id F3AE88FC1D
	for <net@freebsd.org>; Tue, 21 Dec 2010 14:11:29 +0000 (UTC)
Received: from grosbein.pp.ru (localhost [127.0.0.1])
	by grosbein.pp.ru (8.14.4/8.14.4) with ESMTP id oBLE0KOY002079;
	Tue, 21 Dec 2010 20:00:21 +0600 (NOVT)
	(envelope-from egrosbein@rdtc.ru)
Message-ID: <4D10B2F4.1060404@rdtc.ru>
Date: Tue, 21 Dec 2010 20:00:20 +0600
From: Eugene Grosbein <egrosbein@rdtc.ru>
User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US;
	rv:1.9.1.9) Gecko/20100518 Thunderbird/3.0.4
MIME-Version: 1.0
To: Fabien Thomas <fabien.thomas@netasq.com>
References: <4D0CFEFF.3000902@rdtc.ru> <1292844095.1917.136.camel@stormi>
	<4D1083D6.6010707@rdtc.ru>
	<84530C06-AC2E-4E2B-BFD4-693902BB0FA6@netasq.com>
In-Reply-To: <84530C06-AC2E-4E2B-BFD4-693902BB0FA6@netasq.com>
Content-Type: text/plain; charset=KOI8-R
Content-Transfer-Encoding: 7bit
Cc: net@freebsd.org
Subject: Re: lagg/lacp poor traffic distribution
X-BeenThere: freebsd-net@freebsd.org
X-Mailman-Version: 2.1.5
Precedence: list
List-Id: Networking and TCP/IP with FreeBSD <freebsd-net.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-net>,
	<mailto:freebsd-net-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-net>
List-Post: <mailto:freebsd-net@freebsd.org>
List-Help: <mailto:freebsd-net-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-net>,
	<mailto:freebsd-net-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Tue, 21 Dec 2010 14:11:30 -0000

On 21.12.2010 19:11, Fabien Thomas wrote:

>>> I had this problem with igb driver, and I found, that lagg selects
>>> outgoing interface based on packet header flowid field if M_FLOWID field
>>> is set. And in the igb driver code flowid is set as 
>>>
>>> #if __FreeBSD_version >= 800000
>>> <------><------><------>rxr->fmp->m_pkthdr.flowid = que->msix;
>>> <------><------><------>rxr->fmp->m_flags |= M_FLOWID;
>>> #endif
>>>
>>> The same thing in em driver with MULTIQUEUE 
>>>
>>> That does not give enough number of flows to balance traffic well, so I
>>> commented check in if_lagg.c
>>>
>>> lagg_lb_start(struct lagg_softc *sc, struct mbuf *m)
>>> {
>>> <------>struct lagg_lb *lb = (struct lagg_lb *)sc->sc_psc;
>>> <------>struct lagg_port *lp = NULL;
>>> <------>uint32_t p = 0;
>>>
>>> //<---->if (m->m_flags & M_FLOWID)
>>> //<----><------>p = m->m_pkthdr.flowid;
>>> //<---->else
>>>
>>> and with this change I have much better load distribution across interfaces.
>>>
>>> Hope it helps.
>>
>> You are perfectly right. By disabling flow usage I've obtained load sharing
>> close to even (final patch follows). Two questions:
>>
>> 1. Is it a bug or design problem?
> 
> How many queues have you with igb? If it's one it will explain why the flowid is bad for load balancing with lagg.

How do I know? I've read igb(4) manual page and found no words
about queues within igb, nor I have any knowledge about them.

> The problem is that flowid is good if the number of queue is = or a multiple of the number of lagg ports.

Now I see, thanks.

Eugene Grosbein