From owner-freebsd-net@FreeBSD.ORG Tue Dec 21 14:41:59 2010 Return-Path: Delivered-To: net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B08841065674 for ; Tue, 21 Dec 2010 14:41:59 +0000 (UTC) (envelope-from fabien.thomas@netasq.com) Received: from work.netasq.com (mars.netasq.com [91.212.116.3]) by mx1.freebsd.org (Postfix) with ESMTP id 1DA4B8FC0C for ; Tue, 21 Dec 2010 14:41:58 +0000 (UTC) Received: from [10.20.1.1] (unknown [10.2.1.1]) by work.netasq.com (Postfix) with ESMTPSA id 96C6B740008; Tue, 21 Dec 2010 15:39:39 +0100 (CET) Mime-Version: 1.0 (Apple Message framework v1082) Content-Type: text/plain; charset=us-ascii From: Fabien Thomas In-Reply-To: <4D10B2F4.1060404@rdtc.ru> Date: Tue, 21 Dec 2010 15:41:56 +0100 Content-Transfer-Encoding: quoted-printable Message-Id: <90F64F73-F9A8-4FD6-9303-2FC0D3424751@netasq.com> References: <4D0CFEFF.3000902@rdtc.ru> <1292844095.1917.136.camel@stormi> <4D1083D6.6010707@rdtc.ru> <84530C06-AC2E-4E2B-BFD4-693902BB0FA6@netasq.com> <4D10B2F4.1060404@rdtc.ru> To: Eugene Grosbein X-Mailer: Apple Mail (2.1082) Cc: net@freebsd.org Subject: Re: lagg/lacp poor traffic distribution X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 21 Dec 2010 14:41:59 -0000 On Dec 21, 2010, at 3:00 PM, Eugene Grosbein wrote: > On 21.12.2010 19:11, Fabien Thomas wrote: >=20 >>>> I had this problem with igb driver, and I found, that lagg selects >>>> outgoing interface based on packet header flowid field if M_FLOWID = field >>>> is set. And in the igb driver code flowid is set as=20 >>>>=20 >>>> #if __FreeBSD_version >=3D 800000 >>>> <------><------><------>rxr->fmp->m_pkthdr.flowid =3D que->msix; >>>> <------><------><------>rxr->fmp->m_flags |=3D M_FLOWID; >>>> #endif >>>>=20 >>>> The same thing in em driver with MULTIQUEUE=20 >>>>=20 >>>> That does not give enough number of flows to balance traffic well, = so I >>>> commented check in if_lagg.c >>>>=20 >>>> lagg_lb_start(struct lagg_softc *sc, struct mbuf *m) >>>> { >>>> <------>struct lagg_lb *lb =3D (struct lagg_lb *)sc->sc_psc; >>>> <------>struct lagg_port *lp =3D NULL; >>>> <------>uint32_t p =3D 0; >>>>=20 >>>> //<---->if (m->m_flags & M_FLOWID) >>>> //<----><------>p =3D m->m_pkthdr.flowid; >>>> //<---->else >>>>=20 >>>> and with this change I have much better load distribution across = interfaces. >>>>=20 >>>> Hope it helps. >>>=20 >>> You are perfectly right. By disabling flow usage I've obtained load = sharing >>> close to even (final patch follows). Two questions: >>>=20 >>> 1. Is it a bug or design problem? >>=20 >> How many queues have you with igb? If it's one it will explain why = the flowid is bad for load balancing with lagg. >=20 > How do I know? I've read igb(4) manual page and found no words vmstat -i will show the queue (intr for the queue) normally it's the = number of CPU available. > about queues within igb, nor I have any knowledge about them. >=20 >> The problem is that flowid is good if the number of queue is =3D or a = multiple of the number of lagg ports. >=20 > Now I see, thanks. >=20 > Eugene Grosbein