From owner-freebsd-net@FreeBSD.ORG Mon Nov 12 22:13:49 2012 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 1D7977F7 for ; Mon, 12 Nov 2012 22:13:49 +0000 (UTC) (envelope-from oppermann@networx.ch) Received: from c00l3r.networx.ch (c00l3r.networx.ch [62.48.2.2]) by mx1.freebsd.org (Postfix) with ESMTP id 73F558FC08 for ; Mon, 12 Nov 2012 22:13:48 +0000 (UTC) Received: (qmail 20976 invoked from network); 12 Nov 2012 23:48:06 -0000 Received: from c00l3r.networx.ch (HELO [127.0.0.1]) ([62.48.2.2]) (envelope-sender ) by c00l3r.networx.ch (qmail-ldap-1.03) with SMTP for ; 12 Nov 2012 23:48:06 -0000 Message-ID: <50A17497.10401@networx.ch> Date: Mon, 12 Nov 2012 23:13:43 +0100 From: Andre Oppermann User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:16.0) Gecko/20121010 Thunderbird/16.0.1 MIME-Version: 1.0 To: Karim Subject: Re: em/igb if_transmit (drbr) and ALTQ References: <50A14AEC.2040303@gmail.com> <50A15BB0.60102@networx.ch> <50A16E5A.6080401@gmail.com> In-Reply-To: <50A16E5A.6080401@gmail.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-net@freebsd.org X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 Nov 2012 22:13:49 -0000 On 12.11.2012 22:47, Karim wrote: > On 12-11-12 03:27 PM, Andre Oppermann wrote: >> On 12.11.2012 20:15, Karim wrote: >>> Hi all, >>> >>> I have been following the current discussions on igb tx/rx locking with great interest and I though >>> I would point out (as it was before pointed out in kern/138392) that any driver setting if_start to >>> NULL and using if_transmit (multi-queue or not) will effectively break ALTQ. >>> >>> The reason for this can be found in tbr_timeout() in altq_subr.c where the token bucket regulator >>> goes through all ALTQ registered interface and tries to 'kick' them into sending through if_start(). >>> >>> tbr_timeout() : >>> >>> ... >>> >>> for (ifp = TAILQ_FIRST(&V_ifnet); ifp; >>> ifp = TAILQ_NEXT(ifp, if_list)) { >>> /* read from if_snd unlocked */ >>> if (!TBR_IS_ENABLED(&ifp->if_snd)) >>> continue; >>> active++; >>> if (!IFQ_IS_EMPTY(&ifp->if_snd) && >>> ifp->if_start != NULL) /* if_start is NULL if if_transmit is >>> used in em/igb driver */ >>> (*ifp->if_start)(ifp); >>> } >>> >>> ... >>> >>> As you can see if_start is NULL on those new multi-queue enabled drivers which has for net effect to >>> 'break' ALTQ's token bucket regulator. >>> >>> I am writing this because I am interested in your comments on how this can be fixed properly looking >>> forward. The whole range of suggestions; from 'don't compile with EM_MULTIQUEUE defined' to 'here is >>> how you can make ALTQ use drbr' will help. >> >> This whole area needs some serious re-consideration for 10.0. >> Even without the if_start issue ALTQ is somewhat broken because >> the DMA rings nowadays are just too deep. >> >> See this recent message to this list for some thoughts: >> http://lists.freebsd.org/pipermail/freebsd-net/2012-November/033780.html >> >> On top of refining the stack/driver boundary I'm thinking of a >> dedicated ethernet layer to consolidate all the ethernet extensions, >> have a single place to go and to prevent the drivers from re-inventing >> the wheel all over every time. >> >> I'm working towards it and leaning into various drivers while trying >> to bring in the hybrid interrupt/polling mode with life-lock prevention. >> >> It'll take a few weeks until I'm ready to show a stack/ethernet/driver >> prototype for discussion. Parts may surface earlier in my tcp_workqueue >> svn branch. >> > Hi, > > Glad to see someone looking into this :) > > The addition of a common layer for queuing algorithm is an interesting idea and makes me wonder how > alternate queuing techniques would be able to use this shim layer to leverage devices with multiple > queues. > > The current limitation (in terms of performances) with ALTQ going forward is its current inability > to use multiple queues, in part because it was left out when the current (drbr) implementation of > multiqueues was made but mainly because of its current internal structure (using a single global IFQ > lock). > > Is it part of the plan to abstract multiqueuing from alternate queuing algorithm or will it still be > left to ALTQ, through driver managed queues for example, to manage the TX DMA ring lock? I don't know yet how exactly it will look like. ALTQ, in modified form, will remain part of the functionality set. Most multi-queue network cards also support various modes of queue arbitration for different service classes. This may be leveraged for ALTQ as well. There's many different variants of multi-queue usage depending on the goal of the overall system. I try to allow them to co-exist and to be selectable at run-time while remaining at a sane complexity level. -- Andre