Date: Mon, 12 Nov 2012 16:47:06 -0500 From: Karim <fodillemlinkarim@gmail.com> To: Andre Oppermann <oppermann@networx.ch> Cc: freebsd-net@freebsd.org Subject: Re: em/igb if_transmit (drbr) and ALTQ Message-ID: <50A16E5A.6080401@gmail.com> In-Reply-To: <50A15BB0.60102@networx.ch> References: <50A14AEC.2040303@gmail.com> <50A15BB0.60102@networx.ch>
next in thread | previous in thread | raw e-mail | index | archive | help
On 12-11-12 03:27 PM, Andre Oppermann wrote:
> On 12.11.2012 20:15, Karim wrote:
>> Hi all,
>>
>> I have been following the current discussions on igb tx/rx locking
>> with great interest and I though
>> I would point out (as it was before pointed out in kern/138392) that
>> any driver setting if_start to
>> NULL and using if_transmit (multi-queue or not) will effectively
>> break ALTQ.
>>
>> The reason for this can be found in tbr_timeout() in altq_subr.c
>> where the token bucket regulator
>> goes through all ALTQ registered interface and tries to 'kick' them
>> into sending through if_start().
>>
>> tbr_timeout() :
>>
>> ...
>>
>> for (ifp = TAILQ_FIRST(&V_ifnet); ifp;
>> ifp = TAILQ_NEXT(ifp, if_list)) {
>> /* read from if_snd unlocked */
>> if (!TBR_IS_ENABLED(&ifp->if_snd))
>> continue;
>> active++;
>> if (!IFQ_IS_EMPTY(&ifp->if_snd) &&
>> ifp->if_start != NULL) /*
>> if_start is NULL if if_transmit is
>> used in em/igb driver */
>> (*ifp->if_start)(ifp);
>> }
>>
>> ...
>>
>> As you can see if_start is NULL on those new multi-queue enabled
>> drivers which has for net effect to
>> 'break' ALTQ's token bucket regulator.
>>
>> I am writing this because I am interested in your comments on how
>> this can be fixed properly looking
>> forward. The whole range of suggestions; from 'don't compile with
>> EM_MULTIQUEUE defined' to 'here is
>> how you can make ALTQ use drbr' will help.
>
> This whole area needs some serious re-consideration for 10.0.
> Even without the if_start issue ALTQ is somewhat broken because
> the DMA rings nowadays are just too deep.
>
> See this recent message to this list for some thoughts:
> http://lists.freebsd.org/pipermail/freebsd-net/2012-November/033780.html
>
> On top of refining the stack/driver boundary I'm thinking of a
> dedicated ethernet layer to consolidate all the ethernet extensions,
> have a single place to go and to prevent the drivers from re-inventing
> the wheel all over every time.
>
> I'm working towards it and leaning into various drivers while trying
> to bring in the hybrid interrupt/polling mode with life-lock prevention.
>
> It'll take a few weeks until I'm ready to show a stack/ethernet/driver
> prototype for discussion. Parts may surface earlier in my tcp_workqueue
> svn branch.
>
Hi,
Glad to see someone looking into this :)
The addition of a common layer for queuing algorithm is an interesting
idea and makes me wonder how alternate queuing techniques would be able
to use this shim layer to leverage devices with multiple queues.
The current limitation (in terms of performances) with ALTQ going
forward is its current inability to use multiple queues, in part because
it was left out when the current (drbr) implementation of multiqueues
was made but mainly because of its current internal structure (using a
single global IFQ lock).
Is it part of the plan to abstract multiqueuing from alternate queuing
algorithm or will it still be left to ALTQ, through driver managed
queues for example, to manage the TX DMA ring lock?
Thanks,
Karim.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?50A16E5A.6080401>