Date: Mon, 12 Nov 2012 23:13:43 +0100 From: Andre Oppermann <oppermann@networx.ch> To: Karim <fodillemlinkarim@gmail.com> Cc: freebsd-net@freebsd.org Subject: Re: em/igb if_transmit (drbr) and ALTQ Message-ID: <50A17497.10401@networx.ch> In-Reply-To: <50A16E5A.6080401@gmail.com> References: <50A14AEC.2040303@gmail.com> <50A15BB0.60102@networx.ch> <50A16E5A.6080401@gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On 12.11.2012 22:47, Karim wrote:
> On 12-11-12 03:27 PM, Andre Oppermann wrote:
>> On 12.11.2012 20:15, Karim wrote:
>>> Hi all,
>>>
>>> I have been following the current discussions on igb tx/rx locking with great interest and I though
>>> I would point out (as it was before pointed out in kern/138392) that any driver setting if_start to
>>> NULL and using if_transmit (multi-queue or not) will effectively break ALTQ.
>>>
>>> The reason for this can be found in tbr_timeout() in altq_subr.c where the token bucket regulator
>>> goes through all ALTQ registered interface and tries to 'kick' them into sending through if_start().
>>>
>>> tbr_timeout() :
>>>
>>> ...
>>>
>>> for (ifp = TAILQ_FIRST(&V_ifnet); ifp;
>>> ifp = TAILQ_NEXT(ifp, if_list)) {
>>> /* read from if_snd unlocked */
>>> if (!TBR_IS_ENABLED(&ifp->if_snd))
>>> continue;
>>> active++;
>>> if (!IFQ_IS_EMPTY(&ifp->if_snd) &&
>>> ifp->if_start != NULL) /* if_start is NULL if if_transmit is
>>> used in em/igb driver */
>>> (*ifp->if_start)(ifp);
>>> }
>>>
>>> ...
>>>
>>> As you can see if_start is NULL on those new multi-queue enabled drivers which has for net effect to
>>> 'break' ALTQ's token bucket regulator.
>>>
>>> I am writing this because I am interested in your comments on how this can be fixed properly looking
>>> forward. The whole range of suggestions; from 'don't compile with EM_MULTIQUEUE defined' to 'here is
>>> how you can make ALTQ use drbr' will help.
>>
>> This whole area needs some serious re-consideration for 10.0.
>> Even without the if_start issue ALTQ is somewhat broken because
>> the DMA rings nowadays are just too deep.
>>
>> See this recent message to this list for some thoughts:
>> http://lists.freebsd.org/pipermail/freebsd-net/2012-November/033780.html
>>
>> On top of refining the stack/driver boundary I'm thinking of a
>> dedicated ethernet layer to consolidate all the ethernet extensions,
>> have a single place to go and to prevent the drivers from re-inventing
>> the wheel all over every time.
>>
>> I'm working towards it and leaning into various drivers while trying
>> to bring in the hybrid interrupt/polling mode with life-lock prevention.
>>
>> It'll take a few weeks until I'm ready to show a stack/ethernet/driver
>> prototype for discussion. Parts may surface earlier in my tcp_workqueue
>> svn branch.
>>
> Hi,
>
> Glad to see someone looking into this :)
>
> The addition of a common layer for queuing algorithm is an interesting idea and makes me wonder how
> alternate queuing techniques would be able to use this shim layer to leverage devices with multiple
> queues.
>
> The current limitation (in terms of performances) with ALTQ going forward is its current inability
> to use multiple queues, in part because it was left out when the current (drbr) implementation of
> multiqueues was made but mainly because of its current internal structure (using a single global IFQ
> lock).
>
> Is it part of the plan to abstract multiqueuing from alternate queuing algorithm or will it still be
> left to ALTQ, through driver managed queues for example, to manage the TX DMA ring lock?
I don't know yet how exactly it will look like. ALTQ, in modified
form, will remain part of the functionality set. Most multi-queue
network cards also support various modes of queue arbitration for
different service classes. This may be leveraged for ALTQ as well.
There's many different variants of multi-queue usage depending on
the goal of the overall system. I try to allow them to co-exist and
to be selectable at run-time while remaining at a sane complexity level.
--
Andre
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?50A17497.10401>