From owner-freebsd-net@FreeBSD.ORG Mon Nov 12 21:47:13 2012 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id A753D122 for ; Mon, 12 Nov 2012 21:47:13 +0000 (UTC) (envelope-from fodillemlinkarim@gmail.com) Received: from mail-ia0-f182.google.com (mail-ia0-f182.google.com [209.85.210.182]) by mx1.freebsd.org (Postfix) with ESMTP id 6630A8FC14 for ; Mon, 12 Nov 2012 21:47:13 +0000 (UTC) Received: by mail-ia0-f182.google.com with SMTP id x2so211884iad.13 for ; Mon, 12 Nov 2012 13:47:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=message-id:date:from:user-agent:mime-version:to:cc:subject :references:in-reply-to:content-type:content-transfer-encoding; bh=ELNyghnvPiOVTeepFjaWCXMdbh4Ql3E00h5OmLojbq8=; b=F2KSx8BS+MSC+wDGSiOiTBh3og/Byn26eSkLYkJaydDyteAdi+JVSspIoxx7cXLz5h 4d3tlOKqZHyR79r4bNPX45sOXMnbN+QxdPfGMcE9OFRn7gA6VHiQ9fxiDAUBPxJvmEC5 TdT7Io/kOcPCkhWO29K6DaaK++r+OyWQRF0Kf6qAJoXT5xOl3Ubg6c0kCHvLDAiUNlby /MbuZ3TmMshs0FneBVYc4GEBBU69yQhJ4wwvc7dnigxi9R2nqf98wGdRo6t3Wv8Be7Bg N772zWFgZQftnjq5KgMCjLibm0bPB+LqMCUFPm5bWeBsNMzog4jFfH68jhpOgO0/4f5C jDDw== Received: by 10.50.188.136 with SMTP id ga8mr9315432igc.24.1352756832868; Mon, 12 Nov 2012 13:47:12 -0800 (PST) Received: from [192.168.1.71] ([208.85.112.101]) by mx.google.com with ESMTPS id x7sm7274530igk.8.2012.11.12.13.47.11 (version=SSLv3 cipher=OTHER); Mon, 12 Nov 2012 13:47:12 -0800 (PST) Message-ID: <50A16E5A.6080401@gmail.com> Date: Mon, 12 Nov 2012 16:47:06 -0500 From: Karim User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:11.0) Gecko/20120410 Thunderbird/11.0.1 MIME-Version: 1.0 To: Andre Oppermann Subject: Re: em/igb if_transmit (drbr) and ALTQ References: <50A14AEC.2040303@gmail.com> <50A15BB0.60102@networx.ch> In-Reply-To: <50A15BB0.60102@networx.ch> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-net@freebsd.org X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 Nov 2012 21:47:13 -0000 On 12-11-12 03:27 PM, Andre Oppermann wrote: > On 12.11.2012 20:15, Karim wrote: >> Hi all, >> >> I have been following the current discussions on igb tx/rx locking >> with great interest and I though >> I would point out (as it was before pointed out in kern/138392) that >> any driver setting if_start to >> NULL and using if_transmit (multi-queue or not) will effectively >> break ALTQ. >> >> The reason for this can be found in tbr_timeout() in altq_subr.c >> where the token bucket regulator >> goes through all ALTQ registered interface and tries to 'kick' them >> into sending through if_start(). >> >> tbr_timeout() : >> >> ... >> >> for (ifp = TAILQ_FIRST(&V_ifnet); ifp; >> ifp = TAILQ_NEXT(ifp, if_list)) { >> /* read from if_snd unlocked */ >> if (!TBR_IS_ENABLED(&ifp->if_snd)) >> continue; >> active++; >> if (!IFQ_IS_EMPTY(&ifp->if_snd) && >> ifp->if_start != NULL) /* >> if_start is NULL if if_transmit is >> used in em/igb driver */ >> (*ifp->if_start)(ifp); >> } >> >> ... >> >> As you can see if_start is NULL on those new multi-queue enabled >> drivers which has for net effect to >> 'break' ALTQ's token bucket regulator. >> >> I am writing this because I am interested in your comments on how >> this can be fixed properly looking >> forward. The whole range of suggestions; from 'don't compile with >> EM_MULTIQUEUE defined' to 'here is >> how you can make ALTQ use drbr' will help. > > This whole area needs some serious re-consideration for 10.0. > Even without the if_start issue ALTQ is somewhat broken because > the DMA rings nowadays are just too deep. > > See this recent message to this list for some thoughts: > http://lists.freebsd.org/pipermail/freebsd-net/2012-November/033780.html > > On top of refining the stack/driver boundary I'm thinking of a > dedicated ethernet layer to consolidate all the ethernet extensions, > have a single place to go and to prevent the drivers from re-inventing > the wheel all over every time. > > I'm working towards it and leaning into various drivers while trying > to bring in the hybrid interrupt/polling mode with life-lock prevention. > > It'll take a few weeks until I'm ready to show a stack/ethernet/driver > prototype for discussion. Parts may surface earlier in my tcp_workqueue > svn branch. > Hi, Glad to see someone looking into this :) The addition of a common layer for queuing algorithm is an interesting idea and makes me wonder how alternate queuing techniques would be able to use this shim layer to leverage devices with multiple queues. The current limitation (in terms of performances) with ALTQ going forward is its current inability to use multiple queues, in part because it was left out when the current (drbr) implementation of multiqueues was made but mainly because of its current internal structure (using a single global IFQ lock). Is it part of the plan to abstract multiqueuing from alternate queuing algorithm or will it still be left to ALTQ, through driver managed queues for example, to manage the TX DMA ring lock? Thanks, Karim.