From owner-freebsd-net@FreeBSD.ORG Wed Dec 12 14:36:24 2012 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 3A658C58; Wed, 12 Dec 2012 14:36:24 +0000 (UTC) (envelope-from fodillemlinkarim@gmail.com) Received: from mail-ie0-f182.google.com (mail-ie0-f182.google.com [209.85.223.182]) by mx1.freebsd.org (Postfix) with ESMTP id D4DEB8FC0A; Wed, 12 Dec 2012 14:36:23 +0000 (UTC) Received: by mail-ie0-f182.google.com with SMTP id s9so2016899iec.13 for ; Wed, 12 Dec 2012 06:36:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=message-id:date:from:user-agent:mime-version:to:cc:subject :references:in-reply-to:content-type:content-transfer-encoding; bh=PrCEWtpTs8vy36fdue68gM/1Y+ValUvZJoC/V1SkPyg=; b=YqiBuyPxYlFJk3OoI5gjdvk6NjvB8qsMdz9iobBOTTilvvhGuXLOCze5ZnZLB3lK5I GMw7r9M83t6tLUGI4w0CXBeiQCK6EJ9yEvQNXKU3+9+gEB1gcMwYyH949lCRk4KAfv4M C65bDUk9EuUOGteMNRWoCu+v7F+UzIp2wxtPwMdQKVQgGJ00nIOut9t+cOaGNYbWKVek Xmg4QUVZ16fA2Nl8gTltdd8//7tqP3K2oFeBPUqrgnr+mMjBwYnGkKBsiOmWAfOZ664i VdJxtMUZi+k+Yx1GO8q40JutJMVVyzhIZ695qnT4fKEk3p1XwYqS9HirpQguCXtgnJez ENoA== Received: by 10.50.33.174 with SMTP id s14mr1009426igi.11.1355322982967; Wed, 12 Dec 2012 06:36:22 -0800 (PST) Received: from [192.168.1.73] ([208.85.112.101]) by mx.google.com with ESMTPS id uj11sm1871421igb.15.2012.12.12.06.36.20 (version=SSLv3 cipher=OTHER); Wed, 12 Dec 2012 06:36:21 -0800 (PST) Message-ID: <50C89660.8070806@gmail.com> Date: Wed, 12 Dec 2012 09:36:16 -0500 From: Karim Fodil-Lemelin User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:17.0) Gecko/17.0 Thunderbird/17.0 MIME-Version: 1.0 To: =?ISO-8859-1?Q?Ermal_Lu=E7i?= Subject: Re: igb and ALTQ in 9.1-rc3 References: <20121211075853.GU48639@FreeBSD.org> <1355231140.51621.YahooMailClassic@web121603.mail.ne1.yahoo.com> <50C74990.2090803@gmail.com> <50C79252.5000509@gmail.com> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 8bit Cc: Karim Fodil-Lemelin , freebsd-net , "Clement Hermann \(nodens\)" X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 12 Dec 2012 14:36:24 -0000 On 12/12/2012 2:49 AM, Ermal Luçi wrote: > On Tue, Dec 11, 2012 at 9:06 PM, Karim Fodil-Lemelin < > fodillemlinkarim@gmail.com> wrote: > >> On 11/12/2012 11:27 AM, Ermal Luçi wrote: >> >>> On Tue, Dec 11, 2012 at 3:56 PM, Karim Fodil-Lemelin < >>> fodillemlinkarim@gmail.com> wrote: >>> >>> On 11/12/2012 9:15 AM, Ermal Luçi wrote: >>>> On Tue, Dec 11, 2012 at 2:05 PM, Barney Cordoba < >>>>> barney_cordoba@yahoo.com >>>>> >>>>>> **wrote: >>>>>> >>>>> --- On Tue, 12/11/12, Gleb Smirnoff wrote: >>>>>> From: Gleb Smirnoff >>>>>> >>>>>>> Subject: Re: igb and ALTQ in 9.1-rc3 >>>>>>> To: "Jack Vogel" >>>>>>> Cc: "Clement Hermann (nodens)" , "Barney >>>>>>> Cordoba" >>>>>>> >>>>>>> , freebsd-net@FreeBSD.org >>>>>> Date: Tuesday, December 11, 2012, 2:58 AM >>>>>>> On Mon, Dec 10, 2012 at 03:31:19PM >>>>>>> -0800, Jack Vogel wrote: >>>>>>> J> UH, maybe asking the owner of the driver would help >>>>>>> :) >>>>>>> J> >>>>>>> J> ... and no, I've never been aware of doing anything to >>>>>>> stop supporting altq >>>>>>> J> so you wouldn't see any commits. If there's something >>>>>>> in the altq code or >>>>>>> J> support (which I have nothing to do with) that caused >>>>>>> this no-one informed >>>>>>> J> me. >>>>>>> >>>>>>> Switching from if_start to if_transmit effectively disables >>>>>>> ALTQ support. >>>>>>> >>>>>>> AFAIR, there is some magic implemented in other drivers that >>>>>>> makes them >>>>>>> modern (that means using if_transmit), but still capable to >>>>>>> switch to queueing >>>>>>> mode if SIOCADDALTQ was casted upon them. >>>>>>> >>>>>>> It seems pretty difficult to say that something is compatible with >>>>>>> >>>>>> something else if it hasn't been tested in a few years. >>>>>> >>>>>> It seems to me that ATLQ is the one that should handle if_transmit. >>>>>> although it's a good argument for having a raw "send" function in >>>>>> drivers. Ethernet drivers don't need more than a send() routing that >>>>>> loads a packet into the ring. The decision on what to do if you can't >>>>>> queue a packet should be in the network layer, if we must still call >>>>>> things layers. >>>>>> >>>>>> "start" is a leftover from a day when you stuffed a buffer and waited >>>>>> for an interrupt to stuff in another. The whole idea is antiquated. >>>>>> >>>>>> Imagine drivers that pull packets off of a card and simply queue it; >>>>>> and that you simply submit a packet to be queued for transmit. Instead >>>>>> of trying to find 35 programmers that understand all of the lock BS, >>>>>> you only need to have a couple. >>>>>> >>>>>> I always disable all of the gobbledegook like checksum offloading. They >>>>>> just muddy the water and have very little effect on performance. A >>>>>> modern >>>>>> cpu can do a checksum as fast as you can manage the "capabilities" >>>>>> without >>>>>> disrupting the processing path. >>>>>> >>>>>> With FreeBSD, every driver is an experience. Some suck so bad that they >>>>>> should come with a warning. The MSK driver is completely useless, as >>>>>> an example. >>>>>> >>>>>> >>>>>> BC >>>>>> ______________________________****_________________ >>>>>> freebsd-net@freebsd.org mailing list >>>>>> http://lists.freebsd.org/****mailman/listinfo/freebsd-net >>>>>> >>>>>> To unsubscribe, send any mail to "freebsd-net-unsubscribe@**fre** >>>>>> ebsd.org >>>>>> " >>>>>> >>>>>> During implementation of if_transmit altq was not considered at all. >>>>>> >>>>> The default if_transmit provides some compatibility but that is void >>>>> since >>>>> altq has not been converted to call if_transmit after processing the >>>>> mbuf. >>>>> >>>>> ALTQ can be adapted quite easily to if_transmit model it just wasn't >>>>> done >>>>> at the time. >>>>> With if_transmit model it can even be modularized and not be a compile >>>>> kernel option since the queue of the iface is abstracted now. >>>>> >>>>> I have always wanted to do a diff but have not yet got to it. >>>>> The change is quite simple just provide an altq_transmit default method >>>>> and >>>>> just hook into if_transmit model on the fly. >>>>> You surely need to handle some iface events and enable altq based on >>>>> request but its is not a hard to implement. >>>>> >>>>> I will always have this in my TODO but not sure when i can get to it. >>>>> >>>>> The issue is not only that igb doesn't support if_transmit or if_start >>>>> >>>> method but that ALTQ isn't multiqueue ready and still uses the IFQ_LOCK >>>> for >>>> all of its enqueue/dequeue operations. A simple drop in of if_transmit is >>>> bound to cause race conditions on any multiqueue driver with ALTQ. >>>> >>>> I do have a patch set for this on igb but its ugly and needs more work >>>> although it should get you going. Let me know if your interested I will >>>> clean it up and send it over. For more information on ALTQ discussion and >>>> igb please read this thread: http://freebsd.1045724.n5.** >>>> nabble.com/em-igb-if-transmit-****drbr-and-ALTQ-td5760338.html >>>> **>>> drbr-and-ALTQ-td5760338.html >>>> Best regards, >>>> >>>> Karim. >>>> >>>> >>>> ALTQ needs to do serialization of packets to apply its policy. >>> While it can be extended to map driver queues to ALTQ queues that is not >>> easy done without getting some interface to the driver to communicate. >>> The poor man implementation would be to pass an index argument when >>> calling >>> driver transmit routine. >>> >>> I doubt it is realistic to map ALTQ queues to driver queues since ALTQ >> can have many more queues then what any driver would need and those queues >> are controlled by users in so many ways it is almost impossible to cater >> for all cases in all device drivers. This could lead to TX rings being >> quickly filled up and packets being dropped. A good example of this would >> be to have two ALTQ queues. One very fast 1Gbps and one very slow 512Kbps. >> Having to reference packets through indexes inside the driver queues for >> each of the corresponding packets in ALTQ's fast queue would result in too >> much overhead IMO and a potential ring exhaustion by the slow queue. Both >> queues having different queue backlog criteria for achieving their rate >> they must be maintained somewhat separately (which is what ALTQ does >> through its class_queue). >> >> That is a matter of policy rather than implementation. > If the implementation is there the policy maker can decide to do 1:1 > mapping with hw queues since it provides quite a bit of flexibility and > avoids buffering to much. Still unrealistic in my opinion since there is a chance there won't be enough HW queues for a 1:1 mapping. 1:1 implies the hardware supports enough HW queues for a given policy. I can easily use over 30 ALTQ HFSC classes on igb currently and igb only support 8 HW queues. > > >> ALTQ needs to be both a consumer and a producer of a device driver (or >> ideally to some interface to it). It needs to be able to 'consume' packets >> from an outgoing path to enqueue them and some way (historically if_start) >> to tell a driver to send a given packet. While the enqueue operation is >> usually through a direct dispatch the dequeuing can be done asynchronously >> by a timer through ALTQ's token bucket regulator. Not unlike buf ring ALTQ >> can, in the same if_transmit works, enqueue a packet and dequeue another >> packet to be sent. >> >> > Dummynet does the same and ALTQ would be quite similar to it in that regard > when it gets moved to if_transmit. > They just hook in different level and ALTQ does not need a looping by the > place it chose to hook in. I haven't looked at dummynet in years (I use it everyday but I haven't looked at the 8.x+ code) but I last time I checked, dummnet maintains its own sets of queues and does its own serialization separately from the driver queues. And in the same way ALTQ can sometimes queue and not transmit on an enqueue operation, dummynet can queue packets in its queues and wait for a timer to trigger the transmit of a frame. If you want to delay a packet you must be able to hold the transmissions and trigger it at a later time and for this I think it would be a mistake to clog the small number driver queues. > >> What is needed is a clear API for sending packets and a comprehensive set >> of rules for locking TX queues. Right now FreeBSD has ifqueue (if_start) >> with its well known IFQ_LOCK and if_transmit (drbr) with driver managed TX >> locks and while it seems the trend is to move towards if_transmit, the >> transition is not completed and the locking too much dependent on the >> driver implementation to be currently useable by ALTQ. >> >> Karim. >> > >