From owner-freebsd-net@FreeBSD.ORG Wed Dec 12 07:49:33 2012 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 10AEE9A7 for ; Wed, 12 Dec 2012 07:49:33 +0000 (UTC) (envelope-from ermal.luci@gmail.com) Received: from mail-qc0-f182.google.com (mail-qc0-f182.google.com [209.85.216.182]) by mx1.freebsd.org (Postfix) with ESMTP id AF74C8FC08 for ; Wed, 12 Dec 2012 07:49:32 +0000 (UTC) Received: by mail-qc0-f182.google.com with SMTP id k19so175544qcs.13 for ; Tue, 11 Dec 2012 23:49:32 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=xhiQw2Helfx1yj/4jeLUyT/3mT31rrWa1in5fIMIrpE=; b=OHn08aKKBsoUN5aTY6shxotHECSQYvKxH6PgqUJNRLFoBtMS+9knRrtt/MQdlDfNG1 Ltw1CvxLoqlZI+tXjA/UoNlvTzOen9T5cIi0cHbGECOrdWhn3lzgZL0UPAnbKuYV9Fy7 Ii4Pm1/AD4a5Rgm9FlSg8Kljh08oa0QhqbAb8RUj3MjaqkRkbJR/viPJpi4DGFVlzUd7 tD0iakO3nOr22DJvmjVNMVpN8i1IScr7X7zCY9w8hGFnjCK5IPL4kH9hvnPKmCXdhdPH +xaTXxqP+kGE52T9y9Mlo3tOm6KuaL1SnizVbg/PUouMxV8sCHmuxr5p5oDokfBtgM88 nCpw== MIME-Version: 1.0 Received: by 10.49.132.69 with SMTP id os5mr234184qeb.28.1355298571882; Tue, 11 Dec 2012 23:49:31 -0800 (PST) Sender: ermal.luci@gmail.com Received: by 10.49.27.197 with HTTP; Tue, 11 Dec 2012 23:49:31 -0800 (PST) In-Reply-To: <50C79252.5000509@gmail.com> References: <20121211075853.GU48639@FreeBSD.org> <1355231140.51621.YahooMailClassic@web121603.mail.ne1.yahoo.com> <50C74990.2090803@gmail.com> <50C79252.5000509@gmail.com> Date: Wed, 12 Dec 2012 08:49:31 +0100 X-Google-Sender-Auth: t0-N_rV5l-KACPMjlgEMMLISIOw Message-ID: Subject: Re: igb and ALTQ in 9.1-rc3 From: =?ISO-8859-1?Q?Ermal_Lu=E7i?= To: Karim Fodil-Lemelin Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: freebsd-net , "Clement Hermann \(nodens\)" X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 12 Dec 2012 07:49:33 -0000 On Tue, Dec 11, 2012 at 9:06 PM, Karim Fodil-Lemelin < fodillemlinkarim@gmail.com> wrote: > On 11/12/2012 11:27 AM, Ermal Lu=E7i wrote: > >> On Tue, Dec 11, 2012 at 3:56 PM, Karim Fodil-Lemelin < >> fodillemlinkarim@gmail.com> wrote: >> >> On 11/12/2012 9:15 AM, Ermal Lu=E7i wrote: >>> >>> On Tue, Dec 11, 2012 at 2:05 PM, Barney Cordoba < >>>> barney_cordoba@yahoo.com >>>> >>>>> **wrote: >>>>> >>>> >>>> --- On Tue, 12/11/12, Gleb Smirnoff wrote: >>>>> >>>>> From: Gleb Smirnoff >>>>> >>>>>> Subject: Re: igb and ALTQ in 9.1-rc3 >>>>>> To: "Jack Vogel" >>>>>> Cc: "Clement Hermann (nodens)" , "Barney >>>>>> Cordoba" >>>>>> >>>>>> , freebsd-net@FreeBSD.org >>>>> >>>>> Date: Tuesday, December 11, 2012, 2:58 AM >>>>>> On Mon, Dec 10, 2012 at 03:31:19PM >>>>>> -0800, Jack Vogel wrote: >>>>>> J> UH, maybe asking the owner of the driver would help >>>>>> :) >>>>>> J> >>>>>> J> ... and no, I've never been aware of doing anything to >>>>>> stop supporting altq >>>>>> J> so you wouldn't see any commits. If there's something >>>>>> in the altq code or >>>>>> J> support (which I have nothing to do with) that caused >>>>>> this no-one informed >>>>>> J> me. >>>>>> >>>>>> Switching from if_start to if_transmit effectively disables >>>>>> ALTQ support. >>>>>> >>>>>> AFAIR, there is some magic implemented in other drivers that >>>>>> makes them >>>>>> modern (that means using if_transmit), but still capable to >>>>>> switch to queueing >>>>>> mode if SIOCADDALTQ was casted upon them. >>>>>> >>>>>> It seems pretty difficult to say that something is compatible with >>>>>> >>>>> something else if it hasn't been tested in a few years. >>>>> >>>>> It seems to me that ATLQ is the one that should handle if_transmit. >>>>> although it's a good argument for having a raw "send" function in >>>>> drivers. Ethernet drivers don't need more than a send() routing that >>>>> loads a packet into the ring. The decision on what to do if you can't >>>>> queue a packet should be in the network layer, if we must still call >>>>> things layers. >>>>> >>>>> "start" is a leftover from a day when you stuffed a buffer and waited >>>>> for an interrupt to stuff in another. The whole idea is antiquated. >>>>> >>>>> Imagine drivers that pull packets off of a card and simply queue it; >>>>> and that you simply submit a packet to be queued for transmit. Instea= d >>>>> of trying to find 35 programmers that understand all of the lock BS, >>>>> you only need to have a couple. >>>>> >>>>> I always disable all of the gobbledegook like checksum offloading. Th= ey >>>>> just muddy the water and have very little effect on performance. A >>>>> modern >>>>> cpu can do a checksum as fast as you can manage the "capabilities" >>>>> without >>>>> disrupting the processing path. >>>>> >>>>> With FreeBSD, every driver is an experience. Some suck so bad that th= ey >>>>> should come with a warning. The MSK driver is completely useless, as >>>>> an example. >>>>> >>>>> >>>>> BC >>>>> ______________________________****_________________ >>>>> freebsd-net@freebsd.org mailing list >>>>> http://lists.freebsd.org/****mailman/listinfo/freebsd-net >>>>> >>>>> > >>>>> To unsubscribe, send any mail to "freebsd-net-unsubscribe@**fre** >>>>> ebsd.org >>>>> > >>>>> >>>>> " >>>>> >>>>> During implementation of if_transmit altq was not considered at all= . >>>>> >>>> The default if_transmit provides some compatibility but that is void >>>> since >>>> altq has not been converted to call if_transmit after processing the >>>> mbuf. >>>> >>>> ALTQ can be adapted quite easily to if_transmit model it just wasn't >>>> done >>>> at the time. >>>> With if_transmit model it can even be modularized and not be a compile >>>> kernel option since the queue of the iface is abstracted now. >>>> >>>> I have always wanted to do a diff but have not yet got to it. >>>> The change is quite simple just provide an altq_transmit default metho= d >>>> and >>>> just hook into if_transmit model on the fly. >>>> You surely need to handle some iface events and enable altq based on >>>> request but its is not a hard to implement. >>>> >>>> I will always have this in my TODO but not sure when i can get to it. >>>> >>>> The issue is not only that igb doesn't support if_transmit or if_sta= rt >>>> >>> method but that ALTQ isn't multiqueue ready and still uses the IFQ_LOCK >>> for >>> all of its enqueue/dequeue operations. A simple drop in of if_transmit = is >>> bound to cause race conditions on any multiqueue driver with ALTQ. >>> >>> I do have a patch set for this on igb but its ugly and needs more work >>> although it should get you going. Let me know if your interested I will >>> clean it up and send it over. For more information on ALTQ discussion a= nd >>> igb please read this thread: http://freebsd.1045724.n5.** >>> nabble.com/em-igb-if-transmit-****drbr-and-ALTQ-td5760338.html >>> **>> drbr-and-ALTQ-td5760338.html >>> > >>> >>> Best regards, >>> >>> Karim. >>> >>> >>> ALTQ needs to do serialization of packets to apply its policy. >> While it can be extended to map driver queues to ALTQ queues that is not >> easy done without getting some interface to the driver to communicate. >> The poor man implementation would be to pass an index argument when >> calling >> driver transmit routine. >> >> I doubt it is realistic to map ALTQ queues to driver queues since ALTQ > can have many more queues then what any driver would need and those queue= s > are controlled by users in so many ways it is almost impossible to cater > for all cases in all device drivers. This could lead to TX rings being > quickly filled up and packets being dropped. A good example of this would > be to have two ALTQ queues. One very fast 1Gbps and one very slow 512Kbps= . > Having to reference packets through indexes inside the driver queues for > each of the corresponding packets in ALTQ's fast queue would result in to= o > much overhead IMO and a potential ring exhaustion by the slow queue. Both > queues having different queue backlog criteria for achieving their rate > they must be maintained somewhat separately (which is what ALTQ does > through its class_queue). > > That is a matter of policy rather than implementation. If the implementation is there the policy maker can decide to do 1:1 mapping with hw queues since it provides quite a bit of flexibility and avoids buffering to much. > ALTQ needs to be both a consumer and a producer of a device driver (or > ideally to some interface to it). It needs to be able to 'consume' packet= s > from an outgoing path to enqueue them and some way (historically if_start= ) > to tell a driver to send a given packet. While the enqueue operation is > usually through a direct dispatch the dequeuing can be done asynchronousl= y > by a timer through ALTQ's token bucket regulator. Not unlike buf ring ALT= Q > can, in the same if_transmit works, enqueue a packet and dequeue another > packet to be sent. > > Dummynet does the same and ALTQ would be quite similar to it in that regard when it gets moved to if_transmit. They just hook in different level and ALTQ does not need a looping by the place it chose to hook in. > What is needed is a clear API for sending packets and a comprehensive set > of rules for locking TX queues. Right now FreeBSD has ifqueue (if_start) > with its well known IFQ_LOCK and if_transmit (drbr) with driver managed T= X > locks and while it seems the trend is to move towards if_transmit, the > transition is not completed and the locking too much dependent on the > driver implementation to be currently useable by ALTQ. > > Karim. > --=20 Ermal