Date: Tue, 11 Dec 2012 15:06:42 -0500 From: Karim Fodil-Lemelin <fodillemlinkarim@gmail.com> To: =?ISO-8859-1?Q?Ermal_Lu=E7i?= <eri@freebsd.org> Cc: Karim Fodil-Lemelin <fodillemlinkarim@gmail.com>, freebsd-net <freebsd-net@freebsd.org>, "Clement Hermann \(nodens\)" <nodens2099@gmail.com> Subject: Re: igb and ALTQ in 9.1-rc3 Message-ID: <50C79252.5000509@gmail.com> In-Reply-To: <CAPBZQG3d1uzyEgWK3SALiVwkaEXdCZUmkM3VZA-9QB1EsL-=JQ@mail.gmail.com> References: <20121211075853.GU48639@FreeBSD.org> <1355231140.51621.YahooMailClassic@web121603.mail.ne1.yahoo.com> <CAPBZQG030sSLT__b4cYA2e07t-77pvv9S3f8HnZfYpsXKEqTkQ@mail.gmail.com> <50C74990.2090803@gmail.com> <CAPBZQG3d1uzyEgWK3SALiVwkaEXdCZUmkM3VZA-9QB1EsL-=JQ@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On 11/12/2012 11:27 AM, Ermal Luçi wrote: > On Tue, Dec 11, 2012 at 3:56 PM, Karim Fodil-Lemelin < > fodillemlinkarim@gmail.com> wrote: > >> On 11/12/2012 9:15 AM, Ermal Luçi wrote: >> >>> On Tue, Dec 11, 2012 at 2:05 PM, Barney Cordoba <barney_cordoba@yahoo.com >>>> **wrote: >>> >>>> --- On Tue, 12/11/12, Gleb Smirnoff <glebius@FreeBSD.org> wrote: >>>> >>>> From: Gleb Smirnoff <glebius@FreeBSD.org> >>>>> Subject: Re: igb and ALTQ in 9.1-rc3 >>>>> To: "Jack Vogel" <jfvogel@gmail.com> >>>>> Cc: "Clement Hermann (nodens)" <nodens2099@gmail.com>, "Barney Cordoba" >>>>> >>>> <barney_cordoba@yahoo.com>, freebsd-net@FreeBSD.org >>>> >>>>> Date: Tuesday, December 11, 2012, 2:58 AM >>>>> On Mon, Dec 10, 2012 at 03:31:19PM >>>>> -0800, Jack Vogel wrote: >>>>> J> UH, maybe asking the owner of the driver would help >>>>> :) >>>>> J> >>>>> J> ... and no, I've never been aware of doing anything to >>>>> stop supporting altq >>>>> J> so you wouldn't see any commits. If there's something >>>>> in the altq code or >>>>> J> support (which I have nothing to do with) that caused >>>>> this no-one informed >>>>> J> me. >>>>> >>>>> Switching from if_start to if_transmit effectively disables >>>>> ALTQ support. >>>>> >>>>> AFAIR, there is some magic implemented in other drivers that >>>>> makes them >>>>> modern (that means using if_transmit), but still capable to >>>>> switch to queueing >>>>> mode if SIOCADDALTQ was casted upon them. >>>>> >>>>> It seems pretty difficult to say that something is compatible with >>>> something else if it hasn't been tested in a few years. >>>> >>>> It seems to me that ATLQ is the one that should handle if_transmit. >>>> although it's a good argument for having a raw "send" function in >>>> drivers. Ethernet drivers don't need more than a send() routing that >>>> loads a packet into the ring. The decision on what to do if you can't >>>> queue a packet should be in the network layer, if we must still call >>>> things layers. >>>> >>>> "start" is a leftover from a day when you stuffed a buffer and waited >>>> for an interrupt to stuff in another. The whole idea is antiquated. >>>> >>>> Imagine drivers that pull packets off of a card and simply queue it; >>>> and that you simply submit a packet to be queued for transmit. Instead >>>> of trying to find 35 programmers that understand all of the lock BS, >>>> you only need to have a couple. >>>> >>>> I always disable all of the gobbledegook like checksum offloading. They >>>> just muddy the water and have very little effect on performance. A modern >>>> cpu can do a checksum as fast as you can manage the "capabilities" >>>> without >>>> disrupting the processing path. >>>> >>>> With FreeBSD, every driver is an experience. Some suck so bad that they >>>> should come with a warning. The MSK driver is completely useless, as >>>> an example. >>>> >>>> >>>> BC >>>> ______________________________**_________________ >>>> freebsd-net@freebsd.org mailing list >>>> http://lists.freebsd.org/**mailman/listinfo/freebsd-net<http://lists.freebsd.org/mailman/listinfo/freebsd-net>; >>>> To unsubscribe, send any mail to "freebsd-net-unsubscribe@**freebsd.org<freebsd-net-unsubscribe@freebsd.org> >>>> " >>>> >>>> During implementation of if_transmit altq was not considered at all. >>> The default if_transmit provides some compatibility but that is void since >>> altq has not been converted to call if_transmit after processing the mbuf. >>> >>> ALTQ can be adapted quite easily to if_transmit model it just wasn't done >>> at the time. >>> With if_transmit model it can even be modularized and not be a compile >>> kernel option since the queue of the iface is abstracted now. >>> >>> I have always wanted to do a diff but have not yet got to it. >>> The change is quite simple just provide an altq_transmit default method >>> and >>> just hook into if_transmit model on the fly. >>> You surely need to handle some iface events and enable altq based on >>> request but its is not a hard to implement. >>> >>> I will always have this in my TODO but not sure when i can get to it. >>> >>> The issue is not only that igb doesn't support if_transmit or if_start >> method but that ALTQ isn't multiqueue ready and still uses the IFQ_LOCK for >> all of its enqueue/dequeue operations. A simple drop in of if_transmit is >> bound to cause race conditions on any multiqueue driver with ALTQ. >> >> I do have a patch set for this on igb but its ugly and needs more work >> although it should get you going. Let me know if your interested I will >> clean it up and send it over. For more information on ALTQ discussion and >> igb please read this thread: http://freebsd.1045724.n5.** >> nabble.com/em-igb-if-transmit-**drbr-and-ALTQ-td5760338.html<http://freebsd.1045724.n5.nabble.com/em-igb-if-transmit-drbr-and-ALTQ-td5760338.html>; >> >> Best regards, >> >> Karim. >> >> > ALTQ needs to do serialization of packets to apply its policy. > While it can be extended to map driver queues to ALTQ queues that is not > easy done without getting some interface to the driver to communicate. > The poor man implementation would be to pass an index argument when calling > driver transmit routine. > I doubt it is realistic to map ALTQ queues to driver queues since ALTQ can have many more queues then what any driver would need and those queues are controlled by users in so many ways it is almost impossible to cater for all cases in all device drivers. This could lead to TX rings being quickly filled up and packets being dropped. A good example of this would be to have two ALTQ queues. One very fast 1Gbps and one very slow 512Kbps. Having to reference packets through indexes inside the driver queues for each of the corresponding packets in ALTQ's fast queue would result in too much overhead IMO and a potential ring exhaustion by the slow queue. Both queues having different queue backlog criteria for achieving their rate they must be maintained somewhat separately (which is what ALTQ does through its class_queue). ALTQ needs to be both a consumer and a producer of a device driver (or ideally to some interface to it). It needs to be able to 'consume' packets from an outgoing path to enqueue them and some way (historically if_start) to tell a driver to send a given packet. While the enqueue operation is usually through a direct dispatch the dequeuing can be done asynchronously by a timer through ALTQ's token bucket regulator. Not unlike buf ring ALTQ can, in the same if_transmit works, enqueue a packet and dequeue another packet to be sent. What is needed is a clear API for sending packets and a comprehensive set of rules for locking TX queues. Right now FreeBSD has ifqueue (if_start) with its well known IFQ_LOCK and if_transmit (drbr) with driver managed TX locks and while it seems the trend is to move towards if_transmit, the transition is not completed and the locking too much dependent on the driver implementation to be currently useable by ALTQ. Karim.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?50C79252.5000509>