Date: Tue, 4 Dec 2012 13:31:23 -0800 From: Adrian Chadd <adrian@freebsd.org> To: Andre Oppermann <oppermann@networx.ch> Cc: Barney Cordoba <barney_cordoba@yahoo.com>, John Baldwin <jhb@freebsd.org>, freebsd-net@freebsd.org Subject: Re: Latency issues with buf_ring Message-ID: <CAJ-Vmok%2BW_LgSCnETLOAogucqUSy%2ByBixsdNj-2Aepy%2B1Lo7gw@mail.gmail.com> In-Reply-To: <50BE56C8.1030804@networx.ch> References: <1353259441.19423.YahooMailClassic@web121605.mail.ne1.yahoo.com> <201212041108.17645.jhb@freebsd.org> <CAJ-Vmo=tFFkeK2uADMPuBrgX6wN_9TSjAgs0WKPCrEfyhkG6Pw@mail.gmail.com> <50BE56C8.1030804@networx.ch>
next in thread | previous in thread | raw e-mail | index | archive | help
On 4 December 2012 12:02, Andre Oppermann <oppermann@networx.ch> wrote: > Our IF_* stack/driver boundary handoff isn't up to the task anymore. Right. well, the current hand off is really "here's a packet, go do stuff!" and the legacy if_start() method is just plain broken for SMP, preemption and direct dispatch. Things are also very special in the net80211 world, with the stack layer having to get its grubby fingers into things. I'm sure that the other examples of layered protocols (eg doing MPLS, or even just straight PPPoE style tunneling) has the same issues. Anything with sequence numbers and encryption being done by some other layer is going to have the same issue, unless it's all enforced via some other queue and a single thread handling the network stack "stuff". I bet direct-dispatch netgraph will have similar issues too, if it ever comes into existence. :-) > Also the interactions are either poorly defined or understood in many > places. I've had a few chats with yongari@ and am experimenting with > a modernized interface in my branch. > > The reason I stumbled across it was because I'm extending the hardware > offload feature set and found out that the stack and the drivers (and > the drivers among themself) are not really in sync with regards to behavior. > > For most if not all ethernet drivers from 100Mbit/s the TX DMA rings > are so large that buffering at the IFQ level doesn't make sense anymore > and only adds latency. So it could simply directly put everything into > the TX DMA and not even try to soft-queue. If the TX DMA ring is full > ENOBUFS is returned instead of filling yet another queue. However there > are ALTQ interactions and other mechanisms which have to be considered > too making it a bit more involved. net80211 has slightly different problems. We have requirements for per-node, per-TID/per-AC state (not just for QOS, but separate sequence numbers, different state machine handling for things like aggregation and (later) U-APSD handling, etc) so we do need to direct frames into different queues and then correctly serialise that mess. > I'm coming up with a draft and some benchmark results for an updated > stack/driver boundary in the next weeks before xmas. Ok. Please don't rush into it though; I'd like time to think about it after NY (as I may actually _have_ a holiday this xmas!) and I'd like to try and rope in people from non-ethernet-packet-pushing backgrounds to comment. They may have much stricter and/or stranger requirements when it comes to how the network layer passes, serialises and pushes packets to other layers. Thanks, Adrian
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAJ-Vmok%2BW_LgSCnETLOAogucqUSy%2ByBixsdNj-2Aepy%2B1Lo7gw>