Date: Tue, 16 Oct 2012 18:26:40 -0700 From: Adrian Chadd <adrian@freebsd.org> To: John Baldwin <jhb@freebsd.org> Cc: "Alexander V. Chernikov" <melifaro@freebsd.org>, freebsd-net@freebsd.org, Jack Vogel <jfvogel@gmail.com>, net@freebsd.org, Luigi Rizzo <rizzo@iet.unipi.it> Subject: Re: ixgbe & if_igb RX ring locking Message-ID: <CAJ-Vmo=iYD2N36eq5wzKGh8f%2B792mBBG9LKvi6fVTjqrEpgCRg@mail.gmail.com> In-Reply-To: <201210160838.17741.jhb@freebsd.org> References: <5079A9A1.4070403@FreeBSD.org> <201210151414.27318.jhb@freebsd.org> <CAJ-Vmo=qMJXwYDUEPHRn49SrAOP%2BNt3v9FaxFXq8wB6Q_uCmPg@mail.gmail.com> <201210160838.17741.jhb@freebsd.org>
next in thread | previous in thread | raw e-mail | index | archive | help
On 16 October 2012 05:38, John Baldwin <jhb@freebsd.org> wrote: > I don't follow how this is related to this thread at all (which has more to do > with ixgbe scheduling duplicate work). However, is your issue that the stack > locks (e.g. socket and protocol layer locks) are held across > if_start/if_transmit? It's a comment on the larger scale architectural problem. Since if_transmit and if_start are called from multiple thread contexts, the current ways drivers implement this are: * support direct dispatch to hardware, but wrap the whole sending process in one enormous lock, to prevent packet reordering issues; or * drop TX and TX completion into a TX taskqueue (or multiple, one per hardware send queue) and push frames into that taskqueue via some queue and then wake said taskqueue up; or * some bastardised version of both. For the intel drivers, the locks are held for a (potentially) very long time. Both igb and ixgb both hold the locks for the entirety of the TX process. It's not protecting something like a queue operation, it's effectively serialising the entirety of the TX and TX completion process. That works ok-ish for ethernet drivers which are "send and ignore", but for wireless drivers where the stack implements a lot more state, it really does quite suck. And since wireless drivers have a top level idea of sequence and encryption (ie, it's not per-TCP stream, it's across multiple sending streams to a given node), I can't model the locking and serialisation on what the TCP/UDP code does. I wish we had a better way of implementing "serialisation without long, long held locks" but short of stuffing everything into a taskqueue and only locking the send queue involved, I can't really think of anything. Adrian
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAJ-Vmo=iYD2N36eq5wzKGh8f%2B792mBBG9LKvi6fVTjqrEpgCRg>