Date: Tue, 16 Oct 2012 08:38:17 -0400 From: John Baldwin <jhb@freebsd.org> To: Adrian Chadd <adrian@freebsd.org> Cc: "Alexander V. Chernikov" <melifaro@freebsd.org>, freebsd-net@freebsd.org, Jack Vogel <jfvogel@gmail.com>, net@freebsd.org, Luigi Rizzo <rizzo@iet.unipi.it> Subject: Re: ixgbe & if_igb RX ring locking Message-ID: <201210160838.17741.jhb@freebsd.org> In-Reply-To: <CAJ-Vmo=qMJXwYDUEPHRn49SrAOP%2BNt3v9FaxFXq8wB6Q_uCmPg@mail.gmail.com> References: <5079A9A1.4070403@FreeBSD.org> <201210151414.27318.jhb@freebsd.org> <CAJ-Vmo=qMJXwYDUEPHRn49SrAOP%2BNt3v9FaxFXq8wB6Q_uCmPg@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Monday, October 15, 2012 6:36:57 pm Adrian Chadd wrote: > The reason why I've started moving net80211 and ath _away_ from using > direct dispatch (for now) and to using a taskqueue for TX (and RX) is > because it's too freaking annoying right now to deal with all the > crazy long-held locks to guarantee consistency between multiple > transmitting threads. > > Considering that the driver and net80211 stack: > > * sometimes is PCI, sometimes is USB (with all the differing thread > models that exist there); > * sometimes bridge traffic, sometimes route traffic, sometimes source > or terminate TCP/UDP connections; > * sometimes has one sender, sometimes has multiple senders, with some > other modules in between (bridge, pf, ipfw, etc) with locks being held > here and there; > * since the stack(s) like doing direct dispatch, RX very often causes > TX to occur, which for some drivers will block on a long-held driver > lock (with all the LORs that occur) - and drivers that do this (eg > iwn) will simply drop the lock before passing the packet up. Dropping > the lock before passing net80211_input*() .. is just plain silly. > > Now, I'd _like_ to eventually make net80211/ath support direct > dispatch, but that also requires making sure only -one- transmitter is > working at once. I'd like to not have the extra context switch > overhead, but I haven't seen a better way of doing it yet. > > It's fun to see the gige/10ge driver have lots of long held locks with > lots of concurrent sender processes possibly blocking until TX > completes.. so I wonder if that has scaling issues for lots of > connections/sending processes. I don't follow how this is related to this thread at all (which has more to do with ixgbe scheduling duplicate work). However, is your issue that the stack locks (e.g. socket and protocol layer locks) are held across if_start/if_transmit? -- John Baldwin
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?201210160838.17741.jhb>