From owner-freebsd-net@FreeBSD.ORG Tue Oct 16 12:49:19 2012 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 7455311C; Tue, 16 Oct 2012 12:49:19 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from bigwig.baldwin.cx (bigknife-pt.tunnel.tserv9.chi1.ipv6.he.net [IPv6:2001:470:1f10:75::2]) by mx1.freebsd.org (Postfix) with ESMTP id 42D758FC14; Tue, 16 Oct 2012 12:49:19 +0000 (UTC) Received: from jhbbsd.localnet (unknown [209.249.190.124]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id 8BF06B980; Tue, 16 Oct 2012 08:49:18 -0400 (EDT) From: John Baldwin To: Adrian Chadd Subject: Re: ixgbe & if_igb RX ring locking Date: Tue, 16 Oct 2012 08:38:17 -0400 User-Agent: KMail/1.13.5 (FreeBSD/8.2-CBSD-20110714-p20; KDE/4.5.5; amd64; ; ) References: <5079A9A1.4070403@FreeBSD.org> <201210151414.27318.jhb@freebsd.org> In-Reply-To: MIME-Version: 1.0 Content-Type: Text/Plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Message-Id: <201210160838.17741.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7 (bigwig.baldwin.cx); Tue, 16 Oct 2012 08:49:18 -0400 (EDT) Cc: "Alexander V. Chernikov" , freebsd-net@freebsd.org, Jack Vogel , net@freebsd.org, Luigi Rizzo X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 16 Oct 2012 12:49:19 -0000 On Monday, October 15, 2012 6:36:57 pm Adrian Chadd wrote: > The reason why I've started moving net80211 and ath _away_ from using > direct dispatch (for now) and to using a taskqueue for TX (and RX) is > because it's too freaking annoying right now to deal with all the > crazy long-held locks to guarantee consistency between multiple > transmitting threads. > > Considering that the driver and net80211 stack: > > * sometimes is PCI, sometimes is USB (with all the differing thread > models that exist there); > * sometimes bridge traffic, sometimes route traffic, sometimes source > or terminate TCP/UDP connections; > * sometimes has one sender, sometimes has multiple senders, with some > other modules in between (bridge, pf, ipfw, etc) with locks being held > here and there; > * since the stack(s) like doing direct dispatch, RX very often causes > TX to occur, which for some drivers will block on a long-held driver > lock (with all the LORs that occur) - and drivers that do this (eg > iwn) will simply drop the lock before passing the packet up. Dropping > the lock before passing net80211_input*() .. is just plain silly. > > Now, I'd _like_ to eventually make net80211/ath support direct > dispatch, but that also requires making sure only -one- transmitter is > working at once. I'd like to not have the extra context switch > overhead, but I haven't seen a better way of doing it yet. > > It's fun to see the gige/10ge driver have lots of long held locks with > lots of concurrent sender processes possibly blocking until TX > completes.. so I wonder if that has scaling issues for lots of > connections/sending processes. I don't follow how this is related to this thread at all (which has more to do with ixgbe scheduling duplicate work). However, is your issue that the stack locks (e.g. socket and protocol layer locks) are held across if_start/if_transmit? -- John Baldwin