From owner-freebsd-net@FreeBSD.ORG Mon Oct 15 22:36:58 2012 Return-Path: Delivered-To: net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 18D8B52B; Mon, 15 Oct 2012 22:36:58 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-pb0-f54.google.com (mail-pb0-f54.google.com [209.85.160.54]) by mx1.freebsd.org (Postfix) with ESMTP id C171B8FC0A; Mon, 15 Oct 2012 22:36:57 +0000 (UTC) Received: by mail-pb0-f54.google.com with SMTP id rp8so5740118pbb.13 for ; Mon, 15 Oct 2012 15:36:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=NBeUqkxXqFANk8PmvxmkSkAK7q/Xh5Ri1zXO4hpRjz4=; b=q4DUF1Hs0bpzF44xS+NaYIkjvnNknLLQkoUlWn7lEQsaw7NsbCjv5iQz3pyqVpiJS3 MqEishawiHO4OCp11L/jcojcbDAtxuYNicKNPbkKr2kAgsgYZlTj6BM0tXnohiZtzTFU mCdqfXFPLuFM1oPuSKRVqsWyth1ugSyYAYFW5IRXeLGtEN26fuY+ir2SnD3iJozCPgAq 9wukRk5pI/1ZyKdJwcmyzkk5obX1oR1yydgwj9/TXlCBRctFCx4xhIOgPRvPjPNikZbh 4WdW7XVYMGqc+dXMRJKCESvoVJV3FKVPuC1HNJAJ/m48INUsNJ316D+5oJf8cZKaOIG0 4NXg== MIME-Version: 1.0 Received: by 10.66.80.133 with SMTP id r5mr10706792pax.24.1350340617342; Mon, 15 Oct 2012 15:36:57 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.68.146.233 with HTTP; Mon, 15 Oct 2012 15:36:57 -0700 (PDT) In-Reply-To: <201210151414.27318.jhb@freebsd.org> References: <5079A9A1.4070403@FreeBSD.org> <201210150904.27567.jhb@freebsd.org> <20121015163210.GW89655@FreeBSD.org> <201210151414.27318.jhb@freebsd.org> Date: Mon, 15 Oct 2012 15:36:57 -0700 X-Google-Sender-Auth: KyTp3ym2n1JjTzY8n01J-YeRRuM Message-ID: Subject: Re: ixgbe & if_igb RX ring locking From: Adrian Chadd To: John Baldwin Content-Type: text/plain; charset=ISO-8859-1 Cc: "Alexander V. Chernikov" , freebsd-net@freebsd.org, Jack Vogel , net@freebsd.org, Luigi Rizzo X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 15 Oct 2012 22:36:58 -0000 The reason why I've started moving net80211 and ath _away_ from using direct dispatch (for now) and to using a taskqueue for TX (and RX) is because it's too freaking annoying right now to deal with all the crazy long-held locks to guarantee consistency between multiple transmitting threads. Considering that the driver and net80211 stack: * sometimes is PCI, sometimes is USB (with all the differing thread models that exist there); * sometimes bridge traffic, sometimes route traffic, sometimes source or terminate TCP/UDP connections; * sometimes has one sender, sometimes has multiple senders, with some other modules in between (bridge, pf, ipfw, etc) with locks being held here and there; * since the stack(s) like doing direct dispatch, RX very often causes TX to occur, which for some drivers will block on a long-held driver lock (with all the LORs that occur) - and drivers that do this (eg iwn) will simply drop the lock before passing the packet up. Dropping the lock before passing net80211_input*() .. is just plain silly. Now, I'd _like_ to eventually make net80211/ath support direct dispatch, but that also requires making sure only -one- transmitter is working at once. I'd like to not have the extra context switch overhead, but I haven't seen a better way of doing it yet. It's fun to see the gige/10ge driver have lots of long held locks with lots of concurrent sender processes possibly blocking until TX completes.. so I wonder if that has scaling issues for lots of connections/sending processes. Adrian