From owner-freebsd-net@FreeBSD.ORG Tue Aug 1 13:48:18 2006 Return-Path: X-Original-To: net@FreeBSD.org Delivered-To: freebsd-net@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id A811016A4E2; Tue, 1 Aug 2006 13:48:18 +0000 (UTC) (envelope-from gallatin@cs.duke.edu) Received: from duke.cs.duke.edu (duke.cs.duke.edu [152.3.140.1]) by mx1.FreeBSD.org (Postfix) with ESMTP id 2D11143DA2; Tue, 1 Aug 2006 13:47:59 +0000 (GMT) (envelope-from gallatin@cs.duke.edu) Received: from grasshopper.cs.duke.edu (grasshopper.cs.duke.edu [152.3.145.30]) by duke.cs.duke.edu (8.13.6/8.13.6) with ESMTP id k71Dlxbp012757 (version=TLSv1/SSLv3 cipher=EDH-RSA-DES-CBC3-SHA bits=168 verify=NO); Tue, 1 Aug 2006 09:47:59 -0400 (EDT) Received: (from gallatin@localhost) by grasshopper.cs.duke.edu (8.12.9p2/8.12.9/Submit) id k71Dlr6g061084; Tue, 1 Aug 2006 09:47:53 -0400 (EDT) (envelope-from gallatin) From: Andrew Gallatin MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <17615.23433.918293.466584@grasshopper.cs.duke.edu> Date: Tue, 1 Aug 2006 09:47:53 -0400 (EDT) To: Robert Watson In-Reply-To: <20060801142558.M64452@fledge.watson.org> References: <20060730141642.D16341@fledge.watson.org> <17615.18793.700752.342809@grasshopper.cs.duke.edu> <20060801142558.M64452@fledge.watson.org> X-Mailer: VM 6.75 under 21.1 (patch 12) "Channel Islands" XEmacs Lucid Cc: arch@FreeBSD.org, net@FreeBSD.org Subject: Re: Changes in the network interface queueing handoff model X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 01 Aug 2006 13:48:18 -0000 Robert Watson writes: > > On Tue, 1 Aug 2006, Andrew Gallatin wrote: > > > > - The ifnet send queue is a separately locked object from the device driver, > > > meaning that for a single enqueue/dequeue pair, we pay an extra four lock > > > operations (two for insert, two for remove) per packet. > > > > Going forward, especially now that we support sun4v CoolThreads hardware, > > we're going to want to rethink the "single lock" per transmit routine model > > that most drivers have. The most expensive operation in transmit routines > > is bus_dmamap_load_mbuf_sg(), especially when there is an IOMMU involved > > (like on CoolThreads machines) and there is no reason why this needs to be > > called with a driver's transmit lock held. I have hard data (from Solaris) > > about how much fine grained locking in a 10GbE driver's transmit routine > > helps. > > Right now, with the exception of locking for the ifnet dispatch queue, I > believe our ifnet API pretty much leaves decisions about the nature and > granularity of synchronization to the device driver author. The ifnet queue > is high on my list to address (hence this thread) -- are there any other parts > of our device driver framework that stand in the way from a device driver > being modified to support greater parallelism in sending? No, not that is directly related to ethernet drivers. However, busdma is a pain. Specifically, I hate that bus_dmamap_load_mbuf_sg() requires a bus_dmamap_t. That means that any fine-grained driver will need to "allocate" a bus_dmamap_t either via bus_dmamap_create(), or by pulling a pre-allocated bus_dmamap_t from a pre-allocated pool. Either will require a lock. Solaris has a similar problem, and I use the pool approach in my Solaris driver. Linux's pci_map_single()/pci_unmap_addr_set()/pci_unmap_len_set() is just so much nicer to use... Drew