From owner-freebsd-arch@FreeBSD.ORG Tue Aug 1 12:30:49 2006 Return-Path: X-Original-To: arch@FreeBSD.org Delivered-To: freebsd-arch@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id BB52F16A4E0; Tue, 1 Aug 2006 12:30:49 +0000 (UTC) (envelope-from gallatin@cs.duke.edu) Received: from duke.cs.duke.edu (duke.cs.duke.edu [152.3.140.1]) by mx1.FreeBSD.org (Postfix) with ESMTP id 3AC9C43D73; Tue, 1 Aug 2006 12:30:43 +0000 (GMT) (envelope-from gallatin@cs.duke.edu) Received: from grasshopper.cs.duke.edu (grasshopper.cs.duke.edu [152.3.145.30]) by duke.cs.duke.edu (8.13.6/8.13.6) with ESMTP id k71CUgBl026750 (version=TLSv1/SSLv3 cipher=EDH-RSA-DES-CBC3-SHA bits=168 verify=NO); Tue, 1 Aug 2006 08:30:42 -0400 (EDT) Received: (from gallatin@localhost) by grasshopper.cs.duke.edu (8.12.9p2/8.12.9/Submit) id k71CUXn6061010; Tue, 1 Aug 2006 08:30:33 -0400 (EDT) (envelope-from gallatin) From: Andrew Gallatin MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <17615.18793.700752.342809@grasshopper.cs.duke.edu> Date: Tue, 1 Aug 2006 08:30:33 -0400 (EDT) To: Robert Watson In-Reply-To: <20060730141642.D16341@fledge.watson.org> References: <20060730141642.D16341@fledge.watson.org> X-Mailer: VM 6.75 under 21.1 (patch 12) "Channel Islands" XEmacs Lucid Cc: arch@FreeBSD.org, net@FreeBSD.org Subject: Re: Changes in the network interface queueing handoff model X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 01 Aug 2006 12:30:49 -0000 Robert Watson writes: > > 5BOne of the ideas that I, Scott Long, and a few others have been bouncing > around for some time is a restructuring of the network interface packet > transmission API to reduce the number of locking operations and allow network > device drivers increased control of the queueing behavior. Right now, it <....> > - The ifnet send queue is a separately locked object from the device driver, > meaning that for a single enqueue/dequeue pair, we pay an extra four lock > operations (two for insert, two for remove) per packet. > Going forward, especially now that we support sun4v CoolThreads hardware, we're going to want to rethink the "single lock" per transmit routine model that most drivers have. The most expensive operation in transmit routines is bus_dmamap_load_mbuf_sg(), especially when there is an IOMMU involved (like on CoolThreads machines) and there is no reason why this needs to be called with a driver's transmit lock held. I have hard data (from Solaris) about how much fine grained locking in a 10GbE driver's transmit routine helps. Drew