From owner-svn-src-all@FreeBSD.ORG Tue Sep 22 13:44:44 2009 Return-Path: Delivered-To: svn-src-all@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B04FA106566B; Tue, 22 Sep 2009 13:44:44 +0000 (UTC) (envelope-from gallatin@cs.duke.edu) Received: from duke.cs.duke.edu (duke.cs.duke.edu [152.3.140.1]) by mx1.freebsd.org (Postfix) with ESMTP id 645BB8FC19; Tue, 22 Sep 2009 13:44:44 +0000 (UTC) Received: from [172.31.193.10] (cpe-069-134-110-200.nc.res.rr.com [69.134.110.200]) (authenticated bits=0) by duke.cs.duke.edu (8.14.2/8.14.2) with ESMTP id n8MDihG7008811 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Tue, 22 Sep 2009 09:44:43 -0400 (EDT) X-DKIM: Sendmail DKIM Filter v2.8.3 duke.cs.duke.edu n8MDihG7008811 DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=cs.duke.edu; s=mail; t=1253627083; bh=ipLbrohWHpuIS1F/YNaCSAijjD6N8X4sYFN342HQD3Y=; h=Message-ID:Date:From:MIME-Version:To:CC:Subject:References: In-Reply-To:Content-Type:Content-Transfer-Encoding; b=CXzi1Rt4eWF8V3a6UgM+v8lsToGpCwI+opS7m/CoJOVBlhYEOx/8S5i/YCWae8qNc iLkVnMGj3TUt4coPHQbvz2MvA6+3mEEd3uXt2lVeiVJg0KKG99RaoZ9rW2pwSBU4B7 hllx/GH1JRl2Tu4YERkBJ8YOM2ApcyXuXkez3XqI= Message-ID: <4AB8D4C4.8090604@cs.duke.edu> Date: Tue, 22 Sep 2009 09:44:36 -0400 From: Andrew Gallatin User-Agent: Thunderbird 2.0.0.22 (X11/20090608) MIME-Version: 1.0 To: Robert Watson References: <200909211441.n8LEf721092082@svn.freebsd.org> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: svn-src-head@FreeBSD.org, svn-src-all@FreeBSD.org, src-committers@FreeBSD.org Subject: Re: svn commit: r197391 - head/sys/dev/mxge X-BeenThere: svn-src-all@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "SVN commit messages for the entire src tree \(except for " user" and " projects" \)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 22 Sep 2009 13:44:44 -0000 Robert Watson wrote: > > On Mon, 21 Sep 2009, Andrew Gallatin wrote: > >> Add support for throttling transmit bandwidth. This is most commonly >> used to reduce packet loss on high delay (WAN) paths with a >> slow link. > > Hi Drew-- > > Could you say a little more about the situations in which this is used? > I see (or think I see) that the card supports internal throttling, but As you say, the card supports it, and I ported it from our Linux and Solaris drivers because we've had a few requests for it. I don't pretend to know a lot about WAN tuning, but our Linux customers claim this works better for them than the Linux host based traffic shaping. > is there a reason, other than the hardware supporting it, for not to do > something like this higher in the stack before cycles are burned and PCI > bus bandwidth has been wasted? This throttling increases neither CPU usage nor PCI bus bandwidth usage. It uses a very simple mechanism of having the NIC insert delays between issuing PCIe DMA reads (its slightly more complex than this, but that's the easiest way to think of it). This decreases the maximum transmit bandwidth opaquely to the host. Throttle it enough, and it is almost as if you placed the NIC into a slower PCIe slot. The key is that only DMA reads are slowed, so the NIC can still receive at full bandwidth. The advantage over host based traffic shaping is that no software in the host is involved, so it uses no host CPU. It is also effective at throttling TCP connections where TSO is in use. This means you don't either have to disable TSO, or worry about 64KB bursts at full bandwidth like you would with a host based traffic shaping solution. In addition to WAN scenarios, I should also have mentioned that it is useful in LAN scenarios. Picture a cluster of machines based around some PCIe chipset which has an unbalanced read/write DMA performance (the HT2000 is one such chipset which can send at line rate, but can only receive at 7Gb/s). If link level flow control is not effective then you wind up with massive packet loss. TCP can sometimes deal with this, but UDP is just miserable when you have a steady state where the sender can send faster than the receiver can receive. In fact, one of the scenarios where this was most helpful was a project where data was being collected on machine A, and forwarded to machine B via UDP. For security reasons, the physical link was simplex (only one physical fiber between the TX laser on A and RX on B), so neither TCP, nor link level flow control was possible. These were also "unbalanced" machines. Drew