From owner-freebsd-current@FreeBSD.ORG Thu Mar 1 14:45:50 2007 Return-Path: X-Original-To: freebsd-current@freebsd.org Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 77DCC16A406; Thu, 1 Mar 2007 14:45:50 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from cyrus.watson.org (cyrus.watson.org [209.31.154.42]) by mx1.freebsd.org (Postfix) with ESMTP id 3B85113C4B4; Thu, 1 Mar 2007 14:45:50 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from fledge.watson.org (fledge.watson.org [209.31.154.41]) by cyrus.watson.org (Postfix) with ESMTP id ACD0646F3F; Thu, 1 Mar 2007 09:45:49 -0500 (EST) Date: Thu, 1 Mar 2007 14:45:49 +0000 (GMT) From: Robert Watson X-X-Sender: robert@fledge.watson.org To: Andre Oppermann In-Reply-To: <45E6E2E8.5060408@freebsd.org> Message-ID: <20070301144321.O98688@fledge.watson.org> References: <45E6E2E8.5060408@freebsd.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: freebsd-net@freebsd.org, gallatin@freebsd.org, freebsd-current@freebsd.org, kmacy@freebsd.org Subject: Re: Large TCP send socket buffer optimizations X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 01 Mar 2007 14:45:50 -0000 On Thu, 1 Mar 2007, Andre Oppermann wrote: > With the TCP socket buffer autosizing and generally larger socket buffers > for high bandwidth and high delay connections tcp_output() has become > increasingly inefficient for sending segments. For every segment sent it is > traversing the entire socket buffer mbuf chain until it finds the offset to > continue from. Usually this is close to the end of the chain. Once it got > past a few dozen mbufs it starts to bust the CPU caches and performance > starts to fall off. > > This patch solves the problem by maintaining an offset pointer in the socket > buffer to give tcp_output() the closest mbuf right away avoiding the > traversal from the beginning. > > With this patch we should be able to compete nicely for the Internet land > speed record again. > > The patch is here: > > http://people.freebsd.org/~andre/sockbuf_sndptr-20070301.diff > > Any testing, especially on 10Gig cards, and feedback appreciated. I'm going to be on- and off-line from now through about the last week of March on travel to LinuxForum.dk, AsiaBSDCon, and UKUUG. However, once I get back online from that in late March through the end of April, I will be spending quite a bit of time looking at high performance packet capture issues, so can certainly spend some time looking at this. Per our earlier discussions and my previous exploration of this issue, I see precisely this sort of problem with TCP as socket buffer size grows, so should be set up to fairly easily test for an improvement once I have my feet on the ground. If I'm lucky I may get a breather for a few days from about 15 March - 18 March and could take an early look at it then. Thanks, Robert N M Watson Computer Laboratory University of Cambridge