From owner-freebsd-hackers@FreeBSD.ORG Thu Apr 10 15:23:16 2003 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id B355537B401; Thu, 10 Apr 2003 15:23:16 -0700 (PDT) Received: from ebb.errno.com (ebb.errno.com [66.127.85.87]) by mx1.FreeBSD.org (Postfix) with ESMTP id E56F943F3F; Thu, 10 Apr 2003 15:23:15 -0700 (PDT) (envelope-from sam@errno.com) Received: from melange (melange.errno.com [66.127.85.82]) (authenticated bits=0) by ebb.errno.com (8.12.9/8.12.9) with ESMTP id h3AMNCpw048669 (version=TLSv1/SSLv3 cipher=RC4-MD5 bits=128 verify=NO); Thu, 10 Apr 2003 15:23:13 -0700 (PDT) (envelope-from sam@errno.com) Message-ID: <02b201c2ffaf$c68acca0$52557f42@errno.com> From: "Sam Leffler" To: "Sean Chittenden" , "Jin Guojun [NCS]" References: <3E94A22D.174321F0@lbl.gov><20030409230733.GX79923@perrin.int.nxad.com> <3E94B993.D282DEB2@lbl.gov><20030410005846.GA79923@perrin.int.nxad.com> <3E95A37E.36186A9F@lbl.gov><3E95A653.8F5CE89C@lbl.gov> <3E95BEBB.58F86F4F@lbl.gov> <20030410201143.GF79923@perrin.int.nxad.com> Date: Thu, 10 Apr 2003 15:22:47 -0700 Organization: Errno Consulting MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 5.50.4920.2300 X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4920.2300 cc: freebsd-hackers@freebsd.org cc: freebsd-performance@freebsd.org Subject: Re: tcp_output starving -- is due to mbuf get delay? X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 Apr 2003 22:23:17 -0000 > > I guess I overlooked something after applying the patch (attached): > > > > Apr 10 12:11:52 ncs /kernel: sbappend: bad tail 0x0xc209a200 instead of 0x0xc243 > > 6c00 > > Apr 10 12:11:52 ncs /kernel: sbappend: bad tail 0x0xc2436c00 instead of 0x0xc238 > > bf00 > > Apr 10 12:11:52 ncs /kernel: sbappend: bad tail 0x0xc238bf00 instead of 0x0xc243 > > f300 > > ... > > > > A large number of such message was added into /var/log/message. This > > indicates either bad patch code or something I changed in the patch > > to make it work in 4.8 (attached). > > > > Any thought? > > That's likely the sign that the patch isn't appending to the tail of > the list correctly. Doing a tail append where the tail is known > should be an O(1) operation and should make adding an mbuf to a > cluster faster. Right now it has to do a linear scan to append data, > iirc, which is likely _a_ cause of some performance degradation. I'm > not an mbuf expert, but I wonder how free mbuf's are identified. > Regardless, I'll see if I can't figure out where this problem is with > the patch, it should do nothing but make things faster. If this is a repeat of Jason Thorpe's tail pointer optimization for sbappend; the patch may have started from one I did. I never committed it because I could never reproduce the performance gains he saw. I attributed it to a difference between netbsd and freebsd's TCP window setup algorithms. My patch for -stable (now probably very out of date) is in http://www.freebsd.org/~sam/thorpe-stable.patch. I haven't been following this thread closely, but FWIW I routinely get ~700 Mb/s running netperf between two -stable machines connected by a cross-over cable. Each machine has an Intel PRO/1000 card (em driver); 32-bit PCI in one machine and 64-bit in the other, but I've gotten similar performance with 32-bit PCI on both sides. As others have noted you need to watch out for "environmental factors" in understanding performance. Aside from hardware issues (there are many), be especially warry of IRQ entropy harvesting. Sam