From owner-freebsd-current@FreeBSD.ORG Mon Nov 22 21:31:10 2004 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id E344716A4CE for ; Mon, 22 Nov 2004 21:31:10 +0000 (GMT) Received: from mail6.speakeasy.net (mail6.speakeasy.net [216.254.0.206]) by mx1.FreeBSD.org (Postfix) with ESMTP id 970AB43D55 for ; Mon, 22 Nov 2004 21:31:10 +0000 (GMT) (envelope-from jmg@hydrogen.funkthat.com) Received: (qmail 15864 invoked from network); 22 Nov 2004 21:31:10 -0000 Received: from gate.funkthat.com (HELO hydrogen.funkthat.com) ([69.17.45.168]) (envelope-sender ) by mail6.speakeasy.net (qmail-ldap-1.03) with SMTP for ; 22 Nov 2004 21:31:10 -0000 Received: from hydrogen.funkthat.com (zebixs@localhost.funkthat.com [127.0.0.1])iAMLV9B6005705; Mon, 22 Nov 2004 13:31:09 -0800 (PST) (envelope-from jmg@hydrogen.funkthat.com) Received: (from jmg@localhost) by hydrogen.funkthat.com (8.12.10/8.12.10/Submit) id iAMLV8en005704; Mon, 22 Nov 2004 13:31:08 -0800 (PST) Date: Mon, 22 Nov 2004 13:31:08 -0800 From: John-Mark Gurney To: Sean McNeil Message-ID: <20041122213108.GY57546@funkthat.com> Mail-Followup-To: Sean McNeil , Robert Watson , freebsd-stable@freebsd.org, freebsd-current@freebsd.org, Jeremie Le Hen References: <1101154446.79991.13.camel@server.mcneil.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <1101154446.79991.13.camel@server.mcneil.com> User-Agent: Mutt/1.4.1i X-Operating-System: FreeBSD 4.2-RELEASE i386 X-PGP-Fingerprint: B7 EC EF F8 AE ED A7 31 96 7A 22 B3 D8 56 36 F4 X-Files: The truth is out there X-URL: http://resnet.uoregon.edu/~gurney_j/ X-Resume: http://resnet.uoregon.edu/~gurney_j/resume.html cc: Jeremie Le Hen cc: freebsd-stable@freebsd.org cc: freebsd-current@freebsd.org cc: Robert Watson Subject: Re: Re[4]: serious networking (em) performance (ggate and NFS) problem X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list Reply-To: John-Mark Gurney List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 22 Nov 2004 21:31:11 -0000 Sean McNeil wrote this message on Mon, Nov 22, 2004 at 12:14 -0800: > On Mon, 2004-11-22 at 11:34 +0000, Robert Watson wrote: > > On Sun, 21 Nov 2004, Sean McNeil wrote: > > > > > I have to disagree. Packet loss is likely according to some of my > > > tests. With the re driver, no change except placing a 100BT setup with > > > no packet loss to a gigE setup (both linksys switches) will cause > > > serious packet loss at 20Mbps data rates. I have discovered the only > > > way to get good performance with no packet loss was to > > > > > > 1) Remove interrupt moderation > > > 2) defrag each mbuf that comes in to the driver. > > > > Sounds like you're bumping into a queue limit that is made worse by > > interrupting less frequently, resulting in bursts of packets that are > > relatively large, rather than a trickle of packets at a higher rate. > > Perhaps a limit on the number of outstanding descriptors in the driver or > > hardware and/or a limit in the netisr/ifqueue queue depth. You might try > > changing the default IFQ_MAXLEN from 50 to 128 to increase the size of the > > ifnet and netisr queues. You could also try setting net.isr.enable=1 to > > enable direct dispatch, which in the in-bound direction would reduce the > > number of context switches and queueing. It sounds like the device driver > > has a limit of 256 receive and transmit descriptors, which one supposes is > > probably derived from the hardware limit, but I have no documentation on > > hand so can't confirm that. > > I've tried bumping IFQ_MAXLEN and it made no difference. I could rerun And the default for if_re is RL_IFQ_MAXLEN which is already 512... As is mentioned below, the card can do 64 segments (which usually means 32 packets since each packet usually has a header + payload in seperate packets)... > this test to be 100% certain I suppose. It was done a while back. I > haven't tried net.isr.enable=1, but packet loss is in the transmission > direction. The device driver has been modified to have 1024 transmit > and receive descriptors each as that is the hardware limitation. That > didn't matter either. With 1024 descriptors I still lost packets > without the m_defrag. hmmm... you know, I wonder if this is a problem with the if_re not pulling enough data from memory before starting the transmit... Though we currently have it set for unlimited... so, that doesn't seem like it would be it.. > The most difficult thing for me to understand is: if this is some sort > of resource limitation why will it work with a slower phy layer > perfectly and not with the gigE? The only thing I could think of was > that the old driver was doing m_defrag calls when it filled the transmit > descriptor queues up to a certain point. Understanding the effects of > m_defrag would be helpful in figuring this out I suppose. maybe the chip just can't keep the transmit fifo loaded at the higher speeds... is it possible vls is doing a writev for multisegmented UDP packet? I'll have to look at this again... > > It would be interesting on the send and receive sides to inspect the > > counters for drops at various points in the network stack; i.e., are we > > dropping packets at the ifq handoff because we're overfilling the > > descriptors in the driver, are packets dropped on the inbound path going > > into the netisr due to over-filling before the netisr is scheduled, etc. > > And, it's probably interesting to look at stats on filling the socket > > buffers for the same reason: if bursts of packets come up the stack, the > > socket buffers could well be being over-filled before the user thread can > > run. > > Yes, this would be very interesting and should point out the problem. I > would do such a thing if I had enough knowledge of the network pathways. > Alas, I am very green in this area. The receive side has no issues, > though, so I would focus on transmit counters (with assistance). -- John-Mark Gurney Voice: +1 415 225 5579 "All that I will do, has been done, All that I have, has not."