From owner-freebsd-current@FreeBSD.ORG Wed Jul 12 00:30:58 2006 Return-Path: X-Original-To: freebsd-current@freebsd.org Delivered-To: freebsd-current@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 9438216A4E1 for ; Wed, 12 Jul 2006 00:30:58 +0000 (UTC) (envelope-from pyunyh@gmail.com) Received: from nz-out-0102.google.com (nz-out-0102.google.com [64.233.162.202]) by mx1.FreeBSD.org (Postfix) with ESMTP id 5CB8443D5D for ; Wed, 12 Jul 2006 00:30:53 +0000 (GMT) (envelope-from pyunyh@gmail.com) Received: by nz-out-0102.google.com with SMTP id f1so10605nzc for ; Tue, 11 Jul 2006 17:30:53 -0700 (PDT) DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=beta; d=gmail.com; h=received:date:from:to:cc:subject:message-id:reply-to:references:mime-version:content-type:content-disposition:in-reply-to:user-agent; b=ZEySfNPfh06SBtszmyDnQSMWRoiCaXwfVh/Sd61sQ6ylXJQejnC/Bw4cNQjLaM9drXB4RiWAWpmlrA4R4uQ62I7r41tekRfsfFqwPumVTHsGCV/MBmJY0Idj8NEf9+R/dhYOgFOwX8VS0tHwQjLRV+XxInGKEqHa3yTuNY0+u/U= Received: by 10.36.74.5 with SMTP id w5mr76130nza; Tue, 11 Jul 2006 17:30:53 -0700 (PDT) Received: from michelle.cdnetworks.co.kr ( [211.53.35.84]) by mx.gmail.com with ESMTP id 14sm199898nzp.2006.07.11.17.30.50; Tue, 11 Jul 2006 17:30:52 -0700 (PDT) Received: from michelle.cdnetworks.co.kr (localhost.cdnetworks.co.kr [127.0.0.1]) by michelle.cdnetworks.co.kr (8.13.5/8.13.5) with ESMTP id k6C0VC67009911 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Wed, 12 Jul 2006 09:31:12 +0900 (KST) (envelope-from pyunyh@gmail.com) Received: (from yongari@localhost) by michelle.cdnetworks.co.kr (8.13.5/8.13.5/Submit) id k6C0VBVD009910; Wed, 12 Jul 2006 09:31:11 +0900 (KST) (envelope-from pyunyh@gmail.com) Date: Wed, 12 Jul 2006 09:31:10 +0900 From: Pyun YongHyeon To: Bill Paul Message-ID: <20060712003110.GA9542@cdnetworks.co.kr> References: <20060710022737.GB1128@cdnetworks.co.kr> <20060711051532.83ED116A4E0@hub.freebsd.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20060711051532.83ED116A4E0@hub.freebsd.org> User-Agent: Mutt/1.4.2.1i Cc: csaba-ml@creo.hu, lydianconcepts@gmail.com, freebsd-current@FreeBSD.ORG, brueffer@FreeBSD.ORG Subject: Re: Call for stge(4) testers X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: pyunyh@gmail.com List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 12 Jul 2006 00:30:58 -0000 On Tue, Jul 11, 2006 at 05:15:32AM +0000, Bill Paul wrote: > > On Sat, Jul 08, 2006 at 09:24:23PM -0700, Matthew Jacob wrote: > > > Jason THorpe did one for NetBSD. > > > > > > I did one for Solaris. Nice chip. Too bad it wasn't more popular. > > > > > > > I'm satisfied with the performance and easy interface for the chip. > > But the lack of Tx interrupt moderation and extra accesses to a > > status register to check which kind of Tx errors were occurred are > > serious fault. Since the the error condition uses the same status > > bit in interrupt status register it's hard to distingush failures > > from sucess without extra register accesses. > > > > BTW, I still have no clue how jumbo frame work on this chip. The > > documentation from wpaul didn't say any special things regarding > > jumbo frame. For instance the doc. said it supports jumbo frame > > up to 9022 but it seems the NIC can work with larger size. > > Even windows driver has a larger size than this, so I'm curious > > important part for jumbo frame were not released? > > You have failed to specify in what way jumbo frame support doesn't work. > In your previous mail, you only said there was a "jumbo frame problem" > but did not elaborate further. What exactly _is_ the problem? Does it > not transmit jumbo frames correctly? Does it not receive them correctly? > Does C'thulu emerge from inside the machine and devour your soul? > (If so, this is a sure sign you left the 'devour soul' bit in the > control register turned on. I *hate* it when that happens.) > Actually jumbo frame support works but it takes a very long time (30~50sec) to work correctly after selecting an MTU larger than 1500. During the time I could see the the NIC sends the jumbo frame with tcpdump on sender side but the frame doesn't arrive at destination. Sometimes the NIC generated 'Tx underrun' and sometimes it didn't generate any errors. However in the long run it did work after about a minuate without any action from user. I even changed STGE_TxStartThresh to use a full frame size but it seemed that it had no effects. Anyway, STGE_TxStartThresh is not used at all if stge(4) enabled checksum offload capability. > The documentation refers to a 'MaxFrameSize' register (offset 0x86, > 16 bits access) which controls the point at which the receiver will > consider an inbound frame to be a giant. The default is 0x5ea (1514 > bytes). I think you'll need to increase this to 9014 to receive 9K > jumbo frames (or 9018 for VLAN-tagged jumbo frames). It looks like > the register maxes out at 0x3FFF (16K), which is probably why the > Windows driver claimes to support frames larger than 9K. Note that > for this to work, you'll have to set up the RX descriptors with > enough buffer space to hold 9K of packet data. I haven't looked at > the driver code, but you should be able to set up the fragment array > in each RX descriptor to hold 5 mbufs with 2K clusters. You'll be > wasting a bit of space, but not necessarily as much as if you were > to allocate a single 9K buffer per descriptor. > I'm afraid the hardware doesn't support a single 9K buffer per decriptor. It seems that the hardware can support up to 4k buffer per descriptor so I chained received frames to a single jumbo frame. > I don't see anything that limits TX frame size, though you might want > to look around and see if there's a TX threshold setting and try to > increase it if you see underruns. > Linux driver mentioned that Tx size for a single frame should be less than 11008. I can't find that limitation from the chip documentation though. > You need to establish a proper test setup to diagnose the problem. Plug > the NIC directly into another known good gigE card that does jumbo frames, > use ifconfig to set the MTU on both ports to 9000, monitor both sides of > the link with tcpdump/ethereal/whatever, then do "ping -s 8192 " > from both ends and see what happens. > That's exactly what I did it on my tests. From your detailed explanation on jumbo frame internals and guide I've reread source code related to jumbo frame and managed to find a bug in my code. It was my programming error in selecting an MTU from ioctl handler. However I don't understand how the hardware can send/receive larger MTUs than 9022. If you program the hardware to receive 10K it eventually work in a one or two minute. Thanks for the hint and help. -- Regards, Pyun YongHyeon