From owner-freebsd-hackers@FreeBSD.ORG Wed Jul 26 11:35:52 2006 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 99BEF16A4E1 for ; Wed, 26 Jul 2006 11:35:52 +0000 (UTC) (envelope-from anderson@centtech.com) Received: from mh2.centtech.com (moat3.centtech.com [207.200.51.50]) by mx1.FreeBSD.org (Postfix) with ESMTP id 38E2243D58 for ; Wed, 26 Jul 2006 11:35:50 +0000 (GMT) (envelope-from anderson@centtech.com) Received: from [10.177.171.220] (neutrino.centtech.com [10.177.171.220]) by mh2.centtech.com (8.13.1/8.13.1) with ESMTP id k6QBZkcM027143; Wed, 26 Jul 2006 06:35:46 -0500 (CDT) (envelope-from anderson@centtech.com) Message-ID: <44C7539E.5060804@centtech.com> Date: Wed, 26 Jul 2006 06:35:58 -0500 From: Eric Anderson User-Agent: Thunderbird 1.5.0.4 (X11/20060612) MIME-Version: 1.0 To: Murat Balaban References: <20060726111657.GA22358@enderunix.org> In-Reply-To: <20060726111657.GA22358@enderunix.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: ClamAV 0.87.1/1621/Wed Jul 26 01:13:01 2006 on mh2.centtech.com X-Virus-Status: Clean Cc: freebsd-hackers@freebsd.org Subject: Re: sys/dev/em/if_em.c X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 26 Jul 2006 11:35:52 -0000 On 07/26/06 06:16, Murat Balaban wrote: > Hello hackers, > > I have a special-purpose setting where I have a ng_hub like kernel module (ng_lb) > which I've been coding. The box I'm using has two em(4) adapters, and I've > hooked em0's lower with my ng_lb's link0, and em1's lower with ng_lb's link1. > > Situation looks like this: > > lower link0 link1 lower > em0 ---------------> -------------> ng_lb --------------> ---------------> em1 > > Every packet that is received by em0 is handed over to my netgraph module and > after very little modification in the packet ethernet header (changing destination > mac addresss) I NG_FWD_ITEM() the packet to em1. > > I'm generating traffic with a packet generator, and em0 seems to be ok with around > 910 Mbit/s traffic. > > However if I write the packets into em1, em1 seems to drop 40-60 Mbit/s (of 910 Mbit/s) > data. I digged the problem a bit, and found out that, IFQ_HANDOFF, called deep inside > from NG_FWD_ITEM was returning ENOBUFS. > > A little more investigation proved me that the source of ENOBUFS error was that > the em1 was running out of Tx descriptors. The relative logic in dev/em/if_em.c > (em_encap) was that if # of Tx descriptors falls below a threshold, the driver > tries to clean transmit interrupts once. # of available Tx desc. is again checked > and if the number is still not incresed ENOBUFS error is returned. > > What I'd like to ask is, instead of cleaning the transmit interrupts only once, > why not do it many times till the number of available tx descriptors increases > to a moderate level? > > The following patch solved my problem, though I wanted to get your opinions about > it. What if adapter->num_tx_desc_avail never gets above EM_TX_CLEANUP_THRESHOLD ? Maybe adding another check (max loop counter) or something? > --- if_em_murat.c Wed Jul 26 13:59:22 2006 > +++ if_em.c Wed Jul 26 14:01:11 2006 > @@ -1177,11 +1177,9 @@ > * available hits the threshold > */ > if (adapter->num_tx_desc_avail <= EM_TX_CLEANUP_THRESHOLD) { > - em_clean_transmit_interrupts(adapter); > - if (adapter->num_tx_desc_avail <= EM_TX_CLEANUP_THRESHOLD) { > - adapter->no_tx_desc_avail1++; > - return(ENOBUFS); > - } > + do { > + em_clean_transmit_interrupts(adapter); > + while (adapter->num_tx_desc_avail <= EM_TX_CLEANUP_THRESHOLD); > } > > /* Eric -- ------------------------------------------------------------------------ Eric Anderson Sr. Systems Administrator Centaur Technology Anything that works is better than anything that doesn't. ------------------------------------------------------------------------