From owner-freebsd-current@FreeBSD.ORG Fri Mar 5 21:56:04 2010 Return-Path: Delivered-To: current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id F1D32106564A for ; Fri, 5 Mar 2010 21:56:03 +0000 (UTC) (envelope-from pyunyh@gmail.com) Received: from qw-out-2122.google.com (qw-out-2122.google.com [74.125.92.25]) by mx1.freebsd.org (Postfix) with ESMTP id A15598FC0A for ; Fri, 5 Mar 2010 21:56:03 +0000 (UTC) Received: by qw-out-2122.google.com with SMTP id 8so999613qwh.7 for ; Fri, 05 Mar 2010 13:55:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:received:from:date:to:cc :subject:message-id:reply-to:references:mime-version:content-type :content-disposition:in-reply-to:user-agent; bh=B0ijHWpRN+A1lRc1eul0Dsxca2OPn/HkJOsHpyKBvD0=; b=fYMb0p7/EORtvjdvpXuYY3O2lhPmA+TiPV/5NMmOcXuS3iBlQ8WKCE0pbOL3TgqF1c F0/ekuH98kX1JnT+ZVZgK4fXoc8gJfXk7GiyuNwju6pAsE2461nErynRW1Zx8CaQTO6Y HX0LzOEK/aOrZfddinv88oa/SxdRGyN+JsNAA= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=from:date:to:cc:subject:message-id:reply-to:references:mime-version :content-type:content-disposition:in-reply-to:user-agent; b=JjExIGbL/Or1cqMw8fad4yQ7YmBvOXNbJy9F5nzfbQjlRZRVztifKOX7kKZsXQ2VmO SzWQkg2lsWuLml+9TR32+8FotDi2z8ni67QP//r/kD3oxisBwMIVCHQEJRGtPIDFx+6f PqK1Ep/bWvimkSW53X+LB1zEX5tlT1GpC9pRI= Received: by 10.224.87.19 with SMTP id u19mr794547qal.8.1267826157029; Fri, 05 Mar 2010 13:55:57 -0800 (PST) Received: from pyunyh@gmail.com ([174.35.1.224]) by mx.google.com with ESMTPS id 6sm3560785qwd.34.2010.03.05.13.55.55 (version=TLSv1/SSLv3 cipher=RC4-MD5); Fri, 05 Mar 2010 13:55:56 -0800 (PST) Received: by pyunyh@gmail.com (sSMTP sendmail emulation); Fri, 5 Mar 2010 13:55:39 -0800 From: Pyun YongHyeon Date: Fri, 5 Mar 2010 13:55:39 -0800 To: Ian FREISLICH Message-ID: <20100305215539.GG14818@michelle.cdnetworks.com> References: <20100305210435.GF14818@michelle.cdnetworks.com> <20100305184046.GD14818@michelle.cdnetworks.com> <20100305175639.GB14818@michelle.cdnetworks.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.4.2.3i Cc: current@freebsd.org Subject: Re: dev.bce.X.com_no_buffers increasing and packet loss X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: pyunyh@gmail.com List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 05 Mar 2010 21:56:04 -0000 On Fri, Mar 05, 2010 at 11:16:41PM +0200, Ian FREISLICH wrote: > Pyun YongHyeon wrote: > > Thanks for the info. Frankly, I have no idea how to explain the > > issue given that you have no heavy load. > > How many cores would be involved in handling the traffic and runnig > PF rules on this machine? There are 4x > CPU: Quad-Core AMD Opteron(tm) Processor 8354 (2194.51-MHz K8-class CPU) > In this server. I'm also using carp extensively. > pf(4) uses a single lock for processing, number of core would have no much benefit. > > I have a bce(4) patch which fixes a couple of bus_dma(9) issues as > > well as fixing some minor bugs. However I don't know whether the > > patch can fix the RX issue you're suffering from. Anyway, would you > > give it try the patch at the following URL? > > http://people.freebsd.org/~yongari/bce/bce.20100305.diff > > The patch was generated against CURRENT and you may see a message > > like "Disabling COAL_NOW timedout!" during interface up. You can > > ignore that message. > > Thanks. I'll give the patch a go on Monday when there are people > nearby if something goes wrong during the boot. I don't want to > loose the redundancy over the week end. > >From my testing on quad-port BCM5709 controller, it was stable. But I agree that your plan would be better. > Otherwise, is there another interface chip we can try? It's got I guess bce(4) and igb(4) would be one of the best controller. > an igb(4) quad port in there as well, but the performance is worse > on that chip than the bce(4) interface. It's also riddled with Yeah, I also noticed that. I think bce(4) seems to give more better performance numbers than igb(4). > vlan and other hardware offload bugs. I had good success in the > past with em(4), but it looks like igb is the PCI-e version. > It may depend on specific workloads. Last time I tried igb(4), the driver had a couple of bugs and after patching it, igb(4) also seemed to work well even though the performance was slightly slower than I initially expected. One thing I saw was using LRO on igb(4) showed slightly worse performance. Another thing for igb(4) case, it began to support multi-TX queues as well as RSS. Theoretically current multi-TX queue implementation can reorder packets such that it can give negative effects. bce(4) still lacks multi-TX queue support as well as RSS. bce(4) controllers also supports MSI-X as well as RSS so I have plan to implement it in future but it's hard to tell when I can find time to implement that.