From owner-freebsd-stable@FreeBSD.ORG Wed Sep 8 16:52:25 2010 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 46B7210656C3 for ; Wed, 8 Sep 2010 16:52:25 +0000 (UTC) (envelope-from pyunyh@gmail.com) Received: from mail-pw0-f54.google.com (mail-pw0-f54.google.com [209.85.160.54]) by mx1.freebsd.org (Postfix) with ESMTP id 126518FC1A for ; Wed, 8 Sep 2010 16:52:24 +0000 (UTC) Received: by pwi8 with SMTP id 8so160867pwi.13 for ; Wed, 08 Sep 2010 09:52:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:received:from:date:to:subject :message-id:reply-to:references:mime-version:content-type :content-disposition:in-reply-to:user-agent; bh=5kg7aBaossDUAvnbSWr/4t23TmC5u1tjVVou7UJ1ghM=; b=Ee5pRz6D68byVwJEA6+FdoukRkXs+qewSuI4WAxpi28ksYA5soOphHNiuU1SS/Ol9B YDyQkTRjgfisGeZfjsp8tw8TzoH3jXR9kqdyEZW2FGUgPPaX0mlsUV5PZdyESohYTleS 3Vr00QZMee+Xe7ra5DNxsmj/9hYWYXCo7W/RE= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=from:date:to:subject:message-id:reply-to:references:mime-version :content-type:content-disposition:in-reply-to:user-agent; b=ASFO3zAUsCH/Frw+wLkL3wfFrmCMwcZ8Ui+1U7Pm9FGY4JtUTz2HNr0CHZ3lurYyZx sZDSDOlZXnrpbxpIOpkb4KmfQw1VbjWQu7EL2eIXdODNVO16p2c8eQGHwZO1dDOtu8nT NrX9LudRqVnQLjSgOfp/8yVNJMNiOws2StjBg= Received: by 10.114.89.11 with SMTP id m11mr109559wab.23.1283964738557; Wed, 08 Sep 2010 09:52:18 -0700 (PDT) Received: from pyunyh@gmail.com ([174.35.1.224]) by mx.google.com with ESMTPS id q6sm371310waj.10.2010.09.08.09.52.15 (version=TLSv1/SSLv3 cipher=RC4-MD5); Wed, 08 Sep 2010 09:52:16 -0700 (PDT) Received: by pyunyh@gmail.com (sSMTP sendmail emulation); Wed, 8 Sep 2010 09:51:58 -0700 From: Pyun YongHyeon Date: Wed, 8 Sep 2010 09:51:58 -0700 To: "Mahlon E. Smith" , Jeremy Chadwick , freebsd-stable@freebsd.org Message-ID: <20100908165158.GB7203@michelle.cdnetworks.com> References: <20100907210813.GI49065@martini.nu> <20100907222403.GA18595@icarus.home.lan> <20100907233257.GA94092@martini.nu> <20100908002917.GO1439@michelle.cdnetworks.com> <20100908043834.GA27124@icarus.home.lan> <20100908143444.GB27923@martini.nu> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20100908143444.GB27923@martini.nu> User-Agent: Mutt/1.4.2.3i Cc: Subject: Re: Network memory allocation failures X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: pyunyh@gmail.com List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Sep 2010 16:52:25 -0000 On Wed, Sep 08, 2010 at 07:34:44AM -0700, Mahlon E. Smith wrote: > On Tue, Sep 07, 2010, Jeremy Chadwick wrote: > > > > I figured there might memory exhaustion of sorts, possibly in the bce(4) > > driver itself, that could cause the OP's problem. bce(4) might not be > > the problem at all. But the OP's issue seems to only occur when > > transmitting data, not receiving: > > > > http://lists.freebsd.org/pipermail/freebsd-stable/2010-September/058708.html > > More information: > > Looks like 100M wasn't enough of a test burst to tickle the problem in > my original message... 10G is, though. It's definitely happening in > both directions. > > Upgraded to -STABLE on one of the two machines last night, running > GENERIC. > > FreeBSD obb 8.1-STABLE FreeBSD 8.1-STABLE #0: Tue Sep 7 19:48:55 PDT 2010 root@obb:/usr/obj/usr/src/sys/GENERIC amd64 > > > Outgoing: > > obb# scp testfile root@holp:/usr/local/tmp/ > testfile 8% 856MB 37.6MB/s 04:09 ETA > Write failed: Cannot allocate memory > lost connection > obb# scp testfile root@holp:/usr/local/tmp/ > testfile 0% 72MB 34.3MB/s 04:56 ETA > Write failed: Cannot allocate memory > lost connection > > Incoming: > > obb# scp root@holp:/usr/local/tmp/testfile . > testfile 6% 670MB 31.9MB/s 04:59 ETA > Write failed: Cannot allocate memory > lost connection > obb# scp root@holp:/usr/local/tmp/testfile . > testfile 1% 118MB 39.3MB/s 04:17 ETA > Write failed: Cannot allocate memory > lost connection > obb# scp root@holp:/usr/local/tmp/testfile . > testfile 15% 1613MB 29.0MB/s 04:57 ETA > Write failed: Cannot allocate memory > lost connection > I think bce(4) may not be able to return ENOMEM to user land process so I guess it's not a bce(4) issue. To rule out possible driver issue, could you try other controller instead of bce(4)? > > > > The 2nd-to-last paragraph there is worth noting, specifically how > > limiting maximum addressable memory to 32GB via loader.conf seems to > > work around the issue. > > I'd no longer consider this a coincidence, limiting the memory to 16G > eliminates the issue completely. I'll retest with 32G today. > Again, this type of change has nothing to do with driver operation. bce(4) may have some issues on PAE but I don't think that would trigger problems on amd64 systems. > Incoming: > > obb# scp root@holp:/usr/local/tmp/testfile testfile2 > testfile 100% 10GB 17.8MB/s 09:35 > obb# scp root@holp:/usr/local/tmp/testfile testfile2 > testfile 100% 10GB 17.0MB/s 10:02 > > Outgoing: > > obb# scp testfile root@holp:/usr/local/tmp/testfile2 > testfile 100% 10GB 35.7MB/s 04:47 > obb# scp testfile root@holp:/usr/local/tmp/testfile2 > testfile 100% 10GB 35.4MB/s 04:49 > > > > There were other problems with the systems in question back in July, it > > seems. I assume these got hammered out somehow: > > > > http://www.mail-archive.com/freebsd-stable@freebsd.org/msg111408.html > > To a degree -- the initial install and cpu count problems are all fixed > up, thanks to help from the list. The Intel 10G panics were stifled > with a newer driver from Intel's site, but I ran out of time to do > any serious testing with it, and just ended up using the broadcoms to > satisfy my time constraint. > > -- > Mahlon E. Smith > http://www.martini.nu/contact.html