From owner-freebsd-net@FreeBSD.ORG Thu Feb 23 16:13:37 2012 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6D2E31065674 for ; Thu, 23 Feb 2012 16:13:37 +0000 (UTC) (envelope-from jpaetzel@freebsd.org) Received: from out5-smtp.messagingengine.com (out5-smtp.messagingengine.com [66.111.4.29]) by mx1.freebsd.org (Postfix) with ESMTP id 2DEE18FC1E for ; Thu, 23 Feb 2012 16:13:36 +0000 (UTC) Received: from compute4.internal (compute4.nyi.mail.srv.osa [10.202.2.44]) by gateway1.nyi.mail.srv.osa (Postfix) with ESMTP id 01471210F5 for ; Thu, 23 Feb 2012 10:58:00 -0500 (EST) Received: from frontend1.nyi.mail.srv.osa ([10.202.2.160]) by compute4.internal (MEProxy); Thu, 23 Feb 2012 10:58:00 -0500 DKIM-Signature: v=1; a=rsa-sha1; c=relaxed/relaxed; d= messagingengine.com; h=message-id:date:from:mime-version:to:cc :subject:references:in-reply-to:content-type :content-transfer-encoding; s=smtpout; bh=tFkUZ/ElGuDGrBOZqjoavv 40lsY=; b=DY3oQHs1Cbn4WXkbPATbEl77PlStQWD/qVbZvRL/fUHXmxsnPCszlD QukJqoY0fo5t43cVAyyzZgS/UGbOTMmnbS7i2/wzF68aay9kZnMY3WQBgHmT7BmR oliZraaYxKtaXLjAPIdzGsufDH3Zt6EzsKxfe6IhwqtIsBlLEAOgA= X-Sasl-enc: NgZYrRyPr9ZafP2kPqzyYYMZBtlcYUpodeM+AECAKSBE 1330012680 Received: from roadrash.tcbug.org (drawbridge.ixsystems.com [206.40.55.65]) by mail.messagingengine.com (Postfix) with ESMTPSA id C56DE8E0096; Thu, 23 Feb 2012 10:57:59 -0500 (EST) Message-ID: <4F466206.2000100@freebsd.org> Date: Thu, 23 Feb 2012 07:57:58 -0800 From: Josh Paetzel Organization: FreeBSS User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:10.0) Gecko/20120209 Thunderbird/10.0 MIME-Version: 1.0 To: Jack Vogel References: <20120222205231.GA81949@onelab2.iet.unipi.it> <1329944986.2621.46.camel@bwh-desktop> <20120222214433.GA82582@onelab2.iet.unipi.it> In-Reply-To: X-Enigmail-Version: 1.3.5 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: Ben Hutchings , FreeBSD Net , Luigi Rizzo , re , FreeBSD stable Subject: Re: nmbclusters: how do we want to fix this for 8.3 ? X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 23 Feb 2012 16:13:37 -0000 -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 02/22/2012 13:51, Jack Vogel wrote: > > > On Wed, Feb 22, 2012 at 1:44 PM, Luigi Rizzo > wrote: > > On Wed, Feb 22, 2012 at 09:09:46PM +0000, Ben Hutchings wrote: >> On Wed, 2012-02-22 at 21:52 +0100, Luigi Rizzo wrote: > ... >>> I have hit this problem recently, too. Maybe the issue >>> mostly/only exists on 32-bit systems. >> >> No, we kept hitting mbuf pool limits on 64-bit systems when we >> started working on FreeBSD support. > > ok never mind then, the mechanism would be the same, though the > limits (especially VM_LIMIT) would be different. > >>> Here is a possible approach: >>> >>> 1. nmbclusters consume the kernel virtual address space so >>> there must be some upper limit, say >>> >>> VM_LIMIT = 256000 (translates to 512MB of address space) >>> >>> 2. also you don't want the clusters to take up too much of the > available >>> memory. This one would only trigger for minimal-memory >>> systems, or virtual machines, but still... >>> >>> MEM_LIMIT = (physical_ram / 2) / 2048 >>> >>> 3. one may try to set a suitably large, desirable number of >>> buffers >>> >>> TARGET_CLUSTERS = 128000 >>> >>> 4. and finally we could use the current default as the >>> absolute > minimum >>> >>> MIN_CLUSTERS = 1024 + maxusers*64 >>> >>> Then at boot the system could say >>> >>> nmbclusters = min(TARGET_CLUSTERS, VM_LIMIT, MEM_LIMIT) >>> >>> nmbclusters = max(nmbclusters, MIN_CLUSTERS) >>> >>> >>> In turn, i believe interfaces should do their part and by >>> default never try to allocate more than a fraction of the total >>> number of buffers, >> >> Well what fraction should that be? It surely depends on how >> many interfaces are in the system and how many queues the other >> interfaces have. > >>> if necessary reducing the number of active queues. >> >> So now I have too few queues on my interface even after I >> increase the limit. >> >> There ought to be a standard way to configure numbers of queues >> and default queue lengths. > > Jack raised the problem that there is a poorly chosen default for > nmbclusters, causing one interface to consume all the buffers. If > the user explicitly overrides the value then the number of cluster > should be what the user asks (memory permitting). The next step is > on devices: if there are no overrides, the default for a driver is > to be lean. I would say that topping the request between 1/4 and > 1/8 of the total buffers is surely better than the current > situation. Of course if there is an explicit override, then use it > whatever happens to the others. > > cheers luigi > > > Hmmm, well, I could make the default use only 1 queue or something > like that, was thinking more of what actual users of the hardware > would want. > > After the installed kernel is booted and the admin would do > whatever post install modifications they wish it could be changed, > along with nmbclusters. > > This was why i sought opinions, of the algorithm itself, but also > anyone using ixgbe and igb in heavy use, what would you find most > convenient? > > Jack > The default setting is a thorn in our (with my ixsystems servers for freebsd hat on) side. A system with a quad port igb card and two onboard igb NICs won't boot stable/8 or 8.x-R to multiuser. Ditto for a dual port chelsio or ixgbe alongside dual onboard igb interfaces. My vote would be having systems over some minimum threshold of system ram to come up with a higher default for nmbclusters. You don't see too many 10gbe NICs in systems with 2GB of RAM.... - -- Thanks, Josh Paetzel FreeBSD -- The power to serve -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.18 (FreeBSD) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQEcBAEBAgAGBQJPRmIGAAoJEKFq1/n1feG229gIAIciDDKnc/K6/dgBA2YFGuuV V9cYD6+Zm4bVT9nvFhxJCUj+3CTGQFvNwi2sQx6pVMUWQC7Cpb323CShc8BBNjV3 vCzTmvqVshO+LWhx6J8lq4rqTU+PIKajF3GnwIWt4xmZ6WhrjCUySORYSAINQjKr iXJg/HBA7z/tsPUqOvzU0esZ4moUECapoldEOe0EF2jidARuM4q6MD1+QLMs+JSO JUS5yMPV022NVYS79NsUfvJ1cuwd6/I7CPvsJsW0E+zMMF2BjKZesU89zyFDST80 0WptoEqR9cuyApwu0OfDDzKyL7Z6G9yaAr0zkCAHATxkK0KArMJP/j2eT5uzZkE= =b44v -----END PGP SIGNATURE-----