From owner-freebsd-stable@FreeBSD.ORG Wed Feb 22 21:51:49 2012 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 27449106570A; Wed, 22 Feb 2012 21:51:49 +0000 (UTC) (envelope-from jfvogel@gmail.com) Received: from mail-ww0-f50.google.com (mail-ww0-f50.google.com [74.125.82.50]) by mx1.freebsd.org (Postfix) with ESMTP id 53E4D8FC23; Wed, 22 Feb 2012 21:51:47 +0000 (UTC) Received: by wgbdq11 with SMTP id dq11so474666wgb.31 for ; Wed, 22 Feb 2012 13:51:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=Psq8BF48kHsLS7MmUu4YZ6ReVaw91IXluUKaCJdDWA0=; b=mp4FDt53UNOoU9PP8BRUuCNZDb05v9NIb6zmyiDUw66YAXEPLkeywiD5kj/jOvszHo WcJ5EVshpRGoxjG9bKUcD/xM0T7mmsnHGrW7uQWyh3W8gjVjL4oxUcY7iF/WcYYS3wXT TPPMGn56F2y5SvLuFPBtBrxO0+svUCwq3Kvwk= MIME-Version: 1.0 Received: by 10.180.104.4 with SMTP id ga4mr142392wib.17.1329947506782; Wed, 22 Feb 2012 13:51:46 -0800 (PST) Received: by 10.180.102.97 with HTTP; Wed, 22 Feb 2012 13:51:46 -0800 (PST) In-Reply-To: <20120222214433.GA82582@onelab2.iet.unipi.it> References: <20120222205231.GA81949@onelab2.iet.unipi.it> <1329944986.2621.46.camel@bwh-desktop> <20120222214433.GA82582@onelab2.iet.unipi.it> Date: Wed, 22 Feb 2012 13:51:46 -0800 Message-ID: From: Jack Vogel To: Luigi Rizzo Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: Ben Hutchings , FreeBSD Net , FreeBSD stable , re Subject: Re: nmbclusters: how do we want to fix this for 8.3 ? X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 22 Feb 2012 21:51:49 -0000 On Wed, Feb 22, 2012 at 1:44 PM, Luigi Rizzo wrote: > On Wed, Feb 22, 2012 at 09:09:46PM +0000, Ben Hutchings wrote: > > On Wed, 2012-02-22 at 21:52 +0100, Luigi Rizzo wrote: > ... > > > I have hit this problem recently, too. > > > Maybe the issue mostly/only exists on 32-bit systems. > > > > No, we kept hitting mbuf pool limits on 64-bit systems when we started > > working on FreeBSD support. > > ok never mind then, the mechanism would be the same, though > the limits (especially VM_LIMIT) would be different. > > > > Here is a possible approach: > > > > > > 1. nmbclusters consume the kernel virtual address space so there > > > must be some upper limit, say > > > > > > VM_LIMIT = 256000 (translates to 512MB of address space) > > > > > > 2. also you don't want the clusters to take up too much of the > available > > > memory. This one would only trigger for minimal-memory systems, > > > or virtual machines, but still... > > > > > > MEM_LIMIT = (physical_ram / 2) / 2048 > > > > > > 3. one may try to set a suitably large, desirable number of buffers > > > > > > TARGET_CLUSTERS = 128000 > > > > > > 4. and finally we could use the current default as the absolute minimum > > > > > > MIN_CLUSTERS = 1024 + maxusers*64 > > > > > > Then at boot the system could say > > > > > > nmbclusters = min(TARGET_CLUSTERS, VM_LIMIT, MEM_LIMIT) > > > > > > nmbclusters = max(nmbclusters, MIN_CLUSTERS) > > > > > > > > > In turn, i believe interfaces should do their part and by default > > > never try to allocate more than a fraction of the total number > > > of buffers, > > > > Well what fraction should that be? It surely depends on how many > > interfaces are in the system and how many queues the other interfaces > > have. > > > > if necessary reducing the number of active queues. > > > > So now I have too few queues on my interface even after I increase the > > limit. > > > > There ought to be a standard way to configure numbers of queues and > > default queue lengths. > > Jack raised the problem that there is a poorly chosen default for > nmbclusters, causing one interface to consume all the buffers. > If the user explicitly overrides the value then > the number of cluster should be what the user asks (memory permitting). > The next step is on devices: if there are no overrides, the default > for a driver is to be lean. I would say that topping the request between > 1/4 and 1/8 of the total buffers is surely better than the current > situation. Of course if there is an explicit override, then use > it whatever happens to the others. > > cheers > luigi > Hmmm, well, I could make the default use only 1 queue or something like that, was thinking more of what actual users of the hardware would want. After the installed kernel is booted and the admin would do whatever post install modifications they wish it could be changed, along with nmbclusters. This was why i sought opinions, of the algorithm itself, but also anyone using ixgbe and igb in heavy use, what would you find most convenient? Jack