Skip site navigation (1)Skip section navigation (2)
Date:      Wed, 22 Feb 2012 13:51:46 -0800
From:      Jack Vogel <jfvogel@gmail.com>
To:        Luigi Rizzo <rizzo@iet.unipi.it>
Cc:        Ben Hutchings <bhutchings@solarflare.com>, FreeBSD Net <freebsd-net@freebsd.org>, FreeBSD stable <freebsd-stable@freebsd.org>, re <re@freebsd.org>
Subject:   Re: nmbclusters: how do we want to fix this for 8.3 ?
Message-ID:  <CAFOYbc=BWkvGuqAOVehaYEVc7R_4b1Cq1i7Ged=-YEpCekNvfA@mail.gmail.com>
In-Reply-To: <20120222214433.GA82582@onelab2.iet.unipi.it>
References:  <CAFOYbc=oU5DxZDZQZZe4wJhVDoP=ocVOnpDq7bT=HbVkAjffLQ@mail.gmail.com> <20120222205231.GA81949@onelab2.iet.unipi.it> <1329944986.2621.46.camel@bwh-desktop> <20120222214433.GA82582@onelab2.iet.unipi.it>

next in thread | previous in thread | raw e-mail | index | archive | help
On Wed, Feb 22, 2012 at 1:44 PM, Luigi Rizzo <rizzo@iet.unipi.it> wrote:

> On Wed, Feb 22, 2012 at 09:09:46PM +0000, Ben Hutchings wrote:
> > On Wed, 2012-02-22 at 21:52 +0100, Luigi Rizzo wrote:
> ...
> > > I have hit this problem recently, too.
> > > Maybe the issue mostly/only exists on 32-bit systems.
> >
> > No, we kept hitting mbuf pool limits on 64-bit systems when we started
> > working on FreeBSD support.
>
> ok never mind then, the mechanism would be the same, though
> the limits (especially VM_LIMIT) would be different.
>
> > > Here is a possible approach:
> > >
> > > 1. nmbclusters consume the kernel virtual address space so there
> > >    must be some upper limit, say
> > >
> > >         VM_LIMIT = 256000 (translates to 512MB of address space)
> > >
> > > 2. also you don't want the clusters to take up too much of the
> available
> > >    memory. This one would only trigger for minimal-memory systems,
> > >    or virtual machines, but still...
> > >
> > >         MEM_LIMIT = (physical_ram / 2) / 2048
> > >
> > > 3. one may try to set a suitably large, desirable number of buffers
> > >
> > >         TARGET_CLUSTERS = 128000
> > >
> > > 4. and finally we could use the current default as the absolute minimum
> > >
> > >         MIN_CLUSTERS = 1024 + maxusers*64
> > >
> > > Then at boot the system could say
> > >
> > >         nmbclusters = min(TARGET_CLUSTERS, VM_LIMIT, MEM_LIMIT)
> > >
> > >         nmbclusters = max(nmbclusters, MIN_CLUSTERS)
> > >
> > >
> > > In turn, i believe interfaces should do their part and by default
> > > never try to allocate more than a fraction of the total number
> > > of buffers,
> >
> > Well what fraction should that be?  It surely depends on how many
> > interfaces are in the system and how many queues the other interfaces
> > have.
>
> > > if necessary reducing the number of active queues.
> >
> > So now I have too few queues on my interface even after I increase the
> > limit.
> >
> > There ought to be a standard way to configure numbers of queues and
> > default queue lengths.
>
> Jack raised the problem that there is a poorly chosen default for
> nmbclusters, causing one interface to consume all the buffers.
> If the user explicitly overrides the value then
> the number of cluster should be what the user asks (memory permitting).
> The next step is on devices: if there are no overrides, the default
> for a driver is to be lean. I would say that topping the request between
> 1/4 and 1/8 of the total buffers is surely better than the current
> situation. Of course if there is an explicit override, then use
> it whatever happens to the others.
>
> cheers
> luigi
>

Hmmm, well, I could make the default use only 1 queue or something like
that,
was thinking more of what actual users of the hardware would want.

After the installed kernel is booted and the admin would do whatever post
install
modifications they wish it could be changed, along with nmbclusters.

This was why i sought opinions, of the algorithm itself, but also anyone
using
ixgbe and igb in heavy use, what would you find most convenient?

Jack



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAFOYbc=BWkvGuqAOVehaYEVc7R_4b1Cq1i7Ged=-YEpCekNvfA>