Date: Wed, 22 Feb 2012 22:44:33 +0100 From: Luigi Rizzo <rizzo@iet.unipi.it> To: Ben Hutchings <bhutchings@solarflare.com> Cc: FreeBSD Net <freebsd-net@freebsd.org>, FreeBSD stable <freebsd-stable@freebsd.org>, Jack Vogel <jfvogel@gmail.com>, re <re@freebsd.org> Subject: Re: nmbclusters: how do we want to fix this for 8.3 ? Message-ID: <20120222214433.GA82582@onelab2.iet.unipi.it> In-Reply-To: <1329944986.2621.46.camel@bwh-desktop> References: <CAFOYbc=oU5DxZDZQZZe4wJhVDoP=ocVOnpDq7bT=HbVkAjffLQ@mail.gmail.com> <20120222205231.GA81949@onelab2.iet.unipi.it> <1329944986.2621.46.camel@bwh-desktop>
next in thread | previous in thread | raw e-mail | index | archive | help
On Wed, Feb 22, 2012 at 09:09:46PM +0000, Ben Hutchings wrote: > On Wed, 2012-02-22 at 21:52 +0100, Luigi Rizzo wrote: ... > > I have hit this problem recently, too. > > Maybe the issue mostly/only exists on 32-bit systems. > > No, we kept hitting mbuf pool limits on 64-bit systems when we started > working on FreeBSD support. ok never mind then, the mechanism would be the same, though the limits (especially VM_LIMIT) would be different. > > Here is a possible approach: > > > > 1. nmbclusters consume the kernel virtual address space so there > > must be some upper limit, say > > > > VM_LIMIT = 256000 (translates to 512MB of address space) > > > > 2. also you don't want the clusters to take up too much of the available > > memory. This one would only trigger for minimal-memory systems, > > or virtual machines, but still... > > > > MEM_LIMIT = (physical_ram / 2) / 2048 > > > > 3. one may try to set a suitably large, desirable number of buffers > > > > TARGET_CLUSTERS = 128000 > > > > 4. and finally we could use the current default as the absolute minimum > > > > MIN_CLUSTERS = 1024 + maxusers*64 > > > > Then at boot the system could say > > > > nmbclusters = min(TARGET_CLUSTERS, VM_LIMIT, MEM_LIMIT) > > > > nmbclusters = max(nmbclusters, MIN_CLUSTERS) > > > > > > In turn, i believe interfaces should do their part and by default > > never try to allocate more than a fraction of the total number > > of buffers, > > Well what fraction should that be? It surely depends on how many > interfaces are in the system and how many queues the other interfaces > have. > > if necessary reducing the number of active queues. > > So now I have too few queues on my interface even after I increase the > limit. > > There ought to be a standard way to configure numbers of queues and > default queue lengths. Jack raised the problem that there is a poorly chosen default for nmbclusters, causing one interface to consume all the buffers. If the user explicitly overrides the value then the number of cluster should be what the user asks (memory permitting). The next step is on devices: if there are no overrides, the default for a driver is to be lean. I would say that topping the request between 1/4 and 1/8 of the total buffers is surely better than the current situation. Of course if there is an explicit override, then use it whatever happens to the others. cheers luigi
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20120222214433.GA82582>