Date: Wed, 22 Feb 2012 21:52:32 +0100 From: Luigi Rizzo <rizzo@iet.unipi.it> To: Jack Vogel <jfvogel@gmail.com> Cc: FreeBSD Net <freebsd-net@freebsd.org>, FreeBSD stable <freebsd-stable@freebsd.org>, re <re@freebsd.org> Subject: Re: nmbclusters: how do we want to fix this for 8.3 ? Message-ID: <20120222205231.GA81949@onelab2.iet.unipi.it> In-Reply-To: <CAFOYbc=oU5DxZDZQZZe4wJhVDoP=ocVOnpDq7bT=HbVkAjffLQ@mail.gmail.com> References: <CAFOYbc=oU5DxZDZQZZe4wJhVDoP=ocVOnpDq7bT=HbVkAjffLQ@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Wed, Feb 22, 2012 at 11:56:29AM -0800, Jack Vogel wrote: > Using igb and/or ixgbe on a reasonably powered server requires 1K mbuf > clusters per MSIX vector, > that's how many are in a ring. Either driver will configure 8 queues on a > system with that many or more > cores, so 8K clusters per port... > > My test engineer has a system with 2 igb ports, and 2 10G ixgbe, this is > hardly heavy duty, and yet this > exceeds the default mbuf pool on the installed kernel (1024 + maxusers * > 64). > > Now, this can be immediately fixed by a sysadmin after that first boot, but > it does result in the second > driver that gets started to complain about inadequate buffers. > > I think the default calculation is dated and should be changed, but am not > sure the best way, so are > there suggestions/opinions about this, and might we get it fixed before 8.3 > is baked? I have hit this problem recently, too. Maybe the issue mostly/only exists on 32-bit systems. Here is a possible approach: 1. nmbclusters consume the kernel virtual address space so there must be some upper limit, say VM_LIMIT = 256000 (translates to 512MB of address space) 2. also you don't want the clusters to take up too much of the available memory. This one would only trigger for minimal-memory systems, or virtual machines, but still... MEM_LIMIT = (physical_ram / 2) / 2048 3. one may try to set a suitably large, desirable number of buffers TARGET_CLUSTERS = 128000 4. and finally we could use the current default as the absolute minimum MIN_CLUSTERS = 1024 + maxusers*64 Then at boot the system could say nmbclusters = min(TARGET_CLUSTERS, VM_LIMIT, MEM_LIMIT) nmbclusters = max(nmbclusters, MIN_CLUSTERS) In turn, i believe interfaces should do their part and by default never try to allocate more than a fraction of the total number of buffers, if necessary reducing the number of active queues. what do people think ? cheers luigi
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20120222205231.GA81949>