Date: Wed, 07 Nov 2012 22:24:31 -0800 From: Alfred Perlstein <bright@mu.org> To: John Baldwin <jhb@freebsd.org> Cc: src-committers@freebsd.org, Peter Wemm <peter@wemm.org>, svn-src-all@freebsd.org, Alfred Perlstein <alfred@freebsd.org>, svn-src-head@freebsd.org, Konstantin Belousov <kostikbel@gmail.com> Subject: Re: svn commit: r242029 - head/sys/kern Message-ID: <509B501F.5050109@mu.org> In-Reply-To: <201210250950.57161.jhb@freebsd.org> References: <201210250146.q9P1kLi8043704@svn.freebsd.org> <20121025080551.GG35915@deviant.kiev.zoral.com.ua> <201210250950.57161.jhb@freebsd.org>
next in thread | previous in thread | raw e-mail | index | archive | help
This is a multi-part message in MIME format. --------------090002040205010501030408 Content-Type: text/plain; charset=ISO-8859-15; format=flowed Content-Transfer-Encoding: 7bit [[ + peter ]] Folks, I spent quite a bit of time trying to figure out how to resolve maxusers scaling in a happy way for all. I think I came up with a solution. This solution should work for i386, and other 32 bit platforms, as well as scaling well for 64 bit (or higher) platforms that have virtually unlimited AND 64bit with limited kernel address space. Here is how it works: We calculate the maxusers value based on physical memory, and then clamp it down if physical memory exceeds kernel addressable memory. The algorithm actually remains the same for all architectures, with the exception that machines with large kernel address space it is no longer clamped at 384. I've attached a test program that lets you play with various values for VM_MIN_KERNEL_ADDRESS, VM_MAX_KERNEL_ADDRESS and physpages. (argv[1, 2, 3] respectively.) Please give me your feedback. Note a lot of the ugly types/casting is due to building this test program on i386 darwin. I will clean it up before committing. I just want to make sure this the right direction. I would like to commit this shortly. -Alfred On 10/25/12 6:50 AM, John Baldwin wrote: > On Thursday, October 25, 2012 4:05:51 am Konstantin Belousov wrote: >> On Thu, Oct 25, 2012 at 01:46:21AM +0000, Alfred Perlstein wrote: >>> Author: alfred >>> Date: Thu Oct 25 01:46:20 2012 >>> New Revision: 242029 >>> URL: http://svn.freebsd.org/changeset/base/242029 >>> >>> Log: >>> Allow autotune maxusers > 384 on 64 bit machines >>> >>> A default install on large memory machines with multiple 10gigE interfaces >>> were not being given enough mbufs to do full bandwidth TCP or NFS traffic. >>> >>> To keep the value somewhat reasonable, we scale back the number of >>> maxuers by 1/6 past the 384 point. This gives us enough mbufs for most >>> of our pretty basic 10gigE line-speed tests to complete. >>> >>> Modified: >>> head/sys/kern/subr_param.c >>> >>> Modified: head/sys/kern/subr_param.c >>> ============================================================================== >>> --- head/sys/kern/subr_param.c Thu Oct 25 01:27:01 2012 (r242028) >>> +++ head/sys/kern/subr_param.c Thu Oct 25 01:46:20 2012 (r242029) >>> @@ -278,8 +278,16 @@ init_param2(long physpages) >>> maxusers = physpages / (2 * 1024 * 1024 / PAGE_SIZE); >>> if (maxusers < 32) >>> maxusers = 32; >>> - if (maxusers > 384) >>> - maxusers = 384; >>> + /* >>> + * Clips maxusers to 384 on machines with <= 4GB RAM or 32bit. >>> + * Scales it down 6x for large memory machines. >>> + */ >>> + if (maxusers > 384) { >>> + if (sizeof(void *) <= 4) >>> + maxusers = 384; >>> + else >>> + maxusers = 384 + ((maxusers - 384) / 6); >>> + } >> This is unbelievably weird way to express the '64bit'. The >> #ifdef _LP64 is enough there instead of the runtime check. >> >> Also, are you sure that all our 64bit arches do not have KVA limitations ? > There is an active thread on current@ with a different patch that uses a KVA > constant to derive 384 instead. When we have updated tuning in the past > (e.g. adjusting the cap on maxvnodes), there was a bit of discussion rather > than a random drive by commit. I think we should probably hold off on making > changes here and figure out what the right way to fix this is in the thread on > current@ instead. Andre has already suggested divorcing mbuf tuning from > maxusers entirely for example. > --------------090002040205010501030408 Content-Type: text/plain; charset=UTF-8; x-mac-type="0"; x-mac-creator="0"; name="maxusers.c" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="maxusers.c" #include <stdio.h> #include <stdlib.h> #include <sys/types.h> #ifndef __FreeBSD__ #include <machine/types.h> #endif #define VM_MIN_KERNEL_ADDRESS atoull(argv[1]) #define VM_MAX_KERNEL_ADDRESS atoull(argv[2]) #define physpages atol(argv[3]) #define PAGE_SIZE (long long)4096 int maxusers; unsigned long long atoull(const char *str) { unsigned long long ret; if (sscanf(str, "%llu", &ret) != 1) { fprintf(stderr, "sscanf: error\n"); exit(1); } return ret; } int main(int argc, char **argv) { u_int64_t maxkvapages; long maxusers_kva; long maxusers_phys; printf("min: %llu, max: %llu\n", VM_MIN_KERNEL_ADDRESS, VM_MAX_KERNEL_ADDRESS); maxkvapages = (VM_MAX_KERNEL_ADDRESS - VM_MIN_KERNEL_ADDRESS) / (long long)PAGE_SIZE; printf("VM_ADDRESS_SPACE: %llu\n", (long long)((long long)VM_MAX_KERNEL_ADDRESS - (long long)VM_MIN_KERNEL_ADDRESS)); printf("maxkvapages: %ld\n", (long)maxkvapages); maxusers_kva = maxkvapages * 3 / 4 / 512; printf("maxusers_kva: %ld\n", maxusers_kva); maxusers_phys = physpages / 512; printf("maxusers_phys: %ld\n", maxusers_phys); if (maxusers_phys > maxusers_kva) { maxusers = maxusers_kva; } else { maxusers = maxusers_phys; } printf("maxusers: %d\n", maxusers); return 0; } --------------090002040205010501030408--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?509B501F.5050109>