From owner-freebsd-net@FreeBSD.ORG Thu Jun 16 20:55:00 2005 Return-Path: X-Original-To: freebsd-net@freebsd.org Delivered-To: freebsd-net@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 0DF3116A41C for ; Thu, 16 Jun 2005 20:55:00 +0000 (GMT) (envelope-from kfl@xiphos.ca) Received: from mail.net (custpop.ca.mci.com [142.77.1.111]) by mx1.FreeBSD.org (Postfix) with ESMTP id B660A43D49 for ; Thu, 16 Jun 2005 20:54:59 +0000 (GMT) (envelope-from kfl@xiphos.ca) Received: from [216.95.199.148] (account kfl@xiphos.ca HELO [192.168.1.7]) by mail.net (CommuniGate Pro SMTP 4.2.8) with ESMTP id 64053027; Thu, 16 Jun 2005 16:54:58 -0400 Message-ID: <42B1E82E.2050102@xiphos.ca> Date: Thu, 16 Jun 2005 16:59:26 -0400 From: Karim Fodil-Lemelin User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.7.5) Gecko/20041217 X-Accept-Language: en-us, en MIME-Version: 1.0 To: dima <_pppp@mail.ru> References: In-Reply-To: Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Cc: dave baukus , freebsd-net@freebsd.org Subject: Re: (panic) Lots of network memory needed X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 16 Jun 2005 20:55:00 -0000 I use FreeBSD 4.9. Actually I had to _reduce_ VM_KMEM_SIZE in order to be able to create that much mnbclusters ... Somehow it seems that vm_kmem_size cannot go over 1GB without crashing the system so I had to reduce memory map from other subsystems in order to get what I needed for network memory. From: vm_kmem_size = VM_KMEM_SIZE + nmbclusters * PAGE_SIZE; Assuming vm_kmem_size has some sort of hard limit, if I reduce VM_KMEM_SIZE I can get more nmbclusters into the system. dima wrote: >>I'm fairly sure that you are out of kernel virtual memory. >>Look at kern/kern_malloc.c kmeminit (); you can play w/ VM_KMEM_SIZE_MAX >>or TUNABLE_INT_FETCH("kern.vm.kmem.size", &vm_kmem_size); >> >> /* >> * Try to auto-tune the kernel memory size, so that it is >> * more applicable for a wider range of machine sizes. >> * On an X86, a VM_KMEM_SIZE_SCALE value of 4 is good, while >> * a VM_KMEM_SIZE of 12MB is a fair compromise. The >> * VM_KMEM_SIZE_MAX is dependent on the maximum KVA space >> * available, and on an X86 with a total KVA space of 256MB, >> * try to keep VM_KMEM_SIZE_MAX at 80MB or below. >> * >> * Note that the kmem_map is also used by the zone allocator, >> * so make sure that there is enough space. >> */ >> vm_kmem_size = VM_KMEM_SIZE; >> mem_size = cnt.v_page_count * PAGE_SIZE; >> >> >Which version do you use? I see the following in my /sys/kern/kern_malloc.c: > vm_kmem_size = VM_KMEM_SIZE + nmbclusters * PAGE_SIZE; >.... > /* > * Limit kmem virtual size to twice the physical memory. > * This allows for kmem map sparseness, but limits the size > * to something sane. Be careful to not overflow the 32bit > * ints while doing the check. > */ > if (((vm_kmem_size / 2) / PAGE_SIZE) > cnt.v_page_count) > vm_kmem_size = 2 * cnt.v_page_count * PAGE_SIZE; >It's 5.4-p2 > > > > > >>#if defined(VM_KMEM_SIZE_SCALE) >> if ((mem_size / VM_KMEM_SIZE_SCALE) > vm_kmem_size) >> vm_kmem_size = mem_size / VM_KMEM_SIZE_SCALE; >>#endif >> >>#if defined(VM_KMEM_SIZE_MAX) >> if (vm_kmem_size >= VM_KMEM_SIZE_MAX) >> vm_kmem_size = VM_KMEM_SIZE_MAX; >>#endif >> >> /* Allow final override from the kernel environment */ >> TUNABLE_INT_FETCH("kern.vm.kmem.size", &vm_kmem_size); >> >> >>Karim Fodil-Lemelin wrote: >> >> >>>Thanks but the system still crashes (FreeBSD 4.9) with 131072. Here is a >>>backtrace showing just that: >>> >>>Debugger (msg=0xc02b6cdb "panic") at ../../i386/i386/db_interface.c:321 >>>321 } >>>(kgdb) bt >>>#0 Debugger (msg=0xc02b6cdb "panic") at ../../i386/i386/db_interface.c:321 >>>#1 0xc016a230 in panic ( >>> fmt=0xc02ea380 "pmap_enter: invalid page directory pdir=%#llx, >>>va=%#x\n") >>> at ../../kern/kern_shutdown.c:593 >>>#2 0xc0283594 in pmap_enter (pmap=0xc0340460, va=4292141056, m=0xc24b2848, >>> prot=7 '\a', wired=1) at ../../i386/i386/pmap.c:1943 >>>#3 0xc023ddd0 in vm_fault (map=0xc033322c, vaddr=4292141056, >>> fault_type=7 '\a', fault_flags=1) at ../../vm/vm_fault.c:841 >>>#4 0xc023df0a in vm_fault_wire (map=0xc033322c, start=4292141056, >>> end=4292149248) at ../../vm/vm_fault.c:915 >>>#5 0xc0240831 in vm_map_pageable (map=0xc033322c, start=4292141056, >>> real_end=4292149248, new_pageable=0) at ../../vm/vm_map.c:1817 >>>#6 0xc023ea25 in kmem_alloc (map=0xc033322c, size=8192) >>> at ../../vm/vm_kern.c:213 >>>#7 0xc024a46f in _zget (z=0xdb5c6e80) at ../../vm/vm_zone.c:425 >>>#8 0xc024a269 in zalloc (z=0xdb5c6e80) at ../../vm/vm_zone.c:60 >>>#9 0xc0196106 in namei (ndp=0xfa489ef4) at ../../kern/vfs_lookup.c:104 >>>#10 0xc01614ca in execve (p=0xfa482e00, uap=0xfa489f90) >>> at ../../kern/kern_exec.c:165 >>>#11 0xc01590a1 in start_init (dummy=0x0) at ../../kern/init_main.c:543 >>>(kgdb) p nmbclusters >>>$1 = 131072 >>>(kgdb) >>> >>>Goran Spirovski - MorEl On.net wrote: >>> >>> >>> >>>>AFAIK the number of mbufs (and consequently nmbclusters) has to be a >>>>power >>>>of 2, so you should set it to 131072 >>>> >>>>MorEl >>>> >>>>----- Original Message ----- From: "Karim Fodil-Lemelin" >>>>To: >>>>Sent: Wednesday, June 15, 2005 6:08 PM >>>>Subject: (panic) Lots of network memory needed >>>> >>>> >>>> >>>> >>>> >>>> >>>>>Hello, >>>>> >>>>>From kernel tuning page >>>>> >>>>> >>>>> >>>>(http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/configtuning-kern >>>> >>>>el-limits.html) >>>> >>>> >>>> >>>> >>>>>Some sentence about nmbclusters says: >>>>> >>>>>"Under no circumstances should you specify an arbitrarily high value for >>>>> >>>>> >>>>> >>>>this parameter as it could lead to a boot time crash." >>>> >>>> >>>> >>>> >>>>>Now I want to push the limits where I need 4KB buffer for each of the >>>>> >>>>> >>>>> >>>>32000 connections I want this server to handle. If I do the math: >>>> >>>> >>>> >>>> >>>>>(32000 (conns) * 4 (KB/buffer) * 2 (buffer/conn)) / 2048 (KB/cluster) >>>>> >>>>>= 128000 clusters >>>>> >>>>>So I set this arbitrary high value in loader.conf under >>>>> >>>>> >>>>> >>>>(kern.ipc.nmbclusters) and no surprises I get panic: pmap_enter >>>>invalid page >>>>directory pdir=0x3cb063, va=0xfff800 >>>> >>>> >>>> >>>> >>>>>(va has a weird address here) >>>>> >>>>>I know I am pushing the limits here but I have plenty of memory (2GB) on >>>>> >>>>> >>>>> >>>>this system (after all its just 250MB for network memory ;) and this is >>>>mainly just experimentation. >>>> >>>> >>>> >>>> >>>>>I would like some pointers toward fixing this. Is there another variable >>>>> >>>>> >>>>> >>>>tied into this (I guess so)? Could anybody points me to a technical >>>>document >>>>that would explain the relationship with that (those) other(s) presumed >>>>variable(s)? >>>> >>>> >>>> >>>> >>>>>Thank you, >>>>> >>>>> >>>>>Karim >>>>> >>>>> >>>>>_______________________________________________ >>>>>freebsd-net@freebsd.org mailing list >>>>>http://lists.freebsd.org/mailman/listinfo/freebsd-net >>>>>To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" >>>>> >>>>> >>>>> >>>> >>>> >>>> >>>> >>>> >>>> >>-- >>Dave Baukus >>dbaukus@chiaro.com >> Chiaro Networks Ltd. >> Richardson, Texas >> USA >> >>_______________________________________________ >>freebsd-net@freebsd.org mailing list >>http://lists.freebsd.org/mailman/listinfo/freebsd-net >>To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" >> >> >> > > > > -- Karim Fodil-Lemelin Lead Programmer Xiphos Technologies Inc. (514) 848-9640 x223 (514) 848-9644 fax www.xiplink.com -------------------------------------------------------------- The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. If you have received this in error, please contact the sender and delete this communication and any copy immediately. Thank you.