From owner-freebsd-net@FreeBSD.ORG Fri Jun 17 08:15:47 2005 Return-Path: X-Original-To: freebsd-net@freebsd.org Delivered-To: freebsd-net@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 417D216A41C for ; Fri, 17 Jun 2005 08:15:47 +0000 (GMT) (envelope-from _pppp@mail.ru) Received: from f26.mail.ru (f26.mail.ru [194.67.57.194]) by mx1.FreeBSD.org (Postfix) with ESMTP id BFA8D43D49 for ; Fri, 17 Jun 2005 08:15:46 +0000 (GMT) (envelope-from _pppp@mail.ru) Received: from mail by f26.mail.ru with local id 1DjC0u-000JHy-00; Fri, 17 Jun 2005 12:15:44 +0400 Received: from [212.5.170.174] by win.mail.ru with HTTP; Fri, 17 Jun 2005 12:15:44 +0400 From: dima <_pppp@mail.ru> To: Karim Fodil-Lemelin Mime-Version: 1.0 X-Mailer: mPOP Web-Mail 2.19 X-Originating-IP: [212.5.170.174] Date: Fri, 17 Jun 2005 12:15:44 +0400 In-Reply-To: <42B1E82E.2050102@xiphos.ca> Content-Type: text/plain; charset=koi8-r Content-Transfer-Encoding: 8bit Message-Id: Cc: freebsd-net@freebsd.org Subject: Re[2]: (panic) Lots of network memory needed X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: dima <_pppp@mail.ru> List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 17 Jun 2005 08:15:47 -0000 > I use FreeBSD 4.9. Actually I had to _reduce_ VM_KMEM_SIZE in order to > be able to create that much mnbclusters ... > > Somehow it seems that vm_kmem_size cannot go over 1GB without crashing > the system so I had to reduce memory map from other subsystems in order > to get what I needed for network memory. From: > > vm_kmem_size = VM_KMEM_SIZE + nmbclusters * PAGE_SIZE; > > Assuming vm_kmem_size has some sort of hard limit, if I reduce VM_KMEM_SIZE I can get more nmbclusters into the system. Sorry. I only have 5.x systems handy. According to /sys/i386/include/pmap.h /* * Size of Kernel address space. This is the number of page table pages * (4MB each) to use for the kernel. 256 pages == 1 Gigabyte. * This **MUST** be a multiple of 4 (eg: 252, 256, 260, etc). */ #ifndef KVA_PAGES #ifdef PAE #define KVA_PAGES 512 #else #define KVA_PAGES 256 #endif #endif Well, be very carefull if you're going to change that. Check also the kernel base address in /sys/i386/include/vmparam.h > > > dima wrote: > > >>I'm fairly sure that you are out of kernel virtual memory. > >>Look at kern/kern_malloc.c kmeminit (); you can play w/ VM_KMEM_SIZE_MAX > >>or TUNABLE_INT_FETCH("kern.vm.kmem.size", &vm_kmem_size); > >> > >> /* > >> * Try to auto-tune the kernel memory size, so that it is > >> * more applicable for a wider range of machine sizes. > >> * On an X86, a VM_KMEM_SIZE_SCALE value of 4 is good, while > >> * a VM_KMEM_SIZE of 12MB is a fair compromise. The > >> * VM_KMEM_SIZE_MAX is dependent on the maximum KVA space > >> * available, and on an X86 with a total KVA space of 256MB, > >> * try to keep VM_KMEM_SIZE_MAX at 80MB or below. > >> * > >> * Note that the kmem_map is also used by the zone allocator, > >> * so make sure that there is enough space. > >> */ > >> vm_kmem_size = VM_KMEM_SIZE; > >> mem_size = cnt.v_page_count * PAGE_SIZE; > >> > >> > >Which version do you use? I see the following in my /sys/kern/kern_malloc.c: > > vm_kmem_size = VM_KMEM_SIZE + nmbclusters * PAGE_SIZE; > >.... > > /* > > * Limit kmem virtual size to twice the physical memory. > > * This allows for kmem map sparseness, but limits the size > > * to something sane. Be careful to not overflow the 32bit > > * ints while doing the check. > > */ > > if (((vm_kmem_size / 2) / PAGE_SIZE) > cnt.v_page_count) > > vm_kmem_size = 2 * cnt.v_page_count * PAGE_SIZE; > >It's 5.4-p2 > > > > > > > > > > > >>#if defined(VM_KMEM_SIZE_SCALE) > >> if ((mem_size / VM_KMEM_SIZE_SCALE) > vm_kmem_size) > >> vm_kmem_size = mem_size / VM_KMEM_SIZE_SCALE; > >>#endif > >> > >>#if defined(VM_KMEM_SIZE_MAX) > >> if (vm_kmem_size >= VM_KMEM_SIZE_MAX) > >> vm_kmem_size = VM_KMEM_SIZE_MAX; > >>#endif > >> > >> /* Allow final override from the kernel environment */ > >> TUNABLE_INT_FETCH("kern.vm.kmem.size", &vm_kmem_size); > >> > >> > >>Karim Fodil-Lemelin wrote: > >> > >> > >>>Thanks but the system still crashes (FreeBSD 4.9) with 131072. Here is a > >>>backtrace showing just that: > >>> > >>>Debugger (msg=0xc02b6cdb "panic") at ../../i386/i386/db_interface.c:321 > >>>321 } > >>>(kgdb) bt > >>>#0 Debugger (msg=0xc02b6cdb "panic") at ../../i386/i386/db_interface.c:321 > >>>#1 0xc016a230 in panic ( > >>> fmt=0xc02ea380 "pmap_enter: invalid page directory pdir=%#llx, > >>>va=%#x\n") > >>> at ../../kern/kern_shutdown.c:593 > >>>#2 0xc0283594 in pmap_enter (pmap=0xc0340460, va=4292141056, m=0xc24b2848, > >>> prot=7 '\a', wired=1) at ../../i386/i386/pmap.c:1943 > >>>#3 0xc023ddd0 in vm_fault (map=0xc033322c, vaddr=4292141056, > >>> fault_type=7 '\a', fault_flags=1) at ../../vm/vm_fault.c:841 > >>>#4 0xc023df0a in vm_fault_wire (map=0xc033322c, start=4292141056, > >>> end=4292149248) at ../../vm/vm_fault.c:915 > >>>#5 0xc0240831 in vm_map_pageable (map=0xc033322c, start=4292141056, > >>> real_end=4292149248, new_pageable=0) at ../../vm/vm_map.c:1817 > >>>#6 0xc023ea25 in kmem_alloc (map=0xc033322c, size=8192) > >>> at ../../vm/vm_kern.c:213 > >>>#7 0xc024a46f in _zget (z=0xdb5c6e80) at ../../vm/vm_zone.c:425 > >>>#8 0xc024a269 in zalloc (z=0xdb5c6e80) at ../../vm/vm_zone.c:60 > >>>#9 0xc0196106 in namei (ndp=0xfa489ef4) at ../../kern/vfs_lookup.c:104 > >>>#10 0xc01614ca in execve (p=0xfa482e00, uap=0xfa489f90) > >>> at ../../kern/kern_exec.c:165 > >>>#11 0xc01590a1 in start_init (dummy=0x0) at ../../kern/init_main.c:543 > >>>(kgdb) p nmbclusters > >>>$1 = 131072 > >>>(kgdb) > >>> > >>>Goran Spirovski - MorEl On.net wrote: > >>> > >>> > >>> > >>>>AFAIK the number of mbufs (and consequently nmbclusters) has to be a > >>>>power > >>>>of 2, so you should set it to 131072 > >>>> > >>>>MorEl > >>>> > >>>>----- Original Message ----- From: "Karim Fodil-Lemelin" > >>>>To: > >>>>Sent: Wednesday, June 15, 2005 6:08 PM > >>>>Subject: (panic) Lots of network memory needed > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>>>Hello, > >>>>> > >>>>>From kernel tuning page > >>>>> > >>>>> > >>>>> > >>>>(http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/configtuning-kern > >>>> > >>>>el-limits.html) > >>>> > >>>> > >>>> > >>>> > >>>>>Some sentence about nmbclusters says: > >>>>> > >>>>>"Under no circumstances should you specify an arbitrarily high value for > >>>>> > >>>>> > >>>>> > >>>>this parameter as it could lead to a boot time crash." > >>>> > >>>> > >>>> > >>>> > >>>>>Now I want to push the limits where I need 4KB buffer for each of the > >>>>> > >>>>> > >>>>> > >>>>32000 connections I want this server to handle. If I do the math: > >>>> > >>>> > >>>> > >>>> > >>>>>(32000 (conns) * 4 (KB/buffer) * 2 (buffer/conn)) / 2048 (KB/cluster) > >>>>> > >>>>>= 128000 clusters > >>>>> > >>>>>So I set this arbitrary high value in loader.conf under > >>>>> > >>>>> > >>>>> > >>>>(kern.ipc.nmbclusters) and no surprises I get panic: pmap_enter > >>>>invalid page > >>>>directory pdir=0x3cb063, va=0xfff800 > >>>> > >>>> > >>>> > >>>> > >>>>>(va has a weird address here) > >>>>> > >>>>>I know I am pushing the limits here but I have plenty of memory (2GB) on > >>>>> > >>>>> > >>>>> > >>>>this system (after all its just 250MB for network memory ;) and this is > >>>>mainly just experimentation. > >>>> > >>>> > >>>> > >>>> > >>>>>I would like some pointers toward fixing this. Is there another variable > >>>>> > >>>>> > >>>>> > >>>>tied into this (I guess so)? Could anybody points me to a technical > >>>>document > >>>>that would explain the relationship with that (those) other(s) presumed > >>>>variable(s)? > >>>> > >>>> > >>>> > >>>> > >>>>>Thank you, > >>>>> > >>>>> > >>>>>Karim > >>>>> > >>>>> > >>>>>_______________________________________________ > >>>>>freebsd-net@freebsd.org mailing list > >>>>>http://lists.freebsd.org/mailman/listinfo/freebsd-net > >>>>>To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > >>>>> > >>>>> > >>>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>-- > >>Dave Baukus > >>dbaukus@chiaro.com > >> Chiaro Networks Ltd. > >> Richardson, Texas > >> USA > >> > >>_______________________________________________ > >>freebsd-net@freebsd.org mailing list > >>http://lists.freebsd.org/mailman/listinfo/freebsd-net > >>To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > >> > >> > >> > > > > > > > > > > -- > Karim Fodil-Lemelin > Lead Programmer > > Xiphos Technologies Inc. > (514) 848-9640 x223 > (514) 848-9644 fax > www.xiplink.com > > -------------------------------------------------------------- > The information transmitted is intended only for the > person or entity to which it is addressed and may contain > confidential and/or privileged material. If you have > received this in error, please contact the sender and delete > this communication and any copy immediately. Thank you. > > >