Date: Tue, 6 Apr 1999 18:51:26 +0200 From: Arjan de Vet <Arjan.deVet@adv.iae.nl> To: hackers@freebsd.org Subject: Increasing both NBUF and NMBCLUSTERS leads to panics Message-ID: <199904061651.SAA11984@adv.iae.nl>
next in thread | raw e-mail | index | archive | help
Hi,
I'm testing a new 3.1-stable server with 640MB RAM for running Squid and
I would like to increase the NBUF parameter in the kernel configuration
file. The default maximum size of the buffercache seems to be around
8MB irrespective of the amount of physical memory (see nbuf calculation
code in i386/i386/machdep.c).
For this Squid server I would like to have at least 32MB buffer cache or
more because the 2-level directory hierarchy used by Squid contains in
my setup 4096 directories whose size is <=8KB and I would like to cache
all metadata to speedup the open() calls which are becoming a bottleneck
on my current Squid server on BSD/OS 3.0.
If I understand correctly, metadata only gets cached in the buffer cache
whereas other file data can be cached by the VM system too. Some time
ago John Dyson also posted some sysctl settings for keeping metadata in
the buffer cache as much as possible; I was planning on using those too.
Besides increasing the NBUF parameter I also want to increase
NMBCLUSTERS of course because the machine will handle quite some network
traffic. But for some reason increasing both NBUF and NMBCLUSTERS only
leads to panics :-(.
Below you will find various combinations of NBUF and NMBCLUSTERS
settings together with some attempts of VM_KMEM_SIZE_MAX tuning.
I added some printf's in the kernel sources to print the sizes of the
maps being allocated and to determine which map allocation causes the
panic. If I understand correctly the maps have the following relation:
kernel_map
|
|-kmem_map (npg * PAGE_SIZE)
| | = (nmbufs * MSIZE + nmbclusters * MCLBYTES + vm_kmem_size)
| |
| |-mb_map nmbufs * MSIZE + nmbclusters * MCLBYTES
|
|-clean_map (nbuf*BKVASIZE) + (nswbuf*MAXPHYS) + pager_map_size
| |
| |-buffer_map (nbuf*BKVASIZE)
| |
| |-pager_map (nswbuf*MAXPHYS) + pager_map_size
|
|-exec_map 16*(ARG_MAX+(PAGE_SIZE*3))
The buffercache and nmbclusters are allocated in different maps so I
suspect the problem to be in their common parent map kernel_map.
Any explanations or hints for avoiding these panics are greatly
appreciated.
Arjan
=============================================================================
Version : FreeBSD 3.1-stable, cvsup-ped April 6, 1999
Hardware: Pentium Pro-200, 640MB RAM
-----------------------------------------------------------------------------
Increase NBUF:
maxusers 128
options "NBUF=10240"
#options "NMBCLUSTERS=8192"
#options "VM_KMEM_SIZE_MAX=(256*1024*1024)"
This works OK:
initial vm_kmem_size 12582912
vm_kmem_size after memory tuning 219460949
vm_kmem_size after VM_KMEM_SIZE_MAX 83886080
final vm_kmem_size 83886080
kmem_map 90439680
kmem_map done
real memory = 671088640 (655360K bytes)
clean_map 100663296
clean_map done
buffer_map 83886080
buffer_map done
pager_map 16777216
pager_map done
exec_map 1245184
exec_map done
mb_map 6553600
mb_map done
avail memory = 647626752 (632448K bytes)
-----------------------------------------------------------------------------
Now increase NMBCLUSTERS at the same time:
maxusers 128
options "NBUF=10240"
options "NMBCLUSTERS=8192"
#options "VM_KMEM_SIZE_MAX=(256*1024*1024)"
initial vm_kmem_size 12582912
vm_kmem_size after memory tuning 219460949
vm_kmem_size after VM_KMEM_SIZE_MAX 83886080
final vm_kmem_size 83886080
kmem_map 104857600
kmem_map done
real memory = 671088640 (655360K bytes)
clean_map 100663296
clean_map done
buffer_map 83886080
buffer_map done
pager_map 16777216
pager_map done
exec_map 1245184
exec_map done
mb_map 20971520
mb_map done
avail memory = 647589888 (632412K bytes)
Now it panics in some in_pcballoc code:
Doing additional network setup: xntpd portmap
Fatal trap 12: page fault while in kernel mode
fault virtual address = 0x8
fault code = supervisor read, page not present
instruction pointer = 0x8:0xf01c9e13
stack pointer = 0x10:0xffb70eac
frame pointer = 0x10:0xffb70eb4
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, def32 1, gran 1
processor eflags = interrupt enabled, resume, IOPL = 0
current process = 117 (portmap)
interrupt mask = net tty bio cam
kernel: type 12 trap, code=0
Stopped at zalloci+0xf: movl 0x8(%ebx),%eax
db> t
zalloci(0,ff871c80,ff871c80,80000000,ffb70eec) at zalloci+0xf
in_pcballoc(ff871c80,f024a018,ffb39360) at in_pcballoc+0x14
tcp_attach(ff871c80,ffb39360,ff871c80,f02251c8,1) at tcp_attach+0x3c
tcp_usr_attach(ff871c80,6,ffb39360,0,ffb70f94) at tcp_usr_attach+0x29
socreate(2,ffb70f54,1,6,ffb39360) at socreate+0xc7
socket(ffb39360,ffb70f94,804d000,1,10) at socket+0x54
syscall(27,27,10,1,efbfdddc) at syscall+0x187
Xint0x80_syscall() at Xint0x80_syscall+0x2c
db>
-----------------------------------------------------------------------------
same kernel, but now from the loader:
set kern.vm.kmem.size=268435456 # 256*1024*1024
Then it panics when creating kmem_map:
initial vm_kmem_size 12582912
vm_kmem_size after memory tuning 219460949
vm_kmem_size after VM_KMEM_SIZE_MAX 83886080
final vm_kmem_size 268435456
kmem_map 289406976
kmem_suballoc: bad status return of 3.
panic: kmem_suballoc
Debugger("panic")
Stopped at 0xf01d87e7: movl $0,0xf022a4c8
db> t
(null)(f0202bd7) at 0xf01d87e7
(null)(f020cb95,f020cb6c,3,11400000,f022f454) at 0xf014d3b4
(null)(f0239eac,f02326d4,f02326d8,11400000,f020248b) at 0xf01be80a
(null)(0) at 0xf014a2f2
(null)(f0299fb8,f0403000,2000,2000,f01d8ea0) at 0xf013fdde
(null)() at 0xf0116f3b
db>
f014d340 T panic
f01be7b8 T kmem_suballoc
f014a1c8 t kmeminit
-----------------------------------------------------------------------------
Increase VM_KMEM_SIZE_MAX in kernel config instead of using the boot
loader 'set kern.vm.kmem.size' option:
maxusers 128
options "NBUF=10240"
options "NMBCLUSTERS=8192"
options "VM_KMEM_SIZE_MAX=(256*1024*1024)"
Then it panics in cpu_startup:
initial vm_kmem_size 12582912
vm_kmem_size after memory tuning 219460949
vm_kmem_size after VM_KMEM_SIZE_MAX 219460949
final vm_kmem_size 219460949
kmem_map 240431104
kmem_map done
real memory = 671088640 (655360K bytes)
panic: startup: no room for tables
Debugger("panic")
Stopped at 0xf01d87e7: movl $0,0xf022a4c8
db> t
(null)(f0202bd7) at 0xf01d87e7
(null)(f0211d44,f0226dc0,f022f564,29a000,f0226dc0) at 0xf014d3b4
(null)(0) at 0xf01dc91a
(null)(f0299fb8,f0403000,2000,2000,f01d8ea0) at 0xf013fdde
(null)() at 0xf0116f3b
db>
f014d340 T panic
f01dc644 t cpu_startup
-----------------------------------------------------------------------------
Only increase NMBCLUSTERS:
maxusers 128
#options "NBUF=10240"
options "NMBCLUSTERS=8192"
#options "VM_KMEM_SIZE_MAX=(256*1024*1024)"
OK:
initial vm_kmem_size 12582912
vm_kmem_size after memory tuning 219460949
vm_kmem_size after VM_KMEM_SIZE_MAX 83886080
final vm_kmem_size 83886080
kmem_map 104857600
kmem_map done
real memory = 671088640 (655360K bytes)
clean_map 33800192
clean_map done
buffer_map 17022976
buffer_map done
pager_map 16777216
pager_map done
exec_map 1245184
exec_map done
mb_map 20971520
mb_map done
avail memory = 650076160 (634840K bytes)
-----------------------------------------------------------------------------
Increase VM_KMEM_SIZE_MAX in kernel config:
maxusers 128
#options "NBUF=10240"
options "NMBCLUSTERS=8192"
options "VM_KMEM_SIZE_MAX=(256*1024*1024)"
Then it panics when allocating the clean_map:
initial vm_kmem_size 12582912
vm_kmem_size after memory tuning 219460949
vm_kmem_size after VM_KMEM_SIZE_MAX 219460949
final vm_kmem_size 219460949
kmem_map 240431104
kmem_map done
real memory = 671088640 (655360K bytes)
clean_map 33800192
kmem_suballoc: bad status return of 3.
panic: kmem_suballoc
Debugger("panic")
Stopped at 0xf01d87e7: movl $0,0xf022a4c8
db> t
(null)(f0202bd7) at 0xf01d87e7
(null)(f020cb95,f020cb6c,3,f0226dc0,d0768) at 0xf014d3b4
(null)(f0239eac,f024ccdc,f024d200,203c000,f0211d82) at 0xf01be80a
(null)(0) at 0xf01dc97f
(null)(f0299fb8,f0403000,2000,2000,f01d8ea0) at 0xf013fdde
(null)() at 0xf0116f3b
db>
-----------------------------------------------------------------------------
To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-hackers" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?199904061651.SAA11984>