Date: Sat, 20 Jan 2007 20:57:03 -0500 From: John Capo <jc@irbs.com> To: freebsd-questions@freebsd.org Subject: PV entry limit Message-ID: <20070121015703.GA67097@exuma.irbs.com>
next in thread | raw e-mail | index | archive | help
I got the infamous pmap_collect: collecting pv entries message 5
times within 4 hours 12 days ago. The machine is a Cyrus IMAP
server with 4Gigs of memory peaking around 2000 IMAP processes and
a about 200 other processes. The machine is running 4.11 with these
compile tweaks.
options PMAP_SHPGPERPROC=300
options KVA_PAGES=384
and one sysctl boot tweak.
kern.maxfiles=60000
Open files are 30K or so with about 200MB of shared files mmapped
into each IMAP process.
sysctl vm | grep PV every 10 seconds shows this.
PV ENTRY: 28, 3749470, 2501404, 18961, 296551065
PV ENTRY: 28, 3749470, 2501472, 18893, 296563037
PV ENTRY: 28, 3749470, 2513275, 7090, 296601432
PV ENTRY: 28, 3749470, 151597, 2368768, 296689650
PV ENTRY: 28, 3749470, 211052, 2309313, 296783099
PV ENTRY: 28, 3749470, 283356, 2237009, 296896244
Used plus free PV entries is 2520365 which is considerably less
that the 3749470 PV entry limit.
AFAICT, I have plenty of kernel space available to allocate more
PV entries from.
ITEM SIZE LIMIT USED FREE REQUESTS
PIPE: 160, 0, 522, 294, 22806126
SWAPMETA: 160, 233016, 14858, 29455, 3964171
unpcb: 160, 0, 2659, 4541, 15805192
ripcb: 192, 12328, 2, 40, 31222
syncache: 160, 15359, 0, 76, 8077824
tcpcb: 576, 12328, 1391, 2889, 10030866
udpcb: 192, 12328, 33, 73, 11663049
socket: 224, 12328, 4086, 6848, 37530493
DIRHASH: 1024, 0, 1684, 292, 8344073
KNOTE: 64, 0, 2, 126, 9094626
NFSNODE: 352, 0, 6, 14118, 105689
NFSMOUNT: 544, 0, 3, 11, 6
VNODE: 192, 0, 216941, 41, 216941
NAMEI: 1024, 0, 1, 255, 3156465410
VMSPACE: 192, 0, 1352, 2168, 9106225
PROC: 416, 0, 1403, 2223, 15245401
DP fakepg: 64, 0, 0, 0, 0
PV ENTRY: 28, 3749470, 2263440, 256925, 801513627
MAP ENTRY: 48, 0, 64374, 99039, 862331760
KMAP ENTRY: 48, 73807, 8234, 256, 30468915
MAP: 108, 0, 7, 3, 7
VM OBJECT: 92, 0, 215656, 64792, 403861231
vm.zone_kmem_pages: 13773
vm.zone_kmem_kvaspace: 136269824
vm.zone_kern_pages: 21225
vm.kvm_size: 1606414336
vm.kvm_free: 494923776
Obviously I need to bump PMAP_SHPGPERPROC some more.
Do I need more KVA also?
If there is enough KVA now, what kept more PV entries from being
allocated when needed? Used plus free PV entries has been stuck
at 2520365 for days.
I know 4.11 is EOL but switching to [56].something is just not an
option right now.
Thanks,
John
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20070121015703.GA67097>