From owner-freebsd-questions@FreeBSD.ORG Sun Jan 21 02:15:29 2007 Return-Path: X-Original-To: freebsd-questions@freebsd.org Delivered-To: freebsd-questions@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 5FD1916A408 for ; Sun, 21 Jan 2007 02:15:29 +0000 (UTC) (envelope-from jc@irbs.com) Received: from turtle-out.mxes.net (turtle-out.mxes.net [216.86.168.191]) by mx1.freebsd.org (Postfix) with ESMTP id 36C2213C442 for ; Sun, 21 Jan 2007 02:15:29 +0000 (UTC) (envelope-from jc@irbs.com) Received: from mxout-04.mxes.net (mxout-04.mxes.net [216.86.168.179]) by turtle-in.mxes.net (Postfix) with ESMTP id 696B8105A6 for ; Sat, 20 Jan 2007 20:57:06 -0500 (EST) Received: from exuma.irbs.com (exuma.irbs.com [216.86.160.225]) by mxout-04.mxes.net (Postfix) with ESMTP id 59AAAA32A4 for ; Sat, 20 Jan 2007 20:57:03 -0500 (EST) Received: by exuma.irbs.com (Postfix, from userid 2500) id 8C5D1CD7C; Sat, 20 Jan 2007 20:57:03 -0500 (EST) Date: Sat, 20 Jan 2007 20:57:03 -0500 From: John Capo To: freebsd-questions@freebsd.org Message-ID: <20070121015703.GA67097@exuma.irbs.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.4.2.2i Subject: PV entry limit X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: jc@irbs.com List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 21 Jan 2007 02:15:29 -0000 I got the infamous pmap_collect: collecting pv entries message 5 times within 4 hours 12 days ago. The machine is a Cyrus IMAP server with 4Gigs of memory peaking around 2000 IMAP processes and a about 200 other processes. The machine is running 4.11 with these compile tweaks. options PMAP_SHPGPERPROC=300 options KVA_PAGES=384 and one sysctl boot tweak. kern.maxfiles=60000 Open files are 30K or so with about 200MB of shared files mmapped into each IMAP process. sysctl vm | grep PV every 10 seconds shows this. PV ENTRY: 28, 3749470, 2501404, 18961, 296551065 PV ENTRY: 28, 3749470, 2501472, 18893, 296563037 PV ENTRY: 28, 3749470, 2513275, 7090, 296601432 PV ENTRY: 28, 3749470, 151597, 2368768, 296689650 PV ENTRY: 28, 3749470, 211052, 2309313, 296783099 PV ENTRY: 28, 3749470, 283356, 2237009, 296896244 Used plus free PV entries is 2520365 which is considerably less that the 3749470 PV entry limit. AFAICT, I have plenty of kernel space available to allocate more PV entries from. ITEM SIZE LIMIT USED FREE REQUESTS PIPE: 160, 0, 522, 294, 22806126 SWAPMETA: 160, 233016, 14858, 29455, 3964171 unpcb: 160, 0, 2659, 4541, 15805192 ripcb: 192, 12328, 2, 40, 31222 syncache: 160, 15359, 0, 76, 8077824 tcpcb: 576, 12328, 1391, 2889, 10030866 udpcb: 192, 12328, 33, 73, 11663049 socket: 224, 12328, 4086, 6848, 37530493 DIRHASH: 1024, 0, 1684, 292, 8344073 KNOTE: 64, 0, 2, 126, 9094626 NFSNODE: 352, 0, 6, 14118, 105689 NFSMOUNT: 544, 0, 3, 11, 6 VNODE: 192, 0, 216941, 41, 216941 NAMEI: 1024, 0, 1, 255, 3156465410 VMSPACE: 192, 0, 1352, 2168, 9106225 PROC: 416, 0, 1403, 2223, 15245401 DP fakepg: 64, 0, 0, 0, 0 PV ENTRY: 28, 3749470, 2263440, 256925, 801513627 MAP ENTRY: 48, 0, 64374, 99039, 862331760 KMAP ENTRY: 48, 73807, 8234, 256, 30468915 MAP: 108, 0, 7, 3, 7 VM OBJECT: 92, 0, 215656, 64792, 403861231 vm.zone_kmem_pages: 13773 vm.zone_kmem_kvaspace: 136269824 vm.zone_kern_pages: 21225 vm.kvm_size: 1606414336 vm.kvm_free: 494923776 Obviously I need to bump PMAP_SHPGPERPROC some more. Do I need more KVA also? If there is enough KVA now, what kept more PV entries from being allocated when needed? Used plus free PV entries has been stuck at 2520365 for days. I know 4.11 is EOL but switching to [56].something is just not an option right now. Thanks, John