Date: Sat, 1 Aug 2009 11:41:19 -0400 From: Linda Messerschmidt <linda.messerschmidt@gmail.com> To: freebsd-hackers@freebsd.org Subject: Where have all the vnodes gone? Message-ID: <237c27100908010841g5abd3183w479121b2ba7b0c91@mail.gmail.com>
next in thread | raw e-mail | index | archive | help
(Reposted from freebsd-questions due to no replies.) With the last few releases, I've noticed a distinct trend toward disappearing vnodes on one of the machines I look after. This machine isn't doing a whole lot. It runs a couple of small web sites, and once an hour it rsync's some files from one NFS mount to another, but the rsync doesn't stay running; it restarts every hour and runs for 10-15 minutes. I set it to log the number of vnodes every ten minutes and this is what I got: 00:47:59 vfs.numvnodes: 39337 00:57:59 vfs.numvnodes: 40568 01:07:59 vfs.numvnodes: 44554 01:17:59 vfs.numvnodes: 52141 01:27:59 vfs.numvnodes: 55713 01:37:59 vfs.numvnodes: 58643 01:47:59 vfs.numvnodes: 60792 01:57:59 vfs.numvnodes: 67130 02:07:59 vfs.numvnodes: 76035 02:17:59 vfs.numvnodes: 84349 02:27:59 vfs.numvnodes: 92187 02:37:59 vfs.numvnodes: 98114 02:47:59 vfs.numvnodes: 116854 02:57:59 vfs.numvnodes: 124842 03:07:59 vfs.numvnodes: 164173 03:17:59 vfs.numvnodes: 172257 03:27:59 vfs.numvnodes: 178388 03:37:59 vfs.numvnodes: 183066 03:47:59 vfs.numvnodes: 190092 03:57:59 vfs.numvnodes: 198322 04:07:59 vfs.numvnodes: 204598 04:17:59 vfs.numvnodes: 208311 04:27:59 vfs.numvnodes: 214207 04:37:59 vfs.numvnodes: 221028 04:47:59 vfs.numvnodes: 227792 04:57:59 vfs.numvnodes: 233214 05:07:59 vfs.numvnodes: 240112 05:17:59 vfs.numvnodes: 247572 05:27:59 vfs.numvnodes: 256090 05:37:59 vfs.numvnodes: 262720 05:47:59 vfs.numvnodes: 269755 05:57:59 vfs.numvnodes: 274986 06:07:59 vfs.numvnodes: 279879 06:17:59 vfs.numvnodes: 287039 06:27:59 vfs.numvnodes: 291984 06:37:59 vfs.numvnodes: 294267 06:47:59 vfs.numvnodes: 296658 06:57:59 vfs.numvnodes: 299086 07:07:59 vfs.numvnodes: 301825 07:17:59 vfs.numvnodes: 309060 07:27:59 vfs.numvnodes: 312955 07:37:59 vfs.numvnodes: 317400 07:47:59 vfs.numvnodes: 320047 At that point the machine crashed with: panic: kmem_malloc(16384): kmem_map too small: 334745600 total allocated If I don't tune kern.maxvnodes up to the point where it panics, then eventually it runs out of vnodes and all sorts of stuff gets stuck in vlruwk. The machine in question is running 7.2-RELEASE-p3, but I already upgraded it from 7.1 trying to get this to go away, so it's a problem that's been around for awhile. My guess is that they're leaking in the kernel somewhere because of the rsync, because there's just not much else going on, but unless I can figure out how many vnodes are being used on a per-process basis, I can't make any headway on proving or disproving that. I do know that according to fstat, there are only 1000-1500 descriptors open at any given time, and kern.openfiles ranges 250-500. So I'm just mystified what the other 300000+ could be. Is there any way to figure out where all these vnodes are going? TIA for any advice!
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?237c27100908010841g5abd3183w479121b2ba7b0c91>