Date: Wed, 1 Sep 2004 15:19:27 -0300 (ADT) From: "Marc G. Fournier" <scrappy@hub.org> To: freebsd-current@freebsd.org Subject: vnode leak in FFS code ... ? Message-ID: <20040901151405.G47186@ganymede.hub.org>
next in thread | raw e-mail | index | archive | help
I don't know if this is applicable to -current as well, but so far,
anything like this I've uncovered in 4.x has needed an equivalent fix in
5.x, so figured it can't hurt to ask, especially with everyone working
towards a STABLE 5.x branch ... I do not have a 5.x machine running this
sort of load at the moment, so can't test, or provide feedback there ...
all my 5.x machines are more or less desktops ...
On Saturday, I'm going to try an unmount of the bigger file system, to see
if it frees everything up without a reboot ... but if someone can suggest
something to check to see if it is a) a leak and b) is fixable between now
and then, please let me know ... again, this is a 4.10 system, but most of
the work that Tor and David have done (re: vnodes) in the past relating to
my servers have been applied to 5.x first, and MFC'd afterwards, so I
suspect that this too many be something that applies to both branches ...
-----------------
I have two servers, both running 4.10 of within a few days (Aug 5 for
venus, Aug 7 for neptune) ... both running jail environments ... one with
~60 running, the other with ~80 ... the one with 60 has been running for
~25 days now, and is at the border of running out of vnodes:
Aug 31 20:58:00 venus root: debug.numvnodes: 519920 - debug.freevnodes: 11058 - debug.vnlru_nowhere: 256463 - vlrup
Aug 31 20:59:01 venus root: debug.numvnodes: 519920 - debug.freevnodes: 13155 - debug.vnlru_nowhere: 256482 - vlrup
Aug 31 21:00:03 venus root: debug.numvnodes: 519920 - debug.freevnodes: 13092 - debug.vnlru_nowhere: 256482 - vlruwt
while the other one has been up for ~1 days, but is using alot less, for
more processes:
Aug 31 20:58:00 neptune root: debug.numvnodes: 344062 - debug.freevnodes: 208655 - debug.vnlru_nowhere: 0 - vlruwt
Aug 31 20:59:00 neptune root: debug.numvnodes: 344062 - debug.freevnodes: 208602 - debug.vnlru_nowhere: 0 - vlruwt
Aug 31 21:00:03 neptune root: debug.numvnodes: 344062 - debug.freevnodes: 208319 - debug.vnlru_nowhere: 0 - vlruwt
I've tried shutting down all of the VMs on venus, and umount'd all of the
unionfs mounts, as well as the one nfs mount we have ... the above #s are
after the VMs (and mounts are recreated ...
Now, my understanding of the vnodes is that for every file opened, a vnode
is created ... in my case, since I'm using unionfs, there are two vnodes
per file ... if it possible that there are 'stale' vnodes that aren't
being freed up? Is there some way of 'viewing' the vnode structure?
For instance, fstat shows:
venus# fstat | wc -l
19531
So, obviously it isn't just open files that I'm dealing with here, for
even if I double that, that is nowhere near 519920 ...
So, where else are the vnodes going? Is there a 'leak'? What can I look
at to try and narrow this down / provide more information?
Even some way of determining a specific process that is sucking back alot
of them, to move that to a different machine ... ?
Looking at vmstat -m .. specifically the work that David did on seperating
the union vs regular vnodes:
UNION mount 60 2K 3K204800K 162 0 0 32
undcac 0 0K 1K204800K343638713 0 0 16
unpath 13146 227K 1025K204800K 43541149 0 0 16,32,64,128
Export Host 1 1K 1K204800K 164 0 0 256
vnodes 141 7K 8K204800K 613 0 0 16,32,64,128,256
Why does 'vnodes' show only 141 InUse? Or, in this case, should I be
looking at:
FFS node496600124150K 127870K204800K401059293 0 0 256
496k FFS nodes, if I'm reading right?
vs neptune, which is showing only:
FFS node300433 75109K 80257K204800K 3875307 0 0 256
Hrmmm, maybe I'm mis-reading all of this, and going down the wrong paths
here, so hopefully someone will correct if I am ... but, for now ...
Looking at vmstat -m a bit further, the top of the report has:
Memory statistics by bucket size
Size In Use Free Requests HighWater Couldfree
16 13116 28356 2063580697 1280 7822
32 77734 7002 168084205 640 316065
64 465006 48402 2804541088 320 637084
128 100182 60010 591859866 160 1850304
256 500029 12163 1178322001 80 123078
Now, the only things that are using alot of the '256 Size' memory are:
FFS node494513123629K 127870K204800K401104542 0 0 256
vfscache449709 29178K 32434K204800K737673766 0 0 64,128,256,512K
Since only 500029 are 'InUse', and since FFS node is exclusively 256 ...
I'm going to guess that most of vfscache is using something else ... so,
my question becomes if 123000 'Could be Freed', why aren't they?
Assuming, of course, I'm not on the wrong trail here :(
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20040901151405.G47186>