Date: Mon, 19 Apr 1999 21:51:09 +0200 (CEST) From: Arjan de Vet <Arjan.deVet@adv.iae.nl> To: hackers@freebsd.org Subject: Re: Directories not VMIO cached at all! Message-ID: <199904191951.VAA05105@adv.iae.nl> In-Reply-To: <199904171844.LAA75452@apollo.backplane.com>
next in thread | previous in thread | raw e-mail | index | archive | help
In article <199904171844.LAA75452@apollo.backplane.com> Matt Dillon
writes:
> I've been playing with my new large-memory configured box and,
> especially, looking at disk I/O.
I've been doing this with a 640MB Squid server too the last weeks.
> When I scan enough directories ( e.g. a megabyte worth of directories
> on a 1 GB machine), then scan again, the data is re-fetched from disk.
I've been tuning my machine to be able to cache at least 32MB worth of
directories, on which I which I mailed on April 6 already.
> [...]
>
> Right now, the buffer cache appears to limit itself to 8 MBytes or so,
> and the maximum malloc space limits itself to only 400K! Rather absurdly
> small for a directory cache, I think, yet I also believe that increasing
> the size of the buffer cache may be detrimental due to the amount of I/O
> it can bind up.
I did some tests as explained below and my results (on a 3.1-stable
machine, I don't now how much different -current already is) seems to
indicate that directories are not limited to the malloc space only. I
reran the test after applying your patch and the results are indeed
different and as expected.
I'm still doing some performance testing with Squid and I'll try to
report on it later; Squid's usage of the filesystem and I/O behavior
under heavy load is quite absurd and a good test-case for these kind of
things I think.
Arjan
-----------------------------------------------------------------------------
- 3.1 stable April 6 + big-KVM patches, 640MB RAM
- /cache contains 4096 Squid directories (64 dirs with 64 subdirs each)
with 384 files per directory (maximum, so dirs stay <8K). Total size
of the directories is 22MB (that's 5632 bytes/directory on average).
- Buffer cache size limited to 80MB instead of 8MB by patching
machdep.c:
diff -u -w -r1.322.2.4 machdep.c
--- machdep.c 1999/02/17 13:08:41 1.322.2.4
+++ machdep.c 1999/04/09 08:23:31
@@ -369,7 +369,7 @@
if (nbuf == 0) {
nbuf = 30;
if( physmem > 1024)
- nbuf += min((physmem - 1024) / 8, 2048);
+ nbuf += min((physmem - 1024) / 8, 20480);
}
nswbuf = max(min(nbuf/4, 64), 16);
- in rc.local:
tmp=`sysctl -n vfs.maxvmiobufspace`
sysctl -w vfs.maxvmiobufspace=`expr $tmp / 4`
to favor metadata in the buffer cache as suggested by John Dyson some
time ago.
- After a fresh reboot:
25544 wire
3740 act
3552 inact
0 cache
610048 free
7702 buf
vfs.maxbufspace: 83251200
vfs.bufspace: 7875584
vfs.maxvmiobufspace: 13875200
vfs.vmiospace: 7613440
vfs.maxmallocbufspace: 4162560
vfs.bufmallocspace: 57344
- Read all directories:
[/cache] > time ls -R > /dev/null
5.622u 2.398s 0:46.45 17.2% 210+400k 4975+0io 6pf+0w
57756 wire
3768 act
3688 inact
0 cache
577672 free
37838 buf
vfs.maxbufspace: 83251200
vfs.bufspace: 38746112
vfs.maxvmiobufspace: 13875200
vfs.vmiospace: 14290944
vfs.maxmallocbufspace: 4162560
vfs.bufmallocspace: 1472512
bufspace has increased by 30MB, vmiospace has increased by 7MB (I'm
wondering what data is in it...) and bufmallocspace has increased by
1.4MB. There was no other activity on the system so all this should be
due to reading the directories I guess. Note that 22MB (size of all
directories) plus that strange 7MB vmio data is close to the 30MB
increase in bufspace...
- Check whether they're really cached now:
[/cache] > time ls -R > /dev/null
5.658u 1.147s 0:06.94 97.8% 208+396k 0+0io 0pf+0w
OK, zero I/O.
- Now add the -l option so all files needs to be stat()-ed too:
[/cache] > time ls -lR > /dev/null
40.124u 54.509s 2:46.99 56.6% 209+464k 12370+0io 0pf+0w
99140 wire
3948 act
61376 inact
0 cache
478420 free
81302 buf
vfs.maxbufspace: 83251200
vfs.bufspace: 83252224
vfs.maxvmiobufspace: 13875200
vfs.vmiospace: 58797056
vfs.maxmallocbufspace: 4162560
vfs.bufmallocspace: 1472512
bufspace has reached its maximum value and 58MB of pages have been
moved from the buffer cache to the VM inact queue. vmiospace has
increased by 44MB, bufmallocspace stayed the same. The amount of
non-vmio and non-malloc space in the buffer cache is now 22MB... which
is the size of all directories. Coincidence or not? If not, John
Dyson's suggestion about vfs.maxvmiobufspace seems to work.
- Check whether everything is really cached:
[/cache] > time ls -lR > /dev/null
40.045u 54.867s 1:37.09 97.7% 208+463k 0+0io 0pf+0w
OK, zero I/O again.
- Applied Matt's patches, installed the new kernel, removed the
vfs.maxvmiobufspace hack from rc.local, and rebooted:
26168 wire
3964 act
3696 inact
0 cache
609048 free
8411 buf
vfs.maxbufspace: 83251200
vfs.bufspace: 8621056
vfs.maxvmiobufspace: 55500800
vfs.vmiospace: 8376320
vfs.maxmallocbufspace: 4162560
vfs.bufmallocspace: 44032
- [/cache] > time ls -R > /dev/null
5.572u 2.600s 0:46.82 17.4% 213+405k 4975+0io 6pf+0w
63076 wire
3852 act
3832 inact
0 cache
572112 free
38513 buf
vfs.maxbufspace: 83251200
vfs.bufspace: 39437312
vfs.maxvmiobufspace: 55500800
vfs.vmiospace: 39192576
vfs.maxmallocbufspace: 4162560
vfs.bufmallocspace: 44032
Indeed, bufmallocspace did not increase and bufspace has increased by
a slightly higher amount than in the old case because of this.
vmiospace clearly differs, 39MB instead of 14MB. So all directory data
seems indeed to be VMIO'ed now.
- [/cache] > time ls -R > /dev/null
5.535u 1.203s 0:07.02 95.8% 211+402k 0+0io 0pf+0w
- [/cache] > time ls -lR > /dev/null
40.492u 57.054s 2:51.16 56.9% 207+460k 12370+0io 0pf+0w
103744 wire
3976 act
62256 inact
0 cache
472900 free
81304 buf
vfs.maxbufspace: 83251200
vfs.bufspace: 83246080
vfs.maxvmiobufspace: 55500800
vfs.vmiospace: 83000320
vfs.maxmallocbufspace: 4162560
vfs.bufmallocspace: 45056
To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-hackers" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?199904191951.VAA05105>