Date: Sun, 26 Jan 2003 18:09:16 -0800 From: Terry Lambert <tlambert2@mindspring.com> To: Matthew Dillon <dillon@apollo.backplane.com> Cc: Sean Hamilton <sh@bel.bc.ca>, hackers@FreeBSD.ORG Subject: Re: Random disk cache expiry Message-ID: <3E3494CC.5895492D@mindspring.com> References: <000501c2c4dd$f43ed450$16e306cf@slugabed.org> <200301261931.h0QJVCp8052101@apollo.backplane.com> <3E348B51.6F4D6096@mindspring.com> <200301270142.h0R1guR3070182@apollo.backplane.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Matthew Dillon wrote: > :I really dislike the idea of random expiration; I don't understand > :the point, unless you are trying to get better numbers on some > :.. > > Well, the basic scenario is something like this: Lets say you have > 512MB of ram and you are reading a 1GB file sequentially, over and over > again. The cache scenario that gives you the greatest performance is > basically to keep 512MB worth of the file in memory and throw away the > rest as you read it. You don't need random expiration for this; in fact, random expiration will give you 50% worse performance than you expect. If you are doing this, the proper thing is to map the file window you want to keep seperately from the rest of the file, or, if it's small enough you can mmap the whole thing, then to madvise the part of the file you want to not cache as "sequential", and drop the cache. That this is not well supported is a bug in FreeBSD (IMO). > This is effectively what random disk cache expiration gives you (assuming > you expire whole tracks worth of data, not just random pages). i.e. it's > only useful when you are cycling through a data set that is larger then > main memory over and over again. And not very, there. Because you destroy locality of reference -- or, as in this case, you assume that there is no locality in the first place. I think it works at 50% of the efficiency of the madvise that I suggest, above, which, at least, gets hits on most of the file, without the risk of page replacement of the cached area of the file that you intend to read subsequently (25% hit rate in the hittable area, assuming this, since page replacement is random, if the replacement range is 50% of the size of the data set... i.e.: numbers in real life will vary downward, the larger the data set). > Most people deal with this issue simply by beefing up their disk > subsystem. After all, you don't have to stripe very many disks (only > 2 or 3) to get 100MB/sec+ in throughput from the disk subsystem. I think most people don't have this problem, because this is a very degenerate case, which requires either a serindipitously incredibly bad design, or intential screwing of caching, in order to do something like benchmark the disk I/O subsystem seperately from the case (for legitimate cases) or to "prove" that caching is not useful because you have an axe to grind (e.g. the proxy cache "benchmark" suite that does this because the author is a mathematician that has developed a pessimizing hueristic because he is philosophically opposed to proxy caching, due to his own narrow interpretation of "end to end guarantees"). In general, this is not a useful case to try to optimize, and the people who try to optimize this case are doing it to cheat on a benchmark, not because there's any treal-world application that requires it. 8-(. -- Terry To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-hackers" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?3E3494CC.5895492D>