Date: Fri, 16 Jul 2010 18:33:06 -0500 From: Alan Cox <alc@cs.rice.edu> To: Peter Jeremy <peterjeremy@acm.org> Cc: alc@freebsd.org, freebsd-hackers@freebsd.org Subject: Re: disk I/O, VFS hirunningspace Message-ID: <4C40EC32.3030700@cs.rice.edu> In-Reply-To: <20100716093041.GB26367@server.vk2pj.dyndns.org> References: <AANLkTinm3kFm7pF_cxoNz1Cgyd5UvnmgZzCpbjak-zzy@mail.gmail.com> <20100714090454.1177b96b@ernst.jennejohn.org> <AANLkTinhqQUYMqrtV9l0SKwi2qFmx8t88jBYRmdiU10M@mail.gmail.com> <i1n0ra$ahs$1@dough.gmane.org> <AANLkTiniWXgEsrtEy3ZaL2_rQLiIMwI_Lt21EJCzqJZu@mail.gmail.com> <20100716093041.GB26367@server.vk2pj.dyndns.org>
next in thread | previous in thread | raw e-mail | index | archive | help
Peter Jeremy wrote: > Regarding vfs.lorunningspace and vfs.hirunningspace... > > On 2010-Jul-15 13:52:43 -0500, Alan Cox<alan.l.cox@gmail.com> wrote: > >> Keep in mind that we still run on some fairly small systems with limited I/O >> capabilities, e.g., a typical arm platform. More generally, with the range >> of systems that FreeBSD runs on today, any particular choice of constants is >> going to perform poorly for someone. If nothing else, making these sysctls >> a function of the buffer cache size is probably better than any particular >> constants. >> > > That sounds reasonable but brings up a related issue - the buffer > cache. Given the unified VM system no longer needs a traditional Unix > buffer cache, what is the buffer cache still used for? Today, it is essentially a mapping cache. So, what does that mean? After you've set aside a modest amount of physical memory for the kernel to hold its own internal data structures, all of the remaining physical memory can potentially be used to cache file data. However, on many architectures this is far more memory than the kernel can instantaneously access. Consider i386. You might have 4+ GB of physical memory, but the kernel address space is (by default) only 1 GB. So, at any instant in time, only a fraction of the physical memory is instantaneously accessible to the kernel. In general, to access an arbitrary physical page, the kernel is going to have to replace an existing virtual-to-physical mapping in its address space with one for the desired page. (Generally speaking, on most architectures, even the kernel can't directly access physical memory that isn't mapped by a virtual address.) The buffer cache is essentially a region of the kernel address space that is dedicated to mappings to physical pages containing cached file data. As applications access files, the kernel dynamically maps (and unmaps) physical pages containing cached file data into this region. Once the desired pages are mapped, then read(2) and write(2) can essentially "bcopy" from the buffer cache mapping to the application's buffer. (Understand that this buffer cache mapping is a prerequisite for the copy out to occur.) So, why did I call it a mapping cache? There is generally locality in the access to file data. So, rather than map and unmap the desired physical pages on every read and write, the mappings to file data are allowed to persist and are managed much like many other kinds of caches. When the kernel needs to map a new set of file pages, it finds an older, not-so-recently used mapping and destroys it, allowing those kernel virtual addresses to be remapped to the new pages. So far, I've used i386 as a motivating example. What of other architectures? Most 64-bit machines take advantage of their large address space by implementing some form of "direct map" that provides instantaneous access to all of physical memory. (Again, I use "instantaneous" to mean that the kernel doesn't have to dynamically create a virtual-to-physical mapping before being able to access the data.) On these machines, you could, in principle, use the direct map to implement the "bcopy" to the application's buffer. So, what is the point of the buffer cache on these machines? A trivial benefit is that the file pages are mapped contiguously in the buffer cache. Even though the underlying physical pages may be scattered throughout the physical address space, they are mapped contiguously. So, the "bcopy" doesn't need to worry about every page boundary, only buffer boundaries. The buffer cache also plays a role in the page replacement mechanism. Once mapped into the buffer cache, a page is "wired", that is, it removed from the paging lists, where the page daemon could reclaim it. However, a page in the buffer cache should really be thought of as being "active". In fact, when a page is unmapped from the buffer cache, it is placed at the tail of the virtual memory system's "inactive" list. The same place where the virtual memory system would place a physical page that it is transitioning from "active" to "inactive". If an application later performs a read(2) from or write(2) to the same page, that page will be removed from the "inactive" list and mapped back into the buffer cache. So, the mapping and unmapping process contributes to creating an LRU-ordered "inactive" queue. Finally, the buffer cache limits the amount of dirty file system data that is cached in memory. > ... Is the current > tuning formula still reasonable (for virtually all current systems > it's basically 10MB + 10% RAM)? It's probably still good enough. However, this is not a statement for which I have supporting data. So, I reserve the right to change my opinion. :-) Consider what the buffer cache now does. It's just a mapping cache. Increasing the buffer cache size doesn't affect (much) the amount of physical memory available for caching file data. So, unlike ancient times, increasing the size of the buffer cache isn't going to have nearly the same effect on the amount of actual I/O that your machine does. For some workloads, increasing the buffer cache size may have greater impact on CPU overhead than I/O overhead. For example, all of your file data might fit into physical memory, but you're doing random read accesses to it. That would cause the buffer cache to thrash, even though you wouldn't do any actual I/O. Unfortunately, mapping pages into the buffer cache isn't trivial. For example, it requires every processor to be interrupted to invalidate some entries from its TLB. (This is a so-called "TLB shootdown".) > ... How can I measure the effectiveness > of the buffer cache? > > I'm not sure that I can give you a short answer to this question. > The buffer cache size is also very tightly constrained (vfs.hibufspace > and vfs.lobufspace differ by 64KB) and at least one of the underlying > tuning parameters have comments at variance with current reality: > In<sys/param.h>: > > * MAXBSIZE - Filesystems are made out of blocks of at most MAXBSIZE bytes > * per block. MAXBSIZE may be made larger without effecting > ... > * > * BKVASIZE - Nominal buffer space per buffer, in bytes. BKVASIZE is the > ... > * The default is 16384, roughly 2x the block size used by a > * normal UFS filesystem. > */ > #define MAXBSIZE 65536 /* must be power of 2 */ > #define BKVASIZE 16384 /* must be power of 2 */ > > There's no mention of the 64KiB limit in newfs(8) and I recall seeing > occasional comments from people who have either tried or suggested > trying larger blocksizes. I believe that larger than 64KB would fail an assertion. > Likewise, the default UFS blocksize has > been 16KiB for quite a while. Are the comments still valid and, if so, > should BKVASIZE be doubled to 32768 and a suitable note added to newfs(8) > regarding the maximum block size? > > If I recall correctly, increasing BKVASIZE would only reduce the number buffer headers. In other words, it might avoid wasting some memory on buffer headers that won't be used. Alan
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?4C40EC32.3030700>