Skip site navigation (1)Skip section navigation (2)
Date:      Fri, 16 Jul 2010 18:33:06 -0500
From:      Alan Cox <alc@cs.rice.edu>
To:        Peter Jeremy <peterjeremy@acm.org>
Cc:        alc@freebsd.org, freebsd-hackers@freebsd.org
Subject:   Re: disk I/O, VFS hirunningspace
Message-ID:  <4C40EC32.3030700@cs.rice.edu>
In-Reply-To: <20100716093041.GB26367@server.vk2pj.dyndns.org>
References:  <AANLkTinm3kFm7pF_cxoNz1Cgyd5UvnmgZzCpbjak-zzy@mail.gmail.com> <20100714090454.1177b96b@ernst.jennejohn.org> <AANLkTinhqQUYMqrtV9l0SKwi2qFmx8t88jBYRmdiU10M@mail.gmail.com> <i1n0ra$ahs$1@dough.gmane.org> <AANLkTiniWXgEsrtEy3ZaL2_rQLiIMwI_Lt21EJCzqJZu@mail.gmail.com> <20100716093041.GB26367@server.vk2pj.dyndns.org>

next in thread | previous in thread | raw e-mail | index | archive | help
Peter Jeremy wrote:
> Regarding vfs.lorunningspace and vfs.hirunningspace...
>
> On 2010-Jul-15 13:52:43 -0500, Alan Cox<alan.l.cox@gmail.com>  wrote:
>    
>> Keep in mind that we still run on some fairly small systems with limited I/O
>> capabilities, e.g., a typical arm platform.  More generally, with the range
>> of systems that FreeBSD runs on today, any particular choice of constants is
>> going to perform poorly for someone.  If nothing else, making these sysctls
>> a function of the buffer cache size is probably better than any particular
>> constants.
>>      
>
> That sounds reasonable but brings up a related issue - the buffer
> cache.  Given the unified VM system no longer needs a traditional Unix
> buffer cache, what is the buffer cache still used for?

Today, it is essentially a mapping cache.  So, what does that mean?

After you've set aside a modest amount of physical memory for the kernel 
to hold its own internal data structures, all of the remaining physical 
memory can potentially be used to cache file data.  However, on many 
architectures this is far more memory than the kernel can 
instantaneously access.  Consider i386.  You might have 4+ GB of 
physical memory, but the kernel address space is (by default) only 1 
GB.  So, at any instant in time, only a fraction of the physical memory 
is instantaneously accessible to the kernel.  In general, to access an 
arbitrary physical page, the kernel is going to have to replace an 
existing virtual-to-physical mapping in its address space with one for 
the desired page.  (Generally speaking, on most architectures, even the 
kernel can't directly access physical memory that isn't mapped by a 
virtual address.)

The buffer cache is essentially a region of the kernel address space 
that is dedicated to mappings to physical pages containing cached file 
data.  As applications access files, the kernel dynamically maps (and 
unmaps) physical pages containing cached file data into this region.  
Once the desired pages are mapped, then read(2) and write(2) can 
essentially "bcopy" from the buffer cache mapping to the application's 
buffer.  (Understand that this buffer cache mapping is a prerequisite 
for the copy out to occur.)

So, why did I call it a mapping cache?  There is generally locality in 
the access to file data.  So, rather than map and unmap the desired 
physical pages on every read and write, the mappings to file data are 
allowed to persist and are managed much like many other kinds of 
caches.  When the kernel needs to map a new set of file pages, it finds 
an older, not-so-recently used mapping and destroys it, allowing those 
kernel virtual addresses to be remapped to the new pages.

So far, I've used i386 as a motivating example.  What of other 
architectures?  Most 64-bit machines take advantage of their large 
address space by implementing some form of "direct map" that provides 
instantaneous access to all of physical memory.  (Again, I use 
"instantaneous" to mean that the kernel doesn't have to dynamically 
create a virtual-to-physical mapping before being able to access the 
data.)  On these machines, you could, in principle, use the direct map 
to implement the "bcopy" to the application's buffer.  So, what is the 
point of the buffer cache on these machines?

A trivial benefit is that the file pages are mapped contiguously in the 
buffer cache.  Even though the underlying physical pages may be 
scattered throughout the physical address space, they are mapped 
contiguously.  So, the "bcopy" doesn't need to worry about every page 
boundary, only buffer boundaries.

The buffer cache also plays a role in the page replacement mechanism.  
Once mapped into the buffer cache, a page is "wired", that is, it 
removed from the paging lists, where the page daemon could reclaim it.  
However, a page in the buffer cache should really be thought of as being 
"active".  In fact, when a page is unmapped from the buffer cache, it is 
placed at the tail of the virtual memory system's "inactive" list.  The 
same place where the virtual memory system would place a physical page 
that it is transitioning from "active" to "inactive".  If an application 
later performs a read(2) from or write(2) to the same page, that page 
will be removed from the "inactive" list and mapped back into the buffer 
cache.  So, the mapping and unmapping process contributes to creating an 
LRU-ordered "inactive" queue.

Finally, the buffer cache limits the amount of dirty file system data 
that is cached in memory.

> ...  Is the current
> tuning formula still reasonable (for virtually all current systems
> it's basically 10MB + 10% RAM)?

It's probably still good enough.  However, this is not a statement for 
which I have supporting data.  So, I reserve the right to change my 
opinion.  :-)

Consider what the buffer cache now does.  It's just a mapping cache.  
Increasing the buffer cache size doesn't affect (much) the amount of 
physical memory available for caching file data.  So, unlike ancient 
times, increasing the size of the buffer cache isn't going to have 
nearly the same effect on the amount of actual I/O that your machine 
does.  For some workloads, increasing the buffer cache size may have 
greater impact on CPU overhead than I/O overhead.  For example, all of 
your file data might fit into physical memory, but you're doing random 
read accesses to it.  That would cause the buffer cache to thrash, even 
though you wouldn't do any actual I/O.  Unfortunately, mapping pages 
into the buffer cache isn't trivial.  For example, it requires every 
processor to be interrupted to invalidate some entries from its TLB.  
(This is a so-called "TLB shootdown".)

> ...  How can I measure the effectiveness
> of the buffer cache?
>
>    

I'm not sure that I can give you a short answer to this question.

> The buffer cache size is also very tightly constrained (vfs.hibufspace
> and vfs.lobufspace differ by 64KB) and at least one of the underlying
> tuning parameters have comments at variance with current reality:
> In<sys/param.h>:
>
>   * MAXBSIZE -   Filesystems are made out of blocks of at most MAXBSIZE bytes
>   *              per block.  MAXBSIZE may be made larger without effecting
> ...
>   *
>   * BKVASIZE -   Nominal buffer space per buffer, in bytes.  BKVASIZE is the
> ...
>   *              The default is 16384, roughly 2x the block size used by a
>   *              normal UFS filesystem.
>   */
> #define MAXBSIZE        65536   /* must be power of 2 */
> #define BKVASIZE        16384   /* must be power of 2 */
>
> There's no mention of the 64KiB limit in newfs(8) and I recall seeing
> occasional comments from people who have either tried or suggested
> trying larger blocksizes.

I believe that larger than 64KB would fail an assertion.

>    Likewise, the default UFS blocksize has
> been 16KiB for quite a while.  Are the comments still valid and, if so,
> should BKVASIZE be doubled to 32768 and a suitable note added to newfs(8)
> regarding the maximum block size?
>
>    

If I recall correctly, increasing BKVASIZE would only reduce the number 
buffer headers.  In other words, it might avoid wasting some memory on 
buffer headers that won't be used.

Alan





Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?4C40EC32.3030700>