Date: Fri, 10 Aug 2001 02:53:39 -0700 From: Terry Lambert <tlambert2@mindspring.com> To: smail <smail@eStart.ru> Cc: freebsd-hackers@FreeBSD.ORG Subject: Re: need help Message-ID: <3B73AF23.43CC6C05@mindspring.com> References: <15541463687.20010801001742@estart.ru>
next in thread | previous in thread | raw e-mail | index | archive | help
smail wrote:
>
> Hello freebsd-hackers,
>
> i need some help. my problem is about memory limit in mmap function.
> i can't mmap files infinitely, after some number of file mmaped in
> memory i've got an error, probably causing memory limit of 2 or 4 Gb.
> can you help me? my platform is FreeBSD 4.3/i386 [128Mb RAM, 4Gb
> swap].
This is probably your homework, isn't it? 8-)
Your address space is by default limited to 3G (the kernel virtual
address space is 1G and the user address space is 3G: the kernel
needs to be able to get at all memory.
You can change this a little, but you will still bump your head
on the limiting fact that you have only 32 bits with which to
address all the memory you use.
To exceed this limit, you will have to go to indirect mappings;
using indirect mappings, you divide your address space up into
chunks, and then when you need to access an additional chunk of
data, and all your chuncks are busy, you evict a previous mapping
and map the chunk there instead. A common algorithm for this
type of eviction is LRU.
Effectively, you are managing the mapping to implement in software
a virtual address space in excess of your 3G limit; in effect,
there is no practical limit on the amount of data you can access
in this way, since you could go to eviction of your table of ranges,
after going to a hierarchy of tables of ranges of file data that
you map.
A simple example of this technique would be to map a smaller
portion of the file (say 8MB of the file) at a time, and keep a
quad word (C type "long long", FreeBSD type "off_t") of the offset.
To access the first 8MB of the file, mmap it in memory, and iterate
through it. Now iterate through 8GB of the file by moving the
mapping, only when necessary.
This technique is known as "windowing"; here is a simple example,
which won't compile without header files, probably needs "LL"
constant identifiers to make them 64 bits, and may have a typo
even after you add the header files; it will also be incredibly
slow, since the algorithm parameters should be tuned to the data
you will be accessing:
/* I call this program "How much is that file in the window?"*/
void *basep; /* window address*/
off_t relbase; /* window base*/
off_t eight_meg = 0x000000000080000; /* window size*/
int memfd; /* lazy global*/
/*
* Stupid program to dump out the first 8G of a file, one
* byte at a time, using a windowed "get byte" function.
*/
main()
{
off_t curoff;
unsigned char c;
memfd = open( "my_overly_large_file", O_RDONLY, 0);
/* Go from 0 to 8G, one byte at a time...*/
for( curoff = 0; curoff < 0x0000000100000000; curoff++) {
c = byte_at_offset( curoff);
putchar(c);
}
}
/*
* Access a character in an arbitrary length file by mapping
* it into memory in 8M windows, changing the mapping when
* the requested character lands before or after the current
* window, so that a lot of mapping and unmapping isn't needed.
*/
unsigned char
byte_at_offset(off_t offset)
{
off_t reloffset = offset - relbase;
/*
* if the offset is before, or after the window, or
* if we haven't yet set up a window, then we need to
* modify the window we are using.
*/
if( relbase > offset || reloffset > eight_meg || basep == NULL) {
/* if this is the first time, there is nothing to unmap*/
if( basep != NULL) {
munmap( basep, eight_meg);
}
/* get a new relative base, offset, and a new window*/
relbase = offset - (offset % eight_meg);
reloffset = offest - relbase;
basep = mmap( NULL, eight_meg, PROT_READ, MAP_SHARED,
memfd, relbase;
}
/*
* return the requested character from its relative
* location in the window.
*/
return *(unsigned char *)&basep[ reloffset];
}
-- Terry
To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-hackers" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?3B73AF23.43CC6C05>