From owner-freebsd-hackers Fri Aug 10 2:53:26 2001 Delivered-To: freebsd-hackers@freebsd.org Received: from robin.mail.pas.earthlink.net (robin.mail.pas.earthlink.net [207.217.120.65]) by hub.freebsd.org (Postfix) with ESMTP id B127637B403 for ; Fri, 10 Aug 2001 02:53:06 -0700 (PDT) (envelope-from tlambert2@mindspring.com) Received: from mindspring.com (dialup-209.244.105.137.Dial1.SanJose1.Level3.net [209.244.105.137]) by robin.mail.pas.earthlink.net (EL-8_9_3_3/8.9.3) with ESMTP id CAA11348; Fri, 10 Aug 2001 02:52:58 -0700 (PDT) Message-ID: <3B73AF23.43CC6C05@mindspring.com> Date: Fri, 10 Aug 2001 02:53:39 -0700 From: Terry Lambert Reply-To: tlambert2@mindspring.com X-Mailer: Mozilla 4.7 [en]C-CCK-MCD {Sony} (Win98; U) X-Accept-Language: en MIME-Version: 1.0 To: smail Cc: freebsd-hackers@FreeBSD.ORG Subject: Re: need help References: <15541463687.20010801001742@estart.ru> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-freebsd-hackers@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG smail wrote: > > Hello freebsd-hackers, > > i need some help. my problem is about memory limit in mmap function. > i can't mmap files infinitely, after some number of file mmaped in > memory i've got an error, probably causing memory limit of 2 or 4 Gb. > can you help me? my platform is FreeBSD 4.3/i386 [128Mb RAM, 4Gb > swap]. This is probably your homework, isn't it? 8-) Your address space is by default limited to 3G (the kernel virtual address space is 1G and the user address space is 3G: the kernel needs to be able to get at all memory. You can change this a little, but you will still bump your head on the limiting fact that you have only 32 bits with which to address all the memory you use. To exceed this limit, you will have to go to indirect mappings; using indirect mappings, you divide your address space up into chunks, and then when you need to access an additional chunk of data, and all your chuncks are busy, you evict a previous mapping and map the chunk there instead. A common algorithm for this type of eviction is LRU. Effectively, you are managing the mapping to implement in software a virtual address space in excess of your 3G limit; in effect, there is no practical limit on the amount of data you can access in this way, since you could go to eviction of your table of ranges, after going to a hierarchy of tables of ranges of file data that you map. A simple example of this technique would be to map a smaller portion of the file (say 8MB of the file) at a time, and keep a quad word (C type "long long", FreeBSD type "off_t") of the offset. To access the first 8MB of the file, mmap it in memory, and iterate through it. Now iterate through 8GB of the file by moving the mapping, only when necessary. This technique is known as "windowing"; here is a simple example, which won't compile without header files, probably needs "LL" constant identifiers to make them 64 bits, and may have a typo even after you add the header files; it will also be incredibly slow, since the algorithm parameters should be tuned to the data you will be accessing: /* I call this program "How much is that file in the window?"*/ void *basep; /* window address*/ off_t relbase; /* window base*/ off_t eight_meg = 0x000000000080000; /* window size*/ int memfd; /* lazy global*/ /* * Stupid program to dump out the first 8G of a file, one * byte at a time, using a windowed "get byte" function. */ main() { off_t curoff; unsigned char c; memfd = open( "my_overly_large_file", O_RDONLY, 0); /* Go from 0 to 8G, one byte at a time...*/ for( curoff = 0; curoff < 0x0000000100000000; curoff++) { c = byte_at_offset( curoff); putchar(c); } } /* * Access a character in an arbitrary length file by mapping * it into memory in 8M windows, changing the mapping when * the requested character lands before or after the current * window, so that a lot of mapping and unmapping isn't needed. */ unsigned char byte_at_offset(off_t offset) { off_t reloffset = offset - relbase; /* * if the offset is before, or after the window, or * if we haven't yet set up a window, then we need to * modify the window we are using. */ if( relbase > offset || reloffset > eight_meg || basep == NULL) { /* if this is the first time, there is nothing to unmap*/ if( basep != NULL) { munmap( basep, eight_meg); } /* get a new relative base, offset, and a new window*/ relbase = offset - (offset % eight_meg); reloffset = offest - relbase; basep = mmap( NULL, eight_meg, PROT_READ, MAP_SHARED, memfd, relbase; } /* * return the requested character from its relative * location in the window. */ return *(unsigned char *)&basep[ reloffset]; } -- Terry To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-hackers" in the body of the message