Date: Tue, 28 Mar 2006 21:27:21 +1100 From: Peter Jeremy <peterjeremy@optushome.com.au> To: Mikhail Teterin <mi+mx@aldan.algebra.com> Cc: stable@freebsd.org Subject: Re: Reading via mmap stinks (Re: weird bugs with mmap-ing via NFS) Message-ID: <20060328102721.GA2352@turion.vk2pj.dyndns.org> In-Reply-To: <20060325103927.GE703@turion.vk2pj.dyndns.org> References: <200603232352.k2NNqPS8018729@gate.bitblocks.com> <200603241518.01027.mi%2Bmx@aldan.algebra.com> <20060325103927.GE703@turion.vk2pj.dyndns.org>
next in thread | previous in thread | raw e-mail | index | archive | help
On Sat, 2006-Mar-25 21:39:27 +1100, Peter Jeremy wrote:
>What happens if you simulate read-ahead yourself? Have your main
>program fork and the child access pages slightly ahead of the parent
>but do nothing else.
I suspect something like this may be the best approach for your application.
My suggestion would be to split the backup into 3 processes that share
memory. I wrote a program that is designed to buffer data in what looks
like a big FIFO and "dump | myfifo | gzip > file.gz" is significantly
faster than "dump | gzip > file.gz" so I suspect it will help you as well.
Process 1 reads the input file into mmap A.
Process 2 {b,gz}ips's mmap A into mmap B.
Process 3 writes mmap B into the output file.
Process 3 and mmap B may be optional, depending on your target's write
performance.
mmap A could be the real file with process 1 just accessing pages to
force them into RAM.
I'd suggest that each mmap be capable of storing several hundred msec of
data as a minumum (maybe 10MB input and 5MB output, preferably more).
Synchronisation can be done by writing tokens into pipes shared with the
mmap's, optimised by sharing read/write pointers (so you only really need
the tokens when the shared buffer is full/empty).
--
Peter Jeremy
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20060328102721.GA2352>