Date: Mon, 13 Sep 1999 02:24:14 +0000 (GMT) From: Alfred Perlstein <bright@wintelcom.net> To: "Ronald F. Guilmette" <rfg@monkeys.com> Cc: questions@FreeBSD.ORG Subject: Re: Need persistant heap code. Message-ID: <Pine.BSF.4.05.9909130217290.6392-100000@fw.wintelcom.net> In-Reply-To: <29573.937210706@monkeys.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Mon, 13 Sep 1999, Ronald F. Guilmette wrote: > > >On Sun, 12 Sep 1999, Ronald F. Guilmette wrote: > > > >> > >> I need to find (or, if necessary, build) some package that will provide > >> essentially the same functionality as malloc and free, except for the > >> fact that data placed into the allocated regions should be persistant, > >> i.e. stored in a disk file, even after the program using these primitives > >> terminates. > >> > >> Ideally, the code should do allocation very efficiently as, for example, > >> the GNU malloc package does. > >> > >> It is also essential that the code be able to expand the disk file, as > >> necessary, to accomodate new allocation requests. But the total amount > >> of space that will be allocated in this persistant heap at any given time > >> will never exceed 2GB. > >> > >> And of course, I'd like this as C source code and I need it to run on > >> FreeBSD. > >> > >> Anybody ever heard of such a thing? If so, where might I be able to > >> get a copy? > >> > >> Please understand, I don't need any extraordinarily fancy persistance > >> package... just persistant versions of malloc and free, with maybe > >> calloc thrown in, just for good measure. > >> > >> Any help would be appreciated. > > > >The problem is that you didn't define a "restart" proceedure, > >ie, how is one to identify or get a handle to what is actually > >stored in this on disk heap? > > Humm... Good point. > > I guess that I would be happy to just treat this "persistant heap" thing > as sort of an independent address space. Whatever routine filled the role > of "malloc" for this persistant heap would just return a 32-bit offset, > which would be taken as being relative to the start of the heap file. You still need a handle to the initial allocation unless the initial allocation is garanteed to be the first thing in the file. > >It would be an interesting project to setup such an allocation > >mechanism, I imagine one would need a special pointer type to > >distinguish between real memory and on disk memory. > > I don't think so. > > I mean yea sure, if we were talking C++ code, then yes, you would want to > create a new pointer type... a "persistant heap pointer". But I have gotten > rather un-enamored of C++ and I much prefer the simplicity of C these days. > > In good old C, you could just use void* and/or char* types for these new > persistant heap pointers. Then you would just have to remember to always > add the base address of the place where the persistant file got mmapped > into the current process... assuming that mmap() is used to implement this > whole scheme somehow... before you ever try to dereference one of these > things. If you screw up and forget to do that add, then yea, all hell > breaks loose and you are hosed. But if you're careful, then you don't > need a new type. This is what I meant, you'd need to treat the pointers specially, what if at one point you want to point to something not on this special heap? > >Perhaps if you put up a page explaining the API and restart proceedure > >someone would take an interest in it, then again maybe something is > >already out there that I am unaware of. > > Yes, I should probably do that, but I'm somewhat flexible about the API > at this point, and I'm just hoping that _something_ will turn up like > what I want... something that is already built. (I don't want to *preclude* > any possible solutions just yet.) > > I suppose that I could just grab GNU malloc and then munge it awhile so > as to come up with a version that would just try to do allocations out > of some mmapped region (associated with some disk file) but here is the > problem: How do you EXPAND an mmapped region? I didn't see any such > capability on the mmap() man page and thus, I assume that there is no > such capability. (Should I put in an enhancement request to the kernel > developers?) Yes, there is no mremap as far as I can tell... Why would you want to use a GNU based solution when it locks you into GPL? FreeBSD's malloc is extremely fast and afaik already based on mmap. > The only way I see to expand an mmapped region is to munmap it, extend > the file (using seek/write) and then re-mmap the file again. Or just mmap another region of it, the new area will appear sequentially in your address space. > Seems rather messy, but I guess it would work. It is, you don't have to do that, just extend the file via trunc then mmap the exteneded part. > So I could do that, and I could just hack on GNU malloc until I came up > with a version that knew how to do allocations against (and also how to > expand) and mmapped file. > > I _could_ do all of this, but I'm lazy, and I was just vaguely hoping that > someone else had already done it for me. > (1/2 :-) > It's not a particulary difficult task, you just need to define handles, an alternative might be to use Berkeley DB, it works somewhat like a persistant malloc. -Alfred To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-questions" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?Pine.BSF.4.05.9909130217290.6392-100000>