Date: Fri, 19 Jun 1998 12:37:29 -0700 (PDT) From: Kevin Marcus <kmarcus@saturn.infospace.com> To: Steven Yang <syang@DIRHIT.COM> Cc: "'freebsd-database@FreeBSD.org'" <freebsd-database@FreeBSD.ORG> Subject: Re: gdbm scalability question Message-ID: <Pine.LNX.3.96.980619123327.19523A-100000@saturn.infospace.com> In-Reply-To: <839A86AB6CE4D111A52200104B938D435B41@MOE>
next in thread | previous in thread | raw e-mail | index | archive | help
My observations suggest that gdbm's extendable hashing mechanism does not
scale very well. In particular, it seems that the performance degrades
pretty horribly once you get to around the 1 or 2 million record range.
On the other hand, the berkeley DB routines, using a B-tree structure seem
to perform significantly better. For a database the size you are looking
at, I think the berkeley db routines with a b-tree size of 2 or 4 should
perform quite well.
However, I'm not too sure about memory requirements on the db routines.
Regardless, with 128M of ram, you should be just fine with either gdbm or
the db routines, so I'm sort of surprised you'd get this sort of error.
Kevin Marcus -*- mailto:kmarcus@infospace.com -*- Sr. Software Engr
-*@*- I n f o S p a c e, I n c. -*@*-
"The Ultimate Directory"
Redmond, WA http://www.infospace.com (425)882.1602
On Fri, 19 Jun 1998, Steven Yang wrote:
> Hi, I am having trouble creating a gdbm database larger than 16.9MB. I
> have 128MB of RAM, and would like my database to be as large as 2GB.
> However, as I generate a test database, when the file reaches 16.9MB in
> size, I get:
>
> gdbm fatal: malloc error
>
> It appears that there is a per-process limitation of 16MB of RAM with a
> generic kernel. But even if I were to get around that per-process
> thing, I wouldn't be able to generate a 2GB database file with only
> 128MB of RAM given what I am doing. Does gdbm have to load the entire
> database into RAM in order to insert a record? Is there a setting I
> have to change?
>
> Does anybody out there know how to make a large gdbm database, like one
> with 10 million records where the content of each record is 1-2KB? Once
> the database is created, it is meant to be read-only.
>
> - Steven Yang
>
> To Unsubscribe: send mail to majordomo@FreeBSD.org
> with "unsubscribe freebsd-database" in the body of the message
>
To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-database" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?Pine.LNX.3.96.980619123327.19523A-100000>