Date: Wed, 24 Jun 1998 13:46:25 -0400 From: Steven Yang <syang@DIRHIT.COM> To: "'freebsd-database@FreeBSD.org'" <freebsd-database@FreeBSD.ORG> Subject: gdbm, help! Message-ID: <839A86AB6CE4D111A52200104B938D435B4B@MOE>
next in thread | raw e-mail | index | archive | help
Hi, I asked for help last week and got lots of useful tips and some
people also suggested alternatives to GDBM. I said that I had a problem
where I got a gdbm error message of the form: "gdbm fatal: malloc error"
I also previously said I had a file size limitation of 16.9MB.
I have since determined that the file size limitation was never the
issue. That file size was a coincidence because my keys were integers
and my values were integers, so the file size for 130,000 records was
relatively small. By changing the values for the keys to 4K strings, I
could easily make larger files, but not indexes...
Now I have a test script that generates keys that look like "1Query
1","2Query 2",3Query 3","4Query 4"...
and values that look like "6","7","8","9". The values happen to be
(keynum+5), so "145Query145" corresponds to 150.
So key.dsize=strlen("nQuery n")+1 because it is null terminated
and value.dsize=4 because it is an integer. FYI, to store value.dptr, I
have to cast the address of the int to char*
So, I tried to create a GDBM database of 200000 records of these, and
when I try to gdbm_store record 112200, I get
"gdbm fatal: malloc error". I have determined that the malloc error
message gets printed after GDBM tries to split a bucket and receives an
error when it tries to malloc. Specifically, I think it is trying to
allocate more memory for the hash directory (based on looking at GDBM
source code). I have verified through the gdb debugger that the error
occurs during a call to gdbm_store.
I have conducted tests where the value was a large string (5K each), and
when the value is an integer, as I explained for my current test script.
Either way, I get the same malloc error at the same record insertion, so
the limitation I am running it definitely depends on the key.
My previous plea for help generated some nice suggestions like changing
login.conf settings and using db instead of gdbm.
I can change my login.conf settings and give root maximum permissions,
like resource allocations that equal inifinity (most are like this by
default). I get the same problem as root, so I don't think login.conf
is the problem. I also looked into db, but it looks difficult to use
because it seems I need to feed it a hash function, or if I do B trees,
a comparator. Considering all of the time I have invested into GDBM, I
don't really want to spend hours of frustration trying to get db to work
for me. Does anybody have any simple source code that creates a simple
db database and does a lookup??? On that note, does anybody have any
GDBM source code that can make a key of the type I specified and
generate one file with 1 million records?
I'm really frustrated because if I can't make a single GDBM database
with more than 112,199 records, I will probably end up splitting up the
database over many database files. Once generated, these files are to
be READ-ONLY. On the other hand, maybe splitting up the database over
many files might enhance lookup time... any thoughts on that?
I have even tried recompiling the kernel with a few memory-related
modifications, but those failed too (I have FreeBSD 2.2.5).
Somebody suggested using DB (as opposed to db), but I don't know where
to get DB. DB is supposed to have a GDBM-like interface.
So to summarize what help I'm looking for:
- any c/c++ source code you think might help me out for any free
database package
- FreeBSD configuration suggestions
- how to get DB (and hopefully how to use it)
- words of wisdom
- onsite consulting help if you live near Wellesley, MA (we'll pay you
competitive consulting rates).
If you'd like to help me out and would like to see my test scripts, just
ask!
Oh, and an unrelated question: We have a firewall. With either my
FreeBSD machine or my NT machine, I can ping a domain and DNS gives me
the IP, but I also get a reply from a different IP that says
"destination unreachable." Will I have problems doing Internet uploads
of FreeBSD ports? If so, what's the solution?
Thanks for all of the help you guys have provided me so far!
Steven Yang
To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-database" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?839A86AB6CE4D111A52200104B938D435B4B>