From owner-freebsd-database Wed Jun 24 10:48:15 1998 Return-Path: Received: (from majordom@localhost) by hub.freebsd.org (8.8.8/8.8.8) id KAA20868 for freebsd-database-outgoing; Wed, 24 Jun 1998 10:48:15 -0700 (PDT) (envelope-from owner-freebsd-database@FreeBSD.ORG) Received: from elektra.ultra.net (elektra.ultra.net [199.232.56.13]) by hub.freebsd.org (8.8.8/8.8.8) with ESMTP id KAA20780 for ; Wed, 24 Jun 1998 10:47:46 -0700 (PDT) (envelope-from syang@DIRHIT.COM) Received: from moe.dirhit.com ([10.4.18.2]) by elektra.ultra.net (8.8.8/ult.n14767) with ESMTP id NAA25272 for ; Wed, 24 Jun 1998 13:47:41 -0400 (EDT) Received: by MOE with Internet Mail Service (5.0.1459.74) id ; Wed, 24 Jun 1998 13:46:28 -0400 Message-ID: <839A86AB6CE4D111A52200104B938D435B4B@MOE> From: Steven Yang To: "'freebsd-database@FreeBSD.org'" Subject: gdbm, help! Date: Wed, 24 Jun 1998 13:46:25 -0400 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.0.1459.74) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-freebsd-database@FreeBSD.ORG Precedence: bulk Hi, I asked for help last week and got lots of useful tips and some people also suggested alternatives to GDBM. I said that I had a problem where I got a gdbm error message of the form: "gdbm fatal: malloc error" I also previously said I had a file size limitation of 16.9MB. I have since determined that the file size limitation was never the issue. That file size was a coincidence because my keys were integers and my values were integers, so the file size for 130,000 records was relatively small. By changing the values for the keys to 4K strings, I could easily make larger files, but not indexes... Now I have a test script that generates keys that look like "1Query 1","2Query 2",3Query 3","4Query 4"... and values that look like "6","7","8","9". The values happen to be (keynum+5), so "145Query145" corresponds to 150. So key.dsize=strlen("nQuery n")+1 because it is null terminated and value.dsize=4 because it is an integer. FYI, to store value.dptr, I have to cast the address of the int to char* So, I tried to create a GDBM database of 200000 records of these, and when I try to gdbm_store record 112200, I get "gdbm fatal: malloc error". I have determined that the malloc error message gets printed after GDBM tries to split a bucket and receives an error when it tries to malloc. Specifically, I think it is trying to allocate more memory for the hash directory (based on looking at GDBM source code). I have verified through the gdb debugger that the error occurs during a call to gdbm_store. I have conducted tests where the value was a large string (5K each), and when the value is an integer, as I explained for my current test script. Either way, I get the same malloc error at the same record insertion, so the limitation I am running it definitely depends on the key. My previous plea for help generated some nice suggestions like changing login.conf settings and using db instead of gdbm. I can change my login.conf settings and give root maximum permissions, like resource allocations that equal inifinity (most are like this by default). I get the same problem as root, so I don't think login.conf is the problem. I also looked into db, but it looks difficult to use because it seems I need to feed it a hash function, or if I do B trees, a comparator. Considering all of the time I have invested into GDBM, I don't really want to spend hours of frustration trying to get db to work for me. Does anybody have any simple source code that creates a simple db database and does a lookup??? On that note, does anybody have any GDBM source code that can make a key of the type I specified and generate one file with 1 million records? I'm really frustrated because if I can't make a single GDBM database with more than 112,199 records, I will probably end up splitting up the database over many database files. Once generated, these files are to be READ-ONLY. On the other hand, maybe splitting up the database over many files might enhance lookup time... any thoughts on that? I have even tried recompiling the kernel with a few memory-related modifications, but those failed too (I have FreeBSD 2.2.5). Somebody suggested using DB (as opposed to db), but I don't know where to get DB. DB is supposed to have a GDBM-like interface. So to summarize what help I'm looking for: - any c/c++ source code you think might help me out for any free database package - FreeBSD configuration suggestions - how to get DB (and hopefully how to use it) - words of wisdom - onsite consulting help if you live near Wellesley, MA (we'll pay you competitive consulting rates). If you'd like to help me out and would like to see my test scripts, just ask! Oh, and an unrelated question: We have a firewall. With either my FreeBSD machine or my NT machine, I can ping a domain and DNS gives me the IP, but I also get a reply from a different IP that says "destination unreachable." Will I have problems doing Internet uploads of FreeBSD ports? If so, what's the solution? Thanks for all of the help you guys have provided me so far! Steven Yang To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-database" in the body of the message