From owner-freebsd-database Sun Jun 14 10:45:13 1998 Return-Path: Received: (from majordom@localhost) by hub.freebsd.org (8.8.8/8.8.8) id KAA03351 for freebsd-database-outgoing; Sun, 14 Jun 1998 10:45:13 -0700 (PDT) (envelope-from owner-freebsd-database@FreeBSD.ORG) Received: from mail.actrix.gen.nz (root@mail.actrix.gen.nz [203.96.16.37]) by hub.freebsd.org (8.8.8/8.8.8) with ESMTP id KAA03346; Sun, 14 Jun 1998 10:45:10 -0700 (PDT) (envelope-from andrew@squiz.co.nz) Received: from [192.168.1.2] (aniwa.actrix.gen.nz [203.96.56.186]) by mail.actrix.gen.nz (8.8.8/8.8.5) with SMTP id FAA15821; Mon, 15 Jun 1998 05:45:00 +1200 (NZST) X-Sender: andrew@192.168.1.1 Message-Id: Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Date: Mon, 15 Jun 1998 05:47:51 +1200 To: questions@FreeBSD.ORG, database@FreeBSD.ORG From: andrew@squiz.co.nz (Andrew McNaughton) Subject: has anyone run freewais-sf-2.2? Sender: owner-freebsd-database@FreeBSD.ORG Precedence: bulk Has anyone evr gotten freewais-sf-2.2 to run under freebsd? >From what I can see, freewais-sf-2.2.10 was the latest developed, and that some time ago, but freebsd does not seem to have a port for this. Has anyone gotten it to run on their system, or had a serious look at what's involved? Andrew McNaughton ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Andrew McNaughton = ++64 4 389 6891 Any sufficiently advanced = andrew@squiz.co.nz bug is indistinguishable = http://www.newsroom.co from a feature. = -- Rich Kulawiec = To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-database" in the body of the message From owner-freebsd-database Fri Jun 19 10:20:18 1998 Return-Path: Received: (from majordom@localhost) by hub.freebsd.org (8.8.8/8.8.8) id KAA24403 for freebsd-database-outgoing; Fri, 19 Jun 1998 10:20:18 -0700 (PDT) (envelope-from owner-freebsd-database@FreeBSD.ORG) Received: from elektra.ultra.net (elektra.ultra.net [199.232.56.13]) by hub.freebsd.org (8.8.8/8.8.8) with ESMTP id KAA24294 for ; Fri, 19 Jun 1998 10:19:39 -0700 (PDT) (envelope-from syang@DIRHIT.COM) Received: from moe.dirhit.com (d-hit.dirhit.com [146.115.235.211]) by elektra.ultra.net (8.8.8/ult.n14767) with ESMTP id NAA25010 for ; Fri, 19 Jun 1998 13:19:32 -0400 (EDT) Received: by MOE with Internet Mail Service (5.0.1459.74) id ; Fri, 19 Jun 1998 13:01:21 -0400 Message-ID: <839A86AB6CE4D111A52200104B938D435B41@MOE> From: Steven Yang To: "'freebsd-database@FreeBSD.org'" Subject: gdbm scalability question Date: Fri, 19 Jun 1998 13:01:20 -0400 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.0.1459.74) Content-Type: text/plain Sender: owner-freebsd-database@FreeBSD.ORG Precedence: bulk Hi, I am having trouble creating a gdbm database larger than 16.9MB. I have 128MB of RAM, and would like my database to be as large as 2GB. However, as I generate a test database, when the file reaches 16.9MB in size, I get: gdbm fatal: malloc error It appears that there is a per-process limitation of 16MB of RAM with a generic kernel. But even if I were to get around that per-process thing, I wouldn't be able to generate a 2GB database file with only 128MB of RAM given what I am doing. Does gdbm have to load the entire database into RAM in order to insert a record? Is there a setting I have to change? Does anybody out there know how to make a large gdbm database, like one with 10 million records where the content of each record is 1-2KB? Once the database is created, it is meant to be read-only. - Steven Yang To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-database" in the body of the message From owner-freebsd-database Fri Jun 19 12:38:58 1998 Return-Path: Received: (from majordom@localhost) by hub.freebsd.org (8.8.8/8.8.8) id MAA20615 for freebsd-database-outgoing; Fri, 19 Jun 1998 12:38:58 -0700 (PDT) (envelope-from owner-freebsd-database@FreeBSD.ORG) Received: from saturn.infospace.com (kmarcus@saturn.infospace.com [206.129.166.107]) by hub.freebsd.org (8.8.8/8.8.8) with ESMTP id MAA20578 for ; Fri, 19 Jun 1998 12:38:41 -0700 (PDT) (envelope-from kmarcus@saturn.infospace.com) Received: from localhost (kmarcus@localhost) by saturn.infospace.com (8.8.7/8.8.7) with SMTP id MAA19711; Fri, 19 Jun 1998 12:37:30 -0700 Date: Fri, 19 Jun 1998 12:37:29 -0700 (PDT) From: Kevin Marcus To: Steven Yang cc: "'freebsd-database@FreeBSD.org'" Subject: Re: gdbm scalability question In-Reply-To: <839A86AB6CE4D111A52200104B938D435B41@MOE> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-database@FreeBSD.ORG Precedence: bulk My observations suggest that gdbm's extendable hashing mechanism does not scale very well. In particular, it seems that the performance degrades pretty horribly once you get to around the 1 or 2 million record range. On the other hand, the berkeley DB routines, using a B-tree structure seem to perform significantly better. For a database the size you are looking at, I think the berkeley db routines with a b-tree size of 2 or 4 should perform quite well. However, I'm not too sure about memory requirements on the db routines. Regardless, with 128M of ram, you should be just fine with either gdbm or the db routines, so I'm sort of surprised you'd get this sort of error. Kevin Marcus -*- mailto:kmarcus@infospace.com -*- Sr. Software Engr -*@*- I n f o S p a c e, I n c. -*@*- "The Ultimate Directory" Redmond, WA http://www.infospace.com (425)882.1602 On Fri, 19 Jun 1998, Steven Yang wrote: > Hi, I am having trouble creating a gdbm database larger than 16.9MB. I > have 128MB of RAM, and would like my database to be as large as 2GB. > However, as I generate a test database, when the file reaches 16.9MB in > size, I get: > > gdbm fatal: malloc error > > It appears that there is a per-process limitation of 16MB of RAM with a > generic kernel. But even if I were to get around that per-process > thing, I wouldn't be able to generate a 2GB database file with only > 128MB of RAM given what I am doing. Does gdbm have to load the entire > database into RAM in order to insert a record? Is there a setting I > have to change? > > Does anybody out there know how to make a large gdbm database, like one > with 10 million records where the content of each record is 1-2KB? Once > the database is created, it is meant to be read-only. > > - Steven Yang > > To Unsubscribe: send mail to majordomo@FreeBSD.org > with "unsubscribe freebsd-database" in the body of the message > To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-database" in the body of the message From owner-freebsd-database Fri Jun 19 13:31:45 1998 Return-Path: Received: (from majordom@localhost) by hub.freebsd.org (8.8.8/8.8.8) id NAA29575 for freebsd-database-outgoing; Fri, 19 Jun 1998 13:31:45 -0700 (PDT) (envelope-from owner-freebsd-database@FreeBSD.ORG) Received: from nomis.simon-shapiro.org ([209.86.126.163]) by hub.freebsd.org (8.8.8/8.8.8) with SMTP id NAA28087 for ; Fri, 19 Jun 1998 13:20:54 -0700 (PDT) (envelope-from shimon@nomis.Simon-Shapiro.ORG) Received: (qmail 7326 invoked by uid 1000); 19 Jun 1998 20:20:59 -0000 Message-ID: X-Mailer: XFMail 1.3 [p0] on FreeBSD X-Priority: 3 (Normal) MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="_=XFMail.1.3.p0.FreeBSD:980619162058:3015=_" In-Reply-To: <839A86AB6CE4D111A52200104B938D435B41@MOE> Date: Fri, 19 Jun 1998 16:20:58 -0400 (EDT) Reply-To: shimon@simon-shapiro.org Organization: The Simon Shapiro Foundation From: Simon Shapiro To: Steven Yang Subject: RE: gdbm scalability question Cc: "freebsd-database@FreeBSD.org" Sender: owner-freebsd-database@FreeBSD.ORG Precedence: bulk This message is in MIME format --_=XFMail.1.3.p0.FreeBSD:980619162058:3015=_ Content-Type: text/plain; charset=us-ascii Try the ulimit command. Then try to glean from my kernel configuration below what is it that I do to get your problem solved. I simnply do not remember. Simon On 19-Jun-98 Steven Yang wrote: > Hi, I am having trouble creating a gdbm database larger than 16.9MB. I > have 128MB of RAM, and would like my database to be as large as 2GB. > However, as I generate a test database, when the file reaches 16.9MB in > size, I get: > > gdbm fatal: malloc error > > It appears that there is a per-process limitation of 16MB of RAM with a > generic kernel. But even if I were to get around that per-process > thing, I wouldn't be able to generate a 2GB database file with only > 128MB of RAM given what I am doing. Does gdbm have to load the entire > database into RAM in order to insert a record? Is there a setting I > have to change? > > Does anybody out there know how to make a large gdbm database, like one > with 10 million records where the content of each record is 1-2KB? Once > the database is created, it is meant to be read-only. > > - Steven Yang > > To Unsubscribe: send mail to majordomo@FreeBSD.org > with "unsubscribe freebsd-database" in the body of the message ---------- Sincerely Yours, Simon Shapiro Shimon@Simon-Shapiro.ORG Voice: 503.799.2313 --_=XFMail.1.3.p0.FreeBSD:980619162058:3015=_ Content-Disposition: attachment; filename="NOMIS-smp" Content-Transfer-Encoding: base64 Content-Description: Large kernel configuration Content-Type: application/octet-stream; name=NOMIS-smp; SizeOnDisk=11014 IwojIE5PTUlTLXNtcCAtLSBUZXN0IERQVCBTTVAga2VybmVsIGZvciBOT01JUy1saWtlIG1hY2hp bmVzCiMKIwkkSWQ6IE5PTUlTLXNtcCx2IDEuMTYgMTk5OC8wNi8xNSAxNjowMDoxMiBTaGltb25S IEV4cCBTaGltb25SICQKCm1hY2hpbmUJCSJpMzg2IgpjcHUJCSJJNjg2X0NQVSIKaWRlbnQJCU5P TUlTLXNtcAptYXh1c2Vycwk2NAoKb3B0aW9ucwkJSU5FVAkJCSNJbnRlck5FVHdvcmtpbmcKb3B0 aW9ucwkJRkZTCQkJI0JlcmtlbGV5IEZhc3QgRmlsZXN5c3RlbQpvcHRpb25zCQlORlMJCQkjTmV0 d29yayBGaWxlc3lzdGVtCm9wdGlvbnMJCU1TRE9TRlMJCQkjTVNET1MgRmlsZXN5c3RlbQpvcHRp b25zCQkiQ0Q5NjYwIgkJI0lTTyA5NjYwIEZpbGVzeXN0ZW0Kb3B0aW9ucwkJUFJPQ0ZTCQkJI1By b2Nlc3MgZmlsZXN5c3RlbQpvcHRpb25zCQkiQ09NUEFUXzQzIgkJI0NvbXBhdGlibGUgd2l0aCBC U0QgNC4zIFtLRUVQIFRISVMhXQpvcHRpb25zCQlCT1VOQ0VfQlVGRkVSUwkJI2luY2x1ZGUgc3Vw cG9ydCBmb3IgRE1BIGJvdW5jZSBidWZmZXJzCm9wdGlvbnMJCVVDT05TT0xFCQkjQWxsb3cgdXNl cnMgdG8gZ3JhYiB0aGUgY29uc29sZQpvcHRpb25zCQlGQUlMU0FGRQkJI0JlIGNvbnNlcnZhdGl2 ZQpvcHRpb25zCQlVU0VSQ09ORklHCQkjYm9vdCAtYyBlZGl0b3IKb3B0aW9ucwkJVklTVUFMX1VT RVJDT05GSUcJI3Zpc3VhbCBib290IC1jIGVkaXRvcgoKb3B0aW9ucwkJRERCCm9wdGlvbnMJCUtU UkFDRQpvcHRpb25zCQlQRVJGTU9OCgpvcHRpb25zCQlTTVAgICAgICAgICAgICAgICAgICAgICAj IFN5bW1ldHJpYyBNdWx0aVByb2Nlc3NvciBLZXJuZWwKb3B0aW9ucwkJQVBJQ19JTyAgICAgICAg ICAgICAgICAgIyBTeW1tZXRyaWMgKEFQSUMpIEkvTwoKY29uZmlnCQlrZXJuZWwJcm9vdCBvbiB3 ZDAKCmNvbnRyb2xsZXIJaXNhMApjb250cm9sbGVyCXBjaTAKCmNvbnRyb2xsZXIJZmRjMAlhdCBp c2E/IHBvcnQgIklPX0ZEMSIgYmlvIGlycSA2IGRycSAyIHZlY3RvciBmZGludHIKZGlzawkJZmQw CWF0IGZkYzAgZHJpdmUgMApkaXNrCQlmZDEJYXQgZmRjMCBkcml2ZSAxCgpjb250cm9sbGVyCXdk YzAJYXQgaXNhPyBwb3J0ICJJT19XRDEiIGJpbyBpcnEgMTQgdmVjdG9yIHdkaW50cgpkaXNrCQl3 ZDAJYXQgd2RjMCBkcml2ZSAwCmRpc2sJCXdkMQlhdCB3ZGMwIGRyaXZlIDEKCmNvbnRyb2xsZXIJ d2RjMQlhdCBpc2E/IHBvcnQgIklPX1dEMiIgYmlvIGlycSAxNSB2ZWN0b3Igd2RpbnRyCmRpc2sJ CXdkMglhdCB3ZGMxIGRyaXZlIDAKZGlzawkJd2QzCWF0IHdkYzEgZHJpdmUgMQoKb3B0aW9ucwkJ QVRBUEkKb3B0aW9ucwkJQVRBUElfU1RBVElDCmRldmljZQkJd2NkMAoKIyBXZSBhc3N1bWUgd2lk ZSBTQ1NJLCBub3QgRkNBTCwgd2l0aCB0aGlzIHJlbGVhc2UuCiMgVGhlcmUgaXMgbmV2ZXIgYSB0 YXJnZXQgNywgbm9yIHRhcmdldCAxNTsgVGhleSBhcmUgcmVzZXJ2ZWQgZm9yIGluaXRpYXRvcnMh CiMgRWFjaCBBREFQVEVSIHRha2VzIDQgYnVzc2VzIGJ1dCB1c2VzIG9ubHkgYnVzc2VzIDAtMi4K IyBFYWNoIGJ1cyBjb25zdW1lcyAxNiBhZGRyZXNzZXMsIGJ1dCB1c2VzIG5vIG1vcmUgdGhhbiAx MyBkZXZpY2VzIHBlciBidXMuCiMgRGV2aWNlcyB3aXRoIG5vIExVTnMgaGF2ZSBvbmx5IExVTiAw LiAgTHVuICE9IDAgbWVhbnMgc2Vjb25kYXJ5IExVTlMgMS03CiMgTm9uLXplcm8gTFVOcyBoYXZl IHRoZSBzYW1lIElEIGFzIExVTiAwIGJ1dCBiaXQgNyBpcyBzZXQgb24gKDY0ICsgdGFyZ2V0KQoK Y29udHJvbGxlciAgICAgIGRwdDAKY29udHJvbGxlcglkcHQxCgpvcHRpb25zCQlEUFRfTUVBU1VS RV9QRVJGT1JNQU5DRQpvcHRpb25zCQlEUFRfTE9TVF9JUlEKCmNvbnRyb2xsZXIJc2NidXMwICBh dCBkcHQwIGJ1cyAwCmNvbnRyb2xsZXIJc2NidXMxICBhdCBkcHQwIGJ1cyAxCmNvbnRyb2xsZXIJ c2NidXMyICBhdCBkcHQwIGJ1cyAyCgpjb250cm9sbGVyCXNjYnVzNCAgYXQgZHB0MSBidXMgMApj b250cm9sbGVyCXNjYnVzNSAgYXQgZHB0MSBidXMgMQpjb250cm9sbGVyCXNjYnVzNiAgYXQgZHB0 MSBidXMgMgoKZGV2aWNlCQljaDAKZGV2aWNlCQlzZDAKZGV2aWNlCQlvZDAKZGV2aWNlCQlzdDAK ZGV2aWNlCQljZDAKCiMgRmlyc3QgRFBUIHN0YXJ0cyBoZXJlCgojIE9uIGJ1cyAwLCB5b3UgY2Fu IGhhdmUgYm90aCBhIENkIGFuZCBhIHdvcm0gYnV0IG5vIG11bHRpLWluaXRpYXRvcgpkZXZpY2Ug ICAgICB3b3JtMCAgIGF0IHNjYnVzMCAgdGFyZ2V0ICA0IHVuaXQgMAp0YXBlICAgICAgICBzdDAg ICAgIGF0IHNjYnVzMCAgdGFyZ2V0ICA1IHVuaXQgMApkZXZpY2UgICAgICBjZDAgICAgIGF0IHNj YnVzMCAgdGFyZ2V0ICA2IHVuaXQgMAoKZGlzayAgICAgICAgc2QwICAgICBhdCBzY2J1czAgIHRh cmdldCAgMCB1bml0IDAKZGlzayAgICAgICAgc2QxICAgICBhdCBzY2J1czAgIHRhcmdldCAgMSB1 bml0IDAKZGlzayAgICAgICAgc2QyICAgICBhdCBzY2J1czAgIHRhcmdldCAgMiB1bml0IDAKZGlz ayAgICAgICAgc2QzICAgICBhdCBzY2J1czAgIHRhcmdldCAgMyB1bml0IDAKZGlzayAgICAgICAg c2Q0ICAgICBhdCBzY2J1czAgIHRhcmdldCAgNCB1bml0IDAKZGlzayAgICAgICAgc2Q1ICAgICBh dCBzY2J1czAgIHRhcmdldCAgNSB1bml0IDAKZGlzayAgICAgICAgc2Q2ICAgICBhdCBzY2J1czAg IHRhcmdldCAgNiB1bml0IDAKZGlzayAgICAgICAgc2Q4ICAgICBhdCBzY2J1czAgIHRhcmdldCAg OCB1bml0IDAKZGlzayAgICAgICAgc2Q5ICAgICBhdCBzY2J1czAgIHRhcmdldCAgOSB1bml0IDAK ZGlzayAgICAgICAgc2QxMCAgICBhdCBzY2J1czAgIHRhcmdldCAxMCB1bml0IDAKZGlzayAgICAg ICAgc2QxMSAgICBhdCBzY2J1czAgIHRhcmdldCAxMSB1bml0IDAKZGlzayAgICAgICAgc2QxMiAg ICBhdCBzY2J1czAgIHRhcmdldCAxMiB1bml0IDAKZGlzayAgICAgICAgc2QxMyAgICBhdCBzY2J1 czAgIHRhcmdldCAxMyB1bml0IDAKZGlzayAgICAgICAgc2QxNCAgICBhdCBzY2J1czAgIHRhcmdl dCAxNCB1bml0IDAKCiMgT24gYnVzIDEgeW91IGNhbiBoYXZlIGVpdGhlciBhIENEIG9yIGEgV09S TSwgYW5kIG11bHRpLWluaXRpYXRvcnMKZGV2aWNlICAgICAgd29ybTEgICBhdCBzY2J1czEgIHRh cmdldCAgNCB1bml0IDAKZGV2aWNlICAgICAgY2QxICAgICBhdCBzY2J1czEgIHRhcmdldCAgNCB1 bml0IDAKdGFwZSAgICAgICAgc3QxICAgICBhdCBzY2J1czEgIHRhcmdldCAgNSB1bml0IDAKZGV2 aWNlCQlwdDEJCWF0IHNjYnVzMSAgdGFyZ2V0ICA2IHVuaXQgMAkjIFNDU0kgcHJvY2Vzc29yIHR5 cGUKZGV2aWNlCQlwdDEJCWF0IHNjYnVzMSAgdGFyZ2V0ICA3IHVuaXQgMAkjIFNDU0kgcHJvY2Vz c29yIHR5cGUKCmRpc2sgICAgICAgIHNkMTYgICAgYXQgc2NidXMxICB0YXJnZXQgIDAgdW5pdCAw CmRpc2sgICAgICAgIHNkMTcgICAgYXQgc2NidXMxICB0YXJnZXQgIDEgdW5pdCAwCmRpc2sgICAg ICAgIHNkMTggICAgYXQgc2NidXMxICB0YXJnZXQgIDIgdW5pdCAwCmRpc2sgICAgICAgIHNkMTkg ICAgYXQgc2NidXMxICB0YXJnZXQgIDMgdW5pdCAwCmRpc2sgICAgICAgIHNkMjAgICAgYXQgc2Ni dXMxICB0YXJnZXQgIDQgdW5pdCAwCmRpc2sgICAgICAgIHNkMjEgICAgYXQgc2NidXMxICB0YXJn ZXQgIDUgdW5pdCAwCmRpc2sgICAgICAgIHNkMjIgICAgYXQgc2NidXMxICB0YXJnZXQgIDYgdW5p dCAwCmRpc2sgICAgICAgIHNkMjQgICAgYXQgc2NidXMxICB0YXJnZXQgIDggdW5pdCAwCmRpc2sg ICAgICAgIHNkMjUgICAgYXQgc2NidXMxICB0YXJnZXQgIDkgdW5pdCAwCmRpc2sgICAgICAgIHNk MjYgICAgYXQgc2NidXMxICB0YXJnZXQgMTAgdW5pdCAwCmRpc2sgICAgICAgIHNkMjcgICAgYXQg c2NidXMxICB0YXJnZXQgMTEgdW5pdCAwCmRpc2sgICAgICAgIHNkMjggICAgYXQgc2NidXMxICB0 YXJnZXQgMTIgdW5pdCAwCmRpc2sgICAgICAgIHNkMjkgICAgYXQgc2NidXMxICB0YXJnZXQgMTMg dW5pdCAwCmRpc2sgICAgICAgIHNkMzAgICAgYXQgc2NidXMxICB0YXJnZXQgMTQgdW5pdCAwCgoj IE9uIGJ1cyAyIHlvdSBjYW4gaGF2ZSBlaXRoZXIgYSBDRCBvciBhIFdPUk0sIGFuZCBtdWx0aS1p bml0aWF0b3JzCmRldmljZSAgICAgIHdvcm0yICAgYXQgc2NidXMyICB0YXJnZXQgIDQgdW5pdCAw CmRldmljZSAgICAgIGNkMiAgICAgYXQgc2NidXMyICB0YXJnZXQgIDQgdW5pdCAwCnRhcGUgICAg ICAgIHN0MiAgICAgYXQgc2NidXMyICB0YXJnZXQgIDUgdW5pdCAwCmRldmljZQkJcHQyCQlhdCBz Y2J1czIgIHRhcmdldCAgNiB1bml0IDAJIyBTQ1NJIHByb2Nlc3NvciB0eXBlCmRldmljZQkJcHQy CQlhdCBzY2J1czIgIHRhcmdldCAgNyB1bml0IDAJIyBTQ1NJIHByb2Nlc3NvciB0eXBlCgpkaXNr ICAgICAgICBzZDMyICAgIGF0IHNjYnVzMiAgdGFyZ2V0ICAwIHVuaXQgMApkaXNrICAgICAgICBz ZDMzICAgIGF0IHNjYnVzMiAgdGFyZ2V0ICAxIHVuaXQgMApkaXNrICAgICAgICBzZDM0ICAgIGF0 IHNjYnVzMiAgdGFyZ2V0ICAyIHVuaXQgMApkaXNrICAgICAgICBzZDM1ICAgIGF0IHNjYnVzMiAg dGFyZ2V0ICAzIHVuaXQgMApkaXNrICAgICAgICBzZDM2ICAgIGF0IHNjYnVzMiAgdGFyZ2V0ICA0 IHVuaXQgMApkaXNrICAgICAgICBzZDM3ICAgIGF0IHNjYnVzMiAgdGFyZ2V0ICA1IHVuaXQgMApk aXNrICAgICAgICBzZDM4ICAgIGF0IHNjYnVzMiAgdGFyZ2V0ICA2IHVuaXQgMApkaXNrICAgICAg ICBzZDQxICAgIGF0IHNjYnVzMiAgdGFyZ2V0ICA5IHVuaXQgMApkaXNrICAgICAgICBzZDQyICAg IGF0IHNjYnVzMiAgdGFyZ2V0IDEwIHVuaXQgMApkaXNrICAgICAgICBzZDQzICAgIGF0IHNjYnVz MiAgdGFyZ2V0IDExIHVuaXQgMApkaXNrICAgICAgICBzZDQ0ICAgIGF0IHNjYnVzMiAgdGFyZ2V0 IDEyIHVuaXQgMApkaXNrICAgICAgICBzZDQ1ICAgIGF0IHNjYnVzMiAgdGFyZ2V0IDEzIHVuaXQg MApkaXNrICAgICAgICBzZDQ2ICAgIGF0IHNjYnVzMiAgdGFyZ2V0IDE0IHVuaXQgMAoKIyBTZWNv bmQgRFBUIHN0YXJ0cyBoZXJlCgojIE9uIGJ1cyAwLCB5b3UgY2FuIGhhdmUgYm90aCBhIENEIGFu ZCBhIHdvcm0gYnV0IG5vIG11bHRpLWluaXRpYXRvcgpkZXZpY2UgICAgICB3b3JtNCAgIGF0IHNj YnVzNCAgdGFyZ2V0ICA0IHVuaXQgMAp0YXBlICAgICAgICBzdDQgICAgIGF0IHNjYnVzNCAgdGFy Z2V0ICA1IHVuaXQgMApkZXZpY2UgICAgICBjZDQgICAgIGF0IHNjYnVzNCAgdGFyZ2V0ICA2IHVu aXQgMAoKZGlzayAgICAgICAgc2Q2NCAgICBhdCBzY2J1czQgIHRhcmdldCAgMCB1bml0IDAKZGlz ayAgICAgICAgc2Q2NSAgICBhdCBzY2J1czQgIHRhcmdldCAgMSB1bml0IDAKZGlzayAgICAgICAg c2Q2NiAgICBhdCBzY2J1czQgIHRhcmdldCAgMiB1bml0IDAKZGlzayAgICAgICAgc2Q2NyAgICBh dCBzY2J1czQgIHRhcmdldCAgMyB1bml0IDAKZGlzayAgICAgICAgc2Q2OCAgICBhdCBzY2J1czQg IHRhcmdldCAgNCB1bml0IDAKZGlzayAgICAgICAgc2Q2OSAgICBhdCBzY2J1czQgIHRhcmdldCAg NSB1bml0IDAKZGlzayAgICAgICAgc2Q3MCAgICBhdCBzY2J1czQgIHRhcmdldCAgNiB1bml0IDAK ZGlzayAgICAgICAgc2Q3MiAgICBhdCBzY2J1czQgIHRhcmdldCAgOCB1bml0IDAKZGlzayAgICAg ICAgc2Q3MyAgICBhdCBzY2J1czQgIHRhcmdldCAgOSB1bml0IDAKZGlzayAgICAgICAgc2Q3NCAg ICBhdCBzY2J1czQgIHRhcmdldCAxMCB1bml0IDAKZGlzayAgICAgICAgc2Q3NSAgICBhdCBzY2J1 czQgIHRhcmdldCAxMSB1bml0IDAKZGlzayAgICAgICAgc2Q3NiAgICBhdCBzY2J1czQgIHRhcmdl dCAxMiB1bml0IDAKZGlzayAgICAgICAgc2Q3NyAgICBhdCBzY2J1czQgIHRhcmdldCAxMyB1bml0 IDAKZGlzayAgICAgICAgc2Q3OCAgICBhdCBzY2J1czQgIHRhcmdldCAxNCB1bml0IDAKZGlzayAg ICAgICAgc2Q3OSAgICBhdCBzY2J1czQgIHRhcmdldCAxNSB1bml0IDAKCiMgT24gYnVzIDEgeW91 IGNhbiBoYXZlIGVpdGhlciBhIENEIG9yIGEgV09STSwgYW5kIG11bHRpLWluaXRpYXRvcnMKZGV2 aWNlICAgICAgd29ybTUgICBhdCBzY2J1czUgIHRhcmdldCAgNCB1bml0IDAKZGV2aWNlICAgICAg Y2Q1ICAgICBhdCBzY2J1czUgIHRhcmdldCAgNCB1bml0IDAKdGFwZSAgICAgICAgc3Q1ICAgICBh dCBzY2J1czUgIHRhcmdldCAgNSB1bml0IDAKZGV2aWNlCQlwdDUJCWF0IHNjYnVzMSAgdGFyZ2V0 ICA2IHVuaXQgMAkjIFNDU0kgcHJvY2Vzc29yIHR5cGUKZGV2aWNlCQlwdDUJCWF0IHNjYnVzMSAg dGFyZ2V0ICA3IHVuaXQgMAkjIFNDU0kgcHJvY2Vzc29yIHR5cGUKCmRpc2sgICAgICAgIHNkODAg ICAgYXQgc2NidXM1ICB0YXJnZXQgIDAgdW5pdCAwCmRpc2sgICAgICAgIHNkODEgICAgYXQgc2Ni dXM1ICB0YXJnZXQgIDEgdW5pdCAwCmRpc2sgICAgICAgIHNkODIgICAgYXQgc2NidXM1ICB0YXJn ZXQgIDIgdW5pdCAwCmRpc2sgICAgICAgIHNkODMgICAgYXQgc2NidXM1ICB0YXJnZXQgIDMgdW5p dCAwCmRpc2sgICAgICAgIHNkODQgICAgYXQgc2NidXM1ICB0YXJnZXQgIDQgdW5pdCAwCmRpc2sg ICAgICAgIHNkODUgICAgYXQgc2NidXM1ICB0YXJnZXQgIDUgdW5pdCAwCmRpc2sgICAgICAgIHNk ODYgICAgYXQgc2NidXM1ICB0YXJnZXQgIDYgdW5pdCAwCmRpc2sgICAgICAgIHNkODggICAgYXQg c2NidXM1ICB0YXJnZXQgIDggdW5pdCAwCmRpc2sgICAgICAgIHNkODkgICAgYXQgc2NidXM1ICB0 YXJnZXQgIDkgdW5pdCAwCmRpc2sgICAgICAgIHNkOTAgICAgYXQgc2NidXM1ICB0YXJnZXQgMTAg dW5pdCAwCmRpc2sgICAgICAgIHNkOTEgICAgYXQgc2NidXM1ICB0YXJnZXQgMTEgdW5pdCAwCmRp c2sgICAgICAgIHNkOTIgICAgYXQgc2NidXM1ICB0YXJnZXQgMTIgdW5pdCAwCmRpc2sgICAgICAg IHNkOTMgICAgYXQgc2NidXM1ICB0YXJnZXQgMTMgdW5pdCAwCmRpc2sgICAgICAgIHNkOTQgICAg YXQgc2NidXM1ICB0YXJnZXQgMTQgdW5pdCAwCmRpc2sgICAgICAgIHNkOTUgICAgYXQgc2NidXM1 ICB0YXJnZXQgMTUgdW5pdCAwCiAKIyBPbiBidXMgMiB5b3UgY2FuIGhhdmUgZWl0aGVyIGEgQ0Qg b3IgYSBXT1JNLCBhbmQgbXVsdGktaW5pdGlhdG9ycwpkZXZpY2UgICAgICB3b3JtNiAgIGF0IHNj YnVzNiAgdGFyZ2V0ICA0IHVuaXQgMApkZXZpY2UgICAgICBjZDYgICAgIGF0IHNjYnVzNiAgdGFy Z2V0ICA0IHVuaXQgMAp0YXBlICAgICAgICBzdDYgICAgIGF0IHNjYnVzNiAgdGFyZ2V0ICA1IHVu aXQgMApkZXZpY2UJCXB0NgkJYXQgc2NidXMxICB0YXJnZXQgIDYgdW5pdCAwCSMgU0NTSSBwcm9j ZXNzb3IgdHlwZQpkZXZpY2UJCXB0NgkJYXQgc2NidXMxICB0YXJnZXQgIDcgdW5pdCAwCSMgU0NT SSBwcm9jZXNzb3IgdHlwZQoKZGlzayAgICAgICAgc2Q5NiAgICBhdCBzY2J1czYgIHRhcmdldCAg MCB1bml0IDAKZGlzayAgICAgICAgc2Q5NyAgICBhdCBzY2J1czYgIHRhcmdldCAgMSB1bml0IDAK ZGlzayAgICAgICAgc2Q5OCAgICBhdCBzY2J1czYgIHRhcmdldCAgMiB1bml0IDAKZGlzayAgICAg ICAgc2Q5OSAgICBhdCBzY2J1czYgIHRhcmdldCAgMyB1bml0IDAKZGlzayAgICAgICAgc2QxMDAg ICBhdCBzY2J1czYgIHRhcmdldCAgNCB1bml0IDAKZGlzayAgICAgICAgc2QxMDEgICBhdCBzY2J1 czYgIHRhcmdldCAgNSB1bml0IDAKZGlzayAgICAgICAgc2QxMDIgICBhdCBzY2J1czYgIHRhcmdl dCAgNiB1bml0IDAKZGlzayAgICAgICAgc2QxMDQgICBhdCBzY2J1czYgIHRhcmdldCAgOCB1bml0 IDAKZGlzayAgICAgICAgc2QxMDUgICBhdCBzY2J1czYgIHRhcmdldCAgOSB1bml0IDAKZGlzayAg ICAgICAgc2QxMDYgICBhdCBzY2J1czYgIHRhcmdldCAxMCB1bml0IDAKZGlzayAgICAgICAgc2Qx MDcgICBhdCBzY2J1czYgIHRhcmdldCAxMSB1bml0IDAKZGlzayAgICAgICAgc2QxMDggICBhdCBz Y2J1czYgIHRhcmdldCAxMiB1bml0IDAKZGlzayAgICAgICAgc2QxMDkgICBhdCBzY2J1czYgIHRh cmdldCAxMyB1bml0IDAKZGlzayAgICAgICAgc2QxMTAgICBhdCBzY2J1czYgIHRhcmdldCAxNCB1 bml0IDAKZGlzayAgICAgICAgc2QxMTEgICBhdCBzY2J1czYgIHRhcmdldCAxNSB1bml0IDAKCmNv bnRyb2xsZXIJYWhjMApvcHRpb25zICAgICAgICAgQUhDX1RBR0VOQUJMRQpvcHRpb25zICAgICAg ICAgQUhDX1NDQlBBR0lOR19FTkFCTEUKb3B0aW9ucwkJQUhDX0FMTE9XX01FTUlPCgpjb250cm9s bGVyIHNjYnVzOCBhdCBhaGMwCgojIE9uIGFuIEFkYXB0ZWMgeW91IGNhbiBoYXZlIGJvdGggYSBD RCBvciBhIFdPUk0sIGJ1dCBubyBtdWx0aS1pbml0aWF0b3JzCmRldmljZSAgICAgd29ybTggYXQg c2NidXM2IHRhcmdldCAgNCB1bml0IDAKdGFwZSAgICAgICBzdDggICBhdCBzY2J1czYgdGFyZ2V0 ICA1IHVuaXQgMApkZXZpY2UgICAgIGNkOCAgIGF0IHNjYnVzNiB0YXJnZXQgIDYgdW5pdCAwCgpk aXNrICAgICAgIHNkMTI4IGF0IHNjYnVzOCB0YXJnZXQgIDAgdW5pdCAwCmRpc2sgICAgICAgc2Qx MjkgYXQgc2NidXM4IHRhcmdldCAgMSB1bml0IDAKZGlzayAgICAgICBzZDEzMCBhdCBzY2J1czgg dGFyZ2V0ICAyIHVuaXQgMApkaXNrICAgICAgIHNkMTMxIGF0IHNjYnVzOCB0YXJnZXQgIDMgdW5p dCAwCmRpc2sgICAgICAgc2QxMzIgYXQgc2NidXM4IHRhcmdldCAgNCB1bml0IDAKZGlzayAgICAg ICBzZDEzMyBhdCBzY2J1czggdGFyZ2V0ICA1IHVuaXQgMApkaXNrICAgICAgIHNkMTM0IGF0IHNj YnVzOCB0YXJnZXQgIDYgdW5pdCAwCmRpc2sgICAgICAgc2QxMzYgYXQgc2NidXM4IHRhcmdldCAg OCB1bml0IDAKZGlzayAgICAgICBzZDEzNyBhdCBzY2J1czggdGFyZ2V0ICA5IHVuaXQgMApkaXNr ICAgICAgIHNkMTM4IGF0IHNjYnVzOCB0YXJnZXQgMTAgdW5pdCAwCmRpc2sgICAgICAgc2QxMzkg YXQgc2NidXM4IHRhcmdldCAxMSB1bml0IDAKZGlzayAgICAgICBzZDE0MCBhdCBzY2J1czggdGFy Z2V0IDEyIHVuaXQgMApkaXNrICAgICAgIHNkMTQxIGF0IHNjYnVzOCB0YXJnZXQgMTMgdW5pdCAw CmRpc2sgICAgICAgc2QxNDIgYXQgc2NidXM4IHRhcmdldCAxNCB1bml0IDAKZGlzayAgICAgICBz ZDE0MyBhdCBzY2J1czggdGFyZ2V0IDE1IHVuaXQgMAoKZGV2aWNlCQlzYzAgIGF0IGlzYT8gcG9y dCAiSU9fS0JEIiB0dHkgaXJxIDEgdmVjdG9yIHNjaW50cgpkZXZpY2UJCXBzbTAgYXQgaXNhPyBw b3J0ICJJT19LQkQiIGNvbmZsaWN0cyB0dHkgaXJxIDEyIHZlY3RvciBwc21pbnRyCgpkZXZpY2UJ CW5weDAJYXQgaXNhPyBwb3J0ICJJT19OUFgiIGlycSAxMyB2ZWN0b3IgbnB4aW50cgoKb3B0aW9u cwkJQ09OU1BFRUQ9OTYwMAkJI2RlZmF1bHQgc3BlZWQgZm9yIHNlcmlhbCBjb25zb2xlIChkZWZh dWx0IDk2MDApCmRldmljZQkJc2lvMCBhdCBpc2E/IHBvcnQgIklPX0NPTTEiIGZsYWdzIDB4MTAg dHR5IGlycSA0IHZlY3RvciBzaW9pbnRyCmRldmljZQkJc2lvMSBhdCBpc2E/IHBvcnQgIklPX0NP TTIiIHR0eSBpcnEgMyB2ZWN0b3Igc2lvaW50cgpkZXZpY2UJCXNpbzIJYXQgaXNhPyBwb3J0ICJJ T19DT00zIiB0dHkgaXJxIDUgdmVjdG9yIHNpb2ludHIKZGV2aWNlCQlzaW8zCWF0IGlzYT8gcG9y dCAiSU9fQ09NNCIgdHR5IGlycSA5IHZlY3RvciBzaW9pbnRyCgpkZXZpY2UJCWxwdDAJYXQgaXNh PyBwb3J0PyB0dHkgaXJxIDcgdmVjdG9yIGxwdGludHIKCiMgT3JkZXIgaXMgaW1wb3J0YW50IGhl cmUgZHVlIHRvIGludHJ1c2l2ZSBwcm9iZXMsIGRvICpub3QqIGFscGhhYmV0aXplLgpkZXZpY2UJ CWRlMApkZXZpY2UJCWZ4cDAKZGV2aWNlCQlmcGEwCgpkZXZpY2UJCWVkMCBhdCBpc2E/IHBvcnQg MHgzMDAgbmV0IGlycSAxMCBpb21lbSAweGQ4MDAwIHZlY3RvciBlZGludHIKCnBzZXVkby1kZXZp Y2UJbG9vcApwc2V1ZG8tZGV2aWNlCWV0aGVyCnBzZXVkby1kZXZpY2UJZmRkaQpwc2V1ZG8tZGV2 aWNlCXBwcAkJOApwc2V1ZG8tZGV2aWNlCWJwZmlsdGVyCTE2CnBzZXVkby1kZXZpY2UJdHVuCQk0 CnBzZXVkby1kZXZpY2UJcHR5CQkxNgpwc2V1ZG8tZGV2aWNlCWd6aXAKCm9wdGlvbnMJCU1ST1VU SU5HCQkjIE11bHRpY2FzdCByb3V0aW5nCm9wdGlvbnMJCUlQRklSRVdBTEwgICAgICAgICAgICAg ICNmaXJld2FsbApvcHRpb25zCQlJUEZJUkVXQUxMX1ZFUkJPU0UgICAgICAjcHJpbnQgaW5mb3Jt YXRpb24gYWJvdXQKCQkJCQkjIGRyb3BwZWQgcGFja2V0cwpvcHRpb25zCQlJUERJVkVSVAkJI2Rp dmVydCBzb2NrZXRzCgpvcHRpb25zCQlTWVNWU0hNCm9wdGlvbnMJCVNZU1ZTRU0Kb3B0aW9ucwkJ U1lTVk1TRwoKI3BzZXVkby1kZXZpY2UJZGxtCiNvcHRpb25zCQlETE1fTUFYX0NIQU5ORUxTPTQK I29wdGlvbnMJCURMTV9MT0NLU19QRVJfQ0hBTk5FTD00MDk2CiNvcHRpb25zCQlETE1fU1RBVEVf QklUUz04CiNvcHRpb25zCQlETE1fTUFYX0xPQ0tFUlM9MTI4CiNvcHRpb25zCQlETE1fS0VZX0xF Tj0xNgoKcHNldWRvLWRldmljZQlzcHBwCnBzZXVkby1kZXZpY2UJZGlzYwpwc2V1ZG8tZGV2aWNl CXNwZWFrZXIKcHNldWRvLWRldmljZQl2bgkJI1Zub2RlIGRyaXZlciAodHVybnMgYSBmaWxlIGlu dG8gYSBkZXZpY2UpCnBzZXVkby1kZXZpY2UJc25wCTMJI1Nub29wIGRldmljZSAtIHRvIGxvb2sg YXQgcHR5L3Z0eS9ldGMuLgpwc2V1ZG8tZGV2aWNlCWNjZAk2NAkjQ29uY2F0ZW5hdGVkIGRpc2sg ZHJpdmVyCgpvcHRpb25zCQlYU0VSVkVSCQkJIyBpbmNsdWRlIGNvZGUgZm9yIFhGcmVlODYKb3B0 aW9ucwkJTUFYQ09OUz0xMgkJIyBudW1iZXIgb2YgdmlydHVhbCBjb25zb2xlcwpvcHRpb25zCQlD T01QQVRfTElOVVgKb3B0aW9ucwkJIkVYVDJGUyIKb3B0aW9ucwkJIklCQ1MyIgoKb3B0aW9ucwkJ U0hPV19CVVNZQlVGUwoKY29udHJvbGxlcglzbmQwCmRldmljZQlzYjAJYXQgaXNhPyBwb3J0IDB4 MjIwIGlycSA3IGRycSAxIHZlY3RvciBzYmludHIKZGV2aWNlCXNieHZpMAlhdCBpc2E/IGRycSA1 CmRldmljZQlzYm1pZGkwCWF0IGlzYT8gcG9ydCAweDMwMApkZXZpY2Ugb3BsMCAgICAgYXQgaXNh PyBwb3J0IDB4Mzg4Cg== --_=XFMail.1.3.p0.FreeBSD:980619162058:3015=_-- End of MIME message To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-database" in the body of the message From owner-freebsd-database Sat Jun 20 10:06:08 1998 Return-Path: Received: (from majordom@localhost) by hub.freebsd.org (8.8.8/8.8.8) id KAA10726 for freebsd-database-outgoing; Sat, 20 Jun 1998 10:06:08 -0700 (PDT) (envelope-from owner-freebsd-database@FreeBSD.ORG) Received: from misery.sdf.com (misery.sdf.com [204.244.213.32]) by hub.freebsd.org (8.8.8/8.8.8) with SMTP id KAA10721 for ; Sat, 20 Jun 1998 10:06:04 -0700 (PDT) (envelope-from tom@sdf.com) Received: from tom by misery.sdf.com with smtp (Exim 1.82 #3) id 0ynQZx-0006vI-00; Sat, 20 Jun 1998 09:33:25 -0700 Date: Sat, 20 Jun 1998 09:33:24 -0700 (PDT) From: Tom To: Steven Yang cc: "'freebsd-database@FreeBSD.org'" Subject: Re: gdbm scalability question In-Reply-To: <839A86AB6CE4D111A52200104B938D435B41@MOE> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-database@FreeBSD.ORG Precedence: bulk On Fri, 19 Jun 1998, Steven Yang wrote: > It appears that there is a per-process limitation of 16MB of RAM with a > generic kernel. But even if I were to get around that per-process It is not the kernel. You need to put your user in a class with a larger datasize. See /etc/login.conf > thing, I wouldn't be able to generate a 2GB database file with only > 128MB of RAM given what I am doing. Does gdbm have to load the entire > database into RAM in order to insert a record? Is there a setting I > have to change? I'd use the BSD db routines as they are much better, even the 1.85 stuff used by default in FreeBSD. The db 2.x library has even more improvements. Tom To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-database" in the body of the message