From owner-freebsd-hackers@FreeBSD.ORG Sat May 12 18:58:46 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 2DCB916A405 for ; Sat, 12 May 2007 18:58:46 +0000 (UTC) (envelope-from xistence@0x58.com) Received: from mailexchange.osnn.net (1e.66.5646.static.theplanet.com [70.86.102.30]) by mx1.freebsd.org (Postfix) with SMTP id E75D013C45E for ; Sat, 12 May 2007 18:58:45 +0000 (UTC) (envelope-from xistence@0x58.com) Received: (qmail 13932 invoked by uid 0); 12 May 2007 18:28:18 -0000 Received: from unknown (HELO ?10.10.10.22?) (xistence@0x58.com@72.208.132.56) by mailexchange.osnn.net with SMTP; 12 May 2007 18:28:18 -0000 Mime-Version: 1.0 (Apple Message framework v752.3) In-Reply-To: <4645AFAF.7010704@free.fr> References: <200705102105.27271.blackdragon@highveldmail.co.za> <20070512155059.92011d54.stas@FreeBSD.org> <4645AFAF.7010704@free.fr> Content-Type: multipart/signed; micalg=sha1; boundary=Apple-Mail-4-971540728; protocol="application/pkcs7-signature" Message-Id: <8916C4D5-4DB5-49C0-AF8D-07F9FFA0A6E0@0x58.com> From: Bert JW Regeer Date: Sat, 12 May 2007 11:31:59 -0700 To: freebsd-hackers@freebsd.org X-Mailer: Apple Mail (2.752.3) X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Subject: Re: New FreeBSD package system (a.k.a. Daemon Package System (dps)) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 12 May 2007 18:58:46 -0000 --Apple-Mail-4-971540728 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=ISO-8859-1; delsp=yes; format=flowed On May 12, 2007, at 5:14 AM, Philippe Laquet wrote: > Stanislav Sedov a =E9crit : >> On Fri, 11 May 2007 02:10:05 +0200 >> Ivan Voras mentioned: >> >> >>> - I think it's time to give up on using BDB+directory tree full =20 >>> of text >>> files for storing the installed packages database, and I propose =20 >>> all of >>> this be replaced by a single SQLite database. SQLite is public =20 >>> domain >>> (can be slurped into base system), embeddable, stores all data in a >>> single file, lightweight, fast, and can be used to do fancy =20 >>> things such >>> as reporting. >>> >> >> What is the reason to use SQL-based database? You'll perform direct >> queries to database? The packaging system is for ordinal users, =20 >> not sql >> geeks, so they should not have to use sql for managing packages. So a >> simple set of hashes will suffer or needs. I agree with Julian =20 >> that we >> should have a backup of packaging database in plain text format, and >> utility to rebuild it. This way we can always restore the database if >> something goes wrong. Furhtermore, that should not make a great =20 >> impact >> on performance, since we don't have to rebuild it every day. >> > I agree with Stan ;) > > "fast and improved" package utilities uses mainly some indexed =20 > berkeley DB combined with flat files, aren't they? I, and may be =20 > many other FreeBSD users use light systems for efficiency and =20 > eaiser management, if we use some database system it will require =20 > Disk Space, ressources for the DB to run, dependencies and so on... =20= > And we also may be exposed to a "that DB is better" war ;) > SQLite is compiled inside a program, and as such does not require any =20= resources other than one file handle and some CPU time when querying. =20= The file is stored on disk, and requires no separate process to be =20 running to query. Maybe I misunderstood what you were trying to say. =20 SQLite will require less resources than flat text files, since SQLite =20= is a one time open then process, instead of what is currently =20 happening, having to open and close hundreds of files depending on =20 how many ports are installed. With this regard, SQLite is like BDB. =20 Where SQLite uses standards compliant SQL statements to get data. > >> -- >> Stanislav Sedov >> ST4096-RIPE >> > I am able to understand many of the gripes with using a databases, =20 and have to import yet another code base into the FreeBSD base, =20 however as one of the young ones, and knowing sed/awk/grep and SQL, I =20= prefer SQL over having to process hundreds of text files using text =20 processing tools. It saddens me each time I run one of the pkg_* =20 tools that needs to parse the flat file structure since it takes so =20 long. I have friends running Ubuntu and their apt-get returns results =20= much faster. In a world where hard drives are becoming more reliable, and are =20 automatically relocating sectors that go bad, do we really have to =20 worry about database corruption as much? I feel that many of the =20 fears that are being put forward will do harm to a text based =20 "storage" system as well. If one block drops out, it can cause tools =20 to not be able to parse the files. Create a backup copy of the =20 database after each successful transaction? There are ways to battle =20 data corruption. Using BDB is not an real option either. I can not even count the =20 amount of times that the BDB database that portupgrade created has =20 become corrupt because I accidently ran two portupgrades at the same =20 time, or even remembered that I did not want to upgrade something and =20= hit Ctrl+C. The experience I got from running SVN with BDB as the =20 back-end database to store my data, I say no thanks. In that case I =20 would much rather stick with the flat text files than go with a =20 database. Bert JW Regeer= --Apple-Mail-4-971540728--