From owner-freebsd-hackers@FreeBSD.ORG  Tue May 13 04:20:42 2008
Return-Path: <owner-freebsd-hackers@FreeBSD.ORG>
Delivered-To: freebsd-hackers@freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id 955DD106566B
	for <freebsd-hackers@freebsd.org>; Tue, 13 May 2008 04:20:42 +0000 (UTC)
	(envelope-from kientzle@freebsd.org)
Received: from kientzle.com (h-66-166-149-50.snvacaid.covad.net
	[66.166.149.50])
	by mx1.freebsd.org (Postfix) with ESMTP id 6B0F18FC2C
	for <freebsd-hackers@freebsd.org>; Tue, 13 May 2008 04:20:42 +0000 (UTC)
	(envelope-from kientzle@freebsd.org)
Received: from [10.0.0.128] (p54.kientzle.com [66.166.149.54])
	by kientzle.com (8.12.9/8.12.9) with ESMTP id m4D4Kftv098830;
	Mon, 12 May 2008 21:20:41 -0700 (PDT)
	(envelope-from kientzle@freebsd.org)
Message-ID: <48291719.5010801@freebsd.org>
Date: Mon, 12 May 2008 21:20:41 -0700
From: Tim Kientzle <kientzle@freebsd.org>
User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.7.12) Gecko/20060422
X-Accept-Language: en-us, en
MIME-Version: 1.0
To: Jeremy Lea <reg@freebsd.org>
References: <op.uavxx8ip2n4ijf@duckjen.nextgentel.no>
	<20080512204558.GA44114@flint.openpave.org>
In-Reply-To: <20080512204558.GA44114@flint.openpave.org>
Content-Type: text/plain; charset=us-ascii; format=flowed
Content-Transfer-Encoding: 7bit
Cc: freebsd-hackers@freebsd.org, Anders Nore <andenore@start.no>
Subject: Re: Adding .db support to pkg_tools
X-BeenThere: freebsd-hackers@freebsd.org
X-Mailman-Version: 2.1.5
Precedence: list
List-Id: Technical Discussions relating to FreeBSD
	<freebsd-hackers.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-hackers>, 
	<mailto:freebsd-hackers-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-hackers>
List-Post: <mailto:freebsd-hackers@freebsd.org>
List-Help: <mailto:freebsd-hackers-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-hackers>,
	<mailto:freebsd-hackers-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Tue, 13 May 2008 04:20:42 -0000

> I think this is a really bad idea.  The problem with the tools is not
> with the files.  It is that the files need to be parsed on each run,
> often recursively, and your solution would not help at all.

Parsing one file isn't expensive; parsing several hundred files
to find one bit of information is expensive.

> The database(s) should just be a cache of the information stored in the
> files.

Bingo!  As long as the .db version can be easily recreated
from scratch from the master data stored in the same files
as always, it doesn't really matter if the BDB is occasionally
corrupted, as long as it can be rebuilt fairly quickly.

So, the key points are:
   * Use the .db file for lookups that can't be done quickly
against the existing data.  In particular, look for operations
where the pkg tools have to scan a lot of files to verify a single
fact.
   * Ensure the .db file can be deleted and rebuilt from scratch
from the data in the regular files.

Tim Kientzle