From owner-freebsd-hackers@FreeBSD.ORG Wed Nov 6 10:39:54 2013 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 47752A81 for ; Wed, 6 Nov 2013 10:39:54 +0000 (UTC) (envelope-from baptiste.daroussin@gmail.com) Received: from mail-wg0-x230.google.com (mail-wg0-x230.google.com [IPv6:2a00:1450:400c:c00::230]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id D41B1231C for ; Wed, 6 Nov 2013 10:39:53 +0000 (UTC) Received: by mail-wg0-f48.google.com with SMTP id b13so4740096wgh.27 for ; Wed, 06 Nov 2013 02:39:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:date:from:to:cc:subject:message-id:references:mime-version :content-type:content-disposition:in-reply-to:user-agent; bh=KQbVP1+iunScqREY4IPXoEYUo8X8JHDZZ6io43ipd7E=; b=HcO9A/sCCCT6H+pFt/KQk3Rtc6nhgk2Df+xoE9MUtNdkc7wMf4594VmtK/I6CPQj+O S5oAVkwUMzdZIEybTeE9poonhGRWSRV5/ZzzcsOwNJfmYBW06oB1E1jnm+glTCX/46cy HLwGrdc7PAgGwF8GnzxPg9N3B/eEdDQph6BXxTWQIcyRslB0DO2uvtS8s8/9RF7sDVzj tCgIo3lD9hwDSGtzqHrJSEoMszOEjUHNul+O6t7dsi4EHel1wYPpJoJBpVU/mZ/X+puo thLlHj8+spf1el7rHiURf8Y5XNZUF3Q5TUU6kMhTjtlKJBnpPSiQtC8wn3fB9hBgpaFd 5D3A== X-Received: by 10.180.73.70 with SMTP id j6mr20287292wiv.47.1383734391868; Wed, 06 Nov 2013 02:39:51 -0800 (PST) Received: from ithaqua.etoilebsd.net (ithaqua.etoilebsd.net. [37.59.37.188]) by mx.google.com with ESMTPSA id qc10sm23058162wic.9.2013.11.06.02.39.50 for (version=TLSv1 cipher=RC4-SHA bits=128/128); Wed, 06 Nov 2013 02:39:50 -0800 (PST) Sender: Baptiste Daroussin Date: Wed, 6 Nov 2013 11:39:48 +0100 From: Baptiste Daroussin To: Atte Peltomaki Subject: Re: Importing netbsd cdb Message-ID: <20131106103947.GI56315@ithaqua.etoilebsd.net> References: <20131027232628.GB74512@ithaqua.etoilebsd.net> <20131105220225.GF37879@caravan.chchile.org> <20131106084550.GA21489@ass.pp.htv.fi> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="TA4f0niHM6tHt3xR" Content-Disposition: inline In-Reply-To: <20131106084550.GA21489@ass.pp.htv.fi> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: hackers@FreeBSD.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Nov 2013 10:39:54 -0000 --TA4f0niHM6tHt3xR Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Wed, Nov 06, 2013 at 10:45:50AM +0200, Atte Peltomaki wrote: > On Tue, Nov 05, 2013 at 11:02:26PM +0100, Jeremie Le Hen wrote: > > On Mon, Oct 28, 2013 at 12:26:28AM +0100, Baptiste Daroussin wrote: > > >=20 > > > Here is a code that imports the cdbrw from netbsd into a new lib/libc= dbrw > > > library, the read part is also added to libc but not exposed. > > >=20 > > > As an example of using that library I also got the service_mkdb patch= es from > > > netbsd that makes it by default emit a cdb database and add a switch = to allow it > > > to still create the old .db database format. > > >=20 > > > in the libc, getservent has been modified to first try to read the .c= db files > > > and fallback on reading the old .db file. (I'm not sure if it is wort= h keeping > > > reading the old db format.) > > >=20 > > > http://people.freebsd.org/~bapt/cdbrw.diff > > >=20 > > > The plan after that is to get pw_util(3) directly generating a cdb fi= le for > > > pwd.db and spwd.db and modifiy pwd_mkdb(8) so that by default it uses= the API > > > from pw_util(3) and have a switch to fallback on creating in bdb form= at. > > >=20 > > > I also plan to do the same for cap_mkdb(1) and getcap(3). > > >=20 > > > With cdb querying is way faster (I don't have number but I can get so= me it > > > needed) > > > the size of the db is also way smaller: > > > 64K /var/db/services.cdb > > > 2,1M /var/db/services.db > > >=20 > > > Any objection? > >=20 > > What are the benefits of this, beside generating a much smaller > > database? Do you know about benchmarks that compare it against the > > current database engine used? It would be unfortunate to notice a > > regression afterward :). >=20 > At least randomly googled third-party benchmarks suggest that CDB is > vastly superior to BDB: > http://www.dmo.ca/blog/benchmarking-hash-databases-on-large-data/ >=20 =46rom the netbsd commit log, it says that in case of services it is at lea= st as slow, and at best faster, from my testing it is way faster, (sorry no numbe= rs) Concerning the migration, the way I did it for now, is that getservent is t= rying to open the cdb database and fallsback on the bdb one, so migrating to 11.0-RELEASE should be ootb. Running: services_mkdb Will generate the cdb version of the db services_mkdb -V db Will generate the old format db. regards, Bapt --TA4f0niHM6tHt3xR Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.19 (FreeBSD) iEYEARECAAYFAlJ6HHMACgkQ8kTtMUmk6Ex53wCfXkcvKJVQD2+UFNYk4TENeH3P Z2MAnjAxndTFcClDUVPYiOFXbQQjfePf =G0I4 -----END PGP SIGNATURE----- --TA4f0niHM6tHt3xR--