Date: Fri, 20 Aug 2010 12:00:36 +0200 From: Ivan Voras <ivoras@freebsd.org> To: Garrett Cooper <gcooper@freebsd.org> Cc: bapt@freebsd.org, Florent Thoumie <flz@freebsd.org>, Julien Laffaye <jlaffaye@freebsd.org>, David Forsythe <dforsyth@freebsd.org>, Tim Kientzle <kientzle@freebsd.org>, freebsd-ports@freebsd.org Subject: Re: what next for the pkg_install rewrite Message-ID: <AANLkTi=F_sbOWXkXkqAZ00xu9E44m8gPhD39kGet01vu@mail.gmail.com> In-Reply-To: <AANLkTinn4utHttMPsCN1GKp1dTPOdTJAtLgG3m8BfLgw@mail.gmail.com> References: <20100819143830.GJ35140@azathoth.lan> <AANLkTimY=FJas-oXkWwO07QtaD%2BGrLockgJ_SZQJ7UHM@mail.gmail.com> <AANLkTimhh2vOtXUb-frzWcZmANWyEC7oPtTgepzvOtSB@mail.gmail.com> <4C6DA0FA.7080203@DataIX.net> <AANLkTi=h_GdAFbZ2X0agCAtyLLiwNrMoLx_ZunhBBx2=@mail.gmail.com> <AANLkTinn4utHttMPsCN1GKp1dTPOdTJAtLgG3m8BfLgw@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On 20/08/2010, Garrett Cooper <gcooper@freebsd.org> wrote: > 1. SQLite was killed before because of complexity and because it was > needs another package in base that isn't BSD licensed. That's why And... both ideas are completely wrong. SQLite can be imported as a single C file + header, which you must agree is practically the optimum, and its license is "public domain" which is, if anything, "freer" than BSDL and eminently compatible with it. > everyone in the know has been pushing for BDB 1.8x (in base). People BDB in base offer exactly one (1) single "feature", if you can call it that: storing and retrieving key-value binary blobs. It has no practical concurrency support, it's locking is laughable and upgrading data stored as binary blobs is even more nightmarish than maintaining the current plist format (and it will lead to similar uglyness soon - rather than upgrading the data piece called "x", I'm sure developers will introduce new keys called "x_extdata", "x_moredata" etc etc). SQLite on the other hand solves all these in the following way: 1) stores proper records, not blobs 2) has proper locking & concurrency support, ACID 3) the database schema can be modified on the fly for upgrading - fields can be added to existing table while preserving data. 4) is endian-agnostic, 32/64-bit agnostic and portable (I mean the database file) to an extremely large number of platforms. It is already used as a system database in OS X, iPhone and Android. Note that I'm promoting SQLite to replace the /var/db/pkg/* tree of directories and files with ad-hoc structure - all data in there should be in one SQLite database, which is stored in a single file. > suggested that to me back when I was trying to get off the ground in > GSoC 2006, they did it to you before Ivan, and I'm sure they've done > it before in the past. We need to implement things in terms of BDB > first, but make the APIs generic enough so that it _could_ be extended > to SQLite if people chose to do the work later on. Not planning an API for a transactional database in the first place will bring the whole thing down in the long term, and in any case will certainly push things 10 more years in the future. Note that SQLite can *not* be considered a drop-in replacement for BDB (any version of BDB) because in that case you will gain far less than is optimal. If you haven't used SQLite yet, please try it, from C, and then see if you can reevaluate your comment on its complexity. If you don't like SQL, this is also fine. You are not out of the game, there are many other parts to work on. As for backward compatibility: basically it can be done in two ways: 1) build a one-time converter that will read the present plist database and output a new database or 2) build a compatibility layer which would support both the old and the new format at the same time, so people upgrading will continue to use their old data. I consider the latter wasteful in terms of developer resources and because bit-rot will eventually make support for the old format decrepit. > 2. XML is a bad idea. Great in theory, wonderful in my browser, but a > bloated plaintext file with a lot of complexity. I would prefer a > database solution that could be dumped to a simple text file format if > needed. XML, at least in my proposal, would not be used for the system package database, but would replace (either in part or all with a single file) today's "+" files in the packages, together with other purposes where some metadata transfer is required. That said, I would also be happy with other self-described formats like JSON. XML is the front runner because it's more standard (industry standard, whether someone likes this fact or not!) and there is already a parser in base. > Again (and I can't overemphasize this): no matter what we do, say, > etc, unless we have buy-in from portmgr beforehand on anything here, > the work will never see a ports/src CVS/svn repo, etc. This includes > work done for past GSoCs, and current GSoCs. With all respect to portmgr, I expect it to keep a cool head and acknowledge the following: 1) Decisions must be made on objective terms which include consideration for clean / elegant implementation, long-term maintainability and standards. If there is to be a rewrite, it must be built to stand the test of next 10 years of usage, and not start compromising even before it is started. 2) Except in extreme cases, portmgr should decide based on functionality and not have that much say in the specifics of the implementation. Basically, the portmgr should in the first place approve the feature spec, and the src people should worry about the details of what can and cannot be done. This is not a src vs portmgr war, this is separation of duty. (of course there are overlaps in people's memberships)
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?AANLkTi=F_sbOWXkXkqAZ00xu9E44m8gPhD39kGet01vu>