From owner-freebsd-hackers@FreeBSD.ORG Sun May 6 05:03:50 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 0527216A401 for ; Sun, 6 May 2007 05:03:50 +0000 (UTC) (envelope-from youshi10@u.washington.edu) Received: from mxout1.cac.washington.edu (mxout1.cac.washington.edu [140.142.32.134]) by mx1.freebsd.org (Postfix) with ESMTP id D7A8F13C459 for ; Sun, 6 May 2007 05:03:49 +0000 (UTC) (envelope-from youshi10@u.washington.edu) Received: from smtp.washington.edu (smtp.washington.edu [140.142.32.139]) by mxout1.cac.washington.edu (8.13.7+UW06.06/8.13.7+UW07.03) with ESMTP id l4653mC5015739 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Sat, 5 May 2007 22:03:49 -0700 X-Auth-Received: from [192.168.10.45] (c-67-187-164-17.hsd1.ca.comcast.net [67.187.164.17]) (authenticated authid=youshi10) by smtp.washington.edu (8.13.7+UW06.06/8.13.7+UW07.03) with ESMTP id l4653mtS024078 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NOT); Sat, 5 May 2007 22:03:48 -0700 Message-ID: <463D61DA.9020409@u.washington.edu> Date: Sat, 05 May 2007 22:04:26 -0700 From: Garrett Cooper User-Agent: Thunderbird 2.0.0.0 (Windows/20070326) MIME-Version: 1.0 To: FreeBSD User Giacomo References: <20070505195033.GA30661@eclypse.it> In-Reply-To: <20070505195033.GA30661@eclypse.it> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 8bit X-PMX-Version: 5.3.1.294258, Antispam-Engine: 2.5.1.298604, Antispam-Data: 2007.5.5.214733 X-Uwash-Spam: Gauge=IIIIIII, Probability=7%, Report='__CT 0, __CTE 0, __CT_TEXT_PLAIN 0, __HAS_MSGID 0, __MIME_TEXT_ONLY 0, __MIME_VERSION 0, __SANE_MSGID 0, __STOCK_PHRASE_25 0, __USER_AGENT 0' Cc: freebsd-hackers@freebsd.org Subject: Re: Samsung DVD writer and FreeBSD kernel. X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 06 May 2007 05:03:50 -0000 FreeBSD User Giacomo wrote: > Hi, > Excuse me for a long email. > > I have a problem with my external USB DVD writer (model SE-S184M/EUBN). > I do not succeed in to burn DVD (only CD). > The system (FreeBSD eclypse.it 6.1-RELEASE FreeBSD 6.1-RELEASE #0: Sun May 7 04:32:43 UTC 2006 > root@opus.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC i386) recognizes masterizzatore like: > > ------------------------------------------------------------------------------------------------------- > umass0: TSST corp USB Mass Storage Device, rev 2.00/0.00, addr 3 cd0 at umass-sim0 bus 0 target 0 lun 0 > cd0: device Removable CD-ROM SCSI-0 > cd0: 40.000MB/s transfers > cd0: Device Attempt to query size failed: NOT READY, Medium not present - tray close > > > If I use the comand whith a dvd+rw : dvd+rw- format /dev/cd0 > I obtain > > * DVD±RW/-RAM format utility by , version 6.0. > :-( (unable to GET CONFIGURATION: > Input/output error and on the consul: umass0: > Unsupported ATAPI command 0x46. > ------------------------------------------------------------------------------------------------------- > > 1. I have asked the list "questions" , they advised me to contact the > cdrecord maintainer (Jorg Schilling) for this problem. > 2. He advised to me to upgrade of the program. After an upgrade the > problem persisted. > 3. After He says > > "Can you verify that FreeBSD did not again break the SCSI implementation > and does not do an auto-request Sense? > > Please use "scgcheck" to verify the correctness of your SCSI implementation. > > Jörg" > > My verify is: > ------------------------------------------------------------------------------------------------------- > Scgcheck 2.01 (i386-unknown-freebsd6.1) SCSI user level transport library ABI checker. > Copyright (C) 1998,2001 Jörg Schilling > Checking if your implementation supports to scan the SCSI bus. > Trying to open device: '(NULL POINTER)'. > Using libscg version 'schily-0.9' > Using libscg transport code version 'schily-scsi-bsd.c-1.44' > Using kernel transport code version '' > Using remote transport code version '-' > Max DMA buffer size: 65536 > scsibus1: > 1,0,0 100) 'ATAPI ' 'CD-ROM 52X ' '172A' Removable CD-ROM > 1,1,0 101) 'HL-DT-ST' 'CD-RW GCE-8481B ' '1.00' Removable CD-ROM > 1,2,0 102) * > 1,3,0 103) * > 1,4,0 104) * > 1,5,0 105) * > 1,6,0 106) * > 1,7,0 107) * > scsibus2: > 2,0,0 200) 'TSSTcorp' 'CD/DVDW SH-S182M' 'SB04' Removable CD-ROM > 2,1,0 201) * > 2,2,0 202) * > 2,3,0 203) * > 2,4,0 204) * > 2,5,0 205) * > 2,6,0 206) * > 2,7,0 207) * > ----------> SCSI scan bus test PASSED > For the next test we need to open a single SCSI device. > Best results will be obtained if you specify a modern CD-ROM drive. > Trying to open device: '2,0,0'. > Using libscg version 'schily-0.9' > Using libscg transport code version 'schily-scsi-bsd.c-1.44' > Using kernel transport code version '' > Using remote transport code version '-' > Max DMA buffer size: 65536 > Device type : Removable CD-ROM > Version : 0 > Response Format: 2 > Capabilities : > Vendor_info : 'TSSTcorp' > Identifikation : 'CD/DVDW SH-S182M' > Revision : 'SB04' > First SCSI open OK - device usable > Checking for second SCSI open. > Second SCSI open for same device succeeded, 1 file descriptor(s) used. > Second SCSI open is usable > Closing second SCSI. > Checking first SCSI. > First SCSI open is still usable > Second SCSI open test passed. > > Executing 'inquiry' command on Bus 2 Target 0, Lun 0 timeout 40s > CDB: 12 00 00 00 24 00 > cmd finished after 0.001s timeout 40s > ----------> SCSI succeeded command test PASSED > **********> Testing for failed SCSI command. > Inquiry did not fail. > scgcheck: Input/output error. test unit ready: scsi sendcmd: retryable error > CDB: 00 00 00 00 00 00 > status: 0x2 (CHECK CONDITION) > Sense Bytes: 70 00 02 00 00 00 00 0A 00 00 00 00 3A 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 > Sense Key: 0x2 Not Ready, Segment 0 > Sense Code: 0x3A Qual 0x01 (medium not present - tray closed) Fru 0x0 > Sense flags: Blk 0 (not valid) > cmd finished after 0.002s timeout 40s > ----------> SCSI Transport return != SCG_NO_ERROR (1) > ----------> SCSI failed command test FAILED > **********> Testing for SCSI sense data count. > **********> Testing if at least CCS_SENSE_LEN (18) is supported... > Sense Data: 70 00 02 00 00 00 00 0A 00 00 00 00 3A 01 00 00 00 00 > Sense Data: 70 00 02 00 00 00 00 0A 00 00 00 00 3A 01 00 00 00 00 > ----------> Wanted 18 sense bytes, got it. > ----------> Libscg says 32 sense bytes but got (18) > **********> Testing for 32 bytes of sense data... > Sense Data: 70 00 02 00 00 00 00 0A 00 00 00 00 3A 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 > Sense Data: 70 00 02 00 00 00 00 0A 00 00 00 00 3A 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 > ----------> Wanted 32 sense bytes, got it. > ----------> Got a maximum of 32 sense bytes > ----------> SCSI sense count test FAILED > ----------> SCSI status byte test NOT YET READY > **********> Testing for working DMA residual count. > **********> Testing for working DMA residual count == 0. > ----------> Wanted 36 bytes, got it. > ----------> SCSI DMA residual count == 0 test PASSED > **********> Testing for working DMA residual count == DMA count. > ----------> Wanted 0 bytes, got it. > ----------> SCSI DMA residual count == DMA count test PASSED > **********> Testing for working DMA residual count == 1. > ----------> Wanted 36 bytes, got it. > ----------> SCSI DMA residual count == 1 test PASSED > **********> Testing for working DMA overrun test. > ----------> SCSI DMA overrun test FAILED > ----------> SCSI transport code test NOT YET READY > ------------------------------------------------------------------------------------------------------------ > > 4. And I have obtained this answer: > > ------------------------------------------------------------------------------------------------------------ >> Hi, >> the result of scgcheck command is (much long): > >> ----------> SCSI succeeded command test PASSED >> Ready to start test for failing command? Enter to continue: >> Inquiry did not fail. >> This may be because the firmware in your drive is buggy. >> If the current drive is not a CD-ROM drive please restart >> the test utility. Otherwise remove any medium from the drive. >> Ready to start test for failing command? Enter to continue: >> scgcheck: Input/output error. test unit ready: scsi sendcmd: retryable error >> CDB: 00 00 00 00 00 00 >> status: 0x2 (CHECK CONDITION) >> Sense Bytes: 70 00 02 00 00 00 00 0A 00 00 00 00 3A 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >> Sense Key: 0x2 Not Ready, Segment 0 >> Sense Code: 0x3A Qual 0x01 (medium not present - tray closed) Fru 0x0 >> Sense flags: Blk 0 (not valid) >> cmd finished after 0.002s timeout 40s >> ----------> SCSI Transport return != SCG_NO_ERROR (1) >> ----------> SCSI failed command test FAILED > > It looks like I would need to see whether I could enhance the > libscg adptation layer for freeBSD/CAM. > > But in this case you correctly receive a CHECK CONDITION. > > In order to check whether this layer has more problems I would need > to add debug code that s currently missing. > > Please note that the probability that the problem is inside your kernel > is much higher than a problem in libscg. This is because the SCSI status > byte is directly copied from the FreeBSD Kernel. > > Did you try to send a bug report to the FreeBSD kernel team? > > > > Jörg Please post this as a PR if no one replies back soon with any good ideas on how to solve this. -Garrett From owner-freebsd-hackers@FreeBSD.ORG Sun May 6 08:27:48 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id D093016A401 for ; Sun, 6 May 2007 08:27:48 +0000 (UTC) (envelope-from dds@aueb.gr) Received: from mx-out-01.forthnet.gr (mx-out.forthnet.gr [193.92.150.103]) by mx1.freebsd.org (Postfix) with ESMTP id 4F69213C455 for ; Sun, 6 May 2007 08:27:47 +0000 (UTC) (envelope-from dds@aueb.gr) Received: from mx-av-02.forthnet.gr (mx-av.forthnet.gr [193.92.150.27]) by mx-out-01.forthnet.gr (8.13.8/8.13.8) with ESMTP id l468Rk6V018564; Sun, 6 May 2007 11:27:46 +0300 Received: from MX-IN-02.forthnet.gr (mx-in-02.forthnet.gr [193.92.150.185]) by mx-av-02.forthnet.gr (8.14.1/8.14.1) with ESMTP id l468RkUn021490; Sun, 6 May 2007 11:27:46 +0300 Received: from [192.168.136.18] (ppp124-213.adsl.forthnet.gr [193.92.231.213]) by MX-IN-02.forthnet.gr (8.14.1/8.14.1) with ESMTP id l468Rioj024545; Sun, 6 May 2007 11:27:45 +0300 Authentication-Results: MX-IN-02.forthnet.gr from=dds@aueb.gr; sender-id=neutral; spf=neutral In-Reply-To: References: <463CCE1E.1080906@geri.cc.fer.hr> Mime-Version: 1.0 (Apple Message framework v752.2) Content-Type: text/plain; charset=US-ASCII; delsp=yes; format=flowed Message-Id: <457ABFA4-D793-4FD6-85E4-5CE380FE2CB9@aueb.gr> Content-Transfer-Encoding: 7bit From: Diomidis Spinellis Date: Sun, 6 May 2007 11:27:40 +0300 To: Sonja Milicic X-Mailer: Apple Mail (2.752.2) Cc: freebsd-hackers@freebsd.org Subject: Re: Writing to a file X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 06 May 2007 08:27:48 -0000 On May 5, 2007, at 10:29 PM, Diomidis Spinellis wrote: > On May 5, 2007, at 9:34 PM, Sonja Milicic wrote: >> I'm working on an IO logging utility for FreeBSD as my GSoC >> project, and >> I have some questions about writing a kernel functions that would >> open >> an existing or create a new file (with the file name as a parameter, >> returns a vnode * for the file) and write data to that file (with >> pointer to data as parameter). I've found some functions in existing >> code that do similar things and might help me understand how to >> solve my >> problem, but as there isn't much documentation out there I still >> don't >> understand a lot of things. So, could anyone please give me a >> detailed >> explanation of how to open a file in kernel and write to it - best >> data >> types to use, functions, what to look out for, maybe a link to >> tutorial >> or manual that deals with this (if such a thing exists), etc.? > > A good strategy for dealing with such questions is to look for code > that does a task similar to the one you want to implement. Two > kernel subsystems that come to my mind is the kernel logging > facility, which writes data to a user space process via a socket, > and the process accounting facility, which writes data to an > already opened file. There are reasons (performance, flexibility) > why these two facilities have been designed in this way, and it > would be a good idea to see whether some of their design decisions > are also applicable to your problem. Some clarifications on the above. You can find the kernel-side of the accounting code at /usr/src/sys/kern/kern_acct.c acct opens an existing file for appending; acct_process (look for vn_rdwr) will write to that file. Similarly, you can find the kernel-side of the system log code at / usr/src/sys/kern/subr_prf.c. The userland client, which actually writes the message buffers to files, is at /usr/src/usr.sbin/syslogd. In general, coding in the kernel environment is tricky. You have to be careful with locking, many standard C facilities are missing, and bugs can bring down the entire system. Therefore, it is often better to split complex tasks into two: a simple part in the kernel will communicate with a userland process, where you can put all the complexity. Another example of this pattern is the routing code. Diomidis Spinellis - http://www.spinellis.gr From owner-freebsd-hackers@FreeBSD.ORG Sun May 6 08:34:40 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 6F7B116A403; Sun, 6 May 2007 08:34:40 +0000 (UTC) (envelope-from danny@cs.huji.ac.il) Received: from cs1.cs.huji.ac.il (cs1.cs.huji.ac.il [132.65.16.10]) by mx1.freebsd.org (Postfix) with ESMTP id 11A7313C468; Sun, 6 May 2007 08:34:40 +0000 (UTC) (envelope-from danny@cs.huji.ac.il) Received: from pampa.cs.huji.ac.il ([132.65.80.32]) by cs1.cs.huji.ac.il with esmtp id 1HkcCT-0004uW-76; Sun, 06 May 2007 11:34:37 +0300 X-Mailer: exmh version 2.7.2 01/07/2005 with nmh-1.2 To: Daichi GOTO Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Sun, 06 May 2007 11:34:35 +0300 From: Danny Braniss Message-ID: Cc: Stanislav Sedov , Masanori OZAWA , Craig Rodrigues , FreeBSD Hackers , Ed Schouten , freebsd-fs@freebsd.org, FreeBSD Current , Kris Kennaway Subject: Re: [ANN] unionfs patchset-19-20070504 release, it is now MPSAFE and transparent mode as default X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 06 May 2007 08:34:40 -0000 > Hi Guys > > It is my pleasure and honor to announce the availability of > the unionfs patchset-19-20070504. p19 is second patchset after > its merged of FreeBSD. Our improvements works of unionfs are > going step by step. p19 is milestone release. > > Patchset-19-20070504: > For 7-current > http://people.freebsd.org/~daichi/unionfs/unionfs-p19-20070504.diff > > For 6-stable > http://people.freebsd.org/~daichi/unionfs/unionfs6-p19-20070504.diff > > Changes in unionfs-p19-20070504.diff > - It has been became MPSAFE. > - Default copy mode has been changed from traditional-mode to > transparent-mode. Some folks who have reported some issues > have solved with transparent mode. We guess it is time to > change the default copy mode. The transparent-mode is the > best in most situations. > - Fixed kern/111262 issue. > - Added support of vfs_cache on unionfs. As a result, you > can use applications that use procfs on unionfs. > - Removed unionfs internal cache mechanism because it has > vfs_cache support instead. As a result, it just simplified > code of unionfs. > - Added whiteout behavior option. ``-o whiteout=always'' is > default mode(it is established practice) and > ``-o whiteout=whenneeded'' is less disk-space using mode > especially for resource restricted environments like embedded > environments. (Contributed by Ed Schouten. Thanks) > - Fixed a mtx lock issue happened with nullfs. > - Fixed lock issues around unionfs. > - Added NULL check code pointed out by Coverity. (Pointed out > by Stanislav Sedov. Thanks) > > The documents of those unionfs patches: > > http://people.freebsd.org/~daichi/unionfs/ (English) > http://people.freebsd.org/~daichi/unionfs/index-ja.html (Japanese) > > Request for Test: > > Unionfs lovers including FreeSBIE developers, ports cluster managers, > heavy memory-fs users and folks use unionfs, could you try p19 please? > > Merge plan: > > I have plan to commit unionfs-p19-20070504.diff to -current after > received unionfs users responses. > > Thanks > > P.S. > > I am going to join BSDCan 2007. Lets meet at Ottawa, Canada :) > > -- > Daichi GOTO, http://people.freebsd.org/~daichi Hi, So far I've tested it under -current, in my diskless env. where /etc & /compat/linux are unionfs'ed with a mfs, and so all seems OK. thanks, danny From owner-freebsd-hackers@FreeBSD.ORG Sun May 6 09:53:29 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 6842616A403 for ; Sun, 6 May 2007 09:53:29 +0000 (UTC) (envelope-from joseph.koshy@gmail.com) Received: from nz-out-0506.google.com (nz-out-0506.google.com [64.233.162.229]) by mx1.freebsd.org (Postfix) with ESMTP id 2957B13C4B7 for ; Sun, 6 May 2007 09:53:28 +0000 (UTC) (envelope-from joseph.koshy@gmail.com) Received: by nz-out-0506.google.com with SMTP id s1so1338959nze for ; Sun, 06 May 2007 02:53:28 -0700 (PDT) DKIM-Signature: a=rsa-sha1; c=relaxed/relaxed; d=gmail.com; s=beta; h=domainkey-signature:received:received:message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; b=XjVpzdPYNWJkLe0zjAdG4Oj16eU5KJfGT/yumNkgvCxiu3BzgCT/9evMJZv0ng8YKekA3ukWdoOnzaXprv/niJVAUKqF4f2mWErP3CQTBB+wuir3hNnCdaf4uJ9Ew25cSazNwqzE5XUsVDL+GDTv6msg86EIui9BU5eMbvIqjtw= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=beta; h=received:message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; b=XnBZd4k9pTDHx7hPHf1THpTvwS3pXraS8g/oCGUAqFLF2pstlWJm/ZpnvHcbHiA/n1J42DTAn026crKUI2KDGGdbQGzdwSznHIks/wAu0SkLPrAHX8gIOlWRNk9pK+y7oHV/k86QDVIZwivy106zorzKL5MgceyxKXd/qqJ167c= Received: by 10.65.248.19 with SMTP id a19mr8197043qbs.1178443698427; Sun, 06 May 2007 02:28:18 -0700 (PDT) Received: by 10.65.81.13 with HTTP; Sun, 6 May 2007 02:28:18 -0700 (PDT) Message-ID: <84dead720705060228t2c79529cr27a90e34bbbb3e7a@mail.gmail.com> Date: Sun, 6 May 2007 09:28:18 +0000 From: "Joseph Koshy" To: "Sonja Milicic" In-Reply-To: <463CCE1E.1080906@geri.cc.fer.hr> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <463CCE1E.1080906@geri.cc.fer.hr> Cc: freebsd-hackers@freebsd.org Subject: Re: Writing to a file X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 06 May 2007 09:53:29 -0000 > So, could anyone please give me a detailed > explanation of how to open a file in kernel and write to it - best data > types to use, functions, what to look out for, maybe a link to tutorial > or manual that deals with this (if such a thing exists), etc.? I'm not aware of such a tutorial. I would suggest that you start with the source for alq_open() in "/sys/kern/kern_alq.c", looking up the manual pages for all the functions it calls as needed. -- FreeBSD Volunteer, http://people.freebsd.org/~jkoshy From owner-freebsd-hackers@FreeBSD.ORG Sun May 6 11:04:25 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 4FD5E16A401 for ; Sun, 6 May 2007 11:04:25 +0000 (UTC) (envelope-from hselasky@c2i.net) Received: from swip.net (mailfe07.swip.net [212.247.154.193]) by mx1.freebsd.org (Postfix) with ESMTP id E317613C44B for ; Sun, 6 May 2007 11:04:24 +0000 (UTC) (envelope-from hselasky@c2i.net) X-Cloudmark-Score: 0.000000 [] Received: from [81.191.58.152] (account mc467741@c2i.net HELO laptop.lan) by mailfe07.swip.net (CommuniGate Pro SMTP 5.1.7) with ESMTPA id 483116387 for freebsd-hackers@freebsd.org; Sun, 06 May 2007 13:04:20 +0200 From: Hans Petter Selasky To: freebsd-hackers@freebsd.org User-Agent: KMail/1.9.5 MIME-Version: 1.0 Content-Disposition: inline Date: Sun, 6 May 2007 13:04:04 +0200 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Message-Id: <200705061304.05093.hselasky@c2i.net> Subject: Missing LIST_PREV() ? X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 06 May 2007 11:04:25 -0000 Hi, Why should LISTs only be forward traversable? The following piece of code make lists backward traversable: /sys/sys/queue.h: +#define LIST_PREV(head,elm,field) \ + (((elm) == LIST_FIRST(head)) ? ((__typeof(elm))0) : \ + ((__typeof(elm))(((uint8_t *)((elm)->field.le_prev)) - \ + ((uint8_t *)&LIST_NEXT((__typeof(elm))0,field))))) Any comments? --HPS From owner-freebsd-hackers@FreeBSD.ORG Sun May 6 09:30:41 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id B5D5916A404; Sun, 6 May 2007 09:30:41 +0000 (UTC) (envelope-from rdivacky@vlk.vlakno.cz) Received: from vlakno.cz (vlk.vlakno.cz [62.168.28.247]) by mx1.freebsd.org (Postfix) with ESMTP id 634BB13C480; Sun, 6 May 2007 09:30:41 +0000 (UTC) (envelope-from rdivacky@vlk.vlakno.cz) Received: from localhost (localhost [127.0.0.1]) by vlakno.cz (Postfix) with ESMTP id 6C16F8BD3D9; Sun, 6 May 2007 11:30:39 +0200 (CEST) X-Virus-Scanned: amavisd-new at vlakno.cz Received: from vlakno.cz ([127.0.0.1]) by localhost (vlk.vlakno.cz [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id kRzYUAzVx3NI; Sun, 6 May 2007 11:30:37 +0200 (CEST) Received: from vlk.vlakno.cz (localhost [127.0.0.1]) by vlakno.cz (Postfix) with ESMTP id BB2408BD3AD; Sun, 6 May 2007 11:30:37 +0200 (CEST) Received: (from rdivacky@localhost) by vlk.vlakno.cz (8.13.8/8.13.8/Submit) id l469UYbV043005; Sun, 6 May 2007 11:30:34 +0200 (CEST) (envelope-from rdivacky) Date: Sun, 6 May 2007 11:30:34 +0200 From: Roman Divacky To: Danny Braniss Message-ID: <20070506093034.GA42976@freebsd.org> References: Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.4.2.2i X-Mailman-Approved-At: Sun, 06 May 2007 11:27:36 +0000 Cc: Craig Rodrigues , Masanori OZAWA , FreeBSD Hackers , Daichi GOTO , Ed Schouten , Stanislav Sedov , FreeBSD Current , freebsd-fs@freebsd.org, Kris Kennaway Subject: Re: [ANN] unionfs patchset-19-20070504 release, it is now MPSAFE and transparent mode as default X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 06 May 2007 09:30:42 -0000 > Hi, > So far I've tested it under -current, in my diskless env. where > /etc & /compat/linux are unionfs'ed with a mfs, and so all seems OK. there's an XXX in the linuxulator code saying: XXX Untested vs. mount -o union; probably does the wrong thing. can you confirm that it works ok with unionfs from daichi? thnx roman From owner-freebsd-hackers@FreeBSD.ORG Sun May 6 09:48:25 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id D819716A403; Sun, 6 May 2007 09:48:25 +0000 (UTC) (envelope-from danny@cs.huji.ac.il) Received: from cs1.cs.huji.ac.il (cs1.cs.huji.ac.il [132.65.16.10]) by mx1.freebsd.org (Postfix) with ESMTP id 872FE13C487; Sun, 6 May 2007 09:48:25 +0000 (UTC) (envelope-from danny@cs.huji.ac.il) Received: from pampa.cs.huji.ac.il ([132.65.80.32]) by cs1.cs.huji.ac.il with esmtp id 1HkdLr-000C5C-Aq; Sun, 06 May 2007 12:48:23 +0300 X-Mailer: exmh version 2.7.2 01/07/2005 with nmh-1.2 To: Roman Divacky In-reply-to: <20070506093034.GA42976@freebsd.org> References: <20070506093034.GA42976@freebsd.org> Comments: In-reply-to Roman Divacky message dated "Sun, 06 May 2007 11:30:34 +0200." Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Sun, 06 May 2007 12:48:21 +0300 From: Danny Braniss Message-ID: X-Mailman-Approved-At: Sun, 06 May 2007 11:27:42 +0000 Cc: Craig Rodrigues , Masanori OZAWA , FreeBSD Hackers , Daichi GOTO , Ed Schouten , Stanislav Sedov , FreeBSD Current , freebsd-fs@freebsd.org, Kris Kennaway Subject: Re: [ANN] unionfs patchset-19-20070504 release, it is now MPSAFE and transparent mode as default X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 06 May 2007 09:48:26 -0000 > > Hi, > > So far I've tested it under -current, in my diskless env. where > > /etc & /compat/linux are unionfs'ed with a mfs, and so all seems OK. > > there's an XXX in the linuxulator code saying: > > XXX Untested vs. mount -o union; probably does the wrong thing. > > can you confirm that it works ok with unionfs from daichi? > sure, but what/how do you want me to test? danny From owner-freebsd-hackers@FreeBSD.ORG Sun May 6 11:40:17 2007 Return-Path: X-Original-To: freebsd-hackers@FreeBSD.ORG Delivered-To: freebsd-hackers@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 313D216A402 for ; Sun, 6 May 2007 11:40:17 +0000 (UTC) (envelope-from mark@grondar.org) Received: from storm.uk.FreeBSD.org (storm.uk.FreeBSD.org [194.242.157.42]) by mx1.freebsd.org (Postfix) with ESMTP id C5F9513C46C for ; Sun, 6 May 2007 11:40:16 +0000 (UTC) (envelope-from mark@grondar.org) Received: from storm.uk.FreeBSD.org (uucp@localhost [127.0.0.1]) by storm.uk.FreeBSD.org (8.13.8/8.13.8) with ESMTP id l46BeBD1039124; Sun, 6 May 2007 12:40:12 +0100 (BST) (envelope-from mark@grondar.org) Received: (from uucp@localhost) by storm.uk.FreeBSD.org (8.13.8/8.12.11/Submit) with UUCP id l46BeBwD039123; Sun, 6 May 2007 12:40:11 +0100 (BST) (envelope-from mark@grondar.org) Received: from greatest.grondar.org (localhost [127.0.0.1]) by greatest.grondar.org (8.13.8/8.13.8) with ESMTP id l46BdssN017108; Sun, 6 May 2007 13:39:55 +0200 (SAST) (envelope-from mark@greatest.grondar.org) Message-Id: <200705061139.l46BdssN017108@greatest.grondar.org> X-Mailer: exmh version 2.7.2 01/07/2005 with nmh-1.2 To: Hans Petter Selasky In-Reply-To: Message from Hans Petter Selasky of "Sun, 06 May 2007 13:04:04 +0200." <200705061304.05093.hselasky@c2i.net> From: Mark Murray From: Mark Murray Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Sun, 06 May 2007 12:39:53 +0100 Sender: mark@grondar.org Cc: freebsd-hackers@FreeBSD.ORG Subject: Re: Missing LIST_PREV() ? X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 06 May 2007 11:40:17 -0000 Hans Petter Selasky writes: > Hi, > > Why should LISTs only be forward traversable? The following piece of > code make lists backward traversable: No objection to the concept. But... > /sys/sys/queue.h: > > +#define LIST_PREV(head,elm,field) \ > + (((elm) == LIST_FIRST(head)) ? ((__typeof(elm))0) : \ > + ((__typeof(elm))(((uint8_t *)((elm)->field.le_prev)) - \ > + ((uint8_t *)&LIST_NEXT((__typeof(elm))0,field))))) Please don't use typeof; it is a GCCism. Do you really mean NULL? M -- Mark R V Murray - Cert APS(Open) Dip Phys(Open) BSc Open(Open) From owner-freebsd-hackers@FreeBSD.ORG Sun May 6 11:57:08 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id AC8C816A402 for ; Sun, 6 May 2007 11:57:08 +0000 (UTC) (envelope-from hselasky@c2i.net) Received: from swip.net (mailfe07.swip.net [212.247.154.193]) by mx1.freebsd.org (Postfix) with ESMTP id 49A1013C44B for ; Sun, 6 May 2007 11:57:08 +0000 (UTC) (envelope-from hselasky@c2i.net) X-Cloudmark-Score: 0.000000 [] Received: from [81.191.58.152] (account mc467741@c2i.net HELO laptop.lan) by mailfe07.swip.net (CommuniGate Pro SMTP 5.1.7) with ESMTPA id 483141486; Sun, 06 May 2007 13:57:06 +0200 From: Hans Petter Selasky To: Mark Murray Date: Sun, 6 May 2007 13:56:50 +0200 User-Agent: KMail/1.9.5 References: <200705061139.l46BdssN017108@greatest.grondar.org> In-Reply-To: <200705061139.l46BdssN017108@greatest.grondar.org> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200705061356.50911.hselasky@c2i.net> Cc: freebsd-hackers@freebsd.org Subject: Re: Missing LIST_PREV() ? X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 06 May 2007 11:57:08 -0000 On Sunday 06 May 2007 13:39, Mark Murray wrote: > Hans Petter Selasky writes: > > Hi, > > > > Why should LISTs only be forward traversable? The following piece of > > code make lists backward traversable: > > No objection to the concept. > > But... > > > /sys/sys/queue.h: > > > > +#define LIST_PREV(head,elm,field) \ > > + (((elm) == LIST_FIRST(head)) ? ((__typeof(elm))0) : \ > > + ((__typeof(elm))(((uint8_t *)((elm)->field.le_prev)) - \ > > + ((uint8_t *)&LIST_NEXT((__typeof(elm))0,field))))) > > Please don't use typeof; it is a GCCism. Do you really mean NULL? Thanks for pointing that out. Then you will have to pass an additional argument, namely the "type": #define LIST_PREV(head,elm,field,type) \ (((elm) == LIST_FIRST(head)) ? ((struct type *)0) : \ ((struct type *)(((uint8_t *)((elm)->field.le_prev)) - \ ((uint8_t *)&LIST_NEXT((struct type *)0,field))))) How about the order of the arguments? Is this better? If this is accepted I will commit it to my FreeBSD P4 USB project first. Then someone else can commit it to HEAD. --HPS From owner-freebsd-hackers@FreeBSD.ORG Mon May 7 08:13:56 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 8181016A400 for ; Mon, 7 May 2007 08:13:56 +0000 (UTC) (envelope-from valiy-td@mail.ru) Received: from mx6.mail.ru (mx6.mail.ru [194.67.23.26]) by mx1.freebsd.org (Postfix) with ESMTP id 42C4613C43E for ; Mon, 7 May 2007 08:13:56 +0000 (UTC) (envelope-from valiy-td@mail.ru) Received: from [195.218.186.80] (port=11733 helo=[195.218.186.80]) by mx6.mail.ru with asmtp id 1HkyLy-0002Pg-00 for freebsd-hackers@freebsd.org; Mon, 07 May 2007 12:13:54 +0400 Message-ID: <463EDED0.2090805@mail.ru> Date: Mon, 07 May 2007 12:09:52 +0400 From: valiy User-Agent: Thunderbird 1.5.0.2 (X11/20060519) MIME-Version: 1.0 To: freebsd-hackers@freebsd.org Content-Type: text/plain; charset=windows-1251; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: FreeBSD 6.1 6.2 C++ stl set.clear() very slow X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 07 May 2007 08:13:56 -0000 FreeBSD 6.2-RELEASE FreeBSD 6.2-RELEASE #0: Mon Jan 15 14:39:55 MSK 2007 root@valiy:/usr/src/sys/i386/compile/MY i386 valiy# g++ test_set.cpp valiy# ./a.out filling clearing done alloc time 3.783 clear time 7.987 valiy# g++34 test_set.cpp valiy# ./a.out filling clearing done alloc time 3.850 clear time 7.889 valiy# ./a.out filling clearing done alloc time 4.253 clear time 7.994 valiy# ./a.out filling clearing done alloc time 3.573 clear time 0.293 From owner-freebsd-hackers@FreeBSD.ORG Mon May 7 08:47:29 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 6DDF716A400 for ; Mon, 7 May 2007 08:47:29 +0000 (UTC) (envelope-from dudu@dudu.ro) Received: from mu-out-0910.google.com (mu-out-0910.google.com [209.85.134.184]) by mx1.freebsd.org (Postfix) with ESMTP id 0B26513C447 for ; Mon, 7 May 2007 08:47:28 +0000 (UTC) (envelope-from dudu@dudu.ro) Received: by mu-out-0910.google.com with SMTP id w9so294803mue for ; Mon, 07 May 2007 01:47:27 -0700 (PDT) Received: by 10.82.180.17 with SMTP id c17mr10460212buf.1178525944837; Mon, 07 May 2007 01:19:04 -0700 (PDT) Received: by 10.82.148.19 with HTTP; Mon, 7 May 2007 01:19:04 -0700 (PDT) Message-ID: Date: Mon, 7 May 2007 11:19:04 +0300 From: "Vlad GALU" To: valiy In-Reply-To: <463EDED0.2090805@mail.ru> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <463EDED0.2090805@mail.ru> Cc: freebsd-hackers@freebsd.org Subject: Re: FreeBSD 6.1 6.2 C++ stl set.clear() very slow X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 07 May 2007 08:47:29 -0000 On 5/7/07, valiy wrote: [...] It might be related to phkmalloc vs jemalloc. You may want to try to link against jemalloc to see if the slowdown still occurs. -- If it's there, and you can see it, it's real. If it's not there, and you can see it, it's virtual. If it's there, and you can't see it, it's transparent. If it's not there, and you can't see it, you erased it. From owner-freebsd-hackers@FreeBSD.ORG Mon May 7 14:22:34 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 3AC2916A402 for ; Mon, 7 May 2007 14:22:34 +0000 (UTC) (envelope-from dudu@dudu.ro) Received: from ug-out-1314.google.com (ug-out-1314.google.com [66.249.92.169]) by mx1.freebsd.org (Postfix) with ESMTP id D17EA13C455 for ; Mon, 7 May 2007 14:22:33 +0000 (UTC) (envelope-from dudu@dudu.ro) Received: by ug-out-1314.google.com with SMTP id 71so916553ugh for ; Mon, 07 May 2007 07:22:32 -0700 (PDT) Received: by 10.82.113.6 with SMTP id l6mr10603989buc.1178547752225; Mon, 07 May 2007 07:22:32 -0700 (PDT) Received: by 10.82.148.19 with HTTP; Mon, 7 May 2007 07:22:32 -0700 (PDT) Message-ID: Date: Mon, 7 May 2007 17:22:32 +0300 From: "Vlad GALU" To: valiy In-Reply-To: <463F321A.1010803@mail.ru> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <463EDED0.2090805@mail.ru> <463F321A.1010803@mail.ru> Cc: freebsd-hackers@freebsd.org Subject: Re: FreeBSD 6.1 6.2 C++ stl set.clear() very slow X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 07 May 2007 14:22:34 -0000 On 5/7/07, valiy wrote: > Vlad GALU wrote: > > On 5/7/07, valiy wrote: > > [...] > > It might be related to phkmalloc vs jemalloc. You may want to try > > to link against jemalloc to see if the slowdown still occurs. > > > http://lists.freebsd.org/pipermail/freebsd-stable/2006-October/029186.html > Fetch the HEAD revision of http://www.freebsd.org/cgi/cvsweb.cgi/src/sys/sys/tree.h and overwrite your existing file using it. Afterwards, fetch the HEAD revision of http://www.freebsd.org/cgi/cvsweb.cgi/src/lib/libc/stdlib/malloc.c. Last step: gcc -fPIC -O -Wall -I/usr/src/lib/libc/include -I/usr/src/sys -shared -o malloc.so malloc.c Or if you want to compile it into a simple object that you can then link directly in your application, gcc -O -Wall -I/usr/src/lib/libc/include -I/usr/src/sys -o malloc.o malloc.c HTH. > > [root@valiy ~]# cp /7.0_CURRENT/src/lib/libc/stdlib/malloc.c > /tmp/jemalloc.so > [root@valiy ~]# gcc -O -Wall -I/usr/src/lib/libc/include -shared -o > /lib/jemalloc.so /tmp/jemalloc.c > /tmp/jemalloc.c:1197: warning: type defaults to `int' in declaration of > `RB_GENERATE_STATIC' > /tmp/jemalloc.c:1197: warning: parameter names (without types) in > function declaration > /tmp/jemalloc.c:1197: warning: data definition has no type or storage class > /tmp/jemalloc.c: In function `chunk_alloc': > /tmp/jemalloc.c:1267: warning: implicit declaration of function > `chunk_tree_s_RB_MINMAX' > /tmp/jemalloc.c:1267: warning: assignment makes pointer from integer > without a cast > /tmp/jemalloc.c:1273: warning: implicit declaration of function > `chunk_tree_s_RB_NEXT' > /tmp/jemalloc.c:1273: warning: assignment makes pointer from integer > without a cast > /tmp/jemalloc.c:1276: warning: implicit declaration of function > `chunk_tree_s_RB_REMOVE' > /tmp/jemalloc.c: In function `chunk_dealloc': > /tmp/jemalloc.c:1444: warning: implicit declaration of function > `chunk_tree_s_RB_FIND' > /tmp/jemalloc.c:1444: warning: comparison between pointer and integer > /tmp/jemalloc.c:1451: warning: implicit declaration of function > `chunk_tree_s_RB_INSERT' > /tmp/jemalloc.c: At top level: > /tmp/jemalloc.c:1597: warning: type defaults to `int' in declaration of > `RB_GENERATE_STATIC' > /tmp/jemalloc.c:1597: warning: parameter names (without types) in > function declaration > /tmp/jemalloc.c:1597: warning: data definition has no type or storage class > /tmp/jemalloc.c: In function `arena_chunk_alloc': > /tmp/jemalloc.c:1794: warning: implicit declaration of function > `arena_chunk_tree_s_RB_INSERT' > /tmp/jemalloc.c: In function `arena_chunk_dealloc': > /tmp/jemalloc.c:1875: warning: implicit declaration of function > `arena_chunk_tree_s_RB_REMOVE' > /tmp/jemalloc.c: In function `arena_run_alloc': > /tmp/jemalloc.c:2051: warning: implicit declaration of function > `arena_chunk_tree_s_RB_MINMAX' > /tmp/jemalloc.c:2051: warning: assignment makes pointer from integer > without a cast > /tmp/jemalloc.c:2051: warning: implicit declaration of function > `arena_chunk_tree_s_RB_NEXT' > /tmp/jemalloc.c:2051: warning: assignment makes pointer from integer > without a cast > /tmp/jemalloc.c: In function `huge_dalloc': > /tmp/jemalloc.c:2730: warning: assignment makes pointer from integer > without a cast > /tmp/jemalloc.c: In function `isalloc': > /tmp/jemalloc.c:2944: warning: assignment makes pointer from integer > without a cast > /tmp/jemalloc.c: In function `malloc_init_hard': > /tmp/jemalloc.c:3114: warning: 'opts' might be used uninitialized in > this function > /tmp/jemalloc.c: In function `calloc': > /tmp/jemalloc.c:3559: warning: 'num_size' might be used uninitialized in > this function > /tmp/jemalloc.c: At top level: > /tmp/jemalloc.c:1183: warning: 'chunk_comp' defined but not used > /tmp/jemalloc.c:1583: warning: 'arena_chunk_comp' defined but not used > > > could you please send me you version of malloc.c ? > -- If it's there, and you can see it, it's real. If it's not there, and you can see it, it's virtual. If it's there, and you can't see it, it's transparent. If it's not there, and you can't see it, you erased it. From owner-freebsd-hackers@FreeBSD.ORG Mon May 7 17:05:19 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 40C2616A418 for ; Mon, 7 May 2007 17:05:19 +0000 (UTC) (envelope-from freebsd-hackers@m.gmane.org) Received: from ciao.gmane.org (main.gmane.org [80.91.229.2]) by mx1.freebsd.org (Postfix) with ESMTP id C40C013C4B7 for ; Mon, 7 May 2007 17:05:18 +0000 (UTC) (envelope-from freebsd-hackers@m.gmane.org) Received: from list by ciao.gmane.org with local (Exim 4.43) id 1Hl6FV-0001d7-Ex for freebsd-hackers@freebsd.org; Mon, 07 May 2007 18:39:45 +0200 Received: from 78-1-95-82.adsl.net.t-com.hr ([78.1.95.82]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Mon, 07 May 2007 18:39:45 +0200 Received: from ivoras by 78-1-95-82.adsl.net.t-com.hr with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Mon, 07 May 2007 18:39:45 +0200 X-Injected-Via-Gmane: http://gmane.org/ To: freebsd-hackers@freebsd.org From: Ivan Voras Date: Mon, 07 May 2007 18:39:34 +0200 Lines: 75 Message-ID: Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="------------enig7CF7858D4EB10347BFBC0CF8" X-Complaints-To: usenet@sea.gmane.org X-Gmane-NNTP-Posting-Host: 78-1-95-82.adsl.net.t-com.hr User-Agent: Thunderbird 1.5.0.10 (Windows/20070221) X-Enigmail-Version: 0.94.3.0 Sender: news Subject: MIME-ifying sysctl types X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 07 May 2007 17:05:19 -0000 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enig7CF7858D4EB10347BFBC0CF8 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable In the output of sysctl -oa I see values like: kern.proc.all: Format:S,proc Length:75264 Dump:0x00030000000000002011fac380d4fec3... kern.proc.proc_td: Format:N Length:75264 Dump:0x00030000000000002011fac380d4fec3... kern.file: Format:S,xfile Length:18616 Dump:0x340000003e0c0000e803000006000000... vfs.nfs.diskless_rootaddr: Format:%Ssockaddr_in Length:16 Dump:0x00000000000000000000000000000000... debug.hashstat.rawnchash: Format:S,int Length:262144 Dump:0x00000000000000000000000000000000... , etc., and some like: kern.ipc.msqids: Format: Length:3520 Dump:0x00000000000000000000000000000000... kern.ipc.sema: Format: Length:600 Dump:0x00000000000000000000000000000000= =2E.. and even some like: machdep.consdev: Format:T,struct cdev * Length:4 Dump:0x00000000... The first group has widely nonuniform "Format" fields, the second don't have it at all, and the third just looks wrong (is that a NULL pointer being exported from the kernel, just in case the userland doesn't have it? :) ). My proposal is to MIME-ify the Format fields, best presented in examples:= "S,proc" -> "x-struct/proc" "S,xfile" -> "x-struct/xfile" Etc. GEOM tree is the easiest, it's just "text/xml", and the already present text fields would be "text/plain". If possible, the types should conform to http://www.iana.org/assignments/media-types/ - but except for text and XML types they usually won't. The next step, obviously, but of dubious benefit, would be to stop exporting binary data from the kernel and do it via XML, but that's another thing, possibly another GSoC proposal :) I can provide the patches for the format names. At this time I'm looking for input: is this idea sane? Are there any utilities that actually parse the type name? The benefit of this conversion is that MIME types are easier on the eyes and will help unfamiliar users understand what's going on. --------------enig7CF7858D4EB10347BFBC0CF8 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.4 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFGP1ZGldnAQVacBcgRAvZwAKCbB0K89B2a80yVfjqATZIBotZpxgCbB+Cs XS/NOYjnPeWTpm9k9XqNMdQ= =bsyG -----END PGP SIGNATURE----- --------------enig7CF7858D4EB10347BFBC0CF8-- From owner-freebsd-hackers@FreeBSD.ORG Mon May 7 17:17:35 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 452F216A402 for ; Mon, 7 May 2007 17:17:35 +0000 (UTC) (envelope-from anderson@freebsd.org) Received: from mh2.centtech.com (moat3.centtech.com [64.129.166.50]) by mx1.freebsd.org (Postfix) with ESMTP id 0DB0413C483 for ; Mon, 7 May 2007 17:17:34 +0000 (UTC) (envelope-from anderson@freebsd.org) Received: from neutrino.centtech.com (neutrino.centtech.com [10.177.171.220]) by mh2.centtech.com (8.13.8/8.13.8) with ESMTP id l47HHP4h059911; Mon, 7 May 2007 12:17:26 -0500 (CDT) (envelope-from anderson@freebsd.org) Message-ID: <463F5F25.5030201@freebsd.org> Date: Mon, 07 May 2007 12:17:25 -0500 From: Eric Anderson User-Agent: Thunderbird 2.0.0.0 (X11/20070420) MIME-Version: 1.0 To: Ivan Voras References: In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: ClamAV 0.88.4/3216/Mon May 7 02:36:02 2007 on mh2.centtech.com X-Virus-Status: Clean X-Spam-Status: No, score=-2.5 required=8.0 tests=AWL,BAYES_00 autolearn=ham version=3.1.6 X-Spam-Checker-Version: SpamAssassin 3.1.6 (2006-10-03) on mh2.centtech.com Cc: freebsd-hackers@freebsd.org Subject: Re: MIME-ifying sysctl types X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 07 May 2007 17:17:35 -0000 On 05/07/07 11:39, Ivan Voras wrote: > In the output of sysctl -oa I see values like: > > kern.proc.all: Format:S,proc Length:75264 > Dump:0x00030000000000002011fac380d4fec3... > kern.proc.proc_td: Format:N Length:75264 > Dump:0x00030000000000002011fac380d4fec3... > kern.file: Format:S,xfile Length:18616 > Dump:0x340000003e0c0000e803000006000000... > vfs.nfs.diskless_rootaddr: Format:%Ssockaddr_in Length:16 > Dump:0x00000000000000000000000000000000... > debug.hashstat.rawnchash: Format:S,int Length:262144 > Dump:0x00000000000000000000000000000000... > > , etc., and some like: > > kern.ipc.msqids: Format: Length:3520 > Dump:0x00000000000000000000000000000000... > kern.ipc.sema: Format: Length:600 Dump:0x00000000000000000000000000000000... > > and even some like: > machdep.consdev: Format:T,struct cdev * Length:4 Dump:0x00000000... > > The first group has widely nonuniform "Format" fields, the second don't > have it at all, and the third just looks wrong (is that a NULL pointer > being exported from the kernel, just in case the userland doesn't have > it? :) ). > > My proposal is to MIME-ify the Format fields, best presented in examples: > > "S,proc" -> "x-struct/proc" > "S,xfile" -> "x-struct/xfile" > > Etc. > > GEOM tree is the easiest, it's just "text/xml", and the already present > text fields would be "text/plain". > > If possible, the types should conform to > http://www.iana.org/assignments/media-types/ - but except for text and > XML types they usually won't. > > The next step, obviously, but of dubious benefit, would be to stop > exporting binary data from the kernel and do it via XML, but that's > another thing, possibly another GSoC proposal :) > > I can provide the patches for the format names. At this time I'm looking > for input: is this idea sane? Are there any utilities that actually > parse the type name? > > The benefit of this conversion is that MIME types are easier on the eyes > and will help unfamiliar users understand what's going on. > How about adding a -m switch to sysctl that outputs the MIME compatible format, leaving the other formats unchanged to avoid breaking something someone is expecting? Eric From owner-freebsd-hackers@FreeBSD.ORG Mon May 7 17:51:54 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id B067216A4EA for ; Mon, 7 May 2007 17:51:54 +0000 (UTC) (envelope-from jmg@hydrogen.funkthat.com) Received: from hydrogen.funkthat.com (gate.funkthat.com [69.17.45.168]) by mx1.freebsd.org (Postfix) with ESMTP id 7F35013C506 for ; Mon, 7 May 2007 17:51:54 +0000 (UTC) (envelope-from jmg@hydrogen.funkthat.com) Received: from hydrogen.funkthat.com (1cgk8nsr1zoo9fhw@localhost.funkthat.com [127.0.0.1]) by hydrogen.funkthat.com (8.13.6/8.13.3) with ESMTP id l47Hpiji032055; Mon, 7 May 2007 10:51:44 -0700 (PDT) (envelope-from jmg@hydrogen.funkthat.com) Received: (from jmg@localhost) by hydrogen.funkthat.com (8.13.6/8.13.3/Submit) id l47HphlE032054; Mon, 7 May 2007 10:51:43 -0700 (PDT) (envelope-from jmg) Date: Mon, 7 May 2007 10:51:43 -0700 From: John-Mark Gurney To: Ivan Voras Message-ID: <20070507175143.GR17958@funkthat.com> Mail-Followup-To: Ivan Voras , freebsd-hackers@freebsd.org References: Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.4.2.1i X-Operating-System: FreeBSD 5.4-RELEASE-p6 i386 X-Files: The truth is out there X-URL: http://resnet.uoregon.edu/~gurney_j/ X-Resume: http://resnet.uoregon.edu/~gurney_j/resume.html Cc: freebsd-hackers@freebsd.org Subject: Re: MIME-ifying sysctl types X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: John-Mark Gurney List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 07 May 2007 17:51:54 -0000 Ivan Voras wrote this message on Mon, May 07, 2007 at 18:39 +0200: > My proposal is to MIME-ify the Format fields, best presented in examples: > > "S,proc" -> "x-struct/proc" > "S,xfile" -> "x-struct/xfile" If you do, don't forget to add the version of the field to the mime-type as these structures change over time.. -- John-Mark Gurney Voice: +1 415 225 5579 "All that I will do, has been done, All that I have, has not." From owner-freebsd-hackers@FreeBSD.ORG Mon May 7 17:55:25 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id C564A16A400 for ; Mon, 7 May 2007 17:55:25 +0000 (UTC) (envelope-from ivoras@fer.hr) Received: from ls405.t-com.hr (ls405.t-com.hr [195.29.150.135]) by mx1.freebsd.org (Postfix) with ESMTP id 80E4813C448 for ; Mon, 7 May 2007 17:55:25 +0000 (UTC) (envelope-from ivoras@fer.hr) Received: from ls248.t-com.hr (ls248.t-com.hr [195.29.150.237]) by ls405.t-com.hr (Postfix) with ESMTP id C377C1444CA; Mon, 7 May 2007 19:23:24 +0200 (CEST) Received: from ls248.t-com.hr (ls248.t-com.hr [127.0.0.1]) by ls248.t-com.hr (Qmlai) with ESMTP id C02D4D5004A; Mon, 7 May 2007 19:23:24 +0200 (CEST) Received: from ls248.t-com.hr (ls248.t-com.hr [127.0.0.1]) by ls248.t-com.hr (Qmlai) with ESMTP id AB118D50047; Mon, 7 May 2007 19:23:24 +0200 (CEST) X-Envelope-Sender-Info: g5URFa92gX9K/Rg9VFA/rHzRAIC33GmuD4/BZ7fTnng6StkSH1j7CT0zJW9WjWDV X-Envelope-Sender: ivoras@fer.hr Received: from [10.0.0.100] (78-1-95-82.adsl.net.t-com.hr [78.1.95.82]) by ls248.t-com.hr (Qmali) with ESMTP id 6806D5E005E; Mon, 7 May 2007 19:23:24 +0200 (CEST) Message-ID: <463F6086.4020909@fer.hr> Date: Mon, 07 May 2007 19:23:18 +0200 From: Ivan Voras User-Agent: Thunderbird 1.5.0.10 (Windows/20070221) MIME-Version: 1.0 To: Eric Anderson References: <463F5F25.5030201@freebsd.org> In-Reply-To: <463F5F25.5030201@freebsd.org> X-Enigmail-Version: 0.94.3.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="------------enig435ED7C5DD1C3A0427681A50" Cc: freebsd-hackers@freebsd.org Subject: Re: MIME-ifying sysctl types X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 07 May 2007 17:55:25 -0000 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enig435ED7C5DD1C3A0427681A50 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Eric Anderson wrote: > How about adding a -m switch to sysctl that outputs the MIME compatible= > format, leaving the other formats unchanged to avoid breaking something= > someone is expecting? I don't think it's a good idea because it either means adding another "format-like" field in the kernel or adding a translation table in the sysctl program, and both are bad hacks. --------------enig435ED7C5DD1C3A0427681A50 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.4 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFGP2CMldnAQVacBcgRAg0+AKCDiRVy6gWpnwbK4nny7N3M6ICSEgCg8kIS PIOCfX0kx1kOv4Cp7on9cMs= =9Cjn -----END PGP SIGNATURE----- --------------enig435ED7C5DD1C3A0427681A50-- From owner-freebsd-hackers@FreeBSD.ORG Mon May 7 18:12:50 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 8E4AB16A407 for ; Mon, 7 May 2007 18:12:50 +0000 (UTC) (envelope-from freebsd-hackers@m.gmane.org) Received: from ciao.gmane.org (main.gmane.org [80.91.229.2]) by mx1.freebsd.org (Postfix) with ESMTP id 47A6B13C46A for ; Mon, 7 May 2007 18:12:50 +0000 (UTC) (envelope-from freebsd-hackers@m.gmane.org) Received: from list by ciao.gmane.org with local (Exim 4.43) id 1Hl7hR-0006v7-6s for freebsd-hackers@freebsd.org; Mon, 07 May 2007 20:12:41 +0200 Received: from 78-1-95-82.adsl.net.t-com.hr ([78.1.95.82]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Mon, 07 May 2007 20:12:41 +0200 Received: from ivoras by 78-1-95-82.adsl.net.t-com.hr with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Mon, 07 May 2007 20:12:41 +0200 X-Injected-Via-Gmane: http://gmane.org/ To: freebsd-hackers@freebsd.org From: Ivan Voras Date: Mon, 07 May 2007 20:12:26 +0200 Lines: 37 Message-ID: References: <20070507175143.GR17958@funkthat.com> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="------------enig8EFC03A977FA1D627CA241E0" X-Complaints-To: usenet@sea.gmane.org X-Gmane-NNTP-Posting-Host: 78-1-95-82.adsl.net.t-com.hr User-Agent: Thunderbird 1.5.0.10 (Windows/20070221) In-Reply-To: <20070507175143.GR17958@funkthat.com> X-Enigmail-Version: 0.94.3.0 Sender: news Subject: Re: MIME-ifying sysctl types X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 07 May 2007 18:12:50 -0000 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enig8EFC03A977FA1D627CA241E0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable John-Mark Gurney wrote: > Ivan Voras wrote this message on Mon, May 07, 2007 at 18:39 +0200: >> My proposal is to MIME-ify the Format fields, best presented in exampl= es: >> >> "S,proc" -> "x-struct/proc" >> "S,xfile" -> "x-struct/xfile" >=20 > If you do, don't forget to add the version of the field to the > mime-type as these structures change over time.. It's a good idea, maybe something like "x-struct/proc-v1", but is it necessary without the XMLizing part? AFAIK the fields are currently unversioned? --------------enig8EFC03A977FA1D627CA241E0 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.4 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFGP2wPldnAQVacBcgRAnPjAKC5GBSjit/R7tJhg0a15JJlAap54ACfYSGY Tac1GU4GufAns2nMp78MYwI= =syEP -----END PGP SIGNATURE----- --------------enig8EFC03A977FA1D627CA241E0-- From owner-freebsd-hackers@FreeBSD.ORG Mon May 7 18:29:35 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id EBC1716A400 for ; Mon, 7 May 2007 18:29:35 +0000 (UTC) (envelope-from joerg@britannica.bec.de) Received: from www.pkgsrc-box.org (www.ostsee-abc.de [62.206.222.50]) by mx1.freebsd.org (Postfix) with ESMTP id AE74413C447 for ; Mon, 7 May 2007 18:29:35 +0000 (UTC) (envelope-from joerg@britannica.bec.de) Received: from britannica.bec.de (www.pkgsrc-box.org [127.0.0.1]) by www.pkgsrc-box.org (Postfix) with ESMTP id E559DE7A3F9 for ; Mon, 7 May 2007 17:35:13 +0000 (UTC) Received: by britannica.bec.de (Postfix, from userid 1000) id 72BB922C; Mon, 7 May 2007 20:00:16 +0200 (CEST) Date: Mon, 7 May 2007 20:00:15 +0200 From: Joerg Sonnenberger To: freebsd-hackers@freebsd.org Message-ID: <20070507180015.GA694@britannica.bec.de> Mail-Followup-To: freebsd-hackers@freebsd.org References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.13 (2006-08-11) Subject: Re: MIME-ifying sysctl types X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 07 May 2007 18:29:36 -0000 On Mon, May 07, 2007 at 06:39:34PM +0200, Ivan Voras wrote: > My proposal is to MIME-ify the Format fields, best presented in examples: > > "S,proc" -> "x-struct/proc" > "S,xfile" -> "x-struct/xfile" The format field is intended for the interpretation of userland programs like sysctl. For those fields, the human user is barely able to find meaning in the fields -- so this doesn't really help much but changing the kernel ABI. Which should be considered A Bad Thing (TM). Joerg From owner-freebsd-hackers@FreeBSD.ORG Mon May 7 18:53:13 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 46AFF16A401 for ; Mon, 7 May 2007 18:53:13 +0000 (UTC) (envelope-from jmg@hydrogen.funkthat.com) Received: from hydrogen.funkthat.com (gate.funkthat.com [69.17.45.168]) by mx1.freebsd.org (Postfix) with ESMTP id 21DAB13C458 for ; Mon, 7 May 2007 18:53:12 +0000 (UTC) (envelope-from jmg@hydrogen.funkthat.com) Received: from hydrogen.funkthat.com (ipwm8adx3ztght2l@localhost.funkthat.com [127.0.0.1]) by hydrogen.funkthat.com (8.13.6/8.13.3) with ESMTP id l47Ir7wO032961; Mon, 7 May 2007 11:53:07 -0700 (PDT) (envelope-from jmg@hydrogen.funkthat.com) Received: (from jmg@localhost) by hydrogen.funkthat.com (8.13.6/8.13.3/Submit) id l47Ir3Xr032960; Mon, 7 May 2007 11:53:03 -0700 (PDT) (envelope-from jmg) Date: Mon, 7 May 2007 11:53:03 -0700 From: John-Mark Gurney To: Ivan Voras Message-ID: <20070507185303.GS17958@funkthat.com> Mail-Followup-To: Ivan Voras , freebsd-hackers@freebsd.org References: <20070507175143.GR17958@funkthat.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.4.2.1i X-Operating-System: FreeBSD 5.4-RELEASE-p6 i386 X-Files: The truth is out there X-URL: http://resnet.uoregon.edu/~gurney_j/ X-Resume: http://resnet.uoregon.edu/~gurney_j/resume.html Cc: freebsd-hackers@freebsd.org Subject: Re: MIME-ifying sysctl types X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: John-Mark Gurney List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 07 May 2007 18:53:13 -0000 Ivan Voras wrote this message on Mon, May 07, 2007 at 20:12 +0200: > John-Mark Gurney wrote: > > Ivan Voras wrote this message on Mon, May 07, 2007 at 18:39 +0200: > >> My proposal is to MIME-ify the Format fields, best presented in examples: > >> > >> "S,proc" -> "x-struct/proc" > >> "S,xfile" -> "x-struct/xfile" > > > > If you do, don't forget to add the version of the field to the > > mime-type as these structures change over time.. > > It's a good idea, maybe something like "x-struct/proc-v1", but is it > necessary without the XMLizing part? AFAIK the fields are currently > unversioned? no, I was thinking: "x-struct/proc;version=1"... Doesn't mean they shouldn't be.. Currently most tools detect different versions of structs by examining the size.. This isn't always safe as fields could get shuffled or meanings changed w/o changing the size.. -- John-Mark Gurney Voice: +1 415 225 5579 "All that I will do, has been done, All that I have, has not." From owner-freebsd-hackers@FreeBSD.ORG Mon May 7 19:18:53 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 3700916A402 for ; Mon, 7 May 2007 19:18:53 +0000 (UTC) (envelope-from freebsd-hackers@m.gmane.org) Received: from ciao.gmane.org (main.gmane.org [80.91.229.2]) by mx1.freebsd.org (Postfix) with ESMTP id E3C2313C4BB for ; Mon, 7 May 2007 19:18:52 +0000 (UTC) (envelope-from freebsd-hackers@m.gmane.org) Received: from list by ciao.gmane.org with local (Exim 4.43) id 1Hl8jL-00019V-64 for freebsd-hackers@freebsd.org; Mon, 07 May 2007 21:18:43 +0200 Received: from 78-1-95-82.adsl.net.t-com.hr ([78.1.95.82]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Mon, 07 May 2007 21:18:43 +0200 Received: from ivoras by 78-1-95-82.adsl.net.t-com.hr with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Mon, 07 May 2007 21:18:43 +0200 X-Injected-Via-Gmane: http://gmane.org/ To: freebsd-hackers@freebsd.org From: Ivan Voras Date: Mon, 07 May 2007 21:18:23 +0200 Lines: 31 Message-ID: References: <20070507175143.GR17958@funkthat.com> <20070507185303.GS17958@funkthat.com> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="------------enig4B1D66BFC4793EABE3B114F7" X-Complaints-To: usenet@sea.gmane.org X-Gmane-NNTP-Posting-Host: 78-1-95-82.adsl.net.t-com.hr User-Agent: Thunderbird 1.5.0.10 (Windows/20070221) In-Reply-To: <20070507185303.GS17958@funkthat.com> X-Enigmail-Version: 0.94.3.0 Sender: news Subject: Re: MIME-ifying sysctl types X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 07 May 2007 19:18:53 -0000 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enig4B1D66BFC4793EABE3B114F7 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable John-Mark Gurney wrote: > no, I was thinking: "x-struct/proc;version=3D1"... Doesn't mean they > shouldn't be.. Currently most tools detect different versions of struc= ts > by examining the size.. This isn't always safe as fields could get > shuffled or meanings changed w/o changing the size.. I agree, this is better. --------------enig4B1D66BFC4793EABE3B114F7 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.4 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFGP3uFldnAQVacBcgRAveDAKDmR0VbR/YCysEiiGNRpQ8kp1ZOtACeICUV Tp3L78YESigyWUFRQ++Zuds= =N7QF -----END PGP SIGNATURE----- --------------enig4B1D66BFC4793EABE3B114F7-- From owner-freebsd-hackers@FreeBSD.ORG Tue May 8 02:40:11 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id ADBC716A401 for ; Tue, 8 May 2007 02:40:11 +0000 (UTC) (envelope-from freebsd-hackers@m.gmane.org) Received: from ciao.gmane.org (main.gmane.org [80.91.229.2]) by mx1.freebsd.org (Postfix) with ESMTP id 3637313C457 for ; Tue, 8 May 2007 02:40:10 +0000 (UTC) (envelope-from freebsd-hackers@m.gmane.org) Received: from root by ciao.gmane.org with local (Exim 4.43) id 1HlFcQ-0003ii-QT for freebsd-hackers@freebsd.org; Tue, 08 May 2007 04:40:02 +0200 Received: from nat155.aci.icn.bmstu.ru ([80.92.248.155]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Tue, 08 May 2007 04:40:02 +0200 Received: from thIOretic by nat155.aci.icn.bmstu.ru with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Tue, 08 May 2007 04:40:02 +0200 X-Injected-Via-Gmane: http://gmane.org/ To: freebsd-hackers@freebsd.org From: "Maxim Zhuravlev" Date: Tue, 8 May 2007 02:55:48 +0400 Lines: 16 Message-ID: References: <463CCE1E.1080906@geri.cc.fer.hr> Mime-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=original Content-Transfer-Encoding: 7bit X-Complaints-To: usenet@sea.gmane.org X-Gmane-NNTP-Posting-Host: nat155.aci.icn.bmstu.ru In-Reply-To: <463CCE1E.1080906@geri.cc.fer.hr> X-MSMail-Priority: Normal X-Newsreader: Microsoft Windows Mail 6.0.5744.16384 X-MimeOLE: Produced By Microsoft MimeOLE V6.0.5744.16384 Sender: news Subject: Re: Writing to a file X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Maxim Zhuravlev List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 May 2007 02:40:11 -0000 "Sonja Milicic" wrote > So, could anyone please give me a detailed > explanation of how to open a file in kernel and write to it - best data > types to use, functions, what to look out for, maybe a link to tutorial > or manual that deals with this (if such a thing exists), etc.? Not really an explanation, but may be you should start from vfs kernel subsystem (it should be nicely documented, I guess :)). Even more adequate, study file-related syscalls implementation. If not done already. WBR, Maxim From owner-freebsd-hackers@FreeBSD.ORG Tue May 8 23:43:14 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id DCDA316A403 for ; Tue, 8 May 2007 23:43:14 +0000 (UTC) (envelope-from subhashg.unix@gmail.com) Received: from qb-out-0506.google.com (qb-out-0506.google.com [72.14.204.230]) by mx1.freebsd.org (Postfix) with ESMTP id 9A9B613C45D for ; Tue, 8 May 2007 23:43:14 +0000 (UTC) (envelope-from subhashg.unix@gmail.com) Received: by qb-out-0506.google.com with SMTP id e6so15675qbe for ; Tue, 08 May 2007 16:43:13 -0700 (PDT) DKIM-Signature: a=rsa-sha1; c=relaxed/relaxed; d=gmail.com; s=beta; h=domainkey-signature:received:received:message-id:date:from:to:subject:mime-version:content-type; b=qtBTeJszFs/quznn8N2Zwu/zb+pEjZV/0h1lrF/SgcpL6T7o4dOZh0OQ0aicOnM5ix2ES7FXrQ+A2jriHK3kkP8v9VRTERecD1umpBNpxv5GtwONR//efk0XvzmPS8k9fj9xG4oPHQATt0Hp9+zULLmN3TB4Q+VLzLiXh5CaMbg= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=beta; h=received:message-id:date:from:to:subject:mime-version:content-type; b=YWajvF0K07OKSxG+PF49a90YYUdpxRlvUS7pcmOlf8jW85smSQIdq83ab6I3k6QCWuPmIlta+U1Zvwu3LoG1feNjdUaIEhteUI+ck3N9zPLk6lmX6fvtla8JRlfe7v3KHn+ZznuQYbAse0/SmTgg2VnhIJM0/GyL/biwkwI4YGc= Received: by 10.114.94.1 with SMTP id r1mr2807165wab.1178667792245; Tue, 08 May 2007 16:43:12 -0700 (PDT) Received: by 10.115.91.5 with HTTP; Tue, 8 May 2007 16:43:12 -0700 (PDT) Message-ID: <5db9d2e0705081643l402857aeqb042c26d3cebdaac@mail.gmail.com> Date: Tue, 8 May 2007 16:43:12 -0700 From: "Subhash Gopinath" To: freebsd-hackers@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Content-Disposition: inline X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Subject: ipv6 Secure NDP X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 May 2007 23:43:14 -0000 Hello folks, Is there a plan to implement ipv6 SEND (secure NDP) in FreeBSD? There's a userspace implementation by DoCoMo, but thats not yet in the official ports collection. Thanks, --Subhash From owner-freebsd-hackers@FreeBSD.ORG Wed May 9 07:51:07 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id C7CE716A400 for ; Wed, 9 May 2007 07:51:07 +0000 (UTC) (envelope-from mohacsi@niif.hu) Received: from mail.ki.iif.hu (mail.ki.iif.hu [193.6.222.241]) by mx1.freebsd.org (Postfix) with ESMTP id 8A68213C45E for ; Wed, 9 May 2007 07:51:07 +0000 (UTC) (envelope-from mohacsi@niif.hu) Received: by mail.ki.iif.hu (Postfix, from userid 1003) id 3C0B455ED; Wed, 9 May 2007 09:51:06 +0200 (CEST) Received: from localhost (localhost [127.0.0.1]) by mail.ki.iif.hu (Postfix) with ESMTP id 3A60755EA; Wed, 9 May 2007 09:51:06 +0200 (CEST) Date: Wed, 9 May 2007 09:51:06 +0200 (CEST) From: Mohacsi Janos X-X-Sender: mohacsi@mignon.ki.iif.hu To: Subhash Gopinath In-Reply-To: <5db9d2e0705081643l402857aeqb042c26d3cebdaac@mail.gmail.com> Message-ID: <20070509093717.R37033@mignon.ki.iif.hu> References: <5db9d2e0705081643l402857aeqb042c26d3cebdaac@mail.gmail.com> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: freebsd-hackers@freebsd.org Subject: Re: ipv6 Secure NDP X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 May 2007 07:51:07 -0000 Hi, There was a plan to implement SEND at KAME, but due to IPR issues KAME did not start the implementation: http://www.kame.net/newsletter/20040525/ Maybe it could be implemented in Google Summer of Code for any BSD. Regards, Janos Mohacsi Network Engineer, Research Associate, Head of Network Planning and Projects NIIF/HUNGARNET, HUNGARY Key 70EF9882: DEC2 C685 1ED4 C95A 145F 4300 6F64 7B00 70EF 9882 On Tue, 8 May 2007, Subhash Gopinath wrote: > Hello folks, > > Is there a plan to implement ipv6 SEND (secure NDP) in FreeBSD? > There's a userspace implementation by DoCoMo, but thats not yet > in the official ports collection. > > Thanks, > --Subhash > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" > From owner-freebsd-hackers@FreeBSD.ORG Thu May 10 06:25:09 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id C964116A404 for ; Thu, 10 May 2007 06:25:09 +0000 (UTC) (envelope-from dunceor@gmail.com) Received: from wx-out-0506.google.com (wx-out-0506.google.com [66.249.82.224]) by mx1.freebsd.org (Postfix) with ESMTP id 76A4413C458 for ; Thu, 10 May 2007 06:25:09 +0000 (UTC) (envelope-from dunceor@gmail.com) Received: by wx-out-0506.google.com with SMTP id s18so442822wxc for ; Wed, 09 May 2007 23:25:09 -0700 (PDT) DKIM-Signature: a=rsa-sha1; c=relaxed/relaxed; d=gmail.com; s=beta; h=domainkey-signature:received:received:message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; b=Q/VQ0fIPL7/yoEm1RT/k/H61yx+RntuoBNOB4BpJ1ouHOTzDNyb+/w9+PeBtCnkELm3f8Had4sxr7bI8KOgGmdChjSAyp7gc9jNcILIZvWZmD5+v0c90UWcjNa+0dtKTSU4Rr29BR9af05AJWPnaKwjJprJqDGZ3ZSFW+zJB1vc= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=beta; h=received:message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; b=XB6HGfAXYeOkL+Qtp44WddwG3I51I7hEC1aXkYWkL0t/YfVStk71zqyKL0unkJ6qjDxQym501ktZ85GtPIlVByfuzGAEK/y4zfiag1tW/02JqUQd74assLbjBpOUYO3cDoT3hhK62DTra4eUVOIw5B1HYnUkHagZbKKMjYIUVns= Received: by 10.78.176.20 with SMTP id y20mr309107hue.1178776784957; Wed, 09 May 2007 22:59:44 -0700 (PDT) Received: by 10.78.165.12 with HTTP; Wed, 9 May 2007 22:59:44 -0700 (PDT) Message-ID: <5d84cb30705092259n121f1b4eu454df8e553700f49@mail.gmail.com> Date: Thu, 10 May 2007 07:59:44 +0200 From: "=?ISO-8859-1?Q?Karl_Sj=F6dahl_-_dunceor?=" To: "Mohacsi Janos" In-Reply-To: <20070509093717.R37033@mignon.ki.iif.hu> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <5db9d2e0705081643l402857aeqb042c26d3cebdaac@mail.gmail.com> <20070509093717.R37033@mignon.ki.iif.hu> Cc: freebsd-hackers@freebsd.org, Subhash Gopinath Subject: Re: ipv6 Secure NDP X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 May 2007 06:25:09 -0000 On 5/9/07, Mohacsi Janos wrote: > Hi, > There was a plan to implement SEND at KAME, but due to IPR issues > KAME did not start the implementation: > http://www.kame.net/newsletter/20040525/ > > Maybe it could be implemented in Google Summer of Code for any BSD. > > Regards, > > Janos Mohacsi > Network Engineer, Research Associate, Head of Network Planning and Projects > NIIF/HUNGARNET, HUNGARY > Key 70EF9882: DEC2 C685 1ED4 C95A 145F 4300 6F64 7B00 70EF 9882 > > On Tue, 8 May 2007, Subhash Gopinath wrote: > > > Hello folks, > > > > Is there a plan to implement ipv6 SEND (secure NDP) in FreeBSD? > > There's a userspace implementation by DoCoMo, but thats not yet > > in the official ports collection. > > > > Thanks, > > --Subhash There is still patent issues with SEND as far as I know so I do not see how this could be implemented in a GSoC? Microsoft which has claims wants a licence from them I think. I got interested in this now and since I work at Ericsson I will look this up and see what the status is from the Ericsson side of it. BR Karl From owner-freebsd-hackers@FreeBSD.ORG Thu May 10 11:13:26 2007 Return-Path: X-Original-To: hackers@freebsd.org Delivered-To: freebsd-hackers@FreeBSD.ORG Received: by hub.freebsd.org (Postfix, from userid 680) id 50E3816A407; Thu, 10 May 2007 11:13:26 +0000 (UTC) Date: Thu, 10 May 2007 11:13:26 +0000 From: Darren Reed To: hackers@freebsd.org Message-ID: <20070510111326.GA94093@hub.freebsd.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.4.2.1i X-Mailman-Approved-At: Thu, 10 May 2007 11:22:27 +0000 Cc: Subject: Experiences with 7.0-CURRENT and vmware. X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 May 2007 11:13:26 -0000 I'm using FreeBSD 7.0-CURRENT under vmware and there are a few issues. First, time. hint.hw.acpi.disabled="1" This appears to make _no_ difference to time keeping on FreeBSD 7 and nor does it seem to have any impact on ACPI being loaded. Do I need to recompile a new kernel without it or is there a new way to disable ACPI? I should add that FreeBSD 6, with the same setting, is no better and that I need to run ntpdate every 5-10 minutes via crontab in order to keep good time (timekeeping is *really* bad.) In one instance, i was watching "zpool iostat 1" and it appeared like the rows were muching up at a rate of 2 a second for a minute or so. How do I disable TSC timekeeping? (NetBSD has this disabled by default in their kernels.) Or is there somethign else I must do? Second, networking. Prior to FreeBSD-7, the driver to use inside vmware workstation was lnc. It has worked and contiues to work great. No problemo. FreeBSD-7 uses the "em" driver. To put it simply, it sucks in comparison. When things really get bad I start seeing "em0: watchdog timeout" messages on the console. I looked and I don't see a lnc driver anywhere. Is there another alternative (le?) driver that I can use in place of em, if so, how? Apart from these two issues (which are very central ones :-(), I'm using FreeBSD in a 64bit vmware workstation environment quick successfully and ZFS is quite happy with all the kva :-) ZFS and zpools are working just as I expect, even if a bit slower due to vmware but I'm not cranking out benchmarks here. Oh, and how do I fix ssh/rsh to do passwordless sessions? I'm trying to setup cron jobs to automate various tasks but there's this small hurdle called a password prompt that I can't seem to get rid of :-/ Cheers, Darren From owner-freebsd-hackers@FreeBSD.ORG Thu May 10 12:17:40 2007 Return-Path: X-Original-To: hackers@freebsd.org Delivered-To: freebsd-hackers@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 2E85D16A403 for ; Thu, 10 May 2007 12:17:40 +0000 (UTC) (envelope-from dom@helenmarks.co.uk) Received: from ws.helenmarks.co.uk (ws.helenmarks.co.uk [81.19.179.151]) by mx1.freebsd.org (Postfix) with ESMTP id F18F713C448 for ; Thu, 10 May 2007 12:17:39 +0000 (UTC) (envelope-from dom@helenmarks.co.uk) X-Virus-Scanned: mail.goodforbusiness.co.uk Message-ID: <4504.195.12.22.194.1178798447.squirrel@mail.helenmarks.co.uk> In-Reply-To: <20070510111326.GA94093@hub.freebsd.org> References: <20070510111326.GA94093@hub.freebsd.org> Date: Thu, 10 May 2007 13:00:47 +0100 (BST) From: "Dominic Marks" To: "Darren Reed" User-Agent: SquirrelMail/1.4.9a MIME-Version: 1.0 Content-Type: text/plain;charset=iso-8859-1 Content-Transfer-Encoding: 8bit X-Priority: 3 (Normal) Importance: Normal Cc: hackers@freebsd.org Subject: Re: Experiences with 7.0-CURRENT and vmware. X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 May 2007 12:17:40 -0000 Darren Reed wrote: > > I'm using FreeBSD 7.0-CURRENT under vmware and there are a few issues. > > First, time. > hint.hw.acpi.disabled="1" > This appears to make _no_ difference to time keeping on FreeBSD 7 > and nor does it seem to have any impact on ACPI being loaded. Do > I need to recompile a new kernel without it or is there a new way > to disable ACPI? > > I should add that FreeBSD 6, with the same setting, is no better > and that I need to run ntpdate every 5-10 minutes via crontab in > order to keep good time (timekeeping is *really* bad.) In one > instance, i was watching "zpool iostat 1" and it appeared like the > rows were muching up at a rate of 2 a second for a minute or so. > How do I disable TSC timekeeping? (NetBSD has this disabled by > default in their kernels.) Or is there somethign else I must do? Does: hint.apic.0.disabled=1 Help? > Second, networking. > Prior to FreeBSD-7, the driver to use inside vmware workstation > was lnc. It has worked and contiues to work great. No problemo. > FreeBSD-7 uses the "em" driver. To put it simply, it sucks in > comparison. When things really get bad I start seeing "em0: watchdog > timeout" messages on the console. I looked and I don't see a lnc > driver anywhere. Is there another alternative (le?) driver that > I can use in place of em, if so, how? > > Apart from these two issues (which are very central ones :-(), > I'm using FreeBSD in a 64bit vmware workstation environment > quick successfully and ZFS is quite happy with all the kva :-) > ZFS and zpools are working just as I expect, even if a bit > slower due to vmware but I'm not cranking out benchmarks here. > > Oh, and how do I fix ssh/rsh to do passwordless sessions? > I'm trying to setup cron jobs to automate various tasks but > there's this small hurdle called a password prompt that I > can't seem to get rid of :-/ Use a password-free (R|D)SA key. Dominic From owner-freebsd-hackers@FreeBSD.ORG Thu May 10 12:58:02 2007 Return-Path: X-Original-To: hackers@freebsd.org Delivered-To: freebsd-hackers@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 9280616A405 for ; Thu, 10 May 2007 12:58:02 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from cyrus.watson.org (cyrus.watson.org [209.31.154.42]) by mx1.freebsd.org (Postfix) with ESMTP id 4838913C44B for ; Thu, 10 May 2007 12:58:02 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from fledge.watson.org (fledge.watson.org [209.31.154.41]) by cyrus.watson.org (Postfix) with ESMTP id E8EFF46CD2; Thu, 10 May 2007 08:28:16 -0400 (EDT) Date: Thu, 10 May 2007 13:28:16 +0100 (BST) From: Robert Watson X-X-Sender: robert@fledge.watson.org To: Darren Reed In-Reply-To: <20070510111326.GA94093@hub.freebsd.org> Message-ID: <20070510132153.A91312@fledge.watson.org> References: <20070510111326.GA94093@hub.freebsd.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: hackers@freebsd.org Subject: Re: Experiences with 7.0-CURRENT and vmware. X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 May 2007 12:58:02 -0000 On Thu, 10 May 2007, Darren Reed wrote: > I'm using FreeBSD 7.0-CURRENT under vmware and there are a few issues. Generally speaking, I would suggest sending this post to current@, not hackers@, since your comments largely have to do between differences between -STABLE and -CURRENT, making it ideal fodder for the mailing list dedicated to -CURRENT development. > First, time. hint.hw.acpi.disabled="1" This appears to make _no_ difference > to time keeping on FreeBSD 7 and nor does it seem to have any impact on ACPI > being loaded. Do I need to recompile a new kernel without it or is there a > new way to disable ACPI? Have you tried hint.acpi.0.disabled=1 instead? This is what appears in acpi(4), and is what is used in various existing boot loader bits when I grep around. > I should add that FreeBSD 6, with the same setting, is no better and that I > need to run ntpdate every 5-10 minutes via crontab in order to keep good > time (timekeeping is *really* bad.) In one instance, i was watching "zpool > iostat 1" and it appeared like the rows were muching up at a rate of 2 a > second for a minute or so. How do I disable TSC timekeeping? (NetBSD has > this disabled by default in their kernels.) Or is there somethign else I > must do? kern.timecounter.hardware: ACPI-fast kern.timecounter.choice: TSC(800) ACPI-fast(1000) i8254(0) dummy(-1000000) I believe you can simply set kern.timecounter.hardware=APCI-fast and it will do what you expect. An interesting question is why it selects what is arguably the wrong one; a post to current@ might help resolve that. > Second, networking. Prior to FreeBSD-7, the driver to use inside vmware > workstation was lnc. It has worked and contiues to work great. No > problemo. FreeBSD-7 uses the "em" driver. To put it simply, it sucks in > comparison. When things really get bad I start seeing "em0: watchdog > timeout" messages on the console. I looked and I don't see a lnc driver > anywhere. Is there another alternative (le?) driver that I can use in place > of em, if so, how? Has VMware changed what network hardware they emulate, and/or does VMware offer options about what virtual hardware to expose? The if_em driver is for Intel ethernet cards; historically VMware has exposed a Lance ethernet device supported by the lnc(4) device driver; now that driver has indeed been replaced with le(4). But if if_em is probing, it suggests a VMware change rather than a FreeBSD change, which you may be able to revert by telling it to expose a Lance-style device as opposed to an Intel device. There was recently a rather large overhaul of the if_em driver in 7.x--I suggest e-mailing Jack Vogel (jfv) who is bother a FreeBSD committer and the Intel employee responsible for the if_em driver. He may rightly point out that this isn't real hardware, rather, virtual hardware, and therefore not supported by Intel, but it might also be that the new version of the driver contains a bug, there's an ACPI issue of some sort, etc. > Apart from these two issues (which are very central ones :-(), I'm using > FreeBSD in a 64bit vmware workstation environment quick successfully and ZFS > is quite happy with all the kva :-) ZFS and zpools are working just as I > expect, even if a bit slower due to vmware but I'm not cranking out > benchmarks here. > > Oh, and how do I fix ssh/rsh to do passwordless sessions? I'm trying to > setup cron jobs to automate various tasks but there's this small hurdle > called a password prompt that I can't seem to get rid of :-/ Generally speaking, this would be a discouraged configuration, but you will probably need to frob two settings: first, PermitEmptyPasswords in sshd_config, and second, force non-PAM validation by setting UsePAM to false. Instead of doing this, I would advise instead setting up an SSH key for the account, and not set a passphrase on the SSH key. This doesn't require any changing of the global sshd configuration and should offer most of the same benefits. Robert N M Watson Computer Laboratory University of Cambridge From owner-freebsd-hackers@FreeBSD.ORG Thu May 10 12:54:45 2007 Return-Path: X-Original-To: hackers@freebsd.org Delivered-To: freebsd-hackers@FreeBSD.ORG Received: by hub.freebsd.org (Postfix, from userid 680) id 6706816A407; Thu, 10 May 2007 12:54:45 +0000 (UTC) Date: Thu, 10 May 2007 12:54:45 +0000 From: Darren Reed To: Robert Watson Message-ID: <20070510125445.GA5460@hub.freebsd.org> References: <20070510111326.GA94093@hub.freebsd.org> <20070510132153.A91312@fledge.watson.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20070510132153.A91312@fledge.watson.org> User-Agent: Mutt/1.4.2.1i X-Mailman-Approved-At: Thu, 10 May 2007 13:08:25 +0000 Cc: current@freebsd.org Subject: Re: Experiences with 7.0-CURRENT and vmware. X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 May 2007 12:54:45 -0000 On Thu, May 10, 2007 at 01:28:16PM +0100, Robert Watson wrote: > > On Thu, 10 May 2007, Darren Reed wrote: > > >I'm using FreeBSD 7.0-CURRENT under vmware and there are a few issues. Redirecting to current@... > >First, time. hint.hw.acpi.disabled="1" This appears to make _no_ > >difference to time keeping on FreeBSD 7 and nor does it seem to have any > >impact on ACPI being loaded. Do I need to recompile a new kernel without > >it or is there a new way to disable ACPI? > > Have you tried hint.acpi.0.disabled=1 instead? This is what appears in > acpi(4), and is what is used in various existing boot loader bits when I > grep around. In another reply it was "hint.apic.0.disabled=1". My current loader.conf: vm.kmem_size=536870912 vm.kmem_size_max=536870912 unset acpi_load hint.acpi.0.disabled=1 hint.apci.0.disabled=1 hint.acpi.0.disabled="1" hint.apci.0.disabled="1" vfs.zfs.arc_max=402653184 Booting with this gives me: kernel: Timecounter "ACPI-safe" frequency 3579545 Hz quality 1000 and ACPI enabled. > >I should add that FreeBSD 6, with the same setting, is no better and that > >I need to run ntpdate every 5-10 minutes via crontab in order to keep good > >time (timekeeping is *really* bad.) In one instance, i was watching > >"zpool iostat 1" and it appeared like the rows were muching up at a rate > >of 2 a second for a minute or so. How do I disable TSC timekeeping? > >(NetBSD has this disabled by default in their kernels.) Or is there > >somethign else I must do? > > kern.timecounter.hardware: ACPI-fast > kern.timecounter.choice: TSC(800) ACPI-fast(1000) i8254(0) dummy(-1000000) > > I believe you can simply set kern.timecounter.hardware=APCI-fast and it > will do what you expect. An interesting question is why it selects what is > arguably the wrong one; a post to current@ might help resolve that. Hmm. # sysctl kern.timecounter.hardware="ACPI-fast" kern.timecounter.hardware: ACPI-safe sysctl: kern.timecounter.hardware: Invalid argument Or is this a loader.conf setting? > >Second, networking. Prior to FreeBSD-7, the driver to use inside vmware > >workstation was lnc. It has worked and contiues to work great. No > >problemo. FreeBSD-7 uses the "em" driver. To put it simply, it sucks in > >comparison. When things really get bad I start seeing "em0: watchdog > >timeout" messages on the console. I looked and I don't see a lnc driver > >anywhere. Is there another alternative (le?) driver that I can use in > >place of em, if so, how? > > Has VMware changed what network hardware they emulate, and/or does VMware > offer options about what virtual hardware to expose? I don't believe so. It still probes as pcn under NetBSD. > The if_em driver is > for Intel ethernet cards; historically VMware has exposed a Lance ethernet > device supported by the lnc(4) device driver; now that driver has indeed > been replaced with le(4). Right. I believe it still is lance, but somehow em is showing up. > But if if_em is probing, it suggests a VMware > change rather than a FreeBSD change, which you may be able to revert by > telling it to expose a Lance-style device as opposed to an Intel device. There's no way to choose the type of card vmware emulates. > Generally speaking, this would be a discouraged configuration, but you will > probably need to frob two settings: first, PermitEmptyPasswords in > sshd_config, and second, force non-PAM validation by setting UsePAM to > false. Instead of doing this, I would advise instead setting up an SSH key > for the account, and not set a passphrase on the SSH key. This doesn't > require any changing of the global sshd configuration and should offer most > of the same benefits. btw, there are instances where you can be promopted 6 times for a password when logging in with ssh, 3 times with "Password:" prompt and another three with "root@hostname's password:" promopt. Darren From owner-freebsd-hackers@FreeBSD.ORG Thu May 10 14:38:58 2007 Return-Path: X-Original-To: hackers@freebsd.org Delivered-To: freebsd-hackers@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id EBA8916A400; Thu, 10 May 2007 14:38:58 +0000 (UTC) (envelope-from doublef-ctm@yandex.ru) Received: from smtp1.yandex.ru (smtp1.yandex.ru [213.180.223.87]) by mx1.freebsd.org (Postfix) with ESMTP id 0902813C46A; Thu, 10 May 2007 14:38:57 +0000 (UTC) (envelope-from doublef-ctm@yandex.ru) Received: from [85.172.94.196] ([85.172.94.196]:731 "EHLO shark" smtp-auth: "doublef-ctm" TLS-CIPHER: TLS-PEER-CN1: ) by mail.yandex.ru with ESMTP id S3510375AbXEJOVx (ORCPT + 1 other); Thu, 10 May 2007 18:21:53 +0400 X-Comment: RFC 2476 MSA function at smtp1.yandex.ru logged sender identity as: doublef-ctm Received: by shark (Postfix, from userid 1000) id 278D817A3C; Thu, 10 May 2007 18:21:48 +0400 (MSD) Date: Thu, 10 May 2007 18:21:48 +0400 From: Sergey Zaharchenko To: hackers@freebsd.org Message-ID: <20070510142147.GA83797@shark.localdomain> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="pf9I7BMVVzbSWLtt" Content-Disposition: inline X-Listening-To: Silence User-Agent: Mutt/1.5.11 Cc: brian@FreeBSD.org Subject: My PPP timer PR [nag] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 May 2007 14:38:59 -0000 --pf9I7BMVVzbSWLtt Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Hello hackers@, bin/102747 has been sitting there for about 8 months, with no activity since it was assigned to brian@, all my mail to whom bounces [CC'd just in case]. The patch attached in the PR has been working for me since, so it not being fixed in the main tree isn't a problem with me. I just thought someone would benefit from it being committed... HAND, --=20 DoubleF No virus detected in this message. Ehrm, wait a minute... /kernel: pid 56921 (antivirus), uid 32000: exited on signal 9 Oh yes, no virus:) --pf9I7BMVVzbSWLtt Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.1 (FreeBSD) iD8DBQFGQyp7wo7hT/9lVdwRAq+FAJ9tdvFv9TMkeYd6OI8fRRg4LypX7QCcDyGs XK9dwhVLZYDnERLjHgyA3m0= =LvKO -----END PGP SIGNATURE----- --pf9I7BMVVzbSWLtt-- From owner-freebsd-hackers@FreeBSD.ORG Thu May 10 19:21:21 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id B4C1516A404 for ; Thu, 10 May 2007 19:21:21 +0000 (UTC) (envelope-from duane@dwlabs.ca) Received: from smtpout.eastlink.ca (smtpout.eastlink.ca [24.222.0.30]) by mx1.freebsd.org (Postfix) with ESMTP id 77ED313C455 for ; Thu, 10 May 2007 19:21:21 +0000 (UTC) (envelope-from duane@dwlabs.ca) Received: from ip04.eastlink.ca ([24.222.10.20]) by mta02.eastlink.ca (Sun Java System Messaging Server 6.2-4.03 (built Sep 22 2005)) with ESMTP id <0JHU00F4S9PK7TF0@mta02.eastlink.ca> for freebsd-hackers@freebsd.org; Thu, 10 May 2007 15:51:20 -0300 (ADT) Received: from blk-224-199-230.eastlink.ca (HELO dwpc.dwlabs.ca) ([24.224.199.230]) by ip04.eastlink.ca with ESMTP; Thu, 10 May 2007 15:23:10 -0300 Received: from dwpc.dwlabs.ca (imap.dwlabs.ca [192.168.0.10]) by dwpc.dwlabs.ca (8.13.8/8.13.8) with ESMTP id l4AIpOh8015412; Thu, 10 May 2007 15:51:30 -0300 (ADT envelope-from duane@dwpc.dwlabs.ca) Received: (from duane@localhost) by dwpc.dwlabs.ca (8.13.8/8.13.8/Submit) id l4AIpOnZ015411; Thu, 10 May 2007 15:51:24 -0300 (ADT envelope-from duane) Date: Thu, 10 May 2007 15:51:24 -0300 From: Duane Whitty In-reply-to: <20070510125445.GA5460@hub.freebsd.org> To: Darren Reed Message-id: <20070510185124.GA11407@dwpc.dwlabs.ca> MIME-version: 1.0 Content-type: text/plain; charset=us-ascii Content-transfer-encoding: 7BIT Content-disposition: inline X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: Ao8CAHAGQ0YY4MfmdGdsb2JhbACBXI4iATY X-IronPort-AV: i="4.14,518,1170648000"; d="scan'208"; a="213110432:sNHT37715067" X-Virus-Scanned: ClamAV 0.88.6/3225/Thu May 10 05:08:21 2007 on dwpc.dwlabs.ca X-Virus-Status: Clean X-Spam-Checker-Version: SpamAssassin 3.1.4 (2006-07-25) on dwpc.dwlabs.ca References: <20070510111326.GA94093@hub.freebsd.org> <20070510132153.A91312@fledge.watson.org> <20070510125445.GA5460@hub.freebsd.org> User-Agent: Mutt/1.4.2.2i X-Spam-Status: No, score=-2.1 required=5.0 tests=AWL,BAYES_00, UNPARSEABLE_RELAY autolearn=ham version=3.1.4 Cc: freebsd-hackers@freebsd.org Subject: Re: Experiences with 7.0-CURRENT and vmware. X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Duane Whitty List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 May 2007 19:21:21 -0000 On Thursday, 10 May 2007 at 12:54:45 +0000, Darren Reed wrote: > On Thu, May 10, 2007 at 01:28:16PM +0100, Robert Watson wrote: > > > > On Thu, 10 May 2007, Darren Reed wrote: > > > > >I'm using FreeBSD 7.0-CURRENT under vmware and there are a few issues. > > Redirecting to current@... > > > >First, time. hint.hw.acpi.disabled="1" This appears to make _no_ > > >difference to time keeping on FreeBSD 7 and nor does it seem to have any > > >impact on ACPI being loaded. Do I need to recompile a new kernel without > > >it or is there a new way to disable ACPI? > > > > Have you tried hint.acpi.0.disabled=1 instead? This is what appears in > > acpi(4), and is what is used in various existing boot loader bits when I > > grep around. > > In another reply it was "hint.apic.0.disabled=1". > My current loader.conf: > > vm.kmem_size=536870912 > vm.kmem_size_max=536870912 > unset acpi_load > hint.acpi.0.disabled=1 > hint.apci.0.disabled=1 > hint.acpi.0.disabled="1" > hint.apci.0.disabled="1" > vfs.zfs.arc_max=402653184 > > Booting with this gives me: > kernel: Timecounter "ACPI-safe" frequency 3579545 Hz quality 1000 > > and ACPI enabled. > > > >I should add that FreeBSD 6, with the same setting, is no better and that > > >I need to run ntpdate every 5-10 minutes via crontab in order to keep good > > >time (timekeeping is *really* bad.) In one instance, i was watching > > >"zpool iostat 1" and it appeared like the rows were muching up at a rate > > >of 2 a second for a minute or so. How do I disable TSC timekeeping? > > >(NetBSD has this disabled by default in their kernels.) Or is there > > >somethign else I must do? > > > > kern.timecounter.hardware: ACPI-fast > > kern.timecounter.choice: TSC(800) ACPI-fast(1000) i8254(0) dummy(-1000000) > > > > I believe you can simply set kern.timecounter.hardware=APCI-fast and it > > will do what you expect. An interesting question is why it selects what is > > arguably the wrong one; a post to current@ might help resolve that. > > Hmm. > > # sysctl kern.timecounter.hardware="ACPI-fast" > kern.timecounter.hardware: ACPI-safe > sysctl: kern.timecounter.hardware: Invalid argument > > Or is this a loader.conf setting? > > > >Second, networking. Prior to FreeBSD-7, the driver to use inside vmware > > >workstation was lnc. It has worked and contiues to work great. No > > >problemo. FreeBSD-7 uses the "em" driver. To put it simply, it sucks in > > >comparison. When things really get bad I start seeing "em0: watchdog > > >timeout" messages on the console. I looked and I don't see a lnc driver > > >anywhere. Is there another alternative (le?) driver that I can use in > > >place of em, if so, how? > > > > Has VMware changed what network hardware they emulate, and/or does VMware > > offer options about what virtual hardware to expose? > > I don't believe so. It still probes as pcn under NetBSD. > > > The if_em driver is > > for Intel ethernet cards; historically VMware has exposed a Lance ethernet > > device supported by the lnc(4) device driver; now that driver has indeed > > been replaced with le(4). > > Right. I believe it still is lance, but somehow em is showing up. > > > But if if_em is probing, it suggests a VMware > > change rather than a FreeBSD change, which you may be able to revert by > > telling it to expose a Lance-style device as opposed to an Intel device. > > There's no way to choose the type of card vmware emulates. > > > Generally speaking, this would be a discouraged configuration, but you will > > probably need to frob two settings: first, PermitEmptyPasswords in > > sshd_config, and second, force non-PAM validation by setting UsePAM to > > false. Instead of doing this, I would advise instead setting up an SSH key > > for the account, and not set a passphrase on the SSH key. This doesn't > > require any changing of the global sshd configuration and should offer most > > of the same benefits. > > btw, there are instances where you can be promopted 6 times for a > password when logging in with ssh, 3 times with "Password:" prompt > and another three with "root@hostname's password:" promopt. > > Darren This is on a 6 BETA install of VMware. As you can see from below I'm using if_le. This happened by default AFAIR. Maybe I can check my config though to see if there's a way to set the interface. I don't get a chance to use this as much as I'd like though so I can't claim to have much VMware knowledge. FreeBSD beastie.dwlabs.ca 7.0-CURRENT FreeBSD 7.0-CURRENT #2: Wed Mar 21 03:25:17 ADT 2007 duane@beastie.dwlabs.ca:/usr/obj/usr/src/sys/GENERIC i386 le0: flags=8843 metric 0 mtu 1500 options=8 ether 00:0c:29:a4:2a:26 inet 192.168.0.102 netmask 0xffffff00 broadcast 192.168.0.255 media: Ethernet autoselect status: active plip0: flags=108810 metric 0 mtu 1500 lo0: flags=8049 metric 0 mtu 16384 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3 inet6 ::1 prefixlen 128 inet 127.0.0.1 netmask 0xff000000 From owner-freebsd-hackers@FreeBSD.ORG Thu May 10 19:32:42 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id A1B4716A400 for ; Thu, 10 May 2007 19:32:42 +0000 (UTC) (envelope-from naylor.b.david@gmail.com) Received: from wx-out-0506.google.com (wx-out-0506.google.com [66.249.82.233]) by mx1.freebsd.org (Postfix) with ESMTP id 5BF9913C447 for ; Thu, 10 May 2007 19:32:42 +0000 (UTC) (envelope-from naylor.b.david@gmail.com) Received: by wx-out-0506.google.com with SMTP id s18so658038wxc for ; Thu, 10 May 2007 12:32:42 -0700 (PDT) DKIM-Signature: a=rsa-sha1; c=relaxed/relaxed; d=gmail.com; s=beta; h=domainkey-signature:received:received:from:to:subject:date:user-agent:cc:mime-version:content-type:content-transfer-encoding:message-id:sender; b=YKh5SqYgBz2PUc+xHxR8cBlR254vzm5SoSsWKP3Qb0CX/iruDVs+az2cSrL9kKbfawf5atrENkVx9s5oKtKHyJrmfVVGCJRKMRkvyobXVi6g4zKEfoGZl/WxhofFV5tKcxxfLqNFGxxXdwLVvM7gIy5m0JG8h9dSWcPdNx8xiqQ= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=beta; h=received:from:to:subject:date:user-agent:cc:mime-version:content-type:content-transfer-encoding:message-id:sender; b=N+TnNVTuQJ2/PxTpbCN+yDDvxdjWA0rZ9WwLhFvy2MP59i+XR6Je6wJynClDBPOQkF7UoI9SUCzESLDe0a8Fw4bVNxXGpR4VOS/+MEq0PKpv1yCdK3O+Uom90xA8uhEncoYWdLXwkbAA0M2l6g7kDeoTfQ8V7+CHhaFGMNxwuqA= Received: by 10.90.113.18 with SMTP id l18mr2174165agc.1178823941166; Thu, 10 May 2007 12:05:41 -0700 (PDT) Received: from ?0.0.0.0? ( [196.37.90.163]) by mx.google.com with ESMTP id q30sm13300905wrq.2007.05.10.12.05.36; Thu, 10 May 2007 12:05:39 -0700 (PDT) From: David Naylor To: jkh@freebsd.org Date: Thu, 10 May 2007 21:05:21 +0200 User-Agent: KMail/1.9.5 MIME-Version: 1.0 Content-Type: multipart/signed; boundary="nextPart15314634.UlFRPv0S4Q"; protocol="application/pgp-signature"; micalg=pgp-sha1 Content-Transfer-Encoding: 7bit Message-Id: <200705102105.27271.blackdragon@highveldmail.co.za> Sender: David Naylor X-Mailman-Approved-At: Thu, 10 May 2007 23:39:18 +0000 Cc: freebsd-hackers@freebsd.org Subject: New FreeBSD package system (a.k.a. Daemon Package System (dps)) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 May 2007 19:32:42 -0000 --nextPart15314634.UlFRPv0S4Q Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Content-Disposition: inline Dear Jordan Recently I stumbled across a document you wrote in 2001, entitled "FreeBSD= =20 installation and package tools, past, present and future". I find FreeBSD= =20 appealing and I would like to contribute it its success, and as your articl= e=20 describes, the installation and packaging system is lacking. Since the=20 installation system is being tackled under a SoC project I am hoping to giv= e=20 the packaging system a go. =20 I was hoping you could help me with an update about the situation with pkg.= I=20 have searched the FreeBSD mailing lists and have found little information o= n=20 the package system. Once I have a (much more) complete understanding of th= e=20 packaging system (and providing there is work to be done) I would like to=20 write up a proposal to solve the problems, and perhaps provide some=20 innovating new capabilities. =20 After that I will gladly contribute what I can to this (possible) project a= nd=20 hopefully further and improve FreeBSD. Any assistance or information you c= an=20 give I will be greatly appreciate. =20 I look forward to your reply. =20 David --nextPart15314634.UlFRPv0S4Q Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.5 (GNU/Linux) iD8DBQBGQ2z3yqzxLKpyZI8RApAmAJwKWitmM00p8Z3XfpDNr9LMct6oYwCdF78w p61fwhYjE8kGRIrTgptM+gc= =PBfW -----END PGP SIGNATURE----- --nextPart15314634.UlFRPv0S4Q-- From owner-freebsd-hackers@FreeBSD.ORG Fri May 11 00:11:22 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id B385F16A400 for ; Fri, 11 May 2007 00:11:22 +0000 (UTC) (envelope-from freebsd-hackers@m.gmane.org) Received: from ciao.gmane.org (main.gmane.org [80.91.229.2]) by mx1.freebsd.org (Postfix) with ESMTP id 387E313C44C for ; Fri, 11 May 2007 00:11:22 +0000 (UTC) (envelope-from freebsd-hackers@m.gmane.org) Received: from list by ciao.gmane.org with local (Exim 4.43) id 1HmIiP-0003Jm-QR for freebsd-hackers@freebsd.org; Fri, 11 May 2007 02:10:38 +0200 Received: from 89-172-50-90.adsl.net.t-com.hr ([89.172.50.90]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Fri, 11 May 2007 02:10:33 +0200 Received: from ivoras by 89-172-50-90.adsl.net.t-com.hr with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Fri, 11 May 2007 02:10:33 +0200 X-Injected-Via-Gmane: http://gmane.org/ To: freebsd-hackers@freebsd.org From: Ivan Voras Date: Fri, 11 May 2007 02:10:05 +0200 Lines: 61 Message-ID: References: <200705102105.27271.blackdragon@highveldmail.co.za> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="------------enigDB6EC1D000090CDB58D57677" X-Complaints-To: usenet@sea.gmane.org X-Gmane-NNTP-Posting-Host: 89-172-50-90.adsl.net.t-com.hr User-Agent: Thunderbird 1.5.0.10 (Windows/20070221) In-Reply-To: <200705102105.27271.blackdragon@highveldmail.co.za> X-Enigmail-Version: 0.94.3.0 Sender: news Subject: Re: New FreeBSD package system (a.k.a. Daemon Package System (dps)) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 May 2007 00:11:22 -0000 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enigDB6EC1D000090CDB58D57677 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable David Naylor wrote: > Dear Jordan >=20 > Recently I stumbled across a document you wrote in 2001, entitled "Free= BSD=20 > installation and package tools, past, present and future". I find Free= BSD=20 > appealing and I would like to contribute it its success, and as your ar= ticle=20 > describes, the installation and packaging system is lacking. Since the= =20 > installation system is being tackled under a SoC project I am hoping to= give=20 > the packaging system a go. =20 I've just read the document, and since I was also thinking about the ports/packages system lately, here are a few random ideas: - I think it's time to give up on using BDB+directory tree full of text files for storing the installed packages database, and I propose all of this be replaced by a single SQLite database. SQLite is public domain (can be slurped into base system), embeddable, stores all data in a single file, lightweight, fast, and can be used to do fancy things such as reporting. The current pkg_info's behaviour that takes *noticable* *time* to generate 1 line of output per package is horrible in this day and age. And the upcoming X11/X.Org 7.2 with cca 400 packages (which is in itself horrible just for a X11 system) will make it unbearable (also portupgrade which already looks slower by the day). - A quick test confirms that the current bsdtar will happily ignore any extra data at the end of a tgz/tbz archive, so package metadata can be embedded there, thus conserving existing infrastructure and being fast to parse. I suggest encoding this metadata in a sane and easy to parse XML structure. I cannot currently actively participate in implementing proposed things, but I can give advice on sqlite, database and xml schemas if anyone wants to... --------------enigDB6EC1D000090CDB58D57677 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.4 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFGQ7RdldnAQVacBcgRAsC5AJ9eOrtH/PZ1f0FCpUNCd5e8P0cepACgjJn1 4RDFx35acLcdnefjtDqgSEE= =/yM6 -----END PGP SIGNATURE----- --------------enigDB6EC1D000090CDB58D57677-- From owner-freebsd-hackers@FreeBSD.ORG Fri May 11 01:33:17 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 8AEE416A403 for ; Fri, 11 May 2007 01:33:17 +0000 (UTC) (envelope-from julian@elischer.org) Received: from outF.internet-mail-service.net (outF.internet-mail-service.net [216.240.47.229]) by mx1.freebsd.org (Postfix) with ESMTP id 7466213C447 for ; Fri, 11 May 2007 01:33:17 +0000 (UTC) (envelope-from julian@elischer.org) Received: from mx0.idiom.com (HELO idiom.com) (216.240.32.160) by out.internet-mail-service.net (qpsmtpd/0.32) with ESMTP; Thu, 10 May 2007 18:33:17 -0700 Received: from julian-mac.elischer.org (home.elischer.org [216.240.48.38]) by idiom.com (Postfix) with ESMTP id 8AB99125B4A; Thu, 10 May 2007 18:33:16 -0700 (PDT) Message-ID: <4643C7DB.6000408@elischer.org> Date: Thu, 10 May 2007 18:33:15 -0700 From: Julian Elischer User-Agent: Thunderbird 2.0.0.0 (Macintosh/20070326) MIME-Version: 1.0 To: Ivan Voras References: <200705102105.27271.blackdragon@highveldmail.co.za> In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-hackers@freebsd.org Subject: Re: New FreeBSD package system (a.k.a. Daemon Package System (dps)) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 May 2007 01:33:17 -0000 Ivan Voras wrote: > David Naylor wrote: >> Dear Jordan >> >> Recently I stumbled across a document you wrote in 2001, entitled "FreeBSD >> installation and package tools, past, present and future". I find FreeBSD >> appealing and I would like to contribute it its success, and as your article >> describes, the installation and packaging system is lacking. Since the >> installation system is being tackled under a SoC project I am hoping to give >> the packaging system a go. > > I've just read the document, and since I was also thinking about the > ports/packages system lately, here are a few random ideas: > > - I think it's time to give up on using BDB+directory tree full of text > files for storing the installed packages database, and I propose all of > this be replaced by a single SQLite database. SQLite is public domain > (can be slurped into base system), embeddable, stores all data in a > single file, lightweight, fast, and can be used to do fancy things such > as reporting. The current pkg_info's behaviour that takes *noticable* > *time* to generate 1 line of output per package is horrible in this day > and age. And the upcoming X11/X.Org 7.2 with cca 400 packages (which is > in itself horrible just for a X11 system) will make it unbearable (also > portupgrade which already looks slower by the day). ok, let me give you some words that come from the wisdom of having done this for 30 years. "Use an SQL DB file for this over my dead body".. Now having said that, I need to modify it a bit and explain. Explanation: The number of times I've gone into /var/db/pkg to read the files to figure out what the "fsck" is going on, is very large. It is also possible to delete files and edit them. This is very important to me in this imperfect world when the pkg system gets things wrong (oh no, don't tell me that the new pkg system never gets things wrong.. like including a file in 2 packages). Modification: Now, I have no objection to a DB file of some sort IF (and only if) it is somehow rebuildable from a text base which is kept somewhere, like the password database is done. but PLEASE, NO SQL in the base system! > > - A quick test confirms that the current bsdtar will happily ignore any > extra data at the end of a tgz/tbz archive, so package metadata can be > embedded there, thus conserving existing infrastructure and being fast > to parse. I suggest encoding this metadata in a sane and easy to parse > XML structure. > > I cannot currently actively participate in implementing proposed things, > but I can give advice on sqlite, database and xml schemas if anyone > wants to... > > From owner-freebsd-hackers@FreeBSD.ORG Fri May 11 01:48:22 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id A0D0216A402 for ; Fri, 11 May 2007 01:48:22 +0000 (UTC) (envelope-from mwm-keyword-freebsdhackers2.e313df@mired.org) Received: from mired.org (vpn.mired.org [66.92.153.74]) by mx1.freebsd.org (Postfix) with SMTP id 2E08813C44C for ; Fri, 11 May 2007 01:48:22 +0000 (UTC) (envelope-from mwm-keyword-freebsdhackers2.e313df@mired.org) Received: (qmail 29728 invoked by uid 1001); 11 May 2007 01:47:49 -0000 Received: by bhuda.mired.org (tmda-sendmail, from uid 1001); Thu, 10 May 2007 21:47:49 -0400 MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <17987.52037.112351.872442@bhuda.mired.org> Date: Thu, 10 May 2007 21:47:49 -0400 To: Ivan Voras In-Reply-To: References: <200705102105.27271.blackdragon@highveldmail.co.za> X-Mailer: VM 7.19 under Emacs 21.3.1 X-Primary-Address: mwm@mired.org X-face: "5Mnwy%?j>IIV\)A=):rjWL~NB2aH[}Yq8Z=u~vJ`"(,&SiLvbbz2W`; h9L,Yg`+vb1>RG% *h+%X^n0EZd>TM8_IB;a8F?(Fb"lw'IgCoyM.[Lg#r\ X-Delivery-Agent: TMDA/1.1.11 (Ladyburn) From: Mike Meyer Cc: freebsd-hackers@freebsd.org Subject: Re: New FreeBSD package system (a.k.a. Daemon Package System (dps)) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 May 2007 01:48:22 -0000 In , Ivan Voras typed: > I cannot currently actively participate in implementing proposed things, > but I can give advice on sqlite, database and xml schemas if anyone > wants to... One of the things that would be nice for a replacement to do would be to correctly install i386 packages on amd64 platforms (and similar things). The binaries will run if we have all the libraries installed, but getting them installed is painful. At the very least, an installed package needs to know what platform it was built for, so that packages being installed cross-platform have a chance of figuring out whether or not the installed version of FOO will work for them, or they need to get a version for their specific platform. If the new version can't make cross-platform installs work, hopefully that goal can be kept in mind while the work is beign done. Personally, I'd still like LOCALBASE to move out of /usr/local. Maybe it's time to reconsider that. http://www.mired.org/consulting.html Independent Network/Unix/Perforce consultant, email for more information. From owner-freebsd-hackers@FreeBSD.ORG Fri May 11 01:51:57 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 6058616A405 for ; Fri, 11 May 2007 01:51:57 +0000 (UTC) (envelope-from kris@obsecurity.org) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.freebsd.org (Postfix) with ESMTP id 4C4DD13C44C for ; Fri, 11 May 2007 01:51:57 +0000 (UTC) (envelope-from kris@obsecurity.org) Received: from obsecurity.dyndns.org (elvis.mu.org [192.203.228.196]) by elvis.mu.org (Postfix) with ESMTP id DD46D1A4D89; Thu, 10 May 2007 18:52:41 -0700 (PDT) Received: by obsecurity.dyndns.org (Postfix, from userid 1000) id 9910451226; Thu, 10 May 2007 21:51:56 -0400 (EDT) Date: Thu, 10 May 2007 21:51:56 -0400 From: Kris Kennaway To: Mike Meyer Message-ID: <20070511015156.GA77895@xor.obsecurity.org> References: <200705102105.27271.blackdragon@highveldmail.co.za> <17987.52037.112351.872442@bhuda.mired.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <17987.52037.112351.872442@bhuda.mired.org> User-Agent: Mutt/1.4.2.2i Cc: freebsd-hackers@freebsd.org, Ivan Voras Subject: Re: New FreeBSD package system (a.k.a. Daemon Package System (dps)) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 May 2007 01:51:57 -0000 On Thu, May 10, 2007 at 09:47:49PM -0400, Mike Meyer wrote: > In , Ivan Voras typed: > > I cannot currently actively participate in implementing proposed things, > > but I can give advice on sqlite, database and xml schemas if anyone > > wants to... > > One of the things that would be nice for a replacement to do would be > to correctly install i386 packages on amd64 platforms (and similar > things). This has nothing to do with a new packaging system and can be done today if someone cares enough to work on it. > The binaries will run if we have all the libraries installed, > but getting them installed is painful. At the very least, an installed > package needs to know what platform it was built for, so that packages > being installed cross-platform have a chance of figuring out whether > or not the installed version of FOO will work for them, or they need > to get a version for their specific platform. If the new version can't > make cross-platform installs work, hopefully that goal can be kept in > mind while the work is beign done. > > Personally, I'd still like LOCALBASE to move out of /usr/local. Maybe > it's time to reconsider that. Not gonna happen as a default, but you can change it on your systems if you like. Kris From owner-freebsd-hackers@FreeBSD.ORG Fri May 11 02:03:55 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id E3DAC16A405 for ; Fri, 11 May 2007 02:03:55 +0000 (UTC) (envelope-from mwm-keyword-freebsdhackers2.e313df@mired.org) Received: from mired.org (vpn.mired.org [66.92.153.74]) by mx1.freebsd.org (Postfix) with SMTP id 8673913C45E for ; Fri, 11 May 2007 02:03:55 +0000 (UTC) (envelope-from mwm-keyword-freebsdhackers2.e313df@mired.org) Received: (qmail 29960 invoked by uid 1001); 11 May 2007 02:03:22 -0000 Received: by bhuda.mired.org (tmda-sendmail, from uid 1001); Thu, 10 May 2007 22:03:22 -0400 MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <17987.52970.398402.580727@bhuda.mired.org> Date: Thu, 10 May 2007 22:03:22 -0400 To: Kris Kennaway In-Reply-To: <20070511015156.GA77895@xor.obsecurity.org> References: <200705102105.27271.blackdragon@highveldmail.co.za> <17987.52037.112351.872442@bhuda.mired.org> <20070511015156.GA77895@xor.obsecurity.org> X-Mailer: VM 7.19 under Emacs 21.3.1 X-Primary-Address: mwm@mired.org X-face: "5Mnwy%?j>IIV\)A=):rjWL~NB2aH[}Yq8Z=u~vJ`"(,&SiLvbbz2W`; h9L,Yg`+vb1>RG% *h+%X^n0EZd>TM8_IB;a8F?(Fb"lw'IgCoyM.[Lg#r\ X-Delivery-Agent: TMDA/1.1.11 (Ladyburn) From: Mike Meyer Cc: freebsd-hackers@freebsd.org, Ivan Voras Subject: Re: New FreeBSD package system (a.k.a. Daemon Package System (dps)) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 May 2007 02:03:56 -0000 In <20070511015156.GA77895@xor.obsecurity.org>, Kris Kennaway typed: > On Thu, May 10, 2007 at 09:47:49PM -0400, Mike Meyer wrote: > > In , Ivan Voras typed: > > > I cannot currently actively participate in implementing proposed things, > > > but I can give advice on sqlite, database and xml schemas if anyone > > > wants to... > > One of the things that would be nice for a replacement to do would be > > to correctly install i386 packages on amd64 platforms (and similar > > things). > This has nothing to do with a new packaging system and can be done > today if someone cares enough to work on it. Well, yeah - *anything* can be done if someone cares enough to work on it - it's all just SMOP. You could definitely put enough smarts into the package installer do this without actually changing the packaging system. But if we're gonna change the packaging system anyway, why not consider adding information that the package building software already has so that the package installer software doesn't have to try and figure it out? > > The binaries will run if we have all the libraries installed, > > but getting them installed is painful. At the very least, an installed > > package needs to know what platform it was built for, so that packages > > being installed cross-platform have a chance of figuring out whether > > or not the installed version of FOO will work for them, or they need > > to get a version for their specific platform. If the new version can't > > make cross-platform installs work, hopefully that goal can be kept in > > mind while the work is beign done. > > > > Personally, I'd still like LOCALBASE to move out of /usr/local. Maybe > > it's time to reconsider that. > Not gonna happen as a default, but you can change it on your systems > if you like. Not very reliably. http://www.mired.org/consulting.html Independent Network/Unix/Perforce consultant, email for more information. From owner-freebsd-hackers@FreeBSD.ORG Fri May 11 02:12:50 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id E336A16A406 for ; Fri, 11 May 2007 02:12:50 +0000 (UTC) (envelope-from kris@obsecurity.org) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.freebsd.org (Postfix) with ESMTP id CE46413C45E for ; Fri, 11 May 2007 02:12:50 +0000 (UTC) (envelope-from kris@obsecurity.org) Received: from obsecurity.dyndns.org (elvis.mu.org [192.203.228.196]) by elvis.mu.org (Postfix) with ESMTP id 6DDFA1A3C19; Thu, 10 May 2007 19:13:35 -0700 (PDT) Received: by obsecurity.dyndns.org (Postfix, from userid 1000) id 18C88513BC; Thu, 10 May 2007 22:12:50 -0400 (EDT) Date: Thu, 10 May 2007 22:12:49 -0400 From: Kris Kennaway To: Mike Meyer Message-ID: <20070511021249.GA78729@xor.obsecurity.org> References: <200705102105.27271.blackdragon@highveldmail.co.za> <17987.52037.112351.872442@bhuda.mired.org> <20070511015156.GA77895@xor.obsecurity.org> <17987.52970.398402.580727@bhuda.mired.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <17987.52970.398402.580727@bhuda.mired.org> User-Agent: Mutt/1.4.2.2i Cc: freebsd-hackers@freebsd.org, Ivan Voras , Kris Kennaway Subject: Re: New FreeBSD package system (a.k.a. Daemon Package System (dps)) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 May 2007 02:12:51 -0000 On Thu, May 10, 2007 at 10:03:22PM -0400, Mike Meyer wrote: > In <20070511015156.GA77895@xor.obsecurity.org>, Kris Kennaway typed: > > On Thu, May 10, 2007 at 09:47:49PM -0400, Mike Meyer wrote: > > > In , Ivan Voras typed: > > > > I cannot currently actively participate in implementing proposed things, > > > > but I can give advice on sqlite, database and xml schemas if anyone > > > > wants to... > > > One of the things that would be nice for a replacement to do would be > > > to correctly install i386 packages on amd64 platforms (and similar > > > things). > > This has nothing to do with a new packaging system and can be done > > today if someone cares enough to work on it. > > Well, yeah - *anything* can be done if someone cares enough to work on > it - it's all just SMOP. You could definitely put enough smarts into > the package installer do this without actually changing the packaging > system. But if we're gonna change the packaging system anyway, why not > consider adding information that the package building software already > has so that the package installer software doesn't have to try and > figure it out? Sure, we could pile on some more features onto a 6 year old design document that never got out of the design phase, or someone could just go and make the relatively minor changes to support i386 packages on amd64 now. I guess it's always more fun to build dream castles though :) > > Not gonna happen as a default, but you can change it on your systems > > if you like. > > Not very reliably. Best I can offer ;) Kris From owner-freebsd-hackers@FreeBSD.ORG Fri May 11 03:16:10 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 63AEF16A400 for ; Fri, 11 May 2007 03:16:10 +0000 (UTC) (envelope-from mwm-keyword-freebsdhackers2.e313df@mired.org) Received: from mired.org (vpn.mired.org [66.92.153.74]) by mx1.freebsd.org (Postfix) with SMTP id 070BC13C4AE for ; Fri, 11 May 2007 03:16:09 +0000 (UTC) (envelope-from mwm-keyword-freebsdhackers2.e313df@mired.org) Received: (qmail 31661 invoked by uid 1001); 11 May 2007 03:15:37 -0000 Received: by bhuda.mired.org (tmda-sendmail, from uid 1001); Thu, 10 May 2007 23:15:37 -0400 MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <17987.57305.7130.873114@bhuda.mired.org> Date: Thu, 10 May 2007 23:15:37 -0400 To: Kris Kennaway In-Reply-To: <20070511021249.GA78729@xor.obsecurity.org> References: <200705102105.27271.blackdragon@highveldmail.co.za> <17987.52037.112351.872442@bhuda.mired.org> <20070511015156.GA77895@xor.obsecurity.org> <17987.52970.398402.580727@bhuda.mired.org> <20070511021249.GA78729@xor.obsecurity.org> X-Mailer: VM 7.19 under Emacs 21.3.1 X-Primary-Address: mwm@mired.org X-face: "5Mnwy%?j>IIV\)A=):rjWL~NB2aH[}Yq8Z=u~vJ`"(,&SiLvbbz2W`; h9L,Yg`+vb1>RG% *h+%X^n0EZd>TM8_IB;a8F?(Fb"lw'IgCoyM.[Lg#r\ X-Delivery-Agent: TMDA/1.1.11 (Ladyburn) From: Mike Meyer Cc: freebsd-hackers@freebsd.org, Ivan Voras Subject: Re: New FreeBSD package system (a.k.a. Daemon Package System (dps)) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 May 2007 03:16:10 -0000 In <20070511021249.GA78729@xor.obsecurity.org>, Kris Kennaway typed: > On Thu, May 10, 2007 at 10:03:22PM -0400, Mike Meyer wrote: > > In <20070511015156.GA77895@xor.obsecurity.org>, Kris Kennaway typed: > > > On Thu, May 10, 2007 at 09:47:49PM -0400, Mike Meyer wrote: > > > > In , Ivan Voras typed: > > > > > I cannot currently actively participate in implementing proposed things, > > > > > but I can give advice on sqlite, database and xml schemas if anyone > > > > > wants to... > > > > One of the things that would be nice for a replacement to do would be > > > > to correctly install i386 packages on amd64 platforms (and similar > > > > things). > > > This has nothing to do with a new packaging system and can be done > > > today if someone cares enough to work on it. > > > > Well, yeah - *anything* can be done if someone cares enough to work on > > it - it's all just SMOP. You could definitely put enough smarts into > > the package installer do this without actually changing the packaging > > system. But if we're gonna change the packaging system anyway, why not > > consider adding information that the package building software already > > has so that the package installer software doesn't have to try and > > figure it out? > > Sure, we could pile on some more features onto a 6 year old design > document that never got out of the design phase, or someone could just > go and make the relatively minor changes to support i386 packages on > amd64 now. I guess it's always more fun to build dream castles though > :) Last time I looked into this, it didn't look relatively minor to me. But hey, if there's a document listing what needs to be done somewhere and it's really relatively minor, I need this bad enough to deal with that. On the other hand, if no one has actually done the work to figure out what this would really take, is wishful thinking really enough to keep a very desirable feature (well, it's desirable enough that most Linux platforms seem to offer it) from even being considered? > > > Not gonna happen as a default, but you can change it on your systems > > > if you like. > > Not very reliably. > Best I can offer ;) Is this the new motto for FreeBSD? Good QA practices would have the ports built with that knob set to something something other than the default at regular intervals. Lately things have been better, but I've found broken things in the last twelve months. http://www.mired.org/consulting.html Independent Network/Unix/Perforce consultant, email for more information. From owner-freebsd-hackers@FreeBSD.ORG Fri May 11 03:20:45 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id AEE5C16A403; Fri, 11 May 2007 03:20:45 +0000 (UTC) (envelope-from youshi10@u.washington.edu) Received: from mxout2.cac.washington.edu (mxout2.cac.washington.edu [140.142.33.4]) by mx1.freebsd.org (Postfix) with ESMTP id 8BFCD13C483; Fri, 11 May 2007 03:20:45 +0000 (UTC) (envelope-from youshi10@u.washington.edu) Received: from smtp.washington.edu (smtp.washington.edu [140.142.33.7] (may be forged)) by mxout2.cac.washington.edu (8.13.7+UW06.06/8.13.7+UW07.03) with ESMTP id l4B3KiYI026990 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Thu, 10 May 2007 20:20:44 -0700 X-Auth-Received: from [192.168.10.45] (c-67-187-164-17.hsd1.ca.comcast.net [67.187.164.17]) (authenticated authid=youshi10) by smtp.washington.edu (8.13.7+UW06.06/8.13.7+UW07.03) with ESMTP id l4B3Kh4Z028871 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NOT); Thu, 10 May 2007 20:20:44 -0700 Message-ID: <4643E10A.5030104@u.washington.edu> Date: Thu, 10 May 2007 20:20:42 -0700 From: Garrett Cooper User-Agent: Thunderbird 2.0.0.0 (Windows/20070326) MIME-Version: 1.0 To: David Naylor References: <200705102105.27271.blackdragon@highveldmail.co.za> In-Reply-To: <200705102105.27271.blackdragon@highveldmail.co.za> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-PMX-Version: 5.3.1.294258, Antispam-Engine: 2.5.1.298604, Antispam-Data: 2007.5.10.195834 X-Uwash-Spam: Gauge=IIIIIII, Probability=7%, Report='__CT 0, __CTE 0, __CT_TEXT_PLAIN 0, __HAS_MSGID 0, __MIME_TEXT_ONLY 0, __MIME_VERSION 0, __SANE_MSGID 0, __USER_AGENT 0' Cc: jkh@freebsd.org, freebsd-hackers@freebsd.org Subject: Re: New FreeBSD package system (a.k.a. Daemon Package System (dps)) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 May 2007 03:20:45 -0000 David Naylor wrote: > Dear Jordan > > Recently I stumbled across a document you wrote in 2001, entitled "FreeBSD > installation and package tools, past, present and future". I find FreeBSD > appealing and I would like to contribute it its success, and as your article > describes, the installation and packaging system is lacking. Since the > installation system is being tackled under a SoC project I am hoping to give > the packaging system a go. > > I was hoping you could help me with an update about the situation with pkg. I > have searched the FreeBSD mailing lists and have found little information on > the package system. Once I have a (much more) complete understanding of the > packaging system (and providing there is work to be done) I would like to > write up a proposal to solve the problems, and perhaps provide some > innovating new capabilities. > > After that I will gladly contribute what I can to this (possible) project and > hopefully further and improve FreeBSD. Any assistance or information you can > give I will be greatly appreciate. > > I look forward to your reply. > > David Yipes. The name of the game is to get something working in the base system, instead of dragging in multiple 3rd party packages, with licensing schemes that may not be aligned with the BSD license. SQL's great, SQL's wonderful for db use, but the problem is that supporting it from my POV would cause a lot more grief and waiting than having me wait a few months to get a BDB compatible scheme out the door. If only Oracle didn't make BDB 3.x non-BSD license friendly though.. that would be nice.. -Garrett From owner-freebsd-hackers@FreeBSD.ORG Fri May 11 05:18:54 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 54B1A16A407 for ; Fri, 11 May 2007 05:18:54 +0000 (UTC) (envelope-from kris@obsecurity.org) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.freebsd.org (Postfix) with ESMTP id 1EDAE13C469 for ; Fri, 11 May 2007 05:18:54 +0000 (UTC) (envelope-from kris@obsecurity.org) Received: from obsecurity.dyndns.org (elvis.mu.org [192.203.228.196]) by elvis.mu.org (Postfix) with ESMTP id 968111A3C19; Thu, 10 May 2007 22:19:38 -0700 (PDT) Received: by obsecurity.dyndns.org (Postfix, from userid 1000) id BF9405160C; Fri, 11 May 2007 01:18:52 -0400 (EDT) Date: Fri, 11 May 2007 01:18:52 -0400 From: Kris Kennaway To: Mike Meyer Message-ID: <20070511051852.GA89359@xor.obsecurity.org> References: <200705102105.27271.blackdragon@highveldmail.co.za> <17987.52037.112351.872442@bhuda.mired.org> <20070511015156.GA77895@xor.obsecurity.org> <17987.52970.398402.580727@bhuda.mired.org> <20070511021249.GA78729@xor.obsecurity.org> <17987.57305.7130.873114@bhuda.mired.org> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="fUYQa+Pmc3FrFX/N" Content-Disposition: inline In-Reply-To: <17987.57305.7130.873114@bhuda.mired.org> User-Agent: Mutt/1.4.2.2i Cc: freebsd-hackers@freebsd.org, Ivan Voras , Kris Kennaway Subject: Re: New FreeBSD package system (a.k.a. Daemon Package System (dps)) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 May 2007 05:18:54 -0000 --fUYQa+Pmc3FrFX/N Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Thu, May 10, 2007 at 11:15:37PM -0400, Mike Meyer wrote: > In <20070511021249.GA78729@xor.obsecurity.org>, Kris Kennaway typed: > > On Thu, May 10, 2007 at 10:03:22PM -0400, Mike Meyer wrote: > > > In <20070511015156.GA77895@xor.obsecurity.org>, Kris Kennaway typed: > > > > On Thu, May 10, 2007 at 09:47:49PM -0400, Mike Meyer wrote: > > > > > In , Ivan Voras typed: > > > > > > I cannot currently actively participate in implementing propose= d things, > > > > > > but I can give advice on sqlite, database and xml schemas if an= yone > > > > > > wants to... > > > > > One of the things that would be nice for a replacement to do woul= d be > > > > > to correctly install i386 packages on amd64 platforms (and similar > > > > > things).=20 > > > > This has nothing to do with a new packaging system and can be done > > > > today if someone cares enough to work on it. > > >=20 > > > Well, yeah - *anything* can be done if someone cares enough to work on > > > it - it's all just SMOP. You could definitely put enough smarts into > > > the package installer do this without actually changing the packaging > > > system. But if we're gonna change the packaging system anyway, why not > > > consider adding information that the package building software already > > > has so that the package installer software doesn't have to try and > > > figure it out? > >=20 > > Sure, we could pile on some more features onto a 6 year old design > > document that never got out of the design phase, or someone could just > > go and make the relatively minor changes to support i386 packages on > > amd64 now. I guess it's always more fun to build dream castles though > > :) >=20 > Last time I looked into this, it didn't look relatively minor to > me. But hey, if there's a document listing what needs to be done > somewhere and it's really relatively minor, I need this bad enough to > deal with that. There are a few ways you can go. The simplest is to install a complete i386 world in e.g. /compat/ia32 and have i386 packages installed there, and change the kernel to do a "magic directory lookup" for i386 binaries that does path munging like the linuxulator does with /compat/linux. If you want the i386 packages to live in /usr/local alongside the amd64 packages, you'll need to do something like adding an @arch directive to the +CONTENTS file recorded in /var/db/pkg, and add some checking to pkg_add to ensure that when you install a package, everything it depends on was built for the same architecture. You'd need to also add these checks to bsd.port.mk so it exits gracefully when someone tries to natively compile a port for which non-native dependencies are installed (e.g. it's going to be really unhappy if it finds i386 headers or libraries). This method might be more trouble than it's worth. > On the other hand, if no one has actually done the > work to figure out what this would really take, is wishful thinking > really enough to keep a very desirable feature (well, it's desirable > enough that most Linux platforms seem to offer it) from even being > considered? You misunderstand me: by all means people should work on improving our package infrastructure -- but it's wrong to pin your hopes on a possibly mythical future event when you can easily solve a problem today. > > > > Not gonna happen as a default, but you can change it on your systems > > > > if you like. > > > Not very reliably. > > Best I can offer ;) >=20 > Is this the new motto for FreeBSD? Good QA practices would have the > ports built with that knob set to something something other than the > default at regular intervals. Lately things have been better, but > I've found broken things in the last twelve months. You'll be delighted to know that I have been doing precisely this for the past few years. If you're interested I'll CC you the errors from my next build so you can help work on improving compliance. Kris --fUYQa+Pmc3FrFX/N Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (FreeBSD) iD8DBQFGQ/y8Wry0BWjoQKURAjjVAKDBKgwGZme0Qnl96dvBsB8Q3obR1wCdGNYR 8h35fdUFr6E8wShQq4STalE= =m/Cd -----END PGP SIGNATURE----- --fUYQa+Pmc3FrFX/N-- From owner-freebsd-hackers@FreeBSD.ORG Fri May 11 07:35:37 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 00E9716A405 for ; Fri, 11 May 2007 07:35:37 +0000 (UTC) (envelope-from ota@j.email.ne.jp) Received: from mail.asahi-net.or.jp (mail1.asahi-net.or.jp [202.224.39.197]) by mx1.freebsd.org (Postfix) with ESMTP id CB2F913C487 for ; Fri, 11 May 2007 07:35:36 +0000 (UTC) (envelope-from ota@j.email.ne.jp) Received: from amd64-freebsd.advok.com (pool-141-151-87-91.phlapa.east.verizon.net [141.151.87.91]) by mail.asahi-net.or.jp (Postfix) with ESMTP id DF8943EEBB; Fri, 11 May 2007 16:15:52 +0900 (JST) Date: Fri, 11 May 2007 03:10:48 -0400 From: Yoshihiro Ota To: Mike Meyer Message-Id: <20070511031048.30706c49.ota@j.email.ne.jp> In-Reply-To: <17987.52970.398402.580727@bhuda.mired.org> References: <200705102105.27271.blackdragon@highveldmail.co.za> <17987.52037.112351.872442@bhuda.mired.org> <20070511015156.GA77895@xor.obsecurity.org> <17987.52970.398402.580727@bhuda.mired.org> X-Mailer: Sylpheed version 2.2.9 (GTK+ 2.10.12; i386-portbld-freebsd6.1) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Cc: freebsd-hackers@freebsd.org, Ivan Voras , Kris Kennaway Subject: Re: New FreeBSD package system (a.k.a. Daemon Package System (dps)) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 May 2007 07:35:37 -0000 On Thu, 10 May 2007 22:03:22 -0400 Mike Meyer wrote: > In <20070511015156.GA77895@xor.obsecurity.org>, Kris Kennaway typed: > > On Thu, May 10, 2007 at 09:47:49PM -0400, Mike Meyer wrote: > > > Personally, I'd still like LOCALBASE to move out of /usr/local. Maybe > > > it's time to reconsider that. > > Not gonna happen as a default, but you can change it on your systems > > if you like. > > Not very reliably. > > X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 1502316A404 for ; Fri, 11 May 2007 07:58:59 +0000 (UTC) (envelope-from ed@hoeg.nl) Received: from palm.hoeg.nl (mx0.hoeg.nl [83.98.131.211]) by mx1.freebsd.org (Postfix) with ESMTP id D60DB13C458 for ; Fri, 11 May 2007 07:58:58 +0000 (UTC) (envelope-from ed@hoeg.nl) Received: by palm.hoeg.nl (Postfix, from userid 1000) id 3C8401CCC5; Fri, 11 May 2007 09:58:57 +0200 (CEST) Date: Fri, 11 May 2007 09:58:57 +0200 From: Ed Schouten To: Jan Knepper Message-ID: <20070511075857.GL23313@hoeg.nl> References: <45F1C355.8030504@digitaldaemon.com> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="Z1OTrj3C7qypP14j" Content-Disposition: inline In-Reply-To: <45F1C355.8030504@digitaldaemon.com> User-Agent: Mutt/1.5.15 (2007-04-06) Cc: FreeBSD Hackers Subject: Re: Multiple IP Jail's patch for FreeBSD 6.2 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 May 2007 07:58:59 -0000 --Z1OTrj3C7qypP14j Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Hello, It may be interesting to mention that yesterday there was a presentation at the NLUUG (Netherlands UNIX Users Group) conference by Marco Zec, who once wrote a patchset for FreeBSD 4.11 (and is in the process of porting it to FreeBSD 7.x) that gives each jail its own networking stack. You can hook up physical interfaces to jails or perform bridging between jails through netgraph bridging code. That way you can create virtual network topologies on a single box. This will allow you to use multiple IPv4 and IPv6 addresses on each instance. You can even use (I)PF(W) inside jails. More information about this can be found at: http://www.tel.fer.hr/zec/vimage/ http://imunes.net/ Yours, --=20 Ed Schouten WWW: http://g-rave.nl/ --Z1OTrj3C7qypP14j Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (FreeBSD) iD8DBQFGRCJB52SDGA2eCwURAuanAJ9K78wd0oVSjw491WsTsABi5r3uRQCfWSt+ t6Qlu4Ci1CWk6FSzogTqxls= =Bj/i -----END PGP SIGNATURE----- --Z1OTrj3C7qypP14j-- From owner-freebsd-hackers@FreeBSD.ORG Fri May 11 08:18:25 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id E914516A405 for ; Fri, 11 May 2007 08:18:25 +0000 (UTC) (envelope-from des@des.no) Received: from tim.des.no (tim.des.no [194.63.250.121]) by mx1.freebsd.org (Postfix) with ESMTP id AD20A13C45D for ; Fri, 11 May 2007 08:18:23 +0000 (UTC) (envelope-from des@des.no) Received: from tim.des.no (localhost [127.0.0.1]) by spam.des.no (Postfix) with ESMTP id 180EE20A7; Fri, 11 May 2007 10:18:20 +0200 (CEST) X-Spam-Tests: AWL X-Spam-Learn: disabled X-Spam-Score: 0.0/3.0 X-Spam-Checker-Version: SpamAssassin 3.1.7 (2006-10-05) on tim.des.no Received: from dwp.des.no (des.no [80.203.243.180]) by smtp.des.no (Postfix) with ESMTP id 0BDCB20A6; Fri, 11 May 2007 10:18:20 +0200 (CEST) Received: by dwp.des.no (Postfix, from userid 1001) id DF9A55324; Fri, 11 May 2007 10:18:19 +0200 (CEST) From: des@des.no (Dag-Erling =?utf-8?Q?Sm=C3=B8rgrav?=) To: Ivan Voras References: <200705102105.27271.blackdragon@highveldmail.co.za> Date: Fri, 11 May 2007 10:18:19 +0200 In-Reply-To: (Ivan Voras's message of "Fri\, 11 May 2007 02\:10\:05 +0200") Message-ID: <864pmj4vuc.fsf@dwp.des.no> User-Agent: Gnus/5.110006 (No Gnus v0.6) Emacs/21.3 (berkeley-unix) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Cc: freebsd-hackers@freebsd.org Subject: Re: New FreeBSD package system (a.k.a. Daemon Package System (dps)) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 May 2007 08:18:26 -0000 Ivan Voras writes: > - A quick test confirms that the current bsdtar will happily ignore any > extra data at the end of a tgz/tbz archive, so package metadata can be > embedded there, thus conserving existing infrastructure and being fast > to parse. I suggest encoding this metadata in a sane and easy to parse > XML structure. This has been discussed a million times before. The metadata needs to be at the *start* of the package file, otherwise networked installs (such as pkg_add -r) of large packages will require huge amounts of memory and / or temporary disk storage. DES --=20 Dag-Erling Sm=C3=B8rgrav - des@des.no From owner-freebsd-hackers@FreeBSD.ORG Fri May 11 08:19:50 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id EBEA616A403 for ; Fri, 11 May 2007 08:19:50 +0000 (UTC) (envelope-from des@des.no) Received: from tim.des.no (tim.des.no [194.63.250.121]) by mx1.freebsd.org (Postfix) with ESMTP id 9799013C480 for ; Fri, 11 May 2007 08:19:50 +0000 (UTC) (envelope-from des@des.no) Received: from tim.des.no (localhost [127.0.0.1]) by spam.des.no (Postfix) with ESMTP id 104D920A7; Fri, 11 May 2007 10:19:47 +0200 (CEST) X-Spam-Tests: AWL X-Spam-Learn: disabled X-Spam-Score: 0.0/3.0 X-Spam-Checker-Version: SpamAssassin 3.1.7 (2006-10-05) on tim.des.no Received: from dwp.des.no (des.no [80.203.243.180]) by smtp.des.no (Postfix) with ESMTP id 030B720A6; Fri, 11 May 2007 10:19:47 +0200 (CEST) Received: by dwp.des.no (Postfix, from userid 1001) id E20D95326; Fri, 11 May 2007 10:19:46 +0200 (CEST) From: des@des.no (Dag-Erling =?utf-8?Q?Sm=C3=B8rgrav?=) To: Kris Kennaway References: <200705102105.27271.blackdragon@highveldmail.co.za> <17987.52037.112351.872442@bhuda.mired.org> <20070511015156.GA77895@xor.obsecurity.org> Date: Fri, 11 May 2007 10:19:46 +0200 In-Reply-To: <20070511015156.GA77895@xor.obsecurity.org> (Kris Kennaway's message of "Thu\, 10 May 2007 21\:51\:56 -0400") Message-ID: <86zm4b3h7h.fsf@dwp.des.no> User-Agent: Gnus/5.110006 (No Gnus v0.6) Emacs/21.3 (berkeley-unix) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Cc: freebsd-hackers@freebsd.org, Mike Meyer , Ivan Voras Subject: Re: New FreeBSD package system (a.k.a. Daemon Package System (dps)) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 May 2007 08:19:51 -0000 Kris Kennaway writes: > On Thu, May 10, 2007 at 09:47:49PM -0400, Mike Meyer wrote: > > Personally, I'd still like LOCALBASE to move out of /usr/local. Maybe > > it's time to reconsider that. > Not gonna happen as a default, but you can change it on your systems > if you like. Not if you want to use pre-built packages. You made sure of that when you decided (against my objections) to include .la files in packages. DES --=20 Dag-Erling Sm=C3=B8rgrav - des@des.no From owner-freebsd-hackers@FreeBSD.ORG Fri May 11 08:26:59 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 3299516A400 for ; Fri, 11 May 2007 08:26:59 +0000 (UTC) (envelope-from kris@obsecurity.org) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.freebsd.org (Postfix) with ESMTP id 1EAAD13C46A for ; Fri, 11 May 2007 08:26:59 +0000 (UTC) (envelope-from kris@obsecurity.org) Received: from obsecurity.dyndns.org (elvis.mu.org [192.203.228.196]) by elvis.mu.org (Postfix) with ESMTP id 0779B1A3C19; Fri, 11 May 2007 01:27:44 -0700 (PDT) Received: by obsecurity.dyndns.org (Postfix, from userid 1000) id E526353029; Fri, 11 May 2007 04:26:57 -0400 (EDT) Date: Fri, 11 May 2007 04:26:57 -0400 From: Kris Kennaway To: Dag-Erling Sm??rgrav Message-ID: <20070511082657.GA24652@xor.obsecurity.org> References: <200705102105.27271.blackdragon@highveldmail.co.za> <17987.52037.112351.872442@bhuda.mired.org> <20070511015156.GA77895@xor.obsecurity.org> <86zm4b3h7h.fsf@dwp.des.no> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="jI8keyz6grp/JLjh" Content-Disposition: inline In-Reply-To: <86zm4b3h7h.fsf@dwp.des.no> User-Agent: Mutt/1.4.2.2i Cc: freebsd-hackers@freebsd.org, Mike Meyer , Ivan Voras , Kris Kennaway Subject: Re: New FreeBSD package system (a.k.a. Daemon Package System (dps)) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 May 2007 08:26:59 -0000 --jI8keyz6grp/JLjh Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Fri, May 11, 2007 at 10:19:46AM +0200, Dag-Erling Sm??rgrav wrote: > Kris Kennaway writes: > > On Thu, May 10, 2007 at 09:47:49PM -0400, Mike Meyer wrote: > > > Personally, I'd still like LOCALBASE to move out of /usr/local. Maybe > > > it's time to reconsider that. > > Not gonna happen as a default, but you can change it on your systems > > if you like. >=20 > Not if you want to use pre-built packages. You made sure of that when > you decided (against my objections) to include .la files in packages. I have a suspicion you're never going to let that go, but it's not relevant here anyway. Binaries have been hardcoding their build location (e.g. /usr/local) since the dawn of time. The best you can do is to binary edit everything to a string of the same length, and that works for .la files too. Kris --jI8keyz6grp/JLjh Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (FreeBSD) iD8DBQFGRCjQWry0BWjoQKURAoqYAJ0V5AqqLjxhXI0JF83lVxjk22SZgACeNoO1 +WXu1wjpMPDkWDckorWjwog= =lQ9d -----END PGP SIGNATURE----- --jI8keyz6grp/JLjh-- From owner-freebsd-hackers@FreeBSD.ORG Fri May 11 08:28:30 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id EDFB916A406 for ; Fri, 11 May 2007 08:28:30 +0000 (UTC) (envelope-from freebsd-hackers@m.gmane.org) Received: from ciao.gmane.org (main.gmane.org [80.91.229.2]) by mx1.freebsd.org (Postfix) with ESMTP id 7BCE513C458 for ; Fri, 11 May 2007 08:28:30 +0000 (UTC) (envelope-from freebsd-hackers@m.gmane.org) Received: from list by ciao.gmane.org with local (Exim 4.43) id 1HmQUG-0002kh-5P for freebsd-hackers@freebsd.org; Fri, 11 May 2007 10:28:28 +0200 Received: from 78-1-101-222.adsl.net.t-com.hr ([78.1.101.222]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Fri, 11 May 2007 10:28:28 +0200 Received: from ivoras by 78-1-101-222.adsl.net.t-com.hr with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Fri, 11 May 2007 10:28:28 +0200 X-Injected-Via-Gmane: http://gmane.org/ To: freebsd-hackers@freebsd.org From: Ivan Voras Date: Fri, 11 May 2007 10:28:16 +0200 Lines: 77 Message-ID: References: <200705102105.27271.blackdragon@highveldmail.co.za> <4643C7DB.6000408@elischer.org> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="------------enigCFB197788CD94A19B3832926" X-Complaints-To: usenet@sea.gmane.org X-Gmane-NNTP-Posting-Host: 78-1-101-222.adsl.net.t-com.hr User-Agent: Thunderbird 1.5.0.10 (Windows/20070221) In-Reply-To: <4643C7DB.6000408@elischer.org> X-Enigmail-Version: 0.94.3.0 Sender: news Subject: Re: New FreeBSD package system (a.k.a. Daemon Package System (dps)) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 May 2007 08:28:31 -0000 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enigCFB197788CD94A19B3832926 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Julian Elischer wrote: > ok, let me give you some words that come from the wisdom of having done= > this for 30 years. >=20 > "Use an SQL DB file for this over my dead body".. :) I certainly won't be going on a campaign for it, but... > Now having said that, I need to modify it a bit and explain. >=20 > Explanation: > The number of times I've gone into /var/db/pkg to read the files to > figure out what the "fsck" is going on, is very large. > It is also possible to delete files and edit them. > This is very important to me in this imperfect world when the pkg syste= m You can inspect s sqlite database with the provided utility. Unless the database gets corrupted (which it tries to avoid by respecting ACID), I think you can always get more sensible data (and quicker) out of a SQL db then out of a text file. I've seen both the pkg tree (dir of text files) and pkgdb (BDB database) corrupted before, during NORMAL operation (i.e. no power outages & similar disruptive events), so I still vote for a transaction-safe way of doing this. I see it as a choice between annoying many (and I mean a lot of!) users with slow operation and clumsiness vs a little convenience during disaster recovery (e.g. learning basics of SQL). > gets things wrong (oh no, don't tell me that the new pkg system never > gets things wrong.. like including a file in 2 packages). This could be caught by a UNIQUE constraint on a table. > Modification: > Now, I have no objection to a DB file of some sort IF (and only if) > it is somehow rebuildable from a text base which is kept somewhere, That's precisely why the current system is so clumsy - there are at least two places this information is kept, and the "text base" is the limiting factor: slow and not transaction-safe. I don't think it can remain as the primary data store in a modern system. > like the password database is done. but PLEASE, NO SQL in the base syst= em! Maybe the other way around would satisfy you - the ability to rebuild the text db from binary db? Perhaps this is a good time I should mention that I think sqlite would also be good for the password and login databases? :) --------------enigCFB197788CD94A19B3832926 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.4 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFGRCklldnAQVacBcgRAu7wAKDmqar0/IgTWOZiJgeIQBadSnJUyACgugn0 KQg5buS7B9Sm6EpJdayAmdQ= =/r2O -----END PGP SIGNATURE----- --------------enigCFB197788CD94A19B3832926-- From owner-freebsd-hackers@FreeBSD.ORG Fri May 11 08:33:33 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id DC97516A402 for ; Fri, 11 May 2007 08:33:33 +0000 (UTC) (envelope-from des@des.no) Received: from tim.des.no (tim.des.no [194.63.250.121]) by mx1.freebsd.org (Postfix) with ESMTP id 9DBB613C45E for ; Fri, 11 May 2007 08:33:33 +0000 (UTC) (envelope-from des@des.no) Received: from tim.des.no (localhost [127.0.0.1]) by spam.des.no (Postfix) with ESMTP id 0870B20AA; Fri, 11 May 2007 10:33:30 +0200 (CEST) X-Spam-Tests: AWL X-Spam-Learn: disabled X-Spam-Score: 0.0/3.0 X-Spam-Checker-Version: SpamAssassin 3.1.7 (2006-10-05) on tim.des.no Received: from dwp.des.no (des.no [80.203.243.180]) by smtp.des.no (Postfix) with ESMTP id E14CD20A6; Fri, 11 May 2007 10:33:29 +0200 (CEST) Received: by dwp.des.no (Postfix, from userid 1001) id B849A532B; Fri, 11 May 2007 10:33:29 +0200 (CEST) From: des@des.no (Dag-Erling =?utf-8?Q?Sm=C3=B8rgrav?=) To: Kris Kennaway References: <200705102105.27271.blackdragon@highveldmail.co.za> <17987.52037.112351.872442@bhuda.mired.org> <20070511015156.GA77895@xor.obsecurity.org> <86zm4b3h7h.fsf@dwp.des.no> <20070511082657.GA24652@xor.obsecurity.org> Date: Fri, 11 May 2007 10:33:29 +0200 In-Reply-To: <20070511082657.GA24652@xor.obsecurity.org> (Kris Kennaway's message of "Fri\, 11 May 2007 04\:26\:57 -0400") Message-ID: <86sla33gkm.fsf@dwp.des.no> User-Agent: Gnus/5.110006 (No Gnus v0.6) Emacs/21.3 (berkeley-unix) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Cc: freebsd-hackers@freebsd.org, Mike Meyer , Ivan Voras Subject: Re: New FreeBSD package system (a.k.a. Daemon Package System (dps)) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 May 2007 08:33:33 -0000 Kris Kennaway writes: > On Fri, May 11, 2007 at 10:19:46AM +0200, Dag-Erling Sm??rgrav wrote: > > Not if you want to use pre-built packages. You made sure of that when > > you decided (against my objections) to include .la files in packages. > I have a suspicion you're never going to let that go, but it's not > relevant here anyway. Binaries have been hardcoding their build > location (e.g. /usr/local) since the dawn of time. Most don't. > The best you can > do is to binary edit everything to a string of the same length, and > that works for .la files too. The existence of .la files is a bug. We already have a mechanism for recording dependencies between libraries; it's built into the ELF format, and does not require hardcoding any directories. Introducing .la files which override the existing mechanism and *do* hardcode directories is a regression. I don't buy the argument that "KDE won't build without them", or whatever it was you used to justify this. There is nothing an .la file does which can't be done more properly by adding the correct directory to your ldconfig path. DES --=20 Dag-Erling Sm=C3=B8rgrav - des@des.no From owner-freebsd-hackers@FreeBSD.ORG Fri May 11 08:39:14 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 516CC16A402 for ; Fri, 11 May 2007 08:39:14 +0000 (UTC) (envelope-from joerg@britannica.bec.de) Received: from www.pkgsrc-box.org (www.ostsee-abc.de [62.206.222.50]) by mx1.freebsd.org (Postfix) with ESMTP id 503A213C455 for ; Fri, 11 May 2007 08:39:13 +0000 (UTC) (envelope-from joerg@britannica.bec.de) Received: from britannica.bec.de (www.pkgsrc-box.org [127.0.0.1]) by www.pkgsrc-box.org (Postfix) with ESMTP id 59930E7A3F9 for ; Fri, 11 May 2007 08:39:07 +0000 (UTC) Received: by britannica.bec.de (Postfix, from userid 1000) id DF5CA7E95; Fri, 11 May 2007 10:38:56 +0200 (CEST) Date: Fri, 11 May 2007 10:38:56 +0200 From: Joerg Sonnenberger To: freebsd-hackers@freebsd.org Message-ID: <20070511083856.GD228@britannica.bec.de> Mail-Followup-To: freebsd-hackers@freebsd.org References: <200705102105.27271.blackdragon@highveldmail.co.za> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.13 (2006-08-11) Subject: Re: New FreeBSD package system (a.k.a. Daemon Package System (dps)) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 May 2007 08:39:14 -0000 On Fri, May 11, 2007 at 02:10:05AM +0200, Ivan Voras wrote: > - I think it's time to give up on using BDB+directory tree full of text > files for storing the installed packages database, and I propose all of > this be replaced by a single SQLite database. Take a look at the pkgjam presentation during the last pkgsrcCon: http://www.pkgsrcCon.org/2007/presentations.html It has merits and disadvantages. Joerg From owner-freebsd-hackers@FreeBSD.ORG Fri May 11 08:40:28 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 6869F16A406 for ; Fri, 11 May 2007 08:40:28 +0000 (UTC) (envelope-from kris@obsecurity.org) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.freebsd.org (Postfix) with ESMTP id 46FE513C484 for ; Fri, 11 May 2007 08:40:28 +0000 (UTC) (envelope-from kris@obsecurity.org) Received: from obsecurity.dyndns.org (elvis.mu.org [192.203.228.196]) by elvis.mu.org (Postfix) with ESMTP id 2A16A1A3C19; Fri, 11 May 2007 01:41:13 -0700 (PDT) Received: by obsecurity.dyndns.org (Postfix, from userid 1000) id 39747528E4; Fri, 11 May 2007 04:40:27 -0400 (EDT) Date: Fri, 11 May 2007 04:40:27 -0400 From: Kris Kennaway To: Dag-Erling Sm??rgrav Message-ID: <20070511084026.GA59508@xor.obsecurity.org> References: <200705102105.27271.blackdragon@highveldmail.co.za> <17987.52037.112351.872442@bhuda.mired.org> <20070511015156.GA77895@xor.obsecurity.org> <86zm4b3h7h.fsf@dwp.des.no> <20070511082657.GA24652@xor.obsecurity.org> <86sla33gkm.fsf@dwp.des.no> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <86sla33gkm.fsf@dwp.des.no> User-Agent: Mutt/1.4.2.2i Cc: freebsd-hackers@freebsd.org, Mike Meyer , Ivan Voras , Kris Kennaway Subject: Re: New FreeBSD package system (a.k.a. Daemon Package System (dps)) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 May 2007 08:40:28 -0000 On Fri, May 11, 2007 at 10:33:29AM +0200, Dag-Erling Sm??rgrav wrote: > Kris Kennaway writes: > > On Fri, May 11, 2007 at 10:19:46AM +0200, Dag-Erling Sm??rgrav wrote: > > > Not if you want to use pre-built packages. You made sure of that when > > > you decided (against my objections) to include .la files in packages. > > I have a suspicion you're never going to let that go, but it's not > > relevant here anyway. Binaries have been hardcoding their build > > location (e.g. /usr/local) since the dawn of time. > > Most don't. Assertion without proof. In fact a quick survey shows that 90% of my /usr/local/bin references /usr/local. > > The best you can > > do is to binary edit everything to a string of the same length, and > > that works for .la files too. > > The existence of .la files is a bug. > > We already have a mechanism for recording dependencies between > libraries; it's built into the ELF format, and does not require > hardcoding any directories. Introducing .la files which override the > existing mechanism and *do* hardcode directories is a regression. > > I don't buy the argument that "KDE won't build without them", or > whatever it was you used to justify this. I can't help it that you weren't paying attention. Kris From owner-freebsd-hackers@FreeBSD.ORG Fri May 11 08:52:44 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 013EF16A402 for ; Fri, 11 May 2007 08:52:44 +0000 (UTC) (envelope-from joerg@britannica.bec.de) Received: from www.pkgsrc-box.org (www.ostsee-abc.de [62.206.222.50]) by mx1.freebsd.org (Postfix) with ESMTP id 0FB9D13C44C for ; Fri, 11 May 2007 08:52:43 +0000 (UTC) (envelope-from joerg@britannica.bec.de) Received: from britannica.bec.de (www.pkgsrc-box.org [127.0.0.1]) by www.pkgsrc-box.org (Postfix) with ESMTP id AD531E7A3F9 for ; Fri, 11 May 2007 08:52:37 +0000 (UTC) Received: by britannica.bec.de (Postfix, from userid 1000) id 064217E95; Fri, 11 May 2007 10:52:29 +0200 (CEST) Date: Fri, 11 May 2007 10:52:28 +0200 From: Joerg Sonnenberger To: freebsd-hackers@freebsd.org Message-ID: <20070511085228.GB2318@britannica.bec.de> Mail-Followup-To: freebsd-hackers@freebsd.org References: <200705102105.27271.blackdragon@highveldmail.co.za> <17987.52037.112351.872442@bhuda.mired.org> <20070511015156.GA77895@xor.obsecurity.org> <86zm4b3h7h.fsf@dwp.des.no> <20070511082657.GA24652@xor.obsecurity.org> <86sla33gkm.fsf@dwp.des.no> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <86sla33gkm.fsf@dwp.des.no> User-Agent: Mutt/1.5.13 (2006-08-11) Subject: Re: New FreeBSD package system (a.k.a. Daemon Package System (dps)) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 May 2007 08:52:44 -0000 On Fri, May 11, 2007 at 10:33:29AM +0200, Dag-Erling Smørgrav wrote: > We already have a mechanism for recording dependencies between > libraries; it's built into the ELF format, and does not require > hardcoding any directories. Introducing .la files which override the > existing mechanism and *do* hardcode directories is a regression. That is because FreeBSD doesn't use the ELF mechanisms to specify search directories in most cases. It still depends on the a.out orignating ld.so.conf mechanism. Joerg From owner-freebsd-hackers@FreeBSD.ORG Fri May 11 09:01:27 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 226BC16A405 for ; Fri, 11 May 2007 09:01:27 +0000 (UTC) (envelope-from peterjeremy@optushome.com.au) Received: from turion.vk2pj.dyndns.org (c220-239-3-125.belrs4.nsw.optusnet.com.au [220.239.3.125]) by mx1.freebsd.org (Postfix) with ESMTP id 85A2513C45D for ; Fri, 11 May 2007 09:01:26 +0000 (UTC) (envelope-from peterjeremy@optushome.com.au) Received: from turion.vk2pj.dyndns.org (localhost.vk2pj.dyndns.org [127.0.0.1]) by turion.vk2pj.dyndns.org (8.14.1/8.14.1) with ESMTP id l4B91IUs001513; Fri, 11 May 2007 19:01:18 +1000 (EST) (envelope-from peter@turion.vk2pj.dyndns.org) Received: (from peter@localhost) by turion.vk2pj.dyndns.org (8.14.1/8.14.1/Submit) id l4B91IGr001512; Fri, 11 May 2007 19:01:18 +1000 (EST) (envelope-from peter) Date: Fri, 11 May 2007 19:01:18 +1000 From: Peter Jeremy To: Ivan Voras Message-ID: <20070511090118.GE826@turion.vk2pj.dyndns.org> References: <200705102105.27271.blackdragon@highveldmail.co.za> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="GID0FwUMdk1T2AWN" Content-Disposition: inline In-Reply-To: X-PGP-Key: http://members.optusnet.com.au/peterjeremy/pubkey.asc User-Agent: Mutt/1.5.14 (2007-02-12) Cc: freebsd-hackers@freebsd.org Subject: Re: New FreeBSD package system (a.k.a. Daemon Package System (dps)) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 May 2007 09:01:27 -0000 --GID0FwUMdk1T2AWN Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On 2007-May-11 02:10:05 +0200, Ivan Voras wrote: >- I think it's time to give up on using BDB+directory tree full of text >files for storing the installed packages database, Why? > and I propose all of >this be replaced by a single SQLite database. I'll agree with Julian on this one. When it comes to maintenance, you can't beat a collection of documented text files. As a good example of why non-text databases for system configuration information aren't a good idea, I suggest you google "windows registry" :-) > SQLite is public domain >(can be slurped into base system), We don't want to unnecessarily increase the size of the base system. SQLite is also changing at a fairly rapid rate - which is incompatible with the FreeBSD release cycle. There have been 6 releases of SQLite so far this year. This would lead to a situation where even if we imported SQLite, we would still need an SQLite port for people who needed a more up-to-date version. >as reporting. The current pkg_info's behaviour that takes *noticable* >*time* to generate 1 line of output per package is horrible in this day >and age. After warming up the cache, I get one line every 1.5msec. Before that, I get one line every 15-20msec which isn't that bad. If that is too slow, then build a database as a cache - as long as the text files remain the master version. In which case, I'd prefer to use dbm(3) since it's already in the base system. >- A quick test confirms that the current bsdtar will happily ignore any >extra data at the end of a tgz/tbz archive, so package metadata can be >embedded there, thus conserving existing infrastructure and being fast >to parse. I suggest encoding this metadata in a sane and easy to parse >XML structure. Relying on undocumented features of tools is rarely a good idea. tar has other disadvantages (particularly the lack of random access) as a ports archive format. ZIP was suggested as an alternative. I also question the combination of "sane", "easy to parse" and "XML". --=20 Peter Jeremy --GID0FwUMdk1T2AWN Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (FreeBSD) iD8DBQFGRDDe/opHv/APuIcRAtD5AJ4h2eWIfPzE+k4IxHiH7mI+tfQGnQCfTUFB Pidw0pwzUiYDVrpn8xRWyJ0= =n0Ge -----END PGP SIGNATURE----- --GID0FwUMdk1T2AWN-- From owner-freebsd-hackers@FreeBSD.ORG Fri May 11 09:02:41 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id C60A516A406 for ; Fri, 11 May 2007 09:02:41 +0000 (UTC) (envelope-from des@des.no) Received: from tim.des.no (tim.des.no [194.63.250.121]) by mx1.freebsd.org (Postfix) with ESMTP id 8585A13C455 for ; Fri, 11 May 2007 09:02:41 +0000 (UTC) (envelope-from des@des.no) Received: from tim.des.no (localhost [127.0.0.1]) by spam.des.no (Postfix) with ESMTP id A731920AD; Fri, 11 May 2007 11:02:37 +0200 (CEST) X-Spam-Tests: AWL X-Spam-Learn: disabled X-Spam-Score: 0.0/3.0 X-Spam-Checker-Version: SpamAssassin 3.1.7 (2006-10-05) on tim.des.no Received: from dwp.des.no (des.no [80.203.243.180]) by smtp.des.no (Postfix) with ESMTP id 96D0A20A8; Fri, 11 May 2007 11:02:37 +0200 (CEST) Received: by dwp.des.no (Postfix, from userid 1001) id 7A8C85330; Fri, 11 May 2007 11:02:37 +0200 (CEST) From: des@des.no (Dag-Erling =?utf-8?Q?Sm=C3=B8rgrav?=) To: Kris Kennaway References: <200705102105.27271.blackdragon@highveldmail.co.za> <17987.52037.112351.872442@bhuda.mired.org> <20070511015156.GA77895@xor.obsecurity.org> <86zm4b3h7h.fsf@dwp.des.no> <20070511082657.GA24652@xor.obsecurity.org> <86sla33gkm.fsf@dwp.des.no> <20070511084026.GA59508@xor.obsecurity.org> Date: Fri, 11 May 2007 11:02:37 +0200 In-Reply-To: <20070511084026.GA59508@xor.obsecurity.org> (Kris Kennaway's message of "Fri\, 11 May 2007 04\:40\:27 -0400") Message-ID: <86k5vf3f82.fsf@dwp.des.no> User-Agent: Gnus/5.110006 (No Gnus v0.6) Emacs/21.3 (berkeley-unix) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Cc: freebsd-hackers@freebsd.org, Mike Meyer , Ivan Voras Subject: Re: New FreeBSD package system (a.k.a. Daemon Package System (dps)) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 May 2007 09:02:41 -0000 Kris Kennaway writes: > On Fri, May 11, 2007 at 10:33:29AM +0200, Dag-Erling Sm??rgrav wrote: > > The existence of .la files is a bug. > > > > We already have a mechanism for recording dependencies between > > libraries; it's built into the ELF format, and does not require > > hardcoding any directories. Introducing .la files which override the > > existing mechanism and *do* hardcode directories is a regression. > > > > I don't buy the argument that "KDE won't build without them", or > > whatever it was you used to justify this. > I can't help it that you weren't paying attention. My point is that the argument is bogus. I am positively certain that the root issue was not the absence of .la files; installing them merely served as a workaround for the actual problem, which was most likely related to ldconfig and / or LD_LIBRARY_PATH. This is like the (apocryphal) story of the car that wouldn't start on the way back from the store when the owner went to buy ice cream - but only if he bought vanilla. The correct fix is not to buy a different flavor. Once you realize that the vanilla ice cream is right next to the check-out register while the others are deeper within the store and therefore take considerably longer to get, you start looking for vapor lock. DES --=20 Dag-Erling Sm=C3=B8rgrav - des@des.no From owner-freebsd-hackers@FreeBSD.ORG Fri May 11 09:03:49 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id B266C16A408 for ; Fri, 11 May 2007 09:03:49 +0000 (UTC) (envelope-from des@des.no) Received: from tim.des.no (tim.des.no [194.63.250.121]) by mx1.freebsd.org (Postfix) with ESMTP id 7248C13C489 for ; Fri, 11 May 2007 09:03:49 +0000 (UTC) (envelope-from des@des.no) Received: from tim.des.no (localhost [127.0.0.1]) by spam.des.no (Postfix) with ESMTP id D3A4720B0; Fri, 11 May 2007 11:03:45 +0200 (CEST) X-Spam-Tests: AWL X-Spam-Learn: disabled X-Spam-Score: 0.0/3.0 X-Spam-Checker-Version: SpamAssassin 3.1.7 (2006-10-05) on tim.des.no Received: from dwp.des.no (des.no [80.203.243.180]) by smtp.des.no (Postfix) with ESMTP id C488D20AF; Fri, 11 May 2007 11:03:45 +0200 (CEST) Received: by dwp.des.no (Postfix, from userid 1001) id A95875332; Fri, 11 May 2007 11:03:45 +0200 (CEST) From: des@des.no (Dag-Erling =?utf-8?Q?Sm=C3=B8rgrav?=) To: Kris Kennaway References: <200705102105.27271.blackdragon@highveldmail.co.za> <17987.52037.112351.872442@bhuda.mired.org> <20070511015156.GA77895@xor.obsecurity.org> <86zm4b3h7h.fsf@dwp.des.no> <20070511082657.GA24652@xor.obsecurity.org> <86sla33gkm.fsf@dwp.des.no> <20070511084026.GA59508@xor.obsecurity.org> Date: Fri, 11 May 2007 11:03:45 +0200 In-Reply-To: <20070511084026.GA59508@xor.obsecurity.org> (Kris Kennaway's message of "Fri\, 11 May 2007 04\:40\:27 -0400") Message-ID: <86fy633f66.fsf@dwp.des.no> User-Agent: Gnus/5.110006 (No Gnus v0.6) Emacs/21.3 (berkeley-unix) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Cc: freebsd-hackers@freebsd.org, Mike Meyer , Ivan Voras Subject: Re: New FreeBSD package system (a.k.a. Daemon Package System (dps)) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 May 2007 09:03:49 -0000 Kris Kennaway writes: > On Fri, May 11, 2007 at 10:33:29AM +0200, Dag-Erling Sm??rgrav wrote: > > Kris Kennaway writes: > > > On Fri, May 11, 2007 at 10:19:46AM +0200, Dag-Erling Sm??rgrav wrote: > > > > Not if you want to use pre-built packages. You made sure of that w= hen > > > > you decided (against my objections) to include .la files in package= s. > > > I have a suspicion you're never going to let that go, but it's not > > > relevant here anyway. Binaries have been hardcoding their build > > > location (e.g. /usr/local) since the dawn of time. > > Most don't. > Assertion without proof. In fact a quick survey shows that 90% of my > /usr/local/bin references /usr/local. I forgot to answer this bit: of course they do, if they were linked in the presence of .la files. Please understand that they are the problem, not the solution. DES --=20 Dag-Erling Sm=C3=B8rgrav - des@des.no From owner-freebsd-hackers@FreeBSD.ORG Fri May 11 09:20:21 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 73FAE16A400 for ; Fri, 11 May 2007 09:20:21 +0000 (UTC) (envelope-from freebsd-hackers@m.gmane.org) Received: from ciao.gmane.org (main.gmane.org [80.91.229.2]) by mx1.freebsd.org (Postfix) with ESMTP id 01F6113C448 for ; Fri, 11 May 2007 09:20:20 +0000 (UTC) (envelope-from freebsd-hackers@m.gmane.org) Received: from list by ciao.gmane.org with local (Exim 4.43) id 1HmRHx-0001GF-0E for freebsd-hackers@freebsd.org; Fri, 11 May 2007 11:19:49 +0200 Received: from lara.cc.fer.hr ([161.53.72.113]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Fri, 11 May 2007 11:19:48 +0200 Received: from ivoras by lara.cc.fer.hr with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Fri, 11 May 2007 11:19:48 +0200 X-Injected-Via-Gmane: http://gmane.org/ To: freebsd-hackers@freebsd.org From: Ivan Voras Date: Fri, 11 May 2007 11:19:13 +0200 Lines: 94 Message-ID: References: <200705102105.27271.blackdragon@highveldmail.co.za> <20070511090118.GE826@turion.vk2pj.dyndns.org> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="------------enigA1C266F0560A81607C164753" X-Complaints-To: usenet@sea.gmane.org X-Gmane-NNTP-Posting-Host: lara.cc.fer.hr User-Agent: Thunderbird 1.5.0.10 (X11/20060911) In-Reply-To: <20070511090118.GE826@turion.vk2pj.dyndns.org> X-Enigmail-Version: 0.94.2.0 Sender: news Subject: Re: New FreeBSD package system (a.k.a. Daemon Package System (dps)) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 May 2007 09:20:21 -0000 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enigA1C266F0560A81607C164753 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: quoted-printable Peter Jeremy wrote: > On 2007-May-11 02:10:05 +0200, Ivan Voras wrote: >> - I think it's time to give up on using BDB+directory tree full of tex= t >> files for storing the installed packages database, >=20 > Why? - no strict format - slow - not transaction safe - harder to use then SQL (yes, relatively, but many younger people know=20 SQL better than grep). >> and I propose all of >> this be replaced by a single SQLite database. >=20 > I'll agree with Julian on this one. When it comes to maintenance, you > can't beat a collection of documented text files. As a good example > of why non-text databases for system configuration information aren't > a good idea, I suggest you google "windows registry" :-) They fixed in Win2000 and newer version (mostly by using a sane file=20 system instead of FAT32). > We don't want to unnecessarily increase the size of the base system. > SQLite is also changing at a fairly rapid rate - which is incompatible > with the FreeBSD release cycle. There have been 6 releases of SQLite > so far this year. This would lead to a situation where even if we > imported SQLite, we would still need an SQLite port for people who > needed a more up-to-date version. Since fancy new features of sqlite are not needed here, I'd suggest=20 importing the latest 2.x release, which hasn't changed for quite some tim= e. >> as reporting. The current pkg_info's behaviour that takes *noticable* >> *time* to generate 1 line of output per package is horrible in this da= y >> and age. >=20 > After warming up the cache, I get one line every 1.5msec. Before that,= > I get one line every 15-20msec which isn't that bad. I don't think the "common usage pattern here" includes warming up the=20 pkg cache :) If you don't think 15 ms per line is bad performance, then=20 we'll just agree do disagree :) > Relying on undocumented features of tools is rarely a good idea. tar BSDtar is maintained by "our people". > has other disadvantages (particularly the lack of random access) as a > ports archive format. ZIP was suggested as an alternative. I also > question the combination of "sane", "easy to parse" and "XML". Alternatives to XML are: - binary format (yuck) - another plain text format (hacks such as concatenating existing=20 metadata "files" in the .tar - also yuck). des@ mentioned putting metadata info at the front of the file - I don't=20 see how this would help. The most common operation with binary packages=20 *over the network* is "pkg_add -r", which will need to read it whole=20 anyway, and it would help greatly for things such as installers from CD=20 media. (Querying a bunch of packages over the network for their=20 properties, one by one, is not a good idea, but it is on a local media). Anyway, I'm not going to do it, so I'll try and stop bikeshedding :) --------------enigA1C266F0560A81607C164753 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.5 (GNU/Linux) Comment: Using GnuPG with SUSE - http://enigmail.mozdev.org iD8DBQFGRDUXldnAQVacBcgRAq0lAJ49qzgZi6ZrnKc7vC7bRHvF6H2H0QCgz0oZ JQrFEeRfNytWdX9glK4CEC8= =pxIF -----END PGP SIGNATURE----- --------------enigA1C266F0560A81607C164753-- From owner-freebsd-hackers@FreeBSD.ORG Fri May 11 09:35:03 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 5086416A402 for ; Fri, 11 May 2007 09:35:03 +0000 (UTC) (envelope-from des@des.no) Received: from tim.des.no (tim.des.no [194.63.250.121]) by mx1.freebsd.org (Postfix) with ESMTP id 1398113C46E for ; Fri, 11 May 2007 09:35:03 +0000 (UTC) (envelope-from des@des.no) Received: from tim.des.no (localhost [127.0.0.1]) by spam.des.no (Postfix) with ESMTP id 977B220B2; Fri, 11 May 2007 11:34:59 +0200 (CEST) X-Spam-Tests: AWL X-Spam-Learn: disabled X-Spam-Score: 0.0/3.0 X-Spam-Checker-Version: SpamAssassin 3.1.7 (2006-10-05) on tim.des.no Received: from dwp.des.no (des.no [80.203.243.180]) by smtp.des.no (Postfix) with ESMTP id 8925F20B1; Fri, 11 May 2007 11:34:59 +0200 (CEST) Received: by dwp.des.no (Postfix, from userid 1001) id 6E5F0533A; Fri, 11 May 2007 11:34:59 +0200 (CEST) From: des@des.no (Dag-Erling =?utf-8?Q?Sm=C3=B8rgrav?=) To: Ivan Voras References: <200705102105.27271.blackdragon@highveldmail.co.za> <4643C7DB.6000408@elischer.org> Date: Fri, 11 May 2007 11:34:59 +0200 In-Reply-To: (Ivan Voras's message of "Fri\, 11 May 2007 10\:28\:16 +0200") Message-ID: <863b233dq4.fsf@dwp.des.no> User-Agent: Gnus/5.110006 (No Gnus v0.6) Emacs/21.3 (berkeley-unix) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Cc: freebsd-hackers@freebsd.org Subject: Re: New FreeBSD package system (a.k.a. Daemon Package System (dps)) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 May 2007 09:35:03 -0000 Ivan Voras writes: > You can inspect s sqlite database with the provided utility. Unless the > database gets corrupted (which it tries to avoid by respecting ACID), ACID is not something a database "respects", it is a set of guarantees that it provides to the application. Avoiding database corruption is a necessary requirement for, rather than a consequence of, ACID. Perhaps you mean that SQLite tries to avoid database corruption by using locks, and either scatter-gather writes or copy-on-write, and flushing the file between transactions, to ensure consistency? DES --=20 Dag-Erling Sm=C3=B8rgrav - des@des.no From owner-freebsd-hackers@FreeBSD.ORG Fri May 11 09:41:58 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 492EE16A404 for ; Fri, 11 May 2007 09:41:58 +0000 (UTC) (envelope-from des@des.no) Received: from tim.des.no (tim.des.no [194.63.250.121]) by mx1.freebsd.org (Postfix) with ESMTP id 0D26913C45D for ; Fri, 11 May 2007 09:41:58 +0000 (UTC) (envelope-from des@des.no) Received: from tim.des.no (localhost [127.0.0.1]) by spam.des.no (Postfix) with ESMTP id 44E5320A6; Fri, 11 May 2007 11:41:54 +0200 (CEST) X-Spam-Tests: AWL X-Spam-Learn: disabled X-Spam-Score: 0.0/3.0 X-Spam-Checker-Version: SpamAssassin 3.1.7 (2006-10-05) on tim.des.no Received: from dwp.des.no (des.no [80.203.243.180]) by smtp.des.no (Postfix) with ESMTP id BAB4C2088; Fri, 11 May 2007 11:41:53 +0200 (CEST) Received: by dwp.des.no (Postfix, from userid 1001) id A1D9A533C; Fri, 11 May 2007 11:41:53 +0200 (CEST) From: des@des.no (Dag-Erling =?utf-8?Q?Sm=C3=B8rgrav?=) To: Ivan Voras References: <200705102105.27271.blackdragon@highveldmail.co.za> <20070511090118.GE826@turion.vk2pj.dyndns.org> Date: Fri, 11 May 2007 11:41:53 +0200 In-Reply-To: (Ivan Voras's message of "Fri\, 11 May 2007 11\:19\:13 +0200") Message-ID: <86y7jv1yu6.fsf@dwp.des.no> User-Agent: Gnus/5.110006 (No Gnus v0.6) Emacs/21.3 (berkeley-unix) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Cc: freebsd-hackers@freebsd.org Subject: Re: New FreeBSD package system (a.k.a. Daemon Package System (dps)) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 May 2007 09:41:58 -0000 Ivan Voras writes: > des@ mentioned putting metadata info at the front of the file - I > don't see how this would help. The most common operation with binary > packages *over the network* is "pkg_add -r", which will need to read > it whole anyway, and it would help greatly for things such as > installers from CD media. (Querying a bunch of packages over the > network for their properties, one by one, is not a good idea, but it > is on a local media). Having the metadata in front means you don't need to store a temporary copy of the package in memory or on disk; you extract the metadata in memory, and the rest of the package directly in its final location. AFAIK, pkg_create makes sure that +CONTENTS is always the first file in the archive, precisely to make this possible. The fact that pkg_add doesn't take advantage of it is a bug. DES --=20 Dag-Erling Sm=C3=B8rgrav - des@des.no From owner-freebsd-hackers@FreeBSD.ORG Fri May 11 09:51:55 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 2040916A400 for ; Fri, 11 May 2007 09:51:55 +0000 (UTC) (envelope-from des@des.no) Received: from tim.des.no (tim.des.no [194.63.250.121]) by mx1.freebsd.org (Postfix) with ESMTP id D71D613C45D for ; Fri, 11 May 2007 09:51:54 +0000 (UTC) (envelope-from des@des.no) Received: from tim.des.no (localhost [127.0.0.1]) by spam.des.no (Postfix) with ESMTP id EDBD220A6; Fri, 11 May 2007 11:51:48 +0200 (CEST) X-Spam-Tests: AWL X-Spam-Learn: disabled X-Spam-Score: 0.0/3.0 X-Spam-Checker-Version: SpamAssassin 3.1.7 (2006-10-05) on tim.des.no Received: from dwp.des.no (des.no [80.203.243.180]) by smtp.des.no (Postfix) with ESMTP id E0AF32088; Fri, 11 May 2007 11:51:48 +0200 (CEST) Received: by dwp.des.no (Postfix, from userid 1001) id C756A533F; Fri, 11 May 2007 11:51:48 +0200 (CEST) From: des@des.no (Dag-Erling =?utf-8?Q?Sm=C3=B8rgrav?=) To: Ivan Voras References: <200705102105.27271.blackdragon@highveldmail.co.za> <4643C7DB.6000408@elischer.org> <863b233dq4.fsf@dwp.des.no> <46443AF0.2070006@fer.hr> Date: Fri, 11 May 2007 11:51:48 +0200 In-Reply-To: <46443AF0.2070006@fer.hr> (Ivan Voras's message of "Fri\, 11 May 2007 11\:44\:16 +0200") Message-ID: <86tzuj1ydn.fsf@dwp.des.no> User-Agent: Gnus/5.110006 (No Gnus v0.6) Emacs/21.3 (berkeley-unix) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Cc: freebsd-hackers@freebsd.org Subject: Re: New FreeBSD package system (a.k.a. Daemon Package System (dps)) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 May 2007 09:51:55 -0000 Ivan Voras writes: > Dag-Erling Sm=C3=B8rgrav wrote: > > Ivan Voras writes: > > > You can inspect s sqlite database with the provided utility. Unless t= he > > > database gets corrupted (which it tries to avoid by respecting ACID), > > ACID is not something a database "respects", it is a set of guarantees > > that it provides to the application. Avoiding database corruption is > > a necessary requirement for, rather than a consequence of, ACID. > I'm thinking of ACID as a set of ideas / procedures, the consequence > of which is avoiding corruption. Of course, there's a "hierarchy of > reliability" - the db relies on the file system to meet the > requirements, the file system relies on the hardware, etc. but if the > db doesn't make use of those, it's all for nothing. The world would be a much nicer place if people would stop redefining technical terms to mean whatever suits them. DES --=20 Dag-Erling Sm=C3=B8rgrav - des@des.no From owner-freebsd-hackers@FreeBSD.ORG Fri May 11 09:59:49 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id E570816A405 for ; Fri, 11 May 2007 09:59:49 +0000 (UTC) (envelope-from ivoras@fer.hr) Received: from pinus.cc.fer.hr (pinus.cc.fer.hr [161.53.73.18]) by mx1.freebsd.org (Postfix) with ESMTP id 7997813C458 for ; Fri, 11 May 2007 09:59:49 +0000 (UTC) (envelope-from ivoras@fer.hr) Received: from [161.53.72.113] (lara.cc.fer.hr [161.53.72.113]) by pinus.cc.fer.hr (8.12.2/8.12.2) with ESMTP id l4BA9Q3f022054; Fri, 11 May 2007 12:09:26 +0200 (MEST) Message-ID: <46443E8D.7050301@fer.hr> Date: Fri, 11 May 2007 11:59:41 +0200 From: Ivan Voras User-Agent: Thunderbird 1.5.0.10 (X11/20060911) MIME-Version: 1.0 To: =?UTF-8?B?RGFnLUVybGluZyBTbcO4cmdyYXY=?= References: <200705102105.27271.blackdragon@highveldmail.co.za> <4643C7DB.6000408@elischer.org> <863b233dq4.fsf@dwp.des.no> <46443AF0.2070006@fer.hr> <86tzuj1ydn.fsf@dwp.des.no> In-Reply-To: <86tzuj1ydn.fsf@dwp.des.no> X-Enigmail-Version: 0.94.2.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="------------enig4605A72565FDB07D337F2B2B" Cc: freebsd-hackers@freebsd.org Subject: Re: New FreeBSD package system (a.k.a. Daemon Package System (dps)) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 May 2007 09:59:50 -0000 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enig4605A72565FDB07D337F2B2B Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: quoted-printable Dag-Erling Sm=C3=B8rgrav wrote: > The world would be a much nicer place if people would stop redefining > technical terms to mean whatever suits them. I think you're overreacting. You say: if the database is consistent,=20 it's ACID ("Avoiding database corruption is a necessary requirement for, = rather than a consequence of, ACID") and I say: if the database is ACID, = it's consistent. Aren't we both right? --------------enig4605A72565FDB07D337F2B2B Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.5 (GNU/Linux) Comment: Using GnuPG with SUSE - http://enigmail.mozdev.org iD8DBQFGRD6SldnAQVacBcgRApu8AKDYmms1Dunxc2z46EHM6rcysCH2EwCg/dfh IQuDqr0FUZoi1T/MErJzUuc= =u8PR -----END PGP SIGNATURE----- --------------enig4605A72565FDB07D337F2B2B-- From owner-freebsd-hackers@FreeBSD.ORG Fri May 11 10:11:04 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 2351416A400 for ; Fri, 11 May 2007 10:11:04 +0000 (UTC) (envelope-from joerg@britannica.bec.de) Received: from www.pkgsrc-box.org (www.ostsee-abc.de [62.206.222.50]) by mx1.freebsd.org (Postfix) with ESMTP id B810213C45B for ; Fri, 11 May 2007 10:11:03 +0000 (UTC) (envelope-from joerg@britannica.bec.de) Received: from britannica.bec.de (www.pkgsrc-box.org [127.0.0.1]) by www.pkgsrc-box.org (Postfix) with ESMTP id 547F8E7A3F9 for ; Fri, 11 May 2007 10:11:02 +0000 (UTC) Received: by britannica.bec.de (Postfix, from userid 1000) id 2C3CB7E90; Fri, 11 May 2007 12:10:54 +0200 (CEST) Date: Fri, 11 May 2007 12:10:54 +0200 From: Joerg Sonnenberger To: freebsd-hackers@freebsd.org Message-ID: <20070511101054.GA1847@britannica.bec.de> Mail-Followup-To: freebsd-hackers@freebsd.org References: <200705102105.27271.blackdragon@highveldmail.co.za> <4643C7DB.6000408@elischer.org> <863b233dq4.fsf@dwp.des.no> <46443AF0.2070006@fer.hr> <86tzuj1ydn.fsf@dwp.des.no> <46443E8D.7050301@fer.hr> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <46443E8D.7050301@fer.hr> User-Agent: Mutt/1.5.13 (2006-08-11) Subject: Re: New FreeBSD package system (a.k.a. Daemon Package System (dps)) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 May 2007 10:11:04 -0000 On Fri, May 11, 2007 at 11:59:41AM +0200, Ivan Voras wrote: > I think you're overreacting. You say: if the database is consistent, > it's ACID ("Avoiding database corruption is a necessary requirement for, > rather than a consequence of, ACID") and I say: if the database is ACID, > it's consistent. Aren't we both right? Database corruption violate I (Integrety) and D (Durability). So you can't have a database providing ACID without protecting against database corruption. Consistency is actually the second letter of ACID. That said, there are limits. Enough issues that can corrupt a sqlite database can also corrupt the current pkgdb tree. In fact, a number of issues are less likely to hit sqlite. It doesn't mean that either is perfect. Joerg From owner-freebsd-hackers@FreeBSD.ORG Fri May 11 10:37:12 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id E469B16A40B for ; Fri, 11 May 2007 10:37:12 +0000 (UTC) (envelope-from des@des.no) Received: from tim.des.no (tim.des.no [194.63.250.121]) by mx1.freebsd.org (Postfix) with ESMTP id A4FC813C469 for ; Fri, 11 May 2007 10:37:12 +0000 (UTC) (envelope-from des@des.no) Received: from tim.des.no (localhost [127.0.0.1]) by spam.des.no (Postfix) with ESMTP id 0D6AB20AC; Fri, 11 May 2007 12:37:09 +0200 (CEST) X-Spam-Tests: AWL X-Spam-Learn: disabled X-Spam-Score: 0.0/3.0 X-Spam-Checker-Version: SpamAssassin 3.1.7 (2006-10-05) on tim.des.no Received: from dwp.des.no (des.no [80.203.243.180]) by smtp.des.no (Postfix) with ESMTP id 7A6E820AB; Fri, 11 May 2007 12:37:08 +0200 (CEST) Received: by dwp.des.no (Postfix, from userid 1001) id 582F65347; Fri, 11 May 2007 12:37:08 +0200 (CEST) From: des@des.no (Dag-Erling =?utf-8?Q?Sm=C3=B8rgrav?=) To: Ivan Voras References: <200705102105.27271.blackdragon@highveldmail.co.za> <4643C7DB.6000408@elischer.org> <863b233dq4.fsf@dwp.des.no> <46443AF0.2070006@fer.hr> <86tzuj1ydn.fsf@dwp.des.no> <46443E8D.7050301@fer.hr> Date: Fri, 11 May 2007 12:37:08 +0200 In-Reply-To: <46443E8D.7050301@fer.hr> (Ivan Voras's message of "Fri\, 11 May 2007 11\:59\:41 +0200") Message-ID: <86ps571wa3.fsf@dwp.des.no> User-Agent: Gnus/5.110006 (No Gnus v0.6) Emacs/21.3 (berkeley-unix) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Cc: freebsd-hackers@freebsd.org Subject: Re: New FreeBSD package system (a.k.a. Daemon Package System (dps)) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 May 2007 10:37:13 -0000 Ivan Voras writes: > Dag-Erling Sm=C3=B8rgrav wrote: > > The world would be a much nicer place if people would stop redefining > > technical terms to mean whatever suits them. > I think you're overreacting. You say: if the database is consistent, > it's ACID ("Avoiding database corruption is a necessary requirement > for, rather than a consequence of, ACID") and I say: if the database > is ACID, it's consistent. Aren't we both right? First, that's not what I'm saying. Look up "requirement" and "consequence" in a dictionary (and read up on Leibniz if you feel like it) Second, ACID is much more than not corrupting the database. It also means returning consistent results for the duration of a transaction, for instance. DES --=20 Dag-Erling Sm=C3=B8rgrav - des@des.no From owner-freebsd-hackers@FreeBSD.ORG Fri May 11 14:01:36 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 1015016A400 for ; Fri, 11 May 2007 14:01:36 +0000 (UTC) (envelope-from andre@freebsd.org) Received: from c00l3r.networx.ch (c00l3r.networx.ch [62.48.2.2]) by mx1.freebsd.org (Postfix) with ESMTP id 7A18613C448 for ; Fri, 11 May 2007 14:01:35 +0000 (UTC) (envelope-from andre@freebsd.org) Received: (qmail 96448 invoked from network); 11 May 2007 13:21:06 -0000 Received: from dotat.atdotat.at (HELO [62.48.0.47]) ([62.48.0.47]) (envelope-sender ) by c00l3r.networx.ch (qmail-ldap-1.03) with SMTP for ; 11 May 2007 13:21:06 -0000 Message-ID: <4644773E.60909@freebsd.org> Date: Fri, 11 May 2007 16:01:34 +0200 From: Andre Oppermann User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.8b) Gecko/20050217 MIME-Version: 1.0 To: Ed Schouten References: <45F1C355.8030504@digitaldaemon.com> <20070511075857.GL23313@hoeg.nl> In-Reply-To: <20070511075857.GL23313@hoeg.nl> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: FreeBSD Hackers Subject: Re: Multiple IP Jail's patch for FreeBSD 6.2 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 May 2007 14:01:36 -0000 Ed Schouten wrote: > Hello, > > It may be interesting to mention that yesterday there was a presentation > at the NLUUG (Netherlands UNIX Users Group) conference by Marco Zec, who > once wrote a patchset for FreeBSD 4.11 (and is in the process of porting > it to FreeBSD 7.x) that gives each jail its own networking stack. > > You can hook up physical interfaces to jails or perform bridging between > jails through netgraph bridging code. That way you can create virtual > network topologies on a single box. This will allow you to use multiple > IPv4 and IPv6 addresses on each instance. You can even use (I)PF(W) > inside jails. I'm working on a "light" variant of multi-IPv[46] per jail. It doesn't create an entirely new network instance per jail and probably is more suitable for low- to mid-end (virtual) hosting. In those cases you normally want the host administrator to excercise full control over IP address and firewall configuration of the individual jails. For high-end stuff where you offer jail based virtual machines or network and routing simulations Marco's work is more appropriate. -- Andre > More information about this can be found at: > > http://www.tel.fer.hr/zec/vimage/ > http://imunes.net/ > > Yours, From owner-freebsd-hackers@FreeBSD.ORG Fri May 11 14:36:14 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 7E6BD16A400 for ; Fri, 11 May 2007 14:36:14 +0000 (UTC) (envelope-from mwm-keyword-freebsdhackers2.e313df@mired.org) Received: from mired.org (vpn.mired.org [66.92.153.74]) by mx1.freebsd.org (Postfix) with SMTP id 2F6CB13C45A for ; Fri, 11 May 2007 14:36:14 +0000 (UTC) (envelope-from mwm-keyword-freebsdhackers2.e313df@mired.org) Received: (qmail 39024 invoked by uid 1001); 11 May 2007 14:35:42 -0000 Received: by bhuda.mired.org (tmda-sendmail, from uid 1001); Fri, 11 May 2007 10:35:42 -0400 MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <17988.32573.910854.388638@bhuda.mired.org> Date: Fri, 11 May 2007 10:35:41 -0400 To: Kris Kennaway In-Reply-To: <20070511051852.GA89359@xor.obsecurity.org> References: <200705102105.27271.blackdragon@highveldmail.co.za> <17987.52037.112351.872442@bhuda.mired.org> <20070511015156.GA77895@xor.obsecurity.org> <17987.52970.398402.580727@bhuda.mired.org> <20070511021249.GA78729@xor.obsecurity.org> <17987.57305.7130.873114@bhuda.mired.org> <20070511051852.GA89359@xor.obsecurity.org> X-Mailer: VM 7.19 under Emacs 21.3.1 X-Primary-Address: mwm@mired.org X-face: "5Mnwy%?j>IIV\)A=):rjWL~NB2aH[}Yq8Z=u~vJ`"(,&SiLvbbz2W`; h9L,Yg`+vb1>RG% *h+%X^n0EZd>TM8_IB;a8F?(Fb"lw'IgCoyM.[Lg#r\ X-Delivery-Agent: TMDA/1.1.11 (Ladyburn) From: Mike Meyer Cc: freebsd-hackers@freebsd.org, Ivan Voras Subject: Re: New FreeBSD package system (a.k.a. Daemon Package System (dps)) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 May 2007 14:36:14 -0000 In <20070511051852.GA89359@xor.obsecurity.org>, Kris Kennaway typed: > There are a few ways you can go. The simplest is to install a > complete i386 world in e.g. /compat/ia32 and have i386 packages > installed there, and change the kernel to do a "magic directory > lookup" for i386 binaries that does path munging like the linuxulator > does with /compat/linux. Actually, the simplest is to install a virtual machine and just use that. But none of these is quite ideal. > If you want the i386 packages to live in /usr/local alongside the > amd64 packages, you'll need to do something like adding an @arch > directive to the +CONTENTS file recorded in /var/db/pkg, and add some > checking to pkg_add to ensure that when you install a package, > everything it depends on was built for the same architecture. Which is pretty much what I pointed out in my original request. Except you don't really need everything it depends on to be built for the same architecture. If your port needs gmake to build, or uses ghostscript to do ps->jpeg conversion or some such, it won't really care what architecture the executables were built for - just that they run. It would help if requirements were flagged with whether or not they required the same architecture. > You'd need to also add these checks to bsd.port.mk so it exits > gracefully when someone tries to natively compile a port for which > non-native dependencies are installed (e.g. it's going to be really > unhappy if it finds i386 headers or libraries). This method might be > more trouble than it's worth. Again, it might or might not be unhappy. But now you're venturing into ports as opposed to the package system. I have a whole other list of things to consider there. > > On the other hand, if no one has actually done the > > work to figure out what this would really take, is wishful thinking > > really enough to keep a very desirable feature (well, it's desirable > > enough that most Linux platforms seem to offer it) from even being > > considered? > You misunderstand me: by all means people should work on improving our > package infrastructure -- but it's wrong to pin your hopes on a > possibly mythical future event when you can easily solve a problem > today. Oh, I've been dealing with BSD long enough to know better than to pin my hopes on such things. Even doing it may not be enough - you still have to get someone to *commit* it. On the other hand, someone stepped up and said "We're going to rework the package system." I don't think it's to much to ask that they keep this requirement in mind. > > > > > Not gonna happen as a default, but you can change it on your systems > > > > > if you like. > > > > Not very reliably. > > > Best I can offer ;) > > > > Is this the new motto for FreeBSD? Good QA practices would have the > > ports built with that knob set to something something other than the > > default at regular intervals. Lately things have been better, but > > I've found broken things in the last twelve months. > You'll be delighted to know that I have been doing precisely this for > the past few years. If you're interested I'll CC you the errors from > my next build so you can help work on improving compliance. Actually, what i'm delighted about is that I actually noticed things were getting better - the last time I did a near-complete set of upgrades, nothing broke because of LOCALBASE having a non-default setting. I had figured this was just do to people reporting such bugs - I know I always do. It seems that you're responsible. Thank you. I still think we ought to quit pretending that ports/packages aren't part of BSD, and default LOCALBASE to /usr. But if changing it is being tested, that's a big help. Thanks, http://www.mired.org/consulting.html Independent Network/Unix/Perforce consultant, email for more information. From owner-freebsd-hackers@FreeBSD.ORG Fri May 11 14:12:27 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id AE0EF16A400 for ; Fri, 11 May 2007 14:12:27 +0000 (UTC) (envelope-from alexander@leidinger.net) Received: from redbull.bpaserver.net (redbullneu.bpaserver.net [213.198.78.217]) by mx1.freebsd.org (Postfix) with ESMTP id 20ED213C480 for ; Fri, 11 May 2007 14:12:27 +0000 (UTC) (envelope-from alexander@leidinger.net) Received: from outgoing.leidinger.net (p54a5e6b4.dip.t-dialin.net [84.165.230.180]) by redbull.bpaserver.net (Postfix) with ESMTP id 67F662E1A5; Fri, 11 May 2007 16:12:17 +0200 (CEST) Received: from webmail.leidinger.net (webmail.Leidinger.net [192.168.1.102]) by outgoing.leidinger.net (Postfix) with ESMTP id B5CAC5B48A3; Fri, 11 May 2007 16:12:01 +0200 (CEST) Received: (from www@localhost) by webmail.leidinger.net (8.13.8/8.13.8/Submit) id l4BEC0nt061925; Fri, 11 May 2007 16:12:00 +0200 (CEST) (envelope-from Alexander@Leidinger.net) Received: from pslux.cec.eu.int (pslux.cec.eu.int [158.169.9.14]) by webmail.leidinger.net (Horde MIME library) with HTTP; Fri, 11 May 2007 16:12:00 +0200 Message-ID: <20070511161200.w7tzoc5mkg8og44c@webmail.leidinger.net> X-Priority: 3 (Normal) Date: Fri, 11 May 2007 16:12:00 +0200 From: Alexander Leidinger To: Dag-Erling =?utf-8?b?U23Dg8K4cmdyYXY=?= References: <200705102105.27271.blackdragon@highveldmail.co.za> <17987.52037.112351.872442@bhuda.mired.org> <20070511015156.GA77895@xor.obsecurity.org> <86zm4b3h7h.fsf@dwp.des.no> <20070511082657.GA24652@xor.obsecurity.org> <86sla33gkm.fsf@dwp.des.no> In-Reply-To: <86sla33gkm.fsf@dwp.des.no> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8; DelSp="Yes"; format="flowed" Content-Disposition: inline Content-Transfer-Encoding: quoted-printable User-Agent: Internet Messaging Program (IMP) H3 (4.1.4) / FreeBSD-7.0 X-BPAnet-MailScanner-Information: Please contact the ISP for more information X-BPAnet-MailScanner: Found to be clean X-BPAnet-MailScanner-SpamCheck: not spam, SpamAssassin (not cached, score=-14.564, required 8, BAYES_00 -15.00, DK_POLICY_SIGNSOME 0.00, FORGED_RCVD_HELO 0.14, MIME_8BIT_HEADER 0.30) X-BPAnet-MailScanner-From: alexander@leidinger.net X-Spam-Status: No X-Mailman-Approved-At: Fri, 11 May 2007 14:48:04 +0000 Cc: freebsd-hackers@freebsd.org, Mike, Meyer , Ivan Voras , Kris Kennaway Subject: Re: New FreeBSD package system (a.k.a. Daemon Package System (dps)) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 May 2007 14:12:27 -0000 Quoting Dag-Erling Sm=C3=83=C2=B8rgrav (from Fri, 11 May 2007 = =20 10:33:29 +0200): > The existence of .la files is a bug. I fully agree. > We already have a mechanism for recording dependencies between > libraries; it's built into the ELF format, and does not require > hardcoding any directories. Introducing .la files which override the > existing mechanism and *do* hardcode directories is a regression. > > I don't buy the argument that "KDE won't build without them", or > whatever it was you used to justify this. There is nothing an .la file > does which can't be done more properly by adding the correct directory > to your ldconfig path. Unfortunately you are addressing this to the wrong people. You need to =20 talk with the libtool people. Trying to use a libtool version which is =20 as closest as possible to the version distributed by the authors is a =20 very sane requirement for our ports collection and the users which =20 expect a sane behavior of libtool when they want to create a tarball =20 which is supposed to be cross-platform (for an appropriate value of =20 cross-platform). So you need to address the issues upstream. As soon as a new release =20 comes out with the improvements you suggest, we will get them in =20 FreeBSD. The problem is that a lot of users of libltdl try to open the =20 .la file instead of the lib (AFAIR libltdl tries to first open the .la =20 then the .so if you don't have a .la or .so ending in the dlopen() =20 string, but most people specify the .la... I could misremember this, =20 it's been a while since I looked at it). The way to go is to teach libtool about the ELF features and to not =20 produce the .la files on ELF systems. Additionally a warning in =20 libltdl needs to printed in case the .la file is used (as a second =20 step maybe adding a long delay to the loading in case a .la file is =20 used, to provoke the transition). While it may be more easy to talk about doing something like this in =20 our ports collection, the way to go is to address this upstream with =20 the libtool authors, and not with portmgr/ade or on hackers/ports. Bye, Alexander. --=20 Law of the Yukon: =09Only the lead dog gets a change of scenery. http://www.Leidinger.net Alexander @ Leidinger.net: PGP ID =3D B0063FE7 http://www.FreeBSD.org netchild @ FreeBSD.org : PGP ID =3D 72077137 From owner-freebsd-hackers@FreeBSD.ORG Fri May 11 15:23:32 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 9288B16A403 for ; Fri, 11 May 2007 15:23:32 +0000 (UTC) (envelope-from mwm-keyword-freebsdhackers2.e313df@mired.org) Received: from mired.org (vpn.mired.org [66.92.153.74]) by mx1.freebsd.org (Postfix) with SMTP id 368AD13C45B for ; Fri, 11 May 2007 15:23:32 +0000 (UTC) (envelope-from mwm-keyword-freebsdhackers2.e313df@mired.org) Received: (qmail 39613 invoked by uid 1001); 11 May 2007 15:23:00 -0000 Received: by bhuda.mired.org (tmda-sendmail, from uid 1001); Fri, 11 May 2007 11:23:00 -0400 MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <17988.35412.231093.411177@bhuda.mired.org> Date: Fri, 11 May 2007 11:23:00 -0400 To: Ivan Voras In-Reply-To: References: <200705102105.27271.blackdragon@highveldmail.co.za> <4643C7DB.6000408@elischer.org> X-Mailer: VM 7.19 under Emacs 21.3.1 X-Primary-Address: mwm@mired.org X-face: "5Mnwy%?j>IIV\)A=):rjWL~NB2aH[}Yq8Z=u~vJ`"(,&SiLvbbz2W`; h9L,Yg`+vb1>RG% *h+%X^n0EZd>TM8_IB;a8F?(Fb"lw'IgCoyM.[Lg#r\ X-Delivery-Agent: TMDA/1.1.11 (Ladyburn) From: Mike Meyer Cc: freebsd-hackers@freebsd.org Subject: SQL in the base system (Was: New FreeBSD package system (a.k.a. Daemon Package System (dps))) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 May 2007 15:23:32 -0000 > Perhaps this is a good time I should mention that I think sqlite would > also be good for the password and login databases? :) Someone has already pointed out the horror that is the Windows registry. IIUC, even MS has figured out this is a bad idea, and gotten away from it with Vista. But it's been tried on Unix systems before as well. People who've dealt with AIX tend to feel the same way about it's config system as they do about the Windows Registry. Having a magic tool to edit these files is a pain - it means you now have two sets of things to learn: the magic tool, and the specific details of the file you're dealing with. On a Unix system, knowing how to tweak text files is pretty much a given. So all you need is the details for that file. Using a binary format brings it's own problems. How hard is it to fix a corrupt database? How hard is it to figure out that the database is so corrupt you aren't going to get anything out of it, so you might as well give up? How robust is it - can a corrupt block fry the entire database? How about portability - can I move the file to a completely different architecture and still get the data from it? If any of these are noticably worse than they are for text files, changing is probably not a good idea. Someone else mentioned XML. Well, it's easy to parse - assuming you read that as "someone else wrote the parser for us". That's true for lots of things. I also question the sanity of using it. But I wind up doing a lot of it, because my clients like the buzzword compliance. I regularly beat on them to take advantage of what XML can do that other formats can't. Meaning I make them install validation software, and pay me to write schemas for them, and get them to add hooks to the repository so you can't check in xml files that don't validate. But if you're not going to do that kind of thing, the major feature XML brings is the buzzword compliance. http://www.mired.org/consulting.html Independent Network/Unix/Perforce consultant, email for more information. From owner-freebsd-hackers@FreeBSD.ORG Fri May 11 15:29:12 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 6508216A404 for ; Fri, 11 May 2007 15:29:12 +0000 (UTC) (envelope-from kientzle@freebsd.org) Received: from kientzle.com (h-66-166-149-50.snvacaid.covad.net [66.166.149.50]) by mx1.freebsd.org (Postfix) with ESMTP id 3FBC013C459 for ; Fri, 11 May 2007 15:29:12 +0000 (UTC) (envelope-from kientzle@freebsd.org) Received: from [10.0.0.222] (p54.kientzle.com [66.166.149.54]) by kientzle.com (8.12.9/8.12.9) with ESMTP id l4BEw2H7004531; Fri, 11 May 2007 07:58:02 -0700 (PDT) (envelope-from kientzle@freebsd.org) Message-ID: <4644847A.5060702@freebsd.org> Date: Fri, 11 May 2007 07:58:02 -0700 From: Tim Kientzle User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.7.12) Gecko/20060422 X-Accept-Language: en-us, en MIME-Version: 1.0 To: Ivan Voras References: <200705102105.27271.blackdragon@highveldmail.co.za> In-Reply-To: Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-hackers@freebsd.org Subject: Re: New FreeBSD package system (a.k.a. Daemon Package System (dps)) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 May 2007 15:29:12 -0000 Ivan Voras wrote: > - A quick test confirms that the current bsdtar will happily ignore any > extra data at the end of a tgz/tbz archive, so package metadata can be > embedded there, thus conserving existing infrastructure... Not a good idea at all. 1) Keeping everything within the archive makes it possible for people to do surgery on the packages. Being able to extract/modify/rebuild packages using just tar makes it much easier to debug and experiment with new techniques. 2) The existing metadata format is not very pretty, but it can be reasonably extended. (Personally, I don't find XML very pretty, either. ;-) 3) As DES pointed out, the package tools must be able to read the metadata before the files. If you really need a completely separate metadata file, make it the second file in the archive. Tim Kientzle From owner-freebsd-hackers@FreeBSD.ORG Fri May 11 15:34:59 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 1602316A400 for ; Fri, 11 May 2007 15:34:59 +0000 (UTC) (envelope-from joerg@britannica.bec.de) Received: from www.pkgsrc-box.org (www.ostsee-abc.de [62.206.222.50]) by mx1.freebsd.org (Postfix) with ESMTP id C91AE13C447 for ; Fri, 11 May 2007 15:34:58 +0000 (UTC) (envelope-from joerg@britannica.bec.de) Received: from britannica.bec.de (www.pkgsrc-box.org [127.0.0.1]) by www.pkgsrc-box.org (Postfix) with ESMTP id AB6C6E7A3F9 for ; Fri, 11 May 2007 15:34:57 +0000 (UTC) Received: by britannica.bec.de (Postfix, from userid 1000) id A21747E90; Fri, 11 May 2007 17:34:48 +0200 (CEST) Date: Fri, 11 May 2007 17:34:48 +0200 From: Joerg Sonnenberger To: freebsd-hackers@freebsd.org Message-ID: <20070511153448.GA7516@britannica.bec.de> Mail-Followup-To: freebsd-hackers@freebsd.org References: <200705102105.27271.blackdragon@highveldmail.co.za> <4644847A.5060702@freebsd.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4644847A.5060702@freebsd.org> User-Agent: Mutt/1.5.13 (2006-08-11) Subject: Re: New FreeBSD package system (a.k.a. Daemon Package System (dps)) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 May 2007 15:34:59 -0000 On Fri, May 11, 2007 at 07:58:02AM -0700, Tim Kientzle wrote: > 3) As DES pointed out, the package tools must be able > to read the metadata before the files. If you really > need a completely separate metadata file, make it > the second file in the archive. Actually, the argument is pretty weak. Being able to extract them streamable and access the meta-data easily is fine. The remote access argument is very weak as it doesn't allow e.g. signature checks. Joerg From owner-freebsd-hackers@FreeBSD.ORG Fri May 11 16:13:06 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 2386A16A403 for ; Fri, 11 May 2007 16:13:06 +0000 (UTC) (envelope-from freebsd-hackers@m.gmane.org) Received: from ciao.gmane.org (main.gmane.org [80.91.229.2]) by mx1.freebsd.org (Postfix) with ESMTP id A14AF13C45B for ; Fri, 11 May 2007 16:13:05 +0000 (UTC) (envelope-from freebsd-hackers@m.gmane.org) Received: from list by ciao.gmane.org with local (Exim 4.43) id 1HmXU2-0008Gp-Vw for freebsd-hackers@freebsd.org; Fri, 11 May 2007 17:56:43 +0200 Received: from lara.cc.fer.hr ([161.53.72.113]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Fri, 11 May 2007 17:56:42 +0200 Received: from ivoras by lara.cc.fer.hr with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Fri, 11 May 2007 17:56:42 +0200 X-Injected-Via-Gmane: http://gmane.org/ To: freebsd-hackers@freebsd.org From: Ivan Voras Date: Fri, 11 May 2007 17:54:09 +0200 Lines: 86 Message-ID: References: <200705102105.27271.blackdragon@highveldmail.co.za> <4643C7DB.6000408@elischer.org> <17988.35412.231093.411177@bhuda.mired.org> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="------------enigA0F802F6F6CE7264877492FB" X-Complaints-To: usenet@sea.gmane.org X-Gmane-NNTP-Posting-Host: lara.cc.fer.hr User-Agent: Thunderbird 1.5.0.10 (X11/20060911) In-Reply-To: <17988.35412.231093.411177@bhuda.mired.org> X-Enigmail-Version: 0.94.2.0 Sender: news Subject: Re: SQL in the base system X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 May 2007 16:13:06 -0000 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enigA0F802F6F6CE7264877492FB Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: quoted-printable Mike Meyer wrote: >> Perhaps this is a good time I should mention that I think sqlite would= >> also be good for the password and login databases? :) >=20 > Someone has already pointed out the horror that is the Windows > registry. IIUC, even MS has figured out this is a bad idea, and gotten > away from it with Vista. But it's been tried on Unix systems before as This is the first time I hear about Vista - AFAIK it relies even more on = registry, to the point that the boot process also uses it. Registry was pretty bad in Win9x, but AFAIK most of those issues were=20 because FAT32 is a bad file system. I never had a registry problem (on=20 many machines) running Win2k and WinXP. > Using a binary format brings it's own problems. How hard is it to fix > a corrupt database? How hard is it to figure out that the database is > so corrupt you aren't going to get anything out of it, so you might as > well give up? How robust is it - can a corrupt block fry the entire > database? How about portability - can I move the file to a completely > different architecture and still get the data from it? If any of these > are noticably worse than they are for text files, changing is probably > not a good idea. Most of those issues are valid, but don't strongly advocate against=20 databases, because the same issues (corruption, rebuilding, manual=20 inspection) are present for directories of text files. Endian issues are = solved in sqlite. I don't think databases are so scary (but possibly that's lack of=20 experience on my part). If you would get a corrupt block in the middle=20 of a complex text file, it would wedge your system just as bad as if you = got a bad block in a table in the database (anecdotally, sqlite database = can be read even in those circumstances, if you avoid the corrupt=20 table). (And a corrupt block in an important metadata section is the=20 same as a corrupt block in the directory record on the file system). The = objective downside is that there are more blocks in a database. > Someone else mentioned XML. Well, it's easy to parse - assuming you > read that as "someone else wrote the parser for us". That's true for > lots of things. I also question the sanity of using it. But I wind up > doing a lot of it, because my clients like the buzzword compliance. I > regularly beat on them to take advantage of what XML can do that other > formats can't. Meaning I make them install validation software, and > pay me to write schemas for them, and get them to add hooks to the > repository so you can't check in xml files that don't validate. But if > you're not going to do that kind of thing, the major feature XML > brings is the buzzword compliance. Bofore I get misunderstood, I'd like to say that I'm not actually such a = big fan of XML, but I think it's currently the lesser of evils. The=20 alternative is either to create a one-off file format for each and every = purpose (aka "the unix way"), or use some XML-replacement (JSON &=20 others) which has most of the evils of XML and introduce lack of support = from existing tools. --------------enigA0F802F6F6CE7264877492FB Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.5 (GNU/Linux) Comment: Using GnuPG with SUSE - http://enigmail.mozdev.org iD8DBQFGRJGtldnAQVacBcgRAo22AJ9m/aws78NY/jbgLEwV98L9E++3NwCglfwN aV3f2xwPd/Yje/fUIvfFF/s= =m8Eq -----END PGP SIGNATURE----- --------------enigA0F802F6F6CE7264877492FB-- From owner-freebsd-hackers@FreeBSD.ORG Fri May 11 16:14:17 2007 Return-Path: X-Original-To: hackers@freebsd.org Delivered-To: freebsd-hackers@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 01E3716A402 for ; Fri, 11 May 2007 16:14:17 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from cyrus.watson.org (cyrus.watson.org [209.31.154.42]) by mx1.freebsd.org (Postfix) with ESMTP id CB76813C447 for ; Fri, 11 May 2007 16:14:16 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from fledge.watson.org (fledge.watson.org [209.31.154.41]) by cyrus.watson.org (Postfix) with ESMTP id 82C2546CB1; Fri, 11 May 2007 12:14:16 -0400 (EDT) Date: Fri, 11 May 2007 17:14:16 +0100 (BST) From: Robert Watson X-X-Sender: robert@fledge.watson.org To: Sergey Zaharchenko In-Reply-To: <20070510142147.GA83797@shark.localdomain> Message-ID: <20070511171345.M86304@fledge.watson.org> References: <20070510142147.GA83797@shark.localdomain> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: hackers@freebsd.org, brian@FreeBSD.org Subject: Re: My PPP timer PR [nag] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 May 2007 16:14:17 -0000 On Thu, 10 May 2007, Sergey Zaharchenko wrote: > bin/102747 has been sitting there for about 8 months, with no activity since > it was assigned to brian@, all my mail to whom bounces [CC'd just in case]. > > The patch attached in the PR has been working for me since, so it not being > fixed in the main tree isn't a problem with me. I just thought someone would > benefit from it being committed... In general, I would suggest sending this sort of thing to current@ and/or net@, but not hackers@. Robert N M Watson Computer Laboratory University of Cambridge From owner-freebsd-hackers@FreeBSD.ORG Fri May 11 16:45:11 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 6C8C816A403 for ; Fri, 11 May 2007 16:45:11 +0000 (UTC) (envelope-from mwm-keyword-freebsdhackers2.e313df@mired.org) Received: from mired.org (vpn.mired.org [66.92.153.74]) by mx1.freebsd.org (Postfix) with SMTP id 1EAF413C447 for ; Fri, 11 May 2007 16:45:10 +0000 (UTC) (envelope-from mwm-keyword-freebsdhackers2.e313df@mired.org) Received: (qmail 40928 invoked by uid 1001); 11 May 2007 16:44:39 -0000 Received: by bhuda.mired.org (tmda-sendmail, from uid 1001); Fri, 11 May 2007 12:44:39 -0400 MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <17988.40311.210855.381093@bhuda.mired.org> Date: Fri, 11 May 2007 12:44:39 -0400 To: Ivan Voras In-Reply-To: References: <200705102105.27271.blackdragon@highveldmail.co.za> <4643C7DB.6000408@elischer.org> <17988.35412.231093.411177@bhuda.mired.org> X-Mailer: VM 7.19 under Emacs 21.3.1 X-Primary-Address: mwm@mired.org X-face: "5Mnwy%?j>IIV\)A=):rjWL~NB2aH[}Yq8Z=u~vJ`"(,&SiLvbbz2W`; h9L,Yg`+vb1>RG% *h+%X^n0EZd>TM8_IB;a8F?(Fb"lw'IgCoyM.[Lg#r\ X-Delivery-Agent: TMDA/1.1.11 (Ladyburn) From: Mike Meyer Cc: freebsd-hackers@freebsd.org Subject: Re: SQL in the base system X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 May 2007 16:45:11 -0000 In , Ivan Voras typed: > Mike Meyer wrote: > >> Perhaps this is a good time I should mention that I think sqlite would > >> also be good for the password and login databases? :) > > > > Someone has already pointed out the horror that is the Windows > > registry. IIUC, even MS has figured out this is a bad idea, and gotten > > away from it with Vista. But it's been tried on Unix systems before as > > This is the first time I hear about Vista - AFAIK it relies even more on > registry, to the point that the boot process also uses it. That may be so. I haven't dealt with Vista, but had heard it dropped the registry dependencies. > Registry was pretty bad in Win9x, but AFAIK most of those issues were > because FAT32 is a bad file system. I never had a registry problem (on > many machines) running Win2k and WinXP. No, that only deals with it getting corrupted by the file system. I've had registery problems as late as XP/SP2. And of course, you can't blame the issues with the AIX object manager on FAT32. > > Using a binary format brings it's own problems. How hard is it to fix > > a corrupt database? How hard is it to figure out that the database is > > so corrupt you aren't going to get anything out of it, so you might as > > well give up? How robust is it - can a corrupt block fry the entire > > database? How about portability - can I move the file to a completely > > different architecture and still get the data from it? If any of these > > are noticably worse than they are for text files, changing is probably > > not a good idea. > Most of those issues are valid, but don't strongly advocate against > databases, because the same issues (corruption, rebuilding, manual > inspection) are present for directories of text files. Endian issues are > solved in sqlite. Yes, they are present no matter what representation you use. The question is - how do the answers change if you change the format. These days, cross-platform means you deal with length as well as endian issues. Or maybe you don't, depending on the db. I know the answers for text files (easy, easy, very, yes). Can you propose a db scheme that gets has the same answers? > > Someone else mentioned XML. Well, it's easy to parse - assuming you > > read that as "someone else wrote the parser for us". That's true for > > lots of things. I also question the sanity of using it. But I wind up > > doing a lot of it, because my clients like the buzzword compliance. I > > regularly beat on them to take advantage of what XML can do that other > > formats can't. Meaning I make them install validation software, and > > pay me to write schemas for them, and get them to add hooks to the > > repository so you can't check in xml files that don't validate. But if > > you're not going to do that kind of thing, the major feature XML > > brings is the buzzword compliance. > Bofore I get misunderstood, I'd like to say that I'm not actually such a > big fan of XML, but I think it's currently the lesser of evils. The > alternative is either to create a one-off file format for each and every > purpose (aka "the unix way"), or use some XML-replacement (JSON & > others) which has most of the evils of XML and introduce lack of support > from existing tools. I hate to tell you this, but your XML solution would still consist of a bunch of one-of file formats for each and every purpose. Using XML just fixes the syntax for the file, not the semantics. Settling on XML (or JSON, or INI, or cap files, or ...) is sort of like settling on UTF, only less obviously a win. Sure, you get to use canned code that will turn you text file into a structure in memory. But you still have to figure out what it all means. As you say, the XML toolset is the real win. Smart editors, validators, schemas (which make the editors and validators even more powerful) are all good things. Most people don't really seem interested in this beyond editors. That's not really much of a win. http://www.mired.org/consulting.html Independent Network/Unix/Perforce consultant, email for more information. From owner-freebsd-hackers@FreeBSD.ORG Fri May 11 17:01:25 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id C185816A403 for ; Fri, 11 May 2007 17:01:25 +0000 (UTC) (envelope-from freebsd-hackers@m.gmane.org) Received: from ciao.gmane.org (main.gmane.org [80.91.229.2]) by mx1.freebsd.org (Postfix) with ESMTP id 4969B13C458 for ; Fri, 11 May 2007 17:01:25 +0000 (UTC) (envelope-from freebsd-hackers@m.gmane.org) Received: from list by ciao.gmane.org with local (Exim 4.43) id 1HmYUP-0004oK-6V for freebsd-hackers@freebsd.org; Fri, 11 May 2007 19:01:09 +0200 Received: from lara.cc.fer.hr ([161.53.72.113]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Fri, 11 May 2007 19:01:09 +0200 Received: from ivoras by lara.cc.fer.hr with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Fri, 11 May 2007 19:01:09 +0200 X-Injected-Via-Gmane: http://gmane.org/ To: freebsd-hackers@freebsd.org From: Ivan Voras Date: Fri, 11 May 2007 19:00:41 +0200 Lines: 68 Message-ID: References: <200705102105.27271.blackdragon@highveldmail.co.za> <4643C7DB.6000408@elischer.org> <17988.35412.231093.411177@bhuda.mired.org> <17988.40311.210855.381093@bhuda.mired.org> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="------------enigF3CBC1EA99334E21A7E11CAB" X-Complaints-To: usenet@sea.gmane.org X-Gmane-NNTP-Posting-Host: lara.cc.fer.hr User-Agent: Thunderbird 1.5.0.10 (X11/20060911) In-Reply-To: <17988.40311.210855.381093@bhuda.mired.org> X-Enigmail-Version: 0.94.2.0 Sender: news Subject: Re: SQL in the base system X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 May 2007 17:01:25 -0000 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enigF3CBC1EA99334E21A7E11CAB Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: quoted-printable Mike Meyer wrote: > Yes, they are present no matter what representation you use. The > question is - how do the answers change if you change the > format. These days, cross-platform means you deal with length as well > as endian issues. Or maybe you don't, depending on the db. I know the > answers for text files (easy, easy, very, yes). Can you propose a db > scheme that gets has the same answers? I think I don't understand the question. If the database contains number = "42" in a field typed "int32", in a row, and handles endianess well, why = would I get a different number on different platforms? (A side note about sqlite: it's actually weakly typed - you store and=20 receive strings). > I hate to tell you this, but your XML solution would still consist of > a bunch of one-of file formats for each and every purpose. Using XML > just fixes the syntax for the file, not the semantics. Settling on XML > (or JSON, or INI, or cap files, or ...) is sort of like settling on > UTF, only less obviously a win. Sure, you get to use canned code that > will turn you text file into a structure in memory. But you still have > to figure out what it all means. >=20 > As you say, the XML toolset is the real win. Smart editors, > validators, schemas (which make the editors and validators even more > powerful) are all good things. Most people don't really seem > interested in this beyond editors. That's not really much of a win. I agree that validation in XML is a strong point - but one of the reason = people like text files is that they DON'T usually have validation=20 features :) | pro | contra ---------------------------------------------------------------------- XML | standard tools, validation, | evil manual parsing, bad rep | can embed multiple data | | structures in a standard way | ---------------------------------------------------------------------- text | standard tools, sometimes | no validation, manual parsing, | human readable | usually one data structure per | | file --------------enigF3CBC1EA99334E21A7E11CAB Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.5 (GNU/Linux) Comment: Using GnuPG with SUSE - http://enigmail.mozdev.org iD8DBQFGRKE5ldnAQVacBcgRAvLpAKDAshAiP7IZW3RFYZLQ6SHMd6oKJgCgjgQj emQqM6aQhwfS+Rlh8duNq2k= =iJpD -----END PGP SIGNATURE----- --------------enigF3CBC1EA99334E21A7E11CAB-- From owner-freebsd-hackers@FreeBSD.ORG Fri May 11 17:22:32 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id BA82716A402 for ; Fri, 11 May 2007 17:22:32 +0000 (UTC) (envelope-from jos@catnook.com) Received: from lizzy.dyndns.org (209-204-188-132.dsl.static.sonic.net [209.204.188.132]) by mx1.freebsd.org (Postfix) with SMTP id 5D77513C44B for ; Fri, 11 May 2007 17:22:32 +0000 (UTC) (envelope-from jos@catnook.com) Received: (qmail 48635 invoked by uid 1000); 11 May 2007 16:56:12 -0000 Date: Fri, 11 May 2007 09:56:12 -0700 From: Jos Backus To: freebsd-hackers@freebsd.org Message-ID: <20070511165612.GA48097@lizzy.catnook.local> Mail-Followup-To: freebsd-hackers@freebsd.org References: <200705102105.27271.blackdragon@highveldmail.co.za> <4643C7DB.6000408@elischer.org> <17988.35412.231093.411177@bhuda.mired.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <17988.35412.231093.411177@bhuda.mired.org> User-Agent: Mutt/1.5.15 (2007-04-06) Subject: Re: SQL in the base system (Was: New FreeBSD package system (a.k.a. Daemon Package System (dps))) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: jos@catnook.com List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 May 2007 17:22:32 -0000 On Fri, May 11, 2007 at 11:23:00AM -0400, Mike Meyer wrote: [snip] > How robust is it - can a corrupt block fry the entire database? Dunno, but "Transactions are atomic, consistent, isolated, and durable (ACID) even after system crashes and power failures.". So it appears to try hard to minimize the chance of corruption. > How about portability - can I move the file to a completely > different architecture and still get the data from it? "Database files can be freely shared between machines with different byte orders." (Quotes taken from http://www.sqlite.org/) Also, the code is in the public domain. -- Jos Backus jos at catnook.com From owner-freebsd-hackers@FreeBSD.ORG Fri May 11 17:27:34 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 46A4816A402 for ; Fri, 11 May 2007 17:27:34 +0000 (UTC) (envelope-from mwm-keyword-freebsdhackers2.e313df@mired.org) Received: from mired.org (vpn.mired.org [66.92.153.74]) by mx1.freebsd.org (Postfix) with SMTP id D994313C459 for ; Fri, 11 May 2007 17:27:33 +0000 (UTC) (envelope-from mwm-keyword-freebsdhackers2.e313df@mired.org) Received: (qmail 41814 invoked by uid 1001); 11 May 2007 17:27:02 -0000 Received: by bhuda.mired.org (tmda-sendmail, from uid 1001); Fri, 11 May 2007 13:27:02 -0400 MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <17988.42853.894408.181184@bhuda.mired.org> Date: Fri, 11 May 2007 13:27:01 -0400 To: Ivan Voras In-Reply-To: References: <200705102105.27271.blackdragon@highveldmail.co.za> <4643C7DB.6000408@elischer.org> <17988.35412.231093.411177@bhuda.mired.org> <17988.40311.210855.381093@bhuda.mired.org> X-Mailer: VM 7.19 under Emacs 21.3.1 X-Primary-Address: mwm@mired.org X-face: "5Mnwy%?j>IIV\)A=):rjWL~NB2aH[}Yq8Z=u~vJ`"(,&SiLvbbz2W`; h9L,Yg`+vb1>RG% *h+%X^n0EZd>TM8_IB;a8F?(Fb"lw'IgCoyM.[Lg#r\ X-Delivery-Agent: TMDA/1.1.11 (Ladyburn) From: Mike Meyer Cc: freebsd-hackers@freebsd.org Subject: Re: SQL in the base system X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 May 2007 17:27:34 -0000 In , Ivan Voras typed: > Mike Meyer wrote: > > > Yes, they are present no matter what representation you use. The > > question is - how do the answers change if you change the > > format. These days, cross-platform means you deal with length as well > > as endian issues. Or maybe you don't, depending on the db. I know the > > answers for text files (easy, easy, very, yes). Can you propose a db > > scheme that gets has the same answers? > I think I don't understand the question. If the database contains number > "42" in a field typed "int32", in a row, and handles endianess well, why > would I get a different number on different platforms? The question is "How hard is it to move a db to a radically different platform and still use it?" Endianness is a biggie; world length (what does int32 turn into on a platform with 40 bit words and 20 bit half-words?) is less of a problem than it used to be; pointer sizes may be an issue, etc. Text files work on every Unix platform that uses the character set of your system. For the application at hand, working on any FreeBSD system would be equivalent. > (A side note about sqlite: it's actually weakly typed - you store and > receive strings). Which is part of why answering the questions needs a detailed proposal. > > I hate to tell you this, but your XML solution would still consist of > > a bunch of one-of file formats for each and every purpose. Using XML > > just fixes the syntax for the file, not the semantics. Settling on XML > > (or JSON, or INI, or cap files, or ...) is sort of like settling on > > UTF, only less obviously a win. Sure, you get to use canned code that > > will turn you text file into a structure in memory. But you still have > > to figure out what it all means. > > > > As you say, the XML toolset is the real win. Smart editors, > > validators, schemas (which make the editors and validators even more > > powerful) are all good things. Most people don't really seem > > interested in this beyond editors. That's not really much of a win. > > I agree that validation in XML is a strong point - but one of the reason > people like text files is that they DON'T usually have validation > features :) No, it's that the validation is usually simple enough that it can be done manually. Of course, for both XML and other text formats, people generally don't bother - they feed the data to the program, and let it blow up if the file is invalid. I suspect it's because the validation is generally a separate step. If we changed the config files to XML, and vi were configured to interactively underline in red everything that wasn't valid according to the schema, admins would almost certainly *love* it.(*) > | pro | contra > ---------------------------------------------------------------------- > XML | standard tools, validation, | evil manual parsing, bad rep > | can embed multiple data | > | structures in a standard way | bloated compared to alternatives, one document element per file. Since XML is text, you really want this row to be named someething like "ad-hoc". > ---------------------------------------------------------------------- > text | standard tools, sometimes | no validation, manual parsing, > | human readable | usually one data structure per > | | file Choosing a specific format - XML, JSON, S-expressions, cap, INI, etc. - just wires down the tokenizer (trivial) and structuring (varies from trivial to complex). If you're not taking advantage of anything beyond that that XML offers, there's not much point in picking XML over anything else - including ad-hoc formats. http://www.mired.org/consulting.html Independent Network/Unix/Perforce consultant, email for more information. From owner-freebsd-hackers@FreeBSD.ORG Fri May 11 17:34:39 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 400A216A403 for ; Fri, 11 May 2007 17:34:39 +0000 (UTC) (envelope-from mwm-keyword-freebsdhackers2.e313df@mired.org) Received: from mired.org (vpn.mired.org [66.92.153.74]) by mx1.freebsd.org (Postfix) with SMTP id D888B13C468 for ; Fri, 11 May 2007 17:34:38 +0000 (UTC) (envelope-from mwm-keyword-freebsdhackers2.e313df@mired.org) Received: (qmail 41897 invoked by uid 1001); 11 May 2007 17:34:07 -0000 Received: by bhuda.mired.org (tmda-sendmail, from uid 1001); Fri, 11 May 2007 13:34:07 -0400 MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <17988.43279.93248.608136@bhuda.mired.org> Date: Fri, 11 May 2007 13:34:07 -0400 To: jos@catnook.com In-Reply-To: <20070511165612.GA48097@lizzy.catnook.local> References: <200705102105.27271.blackdragon@highveldmail.co.za> <4643C7DB.6000408@elischer.org> <17988.35412.231093.411177@bhuda.mired.org> <20070511165612.GA48097@lizzy.catnook.local> X-Mailer: VM 7.19 under Emacs 21.3.1 X-Primary-Address: mwm@mired.org X-face: "5Mnwy%?j>IIV\)A=):rjWL~NB2aH[}Yq8Z=u~vJ`"(,&SiLvbbz2W`; h9L,Yg`+vb1>RG% *h+%X^n0EZd>TM8_IB;a8F?(Fb"lw'IgCoyM.[Lg#r\ X-Delivery-Agent: TMDA/1.1.11 (Ladyburn) From: Mike Meyer Cc: freebsd-hackers@freebsd.org Subject: Re: SQL in the base system (Was: New FreeBSD package system (a.k.a. Daemon Package System (dps))) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 May 2007 17:34:39 -0000 In <20070511165612.GA48097@lizzy.catnook.local>, Jos Backus typed: > On Fri, May 11, 2007 at 11:23:00AM -0400, Mike Meyer wrote: > [snip] > > How robust is it - can a corrupt block fry the entire database? > > Dunno, but "Transactions are atomic, consistent, isolated, and durable (ACID) > even after system crashes and power failures.". So it appears to try hard to > minimize the chance of corruption. Right. This is a good thing. However, the db *will* become corrupt. A disk block will fail to read, whatever. The question is asking how much data will be lost outside the corrupt data block? > > How about portability - can I move the file to a completely > > different architecture and still get the data from it? > "Database files can be freely shared between machines with different byte > orders." That sounds like a "somewhat". The desired answer is "If the version of sqllite runs on a platform, all database files will work on it." That they felt the need to point out that they are byte order independent implies that other architectural issues may be a problem. Of course, it could be that nobody has asked the right people that question. > Also, the code is in the public domain. Wow. That's everylicensecompliant. http://www.mired.org/consulting.html Independent Network/Unix/Perforce consultant, email for more information. From owner-freebsd-hackers@FreeBSD.ORG Fri May 11 17:37:50 2007 Return-Path: X-Original-To: hackers@freebsd.org Delivered-To: freebsd-hackers@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 781FE16A403 for ; Fri, 11 May 2007 17:37:50 +0000 (UTC) (envelope-from fcash@ocis.net) Received: from smtp.sd73.bc.ca (smtp.sd73.bc.ca [142.24.13.140]) by mx1.freebsd.org (Postfix) with ESMTP id 5877813C4AD for ; Fri, 11 May 2007 17:37:50 +0000 (UTC) (envelope-from fcash@ocis.net) Received: from localhost (localhost [127.0.0.1]) by localhost.sd73.bc.ca (Postfix) with ESMTP id 2EA5E1A000B0D for ; Fri, 11 May 2007 10:11:55 -0700 (PDT) X-Virus-Scanned: Debian amavisd-new at smtp.sd73.bc.ca Received: from smtp.sd73.bc.ca ([127.0.0.1]) by localhost (smtp.sd73.bc.ca [127.0.0.1]) (amavisd-new, port 10024) with LMTP id 4TXA4WoKPRxl for ; Fri, 11 May 2007 10:11:53 -0700 (PDT) Received: from coal (s10.sbo [192.168.0.10]) by smtp.sd73.bc.ca (Postfix) with ESMTP id 927AE1A000B0B for ; Fri, 11 May 2007 10:11:53 -0700 (PDT) From: Freddie Cash To: hackers@freebsd.org Date: Fri, 11 May 2007 10:11:51 -0700 User-Agent: KMail/1.9.5 References: <200705102105.27271.blackdragon@highveldmail.co.za> <20070511051852.GA89359@xor.obsecurity.org> <17988.32573.910854.388638@bhuda.mired.org> In-Reply-To: <17988.32573.910854.388638@bhuda.mired.org> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200705111011.52212.fcash@ocis.net> Cc: Subject: Re: New FreeBSD package system (a.k.a. Daemon Package System (dps)) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 May 2007 17:37:50 -0000 On Friday 11 May 2007 07:35 am, Mike Meyer wrote: > I still think we ought to quit pretending that ports/packages aren't > part of BSD, and default LOCALBASE to /usr. But if changing it is > being tested, that's a big help. Personally, this is the one thing I like *most* about BSD. There is a clear separation between what ships as part of the OS, and what apps I install on it later. There's a consistency to things, that you just can't find anywhere else. / and /usr are the OS. /usr/local is what the ports tree installs. /whatever/i/want/ is where I install things from source to keep them separate. One could make the case for /usr to be the OS, /usr/pkg (or whatever) for port installs, and /usr/local for local source installs. So long as the OS is separate from the apps. With the OS and apps separate, you can upgrade them asynchronously. There's a nice feeling to running the latest version of appX on FreeBSD 5.3. Or an older version of appY on FreeBSD 6-STABLE. Try getting something similar on a Linux system. -- Freddie Cash fcash@ocis.net From owner-freebsd-hackers@FreeBSD.ORG Fri May 11 17:23:50 2007 Return-Path: X-Original-To: hackers@FreeBSD.org Delivered-To: freebsd-hackers@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id AD7C916A403; Fri, 11 May 2007 17:23:50 +0000 (UTC) (envelope-from brian@Awfulhak.org) Received: from storm.uk.FreeBSD.org (storm.uk.FreeBSD.org [194.242.157.42]) by mx1.freebsd.org (Postfix) with ESMTP id 708FE13C447; Fri, 11 May 2007 17:23:49 +0000 (UTC) (envelope-from brian@Awfulhak.org) Received: from store.lan.Awfulhak.org (store.lan.Awfulhak.org [172.16.0.35]) by storm.uk.FreeBSD.org (8.14.1/8.14.1) with ESMTP id l4BHMYq1042799; Fri, 11 May 2007 18:22:38 +0100 (BST) (envelope-from brian@Awfulhak.org) Received: from store.lan.Awfulhak.org (localhost.localdomain [127.0.0.1]) by localhost (Postfix) with SMTP id BAC261957C73; Fri, 11 May 2007 17:22:36 +0000 (GMT) Received: from gw.Awfulhak.org (gw.lan.Awfulhak.org [172.16.0.1]) by store.lan.Awfulhak.org (Postfix) with ESMTP id A1CDE1957C69; Fri, 11 May 2007 17:22:30 +0000 (GMT) Received: from conflict.ca.sophos.com (brian@localhost [127.0.0.1]) by gw.Awfulhak.org (8.14.1/8.14.1) with ESMTP id l4BHMECA049313; Fri, 11 May 2007 10:22:15 -0700 (PDT) (envelope-from brian@Awfulhak.org) Date: Fri, 11 May 2007 10:22:14 -0700 From: Brian Somers To: Robert Watson Message-ID: <20070511102214.44e42bf2@conflict.ca.sophos.com> In-Reply-To: <20070511171345.M86304@fledge.watson.org> References: <20070510142147.GA83797@shark.localdomain> <20070511171345.M86304@fledge.watson.org> X-Mailer: Claws Mail 2.9.1 (GTK+ 2.10.12; i386-portbld-freebsd7.0) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Spam-Status: No, score=-1.4 required=5.0 tests=ALL_TRUSTED,SPF_HELO_PASS, SPF_PASS autolearn=failed version=3.1.8 X-Spam-Checker-Version: SpamAssassin 3.1.8 (2007-02-13) on gw.lan.Awfulhak.org X-Mailman-Approved-At: Fri, 11 May 2007 18:42:59 +0000 Cc: hackers@FreeBSD.org, brian@FreeBSD.org, Sergey Zaharchenko Subject: Re: My PPP timer PR [nag] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 May 2007 17:23:50 -0000 On Fri, 11 May 2007 17:14:16 +0100 (BST) Robert Watson wrote: > > On Thu, 10 May 2007, Sergey Zaharchenko wrote: > > > bin/102747 has been sitting there for about 8 months, with no activity since > > it was assigned to brian@, all my mail to whom bounces [CC'd just in case]. > > > > The patch attached in the PR has been working for me since, so it not being > > fixed in the main tree isn't a problem with me. I just thought someone would > > benefit from it being committed... > > In general, I would suggest sending this sort of thing to current@ and/or > net@, but not hackers@. I'll have a look at it RSN - or at BSDCan next week at the latest. -- Brian Somers Don't _EVER_ lose your sense of humour ! From owner-freebsd-hackers@FreeBSD.ORG Fri May 11 18:43:01 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 79C1A16A419 for ; Fri, 11 May 2007 18:43:01 +0000 (UTC) (envelope-from kris@obsecurity.org) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.freebsd.org (Postfix) with ESMTP id 5B63513C43E for ; Fri, 11 May 2007 18:43:01 +0000 (UTC) (envelope-from kris@obsecurity.org) Received: from obsecurity.dyndns.org (elvis.mu.org [192.203.228.196]) by elvis.mu.org (Postfix) with ESMTP id CC7C11A3C19; Fri, 11 May 2007 11:43:46 -0700 (PDT) Received: by obsecurity.dyndns.org (Postfix, from userid 1000) id D7C7051F44; Fri, 11 May 2007 14:42:59 -0400 (EDT) Date: Fri, 11 May 2007 14:42:59 -0400 From: Kris Kennaway To: Mike Meyer Message-ID: <20070511184259.GA23483@xor.obsecurity.org> References: <200705102105.27271.blackdragon@highveldmail.co.za> <17987.52037.112351.872442@bhuda.mired.org> <20070511015156.GA77895@xor.obsecurity.org> <17987.52970.398402.580727@bhuda.mired.org> <20070511021249.GA78729@xor.obsecurity.org> <17987.57305.7130.873114@bhuda.mired.org> <20070511051852.GA89359@xor.obsecurity.org> <17988.32573.910854.388638@bhuda.mired.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <17988.32573.910854.388638@bhuda.mired.org> User-Agent: Mutt/1.4.2.2i Cc: freebsd-hackers@freebsd.org, Ivan Voras , Kris Kennaway Subject: Re: New FreeBSD package system (a.k.a. Daemon Package System (dps)) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 May 2007 18:43:01 -0000 On Fri, May 11, 2007 at 10:35:41AM -0400, Mike Meyer wrote: > In <20070511051852.GA89359@xor.obsecurity.org>, Kris Kennaway typed: > > There are a few ways you can go. The simplest is to install a > > complete i386 world in e.g. /compat/ia32 and have i386 packages > > installed there, and change the kernel to do a "magic directory > > lookup" for i386 binaries that does path munging like the linuxulator > > does with /compat/linux. > > Actually, the simplest is to install a virtual machine and just use > that. But none of these is quite ideal. > > > If you want the i386 packages to live in /usr/local alongside the > > amd64 packages, you'll need to do something like adding an @arch > > directive to the +CONTENTS file recorded in /var/db/pkg, and add some > > checking to pkg_add to ensure that when you install a package, > > everything it depends on was built for the same architecture. > > Which is pretty much what I pointed out in my original request. Except > you don't really need everything it depends on to be built for the > same architecture. If your port needs gmake to build, or uses > ghostscript to do ps->jpeg conversion or some such, it won't really > care what architecture the executables were built for - just that they > run. It would help if requirements were flagged with whether or not > they required the same architecture. You have to be more careful than this, because a lot of ports have architecture-dependent on-disk formats for their data files (obviously not the jpeg example, but there are lots of other cases that use a non-portable storage format). In practise avoiding all these conflicts will be very restrictive: as soon as you have installed a moderate number of amd64 packages, you lose the ability to install most i386 packages because of these header/library/data file conflicts (e.g. you can't install an i386 gtk app if you are running a gnome desktop). Which means you're back to installing the i386 packages in a separate prefix, i.e. option 1. > > You'd need to also add these checks to bsd.port.mk so it exits > > gracefully when someone tries to natively compile a port for which > > non-native dependencies are installed (e.g. it's going to be really > > unhappy if it finds i386 headers or libraries). This method might be > > more trouble than it's worth. > > Again, it might or might not be unhappy. But now you're venturing into > ports as opposed to the package system. I have a whole other list of > things to consider there. The point is that the real problem is: "how do you arrange the bits on disk", not "how do you wrap that in a package system". Until you figure out a workable on-disk arrangement for the files, questions about packaging are not relevant. And it seems that option 1 is the only workable one in practise (unless you have some other idea), which can easily be achieved today with a couple of hours of kernel hacking. Kris From owner-freebsd-hackers@FreeBSD.ORG Fri May 11 19:01:46 2007 Return-Path: X-Original-To: hackers@freebsd.org Delivered-To: freebsd-hackers@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id AC23B16A40A for ; Fri, 11 May 2007 19:01:46 +0000 (UTC) (envelope-from mwm-keyword-freebsdhackers2.e313df@mired.org) Received: from mired.org (vpn.mired.org [66.92.153.74]) by mx1.freebsd.org (Postfix) with SMTP id 4193313C45E for ; Fri, 11 May 2007 19:01:46 +0000 (UTC) (envelope-from mwm-keyword-freebsdhackers2.e313df@mired.org) Received: (qmail 42447 invoked by uid 1001); 11 May 2007 18:34:33 -0000 Received: by bhuda.mired.org (tmda-sendmail, from uid 1001); Fri, 11 May 2007 14:34:33 -0400 MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <17988.46905.9205.461424@bhuda.mired.org> Date: Fri, 11 May 2007 14:34:33 -0400 To: Freddie Cash In-Reply-To: <200705111011.52212.fcash@ocis.net> References: <200705102105.27271.blackdragon@highveldmail.co.za> <20070511051852.GA89359@xor.obsecurity.org> <17988.32573.910854.388638@bhuda.mired.org> <200705111011.52212.fcash@ocis.net> X-Mailer: VM 7.19 under Emacs 21.3.1 X-Primary-Address: mwm@mired.org X-face: "5Mnwy%?j>IIV\)A=):rjWL~NB2aH[}Yq8Z=u~vJ`"(,&SiLvbbz2W`; h9L,Yg`+vb1>RG% *h+%X^n0EZd>TM8_IB;a8F?(Fb"lw'IgCoyM.[Lg#r\ X-Delivery-Agent: TMDA/1.1.11 (Ladyburn) From: Mike Meyer Cc: hackers@freebsd.org Subject: Re: New FreeBSD package system (a.k.a. Daemon Package System (dps)) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 May 2007 19:01:46 -0000 In <200705111011.52212.fcash@ocis.net>, Freddie Cash typed: > On Friday 11 May 2007 07:35 am, Mike Meyer wrote: > > I still think we ought to quit pretending that ports/packages aren't > > part of BSD, and default LOCALBASE to /usr. But if changing it is > > being tested, that's a big help. > Personally, this is the one thing I like *most* about BSD. There is a > clear separation between what ships as part of the OS, and what apps I > install on it later. There's a consistency to things, that you just > can't find anywhere else. > > / and /usr are the OS. > > /usr/local is what the ports tree installs. Moving the OS into the package system has been on the "todo" list for a long time (assuming it's still there - there are people opposed to that). What happens to your distinction in that case? This is why I think the distinction is an illusion. > One could make the case for /usr to be the OS, /usr/pkg (or whatever) for > port installs, and /usr/local for local source installs. So long as the > OS is separate from the apps. I think that would be an improvement. There's a real distinction between "things installed from ports/packages" and "things I built and installed myself". The former I expect to reinstall from FreeBSD media; the latter I can't. Since most packages in the latter install in /usr/local, using that for the former makes life a bit more painful if you want to keep them separate. The downside is that making the default something else makes things a bit harder for people doing ports, which we promptly throw away by providing a settable LOCALBASE. > With the OS and apps separate, you can upgrade them asynchronously. > There's a nice feeling to running the latest version of appX on FreeBSD > 5.3. Or an older version of appY on FreeBSD 6-STABLE. How would setting LOCALBASE=/usr break this? Of course, equally valid is the question "what will break if I set LOCALBASE=/usr"? Hmm. I think I may found out.... Personally, what I like about FreeBSD is that it provides flexibility for things like this. If I'd rather have ports out of /usr/local, it's not really hard to do. Not as easy as doing things the default way, but not any worse than anyone else who wants to build from source. Other systems seem intent on making me do things their way, no matter what I may think of it, and trying to change something like this is a major change. http://www.mired.org/consulting.html Independent Network/Unix/Perforce consultant, email for more information. From owner-freebsd-hackers@FreeBSD.ORG Fri May 11 19:07:59 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id CFC8C16A405 for ; Fri, 11 May 2007 19:07:59 +0000 (UTC) (envelope-from mwm-keyword-freebsdhackers2.e313df@mired.org) Received: from mired.org (vpn.mired.org [66.92.153.74]) by mx1.freebsd.org (Postfix) with SMTP id 6BCC913C45D for ; Fri, 11 May 2007 19:07:59 +0000 (UTC) (envelope-from mwm-keyword-freebsdhackers2.e313df@mired.org) Received: (qmail 42883 invoked by uid 1001); 11 May 2007 19:07:28 -0000 Received: by bhuda.mired.org (tmda-sendmail, from uid 1001); Fri, 11 May 2007 15:07:27 -0400 MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <17988.48879.828384.170557@bhuda.mired.org> Date: Fri, 11 May 2007 15:07:27 -0400 To: Kris Kennaway In-Reply-To: <20070511184259.GA23483@xor.obsecurity.org> References: <200705102105.27271.blackdragon@highveldmail.co.za> <17987.52037.112351.872442@bhuda.mired.org> <20070511015156.GA77895@xor.obsecurity.org> <17987.52970.398402.580727@bhuda.mired.org> <20070511021249.GA78729@xor.obsecurity.org> <17987.57305.7130.873114@bhuda.mired.org> <20070511051852.GA89359@xor.obsecurity.org> <17988.32573.910854.388638@bhuda.mired.org> <20070511184259.GA23483@xor.obsecurity.org> X-Mailer: VM 7.19 under Emacs 21.3.1 X-Primary-Address: mwm@mired.org X-face: "5Mnwy%?j>IIV\)A=):rjWL~NB2aH[}Yq8Z=u~vJ`"(,&SiLvbbz2W`; h9L,Yg`+vb1>RG% *h+%X^n0EZd>TM8_IB;a8F?(Fb"lw'IgCoyM.[Lg#r\ X-Delivery-Agent: TMDA/1.1.11 (Ladyburn) From: Mike Meyer Cc: freebsd-hackers@freebsd.org, Ivan Voras Subject: Re: New FreeBSD package system (a.k.a. Daemon Package System (dps)) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 May 2007 19:07:59 -0000 In <20070511184259.GA23483@xor.obsecurity.org>, Kris Kennaway typed: > The point is that the real problem is: "how do you arrange the bits on > disk", not "how do you wrap that in a package system". Until you > figure out a workable on-disk arrangement for the files, questions > about packaging are not relevant. And it seems that option 1 is the > only workable one in practise (unless you have some other idea), which > can easily be achieved today with a couple of hours of kernel hacking. There are clearly other workable ideas - as I said, the linux folks managed to make it work. But it's not an easy problem. I certainly wouldn't suggest rebuilding the packaging system to deal with this, except as part of a larger effort. On the other hand, since people are working on the ports/package system (I see port/pkg database and some ports infrastructure work in the current SoC projects list), not keeping this goal in mind would seem to be a bit short-sighted. I wouldn't be surprised if your option #1 could benefit from this as well. http://www.mired.org/consulting.html Independent Network/Unix/Perforce consultant, email for more information. From owner-freebsd-hackers@FreeBSD.ORG Fri May 11 19:10:59 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 1254716A406 for ; Fri, 11 May 2007 19:10:59 +0000 (UTC) (envelope-from kris@obsecurity.org) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.freebsd.org (Postfix) with ESMTP id EF4C713C46A for ; Fri, 11 May 2007 19:10:58 +0000 (UTC) (envelope-from kris@obsecurity.org) Received: from obsecurity.dyndns.org (elvis.mu.org [192.203.228.196]) by elvis.mu.org (Postfix) with ESMTP id 754971A3C19; Fri, 11 May 2007 12:11:44 -0700 (PDT) Received: by obsecurity.dyndns.org (Postfix, from userid 1000) id 0601E52CD2; Fri, 11 May 2007 15:10:57 -0400 (EDT) Date: Fri, 11 May 2007 15:10:57 -0400 From: Kris Kennaway To: Mike Meyer Message-ID: <20070511191057.GA25833@xor.obsecurity.org> References: <17987.52037.112351.872442@bhuda.mired.org> <20070511015156.GA77895@xor.obsecurity.org> <17987.52970.398402.580727@bhuda.mired.org> <20070511021249.GA78729@xor.obsecurity.org> <17987.57305.7130.873114@bhuda.mired.org> <20070511051852.GA89359@xor.obsecurity.org> <17988.32573.910854.388638@bhuda.mired.org> <20070511184259.GA23483@xor.obsecurity.org> <17988.48879.828384.170557@bhuda.mired.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <17988.48879.828384.170557@bhuda.mired.org> User-Agent: Mutt/1.4.2.2i Cc: freebsd-hackers@freebsd.org, Ivan Voras , Kris Kennaway Subject: Re: New FreeBSD package system (a.k.a. Daemon Package System (dps)) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 May 2007 19:10:59 -0000 On Fri, May 11, 2007 at 03:07:27PM -0400, Mike Meyer wrote: > In <20070511184259.GA23483@xor.obsecurity.org>, Kris Kennaway typed: > > The point is that the real problem is: "how do you arrange the bits on > > disk", not "how do you wrap that in a package system". Until you > > figure out a workable on-disk arrangement for the files, questions > > about packaging are not relevant. And it seems that option 1 is the > > only workable one in practise (unless you have some other idea), which > > can easily be achieved today with a couple of hours of kernel hacking. > > There are clearly other workable ideas - as I said, the linux folks > managed to make it work. But it's not an easy problem. I certainly > wouldn't suggest rebuilding the packaging system to deal with this, > except as part of a larger effort. On the other hand, since people are > working on the ports/package system (I see port/pkg database and some > ports infrastructure work in the current SoC projects list), not > keeping this goal in mind would seem to be a bit short-sighted. I > wouldn't be surprised if your option #1 could benefit from this as > well. But you said you were interested in working on it...so what is your idea? Kris From owner-freebsd-hackers@FreeBSD.ORG Fri May 11 19:15:50 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id E531616A404 for ; Fri, 11 May 2007 19:15:50 +0000 (UTC) (envelope-from jos@catnook.com) Received: from lizzy.dyndns.org (209-204-188-132.dsl.static.sonic.net [209.204.188.132]) by mx1.freebsd.org (Postfix) with SMTP id 9868313C459 for ; Fri, 11 May 2007 19:15:50 +0000 (UTC) (envelope-from jos@catnook.com) Received: (qmail 50448 invoked by uid 1000); 11 May 2007 19:16:12 -0000 Date: Fri, 11 May 2007 12:16:12 -0700 From: Jos Backus To: freebsd-hackers@freebsd.org Message-ID: <20070511191612.GA50329@lizzy.catnook.local> Mail-Followup-To: freebsd-hackers@freebsd.org References: <200705102105.27271.blackdragon@highveldmail.co.za> <4643C7DB.6000408@elischer.org> <17988.35412.231093.411177@bhuda.mired.org> <20070511165612.GA48097@lizzy.catnook.local> <17988.43279.93248.608136@bhuda.mired.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <17988.43279.93248.608136@bhuda.mired.org> User-Agent: Mutt/1.5.15 (2007-04-06) Subject: Re: SQL in the base system (Was: New FreeBSD package system (a.k.a. Daemon Package System (dps))) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: jos@catnook.com List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 May 2007 19:15:51 -0000 On Fri, May 11, 2007 at 01:34:07PM -0400, Mike Meyer wrote: > In <20070511165612.GA48097@lizzy.catnook.local>, Jos Backus typed: > > On Fri, May 11, 2007 at 11:23:00AM -0400, Mike Meyer wrote: > > [snip] > > > How robust is it - can a corrupt block fry the entire database? > > > > Dunno, but "Transactions are atomic, consistent, isolated, and durable (ACID) > > even after system crashes and power failures.". So it appears to try hard to > > minimize the chance of corruption. > > Right. This is a good thing. However, the db *will* become corrupt. A > disk block will fail to read, whatever. The question is asking how > much data will be lost outside the corrupt data block? It's a trade-off. Use SMART to monitor your disks, or whatever. Dealing with a gazillion different file formats is a PITA from an automation perspective. > > > How about portability - can I move the file to a completely > > > different architecture and still get the data from it? > > "Database files can be freely shared between machines with different byte > > orders." > > That sounds like a "somewhat". The desired answer is "If the version > of sqllite runs on a platform, all database files will work on it." > That they felt the need to point out that they are byte order > independent implies that other architectural issues may be a > problem. Of course, it could be that nobody has asked the right people > that question. Probably. Why don't you? :-) > > Also, the code is in the public domain. > > Wow. That's everylicensecompliant. I sense some sarcasm. Would the GPL have been an improvement in the context of this discussion? -- Jos Backus jos at catnook.com From owner-freebsd-hackers@FreeBSD.ORG Fri May 11 19:22:10 2007 Return-Path: X-Original-To: hackers@freebsd.org Delivered-To: freebsd-hackers@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 6F6D916A403 for ; Fri, 11 May 2007 19:22:10 +0000 (UTC) (envelope-from fcash@ocis.net) Received: from smtp.sd73.bc.ca (smtp.sd73.bc.ca [142.24.13.140]) by mx1.freebsd.org (Postfix) with ESMTP id 4612F13C45A for ; Fri, 11 May 2007 19:22:10 +0000 (UTC) (envelope-from fcash@ocis.net) Received: from localhost (localhost [127.0.0.1]) by localhost.sd73.bc.ca (Postfix) with ESMTP id DD6B41A000B0B for ; Fri, 11 May 2007 12:21:59 -0700 (PDT) X-Virus-Scanned: Debian amavisd-new at smtp.sd73.bc.ca Received: from smtp.sd73.bc.ca ([127.0.0.1]) by localhost (smtp.sd73.bc.ca [127.0.0.1]) (amavisd-new, port 10024) with LMTP id jHCGrAzLmLCu for ; Fri, 11 May 2007 12:21:58 -0700 (PDT) Received: from coal (s10.sbo [192.168.0.10]) by smtp.sd73.bc.ca (Postfix) with ESMTP id 5CE811A000B0C for ; Fri, 11 May 2007 12:21:58 -0700 (PDT) From: Freddie Cash To: hackers@freebsd.org Date: Fri, 11 May 2007 12:21:56 -0700 User-Agent: KMail/1.9.5 References: <200705102105.27271.blackdragon@highveldmail.co.za> <200705111011.52212.fcash@ocis.net> <17988.46905.9205.461424@bhuda.mired.org> In-Reply-To: <17988.46905.9205.461424@bhuda.mired.org> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200705111221.56899.fcash@ocis.net> Cc: Subject: Re: New FreeBSD package system (a.k.a. Daemon Package System (dps)) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 May 2007 19:22:10 -0000 On Friday 11 May 2007 11:34 am, Mike Meyer wrote: > In <200705111011.52212.fcash@ocis.net>, Freddie Cash typed: > > On Friday 11 May 2007 07:35 am, Mike Meyer wrote: > > > I still think we ought to quit pretending that ports/packages > > > aren't part of BSD, and default LOCALBASE to /usr. But if changing > > > it is being tested, that's a big help. > > > > Personally, this is the one thing I like *most* about BSD. There is > > a clear separation between what ships as part of the OS, and what > > apps I install on it later. There's a consistency to things, that > > you just can't find anywhere else. > > > > / and /usr are the OS. > > > > /usr/local is what the ports tree installs. > > Moving the OS into the package system has been on the "todo" list for > a long time (assuming it's still there - there are people opposed to > that). What happens to your distinction in that case? This is why I > think the distinction is an illusion. Hmmm, I might be misremembering things, but I always thought the plan was to break out some bits of the base OS into ports/packages (like sendmail/bind/etc) but have them installed-by-default. Either way, you are correct that the distinction between "base OS" and "installed apps" would blur or even disappear. > > One could make the case for /usr to be the OS, /usr/pkg (or whatever) > > for port installs, and /usr/local for local source installs. So long > > as the OS is separate from the apps. > > I think that would be an improvement. There's a real distinction > between "things installed from ports/packages" and "things I built and > installed myself". The former I expect to reinstall from FreeBSD > media; the latter I can't. Since most packages in the latter install > in /usr/local, using that for the former makes life a bit more painful > if you want to keep them separate. The downside is that making the > default something else makes things a bit harder for people doing > ports, which we promptly throw away by providing a settable LOCALBASE. > > > With the OS and apps separate, you can upgrade them asynchronously. > > There's a nice feeling to running the latest version of appX on > > FreeBSD 5.3. Or an older version of appY on FreeBSD 6-STABLE. > > How would setting LOCALBASE=/usr break this? Of course, equally valid > is the question "what will break if I set LOCALBASE=/usr"? Hmm. I > think I may found out.... I was thinking more along the lines of how Linux distros operate, where all apps are installed under /usr, and where the only way to get the latest version of an app was to upgrade the entire OS and switch package repositories to the ones for that new version of the OS. Something that irks me to no end when using Linux systems. Not quite the same as what you were getting at, I see, so we can just ignore this entire little bit of the thread. :) > Personally, what I like about FreeBSD is that it provides flexibility > for things like this. If I'd rather have ports out of /usr/local, it's > not really hard to do. Not as easy as doing things the default way, > but not any worse than anyone else who wants to build from > source. Other systems seem intent on making me do things their way, no > matter what I may think of it, and trying to change something like > this is a major change. Yes, flexibility and customisability is very nice, and definitely one of FreeBSD's strength. -- Freddie Cash fcash@ocis.net From owner-freebsd-hackers@FreeBSD.ORG Fri May 11 19:43:48 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 1E95B16A50C for ; Fri, 11 May 2007 19:43:48 +0000 (UTC) (envelope-from peterjeremy@optushome.com.au) Received: from turion.vk2pj.dyndns.org (c220-239-3-125.belrs4.nsw.optusnet.com.au [220.239.3.125]) by mx1.freebsd.org (Postfix) with ESMTP id 5ECE213C53D for ; Fri, 11 May 2007 19:43:33 +0000 (UTC) (envelope-from peterjeremy@optushome.com.au) Received: from turion.vk2pj.dyndns.org (localhost.vk2pj.dyndns.org [127.0.0.1]) by turion.vk2pj.dyndns.org (8.14.1/8.14.1) with ESMTP id l4BJhUYt010044 for ; Sat, 12 May 2007 05:43:30 +1000 (EST) (envelope-from peter@turion.vk2pj.dyndns.org) Received: (from peter@localhost) by turion.vk2pj.dyndns.org (8.14.1/8.14.1/Submit) id l4BJhUw0010043 for freebsd-hackers@freebsd.org; Sat, 12 May 2007 05:43:30 +1000 (EST) (envelope-from peter) Date: Sat, 12 May 2007 05:43:30 +1000 From: Peter Jeremy To: freebsd-hackers@freebsd.org Message-ID: <20070511194330.GL826@turion.vk2pj.dyndns.org> References: <200705102105.27271.blackdragon@highveldmail.co.za> <4644847A.5060702@freebsd.org> <20070511153448.GA7516@britannica.bec.de> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="eqp4TxRxnD4KrmFZ" Content-Disposition: inline In-Reply-To: <20070511153448.GA7516@britannica.bec.de> X-PGP-Key: http://members.optusnet.com.au/peterjeremy/pubkey.asc User-Agent: Mutt/1.5.14 (2007-02-12) Subject: Re: New FreeBSD package system (a.k.a. Daemon Package System (dps)) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 May 2007 19:43:48 -0000 --eqp4TxRxnD4KrmFZ Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On 2007-May-11 17:34:48 +0200, Joerg Sonnenberger = wrote: >On Fri, May 11, 2007 at 07:58:02AM -0700, Tim Kientzle wrote: >> 3) As DES pointed out, the package tools must be able >> to read the metadata before the files. If you really >> need a completely separate metadata file, make it >> the second file in the archive. > >Actually, the argument is pretty weak. Being able to extract them >streamable and access the meta-data easily is fine. The remote access >argument is very weak as it doesn't allow e.g. signature checks. Possible options for signature checks on a stream: 1) Have one of the early metadata files be a signed set of hashes for all the expected files. Validate that file and if it's OK use the hashes to validate the other files as they are unloaded. You can check at the end that you have everything you were expecting. 2) Make pkg_add itself transactional (close to ACID): Unload the files into the final locations with temporary names and mode 000. At the end of stream, check the signature. If it's OK, just rename all the files and fix the modes, otherwise delete them all. Rename/chmod is much cheaper than copying the files (and something tar does anyway - you are just delaying it). This has the added benefit that nothing else can use the package until it's completely unpacked. --=20 Peter Jeremy --eqp4TxRxnD4KrmFZ Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (FreeBSD) iD8DBQFGRMdi/opHv/APuIcRAvJkAKClEK8OeIXCCnAC9dOkE88h27yu8QCeJZYi JX/uZC8JZ7Nbljt0LYTHvCk= =Bmlo -----END PGP SIGNATURE----- --eqp4TxRxnD4KrmFZ-- From owner-freebsd-hackers@FreeBSD.ORG Fri May 11 21:03:42 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 0D02516A403 for ; Fri, 11 May 2007 21:03:42 +0000 (UTC) (envelope-from naylor.b.david@gmail.com) Received: from wx-out-0506.google.com (wx-out-0506.google.com [66.249.82.232]) by mx1.freebsd.org (Postfix) with ESMTP id A288C13C447 for ; Fri, 11 May 2007 21:03:41 +0000 (UTC) (envelope-from naylor.b.david@gmail.com) Received: by wx-out-0506.google.com with SMTP id s18so995040wxc for ; Fri, 11 May 2007 14:03:41 -0700 (PDT) DKIM-Signature: a=rsa-sha1; c=relaxed/relaxed; d=gmail.com; s=beta; h=domainkey-signature:received:received:from:to:subject:date:user-agent:mime-version:content-type:content-transfer-encoding:message-id:sender; b=DMtl8CxJQy1PNI5UPw6rvFkTzHg19AtkfSilAWKGhjgdzEmMa+MNPuqBSHRSLIorQgEhbpH/JxvlSwQuPViPzGxEakwAND24EMNYNFGY0iJPGWxTud22XTisevq6g2DtxazhqcHvA+ql5q7FpUNtMMeV3cB1LZubJDsgPXVUl+M= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=beta; h=received:from:to:subject:date:user-agent:mime-version:content-type:content-transfer-encoding:message-id:sender; b=j9q3nLG5qCH34ft4VRy9vIz1pYFQV+5UKJSsBBFbxIAmnT77tPA+NqmIAPBGofc3GI315FeYVSOYdU7eAzaPY3ntThif+vloh+nxRTLFV0dIRKGaWlKynhB8/Q9A1tD5RzxGNwyWYgTQ91BLJVhCTHsTWM8oOhB3UBsg32Gmg0Y= Received: by 10.90.118.12 with SMTP id q12mr3614030agc.1178917421126; Fri, 11 May 2007 14:03:41 -0700 (PDT) Received: from ?0.0.0.0? ( [196.37.90.163]) by mx.google.com with ESMTP id 44sm14801482wri.2007.05.11.14.03.34; Fri, 11 May 2007 14:03:37 -0700 (PDT) From: David Naylor To: freebsd-hackers@freebsd.org Date: Fri, 11 May 2007 23:02:31 +0200 User-Agent: KMail/1.9.5 MIME-Version: 1.0 Content-Type: multipart/signed; boundary="nextPart1701573.s2DFBrc6Nc"; protocol="application/pgp-signature"; micalg=pgp-sha1 Content-Transfer-Encoding: 7bit Message-Id: <200705112302.35380.dragonsa@highveldmail.co.za> Sender: David Naylor X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Subject: DPS Initial Ideas X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 May 2007 21:03:42 -0000 --nextPart1701573.s2DFBrc6Nc Content-Type: multipart/mixed; boundary="Boundary-01=_nnNRGI9yUkC+3An" Content-Transfer-Encoding: 7bit Content-Disposition: inline --Boundary-01=_nnNRGI9yUkC+3An Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Content-Disposition: inline Hi, Thank you all for your responses, it has given me much to think about. I=20 guess there is consenses that there is room for improvement in the current= =20 pkg system. Attached are some of my initial ideas about what is required a= nd=20 expected in any (and all future) package systems. =20 Since I am going away this weekend I have had limited time to work on this= =20 document and as such it is in early stages of development. My objective is= =20 to get a clear understanding and target of what is required for this new=20 package system. =20 I am looking at a hybrid approach to storing the package metadata, a=20 combination of SQLite and compressed text files. I am hoping to create a=20 situation where if either gets corrupted it can be created from the other. = =20 =46urther more, any changes to the text files (even without recompressing t= hem)=20 will be intergrated back into the database. This will allow administrators= =20 to fiddle around with the text files without having to touch the database. = =20 (( Another idea I have is the ability for the whole package data storage to be= =20 recreated from the repository data if all stored data got "destroyed". Thi= s=20 will occur though a process of inspection of the file system. This could b= e=20 extended to allow all none package files to be combined into a "special"=20 package with the correct dependencies allowing a system to be restored=20 without a single error... Any thoughts? )) I would also like to create package groupings, where by many packages share= =20 the same package file. This will allow easy distribution of ports such as= =20 the Xorg 7.2, reducing the 400 or so packages to only a few physical files= =20 that can then be installed and managed individually. =20 Ultimately I would like this project to be compatible with the current pkg= =20 system (allowing easy transition from one to another), proper integration=20 into the ports system and possibly into the pkgsrc system, but the future=20 will only be revealed in time. =20 All feedback is welcome, I do have some questions to ask: 1) What would random access of a package be required for and how often=20 will/does it occur?=20 2) I have chosen XML for storing the data. Is there any practical alternat= ive=20 (Please keep in mind that multiple packages metadata could be stored in a=20 single file) I apologies if the document is too dense any too cryptic, all ideas are=20 welcome. =20 Once again thank you all for your feedback and have a good weekend. =20 David --Boundary-01=_nnNRGI9yUkC+3An-- --nextPart1701573.s2DFBrc6Nc Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.5 (GNU/Linux) iD8DBQBGRNnryqzxLKpyZI8RAkGLAJ4naFsv5FPVpeyLsGa3XfLKu3yT/ACfQfs2 d9w/Fd8sxNbh5XSMAtAlhUM= =6X17 -----END PGP SIGNATURE----- --nextPart1701573.s2DFBrc6Nc-- From owner-freebsd-hackers@FreeBSD.ORG Fri May 11 21:28:49 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 4529E16A402 for ; Fri, 11 May 2007 21:28:49 +0000 (UTC) (envelope-from kris@obsecurity.org) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.freebsd.org (Postfix) with ESMTP id 3222513C458 for ; Fri, 11 May 2007 21:28:49 +0000 (UTC) (envelope-from kris@obsecurity.org) Received: from obsecurity.dyndns.org (elvis.mu.org [192.203.228.196]) by elvis.mu.org (Postfix) with ESMTP id C919B1A3C19; Fri, 11 May 2007 14:29:34 -0700 (PDT) Received: by obsecurity.dyndns.org (Postfix, from userid 1000) id EFDE4514C7; Fri, 11 May 2007 17:28:47 -0400 (EDT) Date: Fri, 11 May 2007 17:28:47 -0400 From: Kris Kennaway To: David Naylor Message-ID: <20070511212847.GA30211@xor.obsecurity.org> References: <200705112302.35380.dragonsa@highveldmail.co.za> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="u3/rZRmxL6MmkK24" Content-Disposition: inline In-Reply-To: <200705112302.35380.dragonsa@highveldmail.co.za> User-Agent: Mutt/1.4.2.2i Cc: freebsd-hackers@freebsd.org Subject: Re: DPS Initial Ideas X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 May 2007 21:28:49 -0000 --u3/rZRmxL6MmkK24 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Fri, May 11, 2007 at 11:02:31PM +0200, David Naylor wrote: > Hi, >=20 > Thank you all for your responses, it has given me much to think about. I= =20 > guess there is consenses that there is room for improvement in the curren= t=20 > pkg system. Attached are some of my initial ideas about what is required= and=20 > expected in any (and all future) package systems. =20 >=20 > Since I am going away this weekend I have had limited time to work on thi= s=20 > document and as such it is in early stages of development. My objective = is=20 > to get a clear understanding and target of what is required for this new= =20 > package system. =20 There are a couple of ground rules you need to appreciate: 1) Backwards-compatibility with the ports collection is absolutely required. This may not be an issue for you, but some past proposals have included the phrase "rewrite the ports collection to use tool X". This is pretty clearly a non-starter, unless you also provide a workable (i.e. mostly automated) migration strategy. 2) Integration with ongoing work is required. e.g. we have two people working on extending the existing pkg_tools as part of the google summer of code (including one who is working on integrating the metadata into a berkeley DB database, to attempt to solve the scalability problems we are starting to run into as the typical number of installed packages on a user system grows), and we're not going to throw away their work. 3) Dependencies on new code have a high bar for adoption. i.e. if you propose to bring in new software packages into the base system, you need to definitively prove that they are necessary for solving a serious problem. 4) When people hear the phrase "new package system" they take this as an invitation to pile on feature requests, pet peeves, etc that would be "great to have" in a new package system. While it helps to be aware of these ideas, and where appropriate to avoid designing a system that prevents them from being added, the temptation is to undergo feature creep: the proposal expands to engulf all possible features and ends up collapsing under its own weight (also known as "Second System Syndrome"). Stay focused on a core idea you want to achieve, and you might avoid this problem which has killed the last N serious "new package system" projects. I think your current proposal falls short on points 2) and 3). In particular, I don't see where SQLite is necessary to solve any problems we are currently facing, and your proposal conflicts with existing work. Kris --u3/rZRmxL6MmkK24 Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (FreeBSD) iD8DBQFGROAPWry0BWjoQKURAuSYAJ9ud5ef3vVtu/8I0t0huKQ7m1lSeACgnRxq EFd+X5rilmdJCNBL/Zr3wto= =Tl9V -----END PGP SIGNATURE----- --u3/rZRmxL6MmkK24-- From owner-freebsd-hackers@FreeBSD.ORG Sat May 12 00:55:14 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 7C06F16A402 for ; Sat, 12 May 2007 00:55:14 +0000 (UTC) (envelope-from michel@lpthe.jussieu.fr) Received: from shiva.jussieu.fr (shiva.jussieu.fr [134.157.0.129]) by mx1.freebsd.org (Postfix) with ESMTP id 172DE13C44B for ; Sat, 12 May 2007 00:55:13 +0000 (UTC) (envelope-from michel@lpthe.jussieu.fr) Received: from parthe.lpthe.jussieu.fr (parthe.lpthe.jussieu.fr [134.157.10.1]) by shiva.jussieu.fr (8.13.8/jtpda-5.4) with ESMTP id l4C0gBos014476 for ; Sat, 12 May 2007 02:42:11 +0200 (CEST) X-Ids: 165 Received: by parthe.lpthe.jussieu.fr (Postfix, from userid 10096) id 8AF73BF6A0; Sat, 12 May 2007 02:42:10 +0200 (CEST) X-Spam-Checker-Version: SpamAssassin 3.1.8 (2007-02-13) on parthe.lpthe.jussieu.fr X-Spam-Level: X-Spam-Status: No, score=-2.1 required=5.0 tests=AWL,BAYES_00 autolearn=ham version=3.1.8 Received: from niobe.lpthe.jussieu.fr (niobe.lpthe.jussieu.fr [134.157.10.41]) by parthe.lpthe.jussieu.fr (Postfix) with ESMTP id CD0BCBF5F3 for ; Sat, 12 May 2007 02:42:09 +0200 (CEST) Received: by niobe.lpthe.jussieu.fr (Postfix, from userid 2005) id B3F7E88; Sat, 12 May 2007 02:42:09 +0200 (CEST) Date: Sat, 12 May 2007 02:42:09 +0200 From: Michel Talon To: freebsd-hackers@freebsd.org Message-ID: <20070512004209.GA12218@lpthe.jussieu.fr> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.4.2.2i X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-3.0 (shiva.jussieu.fr [134.157.0.165]); Sat, 12 May 2007 02:42:11 +0200 (CEST) X-Virus-Scanned: ClamAV 0.88.7/3227/Fri May 11 11:39:25 2007 on shiva.jussieu.fr X-Virus-Status: Clean X-Miltered: at shiva.jussieu.fr with ID 46450D63.002 by Joe's j-chkmail (http://j-chkmail.ensmp.fr)! X-Mailman-Approved-At: Sat, 12 May 2007 01:11:14 +0000 Subject: Re: DPS Initial Ideas X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 12 May 2007 00:55:14 -0000 Kris Kennaway explained in 4 points why a proposal to introduce a new package system is doomed to failure. In particular he says: > "I think your current proposal falls short on points 2) and 3). In > particular, I don't see where SQLite is necessary to solve any > problems we are currently facing, and your proposal conflicts with > existing work." Existing work consists in integrating metadata in a Berkeley DB database, when new project would like to integrate them in SQLite database. Now existing software (portupgrade) has shown what can be gained by pushing all sort of unstructured data in a Berkeley base :-( In this same thread N arguments have been advanced by various people showing why using something more structured and formal like a SQL database would be beneficial. One of the most obvious being that the sqlite database can be edited as easily as a pure textfile using the sqlite3 program, or can be accessed programmatically from a lot of languages (C, perl, python, ruby) using well known and well tested libraries or connectors. Since sqlite is public domain, there is no licence objection to bring it in FreeBSD. Moreover it is a small program and which could be very useful to a lot of users and for a lot of alternative uses. As a consequence the "high bar of entry" and the "proof of necessity" seem to me an unreasonably stringent condition. Because the problem to be solved is not the "collapse under its own weight" of the proposals, but the collapse of the FreeBSD ports system. Someone (i think des) has said not long ago that Debian or Debian like systems were easier to maintain than FreeBSD. This is an understatement. -- Michel TALON From owner-freebsd-hackers@FreeBSD.ORG Sat May 12 01:10:56 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 2A64916A403 for ; Sat, 12 May 2007 01:10:56 +0000 (UTC) (envelope-from jaj@hcl-club.lu) Received: from 0b10111.de (hcl-club.lu [62.75.155.129]) by mx1.freebsd.org (Postfix) with ESMTP id C14CB13C45D for ; Sat, 12 May 2007 01:10:55 +0000 (UTC) (envelope-from jaj@hcl-club.lu) Received: from [192.168.1.100] (ip-83-99-95-13.dyn.luxdsl.pt.lu [83.99.95.13]) by 0b10111.de (Postfix) with ESMTP id 1147C872C2E2; Sat, 12 May 2007 02:51:56 +0200 (CEST) Message-ID: <46450EE1.300@hcl-club.lu> Date: Sat, 12 May 2007 02:48:33 +0200 From: Jona Joachim User-Agent: Thunderbird 2.0.0.0 (Windows/20070326) MIME-Version: 1.0 To: Ivan Voras References: <200705102105.27271.blackdragon@highveldmail.co.za> In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Mailman-Approved-At: Sat, 12 May 2007 01:11:40 +0000 Cc: freebsd-hackers@freebsd.org Subject: Re: New FreeBSD package system (a.k.a. Daemon Package System (dps)) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 12 May 2007 01:10:56 -0000 Ivan Voras a écrit : > David Naylor wrote: >> Dear Jordan >> >> Recently I stumbled across a document you wrote in 2001, entitled "FreeBSD >> installation and package tools, past, present and future". I find FreeBSD >> appealing and I would like to contribute it its success, and as your article >> describes, the installation and packaging system is lacking. Since the >> installation system is being tackled under a SoC project I am hoping to give >> the packaging system a go. > > I've just read the document, and since I was also thinking about the > ports/packages system lately, here are a few random ideas: > > - I think it's time to give up on using BDB+directory tree full of text > files for storing the installed packages database, and I propose all of > this be replaced by a single SQLite database. SQLite is public domain > (can be slurped into base system), embeddable, stores all data in a > single file, lightweight, fast, and can be used to do fancy things such > as reporting. The current pkg_info's behaviour that takes *noticable* > *time* to generate 1 line of output per package is horrible in this day > and age. And the upcoming X11/X.Org 7.2 with cca 400 packages (which is > in itself horrible just for a X11 system) will make it unbearable (also > portupgrade which already looks slower by the day). I don't think it would be a good idea to use SQLite for this purpose. First of all using the file system is the Unix way of doing things. It's efficient and easy to use, it transparent and user friendly. You can simply run vi to inspect a text file but you can't do this which an sqlite database. You have to learn sqlite to do it. Furthermore I don't think the pkg_* tools are slow. They are quite fast IMO. If you let pkg_info print the entire list of installed ports it's only slow because of your line-buffered console. Just redirect the output to a file and you'll see that it's blazing fast. If I compare it for example to Debians apt-get/apt-cache commands it's much faster. portupgrade is very slow, that's true. First of all it's written in Ruby which is not one of the fastest languages but there is another thing that slows it down considerably, which is rebuilding its database. Furthermore I think it would be a very bad idea to include sqlite in base. There is already a lot of third party stuff in base. The philosophy of the BSDs is to provide and maintain an entire OS. This is quite the opposite of how a GNU/Linux system is designed. Both ways have their pros and cons. An advantage of the BSD way of doing things is that the developers know the code very well and have control over the quality of the code. If you include 3rd party software into the FreeBSD base system you make the FreeBSD project depend on the people that wrote that code. Of course you could fork it but the FreeBSD developers are not necessarily familiar with the code. Security patches would have to be merged all the time and a lot of communication between the two projects is needed. I think the best way to go would be to use only folder hierarchies and text files and write a libary in C that provides portupgrade functionality. The code under src/usr.sbin/pkg_install/lib/ would be a good base for this. Then you could use a frontend program that makes use of this library. This frontend could be a CLI program or a GUI based program. Best regards, Jona From owner-freebsd-hackers@FreeBSD.ORG Sat May 12 01:19:58 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id D4BD716A408 for ; Sat, 12 May 2007 01:19:58 +0000 (UTC) (envelope-from anderson@freebsd.org) Received: from mh2.centtech.com (moat3.centtech.com [64.129.166.50]) by mx1.freebsd.org (Postfix) with ESMTP id 84DB713C455 for ; Sat, 12 May 2007 01:19:58 +0000 (UTC) (envelope-from anderson@freebsd.org) Received: from neutrino.centtech.com (andersonbox1.centtech.com [192.168.42.21]) by mh2.centtech.com (8.13.8/8.13.8) with ESMTP id l4C1JpOG064422; Fri, 11 May 2007 20:19:51 -0500 (CDT) (envelope-from anderson@freebsd.org) Message-ID: <46451637.4010900@freebsd.org> Date: Fri, 11 May 2007 20:19:51 -0500 From: Eric Anderson User-Agent: Thunderbird 2.0.0.0 (X11/20070420) MIME-Version: 1.0 To: Jona Joachim References: <200705102105.27271.blackdragon@highveldmail.co.za> <46450EE1.300@hcl-club.lu> In-Reply-To: <46450EE1.300@hcl-club.lu> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 8bit X-Virus-Scanned: ClamAV 0.88.4/3227/Fri May 11 04:39:25 2007 on mh2.centtech.com X-Virus-Status: Clean X-Spam-Status: No, score=-2.5 required=8.0 tests=AWL,BAYES_00 autolearn=ham version=3.1.6 X-Spam-Checker-Version: SpamAssassin 3.1.6 (2006-10-03) on mh2.centtech.com Cc: freebsd-hackers@freebsd.org, Ivan Voras Subject: Re: New FreeBSD package system (a.k.a. Daemon Package System (dps)) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 12 May 2007 01:19:59 -0000 On 05/11/07 19:48, Jona Joachim wrote: > Ivan Voras a écrit : >> David Naylor wrote: >>> Dear Jordan >>> >>> Recently I stumbled across a document you wrote in 2001, entitled "FreeBSD >>> installation and package tools, past, present and future". I find FreeBSD >>> appealing and I would like to contribute it its success, and as your article >>> describes, the installation and packaging system is lacking. Since the >>> installation system is being tackled under a SoC project I am hoping to give >>> the packaging system a go. >> I've just read the document, and since I was also thinking about the >> ports/packages system lately, here are a few random ideas: >> >> - I think it's time to give up on using BDB+directory tree full of text >> files for storing the installed packages database, and I propose all of >> this be replaced by a single SQLite database. SQLite is public domain >> (can be slurped into base system), embeddable, stores all data in a >> single file, lightweight, fast, and can be used to do fancy things such >> as reporting. The current pkg_info's behaviour that takes *noticable* >> *time* to generate 1 line of output per package is horrible in this day >> and age. And the upcoming X11/X.Org 7.2 with cca 400 packages (which is >> in itself horrible just for a X11 system) will make it unbearable (also >> portupgrade which already looks slower by the day). > > I don't think it would be a good idea to use SQLite for this purpose. > First of all using the file system is the Unix way of doing things. It's > efficient and easy to use, it transparent and user friendly. You can > simply run vi to inspect a text file but you can't do this which an > sqlite database. You have to learn sqlite to do it. > Furthermore I don't think the pkg_* tools are slow. They are quite fast > IMO. If you let pkg_info print the entire list of installed ports it's > only slow because of your line-buffered console. Just redirect the > output to a file and you'll see that it's blazing fast. If I compare it Hmm.. Tried to stay out of this whole 'discussion', but, I just disagree here flat out. # time pkg_info > /dev/null real 0m16.952s user 0m0.751s sys 0m0.163s During that entire time, my hard disk was being hammered. Blazing fast.. ? Maybe I need a 15k RPM SCSI disk, or something. Anyway, to me, a file system *IS* a database, and if it's slow, it probably could be used better, and get much better performance. I've seen some very ingenious file system layouts that have several million records easily used on slow systems, and it works pretty well. [..snip warm fuzzies..] > I think the best way to go would be to use only folder hierarchies and > text files and write a libary in C that provides portupgrade > functionality. The code under src/usr.sbin/pkg_install/lib/ would be a > good base for this. Then you could use a frontend program that makes use > of this library. This frontend could be a CLI program or a GUI based > program. That's not a bad idea, if it isn't already available somewhere. Eric From owner-freebsd-hackers@FreeBSD.ORG Sat May 12 01:42:01 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 4E06216A404 for ; Sat, 12 May 2007 01:42:01 +0000 (UTC) (envelope-from wmoran@collaborativefusion.com) Received: from mx00.pub.collaborativefusion.com (mx00.pub.collaborativefusion.com [206.210.89.199]) by mx1.freebsd.org (Postfix) with ESMTP id 2668B13C44B for ; Sat, 12 May 2007 01:42:01 +0000 (UTC) (envelope-from wmoran@collaborativefusion.com) Received: from working (c-71-60-105-193.hsd1.pa.comcast.net [71.60.105.193]) (AUTH: LOGIN wmoran, SSL: TLSv1/SSLv3,256bits,AES256-SHA) by wingspan with esmtp; Fri, 11 May 2007 21:31:55 -0400 id 0005646F.4645190B.00016198 Date: Fri, 11 May 2007 21:31:54 -0400 From: Bill Moran To: Michel Talon Message-Id: <20070511213154.4fdcd7d6.wmoran@collaborativefusion.com> In-Reply-To: <20070512004209.GA12218@lpthe.jussieu.fr> References: <20070512004209.GA12218@lpthe.jussieu.fr> Organization: Collaborative Fusion Inc. X-Mailer: Sylpheed version 2.2.10 (GTK+ 2.10.9; i386-portbld-freebsd6.2) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Cc: freebsd-hackers@freebsd.org Subject: Re: DPS Initial Ideas X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 12 May 2007 01:42:01 -0000 Michel Talon wrote: > > Kris Kennaway explained in 4 points why a proposal to introduce a new > package system is doomed to failure. What the hell? You're making like any effort to improve the packaging system is doomed to failure without SQLite. Before you even go in to any more of a rant, answer me this one question: Why should the FreeBSD project take on the load of importing _yet_ _another_ database library? What's wrong with using bdb, which has been in the base system for as long as I can remember? It's what various package managing tools already use with great success. > In particular he says: > > "I think your current proposal falls short on points 2) and 3). In > > particular, I don't see where SQLite is necessary to solve any > > problems we are currently facing, and your proposal conflicts with > > existing work." > > Existing work consists in integrating metadata in a Berkeley DB > database, when new project would like to integrate them in SQLite > database. Now existing software (portupgrade) has shown what can be > gained by pushing all sort of unstructured data in a Berkeley base :-( > In this same thread N arguments have been advanced by various people > showing why using something more structured and formal like a SQL > database would be beneficial. > > One of the most obvious being that the sqlite database can be edited > as easily as a pure textfile using the sqlite3 program, or can be > accessed programmatically from a lot of languages (C, perl, python, > ruby) using well known and well tested libraries or connectors. > > Since sqlite is public domain, there is no licence objection to bring it > in FreeBSD. Moreover it is a small program and which could be very > useful to a lot of users and for a lot of alternative uses. > As a consequence the "high bar of entry" and the "proof of necessity" > seem to me an unreasonably stringent condition. > > > Because the problem to be solved is not the "collapse under its own > weight" of the proposals, but the collapse of the FreeBSD ports system. > Someone (i think des) has said not long ago that Debian or Debian like > systems were easier to maintain than FreeBSD. This is an understatement. > > > -- > > Michel TALON > > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" > > > > > > -- Bill Moran Collaborative Fusion Inc. wmoran@collaborativefusion.com Phone: 412-422-3463x4023 **************************************************************** IMPORTANT: This message contains confidential information and is intended only for the individual named. If the reader of this message is not an intended recipient (or the individual responsible for the delivery of this message to an intended recipient), please be advised that any re-use, dissemination, distribution or copying of this message is prohibited. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. E-mail transmission cannot be guaranteed to be secure or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. The sender therefore does not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail transmission. **************************************************************** From owner-freebsd-hackers@FreeBSD.ORG Sat May 12 02:02:21 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 47FE816A403 for ; Sat, 12 May 2007 02:02:21 +0000 (UTC) (envelope-from mwm-keyword-freebsdhackers2.e313df@mired.org) Received: from mired.org (vpn.mired.org [66.92.153.74]) by mx1.freebsd.org (Postfix) with SMTP id CFE8F13C44B for ; Sat, 12 May 2007 02:02:20 +0000 (UTC) (envelope-from mwm-keyword-freebsdhackers2.e313df@mired.org) Received: (qmail 82391 invoked by uid 1001); 12 May 2007 02:01:50 -0000 Received: by bhuda.mired.org (tmda-sendmail, from uid 1001); Fri, 11 May 2007 22:01:49 -0400 MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <17989.8202.624522.136573@bhuda.mired.org> Date: Fri, 11 May 2007 22:01:46 -0400 To: Michel Talon In-Reply-To: <20070512004209.GA12218@lpthe.jussieu.fr> References: <20070512004209.GA12218@lpthe.jussieu.fr> X-Mailer: VM 7.19 under Emacs 21.3.1 X-Primary-Address: mwm@mired.org X-face: "5Mnwy%?j>IIV\)A=):rjWL~NB2aH[}Yq8Z=u~vJ`"(,&SiLvbbz2W`; h9L,Yg`+vb1>RG% *h+%X^n0EZd>TM8_IB;a8F?(Fb"lw'IgCoyM.[Lg#r\ X-Delivery-Agent: TMDA/1.1.11 (Ladyburn) From: Mike Meyer Cc: freebsd-hackers@freebsd.org Subject: Re: DPS Initial Ideas X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 12 May 2007 02:02:21 -0000 In <20070512004209.GA12218@lpthe.jussieu.fr>, Michel Talon typed: > One of the most obvious being that the sqlite database can be edited > as easily as a pure textfile using the sqlite3 program Huh? They can? With a pure textfile, if vi is busted, I can use ed. If ed is also busted, I can use sed. What do I use on an sqlite database if sqlite3 is busted? I'm currently reinstalling all the ports on my system, changing LOCALBASE from /usr/opt to /usr, just to see if this breaks anything. Adjusting all those pure textfile configs for this is trivial: find /etc /usr/etc /usr/X11R6/etc -type f | xargs sed -e 's;/opt;;g' -i .opt How can I do this "as easily" if I were using sqlite database(s)? http://www.mired.org/consulting.html Independent Network/Unix/Perforce consultant, email for more information. From owner-freebsd-hackers@FreeBSD.ORG Sat May 12 02:29:09 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id E3F0316A400 for ; Sat, 12 May 2007 02:29:09 +0000 (UTC) (envelope-from duane@dwlabs.ca) Received: from smtpout.eastlink.ca (smtpout.eastlink.ca [24.222.0.30]) by mx1.freebsd.org (Postfix) with ESMTP id AC08A13C43E for ; Sat, 12 May 2007 02:29:09 +0000 (UTC) (envelope-from duane@dwlabs.ca) Received: from ip04.eastlink.ca ([24.222.10.20]) by mta02.eastlink.ca (Sun Java System Messaging Server 6.2-4.03 (built Sep 22 2005)) with ESMTP id <0JHW0071CPKKGGP0@mta02.eastlink.ca> for freebsd-hackers@freebsd.org; Fri, 11 May 2007 23:29:08 -0300 (ADT) Received: from blk-224-199-230.eastlink.ca (HELO dwpc.dwlabs.ca) ([24.224.199.230]) by ip04.eastlink.ca with ESMTP; Fri, 11 May 2007 22:59:56 -0300 Received: from dwpc.dwlabs.ca (www.dwlabs.ca [192.168.0.10]) by dwpc.dwlabs.ca (8.13.8/8.13.8) with ESMTP id l4C2TOA7002423; Fri, 11 May 2007 23:29:30 -0300 (ADT envelope-from duane@dwpc.dwlabs.ca) Received: (from duane@localhost) by dwpc.dwlabs.ca (8.13.8/8.13.8/Submit) id l4C2TOrC002422; Fri, 11 May 2007 23:29:24 -0300 (ADT envelope-from duane) Date: Fri, 11 May 2007 23:29:24 -0300 From: Duane Whitty In-reply-to: <20070511212847.GA30211@xor.obsecurity.org> To: Kris Kennaway Message-id: <20070512022924.GA1017@dwpc.dwlabs.ca> MIME-version: 1.0 Content-type: text/plain; charset=us-ascii Content-transfer-encoding: 7BIT Content-disposition: inline X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: Ao8CAAzDREYY4MfmdGdsb2JhbACQAAE2 X-IronPort-AV: i="4.14,525,1170648000"; d="scan'208"; a="214492920:sNHT27859770" X-Virus-Scanned: ClamAV version 0.88.6, clamav-milter version 0.88.6 on dwpc.dwlabs.ca X-Virus-Status: Clean X-Spam-Checker-Version: SpamAssassin 3.1.4 (2006-07-25) on dwpc.dwlabs.ca References: <200705112302.35380.dragonsa@highveldmail.co.za> <20070511212847.GA30211@xor.obsecurity.org> User-Agent: Mutt/1.4.2.2i X-Spam-Status: No, score=-2.1 required=5.0 tests=AWL,BAYES_00, UNPARSEABLE_RELAY autolearn=ham version=3.1.4 Cc: freebsd-hackers@freebsd.org Subject: Re: DPS Initial Ideas X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: freebsd-hackers@freebsd.org List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 12 May 2007 02:29:10 -0000 On Friday, 11 May 2007 at 17:28:47 -0400, Kris Kennaway wrote: > On Fri, May 11, 2007 at 11:02:31PM +0200, David Naylor wrote: > > Hi, > > > > Thank you all for your responses, it has given me much to think about. I > > guess there is consenses that there is room for improvement in the current > > pkg system. Attached are some of my initial ideas about what is required and > > expected in any (and all future) package systems. > > > > Since I am going away this weekend I have had limited time to work on this > > document and as such it is in early stages of development. My objective is > > to get a clear understanding and target of what is required for this new > > package system. > > There are a couple of ground rules you need to appreciate: > > 1) Backwards-compatibility with the ports collection is absolutely > required. This may not be an issue for you, but some past proposals > have included the phrase "rewrite the ports collection to use tool X". > This is pretty clearly a non-starter, unless you also provide a > workable (i.e. mostly automated) migration strategy. > > 2) Integration with ongoing work is required. e.g. we have two people > working on extending the existing pkg_tools as part of the google > summer of code (including one who is working on integrating the > metadata into a berkeley DB database, to attempt to solve the > scalability problems we are starting to run into as the typical number > of installed packages on a user system grows), and we're not going to > throw away their work. > > 3) Dependencies on new code have a high bar for adoption. i.e. if you > propose to bring in new software packages into the base system, you > need to definitively prove that they are necessary for solving a > serious problem. > > 4) When people hear the phrase "new package system" they take this as > an invitation to pile on feature requests, pet peeves, etc that would > be "great to have" in a new package system. While it helps to be > aware of these ideas, and where appropriate to avoid designing a > system that prevents them from being added, the temptation is to > undergo feature creep: the proposal expands to engulf all possible > features and ends up collapsing under its own weight (also known as > "Second System Syndrome"). Stay focused on a core idea you want to > achieve, and you might avoid this problem which has killed the last N > serious "new package system" projects. > > I think your current proposal falls short on points 2) and 3). In > particular, I don't see where SQLite is necessary to solve any > problems we are currently facing, and your proposal conflicts with > existing work. > > Kris Kris, In your opinion what is the biggest problem(s) the ports system and the package building system currently face? Is this a common problem, i.e., is the issue facing building from ports the same as installing from pre-built packages? I ask this in the context of infrastructure as opposed to any tools currently being used. Is it hoped / planned that storing the metadata in a berkeley DB database will help with the parallelization of package building? If only one thing was going to be done to improve the ports system, not including drafting more volunteers :) , what would you recommend that one thing be? Duane From owner-freebsd-hackers@FreeBSD.ORG Sat May 12 03:32:34 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 23DCD16A404 for ; Sat, 12 May 2007 03:32:34 +0000 (UTC) (envelope-from ivoras@fer.hr) Received: from pinus.cc.fer.hr (pinus.cc.fer.hr [161.53.73.18]) by mx1.freebsd.org (Postfix) with ESMTP id 7471413C469 for ; Sat, 12 May 2007 03:32:33 +0000 (UTC) (envelope-from ivoras@fer.hr) Received: from [161.53.72.113] (lara.cc.fer.hr [161.53.72.113]) by pinus.cc.fer.hr (8.12.2/8.12.2) with ESMTP id l4B9s23f019644; Fri, 11 May 2007 11:54:03 +0200 (MEST) Message-ID: <46443AF0.2070006@fer.hr> Date: Fri, 11 May 2007 11:44:16 +0200 From: Ivan Voras User-Agent: Thunderbird 1.5.0.10 (X11/20060911) MIME-Version: 1.0 To: =?UTF-8?B?RGFnLUVybGluZyBTbcO4cmdyYXY=?= References: <200705102105.27271.blackdragon@highveldmail.co.za> <4643C7DB.6000408@elischer.org> <863b233dq4.fsf@dwp.des.no> In-Reply-To: <863b233dq4.fsf@dwp.des.no> X-Enigmail-Version: 0.94.2.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="------------enigCEAEF8B7D5341ACCA7BDCDFB" Cc: freebsd-hackers@freebsd.org Subject: Re: New FreeBSD package system (a.k.a. Daemon Package System (dps)) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 12 May 2007 03:32:34 -0000 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enigCEAEF8B7D5341ACCA7BDCDFB Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: quoted-printable Dag-Erling Sm=C3=B8rgrav wrote: > Ivan Voras writes: >> You can inspect s sqlite database with the provided utility. Unless th= e >> database gets corrupted (which it tries to avoid by respecting ACID), >=20 > ACID is not something a database "respects", it is a set of guarantees > that it provides to the application. Avoiding database corruption is > a necessary requirement for, rather than a consequence of, ACID. I'm thinking of ACID as a set of ideas / procedures, the consequence of=20 which is avoiding corruption. Of course, there's a "hierarchy of=20 reliability" - the db relies on the file system to meet the=20 requirements, the file system relies on the hardware, etc. but if the db = doesn't make use of those, it's all for nothing. > Perhaps you mean that SQLite tries to avoid database corruption by usin= g > locks, and either scatter-gather writes or copy-on-write, and flushing > the file between transactions, to ensure consistency? It uses locks, a journal file (for DML transactions), and flushes data=20 between transactions. More info at: http://www.sqlite.org/lockingv3.html --------------enigCEAEF8B7D5341ACCA7BDCDFB Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.5 (GNU/Linux) Comment: Using GnuPG with SUSE - http://enigmail.mozdev.org iD8DBQFGRDr3ldnAQVacBcgRAth4AKDvmRRXyUcDKXlAEwCV3UZpSQa4XACgwwAT bYQej+XTB2FSuR/wrlQJdBA= =RAts -----END PGP SIGNATURE----- --------------enigCEAEF8B7D5341ACCA7BDCDFB-- From owner-freebsd-hackers@FreeBSD.ORG Sat May 12 04:43:02 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id D91CE16A400 for ; Sat, 12 May 2007 04:43:02 +0000 (UTC) (envelope-from kris@obsecurity.org) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.freebsd.org (Postfix) with ESMTP id C50DB13C43E for ; Sat, 12 May 2007 04:43:02 +0000 (UTC) (envelope-from kris@obsecurity.org) Received: from obsecurity.dyndns.org (elvis.mu.org [192.203.228.196]) by elvis.mu.org (Postfix) with ESMTP id AC3561A3C19 for ; Fri, 11 May 2007 21:43:48 -0700 (PDT) Received: by obsecurity.dyndns.org (Postfix, from userid 1000) id CEA00513B7; Sat, 12 May 2007 00:43:01 -0400 (EDT) Date: Sat, 12 May 2007 00:43:01 -0400 From: Kris Kennaway To: freebsd-hackers@freebsd.org Message-ID: <20070512044301.GA43885@xor.obsecurity.org> References: <200705112302.35380.dragonsa@highveldmail.co.za> <20070511212847.GA30211@xor.obsecurity.org> <20070512022924.GA1017@dwpc.dwlabs.ca> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="9amGYk9869ThD9tj" Content-Disposition: inline In-Reply-To: <20070512022924.GA1017@dwpc.dwlabs.ca> User-Agent: Mutt/1.4.2.2i Cc: Kris Kennaway Subject: Re: DPS Initial Ideas X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 12 May 2007 04:43:03 -0000 --9amGYk9869ThD9tj Content-Type: text/plain; charset=us-ascii Content-Disposition: inline On Fri, May 11, 2007 at 11:29:24PM -0300, Duane Whitty wrote: > Is it hoped / planned that storing the metadata in a berkeley DB > database will help with the parallelization of package building? That's somewhat orthogonal: the problem there is mutual exclusion and job ordering. > In your opinion what is the biggest problem(s) the ports system and > the package building system currently face? Is this a common problem, > i.e., is the issue facing building from ports the same as installing > from pre-built packages? I ask this in the context of infrastructure > as opposed to any tools currently being used. > If only one thing was going to be done to improve the ports system, > not including drafting more volunteers :) , what would you recommend > that one thing be? Those are both excellent questions, and I'm not sure I have all the answers. It's something I'll try to discuss with people at the BSDCan devsummit next week. I think the major area that needs further work is to do with improving the upgrade process. Historically, the ports collection and package tools did not support any notion of "do what needs to be done to upgrade this package". This was fine in the days when systems typically only had a handful of ports installed, and they each had only a few dependencies: you could update them by hand without much trouble using 'make deinstall' and 'make install'. Of course, it fails utterly to scale to the era of GNOME and modular X.org. This problem was more or less solved by tools like portupgrade [1], but these are bolted on to the side of the ports collection and underlying package tools rather than being properly integrated with them. One consequence of this is that the UPDATING file has a disturbing number of entries for manual steps required to update certain ports. Some of these are because of vendor screwups that we can't really fix (e.g. broken backawards compatibility with existing user files requiring data migration). But a lot of them come down to failures of the FreeBSD upgrade process: either a ports developer was lazy and didn't want to do the extra work to support automatic upgrades, or the upgrade tools are insufficiently flexible to handle the upgrade. My opinion is that every time a ports developer adds an entry to UPDATING that lists manual steps required to update a port (except for the vendor case above), it usually means we have failed as a project. Part of this problem comes down to metadata management and integration with the underlying tools. Some of the information that an upgrade tool needs to plan and manage the upgrades is not efficiently accessible using the pkg_tools (queries are O(N) or worse in the number of packages installed, etc). portupgrade currently solves this by maintaining its own parallel version of the metadata in a database that can be efficiently queried (it also solves part of the problem by ignoring it, which is a limitation that affected e.g. the xorg upgrade and required the UPDATING entry). The problem with maintaining a parallel database is that unless users always manipulate their ports and packages using portupgrade this database can become stale. Addressing this is the goal of Garrett Cooper's SoC project, by pushing down the database management into pkg_tools so hopefully portupgrade and pkg_tools can be made to share the database (or portupgrade can revert to querying efficiently via the pkg_tools). There are other problems affecting package and upgrade management that are due to scaling limitations and data management issues that have become relevant as the ports collection has grown in size and complexity. Some information (e.g. dependency lists) can currently only be obtained by recursively walking the tree, which is a very expensive operation for things like GNOME. There are probably other operations that are O(N) in some large number N that I can't think of right now. It would make sense to explore whether there are ways of avoiding or optimizing these expensive operations. Kris [1] There are other upgrade tools, but IMO none of them are as mature as portupgrade in terms of feature support or robustness, and they also do not attempt to solve the metadata management issues at all. --9amGYk9869ThD9tj Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (FreeBSD) iD8DBQFGRUXVWry0BWjoQKURAo7XAJ4+zEVT9XvIVjU5Vj4CZmcPnnltHwCfddRu fIykdGkJQ1foR2Mgf88DFxg= =hAGV -----END PGP SIGNATURE----- --9amGYk9869ThD9tj-- From owner-freebsd-hackers@FreeBSD.ORG Sat May 12 04:56:18 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 37B4916A403 for ; Sat, 12 May 2007 04:56:18 +0000 (UTC) (envelope-from mwm-keyword-freebsdhackers2.e313df@mired.org) Received: from mired.org (vpn.mired.org [66.92.153.74]) by mx1.freebsd.org (Postfix) with SMTP id DEC7913C448 for ; Sat, 12 May 2007 04:56:17 +0000 (UTC) (envelope-from mwm-keyword-freebsdhackers2.e313df@mired.org) Received: (qmail 72632 invoked by uid 1001); 12 May 2007 04:55:46 -0000 Received: by bhuda.mired.org (tmda-sendmail, from uid 1001); Sat, 12 May 2007 00:55:45 -0400 MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <17989.18640.420958.755412@bhuda.mired.org> Date: Sat, 12 May 2007 00:55:44 -0400 To: Kris Kennaway In-Reply-To: <20070511191057.GA25833@xor.obsecurity.org> References: <17987.52037.112351.872442@bhuda.mired.org> <20070511015156.GA77895@xor.obsecurity.org> <17987.52970.398402.580727@bhuda.mired.org> <20070511021249.GA78729@xor.obsecurity.org> <17987.57305.7130.873114@bhuda.mired.org> <20070511051852.GA89359@xor.obsecurity.org> <17988.32573.910854.388638@bhuda.mired.org> <20070511184259.GA23483@xor.obsecurity.org> <17988.48879.828384.170557@bhuda.mired.org> <20070511191057.GA25833@xor.obsecurity.org> X-Mailer: VM 7.19 under Emacs 21.3.1 X-Primary-Address: mwm@mired.org X-face: "5Mnwy%?j>IIV\)A=):rjWL~NB2aH[}Yq8Z=u~vJ`"(,&SiLvbbz2W`; h9L,Yg`+vb1>RG% *h+%X^n0EZd>TM8_IB;a8F?(Fb"lw'IgCoyM.[Lg#r\ X-Delivery-Agent: TMDA/1.1.11 (Ladyburn) From: Mike Meyer Cc: freebsd-hackers@freebsd.org, Ivan Voras Subject: Re: New FreeBSD package system (a.k.a. Daemon Package System (dps)) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 12 May 2007 04:56:18 -0000 In <20070511191057.GA25833@xor.obsecurity.org>, Kris Kennaway typed: > > There are clearly other workable ideas - as I said, the linux folks > > managed to make it work. But it's not an easy problem. I certainly > > wouldn't suggest rebuilding the packaging system to deal with this, > > except as part of a larger effort. On the other hand, since people are > > working on the ports/package system (I see port/pkg database and some > > ports infrastructure work in the current SoC projects list), not > > keeping this goal in mind would seem to be a bit short-sighted. I > > wouldn't be surprised if your option #1 could benefit from this as > > well. > But you said you were interested in working on it...so what is your > idea? Actually, I said it's on my list of things to do. That's a rather long list. I'm more interested in port building, but the dependencies to that one look like: 1) Get a gcc build whose driver correctly handles "-m32" for cross-compiling. The system gcc can do cross-compilation, but the driver doesn't know where the various bits it needs to link the generated objects with live, so you have to tell it about those things by hand. Possibly one of the versions in ports does this, but it would be nice if the one in the base system did this. I believe it's mostly a matter of properly configuring the gcc build. 2) Rework the ports dependency/conflict detection facilities. Mostly, figuring out the architecture of the current build vs. any previously installed packages or ports it depends on or conflicts with. There are also some things that could be done to make the dependencies more reliable, like auto-detecting shared library dependencies and adding those. Having the target platform for an installed package in the package database would be a nice jump on this. 3) Now we get to the interesting part: figuring out where the bits for the cross-installation are going to land. A lot depends on how far apart we want to keep the two worlds. I'd like them to be integrated, as the point of being able to install 32-bit binaries on a 64-bit system is to run applications that aren't yet 32-bit clean, so the only collisions should be shared libraries common to applications from both sets. A lot will depend on how badly things collide. The other factor is that I'd like the build platform to be tranparent to the end user: you should be able to build 32-bit packages on either 32 or 64 bit platforms, and use them on either. This ties back to the "How do we figure out where our libraries are" issue. I don't know if all these goals can be met. 4) Once that's done, start working on getting cross-building and installing ports working. This is a long way from any work being done on it, so I don't have any details. Doing step #2 would help with fleshing out these step. 5) Now make make building packages from those ports and installing from those packages work. http://www.mired.org/consulting.html Independent Network/Unix/Perforce consultant, email for more information. From owner-freebsd-hackers@FreeBSD.ORG Sat May 12 04:56:20 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id B3DBB16A404 for ; Sat, 12 May 2007 04:56:20 +0000 (UTC) (envelope-from mwm-keyword-freebsdhackers2.e313df@mired.org) Received: from mired.org (vpn.mired.org [66.92.153.74]) by mx1.freebsd.org (Postfix) with SMTP id 6794113C44C for ; Sat, 12 May 2007 04:56:20 +0000 (UTC) (envelope-from mwm-keyword-freebsdhackers2.e313df@mired.org) Received: (qmail 72659 invoked by uid 1001); 12 May 2007 04:55:49 -0000 Received: by bhuda.mired.org (tmda-sendmail, from uid 1001); Sat, 12 May 2007 00:55:48 -0400 Message-ID: <17989.18644.687543.657988@bhuda.mired.org> Date: Sat, 12 May 2007 00:55:48 -0400 MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit To: Kris Kennaway In-Reply-To: <20070511191057.GA25833@xor.obsecurity.org> References: <17987.52037.112351.872442@bhuda.mired.org> <20070511015156.GA77895@xor.obsecurity.org> <17987.52970.398402.580727@bhuda.mired.org> <20070511021249.GA78729@xor.obsecurity.org> <17987.57305.7130.873114@bhuda.mired.org> <20070511051852.GA89359@xor.obsecurity.org> <17988.32573.910854.388638@bhuda.mired.org> <20070511184259.GA23483@xor.obsecurity.org> <17988.48879.828384.170557@bhuda.mired.org> <20070511191057.GA25833@xor.obsecurity.org> X-Mailer: VM 7.19 under Emacs 21.3.1 X-Primary-Address: mwm@mired.org X-face: "5Mnwy%?j>IIV\)A=):rjWL~NB2aH[}Yq8Z=u~vJ`"(,&SiLvbbz2W`; h9L,Yg`+vb1>RG% *h+%X^n0EZd>TM8_IB;a8F?(Fb"lw'IgCoyM.[Lg#r\ X-Delivery-Agent: TMDA/1.1.11 (Ladyburn) From: Mike Meyer Cc: freebsd-hackers@freebsd.org, Ivan Voras Subject: Re: New FreeBSD package system (a.k.a. Daemon Package System (dps)) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 12 May 2007 04:56:20 -0000 In <20070511191057.GA25833@xor.obsecurity.org>, Kris Kennaway typed: > > There are clearly other workable ideas - as I said, the linux folks > > managed to make it work. But it's not an easy problem. I certainly > > wouldn't suggest rebuilding the packaging system to deal with this, > > except as part of a larger effort. On the other hand, since people are > > working on the ports/package system (I see port/pkg database and some > > ports infrastructure work in the current SoC projects list), not > > keeping this goal in mind would seem to be a bit short-sighted. I > > wouldn't be surprised if your option #1 could benefit from this as > > well. > But you said you were interested in working on it...so what is your > idea? Actually, I said it's on my list of things to do. That's a rather long list. I'm more interested in port building, but the dependencies to that one look like: 1) Get a gcc build whose driver correctly handles "-m32" for cross-compiling. The system gcc can do cross-compilation, but the driver doesn't know where the various bits it needs to link the generated objects with live, so you have to tell it about those things by hand. Possibly one of the versions in ports does this, but it would be nice if the one in the base system did this. I believe it's mostly a matter of properly configuring the gcc build. 2) Rework the ports dependency/conflict detection facilities. Mostly, figuring out the architecture of the current build vs. any previously installed packages or ports it depends on or conflicts with. There are also some things that could be done to make the dependencies more reliable, like auto-detecting shared library dependencies and adding those. Having the target platform for an installed package in the package database would be a nice jump on this. 3) Now we get to the interesting part: figuring out where the bits for the cross-installation are going to land. A lot depends on how far apart we want to keep the two worlds. I'd like them to be integrated, as the point of being able to install 32-bit binaries on a 64-bit system is to run applications that aren't yet 32-bit clean, so the only collisions should be shared libraries common to applications from both sets. A lot will depend on how badly things collide. The other factor is that I'd like the build platform to be tranparent to the end user: you should be able to build 32-bit packages on either 32 or 64 bit platforms, and use them on either. This ties back to the "How do we figure out where our libraries are" issue. I don't know if all these goals can be met. 4) Once that's done, start working on getting cross-building and installing ports working. This is a long way from any work being done on it, so I don't have any details. Doing step #2 would help with fleshing out these step. 5) Now make make building packages from those ports and installing from those packages work. http://www.mired.org/consulting.html Independent Network/Unix/Perforce consultant, email for more information. From owner-freebsd-hackers@FreeBSD.ORG Sat May 12 05:20:39 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 9CBFF16A400 for ; Sat, 12 May 2007 05:20:39 +0000 (UTC) (envelope-from kris@obsecurity.org) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.freebsd.org (Postfix) with ESMTP id 85F0A13C459 for ; Sat, 12 May 2007 05:20:39 +0000 (UTC) (envelope-from kris@obsecurity.org) Received: from obsecurity.dyndns.org (elvis.mu.org [192.203.228.196]) by elvis.mu.org (Postfix) with ESMTP id 74A401A3C19; Fri, 11 May 2007 22:21:25 -0700 (PDT) Received: by obsecurity.dyndns.org (Postfix, from userid 1000) id 9A658513B7; Sat, 12 May 2007 01:20:38 -0400 (EDT) Date: Sat, 12 May 2007 01:20:38 -0400 From: Kris Kennaway To: Mike Meyer Message-ID: <20070512052038.GA57479@xor.obsecurity.org> References: <20070511015156.GA77895@xor.obsecurity.org> <17987.52970.398402.580727@bhuda.mired.org> <20070511021249.GA78729@xor.obsecurity.org> <17987.57305.7130.873114@bhuda.mired.org> <20070511051852.GA89359@xor.obsecurity.org> <17988.32573.910854.388638@bhuda.mired.org> <20070511184259.GA23483@xor.obsecurity.org> <17988.48879.828384.170557@bhuda.mired.org> <20070511191057.GA25833@xor.obsecurity.org> <17989.18640.420958.755412@bhuda.mired.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <17989.18640.420958.755412@bhuda.mired.org> User-Agent: Mutt/1.4.2.2i Cc: freebsd-hackers@freebsd.org, Ivan Voras , Kris Kennaway Subject: Re: New FreeBSD package system (a.k.a. Daemon Package System (dps)) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 12 May 2007 05:20:39 -0000 On Sat, May 12, 2007 at 12:55:44AM -0400, Mike Meyer wrote: > In <20070511191057.GA25833@xor.obsecurity.org>, Kris Kennaway typed: > > > There are clearly other workable ideas - as I said, the linux folks > > > managed to make it work. But it's not an easy problem. I certainly > > > wouldn't suggest rebuilding the packaging system to deal with this, > > > except as part of a larger effort. On the other hand, since people are > > > working on the ports/package system (I see port/pkg database and some > > > ports infrastructure work in the current SoC projects list), not > > > keeping this goal in mind would seem to be a bit short-sighted. I > > > wouldn't be surprised if your option #1 could benefit from this as > > > well. > > But you said you were interested in working on it...so what is your > > idea? > > Actually, I said it's on my list of things to do. You might have meant this, but to quote what you actually said: -- But hey, if there's a document listing what needs to be done somewhere and it's really relatively minor, I need this bad enough to deal with that. -- I replied with an extremely minor way to solve the problem (adjust freebsd32 file lookups to check /compat/ia32 first), but apparently you don't really need it bad enough to actually go and solve the problem in a day when you could not-really-solve it in a year instead :) > That's a rather long > list. I'm more interested in port building, but the dependencies to > that one look like: > > 1) Get a gcc build whose driver correctly handles "-m32" for > cross-compiling. The system gcc can do cross-compilation, but the > driver doesn't know where the various bits it needs to link the > generated objects with live, so you have to tell it about those > things by hand. Possibly one of the versions in ports does this, > but it would be nice if the one in the base system did this. I > believe it's mostly a matter of properly configuring the gcc build. Yes, this would be nice. To a first approximation it's not needed though: you can either use the precompiled i386 packages, or build in your /compat/ia32 chroot (cf Gabor's work to automatically install ports inside a chroot, which would extend trivially to this case to work automatically). > 2) Rework the ports dependency/conflict detection facilities. Mostly, > figuring out the architecture of the current build vs. any > previously installed packages or ports it depends on or conflicts > with. There are also some things that could be done to make the > dependencies more reliable, like auto-detecting shared library > dependencies and adding those. Having the target platform for an > installed package in the package database would be a nice jump on > this. This was part of my "option 2" > 3) Now we get to the interesting part: figuring out where the bits for > the cross-installation are going to land. A lot depends on how far > apart we want to keep the two worlds. I'd like them to be > integrated, as the point of being able to install 32-bit binaries > on a 64-bit system is to run applications that aren't yet 32-bit > clean, so the only collisions should be shared libraries common to > applications from both sets. A lot will depend on how badly things > collide. The other factor is that I'd like the build platform to be > tranparent to the end user: you should be able to build 32-bit > packages on either 32 or 64 bit platforms, and use them on > either. This ties back to the "How do we figure out where our > libraries are" issue. I don't know if all these goals can be met. So was this. As I discussed, my experience suggests that it is unlikely this can be solved without imposing severe limitations on the packages that may be concurrently installed, so in practise I expect it to be an insufficiently general solution and not worth pursuing, as opposed to the trivially implemented, completely general "Option 1". Anyway, either you're interested in doing option 1 or you're not...if you are, I'll be happy to support your work and shepherd it into the tree. Otherwise, best of luck with that to-do list and we'll chat some more in a year or two :) Kris From owner-freebsd-hackers@FreeBSD.ORG Sat May 12 05:46:13 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id B209816A400 for ; Sat, 12 May 2007 05:46:13 +0000 (UTC) (envelope-from kientzle@freebsd.org) Received: from kientzle.com (h-66-166-149-50.snvacaid.covad.net [66.166.149.50]) by mx1.freebsd.org (Postfix) with ESMTP id 8E3F713C457 for ; Sat, 12 May 2007 05:46:13 +0000 (UTC) (envelope-from kientzle@freebsd.org) Received: from [10.0.0.222] (p54.kientzle.com [66.166.149.54]) by kientzle.com (8.12.9/8.12.9) with ESMTP id l4C5k6H7009206; Fri, 11 May 2007 22:46:11 -0700 (PDT) (envelope-from kientzle@freebsd.org) Message-ID: <4645549E.1000407@freebsd.org> Date: Fri, 11 May 2007 22:46:06 -0700 From: Tim Kientzle User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.7.12) Gecko/20060422 X-Accept-Language: en-us, en MIME-Version: 1.0 To: Joerg Sonnenberger References: <200705102105.27271.blackdragon@highveldmail.co.za> <4644847A.5060702@freebsd.org> <20070511153448.GA7516@britannica.bec.de> In-Reply-To: <20070511153448.GA7516@britannica.bec.de> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-hackers@freebsd.org Subject: Re: New FreeBSD package system (a.k.a. Daemon Package System (dps)) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 12 May 2007 05:46:13 -0000 >>3) As DES pointed out, the package tools must be able >> to read the metadata before the files. > > Actually, the argument is pretty weak. Being able to extract them > streamable and access the meta-data easily is fine. The remote access > argument is very weak as it doesn't allow e.g. signature checks. I presume you mean that you have to scan the entire package to verify the signature before doing installation? I don't think you do, really. If you can roll back an installation, then you can verify the signature during a streaming install; if the signature fails, you roll back. A good package installer needs to support rollback anyway to do robust dependency handling. I know two relatively straightforward ways to structure the installation process to support rollback. So many ideas, so little time... ;-) Tim Kientzle From owner-freebsd-hackers@FreeBSD.ORG Sat May 12 05:55:15 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id A64DE16A407 for ; Sat, 12 May 2007 05:55:15 +0000 (UTC) (envelope-from mwm-keyword-freebsdhackers2.e313df@mired.org) Received: from mired.org (vpn.mired.org [66.92.153.74]) by mx1.freebsd.org (Postfix) with SMTP id 57D7B13C483 for ; Sat, 12 May 2007 05:55:15 +0000 (UTC) (envelope-from mwm-keyword-freebsdhackers2.e313df@mired.org) Received: (qmail 8670 invoked by uid 1001); 12 May 2007 05:54:44 -0000 Received: by bhuda.mired.org (tmda-sendmail, from uid 1001); Sat, 12 May 2007 01:54:44 -0400 MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <17989.22179.773570.104272@bhuda.mired.org> Date: Sat, 12 May 2007 01:54:43 -0400 To: Kris Kennaway In-Reply-To: <20070512052038.GA57479@xor.obsecurity.org> References: <20070511015156.GA77895@xor.obsecurity.org> <17987.52970.398402.580727@bhuda.mired.org> <20070511021249.GA78729@xor.obsecurity.org> <17987.57305.7130.873114@bhuda.mired.org> <20070511051852.GA89359@xor.obsecurity.org> <17988.32573.910854.388638@bhuda.mired.org> <20070511184259.GA23483@xor.obsecurity.org> <17988.48879.828384.170557@bhuda.mired.org> <20070511191057.GA25833@xor.obsecurity.org> <17989.18640.420958.755412@bhuda.mired.org> <20070512052038.GA57479@xor.obsecurity.org> X-Mailer: VM 7.19 under Emacs 21.3.1 X-Primary-Address: mwm@mired.org X-face: "5Mnwy%?j>IIV\)A=):rjWL~NB2aH[}Yq8Z=u~vJ`"(,&SiLvbbz2W`; h9L,Yg`+vb1>RG% *h+%X^n0EZd>TM8_IB;a8F?(Fb"lw'IgCoyM.[Lg#r\ X-Delivery-Agent: TMDA/1.1.11 (Ladyburn) From: Mike Meyer Cc: freebsd-hackers@freebsd.org, Ivan Voras Subject: Re: New FreeBSD package system (a.k.a. Daemon Package System (dps)) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 12 May 2007 05:55:15 -0000 In <20070512052038.GA57479@xor.obsecurity.org>, Kris Kennaway typed: > > > But you said you were interested in working on it...so what is your > > > idea? > > Actually, I said it's on my list of things to do. > You might have meant this, but to quote what you actually said: > -- > But hey, if there's a document listing what needs to be done > somewhere and it's really relatively minor, I need this bad enough to > deal with that. > -- Ah, sorry - I thought you were referring to my comment when I first brought this up. Yeah, if there were a minor fix that gave me the integrated environment I want, that I'd do. > I replied with an extremely minor way to solve the problem (adjust > freebsd32 file lookups to check /compat/ia32 first), but apparently > you don't really need it bad enough to actually go and solve the > problem in a day when you could not-really-solve it in a year instead > :) There are already solutions using an emulated environment that don't require any work on my part. > > That's a rather long > > list. I'm more interested in port building, but the dependencies to > > that one look like: > > > > 1) Get a gcc build whose driver correctly handles "-m32" for > > cross-compiling. The system gcc can do cross-compilation, but the > > driver doesn't know where the various bits it needs to link the > > generated objects with live, so you have to tell it about those > > things by hand. Possibly one of the versions in ports does this, > > but it would be nice if the one in the base system did this. I > > believe it's mostly a matter of properly configuring the gcc build. > Yes, this would be nice. To a first approximation it's not needed > though: you can either use the precompiled i386 packages, or build in > your /compat/ia32 chroot (cf Gabor's work to automatically install > ports inside a chroot, which would extend trivially to this case to > work automatically). Actually, I need this more than I need 32 bit packages to install. > So was this. As I discussed, my experience suggests that it is > unlikely this can be solved without imposing severe limitations on the > packages that may be concurrently installed, so in practise I expect > it to be an insufficiently general solution and not worth pursuing, as > opposed to the trivially implemented, completely general "Option 1". You may well be right. On the other hand, the linux folks seem to have come up with something they thought was worthwhile, which to me indicates otherwise. > Anyway, either you're interested in doing option 1 or you're not...if > you are, I'll be happy to support your work and shepherd it into the > tree. Otherwise, best of luck with that to-do list and we'll chat > some more in a year or two :) Not interested in doing option 1. There are easier ways to get a segregated 32-bit environment. Yeah, it'd be a little bit better coupled, and some people might think that's worth spending a day or two on, but I'm not one of them. Personally, I think a year or two might be a bit optimistic. Thanks, http://www.mired.org/consulting.html Independent Network/Unix/Perforce consultant, email for more information. From owner-freebsd-hackers@FreeBSD.ORG Sat May 12 06:31:43 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 8C81016A402 for ; Sat, 12 May 2007 06:31:43 +0000 (UTC) (envelope-from jos@catnook.com) Received: from lizzy.dyndns.org (209-204-188-132.dsl.static.sonic.net [209.204.188.132]) by mx1.freebsd.org (Postfix) with SMTP id 2EE5C13C43E for ; Sat, 12 May 2007 06:31:43 +0000 (UTC) (envelope-from jos@catnook.com) Received: (qmail 58873 invoked by uid 1000); 12 May 2007 06:32:04 -0000 Date: Fri, 11 May 2007 23:32:04 -0700 From: Jos Backus To: freebsd-hackers@freebsd.org Message-ID: <20070512063204.GA58563@lizzy.catnook.local> Mail-Followup-To: freebsd-hackers@freebsd.org References: <200705102105.27271.blackdragon@highveldmail.co.za> <4643C7DB.6000408@elischer.org> <17988.35412.231093.411177@bhuda.mired.org> <20070511165612.GA48097@lizzy.catnook.local> <17988.43279.93248.608136@bhuda.mired.org> <20070511191612.GA50329@lizzy.catnook.local> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20070511191612.GA50329@lizzy.catnook.local> User-Agent: Mutt/1.5.15 (2007-04-06) Subject: Re: SQL in the base system (Was: New FreeBSD package system (a.k.a. Daemon Package System (dps))) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: jos@catnook.com List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 12 May 2007 06:31:43 -0000 On Fri, May 11, 2007 at 12:16:12PM -0700, Jos Backus wrote: > On Fri, May 11, 2007 at 01:34:07PM -0400, Mike Meyer wrote: [snip] > > > > How about portability - can I move the file to a completely > > > > different architecture and still get the data from it? [snip] The answer ("Yes") can be found in this thread: http://www.mail-archive.com/sqlite-users@sqlite.org/msg00120.html As another data point, `yum' has been using SQLite for quite some time now. -- Jos Backus jos at catnook.com From owner-freebsd-hackers@FreeBSD.ORG Sat May 12 11:28:15 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 99DCB16A408 for ; Sat, 12 May 2007 11:28:15 +0000 (UTC) (envelope-from freebsd-hackers@m.gmane.org) Received: from ciao.gmane.org (main.gmane.org [80.91.229.2]) by mx1.freebsd.org (Postfix) with ESMTP id 236B213C468 for ; Sat, 12 May 2007 11:28:14 +0000 (UTC) (envelope-from freebsd-hackers@m.gmane.org) Received: from list by ciao.gmane.org with local (Exim 4.43) id 1HmplY-00087p-1K for freebsd-hackers@freebsd.org; Sat, 12 May 2007 13:28:00 +0200 Received: from 89-172-56-80.adsl.net.t-com.hr ([89.172.56.80]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Sat, 12 May 2007 13:28:00 +0200 Received: from ivoras by 89-172-56-80.adsl.net.t-com.hr with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Sat, 12 May 2007 13:28:00 +0200 X-Injected-Via-Gmane: http://gmane.org/ To: freebsd-hackers@freebsd.org From: Ivan Voras Date: Sat, 12 May 2007 13:27:45 +0200 Lines: 106 Message-ID: References: <200705102105.27271.blackdragon@highveldmail.co.za> <46450EE1.300@hcl-club.lu> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="------------enigCFE6EDE01F8D722223639129" X-Complaints-To: usenet@sea.gmane.org X-Gmane-NNTP-Posting-Host: 89-172-56-80.adsl.net.t-com.hr User-Agent: Thunderbird 1.5.0.10 (Windows/20070221) In-Reply-To: <46450EE1.300@hcl-club.lu> X-Enigmail-Version: 0.94.3.0 Sender: news Subject: Re: New FreeBSD package system (a.k.a. Daemon Package System (dps)) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 12 May 2007 11:28:15 -0000 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enigCFE6EDE01F8D722223639129 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Jona Joachim wrote: > I don't think it would be a good idea to use SQLite for this purpose. > First of all using the file system is the Unix way of doing things. It'= s > efficient and easy to use, it transparent and user friendly. You can > simply run vi to inspect a text file but you can't do this which an > sqlite database. You have to learn sqlite to do it. That particular barrier to entry / learning curve is very shallow. In the end, it's a tradeoff between speed in the general case and ease of use (from the developer side), vs convenience in the extreme cases. > Furthermore I don't think the pkg_* tools are slow. They are quite fast= > IMO. If you let pkg_info print the entire list of installed ports it's > only slow because of your line-buffered console. Just redirect the > output to a file and you'll see that it's blazing fast. If I compare it= Err, nope. :) I see Eric has provided the numbers. > for example to Debians apt-get/apt-cache commands it's much faster. AFAIK Debian has the same dichotomy FreeBSD has: a tree of text files used by dpkg, and a binary database (cache) used by apt. > portupgrade is very slow, that's true. First of all it's written in Rub= y > which is not one of the fastest languages but there is another thing > that slows it down considerably, which is rebuilding its database. In my (limited) experience, this sort of task should not depend much on the speed of the language. The most CPU-intensive task portupgrade does is resolving dependencies, and on a running system this is a DAG forest of about 500 nodes. I know portupgrade has some highly unoptimal code in it (if I understand the code correctly, there's a brute force check for cyclic dependancies in it!), but still, in itself, I think the choice of Ruby isn't performance-critical. > Furthermore I think it would be a very bad idea to include sqlite in > base. There is already a lot of third party stuff in base. The > philosophy of the BSDs is to provide and maintain an entire OS. This is= > quite the opposite of how a GNU/Linux system is designed. Both ways hav= e > their pros and cons. An advantage of the BSD way of doing things is tha= t > the developers know the code very well and have control over the qualit= y > of the code. If you include 3rd party software into the FreeBSD base > system you make the FreeBSD project depend on the people that wrote tha= t > code. Of course you could fork it but the FreeBSD developers are not > necessarily familiar with the code. Security patches would have to be > merged all the time and a lot of communication between the two projects= > is needed. I think this line of reasoning was made invalid by the continuing inclusion of sendmail, bind, expat (xml parser!), etc. Not that I don't realize this increases the burden on maintainability, but including a "frozen" branch of a library, which is supported, but won't be changed for ages isn't going to increase it much. Offloading much of the "smarts" to a database would also permit easier reimplementation of portupgrade-like tool in C, since the heavy parsing / regex facilities scripting languages offer won't be used as much. But yes, it's a heavy departure from "the unix way". > I think the best way to go would be to use only folder hierarchies and > text files and write a libary in C that provides portupgrade > functionality. The code under src/usr.sbin/pkg_install/lib/ would be a > good base for this. Then you could use a frontend program that makes us= e > of this library. This frontend could be a CLI program or a GUI based > program. The issue in this thread (at least for me) is performance and reliability, and creating a C wrapper around the current situation won't solve neither. --------------enigCFE6EDE01F8D722223639129 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.4 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFGRaS3ldnAQVacBcgRAsXCAKCLDd+4As/BZ7ArsIhQkjSw/Pqz/ACgyHXT VOfLZaByz26fAic+QVs52Mk= =kY0l -----END PGP SIGNATURE----- --------------enigCFE6EDE01F8D722223639129-- From owner-freebsd-hackers@FreeBSD.ORG Sat May 12 11:52:14 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 8FFE916A402 for ; Sat, 12 May 2007 11:52:14 +0000 (UTC) (envelope-from stas@FreeBSD.org) Received: from com1.ht-systems.ru (com1.ht-systems.ru [83.97.104.204]) by mx1.freebsd.org (Postfix) with ESMTP id 14C8D13C45A for ; Sat, 12 May 2007 11:52:14 +0000 (UTC) (envelope-from stas@FreeBSD.org) Received: from [83.97.106.70] (helo=phonon.SpringDaemons.com ident=postfix) by com1.ht-systems.ru with esmtpa (Exim 4.62) (envelope-from ) id 1Hmq8y-0007PG-5D; Sat, 12 May 2007 15:52:12 +0400 Received: from localhost (localhost [127.0.0.1]) by phonon.SpringDaemons.com (Postfix) with SMTP id 3506611403; Sat, 12 May 2007 15:50:59 +0400 (MSD) Date: Sat, 12 May 2007 15:50:59 +0400 From: Stanislav Sedov To: Ivan Voras Message-Id: <20070512155059.92011d54.stas@FreeBSD.org> In-Reply-To: References: <200705102105.27271.blackdragon@highveldmail.co.za> Organization: The FreeBSD Project X-Mailer: carrier-pigeon X-Voice: +7 916 849 20 23 X-XMPP: ssedov@jabber.ru X-ICQ: 208105021 X-Yahoo: stanislav_sedov X-PGP-Fingerprint: F21E D6CC 5626 9609 6CE2 A385 2BF5 5993 EB26 9581 X-University: MEPhI Mime-Version: 1.0 Content-Type: multipart/signed; protocol="application/pgp-signature"; micalg="PGP-SHA1"; boundary="Signature=_Sat__12_May_2007_15_50_59_+0400_W_8kc=3K=KF8i/xv" X-Spam-Flag: SKIP Cc: freebsd-hackers@freebsd.org Subject: Re: New FreeBSD package system (a.k.a. Daemon Package System (dps)) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 12 May 2007 11:52:14 -0000 --Signature=_Sat__12_May_2007_15_50_59_+0400_W_8kc=3K=KF8i/xv Content-Type: text/plain; charset=US-ASCII Content-Disposition: inline Content-Transfer-Encoding: 7bit On Fri, 11 May 2007 02:10:05 +0200 Ivan Voras mentioned: > - I think it's time to give up on using BDB+directory tree full of text > files for storing the installed packages database, and I propose all of > this be replaced by a single SQLite database. SQLite is public domain > (can be slurped into base system), embeddable, stores all data in a > single file, lightweight, fast, and can be used to do fancy things such > as reporting. What is the reason to use SQL-based database? You'll perform direct queries to database? The packaging system is for ordinal users, not sql geeks, so they should not have to use sql for managing packages. So a simple set of hashes will suffer or needs. I agree with Julian that we should have a backup of packaging database in plain text format, and utility to rebuild it. This way we can always restore the database if something goes wrong. Furhtermore, that should not make a great impact on performance, since we don't have to rebuild it every day. > > - A quick test confirms that the current bsdtar will happily ignore any > extra data at the end of a tgz/tbz archive, so package metadata can be > embedded there, thus conserving existing infrastructure and being fast > to parse. I suggest encoding this metadata in a sane and easy to parse > XML structure. > > I cannot currently actively participate in implementing proposed things, > but I can give advice on sqlite, database and xml schemas if anyone > wants to... > Why use XML for that? It's hard to parse and hard to read format, and I personally see no benefits of using it. If you're suggesting XML a simple bracket-structure format (like bind's config) will fit our needs much better (easier to parse and read and same benefits as XML). Also we might consider YAML, thought I like this idea much fewer. -- Stanislav Sedov ST4096-RIPE --Signature=_Sat__12_May_2007_15_50_59_+0400_W_8kc=3K=KF8i/xv Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (FreeBSD) iD8DBQFGRaojK/VZk+smlYERAn+nAJ0YUywP6wkswNr5GUMbBi5Qqk7nAgCfSxJE phiRoCqrRayKR4TgokUXbts= =l779 -----END PGP SIGNATURE----- --Signature=_Sat__12_May_2007_15_50_59_+0400_W_8kc=3K=KF8i/xv-- From owner-freebsd-hackers@FreeBSD.ORG Sat May 12 09:09:40 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 1B58216A405 for ; Sat, 12 May 2007 09:09:40 +0000 (UTC) (envelope-from michel@lpthe.jussieu.fr) Received: from shiva.jussieu.fr (shiva.jussieu.fr [134.157.0.129]) by mx1.freebsd.org (Postfix) with ESMTP id BDF4F13C43E for ; Sat, 12 May 2007 09:09:39 +0000 (UTC) (envelope-from michel@lpthe.jussieu.fr) Received: from parthe.lpthe.jussieu.fr (parthe.lpthe.jussieu.fr [134.157.10.1]) by shiva.jussieu.fr (8.13.8/jtpda-5.4) with ESMTP id l4C99bIx065888 ; Sat, 12 May 2007 11:09:37 +0200 (CEST) X-Ids: 165 Received: by parthe.lpthe.jussieu.fr (Postfix, from userid 10096) id 92459BF6A0; Sat, 12 May 2007 11:09:36 +0200 (CEST) X-Spam-Checker-Version: SpamAssassin 3.1.8 (2007-02-13) on parthe.lpthe.jussieu.fr X-Spam-Level: X-Spam-Status: No, score=-2.1 required=5.0 tests=AWL,BAYES_00 autolearn=ham version=3.1.8 Received: from niobe.lpthe.jussieu.fr (niobe.lpthe.jussieu.fr [134.157.10.41]) by parthe.lpthe.jussieu.fr (Postfix) with ESMTP id A1400BF60E; Sat, 12 May 2007 11:09:35 +0200 (CEST) Received: by niobe.lpthe.jussieu.fr (Postfix, from userid 2005) id 7E1E688; Sat, 12 May 2007 11:09:35 +0200 (CEST) Date: Sat, 12 May 2007 11:09:35 +0200 From: Michel Talon To: Mike Meyer Message-ID: <20070512090935.GA13929@lpthe.jussieu.fr> References: <20070512004209.GA12218@lpthe.jussieu.fr> <17989.8202.624522.136573@bhuda.mired.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <17989.8202.624522.136573@bhuda.mired.org> User-Agent: Mutt/1.4.2.2i X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-3.0 (shiva.jussieu.fr [134.157.0.165]); Sat, 12 May 2007 11:09:38 +0200 (CEST) X-Virus-Scanned: ClamAV 0.88.7/3229/Sat May 12 07:59:29 2007 on shiva.jussieu.fr X-Virus-Status: Clean X-j-chkmail-Score: MSGID : 46458451.003 on shiva.jussieu.fr : j-chkmail score : X : 0/50 0 0.504 -> 1 X-Miltered: at shiva.jussieu.fr with ID 46458451.003 by Joe's j-chkmail (http://j-chkmail.ensmp.fr)! X-Mailman-Approved-At: Sat, 12 May 2007 12:01:08 +0000 Cc: freebsd-hackers@freebsd.org Subject: Re: DPS Initial Ideas X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 12 May 2007 09:09:40 -0000 On Fri, May 11, 2007 at 10:01:46PM -0400, Mike Meyer wrote: > In <20070512004209.GA12218@lpthe.jussieu.fr>, Michel Talon typed: > > One of the most obvious being that the sqlite database can be edited > > as easily as a pure textfile using the sqlite3 program > > Huh? They can? With a pure textfile, if vi is busted, I can use ed. If > ed is also busted, I can use sed. What do I use on an sqlite database > if sqlite3 is busted? Answering both you and Bill Moran: - first i don't suppose sqlite3 is busted, since i suppose it is in the base system and it works by definition. Your hypothesis is alike, what do i do to edit my config files if vi and ed are busted? Moreover if sqlite3 gets really busted i can import a copy and hope it works, it requires very few libraries and other files, not much more than vi, plus the sqlite3 library, of course. The combined size of sqlite3 and libsqlite3 is less than 400k. - second, if i am sql allergic, it takes one command to export the table to a straight file, each row in a line, each field separated by | or anything else of my choice. Exactly the same tools that you have mentioned allow to edit this file, and then one command allows to load it in the database. - so what are the benefits? They are that non sql impaired people can make good use of the power of sql queries to simplify their work. And this without reducing the possibilities of sql impaired people. Moreover one can use general tools like graphic sql tools to present the contents of the database to the end user in a pleasant way if it is desired. And finally it may be that the transactional properties of sqlite can be used to gain better reliablity. - is the cost of including sqlite in the base system so high that the above benefits are insufficient? Personnally i don't know, but i think some discussion is at least in order. - and finally to answer one of Bill's critiques, why sqlite rather than a Berkeley database? Precisely because sqlite offer a lot of facilities that Berkeley db doesn't offer, such as export and import to and from csv files, auto documentation of the table contents, while it requires in fact programming and knowledge of the api of the database to hand edit the Berkeley db. Anyways, i have read that Marc Espie is envisioning using sqlite3 for OpenBSD package system, and that he is very satisfied with what he has seen up to now. If this enters production, perhaps this will confer BSD legitimity to such practices ... Seriously, the FreeBSD package system is in great need of a profound overhaul, pretending it works well is complete denial of reality. I hope that young people working on summer code projects will infuse *new* ideas, and not spend their vacations polishing inadequate tools. -- Michel TALON From owner-freebsd-hackers@FreeBSD.ORG Sat May 12 10:40:18 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 2706D16A403 for ; Sat, 12 May 2007 10:40:18 +0000 (UTC) (envelope-from andy.kosela@gmail.com) Received: from an-out-0708.google.com (an-out-0708.google.com [209.85.132.250]) by mx1.freebsd.org (Postfix) with ESMTP id D899A13C448 for ; Sat, 12 May 2007 10:40:17 +0000 (UTC) (envelope-from andy.kosela@gmail.com) Received: by an-out-0708.google.com with SMTP id d23so286776and for ; Sat, 12 May 2007 03:40:17 -0700 (PDT) DKIM-Signature: a=rsa-sha1; c=relaxed/relaxed; d=gmail.com; s=beta; h=domainkey-signature:received:received:message-id:date:from:to:subject:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; b=O8n757OFJRiOy/sPXb/4UpkFCYn9gt8MmfYF1kMavWI8lHLfY7Kwog8l0clH3ef0U+yWF9WnHab6dfZWFf45eDTPtDmBcMxDP6dDOHrkhfQy7CuPEYiVtV16nUh51K+edI1EQxKMxJtj6GcP+AwG8pAnlGYZsw1DTafbbSj6br8= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=beta; h=received:message-id:date:from:to:subject:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; b=cHbcNZmyT7K2OrGeiW41UsV4BIWDX86Y414oi6U3hEjrBawz2xlS6gVRxuaCYN3xGrUJiMhobzI35CZOPyIhUV+DcBQ3n+B6cgrLb1JtKVMJ5AfTtualg34meKwqg7C/R6TYrDf5jQH4PBWSGht6Nl2jXN9nBlDYU7LIGg5c/Ic= Received: by 10.114.126.1 with SMTP id y1mr61103wac.1178964928630; Sat, 12 May 2007 03:15:28 -0700 (PDT) Received: by 10.114.12.17 with HTTP; Sat, 12 May 2007 03:15:28 -0700 (PDT) Message-ID: <3cc535c80705120315j25be1113k933ec440d76a14e4@mail.gmail.com> Date: Sat, 12 May 2007 12:15:28 +0200 From: "Andy Kosela" To: freebsd-hackers@freebsd.org In-Reply-To: <1178963343.591874.231130@o5g2000hsb.googlegroups.com> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <1178963343.591874.231130@o5g2000hsb.googlegroups.com> X-Mailman-Approved-At: Sat, 12 May 2007 12:01:08 +0000 Subject: [NEW PORT] ports-mgmt/pkg - smart tool for managing FreeBSD ports X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 12 May 2007 10:40:18 -0000 Hi all, I would like to present to you the new utility to deal with the ports system. The main goal of this project is to provide one common tool for managing ports and packages instead of relying on many applications (pkg_add, pkg_delete, pkg_info, pkg_version etc.). Actually it is a smart wrapper written in /bin/sh to the previously mentioned applications. It also uses external tool portmaster written also in /bin/sh by Doug Barton to work with the ports compiled from source. Pkg tool automates upgrading installed packages, outputs valuable information about packages/ports and overall simplifies working with the FreeBSD Ports Collection. It uses no external databases like portupgrade, just simplicity and minimalism are its main goals. You can test the latest version by installing the package from here http://home.si.rr.com/pyn/pf/pkg-1.1.tbz I commited pkg-1.0 with send-pr to the ports tree a few days ago. It is awaiting approval... http://www.freebsd.org/cgi/query-pr.cgi?pr=ports/112572 Feel free to send any suggestions, new ideas and of course bug reports... Thank you, Andy Kosela Pythagoras Foundation From owner-freebsd-hackers@FreeBSD.ORG Sat May 12 12:06:45 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id A204916A403 for ; Sat, 12 May 2007 12:06:45 +0000 (UTC) (envelope-from rdivacky@vlk.vlakno.cz) Received: from vlakno.cz (vlk.vlakno.cz [62.168.28.247]) by mx1.freebsd.org (Postfix) with ESMTP id 5EEF913C45E for ; Sat, 12 May 2007 12:06:45 +0000 (UTC) (envelope-from rdivacky@vlk.vlakno.cz) Received: from localhost (localhost [127.0.0.1]) by vlakno.cz (Postfix) with ESMTP id 6B38D8BD57D; Sat, 12 May 2007 14:06:44 +0200 (CEST) X-Virus-Scanned: amavisd-new at vlakno.cz Received: from vlakno.cz ([127.0.0.1]) by localhost (vlk.vlakno.cz [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 7kuvIEK3QDvC; Sat, 12 May 2007 14:06:43 +0200 (CEST) Received: from vlk.vlakno.cz (localhost [127.0.0.1]) by vlakno.cz (Postfix) with ESMTP id ABD968BD55D; Sat, 12 May 2007 14:06:43 +0200 (CEST) Received: (from rdivacky@localhost) by vlk.vlakno.cz (8.13.8/8.13.8/Submit) id l4CC6hIW034843; Sat, 12 May 2007 14:06:43 +0200 (CEST) (envelope-from rdivacky) Date: Sat, 12 May 2007 14:06:43 +0200 From: Roman Divacky To: Ivan Voras Message-ID: <20070512120643.GA34770@freebsd.org> References: <200705102105.27271.blackdragon@highveldmail.co.za> <46450EE1.300@hcl-club.lu> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.4.2.2i Cc: freebsd-hackers@freebsd.org Subject: Re: New FreeBSD package system (a.k.a. Daemon Package System (dps)) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 12 May 2007 12:06:45 -0000 > cyclic dependancies in it!), but still, in itself, I think the choice of > Ruby isn't performance-critical. ruby2.0 will come with a virtual machine which should speed up things. ruby2.0 is expected "soon enough" (2008?) From owner-freebsd-hackers@FreeBSD.ORG Sat May 12 12:33:06 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 610F516A400; Sat, 12 May 2007 12:33:06 +0000 (UTC) (envelope-from stom@free.fr) Received: from postfix1-g20.free.fr (postfix1-g20.free.fr [212.27.60.42]) by mx1.freebsd.org (Postfix) with ESMTP id ED0C313C469; Sat, 12 May 2007 12:33:05 +0000 (UTC) (envelope-from stom@free.fr) Received: from smtp2-g19.free.fr (smtp2-g19.free.fr [212.27.42.28]) by postfix1-g20.free.fr (Postfix) with ESMTP id EC29FF54F24; Sat, 12 May 2007 14:15:08 +0200 (CEST) Received: from btn.mine.nu (tok69-1-82-67-36-224.fbx.proxad.net [82.67.36.224]) by smtp2-g19.free.fr (Postfix) with ESMTP id 5C72496E51; Sat, 12 May 2007 14:14:47 +0200 (CEST) Received: from localhost (localhost.localdomain [127.0.0.1]) by btn.mine.nu (Postfix) with ESMTP id C769A47CC3; Sat, 12 May 2007 13:12:34 +0200 (CEST) Received: from btn.mine.nu ([127.0.0.1]) by localhost (btn.mine.nu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id GOLCyecGFi5S; Sat, 12 May 2007 13:12:31 +0200 (CEST) Received: from [192.168.30.10] (88-137-169-58.adslgp.cegetel.net [88.137.169.58]) by btn.mine.nu (Postfix) with ESMTP id F2B7147CC0; Sat, 12 May 2007 13:12:29 +0200 (CEST) Message-ID: <4645AFAF.7010704@free.fr> Date: Sat, 12 May 2007 14:14:39 +0200 From: Philippe Laquet User-Agent: Thunderbird 1.5.0.10 (Windows/20070221) MIME-Version: 1.0 To: Stanislav Sedov References: <200705102105.27271.blackdragon@highveldmail.co.za> <20070512155059.92011d54.stas@FreeBSD.org> In-Reply-To: <20070512155059.92011d54.stas@FreeBSD.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 8bit Cc: freebsd-hackers@freebsd.org, Ivan Voras Subject: Re: New FreeBSD package system (a.k.a. Daemon Package System (dps)) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 12 May 2007 12:33:06 -0000 Stanislav Sedov a écrit : > On Fri, 11 May 2007 02:10:05 +0200 > Ivan Voras mentioned: > > >> - I think it's time to give up on using BDB+directory tree full of text >> files for storing the installed packages database, and I propose all of >> this be replaced by a single SQLite database. SQLite is public domain >> (can be slurped into base system), embeddable, stores all data in a >> single file, lightweight, fast, and can be used to do fancy things such >> as reporting. >> > > What is the reason to use SQL-based database? You'll perform direct > queries to database? The packaging system is for ordinal users, not sql > geeks, so they should not have to use sql for managing packages. So a > simple set of hashes will suffer or needs. I agree with Julian that we > should have a backup of packaging database in plain text format, and > utility to rebuild it. This way we can always restore the database if > something goes wrong. Furhtermore, that should not make a great impact > on performance, since we don't have to rebuild it every day. > I agree with Stan ;) "fast and improved" package utilities uses mainly some indexed berkeley DB combined with flat files, aren't they? I, and may be many other FreeBSD users use light systems for efficiency and eaiser management, if we use some database system it will require Disk Space, ressources for the DB to run, dependencies and so on... And we also may be exposed to a "that DB is better" war ;) > >> - A quick test confirms that the current bsdtar will happily ignore any >> extra data at the end of a tgz/tbz archive, so package metadata can be >> embedded there, thus conserving existing infrastructure and being fast >> to parse. I suggest encoding this metadata in a sane and easy to parse >> XML structure. >> >> I cannot currently actively participate in implementing proposed things, >> but I can give advice on sqlite, database and xml schemas if anyone >> wants to... >> >> > > Why use XML for that? It's hard to parse and hard to read format, and I > personally see no benefits of using it. If you're suggesting XML a > simple bracket-structure format (like bind's config) will fit our needs > much better (easier to parse and read and same benefits as XML). Also > we might consider YAML, thought I like this idea much fewer. > XML could be an altertative to order packages, it can be parsed with some limited dependencies like PERL. The userland tools to manage packages could be based on that language? It is well known by many users, quite simple, required by many other packages so the whole system won't be much heavier. PERL XML Parser can't be a good choice? * PERL-DB for managing packages databases * PERL-XML for parsing categories, dependencies ... PERL also give , in most cases, good performance issues. This is solely ma humble opinion ;) > -- > Stanislav Sedov > ST4096-RIPE > From owner-freebsd-hackers@FreeBSD.ORG Sat May 12 12:37:48 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 3559316A404 for ; Sat, 12 May 2007 12:37:48 +0000 (UTC) (envelope-from michel@lpthe.jussieu.fr) Received: from shiva.jussieu.fr (shiva.jussieu.fr [134.157.0.129]) by mx1.freebsd.org (Postfix) with ESMTP id D92FC13C447 for ; Sat, 12 May 2007 12:37:47 +0000 (UTC) (envelope-from michel@lpthe.jussieu.fr) Received: from parthe.lpthe.jussieu.fr (parthe.lpthe.jussieu.fr [134.157.10.1]) by shiva.jussieu.fr (8.13.8/jtpda-5.4) with ESMTP id l4CCbk9v092074 for ; Sat, 12 May 2007 14:37:46 +0200 (CEST) X-Ids: 165 Received: by parthe.lpthe.jussieu.fr (Postfix, from userid 10096) id 62246BF6A0; Sat, 12 May 2007 14:37:45 +0200 (CEST) X-Spam-Checker-Version: SpamAssassin 3.1.8 (2007-02-13) on parthe.lpthe.jussieu.fr X-Spam-Level: X-Spam-Status: No, score=-2.2 required=5.0 tests=AWL,BAYES_00 autolearn=ham version=3.1.8 Received: from niobe.lpthe.jussieu.fr (niobe.lpthe.jussieu.fr [134.157.10.41]) by parthe.lpthe.jussieu.fr (Postfix) with ESMTP id 7B88BBF621 for ; Sat, 12 May 2007 14:37:44 +0200 (CEST) Received: by niobe.lpthe.jussieu.fr (Postfix, from userid 2005) id 66F8588; Sat, 12 May 2007 14:37:44 +0200 (CEST) Date: Sat, 12 May 2007 14:37:44 +0200 From: Michel Talon To: freebsd-hackers@freebsd.org Message-ID: <20070512123744.GA14586@lpthe.jussieu.fr> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.4.2.2i X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-3.0 (shiva.jussieu.fr [134.157.0.165]); Sat, 12 May 2007 14:37:46 +0200 (CEST) X-Virus-Scanned: ClamAV 0.88.7/3230/Sat May 12 12:34:18 2007 on shiva.jussieu.fr X-Virus-Status: Clean X-Miltered: at shiva.jussieu.fr with ID 4645B51A.000 by Joe's j-chkmail (http://j-chkmail.ensmp.fr)! X-Mailman-Approved-At: Sat, 12 May 2007 12:41:46 +0000 Subject: Re: New FreeBSD package system (a.k.a. Daemon Package System (dps)) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 12 May 2007 12:37:48 -0000 Ivan Voras wrote: > In my (limited) experience, this sort of task should not depend much on > the speed of the language. The most CPU-intensive task portupgrade does > is resolving dependencies, and on a running system this is a DAG forest > of about 500 nodes. I know portupgrade has some highly unoptimal code in > it (if I understand the code correctly, there's a brute force check for > cyclic dependancies in it!), but still, in itself, I think the choice of > Ruby isn't performance-critical. If i remember well, portupgrade uses a reasonable algorithm to sort the DAG in question. But from time to time, it runs make depends or something of the sort in some ports directory, and this is the slow step which kills any package manager whatsoever, be it written in super fast C or superslow ruby. This is because as soon as you run "make" in such a directory it has to read and interpret the 4000 lines file bsd.ports.mk. This takes 1/10 s on my old laptop and perhaps 5 times faster on a powerful machine. Add to that the natural slowness of ruby, the fact that portupgrade constantly does queries in the Berkeley pkgdb.db, portsdb.db etc. while it could cache the whole content in memory without any problem in modern machines, and you have something which is slow as a mollasse. Then you have more structural problems such as the "guessing facilities" of portupgrade, which aim at guessing what is the new origin of a port whose old origin has disappeared. To do that it counts similarities on names, adds some snake oil and other satanic ingredients and comes out with a guess. For some time i was impressed by the exactness of these guesses, but recently i have seen incredibly hilarious results. As a consequence i think that portupgrade is a completely inadequate tool to maintain a FreeBSD machine in an automated way in spite of the remarkable insight which is coded in it, and that Debian like systems run circles around FreeBSD for ease of maintenance. Yes the FreeBSD system is very good for initial installation of software and the flexibility it allows to do that. But it sucks completely as soon as one wants to upgrade stuff. The best way by far is wipe out every port and reinstall. This will take 1/3 of the time to run portupgrade -a and with a much better chance of success and coherency at the end. Note that these innocent looking facts have little consequences: FreeBSD was in position 11 on Distrowatch in 2005, it is now in position 17 and falling like a brick. -- Michel TALON From owner-freebsd-hackers@FreeBSD.ORG Sat May 12 12:52:08 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id C264316A400 for ; Sat, 12 May 2007 12:52:08 +0000 (UTC) (envelope-from stas@FreeBSD.org) Received: from com1.ht-systems.ru (com1.ht-systems.ru [83.97.104.204]) by mx1.freebsd.org (Postfix) with ESMTP id 783C913C468 for ; Sat, 12 May 2007 12:52:08 +0000 (UTC) (envelope-from stas@FreeBSD.org) Received: from [83.97.106.70] (helo=phonon.SpringDaemons.com ident=postfix) by com1.ht-systems.ru with esmtpa (Exim 4.62) (envelope-from ) id 1Hmr4w-0000ic-Kc; Sat, 12 May 2007 16:52:06 +0400 Received: from localhost (localhost [127.0.0.1]) by phonon.SpringDaemons.com (Postfix) with SMTP id B8CD011403; Sat, 12 May 2007 16:50:53 +0400 (MSD) Date: Sat, 12 May 2007 16:50:53 +0400 From: Stanislav Sedov To: Philippe Laquet Message-Id: <20070512165053.8e53f5a3.stas@FreeBSD.org> In-Reply-To: <4645AFAF.7010704@free.fr> References: <200705102105.27271.blackdragon@highveldmail.co.za> <20070512155059.92011d54.stas@FreeBSD.org> <4645AFAF.7010704@free.fr> Organization: The FreeBSD Project X-Mailer: carrier-pigeon X-Voice: +7 916 849 20 23 X-XMPP: ssedov@jabber.ru X-ICQ: 208105021 X-Yahoo: stanislav_sedov X-PGP-Fingerprint: F21E D6CC 5626 9609 6CE2 A385 2BF5 5993 EB26 9581 X-University: MEPhI Mime-Version: 1.0 Content-Type: multipart/signed; protocol="application/pgp-signature"; micalg="PGP-SHA1"; boundary="Signature=_Sat__12_May_2007_16_50_53_+0400_feIP48SCkXX.73fL" X-Spam-Flag: SKIP Cc: freebsd-hackers@freebsd.org, Ivan Voras Subject: Re: New FreeBSD package system (a.k.a. Daemon Package System (dps)) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 12 May 2007 12:52:08 -0000 --Signature=_Sat__12_May_2007_16_50_53_+0400_feIP48SCkXX.73fL Content-Type: text/plain; charset=US-ASCII Content-Disposition: inline Content-Transfer-Encoding: 7bit On Sat, 12 May 2007 14:14:39 +0200 Philippe Laquet mentioned: > > > XML could be an altertative to order packages, it can be parsed with > some limited dependencies like PERL. The userland tools to manage > packages could be based on that language? It is well known by many > users, quite simple, required by many other packages so the whole system > won't be much heavier. PERL XML Parser can't be a good choice? > > * PERL-DB for managing packages databases > * PERL-XML for parsing categories, dependencies ... > > PERL also give , in most cases, good performance issues. > I agree, that there's a lot of ready tools for parsing xml, but why not use much simple language that can be parsed by sed or awk in few lines? Will not require dependecies at all and much simplier (read better). The entire FreeBSD's ideology is to be as simple as possible but powerful. -- Stanislav Sedov ST4096-RIPE --Signature=_Sat__12_May_2007_16_50_53_+0400_feIP48SCkXX.73fL Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (FreeBSD) iD8DBQFGRbgtK/VZk+smlYERAtLmAJ9RtK5oETBR0ZIucNYKoNBeJLpaawCff3po 9OHyqsGVgfBS+mF2ku0f7xM= =3PLh -----END PGP SIGNATURE----- --Signature=_Sat__12_May_2007_16_50_53_+0400_feIP48SCkXX.73fL-- From owner-freebsd-hackers@FreeBSD.ORG Sat May 12 13:11:27 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id A7B6316A409 for ; Sat, 12 May 2007 13:11:27 +0000 (UTC) (envelope-from peterjeremy@optushome.com.au) Received: from turion.vk2pj.dyndns.org (c220-239-3-125.belrs4.nsw.optusnet.com.au [220.239.3.125]) by mx1.freebsd.org (Postfix) with ESMTP id 1E6B313C469 for ; Sat, 12 May 2007 13:11:26 +0000 (UTC) (envelope-from peterjeremy@optushome.com.au) Received: from turion.vk2pj.dyndns.org (localhost.vk2pj.dyndns.org [127.0.0.1]) by turion.vk2pj.dyndns.org (8.14.1/8.14.1) with ESMTP id l4CDBP4t015921; Sat, 12 May 2007 23:11:25 +1000 (EST) (envelope-from peter@turion.vk2pj.dyndns.org) Received: (from peter@localhost) by turion.vk2pj.dyndns.org (8.14.1/8.14.1/Submit) id l4CDBOIX015770; Sat, 12 May 2007 23:11:24 +1000 (EST) (envelope-from peter) Date: Sat, 12 May 2007 23:11:24 +1000 From: Peter Jeremy To: Michel Talon Message-ID: <20070512131124.GK10647@turion.vk2pj.dyndns.org> References: <20070512004209.GA12218@lpthe.jussieu.fr> <17989.8202.624522.136573@bhuda.mired.org> <20070512090935.GA13929@lpthe.jussieu.fr> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="17/8oYur5Y32USnW" Content-Disposition: inline In-Reply-To: <20070512090935.GA13929@lpthe.jussieu.fr> X-PGP-Key: http://members.optusnet.com.au/peterjeremy/pubkey.asc User-Agent: Mutt/1.5.14 (2007-02-12) Cc: freebsd-hackers@freebsd.org, Mike Meyer Subject: Re: DPS Initial Ideas X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 12 May 2007 13:11:27 -0000 --17/8oYur5Y32USnW Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On 2007-May-12 11:09:35 +0200, Michel Talon wrote: >- first i don't suppose sqlite3 is busted, since i suppose it is in the > base system and it works by definition. It can happen that base system utilities become unusable for various reasons: Maybe an installworld went wrong, maybe someone accidently deleted a shared library, maybe the disk developed an inconvenient bad sector. If sqlite3 is being used solely for ports management then it may be a reasonable assumption that sqlite3 is always working but if (as has been suggested) SQLite is used for base system config files then it is essential that those files be repairable. > Your hypothesis is alike, what >do i do to edit my config files if vi and ed are busted? Use emacs :-) Seriously, you can probably get away with sh builtins for most purposes: while read line do case "$line" in foo*) echo bar ;; *) echo "$line" ;; esac done < file > file.tmp mv file.tmp file > Moreover if >sqlite3 gets really busted i can import a copy and hope it works, I agree that sqlite3 is good in this respect. "Import" implies that you are able to get the system to a point where it can communicate externally without needing whatever tool is broken. >- second, if i am sql allergic, it takes one command to export the table > to a straight file, This is only usable if the schema is designed so that the tables are reasonably independent. It's certainly possible to design something that could not be usefully exported table-by-table and edited. >- is the cost of including sqlite in the base system so high that =20 >the above benefits are insufficient? Personnally i don't know, but i >think some discussion is at least in order. I agree that this topic is worth discussing. There is a very high bar for including new utilities in the base system because every time a new utility is added, the maintenance effort goes up. So far I haven't seen anything that would make me say "SQLite should be imported". >- and finally to answer one of Bill's critiques, why sqlite rather than=20 >a Berkeley database? Precisely because sqlite offer a lot of facilities=20 >that Berkeley db doesn't offer, such as export and import to and from=20 >csv files, This is a function of sqlite3(1) rather than the SQLite database itself. It wouldn't be that difficult to write a tool to convert a BDB into a flat file of key,value pairs. > auto documentation of the table contents, This is a big plus for SQL. > while it requires >in fact programming and knowledge of the api of the database to hand >edit the Berkeley db. Very trivial effort - if we had a need for it, someone could write the necessary few dozen lines and commit it. The downside is that since BDB isn't self documenting, a flat file may not be any use. --=20 Peter Jeremy --17/8oYur5Y32USnW Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (FreeBSD) iD8DBQFGRbz8/opHv/APuIcRArL0AKCPiCls9zKufdAnHUMW1otohm6LJACeKA16 YFJS0ZbyuI+1nnAZK0Ql7v4= =11L+ -----END PGP SIGNATURE----- --17/8oYur5Y32USnW-- From owner-freebsd-hackers@FreeBSD.ORG Sat May 12 14:25:49 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 269A016A400 for ; Sat, 12 May 2007 14:25:49 +0000 (UTC) (envelope-from freebsd-hackers@m.gmane.org) Received: from ciao.gmane.org (main.gmane.org [80.91.229.2]) by mx1.freebsd.org (Postfix) with ESMTP id A687813C447 for ; Sat, 12 May 2007 14:25:48 +0000 (UTC) (envelope-from freebsd-hackers@m.gmane.org) Received: from list by ciao.gmane.org with local (Exim 4.43) id 1HmsXJ-0000SF-Qd for freebsd-hackers@freebsd.org; Sat, 12 May 2007 16:25:29 +0200 Received: from 89-172-56-80.adsl.net.t-com.hr ([89.172.56.80]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Sat, 12 May 2007 16:25:29 +0200 Received: from ivoras by 89-172-56-80.adsl.net.t-com.hr with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Sat, 12 May 2007 16:25:29 +0200 X-Injected-Via-Gmane: http://gmane.org/ To: freebsd-hackers@freebsd.org From: Ivan Voras Date: Sat, 12 May 2007 16:25:06 +0200 Lines: 60 Message-ID: References: <200705102105.27271.blackdragon@highveldmail.co.za> <20070512155059.92011d54.stas@FreeBSD.org> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="------------enigBB992BE5955A3EEDFBEF67F3" X-Complaints-To: usenet@sea.gmane.org X-Gmane-NNTP-Posting-Host: 89-172-56-80.adsl.net.t-com.hr User-Agent: Thunderbird 1.5.0.10 (Windows/20070221) In-Reply-To: <20070512155059.92011d54.stas@FreeBSD.org> X-Enigmail-Version: 0.94.3.0 Sender: news Subject: Re: New FreeBSD package system (a.k.a. Daemon Package System (dps)) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 12 May 2007 14:25:49 -0000 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enigBB992BE5955A3EEDFBEF67F3 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Stanislav Sedov wrote: > On Fri, 11 May 2007 02:10:05 +0200 > Ivan Voras mentioned: >=20 >> - I think it's time to give up on using BDB+directory tree full of tex= t >> files for storing the installed packages database, and I propose all o= f >> this be replaced by a single SQLite database. SQLite is public domain >> (can be slurped into base system), embeddable, stores all data in a >> single file, lightweight, fast, and can be used to do fancy things suc= h >> as reporting. >=20 > What is the reason to use SQL-based database? You'll perform direct > queries to database? The packaging system is for ordinal users, not sql= > geeks, so they should not have to use sql for managing packages. So a It's not SQL I'm interested in, it's the "additional" features: - performance - transaction safety ("commit all changes or none") - constraints (like "unique" keys - sqlite unfortunately doesn't support foreign keys) - concurrent access (allowing to run multiple portupgrades at the same ti= me) - easy interface to C programs If a BDB variety or some other storage layer can achieve these things, I'll likely support them. I know "Sleepycat" BDB implementations boast "transaction processing", but can they offer this across multiple stores / databases at the same time (i.e. like one transaction includes updates to multiple tables)? Efficient (performance-wise) storage would probably need to use more than one store, at least to index data by different keys. --------------enigBB992BE5955A3EEDFBEF67F3 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.4 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFGRc5CldnAQVacBcgRAqeoAKCbX8drKd6UJVe9do1cWgInnWLLxQCeP8pd 5jzs0S7eU4uzpj/Pb4bKoNY= =SOb4 -----END PGP SIGNATURE----- --------------enigBB992BE5955A3EEDFBEF67F3-- From owner-freebsd-hackers@FreeBSD.ORG Sat May 12 14:30:10 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id AB6D516A402 for ; Sat, 12 May 2007 14:30:10 +0000 (UTC) (envelope-from freebsd-hackers@m.gmane.org) Received: from ciao.gmane.org (main.gmane.org [80.91.229.2]) by mx1.freebsd.org (Postfix) with ESMTP id 6430213C4AD for ; Sat, 12 May 2007 14:30:10 +0000 (UTC) (envelope-from freebsd-hackers@m.gmane.org) Received: from root by ciao.gmane.org with local (Exim 4.43) id 1Hmsbi-0001Pp-7y for freebsd-hackers@freebsd.org; Sat, 12 May 2007 16:30:02 +0200 Received: from 89-172-56-80.adsl.net.t-com.hr ([89.172.56.80]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Sat, 12 May 2007 16:30:02 +0200 Received: from ivoras by 89-172-56-80.adsl.net.t-com.hr with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Sat, 12 May 2007 16:30:02 +0200 X-Injected-Via-Gmane: http://gmane.org/ To: freebsd-hackers@freebsd.org From: Ivan Voras Date: Sat, 12 May 2007 16:26:53 +0200 Lines: 29 Message-ID: References: <200705102105.27271.blackdragon@highveldmail.co.za> <20070512155059.92011d54.stas@FreeBSD.org> <4645AFAF.7010704@free.fr> <20070512165053.8e53f5a3.stas@FreeBSD.org> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="------------enigA4043894C8C2C3E732955161" X-Complaints-To: usenet@sea.gmane.org X-Gmane-NNTP-Posting-Host: 89-172-56-80.adsl.net.t-com.hr User-Agent: Thunderbird 1.5.0.10 (Windows/20070221) In-Reply-To: <20070512165053.8e53f5a3.stas@FreeBSD.org> X-Enigmail-Version: 0.94.3.0 Sender: news Subject: Re: New FreeBSD package system (a.k.a. Daemon Package System (dps)) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 12 May 2007 14:30:10 -0000 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enigA4043894C8C2C3E732955161 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Stanislav Sedov wrote: > I agree, that there's a lot of ready tools for parsing xml, but why > not use much simple language that can be parsed by sed or awk in few > lines?=20 Because of mindshare. Young people know SQL and XML, but not grep. --------------enigA4043894C8C2C3E732955161 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.4 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFGRc6tldnAQVacBcgRAgyZAJ9M/RAQQAFW9RrSrOw0g9VC17mC2QCg6VhE OFGXOgX4QOjC12kHuSM6gug= =ebnz -----END PGP SIGNATURE----- --------------enigA4043894C8C2C3E732955161-- From owner-freebsd-hackers@FreeBSD.ORG Sat May 12 18:28:52 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 016BA16A407 for ; Sat, 12 May 2007 18:28:52 +0000 (UTC) (envelope-from maslanbsd@gmail.com) Received: from ug-out-1314.google.com (ug-out-1314.google.com [66.249.92.172]) by mx1.freebsd.org (Postfix) with ESMTP id 91F5713C45A for ; Sat, 12 May 2007 18:28:51 +0000 (UTC) (envelope-from maslanbsd@gmail.com) Received: by ug-out-1314.google.com with SMTP id 71so832296ugh for ; Sat, 12 May 2007 11:28:50 -0700 (PDT) DKIM-Signature: a=rsa-sha1; c=relaxed/relaxed; d=gmail.com; s=beta; h=domainkey-signature:received:received:message-id:date:from:to:subject:mime-version:content-type:content-transfer-encoding:content-disposition; b=oV+yimONzjRt8Rduyi6fYGbGox8xBsfdswE8/YHipTw95RjZtkDQrVMjQqGrvx8l/YuQm7HbG9q5i1P/yLzzMK61hQMv51jMAHbQoSDboAsXhjjrVQW+wNh2NqDKWaj/vs5Llh09zpTO+ExM40GSCbkWxchQdryAzlWRmLo34j0= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=beta; h=received:message-id:date:from:to:subject:mime-version:content-type:content-transfer-encoding:content-disposition; b=AAcLystOQfEkyra+VM3YCOwogzRkH2uFte+HRWtHj28UqwTB9awhe7zIfBuERFUN3uG70MOrwjK9LMQLjuuOdFizvAo5EEMelH8g7KcACkSiSYlgFoPmr//O54UO6ir4m1HTSH+UNThkYfBp3bAkxmH5zmjCLWJztPMF/AQQmlA= Received: by 10.67.6.1 with SMTP id j1mr3766220ugi.1178992869616; Sat, 12 May 2007 11:01:09 -0700 (PDT) Received: by 10.67.68.1 with HTTP; Sat, 12 May 2007 11:01:09 -0700 (PDT) Message-ID: <319cceca0705121101p589cbbb7ke8c1d542ee333832@mail.gmail.com> Date: Sat, 12 May 2007 11:01:09 -0700 From: Maslan To: freebsd-hackers@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Content-Disposition: inline Subject: FBSD on HP Pavillion dv6000 Family X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 12 May 2007 18:28:52 -0000 Hi all, I want to install freebsd-6.2 on my new laptop rather than win vista, but by doing some googling i found that almost everything will not work. any resources/links for drivers even if still untested, i would like to help. Thanks -- I'm Searching For Perfection, So Even If U Need Portability U've To Use Assembly ;-) http://libosdk.berlios.de From owner-freebsd-hackers@FreeBSD.ORG Sat May 12 18:47:30 2007 Return-Path: X-Original-To: hackers@freebsd.org Delivered-To: freebsd-hackers@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 0C86016A402 for ; Sat, 12 May 2007 18:47:30 +0000 (UTC) (envelope-from xistence@0x58.com) Received: from mailexchange.osnn.net (1e.66.5646.static.theplanet.com [70.86.102.30]) by mx1.freebsd.org (Postfix) with SMTP id D8EA513C465 for ; Sat, 12 May 2007 18:47:29 +0000 (UTC) (envelope-from xistence@0x58.com) Received: (qmail 14477 invoked by uid 0); 12 May 2007 18:43:45 -0000 Received: from unknown (HELO ?10.10.10.22?) (xistence@0x58.com@72.208.132.56) by mailexchange.osnn.net with SMTP; 12 May 2007 18:43:45 -0000 Mime-Version: 1.0 (Apple Message framework v752.3) In-Reply-To: <20070510125445.GA5460@hub.freebsd.org> References: <20070510111326.GA94093@hub.freebsd.org> <20070510132153.A91312@fledge.watson.org> <20070510125445.GA5460@hub.freebsd.org> Content-Type: multipart/signed; micalg=sha1; boundary=Apple-Mail-5-972466620; protocol="application/pkcs7-signature" Message-Id: From: Bert JW Regeer Date: Sat, 12 May 2007 11:47:25 -0700 To: Darren Reed , current@freebsd.org, hackers@freebsd.org X-Mailer: Apple Mail (2.752.3) X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: Subject: Re: Experiences with 7.0-CURRENT and vmware. X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 12 May 2007 18:47:30 -0000 --Apple-Mail-5-972466620 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=US-ASCII; delsp=yes; format=flowed On May 10, 2007, at 5:54 AM, Darren Reed wrote: > [...] > >> But if if_em is probing, it suggests a VMware >> change rather than a FreeBSD change, which you may be able to >> revert by >> telling it to expose a Lance-style device as opposed to an Intel >> device. > > There's no way to choose the type of card vmware emulates. > I always set my VMWare to expose an intel e1000 card, which gets probed correctly by almost all systems, even Windows with the Intel drivers installed. In your .vmx file you should find a line like this: ethernet0.virtualDev="e1000" If VMWare is to give an em device, if you remove that line it should default back to lnc driver. That being said, I have had no performance problems with the em driver on FreeBSD 6.0 in VMWare, and have not had the timeout problems you mentioned. > Darren Bert JW Regeer --Apple-Mail-5-972466620-- From owner-freebsd-hackers@FreeBSD.ORG Sat May 12 18:50:40 2007 Return-Path: X-Original-To: hackers@freebsd.org Delivered-To: freebsd-hackers@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id F322416A402 for ; Sat, 12 May 2007 18:50:39 +0000 (UTC) (envelope-from xistence@0x58.com) Received: from mailexchange.osnn.net (1e.66.5646.static.theplanet.com [70.86.102.30]) by mx1.freebsd.org (Postfix) with SMTP id C4D6C13C44B for ; Sat, 12 May 2007 18:50:39 +0000 (UTC) (envelope-from xistence@0x58.com) Received: (qmail 14624 invoked by uid 0); 12 May 2007 18:46:54 -0000 Received: from unknown (HELO ?10.10.10.22?) (xistence@0x58.com@72.208.132.56) by mailexchange.osnn.net with SMTP; 12 May 2007 18:46:54 -0000 Mime-Version: 1.0 (Apple Message framework v752.3) In-Reply-To: <4645AFAF.7010704@free.fr> References: <200705102105.27271.blackdragon@highveldmail.co.za> <20070512155059.92011d54.stas@FreeBSD.org> <4645AFAF.7010704@free.fr> Content-Type: multipart/signed; micalg=sha1; boundary=Apple-Mail-6-972656694; protocol="application/pkcs7-signature" Message-Id: <507A6005-05A5-46DB-9746-ADC2E8C1B43B@0x58.com> From: Bert JW Regeer Date: Sat, 12 May 2007 11:50:35 -0700 To: hackers@freebsd.org X-Mailer: Apple Mail (2.752.3) X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: Subject: Re: New FreeBSD package system (a.k.a. Daemon Package System (dps)) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 12 May 2007 18:50:40 -0000 --Apple-Mail-6-972656694 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=ISO-8859-1; delsp=yes; format=flowed On May 12, 2007, at 5:14 AM, Philippe Laquet wrote: > Stanislav Sedov a =E9crit : >> On Fri, 11 May 2007 02:10:05 +0200 >> Ivan Voras mentioned: >> >> >>> - I think it's time to give up on using BDB+directory tree full =20 >>> of text >>> files for storing the installed packages database, and I propose =20 >>> all of >>> this be replaced by a single SQLite database. SQLite is public =20 >>> domain >>> (can be slurped into base system), embeddable, stores all data in a >>> single file, lightweight, fast, and can be used to do fancy =20 >>> things such >>> as reporting. >>> >> >> What is the reason to use SQL-based database? You'll perform direct >> queries to database? The packaging system is for ordinal users, =20 >> not sql >> geeks, so they should not have to use sql for managing packages. So a >> simple set of hashes will suffer or needs. I agree with Julian =20 >> that we >> should have a backup of packaging database in plain text format, and >> utility to rebuild it. This way we can always restore the database if >> something goes wrong. Furhtermore, that should not make a great =20 >> impact >> on performance, since we don't have to rebuild it every day. >> > I agree with Stan ;) > > "fast and improved" package utilities uses mainly some indexed =20 > berkeley DB combined with flat files, aren't they? I, and may be =20 > many other FreeBSD users use light systems for efficiency and =20 > eaiser management, if we use some database system it will require =20 > Disk Space, ressources for the DB to run, dependencies and so on... =20= > And we also may be exposed to a "that DB is better" war ;) > SQLite is compiled inside a program, and as such does not require any =20= resources other than one file handle and some CPU time when querying. =20= The file is stored on disk, and requires no separate process to be =20 running to query. Maybe I misunderstood what you were trying to say. =20 SQLite will require less resources than flat text files, since SQLite =20= is a one time open then process, instead of what is currently =20 happening, having to open and close hundreds of files depending on =20 how many ports are installed. With this regard, SQLite is like BDB. =20 Where SQLite uses standards compliant SQL statements to get data. > >> -- >> Stanislav Sedov >> ST4096-RIPE >> > I am able to understand many of the gripes with using a databases, =20 and have to import yet another code base into the FreeBSD base, =20 however as one of the young ones, and knowing sed/awk/grep and SQL, I =20= prefer SQL over having to process hundreds of text files using text =20 processing tools. It saddens me each time I run one of the pkg_* =20 tools that needs to parse the flat file structure since it takes so =20 long. I have friends running Ubuntu and their apt-get returns results =20= much faster. In a world where hard drives are becoming more reliable, and are =20 automatically relocating sectors that go bad, do we really have to =20 worry about database corruption as much? I feel that many of the =20 fears that are being put forward will do harm to a text based =20 "storage" system as well. If one block drops out, it can cause tools =20 to not be able to parse the files. Create a backup copy of the =20 database after each successful transaction? There are ways to battle =20 data corruption. Using BDB is not an real option either. I can not even count the =20 amount of times that the BDB database that portupgrade created has =20 become corrupt because I accidently ran two portupgrades at the same =20 time, or even remembered that I did not want to upgrade something and =20= hit Ctrl+C. The experience I got from running SVN with BDB as the =20 back-end database to store my data, I say no thanks. In that case I =20 would much rather stick with the flat text files than go with a =20 database. Bert JW Regeer --Apple-Mail-6-972656694-- From owner-freebsd-hackers@FreeBSD.ORG Sat May 12 18:58:46 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 2DCB916A405 for ; Sat, 12 May 2007 18:58:46 +0000 (UTC) (envelope-from xistence@0x58.com) Received: from mailexchange.osnn.net (1e.66.5646.static.theplanet.com [70.86.102.30]) by mx1.freebsd.org (Postfix) with SMTP id E75D013C45E for ; Sat, 12 May 2007 18:58:45 +0000 (UTC) (envelope-from xistence@0x58.com) Received: (qmail 13932 invoked by uid 0); 12 May 2007 18:28:18 -0000 Received: from unknown (HELO ?10.10.10.22?) (xistence@0x58.com@72.208.132.56) by mailexchange.osnn.net with SMTP; 12 May 2007 18:28:18 -0000 Mime-Version: 1.0 (Apple Message framework v752.3) In-Reply-To: <4645AFAF.7010704@free.fr> References: <200705102105.27271.blackdragon@highveldmail.co.za> <20070512155059.92011d54.stas@FreeBSD.org> <4645AFAF.7010704@free.fr> Content-Type: multipart/signed; micalg=sha1; boundary=Apple-Mail-4-971540728; protocol="application/pkcs7-signature" Message-Id: <8916C4D5-4DB5-49C0-AF8D-07F9FFA0A6E0@0x58.com> From: Bert JW Regeer Date: Sat, 12 May 2007 11:31:59 -0700 To: freebsd-hackers@freebsd.org X-Mailer: Apple Mail (2.752.3) X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Subject: Re: New FreeBSD package system (a.k.a. Daemon Package System (dps)) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 12 May 2007 18:58:46 -0000 --Apple-Mail-4-971540728 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=ISO-8859-1; delsp=yes; format=flowed On May 12, 2007, at 5:14 AM, Philippe Laquet wrote: > Stanislav Sedov a =E9crit : >> On Fri, 11 May 2007 02:10:05 +0200 >> Ivan Voras mentioned: >> >> >>> - I think it's time to give up on using BDB+directory tree full =20 >>> of text >>> files for storing the installed packages database, and I propose =20 >>> all of >>> this be replaced by a single SQLite database. SQLite is public =20 >>> domain >>> (can be slurped into base system), embeddable, stores all data in a >>> single file, lightweight, fast, and can be used to do fancy =20 >>> things such >>> as reporting. >>> >> >> What is the reason to use SQL-based database? You'll perform direct >> queries to database? The packaging system is for ordinal users, =20 >> not sql >> geeks, so they should not have to use sql for managing packages. So a >> simple set of hashes will suffer or needs. I agree with Julian =20 >> that we >> should have a backup of packaging database in plain text format, and >> utility to rebuild it. This way we can always restore the database if >> something goes wrong. Furhtermore, that should not make a great =20 >> impact >> on performance, since we don't have to rebuild it every day. >> > I agree with Stan ;) > > "fast and improved" package utilities uses mainly some indexed =20 > berkeley DB combined with flat files, aren't they? I, and may be =20 > many other FreeBSD users use light systems for efficiency and =20 > eaiser management, if we use some database system it will require =20 > Disk Space, ressources for the DB to run, dependencies and so on... =20= > And we also may be exposed to a "that DB is better" war ;) > SQLite is compiled inside a program, and as such does not require any =20= resources other than one file handle and some CPU time when querying. =20= The file is stored on disk, and requires no separate process to be =20 running to query. Maybe I misunderstood what you were trying to say. =20 SQLite will require less resources than flat text files, since SQLite =20= is a one time open then process, instead of what is currently =20 happening, having to open and close hundreds of files depending on =20 how many ports are installed. With this regard, SQLite is like BDB. =20 Where SQLite uses standards compliant SQL statements to get data. > >> -- >> Stanislav Sedov >> ST4096-RIPE >> > I am able to understand many of the gripes with using a databases, =20 and have to import yet another code base into the FreeBSD base, =20 however as one of the young ones, and knowing sed/awk/grep and SQL, I =20= prefer SQL over having to process hundreds of text files using text =20 processing tools. It saddens me each time I run one of the pkg_* =20 tools that needs to parse the flat file structure since it takes so =20 long. I have friends running Ubuntu and their apt-get returns results =20= much faster. In a world where hard drives are becoming more reliable, and are =20 automatically relocating sectors that go bad, do we really have to =20 worry about database corruption as much? I feel that many of the =20 fears that are being put forward will do harm to a text based =20 "storage" system as well. If one block drops out, it can cause tools =20 to not be able to parse the files. Create a backup copy of the =20 database after each successful transaction? There are ways to battle =20 data corruption. Using BDB is not an real option either. I can not even count the =20 amount of times that the BDB database that portupgrade created has =20 become corrupt because I accidently ran two portupgrades at the same =20 time, or even remembered that I did not want to upgrade something and =20= hit Ctrl+C. The experience I got from running SVN with BDB as the =20 back-end database to store my data, I say no thanks. In that case I =20 would much rather stick with the flat text files than go with a =20 database. Bert JW Regeer= --Apple-Mail-4-971540728-- From owner-freebsd-hackers@FreeBSD.ORG Sat May 12 19:33:03 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 423D316A405 for ; Sat, 12 May 2007 19:33:03 +0000 (UTC) (envelope-from kris@obsecurity.org) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.freebsd.org (Postfix) with ESMTP id 2F99213C459 for ; Sat, 12 May 2007 19:33:03 +0000 (UTC) (envelope-from kris@obsecurity.org) Received: from obsecurity.dyndns.org (elvis.mu.org [192.203.228.196]) by elvis.mu.org (Postfix) with ESMTP id 0010A1A3C19; Sat, 12 May 2007 12:33:49 -0700 (PDT) Received: by obsecurity.dyndns.org (Postfix, from userid 1000) id 5DA5352F67; Sat, 12 May 2007 15:33:02 -0400 (EDT) Date: Sat, 12 May 2007 15:33:02 -0400 From: Kris Kennaway To: Michel Talon Message-ID: <20070512193302.GA24673@xor.obsecurity.org> References: <20070512004209.GA12218@lpthe.jussieu.fr> <17989.8202.624522.136573@bhuda.mired.org> <20070512090935.GA13929@lpthe.jussieu.fr> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="UlVJffcvxoiEqYs2" Content-Disposition: inline In-Reply-To: <20070512090935.GA13929@lpthe.jussieu.fr> User-Agent: Mutt/1.4.2.2i Cc: freebsd-hackers@freebsd.org, Mike Meyer Subject: Re: DPS Initial Ideas X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 12 May 2007 19:33:03 -0000 --UlVJffcvxoiEqYs2 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sat, May 12, 2007 at 11:09:35AM +0200, Michel Talon wrote: > Seriously, the FreeBSD package > system is in great need of a profound overhaul, pretending it works well > is complete denial of reality. I hope that young people working on=20 > summer code projects will infuse *new* ideas, and not spend their > vacations polishing inadequate tools. I know that this is your belief, but please try to avoid grasping at straws: there are elements in your argument that are along the lines of "The FreeBSD package system is broken and needs to be fundamentally changed. Rewriting it to use SQLite is a fundamental change. Therefore rewriting it to use SQLite will fix the problems." First figure out what specific problems need to be solved, then figure out how to solve them, not the other way around. So far I have seen little discussion of how SQLite is necessary and sufficient for fixing fundamental issues. The argument in favour of SQL seems to boil down to "It's SQL! You can do more complex queries...if you wanted to". Without a clear demonstration of how this would solve a problem associated with package management, it is not very compelling and basically reduces to change for the sake of change. As I discussed in my email yesterday, there are serious issues to be solved. Some of them can be solved by improving the storage backend of the package database to use a database; but this is in progress using existing tools. Given that this work is happening (or at least will be happening, I am not sure when the SoC officially starts), the best thing is for interested people to work with Garrett to help him achieve the goals of his project. Kris --UlVJffcvxoiEqYs2 Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (FreeBSD) iD4DBQFGRhZuWry0BWjoQKURAhEhAJdq21IHfEXb48sV4ajeTjbsPVciAJ9G6zSu ZwTl/toWmCkIY2wCj2eDNg== =w9uO -----END PGP SIGNATURE----- --UlVJffcvxoiEqYs2-- From owner-freebsd-hackers@FreeBSD.ORG Sat May 12 21:13:56 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 201CC16A403 for ; Sat, 12 May 2007 21:13:56 +0000 (UTC) (envelope-from freebsd-hackers@m.gmane.org) Received: from ciao.gmane.org (main.gmane.org [80.91.229.2]) by mx1.freebsd.org (Postfix) with ESMTP id CE63D13C45A for ; Sat, 12 May 2007 21:13:55 +0000 (UTC) (envelope-from freebsd-hackers@m.gmane.org) Received: from list by ciao.gmane.org with local (Exim 4.43) id 1HmyuS-0004Zz-5L for freebsd-hackers@freebsd.org; Sat, 12 May 2007 23:13:48 +0200 Received: from 89-172-56-80.adsl.net.t-com.hr ([89.172.56.80]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Sat, 12 May 2007 23:13:48 +0200 Received: from ivoras by 89-172-56-80.adsl.net.t-com.hr with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Sat, 12 May 2007 23:13:48 +0200 X-Injected-Via-Gmane: http://gmane.org/ To: freebsd-hackers@freebsd.org From: Ivan Voras Date: Sat, 12 May 2007 23:13:34 +0200 Lines: 38 Message-ID: References: <200705112302.35380.dragonsa@highveldmail.co.za> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="------------enig9769E44987B9D51D5D83CD70" X-Complaints-To: usenet@sea.gmane.org X-Gmane-NNTP-Posting-Host: 89-172-56-80.adsl.net.t-com.hr User-Agent: Thunderbird 1.5.0.10 (Windows/20070221) In-Reply-To: <200705112302.35380.dragonsa@highveldmail.co.za> X-Enigmail-Version: 0.94.3.0 Sender: news Subject: Re: DPS Initial Ideas X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 12 May 2007 21:13:56 -0000 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enig9769E44987B9D51D5D83CD70 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable David Naylor wrote: >=20 > I am looking at a hybrid approach to storing the package metadata, a=20 > combination of SQLite and compressed text files. I am hoping to create= a=20 > situation where if either gets corrupted it can be created from the oth= er. =20 =2E.. throwing away transaction safety, as it means updating 2 completely= unrelated (and unrelatable) data stores. (I'm not against your work, I'm just pointing out an area where you need to be extra careful - some kind of 2pc protocol to update both sides may be required). --------------enig9769E44987B9D51D5D83CD70 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.4 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFGRi4EldnAQVacBcgRApK5AKDZaQp1/YB2HriPf7mYRL1XdO+rlACfYMaP ZP3Fgzjchk4upPhgt8H09s0= =UgLK -----END PGP SIGNATURE----- --------------enig9769E44987B9D51D5D83CD70-- From owner-freebsd-hackers@FreeBSD.ORG Sat May 12 21:26:21 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 81E3916A400 for ; Sat, 12 May 2007 21:26:21 +0000 (UTC) (envelope-from freebsd-hackers@m.gmane.org) Received: from ciao.gmane.org (main.gmane.org [80.91.229.2]) by mx1.freebsd.org (Postfix) with ESMTP id 0C83213C458 for ; Sat, 12 May 2007 21:26:20 +0000 (UTC) (envelope-from freebsd-hackers@m.gmane.org) Received: from list by ciao.gmane.org with local (Exim 4.43) id 1Hmz6T-00067H-NB for freebsd-hackers@freebsd.org; Sat, 12 May 2007 23:26:13 +0200 Received: from 89-172-56-80.adsl.net.t-com.hr ([89.172.56.80]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Sat, 12 May 2007 23:26:13 +0200 Received: from ivoras by 89-172-56-80.adsl.net.t-com.hr with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Sat, 12 May 2007 23:26:13 +0200 X-Injected-Via-Gmane: http://gmane.org/ To: freebsd-hackers@freebsd.org From: Ivan Voras Date: Sat, 12 May 2007 23:25:58 +0200 Lines: 47 Message-ID: References: <20070512004209.GA12218@lpthe.jussieu.fr> <17989.8202.624522.136573@bhuda.mired.org> <20070512090935.GA13929@lpthe.jussieu.fr> <20070512193302.GA24673@xor.obsecurity.org> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="------------enigF6442AB842370A2BED6C0E76" X-Complaints-To: usenet@sea.gmane.org X-Gmane-NNTP-Posting-Host: 89-172-56-80.adsl.net.t-com.hr User-Agent: Thunderbird 1.5.0.10 (Windows/20070221) In-Reply-To: <20070512193302.GA24673@xor.obsecurity.org> X-Enigmail-Version: 0.94.3.0 Sender: news Subject: Re: DPS Initial Ideas X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 12 May 2007 21:26:21 -0000 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enigF6442AB842370A2BED6C0E76 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Kris Kennaway wrote: > First figure out what specific problems need to be solved, then figure > out how to solve them, not the other way around. So far I have seen > little discussion of how SQLite is necessary and sufficient for fixing > fundamental issues. The argument in favour of SQL seems to boil down > to "It's SQL! You can do more complex queries...if you wanted to". I've posted some general ideas (resulting from my experience using the package / port system, not developing for it): 1. speed and simplicity of querying (single query vs traversing a tree of text files) 2. formal data constraints (UNIQUE, CHECK) 3. transaction safety (a consequence of which is the ability to run concurrent installs / updates) 4. easy interface for 3d party tools I admit again that I didn't develop anything with the package / ports subsystems, so there might be other, bigger problems not solvable by sqlite, but I believe the features above could at least solve performance problems. (I also agree there's no point in changing the ports infrastructure itself, just the package tracking database in base system). --------------enigF6442AB842370A2BED6C0E76 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.4 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFGRjDmldnAQVacBcgRAoNjAJ49nGszEsRLWD5nXX9By0+a63Q1vACgyCgB hA/8hk2m1A0IbmRb2Kijtj4= =KArl -----END PGP SIGNATURE----- --------------enigF6442AB842370A2BED6C0E76-- From owner-freebsd-hackers@FreeBSD.ORG Sat May 12 21:33:53 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 00FBE16A402 for ; Sat, 12 May 2007 21:33:53 +0000 (UTC) (envelope-from kris@obsecurity.org) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.freebsd.org (Postfix) with ESMTP id E15F013C465 for ; Sat, 12 May 2007 21:33:52 +0000 (UTC) (envelope-from kris@obsecurity.org) Received: from obsecurity.dyndns.org (elvis.mu.org [192.203.228.196]) by elvis.mu.org (Postfix) with ESMTP id C5B021A3C1A; Sat, 12 May 2007 14:34:39 -0700 (PDT) Received: by obsecurity.dyndns.org (Postfix, from userid 1000) id E1B7052FB7; Sat, 12 May 2007 17:33:51 -0400 (EDT) Date: Sat, 12 May 2007 17:33:51 -0400 From: Kris Kennaway To: Ivan Voras Message-ID: <20070512213351.GA28544@xor.obsecurity.org> References: <20070512004209.GA12218@lpthe.jussieu.fr> <17989.8202.624522.136573@bhuda.mired.org> <20070512090935.GA13929@lpthe.jussieu.fr> <20070512193302.GA24673@xor.obsecurity.org> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="SLDf9lqlvOQaIe6s" Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.4.2.2i Cc: freebsd-hackers@freebsd.org Subject: Re: DPS Initial Ideas X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 12 May 2007 21:33:53 -0000 --SLDf9lqlvOQaIe6s Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sat, May 12, 2007 at 11:25:58PM +0200, Ivan Voras wrote: > Kris Kennaway wrote: >=20 > > First figure out what specific problems need to be solved, then figure > > out how to solve them, not the other way around. So far I have seen > > little discussion of how SQLite is necessary and sufficient for fixing > > fundamental issues. The argument in favour of SQL seems to boil down > > to "It's SQL! You can do more complex queries...if you wanted to". >=20 > I've posted some general ideas (resulting from my experience using the > package / port system, not developing for it): >=20 > 1. speed and simplicity of querying (single query vs traversing a tree > of text files) > 2. formal data constraints (UNIQUE, CHECK) > 3. transaction safety (a consequence of which is the ability to run > concurrent installs / updates) > 4. easy interface for 3d party tools >=20 > I admit again that I didn't develop anything with the package / ports > subsystems, so there might be other, bigger problems not solvable by > sqlite, but I believe the features above could at least solve > performance problems. That is the "sufficient" part but not the "necessary part". 1) and 3) are solvable using existing tools. 2 and 4 not so much, but you haven't described what problems they solve. Homework for SQLite advocates: write a 1-page essay on the following topic. "Rewriting the package tools to use SQLite will solve problem(s) ____ that exist in the current system. Compare and contrast to other possible solutions including Berkeley DB." > (I also agree there's no point in changing the ports infrastructure > itself, just the package tracking database in base system). Well, I was talking about both. Kris --SLDf9lqlvOQaIe6s Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (FreeBSD) iD8DBQFGRjK/Wry0BWjoQKURAkaSAJ4y4ll1Kg1XCFoIfZQELOjTc1+lyQCgg+uF ZEHgEEd1ozP/8OATRqOsOWE= =Dy6V -----END PGP SIGNATURE----- --SLDf9lqlvOQaIe6s-- From owner-freebsd-hackers@FreeBSD.ORG Sat May 12 20:01:05 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 2AABF16A404 for ; Sat, 12 May 2007 20:01:05 +0000 (UTC) (envelope-from mj@feral.com) Received: from ns1.feral.com (ns1.feral.com [192.67.166.1]) by mx1.freebsd.org (Postfix) with ESMTP id 0AAD513C489 for ; Sat, 12 May 2007 20:01:04 +0000 (UTC) (envelope-from mj@feral.com) Received: from ns1.feral.com (localhost [127.0.0.1]) by ns1.feral.com (8.14.1/8.14.1) with ESMTP id l4CJdgJC038842; Sat, 12 May 2007 12:39:50 -0700 (PDT) (envelope-from mj@feral.com) Received: from localhost (mjacob@localhost) by ns1.feral.com (8.14.1/8.14.1/Submit) with ESMTP id l4CJdgdO038839; Sat, 12 May 2007 12:39:42 -0700 (PDT) (envelope-from mj@feral.com) X-Authentication-Warning: ns1.feral.com: mjacob owned process doing -bs Date: Sat, 12 May 2007 12:39:42 -0700 (PDT) From: Matthew Jacob To: Michel Talon In-Reply-To: <20070512193302.GA24673@xor.obsecurity.org> Message-ID: <20070512123652.R38784@ns1.feral.com> References: <20070512004209.GA12218@lpthe.jussieu.fr> <17989.8202.624522.136573@bhuda.mired.org> <20070512090935.GA13929@lpthe.jussieu.fr> <20070512193302.GA24673@xor.obsecurity.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Mailman-Approved-At: Sat, 12 May 2007 21:35:26 +0000 Cc: freebsd-hackers@freebsd.org, Mike Meyer Subject: Re: DPS Initial Ideas X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Matthew Jacob List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 12 May 2007 20:01:05 -0000 > Seriously, the FreeBSD package system is in great need of a profound > overhaul, pretending it works well is complete denial of reality. I > hope that young people working on summer code projects will infuse > *new* ideas, and not spend their vacations polishing inadequate tools. Hmm? Works fine for me and many others who are more than casual users. I think Kris has it right when he asks you to state what problems need to be addressed and *then* how you would address them- not to find a way to address problems first. respectfully -matt From owner-freebsd-hackers@FreeBSD.ORG Sat May 12 22:00:33 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id A6F9E16A404 for ; Sat, 12 May 2007 22:00:33 +0000 (UTC) (envelope-from talon@lpthe.jussieu.fr) Received: from shiva.jussieu.fr (shiva.jussieu.fr [134.157.0.129]) by mx1.freebsd.org (Postfix) with ESMTP id 305ED13C458 for ; Sat, 12 May 2007 22:00:32 +0000 (UTC) (envelope-from talon@lpthe.jussieu.fr) Received: from parthe.lpthe.jussieu.fr (parthe.lpthe.jussieu.fr [134.157.10.1]) by shiva.jussieu.fr (8.13.8/jtpda-5.4) with ESMTP id l4CLiN4d030768 ; Sat, 12 May 2007 23:44:23 +0200 (CEST) X-Ids: 164 Received: by parthe.lpthe.jussieu.fr (Postfix, from userid 10096) id 4501DBF6A0; Sat, 12 May 2007 23:44:22 +0200 (CEST) X-Spam-Checker-Version: SpamAssassin 3.1.8 (2007-02-13) on parthe.lpthe.jussieu.fr X-Spam-Level: X-Spam-Status: No, score=-2.2 required=5.0 tests=AWL,BAYES_00 autolearn=ham version=3.1.8 Received: from asmodee.lpthe.jussieu.fr (asmodee.lpthe.jussieu.fr [134.157.10.34]) by parthe.lpthe.jussieu.fr (Postfix) with ESMTP id 0B49EBE87B; Sat, 12 May 2007 23:44:21 +0200 (CEST) Received: by asmodee.lpthe.jussieu.fr (Postfix, from userid 2005) id C1F2942C5; Sat, 12 May 2007 23:44:22 +0200 (CEST) Date: Sat, 12 May 2007 23:44:22 +0200 From: Michel Talon To: Kris Kennaway Message-ID: <20070512214422.GA88480@lpthe.jussieu.fr> References: <20070512004209.GA12218@lpthe.jussieu.fr> <17989.8202.624522.136573@bhuda.mired.org> <20070512090935.GA13929@lpthe.jussieu.fr> <20070512193302.GA24673@xor.obsecurity.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20070512193302.GA24673@xor.obsecurity.org> User-Agent: Mutt/1.4.2.2i X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-3.0 (shiva.jussieu.fr [134.157.0.164]); Sat, 12 May 2007 23:44:23 +0200 (CEST) X-Virus-Scanned: ClamAV 0.88.7/3231/Sat May 12 17:57:12 2007 on shiva.jussieu.fr X-Virus-Status: Clean X-j-chkmail-Score: MSGID : 46463537.000 on shiva.jussieu.fr : j-chkmail score : X : 0/50 0 0.539 -> 1 X-Miltered: at shiva.jussieu.fr with ID 46463537.000 by Joe's j-chkmail (http://j-chkmail.ensmp.fr)! X-Mailman-Approved-At: Sat, 12 May 2007 22:20:06 +0000 Cc: freebsd-hackers@freebsd.org, Mike Meyer Subject: Re: DPS Initial Ideas X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 12 May 2007 22:00:33 -0000 On Sat, May 12, 2007 at 03:33:02PM -0400, Kris Kennaway wrote: > On Sat, May 12, 2007 at 11:09:35AM +0200, Michel Talon wrote: > > > Seriously, the FreeBSD package > > system is in great need of a profound overhaul, pretending it works well > > is complete denial of reality. I hope that young people working on > > summer code projects will infuse *new* ideas, and not spend their > > vacations polishing inadequate tools. > > I know that this is your belief, but please try to avoid grasping at > straws: there are elements in your argument that are along the lines > of "The FreeBSD package system is broken and needs to be fundamentally > changed. Rewriting it to use SQLite is a fundamental change. > Therefore rewriting it to use SQLite will fix the problems." > Really i don't think at all this way. I think that *perhaps* SQLite may marginally better than a Berkeley database for solving part of the problem, not much more. What i reacted to, was the conservatism which pervades the community as soon as someone emits the idea of using a new tool. > First figure out what specific problems need to be solved, then figure > out how to solve them, not the other way around. So far I have seen > little discussion of how SQLite is necessary and sufficient for fixing > fundamental issues. The argument in favour of SQL seems to boil down > to "It's SQL! You can do more complex queries...if you wanted to". No, for me the main argument is that SQL is more familiar for many people than running a perl script to connect to a Berkeley database. I have also heard that SQLite is more performant, but i would have to see it to beleive it. > > Without a clear demonstration of how this would solve a problem > associated with package management, it is not very compelling and > basically reduces to change for the sake of change. I think that a lot of changes are necessary, and it seems they will happen. So *perhaps* it may be beneficial in this sea of changes to consider a minor change, moving from a more traditional Berkeley database to SQLite. > > As I discussed in my email yesterday, there are serious issues to be > solved. I think some of the issues have nothing to do with the database question. Some of the issues are entirely trivial to solve. One of the worst offenders for misbehaviour of the package system is the constant changes in the port origins and the poor standardisation of the package names. When it will be clear that these name changes bring nothing to the table but introduce a lot of confusion both for end users and automated programs, things will be easier. It may be that borrowing from Debian the idea of "abstract" dependencies which can be fulfilled by several concrete packages may also simplify the dependency problem. For example tomcat may depend on "java" and java my be fulfilled either by diablo-jdk15 or jdk15. This way when you change from diablo-jdk15 to jdk15 you don't need to change anything to tomcat. Another feature that Debian has, and which may happily complete the previous one, is the specification of necessary dependencies with a version number in a certain range (this obviously requires a reasonable standardisation of version numbers, so that comparison of -0.99 to -1.0-rc doesn't depend on arcane rules). This way you don't need to change dependencies which are in the correct range, even if a more recent version exists. This mechanism has been imported in NetBSD pkgsrc. And a problem which has proven useful in Debian is keeping track of the packages which have been required by the end user and those which have been installed as dependencies. This is the difference between apt-get and aptitude. Apparently people are very happy to be able to remove not only a package they have required, but also all its dependencies (which are not required by another program) at one stroke. This also helps in case some big package requires dependency A, but after upgrade, they have changed their mind and require alternative dependency B. With this mechanism, after upgrade A disappears, while without it you will have both an upgraded version of A and B. I have observed on my machine this is an important cause of time monotonic bloat of the package tree. To answer the slowness problem in registering installed packages, one may think about making use of the INDEX file. In fact all the information that is necessary to fill the dependency entries is contained in INDEX, and accessible here in milliseconds with any tool such as awk. It so happens that the ports system doesn't make any use of the INDEX file and systematically recomputes the dependencies through recursive make invocations which are very time consuming. Of course this requires up to date INDEX, or a mechanism to keep INDEX continually up to date. Part of the registration is also filling the +REQUIRED_BY files of the dependencies of a package when one installs a package. If this package has a lot of dependencies this means opening, editing and closing a large number of files. This is expensive. One may imagine using a database containing the global dependency information, then +REQUIRED_BY files are no more necessary, since the information can be recomputed in very little time. In my little python experiments, recomputing the complete set of +REQUIRED_BY files for around 700 ports takes around one second. By the way, topological sorting the DAG of the whole port tree (> 15 000 ports) takes of the order of 2 seconds, so it is clear that if major performance problems occur, they cannot be ascribed to such DAG sorting. > Some of them can be solved by improving the storage backend > of the package database to use a database; but this is in progress > using existing tools. Yes, and i don't buy the idea that using *existing* tools is better than using the best tool for the job (assuming one can prove what is the best tool, considering power, familiarity, etc.). > > Given that this work is happening (or at least will be happening, I am > not sure when the SoC officially starts), the best thing is for > interested people to work with Garrett to help him achieve the goals > of his project. Sure. I am convinced this is the reason why several people, including myself present some ideas in the mailing list now, before Garrett begins working on his project. Of course after that, he will be in charge, with his mentor, and i hope they will do something wonderful. As you are well aware, designing a very good ports system is particularly difficult, unfortunately, particularly in the FreeBSD context where building from source is considered fashionable, which makes designing an efficient upgrade system almost impossible. > > Kris -- Michel TALON From owner-freebsd-hackers@FreeBSD.ORG Sat May 12 22:24:36 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id A2EBD16A402 for ; Sat, 12 May 2007 22:24:36 +0000 (UTC) (envelope-from kris@obsecurity.org) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.freebsd.org (Postfix) with ESMTP id 8CC1F13C447 for ; Sat, 12 May 2007 22:24:36 +0000 (UTC) (envelope-from kris@obsecurity.org) Received: from obsecurity.dyndns.org (elvis.mu.org [192.203.228.196]) by elvis.mu.org (Postfix) with ESMTP id 514301A3C1A; Sat, 12 May 2007 15:25:23 -0700 (PDT) Received: by obsecurity.dyndns.org (Postfix, from userid 1000) id 79A9F52F67; Sat, 12 May 2007 18:24:35 -0400 (EDT) Date: Sat, 12 May 2007 18:24:35 -0400 From: Kris Kennaway To: Michel Talon Message-ID: <20070512222435.GA28981@xor.obsecurity.org> References: <20070512004209.GA12218@lpthe.jussieu.fr> <17989.8202.624522.136573@bhuda.mired.org> <20070512090935.GA13929@lpthe.jussieu.fr> <20070512193302.GA24673@xor.obsecurity.org> <20070512214422.GA88480@lpthe.jussieu.fr> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="k+w/mQv8wyuph6w0" Content-Disposition: inline In-Reply-To: <20070512214422.GA88480@lpthe.jussieu.fr> User-Agent: Mutt/1.4.2.2i Cc: freebsd-hackers@freebsd.org, Mike Meyer , Kris Kennaway Subject: Re: DPS Initial Ideas X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 12 May 2007 22:24:36 -0000 --k+w/mQv8wyuph6w0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sat, May 12, 2007 at 11:44:22PM +0200, Michel Talon wrote: > On Sat, May 12, 2007 at 03:33:02PM -0400, Kris Kennaway wrote: > > On Sat, May 12, 2007 at 11:09:35AM +0200, Michel Talon wrote: > >=20 > > > Seriously, the FreeBSD package > > > system is in great need of a profound overhaul, pretending it works w= ell > > > is complete denial of reality. I hope that young people working on=20 > > > summer code projects will infuse *new* ideas, and not spend their > > > vacations polishing inadequate tools. > >=20 > > I know that this is your belief, but please try to avoid grasping at > > straws: there are elements in your argument that are along the lines > > of "The FreeBSD package system is broken and needs to be fundamentally > > changed. Rewriting it to use SQLite is a fundamental change. > > Therefore rewriting it to use SQLite will fix the problems." > >=20 >=20 > Really i don't think at all this way. I think that *perhaps* SQLite > may marginally better than a Berkeley database for solving part of the > problem, not much more. What i reacted to, was the conservatism which=20 > pervades the community as soon as someone emits the idea of using a new t= ool.=20 It seems to me that you do not appreciate the reasons behind this conservatism. A very important one is that we have two students who have committed to spending their summer working on improving the existing pkg_tools in ways that will solve some of the real problems we are facing, and the project we have agreed upon is that they will be using existing tools rather than rewriting from scratch as part of a not-yet-defined larger project. To some extent it is the timing here that is most unfortunate. If SQLite has been raised as a viable alternative a few months ago it would have made a great project idea, but instead we have committed to improving our existing tools, and the barrier for throwing out these plans is therefore very high. The burden of proof is therefore set much higher than "SQL is awesome and buzzword-compliant and might be better". > It may be that borrowing from Debian the idea of "abstract" dependencies > which can be fulfilled by several concrete packages may also simplify > the dependency problem. For example tomcat may depend on "java" and java > my be fulfilled either by diablo-jdk15 or jdk15. This way when you change > from diablo-jdk15 to jdk15 you don't need to change anything to tomcat. >=20 > Another feature that Debian has, and which may happily complete the previ= ous > one, is the specification of necessary dependencies with a version number > in a certain range (this obviously requires a reasonable standardisation = of > version numbers, so that comparison of -0.99 to=20 > -1.0-rc doesn't depend on arcane rules). This way you don't= need > to change dependencies which are in the correct range, even if a more rec= ent > version exists. This mechanism has been imported in NetBSD pkgsrc. We actually have both of these features in ports, but they have not been pushed down into packages. I think it will be relatively simple to do so, without requiring a rewrite from scratch. > And a problem which has proven useful in Debian is keeping track of the > packages which have been required by the end user and those which have be= en > installed as dependencies. This is the difference between apt-get and > aptitude. Apparently people are very happy to be able to remove not only > a package they have required, but also all its dependencies (which are > not required by another program) at one stroke. This also helps in case > some big package requires dependency A, but after upgrade, they have chan= ged > their mind and require alternative dependency B. With this mechanism, aft= er > upgrade A disappears, while without it you will have both an upgraded ver= sion > of A and B. I have observed on my machine this is an important cause=20 > of time monotonic bloat of the package tree. This one could also be added to the existing tools. > To answer the slowness problem in registering installed packages, one may > think about making use of the INDEX file. In fact all the information that > is necessary to fill the dependency entries is contained in INDEX, and > accessible here in milliseconds with any tool such as awk. It so happens = that > the ports system doesn't make any use of the INDEX file and systematically > recomputes the dependencies through recursive make invocations which are = very > time consuming. Of course this requires up to date INDEX, or a mechanism = to > keep INDEX continually up to date. The problem is that maintaining the INDEX is expensive and/or tricky. p5-FreeBSD-Portindex comes close but seems to have some wrinkles. > Part of the registration is also filling the +REQUIRED_BY files of the > dependencies of a package when one installs a package. If this package h= as a > lot of dependencies this means opening, editing and closing a large numbe= r of > files. This is expensive. One may imagine using a database containing the > global dependency information, then +REQUIRED_BY files are no more necess= ary, > since the information can be recomputed in very little time. Yes, this is one thing that is addressed in the current plan. > > Given that this work is happening (or at least will be happening, I am > > not sure when the SoC officially starts), the best thing is for > > interested people to work with Garrett to help him achieve the goals > > of his project. >=20 > Sure. I am convinced this is the reason why several people, including mys= elf > present some ideas in the mailing list now, before Garrett begins working= on > his project. I think this is a retcon; the original poster did not appear to have this motivation and neither do many others who have posted. Kris --k+w/mQv8wyuph6w0 Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (FreeBSD) iD4DBQFGRj6jWry0BWjoQKURAmUpAJMEv/USguMTVaa80MdBUCm1kjzKAJ4iuNsw rHV3bYRw4I013bH6n/Db8Q== =cIQm -----END PGP SIGNATURE----- --k+w/mQv8wyuph6w0--