From owner-freebsd-ports@FreeBSD.ORG Mon Jul 2 15:10:10 2007 Return-Path: X-Original-To: ports@FreeBSD.org Delivered-To: freebsd-ports@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 5776116A4E0 for ; Mon, 2 Jul 2007 15:10:10 +0000 (UTC) (envelope-from youshi10@u.washington.edu) Received: from mxout3.cac.washington.edu (mxout3.cac.washington.edu [140.142.32.166]) by mx1.freebsd.org (Postfix) with ESMTP id 3234D13C45D for ; Mon, 2 Jul 2007 15:10:10 +0000 (UTC) (envelope-from youshi10@u.washington.edu) Received: from smtp.washington.edu (smtp.washington.edu [140.142.32.141] (may be forged)) by mxout3.cac.washington.edu (8.13.7+UW06.06/8.13.7+UW07.05) with ESMTP id l62FA9Gr007426 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Mon, 2 Jul 2007 08:10:09 -0700 X-Auth-Received: from [192.168.10.45] (c-24-10-12-194.hsd1.ca.comcast.net [24.10.12.194]) (authenticated authid=youshi10) by smtp.washington.edu (8.13.7+UW06.06/8.13.7+UW07.03) with ESMTP id l62FA8DK005622 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NOT); Mon, 2 Jul 2007 08:10:09 -0700 Message-ID: <46891556.4090209@u.washington.edu> Date: Mon, 02 Jul 2007 08:10:14 -0700 From: Garrett Cooper User-Agent: Thunderbird 2.0.0.4 (Windows/20070604) MIME-Version: 1.0 To: Alexander Leidinger References: <46887FD3.3080307@u.washington.edu> <46889F5D.70801@gmx.de> <4688AF6D.90904@u.washington.edu> <20070702115733.3fotau92scgs4g4s@webmail.leidinger.net> In-Reply-To: <20070702115733.3fotau92scgs4g4s@webmail.leidinger.net> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-PMX-Version: 5.3.2.304607, Antispam-Engine: 2.5.1.298604, Antispam-Data: 2007.7.2.75033 X-Uwash-Spam: Gauge=IIIIIII, Probability=7%, Report='__C230066_P2 0, __CT 0, __CTE 0, __CT_TEXT_PLAIN 0, __HAS_MSGID 0, __MIME_TEXT_ONLY 0, __MIME_VERSION 0, __SANE_MSGID 0, __USER_AGENT 0' Cc: ports@FreeBSD.org, "\[LoN\]Kamikaze" Subject: Re: +CONTENTS files X-BeenThere: freebsd-ports@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting software to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 Jul 2007 15:10:10 -0000 Alexander Leidinger wrote: > Quoting Garrett Cooper (from Mon, 02 Jul > 2007 00:55:25 -0700): > >> [LoN]Kamikaze wrote: >>> Garrett Cooper wrote: >>> >>>> Pardon me for being naive, but wouldn't it be wiser for all of the >>>> data >>>> in the +CONTENTS file to be aggregated into sections instead of having >>>> line by line info? >>>> >>>> Example (net/samba_3.0.25a): >>>> >>>> @comment MD5:9e94560ac5e757d3bc5f922dcf3ab4fb >>>> man/man1/log2pcap.1.gz >>>> [~100 lines of repetitive data...] >>>> @comment MD5:9f5fc8df2a1383a175e165ef2e0b10cc >>>> man/man8/vfs_notify_fam.8.gz >>>> >>>> Could be aggregated into: >>>> >>>> @MD5 >>>> 9e94560ac5e757d3bc5f922dcf3ab4fb man/man1/log2pcap.1.gz >>>> c58f068d603a12d4af867c15cf77e636 man/man1/nmblookup.1.gz >>>> [etc..] >>>> @end MD5 >>>> >>>> or something similar to XML. >>>> >>>> This would reduce the filesize from n bytes to n - (9 + 4 -1) * >>>> i_entries + 8. In larger package files this would reduce the amount of >>>> data parsing by a long shot. Also, more powerful scripting languages >>>> like Perl, Python, or smart parsers in C could make short work of this >>>> data and just extract the MD5 elements for comparison. >>>> >>>> Also, by doing a little extra work when creating packages by >>>> organizing all the sections together, I think that the file size could >>>> be reduced by a large degree. >>>> >>>> Similar fields to @comment MD5 could be reduced I believe, but with >>>> less benefit maybe, other than just the @unexec rmdir, etc lines. >>>> >>>> Either that, or the data should be organized into separate files I >>>> think (increases number of files, but reduces overall processing >>>> time IMO). > >>> In some cases the order of data stored is important and thus it >>> cannot be >>> seperated into section. Also, this layout allows for very simple >>> parsing with >>> usual UNIX tools (sed, cut, awk, perl, simply everything). Unlike >>> XML, which is >>> rather complex and thus does not belong into base, in my opinion. > > We have libbsdxml in the base already (an old version of one in the > ports). Ok. >> I didn't say XML exactly. I say XML-like, with implied end and begin >> tags, but keeping with the Makefile like syntax of @MD5 ... @end MD5, >> or something similar. > > The problem is, that a change would break existing installations, as > they can not cope with such a new format. Feel free to propose > improvements, but you need to keep in your mind, that any supported > FreeBSD release has to be able to install packages with only the > package tools available in the basesystem. The point is though that there's a lot of unnecessary bloat, which adds to longer text file sizes, and thus slows down smarter parsers written in C, Perl, or Python. >> My point being is that the +CONTENTS file is bloated a lot by >> useless lines, and it would help speed up package processing if it was >> clipped or reduced somehow I would think. > > You need to provide numbers. Without them this is pure speculation. > > And you have to explain, why the current parsing routines can not be > speed up for the current format, maybe the implementation is just a > little bit outdated compared to todays parsing knowledge... > > Bye, > Alexander. > Ok. I take your challenge and will have preliminary results in 2-3 days. Are Excel formatted spreadsheets ok (thinking graphs)? Thanks, -Garrett