From owner-freebsd-hackers@FreeBSD.ORG Sun May 27 16:31:08 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 1177016A477 for ; Sun, 27 May 2007 16:31:08 +0000 (UTC) (envelope-from mail@maxlor.com) Received: from popeye1.ggamaur.net (popeye1.ggamaur.net [213.160.40.50]) by mx1.freebsd.org (Postfix) with ESMTP id AE72F13C4B7 for ; Sun, 27 May 2007 16:31:07 +0000 (UTC) (envelope-from mail@maxlor.com) Received: from maxlor.mine.nu (maxlor@c-82-192-240-247.customer.ggaweb.ch [82.192.240.247]) by popeye1.ggamaur.net (8.13.7/8.13.7/Submit) with ESMTP id l4RGV3AZ010633; Sun, 27 May 2007 18:31:05 +0200 (CEST) (envelope-from mail@maxlor.com) Received: from localhost (unknown [127.0.0.1]) by maxlor.mine.nu (Postfix) with ESMTP id 9A9722E237; Sun, 27 May 2007 18:31:00 +0200 (CEST) X-Virus-Scanned: amavisd-new at atlantis.intranet Received: from maxlor.mine.nu ([127.0.0.1]) by localhost (atlantis.intranet [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id E586gUBOLxr1; Sun, 27 May 2007 18:31:00 +0200 (CEST) Received: from mini.intranet (mini.intranet [10.0.0.17]) by maxlor.mine.nu (Postfix) with ESMTP id 60EFC2E236; Sun, 27 May 2007 18:31:00 +0200 (CEST) From: Benjamin Lutz To: freebsd-hackers@freebsd.org, karma@freebsd.org Date: Sun, 27 May 2007 18:30:56 +0200 User-Agent: KMail/1.9.5 References: <200705250322.22259.karma@FreeBSD.org> <200705252004.38092.mail@maxlor.com> <200705261149.18510.karma@FreeBSD.org> In-Reply-To: <200705261149.18510.karma@FreeBSD.org> X-Face: $Ov27?7*N,h60fIEfNJdb!m,@#4T/d; 1hw|W0zvsHM(a$Yn6BYQ0^SEEXvi8>D`|V*F"=?utf-8?q?=5F+=0A=09R2?=@Aq>+mNb4`,'[[%z9v0Fa~]AD1}xQO3|>b.z&}l#R-_(P`?@Mz"kS; XC>Eti,i3>%@=?utf-8?q?g=3F=0A=094f?=,\c7|Ghwb&ky$b2PJ^\0b83NkLsFKv|smL/cI4UD%Tu8alAD MIME-Version: 1.0 Content-Type: multipart/signed; boundary="nextPart1297929.ZEhfnqmqU5"; protocol="application/pgp-signature"; micalg=pgp-sha1 Content-Transfer-Encoding: 7bit Message-Id: <200705271830.59646.mail@maxlor.com> X-Scanned-By: MIMEDefang 2.61 on 213.160.40.60 Cc: trustedbsd-discuss@freebsd.org Subject: Re: SoC: Distributed Audit Daemon project X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 27 May 2007 16:31:08 -0000 --nextPart1297929.ZEhfnqmqU5 Content-Type: text/plain; charset="koi8-r" Content-Transfer-Encoding: quoted-printable Content-Disposition: inline On Saturday 26 May 2007 09:49, Alexey Mikhailov wrote: > On Friday 25 May 2007 22:04:34 Benjamin Lutz wrote: > > On Friday 25 May 2007 01:22:21 Alexey Mikhailov wrote: > > > [...] > > > 2. As I said before initial subject of this project was > > > "Distributed audit daemon". But after some discussions we had > > > decided that this project can be done in more general maner. We > > > can perform distributed logging for any user-space app. > > > [...] > > > > This sounds very similar to syslogd. Is it feasible to make dlogd a > > drop-in replacement for syslogd, at least from a > > syslog-using-program point of view? > > Our project concentrates on log shipping. We're paying most attention > to securely and reliable log ships. So our project differs from > syslogd in major way. > > But actually it could be possible to be dlogd used by > syslogd\syslog-ng for logs shipping, as I see it. The thing that bugs me most about syslog is not even the transport to=20 remote syslogd instances; that's relatively easy to fix (put some SSL=20 between the daemons, or use encrypted tunnels, etc). It's that when a=20 process logs a syslog event, it can claim to be anything at all. Iirc,=20 it can even give a bogus timestamp. So what I was hoping for here is for auditd to come with a hook that=20 intercepts syslog(3) calls, adds/validates pid, process name and=20 timestamp, and then puts that information somewhere (some local log, a=20 remote log, a lineprinter). It doesn't even have to give the=20 information back to a syslogd daemon; whatever auditd uses for itself=20 would be fine too. What I'm hoping for here is some way to get a guarantee that the=20 information in a log is actually correct. The way it is at the moment,=20 syslog messages are way too trivial to spoof. Anyway, this is just a=20 feature wish :) I'm happy to see you work on auditd, whether or not it=20 contains these syslog bits. Cheers Benjamin --nextPart1297929.ZEhfnqmqU5 Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) iD8DBQBGWbJDzZEjpyKHuQwRAq2iAJoD1nhQu/O3Ot8QAs2JLAf4vDsrVACcC9tG KXQ5a+jxxnoL+HBNQ/WtEns= =A7tz -----END PGP SIGNATURE----- --nextPart1297929.ZEhfnqmqU5-- From owner-freebsd-hackers@FreeBSD.ORG Sun May 27 17:15:14 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 14B5816A478 for ; Sun, 27 May 2007 17:15:14 +0000 (UTC) (envelope-from bzeeb-lists@lists.zabbadoz.net) Received: from transport.cksoft.de (transport.cksoft.de [62.111.66.27]) by mx1.freebsd.org (Postfix) with ESMTP id BBF1F13C4BA for ; Sun, 27 May 2007 17:15:13 +0000 (UTC) (envelope-from bzeeb-lists@lists.zabbadoz.net) Received: from transport.cksoft.de (localhost [127.0.0.1]) by transport.cksoft.de (Postfix) with ESMTP id E8C751FFDE9; Sun, 27 May 2007 19:15:11 +0200 (CEST) Received: by transport.cksoft.de (Postfix, from userid 66) id B149C1FFDD7; Sun, 27 May 2007 19:15:05 +0200 (CEST) Received: from maildrop.int.zabbadoz.net (maildrop.int.zabbadoz.net [10.111.66.10]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.int.zabbadoz.net (Postfix) with ESMTP id 811EC44487F; Sun, 27 May 2007 17:13:04 +0000 (UTC) Date: Sun, 27 May 2007 17:13:04 +0000 (UTC) From: "Bjoern A. Zeeb" X-X-Sender: bz@maildrop.int.zabbadoz.net To: Benjamin Lutz In-Reply-To: <200705271830.59646.mail@maxlor.com> Message-ID: <20070527165641.I13311@maildrop.int.zabbadoz.net> References: <200705250322.22259.karma@FreeBSD.org> <200705252004.38092.mail@maxlor.com> <200705261149.18510.karma@FreeBSD.org> <200705271830.59646.mail@maxlor.com> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Virus-Scanned: by AMaViS cksoft-s20020300-20031204bz on transport.cksoft.de Cc: karma@freebsd.org, freebsd-hackers@freebsd.org, trustedbsd-discuss@freebsd.org Subject: Re: SoC: Distributed Audit Daemon project X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 27 May 2007 17:15:14 -0000 On Sun, 27 May 2007, Benjamin Lutz wrote: Hi, [ log shippping daemon for audit and other other logs ] [ syslog problems ] sorry I have to cut this short;) What Alexey said - this will be about log shipping not about writing single log records etc. Your syslog problems are outside the scope of this topic as long as it does not come to 'shipping' syslogged log files, i.e. after newsyslog rotated them, to another machine and for that hooking into the distributed log (shipping) daemon. Having a generic, more secure and reliable (local) logging mechnism should be discussed in at least another thread. You may as well think of taking this idea to IETF as RFC 3164 lives there as a Memo these days and it might be a general enough thing for everyone. I wonder if there are no lightwight but more secure and reliable implementations out there already. Maybe time for some research. /bz -- Bjoern A. Zeeb bzeeb at Zabbadoz dot NeT From owner-freebsd-hackers@FreeBSD.ORG Sun May 27 17:31:46 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 234BC16A4D2 for ; Sun, 27 May 2007 17:31:46 +0000 (UTC) (envelope-from artifact.one@googlemail.com) Received: from wx-out-0506.google.com (wx-out-0506.google.com [66.249.82.231]) by mx1.freebsd.org (Postfix) with ESMTP id C0F6A13C4CB for ; Sun, 27 May 2007 17:31:45 +0000 (UTC) (envelope-from artifact.one@googlemail.com) Received: by wx-out-0506.google.com with SMTP id h28so888935wxd for ; Sun, 27 May 2007 10:31:45 -0700 (PDT) DKIM-Signature: a=rsa-sha1; c=relaxed/relaxed; d=googlemail.com; s=beta; h=domainkey-signature:received:received:message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; b=m3fDkDb3NTPkhIESrZXT9e4i8QzZDif67BuyAaOK8XpF2oHAiBe7PmNPRDMC2MxhIEe/9WM0A2tHarTnfgegz0cTPeT4nhugRgbMdr9/aPqJhSdee67/JHIb2xTlCZhwgU1rPYwyvWnY2lAWFlbqYyr8/VQtJuWgxqXMghOhPKs= DomainKey-Signature: a=rsa-sha1; c=nofws; d=googlemail.com; s=beta; h=received:message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; b=h7nYzwjhe0vJSg3VXLqsG+gYh50lQqwb6B3GqDHxaxlCXR3nuP6ZxrgfEXPPn/wJ7itoFbqP4RypQxcl+yAUtuvp69oOuFEtnGPDIrUhkPDj2iIVJUVxZvE6vnHQGJ+j+B86rq7iFjXoSrrnk+Jk7nLHJjOOTPGehz+0xOT5HSw= Received: by 10.90.113.20 with SMTP id l20mr3495383agc.1180287105149; Sun, 27 May 2007 10:31:45 -0700 (PDT) Received: by 10.90.102.14 with HTTP; Sun, 27 May 2007 10:31:45 -0700 (PDT) Message-ID: <8e96a0b90705271031s5349f45dt238ff3a9eb53fe0f@mail.gmail.com> Date: Sun, 27 May 2007 18:31:45 +0100 From: "mal content" To: "Bjoern A. Zeeb" In-Reply-To: <20070527165641.I13311@maildrop.int.zabbadoz.net> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <200705250322.22259.karma@FreeBSD.org> <200705252004.38092.mail@maxlor.com> <200705261149.18510.karma@FreeBSD.org> <200705271830.59646.mail@maxlor.com> <20070527165641.I13311@maildrop.int.zabbadoz.net> Cc: Benjamin Lutz , karma@freebsd.org, trustedbsd-discuss@freebsd.org, freebsd-hackers@freebsd.org Subject: Re: SoC: Distributed Audit Daemon project X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 27 May 2007 17:31:46 -0000 On 27/05/07, Bjoern A. Zeeb wrote: > > Having a generic, more secure and reliable (local) logging mechnism > should be discussed in at least another thread. You may as well think > of taking this idea to IETF as RFC 3164 lives there as a Memo these > days and it might be a general enough thing for everyone. I wonder if > there are no lightwight but more secure and reliable implementations > out there already. Maybe time for some research. http://cr.yp.to/daemontools/multilog.html Obviously the license would require somebody to re-implement it under a BSD license and add networking capabilities, but the design is sound. I use it everywhere in place of the stock syslog (takes some re-architecturing - there's no longer a centralised syslog daemon). MC From owner-freebsd-hackers@FreeBSD.ORG Sun May 27 21:06:57 2007 Return-Path: X-Original-To: hackers@freebsd.org Delivered-To: freebsd-hackers@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id D534E16A4BF for ; Sun, 27 May 2007 21:06:57 +0000 (UTC) (envelope-from stephen@math.missouri.edu) Received: from sccmmhc92.asp.att.net (sccmmhc92.asp.att.net [204.127.203.212]) by mx1.freebsd.org (Postfix) with ESMTP id A230213C483 for ; Sun, 27 May 2007 21:06:57 +0000 (UTC) (envelope-from stephen@math.missouri.edu) Received: from laptop2.montlan (12-216-253-187.client.mchsi.com[12.216.253.187]) by sccmmhc92.asp.att.net (sccmmhc92) with ESMTP id <20070527205216m9200d8race>; Sun, 27 May 2007 20:52:16 +0000 Message-ID: <4659EF80.70100@math.missouri.edu> Date: Sun, 27 May 2007 15:52:16 -0500 From: Stephen Montgomery-Smith User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.8.1.2) Gecko/20070525 SeaMonkey/1.1.1 MIME-Version: 1.0 To: ports@freebsd.org, hackers@freebsd.org Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: Subject: Looking for speed increases in "make index" and pkg_version for ports X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 27 May 2007 21:06:57 -0000 I have been thinking a lot about looking for speed increases for "make index" and pkg_version and things like that. So for example, in pkg_version, it calls "make -V PKGNAME" for every installed package. Now "make -V PKGNAME" should be a speedy operation, but the make has to load in and analyze bsd.port.mk, a quite complicated file with about 200,000 characters in it, when all it is needing to do is to figure out the value of the variable PKGNAME. I suggest rewriting "make" so that variables are only evaluated on a "need to know" basis. So, for example, if all we need to know is PKGNAME, there is no need to evaluate, for example, _RUN_LIB_DEPENDS, unless the writer of that particular port has done something like having PORTNAME depend on the value of _RUN_LIB_DEPENDS. So "make" should analyze all the code it is given, and only figure it out if it is needed to do so. This would include, for example, figuring out .for and .if directives on a need to know basis as well. I have only poked around a little inside the source for make, but I have a sense that this would be a major undertaking. I certainly have not thought through what it entails in more than a cursory manner. However I am quite excited about the possibility of doing this, albeit I may well put off the whole thing for a year or two or even forever depending upon other priorities in my life. However, in the mean time I want to throw this idea out there to get some feedback, either of the form of "this won't work," or of the form "I will do it," or "I have tried to do this." Best regards, Stephen From owner-freebsd-hackers@FreeBSD.ORG Sun May 27 22:40:52 2007 Return-Path: X-Original-To: hackers@freebsd.org Delivered-To: freebsd-hackers@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id D930116A4A0 for ; Sun, 27 May 2007 22:40:52 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from sccrmhc14.comcast.net (sccrmhc14.comcast.net [63.240.77.84]) by mx1.freebsd.org (Postfix) with ESMTP id A26BC13C4BB for ; Sun, 27 May 2007 22:40:52 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from icarus.home.lan (c-67-169-189-187.hsd1.ca.comcast.net[67.169.189.187]) by comcast.net (sccrmhc14) with ESMTP id <2007052722304801400devs3e>; Sun, 27 May 2007 22:30:49 +0000 Received: by icarus.home.lan (Postfix, from userid 1000) id 5057B1FA020; Sun, 27 May 2007 15:30:48 -0700 (PDT) Date: Sun, 27 May 2007 15:30:48 -0700 From: Jeremy Chadwick To: Stephen Montgomery-Smith Message-ID: <20070527223048.GA37505@icarus.home.lan> Mail-Followup-To: Stephen Montgomery-Smith , ports@freebsd.org, hackers@freebsd.org References: <4659EF80.70100@math.missouri.edu> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4659EF80.70100@math.missouri.edu> User-Agent: Mutt/1.5.15 (2007-04-06) Cc: ports@freebsd.org, hackers@freebsd.org Subject: Re: Looking for speed increases in "make index" and pkg_version for ports X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 27 May 2007 22:40:52 -0000 On Sun, May 27, 2007 at 03:52:16PM -0500, Stephen Montgomery-Smith wrote: > I have been thinking a lot about looking for speed increases for "make > index" and pkg_version and things like that. So for example, in > pkg_version, it calls "make -V PKGNAME" for every installed package. Now > "make -V PKGNAME" should be a speedy operation, but the make has to load in > and analyze bsd.port.mk, a quite complicated file with about 200,000 > characters in it, when all it is needing to do is to figure out the value of > the variable PKGNAME. I have a related question, pertaining to "make all-depends-list" and the utter atrocity that is the make variable ALL-DEPENDS-LIST. If you don't know what it is, look for ^ALL-DEPENDS-LIST around line 5175, in bsd.ports.mk. I call it an atrocity because it's a mix of make variable expansion combined with sh scripting, and it's nearly impossible to read. It's not commented either, so folks like myself are left thinking "What IS this mess?!". It's expanded via $$(${ALL-DEPENDS-LIST}) in for loops, throughout several places in bsd.port.mk. I do not entirely understand what ALL-DEPENDS-LIST is about (that should be apparent), but upon performing some of my benchmarks, I found this to be a very slow piece of bsd.port.mk. make -V _DEPEND_DIRS is incredibly fast, but ALL-DEPENDS-LIST is not. Does it need to be done this way? Can we just iterate through all of the ports, call make -V _DEPEND_DIRS, then sort | uniq the results? I suppose that depends on the operation (make vs. make clean vs. others)... The port I used for testing some of the benchmarks was net/gacxtool. It seems to be a good example of a "hefty" port. > I suggest rewriting "make" so that variables are only evaluated on a "need > to know" basis. So, for example, if all we need to know is PKGNAME, there > is no need to evaluate, for example, _RUN_LIB_DEPENDS, unless the writer of > that particular port has done something like having PORTNAME depend on the > value of _RUN_LIB_DEPENDS. So "make" should analyze all the code it is > given, and only figure it out if it is needed to do so. This would include, > for example, figuring out .for and .if directives on a need to know basis as > well. This sounds like a good solution. In fact, I'm lead to believe that heavy reliance on /bin/sh is part of why the ports collection is slow. No, it's not the sole reason, but it's one of many. I'm of the belief that anything we can do to migrate portions into native make would be benefitial. That said, I'll ask this out in the open: am I the only one who sees the benefit of GNU make in regards to this? There's a lot of built-in functions in GNU make which could help in regards to ports. I have no qualms with PMake per se, but if another tool gives us what we need, then maybe we should consider the pros and cons of adapting that. There's also CMake, which is incredibly fast. -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB | From owner-freebsd-hackers@FreeBSD.ORG Sun May 27 23:39:59 2007 Return-Path: X-Original-To: hackers@freebsd.org Delivered-To: freebsd-hackers@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 7A8AE16A4B3; Sun, 27 May 2007 23:39:59 +0000 (UTC) (envelope-from bakul@bitblocks.com) Received: from mail.bitblocks.com (mail.bitblocks.com [64.142.15.60]) by mx1.freebsd.org (Postfix) with ESMTP id 55FCB13C4BC; Sun, 27 May 2007 23:39:58 +0000 (UTC) (envelope-from bakul@bitblocks.com) Received: from bitblocks.com (localhost.bitblocks.com [127.0.0.1]) by mail.bitblocks.com (Postfix) with ESMTP id D0C325B5E; Sun, 27 May 2007 16:12:54 -0700 (PDT) To: Stephen Montgomery-Smith In-reply-to: Your message of "Sun, 27 May 2007 15:52:16 CDT." <4659EF80.70100@math.missouri.edu> Date: Sun, 27 May 2007 16:12:54 -0700 From: Bakul Shah Message-Id: <20070527231254.D0C325B5E@mail.bitblocks.com> Cc: ports@freebsd.org, hackers@freebsd.org Subject: Re: Looking for speed increases in "make index" and pkg_version for ports X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 27 May 2007 23:39:59 -0000 Not quite what you asked for but... Given the size and complexity of the port system I have long felt that rather than do everything via more and more complex Mk/*.mk what is is needed is a ports server and a thin CLI frontend to it. This server can store dependency data in an efficient manner, deal with conditional dependencies, port renames, security and what not. It can build or fetch or serve packages, handle updates etc. Things mentioned in UPDATING file can instead be done by the server. In general it can automate a lot of stuff, remove error prone redundancies etc. If it is small enough and written in C, it can even be shipped with the base system instead of various pkg_* programs. It can provide two interfaces, one for normal users (with commands like add, check, config, delete, info, search, update, which) and one for port developers (command for adding/remove/renaming ports, etc.). Initially it must work with existing Makefiles. > I have been thinking a lot about looking for speed increases for "make > index" and pkg_version and things like that. So for example, in > pkg_version, it calls "make -V PKGNAME" for every installed package. > Now "make -V PKGNAME" should be a speedy operation, but the make has to > load in and analyze bsd.port.mk, a quite complicated file with about > 200,000 characters in it, when all it is needing to do is to figure out > the value of the variable PKGNAME. > > I suggest rewriting "make" so that variables are only evaluated on a > "need to know" basis. So, for example, if all we need to know is > PKGNAME, there is no need to evaluate, for example, _RUN_LIB_DEPENDS, > unless the writer of that particular port has done something like having > PORTNAME depend on the value of _RUN_LIB_DEPENDS. So "make" should > analyze all the code it is given, and only figure it out if it is needed > to do so. This would include, for example, figuring out .for and .if > directives on a need to know basis as well. > > I have only poked around a little inside the source for make, but I have > a sense that this would be a major undertaking. I certainly have not > thought through what it entails in more than a cursory manner. However > I am quite excited about the possibility of doing this, albeit I may > well put off the whole thing for a year or two or even forever depending > upon other priorities in my life. > > However, in the mean time I want to throw this idea out there to get > some feedback, either of the form of "this won't work," or of the form > "I will do it," or "I have tried to do this." > > Best regards, Stephen > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" From owner-freebsd-hackers@FreeBSD.ORG Sun May 27 22:15:33 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 152D716A41F for ; Sun, 27 May 2007 22:15:33 +0000 (UTC) (envelope-from michel@lpthe.jussieu.fr) Received: from shiva.jussieu.fr (shiva.jussieu.fr [134.157.0.129]) by mx1.freebsd.org (Postfix) with ESMTP id AB34613C469 for ; Sun, 27 May 2007 22:15:32 +0000 (UTC) (envelope-from michel@lpthe.jussieu.fr) Received: from parthe.lpthe.jussieu.fr (parthe.lpthe.jussieu.fr [134.157.10.1]) by shiva.jussieu.fr (8.13.8/jtpda-5.4) with ESMTP id l4RMFUHF045047 ; Mon, 28 May 2007 00:15:31 +0200 (CEST) X-Ids: 164 Received: by parthe.lpthe.jussieu.fr (Postfix, from userid 10096) id DA39FBF69A; Mon, 28 May 2007 00:15:29 +0200 (CEST) X-Spam-Checker-Version: SpamAssassin 3.1.8 (2007-02-13) on parthe.lpthe.jussieu.fr X-Spam-Level: X-Spam-Status: No, score=-3.2 required=5.0 tests=ALL_TRUSTED,AWL,BAYES_00 autolearn=ham version=3.1.8 Received: from niobe.lpthe.jussieu.fr (niobe.lpthe.jussieu.fr [134.157.10.41]) by parthe.lpthe.jussieu.fr (Postfix) with ESMTP id 105FEBF5A0; Mon, 28 May 2007 00:15:29 +0200 (CEST) Received: by niobe.lpthe.jussieu.fr (Postfix, from userid 2005) id E91BC80; Mon, 28 May 2007 00:15:28 +0200 (CEST) Date: Mon, 28 May 2007 00:15:28 +0200 From: Michel Talon To: freebsd-hackers@freebsd.org Message-ID: <20070527221528.GA19603@lpthe.jussieu.fr> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.4.2.2i X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-3.0 (shiva.jussieu.fr [134.157.0.164]); Mon, 28 May 2007 00:15:31 +0200 (CEST) X-Virus-Scanned: ClamAV 0.88.7/3305/Sun May 27 20:19:57 2007 on shiva.jussieu.fr X-Virus-Status: Clean X-Miltered: at shiva.jussieu.fr with ID 465A0302.004 by Joe's j-chkmail (http://j-chkmail.ensmp.fr)! X-Mailman-Approved-At: Mon, 28 May 2007 01:02:40 +0000 Cc: ports@freebsd.org Subject: Re: Looking for speed increases in "make index" X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 27 May 2007 22:15:33 -0000 Stephen Montgomery-Smith said: > I suggest rewriting "make" so that variables are only evaluated on a > "need to know" basis. .... > or "I have tried to do this." Of course a lot of people have thinked about it, and quickly realized that it was not going to work. In the bsd.ports.mk, evaluation of one variable may be dependent on some conditional, which may itself be dependant on some other variable, appearing as some target. This constantly happens in bsd.ports.mk. If you think about that, you convince yourself that a "reduced make" needs to understand targets, needs to understand conditionals, and needs to evaluate not only the variable under consideration, but but possibly any other. In other words, you need a full blown make. To gain some performance, a first idea would be to simplify bsd.ports.mk. I am convinced that a substantial part of the 4000 lines are historical crap which serve no useful purpose. There are tons of variables who have probably purely anecdotical interest. There are targets which could be happily suppressed. Of course, due to the complexity of bsd.ports.mk, rewriting it is certainly not an easy task. There is a freebsd port whose aim is to rewrite it, i don't know how far they are. The NetBSD people have completely rewritten the system, i have no idea if the new one is faster. The OpenBSD people have also rewritten bsd.ports.mk, with apparently much faster makefile. A second idea would be to multithread make, since modern machines will have a lot of cores. At present make -j forks submakes and waits for their completion. This doesn't help for the problem at hand. However, multithreading the execution of a single makefile is certainly not trivial due to the interdependencies. Anyways, one of the things which cannot be a big factor is reading bsd.ports.mk from hard disk. It is certainly cached in memory when you run make index. On the other hand it is so big that it probably doesn't fit in cache, or probably only fits in caches of machines generously endowed. I have remarked that the difference of execution speed of make index between machines of similar speed but very different cache size (i am thinking Pentium 4 versus Core 2 Duo) is striking. It is less than 10 minutes on the big cache machine versus 30 minutes on the small cache one. If the cache effect is indeed dominant, then reducing the size of bsd.ports.mk by all possible means would be very beneficial. By the way an alternative system would be to use something other than make to do the job. This is what the Gentoo people are doing, using first a python based system, and now a C++ one (paludis). Here also i don't have any idea if it is faster. -- Michel TALON From owner-freebsd-hackers@FreeBSD.ORG Mon May 28 01:32:04 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 2222016A488 for ; Mon, 28 May 2007 01:32:04 +0000 (UTC) (envelope-from stephen@math.missouri.edu) Received: from math.missouri.edu (math.missouri.edu [128.206.184.200]) by mx1.freebsd.org (Postfix) with ESMTP id E0EE413C44C for ; Mon, 28 May 2007 01:32:03 +0000 (UTC) (envelope-from stephen@math.missouri.edu) Received: from math.missouri.edu (localhost [127.0.0.1]) by math.missouri.edu (8.13.1/8.13.1) with ESMTP id l4S1HjOn011712; Sun, 27 May 2007 20:17:45 -0500 (CDT) (envelope-from stephen@math.missouri.edu) Received: from localhost (stephen@localhost) by math.missouri.edu (8.13.1/8.13.1/Submit) with ESMTP id l4S1HjgN011709; Sun, 27 May 2007 20:17:45 -0500 (CDT) (envelope-from stephen@math.missouri.edu) Date: Sun, 27 May 2007 20:17:44 -0500 (CDT) From: Stephen Montgomery-Smith To: Michel Talon In-Reply-To: <20070527221528.GA19603@lpthe.jussieu.fr> Message-ID: <20070527200954.C11340@math.missouri.edu> References: <20070527221528.GA19603@lpthe.jussieu.fr> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: ports@freebsd.org, freebsd-hackers@freebsd.org Subject: Re: Looking for speed increases in "make index" X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 May 2007 01:32:04 -0000 On Mon, 28 May 2007, Michel Talon wrote: > Stephen Montgomery-Smith said: > >> I suggest rewriting "make" so that variables are only evaluated on a >> "need to know" basis. .... >> or "I have tried to do this." > > Of course a lot of people have thinked about it, and quickly realized > that it was not going to work. In the bsd.ports.mk, evaluation of one > variable may be dependent on some conditional, which may itself be > dependant on some other variable, appearing as some target. This > constantly happens in bsd.ports.mk. > > If you think about that, you convince yourself that a "reduced make" > needs to understand targets, needs to understand conditionals, and needs > to evaluate not only the variable under consideration, but but possibly > any other. In other words, you need a full blown make. Actually I have thought quite a lot about this over the last couple of weeks. You are correct in that the "make" really does have to be a full blown make. What I am suggesting is something rather different - a rather sophisticated make that works hard to only that which it really has to do. And it certainly would require a lot of sophistication, precisely for the reasons you state. > > To gain some performance, a first idea would be to simplify > bsd.ports.mk. I am convinced that a substantial part of the 4000 lines > are historical crap which serve no useful purpose. There are tons of > variables who have probably purely anecdotical interest. There are > targets which could be happily suppressed. Of course, due to the > complexity of bsd.ports.mk, rewriting it is certainly not an easy task. > There is a freebsd port whose aim is to rewrite it, i don't know how > far they are. The NetBSD people have completely rewritten the system, i > have no idea if the new one is faster. The OpenBSD people have also > rewritten bsd.ports.mk, with apparently much faster makefile. > I rewrote a bsd.ports.mk in which ALL targets were removed, and all it did was evalute variables. (I wrote a perl script that did this automatically.) The net effect was no speed gain. In other words, processing unnecessary targets is not the problem. I really have looked at bsd.ports.mk a lot, and given that it is written in the context of what make requires, it seems to be rather well and efficiently written. I did find one speed increase in bsd.gnome.mk, which I am kind of proud of, but even that only gained about a 5-10% speed increase in "make index." > A second idea would be to multithread make, since modern machines will > have a lot of cores. At present make -j forks submakes > and waits for their completion. This doesn't help for the problem at > hand. However, multithreading the execution of a single makefile is > certainly not trivial due to the interdependencies. > I don't think multithreaded make will help in this situation. > Anyways, one of the things which cannot be a big factor is reading > bsd.ports.mk from hard disk. It is certainly cached in memory when you > run make index. On the other hand it is so big that it probably doesn't > fit in cache, or probably only fits in caches of machines generously > endowed. I have remarked that the difference of execution speed of make > index between machines of similar speed but very different cache size > (i am thinking Pentium 4 versus Core 2 Duo) is striking. It is less > than 10 minutes on the big cache machine versus 30 minutes on the small > cache one. If the cache effect is indeed dominant, then reducing the > size of bsd.ports.mk by all possible means would be very beneficial. > My suspicion is that the timne taken to read bsd.ports.mk from the hard disk is a rather small part of what takes it so long. Compare doing a "make" with a simple "cat" and the time differences are very great. I think make spends far more time processing the file than reading it. > By the way an alternative system would be to use something other than > make to do the job. This is what the Gentoo people are doing, using > first a python based system, and now a C++ one (paludis). Here also i > don't have any idea if it is faster. > It would be nice to find a solution that doesn't involve redesigning the whole ports process from scratch. > > > > -- > > Michel TALON > > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" > From owner-freebsd-hackers@FreeBSD.ORG Mon May 28 01:43:50 2007 Return-Path: X-Original-To: hackers@freebsd.org Delivered-To: freebsd-hackers@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 6142316A4A0; Mon, 28 May 2007 01:43:50 +0000 (UTC) (envelope-from stephen@math.missouri.edu) Received: from sccmmhc91.asp.att.net (sccmmhc91.asp.att.net [204.127.203.211]) by mx1.freebsd.org (Postfix) with ESMTP id 1797D13C43E; Mon, 28 May 2007 01:43:49 +0000 (UTC) (envelope-from stephen@math.missouri.edu) Received: from laptop2.montlan (12-216-253-187.client.mchsi.com[12.216.253.187]) by sccmmhc91.asp.att.net (sccmmhc91) with ESMTP id <20070528014348m9100d38lue>; Mon, 28 May 2007 01:43:49 +0000 Message-ID: <465A33D4.1040706@math.missouri.edu> Date: Sun, 27 May 2007 20:43:48 -0500 From: Stephen Montgomery-Smith User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.8.1.2) Gecko/20070525 SeaMonkey/1.1.1 MIME-Version: 1.0 To: Stephen Montgomery-Smith , ports@freebsd.org, hackers@freebsd.org References: <4659EF80.70100@math.missouri.edu> <20070527223048.GA37505@icarus.home.lan> In-Reply-To: <20070527223048.GA37505@icarus.home.lan> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: Subject: Re: Looking for speed increases in "make index" and pkg_version for ports X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 May 2007 01:43:50 -0000 Jeremy Chadwick wrote: > On Sun, May 27, 2007 at 03:52:16PM -0500, Stephen Montgomery-Smith wrote: >> I have been thinking a lot about looking for speed increases for "make >> index" and pkg_version and things like that. So for example, in >> pkg_version, it calls "make -V PKGNAME" for every installed package. Now >> "make -V PKGNAME" should be a speedy operation, but the make has to load in >> and analyze bsd.port.mk, a quite complicated file with about 200,000 >> characters in it, when all it is needing to do is to figure out the value of >> the variable PKGNAME. > > I have a related question, pertaining to "make all-depends-list" and the > utter atrocity that is the make variable ALL-DEPENDS-LIST. If you don't > know what it is, look for ^ALL-DEPENDS-LIST around line 5175, in > bsd.ports.mk. > > I call it an atrocity because it's a mix of make variable expansion > combined with sh scripting, and it's nearly impossible to read. It's > not commented either, so folks like myself are left thinking "What IS > this mess?!". It's expanded via $$(${ALL-DEPENDS-LIST}) in for loops, > throughout several places in bsd.port.mk. > > I do not entirely understand what ALL-DEPENDS-LIST is about (that should > be apparent), but upon performing some of my benchmarks, I found this to > be a very slow piece of bsd.port.mk. make -V _DEPEND_DIRS is incredibly > fast, but ALL-DEPENDS-LIST is not. > > Does it need to be done this way? Can we just iterate through all of > the ports, call make -V _DEPEND_DIRS, then sort | uniq the results? I > suppose that depends on the operation (make vs. make clean vs. > others)... > > The port I used for testing some of the benchmarks was net/gacxtool. It > seems to be a good example of a "hefty" port. I looked very hard at this particular piece of code. Once you understand how it works, you realize that it is rather well written. It basically does what you suggest, except it keeps a list of ports it has already checked, so that it doesn't do them over again. This piece of code is as efficient as it can possibly be, given that the program has to recursively call make on every dependency port at least once (and as it happens only once). > >> I suggest rewriting "make" so that variables are only evaluated on a "need >> to know" basis. So, for example, if all we need to know is PKGNAME, there >> is no need to evaluate, for example, _RUN_LIB_DEPENDS, unless the writer of >> that particular port has done something like having PORTNAME depend on the >> value of _RUN_LIB_DEPENDS. So "make" should analyze all the code it is >> given, and only figure it out if it is needed to do so. This would include, >> for example, figuring out .for and .if directives on a need to know basis as >> well. > > This sounds like a good solution. In fact, I'm lead to believe that > heavy reliance on /bin/sh is part of why the ports collection is slow. > No, it's not the sole reason, but it's one of many. I'm of the belief > that anything we can do to migrate portions into native make would be > benefitial. I have done profiling tests on make, and in its current form, bsd.ports.mk actually spends rather little time inside of bin/sh. Thw writers of bsd.ports.mk have done a very good job of minimizing the bin/sh calls. > > That said, I'll ask this out in the open: am I the only one who sees the > benefit of GNU make in regards to this? There's a lot of built-in > functions in GNU make which could help in regards to ports. I have no > qualms with PMake per se, but if another tool gives us what we need, > then maybe we should consider the pros and cons of adapting that. > There's also CMake, which is incredibly fast. > Maybe I should look at the inner workings of cmake and gmake. Maybe they have some good ideas. However having looked through the source code of make, and also looking at the cvs logs, it does seem to be well written. The only possibility I see of making it go a lot faster is a complete redesign, e.g. my just in time idea for processing variables. Stephen From owner-freebsd-hackers@FreeBSD.ORG Mon May 28 01:47:24 2007 Return-Path: X-Original-To: hackers@freebsd.org Delivered-To: freebsd-hackers@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 590FB16A400; Mon, 28 May 2007 01:47:24 +0000 (UTC) (envelope-from stephen@math.missouri.edu) Received: from sccmmhc92.asp.att.net (sccmmhc92.asp.att.net [204.127.203.212]) by mx1.freebsd.org (Postfix) with ESMTP id 1049C13C455; Mon, 28 May 2007 01:47:23 +0000 (UTC) (envelope-from stephen@math.missouri.edu) Received: from laptop2.montlan (12-216-253-187.client.mchsi.com[12.216.253.187]) by sccmmhc92.asp.att.net (sccmmhc92) with ESMTP id <20070528014722m9200d9nk8e>; Mon, 28 May 2007 01:47:23 +0000 Message-ID: <465A34AA.8000002@math.missouri.edu> Date: Sun, 27 May 2007 20:47:22 -0500 From: Stephen Montgomery-Smith User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.8.1.2) Gecko/20070525 SeaMonkey/1.1.1 MIME-Version: 1.0 To: Bakul Shah References: <20070527231254.D0C325B5E@mail.bitblocks.com> In-Reply-To: <20070527231254.D0C325B5E@mail.bitblocks.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: ports@freebsd.org, hackers@freebsd.org Subject: Re: Looking for speed increases in "make index" and pkg_version for ports X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 May 2007 01:47:24 -0000 I'm looking for something that will work with the existing framework. But yes, I get the feeling that maybe using "make" to process the ports might be the source of the problem. Make is a program primarily designed for figuring out which was made first, the target or the source, but in the ports what we really want is a scripting language that presides over "cd WKSRC; make". (P.S. sorry for top-posting, but I am following your lead.) Bakul Shah wrote: > Not quite what you asked for but... > > Given the size and complexity of the port system I have long > felt that rather than do everything via more and more complex > Mk/*.mk what is is needed is a ports server and a thin CLI > frontend to it. > > This server can store dependency data in an efficient manner, > deal with conditional dependencies, port renames, security > and what not. It can build or fetch or serve packages, > handle updates etc. Things mentioned in UPDATING file can > instead be done by the server. In general it can automate a > lot of stuff, remove error prone redundancies etc. If it is > small enough and written in C, it can even be shipped with > the base system instead of various pkg_* programs. > > It can provide two interfaces, one for normal users (with > commands like add, check, config, delete, info, search, > update, which) and one for port developers (command for > adding/remove/renaming ports, etc.). Initially it must work > with existing Makefiles. > >> I have been thinking a lot about looking for speed increases for "make >> index" and pkg_version and things like that. So for example, in >> pkg_version, it calls "make -V PKGNAME" for every installed package. >> Now "make -V PKGNAME" should be a speedy operation, but the make has to >> load in and analyze bsd.port.mk, a quite complicated file with about >> 200,000 characters in it, when all it is needing to do is to figure out >> the value of the variable PKGNAME. >> >> I suggest rewriting "make" so that variables are only evaluated on a >> "need to know" basis. So, for example, if all we need to know is >> PKGNAME, there is no need to evaluate, for example, _RUN_LIB_DEPENDS, >> unless the writer of that particular port has done something like having >> PORTNAME depend on the value of _RUN_LIB_DEPENDS. So "make" should >> analyze all the code it is given, and only figure it out if it is needed >> to do so. This would include, for example, figuring out .for and .if >> directives on a need to know basis as well. >> >> I have only poked around a little inside the source for make, but I have >> a sense that this would be a major undertaking. I certainly have not >> thought through what it entails in more than a cursory manner. However >> I am quite excited about the possibility of doing this, albeit I may >> well put off the whole thing for a year or two or even forever depending >> upon other priorities in my life. >> >> However, in the mean time I want to throw this idea out there to get >> some feedback, either of the form of "this won't work," or of the form >> "I will do it," or "I have tried to do this." >> >> Best regards, Stephen >> _______________________________________________ >> freebsd-hackers@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-hackers >> To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" > > From owner-freebsd-hackers@FreeBSD.ORG Mon May 28 02:18:01 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id C4A5D16A4B3 for ; Mon, 28 May 2007 02:18:01 +0000 (UTC) (envelope-from edwin@mavetju.org) Received: from mail4out.barnet.com.au (mail4.barnet.com.au [202.83.178.125]) by mx1.freebsd.org (Postfix) with ESMTP id 5B3EE13C487 for ; Mon, 28 May 2007 02:18:01 +0000 (UTC) (envelope-from edwin@mavetju.org) Received: by mail4out.barnet.com.au (Postfix, from userid 1001) id 3133537D751; Mon, 28 May 2007 12:01:44 +1000 (EST) X-Viruscan-Id: <465A38080000A9C4CE97B0@BarNet> Received: from mail4auth.barnet.com.au (mail4.barnet.com.au [202.83.178.125]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mail4.barnet.com.au (Postfix) with ESMTP id EF0974220C8; Mon, 28 May 2007 12:01:43 +1000 (EST) Received: from k7.mavetju (k7.mavetju.org [10.251.1.18]) by mail4auth.barnet.com.au (Postfix) with ESMTP id B0C6D37D402; Mon, 28 May 2007 12:01:43 +1000 (EST) Received: by k7.mavetju (Postfix, from userid 1001) id 8536F423; Mon, 28 May 2007 12:01:43 +1000 (EST) Date: Mon, 28 May 2007 12:01:43 +1000 From: Edwin Groothuis To: ports@freebsd.org, freebsd-hackers@freebsd.org Message-ID: <20070528020143.GB1361@k7.mavetju> Mail-Followup-To: Edwin Groothuis , ports@freebsd.org, freebsd-hackers@freebsd.org References: <20070527221528.GA19603@lpthe.jussieu.fr> <20070528014456.GA24097@soaustin.net> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20070528014456.GA24097@soaustin.net> User-Agent: Mutt/1.4.2.2i Cc: Subject: Re: Looking for speed increases in "make index" X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 May 2007 02:18:01 -0000 On Sun, May 27, 2007 at 08:44:56PM -0500, Mark Linimon wrote: > In summary, the ports infrastructure is really complicated because it's > trying to deal with all kinds of constraints and conditions. I challenge Reading this, I was wondering what the ports infrastructure has ever done for us? See http://www.epicure.demon.co.uk/whattheromans.html Edwin -- Edwin Groothuis | Personal website: http://www.mavetju.org edwin@mavetju.org | Weblog: http://www.mavetju.org/weblog/ From owner-freebsd-hackers@FreeBSD.ORG Mon May 28 02:15:42 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 01D3616A468 for ; Mon, 28 May 2007 02:15:42 +0000 (UTC) (envelope-from linimon@lonesome.com) Received: from mail.soaustin.net (mail.soaustin.net [207.200.4.66]) by mx1.freebsd.org (Postfix) with ESMTP id DC6D713C458 for ; Mon, 28 May 2007 02:15:41 +0000 (UTC) (envelope-from linimon@lonesome.com) Received: by mail.soaustin.net (Postfix, from userid 502) id ED99FAC3; Sun, 27 May 2007 20:44:56 -0500 (CDT) Date: Sun, 27 May 2007 20:44:56 -0500 To: Michel Talon Message-ID: <20070528014456.GA24097@soaustin.net> References: <20070527221528.GA19603@lpthe.jussieu.fr> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20070527221528.GA19603@lpthe.jussieu.fr> User-Agent: Mutt/1.5.9i From: linimon@lonesome.com (Mark Linimon) X-Mailman-Approved-At: Mon, 28 May 2007 02:36:45 +0000 Cc: ports@freebsd.org, freebsd-hackers@freebsd.org Subject: Re: Looking for speed increases in "make index" X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 May 2007 02:15:42 -0000 On Mon, May 28, 2007 at 12:15:28AM +0200, Michel Talon wrote: > To gain some performance, a first idea would be to simplify > bsd.ports.mk. I am convinced that a substantial part of the 4000 lines > are historical crap which serve no useful purpose. 11272 of LOC in bsd.*.mk, but who's counting. > There are tons of variables who have probably purely anecdotical > interest. There are targets which could be happily suppressed. Please let us know which functionality you think is extra. You should use the individual port Makefiles as well as ports/Makefile to figure out what is unused. For extra credit, please include ports/Tools/portbuild/scripts so the build cluster will continue to work. Please don't think I am picking on you specifically; however, about every 6 months or so someone decides that "the ports framework is too complicated" without saying exactly what needs to be removed. Since I look at all the portmgr PRs as they come in, and participate in rejecting in some of the (by our determination) more marginal features, I can assure you that not every single new proposal makes it in there, nor has in the past. Every- thing that's in there is because there was some specific justification for it (at least at the time). Given that we had no install base, a significant rewrite would not be a burden, but that's not the case. Please note, I've agreed for several years that a great deal of the code could be factored out into some kind of C library for speed and reduction of code duplication. Some work is going towards that in the Summer of Code. But the hard part is making it work, in a backwards compatible fashion, and doing the exhaustive regression testing to prove that it solves more problems that it creates. (portmgr spends a _lot_ of time on regression testing, behind the scenes.) In summary, the ports infrastructure is really complicated because it's trying to deal with all kinds of constraints and conditions. I challenge anyone who thinks things can be removed to roll up their sleeves and make a good case for it. I'd be happy to have something easier to read. mcl From owner-freebsd-hackers@FreeBSD.ORG Mon May 28 02:52:41 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 3EE1C16A4A0 for ; Mon, 28 May 2007 02:52:41 +0000 (UTC) (envelope-from mwm-keyword-freebsdhackers2.e313df@mired.org) Received: from mired.org (vpn.mired.org [66.92.153.74]) by mx1.freebsd.org (Postfix) with SMTP id C15D213C480 for ; Mon, 28 May 2007 02:52:40 +0000 (UTC) (envelope-from mwm-keyword-freebsdhackers2.e313df@mired.org) Received: (qmail 40645 invoked by uid 1001); 28 May 2007 02:51:56 -0000 Received: by bhuda.mired.org (tmda-sendmail, from uid 1001); Sun, 27 May 2007 22:51:56 -0400 MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <18010.17356.231693.31047@bhuda.mired.org> Date: Sun, 27 May 2007 22:51:56 -0400 To: Edwin Groothuis In-Reply-To: <20070528020143.GB1361@k7.mavetju> References: <20070527221528.GA19603@lpthe.jussieu.fr> <20070528014456.GA24097@soaustin.net> <20070528020143.GB1361@k7.mavetju> X-Mailer: VM 7.19 under Emacs 21.3.1 X-Primary-Address: mwm@mired.org X-face: "5Mnwy%?j>IIV\)A=):rjWL~NB2aH[}Yq8Z=u~vJ`"(,&SiLvbbz2W`; h9L,Yg`+vb1>RG% *h+%X^n0EZd>TM8_IB;a8F?(Fb"lw'IgCoyM.[Lg#r\ X-Delivery-Agent: TMDA/1.1.11 (Ladyburn) From: Mike Meyer Cc: ports@freebsd.org, freebsd-hackers@freebsd.org Subject: Re: Looking for speed increases in "make index" X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 May 2007 02:52:41 -0000 In <20070528020143.GB1361@k7.mavetju>, Edwin Groothuis typed: > On Sun, May 27, 2007 at 08:44:56PM -0500, Mark Linimon wrote: > > In summary, the ports infrastructure is really complicated because it's > > trying to deal with all kinds of constraints and conditions. I challenge > Reading this, I was wondering what the ports infrastructure has > ever done for us? > See http://www.epicure.demon.co.uk/whattheromans.html While that's funny, it makes me wonder if you're serious when you ask the question. The ports system is a wonder. If you've ever tried installing software off the net without such a thing to help, you'll know what I mean. If you haven't, you should thank jkh for your state of blessedness. That said, it's now a decade old, and I'm sure doing far more than jkh ever expected of it. It's also tightly integarted with the package system, which is in a similar state. Both are suffering in comparison to newer systems, many of which have less ambitious goals. It seems like in the last month or so a lot of people have popped up with an interest in reworking one or both of them. Hopefully, some of them with time to do so will get good advice. http://www.mired.org/consulting.html Independent Network/Unix/Perforce consultant, email for more information. From owner-freebsd-hackers@FreeBSD.ORG Mon May 28 03:41:25 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id F3FBB16A474; Mon, 28 May 2007 03:41:24 +0000 (UTC) (envelope-from edwin@mavetju.org) Received: from mail4out.barnet.com.au (mail4.barnet.com.au [202.83.178.125]) by mx1.freebsd.org (Postfix) with ESMTP id B03C013C44B; Mon, 28 May 2007 03:41:24 +0000 (UTC) (envelope-from edwin@mavetju.org) Received: by mail4out.barnet.com.au (Postfix, from userid 1001) id 771BE37D4DC; Mon, 28 May 2007 13:41:23 +1000 (EST) X-Viruscan-Id: <465A4F63000083D386A6C3@BarNet> Received: from mail4auth.barnet.com.au (mail4.barnet.com.au [202.83.178.125]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mail4.barnet.com.au (Postfix) with ESMTP id 1B2D042208D; Mon, 28 May 2007 13:41:23 +1000 (EST) Received: from k7.mavetju (k7.mavetju.org [10.251.1.18]) by mail4auth.barnet.com.au (Postfix) with ESMTP id D22EA37D402; Mon, 28 May 2007 13:41:22 +1000 (EST) Received: by k7.mavetju (Postfix, from userid 1001) id A2DFB5C8; Mon, 28 May 2007 13:41:22 +1000 (EST) Date: Mon, 28 May 2007 13:41:22 +1000 From: Edwin Groothuis To: Mike Meyer Message-ID: <20070528034122.GA1646@k7.mavetju> Mail-Followup-To: Edwin Groothuis , Mike Meyer , ports@freebsd.org, freebsd-hackers@freebsd.org References: <20070527221528.GA19603@lpthe.jussieu.fr> <20070528014456.GA24097@soaustin.net> <20070528020143.GB1361@k7.mavetju> <18010.17356.231693.31047@bhuda.mired.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <18010.17356.231693.31047@bhuda.mired.org> User-Agent: Mutt/1.4.2.2i Cc: ports@freebsd.org, freebsd-hackers@freebsd.org Subject: Re: Looking for speed increases in "make index" X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 May 2007 03:41:25 -0000 On Sun, May 27, 2007 at 10:51:56PM -0400, Mike Meyer wrote: > In <20070528020143.GB1361@k7.mavetju>, Edwin Groothuis typed: > > On Sun, May 27, 2007 at 08:44:56PM -0500, Mark Linimon wrote: > > > In summary, the ports infrastructure is really complicated because it's > > > trying to deal with all kinds of constraints and conditions. I challenge > > Reading this, I was wondering what the ports infrastructure has > > ever done for us? > > See http://www.epicure.demon.co.uk/whattheromans.html > > While that's funny, it makes me wonder if you're serious when you ask > the question. No I wasn't serious, I was sarcastic. I've spend enough time working in ports/Mk to know what is has done for us. Edwin -- Edwin Groothuis | Personal website: http://www.mavetju.org edwin@mavetju.org | Weblog: http://www.mavetju.org/weblog/ From owner-freebsd-hackers@FreeBSD.ORG Mon May 28 08:13:58 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 3D35016A421; Mon, 28 May 2007 08:13:58 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from cyrus.watson.org (cyrus.watson.org [209.31.154.42]) by mx1.freebsd.org (Postfix) with ESMTP id 0BFF413C44B; Mon, 28 May 2007 08:13:57 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from fledge.watson.org (fledge.watson.org [209.31.154.41]) by cyrus.watson.org (Postfix) with ESMTP id 6780A470BA; Mon, 28 May 2007 04:13:57 -0400 (EDT) Date: Mon, 28 May 2007 09:13:57 +0100 (BST) From: Robert Watson X-X-Sender: robert@fledge.watson.org To: "M. Warner Losh" In-Reply-To: <20070526.021609.-1749708199.imp@bsdimp.com> Message-ID: <20070528090910.X2234@fledge.watson.org> References: <200705250322.22259.karma@FreeBSD.org> <200705252004.38092.mail@maxlor.com> <20070526.021609.-1749708199.imp@bsdimp.com> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: trustedbsd-audit@freebsd.org, karma@ez.pereslavl.ru, karma@freebsd.org, freebsd-hackers@freebsd.org, trustedbsd-discuss@freebsd.org, mail@maxlor.com Subject: Re: SoC: Distributed Audit Daemon project X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 May 2007 08:13:58 -0000 On Sat, 26 May 2007, M. Warner Losh wrote: > In message: <200705252004.38092.mail@maxlor.com> > Benjamin Lutz writes: > : On Friday 25 May 2007 01:22:21 Alexey Mikhailov wrote: > : > [...] > : > 2. As I said before initial subject of this project was "Distributed > : > audit daemon". But after some discussions we had decided that this > : > project can be done in more general maner. We can perform distributed > : > logging for any user-space app. > : > [...] > : > : This sounds very similar to syslogd. Is it feasible to make dlogd a drop-in > : replacement for syslogd, at least from a syslog-using-program point of view? > > I suspect that it is dealing with different data streams. syslog is for > programs sending text voluntarily. auditd is for pulling audit trails out > of the kernel for which the 'target' programs have no knowledge that the > audit trails are being generated, let alone anyway to prevent it. To possibly clarify a few points: (1) A distributed audit daemon wouldn't eliminate the need for local daemons that already manage log streams from various sources -- for example, syslogd for syslog, auditd for audit, Apache generating its own log files, etc. The goal of the distributed audit/log daemon is to manage these log files once log sources (such as auditd) are done with their logs. (2) One of the trickiest parts of the design will be the interaction between log sources and the audit daemon, so that log files can reliably change hands from being managed by the source to the distributed log tool. In the event of a system crash/power loss/network partition/syslogd crash/etc, we still want the log file to be picked up and synchronized. Hence discussion of an explicit hand-off API rather than casually looking in the same directory and hoping we get it right. (3) Unlikely syslogd's network logging support, the goal here is secure, reliable, batched delivery. We've talked a bit about live audit record delivery for IDS, but up front what we actually want to do is push the same sorts of reliability guarantees already present for local trail management to the distributed case. We are looking at pushing existing trail files over the network in a spooled way rather than shipping individual records as they are generated for this reason. Robert N M Watson Computer Laboratory University of Cambridge From owner-freebsd-hackers@FreeBSD.ORG Mon May 28 09:03:03 2007 Return-Path: X-Original-To: hackers@freebsd.org Delivered-To: freebsd-hackers@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id E0FC316A4FF; Mon, 28 May 2007 09:03:03 +0000 (UTC) (envelope-from m.seaman@infracaninophile.co.uk) Received: from smtp.infracaninophile.co.uk (happy-idiot-talk.infracaninophile.co.uk [81.187.76.162]) by mx1.freebsd.org (Postfix) with ESMTP id 339B213C484; Mon, 28 May 2007 09:03:02 +0000 (UTC) (envelope-from m.seaman@infracaninophile.co.uk) Received: from happy-idiot-talk.infracaninophile.co.uk (localhost.infracaninophile.co.uk [IPv6:::1]) by smtp.infracaninophile.co.uk (8.14.1/8.14.1) with ESMTP id l4S8bKn6006910; Mon, 28 May 2007 09:37:22 +0100 (BST) (envelope-from m.seaman@infracaninophile.co.uk) Authentication-Results: smtp.infracaninophile.co.uk from=m.seaman@infracaninophile.co.uk; sender-id=permerror; spf=permerror X-SenderID: Sendmail Sender-ID Filter v0.2.14 smtp.infracaninophile.co.uk l4S8bKn6006910 Message-ID: <465A94C0.2080301@infracaninophile.co.uk> Date: Mon, 28 May 2007 09:37:20 +0100 From: Matthew Seaman Organization: Infracaninophile User-Agent: Thunderbird 2.0.0.0 (X11/20070521) MIME-Version: 1.0 To: Stephen Montgomery-Smith References: <4659EF80.70100@math.missouri.edu> In-Reply-To: <4659EF80.70100@math.missouri.edu> X-Enigmail-Version: 0.95.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-3.0 (smtp.infracaninophile.co.uk [IPv6:::1]); Mon, 28 May 2007 09:37:32 +0100 (BST) X-Virus-Scanned: ClamAV 0.90.2/3311/Mon May 28 01:21:25 2007 on happy-idiot-talk.infracaninophile.co.uk X-Virus-Status: Clean X-Spam-Status: No, score=-2.6 required=5.0 tests=BAYES_00, DKIM_POLICY_TESTING, DK_POLICY_SIGNSOME,NO_RELAYS autolearn=ham version=3.1.8 X-Spam-Checker-Version: SpamAssassin 3.1.8 (2007-02-13) on happy-idiot-talk.infracaninophile.co.uk Cc: ports@freebsd.org, hackers@freebsd.org Subject: Re: Looking for speed increases in "make index" and pkg_version for ports X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 May 2007 09:03:04 -0000 -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 Stephen Montgomery-Smith wrote: > I have been thinking a lot about looking for speed increases for "make > index" and pkg_version and things like that. So for example, in > pkg_version, it calls "make -V PKGNAME" for every installed package. Now > "make -V PKGNAME" should be a speedy operation, but the make has to load > in and analyze bsd.port.mk, a quite complicated file with about 200,000 > characters in it, when all it is needing to do is to figure out the > value of the variable PKGNAME. pkg_version is one thing -- but to build the INDEX you need to extract at least the values of the following variables: PKGNAME .CURDIR PREFIX COMMENT DESCR MAINTAINER CATEGORIES EXTRACT_DEPENDS PATCH_DEPENDS FETCH_DEPENDS BUILD_DEPENDS RUN_DEPENDS LIB_DEPENDS Plus you need to grep in the referenced pkg-descr file for any WWW links. I also extract the values of: MASTER_PORT .MAKEFILE_LIST SUBDIR for my FreeBSD::Portindex stuff. Trouble is, by the time you've extracted all that lot, you have pretty much done the same level of variable processing as you would were you actually going to build the port. One thing that would speed up this process would be a make option to just do parsing of the Makefile and variable expansion, without calling stat(2) on all the various sources and dependencies involved. For instance: happy-idiot-talk:...ports/databases/mysql-connector-java:% truss make -V PKGNAME |& grep stat | wc -l 49 It is quite instructive to see what files make(1) touches while doing that. At least half of them are irrelevant if all make(1) is going to do is print out the values of some variables. Multiply that by 17,000 and it adds up to a big waste of effort. Cheers, Matthew - -- Dr Matthew J Seaman MA, D.Phil. 7 Priory Courtyard Flat 3 PGP: http://www.infracaninophile.co.uk/pgpkey Ramsgate Kent, CT11 9PW -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.3 (FreeBSD) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFGWpTA8Mjk52CukIwRCI0BAJ0bX5hTAJkMCO6Pl+cA4THv3mKulwCgg+39 kCyAGOTYYz9vEzzM9NRe3no= =MqFV -----END PGP SIGNATURE----- From owner-freebsd-hackers@FreeBSD.ORG Mon May 28 11:04:34 2007 Return-Path: X-Original-To: hackers@freebsd.org Delivered-To: freebsd-hackers@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id A39B916A41F; Mon, 28 May 2007 11:04:34 +0000 (UTC) (envelope-from Hartmut.Brandt@dlr.de) Received: from smtp-1.dlr.de (smtp-1.dlr.de [195.37.61.185]) by mx1.freebsd.org (Postfix) with ESMTP id 3E2AD13C480; Mon, 28 May 2007 11:04:34 +0000 (UTC) (envelope-from Hartmut.Brandt@dlr.de) Received: from [129.247.12.6] ([129.247.12.6]) by smtp-1.dlr.de with Microsoft SMTPSVC(6.0.3790.1830); Mon, 28 May 2007 13:04:26 +0200 Message-ID: <465AB737.1030708@dlr.de> Date: Mon, 28 May 2007 13:04:23 +0200 From: Hartmut Brandt User-Agent: Thunderbird 1.5.0.10 (Windows/20070221) MIME-Version: 1.0 To: Matthew Seaman References: <4659EF80.70100@math.missouri.edu> <465A94C0.2080301@infracaninophile.co.uk> In-Reply-To: <465A94C0.2080301@infracaninophile.co.uk> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-OriginalArrivalTime: 28 May 2007 11:04:26.0727 (UTC) FILETIME=[F4D50770:01C7A117] Cc: ports@freebsd.org, hackers@freebsd.org Subject: Re: Looking for speed increases in "make index" and pkg_version for ports X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 May 2007 11:04:34 -0000 Matthew Seaman wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA256 > > Stephen Montgomery-Smith wrote: >> I have been thinking a lot about looking for speed increases for "make >> index" and pkg_version and things like that. So for example, in >> pkg_version, it calls "make -V PKGNAME" for every installed package. Now >> "make -V PKGNAME" should be a speedy operation, but the make has to load >> in and analyze bsd.port.mk, a quite complicated file with about 200,000 >> characters in it, when all it is needing to do is to figure out the >> value of the variable PKGNAME. > > pkg_version is one thing -- but to build the INDEX you need to extract > at least the values of the following variables: > > PKGNAME > .CURDIR > PREFIX > COMMENT > DESCR > MAINTAINER > CATEGORIES > EXTRACT_DEPENDS > PATCH_DEPENDS > FETCH_DEPENDS > BUILD_DEPENDS > RUN_DEPENDS > LIB_DEPENDS > > Plus you need to grep in the referenced pkg-descr file for any WWW > links. I also extract the values of: > > MASTER_PORT > .MAKEFILE_LIST > SUBDIR > > for my FreeBSD::Portindex stuff. > > Trouble is, by the time you've extracted all that lot, you have pretty > much done the same level of variable processing as you would were you > actually going to build the port. > > One thing that would speed up this process would be a make option > to just do parsing of the Makefile and variable expansion, without > calling stat(2) on all the various sources and dependencies involved. > > For instance: > > happy-idiot-talk:...ports/databases/mysql-connector-java:% truss make -V PKGNAME |& grep stat | wc -l > 49 > > It is quite instructive to see what files make(1) touches while doing > that. At least half of them are irrelevant if all make(1) is going to > do is print out the values of some variables. Multiply that by 17,000 > and it adds up to a big waste of effort. Are you sure? A good deal of the stats() is make poking around for the make infrastructure. This should be in the cache. And there are a couple of stats for the *done* files that might be avoided by doing something in the ports infrastructure. But as I already said in my previous mail: numbers, please, no guessing. harti From owner-freebsd-hackers@FreeBSD.ORG Mon May 28 11:04:36 2007 Return-Path: X-Original-To: hackers@freebsd.org Delivered-To: freebsd-hackers@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 2CCD816A41F; Mon, 28 May 2007 11:04:36 +0000 (UTC) (envelope-from Hartmut.Brandt@dlr.de) Received: from smtp-1.dlr.de (smtp-1.dlr.de [195.37.61.185]) by mx1.freebsd.org (Postfix) with ESMTP id BCD7B13C483; Mon, 28 May 2007 11:04:35 +0000 (UTC) (envelope-from Hartmut.Brandt@dlr.de) Received: from [129.247.12.6] ([129.247.12.6]) by smtp-1.dlr.de with Microsoft SMTPSVC(6.0.3790.1830); Mon, 28 May 2007 12:51:16 +0200 Message-ID: <465AB421.10802@dlr.de> Date: Mon, 28 May 2007 12:51:13 +0200 From: Hartmut Brandt User-Agent: Thunderbird 1.5.0.10 (Windows/20070221) MIME-Version: 1.0 To: Stephen Montgomery-Smith References: <4659EF80.70100@math.missouri.edu> In-Reply-To: <4659EF80.70100@math.missouri.edu> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-OriginalArrivalTime: 28 May 2007 10:51:16.0185 (UTC) FILETIME=[1DA1E490:01C7A116] Cc: ports@freebsd.org, hackers@freebsd.org Subject: Re: Looking for speed increases in "make index" and pkg_version for ports X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 May 2007 11:04:36 -0000 Stephen Montgomery-Smith wrote: > I have been thinking a lot about looking for speed increases for "make > index" and pkg_version and things like that. So for example, in > pkg_version, it calls "make -V PKGNAME" for every installed package. Now > "make -V PKGNAME" should be a speedy operation, but the make has to load > in and analyze bsd.port.mk, a quite complicated file with about 200,000 > characters in it, when all it is needing to do is to figure out the > value of the variable PKGNAME. > > I suggest rewriting "make" so that variables are only evaluated on a > "need to know" basis. So, for example, if all we need to know is > PKGNAME, there is no need to evaluate, for example, _RUN_LIB_DEPENDS, > unless the writer of that particular port has done something like having > PORTNAME depend on the value of _RUN_LIB_DEPENDS. So "make" should > analyze all the code it is given, and only figure it out if it is needed > to do so. This would include, for example, figuring out .for and .if > directives on a need to know basis as well. > > I have only poked around a little inside the source for make, but I have > a sense that this would be a major undertaking. I certainly have not > thought through what it entails in more than a cursory manner. However > I am quite excited about the possibility of doing this, albeit I may > well put off the whole thing for a year or two or even forever depending > upon other priorities in my life. > > However, in the mean time I want to throw this idea out there to get > some feedback, either of the form of "this won't work," or of the form > "I will do it," or "I have tried to do this." Having done a great deal of rewriting of make some two years ago I can tell you that even a small change to make is a tough job testing-wise: run all the combinations of !-j and -j on all architectures and run the change through the port-building cluster. That's a warning to start with. Second I would start with careful profiling to find out where the problem actually is. You might be surprised. As an example: several times the idea came up to use a hash structure instead of linear lists for make variables. I got a patch for this and - it makes absolutely no difference performance-wise (well, there was some indication that performance gets worse, but that was around or below noise level). With careful I mean to find out who takes the time: 1. make and its sub-makes for a) reading the file; b) parsing the file (note that .if and .for processing is done while parsing); c) processing targets. 2. sub-shells executed for executing targets commands (note, that make optimizes the subshells away when there are no special shell symbol in the command line) 3. executed programs (find, sort, ...) Until you have numbers for this everything is rather moot. It might be a good idea to put some performance measurement hooks into make for this to do. If anybody wants to work on make, I would rather recommend to implement %-rules :-) And if anybody wants to recommend gmake over make(1) - look into the code, what mess that is :-/ Regards, harti From owner-freebsd-hackers@FreeBSD.ORG Mon May 28 15:18:39 2007 Return-Path: X-Original-To: hackers@freebsd.org Delivered-To: freebsd-hackers@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 0699E16A474; Mon, 28 May 2007 15:18:39 +0000 (UTC) (envelope-from stephen@math.missouri.edu) Received: from sccmmhc91.asp.att.net (sccmmhc91.asp.att.net [204.127.203.211]) by mx1.freebsd.org (Postfix) with ESMTP id B341613C4CA; Mon, 28 May 2007 15:18:38 +0000 (UTC) (envelope-from stephen@math.missouri.edu) Received: from laptop2.montlan (12-216-253-187.client.mchsi.com[12.216.253.187]) by sccmmhc91.asp.att.net (sccmmhc91) with ESMTP id <20070528151837m9100d22jne>; Mon, 28 May 2007 15:18:37 +0000 Message-ID: <465AF2CC.9020506@math.missouri.edu> Date: Mon, 28 May 2007 10:18:36 -0500 From: Stephen Montgomery-Smith User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.8.1.2) Gecko/20070525 SeaMonkey/1.1.1 MIME-Version: 1.0 To: Hartmut Brandt References: <4659EF80.70100@math.missouri.edu> <465AB421.10802@dlr.de> In-Reply-To: <465AB421.10802@dlr.de> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: ports@freebsd.org, hackers@freebsd.org Subject: Re: Looking for speed increases in "make index" and pkg_version for ports X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 May 2007 15:18:39 -0000 Hartmut Brandt wrote: > Having done a great deal of rewriting of make some two years ago I can > tell you that even a small change to make is a tough job testing-wise: > run all the combinations of !-j and -j on all architectures and run > the change through the port-building cluster. That's a warning to start > with. > > Second I would start with careful profiling to find out where the > problem actually is. You might be surprised. As an example: several > times the idea came up to use a hash structure instead of linear lists > for make variables. I got a patch for this and - it makes absolutely no > difference performance-wise (well, there was some indication that > performance gets worse, but that was around or below noise level). With > careful I mean to find out who takes the time: Yes, I must admit that I thought that a hash structure for the variables would greatly increase the speed of make. I rewrote it using Berkeley databases, and like you said - absolutely no difference!! I even tried btrees. From owner-freebsd-hackers@FreeBSD.ORG Mon May 28 15:55:49 2007 Return-Path: X-Original-To: hackers@freebsd.org Delivered-To: freebsd-hackers@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 34B6116A46E; Mon, 28 May 2007 15:55:49 +0000 (UTC) (envelope-from Hartmut.Brandt@dlr.de) Received: from smtp-1.dlr.de (smtp-1.dlr.de [195.37.61.185]) by mx1.freebsd.org (Postfix) with ESMTP id C4AF313C483; Mon, 28 May 2007 15:55:48 +0000 (UTC) (envelope-from Hartmut.Brandt@dlr.de) Received: from [129.247.12.6] ([129.247.12.6]) by smtp-1.dlr.de with Microsoft SMTPSVC(6.0.3790.1830); Mon, 28 May 2007 17:55:47 +0200 Message-ID: <465AFB7B.40105@dlr.de> Date: Mon, 28 May 2007 17:55:39 +0200 From: Hartmut Brandt User-Agent: Thunderbird 1.5.0.10 (Windows/20070221) MIME-Version: 1.0 To: Stephen Montgomery-Smith References: <4659EF80.70100@math.missouri.edu> <465AB421.10802@dlr.de> <465AF2CC.9020506@math.missouri.edu> In-Reply-To: <465AF2CC.9020506@math.missouri.edu> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-OriginalArrivalTime: 28 May 2007 15:55:47.0265 (UTC) FILETIME=[A80B7310:01C7A140] Cc: ports@freebsd.org, hackers@freebsd.org Subject: Re: Looking for speed increases in "make index" and pkg_version for ports X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 May 2007 15:55:49 -0000 Stephen Montgomery-Smith wrote: > Hartmut Brandt wrote: > >> Having done a great deal of rewriting of make some two years ago I can >> tell you that even a small change to make is a tough job testing-wise: >> run all the combinations of !-j and -j on all architectures and run >> the change through the port-building cluster. That's a warning to start >> with. >> >> Second I would start with careful profiling to find out where the >> problem actually is. You might be surprised. As an example: several >> times the idea came up to use a hash structure instead of linear lists >> for make variables. I got a patch for this and - it makes absolutely no >> difference performance-wise (well, there was some indication that >> performance gets worse, but that was around or below noise level). With >> careful I mean to find out who takes the time: > > Yes, I must admit that I thought that a hash structure for the variables > would greatly increase the speed of make. I rewrote it using Berkeley > databases, and like you said - absolutely no difference!! I even tried > btrees. > My guess at that time was that because there are actually many variable tables (one per target and the global one) and only a small number of variables in most of the tables the initialisation overhead outweights what you win through the hashing. As for the profiling - I did some profiling on buildworld then. From the several hours a buildworld took only one or two minutes were used by all the makes. At this point I stopped optimizing make :-) (I don't remember the exact numbers - that was two or three years ago). harti From owner-freebsd-hackers@FreeBSD.ORG Mon May 28 16:13:53 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 29B9F16A478 for ; Mon, 28 May 2007 16:13:53 +0000 (UTC) (envelope-from freebsd-hackers@m.gmane.org) Received: from ciao.gmane.org (main.gmane.org [80.91.229.2]) by mx1.freebsd.org (Postfix) with ESMTP id C02FC13C4C3 for ; Mon, 28 May 2007 16:13:52 +0000 (UTC) (envelope-from freebsd-hackers@m.gmane.org) Received: from list by ciao.gmane.org with local (Exim 4.43) id 1HsQm1-0005Qg-PL for freebsd-hackers@freebsd.org; Sun, 27 May 2007 23:59:37 +0200 Received: from 89-172-38-217.adsl.net.t-com.hr ([89.172.38.217]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Sun, 27 May 2007 23:59:37 +0200 Received: from ivoras by 89-172-38-217.adsl.net.t-com.hr with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Sun, 27 May 2007 23:59:37 +0200 X-Injected-Via-Gmane: http://gmane.org/ To: freebsd-hackers@freebsd.org From: Ivan Voras Date: Sun, 27 May 2007 23:59:07 +0200 Lines: 38 Message-ID: References: <4659EF80.70100__16104.363036766$1180300054$gmane$org@math.missouri.edu> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="------------enig454B174B5ADF28F119C54376" X-Complaints-To: usenet@sea.gmane.org X-Gmane-NNTP-Posting-Host: 89-172-38-217.adsl.net.t-com.hr User-Agent: Thunderbird 1.5.0.10 (Windows/20070221) In-Reply-To: <4659EF80.70100__16104.363036766$1180300054$gmane$org@math.missouri.edu> X-Enigmail-Version: 0.94.3.0 Sender: news Subject: Re: Looking for speed increases in "make index" and pkg_version for ports X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 May 2007 16:13:53 -0000 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enig454B174B5ADF28F119C54376 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Stephen Montgomery-Smith wrote: > I have been thinking a lot about looking for speed increases for "make > index" and pkg_version and things like that. So for example, in > pkg_version, it calls "make -V PKGNAME" for every installed package. No= w > "make -V PKGNAME" should be a speedy operation, but the make has to loa= d > in and analyze bsd.port.mk, a quite complicated file with about 200,000= > characters in it, when all it is needing to do is to figure out the > value of the variable PKGNAME. As long as far-out ideas are being discussed, how about caching such information (including dependenices) in a file (I'd call it a database but then I'd had to start a holy war :) ) so it's calculated only once, preferably on the portsnap / cvsup servers and not at the end-user? --------------enig454B174B5ADF28F119C54376 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.4 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFGWf8xldnAQVacBcgRAhl6AKDaHYZIlEBgza3cOmZjGM4Zb0NADACfYaJy P4uaLCOlza+LjIZmsbl1zmQ= =7oxu -----END PGP SIGNATURE----- --------------enig454B174B5ADF28F119C54376-- From owner-freebsd-hackers@FreeBSD.ORG Mon May 28 16:34:26 2007 Return-Path: X-Original-To: hackers@freebsd.org Delivered-To: freebsd-hackers@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id A890E16A41F; Mon, 28 May 2007 16:34:26 +0000 (UTC) (envelope-from stephen@math.missouri.edu) Received: from sccmmhc91.asp.att.net (sccmmhc91.asp.att.net [204.127.203.211]) by mx1.freebsd.org (Postfix) with ESMTP id 6015A13C46C; Mon, 28 May 2007 16:34:26 +0000 (UTC) (envelope-from stephen@math.missouri.edu) Received: from laptop2.montlan (12-216-253-187.client.mchsi.com[12.216.253.187]) by sccmmhc91.asp.att.net (sccmmhc91) with ESMTP id <20070528163425m9100d2t98e>; Mon, 28 May 2007 16:34:25 +0000 Message-ID: <465B0490.7080203@math.missouri.edu> Date: Mon, 28 May 2007 11:34:24 -0500 From: Stephen Montgomery-Smith User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.8.1.2) Gecko/20070525 SeaMonkey/1.1.1 MIME-Version: 1.0 To: Stephen Montgomery-Smith , ports@freebsd.org, hackers@freebsd.org References: <4659EF80.70100@math.missouri.edu> <20070527223048.GA37505@icarus.home.lan> In-Reply-To: <20070527223048.GA37505@icarus.home.lan> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: Subject: Re: Looking for speed increases in "make index" and pkg_version for ports X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 May 2007 16:34:26 -0000 Jeremy Chadwick wrote: > On Sun, May 27, 2007 at 03:52:16PM -0500, Stephen Montgomery-Smith wrote: >> I have been thinking a lot about looking for speed increases for "make >> index" and pkg_version and things like that. So for example, in >> pkg_version, it calls "make -V PKGNAME" for every installed package. Now >> "make -V PKGNAME" should be a speedy operation, but the make has to load in >> and analyze bsd.port.mk, a quite complicated file with about 200,000 >> characters in it, when all it is needing to do is to figure out the value of >> the variable PKGNAME. > > I have a related question, pertaining to "make all-depends-list" and the > utter atrocity that is the make variable ALL-DEPENDS-LIST. If you don't > know what it is, look for ^ALL-DEPENDS-LIST around line 5175, in > bsd.ports.mk. I posted this to ports@freebsd.org, but now I am realizing that it is hackers@freebsd.org that gets more responses. Anyway, here is a multithreaded program "all-depends-list" that can get you double the speed on dual processor systems, and even some small speed gains on single processor systems. E.g. all-depends-list /usr/ports/x11/xorg http://www.math.missouri.edu/~stephen/all-depends-list.c From owner-freebsd-hackers@FreeBSD.ORG Mon May 28 16:39:30 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 40E0D16A400 for ; Mon, 28 May 2007 16:39:30 +0000 (UTC) (envelope-from stephen@math.missouri.edu) Received: from sccmmhc91.asp.att.net (sccmmhc91.asp.att.net [204.127.203.211]) by mx1.freebsd.org (Postfix) with ESMTP id 0B96A13C4C5 for ; Mon, 28 May 2007 16:39:29 +0000 (UTC) (envelope-from stephen@math.missouri.edu) Received: from laptop2.montlan (12-216-253-187.client.mchsi.com[12.216.253.187]) by sccmmhc91.asp.att.net (sccmmhc91) with ESMTP id <20070528162644m9100d240ee>; Mon, 28 May 2007 16:26:45 +0000 Message-ID: <465B02C4.2020508@math.missouri.edu> Date: Mon, 28 May 2007 11:26:44 -0500 From: Stephen Montgomery-Smith User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.8.1.2) Gecko/20070525 SeaMonkey/1.1.1 MIME-Version: 1.0 To: Ivan Voras References: <4659EF80.70100__16104.363036766$1180300054$gmane$org@math.missouri.edu> In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-hackers@freebsd.org Subject: Re: Looking for speed increases in "make index" and pkg_version for ports X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 May 2007 16:39:30 -0000 Ivan Voras wrote: > Stephen Montgomery-Smith wrote: >> I have been thinking a lot about looking for speed increases for "make >> index" and pkg_version and things like that. So for example, in >> pkg_version, it calls "make -V PKGNAME" for every installed package. Now >> "make -V PKGNAME" should be a speedy operation, but the make has to load >> in and analyze bsd.port.mk, a quite complicated file with about 200,000 >> characters in it, when all it is needing to do is to figure out the >> value of the variable PKGNAME. > > As long as far-out ideas are being discussed, how about caching such > information (including dependenices) in a file (I'd call it a database > but then I'd had to start a holy war :) ) so it's calculated only once, > preferably on the portsnap / cvsup servers and not at the end-user? Because the information is not a constant. For example, the mpg123 port changes its PKGNAME as soon as esound is installed. From owner-freebsd-hackers@FreeBSD.ORG Mon May 28 16:29:03 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 5A87116A400 for ; Mon, 28 May 2007 16:29:03 +0000 (UTC) (envelope-from naylor.b.david@gmail.com) Received: from wr-out-0506.google.com (wr-out-0506.google.com [64.233.184.237]) by mx1.freebsd.org (Postfix) with ESMTP id EB18713C45A for ; Mon, 28 May 2007 16:29:02 +0000 (UTC) (envelope-from naylor.b.david@gmail.com) Received: by wr-out-0506.google.com with SMTP id 70so479149wra for ; Mon, 28 May 2007 09:29:01 -0700 (PDT) DKIM-Signature: a=rsa-sha1; c=relaxed/relaxed; d=gmail.com; s=beta; h=domainkey-signature:received:received:from:to:subject:date:user-agent:references:in-reply-to:mime-version:content-type:content-transfer-encoding:message-id:sender; b=bDr0mdNW1xWo18bmdImqx6uAk4/lS8kwWQikKx0dLCwo4I2R/K1EYVgjAV9U5Ws2JDBRQsrBkSOeAoJiyaPFdzHh+OkFHNMQwgmLl5E/sDXfLShrMg3Aox64nHvAILMdOLL61AKTTU9BV4ILp1joymcQztkq0Pyjr8uN0QIl8ew= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=beta; h=received:from:to:subject:date:user-agent:references:in-reply-to:mime-version:content-type:content-transfer-encoding:message-id:sender; b=NkPI1J9vQxdyRYzCPGF9cXsvR63G9r199A8CkJwBfXvYj102gPu9IbdVb1O0QVkcEMyrQcGuYX17b0ImG8jM1WfD8pUUvEtxK6CRfuDalgBJv4sfkYHYWsGOAZ9RkFeboV4MyIu5WJ8kfC/NwOco8RWM/UWEEWho7NcKTiexvNA= Received: by 10.90.28.13 with SMTP id b13mr638475agb.1180369741720; Mon, 28 May 2007 09:29:01 -0700 (PDT) Received: from ?0.0.0.0? ( [196.37.90.163]) by mx.google.com with ESMTP id c1sm7161084nzd.2007.05.28.09.28.56; Mon, 28 May 2007 09:29:00 -0700 (PDT) From: David Naylor To: freebsd-hackers@freebsd.org Date: Mon, 28 May 2007 18:28:43 +0200 User-Agent: KMail/1.9.5 References: <4659EF80.70100@math.missouri.edu> <20070527223048.GA37505@icarus.home.lan> <465A33D4.1040706@math.missouri.edu> In-Reply-To: <465A33D4.1040706@math.missouri.edu> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="nextPart4943566.SuABLBdVN4"; protocol="application/pgp-signature"; micalg=pgp-sha1 Content-Transfer-Encoding: 7bit Message-Id: <200705281828.48026.blackdragon@highveldmail.co.za> Sender: David Naylor X-Mailman-Approved-At: Mon, 28 May 2007 16:52:06 +0000 Subject: Re: Looking for speed increases in "make index" and pkg_version for ports X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 May 2007 16:29:03 -0000 --nextPart4943566.SuABLBdVN4 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Content-Disposition: inline On Monday 28 May 2007 03:43, you wrote: > Maybe I should look at the inner workings of cmake and gmake. Maybe > they have some good ideas. However having looked through the source > code of make, and also looking at the cvs logs, it does seem to be well > written. The only possibility I see of making it go a lot faster is a > complete redesign, e.g. my just in time idea for processing variables. > > Stephen Just in time (jit), if I remember correctly, is a term used by java=20 interpreters which compile the byte code into machine code!!! Perhaps this= =20 could be developed for makefile's, especially bsd.*.mk. =20 This, I think, could be done in two ways: 1) Develop the bsd.*.mk files in C and link it in with make, or 2) Use the makefiles as source to compile into machine code (passibly via=20 C->ASM). The machine code could be created on demand, or cached and only=20 updated if the source makefile changes. =20 I am not sure if this could work or even if it will have any significant sp= eed=20 increase. However if method 2 does work it has the potential to radically= =20 increase the speed of ports _while_ maintaining the flexability. =20 All that will be needed is an API for the machine code and a compiler??? David --nextPart4943566.SuABLBdVN4 Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.5 (GNU/Linux) iD8DBQBGWwNAyqzxLKpyZI8RAuhFAJ9UVnPXYyC69baLKYQlDXCvqz0dVwCfXDPp qXVLSnKIsL2BpOm/aUmgHQ4= =1gFI -----END PGP SIGNATURE----- --nextPart4943566.SuABLBdVN4-- From owner-freebsd-hackers@FreeBSD.ORG Mon May 28 16:57:10 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id F368016A46B for ; Mon, 28 May 2007 16:57:09 +0000 (UTC) (envelope-from m.seaman@infracaninophile.co.uk) Received: from smtp.infracaninophile.co.uk (ns0.infracaninophile.co.uk [81.187.76.162]) by mx1.freebsd.org (Postfix) with ESMTP id 58A0513C458 for ; Mon, 28 May 2007 16:57:09 +0000 (UTC) (envelope-from m.seaman@infracaninophile.co.uk) Received: from happy-idiot-talk.infracaninophile.co.uk (localhost.infracaninophile.co.uk [IPv6:::1]) by smtp.infracaninophile.co.uk (8.14.1/8.14.1) with ESMTP id l4SGuhQY098618; Mon, 28 May 2007 17:56:45 +0100 (BST) (envelope-from m.seaman@infracaninophile.co.uk) Authentication-Results: smtp.infracaninophile.co.uk from=m.seaman@infracaninophile.co.uk; sender-id=permerror; spf=permerror X-SenderID: Sendmail Sender-ID Filter v0.2.14 smtp.infracaninophile.co.uk l4SGuhQY098618 Message-ID: <465B09CB.8040704@infracaninophile.co.uk> Date: Mon, 28 May 2007 17:56:43 +0100 From: Matthew Seaman Organization: Infracaninophile User-Agent: Thunderbird 2.0.0.0 (X11/20070521) MIME-Version: 1.0 To: Ivan Voras References: <4659EF80.70100__16104.363036766$1180300054$gmane$org@math.missouri.edu> In-Reply-To: X-Enigmail-Version: 0.95.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-3.0 (smtp.infracaninophile.co.uk [IPv6:::1]); Mon, 28 May 2007 17:56:55 +0100 (BST) X-Virus-Scanned: ClamAV 0.90.2/3311/Mon May 28 01:21:25 2007 on happy-idiot-talk.infracaninophile.co.uk X-Virus-Status: Clean X-Spam-Status: No, score=-2.6 required=5.0 tests=BAYES_00, DKIM_POLICY_TESTING, DK_POLICY_SIGNSOME,NO_RELAYS autolearn=ham version=3.1.8 X-Spam-Checker-Version: SpamAssassin 3.1.8 (2007-02-13) on happy-idiot-talk.infracaninophile.co.uk Cc: freebsd-hackers@freebsd.org Subject: Re: Looking for speed increases in "make index" and pkg_version for ports X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 May 2007 16:57:10 -0000 -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 Ivan Voras wrote: > Stephen Montgomery-Smith wrote: >> I have been thinking a lot about looking for speed increases for "make >> index" and pkg_version and things like that. So for example, in >> pkg_version, it calls "make -V PKGNAME" for every installed package. Now >> "make -V PKGNAME" should be a speedy operation, but the make has to load >> in and analyze bsd.port.mk, a quite complicated file with about 200,000 >> characters in it, when all it is needing to do is to figure out the >> value of the variable PKGNAME. > > As long as far-out ideas are being discussed, how about caching such > information (including dependenices) in a file (I'd call it a database > but then I'd had to start a holy war :) ) so it's calculated only once, > preferably on the portsnap / cvsup servers and not at the end-user? Good idea. http://www.infracaninophile.co.uk/portindex/ Been done before though. Cheers, Matthew - -- Dr Matthew J Seaman MA, D.Phil. 7 Priory Courtyard Flat 3 PGP: http://www.infracaninophile.co.uk/pgpkey Ramsgate Kent, CT11 9PW -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.3 (FreeBSD) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFGWwnL8Mjk52CukIwRCNDLAJ4jFCpr5y7uAQi97mVRV3Pc4+c99ACeN9vQ tOc6IzTQ90+wObG34KWQzzw= =XuiO -----END PGP SIGNATURE----- From owner-freebsd-hackers@FreeBSD.ORG Mon May 28 17:10:27 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 6A15016A498 for ; Mon, 28 May 2007 17:10:27 +0000 (UTC) (envelope-from stephen@math.missouri.edu) Received: from math.missouri.edu (math.missouri.edu [128.206.184.200]) by mx1.freebsd.org (Postfix) with ESMTP id 34D7F13C4EA for ; Mon, 28 May 2007 17:10:27 +0000 (UTC) (envelope-from stephen@math.missouri.edu) Received: from math.missouri.edu (localhost [127.0.0.1]) by math.missouri.edu (8.13.1/8.13.1) with ESMTP id l4SHAQ8v040061; Mon, 28 May 2007 12:10:26 -0500 (CDT) (envelope-from stephen@math.missouri.edu) Received: from localhost (stephen@localhost) by math.missouri.edu (8.13.1/8.13.1/Submit) with ESMTP id l4SHAQE9040057; Mon, 28 May 2007 12:10:26 -0500 (CDT) (envelope-from stephen@math.missouri.edu) Date: Mon, 28 May 2007 12:10:25 -0500 (CDT) From: Stephen Montgomery-Smith To: David Naylor In-Reply-To: <200705281828.48026.blackdragon@highveldmail.co.za> Message-ID: <20070528120929.D12634@math.missouri.edu> References: <4659EF80.70100@math.missouri.edu> <20070527223048.GA37505@icarus.home.lan> <465A33D4.1040706@math.missouri.edu> <200705281828.48026.blackdragon@highveldmail.co.za> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: freebsd-hackers@freebsd.org Subject: Re: Looking for speed increases in "make index" and pkg_version for ports X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 May 2007 17:10:27 -0000 On Mon, 28 May 2007, David Naylor wrote: > On Monday 28 May 2007 03:43, you wrote: >> Maybe I should look at the inner workings of cmake and gmake. Maybe >> they have some good ideas. However having looked through the source >> code of make, and also looking at the cvs logs, it does seem to be well >> written. The only possibility I see of making it go a lot faster is a >> complete redesign, e.g. my just in time idea for processing variables. >> >> Stephen > > Just in time (jit), if I remember correctly, is a term used by java > interpreters which compile the byte code into machine code!!! Perhaps this > could be developed for makefile's, especially bsd.*.mk. > > This, I think, could be done in two ways: > 1) Develop the bsd.*.mk files in C and link it in with make, or > 2) Use the makefiles as source to compile into machine code (passibly via > C->ASM). The machine code could be created on demand, or cached and only > updated if the source makefile changes. > > I am not sure if this could work or even if it will have any significant speed > increase. However if method 2 does work it has the potential to radically > increase the speed of ports _while_ maintaining the flexability. > > All that will be needed is an API for the machine code and a compiler??? > My gut reaction is the same as yours - I doubt that this would bring any speed increases. And the programming effort would be huge. From owner-freebsd-hackers@FreeBSD.ORG Mon May 28 17:56:03 2007 Return-Path: X-Original-To: hackers@freebsd.org Delivered-To: freebsd-hackers@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 160F316A53C for ; Mon, 28 May 2007 17:56:03 +0000 (UTC) (envelope-from mwm-keyword-freebsdhackers2.e313df@mired.org) Received: from mired.org (vpn.mired.org [66.92.153.74]) by mx1.freebsd.org (Postfix) with SMTP id 99BBF13C457 for ; Mon, 28 May 2007 17:56:02 +0000 (UTC) (envelope-from mwm-keyword-freebsdhackers2.e313df@mired.org) Received: (qmail 21257 invoked by uid 1001); 28 May 2007 17:55:16 -0000 Received: by bhuda.mired.org (tmda-sendmail, from uid 1001); Mon, 28 May 2007 13:55:15 -0400 MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <18011.6019.436391.128372@bhuda.mired.org> Date: Mon, 28 May 2007 13:55:15 -0400 To: Hartmut Brandt In-Reply-To: <465AB421.10802@dlr.de> References: <4659EF80.70100@math.missouri.edu> <465AB421.10802@dlr.de> X-Mailer: VM 7.19 under Emacs 21.3.1 X-Primary-Address: mwm@mired.org X-face: "5Mnwy%?j>IIV\)A=):rjWL~NB2aH[}Yq8Z=u~vJ`"(,&SiLvbbz2W`; h9L,Yg`+vb1>RG% *h+%X^n0EZd>TM8_IB;a8F?(Fb"lw'IgCoyM.[Lg#r\ X-Delivery-Agent: TMDA/1.1.11 (Ladyburn) From: Mike Meyer Cc: ports@freebsd.org, hackers@freebsd.org Subject: Re: Looking for speed increases in "make index" and pkg_version for ports X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 May 2007 17:56:03 -0000 In <465AB421.10802@dlr.de>, Hartmut Brandt typed: > 1. make and its sub-makes for a) reading the file; b) parsing the file > (note that .if and .for processing is done while parsing); c) processing > targets. Make and submakes have been gone over already. See . I'm not sure it can be applied to the ports tree, though. I haven't looked into it, but recalled this paper when you mentioned measuring makes and sub-makes. http://www.mired.org/consulting.html Independent Network/Unix/Perforce consultant, email for more information. From owner-freebsd-hackers@FreeBSD.ORG Mon May 28 18:13:57 2007 Return-Path: X-Original-To: hackers@freebsd.org Delivered-To: freebsd-hackers@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 594FF16A421; Mon, 28 May 2007 18:13:57 +0000 (UTC) (envelope-from youshi10@u.washington.edu) Received: from mxout5.cac.washington.edu (mxout5.cac.washington.edu [140.142.32.135]) by mx1.freebsd.org (Postfix) with ESMTP id 3656113C448; Mon, 28 May 2007 18:13:57 +0000 (UTC) (envelope-from youshi10@u.washington.edu) Received: from smtp.washington.edu (smtp.washington.edu [140.142.33.7] (may be forged)) by mxout5.cac.washington.edu (8.13.7+UW06.06/8.13.7+UW07.03) with ESMTP id l4SIDs2O015119 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Mon, 28 May 2007 11:13:54 -0700 X-Auth-Received: from [192.168.10.45] (c-67-166-149-71.hsd1.ca.comcast.net [67.166.149.71]) (authenticated authid=youshi10) by smtp.washington.edu (8.13.7+UW06.06/8.13.7+UW07.03) with ESMTP id l4SIDrRC026783 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NOT); Mon, 28 May 2007 11:13:54 -0700 Message-ID: <465B1BE1.7010208@u.washington.edu> Date: Mon, 28 May 2007 11:13:53 -0700 From: Garrett Cooper User-Agent: Thunderbird 2.0.0.0 (Windows/20070326) MIME-Version: 1.0 To: Mike Meyer References: <4659EF80.70100@math.missouri.edu> <465AB421.10802@dlr.de> <18011.6019.436391.128372@bhuda.mired.org> In-Reply-To: <18011.6019.436391.128372@bhuda.mired.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-PMX-Version: 5.3.1.294258, Antispam-Engine: 2.5.1.298604, Antispam-Data: 2007.5.28.110037 X-Uwash-Spam: Gauge=IIIIIII, Probability=7%, Report='__CP_URI_IN_BODY 0, __CT 0, __CTE 0, __CT_TEXT_PLAIN 0, __HAS_MSGID 0, __MIME_TEXT_ONLY 0, __MIME_VERSION 0, __SANE_MSGID 0, __USER_AGENT 0' Cc: ports@freebsd.org, Hartmut Brandt , hackers@freebsd.org Subject: Re: Looking for speed increases in "make index" and pkg_version for ports X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 May 2007 18:13:57 -0000 Mike Meyer wrote: > In <465AB421.10802@dlr.de>, Hartmut Brandt typed: >> 1. make and its sub-makes for a) reading the file; b) parsing the file >> (note that .if and .for processing is done while parsing); c) processing >> targets. > > Make and submakes have been gone over already. See http://miller.emu.id.au/pmiller/books/rmch/ >. > > I'm not sure it can be applied to the ports tree, though. I haven't > looked into it, but recalled this paper when you mentioned measuring > makes and sub-makes. > > X-Original-To: hackers@freebsd.org Delivered-To: freebsd-hackers@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 9B6F016A481; Mon, 28 May 2007 18:43:34 +0000 (UTC) (envelope-from youshi10@u.washington.edu) Received: from mxout4.cac.washington.edu (mxout4.cac.washington.edu [140.142.33.19]) by mx1.freebsd.org (Postfix) with ESMTP id 7795C13C44B; Mon, 28 May 2007 18:43:34 +0000 (UTC) (envelope-from youshi10@u.washington.edu) Received: from smtp.washington.edu (smtp.washington.edu [140.142.32.141] (may be forged)) by mxout4.cac.washington.edu (8.13.7+UW06.06/8.13.7+UW07.03) with ESMTP id l4SIhWcV011933 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Mon, 28 May 2007 11:43:32 -0700 X-Auth-Received: from [192.168.10.45] (c-67-166-149-71.hsd1.ca.comcast.net [67.166.149.71]) (authenticated authid=youshi10) by smtp.washington.edu (8.13.7+UW06.06/8.13.7+UW07.03) with ESMTP id l4SIhVqE008014 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NOT); Mon, 28 May 2007 11:43:31 -0700 Message-ID: <465B22D0.8070907@u.washington.edu> Date: Mon, 28 May 2007 11:43:28 -0700 From: Garrett Cooper User-Agent: Thunderbird 2.0.0.0 (Windows/20070326) MIME-Version: 1.0 To: Mike Meyer References: <4659EF80.70100@math.missouri.edu> <465AB421.10802@dlr.de> <18011.6019.436391.128372@bhuda.mired.org> <465B1BE1.7010208@u.washington.edu> In-Reply-To: <465B1BE1.7010208@u.washington.edu> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-PMX-Version: 5.3.1.294258, Antispam-Engine: 2.5.1.298604, Antispam-Data: 2007.5.28.112535 X-Uwash-Spam: Gauge=IIIIIII, Probability=7%, Report='__CP_URI_IN_BODY 0, __CT 0, __CTE 0, __CT_TEXT_PLAIN 0, __HAS_MSGID 0, __MIME_TEXT_ONLY 0, __MIME_VERSION 0, __SANE_MSGID 0, __USER_AGENT 0' Cc: ports@freebsd.org, Hartmut Brandt , hackers@freebsd.org Subject: Re: Looking for speed increases in "make index" and pkg_version for ports X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 May 2007 18:43:34 -0000 Garrett Cooper wrote: > Mike Meyer wrote: >> In <465AB421.10802@dlr.de>, Hartmut Brandt typed: >>> 1. make and its sub-makes for a) reading the file; b) parsing the file >>> (note that .if and .for processing is done while parsing); c) processing >>> targets. >> >> Make and submakes have been gone over already. See > http://miller.emu.id.au/pmiller/books/rmch/ >. >> >> I'm not sure it can be applied to the ports tree, though. I haven't >> looked into it, but recalled this paper when you mentioned measuring >> makes and sub-makes. >> >> > Reducing the number of variables will certainly cut down on the amount > of overhead in the make/submake context switches by a long shot. > > Maybe someone should consider running a 'pre-make' using the .mk files, > find the variables of interest for all particular sub-ports, and then > carry on the 'root make', i.e. make process in each port directory, with > just the variables of interest. > > -Garrett s/long shot/possibly a lot/g Also, I was thinking in particular of the X.Org 7.2 packages, because the bulk majority of the packages are smaller, and compile in a short amount of time. -Garrett From owner-freebsd-hackers@FreeBSD.ORG Mon May 28 20:08:55 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id BC6DA16A46D for ; Mon, 28 May 2007 20:08:55 +0000 (UTC) (envelope-from freebsd-hackers@m.gmane.org) Received: from ciao.gmane.org (main.gmane.org [80.91.229.2]) by mx1.freebsd.org (Postfix) with ESMTP id 68F8B13C45A for ; Mon, 28 May 2007 20:08:55 +0000 (UTC) (envelope-from freebsd-hackers@m.gmane.org) Received: from list by ciao.gmane.org with local (Exim 4.43) id 1HslVz-00026i-HS for freebsd-hackers@freebsd.org; Mon, 28 May 2007 22:08:27 +0200 Received: from 78-1-119-138.adsl.net.t-com.hr ([78.1.119.138]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Mon, 28 May 2007 22:08:27 +0200 Received: from ivoras by 78-1-119-138.adsl.net.t-com.hr with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Mon, 28 May 2007 22:08:27 +0200 X-Injected-Via-Gmane: http://gmane.org/ To: freebsd-hackers@freebsd.org From: Ivan Voras Date: Mon, 28 May 2007 22:07:12 +0200 Lines: 39 Message-ID: References: <4659EF80.70100__16104.363036766$1180300054$gmane$org@math.missouri.edu> <465B02C4.2020508@math.missouri.edu> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="------------enig279D4A02D1F67CF59DE01083" X-Complaints-To: usenet@sea.gmane.org X-Gmane-NNTP-Posting-Host: 78-1-119-138.adsl.net.t-com.hr User-Agent: Thunderbird 1.5.0.10 (Windows/20070221) In-Reply-To: <465B02C4.2020508@math.missouri.edu> X-Enigmail-Version: 0.94.3.0 Sender: news Subject: Re: Looking for speed increases in "make index" and pkg_version for ports X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 May 2007 20:08:55 -0000 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enig279D4A02D1F67CF59DE01083 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Stephen Montgomery-Smith wrote: > Ivan Voras wrote: >> As long as far-out ideas are being discussed, how about caching such >> information (including dependenices) in a file (I'd call it a database= >> but then I'd had to start a holy war :) ) so it's calculated only once= , >> preferably on the portsnap / cvsup servers and not at the end-user? >=20 > Because the information is not a constant. For example, the mpg123 por= t > changes its PKGNAME as soon as esound is installed. Maybe the time has come to give up on some of the flexibility the ports tree has (and this particular one is confusing to the users) in favour of gaining speed? --------------enig279D4A02D1F67CF59DE01083 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.4 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFGWzZwldnAQVacBcgRAs5mAJ9Qw+s+kU+gip4rYavVInmUpLwWFwCgjq2A A8HqGLyhECWg8LK1hNceMWA= =dkcU -----END PGP SIGNATURE----- --------------enig279D4A02D1F67CF59DE01083-- From owner-freebsd-hackers@FreeBSD.ORG Mon May 28 21:32:54 2007 Return-Path: X-Original-To: hackers@freebsd.org Delivered-To: freebsd-hackers@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 97C2216A479; Mon, 28 May 2007 21:32:54 +0000 (UTC) (envelope-from Hartmut.Brandt@dlr.de) Received: from smtp-1.dlr.de (smtp-1.dlr.de [195.37.61.185]) by mx1.freebsd.org (Postfix) with ESMTP id 328BE13C448; Mon, 28 May 2007 21:32:53 +0000 (UTC) (envelope-from Hartmut.Brandt@dlr.de) Received: from [192.168.2.102] ([172.21.151.1]) by smtp-1.dlr.de with Microsoft SMTPSVC(6.0.3790.1830); Mon, 28 May 2007 23:32:52 +0200 Message-ID: <465B4A84.6050407@dlr.de> Date: Mon, 28 May 2007 23:32:52 +0200 From: Hartmut Brandt User-Agent: Thunderbird 1.5.0.10 (Windows/20070221) MIME-Version: 1.0 To: Mike Meyer References: <4659EF80.70100@math.missouri.edu> <465AB421.10802@dlr.de> <18011.6019.436391.128372@bhuda.mired.org> In-Reply-To: <18011.6019.436391.128372@bhuda.mired.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-OriginalArrivalTime: 28 May 2007 21:32:52.0647 (UTC) FILETIME=[BF4FD370:01C7A16F] Cc: ports@freebsd.org, hackers@freebsd.org Subject: Re: Looking for speed increases in "make index" and pkg_version for ports X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 May 2007 21:32:54 -0000 Mike Meyer wrote: > In <465AB421.10802@dlr.de>, Hartmut Brandt typed: >> 1. make and its sub-makes for a) reading the file; b) parsing the file >> (note that .if and .for processing is done while parsing); c) processing >> targets. > > Make and submakes have been gone over already. See http://miller.emu.id.au/pmiller/books/rmch/ >. > > I'm not sure it can be applied to the ports tree, though. I haven't > looked into it, but recalled this paper when you mentioned measuring > makes and sub-makes. Unfortunately you deleted the sentence before, so I rephrase it: before looking into optimizations find out where the time is actually spend - how many seconds of the hours the process takes, are actually spent in make and sub-makes. If the entire process takes 2 hours of which the makes take 20 seconds then by enhancing performance of make by 50% you win 10 seconds. This is probably not worth a single line of additional code. The paper you point to talks about something entirely different. harti From owner-freebsd-hackers@FreeBSD.ORG Mon May 28 21:38:24 2007 Return-Path: X-Original-To: hackers@freebsd.org Delivered-To: freebsd-hackers@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 539DE16A46B; Mon, 28 May 2007 21:38:24 +0000 (UTC) (envelope-from rdivacky@vlk.vlakno.cz) Received: from vlakno.cz (vlk.vlakno.cz [62.168.28.247]) by mx1.freebsd.org (Postfix) with ESMTP id CA21313C4B0; Mon, 28 May 2007 21:38:23 +0000 (UTC) (envelope-from rdivacky@vlk.vlakno.cz) Received: from localhost (localhost [127.0.0.1]) by vlakno.cz (Postfix) with ESMTP id C47238BD861; Mon, 28 May 2007 23:20:37 +0200 (CEST) X-Virus-Scanned: amavisd-new at vlakno.cz Received: from vlakno.cz ([127.0.0.1]) by localhost (vlk.vlakno.cz [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id EpxsXoLgWNl2; Mon, 28 May 2007 23:20:36 +0200 (CEST) Received: from vlk.vlakno.cz (localhost [127.0.0.1]) by vlakno.cz (Postfix) with ESMTP id 8C70B8BD84B; Mon, 28 May 2007 23:20:36 +0200 (CEST) Received: (from rdivacky@localhost) by vlk.vlakno.cz (8.13.8/8.13.8/Submit) id l4SLKaqJ036060; Mon, 28 May 2007 23:20:36 +0200 (CEST) (envelope-from rdivacky) Date: Mon, 28 May 2007 23:20:36 +0200 From: Roman Divacky To: Stephen Montgomery-Smith Message-ID: <20070528212036.GA35796@freebsd.org> References: <4659EF80.70100@math.missouri.edu> <20070527223048.GA37505@icarus.home.lan> <465B0490.7080203@math.missouri.edu> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <465B0490.7080203@math.missouri.edu> User-Agent: Mutt/1.4.2.2i Cc: ports@freebsd.org, hackers@freebsd.org Subject: Re: Looking for speed increases in "make index" and pkg_version for ports X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 May 2007 21:38:24 -0000 On Mon, May 28, 2007 at 11:34:24AM -0500, Stephen Montgomery-Smith wrote: > Jeremy Chadwick wrote: > >On Sun, May 27, 2007 at 03:52:16PM -0500, Stephen Montgomery-Smith wrote: > >> I have been thinking a lot about looking for speed increases for "make > >> index" and pkg_version and things like that. So for example, in > >> pkg_version, it calls "make -V PKGNAME" for every installed package. Now > >> "make -V PKGNAME" should be a speedy operation, but the make has to load > >> in and analyze bsd.port.mk, a quite complicated file with about 200,000 > >> characters in it, when all it is needing to do is to figure out the > >> value of the variable PKGNAME. > > > >I have a related question, pertaining to "make all-depends-list" and the > >utter atrocity that is the make variable ALL-DEPENDS-LIST. If you don't > >know what it is, look for ^ALL-DEPENDS-LIST around line 5175, in > >bsd.ports.mk. > > I posted this to ports@freebsd.org, but now I am realizing that it is > hackers@freebsd.org that gets more responses. Anyway, here is a > multithreaded program "all-depends-list" that can get you double the > speed on dual processor systems, and even some small speed gains on > single processor systems. E.g. > > all-depends-list /usr/ports/x11/xorg > > http://www.math.missouri.edu/~stephen/all-depends-list.c btw.. stehpen, when are you getting a commit bit? :) I certainly hope that soon enough ;) roman From owner-freebsd-hackers@FreeBSD.ORG Mon May 28 22:46:30 2007 Return-Path: X-Original-To: hackers@freebsd.org Delivered-To: freebsd-hackers@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id D9D5A16A524 for ; Mon, 28 May 2007 22:46:30 +0000 (UTC) (envelope-from mwm-keyword-freebsdhackers2.e313df@mired.org) Received: from mired.org (vpn.mired.org [66.92.153.74]) by mx1.freebsd.org (Postfix) with SMTP id 7CA4F13C4B7 for ; Mon, 28 May 2007 22:46:30 +0000 (UTC) (envelope-from mwm-keyword-freebsdhackers2.e313df@mired.org) Received: (qmail 27875 invoked by uid 1001); 28 May 2007 22:45:45 -0000 Received: by bhuda.mired.org (tmda-sendmail, from uid 1001); Mon, 28 May 2007 18:45:44 -0400 MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <18011.23448.513335.199513@bhuda.mired.org> Date: Mon, 28 May 2007 18:45:44 -0400 To: Hartmut Brandt In-Reply-To: <465B4A84.6050407@dlr.de> References: <4659EF80.70100@math.missouri.edu> <465AB421.10802@dlr.de> <18011.6019.436391.128372@bhuda.mired.org> <465B4A84.6050407@dlr.de> X-Mailer: VM 7.19 under Emacs 21.3.1 X-Primary-Address: mwm@mired.org X-face: "5Mnwy%?j>IIV\)A=):rjWL~NB2aH[}Yq8Z=u~vJ`"(,&SiLvbbz2W`; h9L,Yg`+vb1>RG% *h+%X^n0EZd>TM8_IB;a8F?(Fb"lw'IgCoyM.[Lg#r\ X-Delivery-Agent: TMDA/1.1.11 (Ladyburn) From: Mike Meyer Cc: ports@freebsd.org, hackers@freebsd.org Subject: Re: Looking for speed increases in "make index" and pkg_version for ports X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 May 2007 22:46:31 -0000 In <465B4A84.6050407@dlr.de>, Hartmut Brandt typed: > Mike Meyer wrote: > > In <465AB421.10802@dlr.de>, Hartmut Brandt typed: > >> 1. make and its sub-makes for a) reading the file; b) parsing the file > >> (note that .if and .for processing is done while parsing); c) processing > >> targets. > > > > Make and submakes have been gone over already. See > http://miller.emu.id.au/pmiller/books/rmch/ >. > > > > I'm not sure it can be applied to the ports tree, though. I haven't > > looked into it, but recalled this paper when you mentioned measuring > > makes and sub-makes. > Unfortunately you deleted the sentence before, so I rephrase it: before > looking into optimizations find out where the time is actually spend - > how many seconds of the hours the process takes, are actually spent in > make and sub-makes. If the entire process takes 2 hours of which the > makes take 20 seconds then by enhancing performance of make by 50% you > win 10 seconds. This is probably not worth a single line of additional code. > > The paper you point to talks about something entirely different. It think we're talking about two different things. You're talking about the efficiency of make, whereas he's talking about the efficiency of make. Um, wait. You're talking about what I'll call the *internal* efficiency of make, defined as how fast it does the things it does. He's talking about what I'll call the *external* efficiency of make, which is how well it does at doing the minimum amount of work it needs to do. I hope you can see where the confusion comes from. In particular, he talks about how recursive makefiles screw up evaluating complex variables, causing them to be executed multiple times. So if you're running a makefile to pull some variables value, as opposed to do real commands, and your entire process takes 2 hours and the Makefile takes 20 seconds, but it evaluates all the variables twice, then by fixing your makefile you win at least 59 minutes and 50 seconds. I think cutting the run time by 50% is worth some work. Benchmarking can help you decide which things it pays to work on if all you're worried about is the internal efficiency. However, the goal is to make the process faster, so we need to worry about the external efficiency as well. The problem here is that the worse it is, the less it looks like you stand to gain by looking at your makefile when you look at the benchmarks. Given that the ports system has both highly complex variables and is very recursive, I believe that it warrants investigation if you're going to work on making make in the ports faster. http://www.mired.org/consulting.html Independent Network/Unix/Perforce consultant, email for more information. From owner-freebsd-hackers@FreeBSD.ORG Mon May 28 23:26:08 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 75CB316A539 for ; Mon, 28 May 2007 23:26:08 +0000 (UTC) (envelope-from adamlsd@gmail.com) Received: from an-out-0708.google.com (an-out-0708.google.com [209.85.132.249]) by mx1.freebsd.org (Postfix) with ESMTP id 9963113C457 for ; Mon, 28 May 2007 23:26:07 +0000 (UTC) (envelope-from adamlsd@gmail.com) Received: by an-out-0708.google.com with SMTP id c14so532910anc for ; Mon, 28 May 2007 16:26:06 -0700 (PDT) DKIM-Signature: a=rsa-sha1; c=relaxed/relaxed; d=gmail.com; s=beta; h=domainkey-signature:received:received:mime-version:in-reply-to:references:content-type:message-id:content-transfer-encoding:from:subject:date:to:x-mailer:sender; b=TobenmaGThfphO3y5s5JLh5HfZMVGoiW8tmpuywmMAwgfSp65sLcUJpEkHh0g7vqfri6uY7O0vBbmilHb5wIO+rAg9gRTu30cV6w6+6WznrKul6XHkd90SnrqK2y8hg6iwBenyGaxfhM3X8GhJ2vvA8zE/04UVpT29KK6BG00yc= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=beta; h=received:mime-version:in-reply-to:references:content-type:message-id:content-transfer-encoding:from:subject:date:to:x-mailer:sender; b=aGuWrdKpDS2cRVm7wxYou2L8mE6i4Wx8qzsh1VYxmKi4Y2ZHj/TSZDMNoKLpwmIZp/nyfR9+cnGZpPi+ksKZVWHLD/26xHy6IKu7zpGCWOLzjy1jUiNQAFnzs/7D4pYvfzvNU6y+Ew9SuQR5/ize/CiCWNMaUhVMZruEmu6JLw8= Received: by 10.100.106.5 with SMTP id e5mr4771103anc.1180391381932; Mon, 28 May 2007 15:29:41 -0700 (PDT) Received: from ?192.168.123.216? ( [24.190.113.211]) by mx.google.com with ESMTP id d12sm10730275and.2007.05.28.15.29.40; Mon, 28 May 2007 15:29:40 -0700 (PDT) Mime-Version: 1.0 (Apple Message framework v752.2) In-Reply-To: References: Content-Type: text/plain; charset=US-ASCII; delsp=yes; format=flowed Message-Id: Content-Transfer-Encoding: 7bit From: Adam Martin Date: Mon, 28 May 2007 18:29:42 -0400 To: Ivan Voras , freebsd-hackers@freebsd.org X-Mailer: Apple Mail (2.752.2) Sender: Adam Martin X-Mailman-Approved-At: Mon, 28 May 2007 23:59:29 +0000 Cc: Subject: Re: GPT boot loader? X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 May 2007 23:26:08 -0000 Resent to freebsd-hackers@: Greetings, all. On 2007.05.22, at 03:57, Ivan Voras wrote: > Hi! > > I've had the opportunity to talk to Adam Martin, Marcel Moolenaar and > Peter Wemm about making GPT bootable, but not all of them at the same > time, so I'd like this thread to be the meeting point on the subject. I have offered to work on the MBR and some of the lower level assembler nasties to make this thing work. The most problematic issue is that the GPT requires 128 bit fields to identify the partition type. In a typical MBR we have 488 bytes to work with (512 bytes in one sector, 2 bytes for the boot-identifier, and another 32 bytes for the partition table.) If we insert a few valid GPT labels into the MBR to scan for, this takes up 16 bytes per entry. The lucky part is that the MBR really just needs to find a bootable partition, load a few sectors from it, and let go. > (Adam and Peter have offered to modify the boot loader chain, but > differently.) I can also work on boot2, not just mbr/boot0, to make it understand GPT, but I need to know a bit more about the information it needs to understand. In this situation, /boot/loader still needs GPT support -- according to Marcel, it gets help on the GPT interpretation from the EFI. I do not want to undertake the task of "GPT-ifying" the loader program. It's not worth it to have someone make the earlier boot stages work with GPT, unless the later stages also will support it. I have written a simple MBR code that will look for the first GPT partition marked with a GUID specified at compile time, and load the first 8k of that partition, to facilitate boot2. However, I do not think I will take this any further unless people take up the task of writing the GPT support for other parts of the boot-chain. > Summary: > > The idea is to replace bsdlabels with GPT. The problem is that GPT is > intended to be used with EFI and not to be bootable by regular BIOS > machines. For FreeBSD, there are two distinct cases: > > 1. the machine is GPT-only, there are no other MSDOS and bsdlabel > partitions > 2. the machine is dual-boot or has some other need to retain MSDOS > partitions. > > The second case is more convoluted as it means the MBR will hold a > regular MSDOS partition table, and one of those partitions will hold a > GPT - which is trivially done with GEOM but completely non-standard. Marcel and I discussed a hybrid solution, which requires almost no bootloader code to be written -- We could have the MBR point to a "legacy" boot partition, even down to a really small root filesystem, which the GPT also has an entry for, but the kernel in this root filesystem has a switch set to inform it that it must use GPT and not MBR/BSDlabels. This is not unlike what Apple have done with their EFI on x86 platform. In a related quirk, I have not used MBR partitions for almost 3 years now -- I just raw-BSDlabel my discs. So I know it is possible to do away with the MBR completely. The biggest obstacle to this on x86/amd64 computers is the single-sector bootloader limitations. GPT isn't very PC-friendly, as it also demands that the sectors right after the "fake MBR" be for the GPT. What I would love to see is some ability to have a smarter PC boot procedure. > I'd like to hear more from Adam and Peter (and others!) about their > ideas... Optimally, the PC architecture boot procedure (and perhaps the PC itself?) should die and fade away, leaving behind smarter bootloader and firmware code in its place, instead of shoehorning this functionality into an unnecessarily complicated bootloader chain. Since this does not seem to be in the cards, I guess we need to focus on this concept. -- ADAM David Alan Martin FreeBSD Hacker, TCSH Hacker, general UNIX coding Author of AutoFS for FreeBSD 6.x LSD Kernel Lead Developer Filesystem and Storage Lab, SUNY Stony Brook From owner-freebsd-hackers@FreeBSD.ORG Tue May 29 02:06:22 2007 Return-Path: X-Original-To: hackers@FreeBSD.org Delivered-To: freebsd-hackers@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 2DABA16A46F for ; Tue, 29 May 2007 02:06:22 +0000 (UTC) (envelope-from bland@FreeBSD.org) Received: from smtp1.jp.viruscheck.net (smtp1.jp.viruscheck.net [154.33.69.52]) by mx1.freebsd.org (Postfix) with ESMTP id EE72013C457 for ; Tue, 29 May 2007 02:06:21 +0000 (UTC) (envelope-from bland@FreeBSD.org) Received: from (mail3.jp.viruscheck.net) [154.33.69.39]:17298 by smtp1.jp.viruscheck.net with esmtp id 1Hsqn0-0000j3-GB ; Tue, 29 May 2007 10:46:22 +0900 Received: from (noc.orchid.orchidtechnology.com) [125.206.34.113]:33218 by mail3.jp.viruscheck.net with esmtp id 1Hsqn0-0007NI-1p ; Tue, 29 May 2007 10:46:22 +0900 Received: from [89.60.200.25] ([89.60.200.25]) by noc.orchid.orchidtechnology.com (8.13.4/8.13.4) with ESMTP id l4T1kKPw010849; Tue, 29 May 2007 10:46:21 +0900 (JST) (envelope-from bland@FreeBSD.org) Message-ID: <465B85E7.2090001@FreeBSD.org> Date: Tue, 29 May 2007 10:46:15 +0900 From: Alexander Nedotsukov User-Agent: Thunderbird 2.0.0.0 (Windows/20070326) MIME-Version: 1.0 To: Stephen Montgomery-Smith References: <4659EF80.70100@math.missouri.edu> <20070527223048.GA37505@icarus.home.lan> <465B0490.7080203@math.missouri.edu> In-Reply-To: <465B0490.7080203@math.missouri.edu> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: ports@FreeBSD.org, hackers@FreeBSD.org Subject: Re: Looking for speed increases in "make index" and pkg_version for ports X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 May 2007 02:06:22 -0000 Correct me if I wrong. Don't you missed the fact that chdir(2) changes process wide attribute? Though it's easy to fix with -C option. Stephen Montgomery-Smith wrote: > Jeremy Chadwick wrote: >> On Sun, May 27, 2007 at 03:52:16PM -0500, Stephen Montgomery-Smith >> wrote: >>> I have been thinking a lot about looking for speed increases for >>> "make index" and pkg_version and things like that. So for example, >>> in pkg_version, it calls "make -V PKGNAME" for every installed >>> package. Now "make -V PKGNAME" should be a speedy operation, but >>> the make has to load in and analyze bsd.port.mk, a quite >>> complicated file with about 200,000 characters in it, when all it >>> is needing to do is to figure out the value of the variable PKGNAME. >> >> I have a related question, pertaining to "make all-depends-list" and the >> utter atrocity that is the make variable ALL-DEPENDS-LIST. If you don't >> know what it is, look for ^ALL-DEPENDS-LIST around line 5175, in >> bsd.ports.mk. > > I posted this to ports@freebsd.org, but now I am realizing that it is > hackers@freebsd.org that gets more responses. Anyway, here is a > multithreaded program "all-depends-list" that can get you double the > speed on dual processor systems, and even some small speed gains on > single processor systems. E.g. > > all-depends-list /usr/ports/x11/xorg > > http://www.math.missouri.edu/~stephen/all-depends-list.c > > _______________________________________________ > freebsd-ports@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-ports > To unsubscribe, send any mail to "freebsd-ports-unsubscribe@freebsd.org" From owner-freebsd-hackers@FreeBSD.ORG Tue May 29 02:13:40 2007 Return-Path: X-Original-To: hackers@freebsd.org Delivered-To: freebsd-hackers@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 0B8AD16A4F1; Tue, 29 May 2007 02:13:40 +0000 (UTC) (envelope-from stephen@math.missouri.edu) Received: from sccmmhc92.asp.att.net (sccmmhc92.asp.att.net [204.127.203.212]) by mx1.freebsd.org (Postfix) with ESMTP id B6A5A13C457; Tue, 29 May 2007 02:13:39 +0000 (UTC) (envelope-from stephen@math.missouri.edu) Received: from laptop2.montlan (12-216-253-187.client.mchsi.com[12.216.253.187]) by sccmmhc92.asp.att.net (sccmmhc92) with ESMTP id <20070529021338m9200da0ije>; Tue, 29 May 2007 02:13:38 +0000 Message-ID: <465B8C51.6010806@math.missouri.edu> Date: Mon, 28 May 2007 21:13:37 -0500 From: Stephen Montgomery-Smith User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.8.1.2) Gecko/20070525 SeaMonkey/1.1.1 MIME-Version: 1.0 To: Roman Divacky References: <4659EF80.70100@math.missouri.edu> <20070527223048.GA37505@icarus.home.lan> <465B0490.7080203@math.missouri.edu> <20070528212036.GA35796@freebsd.org> In-Reply-To: <20070528212036.GA35796@freebsd.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: ports@freebsd.org, hackers@freebsd.org Subject: Re: Looking for speed increases in "make index" and pkg_version for ports X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 May 2007 02:13:40 -0000 Roman Divacky wrote: > On Mon, May 28, 2007 at 11:34:24AM -0500, Stephen Montgomery-Smith wrote: >> Jeremy Chadwick wrote: >>> On Sun, May 27, 2007 at 03:52:16PM -0500, Stephen Montgomery-Smith wrote: >>>> I have been thinking a lot about looking for speed increases for "make >>>> index" and pkg_version and things like that. So for example, in >>>> pkg_version, it calls "make -V PKGNAME" for every installed package. Now >>>> "make -V PKGNAME" should be a speedy operation, but the make has to load >>>> in and analyze bsd.port.mk, a quite complicated file with about 200,000 >>>> characters in it, when all it is needing to do is to figure out the >>>> value of the variable PKGNAME. >>> I have a related question, pertaining to "make all-depends-list" and the >>> utter atrocity that is the make variable ALL-DEPENDS-LIST. If you don't >>> know what it is, look for ^ALL-DEPENDS-LIST around line 5175, in >>> bsd.ports.mk. >> I posted this to ports@freebsd.org, but now I am realizing that it is >> hackers@freebsd.org that gets more responses. Anyway, here is a >> multithreaded program "all-depends-list" that can get you double the >> speed on dual processor systems, and even some small speed gains on >> single processor systems. E.g. >> >> all-depends-list /usr/ports/x11/xorg >> >> http://www.math.missouri.edu/~stephen/all-depends-list.c > > btw.. stehpen, when are you getting a commit bit? :) I certainly hope that soon enough ;) Probably not. The program seems to have a bug in it. In particular, I didn't read the fgetln man page sufficiently well. So think of it as a proof of concept rather than a finished product. I'm going to rest from this stuff for a while, but I enjoyed the exchanges and it has given me encouragement to work on it again in the future sometime. Stephen From owner-freebsd-hackers@FreeBSD.ORG Tue May 29 04:11:51 2007 Return-Path: X-Original-To: hackers@freebsd.org Delivered-To: freebsd-hackers@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 0ADCA16A473; Tue, 29 May 2007 04:11:51 +0000 (UTC) (envelope-from youshi10@u.washington.edu) Received: from mxout4.cac.washington.edu (mxout4.cac.washington.edu [140.142.33.19]) by mx1.freebsd.org (Postfix) with ESMTP id DA65613C457; Tue, 29 May 2007 04:11:50 +0000 (UTC) (envelope-from youshi10@u.washington.edu) Received: from smtp.washington.edu (smtp.washington.edu [140.142.33.9] (may be forged)) by mxout4.cac.washington.edu (8.13.7+UW06.06/8.13.7+UW07.03) with ESMTP id l4T4BotN030208 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Mon, 28 May 2007 21:11:50 -0700 X-Auth-Received: from [192.168.10.45] (c-67-166-149-71.hsd1.ca.comcast.net [67.166.149.71]) (authenticated authid=youshi10) by smtp.washington.edu (8.13.7+UW06.06/8.13.7+UW07.03) with ESMTP id l4T4BnpU022624 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NOT); Mon, 28 May 2007 21:11:49 -0700 Message-ID: <465BA81D.7060600@u.washington.edu> Date: Mon, 28 May 2007 21:12:13 -0700 From: Garrett Cooper User-Agent: Thunderbird 2.0.0.0 (Windows/20070326) MIME-Version: 1.0 To: Stephen Montgomery-Smith References: <4659EF80.70100@math.missouri.edu> <20070527223048.GA37505@icarus.home.lan> <465B0490.7080203@math.missouri.edu> <20070528212036.GA35796@freebsd.org> <465B8C51.6010806@math.missouri.edu> In-Reply-To: <465B8C51.6010806@math.missouri.edu> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-PMX-Version: 5.3.1.294258, Antispam-Engine: 2.5.1.298604, Antispam-Data: 2007.5.28.204933 X-Uwash-Spam: Gauge=IIIIIII, Probability=7%, Report='__CP_URI_IN_BODY 0, __CT 0, __CTE 0, __CT_TEXT_PLAIN 0, __HAS_MSGID 0, __MIME_TEXT_ONLY 0, __MIME_VERSION 0, __SANE_MSGID 0, __USER_AGENT 0' Cc: ports@freebsd.org, Roman Divacky , hackers@freebsd.org Subject: Re: Looking for speed increases in "make index" and pkg_version for ports X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 May 2007 04:11:51 -0000 Stephen Montgomery-Smith wrote: > Roman Divacky wrote: >> On Mon, May 28, 2007 at 11:34:24AM -0500, Stephen Montgomery-Smith wrote: >>> Jeremy Chadwick wrote: >>>> On Sun, May 27, 2007 at 03:52:16PM -0500, Stephen Montgomery-Smith >>>> wrote: >>>>> I have been thinking a lot about looking for speed increases for >>>>> "make index" and pkg_version and things like that. So for example, >>>>> in pkg_version, it calls "make -V PKGNAME" for every installed >>>>> package. Now "make -V PKGNAME" should be a speedy operation, but >>>>> the make has to load in and analyze bsd.port.mk, a quite >>>>> complicated file with about 200,000 characters in it, when all it >>>>> is needing to do is to figure out the value of the variable PKGNAME. >>>> I have a related question, pertaining to "make all-depends-list" and >>>> the >>>> utter atrocity that is the make variable ALL-DEPENDS-LIST. If you >>>> don't >>>> know what it is, look for ^ALL-DEPENDS-LIST around line 5175, in >>>> bsd.ports.mk. >>> I posted this to ports@freebsd.org, but now I am realizing that it is >>> hackers@freebsd.org that gets more responses. Anyway, here is a >>> multithreaded program "all-depends-list" that can get you double the >>> speed on dual processor systems, and even some small speed gains on >>> single processor systems. E.g. >>> >>> all-depends-list /usr/ports/x11/xorg >>> >>> http://www.math.missouri.edu/~stephen/all-depends-list.c >> >> btw.. stehpen, when are you getting a commit bit? :) I certainly hope >> that soon enough ;) > > Probably not. The program seems to have a bug in it. In particular, I > didn't read the fgetln man page sufficiently well. So think of it as a > proof of concept rather than a finished product. > > I'm going to rest from this stuff for a while, but I enjoyed the > exchanges and it has given me encouragement to work on it again in the > future sometime. > > Stephen fgetln(2) just scans ahead to the next newline, so the pointer to the next line is returned and the length of the string (with newline char included) is stored in the len variable (2nd parameter to function). -Garrett From owner-freebsd-hackers@FreeBSD.ORG Tue May 29 04:41:42 2007 Return-Path: X-Original-To: hackers@freebsd.org Delivered-To: freebsd-hackers@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 92E6916A46E for ; Tue, 29 May 2007 04:41:42 +0000 (UTC) (envelope-from rick@kiwi-computer.com) Received: from kiwi-computer.com (keira.kiwi-computer.com [63.224.10.3]) by mx1.freebsd.org (Postfix) with SMTP id 2101113C4B7 for ; Tue, 29 May 2007 04:41:41 +0000 (UTC) (envelope-from rick@kiwi-computer.com) Received: (qmail 22786 invoked by uid 2001); 29 May 2007 04:15:08 -0000 Date: Mon, 28 May 2007 23:15:08 -0500 From: "Rick C. Petty" To: Stephen Montgomery-Smith Message-ID: <20070529041508.GA22341@keira.kiwi-computer.com> References: <4659EF80.70100@math.missouri.edu> <20070527223048.GA37505@icarus.home.lan> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20070527223048.GA37505@icarus.home.lan> User-Agent: Mutt/1.4.2.1i Cc: hackers@freebsd.org Subject: Re: Looking for speed increases in "make index" and pkg_version for ports X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: rick-freebsd@kiwi-computer.com List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 May 2007 04:41:42 -0000 On Sun, May 27, 2007 at 03:30:48PM -0700, Jeremy Chadwick wrote: > > That said, I'll ask this out in the open: am I the only one who sees the > benefit of GNU make in regards to this? There's a lot of built-in > functions in GNU make which could help in regards to ports. I have no > qualms with PMake per se, but if another tool gives us what we need, > then maybe we should consider the pros and cons of adapting that. > There's also CMake, which is incredibly fast. Yes, you are. What gmake benefits? Gmake does not provide the flexibility and power that pmake provides. Off the top of my head: gmake does not have ".for" loops, variable expansion modifiers, or even the "!=" shell command variable assigment. I use these in almost every Makefile I write, and the ports uses these things quite a bit. Also, gmake syntax is horrendous compared to pmake. People are already complaining about how ugly the ports makefiles are-- they'd be worse under gmake. Might as well rewrite the whole infrastructure in /bin/sh ... Also, there's the licensing issues. Remember-- any significant changes to this infrastructure has to work with the core utilities.. this leaves out gmake, python, ruby, etc. I doubt anyone will find anything as powerful as pmake without sacrificing the much-used flexibility it provides. -- Rick C. Petty From owner-freebsd-hackers@FreeBSD.ORG Tue May 29 06:10:28 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id AE33A16A46E for ; Tue, 29 May 2007 06:10:28 +0000 (UTC) (envelope-from mashtizadeh@gmail.com) Received: from wx-out-0506.google.com (wx-out-0506.google.com [66.249.82.229]) by mx1.freebsd.org (Postfix) with ESMTP id 5E55C13C45A for ; Tue, 29 May 2007 06:10:28 +0000 (UTC) (envelope-from mashtizadeh@gmail.com) Received: by wx-out-0506.google.com with SMTP id h28so1223423wxd for ; Mon, 28 May 2007 23:10:27 -0700 (PDT) DKIM-Signature: a=rsa-sha1; c=relaxed/relaxed; d=gmail.com; s=beta; h=domainkey-signature:received:received:message-id:date:from:to:subject:mime-version:content-type; b=HrnN6hwlRS/dd4vPox8LMenKLjn6fDfJzuuc4trM3X+fjVesY+1yHdQx1l0J5+0OcLazD89wUpXe0baaYCKdUTeYiL/l4/Ap1kO9n4X5wsfWi3YdW7T8WkKs+TP67kN8uI0TzwYSPghkN8+AsYnzyhtP+mdMpM/9ltOmk6U26Bc= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=beta; h=received:message-id:date:from:to:subject:mime-version:content-type; b=MUPXNrH58S/bOjR9Z9vqZ/S30jL5IeXhTvFPHiPPE9iS9paYbRbRwMh6562ySsTfZ4VlBKK8iZP1iak/tIWFtYehJgHN48IylB6meRZKo+JLWdHW6425xeTNIvvibzs/7meT4jgcdIwRsYffZHCcJUYW6uZnmBFTKbpIX2G2how= Received: by 10.143.4.16 with SMTP id g16mr198721wfi.1180417296210; Mon, 28 May 2007 22:41:36 -0700 (PDT) Received: by 10.142.251.8 with HTTP; Mon, 28 May 2007 22:41:36 -0700 (PDT) Message-ID: <440b3e930705282241u44fa03a8gb0fedbb1617518df@mail.gmail.com> Date: Tue, 29 May 2007 01:41:36 -0400 From: "Ali Mashtizadeh" To: freebsd-hackers@freebsd.org, freebsd-current@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: base64 Content-Disposition: inline X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: Subject: Xorg port problem? X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 May 2007 06:10:28 -0000 SSBzZWVtIHRvIGhhdmUgcHJvYmxlbXMgYnVpbGRpbmcgeG9yZy1zZXJ2ZXIgcG9ydCBvbiBmcmVl YnNkIEhFQUQgd2l0aCBhCnNuYXBzaG90IG9mIHNvdXJjZSB0aGF0IGlzIGFib3V0ICA1IGRheXMg b2xkLiBJJ20gcnVubmluZyBBTUQ2NCBidWlsZApldmVyeXRoaW5nIGkndmUgaW5zdGFsbGVkIGlz IGZpbmUgZXhjZXB0IHdoZW4gYnVpbGRpbmcgeG9yZy1zZXJ2ZXIgZ2NjCjQuMnN0YXJ0cyB0byB1 c2UgYWxsIHRoZSBtZW1vcnkgaSBoYXZlIGFuZCB0aGVuIGV4aXRzIHNheWluZyBub3QKZW5vdWdo IG1lbW9yeQooT3ZlciAyR0JzIG9mIFJBTSArIFN3YXAgYmVpbmcgdXNlZCkuIEl0IGRvZXMgdGhp cyBjb25zaXN0ZW50bHkgd2hlbiBpdAp0cmllcyB0byBjb21waWxlIHhmODZQY2lTY2FuLmMgKGhv cGUgdGhhdHMgdGhlIHJpZ2h0IGZpbGUpLiBTb21ldGltZSB0aGUKc3lzdGVtIGxvY2tzIHVwIGZy b20gdGhpcyBsb2FkIGJ1dCBpIHRoaW5rIHRoYXQgWkZTICdzIGZhdWx0LiBJIHdhcwp3b25kZXJp bmcgbWF5YmUgSSBqdXN0IGhhZCBiYWQgbHVjayB3aXRoIHRoaXMgYnVpbGQ/IEkgaGF2ZW4ndCBz ZWVuIHRoaXMgaW4KcHJldmlvdXMgYnVpbGRzIEknbSB3b25kZXJpbmcgaWYgaXRzIEdDQyA0LjI/ CgpOb3RlIHRoaXMgd2FzIGRvbmUgb24gYSBjb21wbGV0ZWx5IGNsZWFuIG5ldyBzeXN0ZW0uIFdp dGggYSBaRlMgcm9vdCA6LSkKCi0tIApBbGkgTWFzaHRpemFkZWgK2LnZhNuMINmF2LTYqtuMINiy 2KfYr9mHCg== From owner-freebsd-hackers@FreeBSD.ORG Tue May 29 06:18:16 2007 Return-Path: X-Original-To: hackers@freebsd.org Delivered-To: freebsd-hackers@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id AC6DC16A484; Tue, 29 May 2007 06:18:16 +0000 (UTC) (envelope-from Hartmut.Brandt@dlr.de) Received: from smtp-1.dlr.de (smtp-1.dlr.de [195.37.61.185]) by mx1.freebsd.org (Postfix) with ESMTP id 48A1C13C448; Tue, 29 May 2007 06:18:16 +0000 (UTC) (envelope-from Hartmut.Brandt@dlr.de) Received: from [192.168.2.102] ([172.21.151.1]) by smtp-1.dlr.de with Microsoft SMTPSVC(6.0.3790.1830); Tue, 29 May 2007 08:18:14 +0200 Message-ID: <465BC5A6.7090806@dlr.de> Date: Tue, 29 May 2007 08:18:14 +0200 From: Hartmut Brandt User-Agent: Thunderbird 1.5.0.10 (Windows/20070221) MIME-Version: 1.0 References: <4659EF80.70100@math.missouri.edu> <465AB421.10802@dlr.de> <18011.6019.436391.128372@bhuda.mired.org> <465B4A84.6050407@dlr.de> <18011.23448.513335.199513@bhuda.mired.org> In-Reply-To: <18011.23448.513335.199513@bhuda.mired.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-OriginalArrivalTime: 29 May 2007 06:18:14.0332 (UTC) FILETIME=[23B33FC0:01C7A1B9] Cc: ports@freebsd.org, hackers@freebsd.org Subject: Re: Looking for speed increases in "make index" and pkg_version for ports X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 May 2007 06:18:16 -0000 Mike Meyer wrote: > In <465B4A84.6050407@dlr.de>, Hartmut Brandt typed: >> Mike Meyer wrote: >>> In <465AB421.10802@dlr.de>, Hartmut Brandt typed: >>>> 1. make and its sub-makes for a) reading the file; b) parsing the file >>>> (note that .if and .for processing is done while parsing); c) processing >>>> targets. >>> Make and submakes have been gone over already. See >> http://miller.emu.id.au/pmiller/books/rmch/ >. >>> >>> I'm not sure it can be applied to the ports tree, though. I haven't >>> looked into it, but recalled this paper when you mentioned measuring >>> makes and sub-makes. >> Unfortunately you deleted the sentence before, so I rephrase it: before >> looking into optimizations find out where the time is actually spend - >> how many seconds of the hours the process takes, are actually spent in >> make and sub-makes. If the entire process takes 2 hours of which the >> makes take 20 seconds then by enhancing performance of make by 50% you >> win 10 seconds. This is probably not worth a single line of additional code. >> >> The paper you point to talks about something entirely different. > > It think we're talking about two different things. You're talking > about the efficiency of make, whereas he's talking about the > efficiency of make. Um, wait. > > You're talking about what I'll call the *internal* efficiency of make, > defined as how fast it does the things it does. He's talking about > what I'll call the *external* efficiency of make, which is how well it > does at doing the minimum amount of work it needs to do. I hope you > can see where the confusion comes from. Yeah, from that you deleted the other two of my points in your response where I talked about shells and external commands executed by make. You cited the point where I asked for numbers on *internal* efficiency and the point to a paper about *external* efficiency. I've seen no numbers WHAT actually makes the ports stuff so slow. To make my point a last time: until there are numbers, there is no guess around what to do. harti From owner-freebsd-hackers@FreeBSD.ORG Tue May 29 06:36:23 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id F223B16A481 for ; Tue, 29 May 2007 06:36:23 +0000 (UTC) (envelope-from dougb@FreeBSD.org) Received: from mail2.fluidhosting.com (mx24.fluidhosting.com [204.14.89.7]) by mx1.freebsd.org (Postfix) with SMTP id 90A2713C4CA for ; Tue, 29 May 2007 06:36:23 +0000 (UTC) (envelope-from dougb@FreeBSD.org) Received: (qmail 13772 invoked by uid 399); 29 May 2007 06:09:42 -0000 Received: from localhost (HELO lap.dougb.net) (dougb@dougbarton.us@127.0.0.1) by localhost with SMTP; 29 May 2007 06:09:42 -0000 X-Originating-IP: 127.0.0.1 Message-ID: <465BC3A4.7030008@FreeBSD.org> Date: Mon, 28 May 2007 23:09:40 -0700 From: Doug Barton Organization: http://www.FreeBSD.org/ User-Agent: Thunderbird 2.0.0.0 (X11/20070525) MIME-Version: 1.0 To: Ali Mashtizadeh References: <440b3e930705282241u44fa03a8gb0fedbb1617518df@mail.gmail.com> In-Reply-To: <440b3e930705282241u44fa03a8gb0fedbb1617518df@mail.gmail.com> X-Enigmail-Version: 0.95.0 OpenPGP: id=D5B2F0FB Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: freebsd-hackers@freebsd.org, freebsd-current@freebsd.org Subject: Re: Xorg port problem? X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 May 2007 06:36:24 -0000 Ali Mashtizadeh wrote: > I seem to have problems building xorg-server port on freebsd HEAD with a > snapshot of source that is about 5 days old. I'm running AMD64 build > everything i've installed is fine except when building xorg-server gcc > 4.2starts to use all the memory i have and then exits saying not > enough memory > (Over 2GBs of RAM + Swap being used). It does this consistently when it > tries to compile xf86PciScan.c (hope thats the right file). Sometime the > system locks up from this load but i think that ZFS 's fault. I was > wondering maybe I just had bad luck with this build? I haven't seen this in > previous builds I'm wondering if its GCC 4.2? > > Note this was done on a completely clean new system. With a ZFS root :-) May not be the answer you want to hear, but I built all the xorg stuff multiple times on -current systems both pre and post the gcc + symver + version bump eras, and didn't have the problems you're seeing. What I don't have though is zfs, so that may be a place to look. One way to check is if you're not doing it already, use an mfs /tmp and see if that helps. Doug -- This .signature sanitized for your protection From owner-freebsd-hackers@FreeBSD.ORG Tue May 29 06:42:18 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id AC8E116A513 for ; Tue, 29 May 2007 06:42:18 +0000 (UTC) (envelope-from ale@FreeBSD.org) Received: from lab.alexdupre.com (lab.alexdupre.com [81.174.31.42]) by mx1.freebsd.org (Postfix) with ESMTP id EC19213C448 for ; Tue, 29 May 2007 06:42:17 +0000 (UTC) (envelope-from ale@FreeBSD.org) Received: (qmail 26223 invoked from network); 29 May 2007 06:15:36 -0000 Received: from unknown (HELO athlon.alexdupre.com) (192.168.178.2) by lab.alexdupre.com with SMTP; 29 May 2007 06:15:36 -0000 Message-ID: <465BC507.1060902@FreeBSD.org> Date: Tue, 29 May 2007 08:15:35 +0200 From: Alex Dupre User-Agent: Thunderbird 2.0.0.0 (X11/20070420) MIME-Version: 1.0 To: Doug Barton References: <440b3e930705282241u44fa03a8gb0fedbb1617518df@mail.gmail.com> <465BC3A4.7030008@FreeBSD.org> In-Reply-To: <465BC3A4.7030008@FreeBSD.org> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: freebsd-hackers@freebsd.org, Ali Mashtizadeh , freebsd-current@freebsd.org Subject: Re: Xorg port problem? X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 May 2007 06:42:18 -0000 Doug Barton wrote: >> (Over 2GBs of RAM + Swap being used). It does this consistently when it >> tries to compile xf86PciScan.c (hope thats the right file). > > May not be the answer you want to hear, but I built all the xorg stuff > multiple times on -current systems both pre and post the gcc + symver > + version bump eras, and didn't have the problems you're seeing. It's the well-known problem of new gcc 4.2 optimizations (bug). Simply compile with -O0 instead of -O2. -- Alex Dupre From owner-freebsd-hackers@FreeBSD.ORG Tue May 29 07:07:47 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id D532D16A51B for ; Tue, 29 May 2007 07:07:47 +0000 (UTC) (envelope-from dougb@FreeBSD.org) Received: from mail2.fluidhosting.com (mx24.fluidhosting.com [204.14.89.7]) by mx1.freebsd.org (Postfix) with SMTP id 75FDE13C4B0 for ; Tue, 29 May 2007 07:07:47 +0000 (UTC) (envelope-from dougb@FreeBSD.org) Received: (qmail 11579 invoked by uid 399); 29 May 2007 07:07:46 -0000 Received: from localhost (HELO lap.dougb.net) (dougb@dougbarton.us@127.0.0.1) by localhost with SMTP; 29 May 2007 07:07:46 -0000 X-Originating-IP: 127.0.0.1 Message-ID: <465BD140.20504@FreeBSD.org> Date: Tue, 29 May 2007 00:07:44 -0700 From: Doug Barton Organization: http://www.FreeBSD.org/ User-Agent: Thunderbird 2.0.0.0 (X11/20070525) MIME-Version: 1.0 To: Alex Dupre References: <440b3e930705282241u44fa03a8gb0fedbb1617518df@mail.gmail.com> <465BC3A4.7030008@FreeBSD.org> <465BC507.1060902@FreeBSD.org> In-Reply-To: <465BC507.1060902@FreeBSD.org> X-Enigmail-Version: 0.95.0 OpenPGP: id=D5B2F0FB Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: freebsd-hackers@freebsd.org, Ali Mashtizadeh , freebsd-current@freebsd.org Subject: Re: Xorg port problem? X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 May 2007 07:07:47 -0000 Alex Dupre wrote: > Doug Barton wrote: >>> (Over 2GBs of RAM + Swap being used). It does this consistently when it >>> tries to compile xf86PciScan.c (hope thats the right file). >> May not be the answer you want to hear, but I built all the xorg stuff >> multiple times on -current systems both pre and post the gcc + symver >> + version bump eras, and didn't have the problems you're seeing. > > It's the well-known problem of new gcc 4.2 optimizations (bug). Simply > compile with -O0 instead of -O2. Not disputing your answer, but I'm curious. Why would it cause problems on some systems but not others? I haven't done anything with my cflags ... Doug -- This .signature sanitized for your protection From owner-freebsd-hackers@FreeBSD.ORG Tue May 29 07:57:00 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 2405516A506; Tue, 29 May 2007 07:57:00 +0000 (UTC) (envelope-from emss@free.fr) Received: from kellthuzad.dmz.nerim.net (smtp-dmz-232-tuesday.dmz.nerim.net [195.5.254.232]) by mx1.freebsd.org (Postfix) with ESMTP id D216113C4BC; Tue, 29 May 2007 07:56:59 +0000 (UTC) (envelope-from emss@free.fr) Received: from kraid.nerim.net (smtp-102-tuesday.nerim.net [62.4.16.102]) by kellthuzad.dmz.nerim.net (Postfix) with ESMTP id E1B2316A19; Tue, 29 May 2007 09:24:50 +0200 (CEST) Received: from srvbsdnanssv.interne.kisoft-services.com (kisoft.net1.nerim.net [62.212.107.51]) by kraid.nerim.net (Postfix) with ESMTP id 68930CF133; Tue, 29 May 2007 07:24:52 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by srvbsdnanssv.interne.kisoft-services.com (Postfix) with ESMTP id 117BBC93E; Tue, 29 May 2007 09:24:51 +0200 (CEST) X-Virus-Scanned: amavisd-new at interne.kisoft-services.com Received: from srvbsdnanssv.interne.kisoft-services.com ([127.0.0.1]) by localhost (srvbsdnanssv.interne.kisoft-services.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id BRexud9r7wC1; Tue, 29 May 2007 09:24:47 +0200 (CEST) Received: by srvbsdnanssv.interne.kisoft-services.com (Postfix, from userid 1001) id 8FCB5C8FD; Tue, 29 May 2007 09:24:47 +0200 (CEST) To: Doug Barton From: Eric Masson In-Reply-To: <465BD140.20504@FreeBSD.org> (Doug Barton's message of "Tue, 29 May 2007 00:07:44 -0700") References: <440b3e930705282241u44fa03a8gb0fedbb1617518df@mail.gmail.com> <465BC3A4.7030008@FreeBSD.org> <465BC507.1060902@FreeBSD.org> <465BD140.20504@FreeBSD.org> X-Operating-System: FreeBSD 6.2-RELEASE-p4 i386 Date: Tue, 29 May 2007 09:24:47 +0200 Message-ID: <86abvoks74.fsf@srvbsdnanssv.interne.kisoft-services.com> User-Agent: Gnus/5.1007 (Gnus v5.10.7) XEmacs/21.5-b27 (berkeley-unix) MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-15 Content-Transfer-Encoding: 8bit Cc: freebsd-hackers@freebsd.org, Ali Mashtizadeh , freebsd-current@freebsd.org, Alex Dupre Subject: Re: Xorg port problem? X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 May 2007 07:57:00 -0000 Doug Barton writes: Hello, > Not disputing your answer, but I'm curious. Why would it cause > problems on some systems but not others? I haven't done anything with > my cflags ... Can't give any response, but I was hit by the bug Alex talks about during upgrade. My box ran out of swap while compiling xf86scanpci.c with -O2 (inspiron 4150, 512MB ram, 512MB swap, 07-05-27 -current) Éric Masson -- Même à 15 minutes par livre à 250°, un con reste un con, et celui là restera pour encore un bon moment un con majuscule. On devrait le mettre à Sèvre, sous cloche, le con étalon. (c)(r)(tm) -+- RM in http://neuneu.ctw.cc - Comment servir le neuneu -+- From owner-freebsd-hackers@FreeBSD.ORG Tue May 29 08:03:05 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id B80ED16A47B for ; Tue, 29 May 2007 08:03:05 +0000 (UTC) (envelope-from ale@FreeBSD.org) Received: from andxor.it (relay.andxor.it [195.223.2.3]) by mx1.freebsd.org (Postfix) with SMTP id D6BEB13C487 for ; Tue, 29 May 2007 08:03:04 +0000 (UTC) (envelope-from ale@FreeBSD.org) Received: (qmail 23191 invoked from network); 29 May 2007 07:36:22 -0000 Received: from unknown (HELO ale.andxor.it) (192.168.2.5) by andxor.it with SMTP; 29 May 2007 07:36:22 -0000 Message-ID: <465BD7F5.8030109@FreeBSD.org> Date: Tue, 29 May 2007 09:36:21 +0200 From: Alex Dupre User-Agent: Thunderbird 2.0.0.0 (X11/20070428) MIME-Version: 1.0 To: Doug Barton References: <440b3e930705282241u44fa03a8gb0fedbb1617518df@mail.gmail.com> <465BC3A4.7030008@FreeBSD.org> <465BC507.1060902@FreeBSD.org> <465BD140.20504@FreeBSD.org> In-Reply-To: <465BD140.20504@FreeBSD.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-hackers@freebsd.org, Ali Mashtizadeh , freebsd-current@freebsd.org Subject: Re: Xorg port problem? X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 May 2007 08:03:05 -0000 Doug Barton ha scritto: > Not disputing your answer, but I'm curious. Why would it cause > problems on some systems but not others? I haven't done anything with > my cflags ... It depends on how much RAM + Swap do you have. I know people with 1.5Gb that have such problem and others with 2.5Gb that haven't it (if I remember correctly the amounts). The "bug" is the un-optimized optimization :-) There is already a patch that will be included in 4.2.1 release, but I hope will be backported before in FreeBSD gcc. -- Alex Dupre From owner-freebsd-hackers@FreeBSD.ORG Tue May 29 08:16:22 2007 Return-Path: X-Original-To: hackers@freebsd.org Delivered-To: freebsd-hackers@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 1A6BA16A520 for ; Tue, 29 May 2007 08:16:22 +0000 (UTC) (envelope-from peterjeremy@optushome.com.au) Received: from turion.vk2pj.dyndns.org (c220-239-3-125.belrs4.nsw.optusnet.com.au [220.239.3.125]) by mx1.freebsd.org (Postfix) with ESMTP id 96AC113C4B9 for ; Tue, 29 May 2007 08:16:21 +0000 (UTC) (envelope-from peterjeremy@optushome.com.au) Received: from turion.vk2pj.dyndns.org (localhost.vk2pj.dyndns.org [127.0.0.1]) by turion.vk2pj.dyndns.org (8.14.1/8.14.1) with ESMTP id l4T84vQq053845; Tue, 29 May 2007 18:04:57 +1000 (EST) (envelope-from peter@turion.vk2pj.dyndns.org) Received: (from peter@localhost) by turion.vk2pj.dyndns.org (8.14.1/8.14.1/Submit) id l4T84twC053844; Tue, 29 May 2007 18:04:55 +1000 (EST) (envelope-from peter) Date: Tue, 29 May 2007 18:04:55 +1000 From: Peter Jeremy To: Bakul Shah Message-ID: <20070529080455.GM1161@turion.vk2pj.dyndns.org> References: <4659EF80.70100@math.missouri.edu> <20070527231254.D0C325B5E@mail.bitblocks.com> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="Qrgsu6vtpU/OV/zm" Content-Disposition: inline In-Reply-To: <20070527231254.D0C325B5E@mail.bitblocks.com> X-PGP-Key: http://members.optusnet.com.au/peterjeremy/pubkey.asc User-Agent: Mutt/1.5.15 (2007-04-06) Cc: ports@freebsd.org, hackers@freebsd.org Subject: Re: Looking for speed increases in "make index" and pkg_version for ports X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 May 2007 08:16:22 -0000 --Qrgsu6vtpU/OV/zm Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On 2007-May-27 16:12:54 -0700, Bakul Shah wrote: >Given the size and complexity of the port system I have long >felt that rather than do everything via more and more complex >Mk/*.mk what is is needed is a ports server and a thin CLI >frontend to it. I don't believe this is practical. Both package names and port dependencies depend on the options that are selected as well as what other ports are already installed. A centralised ports server is not going to have access to this information. --=20 Peter Jeremy --Qrgsu6vtpU/OV/zm Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (FreeBSD) iD8DBQFGW96n/opHv/APuIcRAkbWAJ43zRo5IXnrl9cQPopsH4WOY+9Z+wCgxE4S BIdmKkeF5ZwKH7q1+c7ijp8= =Ur1Y -----END PGP SIGNATURE----- --Qrgsu6vtpU/OV/zm-- From owner-freebsd-hackers@FreeBSD.ORG Tue May 29 08:32:09 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 0062B16A51C; Tue, 29 May 2007 08:32:08 +0000 (UTC) (envelope-from peterjeremy@optushome.com.au) Received: from turion.vk2pj.dyndns.org (c220-239-3-125.belrs4.nsw.optusnet.com.au [220.239.3.125]) by mx1.freebsd.org (Postfix) with ESMTP id 5B62813C4AD; Tue, 29 May 2007 08:32:07 +0000 (UTC) (envelope-from peterjeremy@optushome.com.au) Received: from turion.vk2pj.dyndns.org (localhost.vk2pj.dyndns.org [127.0.0.1]) by turion.vk2pj.dyndns.org (8.14.1/8.14.1) with ESMTP id l4T8W6Px049107; Tue, 29 May 2007 18:32:06 +1000 (EST) (envelope-from peter@turion.vk2pj.dyndns.org) Received: (from peter@localhost) by turion.vk2pj.dyndns.org (8.14.1/8.14.1/Submit) id l4T8W5wn049106; Tue, 29 May 2007 18:32:05 +1000 (EST) (envelope-from peter) Date: Tue, 29 May 2007 18:32:05 +1000 From: Peter Jeremy To: Michel Talon Message-ID: <20070529083205.GN1161@turion.vk2pj.dyndns.org> References: <20070527221528.GA19603@lpthe.jussieu.fr> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="fWddYNRDgTk9wQGZ" Content-Disposition: inline In-Reply-To: <20070527221528.GA19603@lpthe.jussieu.fr> X-PGP-Key: http://members.optusnet.com.au/peterjeremy/pubkey.asc User-Agent: Mutt/1.5.15 (2007-04-06) Cc: ports@freebsd.org, freebsd-hackers@freebsd.org Subject: Re: Looking for speed increases in "make index" X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 May 2007 08:32:09 -0000 --fWddYNRDgTk9wQGZ Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On 2007-May-28 00:15:28 +0200, Michel Talon wrote: >Of course a lot of people have thinked about it, and quickly realized >that it was not going to work. In the bsd.ports.mk, evaluation of one >variable may be dependent on some conditional, which may itself be >dependant on some other variable, appearing as some target. This >constantly happens in bsd.ports.mk. =2E.. >To gain some performance, a first idea would be to simplify >bsd.ports.mk. I am convinced that a substantial part of the 4000 lines >are historical crap which serve no useful purpose. There are tons of >variables who have probably purely anecdotical interest. There are >targets which could be happily suppressed. These two paragraphs are contradictory. If bsd.ports.mk really is 4000 lines of historical crap and full full of unused variables than an "evaluate on demand" approach would be a big win. >A second idea would be to multithread make, since modern machines will >have a lot of cores. Not yet. Dual-core machines are still relatively new. Of the seven hosts that I regularly use, only one is dual core. Threading adds significant overheads and it's not clear how it would help in this specific instance. >Anyways, one of the things which cannot be a big factor is reading >bsd.ports.mk from hard disk. Reading 210KB from a modern disk is quite fast. > It is certainly cached in memory when you >run make index. Which is the specific case being discussed in this thread. > On the other hand it is so big that it probably doesn't >fit in cache, or probably only fits in caches of machines generously >endowed. It may not stay in L1 cache but unless your system is extremely memory stressed, it should stay in RAM. I'd suggest that if bsd.ports.mk is not cached then you have more serious problems to address before you worry about the performance of make(1). >(i am thinking Pentium 4 versus Core 2 Duo) is striking. It is less >than 10 minutes on the big cache machine versus 30 minutes on the small >cache one. If the cache effect is indeed dominant, then reducing the >size of bsd.ports.mk by all possible means would be very beneficial. This is likely to be more a combination of the crippled pentium 4 core and a largish memory footprint for make than anything to do with bsd.ports.mk itself. >By the way an alternative system would be to use something other than >make to do the job. This is a possibility but it needs to be something that can be part of the base system. This means you are limited to C/C++, awk, sh and make. In particular, python, ruby, perl and similar scripting languages are out. Whilst make seems the obvious choice for the ports infrastructure, in reality, the infrastructure does not really need or use the sort of implicit dependency tracking and target transformations that make excels at. Of course, any alternative to make needs to provide a language for defining dependencies that is equally powerful and easy to use. --=20 Peter Jeremy --fWddYNRDgTk9wQGZ Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (FreeBSD) iD8DBQFGW+UF/opHv/APuIcRAhGyAKC62+7Ap+18egbYrs2OLvpollDxkgCeI8ds daDjRlZq0UdaFn8k1M7fE3c= =2/hK -----END PGP SIGNATURE----- --fWddYNRDgTk9wQGZ-- From owner-freebsd-hackers@FreeBSD.ORG Tue May 29 10:02:59 2007 Return-Path: X-Original-To: hackers@freebsd.org Delivered-To: freebsd-hackers@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id C2D7816A53B; Tue, 29 May 2007 10:02:59 +0000 (UTC) (envelope-from peterjeremy@optushome.com.au) Received: from turion.vk2pj.dyndns.org (c220-239-3-125.belrs4.nsw.optusnet.com.au [220.239.3.125]) by mx1.freebsd.org (Postfix) with ESMTP id A466013C46C; Tue, 29 May 2007 10:02:58 +0000 (UTC) (envelope-from peterjeremy@optushome.com.au) Received: from turion.vk2pj.dyndns.org (localhost.vk2pj.dyndns.org [127.0.0.1]) by turion.vk2pj.dyndns.org (8.14.1/8.14.1) with ESMTP id l4TA2uWg094097; Tue, 29 May 2007 20:02:56 +1000 (EST) (envelope-from peter@turion.vk2pj.dyndns.org) Received: (from peter@localhost) by turion.vk2pj.dyndns.org (8.14.1/8.14.1/Submit) id l4TA2uqo094096; Tue, 29 May 2007 20:02:56 +1000 (EST) (envelope-from peter) Date: Tue, 29 May 2007 20:02:56 +1000 From: Peter Jeremy To: Stephen Montgomery-Smith Message-ID: <20070529100255.GA70055@turion.vk2pj.dyndns.org> References: <4659EF80.70100@math.missouri.edu> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="+HP7ph2BbKc20aGI" Content-Disposition: inline In-Reply-To: <4659EF80.70100@math.missouri.edu> X-PGP-Key: http://members.optusnet.com.au/peterjeremy/pubkey.asc User-Agent: Mutt/1.5.15 (2007-04-06) Cc: ports@freebsd.org, hackers@freebsd.org Subject: Re: Looking for speed increases in "make index" and pkg_version for ports X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 May 2007 10:02:59 -0000 --+HP7ph2BbKc20aGI Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On 2007-May-27 15:52:16 -0500, Stephen Montgomery-Smith wrote: > I have been thinking a lot about looking for speed increases for "make=20 > index" and pkg_version and things like that. So for example, in=20 > pkg_version, it calls "make -V PKGNAME" for every installed package. Now= =20 > "make -V PKGNAME" should be a speedy operation, but the make has to load = in=20 > and analyze bsd.port.mk, a quite complicated file with about 200,000=20 > characters in it, when all it is needing to do is to figure out the value= of=20 > the variable PKGNAME. This would be trivial if some packages didn't change names depending on options and what was installed. I agree that parsing a 210KB file 17,000 times is not going to be fast. Especially since some ports include bsd.ports.mk multiple times... > I suggest rewriting "make" so that variables are only evaluated on a "nee= d=20 > to know" basis. This sounds like a good idea but I suspect it's not going to be feasible. The biggest problem I see is that the make language allows variables to be expanded either when they are assigned or when they are referenced. If a variable expansion is delayed from the assignment to the first use then the expansion must be performed using the state of make as it was when the variable was assigned. The cost of keeping this state probably exceeds the cost of actually evaluating the variable. > So, for example, if all we need to know is PKGNAME, there=20 > is no need to evaluate, for example, _RUN_LIB_DEPENDS, unless the writer = of=20 > that particular port has done something like having PORTNAME depend on th= e=20 > value of _RUN_LIB_DEPENDS. Rather than trying to develop a tool that can quickly expand PKGNAME irrespective of what convoluted code the author has used, how about a partial solution? For most ports, PKGNAME depends solely on 3 or 4 variables that are statically defined in the port Makefile. The obvious solution would seem to be to develop a script that can handle the easy cases itself and punt the difficult cases back to make. The definition of 'easy' can be adjusted over time to increase performance. This approach would seem to have a relatively low bar to entry whilst offering good effort/performance tradeoff at the low end. The various depends lists would seem amenable to the same approach - though the entry level tool will have far lower coverage due to the extensive use of USE_GNOME=3D... and similar 'macro'-style constructs. --=20 Peter Jeremy --+HP7ph2BbKc20aGI Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (FreeBSD) iD8DBQFGW/pP/opHv/APuIcRArw9AJ9WsRYjwIrgxdypFLGTK4G7JnQKPQCeNJvl QvVG9B881APC8Ap2MDhjbyA= =M39M -----END PGP SIGNATURE----- --+HP7ph2BbKc20aGI-- From owner-freebsd-hackers@FreeBSD.ORG Tue May 29 10:34:32 2007 Return-Path: X-Original-To: hackers@freebsd.org Delivered-To: freebsd-hackers@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 6937916A4F1; Tue, 29 May 2007 10:34:32 +0000 (UTC) (envelope-from peterjeremy@optushome.com.au) Received: from turion.vk2pj.dyndns.org (c220-239-3-125.belrs4.nsw.optusnet.com.au [220.239.3.125]) by mx1.freebsd.org (Postfix) with ESMTP id C900813C4BD; Tue, 29 May 2007 10:34:31 +0000 (UTC) (envelope-from peterjeremy@optushome.com.au) Received: from turion.vk2pj.dyndns.org (localhost.vk2pj.dyndns.org [127.0.0.1]) by turion.vk2pj.dyndns.org (8.14.1/8.14.1) with ESMTP id l4TAYU09092399; Tue, 29 May 2007 20:34:30 +1000 (EST) (envelope-from peter@turion.vk2pj.dyndns.org) Received: (from peter@localhost) by turion.vk2pj.dyndns.org (8.14.1/8.14.1/Submit) id l4TAYTnM092394; Tue, 29 May 2007 20:34:29 +1000 (EST) (envelope-from peter) Date: Tue, 29 May 2007 20:34:29 +1000 From: Peter Jeremy To: Stephen Montgomery-Smith , ports@freebsd.org, hackers@freebsd.org Message-ID: <20070529103429.GD70055@turion.vk2pj.dyndns.org> References: <4659EF80.70100@math.missouri.edu> <20070527223048.GA37505@icarus.home.lan> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="MnLPg7ZWsaic7Fhd" Content-Disposition: inline In-Reply-To: <20070527223048.GA37505@icarus.home.lan> X-PGP-Key: http://members.optusnet.com.au/peterjeremy/pubkey.asc User-Agent: Mutt/1.5.15 (2007-04-06) Cc: Subject: Re: Looking for speed increases in "make index" and pkg_version for ports X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 May 2007 10:34:32 -0000 --MnLPg7ZWsaic7Fhd Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On 2007-May-27 15:30:48 -0700, Jeremy Chadwick wrote: >This sounds like a good solution. In fact, I'm lead to believe that >heavy reliance on /bin/sh is part of why the ports collection is slow. Someone needs to enable accounting on a recent -current (with the high-resolution accounting records) and look at where the time is actually going. (My -current box needs upgrading before I could do this). That said, /bin/sh is dynamically linked and a fork/exec is not cheap. Some quick-and-not-necessarily-reliable tests on 6.2-STABLE/amd64 show that /bin/sh takes about 2.5 times as long to start as /rescue/sh (though it's only 2:1 on i386). (These are different boxes so the absolute times aren't comparable). amd64% time sh -c 'i=3D0; while [ $i -lt 10000 ]; do i=3D$(($i+1)); echo fo= o; done' >/dev/null sh -c 'i=3D0; while [ $i -lt 10000 ]; do i=3D$(($i+1)); echo foo; done' > = 0.20s user 0.08s system 98% cpu 0.283 total amd64% time sh -c 'i=3D0; while [ $i -lt 10000 ]; do i=3D$(($i+1)); echo fo= o; done' >/dev/null=20 sh -c 'i=3D0; while [ $i -lt 10000 ]; do i=3D$(($i+1)); echo foo; done' > = 0.22s user 0.06s system 97% cpu 0.287 total amd64% time sh -c 'i=3D0; while [ $i -lt 10000 ]; do i=3D$(($i+1)); echo fo= o; done' >/dev/null sh -c 'i=3D0; while [ $i -lt 10000 ]; do i=3D$(($i+1)); echo foo; done' > = 0.19s user 0.10s system 98% cpu 0.288 total amd64% time sh -c 'i=3D0; while [ $i -lt 10000 ]; do i=3D$(($i+1)); /rescue= /sh -c "echo foo"; done' >/dev/null sh -c > /dev/null 0.84s user 6.12s system 97% cpu 7.162 total amd64% time sh -c 'i=3D0; while [ $i -lt 10000 ]; do i=3D$(($i+1)); /rescue= /sh -c "echo foo"; done' >/dev/null sh -c > /dev/null 1.12s user 6.05s system 97% cpu 7.366 total amd64% time sh -c 'i=3D0; while [ $i -lt 10000 ]; do i=3D$(($i+1)); /bin/sh= -c "echo foo"; done' >/dev/null sh -c > /dev/null 5.72s user 13.40s system 96% cpu 19.734 total amd64% time sh -c 'i=3D0; while [ $i -lt 10000 ]; do i=3D$(($i+1)); /bin/sh= -c "echo foo"; done' >/dev/null=20 sh -c > /dev/null 5.97s user 12.89s system 97% cpu 19.407 total amd64% =20 i386% time sh -c 'i=3D0; while [ $i -lt 10000 ]; do i=3D$(($i+1)); echo foo= ; done' >/dev/null sh -c 'i=3D0; while [ $i -lt 10000 ]; do i=3D$(($i+1)); echo foo; done' > = 0.17s user 0.03s system 95% cpu 0.208 total i386% time sh -c 'i=3D0; while [ $i -lt 10000 ]; do i=3D$(($i+1)); echo foo= ; done' >/dev/null sh -c 'i=3D0; while [ $i -lt 10000 ]; do i=3D$(($i+1)); echo foo; done' > = 0.17s user 0.03s system 99% cpu 0.199 total i386% time sh -c 'i=3D0; while [ $i -lt 10000 ]; do i=3D$(($i+1)); echo foo= ; done' >/dev/null sh -c 'i=3D0; while [ $i -lt 10000 ]; do i=3D$(($i+1)); echo foo; done' > = 0.16s user 0.04s system 99% cpu 0.200 total i386% time sh -c 'i=3D0; while [ $i -lt 10000 ]; do i=3D$(($i+1)); /rescue/= sh -c "echo foo"; done' >/dev/null sh -c > /dev/null 3.68s user 18.19s system 98% cpu 22.212 total i386% time sh -c 'i=3D0; while [ $i -lt 10000 ]; do i=3D$(($i+1)); /rescue/= sh -c "echo foo"; done' >/dev/null sh -c > /dev/null 3.34s user 18.54s system 98% cpu 22.110 total i386% time sh -c 'i=3D0; while [ $i -lt 10000 ]; do i=3D$(($i+1)); /bin/sh = -c "echo foo"; done' >/dev/null=20 sh -c > /dev/null 12.03s user 29.42s system 98% cpu 41.965 total i386% time sh -c 'i=3D0; while [ $i -lt 10000 ]; do i=3D$(($i+1)); /bin/sh = -c "echo foo"; done' >/dev/null sh -c > /dev/null 12.20s user 29.25s system 98% cpu 41.975 total --=20 Peter Jeremy --MnLPg7ZWsaic7Fhd Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (FreeBSD) iD8DBQFGXAG1/opHv/APuIcRAks8AKCdrEBJKOJKodeq4N+IAMNh4AKvrwCfQdGP CxG8wrZlvsAwy/UGKRfiuzc= =lNGY -----END PGP SIGNATURE----- --MnLPg7ZWsaic7Fhd-- From owner-freebsd-hackers@FreeBSD.ORG Tue May 29 11:16:48 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 2AB0716A46D for ; Tue, 29 May 2007 11:16:48 +0000 (UTC) (envelope-from soralx@cydem.org) Received: from cydem.org (S0106000103ce4c9c.vc.shawcable.net [24.87.27.3]) by mx1.freebsd.org (Postfix) with ESMTP id 0D3B813C4B8 for ; Tue, 29 May 2007 11:16:47 +0000 (UTC) (envelope-from soralx@cydem.org) Received: by cydem.org (Postfix/FreeBSD, from userid 58) id 2E8A590A27; Tue, 29 May 2007 04:00:29 -0700 (PDT) X-Spam-Checker-Version: SpamAssassin 3.1.8 (2007-02-13) on cydem.org X-Spam-Level: X-Spam-Status: No, score=-1.4 required=5.0 tests=ALL_TRUSTED autolearn=failed version=3.1.8 Received: from soralx.cydem.org (unknown [192.168.0.249]) by cydem.org (Postfix/FreeBSD) with ESMTP id C6B0D9098C for ; Tue, 29 May 2007 04:00:25 -0700 (PDT) Date: Tue, 29 May 2007 04:00:25 -0700 From: To: Message-ID: <20070529040025.52124254@soralx.cydem.org> In-Reply-To: References: <4659EF80.70100__16104.363036766$1180300054$gmane$org@math.missouri.edu> <465B02C4.2020508@math.missouri.edu> X-Mailer: Claws Mail 2.9.1 (GTK+ 2.10.11; i386-portbld-freebsd5.4) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Subject: Re: Looking for speed increases in "make index" and pkg_version for ports X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 May 2007 11:16:48 -0000 Ivan Voras wrote: > > Because the information is not a constant. For example, the mpg123 > > port changes its PKGNAME as soon as esound is installed. > > Maybe the time has come to give up on some of the flexibility the > ports tree has (and this particular one is confusing to the users) > [...] maybe more confusing to portupgrade -- one day I'll get tired of running 'pkgdb -F' almose every time before portupgrade to fix old dependencies (packages with old config or things like ru-xmms instead of xmms). This is portupgrade's problem really, but doing away from such "flexibility" would make life slightly easier... [SorAlx] ridin' VN1500-B2 From owner-freebsd-hackers@FreeBSD.ORG Tue May 29 11:22:34 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 3550116A46B for ; Tue, 29 May 2007 11:22:34 +0000 (UTC) (envelope-from suresh.kumar.j@gmail.com) Received: from an-out-0708.google.com (an-out-0708.google.com [209.85.132.245]) by mx1.freebsd.org (Postfix) with ESMTP id EAFB113C487 for ; Tue, 29 May 2007 11:22:33 +0000 (UTC) (envelope-from suresh.kumar.j@gmail.com) Received: by an-out-0708.google.com with SMTP id c14so571343anc for ; Tue, 29 May 2007 04:22:33 -0700 (PDT) DKIM-Signature: a=rsa-sha1; c=relaxed/relaxed; d=gmail.com; s=beta; h=domainkey-signature:received:received:message-id:date:from:to:subject:mime-version:content-type:content-transfer-encoding:content-disposition; b=Se7y0Wu3WX7Gf92sDFy18d76uu6ojQo4UWzrbQaMKYCzqPKenQcWdfCcqdRU1KWZV/Pypm4ZMNmKiahwvXMAs7dqhmwBfMj/QvOBo7QrHT/kmeUcIbCuBzgHFuOL7lrjrkS+K7XYkrLCrhvyGX9MvsPOrn0SUtwRk4Wq5cbyxFQ= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=beta; h=received:message-id:date:from:to:subject:mime-version:content-type:content-transfer-encoding:content-disposition; b=kulBWuZXYTFg6aNsKljndF76CkwsFG1ikSKdVM4q/FXSJYR7GIlCd+elLUZkjaIJgGpOS9DPRTnW7mE8/SpTs68QMoQN2TjCMAEIEjNqqDHfU6EUrimZrS6VBryEIiraesie8bRSZQiLOhqW5VKL47m1kh4J1aSz2trB1MZYqCQ= Received: by 10.114.145.1 with SMTP id s1mr3343415wad.1180436068006; Tue, 29 May 2007 03:54:28 -0700 (PDT) Received: by 10.115.46.11 with HTTP; Tue, 29 May 2007 03:54:27 -0700 (PDT) Message-ID: <88b2f6dd0705290354l261e5995ta2af8cc2157d0f89@mail.gmail.com> Date: Tue, 29 May 2007 16:24:27 +0530 From: "Suresh Kumar J" To: freebsd-hackers@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Content-Disposition: inline Subject: Using get_system_info() - Obtaining the CPU states... X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 May 2007 11:22:34 -0000 Hi there, I have an application running on FreeBSD which needs to display the CPU states as like the "top" command shows in the 3rd line as below: --------------------------------------------------------- CPU states: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle --------------------------------------------------------- I learnt that the "top" command uses the get_system_info() function for printing the CPU state detail. But I could not locate the source code of this function. Could anybody help me in locating the header/source file in which the get_system_info() function is located?. Any other ideas are also welcome. Note: I do not want to display the CPU load average... -- Thanks, Suresh Kumar J From owner-freebsd-hackers@FreeBSD.ORG Tue May 29 11:52:26 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 9DDD116A41F for ; Tue, 29 May 2007 11:52:26 +0000 (UTC) (envelope-from lists-freebsd@biaix.org) Received: from grummit.biaix.org (86.Red-213-97-212.staticIP.rima-tde.net [213.97.212.86]) by mx1.freebsd.org (Postfix) with SMTP id C555613C45A for ; Tue, 29 May 2007 11:52:25 +0000 (UTC) (envelope-from lists-freebsd@biaix.org) Received: (qmail 27644 invoked by uid 1012); 29 May 2007 11:26:01 -0000 Date: Tue, 29 May 2007 13:26:01 +0200 From: Joan Picanyol i Puig To: freebsd-hackers@freebsd.org Message-ID: <20070529112601.GC27304@grummit.biaix.org> Mail-Followup-To: freebsd-hackers@freebsd.org References: <4659EF80.70100@math.missouri.edu> <465AB421.10802@dlr.de> <18011.6019.436391.128372@bhuda.mired.org> <465B4A84.6050407@dlr.de> <18011.23448.513335.199513@bhuda.mired.org> <465BC5A6.7090806@dlr.de> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <465BC5A6.7090806@dlr.de> User-Agent: Mutt/1.5.11 X-Mailman-Approved-At: Tue, 29 May 2007 12:15:26 +0000 Subject: Re: Looking for speed increases in "make index" and pkg_version for ports X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 May 2007 11:52:26 -0000 * Hartmut Brandt [20070529 08:21]: > I've seen no numbers WHAT actually makes the ports stuff so slow. To > make my point a last time: until there are numbers, there is no guess > around what to do. It just occured to me that DTrace could be a big help with this task. I suggest someone convinces jb@ to take a look. qvb -- pica From owner-freebsd-hackers@FreeBSD.ORG Tue May 29 12:45:51 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 646CF16A468 for ; Tue, 29 May 2007 12:45:51 +0000 (UTC) (envelope-from mike@urgle.com) Received: from anchor-fallback-96.mail.demon.net (anchor-fallback-96.mail.demon.net [194.217.242.83]) by mx1.freebsd.org (Postfix) with ESMTP id 27F9F13C44C for ; Tue, 29 May 2007 12:45:51 +0000 (UTC) (envelope-from mike@urgle.com) Received: from anchor-post-34.mail.demon.net ([194.217.242.92]:4991 "EHLO anchor-post-34.mail.demon.net") by anchor-fallback-96.mail.demon.net with ESMTP id S448466AbXE2LiA (ORCPT ); Tue, 29 May 2007 11:38:00 +0000 Received: from wafer.urgle.com ([80.177.40.53]) by anchor-post-34.mail.demon.net with esmtp (Exim 4.42) id 1Ht01R-000Pa4-G5; Tue, 29 May 2007 11:37:54 +0000 Received: from mike by wafer.urgle.com with local (Exim 4.67 (FreeBSD)) (envelope-from ) id 1Ht01R-0009pP-Am; Tue, 29 May 2007 11:37:53 +0000 Date: Tue, 29 May 2007 12:37:53 +0100 From: Mike Bristow To: Suresh Kumar J Message-ID: <20070529113753.GA37427@wafer.urgle.com> References: <88b2f6dd0705290354l261e5995ta2af8cc2157d0f89@mail.gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <88b2f6dd0705290354l261e5995ta2af8cc2157d0f89@mail.gmail.com> User-Agent: Mutt/1.5.15 (2007-04-06) Cc: freebsd-hackers@freebsd.org Subject: Re: Using get_system_info() - Obtaining the CPU states... X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 May 2007 12:45:51 -0000 On Tue, May 29, 2007 at 04:24:27PM +0530, Suresh Kumar J wrote: > I learnt that the "top" command uses the get_system_info() function > for printing the CPU state detail. But I could not locate the source > code of this function. Could anybody help me in locating the > header/source file in which the get_system_info() function is > located? /usr/src/usr.bin/top/machine.c (which can be found on the web at http://cvsweb.freebsd.org/src/usr.bin/top/machine.c) -- Shenanigans! Shenanigans! Best of 3! -- Flash From owner-freebsd-hackers@FreeBSD.ORG Tue May 29 12:55:09 2007 Return-Path: X-Original-To: hackers@freebsd.org Delivered-To: freebsd-hackers@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 99D7116A494; Tue, 29 May 2007 12:55:09 +0000 (UTC) (envelope-from jdc@parodius.com) Received: from mx01.sc1.parodius.com (mx01.sc1.parodius.com [72.20.106.3]) by mx1.freebsd.org (Postfix) with ESMTP id 5DA7C13C4BC; Tue, 29 May 2007 12:55:09 +0000 (UTC) (envelope-from jdc@parodius.com) Received: by mx01.sc1.parodius.com (Postfix, from userid 1000) id 0CA1B1CC01C; Tue, 29 May 2007 05:40:05 -0700 (PDT) Date: Tue, 29 May 2007 05:40:05 -0700 From: Jeremy Chadwick To: Peter Jeremy Message-ID: <20070529124005.GA52331@eos.sc1.parodius.com> Mail-Followup-To: Peter Jeremy , Stephen Montgomery-Smith , ports@freebsd.org, hackers@freebsd.org References: <4659EF80.70100@math.missouri.edu> <20070527223048.GA37505@icarus.home.lan> <20070529103429.GD70055@turion.vk2pj.dyndns.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20070529103429.GD70055@turion.vk2pj.dyndns.org> User-Agent: Mutt/1.5.15 (2007-04-06) Cc: ports@freebsd.org, hackers@freebsd.org Subject: Re: Looking for speed increases in "make index" and pkg_version for ports X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 May 2007 12:55:09 -0000 On Tue, May 29, 2007 at 08:34:29PM +1000, Peter Jeremy wrote: > On 2007-May-27 15:30:48 -0700, Jeremy Chadwick wrote: > >This sounds like a good solution. In fact, I'm lead to believe that > >heavy reliance on /bin/sh is part of why the ports collection is slow. > > Someone needs to enable accounting on a recent -current (with the > high-resolution accounting records) and look at where the time is > actually going. (My -current box needs upgrading before I could > do this). The best I was able to do: I have a 6.2-RELEASE box which contains profiled libraries, and managed to make a profiled /bin/sh. This didn't take much work (just some modification of src/bin/sh/Makefile), but one thing which did stump me was the .gmon file never getting written. Turns out trap.c calls _exit(2) not exit(3). Change that and voila. Admittedly I'm not that familiar with gprof, and I'm also left wondering if profiling /bin/sh is going to help us, since we don't have a direct way of determining which shell commands take the most time -- just which C functions are most heavily used. -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB | From owner-freebsd-hackers@FreeBSD.ORG Tue May 29 16:52:55 2007 Return-Path: X-Original-To: hackers@freebsd.org Delivered-To: freebsd-hackers@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 6565B16A41F for ; Tue, 29 May 2007 16:52:55 +0000 (UTC) (envelope-from youshi10@u.washington.edu) Received: from mxout7.cac.washington.edu (mxout7.cac.washington.edu [140.142.32.178]) by mx1.freebsd.org (Postfix) with ESMTP id 47C2513C489 for ; Tue, 29 May 2007 16:52:55 +0000 (UTC) (envelope-from youshi10@u.washington.edu) Received: from hymn01.u.washington.edu (hymn01.u.washington.edu [140.142.8.55]) by mxout7.cac.washington.edu (8.13.7+UW06.06/8.13.7+UW07.03) with ESMTP id l4TGqshS020777 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Tue, 29 May 2007 09:52:55 -0700 Received: from localhost (localhost [127.0.0.1]) by hymn01.u.washington.edu (8.13.7+UW06.06/8.13.7+UW07.03) with ESMTP id l4TGqsj7030577 for ; Tue, 29 May 2007 09:52:54 -0700 X-Auth-Received: from [192.55.52.10] by hymn01.u.washington.edu via HTTP; Tue, 29 May 2007 09:52:54 PDT Date: Tue, 29 May 2007 09:52:54 -0700 (PDT) From: youshi10@u.washington.edu To: hackers@freebsd.org Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-PMX-Version: 5.3.1.294258, Antispam-Engine: 2.5.1.298604, Antispam-Data: 2007.5.29.92633 X-Uwash-Spam: Gauge=IIIIIII, Probability=7%, Report='SUPERLONG_LINE 0.05, NO_REAL_NAME 0, __CT 0, __CT_TEXT_PLAIN 0, __HAS_MSGID 0, __MIME_TEXT_ONLY 0, __MIME_VERSION 0, __SANE_MSGID 0' Cc: Subject: Kernel panicking in 6.2 and 7-CURRENT -- interrupt issues X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 May 2007 16:52:55 -0000 Hello, I'm trying to install 7-CURRENT on my desktop, locally instead of on a virtual machine, and I'm running into an issue where the kernel almost always panics on boot with my motherboard (ASUS P5N-SLI E), due to some sort of interrupt assignment / probing issues. Observations: 1. 7-CURRENT (built) never properly detects the interrupts on the system. 2. 7-CURRENT (May ISO snapshot) doesn't properly detect my USB keyboard, but boots and runs sysinstall. 3. 6.2 RELEASE panics if I don't boot FreeBSD up into "safe mode", due to an issue with the ohci driver (I think the error message had something to deal with device adding / enumeration and not being able to find a memory / interrupt address). First off, if I could get some of the command line arguments to pass to the kernel to emulate safe-mode, that would be much appreciated. Second off, if anybody has any ideas on how to debug this issue, I'll go off and try to determine what the cause is. If it's anything like it was before (clean out /usr/obj, rebuild), I'll be ok. Otherwise, I'll have to purchase more parts and build another dedicated FreeBSD system :(.. Thanks, -Garrett From owner-freebsd-hackers@FreeBSD.ORG Tue May 29 18:27:55 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 1423616A49A for ; Tue, 29 May 2007 18:27:55 +0000 (UTC) (envelope-from rmgls@orange.fr) Received: from smtp24.orange.fr (smtp24.orange.fr [193.252.22.25]) by mx1.freebsd.org (Postfix) with ESMTP id 6F42013C46C for ; Tue, 29 May 2007 18:27:53 +0000 (UTC) (envelope-from rmgls@orange.fr) Received: from smtp24.orange.fr (mwinf2454 [10.232.5.162]) by mwinf2430.orange.fr (SMTP Server) with ESMTP id 9580C1C6E9BF for ; Tue, 29 May 2007 20:10:29 +0200 (CEST) Received: from me-wanadoo.net (localhost [127.0.0.1]) by mwinf2454.orange.fr (SMTP Server) with ESMTP id 786FE70000A9 for ; Tue, 29 May 2007 20:10:27 +0200 (CEST) Received: from orange.fr (ARouen-153-1-44-243.w90-17.abo.wanadoo.fr [90.17.227.243]) by mwinf2454.orange.fr (SMTP Server) with ESMTP id 65AD5700009C for ; Tue, 29 May 2007 20:10:27 +0200 (CEST) X-ME-UUID: 20070529181027416.65AD5700009C@mwinf2454.orange.fr To: freebsd-hackers@freebsd.org From: rmgls@wanadoo.fr Date: Tue, 29 May 2007 20:10:26 +0200 Sender: rmgls@orange.fr Message-Id: <20070529181027.65AD5700009C@mwinf2454.orange.fr> Subject: direct I/O access X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 May 2007 18:27:55 -0000 hi all, Sorry for cross posting, but perhaps hackers is a better list than multimedia for this topic. i am trying to port my old assembler soft for Dos to FreeBSD. i need to write and read directly to the midi and scsi device. when i try something like this i receive a sigbus error SORRY, i am NOT nor a C nor a FreeBSD expert!!! all i know is Assembly language! i made some search in the devel handbook and did not found the solution. What is wrong here? Can you enlight me please? Many thanks in advance and bests regards Raoul rmgls@wanadeoo.fr ---cut--- .data .align 4 params: .word 0x330,2,1 # midi port => enabling IO ??? .text .align 4 .global _start _start: nop pushl params pushl $0x3 movl $0Xa5,%eax int $0x80 addl $0x08,%esp movw $0x331,%dx # status register inb %dx,%al # ... pushl $0 # exit movl $0x1,%eax int $0x80 From owner-freebsd-hackers@FreeBSD.ORG Tue May 29 18:44:53 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id D1DBC16A400; Tue, 29 May 2007 18:44:53 +0000 (UTC) (envelope-from kris@obsecurity.org) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.freebsd.org (Postfix) with ESMTP id BA03513C457; Tue, 29 May 2007 18:44:53 +0000 (UTC) (envelope-from kris@obsecurity.org) Received: from obsecurity.dyndns.org (elvis.mu.org [192.203.228.196]) by elvis.mu.org (Postfix) with ESMTP id 4E5081A4D81; Tue, 29 May 2007 11:46:02 -0700 (PDT) Received: from rot13.obsecurity.org (rot13.obsecurity.org [192.168.1.5]) by obsecurity.dyndns.org (Postfix) with ESMTP id 185E7512AB; Tue, 29 May 2007 14:44:53 -0400 (EDT) Received: by rot13.obsecurity.org (Postfix, from userid 1001) id 0CB58C1DD; Tue, 29 May 2007 14:44:53 -0400 (EDT) Date: Tue, 29 May 2007 14:44:53 -0400 From: Kris Kennaway To: Doug Barton Message-ID: <20070529184452.GA48484@rot13.obsecurity.org> References: <440b3e930705282241u44fa03a8gb0fedbb1617518df@mail.gmail.com> <465BC3A4.7030008@FreeBSD.org> <465BC507.1060902@FreeBSD.org> <465BD140.20504@FreeBSD.org> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="DocE+STaALJfprDB" Content-Disposition: inline In-Reply-To: <465BD140.20504@FreeBSD.org> User-Agent: Mutt/1.4.2.2i Cc: freebsd-hackers@freebsd.org, Ali Mashtizadeh , freebsd-current@freebsd.org, Alex Dupre Subject: Re: Xorg port problem? X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 May 2007 18:44:54 -0000 --DocE+STaALJfprDB Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Tue, May 29, 2007 at 12:07:44AM -0700, Doug Barton wrote: > Alex Dupre wrote: > > Doug Barton wrote: > >>> (Over 2GBs of RAM + Swap being used). It does this consistently when = it > >>> tries to compile xf86PciScan.c (hope thats the right file). > >> May not be the answer you want to hear, but I built all the xorg stuff > >> multiple times on -current systems both pre and post the gcc + symver > >> + version bump eras, and didn't have the problems you're seeing. > >=20 > > It's the well-known problem of new gcc 4.2 optimizations (bug). Simply > > compile with -O0 instead of -O2. >=20 > Not disputing your answer, but I'm curious. Why would it cause > problems on some systems but not others? I haven't done anything with > my cflags ... It wants to use ${BIGNUM} amount of memory to optimize a huge C file, so that might be fatal (or just terminally irritating) on smaller systems. Kris --DocE+STaALJfprDB Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.3 (FreeBSD) iD8DBQFGXHSkWry0BWjoQKURAqOAAJwLB0XknNO5MDikrIYZfzPYrxEinwCeJUIL dw0VatqZb5TFQUC+8YBgREo= =9odR -----END PGP SIGNATURE----- --DocE+STaALJfprDB-- From owner-freebsd-hackers@FreeBSD.ORG Tue May 29 21:14:13 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 8C4CE16A46B for ; Tue, 29 May 2007 21:14:13 +0000 (UTC) (envelope-from didier@aida.org) Received: from derny.org (aida.homeunix.com [82.239.250.150]) by mx1.freebsd.org (Postfix) with ESMTP id 0963313C448 for ; Tue, 29 May 2007 21:14:12 +0000 (UTC) (envelope-from didier@aida.org) Received: from derny.org (localhost.derny.org [127.0.0.1]) by derny.org (Postfix) with ESMTP id EB1B122836 for ; Tue, 29 May 2007 22:56:41 +0200 (CEST) Received: from [192.168.1.5] (nabucco.derny.org [192.168.1.5]) by derny.org (Postfix) with ESMTP for ; Tue, 29 May 2007 22:56:41 +0200 (CEST) Mime-Version: 1.0 (Apple Message framework v752.2) Content-Transfer-Encoding: 7bit Message-Id: <31ED92D6-BE85-4BE6-A27D-F7A59E9C3656@aida.org> Content-Type: text/plain; charset=US-ASCII; format=flowed To: freebsd-hackers@freebsd.org From: didier derny Date: Tue, 29 May 2007 22:56:40 +0200 X-Mailer: Apple Mail (2.752.2) X-Antivirus: ok Subject: Freebsd 6.2 panic X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 May 2007 21:14:13 -0000 I have some machines running php/mysql/postfix from time to time the computers running FreeBSD 6,2 crashes with the following messages fatal trap 12:page fault while in kernel mode cpuid=0; apic id=00 fault virtual address = 0x104 fault code = supervisor read, page not present instruction pointer= 0x20:0xc066c731 stack pointer=0x28:0xe74fec90 frame pointer=0x28:0xe74fec9c code segment=base 0x0,limit 0xfffff,type 0x1b = DPL 0, pres 1, def32 1, gran 1 processor eflag=resume,IOPL=0 current process=5 (thread taskq) trap number=12 panic page fault cpuid=0 uptime:16h7m48s cannot dump. no dump device defined Automatic rebbot in 15 second - press a key on the console to abort Rebooting cpu_reset:stopping other CPU does anyone has an idea on what I can do to solve this problem ? thanks for your help -- didier From owner-freebsd-hackers@FreeBSD.ORG Tue May 29 21:17:17 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id B3AA416A46B for ; Tue, 29 May 2007 21:17:17 +0000 (UTC) (envelope-from dmw@unete.cl) Received: from mail08.ifxnetworks.com (mail08.ifxnetworks.com [190.61.128.18]) by mx1.freebsd.org (Postfix) with ESMTP id 5990813C4BB for ; Tue, 29 May 2007 21:17:17 +0000 (UTC) (envelope-from dmw@unete.cl) Received: (qmail 5152 invoked from network); 29 May 2007 21:17:16 -0000 Received: from unknown (HELO webmail.ifxnw.cl) ([190.61.128.24]) (envelope-sender ) by mail08.ifxnetworks.com (qmail-ldap-1.03) with SMTP for ; 29 May 2007 21:17:16 -0000 Received: from 64.117.137.69 (proxying for 161.131.179.197, 161.131.179.197) (SquirrelMail authenticated user dmw@unete.cl) by webmail.ifxnw.cl with HTTP; Tue, 29 May 2007 17:17:16 -0400 (CLT) Message-ID: <43801.64.117.137.69.1180473436.squirrel@webmail.ifxnw.cl> Date: Tue, 29 May 2007 17:17:16 -0400 (CLT) From: "Daniel Molina Wegener" To: freebsd-hackers@freebsd.org User-Agent: SquirrelMail/1.4.8 MIME-Version: 1.0 Content-Type: text/plain;charset=iso-8859-1 Content-Transfer-Encoding: 8bit X-Priority: 3 (Normal) Importance: Normal Subject: Setting up development environment X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: dmw@unete.cl List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 May 2007 21:17:17 -0000 Hello, Is there any official way to setup a development environment for FreeBSD. I mean, I want to contribute with FreeBSD development. All I know that there is a Developer's Handbook, but what about setting a development environment for FreeBSD-CURRENT and -STABLE including from official c-mode-hooks and c++-mode-hooks for emacs to environment variables for cross-compiling the FreeBSD source. Can anyone guide me, or send me tips to get an optimal chance to contribute with FreeBSD?. Best regards, -- .O. | Daniel Molina Wegener | C/C++ Developer ..O | dmw [at] unete [dot] cl | FOSS Coding Adict OOO | BSD & Linux User | Standards Rocks! From owner-freebsd-hackers@FreeBSD.ORG Tue May 29 21:53:17 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 2E27716A469 for ; Tue, 29 May 2007 21:53:17 +0000 (UTC) (envelope-from dougb@FreeBSD.org) Received: from mail2.fluidhosting.com (mx24.fluidhosting.com [204.14.89.7]) by mx1.freebsd.org (Postfix) with SMTP id C286813C45D for ; Tue, 29 May 2007 21:53:16 +0000 (UTC) (envelope-from dougb@FreeBSD.org) Received: (qmail 9781 invoked by uid 399); 29 May 2007 21:53:16 -0000 Received: from localhost (HELO ?192.168.0.5?) (dougb@dougbarton.us@127.0.0.1) by localhost with SMTP; 29 May 2007 21:53:16 -0000 X-Originating-IP: 127.0.0.1 Message-ID: <465CA0C8.6020706@FreeBSD.org> Date: Tue, 29 May 2007 14:53:12 -0700 From: Doug Barton Organization: http://www.FreeBSD.org/ User-Agent: Thunderbird 2.0.0.0 (Windows/20070326) MIME-Version: 1.0 To: Alex Dupre References: <440b3e930705282241u44fa03a8gb0fedbb1617518df@mail.gmail.com> <465BC3A4.7030008@FreeBSD.org> <465BC507.1060902@FreeBSD.org> <465BD140.20504@FreeBSD.org> <465BD7F5.8030109@FreeBSD.org> In-Reply-To: <465BD7F5.8030109@FreeBSD.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-hackers@freebsd.org, Ali Mashtizadeh , freebsd-current@freebsd.org Subject: Re: Xorg port problem? X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 May 2007 21:53:17 -0000 Alex Dupre wrote: > Doug Barton ha scritto: >> Not disputing your answer, but I'm curious. Why would it cause >> problems on some systems but not others? I haven't done anything with >> my cflags ... > > It depends on how much RAM + Swap do you have. I know people with 1.5Gb > that have such problem and others with 2.5Gb that haven't it (if I > remember correctly the amounts). The "bug" is the un-optimized > optimization :-) There is already a patch that will be included in 4.2.1 > release, but I hope will be backported before in FreeBSD gcc. Ok, this makes sense. I was interested in the original post because I also have 2G of ram, and very little swap configured, so it looked to me like the only variable was that he had ZFS and I didn't. Doug -- This .signature sanitized for your protection From owner-freebsd-hackers@FreeBSD.ORG Tue May 29 21:58:46 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 49D2E16A41F for ; Tue, 29 May 2007 21:58:46 +0000 (UTC) (envelope-from stsp@stsp.name) Received: from hirscho.in-berlin.de (fallback-mx.in-berlin.de [192.109.42.17]) by mx1.freebsd.org (Postfix) with ESMTP id CD45113C480 for ; Tue, 29 May 2007 21:58:45 +0000 (UTC) (envelope-from stsp@stsp.name) Received: from einhorn.in-berlin.de (einhorn.in-berlin.de [192.109.42.8]) by fallback-mx.in-berlin.de (8.13.6/8.13.6/Debian-1) with ESMTP id l4TLaLUQ001899 for ; Tue, 29 May 2007 23:36:21 +0200 X-Envelope-From: stsp@stsp.name X-Envelope-To: Received: from stsp.lan (stsp.in-vpn.de [217.197.85.95]) (authenticated bits=128) by einhorn.in-berlin.de (8.13.6/8.13.6/Debian-1) with ESMTP id l4TLaI5j023279 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NOT) for ; Tue, 29 May 2007 23:36:19 +0200 Received: from ted.stsp.lan (localhost [127.0.0.1]) by stsp.lan (8.13.8/8.13.8) with ESMTP id l4TLZoMp060033 for ; Tue, 29 May 2007 23:35:50 +0200 (CEST) (envelope-from stsp@stsp.name) Received: (from stsp@localhost) by ted.stsp.lan (8.13.8/8.13.8/Submit) id l4TLZnEE060032 for freebsd-hackers@freebsd.org; Tue, 29 May 2007 23:35:49 +0200 (CEST) (envelope-from stsp@stsp.name) X-Authentication-Warning: ted.stsp.lan: stsp set sender to stsp@stsp.name using -f Date: Tue, 29 May 2007 23:35:49 +0200 From: Stefan Sperling To: freebsd-hackers@freebsd.org Message-ID: <20070529213548.GA20565@ted.stsp.lan> Mail-Followup-To: freebsd-hackers@freebsd.org References: <31ED92D6-BE85-4BE6-A27D-F7A59E9C3656@aida.org> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="AhhlLboLdkugWU4S" Content-Disposition: inline In-Reply-To: <31ED92D6-BE85-4BE6-A27D-F7A59E9C3656@aida.org> User-Agent: Mutt/1.5.15 (2007-04-06) X-Scanned-By: MIMEDefang_at_IN-Berlin_e.V. on 192.109.42.8 Subject: Re: Freebsd 6.2 panic X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 May 2007 21:58:46 -0000 --AhhlLboLdkugWU4S Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Tue, May 29, 2007 at 10:56:40PM +0200, didier derny wrote: > I have some machines running php/mysql/postfix > from time to time the computers running FreeBSD 6,2 > crashes with the following messages >=20 > fatal trap 12:page fault while in kernel mode > cpuid=3D0; apic id=3D00 > fault virtual address =3D 0x104 > fault code =3D supervisor read, page not present > instruction pointer=3D 0x20:0xc066c731 > stack pointer=3D0x28:0xe74fec90 > frame pointer=3D0x28:0xe74fec9c > code segment=3Dbase 0x0,limit 0xfffff,type 0x1b > =3D DPL 0, pres 1, def32 1, gran 1 > processor eflag=3Dresume,IOPL=3D0 Is it always the exact same panic or is it random? A box here had a few random "trap 12 page fault" panics recently, especially while booting up cold but also at run time, and the problem turned out to be bad capacitors on the motherboard. If you have physical access look at the capacitors on the board to see if they are bulged of even leak. > cannot dump. no dump device defined You might want to configure dumping, see dumpon(8). Look at the traces you get from the dumps: kgdb /boot/kernel/kernel bt It helps to run a kernel compiled with DEBUG=3D-g. I think this is default in GENERIC on 6.2. If the crashes are at seemingly random points in the kernel code you probably have a hardware problem. If it's always the same trace then post it. --=20 stefan http://stsp.name PGP Key: 0xF59D25F0 --AhhlLboLdkugWU4S Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (FreeBSD) iD8DBQFGXJy05dMCc/WdJfARAgLeAKC2VMWzzoAIOCyTPTn+pDrSTerobACffUA+ 3XF4weUZrpsCKc89glv+xJ4= =Q2Nt -----END PGP SIGNATURE----- --AhhlLboLdkugWU4S-- From owner-freebsd-hackers@FreeBSD.ORG Wed May 30 05:05:33 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id C568516A469 for ; Wed, 30 May 2007 05:05:33 +0000 (UTC) (envelope-from rea-fbsd@codelabs.ru) Received: from pobox.codelabs.ru (pobox.codelabs.ru [144.206.177.45]) by mx1.freebsd.org (Postfix) with ESMTP id 7186F13C4BF for ; Wed, 30 May 2007 05:05:33 +0000 (UTC) (envelope-from rea-fbsd@codelabs.ru) DomainKey-Signature: a=rsa-sha1; q=dns; c=simple; s=one; d=codelabs.ru; h=Received:Date:From:To:Cc:Message-ID:References:MIME-Version:Content-Type:Content-Disposition:In-Reply-To:Sender:X-Spam-Status:Subject; b=DnqvnLI9BrRjy5b/NPTBkYmFCpHTXT+Aqy9/53cX+T/GFpX3NyHOnV2vJT+UU4TCKkr4fZCAy7z4yE49BHrDG53emDr0r+TjBRzxluaeobNSN043Zc1Xp4MS84vDw18j50nWxBtunEIXZzM6w41nZdbnG/VXoEhr83xlJtALulA=; Received: from void.codelabs.ru (void.codelabs.ru [144.206.177.25]) by pobox.codelabs.ru with esmtpsa (TLSv1:AES256-SHA:256) id 1HtG4g-000DTD-Rb; Wed, 30 May 2007 08:46:18 +0400 Date: Wed, 30 May 2007 08:46:14 +0400 From: Eygene Ryabinkin To: rmgls@wanadoo.fr Message-ID: <20070530044614.GX35160@void.codelabs.ru> References: <20070529181027.65AD5700009C@mwinf2454.orange.fr> MIME-Version: 1.0 Content-Type: text/plain; charset=koi8-r Content-Disposition: inline In-Reply-To: <20070529181027.65AD5700009C@mwinf2454.orange.fr> Sender: rea-fbsd@codelabs.ru X-Spam-Status: No, score=-2.9 required=4.0 tests=ALL_TRUSTED,AWL,BAYES_00 Cc: freebsd-hackers@freebsd.org Subject: Re: direct I/O access X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 30 May 2007 05:05:33 -0000 Raoul, good day. Tue, May 29, 2007 at 08:10:26PM +0200, rmgls@wanadoo.fr wrote: > i am trying to port my old assembler soft for Dos to FreeBSD. > i need to write and read directly to the midi and scsi device. > when i try something like this i receive a sigbus error Seems like that the following fragment >params: .word 0x330,2,1 # midi port => enabling IO ??? >[...] > pushl params > pushl $0x3 > movl $0Xa5,%eax > int $0x80 > addl $0x08,%esp translates to the call i386_get_ioperm(0x330, 2, 1). But you should use the i386_set_ioperm(0x330, 2, 1), aren't you? You're trying to grant the IO permissions for your process? Then use i386_set_ioperm that will be equivalent to the first parameter to the 'int 0x80' to 0x04 instead 0x03. With 0x03 you're just trying to make the system to write the _current_ IO permissions starting with port 0x330 to the _addresses_ 0x02 and 0x01. And this is obviously wrong, but it should provoke the segfault error instead of sigbus. I am not sure about the 'pushl params' statement: will it push all three arguments to the stack? My GNU assembler knowledge is rather rusty and incomplete, sorry. But maybe it is the reason why you're getting the sigbus instead of 'correct' sigsegv. -- Eygene From owner-freebsd-hackers@FreeBSD.ORG Wed May 30 05:05:35 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 044E016A41F for ; Wed, 30 May 2007 05:05:35 +0000 (UTC) (envelope-from rea-fbsd@codelabs.ru) Received: from pobox.codelabs.ru (pobox.codelabs.ru [144.206.177.45]) by mx1.freebsd.org (Postfix) with ESMTP id A6D4513C480 for ; Wed, 30 May 2007 05:05:34 +0000 (UTC) (envelope-from rea-fbsd@codelabs.ru) DomainKey-Signature: a=rsa-sha1; q=dns; c=simple; s=one; d=codelabs.ru; h=Received:Date:From:To:Cc:Message-ID:References:MIME-Version:Content-Type:Content-Disposition:In-Reply-To:Sender:X-Spam-Status:Subject; b=hX9kHF4piyQLWbrqIpx+kYNE7m2QGhzRGXQgl6MRCotgIT0WcP5c2zfb+armqUrKyXj2bP689yqyN/SiTPQy+4hx5lCoz23jkLgGXVnIsRmdEcC5rpEa54OGWQfQqdd6VF+uwTY7OdsOsWUmfVMJPgM6Jlqa3WHqHG3//AKtS9I=; Received: from void.codelabs.ru (void.codelabs.ru [144.206.177.25]) by pobox.codelabs.ru with esmtpsa (TLSv1:AES256-SHA:256) id 1HtG7m-000DTV-Ux; Wed, 30 May 2007 08:49:31 +0400 Date: Wed, 30 May 2007 08:49:26 +0400 From: Eygene Ryabinkin To: rmgls@wanadoo.fr Message-ID: <20070530044926.GY35160@void.codelabs.ru> References: <20070529181027.65AD5700009C@mwinf2454.orange.fr> <20070530044614.GX35160@void.codelabs.ru> MIME-Version: 1.0 Content-Type: text/plain; charset=koi8-r Content-Disposition: inline In-Reply-To: <20070530044614.GX35160@void.codelabs.ru> Sender: rea-fbsd@codelabs.ru X-Spam-Status: No, score=-2.0 required=4.0 tests=ALL_TRUSTED,AWL,BAYES_20 Cc: freebsd-hackers@freebsd.org Subject: Re: direct I/O access X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 30 May 2007 05:05:35 -0000 Me again. Wed, May 30, 2007 at 08:46:14AM +0400, Eygene Ryabinkin wrote: > Then use i386_set_ioperm > that will be equivalent to the first parameter to the 'int 0x80' to > 0x04 instead 0x03. Hmm, that should read "set the first parameter for the 'int 0x80' to 0x04 instead of 0x03" -- it is much more understandable ;)) -- Eygene From owner-freebsd-hackers@FreeBSD.ORG Wed May 30 06:21:14 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id B669316A46C for ; Wed, 30 May 2007 06:21:14 +0000 (UTC) (envelope-from des@des.no) Received: from tim.des.no (tim.des.no [194.63.250.121]) by mx1.freebsd.org (Postfix) with ESMTP id 78BCE13C45E for ; Wed, 30 May 2007 06:21:14 +0000 (UTC) (envelope-from des@des.no) Received: from tim.des.no (localhost [127.0.0.1]) by spam.des.no (Postfix) with ESMTP id E7AEA20A6; Wed, 30 May 2007 08:21:09 +0200 (CEST) X-Spam-Tests: AWL X-Spam-Learn: disabled X-Spam-Score: 0.0/3.0 X-Spam-Checker-Version: SpamAssassin 3.1.8 (2007-02-13) on tim.des.no Received: from dwp.des.no (des.no [80.203.243.180]) by smtp.des.no (Postfix) with ESMTP id D9BBC2095; Wed, 30 May 2007 08:21:09 +0200 (CEST) Received: by dwp.des.no (Postfix, from userid 1001) id A5F6852DD; Wed, 30 May 2007 08:21:10 +0200 (CEST) From: =?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?= To: dmw@unete.cl References: <43801.64.117.137.69.1180473436.squirrel@webmail.ifxnw.cl> Date: Wed, 30 May 2007 08:21:10 +0200 In-Reply-To: <43801.64.117.137.69.1180473436.squirrel@webmail.ifxnw.cl> (Daniel Molina Wegener's message of "Tue\, 29 May 2007 17\:17\:16 -0400 \(CLT\)") Message-ID: <86abvm3k89.fsf@dwp.des.no> User-Agent: Gnus/5.110006 (No Gnus v0.6) Emacs/21.3 (berkeley-unix) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Cc: freebsd-hackers@freebsd.org Subject: Re: Setting up development environment X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 30 May 2007 06:21:14 -0000 "Daniel Molina Wegener" writes: > Is there any official way to setup a development environment for > FreeBSD. I mean, I want to contribute with FreeBSD development. All > I know that there is a Developer's Handbook, but what about setting a > development environment for FreeBSD-CURRENT and -STABLE including > from official c-mode-hooks and c++-mode-hooks for emacs to environment > variables for cross-compiling the FreeBSD source. Emacs setup (for both C and C++): (defun des-knf () (interactive) ;; Basic indent is 8 spaces (make-local-variable 'c-basic-offset) (setq c-basic-offset 8) ;; Continuation lines are indented 4 spaces (make-local-variable 'c-offsets-alist) (c-set-offset 'arglist-cont 4) (c-set-offset 'arglist-cont-nonempty 4) (c-set-offset 'statement-cont 4) ;; Labels are flush to the left (c-set-offset 'label [0]) ;; Fill column (make-local-variable 'fill-column) (setq fill-column 74)) (add-hook 'c-mode-common-hook 'des-knf) As for how to cross-build, read build(7). DES --=20 Dag-Erling Sm=C3=B8rgrav - des@des.no From owner-freebsd-hackers@FreeBSD.ORG Wed May 30 06:43:51 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 95ED816A41F for ; Wed, 30 May 2007 06:43:51 +0000 (UTC) (envelope-from mwm-keyword-freebsdhackers2.e313df@mired.org) Received: from mired.org (vpn.mired.org [66.92.153.74]) by mx1.freebsd.org (Postfix) with SMTP id 3BFF213C45A for ; Wed, 30 May 2007 06:43:51 +0000 (UTC) (envelope-from mwm-keyword-freebsdhackers2.e313df@mired.org) Received: (qmail 66337 invoked by uid 1001); 30 May 2007 06:43:07 -0000 Received: by bhuda.mired.org (tmda-sendmail, from uid 1001); Wed, 30 May 2007 02:43:07 -0400 MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <18013.7419.355416.25585@bhuda.mired.org> Date: Wed, 30 May 2007 02:43:07 -0400 To: rmgls@wanadoo.fr In-Reply-To: <20070529181027.65AD5700009C@mwinf2454.orange.fr> References: <20070529181027.65AD5700009C@mwinf2454.orange.fr> X-Mailer: VM 7.19 under Emacs 21.3.1 X-Primary-Address: mwm@mired.org X-face: "5Mnwy%?j>IIV\)A=):rjWL~NB2aH[}Yq8Z=u~vJ`"(,&SiLvbbz2W`; h9L,Yg`+vb1>RG% *h+%X^n0EZd>TM8_IB;a8F?(Fb"lw'IgCoyM.[Lg#r\ X-Delivery-Agent: TMDA/1.1.11 (Ladyburn) From: Mike Meyer Cc: freebsd-hackers@freebsd.org Subject: Re: direct I/O access X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 30 May 2007 06:43:51 -0000 In <20070529181027.65AD5700009C@mwinf2454.orange.fr>, rmgls@wanadoo.fr typed: > Sorry for cross posting, but perhaps hackers is a better list than multimedia > for this topic. Probably. > i am trying to port my old assembler soft for Dos to FreeBSD. You have my sympathy. > i need to write and read directly to the midi and scsi device. > when i try something like this i receive a sigbus error > > SORRY, i am NOT nor a C nor a FreeBSD expert!!! > all i know is Assembly language! You may have to learn some. Or at least be able to read it. And I have to ask. The hardware has changed a lot since the days of DoS, and things that worked then may cause strange results on modern hardware. Do you know modern hardware, or are you still using dos-era hardware? > i made some search in the devel handbook and did not found the solution. > What is wrong here? > Can you enlight me please? > > > Many thanks in advance and bests regards > > Raoul > rmgls@wanadeoo.fr > > ---cut--- > > .data > .align 4 > params: .word 0x330,2,1 # midi port => enabling IO ??? > > .text > .align 4 > .global _start > _start: > nop > pushl params > pushl $0x3 I believe this should be $0x4, as you want to *set* the values, not get them. You also need to open the file "/dev/io". I believe that leaving this file open for anything more than a handful of instructions would be a bad thing, but I'm not going to verify it. Best of luck, movl $0Xa5,%eax > int $0x80 > addl $0x08,%esp > movw $0x331,%dx # status register > inb %dx,%al > # ... > pushl $0 # exit > movl $0x1,%eax > int $0x80 > > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" > -- Mike Meyer http://www.mired.org/consulting.html Independent Network/Unix/Perforce consultant, email for more information. From owner-freebsd-hackers@FreeBSD.ORG Wed May 30 06:50:34 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 5109B16A421 for ; Wed, 30 May 2007 06:50:34 +0000 (UTC) (envelope-from hselasky@c2i.net) Received: from swip.net (mailfe10.swip.net [212.247.155.33]) by mx1.freebsd.org (Postfix) with ESMTP id DCA9413C44B for ; Wed, 30 May 2007 06:50:33 +0000 (UTC) (envelope-from hselasky@c2i.net) X-Cloudmark-Score: 0.000000 [] Received: from [194.248.135.20] (account mc467741@c2i.net HELO laptop.lan) by mailfe10.swip.net (CommuniGate Pro SMTP 5.1.7) with ESMTPA id 336062019; Wed, 30 May 2007 08:50:32 +0200 From: Hans Petter Selasky To: freebsd-hackers@freebsd.org Date: Wed, 30 May 2007 08:50:23 +0200 User-Agent: KMail/1.9.5 References: <20070529181027.65AD5700009C@mwinf2454.orange.fr> In-Reply-To: <20070529181027.65AD5700009C@mwinf2454.orange.fr> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200705300850.23375.hselasky@c2i.net> Cc: rmgls@wanadoo.fr Subject: Re: direct I/O access X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 30 May 2007 06:50:34 -0000 On Tuesday 29 May 2007 20:10, rmgls@wanadoo.fr wrote: > hi all, > > Sorry for cross posting, but perhaps hackers is a better list than > multimedia for this topic. > > i am trying to port my old assembler soft for Dos to FreeBSD. > i need to write and read directly to the midi and scsi device. > when i try something like this i receive a sigbus error > > SORRY, i am NOT nor a C nor a FreeBSD expert!!! > all i know is Assembly language! > > i made some search in the devel handbook and did not found the solution. > What is wrong here? > Can you enlight me please? I think you need to open /dev/io before your program can execute I/O instructions. --HPS From owner-freebsd-hackers@FreeBSD.ORG Wed May 30 08:55:07 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id EA52916A400 for ; Wed, 30 May 2007 08:55:07 +0000 (UTC) (envelope-from rmgls@orange.fr) Received: from smtp28.orange.fr (smtp28.orange.fr [80.12.242.101]) by mx1.freebsd.org (Postfix) with ESMTP id B1C4213C489 for ; Wed, 30 May 2007 08:55:07 +0000 (UTC) (envelope-from rmgls@orange.fr) Received: from me-wanadoo.net (localhost [127.0.0.1]) by mwinf2816.orange.fr (SMTP Server) with ESMTP id 6E2F4700008B; Wed, 30 May 2007 10:55:06 +0200 (CEST) Received: from orange.fr (ARouen-153-1-44-243.w90-17.abo.wanadoo.fr [90.17.227.243]) by mwinf2816.orange.fr (SMTP Server) with ESMTP id 163217000088; Wed, 30 May 2007 10:55:06 +0200 (CEST) X-ME-UUID: 20070530085506909.163217000088@mwinf2816.orange.fr From: rmgls@wanadoo.fr To: Mike Meyer Date: Wed, 30 May 2007 10:55:05 +0200 Sender: rmgls@orange.fr Message-Id: <20070530085506.163217000088@mwinf2816.orange.fr> Cc: freebsd-hackers@freebsd.org Subject: Re: direct I/O access X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 30 May 2007 08:55:08 -0000 >> i am trying to port my old assembler soft for Dos to FreeBSD. > >You have my sympathy. > Thanks, i need it, because its a big deal for me! >> i need to write and read directly to the midi and scsi device. >> when i try something like this i receive a sigbus error >> >> SORRY, i am NOT nor a C nor a FreeBSD expert!!! >> all i know is Assembly language! > >You may have to learn some. Or at least be able to read it. > I read C, and begin to write some short examples. but of course my lack is great on this topic. >And I have to ask. The hardware has changed a lot since the days of >DoS, and things that worked then may cause strange results on modern >hardware. Do you know modern hardware, or are you still using dos-era >hardware? > yes i am aware of this, and have a few understanding of protected mode too, also about music softs. ---cut--- >> >> .data >> .align 4 >> params: .word 0x330,2,1 # midi port => enabling IO ??? >> >> .text >> .align 4 >> .global _start >> _start: >> nop >> pushl params >> pushl $0x4 > >I believe this should be $0x4, as you want to *set* the values, not >get them. > is it necessary to restore the permissions on exit? >You also need to open the file "/dev/io". I believe that leaving this >file open for anything more than a handful of instructions would be a >bad thing, but I'm not going to verify it. > i tried this, but without set_ioperm, it was not clear on the doc that one have to use these two funcions: in io man: i find SEE ALSO: i386_get(set)_ioperm... is my understanding correct? thanks a lot raoul From owner-freebsd-hackers@FreeBSD.ORG Wed May 30 10:45:36 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id EC55E16A47A for ; Wed, 30 May 2007 10:45:36 +0000 (UTC) (envelope-from luiz@aonet.com.br) Received: from aonet.com.br (mail.aonet.com.br [200.148.248.194]) by mx1.freebsd.org (Postfix) with ESMTP id 9401513C4B7 for ; Wed, 30 May 2007 10:45:35 +0000 (UTC) (envelope-from luiz@aonet.com.br) Received: (qmail 97760 invoked by uid 98); 30 May 2007 10:15:43 -0000 Received: from 200.171.11.153 by mail.aonet.com.br (envelope-from , uid 89) with qmail-scanner-1.25 (uvscan: v4.4.00/v4710. Clear:RC:0(200.171.11.153):. Processed in 0.906116 secs); 30 May 2007 10:15:43 -0000 X-Qmail-Scanner-Mail-From: luiz@aonet.com.br via mail.aonet.com.br X-Qmail-Scanner: 1.25 (Clear:RC:0(200.171.11.153):. Processed in 0.906116 secs) Received: from unknown (HELO ?192.168.0.32?) (luiz@aonet.com.br@200.171.11.153) by aonet.com.br with AES256-SHA encrypted SMTP; 30 May 2007 10:15:42 -0000 Message-ID: <465D4F95.1030004@aonet.com.br> Date: Wed, 30 May 2007 07:19:01 -0300 From: Luiz Otavio Souza User-Agent: Thunderbird 1.5.0.10 (Windows/20070221) MIME-Version: 1.0 To: dmw@unete.cl References: <43801.64.117.137.69.1180473436.squirrel@webmail.ifxnw.cl> In-Reply-To: <43801.64.117.137.69.1180473436.squirrel@webmail.ifxnw.cl> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-hackers@freebsd.org Subject: Re: Setting up development environment X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 30 May 2007 10:45:37 -0000 Daniel Molina Wegener escreveu: > Hello, > > Is there any official way to setup a development environment for > FreeBSD. I mean, I want to contribute with FreeBSD development. All > I know that there is a Developer's Handbook, but what about setting a > development environment for FreeBSD-CURRENT and -STABLE including > from official c-mode-hooks and c++-mode-hooks for emacs to environment > variables for cross-compiling the FreeBSD source. > > Can anyone guide me, or send me tips to get an optimal chance to > contribute with FreeBSD?. > > Best regards, > Check the development(7) man page (and send good work, good ideias to the project). luiz From owner-freebsd-hackers@FreeBSD.ORG Wed May 30 12:55:15 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 211D716A46E for ; Wed, 30 May 2007 12:55:15 +0000 (UTC) (envelope-from mathieu.prevot@gmail.com) Received: from mu-out-0910.google.com (mu-out-0910.google.com [209.85.134.184]) by mx1.freebsd.org (Postfix) with ESMTP id A05E313C469 for ; Wed, 30 May 2007 12:55:14 +0000 (UTC) (envelope-from mathieu.prevot@gmail.com) Received: by mu-out-0910.google.com with SMTP id w9so1808059mue for ; Wed, 30 May 2007 05:55:13 -0700 (PDT) DKIM-Signature: a=rsa-sha1; c=relaxed/relaxed; d=gmail.com; s=beta; h=domainkey-signature:received:received:message-id:date:from:sender:to:subject:cc:mime-version:content-type:content-transfer-encoding:content-disposition:x-google-sender-auth; b=Wt8enHDexDvij2gjfnnFz6tXaAHf4bRzxxJtP4AkEW7ukTk5LCFFjYQRCWYnHE8f0YzLsdfp+1EJR5H71TUXChg5nmhklbfzMu6NREr9eh0Mho+tBqMbUbQGmN0qnHr7k6pI+pMXmTYzKUEkBhR22Joto6uHVnv0rV1cpJl0dZs= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=beta; h=received:message-id:date:from:sender:to:subject:cc:mime-version:content-type:content-transfer-encoding:content-disposition:x-google-sender-auth; b=kEcDI1dQKHDMqfXkFiEMItABFQmpXst7tpdxbiUWct+iVJ//Io2x5OdaNXEil6DyhrxF3NkTF9ASgnTMUcHFYr0fOJoDNHXkmNU+ML+rdoOZ9eeANk/Yq4eZe2Vm4heTUTKNi9gq6VCbv7ISFKii5qQwjUsHqUhnvO4utQzP8M0= Received: by 10.82.180.17 with SMTP id c17mr13869600buf.1180528055667; Wed, 30 May 2007 05:27:35 -0700 (PDT) Received: by 10.82.119.14 with HTTP; Wed, 30 May 2007 05:27:35 -0700 (PDT) Message-ID: <3e473cc60705300527g1b38a6beg6b7e136a1493b0d9@mail.gmail.com> Date: Wed, 30 May 2007 14:27:35 +0200 From: "Mathieu Prevot" Sender: mathieu.prevot@gmail.com To: "Joseph Koshy" MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Content-Disposition: inline X-Google-Sender-Auth: 99e2e02386651464 Cc: freebsd-hackers@freebsd.org, FreeBSD SoC admins Subject: SoC / PMC X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 30 May 2007 12:55:15 -0000 2007/5/25, Joseph Koshy : > Do you intend to run FreeBSD under Parallels? If so you > may need to check if Parallels emulates PMC hardware > correctly. The other emulators do not. You may need a > regular i386 or amd64 PC to run FreeBSD on 'bare metal'. I compiled a SMP (seems not to detect 2 CPUs) kernel on a black macbook (Intel Core 2 Duo) under parallels (Build 3188), with 'options HWPMC_HOOKS'. After reboot, I `kldload hwpmc` and obtain: pmc: Unknown Intel CPU. module_register_init: MOD_LOAD (hwpmc, 0xc05b0114, 0xc3662094) error 78 It seems, the module can't achieve initalization.. please help ! Mathieu From owner-freebsd-hackers@FreeBSD.ORG Wed May 30 13:27:28 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id CFBE416A41F for ; Wed, 30 May 2007 13:27:28 +0000 (UTC) (envelope-from redcrash@gmail.com) Received: from an-out-0708.google.com (an-out-0708.google.com [209.85.132.246]) by mx1.freebsd.org (Postfix) with ESMTP id E987B13C487 for ; Wed, 30 May 2007 13:27:24 +0000 (UTC) (envelope-from redcrash@gmail.com) Received: by an-out-0708.google.com with SMTP id c14so808730anc for ; Wed, 30 May 2007 06:27:23 -0700 (PDT) DKIM-Signature: a=rsa-sha1; c=relaxed/relaxed; d=gmail.com; s=beta; h=domainkey-signature:received:received:message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:references; b=JBJakTdWR35+VgaMKnm6lMLYpno8CL9I3BtSv30VNaziDo1DsS5Y45gvZ8w1tMzcHhvQQ2hOIP5p/j9O8xUWfwMbv/iq2RVeFzg1QIgGr+u+dA9cHlm7WOJschpKzqxr/Q73NDeDAKeF+ue/jHXMJ03rmKtVtkWxEs4eFIGEDb8= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=beta; h=received:message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:references; b=op2W3HiATylwiuokNafIOkyUG6RbjAFL00L+8HniIK6Fi1hj6QPHy9yIUlKlOvB4SNJoKZuCbV3nERkxOIyK8krFzpUmZCFSEDcI+OG8wMjWD+T2faLScMOPOgfgmEQTg5nikS5sofkiDlyx/AbLHuOZZ4WkaNuu+GFyn4FCBMs= Received: by 10.100.226.15 with SMTP id y15mr5937306ang.1180531235939; Wed, 30 May 2007 06:20:35 -0700 (PDT) Received: by 10.100.32.4 with HTTP; Wed, 30 May 2007 06:20:35 -0700 (PDT) Message-ID: Date: Wed, 30 May 2007 15:20:35 +0200 From: "Harald Servat" To: "Mathieu Prevot" In-Reply-To: <3e473cc60705300527g1b38a6beg6b7e136a1493b0d9@mail.gmail.com> MIME-Version: 1.0 References: <3e473cc60705300527g1b38a6beg6b7e136a1493b0d9@mail.gmail.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Content-Disposition: inline X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: Joseph Koshy , FreeBSD SoC admins , freebsd-hackers@freebsd.org Subject: Re: SoC / PMC X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 30 May 2007 13:27:29 -0000 2007/5/30, Mathieu Prevot : > > 2007/5/25, Joseph Koshy : > > Do you intend to run FreeBSD under Parallels? If so you > > may need to check if Parallels emulates PMC hardware > > correctly. The other emulators do not. You may need a > > regular i386 or amd64 PC to run FreeBSD on 'bare metal'. > > I compiled a SMP (seems not to detect 2 CPUs) kernel on a black > macbook (Intel Core 2 Duo) under parallels (Build 3188), with 'options > HWPMC_HOOKS'. After reboot, I `kldload hwpmc` and obtain: > > pmc: Unknown Intel CPU. > module_register_init: MOD_LOAD (hwpmc, 0xc05b0114, 0xc3662094) error 78 > > It seems, the module can't achieve initalization.. please help ! Mathieu, what FreeBSD version are you running? I know FreeBSD 6.2's hwpmc module does not support Intel Core 2 Duo processors. I don't know if Joseph has done any advance on this, but I'm pretty sure that http://wiki.freebsd.org/PmcTools/PmcHardware would state this if this had happened. Regards, Mathieu > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" > -- _________________________________________________________________ Empty your memory, with a free()... like a pointer! If you cast a pointer to an integer, it becomes an integer, if you cast a pointer to a struct, it becomes a struct. The pointer can crash..., and can overflow. Be a pointer my friend... From owner-freebsd-hackers@FreeBSD.ORG Wed May 30 14:01:55 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 3495816A400; Wed, 30 May 2007 14:01:55 +0000 (UTC) (envelope-from schwabe@uni-paderborn.de) Received: from mail.uni-paderborn.de (mail.uni-paderborn.de [131.234.142.9]) by mx1.freebsd.org (Postfix) with ESMTP id E1DA413C45E; Wed, 30 May 2007 14:01:54 +0000 (UTC) (envelope-from schwabe@uni-paderborn.de) Received: from isla-de-muerta.cs.uni-paderborn.de ([131.234.12.51]) by mail.uni-paderborn.de with esmtpsa (TLS-1.0:DHE_RSA_AES_256_CBC_SHA:32) (Exim 4.62 omicron-persei-viii) id 1HtOLd-0006tA-8E; Wed, 30 May 2007 15:36:21 +0200 Message-ID: <465D7DD5.4000802@uni-paderborn.de> Date: Wed, 30 May 2007 15:36:21 +0200 From: Arne Schwabe User-Agent: Mozilla/5.0 (X11; U; Linux i686; de; rv:1.8.0.8) Gecko/20060911 SUSE/1.5.0.8-3 Thunderbird/1.5.0.8 Mnenhy/0.7.5.0 MIME-Version: 1.0 To: Mathieu Prevot References: <3e473cc60705300527g1b38a6beg6b7e136a1493b0d9@mail.gmail.com> In-Reply-To: <3e473cc60705300527g1b38a6beg6b7e136a1493b0d9@mail.gmail.com> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-IMT-Authenticated-Sender: schwabe@UNI-PADERBORN.DE X-Mailman-Approved-At: Wed, 30 May 2007 14:20:20 +0000 Cc: Joseph Koshy , FreeBSD SoC admins , freebsd-hackers@freebsd.org Subject: Re: SoC / PMC X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 30 May 2007 14:01:55 -0000 Mathieu Prevot schrieb: > 2007/5/25, Joseph Koshy : >> Do you intend to run FreeBSD under Parallels? If so you >> may need to check if Parallels emulates PMC hardware >> correctly. The other emulators do not. You may need a >> regular i386 or amd64 PC to run FreeBSD on 'bare metal'. > > I compiled a SMP (seems not to detect 2 CPUs) kernel on a black > macbook (Intel Core 2 Duo) under parallels (Build 3188), with 'options > HWPMC_HOOKS'. After reboot, I `kldload hwpmc` and obtain: > Parallels does not support SMP. Arne From owner-freebsd-hackers@FreeBSD.ORG Wed May 30 16:39:54 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 2F54C16A41F for ; Wed, 30 May 2007 16:39:54 +0000 (UTC) (envelope-from mwm-keyword-freebsdhackers2.e313df@mired.org) Received: from mired.org (vpn.mired.org [66.92.153.74]) by mx1.freebsd.org (Postfix) with SMTP id C5EB613C4BF for ; Wed, 30 May 2007 16:39:53 +0000 (UTC) (envelope-from mwm-keyword-freebsdhackers2.e313df@mired.org) Received: (qmail 72312 invoked by uid 1001); 30 May 2007 16:39:10 -0000 Received: by bhuda.mired.org (tmda-sendmail, from uid 1001); Wed, 30 May 2007 12:39:10 -0400 MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <18013.43182.474113.794494@bhuda.mired.org> Date: Wed, 30 May 2007 12:39:10 -0400 To: rmgls@wanadoo.fr In-Reply-To: <20070530085506.163217000088@mwinf2816.orange.fr> References: <20070530085506.163217000088@mwinf2816.orange.fr> X-Mailer: VM 7.19 under Emacs 21.3.1 X-Primary-Address: mwm@mired.org X-face: "5Mnwy%?j>IIV\)A=):rjWL~NB2aH[}Yq8Z=u~vJ`"(,&SiLvbbz2W`; h9L,Yg`+vb1>RG% *h+%X^n0EZd>TM8_IB;a8F?(Fb"lw'IgCoyM.[Lg#r\ X-Delivery-Agent: TMDA/1.1.11 (Ladyburn) From: Mike Meyer Cc: freebsd-hackers@freebsd.org Subject: Re: direct I/O access X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 30 May 2007 16:39:54 -0000 In <20070530085506.163217000088@mwinf2816.orange.fr>, rmgls@wanadoo.fr typed: > >And I have to ask. The hardware has changed a lot since the days of > >DoS, and things that worked then may cause strange results on modern > >hardware. Do you know modern hardware, or are you still using dos-era > >hardware? > > > yes i am aware of this, and have a few understanding of protected mode too, > also about music softs. Actually, protected mode is just the beginnings of it. I've never done much x86 assembly, but going from the '020 to the '030 (or maybe it was the '010 to the '020). I had to start invalidating the hardware caches after certain operations. After that, I switched to RISC hardware, which were designed assuming that the real people writing assembler would be few and far between, and did crazy things like rearrange the instruction sequence behind your back and add extra instructions. Modern systems do this kind of stuff as well. > ---cut--- > >> > >> .data > >> .align 4 > >> params: .word 0x330,2,1 # midi port => enabling IO ??? > >> > >> .text > >> .align 4 > >> .global _start > >> _start: > >> nop > >> pushl params > >> pushl $0x4 > > > >I believe this should be $0x4, as you want to *set* the values, not > >get them. > > > is it necessary to restore the permissions on exit? > > >You also need to open the file "/dev/io". I believe that leaving this > >file open for anything more than a handful of instructions would be a > >bad thing, but I'm not going to verify it. > > > > i tried this, but without set_ioperm, it was not clear on the doc > that one have to use these two funcions: > in io man: i find SEE ALSO: i386_get(set)_ioperm... > > is my understanding correct? Sounds like it. You have to do i386_set_ioperm to say what port range you want access to, and open /dev/io to actually allow it to work. It doesn't look like it's order dependent, so doing the i386_set_ioperm and then opening/closing /dev/io should work. I no longer have root on a machine running i386 FreeBSD, so I can't test it very easily. This really is a kludge, though. You haven't said what you're trying to do. If you're trying to keep an old one-of device working, this is probably the best way. But if it's a real device that other people might be using, then writing a real device driver, or seeing if you can make the device work with something like the iic drivers might be better. http://www.mired.org/consulting.html Independent Network/Unix/Perforce consultant, email for more information. From owner-freebsd-hackers@FreeBSD.ORG Wed May 30 17:22:53 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 8C09816A421 for ; Wed, 30 May 2007 17:22:53 +0000 (UTC) (envelope-from imp@bsdimp.com) Received: from harmony.bsdimp.com (bsdimp.com [199.45.160.85]) by mx1.freebsd.org (Postfix) with ESMTP id 35B0013C46A for ; Wed, 30 May 2007 17:22:53 +0000 (UTC) (envelope-from imp@bsdimp.com) Received: from localhost (localhost [127.0.0.1]) by harmony.bsdimp.com (8.13.8/8.13.4) with ESMTP id l4UHMJXN053187; Wed, 30 May 2007 11:22:19 -0600 (MDT) (envelope-from imp@bsdimp.com) Date: Wed, 30 May 2007 11:22:36 -0600 (MDT) Message-Id: <20070530.112236.-1827345249.imp@bsdimp.com> To: rmgls@wanadoo.fr From: "M. Warner Losh" In-Reply-To: <20070529181027.65AD5700009C@mwinf2454.orange.fr> References: <20070529181027.65AD5700009C@mwinf2454.orange.fr> X-Mailer: Mew version 4.2 on Emacs 21.3 / Mule 5.0 (SAKAKI) Mime-Version: 1.0 Content-Type: Text/Plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-2.0 (harmony.bsdimp.com [127.0.0.1]); Wed, 30 May 2007 11:22:20 -0600 (MDT) Cc: freebsd-hackers@freebsd.org Subject: Re: direct I/O access X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 30 May 2007 17:22:53 -0000 Open "/dev/io" and you should be able to do what you want. How you do this in assembler, I'll leave to others. Warner From owner-freebsd-hackers@FreeBSD.ORG Wed May 30 18:25:07 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id BF6BC16A41F for ; Wed, 30 May 2007 18:25:07 +0000 (UTC) (envelope-from patrick_dkt@yahoo.com.hk) Received: from web54308.mail.yahoo.com (web54308.mail.yahoo.com [206.190.49.118]) by mx1.freebsd.org (Postfix) with SMTP id C695013C480 for ; Wed, 30 May 2007 18:25:06 +0000 (UTC) (envelope-from patrick_dkt@yahoo.com.hk) Received: (qmail 74781 invoked by uid 60001); 30 May 2007 17:58:26 -0000 DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com.hk; h=Message-ID:X-YMail-OSG:Received:Date:From:Subject:To:Cc:MIME-Version:Content-Type:Content-Transfer-Encoding; b=kqSbLZF2QAdWqWt7ht3s9kXIYphx8OaB7J5pmpmxhpWxz8PaRWwttAt896iMmQbRAl74hKSda76LXNuO6QiRhVdkECTxijphMXI5C6pU+SQgWMe4KZTD1nVV8ZXUA1y2Mb3lyeGtBUoLxO2JEyU1eNUr9ediuGAodzV+83RyqYQ= ; Message-ID: <20070530175826.74779.qmail@web54308.mail.yahoo.com> X-YMail-OSG: XQzz8jgVM1kUx_juuvFOvXFcPWClJueoDTt.JTyYVQA3ZVrazaIjpIn.n_2jyiGHWzPYejHbIUyofcg3aDkjwlEJNT1QwOPiz7KgJwEKc41aR04KpL841zQQqveHZT_SZBx9_NkyTxsT.urk.NBby2bKDWtx8tJU2Rv6X65qg32A7ovCoNIi Received: from [61.15.61.52] by web54308.mail.yahoo.com via HTTP; Wed, 30 May 2007 10:58:26 PDT Date: Wed, 30 May 2007 10:58:26 -0700 (PDT) From: Patrick Dung To: freebsd-ports@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: 8bit X-Mailman-Approved-At: Wed, 30 May 2007 19:01:06 +0000 Cc: freebsd-hackers@freebsd.org Subject: can we use disk device for iscsi-target port? X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 30 May 2007 18:25:07 -0000 Hi This port is from netbsd. I have test it, It is great. I found it seems iscsi-target has to take a file instead of disk devicek (eg. /dev/da1) as the target. Below is my test result: Starting iscsi_target. Reading configuration from `/usr/local/etc/iscsi/targets' target0:rw:0.0.0.0/0 extent0:/dev/da1:0:10733223936 DISK: 1 logical units (20963328 blocks, 512 bytes/block), type iscsi fs DISK: LU 0: pid 896:disk.c:778: ***ERROR*** error reading "target0"pid 896:disk.c:895: ***ERROR*** error allocating space for "target0"pid 896:target.c:1487: ***ERROR*** device_init() failed pid 896:iscsi-target.c:150: ***ERROR*** target_init() failed It would be great if disk device can be used directly. BTW, I have not yet test netbsd with raw disk. seems someone test on openbsd and it seems working: http://www.freebsdforums.org/forums/showthread.php?t=43960 Thanks Patrick ____________________________________________________________________________________Take the Internet to Go: Yahoo!Go puts the Internet in your pocket: mail, news, photos & more. http://mobile.yahoo.com/go?refer=1GNXIC From owner-freebsd-hackers@FreeBSD.ORG Wed May 30 21:22:57 2007 Return-Path: X-Original-To: hackers@freebsd.org Delivered-To: freebsd-hackers@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id E74C516A46C; Wed, 30 May 2007 21:22:57 +0000 (UTC) (envelope-from bakul@bitblocks.com) Received: from mail.bitblocks.com (ns1.bitblocks.com [64.142.15.60]) by mx1.freebsd.org (Postfix) with ESMTP id CBE7613C489; Wed, 30 May 2007 21:22:57 +0000 (UTC) (envelope-from bakul@bitblocks.com) Received: from bitblocks.com (localhost.bitblocks.com [127.0.0.1]) by mail.bitblocks.com (Postfix) with ESMTP id 486EF5B49; Wed, 30 May 2007 13:56:08 -0700 (PDT) To: Peter Jeremy In-reply-to: Your message of "Tue, 29 May 2007 18:04:55 +1000." <20070529080455.GM1161@turion.vk2pj.dyndns.org> Date: Wed, 30 May 2007 13:56:08 -0700 From: Bakul Shah Message-Id: <20070530205608.486EF5B49@mail.bitblocks.com> Cc: ports@freebsd.org, hackers@freebsd.org Subject: Re: Looking for speed increases in "make index" and pkg_version for ports X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 30 May 2007 21:22:58 -0000 Peter Jeremy wrote: > On 2007-May-27 16:12:54 -0700, Bakul Shah wrote: > >Given the size and complexity of the port system I have long > >felt that rather than do everything via more and more complex > >Mk/*.mk what is is needed is a ports server and a thin CLI > >frontend to it. > > I don't believe this is practical. Both package names and > port dependencies depend on the options that are selected as > well as what other ports are already installed. A centralised > ports server is not going to have access to this information. I didn't mean a centralized server at freebsd.org but on your freebsd system and can know about what ports are installed. Conditional dependencies have to be dealt with. Perhaps the underlying reason for changing package names can be handled in a different way. What happens now is that mostly static information from various files is recomputed many times. While that can be handled by a local database, it seems to be a daemon provides a lot of benefits. Come to think of it, even a centralized server can work as there are a finite number of combinations and it can cache the ones in use. But all this is just an educated guess. From owner-freebsd-hackers@FreeBSD.ORG Wed May 30 22:03:16 2007 Return-Path: X-Original-To: hackers@freebsd.org Delivered-To: freebsd-hackers@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id D8CBB16A468 for ; Wed, 30 May 2007 22:03:16 +0000 (UTC) (envelope-from stephen@math.missouri.edu) Received: from math.missouri.edu (math.missouri.edu [128.206.184.200]) by mx1.freebsd.org (Postfix) with ESMTP id ADB2313C4C4 for ; Wed, 30 May 2007 22:03:16 +0000 (UTC) (envelope-from stephen@math.missouri.edu) Received: from math.missouri.edu (localhost [127.0.0.1]) by math.missouri.edu (8.13.1/8.13.1) with ESMTP id l4ULZ4Ot067877; Wed, 30 May 2007 16:35:04 -0500 (CDT) (envelope-from stephen@math.missouri.edu) Received: from localhost (stephen@localhost) by math.missouri.edu (8.13.1/8.13.1/Submit) with ESMTP id l4ULZ42X067874; Wed, 30 May 2007 16:35:04 -0500 (CDT) (envelope-from stephen@math.missouri.edu) Date: Wed, 30 May 2007 16:35:04 -0500 (CDT) From: Stephen Montgomery-Smith To: Bakul Shah In-Reply-To: <20070530205608.486EF5B49@mail.bitblocks.com> Message-ID: <20070530162949.M65262@math.missouri.edu> References: <20070530205608.486EF5B49@mail.bitblocks.com> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: ports@freebsd.org, hackers@freebsd.org Subject: Re: Looking for speed increases in "make index" and pkg_version for ports X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 30 May 2007 22:03:16 -0000 On Wed, 30 May 2007, Bakul Shah wrote: > Peter Jeremy wrote: >> On 2007-May-27 16:12:54 -0700, Bakul Shah wrote: >>> Given the size and complexity of the port system I have long >>> felt that rather than do everything via more and more complex >>> Mk/*.mk what is is needed is a ports server and a thin CLI >>> frontend to it. >> >> I don't believe this is practical. Both package names and >> port dependencies depend on the options that are selected as >> well as what other ports are already installed. A centralised >> ports server is not going to have access to this information. > > I didn't mean a centralized server at freebsd.org but on your > freebsd system and can know about what ports are installed. > Conditional dependencies have to be dealt with. Perhaps the > underlying reason for changing package names can be handled > in a different way. > > What happens now is that mostly static information from > various files is recomputed many times. While that can be > handled by a local database, it seems to be a daemon provides > a lot of benefits. > > Come to think of it, even a centralized server can work as > there are a finite number of combinations and it can cache > the ones in use. But all this is just an educated guess. Your idea really looks very fine to me. From reading other emails on this thread, I get the impression that a lot of the underlying work has already been done in perhaps the portupgrade port, and so all you would have to do is to provide an interface from the make file to the database produced by portupgrade. Perhaps this could be made conditional, so that those who don't install portupgrade wouldn't use it. Even so, I also get the feeling that to implement this would be quite some work, so a volunteer needs to step forward. But my gut reaction is that this is almost certain to make things like "make clean" and pkg_version way way faster. And I must admit that I am having more doubts about my "calculate the variables just in time" idea. The thought of working really hard to make it work, and then seeing mediocre speed increases is rather offputting to me. From owner-freebsd-hackers@FreeBSD.ORG Wed May 30 21:55:29 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id D822F16A41F; Wed, 30 May 2007 21:55:29 +0000 (UTC) (envelope-from mark@foster.cc) Received: from sccrmhc11.comcast.net (sccrmhc11.comcast.net [204.127.200.81]) by mx1.freebsd.org (Postfix) with ESMTP id 8A81813C44B; Wed, 30 May 2007 21:55:29 +0000 (UTC) (envelope-from mark@foster.cc) Received: from fosgate.dyndns.org ([71.227.137.90]) by comcast.net (sccrmhc11) with ESMTP id <2007053021452501100hvvgme>; Wed, 30 May 2007 21:45:26 +0000 Received: from localhost (localhost [127.0.0.1]) by fosgate.dyndns.org (Postfix) with ESMTP id 8350E39836; Wed, 30 May 2007 14:43:59 -0700 (PDT) X-Virus-Scanned: amavisd-new at foster.cc Received: from fosgate.dyndns.org ([127.0.0.1]) by localhost (sonar.foster.dmz [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id bTkRkkJEJ2YB; Wed, 30 May 2007 14:43:56 -0700 (PDT) Received: from [10.1.253.78] (fis-gw1.portseattle.org [198.134.96.8]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by fosgate.dyndns.org (Postfix) with ESMTP id A800739835; Wed, 30 May 2007 14:43:55 -0700 (PDT) Message-ID: <465DF06F.9090500@foster.cc> Date: Wed, 30 May 2007 14:45:19 -0700 From: Mark Foster User-Agent: Thunderbird 1.5.0.10 (X11/20070403) MIME-Version: 1.0 To: Patrick Dung References: <20070530175826.74779.qmail@web54308.mail.yahoo.com> In-Reply-To: <20070530175826.74779.qmail@web54308.mail.yahoo.com> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-Mailman-Approved-At: Wed, 30 May 2007 22:38:08 +0000 Cc: freebsd-hackers@freebsd.org, freebsd-ports@freebsd.org Subject: Re: can we use disk device for iscsi-target port? X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 30 May 2007 21:55:29 -0000 Patrick Dung wrote: > It would be great if disk device can be used directly. > > BTW, I have not yet test netbsd with raw disk. > seems someone test on openbsd and it seems working: > http://www.freebsdforums.org/forums/showthread.php?t=43960 > Whole disk or slice? I don't think either works at this point but can't say I've tried. Can you please test an individual slice and report your findings? -- Said one park ranger, 'There is considerable overlap between the intelligence of the smartest bears and the dumbest tourists.' Mark D. Foster, CISSP http://mark.foster.cc/ From owner-freebsd-hackers@FreeBSD.ORG Thu May 31 04:34:32 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id DC7EB16A421 for ; Thu, 31 May 2007 04:34:32 +0000 (UTC) (envelope-from doconnor@gsoft.com.au) Received: from cain.gsoft.com.au (cain.gsoft.com.au [203.31.81.10]) by mx1.freebsd.org (Postfix) with ESMTP id 6835113C457 for ; Thu, 31 May 2007 04:34:32 +0000 (UTC) (envelope-from doconnor@gsoft.com.au) Received: from inchoate.gsoft.com.au (inchoate.gsoft.com.au [203.31.81.61]) (authenticated bits=0) by cain.gsoft.com.au (8.13.8/8.13.8) with ESMTP id l4V49gBM090280 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Thu, 31 May 2007 13:39:42 +0930 (CST) (envelope-from doconnor@gsoft.com.au) Received: from murder ([unix socket]) by cain.gsoft.com.au (Cyrus v2.3.7) with LMTPA; Thu, 31 May 2007 12:46:47 +0930 X-Sieve: CMU Sieve 2.3 Received: from inchoate.gsoft.com.au (inchoate.gsoft.com.au [203.31.81.61]) (authenticated bits=0) by cain.gsoft.com.au (8.13.8/8.13.8) with ESMTP id l4V3GljF083848 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Thu, 31 May 2007 12:46:47 +0930 (CST) (envelope-from doconnor@gsoft.com.au) From: "Daniel O'Connor" Date: Thu, 31 May 2007 13:39:23 +0930 User-Agent: KMail/1.9.5 MIME-Version: 1.0 To: freebsd-hackers@freebsd.org Message-Id: <200705311339.37683.doconnor@gsoft.com.au> X-Spam-Score: -1.111 () BAYES_00,TVD_RCVD_SPACE_BRACKET,UNPARSEABLE_RELAY X-Scanned-By: MIMEDefang 2.58 on 203.31.81.10 X-Scanned-By: MIMEDefang 2.58 on 203.31.81.10 X-Bogosity: No, tests=bogofilter, spamicity=0.000000, version=1.1.5 X-UID: 163987 X-Length: 2848 Content-Type: multipart/signed; boundary="nextPart147574748.L9FOGiGbBa"; protocol="application/pgp-signature"; micalg=pgp-sha1 Content-Transfer-Encoding: 7bit Subject: boot0 vs XP X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 31 May 2007 04:34:32 -0000 --nextPart147574748.L9FOGiGbBa Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Content-Disposition: inline I recently reinstalled Windows XP on my laptop (I barely use it but=20 occasionally it comes in handy :) and when I did the install it made=20 the base drive E (no idea why, and I couldn't see how to change it). Everything proceeds as usual and then I boot a CD to then jump start my=20 system and run boot0cfg to re-write the boot loader. Alas the next time I boot Windows it logs me out straight after logging=20 in. I do some googling and find that it is most likely that the=20 signature bytes Windows uses to determine which drive is what have=20 changed and it has now made the drive C:. Hence the swapfile location=20 is now incorrect :( I was wondering if it would be possible to modify boot0cfg (and boot0 I=20 guess) so that it avoids touching these bytes. I found some details here ->=20 http://www.goodells.net/multiboot/partsigs.htm Basically it would appear to be the 4 bytes just before the partition=20 table. I'm not sure how big boot0 is though :) =2D-=20 Daniel O'Connor software and network engineer for Genesis Software - http://www.gsoft.com.au "The nice thing about standards is that there are so many of them to choose from." -- Andrew Tanenbaum GPG Fingerprint - 5596 B766 97C0 0E94 4347 295E E593 DC20 7B3F CE8C --nextPart147574748.L9FOGiGbBa Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.5 (FreeBSD) iD8DBQBGXkqB5ZPcIHs/zowRAgs3AKCDO0dU6nIaJ0cABv1bea+tSQ44CwCePmaG rCyFyZipLsjPhhwfAJ/Ts/k= =KUvL -----END PGP SIGNATURE----- --nextPart147574748.L9FOGiGbBa-- From owner-freebsd-hackers@FreeBSD.ORG Thu May 31 04:44:40 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 7513416A400 for ; Thu, 31 May 2007 04:44:40 +0000 (UTC) (envelope-from rea-fbsd@codelabs.ru) Received: from pobox.codelabs.ru (pobox.codelabs.ru [144.206.177.45]) by mx1.freebsd.org (Postfix) with ESMTP id 2A8EA13C4BC for ; Thu, 31 May 2007 04:44:39 +0000 (UTC) (envelope-from rea-fbsd@codelabs.ru) DomainKey-Signature: a=rsa-sha1; q=dns; c=simple; s=one; d=codelabs.ru; h=Received:Date:From:To:Cc:Message-ID:References:MIME-Version:Content-Type:Content-Disposition:In-Reply-To:Sender:X-Spam-Status:Subject; b=ZIr9fjajMr013xCUeBdrwsRwPMtUYDY66PKcMSnmCJXiBeRRcCFI2YL2G0UphkklYhjSn0NSY2J+coKd3Dd+j96vEyztpjDszYD5UuhrBLMg3r3yVncnSgkB1Z22/7sWIW9dv1oIPP565pymTGP4tQQXIAZ+tK8IhevoeHS3a1M=; Received: from void.codelabs.ru (void.codelabs.ru [144.206.177.25]) by pobox.codelabs.ru with esmtpsa (TLSv1:AES256-SHA:256) id 1HtcWb-000Etq-JN; Thu, 31 May 2007 08:44:37 +0400 Date: Thu, 31 May 2007 08:44:33 +0400 From: Eygene Ryabinkin To: Mike Meyer Message-ID: <20070531044432.GC35160@void.codelabs.ru> References: <20070529181027.65AD5700009C@mwinf2454.orange.fr> <18013.7419.355416.25585@bhuda.mired.org> MIME-Version: 1.0 Content-Type: text/plain; charset=koi8-r Content-Disposition: inline In-Reply-To: <18013.7419.355416.25585@bhuda.mired.org> Sender: rea-fbsd@codelabs.ru X-Spam-Status: No, score=-2.9 required=4.0 tests=ALL_TRUSTED,AWL,BAYES_00 Cc: rmgls@wanadoo.fr, freebsd-hackers@freebsd.org Subject: Re: direct I/O access X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 31 May 2007 04:44:40 -0000 Mike, good day. Wed, May 30, 2007 at 02:43:07AM -0400, Mike Meyer wrote: > I believe this should be $0x4, as you want to *set* the values, not > get them. Right. > You also need to open the file "/dev/io". I believe that leaving this > file open for anything more than a handful of instructions would be a > bad thing, but I'm not going to verify it. I feel that the i386_set_ioperm directly manipulates the task's I/O bitmap referenced by the task state segment (TSS), so you don't need to mangle with /dev/io. /dev/io itself is the higher-level semi machine-indenepdent abstraction. Opening /dev/io grants the global access to all ports, while using i386_set_ioperm gives the fine-grained access. When you closing /dev/io, the port I/O access is revoked. To summarise: either you open /dev/io and do all your port I/O as in the good old days of the real-mode programs, or you're using i386_set_ioperm to obtain the access permissions to the I/O port range and again, do all port I/O as usual. -- Eygene From owner-freebsd-hackers@FreeBSD.ORG Thu May 31 06:11:20 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 9874B16A41F for ; Thu, 31 May 2007 06:11:20 +0000 (UTC) (envelope-from jkoshy.freebsd@gmail.com) Received: from nz-out-0506.google.com (nz-out-0506.google.com [64.233.162.234]) by mx1.freebsd.org (Postfix) with ESMTP id 5644713C457 for ; Thu, 31 May 2007 06:11:20 +0000 (UTC) (envelope-from jkoshy.freebsd@gmail.com) Received: by nz-out-0506.google.com with SMTP id 14so51576nzn for ; Wed, 30 May 2007 23:11:19 -0700 (PDT) DKIM-Signature: a=rsa-sha1; c=relaxed/relaxed; d=gmail.com; s=beta; h=domainkey-signature:received:received:message-id:date:from:sender:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references:x-google-sender-auth; b=pbnmSurOAuEQsq7fly6RC/HfUwCaPI9W0GI1Ep8fZh/EGKK2RQgyUvrur+p3fAqr54LBiO82YlOOXGNpkhxRMnuHKWttOeSbqbLO7L9MKQKJy3ljeN6A2DE3Na5bcBKFTg+4CFwo7elv9zXRrJqr1lYa3+ayxq5KgIqZL9dEiGY= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=beta; h=received:message-id:date:from:sender:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references:x-google-sender-auth; b=TFf/fV+lSyrAtjlqNxHN+TzvocxZ1OH3XM71TDjXejg+ygJYblcWNBqt+HlI/CmNWS4c6Gf2m5eumAbFIXdfN8ZlQ9Y68vlyENzWwT6Clns/rnBDy/qlknVkPNXYI6TkWApUYV8q4FlBIRduIhukaHXFr+IkDgJJyVOEtyI5vdQ= Received: by 10.114.61.1 with SMTP id j1mr250141waa.1180590223649; Wed, 30 May 2007 22:43:43 -0700 (PDT) Received: by 10.115.92.8 with HTTP; Wed, 30 May 2007 22:43:43 -0700 (PDT) Message-ID: Date: Thu, 31 May 2007 05:43:43 +0000 From: "Joseph Koshy" Sender: jkoshy.freebsd@gmail.com To: "Mathieu Prevot" In-Reply-To: <3e473cc60705300527g1b38a6beg6b7e136a1493b0d9@mail.gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <3e473cc60705300527g1b38a6beg6b7e136a1493b0d9@mail.gmail.com> X-Google-Sender-Auth: c8dbaf644705c397 Cc: freebsd-hackers@freebsd.org, FreeBSD SoC admins Subject: Re: SoC / PMC X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 31 May 2007 06:11:20 -0000 jk> You may need a regular i386 or amd64 PC to run FreeBSD on 'bare metal'. mp> pmc: Unknown Intel CPU. mp> module_register_init: MOD_LOAD (hwpmc, 0xc05b0114, 0xc3662094) error 78 mp> It seems, the module can't achieve initalization.. please help ! When the time comes for hooking up your GUI to hwpmc, I would recommend that you use supported hardware and run FreeBSD on 'bare metal' i.e., not under emulation. -- From owner-freebsd-hackers@FreeBSD.ORG Thu May 31 08:48:00 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id A57B016A469 for ; Thu, 31 May 2007 08:48:00 +0000 (UTC) (envelope-from rmgls@orange.fr) Received: from smtp19.orange.fr (smtp19.orange.fr [80.12.242.17]) by mx1.freebsd.org (Postfix) with ESMTP id 6ED2B13C448 for ; Thu, 31 May 2007 08:48:00 +0000 (UTC) (envelope-from rmgls@orange.fr) Received: from me-wanadoo.net (localhost [127.0.0.1]) by mwinf1907.orange.fr (SMTP Server) with ESMTP id 9B6791C000A5; Thu, 31 May 2007 10:47:55 +0200 (CEST) Received: from orange.fr (ARouen-153-1-50-12.w90-17.abo.wanadoo.fr [90.17.233.12]) by mwinf1907.orange.fr (SMTP Server) with ESMTP id 816A61C0008F; Thu, 31 May 2007 10:47:55 +0200 (CEST) X-ME-UUID: 20070531084755530.816A61C0008F@mwinf1907.orange.fr From: rmgls@wanadoo.fr To: Mike Meyer Date: Thu, 31 May 2007 10:47:54 +0200 Sender: rmgls@orange.fr Message-Id: <20070531084755.816A61C0008F@mwinf1907.orange.fr> Cc: freebsd-hackers@freebsd.org Subject: Re: direct I/O access X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 31 May 2007 08:48:00 -0000 On Wed, 30 May 2007 12:39:10 -0400, Mike Meyer wrote, > Actually, protected mode is just the beginnings of it. I've never done > much x86 assembly, but going from the '020 to the '030 (or maybe it > was the '010 to the '020). I had to start invalidating the hardware > caches after certain operations. After that, I switched to RISC > hardware, which were designed assuming that the real people writing > assembler would be few and far between, and did crazy things like > rearrange the instruction sequence behind your back and add extra > instructions. Modern systems do this kind of stuff as well. do you think that it would be better to rewrite the soft in C??? in this case, i have more work to do! first of all, learning C on the finger tip, before working on my soft. and i thought that it would be very interesting to learn the heart of FreeBSD. > This really is a kludge, though. You haven't said what you're trying > to do. If you're trying to keep an old one-of device working, this is > probably the best way. But if it's a real device that other people > might be using, then writing a real device driver, or seeing if you > can make the device work with something like the iic drivers might be > better. in fact, they are: - a direct to disk recorder, - a sampler. they are not usable by other people, because they were discontinued. bests regards raoul rmgls@wanadoo.fr From owner-freebsd-hackers@FreeBSD.ORG Thu May 31 09:42:16 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id D734216A400 for ; Thu, 31 May 2007 09:42:16 +0000 (UTC) (envelope-from freebsd-hackers@m.gmane.org) Received: from ciao.gmane.org (main.gmane.org [80.91.229.2]) by mx1.freebsd.org (Postfix) with ESMTP id 5E49513C44B for ; Thu, 31 May 2007 09:42:16 +0000 (UTC) (envelope-from freebsd-hackers@m.gmane.org) Received: from list by ciao.gmane.org with local (Exim 4.43) id 1HthAc-0000G1-1U for freebsd-hackers@freebsd.org; Thu, 31 May 2007 11:42:14 +0200 Received: from lara.cc.fer.hr ([161.53.72.113]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Thu, 31 May 2007 11:42:14 +0200 Received: from ivoras by lara.cc.fer.hr with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Thu, 31 May 2007 11:42:14 +0200 X-Injected-Via-Gmane: http://gmane.org/ To: freebsd-hackers@freebsd.org From: Ivan Voras Date: Thu, 31 May 2007 11:42:04 +0200 Lines: 42 Message-ID: References: <200705311339.37683.doconnor@gsoft.com.au> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="------------enig85B7E32AF56088F97A66CA27" X-Complaints-To: usenet@sea.gmane.org X-Gmane-NNTP-Posting-Host: lara.cc.fer.hr User-Agent: Thunderbird 1.5.0.10 (X11/20060911) In-Reply-To: <200705311339.37683.doconnor@gsoft.com.au> X-Enigmail-Version: 0.94.2.0 Sender: news Subject: Re: boot0 vs XP X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 31 May 2007 09:42:16 -0000 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enig85B7E32AF56088F97A66CA27 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: quoted-printable Daniel O'Connor wrote: > I recently reinstalled Windows XP on my laptop (I barely use it but=20 > occasionally it comes in handy :) and when I did the install it made=20 > the base drive E (no idea why, and I couldn't see how to change it). This is a well-known problem (at least to those who deal with=20 dual-booting Windows and FreeBSD often) - if you (re)install Windows at=20 any point AFTER installing FreeBSD, it will complain. This is not=20 FreeBSD-specific: Windows really wants to mess around with its mappings=20 of drives ("C:", etc.) to primary partitions during SETUP (extended=20 partitions are mostly fine). I don't know the solution to this - I've always had to reinstall=20 everything when that happened, but you might try writing down the=20 parameters of your FreeBSD partition, then delete it, then reinstall=20 Windows, verify that everything's working, then restore your FreeBSD=20 partition. This is, of course, extremely risky. For me, it always works if I install Windows first, then FreeBSD later,=20 never the other way around. --------------enig85B7E32AF56088F97A66CA27 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.5 (GNU/Linux) Comment: Using GnuPG with SUSE - http://enigmail.mozdev.org iD8DBQFGXphsldnAQVacBcgRAuqGAJ9Zs1YEwoiTu3anWyN2LSi3MgeBCwCgrjgD mnu1JULmiQJOhO9nvfbc8n8= =nrJX -----END PGP SIGNATURE----- --------------enig85B7E32AF56088F97A66CA27-- From owner-freebsd-hackers@FreeBSD.ORG Thu May 31 08:04:14 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 5822B16A41F for ; Thu, 31 May 2007 08:04:14 +0000 (UTC) (envelope-from patrick_dkt@yahoo.com.hk) Received: from web54302.mail.re2.yahoo.com (web54302.mail.re2.yahoo.com [206.190.49.112]) by mx1.freebsd.org (Postfix) with SMTP id 0F65F13C4C3 for ; Thu, 31 May 2007 08:04:13 +0000 (UTC) (envelope-from patrick_dkt@yahoo.com.hk) Received: (qmail 9824 invoked by uid 60001); 31 May 2007 07:37:33 -0000 DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com.hk; h=X-YMail-OSG:Received:Date:From:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type:Content-Transfer-Encoding:Message-ID; b=xrFLue0F6LR3W6h3pEz/xpH9RSoHT5mY1PBSpmWAtMHQhrnIdOsnRdmHxOZ6Vac1yPdqgYix/Nf7leFr+Pkkef5sB4aoiCVo7eWPuZm1vvKJxwKxKNzWRgdkkMTTt4Q7hWNwCE7a5MrviNCcn6s5ucjeTFBF/FUnVinao5vIVW4=; X-YMail-OSG: nPewYasVM1kSzaCR4Wyf7ye_8vzGRGDNU_.ExCKvUMhn.RO7_BMQ8W2nPogpw0wFOVVGx5sXJr87kuP.zkJfiTgrLLh4xIPJ1JB66WeBxydhUiqG6fXKtZ1ITigfpkeuU55r1oOOrX.zmKr80.0LUSS.tjD0XMxQ11iEdlFwc65fsCylO6D0 Received: from [61.15.61.52] by web54302.mail.re2.yahoo.com via HTTP; Thu, 31 May 2007 00:37:33 PDT Date: Thu, 31 May 2007 00:37:33 -0700 (PDT) From: Patrick Dung To: Mark Foster In-Reply-To: <465DF06F.9090500@foster.cc> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: 8bit Message-ID: <786193.7469.qm@web54302.mail.re2.yahoo.com> X-Mailman-Approved-At: Thu, 31 May 2007 11:26:08 +0000 Cc: freebsd-hackers@freebsd.org, freebsd-ports@freebsd.org Subject: Re: can we use disk device for iscsi-target port? X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 31 May 2007 08:04:14 -0000 My testing env is Vmware, the backend disks are new/empty. Here's my testing result: 1. OpenBSD 4.1 with iscsi-target compiled (I can't find it in the ports) It support disk slice directly, I use /dev/sd1c. 2. NetBSD 4.0-current with iscsi-target in base OS It support disk slice directly, I use /dev/sd1d. 3. FreeBSD 6.2 with iscsi-target port: When the new harddisk is added. It is empty and do not have any individual slice (only have /dev/da1). iscsi-target cannot use /dev/da1 directly (tested yesterday) /dev/da1c is gone in FreeBSD (since 5.0).... I have tried to create a disk slice by fdisk (/dev/da1s1 and /dev/da1s1c is created). But iscsi-target cannot use them directly: Reading configuration from `./targets' target0:rw:0.0.0.0/0 extent0:/dev/da1s1:0:629145600 DISK: 1 logical units (1228800 blocks, 512 bytes/block), type iscsi fs DISK: LU 0: pid 1916:disk.c:778: ***ERROR*** error reading "target0"pid 1916:disk.c:895: ***ERROR*** error allocating space for "target0"pid 1916:target.c:1487: ***ERROR*** device_init() failed pid 1916:iscsi-target.c:150: ***ERROR*** target_init() failed Reading configuration from `./targets' target0:rw:0.0.0.0/0 extent0:/dev/da1s1c:0:629145600 DISK: 1 logical units (1228800 blocks, 512 bytes/block), type iscsi fs DISK: LU 0: pid 1919:disk.c:778: ***ERROR*** error reading "target0"pid 1919:disk.c:895: ***ERROR*** error allocating space for "target0"pid 1919:target.c:1487: ***ERROR*** device_init() failed pid 1919:iscsi-target.c:150: ***ERROR*** target_init() failed Patrick --- Mark Foster wrote: > Patrick Dung wrote: > > It would be great if disk device can be used directly. > > > > BTW, I have not yet test netbsd with raw disk. > > seems someone test on openbsd and it seems working: > > http://www.freebsdforums.org/forums/showthread.php?t=43960 > > > Whole disk or slice? I don't think either works at this point but > can't > say I've tried. Can you please test an individual slice and report > your > findings? > > -- > Said one park ranger, 'There is considerable overlap between the > intelligence of the smartest bears and the dumbest tourists.' > Mark D. Foster, CISSP http://mark.foster.cc/ > > ____________________________________________________________________________________ Be a PS3 game guru. Get your game face on with the latest PS3 news and previews at Yahoo! Games. http://videogames.yahoo.com/platform?platform=120121 From owner-freebsd-hackers@FreeBSD.ORG Thu May 31 12:36:51 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id DA52216A41F for ; Thu, 31 May 2007 12:36:51 +0000 (UTC) (envelope-from rmgls@orange.fr) Received: from smtp23.orange.fr (smtp23.orange.fr [193.252.22.30]) by mx1.freebsd.org (Postfix) with ESMTP id 7BD8613C448 for ; Thu, 31 May 2007 12:36:51 +0000 (UTC) (envelope-from rmgls@orange.fr) Received: from me-wanadoo.net (localhost [127.0.0.1]) by mwinf2332.orange.fr (SMTP Server) with ESMTP id 4E46A1C000BC; Thu, 31 May 2007 14:36:50 +0200 (CEST) Received: from orange.fr (ARouen-153-1-12-144.w90-17.abo.wanadoo.fr [90.17.155.144]) by mwinf2332.orange.fr (SMTP Server) with ESMTP id 2DB081C000AB; Thu, 31 May 2007 14:36:50 +0200 (CEST) X-ME-UUID: 20070531123650187.2DB081C000AB@mwinf2332.orange.fr To: freebsd-hackers@freebsd.org Date: Thu, 31 May 2007 14:36:48 +0200 From: Raoul MEGELAS Message-Id: <20070531123650.2DB081C000AB@mwinf2332.orange.fr> X-Mailman-Approved-At: Thu, 31 May 2007 13:06:27 +0000 Cc: Subject: Re: direct I/O access X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 31 May 2007 12:36:51 -0000 On Thu, 31 May 2007 08:44:33 +0400 Eygene Ryabinkin wrote, Wed, May 30, 2007 at 02:43:07AM -0400, Mike Meyer wrote: >> I believe this should be $0x4, as you want to *set* the values, not >> get them. > Right. >> You also need to open the file "/dev/io". I believe that leaving this >> file open for anything more than a handful of instructions would be a >> bad thing, but I'm not going to verify it. > I feel that the i386_set_ioperm directly manipulates the task's I/O > bitmap referenced by the task state segment (TSS), so you don't > need to mangle with /dev/io. /dev/io itself is the higher-level > semi machine-indenepdent abstraction. Opening /dev/io grants the > global access to all ports, while using i386_set_ioperm gives the > fine-grained access. When you closing /dev/io, the port I/O access > is revoked. > To summarise: either you open /dev/io and do all your port I/O as > in the good old days of the real-mode programs, or you're using > i386_set_ioperm to obtain the access permissions to the I/O port > range and again, do all port I/O as usual. i rewrote my code and this time, all works fine. thanks to Mike and Eygene. here is the code as a proof of the above explanation: # test_io.S .include "system.inc" BUFSIZE = 0x80 .data params: .long 0x70,2,1,0 path: .asciz "/dev/io" .bss buffer: .fill BUFSIZE, 0 .text .global _start err: pushl $0x1 # return failure SysExit _start: addl $8,%esp # discard argc and argv[0] leal params,%eax pushl %eax pushl $4 movl $0xa5,%eax call bsd_kernel addl $12,%esp movl $0X330,%edx inb %dx,%al # if i try to read or write another port than 0x70/0x71, i have a segbus fault. leal path,%ecx pushl $0 # O_RDONLY pushl %ecx SysOpen jc err # open failed addl $8,%esp movl $0X330,%edx inb %dx,%al # all is ok! pushl $0 # return success SysExit Thanks again. Raoul rmgls@wanadoo.fr From owner-freebsd-hackers@FreeBSD.ORG Thu May 31 13:23:20 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 9B92216A468 for ; Thu, 31 May 2007 13:23:20 +0000 (UTC) (envelope-from mwm-keyword-freebsdhackers2.e313df@mired.org) Received: from mired.org (vpn.mired.org [66.92.153.74]) by mx1.freebsd.org (Postfix) with SMTP id 454B213C44C for ; Thu, 31 May 2007 13:23:19 +0000 (UTC) (envelope-from mwm-keyword-freebsdhackers2.e313df@mired.org) Received: (qmail 92843 invoked by uid 1001); 31 May 2007 13:22:38 -0000 Received: by bhuda.mired.org (tmda-sendmail, from uid 1001); Thu, 31 May 2007 09:22:38 -0400 MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <18014.52254.333433.704250@bhuda.mired.org> Date: Thu, 31 May 2007 09:22:38 -0400 To: rmgls@wanadoo.fr In-Reply-To: <20070531084755.816A61C0008F@mwinf1907.orange.fr> References: <20070531084755.816A61C0008F@mwinf1907.orange.fr> X-Mailer: VM 7.19 under Emacs 21.3.1 X-Primary-Address: mwm@mired.org X-face: "5Mnwy%?j>IIV\)A=):rjWL~NB2aH[}Yq8Z=u~vJ`"(,&SiLvbbz2W`; h9L,Yg`+vb1>RG% *h+%X^n0EZd>TM8_IB;a8F?(Fb"lw'IgCoyM.[Lg#r\ X-Delivery-Agent: TMDA/1.1.11 (Ladyburn) From: Mike Meyer Cc: freebsd-hackers@freebsd.org Subject: Re: direct I/O access X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 31 May 2007 13:23:20 -0000 In <20070531084755.816A61C0008F@mwinf1907.orange.fr>, rmgls@wanadoo.fr typed: > On Wed, 30 May 2007 12:39:10 -0400, Mike Meyer wrote, > > > Actually, protected mode is just the beginnings of it. I've never done > > much x86 assembly, but going from the '020 to the '030 (or maybe it > > was the '010 to the '020). I had to start invalidating the hardware > > caches after certain operations. After that, I switched to RISC > > hardware, which were designed assuming that the real people writing > > assembler would be few and far between, and did crazy things like > > rearrange the instruction sequence behind your back and add extra > > instructions. Modern systems do this kind of stuff as well. > > do you think that it would be better to rewrite the soft in C??? > in this case, i have more work to do! > first of all, learning C on the finger tip, before working on my soft. > > and i thought that it would be very interesting to learn > the heart of FreeBSD. Well, if you want to learn "the heart" of FreeBSD, I think you'd be better off working in C. The code that isn't part of FreeBSD on pretty much every platform would seem to be disqualified as the heart. > > This really is a kludge, though. You haven't said what you're trying > > to do. If you're trying to keep an old one-of device working, this is > > probably the best way. But if it's a real device that other people > > might be using, then writing a real device driver, or seeing if you > > can make the device work with something like the iic drivers might be > > better. > > in fact, they are: > - a direct to disk recorder, > - a sampler. > > they are not usable by other people, because they were discontinued. So nobody else has them? You can't pick one up used on eBay? I personally tend to run all but my most critical servers on "discontinued" hardware. It's fast enough to do the job running FreeBSD, and reasonably reliable; it's just not fast enough to run modern versions of Windows. And I can buy a system and a hot spare for a fraction of the cost of a new system. http://www.mired.org/consulting.html Independent Network/Unix/Perforce consultant, email for more information. From owner-freebsd-hackers@FreeBSD.ORG Thu May 31 19:14:11 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id F0CA816A41F for ; Thu, 31 May 2007 19:14:11 +0000 (UTC) (envelope-from blin@gmx.net) Received: from mail.gmx.net (mail.gmx.net [213.165.64.20]) by mx1.freebsd.org (Postfix) with SMTP id 6536313C4AD for ; Thu, 31 May 2007 19:14:10 +0000 (UTC) (envelope-from blin@gmx.net) Received: (qmail invoked by alias); 31 May 2007 18:47:30 -0000 Received: from p3EE03AED.dip0.t-ipconnect.de (EHLO noname) [62.224.58.237] by mail.gmx.net (mp019) with SMTP; 31 May 2007 20:47:30 +0200 X-Authenticated: #162580 X-Provags-ID: V01U2FsdGVkX19K4txOicZKFpSjC2lyWmYPkpBda5RwVDWuBGsk8B 7hLPFi+YYs7pAR From: Kai Blin To: freebsd-hackers@freebsd.org Date: Thu, 31 May 2007 20:47:15 +0200 User-Agent: KMail/1.9.6 MIME-Version: 1.0 Content-Type: multipart/signed; boundary="nextPart2710033.RjlmGSnfmb"; protocol="application/pgp-signature"; micalg=pgp-sha1 Content-Transfer-Encoding: 7bit Message-Id: <200705312047.27316.blin@gmx.net> X-Y-GMX-Trusted: 0 Subject: Getting the Wine preloader to work in FreeBSD X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 31 May 2007 19:14:12 -0000 --nextPart2710033.RjlmGSnfmb Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Content-Disposition: inline Hi folks, after being nagged by a freebsd user(dibridagoe) in #winehq today, I have=20 decided to look into a problem with the Wine preloader on FreeBSD=20 (http://bugs.winehq.org/show_bug.cgi?id=3D5732). Now, the wine preloader is Linux-specific, it seems. I'm not sure if it is= =20 needed in FreeBSD at all.=20 http://www.winehq.org/site/docs/winedev-guide/x2584#ARCH-MEM and=20 http://www.winehq.org/site/docs/winedev-guide/x2853 contain some information about the preloader rationale.=20 The preloader source is at=20 http://source.winehq.org/git/wine.git/?a=3Dblob;f=3Dloader/preloader.c;h=3D= 0034e9a948c3de7c2752f17efe11acb6db42a1ae;hb=3DHEAD=20 the comments should explain some of the rationale there. Now, why am I bugging you folks with that? The reason this hasn't been fixe= d=20 for FreeBSD yet is that we simply don't have developers who run that OS. So= =20 what I am asking for is this: Is someone who knows how to fix this in FreeBSD interested in helping me to= =20 create a patch? It can be a pain to get patches accepted in Wine, especially if you're a=20 first-time committer. That's where I can offer my experience, or at least m= y=20 stubborness in getting Wine patches in. However, I lack the knowledge of th= is=20 low level code. Thus, I would need to have someone helping me with a patch = or=20 even someone to write that patch him/herself, in the latter case I'd offer= =20 help in getting it through Wine's review process. Thanks in advance, Kai =2D-=20 Kai Blin WorldForge developer http://www.worldforge.org/ Wine developer http://wiki.winehq.org/KaiBlin Samba team member http://us1.samba.org/samba/team/ =2D- Will code for cotton. --nextPart2710033.RjlmGSnfmb Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (GNU/Linux) iD8DBQBGXxgzxvBXTyKdCkkRAojpAJ47D8jddokOqzEaKz50dv4qEnCx/QCeIeLM 8RLGE/Pcky/jV34qCIfA9fA= =lE/0 -----END PGP SIGNATURE----- --nextPart2710033.RjlmGSnfmb-- From owner-freebsd-hackers@FreeBSD.ORG Thu May 31 20:38:49 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id E275016A400 for ; Thu, 31 May 2007 20:38:49 +0000 (UTC) (envelope-from root@orange.fr) Received: from smtp21.orange.fr (smtp21.orange.fr [80.12.242.47]) by mx1.freebsd.org (Postfix) with ESMTP id A8A7513C44B for ; Thu, 31 May 2007 20:38:49 +0000 (UTC) (envelope-from root@orange.fr) Received: from smtp21.orange.fr (mwinf2103 [10.232.7.25]) by mwinf2110.orange.fr (SMTP Server) with ESMTP id 487AF1C2518B for ; Thu, 31 May 2007 22:21:13 +0200 (CEST) Received: from me-wanadoo.net (localhost [127.0.0.1]) by mwinf2103.orange.fr (SMTP Server) with ESMTP id D14091C0005E for ; Thu, 31 May 2007 22:21:11 +0200 (CEST) Received: from orange.fr (ARouen-153-1-12-144.w90-17.abo.wanadoo.fr [90.17.155.144]) by mwinf2103.orange.fr (SMTP Server) with ESMTP id BE27F1C00086 for ; Thu, 31 May 2007 22:21:11 +0200 (CEST) X-ME-UUID: 20070531202111778.BE27F1C00086@mwinf2103.orange.fr To: freebsd-hackers@freebsd.org From: rmgls@wanadoo.fr Date: Thu, 31 May 2007 22:21:10 +0200 Sender: root@orange.fr Message-Id: <20070531202111.BE27F1C00086@mwinf2103.orange.fr> Subject: how to add unsupported pccard? X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 31 May 2007 20:38:50 -0000 hi all, I looked in the devel handbook to find some hints about pccards. i have the following hardware: generic 2 uarts ports 16450 pccard, non recognized: as you can see: card activation failed! here is the FreeBSD dmesg with pccard_debug="YES": cbb0: card inserted: event=0x00000006, state=30000910 pccard0: chip_socket_enable cbb_pcic_socket_enable: cbb0: cbb_power: 3V pccard0: read_cis pcib2: pccard0 requested memory range 0xb0000000-0xb00fffff: good cis mem map 0xe46bb000 (resource: 0xb0010000) pccard0: CIS tuple chain: CISTPL_DEVICE type=null speed=null 01 02 00 ff unhandled CISTPL 3 03 00 CISTPL_END ff cis mem map e46bb000 CISTPL_LINKTARGET expected, code ff observed pccard0: check_cis_quirks pccard0: Card has no functions! cbb0: PC Card card activation failed debian reports less things (perhaps without debug on): pccard: PCMCIA card inserted into slot 0 cs: memory probe 0xb0000000-0xb00fffff: excluding 0xb0000000-0xb000ffff pcmcia: registering new device pcmcia0.0 0.0: ttyS0 at I/O 0x2048 (irq = 3) is a 16450 waiting a good section in the developper's handbook on this topic, can a pccard guru give some advices to make the card recognized? this could be a serious help not only for me! thanks in advance. Raoul rmgls@wanadoo.fr From owner-freebsd-hackers@FreeBSD.ORG Thu May 31 21:28:27 2007 Return-Path: X-Original-To: hackers@freebsd.org Delivered-To: freebsd-hackers@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 15CA316A421; Thu, 31 May 2007 21:28:27 +0000 (UTC) (envelope-from youshi10@u.washington.edu) Received: from mxout7.cac.washington.edu (mxout7.cac.washington.edu [140.142.32.178]) by mx1.freebsd.org (Postfix) with ESMTP id E658A13C457; Thu, 31 May 2007 21:28:26 +0000 (UTC) (envelope-from youshi10@u.washington.edu) Received: from hymn03.u.washington.edu (hymn03.u.washington.edu [140.142.12.169]) by mxout7.cac.washington.edu (8.13.7+UW06.06/8.13.7+UW07.05) with ESMTP id l4VLSQ05005978 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Thu, 31 May 2007 14:28:26 -0700 Received: from localhost (localhost [127.0.0.1]) by hymn03.u.washington.edu (8.13.7+UW06.06/8.13.7+UW07.03) with ESMTP id l4VLSQbr021751; Thu, 31 May 2007 14:28:26 -0700 X-Auth-Received: from [192.55.52.4] by hymn03.u.washington.edu via HTTP; Thu, 31 May 2007 14:28:25 PDT Date: Thu, 31 May 2007 14:28:26 -0700 (PDT) From: youshi10@u.washington.edu To: hackers@freebsd.org In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-PMX-Version: 5.3.1.294258, Antispam-Engine: 2.5.1.298604, Antispam-Data: 2007.5.31.140735 X-Uwash-Spam: Gauge=IIIIIII, Probability=7%, Report='SUPERLONG_LINE 0.05, NO_REAL_NAME 0, __CT 0, __CT_TEXT_PLAIN 0, __HAS_MSGID 0, __MIME_TEXT_ONLY 0, __MIME_VERSION 0, __SANE_MSGID 0' Cc: usb@freebsd.org, stable@freebsd.org, doc@freebsd.org, current@freebsd.org Subject: Re: Kernel panicking in 6.2 and 7-CURRENT -- interrupt issues X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 31 May 2007 21:28:27 -0000 On Tue, 29 May 2007 youshi10@u.washington.edu wrote: > Hello, > I'm trying to install 7-CURRENT on my desktop, locally instead of on a > virtual machine, and I'm running into an issue where the kernel almost always > panics on boot with my motherboard (ASUS P5N-SLI E), due to some sort of > interrupt assignment / probing issues. > > Observations: > 1. 7-CURRENT (built) never properly detects the interrupts on the system. > 2. 7-CURRENT (May ISO snapshot) doesn't properly detect my USB keyboard, > but boots and runs sysinstall. > 3. 6.2 RELEASE panics if I don't boot FreeBSD up into "safe mode", due to > an issue with the ohci driver (I think the error message had something to deal > with device adding / enumeration and not being able to find a memory / > interrupt address). > > First off, if I could get some of the command line arguments to pass to > the kernel to emulate safe-mode, that would be much appreciated. Second off, if > anybody has any ideas on how to debug this issue, I'll go off and try to > determine what the cause is. If it's anything like it was before (clean out > /usr/obj, rebuild), I'll be ok. Otherwise, I'll have to purchase more parts and > build another dedicated FreeBSD system :(.. > > Thanks, > -Garrett Sorry for cross-posting so much, but this information is relevant to all of the mailing lists included. Ok I "solved" my problem, to a certain extent. Basically the ASUS P5N-E SLI motherboard (nVidia chipset) isn't bootable with FreeBSD 6.2 RELEASE without going into SAFE MODE (panics after ohci_add_done looking for a non-existent memory address, if the USB controller is enabled in the BIOS with "Legacy Mode", i.e. USB Keyboard compatible), and isn't compatible with FreeBSD 7 at all (kernel can't map IRQs properly on up-to-date CURRENT and panics in kernel mode during boot). Strangely enough the 7-CURRENT snapshot ISOs work, but then again it doesn't setup the OHCI and USB keyboard stuff properly, and I can't get any sort of input from my keyboard, obviously. The ASUS P5B-DLX motherboard (Intel chipset) is compatible with FreeBSD 6.2 and 7 though, without any hacks. Maybe someone should update some documentation on the release notes page for supported hardware? Thanks, -Garrett PS Please CC me in any communication as I'm not subscribed to doc@, stable@, or usb@. From owner-freebsd-hackers@FreeBSD.ORG Fri Jun 1 03:35:58 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 8E70C16A421 for ; Fri, 1 Jun 2007 03:35:58 +0000 (UTC) (envelope-from mistry.7@osu.edu) Received: from mail.united-ware.com (am-productions.biz [69.61.164.22]) by mx1.freebsd.org (Postfix) with ESMTP id 731A813C447 for ; Fri, 1 Jun 2007 03:35:57 +0000 (UTC) (envelope-from mistry.7@osu.edu) Received: from [192.168.1.100] (cpe-71-72-80-132.columbus.res.rr.com [71.72.80.132]) (authenticated bits=0) by mail.united-ware.com (8.13.8/8.13.8) with ESMTP id l513oLEL096697 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Thu, 31 May 2007 23:50:28 -0400 (EDT) (envelope-from mistry.7@osu.edu) From: Anish Mistry To: freebsd-hackers@freebsd.org Date: Thu, 31 May 2007 23:23:20 -0400 User-Agent: KMail/1.9.6 References: <200705312047.27316.blin@gmx.net> In-Reply-To: <200705312047.27316.blin@gmx.net> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="nextPart1772583.Mjklk7GImS"; protocol="application/pgp-signature"; micalg=pgp-sha1 Content-Transfer-Encoding: 7bit Message-Id: <200705312323.20961.mistry.7@osu.edu> X-Spam-Status: No, score=-0.9 required=5.0 tests=BAYES_50,MYFREEBSD2, RCVD_IN_NJABL_DUL,URI_NO_WWW_ANY_CGI autolearn=failed version=3.1.3 X-Spam-Checker-Version: SpamAssassin 3.1.3 (2006-06-01) on mail.united-ware.com X-Virus-Scanned: ClamAV 0.88.7/3335/Thu May 31 06:16:21 2007 on mail.united-ware.com X-Virus-Status: Clean Cc: Kai Blin Subject: Re: Getting the Wine preloader to work in FreeBSD X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 01 Jun 2007 03:35:58 -0000 --nextPart1772583.Mjklk7GImS Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Content-Disposition: inline On Thursday 31 May 2007, Kai Blin wrote: > Hi folks, > > after being nagged by a freebsd user(dibridagoe) in #winehq today, > I have decided to look into a problem with the Wine preloader on > FreeBSD (http://bugs.winehq.org/show_bug.cgi?id=3D5732). > > Now, the wine preloader is Linux-specific, it seems. I'm not sure > if it is needed in FreeBSD at all. > > http://www.winehq.org/site/docs/winedev-guide/x2584#ARCH-MEM and > http://www.winehq.org/site/docs/winedev-guide/x2853 > contain some information about the preloader rationale. > > The preloader source is at > http://source.winehq.org/git/wine.git/?a=3Dblob;f=3Dloader/preloader.c; >h=3D0034e9a948c3de7c2752f17efe11acb6db42a1ae;hb=3DHEAD the comments > should explain some of the rationale there. > > Now, why am I bugging you folks with that? The reason this hasn't > been fixed for FreeBSD yet is that we simply don't have developers > who run that OS. So what I am asking for is this: > > Is someone who knows how to fix this in FreeBSD interested in > helping me to create a patch? > It can be a pain to get patches accepted in Wine, especially if > you're a first-time committer. That's where I can offer my > experience, or at least my stubborness in getting Wine patches in. > However, I lack the knowledge of this low level code. Thus, I would > need to have someone helping me with a patch or even someone to > write that patch him/herself, in the latter case I'd offer help in > getting it through Wine's review process. I'd suggest contacting gerald@FreeBSD.org since he maintains the wine=20 port and might have the knowledge. =2D-=20 Anish Mistry --nextPart1772583.Mjklk7GImS Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.3 (FreeBSD) iD8DBQBGX5EoxqA5ziudZT0RAr5lAKCI2NbLS2IT9p89gp2z7fJOKpbi5gCgvzSw Wed6blvmUbeRnup+dVspapY= =SoDM -----END PGP SIGNATURE----- --nextPart1772583.Mjklk7GImS-- From owner-freebsd-hackers@FreeBSD.ORG Fri Jun 1 10:41:38 2007 Return-Path: X-Original-To: hackers@FreeBSD.org Delivered-To: freebsd-hackers@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 8455F16A41F for ; Fri, 1 Jun 2007 10:41:38 +0000 (UTC) (envelope-from Artis.Caune@latnet.lv) Received: from esbens.latnet.lv (esbens.latnet.lv [159.148.19.115]) by mx1.freebsd.org (Postfix) with ESMTP id 354DD13C43E for ; Fri, 1 Jun 2007 10:41:38 +0000 (UTC) (envelope-from Artis.Caune@latnet.lv) Received: from localhost (localhost.localdomain [127.0.0.1]) by esbens.latnet.lv (Postfix) with ESMTP id BC72916C580 for ; Fri, 1 Jun 2007 13:19:45 +0300 (EEST) X-Virus-Scanned: Debian amavisd-new at esbens.latnet.lv Received: from esbens.latnet.lv ([127.0.0.1]) by localhost (esbens.latnet.lv [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id UrRTP8ORThgS for ; Fri, 1 Jun 2007 13:19:44 +0300 (EEST) Received: from [159.148.108.245] (didii.latnet.lv [159.148.108.245]) by esbens.latnet.lv (Postfix) with ESMTP id 8741316C577 for ; Fri, 1 Jun 2007 13:19:07 +0300 (EEST) Message-ID: <465FF29B.3010307@latnet.lv> Date: Fri, 01 Jun 2007 13:19:07 +0300 From: Artis Caune User-Agent: Thunderbird 1.5.0.10 (X11/20070327) MIME-Version: 1.0 To: hackers@FreeBSD.org Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Cc: Subject: stopping callouts X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 01 Jun 2007 10:41:38 -0000 Hello FreeBSD-hackers! what is the right way to stop callout which is initialized with mutex: callout_init_mtx(&callout_function, &my_mtx, 0); should I lock mutex before calling callout_drain() or not? man page says: callout_drain() is identical to callout_stop() and if the callout has an associated mutex, then that mutex must be held when callout_stop() is called. /* MOD_UNLOAD */ MTX_LOCK; callout_drain() MTX_UNLOCK; From owner-freebsd-hackers@FreeBSD.ORG Fri Jun 1 15:17:33 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 51AE116A421 for ; Fri, 1 Jun 2007 15:17:33 +0000 (UTC) (envelope-from hselasky@c2i.net) Received: from swip.net (mailfe01.swip.net [212.247.154.1]) by mx1.freebsd.org (Postfix) with ESMTP id 44E4E13C4BF for ; Fri, 1 Jun 2007 15:17:32 +0000 (UTC) (envelope-from hselasky@c2i.net) X-Cloudmark-Score: 0.000000 [] Received: from [194.248.135.20] (account mc467741@c2i.net HELO laptop.lan) by mailfe01.swip.net (CommuniGate Pro SMTP 5.1.7) with ESMTPA id 488543002; Fri, 01 Jun 2007 17:17:30 +0200 From: Hans Petter Selasky To: freebsd-hackers@freebsd.org Date: Fri, 1 Jun 2007 17:17:22 +0200 User-Agent: KMail/1.9.5 References: <465FF29B.3010307@latnet.lv> In-Reply-To: <465FF29B.3010307@latnet.lv> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200706011717.22179.hselasky@c2i.net> Cc: hackers@freebsd.org, Artis Caune Subject: Re: stopping callouts X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 01 Jun 2007 15:17:33 -0000 On Friday 01 June 2007 12:19, Artis Caune wrote: > Hello FreeBSD-hackers! > > what is the right way to stop callout which is initialized with mutex: > callout_init_mtx(&callout_function, &my_mtx, 0); > > should I lock mutex before calling callout_drain() or not? > > man page says: callout_drain() is identical to callout_stop() and > if the callout has an associated mutex, then that mutex must be held > when callout_stop() is called. > You should call "callout_drain()" w/o any locks held, because this function might sleep. callout_drain() is not identical to callout_stop(). Then man-page should be updated. --HPS From owner-freebsd-hackers@FreeBSD.ORG Fri Jun 1 16:17:35 2007 Return-Path: X-Original-To: hackers@freebsd.org Delivered-To: freebsd-hackers@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id B3C4816A400 for ; Fri, 1 Jun 2007 16:17:35 +0000 (UTC) (envelope-from hselasky@c2i.net) Received: from swip.net (mailfe01.swip.net [212.247.154.1]) by mx1.freebsd.org (Postfix) with ESMTP id 52AAB13C45D for ; Fri, 1 Jun 2007 16:17:35 +0000 (UTC) (envelope-from hselasky@c2i.net) X-Cloudmark-Score: 0.000000 [] Received: from [194.248.135.20] (account mc467741@c2i.net HELO laptop.lan) by mailfe01.swip.net (CommuniGate Pro SMTP 5.1.7) with ESMTPA id 488543002; Fri, 01 Jun 2007 17:17:30 +0200 From: Hans Petter Selasky To: freebsd-hackers@freebsd.org Date: Fri, 1 Jun 2007 17:17:22 +0200 User-Agent: KMail/1.9.5 References: <465FF29B.3010307@latnet.lv> In-Reply-To: <465FF29B.3010307@latnet.lv> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200706011717.22179.hselasky@c2i.net> Cc: hackers@freebsd.org, Artis Caune Subject: Re: stopping callouts X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 01 Jun 2007 16:17:35 -0000 On Friday 01 June 2007 12:19, Artis Caune wrote: > Hello FreeBSD-hackers! > > what is the right way to stop callout which is initialized with mutex: > callout_init_mtx(&callout_function, &my_mtx, 0); > > should I lock mutex before calling callout_drain() or not? > > man page says: callout_drain() is identical to callout_stop() and > if the callout has an associated mutex, then that mutex must be held > when callout_stop() is called. > You should call "callout_drain()" w/o any locks held, because this function might sleep. callout_drain() is not identical to callout_stop(). Then man-page should be updated. --HPS From owner-freebsd-hackers@FreeBSD.ORG Fri Jun 1 17:02:07 2007 Return-Path: X-Original-To: hackers@freebsd.org Delivered-To: freebsd-hackers@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id E89DB16A530 for ; Fri, 1 Jun 2007 17:02:07 +0000 (UTC) (envelope-from sam@errno.com) Received: from ebb.errno.com (ebb.errno.com [69.12.149.25]) by mx1.freebsd.org (Postfix) with ESMTP id 7ACA013C465 for ; Fri, 1 Jun 2007 17:02:07 +0000 (UTC) (envelope-from sam@errno.com) Received: from trouble.errno.com (trouble.errno.com [10.0.0.248]) (authenticated bits=0) by ebb.errno.com (8.13.6/8.12.6) with ESMTP id l51GPRva097789 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Fri, 1 Jun 2007 09:25:29 -0700 (PDT) (envelope-from sam@errno.com) Message-ID: <46604877.7030007@errno.com> Date: Fri, 01 Jun 2007 09:25:27 -0700 From: Sam Leffler User-Agent: Thunderbird 2.0.0.0 (X11/20070528) MIME-Version: 1.0 To: Artis Caune References: <465FF29B.3010307@latnet.lv> In-Reply-To: <465FF29B.3010307@latnet.lv> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: hackers@freebsd.org Subject: Re: stopping callouts X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 01 Jun 2007 17:02:08 -0000 Artis Caune wrote: > Hello FreeBSD-hackers! > > what is the right way to stop callout which is initialized with mutex: > callout_init_mtx(&callout_function, &my_mtx, 0); > > should I lock mutex before calling callout_drain() or not? > > man page says: callout_drain() is identical to callout_stop() and > if the callout has an associated mutex, then that mutex must be held > when callout_stop() is called. > > > /* MOD_UNLOAD */ > > MTX_LOCK; > callout_drain() > MTX_UNLOCK; If you use callout_init_mtx then use callout_stop while holding my_mtx; if the callout is blocked waiting for my_mtx the request will be discarded. callout_drain should not be called while holding my_mtx because it can sleep. Sam From owner-freebsd-hackers@FreeBSD.ORG Sat Jun 2 22:57:20 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 7C37116A400 for ; Sat, 2 Jun 2007 22:57:20 +0000 (UTC) (envelope-from artifact.one@googlemail.com) Received: from wx-out-0506.google.com (wx-out-0506.google.com [66.249.82.230]) by mx1.freebsd.org (Postfix) with ESMTP id 3F49E13C448 for ; Sat, 2 Jun 2007 22:57:20 +0000 (UTC) (envelope-from artifact.one@googlemail.com) Received: by wx-out-0506.google.com with SMTP id h28so762387wxd for ; Sat, 02 Jun 2007 15:57:19 -0700 (PDT) DKIM-Signature: a=rsa-sha1; c=relaxed/relaxed; d=googlemail.com; s=beta; h=domainkey-signature:received:received:message-id:date:from:to:subject:mime-version:content-type:content-transfer-encoding:content-disposition; b=OSPNUdqynC1xIAKsCVVpMaARlSg8KQoCj5YYZMEkPAJr9oM5z0eCk7ABNXQTudKXHHaHDaqsFf2VXy4uxXsJ7+1pLMLc5QgOOE/gOWByz1ibC9katU9y6RKxG53E1fihFSNDsyPXLp7PFgX/pgytkglVkEnFoamYmd/1b5w0yNk= DomainKey-Signature: a=rsa-sha1; c=nofws; d=googlemail.com; s=beta; h=received:message-id:date:from:to:subject:mime-version:content-type:content-transfer-encoding:content-disposition; b=Rec7t8MkZsPv/2QSzkhwF88IP5lSJQX9D+n87O+gWoSCx1zDv+NRnxBxFTUJSBpN99IQsvFgPAqOaE65cYsm9cLl2fZVZZeLNjvh8n9i70YC8BTXTVx6tncilqFReGHSf1NnFnMS8mAIudWawvWa6GKhQAE8IwTCGv/bYUHngMY= Received: by 10.90.95.11 with SMTP id s11mr2773695agb.1180825039433; Sat, 02 Jun 2007 15:57:19 -0700 (PDT) Received: by 10.90.51.1 with HTTP; Sat, 2 Jun 2007 15:57:19 -0700 (PDT) Message-ID: <8e96a0b90706021557h2168337emb9b7b59a2084530e@mail.gmail.com> Date: Sat, 2 Jun 2007 23:57:19 +0100 From: "mal content" To: freebsd-hackers@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Content-Disposition: inline Subject: Open FIFO non-blocking but have write() block(?) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 02 Jun 2007 22:57:20 -0000 Hello. The wording of the question in the subject is terrible, I know. What I'm trying to do is write a program that creates two named pipes in the filesystem. Logging date appears on one, to be read by any external program and commands are read by the program from the other. If data is written to the logging FIFO and there's no external reader, the log info is discarded. The problem I'm having is that I can create the 'log' fifo with mkfifo but the open call fails because there's no reader: if (mkfifo("control", 0600) == -1) die(); fd1 = open("control", O_RDONLY | O_NONBLOCK); if (fd1 == -1) die(); if (mkfifo("log", 0600) == -1) die(); fd2 = open("log", O_WRONLY | O_NONBLOCK); if (fd2 == -1) die(); Obviously I can remove the O_NONBLOCK from the second open call, but this causes it to block and I don't want that. The behaviour I'm actually looking for is that the first open() call should succeed but the first write made to the fifo (with no reader) should fail. Is it possible to portably get this behaviour from FIFOs? Or should I be looking at something else like unix domain sockets? thanks, MC