From owner-freebsd-arch Sun Jul 14 0:36:11 2002 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.FreeBSD.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id C762F37B400; Sun, 14 Jul 2002 00:36:08 -0700 (PDT) Received: from srv1.cosmo-project.de (srv1.cosmo-project.de [213.83.6.106]) by mx1.FreeBSD.org (Postfix) with ESMTP id 5DC1143E3B; Sun, 14 Jul 2002 00:36:07 -0700 (PDT) (envelope-from ticso@cicely5.cicely.de) Received: from cicely5.cicely.de (cicely5.cicely.de [IPv6:3ffe:400:8d0:301:200:92ff:fe9b:20e7]) (authenticated bits=0) by srv1.cosmo-project.de (8.12.3/8.12.3) with ESMTP id g6E7a1Ma036694 (version=TLSv1/SSLv3 cipher=EDH-RSA-DES-CBC3-SHA bits=168 verify=OK); Sun, 14 Jul 2002 09:36:04 +0200 (CEST) (envelope-from ticso@cicely5.cicely.de) Received: from cicely5.cicely.de (localhost [IPv6:::1]) by cicely5.cicely.de (8.12.1/8.12.1) with ESMTP id g6E7a2FJ067728 (version=TLSv1/SSLv3 cipher=EDH-RSA-DES-CBC3-SHA bits=168 verify=NO); Sun, 14 Jul 2002 09:36:02 +0200 (CEST)?g (envelope-from ticso@cicely5.cicely.de) Received: (from ticso@localhost) by cicely5.cicely.de (8.12.1/8.12.1/Submit) id g6E7a09i067727; Sun, 14 Jul 2002 09:36:00 +0200 (CEST)?g (envelope-from ticso) Date: Sun, 14 Jul 2002 09:36:00 +0200 From: Bernd Walter To: Terry Lambert Cc: Gregory Neil Shapiro , freebsd-arch@FreeBSD.ORG Subject: Re: Mail subsystem defaults, adding authentication. Message-ID: <20020714073559.GY63545@cicely5.cicely.de> Reply-To: ticso@cicely.de References: <20020713034725.GB47677@ussenterprise.ufp.org> <3D2FAFB2.E2E9CF36@mindspring.com> <20020713045704.GA49379@ussenterprise.ufp.org> <3D300FD4.7479A8E5@mindspring.com> <15664.47827.844708.151118@monkeyboy.gshapiro.net> <3D30C4DA.22A255A8@mindspring.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <3D30C4DA.22A255A8@mindspring.com> X-Operating-System: FreeBSD cicely5.cicely.de 5.0-CURRENT i386 User-Agent: Mutt/1.5.1i Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG On Sat, Jul 13, 2002 at 05:24:59PM -0700, Terry Lambert wrote: > > You can (and should) use STARTTLS with SMTP AUTH PLAIN/LOGIN and do not > > (and should not) use SMTP over SSL as it is non-standard. > > IMO, this is broken. Here's why: Implementation of SSL in the > kernel is a foregone conclusion. It is a matter of "when", not > "if", due to work like that of Sam Leffler's recent porting of > the OpenBSD crypto hardware interface framework to FreeBSD. > > Basically, asking for conversion of a socket from one type to > another is not something that will necessarily be supportable. With SSL you still do a normal socket connect anyway and than call SSL_connect/accept on the already existing connection. What's the matter with exchanging packets before doing that? Does that mean that the SSL API changes? > The whole "STARTTLS" thing was introduced to kludge around the > lack of IPSEC support in IPv4. Even if you argue that it's an > issue for IPv4 because IPSEC bloats the hell out of IPv4 even > when it's not being used, IPv6 requires implementation of IPSEC > for it to be called an IPv6 implementation. > > This means that the days of transport crypto decisions like > this one, and the code to implement it, living in user space > are numbered, no matter what. I'm not a cryptographic expert, but I wouldn't prefer a packet encryption over a stream encryption. > I know the sendmail folks don't like SMTP over SSL, but... > there is an IANA assigned number in /etc/services for it, > which makes it about as standard as it can be; I don't think > SSL RFC policy requires a per protocol SSL usage RFC for SSL > to be used (that wouldn't make sense, in terms of promoting > the adoption of SSL). With STARTTLS you can probe for SSL in MTA - MTA comunications. MTAs connect foreign SMTP servers and want to prefer SSL. It's unpractical to try a connect to smpts port first with all those blackhole firewalls out there. The only downside with STARTTLS is that it makes it allmost impossible to use external SSL boxes. -- B.Walter COSMO-Project http://www.cosmo-project.de ticso@cicely.de Usergroup info@cosmo-project.de To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Sun Jul 14 1:19:10 2002 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.FreeBSD.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 1A37C37B401; Sun, 14 Jul 2002 01:19:06 -0700 (PDT) Received: from harrier.mail.pas.earthlink.net (harrier.mail.pas.earthlink.net [207.217.120.12]) by mx1.FreeBSD.org (Postfix) with ESMTP id 5020643E5E; Sun, 14 Jul 2002 01:19:05 -0700 (PDT) (envelope-from tlambert2@mindspring.com) Received: from pool0078.cvx40-bradley.dialup.earthlink.net ([216.244.42.78] helo=mindspring.com) by harrier.mail.pas.earthlink.net with esmtp (Exim 3.33 #1) id 17Teag-0004bK-00; Sun, 14 Jul 2002 01:18:53 -0700 Message-ID: <3D3133AA.8387DA1E@mindspring.com> Date: Sun, 14 Jul 2002 01:17:46 -0700 From: Terry Lambert X-Mailer: Mozilla 4.7 [en]C-CCK-MCD {Sony} (Win98; U) X-Accept-Language: en MIME-Version: 1.0 To: ticso@cicely.de Cc: Gregory Neil Shapiro , freebsd-arch@FreeBSD.ORG Subject: Re: Mail subsystem defaults, adding authentication. References: <20020713034725.GB47677@ussenterprise.ufp.org> <3D2FAFB2.E2E9CF36@mindspring.com> <20020713045704.GA49379@ussenterprise.ufp.org> <3D300FD4.7479A8E5@mindspring.com> <15664.47827.844708.151118@monkeyboy.gshapiro.net> <3D30C4DA.22A255A8@mindspring.com> <20020714073559.GY63545@cicely5.cicely.de> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG Bernd Walter wrote: > > IMO, this is broken. Here's why: Implementation of SSL in the > > kernel is a foregone conclusion. It is a matter of "when", not > > "if", due to work like that of Sam Leffler's recent porting of > > the OpenBSD crypto hardware interface framework to FreeBSD. > > > > Basically, asking for conversion of a socket from one type to > > another is not something that will necessarily be supportable. > > With SSL you still do a normal socket connect anyway and than > call SSL_connect/accept on the already existing connection. > What's the matter with exchanging packets before doing that? > Does that mean that the SSL API changes? Yes; it becomes something like: s = socket( PF_INET, SOCK_SSL, 0); bzero(&sa, sizeof(struct sockaddr_ssl)); sa.sin_family = PF_INET; sa.sin_port = htons( 465); /* SMTPS */ sa.sin_cert = &cert; /* Local Cert. */ connect( s, (struct sockaddr *)&sa, sizeof(struct sockaddr_ssl)); > I'm not a cryptographic expert, but I wouldn't prefer a packet > encryption over a stream encryption. The bloat in the IPSEC case is because of the KAME integration; the IPSEC in IPv4 eats a context structure, even if it's not used. The IPv6 code is better. I took it to be a political statement on IPv4 by the KAME people. 8-). The support being in the kernel instead of user space doesn't make it other than a stream encryption, though. > With STARTTLS you can probe for SSL in MTA - MTA comunications. > MTAs connect foreign SMTP servers and want to prefer SSL. > It's unpractical to try a connect to smpts port first with all those > blackhole firewalls out there. Meaning that it won't reject the initial packet, you will have to time out the connection, and then retry on port 25? Actually, you can use non-blocking I/O, simultaneously connect, and then drop whiever one doesn't pan out, or you could cache the result, or you etc.. could make it a mailer flag (e.g. "mailer smtps has flag 'Q' and uses port 465"), and control it on a peer basis. It's not really for trusted host communications anyway, since the only reason we're eating the encruption overhead is to secure a plaintext password; the that rest of the session is also encrypted is actually annoying overhead, in most cases. > The only downside with STARTTLS is that it makes it allmost impossible > to use external SSL boxes. That's a very big downside. OpenSSL does maybe 200 if you have a fas box, and you eat the whole thing doing crypto. Not worth doing onboard the box itself, if you can possibly avoid it. -- Terry To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Sun Jul 14 2:16:58 2002 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.FreeBSD.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id CD3E737B400 for ; Sun, 14 Jul 2002 02:16:55 -0700 (PDT) Received: from gw.catspoiler.org (217-ip-163.nccn.net [209.79.217.163]) by mx1.FreeBSD.org (Postfix) with ESMTP id 1D8D443E3B for ; Sun, 14 Jul 2002 02:16:50 -0700 (PDT) (envelope-from dl-freebsd@catspoiler.org) Received: from mousie.catspoiler.org (mousie.catspoiler.org [192.168.101.2]) by gw.catspoiler.org (8.12.5/8.12.5) with ESMTP id g6E9Ghwr020552 for ; Sun, 14 Jul 2002 02:16:47 -0700 (PDT) (envelope-from dl-freebsd@catspoiler.org) Message-Id: <200207140916.g6E9Ghwr020552@gw.catspoiler.org> Date: Sun, 14 Jul 2002 02:16:43 -0700 (PDT) From: Don Lewis Subject: wiring the sysctl output buffer To: arch@FreeBSD.org MIME-Version: 1.0 Content-Type: TEXT/plain; charset=us-ascii Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG A number of places in the kernel call SYSCTL_OUT() while holding locks. The problem is that SYSCTL_OUT() can block while it wires down the "old" buffer in user space. If the data is small, such as an integer value, the best solution may be to copy the data and defer calling SYSCTL_OUT() until after the locks are released. This extra copy could be wasteful if the data is large, and it may be cumbersome if the data size is not known ahead of time, since determining the data size may be expensive. The data size could also potentially change between the time the size is calculatated and the time when the data is copied to the temporary buffer if the locks must be released so that MALLOC() can be called to allocate the buffer. I think the best solution to this problem is to add a sysctl system API to prewire the output buffer that can be called before grabbing the locks. Doing so allows the existing code to operate properly with only minimal changes. Here's a patch that implements this new API and illustrates how it can be used to fix the kern.function_list sysctl, which wants to call SYSCTL_OUT() multiple times while walking a locked data structure. I think this is the best fix, though I'd like some feedback on whether this is the best API. Index: sys/sysctl.h =================================================================== RCS file: /home/ncvs/src/sys/sys/sysctl.h,v retrieving revision 1.105 diff -u -r1.105 sysctl.h --- sys/sysctl.h 16 May 2002 21:28:26 -0000 1.105 +++ sys/sysctl.h 14 Jul 2002 07:57:29 -0000 @@ -595,6 +595,7 @@ size_t *retval); int sysctl_find_oid(int *name, u_int namelen, struct sysctl_oid **noid, int *nindx, struct sysctl_req *req); +void sysctl_wire_old_buffer(struct sysctl_req *req); #else /* !_KERNEL */ #include Index: kern/kern_sysctl.c =================================================================== RCS file: /home/ncvs/src/sys/kern/kern_sysctl.c,v retrieving revision 1.126 diff -u -r1.126 kern_sysctl.c --- kern/kern_sysctl.c 29 Jun 2002 02:00:01 -0000 1.126 +++ kern/kern_sysctl.c 14 Jul 2002 07:57:50 -0000 @@ -990,6 +990,18 @@ return (error); } +/* + * Lock the user space destination buffer. + */ +void +sysctl_wire_old_buffer(struct sysctl_req *req) +{ + if (req->lock == 1 && req->oldptr && req->oldfunc == sysctl_old_user) { + vslock(req->oldptr, req->oldlen); + req->lock = 2; + } +} + int sysctl_find_oid(int *name, u_int namelen, struct sysctl_oid **noid, int *nindx, struct sysctl_req *req) Index: kern/kern_linker.c =================================================================== RCS file: /home/ncvs/src/sys/kern/kern_linker.c,v retrieving revision 1.91 diff -u -r1.91 kern_linker.c --- kern/kern_linker.c 7 Jul 2002 22:35:47 -0000 1.91 +++ kern/kern_linker.c 14 Jul 2002 07:58:38 -0000 @@ -1794,6 +1794,7 @@ linker_file_t lf; int error; + sysctl_wire_old_buffer(req); mtx_lock(&kld_mtx); TAILQ_FOREACH(lf, &linker_files, link) { error = LINKER_EACH_FUNCTION_NAME(lf, To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Sun Jul 14 2:30:56 2002 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.FreeBSD.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 45F7D37B400 for ; Sun, 14 Jul 2002 02:30:54 -0700 (PDT) Received: from critter.freebsd.dk (critter.freebsd.dk [212.242.86.163]) by mx1.FreeBSD.org (Postfix) with ESMTP id 716BA43E3B for ; Sun, 14 Jul 2002 02:30:53 -0700 (PDT) (envelope-from phk@critter.freebsd.dk) Received: from critter.freebsd.dk (localhost [127.0.0.1]) by critter.freebsd.dk (8.12.3/8.12.2) with ESMTP id g6E9SZXB051319; Sun, 14 Jul 2002 11:28:41 +0200 (CEST) (envelope-from phk@critter.freebsd.dk) To: Don Lewis Cc: arch@FreeBSD.ORG Subject: Re: wiring the sysctl output buffer In-Reply-To: Your message of "Sun, 14 Jul 2002 02:16:43 PDT." <200207140916.g6E9Ghwr020552@gw.catspoiler.org> Date: Sun, 14 Jul 2002 11:28:35 +0200 Message-ID: <51318.1026638915@critter.freebsd.dk> From: Poul-Henning Kamp Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG In message <200207140916.g6E9Ghwr020552@gw.catspoiler.org>, Don Lewis writes: >I think the best solution to this problem is to add a sysctl system API >to prewire the output buffer that can be called before grabbing the >locks. Doing so allows the existing code to operate properly with only >minimal changes. It used to be that sysctl unconditionally wired the output buffer, but that gave rise to a host of other problems. I'm not against the suggested change, but it should be used with caution. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk@FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Sun Jul 14 2:54:31 2002 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.FreeBSD.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 323F737B400 for ; Sun, 14 Jul 2002 02:54:27 -0700 (PDT) Received: from gw.catspoiler.org (217-ip-163.nccn.net [209.79.217.163]) by mx1.FreeBSD.org (Postfix) with ESMTP id AEDFE43E67 for ; Sun, 14 Jul 2002 02:54:26 -0700 (PDT) (envelope-from dl-freebsd@catspoiler.org) Received: from mousie.catspoiler.org (mousie.catspoiler.org [192.168.101.2]) by gw.catspoiler.org (8.12.5/8.12.5) with ESMTP id g6E9sBwr020599; Sun, 14 Jul 2002 02:54:15 -0700 (PDT) (envelope-from dl-freebsd@catspoiler.org) Message-Id: <200207140954.g6E9sBwr020599@gw.catspoiler.org> Date: Sun, 14 Jul 2002 02:54:11 -0700 (PDT) From: Don Lewis Subject: Re: wiring the sysctl output buffer To: phk@critter.freebsd.dk Cc: arch@FreeBSD.ORG In-Reply-To: <51318.1026638915@critter.freebsd.dk> MIME-Version: 1.0 Content-Type: TEXT/plain; charset=us-ascii Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG On 14 Jul, Poul-Henning Kamp wrote: > In message <200207140916.g6E9Ghwr020552@gw.catspoiler.org>, Don Lewis writes: > >>I think the best solution to this problem is to add a sysctl system API >>to prewire the output buffer that can be called before grabbing the >>locks. Doing so allows the existing code to operate properly with only >>minimal changes. > > It used to be that sysctl unconditionally wired the output buffer, > but that gave rise to a host of other problems. Anything specific that I should be aware of? > I'm not against the suggested change, but it should be used with > caution. I'd only want to use this when it was the best solution, preferably after checking that SYSCTL_OUT() would actually be called. To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Sun Jul 14 2:59:39 2002 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.FreeBSD.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id ABCAD37B400 for ; Sun, 14 Jul 2002 02:59:36 -0700 (PDT) Received: from critter.freebsd.dk (critter.freebsd.dk [212.242.86.163]) by mx1.FreeBSD.org (Postfix) with ESMTP id EBCEE43E4A for ; Sun, 14 Jul 2002 02:59:35 -0700 (PDT) (envelope-from phk@critter.freebsd.dk) Received: from critter.freebsd.dk (localhost [127.0.0.1]) by critter.freebsd.dk (8.12.3/8.12.2) with ESMTP id g6E9vTXB056793; Sun, 14 Jul 2002 11:57:29 +0200 (CEST) (envelope-from phk@critter.freebsd.dk) To: Don Lewis Cc: arch@FreeBSD.ORG Subject: Re: wiring the sysctl output buffer In-Reply-To: Your message of "Sun, 14 Jul 2002 02:54:11 PDT." <200207140954.g6E9sBwr020599@gw.catspoiler.org> Date: Sun, 14 Jul 2002 11:57:29 +0200 Message-ID: <56792.1026640649@critter.freebsd.dk> From: Poul-Henning Kamp Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG In message <200207140954.g6E9sBwr020599@gw.catspoiler.org>, Don Lewis writes: >On 14 Jul, Poul-Henning Kamp wrote: >> In message <200207140916.g6E9Ghwr020552@gw.catspoiler.org>, Don Lewis writes: >> >>>I think the best solution to this problem is to add a sysctl system API >>>to prewire the output buffer that can be called before grabbing the >>>locks. Doing so allows the existing code to operate properly with only >>>minimal changes. >> >> It used to be that sysctl unconditionally wired the output buffer, >> but that gave rise to a host of other problems. > >Anything specific that I should be aware of? No, just bad deadlocks and such. You may also want to restrict the amount of buffer you pin if I hand the kernel a 2G buffer for a 4 byte sysctl integer you wouldn't want to pin it all. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk@FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Sun Jul 14 3: 1:43 2002 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.FreeBSD.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id DEF9337B407; Sun, 14 Jul 2002 03:01:27 -0700 (PDT) Received: from srv1.cosmo-project.de (srv1.cosmo-project.de [213.83.6.106]) by mx1.FreeBSD.org (Postfix) with ESMTP id 5C15243E67; Sun, 14 Jul 2002 03:01:17 -0700 (PDT) (envelope-from ticso@cicely5.cicely.de) Received: from cicely5.cicely.de (cicely5.cicely.de [IPv6:3ffe:400:8d0:301:200:92ff:fe9b:20e7]) (authenticated bits=0) by srv1.cosmo-project.de (8.12.3/8.12.3) with ESMTP id g6EA1C0i037638 (version=TLSv1/SSLv3 cipher=EDH-RSA-DES-CBC3-SHA bits=168 verify=OK); Sun, 14 Jul 2002 12:01:14 +0200 (CEST) (envelope-from ticso@cicely5.cicely.de) Received: from cicely5.cicely.de (localhost [IPv6:::1]) by cicely5.cicely.de (8.12.1/8.12.1) with ESMTP id g6EA1AFJ068900 (version=TLSv1/SSLv3 cipher=EDH-RSA-DES-CBC3-SHA bits=168 verify=NO); Sun, 14 Jul 2002 12:01:11 +0200 (CEST)?g (envelope-from ticso@cicely5.cicely.de) Received: (from ticso@localhost) by cicely5.cicely.de (8.12.1/8.12.1/Submit) id g6EA19HX068899; Sun, 14 Jul 2002 12:01:09 +0200 (CEST)?g (envelope-from ticso) Date: Sun, 14 Jul 2002 12:01:08 +0200 From: Bernd Walter To: Terry Lambert Cc: ticso@cicely.de, Gregory Neil Shapiro , freebsd-arch@FreeBSD.ORG Subject: Re: Mail subsystem defaults, adding authentication. Message-ID: <20020714100108.GA63545@cicely5.cicely.de> Reply-To: ticso@cicely.de References: <20020713034725.GB47677@ussenterprise.ufp.org> <3D2FAFB2.E2E9CF36@mindspring.com> <20020713045704.GA49379@ussenterprise.ufp.org> <3D300FD4.7479A8E5@mindspring.com> <15664.47827.844708.151118@monkeyboy.gshapiro.net> <3D30C4DA.22A255A8@mindspring.com> <20020714073559.GY63545@cicely5.cicely.de> <3D3133AA.8387DA1E@mindspring.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <3D3133AA.8387DA1E@mindspring.com> X-Operating-System: FreeBSD cicely5.cicely.de 5.0-CURRENT i386 User-Agent: Mutt/1.5.1i Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG On Sun, Jul 14, 2002 at 01:17:46AM -0700, Terry Lambert wrote: > Bernd Walter wrote: > > > IMO, this is broken. Here's why: Implementation of SSL in the > > > kernel is a foregone conclusion. It is a matter of "when", not > > > "if", due to work like that of Sam Leffler's recent porting of > > > the OpenBSD crypto hardware interface framework to FreeBSD. > > > > > > Basically, asking for conversion of a socket from one type to > > > another is not something that will necessarily be supportable. > > > > With SSL you still do a normal socket connect anyway and than > > call SSL_connect/accept on the already existing connection. > > What's the matter with exchanging packets before doing that? > > Does that mean that the SSL API changes? > > Yes; it becomes something like: > > s = socket( PF_INET, SOCK_SSL, 0); > bzero(&sa, sizeof(struct sockaddr_ssl)); > sa.sin_family = PF_INET; > sa.sin_port = htons( 465); /* SMTPS */ > sa.sin_cert = &cert; /* Local Cert. */ > connect( s, (struct sockaddr *)&sa, sizeof(struct sockaddr_ssl)); Now I understand what you mean with "socket type conversion". Mmm - That way I can't have a different SSLcontext on incoming connects? Currently I do it after accept, but before I only have the listen descriptor and if accept already does the SSL handshake... Not that it's an important thing to me - I just wonder. > > I'm not a cryptographic expert, but I wouldn't prefer a packet > > encryption over a stream encryption. > > The bloat in the IPSEC case is because of the KAME integration; > the IPSEC in IPv4 eats a context structure, even if it's not > used. The IPv6 code is better. I took it to be a political > statement on IPv4 by the KAME people. 8-). > > The support being in the kernel instead of user space doesn't > make it other than a stream encryption, though. Maybe I should spend more time into IPSEC. > > With STARTTLS you can probe for SSL in MTA - MTA comunications. > > MTAs connect foreign SMTP servers and want to prefer SSL. > > It's unpractical to try a connect to smpts port first with all those > > blackhole firewalls out there. > > Meaning that it won't reject the initial packet, you will have > to time out the connection, and then retry on port 25? > > Actually, you can use non-blocking I/O, simultaneously connect, > and then drop whiever one doesn't pan out, or you could cache > the result, or you etc.. could make it a mailer flag (e.g. > "mailer smtps has flag 'Q' and uses port 465"), and control it > on a peer basis. OK - that's both possible. > It's not really for trusted host communications anyway, since > the only reason we're eating the encruption overhead is to > secure a plaintext password; the that rest of the session is > also encrypted is actually annoying overhead, in most cases. But it's a nice to have feature. It brings encryption of the whole session together with certificates. Certificates are great to deliver safly to dynamic IPs. Together with ETRN this was the first real option to replace rmail only UUCP nodes with SMTP - asuming that IP is available. > > The only downside with STARTTLS is that it makes it allmost impossible > > to use external SSL boxes. > > That's a very big downside. OpenSSL does maybe 200 if you have > a fas box, and you eat the whole thing doing crypto. Not worth > doing onboard the box itself, if you can possibly avoid it. I never saw this as a big point as it was forseeable that one day you can just plug in a crypto card directly in each border mailhost :) -- B.Walter COSMO-Project http://www.cosmo-project.de ticso@cicely.de Usergroup info@cosmo-project.de To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Sun Jul 14 3: 3: 3 2002 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.FreeBSD.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 0DB1337B400 for ; Sun, 14 Jul 2002 03:03:00 -0700 (PDT) Received: from smtp.netcologne.de (smtp.netcologne.de [194.8.194.112]) by mx1.FreeBSD.org (Postfix) with ESMTP id D1F6C43E31 for ; Sun, 14 Jul 2002 03:02:58 -0700 (PDT) (envelope-from tmseck-lists@netcologne.de) Received: from localhost (xdsl-195-14-205-193.netcologne.de [195.14.205.193]) by smtp.netcologne.de (8.12.2/8.12.2) with ESMTP id g6EA2tUt029692 for ; Sun, 14 Jul 2002 12:02:55 +0200 (MEST) Received: (qmail 627 invoked by uid 1001); 14 Jul 2002 09:59:39 -0000 Date: Sun, 14 Jul 2002 11:59:39 +0200 From: Thomas Seck To: freebsd-arch@FreeBSD.ORG Subject: Re: Package system flaws? Message-ID: <20020714095939.GA588@laurel.tmseck.homedns.org> Mail-Followup-To: freebsd-arch@FreeBSD.ORG References: <3D27A296.D58FB4B4@softweyr.com> <20020712121427.GD3678@lummux.tchpc.tcd.ie> <20020712144854.GA756@laurel.tmseck.homedns.org> <20020713054141.A26277@misty.eyesbeyond.com> <20020713011750.GA755@laurel.tmseck.homedns.org> <20020714042237.GD931@lizzy.catnook.com> <20020714042623.GB95460@squall.waterspout.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20020714042623.GB95460@squall.waterspout.com> User-Agent: Mutt/1.4i Organization: private site in Germany X-PGP-KeyID: DF46EE05 X-PGP-Fingerprint: A38F AE66 6B11 6EB9 5D1A B67D 2444 2FE1 DF46 EE05 X-Attribution: tms Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG * Will Andrews (will@csociety.org): > The people who do the work should do it the best way they can. > Or, with a volunteer twist, the most enjoyable way they can. Agreed. But portupgrade(1) is somewhat special, because it is the only way you can keep your locally installed packages up to date without thrashing your pkg db. It thus does the work that the pkg_* tools and the ports system should have been doing. People ask regularly why the tools of the sysutils/portupgrade port are not part of the base system. With them being written in anything else than C{,++}, sed, awk or sh this is impossible. Being Joe Random User I suggest the following: - find someone who is willing to re-implement sysutils/portupgrade in a language present in the base system (i.e. _not_ in Perl). Then - fix the dependency handling whithin the pkg_* tools resp. the ports framework. This will make sysutils/portupgrade hopefully obsolete. - After this has been done, start to think about a new flashy package format hich has all the bells and whistles that are missing today. The whole discussion I read so far seems to do the second or even the third step before the first. Just my 2 Euro cents. -- Thomas Seck To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Sun Jul 14 3:20:19 2002 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.FreeBSD.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 67BF937B400 for ; Sun, 14 Jul 2002 03:20:17 -0700 (PDT) Received: from gw.catspoiler.org (217-ip-163.nccn.net [209.79.217.163]) by mx1.FreeBSD.org (Postfix) with ESMTP id 09AB743E67 for ; Sun, 14 Jul 2002 03:20:14 -0700 (PDT) (envelope-from dl-freebsd@catspoiler.org) Received: from mousie.catspoiler.org (mousie.catspoiler.org [192.168.101.2]) by gw.catspoiler.org (8.12.5/8.12.5) with ESMTP id g6EAK4wr020922; Sun, 14 Jul 2002 03:20:08 -0700 (PDT) (envelope-from dl-freebsd@catspoiler.org) Message-Id: <200207141020.g6EAK4wr020922@gw.catspoiler.org> Date: Sun, 14 Jul 2002 03:20:04 -0700 (PDT) From: Don Lewis Subject: Re: wiring the sysctl output buffer To: phk@critter.freebsd.dk Cc: arch@FreeBSD.ORG In-Reply-To: <56792.1026640649@critter.freebsd.dk> MIME-Version: 1.0 Content-Type: TEXT/plain; charset=us-ascii Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG On 14 Jul, Poul-Henning Kamp wrote: > In message <200207140954.g6E9sBwr020599@gw.catspoiler.org>, Don Lewis writes: >>On 14 Jul, Poul-Henning Kamp wrote: >>> It used to be that sysctl unconditionally wired the output buffer, >>> but that gave rise to a host of other problems. >> >>Anything specific that I should be aware of? > > No, just bad deadlocks and such. > > You may also want to restrict the amount of buffer you pin > if I hand the kernel a 2G buffer for a 4 byte sysctl integer > you wouldn't want to pin it all. Yeah, that could be bad, but the SYSCTL_OUT() in the current API pins the whole user supplied buffer on the first call. My new entry point does the same thing, but allows the caller to do this potentially blocking operation earlier. To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Sun Jul 14 3:30: 6 2002 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.FreeBSD.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id EE1B237B401 for ; Sun, 14 Jul 2002 03:29:58 -0700 (PDT) Received: from critter.freebsd.dk (critter.freebsd.dk [212.242.86.163]) by mx1.FreeBSD.org (Postfix) with ESMTP id 2266443E4A for ; Sun, 14 Jul 2002 03:29:58 -0700 (PDT) (envelope-from phk@critter.freebsd.dk) Received: from critter.freebsd.dk (localhost [127.0.0.1]) by critter.freebsd.dk (8.12.3/8.12.2) with ESMTP id g6EARoXB062434; Sun, 14 Jul 2002 12:27:50 +0200 (CEST) (envelope-from phk@critter.freebsd.dk) To: Don Lewis Cc: arch@FreeBSD.ORG Subject: Re: wiring the sysctl output buffer In-Reply-To: Your message of "Sun, 14 Jul 2002 03:20:04 PDT." <200207141020.g6EAK4wr020922@gw.catspoiler.org> Date: Sun, 14 Jul 2002 12:27:50 +0200 Message-ID: <62432.1026642470@critter.freebsd.dk> From: Poul-Henning Kamp Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG In message <200207141020.g6EAK4wr020922@gw.catspoiler.org>, Don Lewis writes: >On 14 Jul, Poul-Henning Kamp wrote: >> In message <200207140954.g6E9sBwr020599@gw.catspoiler.org>, Don Lewis writes: >>>On 14 Jul, Poul-Henning Kamp wrote: > >>>> It used to be that sysctl unconditionally wired the output buffer, >>>> but that gave rise to a host of other problems. >>> >>>Anything specific that I should be aware of? >> >> No, just bad deadlocks and such. >> >> You may also want to restrict the amount of buffer you pin >> if I hand the kernel a 2G buffer for a 4 byte sysctl integer >> you wouldn't want to pin it all. > >Yeah, that could be bad, but the SYSCTL_OUT() in the current API pins >the whole user supplied buffer on the first call. My new entry point >does the same thing, but allows the caller to do this potentially >blocking operation earlier. I know, but that also affords an opportunity to be smarter (ie: letting the entry point say "I need this much storage"). -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk@FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Sun Jul 14 3:38:18 2002 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.FreeBSD.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 844BC37B400; Sun, 14 Jul 2002 03:38:15 -0700 (PDT) Received: from harrier.mail.pas.earthlink.net (harrier.mail.pas.earthlink.net [207.217.120.12]) by mx1.FreeBSD.org (Postfix) with ESMTP id 2851043E3B; Sun, 14 Jul 2002 03:38:15 -0700 (PDT) (envelope-from tlambert2@mindspring.com) Received: from pool0078.cvx40-bradley.dialup.earthlink.net ([216.244.42.78] helo=mindspring.com) by harrier.mail.pas.earthlink.net with esmtp (Exim 3.33 #1) id 17TglZ-0006Ky-00; Sun, 14 Jul 2002 03:38:13 -0700 Message-ID: <3D315429.4AF6C2D3@mindspring.com> Date: Sun, 14 Jul 2002 03:36:25 -0700 From: Terry Lambert X-Mailer: Mozilla 4.7 [en]C-CCK-MCD {Sony} (Win98; U) X-Accept-Language: en MIME-Version: 1.0 To: ticso@cicely.de Cc: Gregory Neil Shapiro , freebsd-arch@FreeBSD.ORG Subject: Re: Mail subsystem defaults, adding authentication. References: <20020713034725.GB47677@ussenterprise.ufp.org> <3D2FAFB2.E2E9CF36@mindspring.com> <20020713045704.GA49379@ussenterprise.ufp.org> <3D300FD4.7479A8E5@mindspring.com> <15664.47827.844708.151118@monkeyboy.gshapiro.net> <3D30C4DA.22A255A8@mindspring.com> <20020714073559.GY63545@cicely5.cicely.de> <3D3133AA.8387DA1E@mindspring.com> <20020714100108.GA63545@cicely5.cicely.de> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG FWIW: I've taken the response to this off list, as it's drifting pretty far from FreeBSD, unlike "Should we include Cyrus SASL in the base system" or "How might FreeBSD implement SSL in an ideal future" or "If Cyrus SASL is included in the base system, should the senmail included in the base system be linked against it", etc.. -- Terry To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Sun Jul 14 3:46:11 2002 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.FreeBSD.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 1F7C237B400 for ; Sun, 14 Jul 2002 03:46:07 -0700 (PDT) Received: from gw.catspoiler.org (217-ip-163.nccn.net [209.79.217.163]) by mx1.FreeBSD.org (Postfix) with ESMTP id A474C43E31 for ; Sun, 14 Jul 2002 03:46:06 -0700 (PDT) (envelope-from dl-freebsd@catspoiler.org) Received: from mousie.catspoiler.org (mousie.catspoiler.org [192.168.101.2]) by gw.catspoiler.org (8.12.5/8.12.5) with ESMTP id g6EAjvwr020972; Sun, 14 Jul 2002 03:46:01 -0700 (PDT) (envelope-from dl-freebsd@catspoiler.org) Message-Id: <200207141046.g6EAjvwr020972@gw.catspoiler.org> Date: Sun, 14 Jul 2002 03:45:57 -0700 (PDT) From: Don Lewis Subject: Re: wiring the sysctl output buffer To: phk@critter.freebsd.dk Cc: arch@FreeBSD.ORG In-Reply-To: <62432.1026642470@critter.freebsd.dk> MIME-Version: 1.0 Content-Type: TEXT/plain; charset=us-ascii Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG On 14 Jul, Poul-Henning Kamp wrote: > In message <200207141020.g6EAK4wr020922@gw.catspoiler.org>, Don Lewis writes: >>On 14 Jul, Poul-Henning Kamp wrote: >>> You may also want to restrict the amount of buffer you pin >>> if I hand the kernel a 2G buffer for a 4 byte sysctl integer >>> you wouldn't want to pin it all. >> >>Yeah, that could be bad, but the SYSCTL_OUT() in the current API pins >>the whole user supplied buffer on the first call. My new entry point >>does the same thing, but allows the caller to do this potentially >>blocking operation earlier. > > I know, but that also affords an opportunity to be smarter (ie: letting > the entry point say "I need this much storage"). I like this idea, though we'd want to move the new interface to the recommended list. There is also the problem keeping track of how much memory we wired. The current sysctl exit code unwires the entire user supplied buffer. If we only wire a portion, then we would need to add a member to the sysctl_req structure to record how much was wired so that we could unwire that amount on exit. To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Sun Jul 14 4:24:31 2002 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.FreeBSD.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id C9CD537B400 for ; Sun, 14 Jul 2002 04:24:28 -0700 (PDT) Received: from mailout10.sul.t-online.com (mailout10.sul.t-online.com [194.25.134.21]) by mx1.FreeBSD.org (Postfix) with ESMTP id 4771D43E31 for ; Sun, 14 Jul 2002 04:24:28 -0700 (PDT) (envelope-from Alexander@Leidinger.net) Received: from fwd05.sul.t-online.de by mailout10.sul.t-online.com with smtp id 17ThUG-0001bV-00; Sun, 14 Jul 2002 13:24:24 +0200 Received: from Andro-Beta.Leidinger.net (520065502893-0001@[217.229.220.246]) by fmrl05.sul.t-online.com with esmtp id 17ThU6-120osaC; Sun, 14 Jul 2002 13:24:14 +0200 Received: from Magelan.Leidinger.net (Magelan [192.168.1.1]) by Andro-Beta.Leidinger.net (8.11.6/8.11.6) with ESMTP id g6EBO3x07659; Sun, 14 Jul 2002 13:24:03 +0200 (CEST) (envelope-from Alexander@Leidinger.net) Received: from Leidinger.net (netchild@localhost [127.0.0.1]) by Magelan.Leidinger.net (8.12.5/8.12.5) with ESMTP id g6EBNuxQ064288; Sun, 14 Jul 2002 13:23:59 +0200 (CEST) (envelope-from netchild@Leidinger.net) Message-Id: <200207141123.g6EBNuxQ064288@Magelan.Leidinger.net> Date: Sun, 14 Jul 2002 13:23:55 +0200 (CEST) From: Alexander Leidinger Subject: Re: Mail subsystem defaults, adding authentication. To: bicknell@ufp.org Cc: freebsd-arch@FreeBSD.ORG In-Reply-To: <20020714014600.GA70961@ussenterprise.ufp.org> MIME-Version: 1.0 Content-Type: TEXT/plain; charset=us-ascii X-Sender: 520065502893-0001@t-dialin.net Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG On 13 Jul, Leo Bicknell wrote: > Tomorrow I'll write up a better summary with this new info. At > the end of the day it looks like if we add cyrus-sasl, which is > BSD licensed then the default behavior will be unchanged, but it > will be possible through a combination of rc.conf options, running > saslpasswd, and/or running ssl key generation tools to do auth on > a non-encrypted session using challenge response (against sasl > passwords), or do auth against the password file (or any PAM method) > over an ssl session. Thus we could make it as simple as > 'sendmail_auth="unix"' (or pam, or whatever) for an admin to allow > end clients to starttls, auth, and securely send e-mail all with > their existing credential. It would be nice if it would work like the ssh key stuff. If there's no credential at boot time: create one. Bye, Alexander. -- If Bill Gates had a dime for every time a Windows box crashed... ...Oh, wait a minute, he already does. http://www.Leidinger.net Alexander @ Leidinger.net GPG fingerprint = C518 BC70 E67F 143F BE91 3365 79E2 9C60 B006 3FE7 To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Sun Jul 14 6:33:48 2002 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.FreeBSD.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id EA00537B400 for ; Sun, 14 Jul 2002 06:33:46 -0700 (PDT) Received: from whizzo.transsys.com (whizzo.TransSys.COM [144.202.42.10]) by mx1.FreeBSD.org (Postfix) with ESMTP id 4BE3E43E4A for ; Sun, 14 Jul 2002 06:33:46 -0700 (PDT) (envelope-from louie@whizzo.transsys.com) Received: from whizzo.transsys.com (#6@localhost.transsys.com [127.0.0.1]) by whizzo.transsys.com (8.12.5/8.12.5) with ESMTP id g6EDXj0L031673; Sun, 14 Jul 2002 09:33:45 -0400 (EDT) (envelope-from louie@whizzo.transsys.com) Message-Id: <200207141333.g6EDXj0L031673@whizzo.transsys.com> X-Mailer: exmh version 2.5 07/13/2001 with nmh-1.0.4 To: Thomas Seck Cc: freebsd-arch@FreeBSD.ORG X-Image-URL: http://www.transsys.com/louie/images/louie-mail.jpg From: "Louis A. Mamakos" Subject: Re: Package system flaws? References: <3D27A296.D58FB4B4@softweyr.com> <20020712121427.GD3678@lummux.tchpc.tcd.ie> <20020712144854.GA756@laurel.tmseck.homedns.org> <20020713054141.A26277@misty.eyesbeyond.com> <20020713011750.GA755@laurel.tmseck.homedns.org> <20020714042237.GD931@lizzy.catnook.com> <20020714042623.GB95460@squall.waterspout.com> <20020714095939.GA588@laurel.tmseck.homedns.org> In-reply-to: Your message of "Sun, 14 Jul 2002 11:59:39 +0200." <20020714095939.GA588@laurel.tmseck.homedns.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Sun, 14 Jul 2002 09:33:45 -0400 Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG > * Will Andrews (will@csociety.org): > > > The people who do the work should do it the best way they can. > > Or, with a volunteer twist, the most enjoyable way they can. > > Agreed. But portupgrade(1) is somewhat special, because it is the only > way you can keep your locally installed packages up to date without > thrashing your pkg db. It thus does the work that the pkg_* tools and > the ports system should have been doing. People ask regularly why the > tools of the sysutils/portupgrade port are not part of the base system. > With them being written in anything else than C{,++}, sed, awk or sh > this is impossible. If you've decided to install optional software on your system using the ports mechanism, then it doesn't seem too extreme a requirement that you install a port or package to maintain your ports/packages. cvsup isn't in the base system, but we manage to use it to keep both the base system and ports up to date. I suspect the only result of an attempt to re-write sysutils/portupgrade in a different language will be that the current developer of that tool will disappear. I suspect he chose his implementation lanaguge for a reason. Do you want the tool and developer, or a version in awk/sed/C? louie To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Sun Jul 14 8:58:18 2002 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.FreeBSD.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 8F97637B400 for ; Sun, 14 Jul 2002 08:58:15 -0700 (PDT) Received: from smtp.netcologne.de (smtp.netcologne.de [194.8.194.112]) by mx1.FreeBSD.org (Postfix) with ESMTP id 3EFF243E70 for ; Sun, 14 Jul 2002 08:58:14 -0700 (PDT) (envelope-from tmseck-lists@netcologne.de) Received: from localhost (xdsl-195-14-205-132.netcologne.de [195.14.205.132]) by smtp.netcologne.de (8.12.2/8.12.2) with ESMTP id g6EFwAUt002428 for ; Sun, 14 Jul 2002 17:58:11 +0200 (MEST) Received: (qmail 4280 invoked by uid 1001); 14 Jul 2002 15:57:29 -0000 Date: Sun, 14 Jul 2002 17:57:28 +0200 From: Thomas Seck To: freebsd-arch@FreeBSD.ORG Subject: Re: Package system flaws? Message-ID: <20020714155728.GA4237@laurel.tmseck.homedns.org> Mail-Followup-To: freebsd-arch@FreeBSD.ORG References: <20020712121427.GD3678@lummux.tchpc.tcd.ie> <20020712144854.GA756@laurel.tmseck.homedns.org> <20020713054141.A26277@misty.eyesbeyond.com> <20020713011750.GA755@laurel.tmseck.homedns.org> <20020714042237.GD931@lizzy.catnook.com> <20020714042623.GB95460@squall.waterspout.com> <20020714095939.GA588@laurel.tmseck.homedns.org> <200207141333.g6EDXj0L031673@whizzo.transsys.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <200207141333.g6EDXj0L031673@whizzo.transsys.com> User-Agent: Mutt/1.4i Organization: private site in Germany X-PGP-KeyID: DF46EE05 X-PGP-Fingerprint: A38F AE66 6B11 6EB9 5D1A B67D 2444 2FE1 DF46 EE05 X-Attribution: tms Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG * Louis A. Mamakos (louie@TransSys.COM): > If you've decided to install optional software on your system using > the ports mechanism, then it doesn't seem too extreme a requirement > that you install a port or package to maintain your ports/packages. Sorry, I cannot follow this kind of reasoning. The problem with portupgrade is that you _need_ it to correct the flaws of the current pkg_* tools. And the more people start recommending the use of portupgrade to new users, the less likely it is that the real issues with the ports/package system ll ever get fixed. > cvsup isn't in the base system, but we manage to use it to keep both > the base system and ports up to date. What has cvsup got to do with it? You can keep your sources up to date with cvs too. cvsup is designed to be more efficient than cvs. It is not a bandaid like portupgrade. And yes, I do not like the fact, that it is written in Modula 3 instead of C{,++}. > I suspect the only result of an attempt to re-write sysutils/portupgrade > in a different language will be that the current developer of that tool > will disappear. I suspect he chose his implementation lanaguge for a > reason. Do you want the tool and developer, or a version in awk/sed/C? Did you ask knu about it or is this speculation on your part? Again, I did not say that the portupgrade _port_ should be rewritten. I meant that one should re-implement the tools knu wrote and put them into the base system as a short term solution. A mid-term solution would be to correct the issues with the dependency handling within the base system. When this is done, people should think about new ways to pack and transport packages. -- Thomas Seck To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Sun Jul 14 9:24:41 2002 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.FreeBSD.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id A1D6C37B400 for ; Sun, 14 Jul 2002 09:24:37 -0700 (PDT) Received: from whizzo.transsys.com (whizzo.TransSys.COM [144.202.42.10]) by mx1.FreeBSD.org (Postfix) with ESMTP id 03EE343E64 for ; Sun, 14 Jul 2002 09:24:37 -0700 (PDT) (envelope-from louie@whizzo.transsys.com) Received: from whizzo.transsys.com (#6@localhost.transsys.com [127.0.0.1]) by whizzo.transsys.com (8.12.5/8.12.5) with ESMTP id g6EGOa0L033175; Sun, 14 Jul 2002 12:24:36 -0400 (EDT) (envelope-from louie@whizzo.transsys.com) Message-Id: <200207141624.g6EGOa0L033175@whizzo.transsys.com> X-Mailer: exmh version 2.5 07/13/2001 with nmh-1.0.4 To: Thomas Seck Cc: freebsd-arch@FreeBSD.ORG X-Image-URL: http://www.transsys.com/louie/images/louie-mail.jpg From: "Louis A. Mamakos" Subject: Re: Package system flaws? References: <20020712121427.GD3678@lummux.tchpc.tcd.ie> <20020712144854.GA756@laurel.tmseck.homedns.org> <20020713054141.A26277@misty.eyesbeyond.com> <20020713011750.GA755@laurel.tmseck.homedns.org> <20020714042237.GD931@lizzy.catnook.com> <20020714042623.GB95460@squall.waterspout.com> <20020714095939.GA588@laurel.tmseck.homedns.org> <200207141333.g6EDXj0L031673@whizzo.transsys.com> <20020714155728.GA4237@laurel.tmseck.homedns.org> In-reply-to: Your message of "Sun, 14 Jul 2002 17:57:28 +0200." <20020714155728.GA4237@laurel.tmseck.homedns.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Sun, 14 Jul 2002 12:24:36 -0400 Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG > * Louis A. Mamakos (louie@TransSys.COM): > > > If you've decided to install optional software on your system using > > the ports mechanism, then it doesn't seem too extreme a requirement > > that you install a port or package to maintain your ports/packages. > > Sorry, I cannot follow this kind of reasoning. The problem with > portupgrade is that you _need_ it to correct the flaws of the current > pkg_* tools. And the more people start recommending the use of > portupgrade to new users, the less likely it is that the real issues > with the ports/package system ll ever get fixed. And so what's so difficult to understand? Why is it that the only tools "qualified" for use in maintaining the ports on a machine seem to be required to be in the base system? From what I can tell, the direction is to move non-essential stuff out of the base system. Again, I don't see why it's so burdensome to type 'make install' in the /usr/ports/sysutils/portupgrade directory. > > cvsup isn't in the base system, but we manage to use it to keep both > > the base system and ports up to date. > > What has cvsup got to do with it? You can keep your sources up to date > with cvs too. cvsup is designed to be more efficient than cvs. It is not > a bandaid like portupgrade. And yes, I do not like the fact, that it is > written in Modula 3 instead of C{,++}. Those users that don't just install off of CDROM distributions, and try to keep up to date with the -STABLE branch or the HEAD of the CVS tree use cvsup to update their systems. At least when I was involved in running one of the CVS mirrors, the only server running was cvsupd. So there's an even more critical tool, only available as a port or package, in wide use. Why do you care what the implementation language of the tool is if it solves the problem? Shouldn't we be pleased there even exists a tool in the first place? > > I suspect the only result of an attempt to re-write sysutils/portupgrade > > in a different language will be that the current developer of that tool > > will disappear. I suspect he chose his implementation lanaguge for a > > reason. Do you want the tool and developer, or a version in awk/sed/C? > > Did you ask knu about it or is this speculation on your part? Again, I > did not say that the portupgrade _port_ should be rewritten. I meant > that one should re-implement the tools knu wrote and put them into the > base system as a short term solution. A mid-term solution would be to > correct the issues with the dependency handling within the base system. > When this is done, people should think about new ways to pack and > transport packages. I guess no one is stopping you from reimplementing your own wheel, er, tool to replicate the functionality of the one that's already working pretty good. And no, I haven't spoken to knu about his opinion. I based that remark on how I'd react if someone were to come along and say, "I really like your software, but I need you to re-write it in a language that I like for essentially no good reason." louie To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Sun Jul 14 9:42:45 2002 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.FreeBSD.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id B792837B400 for ; Sun, 14 Jul 2002 09:42:43 -0700 (PDT) Received: from w250.z064001178.sjc-ca.dsl.cnc.net (adsl-66.218.45.239.dslextreme.com [66.218.45.239]) by mx1.FreeBSD.org (Postfix) with SMTP id 12ECC43E31 for ; Sun, 14 Jul 2002 09:42:43 -0700 (PDT) (envelope-from jos@catnook.com) Received: (qmail 80547 invoked by uid 1000); 14 Jul 2002 16:43:04 -0000 Date: Sun, 14 Jul 2002 09:42:42 -0701 From: Jos Backus To: freebsd-arch@FreeBSD.ORG Subject: Re: Package system flaws? Message-ID: <20020714164304.GA32774@lizzy.catnook.com> Reply-To: jos@catnook.com Mail-Followup-To: freebsd-arch@FreeBSD.ORG References: <20020712121427.GD3678@lummux.tchpc.tcd.ie> <20020712144854.GA756@laurel.tmseck.homedns.org> <20020713054141.A26277@misty.eyesbeyond.com> <20020713011750.GA755@laurel.tmseck.homedns.org> <20020714042237.GD931@lizzy.catnook.com> <20020714042623.GB95460@squall.waterspout.com> <20020714095939.GA588@laurel.tmseck.homedns.org> <200207141333.g6EDXj0L031673@whizzo.transsys.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <200207141333.g6EDXj0L031673@whizzo.transsys.com> User-Agent: Mutt/1.5.1i Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG On Sun, Jul 14, 2002 at 09:33:45AM -0400, Louis A. Mamakos wrote: > I suspect the only result of an attempt to re-write sysutils/portupgrade > in a different language will be that the current developer of that tool > will disappear. I suspect he chose his implementation lanaguge for a > reason. Do you want the tool and developer, or a version in awk/sed/C? In case it wasn't clear, my previous remark was meant in jest. I would rather have the tool (and the developer). Part of me still thinks it's a pity that we don't have a decent scripting language (such as Ruby) as part of the base OS. Windows has Visual BASIC for Applications - all due respect to the awk creators and maintainers, but surely we can do better than awk/sh? -- Jos Backus _/ _/_/_/ Santa Clara, CA _/ _/ _/ _/ _/_/_/ _/ _/ _/ _/ jos@catnook.com _/_/ _/_/_/ require 'std/disclaimer' To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Sun Jul 14 11:14:50 2002 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.FreeBSD.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id B406837B400 for ; Sun, 14 Jul 2002 11:14:45 -0700 (PDT) Received: from smtp.netcologne.de (smtp.netcologne.de [194.8.194.112]) by mx1.FreeBSD.org (Postfix) with ESMTP id DB7F643E31 for ; Sun, 14 Jul 2002 11:14:44 -0700 (PDT) (envelope-from tmseck-lists@netcologne.de) Received: from localhost (xdsl-213-168-117-153.netcologne.de [213.168.117.153]) by smtp.netcologne.de (8.12.2/8.12.2) with ESMTP id g6EIEcUt028563 for ; Sun, 14 Jul 2002 20:14:39 +0200 (MEST) Received: (qmail 471 invoked by uid 1001); 14 Jul 2002 18:09:33 -0000 Date: Sun, 14 Jul 2002 20:09:33 +0200 From: Thomas Seck To: freebsd-arch@FreeBSD.ORG Subject: Re: Package system flaws? Message-ID: <20020714180933.GA420@laurel.tmseck.homedns.org> Mail-Followup-To: freebsd-arch@FreeBSD.ORG References: <20020712121427.GD3678@lummux.tchpc.tcd.ie> <20020712144854.GA756@laurel.tmseck.homedns.org> <20020713054141.A26277@misty.eyesbeyond.com> <20020713011750.GA755@laurel.tmseck.homedns.org> <20020714042237.GD931@lizzy.catnook.com> <20020714042623.GB95460@squall.waterspout.com> <20020714095939.GA588@laurel.tmseck.homedns.org> <200207141333.g6EDXj0L031673@whizzo.transsys.com> <20020714155728.GA4237@laurel.tmseck.homedns.org> <200207141624.g6EGOa0L033175@whizzo.transsys.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <200207141624.g6EGOa0L033175@whizzo.transsys.com> User-Agent: Mutt/1.4i Organization: private site in Germany X-PGP-KeyID: DF46EE05 X-PGP-Fingerprint: A38F AE66 6B11 6EB9 5D1A B67D 2444 2FE1 DF46 EE05 X-Attribution: tms Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG * Louis A. Mamakos (louie@TransSys.COM): > > * Louis A. Mamakos (louie@TransSys.COM): > > > > > If you've decided to install optional software on your system using > > > the ports mechanism, then it doesn't seem too extreme a requirement > > > that you install a port or package to maintain your ports/packages. > > > > Sorry, I cannot follow this kind of reasoning. The problem with > > portupgrade is that you _need_ it to correct the flaws of the current > > pkg_* tools. And the more people start recommending the use of > > portupgrade to new users, the less likely it is that the real issues > > with the ports/package system ll ever get fixed. > > And so what's so difficult to understand? Why is it that the only > tools "qualified" for use in maintaining the ports on a machine seem > to be required to be in the base system? From what I can tell, the > direction is to move non-essential stuff out of the base system. A consistent package db is something I consider a vital part of the system. If we had tools in the base system which could maintain it, using portupgrade would simply be a matter of taste. In the current state of affairs, I am forced to use it. And I am forced to install a scripting environment I do not want. And all because of design flaws in the ports and package system. And remember: This thread is about package system flaws. > Again, I don't see why it's so burdensome to type 'make install' in > the /usr/ports/sysutils/portupgrade directory. I am sorry that I am unable to make you understand my point. > > What has cvsup got to do with it? You can keep your sources up to date > > with cvs too. cvsup is designed to be more efficient than cvs. It is not > > a bandaid like portupgrade. And yes, I do not like the fact, that it is > > written in Modula 3 instead of C{,++}. > > Those users that don't just install off of CDROM distributions, and > try to keep up to date with the -STABLE branch or the HEAD of the CVS > tree use cvsup to update their systems. No. They are told that they should do so. They could as well use cvs - it would just be less efficient. You can even do binary updates if you like. > Why do you care what the implementation language of the tool is if it > solves the problem? Shouldn't we be pleased there even exists a tool > in the first place? Again: My point is, that until portupgrade appeared on the scene, there was no way to keep the package db consistent. This issue needs to be addressed. portupgrade is a bandaid that should not have been necessary in the first place. Period. > I guess no one is stopping you from reimplementing your own wheel, er, > tool to replicate the functionality of the one that's already working > pretty good. I do not want to reimplement the wheel. Do not over interpret my messages. > And no, I haven't spoken to knu about his opinion. I based that remark > on how I'd react if someone were to come along and say, "I really like > your software, but I need you to re-write it in a language that I like > for essentially no good reason." This is not a matter of "taste" and personal dislikes. It is simply a matter of how many dependencies you have to track to fulfill a task. You can read all I have to say about this topic in <20020713011750.GA755@laurel.tmseck.homedns.org>. I will not waste any more bandwith in repeating this over and over again. -- Thomas Seck To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Sun Jul 14 11:14:54 2002 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.FreeBSD.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 77BCB37B401 for ; Sun, 14 Jul 2002 11:14:46 -0700 (PDT) Received: from smtp.netcologne.de (smtp.netcologne.de [194.8.194.112]) by mx1.FreeBSD.org (Postfix) with ESMTP id 817B643E42 for ; Sun, 14 Jul 2002 11:14:45 -0700 (PDT) (envelope-from tmseck-lists@netcologne.de) Received: from localhost (xdsl-213-168-117-153.netcologne.de [213.168.117.153]) by smtp.netcologne.de (8.12.2/8.12.2) with ESMTP id g6EIEcUv028563 for ; Sun, 14 Jul 2002 20:14:42 +0200 (MEST) Received: (qmail 482 invoked by uid 1001); 14 Jul 2002 18:14:08 -0000 Date: Sun, 14 Jul 2002 20:14:08 +0200 From: Thomas Seck To: freebsd-arch@FreeBSD.ORG Subject: Re: Package system flaws? Message-ID: <20020714181408.GB420@laurel.tmseck.homedns.org> Mail-Followup-To: freebsd-arch@FreeBSD.ORG References: <20020712121427.GD3678@lummux.tchpc.tcd.ie> <20020712144854.GA756@laurel.tmseck.homedns.org> <20020713054141.A26277@misty.eyesbeyond.com> <20020713011750.GA755@laurel.tmseck.homedns.org> <20020714042237.GD931@lizzy.catnook.com> <20020714042623.GB95460@squall.waterspout.com> <20020714095939.GA588@laurel.tmseck.homedns.org> <200207141333.g6EDXj0L031673@whizzo.transsys.com> <20020714164304.GA32774@lizzy.catnook.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20020714164304.GA32774@lizzy.catnook.com> User-Agent: Mutt/1.4i Organization: private site in Germany X-PGP-KeyID: DF46EE05 X-PGP-Fingerprint: A38F AE66 6B11 6EB9 5D1A B67D 2444 2FE1 DF46 EE05 X-Attribution: tms Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG * Jos Backus (jos@catnook.com): > On Sun, Jul 14, 2002 at 09:33:45AM -0400, Louis A. Mamakos wrote: > > I suspect the only result of an attempt to re-write sysutils/portupgrade > > in a different language will be that the current developer of that tool > > will disappear. I suspect he chose his implementation lanaguge for a > > reason. Do you want the tool and developer, or a version in awk/sed/C? > > In case it wasn't clear, my previous remark was meant in jest. I would rather > have the tool (and the developer). > > Part of me still thinks it's a pity that we don't have a decent scripting > language (such as Ruby) as part of the base OS. Windows has Visual BASIC for > Applications - all due respect to the awk creators and maintainers, but surely > we can do better than awk/sh? I think it is good not to have everyones' favourite scripting language in the base system. And VBA is not part of Windows(tm) by the way. No one stops you from doing (cd /usr/ports/lang/ruby && make install). -- Thomas Seck To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Sun Jul 14 11:58:52 2002 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.FreeBSD.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id EF01B37B400; Sun, 14 Jul 2002 11:58:46 -0700 (PDT) Received: from iguana.icir.org (iguana.icir.org [192.150.187.36]) by mx1.FreeBSD.org (Postfix) with ESMTP id 9D02D43E5E; Sun, 14 Jul 2002 11:58:46 -0700 (PDT) (envelope-from rizzo@iguana.icir.org) Received: (from rizzo@localhost) by iguana.icir.org (8.11.6/8.11.3) id g6EIwka78526; Sun, 14 Jul 2002 11:58:46 -0700 (PDT) (envelope-from rizzo) Date: Sun, 14 Jul 2002 11:58:46 -0700 From: Luigi Rizzo To: arch@freebsd.org Subject: reintroduce handling of NULL route in ip_output() Message-ID: <20020714115846.C78146@iguana.icir.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2.5.1i Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG [discussion on -arch, Bcc to -net in case someone is interested there] Hi, I would like to reintroduce the ability to pass a NULL route to p_output(), which has been possible up to rev.1.34 of ip_output(). Making ip_output() unable to handle NULL routes only saved a single "if (ro == NULL)" check, but caused the work of allocating/deallocating the route to be replicated in multiple places in the clients of ip_output(), and the way this is done is very inconsistent, sometimes involving the use of static variables which (apart from potential errors due to misuse) are a no-no if we ever want to go to fine-grained locking in the network stack. See for example how the route to ip_output is passed in netinet/igmp.c netinet/ip_mroute.c netinet/ip_input.c netinet/tcp_subr.c at least in one place there is a "bug" (like resetting some fields of an automatic variable right before exiting from the block). Other usages look suspicious too. My reason to re-allow NULL routes in ip_output() are: * simplify the code in the callers by centralizing a common function; * make another step towards the removal of global variables in the protocol stack; Does any of you have strong and _motivated_ objections ? thanks luigi To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Sun Jul 14 14:12:43 2002 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.FreeBSD.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 253E237B400 for ; Sun, 14 Jul 2002 14:12:40 -0700 (PDT) Received: from 12-234-90-219.client.attbi.com (12-234-90-219.client.attbi.com [12.234.90.219]) by mx1.FreeBSD.org (Postfix) with ESMTP id 33D5643E6A for ; Sun, 14 Jul 2002 14:12:39 -0700 (PDT) (envelope-from DougB@FreeBSD.org) Received: from FreeBSD.org (master.gorean.org [10.0.0.2]) by 12-234-90-219.client.attbi.com (8.12.3/8.12.3) with ESMTP id g6ELCaBu031792; Sun, 14 Jul 2002 14:12:37 -0700 (PDT) (envelope-from DougB@FreeBSD.org) Message-ID: <3D31E944.A8E523E6@FreeBSD.org> Date: Sun, 14 Jul 2002 14:12:36 -0700 From: Doug Barton Organization: Triborough Bridge & Tunnel Authority X-Mailer: Mozilla 4.79 [en] (X11; U; FreeBSD 4.6-RELEASE i386) X-Accept-Language: en MIME-Version: 1.0 To: "Louis A. Mamakos" Cc: Thomas Seck , freebsd-arch@FreeBSD.org Subject: Re: Package system flaws? References: <3D27A296.D58FB4B4@softweyr.com> <20020712121427.GD3678@lummux.tchpc.tcd.ie> <20020712144854.GA756@laurel.tmseck.homedns.org> <20020713054141.A26277@misty.eyesbeyond.com> <20020713011750.GA755@laurel.tmseck.homedns.org> <20020714042237.GD931@lizzy.catnook.com> <20020714042623.GB95460@squall.waterspout.com> <20020714095939.GA588@laurel.tmseck.homedns.org> <200207141333.g6EDXj0L031673@whizzo.transsys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG "Louis A. Mamakos" wrote: > If you've decided to install optional software on your system using > the ports mechanism, then it doesn't seem too extreme a requirement > that you install a port or package to maintain your ports/packages. I think this is an excellent point. My only concern is that as we move to an increasingly modularized system, we want the tools that manage that system to be production quality, whether they are in the base, or not. Therefore, I think it would be reasonable to echo the request that someone implement the functionality of portupgrade in the pkg_ tools. Personally, I've never seen what was so difficult about managing package updates, but then again, I'm not overly concerned about updating the dependency lists when I do it by hand. Doug -- "We have known freedom's price. We have shown freedom's power. And in this great conflict, ... we will see freedom's victory." - George W. Bush, President of the United States State of the Union, January 28, 2002 Do YOU Yahoo!? To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Sun Jul 14 14:20:49 2002 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.FreeBSD.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id DF8DC37B400 for ; Sun, 14 Jul 2002 14:20:47 -0700 (PDT) Received: from 12-234-90-219.client.attbi.com (12-234-90-219.client.attbi.com [12.234.90.219]) by mx1.FreeBSD.org (Postfix) with ESMTP id DE05643E65 for ; Sun, 14 Jul 2002 14:20:46 -0700 (PDT) (envelope-from DougB@FreeBSD.org) Received: from FreeBSD.org (master.gorean.org [10.0.0.2]) by 12-234-90-219.client.attbi.com (8.12.3/8.12.3) with ESMTP id g6ELKeBu031821; Sun, 14 Jul 2002 14:20:46 -0700 (PDT) (envelope-from DougB@FreeBSD.org) Message-ID: <3D31EB28.5D21E654@FreeBSD.org> Date: Sun, 14 Jul 2002 14:20:40 -0700 From: Doug Barton Organization: Triborough Bridge & Tunnel Authority X-Mailer: Mozilla 4.79 [en] (X11; U; FreeBSD 4.6-RELEASE i386) X-Accept-Language: en MIME-Version: 1.0 To: Mark Valentine Cc: arch@FreeBSD.org Subject: Re: Package system flaws? References: <200207081430.g68EUgoZ062947@dotar.thuvia.org> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG Mark Valentine wrote: > 1. Packages install to /usr/local by default; Yes, and it's incredibly likely that they always will. So, you have two alternatives. First, you can set PREFIX to be something else in your /etc/make.conf file, and work with us on fixing that functionality when it breaks. Second, you can set a different directory to be your "really local" directory, and set your "long-standing (since 4.2BSD) cross-platform administrative policies" any way you want. Asking to change the default is totally non-productive at this stage of the game. > 2. I'd like to see improved support for building/installing packages > as non-root. I agree that this is a "nice to have." -- "We have known freedom's price. We have shown freedom's power. And in this great conflict, ... we will see freedom's victory." - George W. Bush, President of the United States State of the Union, January 28, 2002 Do YOU Yahoo!? To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Sun Jul 14 14:46:37 2002 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.FreeBSD.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 7E85937B400 for ; Sun, 14 Jul 2002 14:46:35 -0700 (PDT) Received: from smtp.noos.fr (descartes.noos.net [212.198.2.74]) by mx1.FreeBSD.org (Postfix) with ESMTP id 6249B43E58 for ; Sun, 14 Jul 2002 14:46:34 -0700 (PDT) (envelope-from root@gits.dyndns.org) Received: (qmail 51322372 invoked by uid 0); 14 Jul 2002 21:46:32 -0000 Received: from unknown (HELO gits.gits.dyndns.org) ([212.198.229.153]) (envelope-sender ) by 212.198.2.74 (qmail-ldap-1.03) with SMTP for ; 14 Jul 2002 21:46:32 -0000 Received: from gits.gits.dyndns.org (n3m54hd69a3tcaew@localhost [127.0.0.1]) by gits.gits.dyndns.org (8.12.5/8.12.5) with ESMTP id g6ELkVie001587; Sun, 14 Jul 2002 23:46:32 +0200 (CEST) (envelope-from root@gits.dyndns.org) Received: (from root@localhost) by gits.gits.dyndns.org (8.12.5/8.12.5/Submit) id g6ELkVhe001586; Sun, 14 Jul 2002 23:46:31 +0200 (CEST) (envelope-from root) Date: Sun, 14 Jul 2002 23:46:30 +0200 From: Cyrille Lefevre To: "Louis A. Mamakos" Cc: Thomas Seck , freebsd-arch@FreeBSD.ORG Subject: Re: Package system flaws? Message-ID: <20020714214630.GA1455@gits.dyndns.org> Mail-Followup-To: Cyrille Lefevre , "Louis A. Mamakos" , Thomas Seck , freebsd-arch@FreeBSD.ORG References: <20020712121427.GD3678@lummux.tchpc.tcd.ie> <20020712144854.GA756@laurel.tmseck.homedns.org> <20020713054141.A26277@misty.eyesbeyond.com> <20020713011750.GA755@laurel.tmseck.homedns.org> <20020714042237.GD931@lizzy.catnook.com> <20020714042623.GB95460@squall.waterspout.com> <20020714095939.GA588@laurel.tmseck.homedns.org> <200207141333.g6EDXj0L031673@whizzo.transsys.com> <20020714155728.GA4237@laurel.tmseck.homedns.org> <200207141624.g6EGOa0L033175@whizzo.transsys.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <200207141624.g6EGOa0L033175@whizzo.transsys.com> User-Agent: Mutt/1.3.99i Organization: ACME X-Face: V|+c;4!|B?E%BE^{E6);aI.[< List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG On Sun, Jul 14, 2002 at 12:24:36PM -0400, Louis A. Mamakos wrote: [snip] > Why do you care what the implementation language of the tool is if it > solves the problem? Shouldn't we be pleased there even exists a tool the answer is simple, `portupgrade' is written in `ruby' and `ruby' isn't part of the base system, so, `portupgrade' can't be part in the base system 'til it is rewritten in another language available in the base system. PS : I don't use `portupgrade' but home made tools written in `ksh'. Cyrille. -- Cyrille Lefevre mailto:cyrille.lefevre@laposte.net To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Sun Jul 14 14:51:12 2002 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.FreeBSD.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 972D937B400 for ; Sun, 14 Jul 2002 14:51:10 -0700 (PDT) Received: from smtp.netcologne.de (smtp.netcologne.de [194.8.194.112]) by mx1.FreeBSD.org (Postfix) with ESMTP id 5F5AE43E3B for ; Sun, 14 Jul 2002 14:51:09 -0700 (PDT) (envelope-from tmseck-lists@netcologne.de) Received: from localhost (xdsl-213-168-110-15.netcologne.de [213.168.110.15]) by smtp.netcologne.de (8.12.2/8.12.2) with ESMTP id g6ELp3Ut008909 for ; Sun, 14 Jul 2002 23:51:04 +0200 (MEST) Received: (qmail 1290 invoked by uid 1001); 14 Jul 2002 21:49:59 -0000 Date: Sun, 14 Jul 2002 23:49:58 +0200 From: Thomas Seck To: freebsd-arch@FreeBSD.ORG Subject: Re: Package system flaws? Message-ID: <20020714214958.GA1228@laurel.tmseck.homedns.org> Mail-Followup-To: freebsd-arch@FreeBSD.ORG References: <20020712121427.GD3678@lummux.tchpc.tcd.ie> <20020712144854.GA756@laurel.tmseck.homedns.org> <20020713054141.A26277@misty.eyesbeyond.com> <20020713011750.GA755@laurel.tmseck.homedns.org> <20020714042237.GD931@lizzy.catnook.com> <20020714042623.GB95460@squall.waterspout.com> <20020714095939.GA588@laurel.tmseck.homedns.org> <200207141333.g6EDXj0L031673@whizzo.transsys.com> <3D31E944.A8E523E6@FreeBSD.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <3D31E944.A8E523E6@FreeBSD.org> User-Agent: Mutt/1.4i Organization: private site in Germany X-PGP-KeyID: DF46EE05 X-PGP-Fingerprint: A38F AE66 6B11 6EB9 5D1A B67D 2444 2FE1 DF46 EE05 X-Attribution: tms Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG * Doug Barton (DougB@FreeBSD.ORG): > "Louis A. Mamakos" wrote: > > > If you've decided to install optional software on your system using > > the ports mechanism, then it doesn't seem too extreme a requirement > > that you install a port or package to maintain your ports/packages. > > I think this is an excellent point. Normally I would second that but IMHO portupgrade(1) is dangerous in a way that it is a fine bandaid for the flaws in the current implementation of the package system. There is no urge for the developers to address these issues because "we have portupgrade". I do not fight a religious war against portupgrade(1), I use it myself. But I wish I would not have to. That is my point. > Personally, I've never seen what was so difficult about managing package > updates, but then again, I'm not overly concerned about updating the > dependency lists when I do it by hand. You are able to do so. I am probably too if I wanted to. But think of the people who just start using FreeBSD. Let them install packages at their will for a year and they will have rendered their package db unusable. Personally I think it rather unprofessional to tell them on -questions or -stable "hey, do not rely on our tools, use portupgrade instead". Remember, one of our marketing arguments is the ports collection. And honestly, I have better things to do than repairing a package database by hand, especially when it comes to things like Gnome or KDE. As I said before: Before thinking about new package formats, the current issues should be addressed. Just my 2 Euro cents from a users perspective. -- Thomas Seck To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Sun Jul 14 14:56:13 2002 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.FreeBSD.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 3D77337B487; Sun, 14 Jul 2002 14:56:05 -0700 (PDT) Received: from thuvia.demon.co.uk (thuvia.demon.co.uk [193.237.34.248]) by mx1.FreeBSD.org (Postfix) with ESMTP id 4134343E42; Sun, 14 Jul 2002 14:56:03 -0700 (PDT) (envelope-from mark@thuvia.demon.co.uk) Received: from dotar.thuvia.org (dotar.thuvia.org [10.0.0.4]) by phaidor.thuvia.org (8.11.6/8.11.6) with ESMTP id g6ELu1b45587; Sun, 14 Jul 2002 22:56:01 +0100 (BST) (envelope-from mark@thuvia.demon.co.uk) Received: from dotar.thuvia.org (localhost [IPv6:::1]) by dotar.thuvia.org (8.12.3/8.12.3) with ESMTP id g6ELu0KC080284; Sun, 14 Jul 2002 22:56:00 +0100 (BST) (envelope-from mark@dotar.thuvia.org) Received: (from mark@localhost) by dotar.thuvia.org (8.12.3/8.12.3/Submit) id g6ELu0E4080283; Sun, 14 Jul 2002 22:56:00 +0100 (BST) Date: Sun, 14 Jul 2002 22:56:00 +0100 (BST) From: Mark Valentine Message-Id: <200207142156.g6ELu0E4080283@dotar.thuvia.org> In-Reply-To: Doug Barton's message of Jul 14, 2:20pm X-Mailer: Mail User's Shell (7.2.6 beta(5) 10/07/98) To: Doug Barton Subject: Re: Package system flaws? Cc: arch@freebsd.org Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG > From: Doug Barton > Date: Sun 14 Jul, 2002 > Subject: Re: Package system flaws? > Mark Valentine wrote: > > 1. Packages install to /usr/local by default; > > Yes, and it's incredibly likely that they always will. Unfortunately there are too many people who don't seem to see the bigger picture. :-( > First, you can set PREFIX to be something else in your > /etc/make.conf file, Doesn't work for packages, only ports. Hence my interest in the recent discussion of making install-time relocation work. > Second, you can set a different directory to be your "really > local" directory, and set your "long-standing (since 4.2BSD) > cross-platform administrative policies" any way you want. Right, change the world because of a single OS vendor's mistake. I then have to deal with conflicts with systems and software which _want_ you to use /usr/local for this purpose. I hate kludges and inconsistencies. My workaround to date has been to steer clear of FreeBSD packages (and while I was at it I avoided ports too, since if I'm already building from source I may as well retain my own policies). Unfortunately, it would be unreasonable for me to suggest my clients do the same, so I am forced to support a different strategy (in this case, if you can't beat 'em, join 'em - everything in /usr/local is a package, and I now have an additional [non-standard] place where local policy is maintained). Maybe one day I'll get used to it enough to migrate my main server. > Asking to > change the default is totally non-productive at this stage of the game. I'm an idealist who believes that mistakes can usually be corrected, given a bit of care with the transition. Cheers, Mark. -- Mark Valentine, Thuvia Labs "Tigers will do ANYTHING for a tuna fish sandwich." Mark Valentine uses "We're kind of stupid that way." *munch* *munch* and endorses FreeBSD -- To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Sun Jul 14 15: 8:31 2002 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.FreeBSD.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id E8EF737B400 for ; Sun, 14 Jul 2002 15:08:25 -0700 (PDT) Received: from mail.rpi.edu (mail.rpi.edu [128.113.22.40]) by mx1.FreeBSD.org (Postfix) with ESMTP id 183D043E8A for ; Sun, 14 Jul 2002 15:06:04 -0700 (PDT) (envelope-from drosih@rpi.edu) Received: from [128.113.24.47] (gilead.netel.rpi.edu [128.113.24.47]) by mail.rpi.edu (8.12.1/8.12.1) with ESMTP id g6EM5sfc104770; Sun, 14 Jul 2002 18:05:54 -0400 Mime-Version: 1.0 X-Sender: drosih@mail.rpi.edu Message-Id: In-Reply-To: <20020714095939.GA588@laurel.tmseck.homedns.org> References: <3D27A296.D58FB4B4@softweyr.com> <20020712121427.GD3678@lummux.tchpc.tcd.ie> <20020712144854.GA756@laurel.tmseck.homedns.org> <20020713054141.A26277@misty.eyesbeyond.com> <20020713011750.GA755@laurel.tmseck.homedns.org> <20020714042237.GD931@lizzy.catnook.com> <20020714042623.GB95460@squall.waterspout.com> <20020714095939.GA588@laurel.tmseck.homedns.org> Date: Sun, 14 Jul 2002 18:05:53 -0400 To: Thomas Seck , freebsd-arch@FreeBSD.ORG From: Garance A Drosihn Subject: Re: Package system flaws? Content-Type: text/plain; charset="us-ascii" ; format="flowed" X-Scanned-By: MIMEDefang 2.3 (www dot roaringpenguin dot com slash mimedefang) Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG At 11:59 AM +0200 7/14/02, Thomas Seck wrote: >* Will Andrews (will@csociety.org): > >> The people who do the work should do it the best way they can. >> Or, with a volunteer twist, the most enjoyable way they can. > >Agreed. But portupgrade(1) is somewhat special, because it is >the only way you can keep your locally installed packages up >to date without thrashing your pkg db. "If you write it, they will come". If a superior solution shows up in C, people will be happy to use it. People are happy to use portupgrade because it is better than not using it. >Being Joe Random User I suggest the following: > > - find someone who is willing to re-implement sysutils/portupgrade > in a language present in the base system (i.e. _not_ in Perl). > Then > > - fix the dependency handling whithin the pkg_* tools resp. > the ports framework. This will make sysutils/portupgrade > hopefully obsolete. > > - After this has been done, start to think about a new flashy > package format hich has all the bells and whistles that are > missing today. > >The whole discussion I read so far seems to do the second or >even the third step before the first. Portupgrade has been improving rapidly over the last year or so, and that's been because we were lucky enough to have someone who was interested in making continuous improvements to it. I can think of areas where it could use more improvement, so that is what I comment on. The fact that it is written in ruby has not caused me any problems so far, so I do not care about that aspect of it. I guess what I'm saying is that the above priority list is your priority list. I want to talk about "step 2" first, because it is not "step 2" on my priority list, it is "step 1". The fact that so few people are talking about your "step 1" is merely a strong hint that most people do not have your list of priorities. -- Garance Alistair Drosehn = gad@gilead.netel.rpi.edu Senior Systems Programmer or gad@freebsd.org Rensselaer Polytechnic Institute or drosih@rpi.edu To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Sun Jul 14 15:16:59 2002 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.FreeBSD.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 2858037B400 for ; Sun, 14 Jul 2002 15:16:56 -0700 (PDT) Received: from gull.mail.pas.earthlink.net (gull.mail.pas.earthlink.net [207.217.120.84]) by mx1.FreeBSD.org (Postfix) with ESMTP id 9900643E4A for ; Sun, 14 Jul 2002 15:16:55 -0700 (PDT) (envelope-from tlambert2@mindspring.com) Received: from pool0036.cvx21-bradley.dialup.earthlink.net ([209.179.192.36] helo=mindspring.com) by gull.mail.pas.earthlink.net with esmtp (Exim 3.33 #1) id 17TrfW-0007Zn-00; Sun, 14 Jul 2002 15:16:43 -0700 Message-ID: <3D31F81E.290289FD@mindspring.com> Date: Sun, 14 Jul 2002 15:15:58 -0700 From: Terry Lambert X-Mailer: Mozilla 4.79 [en]C-CCK-MCD {Sony} (Win98; U) X-Accept-Language: en MIME-Version: 1.0 To: "Louis A. Mamakos" Cc: Thomas Seck , freebsd-arch@FreeBSD.ORG Subject: Re: Package system flaws? References: <20020712121427.GD3678@lummux.tchpc.tcd.ie> <20020712144854.GA756@laurel.tmseck.homedns.org> <20020713054141.A26277@misty.eyesbeyond.com> <20020713011750.GA755@laurel.tmseck.homedns.org> <20020714042237.GD931@lizzy.catnook.com> <20020714042623.GB95460@squall.waterspout.com> <20020714095939.GA588@laurel.tmseck.homedns.org> <200207141333.g6EDXj0L031673@whizzo.transsys.com> <20020714155728.GA4237@laurel.tmseck.homedns.org> <200207141624.g6EGOa0L033175@whizzo.transsys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG "Louis A. Mamakos" wrote: > And so what's so difficult to understand? Why is it that the only > tools "qualified" for use in maintaining the ports on a machine seem > to be required to be in the base system? From what I can tell, the > direction is to move non-essential stuff out of the base system. Because *in order to* move stuff out of the base system as transparently as possible, the base system itself needs to be under the purview of the package management system. In other words, the difference between a "PicoBSD" and a standard minimal installation and a full installation should come down to two things: 1) What shows up when you list installed components 2) What configuration data is mandatory > Again, I don't see why it's so burdensome to type 'make install' in > the /usr/ports/sysutils/portupgrade directory. It doesn't solve some problems that people would like to see solved for them. 8-). [ ... ] > Why do you care what the implementation language of the tool is if it > solves the problem? Shouldn't we be pleased there even exists a tool > in the first place? The implementation language of any tool is irrelevent, so long as the tool works, and so long as it can be maintained if the person who wrote/maintains it get hits on the head by a falling Sputnik. -- Terry To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Sun Jul 14 15:41: 4 2002 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.FreeBSD.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 7E9BB37B400 for ; Sun, 14 Jul 2002 15:41:01 -0700 (PDT) Received: from smtp.netcologne.de (smtp.netcologne.de [194.8.194.112]) by mx1.FreeBSD.org (Postfix) with ESMTP id 6E7EE43E42 for ; Sun, 14 Jul 2002 15:41:00 -0700 (PDT) (envelope-from tmseck-lists@netcologne.de) Received: from localhost (xdsl-213-168-117-34.netcologne.de [213.168.117.34]) by smtp.netcologne.de (8.12.2/8.12.2) with ESMTP id g6EMeuUt014116 for ; Mon, 15 Jul 2002 00:40:56 +0200 (MEST) Received: (qmail 1499 invoked by uid 1001); 14 Jul 2002 22:37:48 -0000 Date: Mon, 15 Jul 2002 00:37:48 +0200 From: Thomas Seck To: freebsd-arch@FreeBSD.ORG Subject: Re: Package system flaws? Message-ID: <20020714223748.GA1460@laurel.tmseck.homedns.org> Mail-Followup-To: freebsd-arch@FreeBSD.ORG References: <20020712121427.GD3678@lummux.tchpc.tcd.ie> <20020712144854.GA756@laurel.tmseck.homedns.org> <20020713054141.A26277@misty.eyesbeyond.com> <20020713011750.GA755@laurel.tmseck.homedns.org> <20020714042237.GD931@lizzy.catnook.com> <20020714042623.GB95460@squall.waterspout.com> <20020714095939.GA588@laurel.tmseck.homedns.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.4i Organization: private site in Germany X-PGP-KeyID: DF46EE05 X-PGP-Fingerprint: A38F AE66 6B11 6EB9 5D1A B67D 2444 2FE1 DF46 EE05 X-Attribution: tms Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG * Garance A Drosihn (drosih@rpi.edu): > At 11:59 AM +0200 7/14/02, Thomas Seck wrote: > >Agreed. But portupgrade(1) is somewhat special, because it is > >the only way you can keep your locally installed packages up > >to date without thrashing your pkg db. > > "If you write it, they will come". If a superior solution shows > up in C, people will be happy to use it. People are happy to > use portupgrade because it is better than not using it. "You want it fixed, you fix it", I know. I would like to see portupgrade obsoleted in the near future. [...] > >The whole discussion I read so far seems to do the second or > >even the third step before the first. > > Portupgrade has been improving rapidly over the last year or so, > and that's been because we were lucky enough to have someone who > was interested in making continuous improvements to it. I can > think of areas where it could use more improvement, so that is > what I comment on. The fact that it is written in ruby has not > caused me any problems so far, so I do not care about that aspect > of it. Please do not get me wrong: I do appreciate the work knu has invested and the quality of the tools in the sysutils/portupgrade suite is high. > I guess what I'm saying is that the above priority list is your > priority list. I want to talk about "step 2" first, because it > is not "step 2" on my priority list, it is "step 1". The fact > that so few people are talking about your "step 1" is merely a > strong hint that most people do not have your list of priorities. I meant it literally, step 2 on my list was meant to be the "first step". As a short term solution, "step 0.5" would be to import knu's tools into the base system until a better solution is found or the pkg_* tools are rewritten - this would mean that they had to be rewritten. That was what I meant with the first item on my list. -- Thomas Seck To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Sun Jul 14 15:58:49 2002 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.FreeBSD.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 145DE37B400; Sun, 14 Jul 2002 15:58:47 -0700 (PDT) Received: from mail.rpi.edu (mail.rpi.edu [128.113.22.40]) by mx1.FreeBSD.org (Postfix) with ESMTP id 6FE2143E3B; Sun, 14 Jul 2002 15:58:46 -0700 (PDT) (envelope-from drosih@rpi.edu) Received: from [128.113.24.47] (gilead.netel.rpi.edu [128.113.24.47]) by mail.rpi.edu (8.12.1/8.12.1) with ESMTP id g6EMwifc180832; Sun, 14 Jul 2002 18:58:44 -0400 Mime-Version: 1.0 X-Sender: drosih@mail.rpi.edu Message-Id: In-Reply-To: <200207142156.g6ELu0E4080283@dotar.thuvia.org> References: <200207142156.g6ELu0E4080283@dotar.thuvia.org> Date: Sun, 14 Jul 2002 18:58:43 -0400 To: Mark Valentine , Doug Barton From: Garance A Drosihn Subject: Re: Package system flaws? Cc: arch@FreeBSD.ORG Content-Type: text/plain; charset="us-ascii" ; format="flowed" X-Scanned-By: MIMEDefang 2.3 (www dot roaringpenguin dot com slash mimedefang) Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG At 10:56 PM +0100 7/14/02, Mark Valentine wrote: >someone (Doug Barton ?) wrote: > > First, you can set PREFIX to be something else in your >> /etc/make.conf file, > >Doesn't work for packages, only ports. > >Hence my interest in the recent discussion of making install-time >relocation work. > I would like to see that too, but I think the problem is that many of the original programs do not understand that concept. The main idea of ports/packages is to take some program which already exists, then then automate the steps for compiling that on freebsd. The thing is, many programs (say, anything with autoconf) does understand the "install prefix" and maybe even things like an "exec prefix", but they do not understand "do a make-install to location-A, and then some other process will move files from there to location-B, so the programs installed in location-A have to be easily relocatable after the make-install". In some cases you can use a long path for the make-install PREFIX, and then just patch the object-files with shorter paths (padded with nulls) to relocate them, but in other cases you have to get into the original program and rework how it does things. With 7000 programs in ports, that extra work could get messy... [but I'd definitely like it if it could be done... I think that could be useful for both ports and packages, particularly in the step of getting the packing-list correctly when installing a port] -- Garance Alistair Drosehn = gad@gilead.netel.rpi.edu Senior Systems Programmer or gad@freebsd.org Rensselaer Polytechnic Institute or drosih@rpi.edu To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Sun Jul 14 16:48:41 2002 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.FreeBSD.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 82AE537B400 for ; Sun, 14 Jul 2002 16:48:38 -0700 (PDT) Received: from whizzo.transsys.com (whizzo.TransSys.COM [144.202.42.10]) by mx1.FreeBSD.org (Postfix) with ESMTP id B3A0A43E4A for ; Sun, 14 Jul 2002 16:48:37 -0700 (PDT) (envelope-from louie@whizzo.transsys.com) Received: from whizzo.transsys.com (#6@localhost.transsys.com [127.0.0.1]) by whizzo.transsys.com (8.12.5/8.12.5) with ESMTP id g6ENla0L039627; Sun, 14 Jul 2002 19:47:36 -0400 (EDT) (envelope-from louie@whizzo.transsys.com) Message-Id: <200207142347.g6ENla0L039627@whizzo.transsys.com> X-Mailer: exmh version 2.5 07/13/2001 with nmh-1.0.4 To: Terry Lambert Cc: Thomas Seck , freebsd-arch@FreeBSD.ORG X-Image-URL: http://www.transsys.com/louie/images/louie-mail.jpg From: "Louis A. Mamakos" Subject: Re: Package system flaws? References: <20020712121427.GD3678@lummux.tchpc.tcd.ie> <20020712144854.GA756@laurel.tmseck.homedns.org> <20020713054141.A26277@misty.eyesbeyond.com> <20020713011750.GA755@laurel.tmseck.homedns.org> <20020714042237.GD931@lizzy.catnook.com> <20020714042623.GB95460@squall.waterspout.com> <20020714095939.GA588@laurel.tmseck.homedns.org> <200207141333.g6EDXj0L031673@whizzo.transsys.com> <20020714155728.GA4237@laurel.tmseck.homedns.org> <200207141624.g6EGOa0L033175@whizzo.transsys.com> <3D31F81E.290289FD@mindspring.com> In-reply-to: Your message of "Sun, 14 Jul 2002 15:15:58 PDT." <3D31F81E.290289FD@mindspring.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Sun, 14 Jul 2002 19:47:36 -0400 Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG > "Louis A. Mamakos" wrote: > > And so what's so difficult to understand? Why is it that the only > > tools "qualified" for use in maintaining the ports on a machine seem > > to be required to be in the base system? From what I can tell, the > > direction is to move non-essential stuff out of the base system. > > Because *in order to* move stuff out of the base system as > transparently as possible, the base system itself needs to > be under the purview of the package management system. > > In other words, the difference between a "PicoBSD" and a > standard minimal installation and a full installation should > come down to two things: > > 1) What shows up when you list installed components > > 2) What configuration data is mandatory I disagree. I think of two classes of users of the FreeBSD system. They are: A) Those that install releases from a CDROM from time-to-time. 2) Those that follow FreeBSD development on an on-going basis (e.g., bleeding edge -CURRENT or production -STABLE users). The "A" class of users don't need a package management system to maintain their systems as part of the FreeBSD base system. E.g., they use tools like sysinstall which isn't even built by "make buildworld", but is available on the distribution CDROM they booted. The "2" class of users use tools like cvsup which isn't part of the base system to keep their source code/repositories up to date. They manage to do this by using a tool in the ports/packages system. Many of them also choose to use tools like portupgrade, also in ports/packages. So why is it the package management infrastructure is required to be in the base system, when to a large extent it's not today? What's it mean to be part of the "base system" when everything turns into optional components? louie To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Sun Jul 14 18: 6:36 2002 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.FreeBSD.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 245F937B400 for ; Sun, 14 Jul 2002 18:06:34 -0700 (PDT) Received: from thuvia.demon.co.uk (thuvia.demon.co.uk [193.237.34.248]) by mx1.FreeBSD.org (Postfix) with ESMTP id 5975043E42 for ; Sun, 14 Jul 2002 18:06:32 -0700 (PDT) (envelope-from mark@thuvia.demon.co.uk) Received: from dotar.thuvia.org (dotar.thuvia.org [10.0.0.4]) by phaidor.thuvia.org (8.11.6/8.11.6) with ESMTP id g6F16Ib46138; Mon, 15 Jul 2002 02:06:18 +0100 (BST) (envelope-from mark@thuvia.demon.co.uk) Received: from dotar.thuvia.org (localhost [IPv6:::1]) by dotar.thuvia.org (8.12.3/8.12.3) with ESMTP id g6F16IKC084525; Mon, 15 Jul 2002 02:06:18 +0100 (BST) (envelope-from mark@dotar.thuvia.org) Received: (from mark@localhost) by dotar.thuvia.org (8.12.3/8.12.3/Submit) id g6F16HA7084524; Mon, 15 Jul 2002 02:06:17 +0100 (BST) Date: Mon, 15 Jul 2002 02:06:17 +0100 (BST) From: Mark Valentine Message-Id: <200207150106.g6F16HA7084524@dotar.thuvia.org> In-Reply-To: "Louis A. Mamakos"'s message of Jul 14, 11:53pm X-Mailer: Mail User's Shell (7.2.6 beta(5) 10/07/98) To: louie@TransSys.COM ("Louis A. Mamakos"), Terry Lambert Subject: Re: Package system flaws? Cc: Thomas Seck , freebsd-arch@freebsd.org Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG > From: louie@TransSys.COM ("Louis A. Mamakos") > Date: Sun 14 Jul, 2002 > Subject: Re: Package system flaws? > I think of two classes of users of the FreeBSD system. They are: > > A) Those that install releases from a CDROM from time-to-time. > > 2) Those that follow FreeBSD development on an on-going basis (e.g., > bleeding edge -CURRENT or production -STABLE users). > > The "A" class of users don't need a package management system to > maintain their systems as part of the FreeBSD base system. How then do they install the package management system package? ;-) > E.g., they > use tools like sysinstall which isn't even built by "make buildworld", > but is available on the distribution CDROM they booted. sysinstall and pkg_add are part of the base system, regardless of what a source build does (source builds being irrelevant to this class of users). [And sysinstall is part of buildworld in 5.0 anyway.] The "A" class still needs tools which will enable them to install additional packages, and to update components of the base system (e.g. for security fixes), ideally without ever needing any part of the source tree. > What's it > mean to be part of the "base system" when everything turns into > optional components? The "base system" is a pre-defined set of packages approximately equivalent to what today comprises the base system (minus a few components which would be optional now if the ports system were up to it). This subset is "standard FreeBSD", and third party developers can reasonably assume that all of its components are installed - you can install less if you don't care about that. Cheers, Mark. -- Mark Valentine, Thuvia Labs "Tigers will do ANYTHING for a tuna fish sandwich." Mark Valentine uses "We're kind of stupid that way." *munch* *munch* and endorses FreeBSD -- To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Sun Jul 14 18:26:55 2002 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.FreeBSD.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 23A4037B400 for ; Sun, 14 Jul 2002 18:26:51 -0700 (PDT) Received: from mailman.zeta.org.au (mailman.zeta.org.au [203.26.10.16]) by mx1.FreeBSD.org (Postfix) with ESMTP id E9E8843E6D for ; Sun, 14 Jul 2002 18:26:49 -0700 (PDT) (envelope-from bde@zeta.org.au) Received: from bde.zeta.org.au (bde.zeta.org.au [203.2.228.102]) by mailman.zeta.org.au (8.9.3/8.8.7) with ESMTP id VAA08378; Sun, 14 Jul 2002 21:32:11 +1000 Date: Sun, 14 Jul 2002 21:35:37 +1000 (EST) From: Bruce Evans X-X-Sender: bde@gamplex.bde.org To: Don Lewis Cc: arch@FreeBSD.ORG Subject: Re: wiring the sysctl output buffer In-Reply-To: <200207140916.g6E9Ghwr020552@gw.catspoiler.org> Message-ID: <20020714201020.F35894-100000@gamplex.bde.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG On Sun, 14 Jul 2002, Don Lewis wrote: > A number of places in the kernel call SYSCTL_OUT() while holding locks. > The problem is that SYSCTL_OUT() can block while it wires down the "old" > buffer in user space. If the data is small, such as an integer value, This seems to have been broken by FreeBSD. In rev.1.1, vslock() was called at the start of the sysctl, so the only problem with it was its pessimalness (*). Now I think the vslock() is just a pessimization, since the point of doing it was to avoid blocking during the copyout(), but blocking during vslock() is equivalent. Also, vslock() is insufficent if the kernel is preemptible. > the best solution may be to copy the data and defer calling SYSCTL_OUT() > until after the locks are released. This extra copy could be wasteful > if the data is large, and it may be cumbersome if the data size is not > known ahead of time, since determining the data size may be expensive. The extra copy is unlikely to be more wasteful than vslock()'ing, since vfslock() is a very expensive operation. Some instruction counts on a UP i386 kernel with no INVARIANTS or WITNESS: (*) vslock(): about 2700 instructions (for a length of just 16) vsunlock(): about 1400 useracc(): about 400 `oldlen' is known ahead of time, so a large enough buffer could be allocated in most cases. Perhaps vslock() iff oldlen is very large. The data could still change while it is being copied to a malloc()'ed buffer (or earlier) if the kernel is preemptible. > The data size could also potentially change between the time the size is > calculatated and the time when the data is copied to the temporary > buffer if the locks must be released so that MALLOC() can be called to > allocate the buffer. Hopefully the sysctl locking is enough to prevent at least *req changing underneath us. > I think the best solution to this problem is to add a sysctl system API > to prewire the output buffer that can be called before grabbing the > locks. Doing so allows the existing code to operate properly with only > minimal changes. > Here's a patch that implements this new API and illustrates how it can > be used to fix the kern.function_list sysctl, which wants to call > SYSCTL_OUT() multiple times while walking a locked data structure. I > think this is the best fix, though I'd like some feedback on whether > this is the best API. I think the callers of SYSCTL_OUT() don't need a new API, but they should supply the data in a form suitable for copying out directly. This probably involves them making a copy while holding suitable domain-specific locks. They can't just point to an active kernel struct since it might change. I would just make a copy at a high level for now. Bruce To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Sun Jul 14 18:52:54 2002 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.FreeBSD.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 2D23637B400 for ; Sun, 14 Jul 2002 18:52:48 -0700 (PDT) Received: from flamingo.mail.pas.earthlink.net (flamingo.mail.pas.earthlink.net [207.217.120.232]) by mx1.FreeBSD.org (Postfix) with ESMTP id C7CEE43E65 for ; Sun, 14 Jul 2002 18:52:46 -0700 (PDT) (envelope-from tlambert2@mindspring.com) Received: from pool0508.cvx40-bradley.dialup.earthlink.net ([216.244.43.253] helo=mindspring.com) by flamingo.mail.pas.earthlink.net with esmtp (Exim 3.33 #1) id 17Tv2W-00023c-00; Sun, 14 Jul 2002 18:52:41 -0700 Message-ID: <3D322AB8.4CCD1A63@mindspring.com> Date: Sun, 14 Jul 2002 18:51:52 -0700 From: Terry Lambert X-Mailer: Mozilla 4.79 [en]C-CCK-MCD {Sony} (Win98; U) X-Accept-Language: en MIME-Version: 1.0 To: "Louis A. Mamakos" Cc: Thomas Seck , freebsd-arch@FreeBSD.ORG Subject: Re: Package system flaws? References: <20020712121427.GD3678@lummux.tchpc.tcd.ie> <20020712144854.GA756@laurel.tmseck.homedns.org> <20020713054141.A26277@misty.eyesbeyond.com> <20020713011750.GA755@laurel.tmseck.homedns.org> <20020714042237.GD931@lizzy.catnook.com> <20020714042623.GB95460@squall.waterspout.com> <20020714095939.GA588@laurel.tmseck.homedns.org> <200207141333.g6EDXj0L031673@whizzo.transsys.com> <20020714155728.GA4237@laurel.tmseck.homedns.org> <200207141624.g6EGOa0L033175@whizzo.transsys.com> <3D31F81E.290289FD@mindspring.com> <200207142347.g6ENla0L039627@whizzo.transsys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG "Louis A. Mamakos" wrote: > I think of two classes of users of the FreeBSD system. They are: > > A) Those that install releases from a CDROM from time-to-time. > > 2) Those that follow FreeBSD development on an on-going basis (e.g., > bleeding edge -CURRENT or production -STABLE users). > > The "A" class of users don't need a package management system to > maintain their systems as part of the FreeBSD base system. E.g., they > use tools like sysinstall which isn't even built by "make buildworld", > but is available on the distribution CDROM they booted. > > The "2" class of users use tools like cvsup which isn't part of the > base system to keep their source code/repositories up to date. They > manage to do this by using a tool in the ports/packages system. Many > of them also choose to use tools like portupgrade, also in ports/packages. I respectfully disagree (and not just with your numbering not being from a single canonical ordinal set ;^)). I think the majority of FreeBSD users fall into an excluded middle: iii) Those that follow -stable or -security, and *ignore* FreeBSD developement on an ongoing basis. The "iii" class of users, I would argue, represents the *vast* majority of FreeBSD users, and I'm not just talking "by volume" because of Yahoo and the thousands of InterJet users, I'm talking *by user*. FreeBSD tries very hard to avoid getting into the desktop domain and competing there, for fear that it would lose. That's where I would say the vast majority of your "A" users live: they take what you give them, and they live with it. FreeBSD has consistently shyed away from serving this user base, any time it had a choice. Your "2" class of users are developers, who, by the nature of the inherent biases in the project processes and management, end up doing all their work on -current, so that the work product is used by the project, rather than being discarded. I think you've fused a number of axis', without providing justification, and the space is more than the two values one dimensional line that you paint it as. Here's your picture: Creators ("us") Consumers ("them") o o - uses CVSup - uses CDROM - subscribes to hackers/current/arch - subscrives to questions etc. etc. I think this is a very narrow view. I think that there is *at least* three major axis', and I don't pretend that my idea of "major" matches everyone elses, but I don't have to for the model to be more valid than the "us/them" model: FreeBSD as a Platform ^ | _. General Purpose | /| | / Consumers | / Creators <-----------------*---------------> / | / | |/ | Embedded `- | v FreeBSD as an ends in itself That's 8 distinct sectors for any given user to be measured into, in order to categorize them; e.g. numbering clockwise fr0m the upper left, and from back to front, option base 0, PicoBSD is somewhere in sector #5: PicoBSD - Embedded - Creators - FreeBSD as a Platform FreeBSD on a CDROM is much less localized; it's a potato shaped region with most of it's mass in sector 3. It's not incredibly useful for developers, ecept as a bootstrapping tool, and it's not incredibly useful as a platform, because the configuration you git with a naieve install is not really safe to run as a hardened server in a rack mount system... and most rack boxes for embedded uses don't have a CDROM drive, anyway, because it takes away front panel space, etc.. > So why is it the package management infrastructure is required to be > in the base system, when to a large extent it's not today? That's exactly it: because, to a large extent, it's not today. And that fails to adequately serve the user base, as a result. > What's it mean to be part of the "base system" when everything > turns into optional components? It means "the pieces fit together". You wouldn't put a left front quarter panel from a Chevy Nova on a Honda Civic. A package system is only intended to mean that, when you install components via one, that they will work together, as expected. If you have a Honda Civic (FreeBSD 4.6) and you select "install new left front quarter panel" (e.g. KDE 3.0), you don't end up with something that was intended for a Chevy Nova (FreeBSD 5.0). Things need to fit. Several times in this discussion, I've beaten around the bush of the fact that one of the major goals of modification of the package system must be to ensure that one of the emergent properties of developement going forward is project cohesiveness. Right now, there is significant work that happens in the PicoBSD realm (the "freebsd-small" mailing list), e.g. with Soekris boxes (as one example), and the ability to build small system images for use in special purpose situations. In other words, the ability to delegate a box to a particular role. None of this work is being captured in the main line FreeBSD; it is duplicated, or it is lost, as the main line FreeBSD moves forward. Another place that's very obvious on this is that every place I have ever worked that was a "FreeBSD shop" ended up having to roll its own developement environment to mirror that of the end product they intended to build. These places did not use FreeBSD CDROMs, or, if they did, it was merely as a starting point... and once past that, there was substantial effort to cross the gap between the CDROM, as distributed, and the target environment. Most of these places ended up building their own CDROM ISO images, burning them, and installing from those images instead any FreeBSD release CDROM, as released by the FreeBSD project. If the FreeBSD project doesn't *learn* how to capture this work, then it will either *continue* to be lost, or it will eventually fork the project, when the value of the work is too high for it to be duplicated easily, or allowed to be lost. PicoBSD is slowly but surely turning itself into a seperate project with a shared repository and mailing list server. Rather than trusting people with this, I would like to see the ability to learn institutionalized in the project and the tools, so that when individuals responsible leave, either voluntarily, or by getting hit by a bus, that the learning is not lost. One of the biggest mistakes a company can make is outsourcing its intellectual property to third party contractors, because as soon as the contract is up, unless there were well-thought-out systems in place, the institutional learning disappears with the people who embody it. The only fix for this -- allowing the use of contractors -- is a systemic fix. In the limit, and once you're past the founders, *everyone* in an Open Source project is effectively a contractor. -- Terry To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Sun Jul 14 21:28:33 2002 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.FreeBSD.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 1132737B400 for ; Sun, 14 Jul 2002 21:28:29 -0700 (PDT) Received: from thuvia.demon.co.uk (thuvia.demon.co.uk [193.237.34.248]) by mx1.FreeBSD.org (Postfix) with ESMTP id 2B6C843E67 for ; Sun, 14 Jul 2002 21:28:28 -0700 (PDT) (envelope-from mark@thuvia.demon.co.uk) Received: from dotar.thuvia.org (dotar.thuvia.org [10.0.0.4]) by phaidor.thuvia.org (8.11.6/8.11.6) with ESMTP id g6F4SCb47127; Mon, 15 Jul 2002 05:28:12 +0100 (BST) (envelope-from mark@thuvia.demon.co.uk) Received: from dotar.thuvia.org (localhost [IPv6:::1]) by dotar.thuvia.org (8.12.3/8.12.3) with ESMTP id g6F4SBKC089567; Mon, 15 Jul 2002 05:28:11 +0100 (BST) (envelope-from mark@dotar.thuvia.org) Received: (from mark@localhost) by dotar.thuvia.org (8.12.3/8.12.3/Submit) id g6F4SAsT089566; Mon, 15 Jul 2002 05:28:10 +0100 (BST) Date: Mon, 15 Jul 2002 05:28:10 +0100 (BST) From: Mark Valentine Message-Id: <200207150428.g6F4SAsT089566@dotar.thuvia.org> In-Reply-To: Terry Lambert's message of Jul 15, 2:03am X-Mailer: Mail User's Shell (7.2.6 beta(5) 10/07/98) To: tlambert2@mindspring.com (Terry Lambert), "Louis A. Mamakos" Subject: Re: Package system flaws? Cc: Thomas Seck , freebsd-arch@freebsd.org Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG > From: tlambert2@mindspring.com (Terry Lambert) > Date: Mon 15 Jul, 2002 > Subject: Re: Package system flaws? > "Louis A. Mamakos" wrote: > > I think of two classes of users of the FreeBSD system. They are: > > > > A) Those that install releases from a CDROM from time-to-time. > > > > 2) Those that follow FreeBSD development on an on-going basis (e.g., > > bleeding edge -CURRENT or production -STABLE users). > > > > The "A" class of users don't need a package management system to > > maintain their systems as part of the FreeBSD base system. E.g., they > > use tools like sysinstall which isn't even built by "make buildworld", > > but is available on the distribution CDROM they booted. > > > > The "2" class of users use tools like cvsup which isn't part of the > > base system to keep their source code/repositories up to date. They > > manage to do this by using a tool in the ports/packages system. Many > > of them also choose to use tools like portupgrade, also in ports/packages. > > I respectfully disagree (and not just with your numbering not > being from a single canonical ordinal set ;^)). I think the > majority of FreeBSD users fall into an excluded middle: > > iii) Those that follow -stable or -security, and *ignore* FreeBSD > developement on an ongoing basis. > > The "iii" class of users, I would argue, represents the *vast* > majority of FreeBSD users, and I'm not just talking "by volume" > because of Yahoo and the thousands of InterJet users, I'm talking > *by user*. Yes, users just want to get their job done with the minimum of effort. They need Internet-exposed systems to keep up to date with security fixes, but they typically don't have the resources to "make world" to do it. > FreeBSD tries very hard to avoid getting into the desktop domain > and competing there, for fear that it would lose. I don't think there's anything specific to desktops in the problems we're discussing. I'd like to be able to reduce the effort for my clients to maintain their news/mail/web servers and Internet gateways. These are key areas where FreeBSD is a relatively easy pitch, but unlike desktops, these systems exist in numbers too small to warrant a cloning or similar distributed maintenance setup. For desktops, I'm not sure. I've traditionally pitched SPARC systems running my second favourite operating system there (application availability, platform stability, etc.), and of course the binary patches are there. However, it's going to increasingly be the case (partly thanks to Sun abandoning mid-range workstations years ago, but also due to changing climate) where I'm going to want to pitch FreeBSD as a fallback, where the customer probably already has Linux vaguely in mind. I have the improved cohesiveness and reliability of FreeBSD on my side, and I have the ability to set up an environment where the customer can update these systems from a central build, but the admins I see in my market are just scared of operating system source code. So binary updates are probably pretty important there too. > Rather than trusting people with this, I would like to see the > ability to learn institutionalized in the project and the tools, > so that when individuals responsible leave, either voluntarily, > or by getting hit by a bus, that the learning is not lost. I'd like to hear your ideas on how to make _that_ happen! :-) Cheers, Mark. -- Mark Valentine, Thuvia Labs "Tigers will do ANYTHING for a tuna fish sandwich." Mark Valentine uses "We're kind of stupid that way." *munch* *munch* and endorses FreeBSD -- To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Sun Jul 14 23:14:27 2002 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.FreeBSD.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 5CF2037B400 for ; Sun, 14 Jul 2002 23:14:24 -0700 (PDT) Received: from scaup.mail.pas.earthlink.net (scaup.mail.pas.earthlink.net [207.217.120.49]) by mx1.FreeBSD.org (Postfix) with ESMTP id E13C543E58 for ; Sun, 14 Jul 2002 23:14:23 -0700 (PDT) (envelope-from tlambert2@mindspring.com) Received: from pool0066.cvx22-bradley.dialup.earthlink.net ([209.179.198.66] helo=mindspring.com) by scaup.mail.pas.earthlink.net with esmtp (Exim 3.33 #1) id 17Tz7M-0001c2-00; Sun, 14 Jul 2002 23:13:56 -0700 Message-ID: <3D3267F4.2C72E5D@mindspring.com> Date: Sun, 14 Jul 2002 23:13:08 -0700 From: Terry Lambert X-Mailer: Mozilla 4.79 [en] (Win98; U) X-Accept-Language: en MIME-Version: 1.0 To: Mark Valentine Cc: "Louis A. Mamakos" , Thomas Seck , freebsd-arch@freebsd.org Subject: Re: Package system flaws? References: <200207150428.g6F4SAsT089566@dotar.thuvia.org> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG Mark Valentine wrote: > > Rather than trusting people with this, I would like to see the > > ability to learn institutionalized in the project and the tools, > > so that when individuals responsible leave, either voluntarily, > > or by getting hit by a bus, that the learning is not lost. > > I'd like to hear your ideas on how to make _that_ happen! :-) A system is always greater than the sum of its parts. I know this seems incredibly cliche, but it's nonetheless true; that's how things get to be cliches in the first place. The way you get systems with self-sustaining properties that are desirable is by designing the system so that the properties you think are desirable are themselves emergent from the set of simple rules that govern the system. So in the limit, you pick the rules that, when combined, result in the systemic behaviour you desire. Note that the rules don't directly dictate this behaviour: the behaviour is an emergent property. So it comes down to being able to pick "the right rules". FreeBSD has done pretty well; but it could do better. One of the emergent properties of the current "rules of FreeBSD", which include a set of configuration management tools which are not capable of being applied to a rigidly defined base system, is that applications for an OS, which are not a good fit to that base system, are seperately defined outside of it. The most friendly of these is my recent "pet example" PicoBSD. But the rule here is arbitrary: the base system is rigidly defined because it is difficult to change, and it's difficult to change for a lack of tools that permit easy redefinition. It's *not* difficult to change because there's a particular systemic *need* for it to be difficult to change. You *might* argue that tools which permit an easy redefinition will make it easier for the group definition of "the base system" to lose power, and erode. Ignoring the merit of the delegation of power towards that particular ends... by that rationale, the lack of such tools should have prevented the existance of PicoBSD in the first place. The argument is flawed: self organizing systems *will* route around the damage, no matter what/ You might as well embrace them: at least it permits you to keep control over the context in which they emerge. To put it another way, if you're an Oxygen breather, it's in your best interests to embrace other Oxygen breathers, particularly if the altternative is a Chlorine-breater. 8-). -- Terry To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Mon Jul 15 5:16:49 2002 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.FreeBSD.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 4A53337B400 for ; Mon, 15 Jul 2002 05:16:46 -0700 (PDT) Received: from thuvia.demon.co.uk (thuvia.demon.co.uk [193.237.34.248]) by mx1.FreeBSD.org (Postfix) with ESMTP id A120C43E4A for ; Mon, 15 Jul 2002 05:16:44 -0700 (PDT) (envelope-from mark@thuvia.demon.co.uk) Received: from dotar.thuvia.org (dotar.thuvia.org [10.0.0.4]) by phaidor.thuvia.org (8.11.6/8.11.6) with ESMTP id g6FCF8b48598; Mon, 15 Jul 2002 13:15:08 +0100 (BST) (envelope-from mark@thuvia.demon.co.uk) Received: from dotar.thuvia.org (localhost [IPv6:::1]) by dotar.thuvia.org (8.12.3/8.12.3) with ESMTP id g6FCF5KC099684; Mon, 15 Jul 2002 13:15:05 +0100 (BST) (envelope-from mark@dotar.thuvia.org) Received: (from mark@localhost) by dotar.thuvia.org (8.12.3/8.12.3/Submit) id g6FCF43b099683; Mon, 15 Jul 2002 13:15:04 +0100 (BST) Date: Mon, 15 Jul 2002 13:15:04 +0100 (BST) From: Mark Valentine Message-Id: <200207151215.g6FCF43b099683@dotar.thuvia.org> In-Reply-To: Terry Lambert's message of Jul 14, 11:13pm X-Mailer: Mail User's Shell (7.2.6 beta(5) 10/07/98) To: Terry Lambert Subject: Re: Package system flaws? Cc: "Louis A. Mamakos" , Thomas Seck , freebsd-arch@freebsd.org Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG > From: Terry Lambert > Date: Sun 14 Jul, 2002 > Subject: Re: Package system flaws? > Mark Valentine wrote: > > > Rather than trusting people with this, I would like to see the > > > ability to learn institutionalized in the project and the tools, > > > so that when individuals responsible leave, either voluntarily, > > > or by getting hit by a bus, that the learning is not lost. > > > > I'd like to hear your ideas on how to make _that_ happen! :-) > > [vague rant snipped] I didn't see anything in there concrete enough to take action on... We already know about the problem of modularising the base system, and I'm not sure how reminding us of that helps to "institutionalize the ability to learn". Cheers, Mark. -- Mark Valentine, Thuvia Labs "Tigers will do ANYTHING for a tuna fish sandwich." Mark Valentine uses "We're kind of stupid that way." *munch* *munch* and endorses FreeBSD -- To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Mon Jul 15 9:29:37 2002 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.FreeBSD.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 5529637B400 for ; Mon, 15 Jul 2002 09:29:33 -0700 (PDT) Received: from w250.z064001178.sjc-ca.dsl.cnc.net (adsl-66.218.45.239.dslextreme.com [66.218.45.239]) by mx1.FreeBSD.org (Postfix) with SMTP id 469B843E72 for ; Mon, 15 Jul 2002 09:29:32 -0700 (PDT) (envelope-from jos@catnook.com) Received: (qmail 18592 invoked by uid 1000); 15 Jul 2002 16:29:53 -0000 Date: Mon, 15 Jul 2002 09:29:31 -0700 From: Jos Backus To: freebsd-arch@FreeBSD.ORG Subject: Re: Package system flaws? Message-ID: <20020715162953.GA12030@lizzy.catnook.com> Reply-To: jos@catnook.com Mail-Followup-To: freebsd-arch@FreeBSD.ORG References: <20020712121427.GD3678@lummux.tchpc.tcd.ie> <20020712144854.GA756@laurel.tmseck.homedns.org> <20020713054141.A26277@misty.eyesbeyond.com> <20020713011750.GA755@laurel.tmseck.homedns.org> <20020714042237.GD931@lizzy.catnook.com> <20020714042623.GB95460@squall.waterspout.com> <20020714095939.GA588@laurel.tmseck.homedns.org> <200207141333.g6EDXj0L031673@whizzo.transsys.com> <20020714164304.GA32774@lizzy.catnook.com> <20020714181408.GB420@laurel.tmseck.homedns.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20020714181408.GB420@laurel.tmseck.homedns.org> User-Agent: Mutt/1.5.1i Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG On Sun, Jul 14, 2002 at 08:14:08PM +0200, Thomas Seck wrote: > I think it is good not to have everyones' favourite scripting language in > the base system. I agree. Nowhere did I say we should include all of them (that's what the ports system is for). I'm saying that the current set of tools in the base is limiting. Unless of course you _define_ the base to consist only of these tools. We should pick one that has a reasonable chance of being able to be supported in the base and stick with it. Perl had build and packaging issues which made it a nightmare to support, fine, so let's pick another one that does a better job than awk/sh/etc. The problems were not with the fact that having a powerful scripting language in the base OS is necessarily bad but with its implementation in FreeBSD. > And VBA is not part of Windows(tm) by the way. Depends on your definition of "Windows", so I'm not going to argue this one. Suffice it to say that a well-supported scripting language is a Good Thing. > No one stops you from doing (cd /usr/ports/lang/ruby && make install). Of course. -- Jos Backus _/ _/_/_/ Santa Clara, CA _/ _/ _/ _/ _/_/_/ _/ _/ _/ _/ jos@catnook.com _/_/ _/_/_/ require 'std/disclaimer' To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Mon Jul 15 9:37:58 2002 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.FreeBSD.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 11A2237B400 for ; Mon, 15 Jul 2002 09:37:55 -0700 (PDT) Received: from w250.z064001178.sjc-ca.dsl.cnc.net (adsl-66.218.45.239.dslextreme.com [66.218.45.239]) by mx1.FreeBSD.org (Postfix) with SMTP id 9708A43E4A for ; Mon, 15 Jul 2002 09:37:53 -0700 (PDT) (envelope-from jos@catnook.com) Received: (qmail 20572 invoked by uid 1000); 15 Jul 2002 16:38:15 -0000 Date: Mon, 15 Jul 2002 09:37:53 -0701 From: Jos Backus To: freebsd-arch@FreeBSD.ORG Subject: Re: Package system flaws? Message-ID: <20020715163815.GB12030@lizzy.catnook.com> Reply-To: jos@catnook.com Mail-Followup-To: freebsd-arch@FreeBSD.ORG References: <20020712121427.GD3678@lummux.tchpc.tcd.ie> <20020712144854.GA756@laurel.tmseck.homedns.org> <20020713054141.A26277@misty.eyesbeyond.com> <20020713011750.GA755@laurel.tmseck.homedns.org> <20020714042237.GD931@lizzy.catnook.com> <20020714042623.GB95460@squall.waterspout.com> <20020714095939.GA588@laurel.tmseck.homedns.org> <200207141333.g6EDXj0L031673@whizzo.transsys.com> <3D31E944.A8E523E6@FreeBSD.org> <20020714214958.GA1228@laurel.tmseck.homedns.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20020714214958.GA1228@laurel.tmseck.homedns.org> User-Agent: Mutt/1.5.1i Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG On Sun, Jul 14, 2002 at 11:49:58PM +0200, Thomas Seck wrote: > Normally I would second that but IMHO portupgrade(1) is dangerous > in a way that it is a fine bandaid for the flaws in the current > implementation of the package system. There is no urge for the > developers to address these issues because "we have portupgrade". I do > not fight a religious war against portupgrade(1), I use it myself. But I > wish I would not have to. That is my point. While I agree with you in theory, the base-supported tools make writing a portupgrade-like tool non-trivial (that's the whole point of high-level scripting languages) which is one reason why only portupgrade exists today. You could turn this around, too: import Ruby+portupgrade into the base system, write compatibility wrappers for the existing pkg_* tools and remove the old pkg_* tools :-) -- Jos Backus _/ _/_/_/ Santa Clara, CA _/ _/ _/ _/ _/_/_/ _/ _/ _/ _/ jos@catnook.com _/_/ _/_/_/ require 'std/disclaimer' To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Mon Jul 15 10:36:27 2002 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.FreeBSD.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id CAF0137B7F9 for ; Mon, 15 Jul 2002 10:36:05 -0700 (PDT) Received: from thuvia.demon.co.uk (thuvia.demon.co.uk [193.237.34.248]) by mx1.FreeBSD.org (Postfix) with ESMTP id 27F9C442FD for ; Mon, 15 Jul 2002 10:24:54 -0700 (PDT) (envelope-from mark@thuvia.demon.co.uk) Received: from dotar.thuvia.org (dotar.thuvia.org [10.0.0.4]) by phaidor.thuvia.org (8.11.6/8.11.6) with ESMTP id g6FHIlb49956; Mon, 15 Jul 2002 18:18:47 +0100 (BST) (envelope-from mark@thuvia.demon.co.uk) Received: from dotar.thuvia.org (localhost [IPv6:::1]) by dotar.thuvia.org (8.12.3/8.12.3) with ESMTP id g6FHIkKC007663; Mon, 15 Jul 2002 18:18:46 +0100 (BST) (envelope-from mark@dotar.thuvia.org) Received: (from mark@localhost) by dotar.thuvia.org (8.12.3/8.12.3/Submit) id g6FHIkof007662; Mon, 15 Jul 2002 18:18:46 +0100 (BST) Date: Mon, 15 Jul 2002 18:18:46 +0100 (BST) From: Mark Valentine Message-Id: <200207151718.g6FHIkof007662@dotar.thuvia.org> In-Reply-To: Jos Backus's message of Jul 15, 4:38pm X-Mailer: Mail User's Shell (7.2.6 beta(5) 10/07/98) To: jos@catnook.com, freebsd-arch@freebsd.org Subject: Re: Package system flaws? Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG > From: jos@catnook.com (Jos Backus) > Date: Mon 15 Jul, 2002 > Subject: Re: Package system flaws? > On Sun, Jul 14, 2002 at 08:14:08PM +0200, Thomas Seck wrote: > > I think it is good not to have everyones' favourite scripting language in > > the base system. > > I agree. Nowhere did I say we should include all of them (that's what the > ports system is for). I'm saying that the current set of tools in the base is > limiting. I can do a heck of a lot with sh/expr/sed/join/etc (and awk when it gets serious), and I try to stick to the POSIX.2 subset. Beyond that there's a perfectly good C compiler. If I have more specialised needs, I'll be installing third party packages in any case, and I'll pick the tools that work for me. > Unless of course you _define_ the base to consist only of these tools. An individual OS can add whatever frills it wants to the "base" system, but it doesn't mean much to me as a proponent of portable software until all the platforms I support do likewise. > We should pick one that has a reasonable chance of being able to be > supported in the base and stick with it. Well, if we can't handle Tcl in the base system, I doubt much else has a look in... > Perl had build and packaging issues > which made it a nightmare to support, fine, so let's pick another one that > does a better job than awk/sh/etc. I'd like to hear you name one that would fit the bill, never mind find a concensus... Cheers, Mark. -- Mark Valentine, Thuvia Labs "Tigers will do ANYTHING for a tuna fish sandwich." Mark Valentine uses "We're kind of stupid that way." *munch* *munch* and endorses FreeBSD -- To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Mon Jul 15 10:43:59 2002 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.FreeBSD.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 5504937B400 for ; Mon, 15 Jul 2002 10:43:55 -0700 (PDT) Received: from smtp.netcologne.de (smtp.netcologne.de [194.8.194.112]) by mx1.FreeBSD.org (Postfix) with ESMTP id 1690243E8A for ; Mon, 15 Jul 2002 10:43:54 -0700 (PDT) (envelope-from tmseck-lists@netcologne.de) Received: from localhost (xdsl-213-168-118-121.netcologne.de [213.168.118.121]) by smtp.netcologne.de (8.12.2/8.12.2) with ESMTP id g6FHhmUt007289 for ; Mon, 15 Jul 2002 19:43:48 +0200 (MEST) Received: (qmail 1240 invoked by uid 1001); 15 Jul 2002 17:43:19 -0000 Date: Mon, 15 Jul 2002 19:43:19 +0200 From: Thomas Seck To: freebsd-arch@FreeBSD.ORG Subject: Re: Package system flaws? Message-ID: <20020715174318.GA682@laurel.tmseck.homedns.org> Mail-Followup-To: freebsd-arch@FreeBSD.ORG References: <20020712144854.GA756@laurel.tmseck.homedns.org> <20020713054141.A26277@misty.eyesbeyond.com> <20020713011750.GA755@laurel.tmseck.homedns.org> <20020714042237.GD931@lizzy.catnook.com> <20020714042623.GB95460@squall.waterspout.com> <20020714095939.GA588@laurel.tmseck.homedns.org> <200207141333.g6EDXj0L031673@whizzo.transsys.com> <20020714164304.GA32774@lizzy.catnook.com> <20020714181408.GB420@laurel.tmseck.homedns.org> <20020715162953.GA12030@lizzy.catnook.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20020715162953.GA12030@lizzy.catnook.com> User-Agent: Mutt/1.4i Organization: private site in Germany X-PGP-KeyID: DF46EE05 X-PGP-Fingerprint: A38F AE66 6B11 6EB9 5D1A B67D 2444 2FE1 DF46 EE05 X-Attribution: tms Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG * Jos Backus (jos@catnook.com): > On Sun, Jul 14, 2002 at 08:14:08PM +0200, Thomas Seck wrote: > > I think it is good not to have everyones' favourite scripting language in > > the base system. > > I agree. Nowhere did I say we should include all of them (that's what the > ports system is for). I'm saying that the current set of tools in the base is > limiting. Unless of course you _define_ the base to consist only of these > tools. I do. The base system should contain only the minimum amount of tools to keep the system running and "self contained", including basic tools for maintenance of third party applications. Everything else should be left to the user's choice. > We should pick one that has a reasonable chance of being able to be > supported in the base and stick with it. Perl had build and packaging issues > which made it a nightmare to support, fine, so let's pick another one that > does a better job than awk/sh/etc. The problems were not with the fact that > having a powerful scripting language in the base OS is necessarily bad but > with its implementation in FreeBSD. Unless a special scripting language is absolutely necessary for the system to function or to "rebuild" (bootstrap?) itself, it should stay in "ports land". I do not think it is wise to invest developer resources into maintenance and integration work the way it was done with Perl ever again. Apart from the usual problem of the maintainer being hit by a bus, kidnapped by aliens, losing interest, getting a life etc., you may run into political issues, Ruby for instance is GPL'ed. > > And VBA is not part of Windows(tm) by the way. > > Depends on your definition of "Windows", so I'm not going to argue this one. I apologise for being picky about it. Say VBS and I agree. VBA ships with MS Office and can IIRC be licenced for integration into your own apps. > Suffice it to say that a well-supported scripting language is a Good Thing. It depends. See ILY and friends for the benefits of scripting environments on Windows "Workstations". I agree in cmd.exe and the whole Windows toolchest being a joke when it comes to system administration. -- Thomas Seck To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Mon Jul 15 11:12:38 2002 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.FreeBSD.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 6EF2C37B40A for ; Mon, 15 Jul 2002 11:12:30 -0700 (PDT) Received: from smtp.netcologne.de (smtp.netcologne.de [194.8.194.112]) by mx1.FreeBSD.org (Postfix) with ESMTP id 7839F43E64 for ; Mon, 15 Jul 2002 11:12:29 -0700 (PDT) (envelope-from tmseck-lists@netcologne.de) Received: from localhost (xdsl-213-168-118-173.netcologne.de [213.168.118.173]) by smtp.netcologne.de (8.12.2/8.12.2) with ESMTP id g6FICOUt014961 for ; Mon, 15 Jul 2002 20:12:24 +0200 (MEST) Received: (qmail 1360 invoked by uid 1001); 15 Jul 2002 18:10:34 -0000 Date: Mon, 15 Jul 2002 20:10:34 +0200 From: Thomas Seck To: freebsd-arch@FreeBSD.ORG Subject: Re: Package system flaws? Message-ID: <20020715181034.GB682@laurel.tmseck.homedns.org> Mail-Followup-To: freebsd-arch@FreeBSD.ORG References: <20020712144854.GA756@laurel.tmseck.homedns.org> <20020713054141.A26277@misty.eyesbeyond.com> <20020713011750.GA755@laurel.tmseck.homedns.org> <20020714042237.GD931@lizzy.catnook.com> <20020714042623.GB95460@squall.waterspout.com> <20020714095939.GA588@laurel.tmseck.homedns.org> <200207141333.g6EDXj0L031673@whizzo.transsys.com> <3D31E944.A8E523E6@FreeBSD.org> <20020714214958.GA1228@laurel.tmseck.homedns.org> <20020715163815.GB12030@lizzy.catnook.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20020715163815.GB12030@lizzy.catnook.com> User-Agent: Mutt/1.4i Organization: private site in Germany X-PGP-KeyID: DF46EE05 X-PGP-Fingerprint: A38F AE66 6B11 6EB9 5D1A B67D 2444 2FE1 DF46 EE05 X-Attribution: tms Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG * Jos Backus (jos@catnook.com): > On Sun, Jul 14, 2002 at 11:49:58PM +0200, Thomas Seck wrote: > > Normally I would second that but IMHO portupgrade(1) is dangerous > > in a way that it is a fine bandaid for the flaws in the current > > implementation of the package system. There is no urge for the > > developers to address these issues because "we have portupgrade". I do > > not fight a religious war against portupgrade(1), I use it myself. But I > > wish I would not have to. That is my point. > > While I agree with you in theory, the base-supported tools make writing a > portupgrade-like tool non-trivial (that's the whole point of high-level > scripting languages) which is one reason why only portupgrade exists today. It is probably more difficult to do correctly, agreed. But this has to be done only once and maybe put into a library (libh?). > You could turn this around, too: import Ruby+portupgrade into the base system, > write compatibility wrappers for the existing pkg_* tools and remove the old > pkg_* tools :-) Three problems with this: - Yet Another Language to maintain, tying up developer resources - I personally do not need Ruby (or $scripting_language_of_the_season) - Licence issues, thus endless political discussions -- Thomas Seck To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Mon Jul 15 11:25:25 2002 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.FreeBSD.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 2956F37B400 for ; Mon, 15 Jul 2002 11:25:24 -0700 (PDT) Received: from mail.speakeasy.net (mail13.speakeasy.net [216.254.0.213]) by mx1.FreeBSD.org (Postfix) with ESMTP id 68F2A43E4A for ; Mon, 15 Jul 2002 11:25:23 -0700 (PDT) (envelope-from jhb@FreeBSD.org) Received: (qmail 8217 invoked from network); 15 Jul 2002 18:25:22 -0000 Received: from unknown (HELO server.baldwin.cx) ([216.27.160.63]) (envelope-sender ) by mail13.speakeasy.net (qmail-ldap-1.03) with DES-CBC3-SHA encrypted SMTP for ; 15 Jul 2002 18:25:22 -0000 Received: from laptop.baldwin.cx (gw1.twc.weather.com [216.133.140.1]) by server.baldwin.cx (8.11.6/8.11.6) with ESMTP id g6FIPL049452 for ; Mon, 15 Jul 2002 14:25:21 -0400 (EDT) (envelope-from jhb@FreeBSD.org) Message-ID: X-Mailer: XFMail 1.5.2 on FreeBSD X-Priority: 3 (Normal) Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 8bit MIME-Version: 1.0 Date: Mon, 15 Jul 2002 14:25:26 -0400 (EDT) From: John Baldwin To: arch@FreeBSD.org Subject: Splitting up options and files by subsystem Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG Back when I first started working on splitting up NOTES into MI and MD sections, I brought up this idea and was met with mostly apathy. The idea is to allow a subsystem (such as ACPI or a device driver used on several platforms but not all, like atkbdc(4) and friends) to specify its NOTES, files, and options contents into subsystem-specific files that the architecture-specific files could then include. For example, we could have a NOTES.acpi, options.acpi, and files.acpi that were included by sys/{i386,ia64}/NOTES, options.{i386,ia64}, and files.{i386,ia64}. Another example would be a NOTES.atkbdc, files.atkbdc, and options.atkbdc for the atkbd(4), atkbdc(4), and psm(4) drivers and related options. IMO, this scales better than duplicating content across a bunch of different files. Trying to get an Alpha LINT to successfully config(8) is pointing out lots of places where our current model hasn't scaled. -- John Baldwin <>< http://www.FreeBSD.org/~jhb/ "Power Users Use the Power to Serve!" - http://www.FreeBSD.org/ To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Mon Jul 15 11:29:39 2002 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.FreeBSD.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id C1DA737B400 for ; Mon, 15 Jul 2002 11:29:36 -0700 (PDT) Received: from w250.z064001178.sjc-ca.dsl.cnc.net (adsl-66.218.45.239.dslextreme.com [66.218.45.239]) by mx1.FreeBSD.org (Postfix) with SMTP id 14A3A43E58 for ; Mon, 15 Jul 2002 11:29:36 -0700 (PDT) (envelope-from jos@catnook.com) Received: (qmail 47099 invoked by uid 1000); 15 Jul 2002 18:29:57 -0000 Date: Mon, 15 Jul 2002 11:29:35 -0700 From: Jos Backus To: freebsd-arch@freebsd.org Subject: Re: Package system flaws? Message-ID: <20020715182957.GA32690@lizzy.catnook.com> Reply-To: jos@catnook.com Mail-Followup-To: freebsd-arch@freebsd.org References: <200207151718.g6FHIkof007662@dotar.thuvia.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <200207151718.g6FHIkof007662@dotar.thuvia.org> User-Agent: Mutt/1.5.1i Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG On Mon, Jul 15, 2002 at 06:18:46PM +0100, Mark Valentine wrote: > I can do a heck of a lot with sh/expr/sed/join/etc (and awk when it gets > serious), and I try to stick to the POSIX.2 subset. > > Beyond that there's a perfectly good C compiler. The fact that it can be done using these tools doesn't mean that they are the most appropriate choice. Of course many people (especially those who feel scripting languages are silly toys only to be used for non-real-world applications) will disagree with this stance. Anyway, I have said what I wanted to say; I'm not going to argue this any further. > An individual OS can add whatever frills it wants to the "base" system, but > it doesn't mean much to me as a proponent of portable software until all the > platforms I support do likewise. If portability is at all important, I think we should abandon discussing the FreeBSD pkg_* tools (and portupgrade) and focus on what the OpenPackages people are doing. > > We should pick one that has a reasonable chance of being able to be > > supported in the base and stick with it. > > Well, if we can't handle Tcl in the base system, I doubt much else has a look > in... FwIu, Tcl is a bad example because of their internally inconsistent versioning; new versions are not backward-compatible enough, etc. > > Perl had build and packaging issues > > which made it a nightmare to support, fine, so let's pick another one that > > does a better job than awk/sh/etc. > > I'd like to hear you name one that would fit the bill, never mind find a > concensus... First we need to decide if we even _want_ a more powerful scripting language included. It sounds like the current consensus is a resounding NO. However, applications like portupgrade are much easier written in a scripting language than using the standard tools. Scripting languages, for all their faults, also reach a larger audience of potential contributors than the standard tools do. All imo, of course. Cheers, -- Jos Backus _/ _/_/_/ Santa Clara, CA _/ _/ _/ _/ _/_/_/ _/ _/ _/ _/ jos@catnook.com _/_/ _/_/_/ require 'std/disclaimer' To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Mon Jul 15 12: 2:43 2002 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.FreeBSD.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 1ED9837B400 for ; Mon, 15 Jul 2002 12:02:41 -0700 (PDT) Received: from thuvia.demon.co.uk (thuvia.demon.co.uk [193.237.34.248]) by mx1.FreeBSD.org (Postfix) with ESMTP id 72CA943E65 for ; Mon, 15 Jul 2002 12:02:39 -0700 (PDT) (envelope-from mark@thuvia.demon.co.uk) Received: from dotar.thuvia.org (dotar.thuvia.org [10.0.0.4]) by phaidor.thuvia.org (8.11.6/8.11.6) with ESMTP id g6FJ2Xb50483; Mon, 15 Jul 2002 20:02:34 +0100 (BST) (envelope-from mark@thuvia.demon.co.uk) Received: from dotar.thuvia.org (localhost [IPv6:::1]) by dotar.thuvia.org (8.12.3/8.12.3) with ESMTP id g6FJ2XKC010626; Mon, 15 Jul 2002 20:02:33 +0100 (BST) (envelope-from mark@dotar.thuvia.org) Received: (from mark@localhost) by dotar.thuvia.org (8.12.3/8.12.3/Submit) id g6FJ2Xg8010625; Mon, 15 Jul 2002 20:02:33 +0100 (BST) Date: Mon, 15 Jul 2002 20:02:33 +0100 (BST) From: Mark Valentine Message-Id: <200207151902.g6FJ2Xg8010625@dotar.thuvia.org> In-Reply-To: X-Mailer: Mail User's Shell (7.2.6 beta(5) 10/07/98) To: jos@catnook.com, freebsd-arch@freebsd.org Subject: Re: Package system flaws? Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG > From: jos@catnook.com (Jos Backus) > Date: Mon 15 Jul, 2002 > Subject: Re: Package system flaws? > If portability is at all important, I think we should abandon discussing the > FreeBSD pkg_* tools (and portupgrade) and focus on what the OpenPackages > people are doing. I wouldn't go that far. A portable application vendor can always just skip the system's preferred packaging tools and use tar(1) and an install script if necessary. However, packaging the application up for platforms you care about isn't excessively painful, even if a universal set of tools would be the ideal situation. Most if the pain in developing portable applications lies elsewhere; all the packaging tools do is allow the vendor to transport the results of "make install" on his system to the customer's system in a reliable and reproducible way. Cheers, Mark. -- Mark Valentine, Thuvia Labs "Tigers will do ANYTHING for a tuna fish sandwich." Mark Valentine uses "We're kind of stupid that way." *munch* *munch* and endorses FreeBSD -- To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Mon Jul 15 12:16:26 2002 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.FreeBSD.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 3D76F37B400 for ; Mon, 15 Jul 2002 12:16:24 -0700 (PDT) Received: from w250.z064001178.sjc-ca.dsl.cnc.net (adsl-66.218.45.239.dslextreme.com [66.218.45.239]) by mx1.FreeBSD.org (Postfix) with SMTP id BA54643E58 for ; Mon, 15 Jul 2002 12:16:23 -0700 (PDT) (envelope-from jos@catnook.com) Received: (qmail 58335 invoked by uid 1000); 15 Jul 2002 19:16:45 -0000 Date: Mon, 15 Jul 2002 12:16:23 -0700 From: Jos Backus To: freebsd-arch@FreeBSD.ORG Subject: Re: Package system flaws? Message-ID: <20020715191645.GA54500@lizzy.catnook.com> Reply-To: jos@catnook.com Mail-Followup-To: freebsd-arch@FreeBSD.ORG References: <20020713054141.A26277@misty.eyesbeyond.com> <20020713011750.GA755@laurel.tmseck.homedns.org> <20020714042237.GD931@lizzy.catnook.com> <20020714042623.GB95460@squall.waterspout.com> <20020714095939.GA588@laurel.tmseck.homedns.org> <200207141333.g6EDXj0L031673@whizzo.transsys.com> <3D31E944.A8E523E6@FreeBSD.org> <20020714214958.GA1228@laurel.tmseck.homedns.org> <20020715163815.GB12030@lizzy.catnook.com> <20020715181034.GB682@laurel.tmseck.homedns.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20020715181034.GB682@laurel.tmseck.homedns.org> User-Agent: Mutt/1.5.1i Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG On Mon, Jul 15, 2002 at 08:10:34PM +0200, Thomas Seck wrote: > It is probably more difficult to do correctly, agreed. But this has to > be done only once and maybe put into a library (libh?). Well, let's wait for libh then, which will render this whole discussion moot. > - Yet Another Language to maintain, tying up developer resources Granted, it would mean more work initially, but afterwards we would have a better prototyping tool in the base OS (one of the things scripting languages are often used for) and the whole portupgrade discussion would be over, too. Assuming bmake'ing Ruby is easy (Perl certainly wasn't, which was one of the reasons it was dropped) it would be a good candidate in my view. But I realize full well that I am perhaps the only person that feels this way. > - I personally do not need Ruby (or $scripting_language_of_the_season) I personally have not used awk in a while, and am trying to get away from Perl for all the bad habits it has taught me :) > - Licence issues, thus endless political discussions Fwiw, Matz has said that Rite (Ruby 2.0) will have a BSD license. Currently regex.c is GPL'ed but a new regex engine (named Oniguruma) is in the works. See http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-talk/34496 -- Jos Backus _/ _/_/_/ Santa Clara, CA _/ _/ _/ _/ _/_/_/ _/ _/ _/ _/ jos@catnook.com _/_/ _/_/_/ require 'std/disclaimer' To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Mon Jul 15 15: 3:13 2002 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.FreeBSD.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id C1E5837B401 for ; Mon, 15 Jul 2002 15:03:08 -0700 (PDT) Received: from swan.mail.pas.earthlink.net (swan.mail.pas.earthlink.net [207.217.120.123]) by mx1.FreeBSD.org (Postfix) with ESMTP id C269D43E65 for ; Mon, 15 Jul 2002 15:03:06 -0700 (PDT) (envelope-from tlambert2@mindspring.com) Received: from pool0196.cvx21-bradley.dialup.earthlink.net ([209.179.192.196] helo=mindspring.com) by swan.mail.pas.earthlink.net with esmtp (Exim 3.33 #1) id 17UDvX-00027y-00; Mon, 15 Jul 2002 15:02:44 -0700 Message-ID: <3D33464B.7C9BC4D3@mindspring.com> Date: Mon, 15 Jul 2002 15:01:47 -0700 From: Terry Lambert X-Mailer: Mozilla 4.79 [en] (Win98; U) X-Accept-Language: en MIME-Version: 1.0 To: Mark Valentine Cc: "Louis A. Mamakos" , Thomas Seck , freebsd-arch@freebsd.org Subject: Re: Package system flaws? References: <200207151215.g6FCF43b099683@dotar.thuvia.org> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG Mark Valentine wrote: > From: Terry Lambert > > > > Rather than trusting people with this, I would like to see the > > > > ability to learn institutionalized in the project and the tools, > > > > so that when individuals responsible leave, either voluntarily, > > > > or by getting hit by a bus, that the learning is not lost. > > > > > > I'd like to hear your ideas on how to make _that_ happen! :-) > > > > [vague rant snipped] > > I didn't see anything in there concrete enough to take action on... > > We already know about the problem of modularising the base system, > and I'm not sure how reminding us of that helps to "institutionalize > the ability to learn". If you don't want to hear the answer, don't ask the question. If you need something less "vague" than a generalization of conclusions, then try: http://www.americanscientist.org/articles/95articles/Saperstein-full.html It takes systems engineering of the institution itself, in order for it to be able to learn, and thus institutionalize knowledge, rather than embedding it in its people. "Modularizing the base system" is not a problem; it is the solution to a problem -- and not just an ends in itself, as your lack of certainty implies. Performing that task has consequences significantly above and beyond the direct consequences any casual observer would expect. -- Terry To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Mon Jul 15 16:18:45 2002 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.FreeBSD.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id E8AED37B63A for ; Mon, 15 Jul 2002 16:18:38 -0700 (PDT) Received: from mail.nsu.ru (mx.nsu.ru [193.124.215.71]) by mx1.FreeBSD.org (Postfix) with ESMTP id C7DA643E31 for ; Mon, 15 Jul 2002 16:18:32 -0700 (PDT) (envelope-from danfe@regency.nsu.ru) Received: from drweb by mail.nsu.ru with drweb-scanned (Exim 3.20 #1) id 17U7SS-0002QG-00 for freebsd-arch@freebsd.org; Mon, 15 Jul 2002 22:08:16 +0700 Received: from regency.nsu.ru ([193.124.210.26]) by mail.nsu.ru with esmtp (Exim 3.20 #1) id 17U7RB-0002J6-00 for freebsd-arch@freebsd.org; Mon, 15 Jul 2002 22:08:16 +0700 Received: from regency.nsu.ru (localhost [127.0.0.1]) by regency.nsu.ru (8.12.4/8.12.4) with ESMTP id g6FF7MnP073527 for ; Mon, 15 Jul 2002 22:07:22 +0700 (NOVST) (envelope-from danfe@regency.nsu.ru) Received: (from danfe@localhost) by regency.nsu.ru (8.12.4/8.12.4/Submit) id g6FEijl3065721; Mon, 15 Jul 2002 21:44:45 +0700 (NOVST) Date: Mon, 15 Jul 2002 21:44:45 +0700 From: Alexey Dokuchaev To: Jos Backus Cc: freebsd-arch@freebsd.org Subject: Re: Package system flaws? Message-ID: <20020715214445.C53266@regency.nsu.ru> References: <20020712121427.GD3678@lummux.tchpc.tcd.ie> <20020712144854.GA756@laurel.tmseck.homedns.org> <20020713054141.A26277@misty.eyesbeyond.com> <20020713011750.GA755@laurel.tmseck.homedns.org> <20020714042237.GD931@lizzy.catnook.com> <20020714042623.GB95460@squall.waterspout.com> <20020714095939.GA588@laurel.tmseck.homedns.org> <200207141333.g6EDXj0L031673@whizzo.transsys.com> <20020714164304.GA32774@lizzy.catnook.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2.5.1i In-Reply-To: <20020714164304.GA32774@lizzy.catnook.com>; from jos@catnook.com on Sun, Jul 14, 2002 at 09:42:42AM -0701 X-Envelope-To: freebsd-arch@freebsd.org Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG On Sun, Jul 14, 2002 at 09:42:42AM -0701, Jos Backus wrote: > > Part of me still thinks it's a pity that we don't have a decent scripting > language (such as Ruby) as part of the base OS. Windows has Visual BASIC for We did have a very powerful one until recently -- Perl. I guess the fact it was removed from the base is for a very good reason. > Applications - all due respect to the awk creators and maintainers, but surely > we can do better than awk/sh? Oh please, can you show us something essential for base enough that could not be implemented in a sh/sed/awk way? I somewhat doubt it. 'cmon, it's pretty clear that Perl or Ruby is more of an overhead than of worth. Traditionally, UNIX lived for 30+ years without need for a monster like Perl or Ruby, in the base, clearly showing us that sh/awk/sed is a [very] decent scripting facility. ./danfe To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Mon Jul 15 17:34:37 2002 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.FreeBSD.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 7F3BE37B400 for ; Mon, 15 Jul 2002 17:34:35 -0700 (PDT) Received: from w250.z064001178.sjc-ca.dsl.cnc.net (adsl-66.218.45.239.dslextreme.com [66.218.45.239]) by mx1.FreeBSD.org (Postfix) with SMTP id C335543E5E for ; Mon, 15 Jul 2002 17:34:34 -0700 (PDT) (envelope-from jos@catnook.com) Received: (qmail 33889 invoked by uid 1000); 16 Jul 2002 00:34:56 -0000 Date: Mon, 15 Jul 2002 17:34:34 -0700 From: Jos Backus To: freebsd-arch@freebsd.org Subject: Re: Package system flaws? Message-ID: <20020716003456.GD54500@lizzy.catnook.com> Reply-To: jos@catnook.com Mail-Followup-To: freebsd-arch@freebsd.org References: <20020712121427.GD3678@lummux.tchpc.tcd.ie> <20020712144854.GA756@laurel.tmseck.homedns.org> <20020713054141.A26277@misty.eyesbeyond.com> <20020713011750.GA755@laurel.tmseck.homedns.org> <20020714042237.GD931@lizzy.catnook.com> <20020714042623.GB95460@squall.waterspout.com> <20020714095939.GA588@laurel.tmseck.homedns.org> <200207141333.g6EDXj0L031673@whizzo.transsys.com> <20020714164304.GA32774@lizzy.catnook.com> <20020715214445.C53266@regency.nsu.ru> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20020715214445.C53266@regency.nsu.ru> User-Agent: Mutt/1.5.1i Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG On Mon, Jul 15, 2002 at 09:44:45PM +0700, Alexey Dokuchaev wrote: > We did have a very powerful one until recently -- Perl. I guess the > fact it was removed from the base is for a very good reason. I don't think ``we don't want a powerful scripting language in the base system'' was one of them. > Oh please, can you show us something essential for base enough that > could not be implemented in a sh/sed/awk way? I somewhat doubt it. portupgrade :-) (Sure, anything that can be done in Ruby can be done in C; no need to make that argument again). > 'cmon, it's pretty clear that Perl or Ruby is more of an overhead than of > worth. Traditionally, UNIX lived for 30+ years without need for a monster > like Perl or Ruby, in the base, clearly showing us that sh/awk/sed is a > [very] decent scripting facility. As must be clear by now, I respectfully beg to differ. -- Jos Backus _/ _/_/_/ Santa Clara, CA _/ _/ _/ _/ _/_/_/ _/ _/ _/ _/ jos@catnook.com _/_/ _/_/_/ require 'std/disclaimer' To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Mon Jul 15 18:12:39 2002 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.FreeBSD.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id E1C7537B400 for ; Mon, 15 Jul 2002 18:12:36 -0700 (PDT) Received: from thuvia.demon.co.uk (thuvia.demon.co.uk [193.237.34.248]) by mx1.FreeBSD.org (Postfix) with ESMTP id 6578043E4A for ; Mon, 15 Jul 2002 18:12:35 -0700 (PDT) (envelope-from mark@thuvia.demon.co.uk) Received: from dotar.thuvia.org (dotar.thuvia.org [10.0.0.4]) by phaidor.thuvia.org (8.11.6/8.11.6) with ESMTP id g6G1CIb52226; Tue, 16 Jul 2002 02:12:18 +0100 (BST) (envelope-from mark@thuvia.demon.co.uk) Received: from dotar.thuvia.org (localhost [IPv6:::1]) by dotar.thuvia.org (8.12.3/8.12.3) with ESMTP id g6G1CIKC020443; Tue, 16 Jul 2002 02:12:18 +0100 (BST) (envelope-from mark@dotar.thuvia.org) Received: (from mark@localhost) by dotar.thuvia.org (8.12.3/8.12.3/Submit) id g6G1CIgH020442; Tue, 16 Jul 2002 02:12:18 +0100 (BST) Date: Tue, 16 Jul 2002 02:12:18 +0100 (BST) From: Mark Valentine Message-Id: <200207160112.g6G1CIgH020442@dotar.thuvia.org> In-Reply-To: X-Mailer: Mail User's Shell (7.2.6 beta(5) 10/07/98) To: danfe@regency.nsu.ru (Alexey Dokuchaev), Jos Backus Subject: Re: Package system flaws? Cc: freebsd-arch@freebsd.org Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG > From: danfe@regency.nsu.ru (Alexey Dokuchaev) > Date: Mon 15 Jul, 2002 > Subject: Re: Package system flaws? > Traditionally, UNIX lived for 30+ years without need for a > monster like Perl or Ruby, in the base, clearly showing us that > sh/awk/sed is a [very] decent scripting facility. Though I did breathe a sigh of relief when shell functions became common enough to be useful... ;-) I've never really wanted for anything since. Except maybe for make(1) to have a standard conditional syntax. Cheers, Mark. -- Mark Valentine, Thuvia Labs "Tigers will do ANYTHING for a tuna fish sandwich." Mark Valentine uses "We're kind of stupid that way." *munch* *munch* and endorses FreeBSD -- To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Mon Jul 15 18:56:20 2002 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.FreeBSD.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id CFC5937B400 for ; Mon, 15 Jul 2002 18:56:17 -0700 (PDT) Received: from mail.rpi.edu (mail.rpi.edu [128.113.22.40]) by mx1.FreeBSD.org (Postfix) with ESMTP id 1DAB543E64 for ; Mon, 15 Jul 2002 18:56:17 -0700 (PDT) (envelope-from drosih@rpi.edu) Received: from [128.113.24.47] (gilead.netel.rpi.edu [128.113.24.47]) by mail.rpi.edu (8.12.1/8.12.1) with ESMTP id g6G1uA3W177818; Mon, 15 Jul 2002 21:56:14 -0400 Mime-Version: 1.0 X-Sender: drosih@mail.rpi.edu Message-Id: In-Reply-To: <20020715182957.GA32690@lizzy.catnook.com> References: <200207151718.g6FHIkof007662@dotar.thuvia.org> <20020715182957.GA32690@lizzy.catnook.com> Date: Mon, 15 Jul 2002 21:56:07 -0400 To: jos@catnook.com, freebsd-arch@FreeBSD.ORG From: Garance A Drosihn Subject: Re: scripting language in base system? Content-Type: text/plain; charset="us-ascii" ; format="flowed" X-Scanned-By: MIMEDefang 2.3 (www dot roaringpenguin dot com slash mimedefang) Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG It seems to me that this topic has nothing to do with "the package system". It also seems to me that this topic is a bikeshed of immense proportions, and we shouldn't bury the very useful discussion on package-system issues with this bikeshed. At 11:29 AM -0700 7/15/02, Jos Backus wrote: >On Mon, Jul 15, 2002, Mark Valentine wrote: > > I'd like to hear you name one that would fit the bill, never > > mind find a concensus... > >First we need to decide if we even _want_ a more powerful >scripting language included. It sounds like the current >consensus is a resounding NO. However, applications like >portupgrade are much easier written in a scripting language >than using the standard tools. Scripting languages, for all >their faults, also reach a larger audience of potential >contributors than the standard tools do. For what it's worth, I'd like to see ruby in the standard freebsd system. However, I can easily get into arguments with myself (never mind anyone else) when trying to pin down my reasoning for this. I keep trying to write this message, listing my thoughts on scripting languages. While I can come up with a list of ideas that I completely agree with, the problem is that the items in that list conflict with each other, and thus I can't even come to a firm conclusion that *I* like. And in the process of trying, I'd probably manage to irritate everyone who has any opinion on this matter. I do not agree with the claim that sh+awk+sed is a adequate alternative to perl, ruby, or python. I also do not believe that we'll take the time to write everything in C which we might find useful, and thus useful things do not get written because we refuse to include tools which would allow us to write such things. Portupgrade was written in ruby because it was NOT getting written in any of the alternatives. So, so far, I've irritated the standard unix scripters, and the C-programmers... At the same time, I don't think we can ever "be safe" with some standard scripting language in the base system, because that scripting language will change over time. And my guess is that any good scripting language will eventually evolve into it's own "OS-neutral" platform, and thus grow into a monster that we won't want to have in our base system. Thus, any good, standard, popular, and useful scripting language is probably going to be a bad choice over time. That claim should manage to irritate everyone else... This is an unwinnable debate, imo. -- Garance Alistair Drosehn = gad@gilead.netel.rpi.edu Senior Systems Programmer or gad@freebsd.org Rensselaer Polytechnic Institute or drosih@rpi.edu To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Mon Jul 15 18:58:20 2002 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.FreeBSD.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id F36D737B400 for ; Mon, 15 Jul 2002 18:58:18 -0700 (PDT) Received: from mail.rpi.edu (mail.rpi.edu [128.113.22.40]) by mx1.FreeBSD.org (Postfix) with ESMTP id 6E52F43E4A for ; Mon, 15 Jul 2002 18:58:16 -0700 (PDT) (envelope-from drosih@rpi.edu) Received: from [128.113.24.47] (gilead.netel.rpi.edu [128.113.24.47]) by mail.rpi.edu (8.12.1/8.12.1) with ESMTP id g6G1vq3W043692; Mon, 15 Jul 2002 21:57:53 -0400 Mime-Version: 1.0 X-Sender: drosih@mail.rpi.edu Message-Id: In-Reply-To: <200207160112.g6G1CIgH020442@dotar.thuvia.org> References: <200207160112.g6G1CIgH020442@dotar.thuvia.org> Date: Mon, 15 Jul 2002 21:57:51 -0400 To: Mark Valentine , danfe@regency.nsu.ru (Alexey Dokuchaev), Jos Backus From: Garance A Drosihn Subject: Re: scripting language in base system? Cc: freebsd-arch@FreeBSD.ORG Content-Type: text/plain; charset="us-ascii" ; format="flowed" X-Scanned-By: MIMEDefang 2.3 (www dot roaringpenguin dot com slash mimedefang) Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG At 2:12 AM +0100 7/16/02, Mark Valentine wrote: > > From: danfe@regency.nsu.ru (Alexey Dokuchaev) > > > Traditionally, UNIX lived for 30+ years without need for a >> monster like Perl or Ruby, in the base, clearly showing us that >> sh/awk/sed is a [very] decent scripting facility. > >Though I did breathe a sigh of relief when shell functions became >common enough to be useful... ;-) > >I've never really wanted for anything since. Except maybe for >make(1) to have a standard conditional syntax. Perhaps that is the problem. Me, I want a good package-system, but all the 30-year sh/sed/awk veterans don't seem to see a need for it. -- Garance Alistair Drosehn = gad@gilead.netel.rpi.edu Senior Systems Programmer or gad@freebsd.org Rensselaer Polytechnic Institute or drosih@rpi.edu To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Mon Jul 15 20:40:45 2002 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.FreeBSD.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id BDE9237B400 for ; Mon, 15 Jul 2002 20:40:37 -0700 (PDT) Received: from thuvia.demon.co.uk (thuvia.demon.co.uk [193.237.34.248]) by mx1.FreeBSD.org (Postfix) with ESMTP id CCC7943E70 for ; Mon, 15 Jul 2002 20:40:35 -0700 (PDT) (envelope-from mark@thuvia.demon.co.uk) Received: from dotar.thuvia.org (dotar.thuvia.org [10.0.0.4]) by phaidor.thuvia.org (8.11.6/8.11.6) with ESMTP id g6G3ePb53224; Tue, 16 Jul 2002 04:40:25 +0100 (BST) (envelope-from mark@thuvia.demon.co.uk) Received: from dotar.thuvia.org (localhost [IPv6:::1]) by dotar.thuvia.org (8.12.3/8.12.3) with ESMTP id g6G3eOKC024172; Tue, 16 Jul 2002 04:40:24 +0100 (BST) (envelope-from mark@dotar.thuvia.org) Received: (from mark@localhost) by dotar.thuvia.org (8.12.3/8.12.3/Submit) id g6G3eOw1024171; Tue, 16 Jul 2002 04:40:24 +0100 (BST) Date: Tue, 16 Jul 2002 04:40:24 +0100 (BST) From: Mark Valentine Message-Id: <200207160340.g6G3eOw1024171@dotar.thuvia.org> In-Reply-To: X-Disclaimer: tequila X-Mailer: Mail User's Shell (7.2.6 beta(5) 10/07/98) To: drosih@rpi.edu (Garance A Drosihn), jos@catnook.com, freebsd-arch@freebsd.org Subject: Re: scripting language in base system? Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG > From: drosih@rpi.edu (Garance A Drosihn) > Date: Tue 16 Jul, 2002 > Subject: Re: scripting language in base system? > It seems to me that this topic has nothing to do with "the > package system". It also seems to me that this topic is a > bikeshed of immense proportions, and we shouldn't bury the > very useful discussion on package-system issues with this > bikeshed. Every thread is part of a tapestry... > For what it's worth, I'd like to see ruby in the standard freebsd > system. However, I can easily get into arguments with myself > (never mind anyone else) when trying to pin down my reasoning > for this. That's OK, I often get into the same situation, but since I read Philip K. Dick's "Clans of the Alphane Moon" it's become so much clearer. ;-) > I keep trying to write this message, listing my thoughts on > scripting languages. While I can come up with a list of ideas > that I completely agree with, the problem is that the items > in that list conflict with each other, and thus I can't even > come to a firm conclusion that *I* like. There are too many ideas to implement, never mind fit in a single language. You should see the folder of notes I have on _my_ scripting language design which started before I was even inspired/repulsed by Perl... (But there's a use for even the prototype which I did real commercial work with way back then, after having been inspired by a colleague's attempts to avoid having to write all our test suites in C!) > And in the process of > trying, I'd probably manage to irritate everyone who has any > opinion on this matter. Well, I have way too many opinions for my own good, but there's still room for more. Don't give up trying. :-) > I do not agree with the claim that sh+awk+sed is a adequate > alternative to perl, ruby, or python. I think we only disagree on "what for". > I also do not believe that > we'll take the time to write everything in C which we might find > useful, and thus useful things do not get written because we > refuse to include tools which would allow us to write such things. There are many things I don't write because it would take too long from scratch in C. But then I never found the right C libraries for them either, nor the right X toolkit, nor the right interpreted language. Nor... The things I do eventually get working generally have minimal dependencies. I've done Tcl/Tk in earnest, but Tcl's just too simplistic a syntax for me to be truly happy with (even compared with sh(1)), which is probably more important for it than my personal preference; it won me over better than the alternatives, didn't it? > Portupgrade was written in ruby because it was NOT getting written > in any of the alternatives. Indeed. It probably wouldn't exist as a (too ;-) valuable tool otherwise. It's great to have it, but what's coming from this debate is that it would be even better to have at least _some_ of its functionality in the base system. The ruby dependency is probably less of an issue (though it is one) than the fact that the functionality wants to exist as part of the existing base system packaging tools. > So, so far, I've irritated the standard > unix scripters, and the C-programmers... Don't presume... > At the same time, I don't think we can ever "be safe" with some > standard scripting language in the base system, because that > scripting language will change over time. Aha! The point is that a _good_ choice will change very little! Like, um, sh/sed/awk... Actually, even within the base system a scripting language can evolve more quickly than standards can keep up with, so long as it doesn't _break_ the standards and is easy to maintain. (Portable software, of course, can only rely on the "standard" functionality, and even then only on that which has been standard for long enough to be taken for granted.) > And my guess is that > any good scripting language will eventually evolve into it's own > "OS-neutral" platform, and thus grow into a monster that we won't > want to have in our base system. I think that's a tendency, but in some ways also a problem with the scripting language. Even as a proponent of portable software (whom you would think would advocate the "give-me-the-kitchen-sink-with-an-OS-neutral-API" approach), I think that an application should fit well into its environment. It's a case of "a bit of effort on the part of one developer makes it easier for [many] end users". > Thus, any good, standard, > popular, and useful scripting language is probably going to be > a bad choice over time. I guess it depends on "how good", "how standard" and "how popular"! So far the applicable languages seem to be "not perl", "sh" and "perl", respectively. :-) > That claim should manage to irritate > everyone else... The "can't hack it with the usual tools" group? Sure, let's go irritate them... > This is an unwinnable debate, imo. Win?? I thought it was all about spreading mindshare, even if just to a minority. ;-) Cheers, Mark. -- Mark Valentine, Thuvia Labs "Tigers will do ANYTHING for a tuna fish sandwich." Mark Valentine uses "We're kind of stupid that way." *munch* *munch* and endorses FreeBSD -- To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Mon Jul 15 21: 8:43 2002 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.FreeBSD.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 47E7C37B400 for ; Mon, 15 Jul 2002 21:08:39 -0700 (PDT) Received: from thuvia.demon.co.uk (thuvia.demon.co.uk [193.237.34.248]) by mx1.FreeBSD.org (Postfix) with ESMTP id 83C6843E6D for ; Mon, 15 Jul 2002 21:08:37 -0700 (PDT) (envelope-from mark@thuvia.demon.co.uk) Received: from dotar.thuvia.org (dotar.thuvia.org [10.0.0.4]) by phaidor.thuvia.org (8.11.6/8.11.6) with ESMTP id g6G45pb53270; Tue, 16 Jul 2002 05:05:51 +0100 (BST) (envelope-from mark@thuvia.demon.co.uk) Received: from dotar.thuvia.org (localhost [IPv6:::1]) by dotar.thuvia.org (8.12.3/8.12.3) with ESMTP id g6G45pKC024877; Tue, 16 Jul 2002 05:05:51 +0100 (BST) (envelope-from mark@dotar.thuvia.org) Received: (from mark@localhost) by dotar.thuvia.org (8.12.3/8.12.3/Submit) id g6G45peU024876; Tue, 16 Jul 2002 05:05:51 +0100 (BST) Date: Tue, 16 Jul 2002 05:05:51 +0100 (BST) From: Mark Valentine Message-Id: <200207160405.g6G45peU024876@dotar.thuvia.org> In-Reply-To: X-Disclaimer: tequila X-Mailer: Mail User's Shell (7.2.6 beta(5) 10/07/98) To: Garance A Drosihn , danfe@regency.nsu.ru (Alexey Dokuchaev), Jos Backus Subject: Re: scripting language in base system? Cc: freebsd-arch@FreeBSD.ORG Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG > From: Garance A Drosihn > Date: Mon 15 Jul, 2002 > Subject: Re: scripting language in base system? > Perhaps that is the problem. Me, I want a good package-system, but > all the 30-year sh/sed/awk veterans don't seem to see a need for it. Whoa! That's not what this sh/expr/sed veteran said. I can live without a good package system (everything is possible given our minimal toolset; and why the heck do we even need that when we can write compilers in machine code...?). ;-b But it's still nice to have, which is why I'm expending energy on this discussion. (I know, it's such a commitment compared to patches...) I happen to think that a good package system can be written with existing base system tools, but I'm not (right now) stepping forward for the assignment. If libh makes it easier, even better. If it suffers from Second System Syndrome, so be it. (I'm not suggesting that it will.) If it means ruby goes in the base system, I applaud the visionary who makes that a success where Tcl and Perl failed! Cheers, Mark. -- Mark Valentine, Thuvia Labs "Tigers will do ANYTHING for a tuna fish sandwich." Mark Valentine uses "We're kind of stupid that way." *munch* *munch* and endorses FreeBSD -- To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Mon Jul 15 21:44:37 2002 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.FreeBSD.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 2624C37B400 for ; Mon, 15 Jul 2002 21:44:36 -0700 (PDT) Received: from hotmail.com (f117.law3.hotmail.com [209.185.241.117]) by mx1.FreeBSD.org (Postfix) with ESMTP id AD1B243E72 for ; Mon, 15 Jul 2002 21:44:35 -0700 (PDT) (envelope-from gat7634@hotmail.com) Received: from mail pickup service by hotmail.com with Microsoft SMTPSVC; Mon, 15 Jul 2002 21:44:35 -0700 Received: from 149.99.123.132 by lw3fd.law3.hotmail.msn.com with HTTP; Tue, 16 Jul 2002 04:44:34 GMT X-Originating-IP: [149.99.123.132] From: "Gary Thorpe" To: tlambert2@mindspring.com Cc: arch@freebsd.org Subject: Re: Package system flaws? Date: Tue, 16 Jul 2002 00:44:34 -0400 Mime-Version: 1.0 Content-Type: text/plain; format=flowed Message-ID: X-OriginalArrivalTime: 16 Jul 2002 04:44:35.0630 (UTC) FILETIME=[7C39DCE0:01C22C83] Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG >From: Terry Lambert >To: Mark Valentine >CC: "Louis A. Mamakos" ,Thomas Seck >, freebsd-arch@freebsd.org >Subject: Re: Package system flaws? >Date: Mon, 15 Jul 2002 15:01:47 -0700 > [...] >"Modularizing the base system" is not a problem; it is the >solution to a problem -- and not just an ends in itself, as your >lack of certainty implies. Performing that task has consequences >significantly above and beyond the direct consequences any casual >observer would expect. On the topic of modularity, is an standard, official ABI for FreeBSD planned? Are there plans for standardizing binary interfaces for kernel modules. Would any of this (theoretical or actual) allow the kernel to be updated relatively independently of the rest of the system? _________________________________________________________________ Chat with friends online, try MSN Messenger: http://messenger.msn.com To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Mon Jul 15 23:15:46 2002 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.FreeBSD.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 95A3137B400 for ; Mon, 15 Jul 2002 23:15:44 -0700 (PDT) Received: from hawk.mail.pas.earthlink.net (hawk.mail.pas.earthlink.net [207.217.120.22]) by mx1.FreeBSD.org (Postfix) with ESMTP id 370E943E42 for ; Mon, 15 Jul 2002 23:15:44 -0700 (PDT) (envelope-from tlambert2@mindspring.com) Received: from pool0127.cvx21-bradley.dialup.earthlink.net ([209.179.192.127] helo=mindspring.com) by hawk.mail.pas.earthlink.net with esmtp (Exim 3.33 #1) id 17ULcb-0007V1-00; Mon, 15 Jul 2002 23:15:41 -0700 Message-ID: <3D33B9D2.9D10EEDB@mindspring.com> Date: Mon, 15 Jul 2002 23:14:42 -0700 From: Terry Lambert X-Mailer: Mozilla 4.79 [en] (Win98; U) X-Accept-Language: en MIME-Version: 1.0 To: Gary Thorpe Cc: arch@freebsd.org Subject: Re: Package system flaws? References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG Gary Thorpe wrote: > On the topic of modularity, is an standard, official ABI for FreeBSD > planned? Are there plans for standardizing binary interfaces for kernel > modules. Would any of this (theoretical or actual) allow the kernel to be > updated relatively independently of the rest of the system? You're asking two different questions, neither of which have anything to do with the subject. 8-). Not that they shouldn't be addressed, but they should probably at least be *seperately* addressed (IMO). -- Terry To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Tue Jul 16 0:32:36 2002 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.FreeBSD.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 67A9237B400 for ; Tue, 16 Jul 2002 00:32:33 -0700 (PDT) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.FreeBSD.org (Postfix) with ESMTP id 2F33E43E3B for ; Tue, 16 Jul 2002 00:32:33 -0700 (PDT) (envelope-from baka@elvis.mu.org) Received: by elvis.mu.org (Postfix, from userid 1921) id E882DAE165; Tue, 16 Jul 2002 00:32:32 -0700 (PDT) Date: Tue, 16 Jul 2002 00:32:32 -0700 From: Jon Mini To: Garance A Drosihn Cc: jos@catnook.com, freebsd-arch@FreeBSD.ORG Subject: An odd scripting language Message-ID: <20020716073232.GD55378@elvis.mu.org> References: <200207151718.g6FHIkof007662@dotar.thuvia.org> <20020715182957.GA32690@lizzy.catnook.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.3.27i Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG On Mon, Jul 15, 2002 at 09:56:07PM -0400, Garance A Drosihn wrote: > > At the same time, I don't think we can ever "be safe" with some > standard scripting language in the base system, because that > scripting language will change over time. And my guess is that > any good scripting language will eventually evolve into it's own > "OS-neutral" platform, and thus grow into a monster that we won't > want to have in our base system. Thus, any good, standard, > popular, and useful scripting language is probably going to be > a bad choice over time. That claim should manage to irritate > everyone else... > > This is an unwinnable debate, imo. Truely. However, I am intruiged by your assertion here. It implies that perhaps a good direction to try would be to build a scripting language that is tied to FreeBSD, but contains elements that makes it easy to pick up by people who are already comfortable with the popular scripting languages. This is an interesting concept. Fraught with problems, but nonetheless interesting. -- Jonathan Mini http://www.freebsd.org/ To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Tue Jul 16 0:40:27 2002 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.FreeBSD.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 2DA7237B400 for ; Tue, 16 Jul 2002 00:40:26 -0700 (PDT) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.FreeBSD.org (Postfix) with ESMTP id E99AB43E58 for ; Tue, 16 Jul 2002 00:40:25 -0700 (PDT) (envelope-from baka@elvis.mu.org) Received: by elvis.mu.org (Postfix, from userid 1921) id BC738AE1EE; Tue, 16 Jul 2002 00:40:25 -0700 (PDT) Date: Tue, 16 Jul 2002 00:40:25 -0700 From: Jon Mini To: Garance A Drosihn Cc: jos@catnook.com, freebsd-arch@FreeBSD.ORG Subject: Re: An odd scripting language Message-ID: <20020716074025.GE55378@elvis.mu.org> References: <200207151718.g6FHIkof007662@dotar.thuvia.org> <20020715182957.GA32690@lizzy.catnook.com> <20020716073232.GD55378@elvis.mu.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20020716073232.GD55378@elvis.mu.org> User-Agent: Mutt/1.3.27i Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG On Tue, Jul 16, 2002 at 12:32:32AM -0700, Jon Mini wrote: > > However, I am intruiged by your assertion here. It implies that perhaps > a good direction to try would be to build a scripting language that is > tied to FreeBSD, but contains elements that makes it easy to pick up > by people who are already comfortable with the popular scripting languages. T BAKA THE WORLD DOESN'T NEED ANOTHER SCRIPTING LANGUAGE. PLEASE DO NOT ENCOURAGE THEM K THX. T * PLZ TO NOT BE MAKING MORE LANGUAGES K THX BYE -- Jonathan Mini http://www.freebsd.org/ To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Tue Jul 16 1:19:21 2002 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.FreeBSD.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 4143937B400 for ; Tue, 16 Jul 2002 01:19:17 -0700 (PDT) Received: from jkh-gw.queasyweasel.com (adsl-64-173-3-158.dsl.sntc01.pacbell.net [64.173.3.158]) by mx1.FreeBSD.org (Postfix) with ESMTP id 7642543E5E for ; Tue, 16 Jul 2002 01:19:16 -0700 (PDT) (envelope-from jkh@queasyweasel.com) Received: from mango.queasyweasel.com (mango.freebsd.com [64.173.15.99]) by jkh-gw.queasyweasel.com (8.12.5/8.12.5) with ESMTP id g6G8J4ux011916; Tue, 16 Jul 2002 01:19:04 -0700 (PDT) (envelope-from jkh@queasyweasel.com) Date: Tue, 16 Jul 2002 01:19:42 -0700 Subject: Re: Package system flaws? Content-Type: text/plain; charset=US-ASCII; format=flowed Mime-Version: 1.0 (Apple Message framework v535) Cc: freebsd-arch@FreeBSD.ORG To: jos@catnook.com From: Jordan K Hubbard In-Reply-To: <20020716003456.GD54500@lizzy.catnook.com> Message-Id: Content-Transfer-Encoding: 7bit X-Mailer: Apple Mail (2.535) Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG Guys, If libh ever makes it off the ground, you can bet that Tcl will enter the base system fairly rapidly since it will be required for everything from bootstrapping packages onto the system to actually installing the system itself. In such a case, you can also rest assured that Tcl *will* have a more than justifiable purpose and I'm sure that even the anti-bloatists will be forced to grudgingly concede the point. Their principle, and successful, objection before was that Tcl entered the base system long before there was such a demonstrable need for it and it was justifiably kicked right back out again. The issue with Perl is orthogonal since it wasn't quite as insinuated into the brick-and-mortar foundations of the system (some of those stray .pl admin scripts notwithstanding) and was much harder to maintain than the _current_ releases of Tcl are. I also agree that Tcl has had a rocky history in terms of its upgrade strategy ("it must be Tuesday, time to break API stability again!") but, for better or worse, development of the language seems to have reached a plateau with 8.4 and API stability ever since 8.0 was released has been pretty good, so I think the old arguments are simply outdated. In any case, the anti-bloatists can also leave their pitchforks and torches in their closets for the time being since all of this is also contingent on libh finally reaching the point where it is a truly effective replacement for pkg_* and sysinstall, something which is still a ways off if it happens at all. - Jordan On Monday, July 15, 2002, at 05:34 PM, Jos Backus wrote: > On Mon, Jul 15, 2002 at 09:44:45PM +0700, Alexey Dokuchaev wrote: >> We did have a very powerful one until recently -- Perl. I guess the >> fact it was removed from the base is for a very good reason. > > I don't think ``we don't want a powerful scripting language in the base > system'' was one of them. > >> Oh please, can you show us something essential for base enough that >> could not be implemented in a sh/sed/awk way? I somewhat doubt it. > > portupgrade :-) (Sure, anything that can be done in Ruby can be done > in C; no > need to make that argument again). > >> 'cmon, it's pretty clear that Perl or Ruby is more of an overhead >> than of >> worth. Traditionally, UNIX lived for 30+ years without need for a >> monster >> like Perl or Ruby, in the base, clearly showing us that sh/awk/sed is >> a >> [very] decent scripting facility. > > As must be clear by now, I respectfully beg to differ. > > -- > Jos Backus _/ _/_/_/ Santa Clara, CA > _/ _/ _/ > _/ _/_/_/ > _/ _/ _/ _/ > jos@catnook.com _/_/ _/_/_/ require 'std/disclaimer' > > To Unsubscribe: send mail to majordomo@FreeBSD.org > with "unsubscribe freebsd-arch" in the body of the message > -- Jordan K. Hubbard Engineering Manager, BSD technology group Apple Computer To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Tue Jul 16 1:24: 7 2002 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.FreeBSD.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 39B3537B400 for ; Tue, 16 Jul 2002 01:24:06 -0700 (PDT) Received: from jkh-gw.queasyweasel.com (adsl-64-173-3-158.dsl.sntc01.pacbell.net [64.173.3.158]) by mx1.FreeBSD.org (Postfix) with ESMTP id A9A9543E4A for ; Tue, 16 Jul 2002 01:24:05 -0700 (PDT) (envelope-from jkh@queasyweasel.com) Received: from mango.queasyweasel.com (mango.freebsd.com [64.173.15.99]) by jkh-gw.queasyweasel.com (8.12.5/8.12.5) with ESMTP id g6G8Nrux011927; Tue, 16 Jul 2002 01:23:54 -0700 (PDT) (envelope-from jkh@queasyweasel.com) Date: Tue, 16 Jul 2002 01:24:31 -0700 Subject: Re: scripting language in base system? Content-Type: text/plain; charset=US-ASCII; format=flowed Mime-Version: 1.0 (Apple Message framework v535) Cc: Mark Valentine , danfe@regency.nsu.ru (Alexey Dokuchaev), Jos Backus , freebsd-arch@FreeBSD.ORG To: Garance A Drosihn From: Jordan K Hubbard In-Reply-To: Message-Id: <73FF8785-9895-11D6-BCA3-0003938C7B7E@queasyweasel.com> Content-Transfer-Encoding: 7bit X-Mailer: Apple Mail (2.535) Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG On Monday, July 15, 2002, at 06:57 PM, Garance A Drosihn wrote: > Perhaps that is the problem. Me, I want a good package-system, but > all the 30-year sh/sed/awk veterans don't seem to see a need for it. A good package system isn't really intended for them anyway since the 30-year veterans usually just grab the sources off their favorite archive site and compile it themselves anyway. "Package systems are a crutch, man" they say, just before falling asleep in front of the TV with a blanket on their laps. I think package systems and fancy installers are for the other 98%. :-) -- Jordan K. Hubbard Engineering Manager, BSD technology group Apple Computer To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Tue Jul 16 2:24:45 2002 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.FreeBSD.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 7700637B400 for ; Tue, 16 Jul 2002 02:24:42 -0700 (PDT) Received: from flood.ping.uio.no (flood.ping.uio.no [129.240.78.31]) by mx1.FreeBSD.org (Postfix) with ESMTP id 0AB7543E58 for ; Tue, 16 Jul 2002 02:24:42 -0700 (PDT) (envelope-from des@ofug.org) Received: by flood.ping.uio.no (Postfix, from userid 2602) id 16C85534A; Tue, 16 Jul 2002 11:24:40 +0200 (CEST) X-URL: http://www.ofug.org/~des/ X-Disclaimer: The views expressed in this message do not necessarily coincide with those of any organisation or company with which I am or have been affiliated. To: Jordan K Hubbard Cc: jos@catnook.com, freebsd-arch@FreeBSD.ORG Subject: Re: Package system flaws? References: From: Dag-Erling Smorgrav Date: 16 Jul 2002 11:24:39 +0200 In-Reply-To: Message-ID: Lines: 14 User-Agent: Gnus/5.0808 (Gnus v5.8.8) Emacs/21.2 MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG Jordan K Hubbard writes: > If libh ever makes it off the ground, you can bet that Tcl will enter > the base system fairly rapidly since it will be required for > everything from bootstrapping packages onto the system to actually > installing the system itself. I'm not a big fan of having Tcl in the base system, but - what are we waiting for? libh should have entered the tree a long time ago, there's absolutely nothing to be gained from leaving it out in the cold. DES -- Dag-Erling Smorgrav - des@ofug.org To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Tue Jul 16 2:47:45 2002 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.FreeBSD.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 9DC2C37B400 for ; Tue, 16 Jul 2002 02:47:43 -0700 (PDT) Received: from critter.freebsd.dk (critter.freebsd.dk [212.242.86.163]) by mx1.FreeBSD.org (Postfix) with ESMTP id DAAE743E42 for ; Tue, 16 Jul 2002 02:47:42 -0700 (PDT) (envelope-from phk@critter.freebsd.dk) Received: from critter.freebsd.dk (localhost [127.0.0.1]) by critter.freebsd.dk (8.12.3/8.12.2) with ESMTP id g6G9i8XB092475; Tue, 16 Jul 2002 11:44:08 +0200 (CEST) (envelope-from phk@critter.freebsd.dk) To: Dag-Erling Smorgrav Cc: Jordan K Hubbard , jos@catnook.com, freebsd-arch@FreeBSD.ORG Subject: Re: Package system flaws? In-Reply-To: Your message of "16 Jul 2002 11:24:39 +0200." Date: Tue, 16 Jul 2002 11:44:08 +0200 Message-ID: <92474.1026812648@critter.freebsd.dk> From: Poul-Henning Kamp Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG In message , Dag-Erling Smorgrav writes: >Jordan K Hubbard writes: >> If libh ever makes it off the ground, you can bet that Tcl will enter >> the base system fairly rapidly since it will be required for >> everything from bootstrapping packages onto the system to actually >> installing the system itself. > >I'm not a big fan of having Tcl in the base system, but - what are we >waiting for? libh should have entered the tree a long time ago, >there's absolutely nothing to be gained from leaving it out in the >cold. It was my impression that we had abandonned the "lets put it in the tree and hope somebody else finishes it" policy long time ago ? -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk@FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Tue Jul 16 2:54:58 2002 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.FreeBSD.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id A92F837B400 for ; Tue, 16 Jul 2002 02:54:51 -0700 (PDT) Received: from gw.catspoiler.org (217-ip-163.nccn.net [209.79.217.163]) by mx1.FreeBSD.org (Postfix) with ESMTP id 0811B43E4A for ; Tue, 16 Jul 2002 02:54:51 -0700 (PDT) (envelope-from dl-freebsd@catspoiler.org) Received: from mousie.catspoiler.org (mousie.catspoiler.org [192.168.101.2]) by gw.catspoiler.org (8.12.5/8.12.5) with ESMTP id g6G9scwr026538; Tue, 16 Jul 2002 02:54:42 -0700 (PDT) (envelope-from dl-freebsd@catspoiler.org) Message-Id: <200207160954.g6G9scwr026538@gw.catspoiler.org> Date: Tue, 16 Jul 2002 02:54:38 -0700 (PDT) From: Don Lewis Subject: Re: wiring the sysctl output buffer To: bde@zeta.org.au Cc: arch@FreeBSD.ORG In-Reply-To: <20020714201020.F35894-100000@gamplex.bde.org> MIME-Version: 1.0 Content-Type: TEXT/plain; charset=US-ASCII Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG On 14 Jul, Bruce Evans wrote: > On Sun, 14 Jul 2002, Don Lewis wrote: > >> A number of places in the kernel call SYSCTL_OUT() while holding locks. >> The problem is that SYSCTL_OUT() can block while it wires down the "old" >> buffer in user space. If the data is small, such as an integer value, > > This seems to have been broken by FreeBSD. In rev.1.1, vslock() was > called at the start of the sysctl, so the only problem with it was its > pessimalness (*). Now I think the vslock() is just a pessimization, > since the point of doing it was to avoid blocking during the copyout(), > but blocking during vslock() is equivalent. Also, vslock() is insufficent > if the kernel is preemptible. Yes, in some ways it looks like we went backwards. We can solve the problem of the data changing while it is being copied by locking the source of the data, but we need to call vslock() before locking the source of the data to avoid being blocked in vslock() while holding locks. The current implematation which calls vslock() in the first call to SYSCTL_OUT() is the reason that I'm looking at this stuff. >> the best solution may be to copy the data and defer calling SYSCTL_OUT() >> until after the locks are released. This extra copy could be wasteful >> if the data is large, and it may be cumbersome if the data size is not >> known ahead of time, since determining the data size may be expensive. > > The extra copy is unlikely to be more wasteful than vslock()'ing, since > vfslock() is a very expensive operation. Some instruction counts on > a UP i386 kernel with no INVARIANTS or WITNESS: > > (*) vslock(): about 2700 instructions (for a length of just 16) > vsunlock(): about 1400 > useracc(): about 400 > > `oldlen' is known ahead of time, so a large enough buffer could be > allocated in most cases. Perhaps vslock() iff oldlen is very large. > The data could still change while it is being copied to a malloc()'ed > buffer (or earlier) if the kernel is preemptible. I basically like this approach. In most cases the data will be a small and of fixed size, so the handler could even store the temporary copy on the stack. If the data is a type that can be copied atomically, like an int, we don't have to worry about the data changing in mid-copy. If it is larger and we care about getting an atomic snapshot, the handler can lock the source for the duration of the copy to the temporary buffer. For large amounts of data, we probably want to avoid copying to a temporary buffer. PHK raised the issue of the user specifying a buffer size that is way too large. Allocating a huge buffer in wired down kernel memory sounds even worse than wiring down a huge buffer in user space. If we don't want to allocate a temporary buffer of length oldlen because it is much larger than what looks to be reasonable, but the amount of data is not easy to calculate ahead of time, we have another potential problem. We might have to lock a data structure, walk through this data structure and calculate the total data size, and unlock the data structure. After we malloc() the buffer, relock the data structure, and start traversing the data structure again to do the copy, we may discover that the amount of data to be returned has increased, so we would have to drop the lock, free the buffer, and start over again. >> The data size could also potentially change between the time the size is >> calculatated and the time when the data is copied to the temporary >> buffer if the locks must be released so that MALLOC() can be called to >> allocate the buffer. > > Hopefully the sysctl locking is enough to prevent at least *req changing > underneath us. > >> I think the best solution to this problem is to add a sysctl system API >> to prewire the output buffer that can be called before grabbing the >> locks. Doing so allows the existing code to operate properly with only >> minimal changes. > >> Here's a patch that implements this new API and illustrates how it can >> be used to fix the kern.function_list sysctl, which wants to call >> SYSCTL_OUT() multiple times while walking a locked data structure. I >> think this is the best fix, though I'd like some feedback on whether >> this is the best API. > > I think the callers of SYSCTL_OUT() don't need a new API, but they > should supply the data in a form suitable for copying out directly. > This probably involves them making a copy while holding suitable > domain-specific locks. They can't just point to an active kernel > struct since it might change. > > I would just make a copy at a high level for now. I'd vote for a hybrid approach. I would remove the vslock() call from sysctl_old_user() and provide the new API for invoking it from the handler if appropriate. Instead, I would call WITNESS_SLEEP() if vslock() had not been called to wire the buffer to guard against blocking while locks are held. I would keep the call to vsunlock() in the sysctl cleanup code. If the handler makes a copy of the data and releases any locks before calling SYSCTL_OUT(), then we can avoid the expense of vslock(). If appropriate, the handler would still have the ability to wire the buffer before grabbing a lock, and then invoke SYSCTL_OUT() while the lock is held. I'd also like to provide a second argument to the interface to vslock() to allow the handler to specify a maximum, kernel enforced, buffer length to potentially limit the amount of wired memory to something less than oldlen, but unless we add a member to the sysctl_req structure to hold it, we would need to overwrite oldlen so that we can pass the proper value to vsunlock() in the cleanup code. What kind impact would changing sysctl_req have on binary compatibility? I doesn't look like a problem to me, since this structure seems to be mostly treated as an opaque type. To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Tue Jul 16 3:10:16 2002 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.FreeBSD.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 4E1FB37B400 for ; Tue, 16 Jul 2002 03:10:14 -0700 (PDT) Received: from flood.ping.uio.no (flood.ping.uio.no [129.240.78.31]) by mx1.FreeBSD.org (Postfix) with ESMTP id D44F043E31 for ; Tue, 16 Jul 2002 03:10:13 -0700 (PDT) (envelope-from des@ofug.org) Received: by flood.ping.uio.no (Postfix, from userid 2602) id 6C423534A; Tue, 16 Jul 2002 12:10:10 +0200 (CEST) X-URL: http://www.ofug.org/~des/ X-Disclaimer: The views expressed in this message do not necessarily coincide with those of any organisation or company with which I am or have been affiliated. To: Poul-Henning Kamp Cc: Jordan K Hubbard , jos@catnook.com, freebsd-arch@FreeBSD.ORG Subject: Re: Package system flaws? References: <92474.1026812648@critter.freebsd.dk> From: Dag-Erling Smorgrav Date: 16 Jul 2002 12:10:10 +0200 In-Reply-To: <92474.1026812648@critter.freebsd.dk> Message-ID: Lines: 15 User-Agent: Gnus/5.0808 (Gnus v5.8.8) Emacs/21.2 MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG Poul-Henning Kamp writes: > In message , Dag-Erling Smorgrav writes: > > I'm not a big fan of having Tcl in the base system, but - what are we > > waiting for? libh should have entered the tree a long time ago, > > there's absolutely nothing to be gained from leaving it out in the > > cold. > It was my impression that we had abandonned the "lets put it in the > tree and hope somebody else finishes it" policy long time ago ? "somebody else"? Unless alex@ dropped dead while I wasn't looking, libh is actively maintained. DES -- Dag-Erling Smorgrav - des@ofug.org To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Tue Jul 16 9:19:59 2002 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.FreeBSD.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 07C9E37B400 for ; Tue, 16 Jul 2002 09:19:54 -0700 (PDT) Received: from postal.sdsc.edu (postal.sdsc.edu [132.249.20.114]) by mx1.FreeBSD.org (Postfix) with ESMTP id 3A1C143E5E for ; Tue, 16 Jul 2002 09:19:53 -0700 (PDT) (envelope-from okumoto@SDSC.EDU) Received: from multivac.sdsc.edu (multivac.sdsc.edu [132.249.20.57]) by postal.sdsc.edu (8.11.6/8.11.6/server/43) with ESMTP id g6GGJfc11034; Tue, 16 Jul 2002 09:19:41 -0700 (PDT) Received: by multivac (8.11.6+Sun/1.11-SolarisClient) id g6GGJfR07406; Tue, 16 Jul 2002 09:19:41 -0700 (PDT) To: Dag-Erling Smorgrav Cc: Jordan K Hubbard , jos@catnook.com, freebsd-arch@FreeBSD.ORG Subject: Re: Package system flaws? References: From: Max Okumoto Date: 16 Jul 2002 09:19:41 -0700 In-Reply-To: Dag-Erling Smorgrav's message of "16 Jul 2002 11:24:39 +0200" Message-ID: Lines: 30 X-Mailer: Gnus v5.5/XEmacs 20.4 - "Emerald" Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG There has been lot of changes going into the code recently. And we are still working on some of the basic framework. Right now I am working on refactoring the graphic/text backends. In addition I am adding lots of comments to the code. The original programmer did not document his code much. :-P I would like to wait a little bit more before we move it into the main cvs tree. The current work and status is accessable at http://usw4.FreeBSD.org/~libh/ Max Okumoto Dag-Erling Smorgrav writes: > Jordan K Hubbard writes: > > If libh ever makes it off the ground, you can bet that Tcl will enter > > the base system fairly rapidly since it will be required for > > everything from bootstrapping packages onto the system to actually > > installing the system itself. > > I'm not a big fan of having Tcl in the base system, but - what are we > waiting for? libh should have entered the tree a long time ago, > there's absolutely nothing to be gained from leaving it out in the > cold. > > DES > -- > Dag-Erling Smorgrav - des@ofug.org > > To Unsubscribe: send mail to majordomo@FreeBSD.org > with "unsubscribe freebsd-arch" in the body of the message To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Tue Jul 16 9:36: 3 2002 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.FreeBSD.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id B763137B400 for ; Tue, 16 Jul 2002 09:35:58 -0700 (PDT) Received: from mail.rpi.edu (mail.rpi.edu [128.113.22.40]) by mx1.FreeBSD.org (Postfix) with ESMTP id 35C2B43E3B for ; Tue, 16 Jul 2002 09:35:58 -0700 (PDT) (envelope-from drosih@rpi.edu) Received: from [128.113.24.47] (gilead.netel.rpi.edu [128.113.24.47]) by mail.rpi.edu (8.12.1/8.12.1) with ESMTP id g6GGZudW130496; Tue, 16 Jul 2002 12:35:56 -0400 Mime-Version: 1.0 X-Sender: drosih@mail.rpi.edu Message-Id: In-Reply-To: <200207160340.g6G3eOw1024171@dotar.thuvia.org> References: <200207160340.g6G3eOw1024171@dotar.thuvia.org> Date: Tue, 16 Jul 2002 12:35:55 -0400 To: Mark Valentine , freebsd-arch@freebsd.org From: Garance A Drosihn Subject: Re: scripting language in base system? Content-Type: text/plain; charset="us-ascii" ; format="flowed" X-Scanned-By: MIMEDefang 2.3 (www dot roaringpenguin dot com slash mimedefang) Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG At 4:40 AM +0100 7/16/02, Mark Valentine wrote: >drosih@rpi.edu (Garance A Drosihn) writes: > > I keep trying to write this message, listing my thoughts on >> scripting languages. While I can come up with a list of ideas >> that I completely agree with, the problem is that the items >> in that list conflict with each other, and thus I can't even >> come to a firm conclusion that *I* like. > >There are too many ideas to implement, never mind fit in a >single language. You're a step ahead of where I am. I meant listing my thoughts on just whether we even *should* have a higher-level scripting language in the base system -- never mind what it might look like! I have definite, firm opinions on both sides of the matter... > > This is an unwinnable debate, imo. > >Win?? I thought it was all about spreading mindshare, even >if just to a minority. ;-) Win? I can't even decide which side I'm on... [well, I'd definitely be happy to see a BSD-licensed ruby in the base system, but I can't really defend why it should be ruby as opposed to something else like python. I also know that tcl drives me nuts, even though I know other programmers who really like it] -- Garance Alistair Drosehn = gad@gilead.netel.rpi.edu Senior Systems Programmer or gad@freebsd.org Rensselaer Polytechnic Institute or drosih@rpi.edu To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Tue Jul 16 9:43: 8 2002 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.FreeBSD.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id E783637B400 for ; Tue, 16 Jul 2002 09:43:06 -0700 (PDT) Received: from mail1.panix.com (mail1.panix.com [166.84.1.72]) by mx1.FreeBSD.org (Postfix) with ESMTP id 848B643E3B for ; Tue, 16 Jul 2002 09:43:06 -0700 (PDT) (envelope-from rsi@panix.com) Received: from panix1.panix.com (panix1.panix.com [166.84.1.1]) by mail1.panix.com (Postfix) with ESMTP id B19B248719 for ; Tue, 16 Jul 2002 12:43:05 -0400 (EDT) Received: (from rsi@localhost) by panix1.panix.com (8.11.3nb1/8.8.8/PanixN1.0) id g6GGh5H07569; Tue, 16 Jul 2002 12:43:05 -0400 (EDT) Message-Id: <200207161643.g6GGh5H07569@panix1.panix.com> X-Authentication-Warning: panix1.panix.com: rsi set sender to rsi@panix.com using -f To: freebsd-arch@freebsd.org Subject: Re: scripting language in base system? References: <200207160340.g6G3eOw1024171@dotar.thuvia.org> From: Rajappa Iyer Date: 16 Jul 2002 12:43:05 -0400 Reply-To: rsi@panix.com MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG Suggestion for a scripting language: scsh It's an excellent, non-hacky, language, decent performance, stable interface and has a BSD type license. www.scsh.net What do people think? Rajappa -- a.k.a. Rajappa Iyer. They also surf who stand in the waves. To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Tue Jul 16 9:47:57 2002 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.FreeBSD.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id E206037B400 for ; Tue, 16 Jul 2002 09:47:52 -0700 (PDT) Received: from mail.rpi.edu (mail.rpi.edu [128.113.22.40]) by mx1.FreeBSD.org (Postfix) with ESMTP id AC66043E31 for ; Tue, 16 Jul 2002 09:47:51 -0700 (PDT) (envelope-from drosih@rpi.edu) Received: from [128.113.24.47] (gilead.netel.rpi.edu [128.113.24.47]) by mail.rpi.edu (8.12.1/8.12.1) with ESMTP id g6GGlmdW109332; Tue, 16 Jul 2002 12:47:48 -0400 Mime-Version: 1.0 X-Sender: drosih@mail.rpi.edu Message-Id: In-Reply-To: <20020716073232.GD55378@elvis.mu.org> References: <200207151718.g6FHIkof007662@dotar.thuvia.org> <20020715182957.GA32690@lizzy.catnook.com> <20020716073232.GD55378@elvis.mu.org> Date: Tue, 16 Jul 2002 12:47:47 -0400 To: Jon Mini From: Garance A Drosihn Subject: Re: An odd scripting language Cc: freebsd-arch@FreeBSD.ORG Content-Type: text/plain; charset="us-ascii" ; format="flowed" X-Scanned-By: MIMEDefang 2.3 (www dot roaringpenguin dot com slash mimedefang) Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG At 12:32 AM -0700 7/16/02, Jon Mini wrote: >On Mon, Jul 15, 2002, Garance A Drosihn wrote: > > >> At the same time, I don't think we can ever "be safe" with some >> standard scripting language in the base system, because that >> scripting language will change over time. And my guess is that >> any good scripting language will eventually evolve into it's own >> "OS-neutral" platform, and thus grow into a monster that we won't > > want to have in our base system. > >However, I am intruiged by your assertion here. It implies that >perhaps a good direction to try would be to build a scripting >language that is tied to FreeBSD, but contains elements that makes >it easy to pick up by people who are already comfortable with >the popular scripting languages. That is kind of where I'm heading in my own thoughts, but you immediately run into problems over "what should that language look like?". I'd like something that looks like ruby, because I like ruby. I have friends who swear by python, so they'd probably like something that looked like python. If I really were to pursue this, I guess I'd first try to figure out what exactly it is that (say) ruby provides which is missing from plain sh/awk/sed, and then try to provide that minimal set of missing capability. The problem is that I'd rather just write some scripts in ruby than to try to figure out some new, minimalist scripting language... -- Garance Alistair Drosehn = gad@gilead.netel.rpi.edu Senior Systems Programmer or gad@freebsd.org Rensselaer Polytechnic Institute or drosih@rpi.edu To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Tue Jul 16 9:58:33 2002 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.FreeBSD.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 62B5237B400 for ; Tue, 16 Jul 2002 09:58:29 -0700 (PDT) Received: from mail.rpi.edu (mail.rpi.edu [128.113.22.40]) by mx1.FreeBSD.org (Postfix) with ESMTP id 92F4F43E31 for ; Tue, 16 Jul 2002 09:58:28 -0700 (PDT) (envelope-from drosih@rpi.edu) Received: from [128.113.24.47] (gilead.netel.rpi.edu [128.113.24.47]) by mail.rpi.edu (8.12.1/8.12.1) with ESMTP id g6GGwNdW098866; Tue, 16 Jul 2002 12:58:24 -0400 Mime-Version: 1.0 X-Sender: drosih@mail.rpi.edu Message-Id: In-Reply-To: References: Date: Tue, 16 Jul 2002 12:58:22 -0400 To: Jordan K Hubbard , jos@catnook.com From: Garance A Drosihn Subject: Re: scripting language in base system? Cc: freebsd-arch@FreeBSD.ORG Content-Type: text/plain; charset="us-ascii" ; format="flowed" X-Scanned-By: MIMEDefang 2.3 (www dot roaringpenguin dot com slash mimedefang) Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG At 1:19 AM -0700 7/16/02, Jordan K Hubbard wrote: >Guys, > >If libh ever makes it off the ground, you can bet that Tcl will >enter the base system fairly rapidly since it will be required >for everything from bootstrapping packages onto the system to >actually installing the system itself. >I also agree that Tcl has had a rocky history in terms of its >upgrade strategy, but, for better or worse, development of the >language seems to have reached a plateau with 8.4 and API >stability ever since 8.0 was released has been pretty good, so >I think the old arguments are simply outdated. When I think about something like this, I wonder if we should put tcl into the system under some unique name ("tclb"?), just so *we* can decide if the base-system tcl will change when the next great tcl API shows up. [the same would apply to ruby or python in the base system] One of the big pains with perl in the base system was that we wanted that perl to remain stable (at least for the entire lifetime of a freebsd-stable branch), while anyone who used perl heavily would want new versions of perl as they became stable. Ie, perl's "stable branch" is not on the same timetable as freebsd's. -- Garance Alistair Drosehn = gad@gilead.netel.rpi.edu Senior Systems Programmer or gad@freebsd.org Rensselaer Polytechnic Institute or drosih@rpi.edu To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Tue Jul 16 10: 4:34 2002 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.FreeBSD.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id D0A5837B400 for ; Tue, 16 Jul 2002 10:04:31 -0700 (PDT) Received: from blackhelicopters.org (geburah.blackhelicopters.org [209.69.178.18]) by mx1.FreeBSD.org (Postfix) with ESMTP id 53D5343E3B for ; Tue, 16 Jul 2002 10:04:31 -0700 (PDT) (envelope-from mwlucas@blackhelicopters.org) Received: from blackhelicopters.org (mwlucas@localhost [127.0.0.1]) by blackhelicopters.org (8.12.4/8.12.4) with ESMTP id g6GH4QcC016214; Tue, 16 Jul 2002 13:04:26 -0400 (EDT) (envelope-from mwlucas@blackhelicopters.org) Received: (from mwlucas@localhost) by blackhelicopters.org (8.12.4/8.12.4/Submit) id g6GH4QPf016213; Tue, 16 Jul 2002 13:04:26 -0400 (EDT) Date: Tue, 16 Jul 2002 13:04:26 -0400 From: Michael Lucas To: Garance A Drosihn Cc: Mark Valentine , freebsd-arch@FreeBSD.ORG Subject: Re: scripting language in base system? Message-ID: <20020716130426.A16051@blackhelicopters.org> References: <200207160340.g6G3eOw1024171@dotar.thuvia.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2.5i In-Reply-To: ; from drosih@rpi.edu on Tue, Jul 16, 2002 at 12:35:55PM -0400 Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG On Tue, Jul 16, 2002 at 12:35:55PM -0400, Garance A Drosihn wrote: > > > This is an unwinnable debate, imo. (Picking this message to respond to, out of the many in this thread): Guys, Might I suggest a thinking sideways to some of this? There's a difference between "included in the base install" and "installed by default." Perl is not in the base system, but it is installed by default. We could tack a menu into sysinstall, much like the current Perl sysinstall option: "Would you like to install the tools to upgrade your FreeBSD installation?" If they say "Yes" we then install cvsup (without X if no X is installed), portupgrade, and attendant paraphenalia. This gives us the ability to support our users more easily, while satisfying the sed-scarred vets. From an advocacy and support view, this would make life much simpler. ("You're an ignorant newbie who took the defaults? Don't worry, these tools are installed.") OK, I'll shut up now. ==ml -- Michael Lucas mwlucas@FreeBSD.org, mwlucas@BlackHelicopters.org http://www.oreillynet.com/pub/q/Big_Scary_Daemons Absolute BSD: http://www.nostarch.com/abs_bsd.htm To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Tue Jul 16 10:20:57 2002 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.FreeBSD.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 641B937B400 for ; Tue, 16 Jul 2002 10:20:54 -0700 (PDT) Received: from storm.FreeBSD.org.uk (storm.FreeBSD.org.uk [194.242.139.170]) by mx1.FreeBSD.org (Postfix) with ESMTP id 7FB6A43E58 for ; Tue, 16 Jul 2002 10:20:53 -0700 (PDT) (envelope-from mark@grimreaper.grondar.org) Received: from storm.FreeBSD.org.uk (uucp@localhost [127.0.0.1]) by storm.FreeBSD.org.uk (8.12.5/8.12.5) with ESMTP id g6GHKjYm014829; Tue, 16 Jul 2002 18:20:45 +0100 (BST) (envelope-from mark@grimreaper.grondar.org) Received: (from uucp@localhost) by storm.FreeBSD.org.uk (8.12.5/8.12.5/Submit) with UUCP id g6GHKj63014828; Tue, 16 Jul 2002 18:20:45 +0100 (BST) Received: from grimreaper.grondar.org (localhost [127.0.0.1]) by grimreaper.grondar.org (8.12.5/8.12.5) with ESMTP id g6GHFauH034332; Tue, 16 Jul 2002 18:15:36 +0100 (BST) (envelope-from mark@grimreaper.grondar.org) Message-Id: <200207161715.g6GHFauH034332@grimreaper.grondar.org> To: Jon Mini Cc: freebsd-arch@FreeBSD.ORG Subject: Re: An odd scripting language References: <20020716073232.GD55378@elvis.mu.org> In-Reply-To: <20020716073232.GD55378@elvis.mu.org> ; from Jon Mini "Tue, 16 Jul 2002 00:32:32 PDT." Date: Tue, 16 Jul 2002 18:15:36 +0100 From: Mark Murray Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG > However, I am intruiged by your assertion here. It implies that perhaps > a good direction to try would be to build a scripting language that is > tied to FreeBSD, but contains elements that makes it easy to pick up > by people who are already comfortable with the popular scripting languages. A solution here may be to pick up an earlier scripting language that has been abandoned, but is nevertheless useful. I'm specifically thinking of Perl4, but I imagine that TCL, the lisps, the BASICs and other older "Toy" languages may yield up a useful base for an OS-specific scripting language. > This is an interesting concept. Fraught with problems, but nonetheless > interesting. Yes. _Very_ interesting :-) M -- o Mark Murray \_ O.\_ Warning: this .sig is umop ap!sdn To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Tue Jul 16 11:12:43 2002 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.FreeBSD.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 603D237B400 for ; Tue, 16 Jul 2002 11:12:41 -0700 (PDT) Received: from critter.freebsd.dk (critter.freebsd.dk [212.242.86.163]) by mx1.FreeBSD.org (Postfix) with ESMTP id 974D443E64 for ; Tue, 16 Jul 2002 11:12:40 -0700 (PDT) (envelope-from phk@critter.freebsd.dk) Received: from critter.freebsd.dk (localhost [127.0.0.1]) by critter.freebsd.dk (8.12.3/8.12.2) with ESMTP id g6GIAUXB086752; Tue, 16 Jul 2002 20:10:31 +0200 (CEST) (envelope-from phk@critter.freebsd.dk) To: Garance A Drosihn Cc: Mark Valentine , freebsd-arch@FreeBSD.ORG Subject: Re: scripting language in base system? In-Reply-To: Your message of "Tue, 16 Jul 2002 12:35:55 EDT." Date: Tue, 16 Jul 2002 20:10:30 +0200 Message-ID: <86751.1026843030@critter.freebsd.dk> From: Poul-Henning Kamp Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG >You're a step ahead of where I am. I meant listing my thoughts >on just whether we even *should* have a higher-level scripting >language in the base system -- never mind what it might look >like! I have definite, firm opinions on both sides of the >matter... I think the conclusion is that yes, in theory it would be wonderful to have a high-level language in the tree, except that it isn't wonderful in practice. Any piece of externally developed software is a PITA, the more active development the more pain. This disadvantages practically all relevant languages. If the language tried to bind to C API's in binary fashion, it becomes radically MD and gets a architecture hazzle. ("Lex Perl"). Finally, the fact that there isn't a standard which says "this is the official scripting language of any respectable UNIX" [1] means that including any particular language will make it magic to FreeBSD (why does "foo" live in /usr/bin instead of /usr/local/bin ???) I wish we could hold a giant Bake-off, throw all the contenders into the pot, let them solve a number of excercises, have people vote on the results, and then all unix'en accepts the winner and implements it in a uniform fashion. I'd like peace on Balkan while we're at it. Unfortunately, this is like making people decide on the one and only kind and color of car for everybody to use. I would be smarter in a lot of ways, spare-parts, education etc etc, but it just ain't gonna happen. Summary: Forget it. [1] We're talking "higher-level than /bin/sh" here. The crucial feature is C language extension without fork(2)/exec(2) overhead. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk@FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Tue Jul 16 11:19:48 2002 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.FreeBSD.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 12E5937B400 for ; Tue, 16 Jul 2002 11:19:46 -0700 (PDT) Received: from mta05-svc.ntlworld.com (mta05-svc.ntlworld.com [62.253.162.45]) by mx1.FreeBSD.org (Postfix) with ESMTP id 36C3E43E67 for ; Tue, 16 Jul 2002 11:19:45 -0700 (PDT) (envelope-from antony.t.curtis@ntlworld.com) Received: from ntlworld.com ([213.104.146.54]) by mta05-svc.ntlworld.com (InterMail vM.4.01.03.27 201-229-121-127-20010626) with ESMTP id <20020716181943.WAGR28874.mta05-svc.ntlworld.com@ntlworld.com>; Tue, 16 Jul 2002 19:19:43 +0100 Message-ID: <3D346322.1060902@ntlworld.com> Date: Tue, 16 Jul 2002 19:17:06 +0100 From: Antony T Curtis User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.0.0) Gecko/20020715 X-Accept-Language: en-gb, en-us, en MIME-Version: 1.0 To: Mark Murray Cc: Jon Mini , freebsd-arch@FreeBSD.ORG Subject: Re: An odd scripting language References: <20020716073232.GD55378@elvis.mu.org> <200207161715.g6GHFauH034332@grimreaper.grondar.org> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG Mark Murray wrote: >>However, I am intruiged by your assertion here. It implies that perhaps >>a good direction to try would be to build a scripting language that is >>tied to FreeBSD, but contains elements that makes it easy to pick up >>by people who are already comfortable with the popular scripting languages. > > A solution here may be to pick up an earlier scripting language > that has been abandoned, but is nevertheless useful. I'm specifically > thinking of Perl4, but I imagine that TCL, the lisps, the BASICs > and other older "Toy" languages may yield up a useful base for an > OS-specific scripting language. I would suggest REXX... It's a mature scripting language which is drifting off into the sunset. It is easy to pick up - being a bit like BASIC... The only thing is - How would IBM react to it? Actually, I think IBM's Mike Colinshaw would love his language being used... >>This is an interesting concept. Fraught with problems, but nonetheless >>interesting. > > > Yes. _Very_ interesting :-) > > M -- Antony T Curtis BSc Unix Analyst Programmer http://homepage.ntlworld.com/antony.t.curtis/ To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Tue Jul 16 11:26: 9 2002 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.FreeBSD.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id EDB8237B400 for ; Tue, 16 Jul 2002 11:26:06 -0700 (PDT) Received: from critter.freebsd.dk (critter.freebsd.dk [212.242.86.163]) by mx1.FreeBSD.org (Postfix) with ESMTP id 31D5643E3B for ; Tue, 16 Jul 2002 11:26:06 -0700 (PDT) (envelope-from phk@critter.freebsd.dk) Received: from critter.freebsd.dk (localhost [127.0.0.1]) by critter.freebsd.dk (8.12.3/8.12.2) with ESMTP id g6GINdXB089394; Tue, 16 Jul 2002 20:23:44 +0200 (CEST) (envelope-from phk@critter.freebsd.dk) To: Antony T Curtis Cc: Mark Murray , Jon Mini , freebsd-arch@FreeBSD.ORG Subject: Re: An odd scripting language In-Reply-To: Your message of "Tue, 16 Jul 2002 19:17:06 BST." <3D346322.1060902@ntlworld.com> Date: Tue, 16 Jul 2002 20:23:39 +0200 Message-ID: <89393.1026843819@critter.freebsd.dk> From: Poul-Henning Kamp Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG In message <3D346322.1060902@ntlworld.com>, Antony T Curtis writes: >Mark Murray wrote: >>>However, I am intruiged by your assertion here. It implies that perhaps >>>a good direction to try would be to build a scripting language that is >>>tied to FreeBSD, but contains elements that makes it easy to pick up >>>by people who are already comfortable with the popular scripting languages. >> >> A solution here may be to pick up an earlier scripting language >> that has been abandoned, but is nevertheless useful. I'm specifically >> thinking of Perl4, but I imagine that TCL, the lisps, the BASICs >> and other older "Toy" languages may yield up a useful base for an >> OS-specific scripting language. > >I would suggest REXX... It's a mature scripting language which is >drifting off into the sunset. It is easy to pick up - being a bit like >BASIC... The only thing is - How would IBM react to it? > >Actually, I think IBM's Mike Colinshaw would love his language being used... I'd love to see John Hartman's TSO-PIPES under UNIX :-) Get them to release it under BSDL and we'll rock :-) -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk@FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Tue Jul 16 11:42:11 2002 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.FreeBSD.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 6422737B401 for ; Tue, 16 Jul 2002 11:42:09 -0700 (PDT) Received: from jkh-gw.queasyweasel.com (adsl-64-173-3-158.dsl.sntc01.pacbell.net [64.173.3.158]) by mx1.FreeBSD.org (Postfix) with ESMTP id C39D043E3B for ; Tue, 16 Jul 2002 11:42:08 -0700 (PDT) (envelope-from jkh@queasyweasel.com) Received: from mango.queasyweasel.com (jkh@mango.freebsd.com [64.173.15.99]) by jkh-gw.queasyweasel.com (8.12.5/8.12.5) with ESMTP id g6GIftux013090; Tue, 16 Jul 2002 11:41:55 -0700 (PDT) (envelope-from jkh@queasyweasel.com) Date: Tue, 16 Jul 2002 11:42:35 -0700 Subject: Re: Package system flaws? Content-Type: text/plain; charset=US-ASCII; format=flowed Mime-Version: 1.0 (Apple Message framework v535) Cc: jos@catnook.com, freebsd-arch@FreeBSD.ORG To: Dag-Erling Smorgrav From: Jordan K Hubbard In-Reply-To: Message-Id: Content-Transfer-Encoding: 7bit X-Mailer: Apple Mail (2.535) Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG I think we're waiting for it to get to the stage where it does enough that it's not shot down for lack of functionality. It actually seems to be making very good progress as a side-project, so I haven't been tempted to rock the boat by encouraging its merger with the main CVS repo until its own developers feel it's ready for that. For one thing, it would make adding new developers to the libh project that much harder since they'd have to go through the more formal committer process. - Jordan On Tuesday, July 16, 2002, at 02:24 AM, Dag-Erling Smorgrav wrote: > Jordan K Hubbard writes: >> If libh ever makes it off the ground, you can bet that Tcl will enter >> the base system fairly rapidly since it will be required for >> everything from bootstrapping packages onto the system to actually >> installing the system itself. > > I'm not a big fan of having Tcl in the base system, but - what are we > waiting for? libh should have entered the tree a long time ago, > there's absolutely nothing to be gained from leaving it out in the > cold. > > DES > -- > Dag-Erling Smorgrav - des@ofug.org > -- Jordan K. Hubbard Engineering Manager, BSD technology group Apple Computer To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Tue Jul 16 12:11:50 2002 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.FreeBSD.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 98D5237B405 for ; Tue, 16 Jul 2002 12:11:46 -0700 (PDT) Received: from thuvia.demon.co.uk (thuvia.demon.co.uk [193.237.34.248]) by mx1.FreeBSD.org (Postfix) with ESMTP id 157C543E42 for ; Tue, 16 Jul 2002 12:11:45 -0700 (PDT) (envelope-from mark@thuvia.demon.co.uk) Received: from dotar.thuvia.org (dotar.thuvia.org [10.0.0.4]) by phaidor.thuvia.org (8.11.6/8.11.6) with ESMTP id g6GJBPb56740; Tue, 16 Jul 2002 20:11:26 +0100 (BST) (envelope-from mark@thuvia.demon.co.uk) Received: from dotar.thuvia.org (localhost [IPv6:::1]) by dotar.thuvia.org (8.12.3/8.12.3) with ESMTP id g6GJBPKC046399; Tue, 16 Jul 2002 20:11:25 +0100 (BST) (envelope-from mark@dotar.thuvia.org) Received: (from mark@localhost) by dotar.thuvia.org (8.12.3/8.12.3/Submit) id g6GJBOF0046398; Tue, 16 Jul 2002 20:11:24 +0100 (BST) Date: Tue, 16 Jul 2002 20:11:24 +0100 (BST) From: Mark Valentine Message-Id: <200207161911.g6GJBOF0046398@dotar.thuvia.org> In-Reply-To: <73FF8785-9895-11D6-BCA3-0003938C7B7E@queasyweasel.com> X-Disclaimer: tequila X-Mailer: Mail User's Shell (7.2.6 beta(5) 10/07/98) To: Jordan K Hubbard , Garance A Drosihn Subject: Re: scripting language in base system? Cc: danfe@regency.nsu.ru (Alexey Dokuchaev), Jos Backus , freebsd-arch@FreeBSD.ORG Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG > From: Jordan K Hubbard > Date: Tue 16 Jul, 2002 > Subject: Re: scripting language in base system? > A good package system isn't really intended for them anyway since the > 30-year veterans usually just grab the sources off their favorite > archive site and compile it themselves anyway. Unfortunately, we can't reasonably expect our customers to do the same... That's why this veteran is taking an interest in packaging systems. Cheers, Mark. -- Mark Valentine, Thuvia Labs "Tigers will do ANYTHING for a tuna fish sandwich." Mark Valentine uses "We're kind of stupid that way." *munch* *munch* and endorses FreeBSD -- To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Tue Jul 16 12:15:24 2002 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.FreeBSD.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 6264E37B400 for ; Tue, 16 Jul 2002 12:15:22 -0700 (PDT) Received: from histidine.utmb.edu (histidine.utmb.edu [129.109.59.80]) by mx1.FreeBSD.org (Postfix) with ESMTP id 0399343E4A for ; Tue, 16 Jul 2002 12:15:22 -0700 (PDT) (envelope-from bdodson@scms.utmb.edu) Received: from scms.utmb.edu (localhost [127.0.0.1]) by histidine.utmb.edu (8.12.3/8.12.3) with ESMTP id g6GJF2Of057411; Tue, 16 Jul 2002 14:15:09 -0500 (CDT) (envelope-from bdodson@scms.utmb.edu) Message-Id: <200207161915.g6GJF2Of057411@histidine.utmb.edu> Date: Tue, 16 Jul 2002 14:15:02 -0500 (CDT) From: "M. L. Dodson" Reply-To: "M. L. Dodson" Subject: Re: An odd scripting language To: mark@grondar.za Cc: baka@elvis.mu.org, freebsd-arch@FreeBSD.ORG In-Reply-To: <200207161715.g6GHFauH034332@grimreaper.grondar.org> MIME-Version: 1.0 Content-Type: TEXT/plain; charset=US-ASCII Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG On 16 Jul, Mark Murray wrote: >> However, I am intruiged by your assertion here. It implies that perhaps >> a good direction to try would be to build a scripting language that is >> tied to FreeBSD, but contains elements that makes it easy to pick up >> by people who are already comfortable with the popular scripting languages. > > A solution here may be to pick up an earlier scripting language > that has been abandoned, but is nevertheless useful. I'm specifically > thinking of Perl4, but I imagine that TCL, the lisps, the BASICs > and other older "Toy" languages may yield up a useful base for an > OS-specific scripting language. > >> This is an interesting concept. Fraught with problems, but nonetheless >> interesting. > > Yes. _Very_ interesting :-) > > M Icon, Icon, Icon! :-) http://www.cs.arizona.edu/icon/ Professionally designed (by the inventor of snobol), right kind of license, ability to do X graphics, if need be, but not required, even better than perl in string processing (IMO). -- M. L. Dodson bdodson@scms.utmb.edu 409-772-2178 FAX: 409-772-1790 To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Tue Jul 16 12:24:51 2002 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.FreeBSD.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 855BB37B400 for ; Tue, 16 Jul 2002 12:24:49 -0700 (PDT) Received: from thuvia.demon.co.uk (thuvia.demon.co.uk [193.237.34.248]) by mx1.FreeBSD.org (Postfix) with ESMTP id 0333843E31 for ; Tue, 16 Jul 2002 12:24:48 -0700 (PDT) (envelope-from mark@thuvia.demon.co.uk) Received: from dotar.thuvia.org (dotar.thuvia.org [10.0.0.4]) by phaidor.thuvia.org (8.11.6/8.11.6) with ESMTP id g6GJOcb56798; Tue, 16 Jul 2002 20:24:39 +0100 (BST) (envelope-from mark@thuvia.demon.co.uk) Received: from dotar.thuvia.org (localhost [IPv6:::1]) by dotar.thuvia.org (8.12.3/8.12.3) with ESMTP id g6GJOcKC046704; Tue, 16 Jul 2002 20:24:38 +0100 (BST) (envelope-from mark@dotar.thuvia.org) Received: (from mark@localhost) by dotar.thuvia.org (8.12.3/8.12.3/Submit) id g6GJOcOP046703; Tue, 16 Jul 2002 20:24:38 +0100 (BST) Date: Tue, 16 Jul 2002 20:24:38 +0100 (BST) From: Mark Valentine Message-Id: <200207161924.g6GJOcOP046703@dotar.thuvia.org> In-Reply-To: X-Disclaimer: tequila X-Mailer: Mail User's Shell (7.2.6 beta(5) 10/07/98) To: phk@critter.freebsd.dk (Poul-Henning Kamp), Garance A Drosihn Subject: Re: scripting language in base system? Cc: freebsd-arch@freebsd.org Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG > From: phk@critter.freebsd.dk (Poul-Henning Kamp) > Date: Tue 16 Jul, 2002 > Subject: Re: scripting language in base system? > [1] We're talking "higher-level than /bin/sh" here. The crucial > feature is C language extension without fork(2)/exec(2) overhead. Hm, Evil Thought: dynamically loadable shell builtins, anyone? Maybe not very pretty (see wksh for an example of extending ksh to embrace the X Toolkit API, albeit statically), but perhaps with the addition of a real list data type (which is about all Tcl needs), it might be useful. Probably make it a separate binary to keep /bin/sh small and static. Cheers, Mark. -- Mark Valentine, Thuvia Labs "Tigers will do ANYTHING for a tuna fish sandwich." Mark Valentine uses "We're kind of stupid that way." *munch* *munch* and endorses FreeBSD -- To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Tue Jul 16 12:37:19 2002 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.FreeBSD.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id EC95937B400 for ; Tue, 16 Jul 2002 12:37:17 -0700 (PDT) Received: from harmony.village.org (rover.bsdimp.com [204.144.255.66]) by mx1.FreeBSD.org (Postfix) with ESMTP id A4D0E43E64 for ; Tue, 16 Jul 2002 12:37:15 -0700 (PDT) (envelope-from imp@bsdimp.com) Received: from localhost (warner@rover2.village.org [10.0.0.1]) by harmony.village.org (8.12.3/8.12.3) with ESMTP id g6GJb51f015049 for ; Tue, 16 Jul 2002 13:37:08 -0600 (MDT) (envelope-from imp@bsdimp.com) Date: Tue, 16 Jul 2002 13:36:42 -0600 (MDT) Message-Id: <20020716.133642.115264178.imp@bsdimp.com> To: arch@freebsd.org Subject: GET THEE TO A NUNNERY From: "M. Warner Losh" X-Mailer: Mew version 2.1 on Emacs 21.1 / Mule 5.0 (SAKAKI) Mime-Version: 1.0 Content-Type: Text/Plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG OK. It is time to take the package discussion and the scripting discussions to config@. They are not moving forward here and are clogging up the works. Yes, we know the system has flaws. Yes, we know that everybody has their own pet scripting language. No, you are going to reach closure on it without producing code. We've been over that at least 5 times in the last 5 years. You aren't covering new ground here. This discussion has gone on long enough. It is time to move it to a more appropriate place. config@ is that place. Please stop posting about it in arch@. Warner To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Tue Jul 16 15:35:11 2002 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.FreeBSD.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 12E2537B400 for ; Tue, 16 Jul 2002 15:35:10 -0700 (PDT) Received: from smtp.noos.fr (zola.noos.net [212.198.2.76]) by mx1.FreeBSD.org (Postfix) with ESMTP id 1649A43E67 for ; Tue, 16 Jul 2002 15:35:09 -0700 (PDT) (envelope-from root@gits.dyndns.org) Received: (qmail 43637039 invoked by uid 0); 16 Jul 2002 22:31:10 -0000 Received: from unknown (HELO gits.gits.dyndns.org) ([212.198.229.153]) (envelope-sender ) by 212.198.2.76 (qmail-ldap-1.03) with SMTP for ; 16 Jul 2002 22:31:10 -0000 Received: from gits.gits.dyndns.org (73kzbsxshgxsag5r@localhost [127.0.0.1]) by gits.gits.dyndns.org (8.12.5/8.12.5) with ESMTP id g6GMV8ie030808; Wed, 17 Jul 2002 00:31:09 +0200 (CEST) (envelope-from root@gits.dyndns.org) Received: (from root@localhost) by gits.gits.dyndns.org (8.12.5/8.12.5/Submit) id g6GMV8Sq030807; Wed, 17 Jul 2002 00:31:08 +0200 (CEST) (envelope-from root) Date: Wed, 17 Jul 2002 00:31:07 +0200 From: Cyrille Lefevre To: Mark Valentine Cc: Poul-Henning Kamp , Garance A Drosihn , freebsd-arch@FreeBSD.ORG Subject: Re: scripting language in base system? Message-ID: <20020716223107.GC29859@gits.dyndns.org> Mail-Followup-To: Cyrille Lefevre , Mark Valentine , Poul-Henning Kamp , Garance A Drosihn , freebsd-arch@FreeBSD.ORG References: <200207161924.g6GJOcOP046703@dotar.thuvia.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <200207161924.g6GJOcOP046703@dotar.thuvia.org> User-Agent: Mutt/1.3.99i Organization: ACME X-Face: V|+c;4!|B?E%BE^{E6);aI.[< List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG On Tue, Jul 16, 2002 at 08:24:38PM +0100, Mark Valentine wrote: > > From: phk@critter.freebsd.dk (Poul-Henning Kamp) > > Date: Tue 16 Jul, 2002 > > Subject: Re: scripting language in base system? > > > [1] We're talking "higher-level than /bin/sh" here. The crucial > > feature is C language extension without fork(2)/exec(2) overhead. > > Hm, Evil Thought: dynamically loadable shell builtins, anyone? > > Maybe not very pretty (see wksh for an example of extending ksh > to embrace the X Toolkit API, albeit statically), but perhaps with > the addition of a real list data type (which is about all Tcl needs), > it might be useful. http://www.cs.princeton.edu/~jlk/tksh/ > Probably make it a separate binary to keep /bin/sh small and static. Q: in the mean time, how about to switch to pdksh as OpenBSD does ? I'm working w/ it for months w/o any problems right now. evrything it ready, the Makefile, the manual pages and the import to src/contrib. I've also merged diffs from the real one pdksh. are you interrested ? Cyrille. -- Cyrille Lefevre mailto:cyrille.lefevre@laposte.net To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Tue Jul 16 15:55:38 2002 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.FreeBSD.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 1D7C337B40E for ; Tue, 16 Jul 2002 15:55:30 -0700 (PDT) Received: from smtp.noos.fr (verlaine.noos.net [212.198.2.73]) by mx1.FreeBSD.org (Postfix) with ESMTP id 0411D43E31 for ; Tue, 16 Jul 2002 15:55:29 -0700 (PDT) (envelope-from root@gits.dyndns.org) Received: (qmail 33302974 invoked by uid 0); 16 Jul 2002 22:55:27 -0000 Received: from unknown (HELO gits.gits.dyndns.org) ([212.198.229.153]) (envelope-sender ) by 212.198.2.73 (qmail-ldap-1.03) with SMTP for ; 16 Jul 2002 22:55:27 -0000 Received: from gits.gits.dyndns.org (gnwcly3l5jmhpg1h@localhost [127.0.0.1]) by gits.gits.dyndns.org (8.12.5/8.12.5) with ESMTP id g6GMtQie030959; Wed, 17 Jul 2002 00:55:26 +0200 (CEST) (envelope-from root@gits.dyndns.org) Received: (from root@localhost) by gits.gits.dyndns.org (8.12.5/8.12.5/Submit) id g6GMtN1K030958; Wed, 17 Jul 2002 00:55:23 +0200 (CEST) (envelope-from root) Date: Wed, 17 Jul 2002 00:55:23 +0200 From: Cyrille Lefevre To: "M. L. Dodson" Cc: mark@grondar.za, baka@elvis.mu.org, freebsd-arch@FreeBSD.ORG Subject: Re: An odd scripting language Message-ID: <20020716225523.GD29859@gits.dyndns.org> Mail-Followup-To: Cyrille Lefevre , "M. L. Dodson" , mark@grondar.za, baka@elvis.mu.org, freebsd-arch@FreeBSD.ORG References: <200207161715.g6GHFauH034332@grimreaper.grondar.org> <200207161915.g6GJF2Of057411@histidine.utmb.edu> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <200207161915.g6GJF2Of057411@histidine.utmb.edu> User-Agent: Mutt/1.3.99i Organization: ACME X-Face: V|+c;4!|B?E%BE^{E6);aI.[< List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG here is a non-exhaustive list of computer languages, so please, don't list them all :) http://dmoz.org/Computers/Programming/Languages/ Cyrille. -- Cyrille Lefevre mailto:cyrille.lefevre@laposte.net To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Tue Jul 16 16:32:22 2002 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.FreeBSD.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id DDEE337B400 for ; Tue, 16 Jul 2002 16:32:19 -0700 (PDT) Received: from thuvia.demon.co.uk (thuvia.demon.co.uk [193.237.34.248]) by mx1.FreeBSD.org (Postfix) with ESMTP id A51E843E42 for ; Tue, 16 Jul 2002 16:32:15 -0700 (PDT) (envelope-from mark@thuvia.demon.co.uk) Received: from dotar.thuvia.org (dotar.thuvia.org [10.0.0.4]) by phaidor.thuvia.org (8.11.6/8.11.6) with ESMTP id g6GNVtb57797; Wed, 17 Jul 2002 00:31:55 +0100 (BST) (envelope-from mark@thuvia.demon.co.uk) Received: from dotar.thuvia.org (localhost [IPv6:::1]) by dotar.thuvia.org (8.12.3/8.12.3) with ESMTP id g6GNVsKC053110; Wed, 17 Jul 2002 00:31:54 +0100 (BST) (envelope-from mark@dotar.thuvia.org) Received: (from mark@localhost) by dotar.thuvia.org (8.12.3/8.12.3/Submit) id g6GNVsCa053104; Wed, 17 Jul 2002 00:31:54 +0100 (BST) Date: Wed, 17 Jul 2002 00:31:54 +0100 (BST) From: Mark Valentine Message-Id: <200207162331.g6GNVsCa053104@dotar.thuvia.org> In-Reply-To: <20020716223107.GC29859@gits.dyndns.org> X-Disclaimer: tequila X-Mailer: Mail User's Shell (7.2.6 beta(5) 10/07/98) To: Cyrille Lefevre Subject: Re: scripting language in base system? Cc: Poul-Henning Kamp , Garance A Drosihn , freebsd-arch@FreeBSD.ORG Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG > From: Cyrille Lefevre > Date: Wed 17 Jul, 2002 > Subject: Re: scripting language in base system? > On Tue, Jul 16, 2002 at 08:24:38PM +0100, Mark Valentine wrote: > > Hm, Evil Thought: dynamically loadable shell builtins, anyone? > > > > Maybe not very pretty (see wksh for an example of extending ksh > > to embrace the X Toolkit API, albeit statically), but perhaps with > > the addition of a real list data type (which is about all Tcl needs), > > it might be useful. > > http://www.cs.princeton.edu/~jlk/tksh/ Neat. It looks like it's annoyingly close to being free, too. > Q: in the mean time, how about to switch to pdksh as OpenBSD does ? Is this going to become necessary to get a more standards-conformant shell? Is pdksh the best implementation available to us? Cheers, Mark. -- Mark Valentine, Thuvia Labs "Tigers will do ANYTHING for a tuna fish sandwich." Mark Valentine uses "We're kind of stupid that way." *munch* *munch* and endorses FreeBSD -- To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Tue Jul 16 22:17: 2 2002 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.FreeBSD.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 0A8F937B400 for ; Tue, 16 Jul 2002 22:16:59 -0700 (PDT) Received: from mailman.zeta.org.au (mailman.zeta.org.au [203.26.10.16]) by mx1.FreeBSD.org (Postfix) with ESMTP id 1359E43E31 for ; Tue, 16 Jul 2002 22:16:58 -0700 (PDT) (envelope-from bde@zeta.org.au) Received: from bde.zeta.org.au (bde.zeta.org.au [203.2.228.102]) by mailman.zeta.org.au (8.9.3/8.8.7) with ESMTP id PAA27438; Wed, 17 Jul 2002 15:16:47 +1000 Date: Wed, 17 Jul 2002 15:20:31 +1000 (EST) From: Bruce Evans X-X-Sender: bde@gamplex.bde.org To: Don Lewis Cc: arch@FreeBSD.ORG Subject: Re: wiring the sysctl output buffer In-Reply-To: <200207160954.g6G9scwr026538@gw.catspoiler.org> Message-ID: <20020717135901.F3656-100000@gamplex.bde.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG On Tue, 16 Jul 2002, Don Lewis wrote: > On 14 Jul, Bruce Evans wrote: > > ... > > `oldlen' is known ahead of time, so a large enough buffer could be > > allocated in most cases. Perhaps vslock() iff oldlen is very large. > > The data could still change while it is being copied to a malloc()'ed > > buffer (or earlier) if the kernel is preemptible. > > I basically like this approach. In most cases the data will be a small > and of fixed size, so the handler could even store the temporary copy on > the stack. If the data is a type that can be copied atomically, like an > int, we don't have to worry about the data changing in mid-copy. If it It's not clear that ints can be copied atomically unless a suitable lock is held. Longs can't be on some arches (i386's with 64-bit longs for example :-). > is larger and we care about getting an atomic snapshot, the handler can > lock the source for the duration of the copy to the temporary buffer. > > For large amounts of data, we probably want to avoid copying to a > temporary buffer. PHK raised the issue of the user specifying a buffer > size that is way too large. Allocating a huge buffer in wired down > kernel memory sounds even worse than wiring down a huge buffer in user > space. I think this is a small problem provided you only make the buffer large enough to hold what the kernel is supplying in the current SYSCTL_OUT() and not what the application is willing to accept for the whole sysctl. There may be no suppliers of huge kernel data except KERN_VNODE, and KERN_VNODE was turned off because it was too hard to supply that much data correctly even years ago when there were fewer vnodes and less concurrency than now. Any suppliers of huge kernel data may have the same problems. > If we don't want to allocate a temporary buffer of length oldlen because > it is much larger than what looks to be reasonable, but the amount of OK, but allocating if it is <= PAGE_SIZE or small enough to put on the stack should handle most cases and hopefully not signficantly complicate the problem of keeping track of what you allocated. > data is not easy to calculate ahead of time, we have another potential > problem. We might have to lock a data structure, walk through this > data structure and calculate the total data size, and unlock the data > structure. After we malloc() the buffer, relock the data structure, and > start traversing the data structure again to do the copy, we may > discover that the amount of data to be returned has increased, so we > would have to drop the lock, free the buffer, and start over again. Also, the size might turn out to be too huge to allocate. You would then need to backtrack and recalulate the size using a less ambitious algorithm. > > I think the callers of SYSCTL_OUT() don't need a new API, but they > > should supply the data in a form suitable for copying out directly. > > This probably involves them making a copy while holding suitable > > domain-specific locks. They can't just point to an active kernel > > struct since it might change. > > > > I would just make a copy at a high level for now. > > I'd vote for a hybrid approach. I would remove the vslock() call from > sysctl_old_user() and provide the new API for invoking it from the > handler if appropriate. Instead, I would call WITNESS_SLEEP() if > vslock() had not been called to wire the buffer to guard against > blocking while locks are held. I would keep the call to vsunlock() in > the sysctl cleanup code. It's never really appropriate, but I guess you need it for a few sysctls because they are too large to fix all at once. I hope the new API won't be used much. Sysctls with fixed locking can keep using the current code since they will have copied the data to a safe place and not hold any locks when they call SYSCTL_OUT() or care if SYSCTL_OUT() makes another copy or blocks. > I'd also like to provide a second argument to the interface to vslock() > to allow the handler to specify a maximum, kernel enforced, buffer > length to potentially limit the amount of wired memory to something less > than oldlen, but unless we add a member to the sysctl_req structure to > hold it, we would need to overwrite oldlen so that we can pass the > proper value to vsunlock() in the cleanup code. What kind impact would > changing sysctl_req have on binary compatibility? I doesn't look like a > problem to me, since this structure seems to be mostly treated as an > opaque type. Changing its size would probably not be good for some modules. I think no time should be spent changing interfaces for this. The following hack might work well: use the current code in sysctl_old_user() if req->oldlen is not huge. Otherwise, vslock() only the region being copied out to and vsunlock() it after the copy so that we don't have to keep track of what is vslock()ed. Bruce To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Tue Jul 16 23:52:37 2002 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.FreeBSD.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 64A0637B400 for ; Tue, 16 Jul 2002 23:52:18 -0700 (PDT) Received: from iguana.icir.org (iguana.icir.org [192.150.187.36]) by mx1.FreeBSD.org (Postfix) with ESMTP id E5ACA43E42 for ; Tue, 16 Jul 2002 23:52:16 -0700 (PDT) (envelope-from rizzo@iguana.icir.org) Received: (from rizzo@localhost) by iguana.icir.org (8.11.6/8.11.3) id g6H6qGu06926; Tue, 16 Jul 2002 23:52:16 -0700 (PDT) (envelope-from rizzo) Date: Tue, 16 Jul 2002 23:52:16 -0700 From: Luigi Rizzo To: arch@freebsd.org Subject: proposed changes to kern_switch.c and kern_synch.c Message-ID: <20020716235216.B6785@iguana.icir.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2.5.1i Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG Hi, we have implemented a weight-based process scheduler for FreeBSD-stable, and are trying to port it to -current (in the description below replace "process" with "thread/kse" as appropriate for the -current case). In order to make this work, it is convenient to have all scheduler-specific functions and data structures in a single file (kern_switch*.c), and generic support in another one (kern_synch.c). I believe this was also the original BSD design in partitioning the code between the two files. However, in our current code, there are some functions which are scheduler-specific (see below) which are misplaced. So I would like to make the following, purely cosmetic, change identified as step #1 below, both for consistency with what i believe to be the original design, and to simplify further work in this area. These would go to both -current and -stable; I repeat, they are only cosmetic changes. Comments ? I already spoke to julian who has no objections. For those interested, a patch for -current is attached, and the one for -stable is very similar (for the records, in -stable we have a fully functional weight-based process scheduler, where you still control the weights using "nice", and you can switch between the old and the new scheduler at any time using a sysctl variable). --- Re. the multi-scheduler architecture --- The general idea is to make the process/thread/kse scheduler a replaceable piece of the kernel, requiring no modifications to the "struct proc", and with the ability of switching from one scheduler to another one at runtime (this both for testing purposes and for whatever need may arise). The way to achieve this would be the following: 1. identify all scheduler-specific functions, put all of them in one file (e.g. kern_switch.c for the default scheduler), and define function pointers for all of them; 2. use one field in "struct proc" as a link to whatever scheduler-specific data structures are necessary. In -stable, we have the p_pad3 field that can be used for this purpose, much like the if_index field in struct ifnet. 3. implement a function which, under control of a sysctl call, activate a new scheduler (by initialising all the function pointers to appropriate values) and moves all processes from the old to the new one. Step #1 is basically a cosmetic change, requiring mostly a move of some functions from kern_synch.c to kern_switch.c. These are roundrobin(); curpriority_cmp(); resetpriority(); parts of schedcpu() and schedclock(); cheers luigi ---------------------------------- Index: kern_switch.c =================================================================== RCS file: /home/ncvs/src/sys/kern/kern_switch.c,v retrieving revision 1.33 diff -u -r1.33 kern_switch.c --- kern_switch.c 14 Jul 2002 03:43:33 -0000 1.33 +++ kern_switch.c 16 Jul 2002 22:15:06 -0000 @@ -97,6 +97,7 @@ #include #include #include +#include #include CTASSERT((RQB_BPW * RQB_LEN) == RQ_NQS); @@ -107,6 +108,9 @@ static struct runq runq; SYSINIT(runq, SI_SUB_RUN_QUEUE, SI_ORDER_FIRST, runq_init, &runq) +static struct callout roundrobin_callout; + +static void roundrobin(void *arg); static void runq_readjust(struct runq *rq, struct kse *ke); /************************************************************************ * Functions that manipulate runnability from a thread perspective. * @@ -442,9 +446,13 @@ { int i; + callout_init(&roundrobin_callout, 0); + bzero(rq, sizeof *rq); for (i = 0; i < RQ_NQS; i++) TAILQ_INIT(&rq->rq_queues[i]); + + roundrobin(NULL); } /* @@ -719,3 +727,207 @@ } #endif +/* + * -- formerly in kern_synch.c + */ + +int curpriority_cmp(struct thread *td); + +int +curpriority_cmp(struct thread *td) +{ + return (td->td_priority - curthread->td_priority); +} + +/* + * Force switch among equal priority processes every 100ms. + * We don't actually need to force a context switch of the current process. + * The act of firing the event triggers a context switch to softclock() and + * then switching back out again which is equivalent to a preemption, thus + * no further work is needed on the local CPU. + */ +/* ARGSUSED */ +static void +roundrobin(arg) + void *arg; +{ + +#ifdef SMP + mtx_lock_spin(&sched_lock); + forward_roundrobin(); + mtx_unlock_spin(&sched_lock); +#endif + + callout_reset(&roundrobin_callout, sched_quantum, roundrobin, NULL); +} + +/* calculations for digital decay to forget 90% of usage in 5*loadav sec */ +#define loadfactor(loadav) (2 * (loadav)) +#define decay_cpu(loadfac, cpu) (((loadfac) * (cpu)) / ((loadfac) + FSCALE)) +extern fixpt_t ccpu; +#define CCPU_SHIFT 11 /* XXX dup from kern_synch.c! */ + +/* + * Recompute process priorities, every hz ticks. + * MP-safe, called without the Giant mutex. + */ +void schedcpu1(struct ksegrp *kg); +/* ARGSUSED */ +void +schedcpu1(struct ksegrp *kg) +{ + register fixpt_t loadfac = loadfactor(averunnable.ldavg[0]); + struct thread *td; + struct kse *ke; + int realstathz; + int awake; + + realstathz = stathz ? stathz : hz; + + awake = 0; + FOREACH_KSE_IN_GROUP(kg, ke) { + /* + * Increment time in/out of memory and sleep + * time (if sleeping). We ignore overflow; + * with 16-bit int's (remember them?) + * overflow takes 45 days. + */ + /* XXXKSE **WRONG***/ + /* + * the kse slptimes are not touched in wakeup + * because the thread may not HAVE a KSE + */ + if ((ke->ke_state == KES_ONRUNQ) || + ((ke->ke_state == KES_THREAD) && + (ke->ke_thread->td_state == TDS_RUNNING))) { + ke->ke_slptime++; + } else { + ke->ke_slptime = 0; + awake = 1; + } + + /* + * pctcpu is only for ps? + * Do it per kse.. and add them up at the end? + * XXXKSE + */ + ke->ke_pctcpu = (ke->ke_pctcpu * ccpu) >> FSHIFT; + /* + * If the kse has been idle the entire second, + * stop recalculating its priority until + * it wakes up. + */ + if (ke->ke_slptime > 1) { + continue; + } + +#if (FSHIFT >= CCPU_SHIFT) + ke->ke_pctcpu += (realstathz == 100) ? + ((fixpt_t) ke->ke_cpticks) << + (FSHIFT - CCPU_SHIFT) : + 100 * (((fixpt_t) ke->ke_cpticks) << + (FSHIFT - CCPU_SHIFT)) / realstathz; +#else + ke->ke_pctcpu += ((FSCALE - ccpu) * + (ke->ke_cpticks * FSCALE / realstathz)) >> + FSHIFT; +#endif + ke->ke_cpticks = 0; + } /* end of kse loop */ + if (awake == 0) { + kg->kg_slptime++; + } else { + kg->kg_slptime = 0; + } + kg->kg_estcpu = decay_cpu(loadfac, kg->kg_estcpu); + resetpriority(kg); + FOREACH_THREAD_IN_GROUP(kg, td) { + int changedqueue; + if (td->td_priority >= PUSER) { + /* + * Only change the priority + * of threads that are still at their + * user priority. + * XXXKSE This is problematic + * as we may need to re-order + * the threads on the KSEG list. + */ + changedqueue = + ((td->td_priority / RQ_PPQ) != + (kg->kg_user_pri / RQ_PPQ)); + + td->td_priority = kg->kg_user_pri; + if (changedqueue && + td->td_state == TDS_RUNQ) { + /* this could be optimised */ + remrunqueue(td); + td->td_priority = + kg->kg_user_pri; + setrunqueue(td); + } else { + td->td_priority = kg->kg_user_pri; + } + } + } +} + +/* + * Compute the priority of a process when running in user mode. + * Arrange to reschedule if the resulting priority is better + * than that of the current process. + */ +void +resetpriority(kg) + register struct ksegrp *kg; +{ + register unsigned int newpriority; + struct thread *td; + + mtx_lock_spin(&sched_lock); + if (kg->kg_pri_class == PRI_TIMESHARE) { + newpriority = PUSER + kg->kg_estcpu / INVERSE_ESTCPU_WEIGHT + + NICE_WEIGHT * (kg->kg_nice - PRIO_MIN); + newpriority = min(max(newpriority, PRI_MIN_TIMESHARE), + PRI_MAX_TIMESHARE); + kg->kg_user_pri = newpriority; + } + FOREACH_THREAD_IN_GROUP(kg, td) { + maybe_resched(td); /* XXXKSE silly */ + } + mtx_unlock_spin(&sched_lock); +} + +#if 0 +/* + * We adjust the priority of the current process. The priority of + * a process gets worse as it accumulates CPU time. The cpu usage + * estimator (p_estcpu) is increased here. resetpriority() will + * compute a different priority each time p_estcpu increases by + * INVERSE_ESTCPU_WEIGHT + * (until MAXPRI is reached). The cpu usage estimator ramps up + * quite quickly when the process is running (linearly), and decays + * away exponentially, at a rate which is proportionally slower when + * the system is busy. The basic principle is that the system will + * 90% forget that the process used a lot of CPU time in 5 * loadav + * seconds. This causes the system to favor processes which haven't + * run much recently, and to round-robin among other processes. + */ +void +schedclock(td) + struct thread *td; +{ + struct kse *ke; + struct ksegrp *kg; + + KASSERT((td != NULL), ("schedlock: null thread pointer")); + ke = td->td_kse; + kg = td->td_ksegrp; + ke->ke_cpticks++; + kg->kg_estcpu = ESTCPULIM(kg->kg_estcpu + 1); + if ((kg->kg_estcpu % INVERSE_ESTCPU_WEIGHT) == 0) { + resetpriority(kg); + if (td->td_priority >= PUSER) + td->td_priority = kg->kg_user_pri; + } +} +#endif Index: kern_synch.c =================================================================== RCS file: /home/ncvs/src/sys/kern/kern_synch.c,v retrieving revision 1.185 diff -u -r1.185 kern_synch.c --- kern_synch.c 14 Jul 2002 03:43:33 -0000 1.185 +++ kern_synch.c 16 Jul 2002 22:16:46 -0000 @@ -67,6 +67,8 @@ #include +void schedcpu1(struct ksegrp *kg); /* XXX in kern_switch.c */ + static void sched_setup(void *dummy); SYSINIT(sched_setup, SI_SUB_KICK_SCHEDULER, SI_ORDER_FIRST, sched_setup, NULL) @@ -76,7 +78,6 @@ static struct callout loadav_callout; static struct callout schedcpu_callout; -static struct callout roundrobin_callout; struct loadavg averunnable = { {0, 0, 0}, FSCALE }; /* load average, of runnable procs */ @@ -92,7 +93,6 @@ static void endtsleep(void *); static void loadav(void *arg); -static void roundrobin(void *arg); static void schedcpu(void *arg); static int @@ -135,28 +135,6 @@ } /* - * Force switch among equal priority processes every 100ms. - * We don't actually need to force a context switch of the current process. - * The act of firing the event triggers a context switch to softclock() and - * then switching back out again which is equivalent to a preemption, thus - * no further work is needed on the local CPU. - */ -/* ARGSUSED */ -static void -roundrobin(arg) - void *arg; -{ - -#ifdef SMP - mtx_lock_spin(&sched_lock); - forward_roundrobin(); - mtx_unlock_spin(&sched_lock); -#endif - - callout_reset(&roundrobin_callout, sched_quantum, roundrobin, NULL); -} - -/* * Constants for digital decay and forget: * 90% of (p_estcpu) usage in 5 * loadav time * 95% of (p_pctcpu) usage in 60 seconds (load insensitive) @@ -225,7 +203,7 @@ #define decay_cpu(loadfac, cpu) (((loadfac) * (cpu)) / ((loadfac) + FSCALE)) /* decay 95% of `p_pctcpu' in 60 seconds; see CCPU_SHIFT before changing */ -static fixpt_t ccpu = 0.95122942450071400909 * FSCALE; /* exp(-1/20) */ +fixpt_t ccpu = 0.95122942450071400909 * FSCALE; /* exp(-1/20) */ SYSCTL_INT(_kern, OID_AUTO, ccpu, CTLFLAG_RD, &ccpu, 0, ""); /* kernel uses `FSCALE', userland (SHOULD) use kern.fscale */ @@ -255,105 +233,15 @@ schedcpu(arg) void *arg; { - register fixpt_t loadfac = loadfactor(averunnable.ldavg[0]); - struct thread *td; struct proc *p; - struct kse *ke; struct ksegrp *kg; - int realstathz; - int awake; - realstathz = stathz ? stathz : hz; sx_slock(&allproc_lock); FOREACH_PROC_IN_SYSTEM(p) { mtx_lock_spin(&sched_lock); p->p_swtime++; FOREACH_KSEGRP_IN_PROC(p, kg) { - awake = 0; - FOREACH_KSE_IN_GROUP(kg, ke) { - /* - * Increment time in/out of memory and sleep - * time (if sleeping). We ignore overflow; - * with 16-bit int's (remember them?) - * overflow takes 45 days. - */ - /* XXXKSE **WRONG***/ - /* - * the kse slptimes are not touched in wakeup - * because the thread may not HAVE a KSE - */ - if ((ke->ke_state == KES_ONRUNQ) || - ((ke->ke_state == KES_THREAD) && - (ke->ke_thread->td_state == TDS_RUNNING))) { - ke->ke_slptime++; - } else { - ke->ke_slptime = 0; - awake = 1; - } - - /* - * pctcpu is only for ps? - * Do it per kse.. and add them up at the end? - * XXXKSE - */ - ke->ke_pctcpu = (ke->ke_pctcpu * ccpu) >> FSHIFT; - /* - * If the kse has been idle the entire second, - * stop recalculating its priority until - * it wakes up. - */ - if (ke->ke_slptime > 1) { - continue; - } - -#if (FSHIFT >= CCPU_SHIFT) - ke->ke_pctcpu += (realstathz == 100) ? - ((fixpt_t) ke->ke_cpticks) << - (FSHIFT - CCPU_SHIFT) : - 100 * (((fixpt_t) ke->ke_cpticks) << - (FSHIFT - CCPU_SHIFT)) / realstathz; -#else - ke->ke_pctcpu += ((FSCALE - ccpu) * - (ke->ke_cpticks * FSCALE / realstathz)) >> - FSHIFT; -#endif - ke->ke_cpticks = 0; - } /* end of kse loop */ - if (awake == 0) { - kg->kg_slptime++; - } else { - kg->kg_slptime = 0; - } - kg->kg_estcpu = decay_cpu(loadfac, kg->kg_estcpu); - resetpriority(kg); - FOREACH_THREAD_IN_GROUP(kg, td) { - int changedqueue; - if (td->td_priority >= PUSER) { - /* - * Only change the priority - * of threads that are still at their - * user priority. - * XXXKSE This is problematic - * as we may need to re-order - * the threads on the KSEG list. - */ - changedqueue = - ((td->td_priority / RQ_PPQ) != - (kg->kg_user_pri / RQ_PPQ)); - - td->td_priority = kg->kg_user_pri; - if (changedqueue && - td->td_state == TDS_RUNQ) { - /* this could be optimised */ - remrunqueue(td); - td->td_priority = - kg->kg_user_pri; - setrunqueue(td); - } else { - td->td_priority = kg->kg_user_pri; - } - } - } + schedcpu1(kg); } /* end of ksegrp loop */ mtx_unlock_spin(&sched_lock); } /* end of process loop */ @@ -944,32 +832,6 @@ } /* - * Compute the priority of a process when running in user mode. - * Arrange to reschedule if the resulting priority is better - * than that of the current process. - */ -void -resetpriority(kg) - register struct ksegrp *kg; -{ - register unsigned int newpriority; - struct thread *td; - - mtx_lock_spin(&sched_lock); - if (kg->kg_pri_class == PRI_TIMESHARE) { - newpriority = PUSER + kg->kg_estcpu / INVERSE_ESTCPU_WEIGHT + - NICE_WEIGHT * (kg->kg_nice - PRIO_MIN); - newpriority = min(max(newpriority, PRI_MIN_TIMESHARE), - PRI_MAX_TIMESHARE); - kg->kg_user_pri = newpriority; - } - FOREACH_THREAD_IN_GROUP(kg, td) { - maybe_resched(td); /* XXXKSE silly */ - } - mtx_unlock_spin(&sched_lock); -} - -/* * Compute a tenex style load average of a quantity on * 1, 5 and 15 minute intervals. * XXXKSE Needs complete rewrite when correct info is available. @@ -1022,11 +884,9 @@ { callout_init(&schedcpu_callout, 1); - callout_init(&roundrobin_callout, 0); callout_init(&loadav_callout, 0); /* Kick off timeout driven events by calling first time. */ - roundrobin(NULL); schedcpu(NULL); loadav(NULL); } ----- End forwarded message ----- To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Wed Jul 17 0:25:12 2002 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.FreeBSD.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id A314C37B401 for ; Wed, 17 Jul 2002 00:25:10 -0700 (PDT) Received: from flamingo.mail.pas.earthlink.net (flamingo.mail.pas.earthlink.net [207.217.120.232]) by mx1.FreeBSD.org (Postfix) with ESMTP id 4199C43E58 for ; Wed, 17 Jul 2002 00:25:10 -0700 (PDT) (envelope-from tlambert2@mindspring.com) Received: from pool0588.cvx22-bradley.dialup.earthlink.net ([209.179.200.78] helo=mindspring.com) by flamingo.mail.pas.earthlink.net with esmtp (Exim 3.33 #1) id 17UjBI-0001Ww-00; Wed, 17 Jul 2002 00:25:04 -0700 Message-ID: <3D351B99.311431F@mindspring.com> Date: Wed, 17 Jul 2002 00:24:09 -0700 From: Terry Lambert X-Mailer: Mozilla 4.79 [en] (Win98; U) X-Accept-Language: en MIME-Version: 1.0 To: Luigi Rizzo Cc: arch@freebsd.org Subject: Re: proposed changes to kern_switch.c and kern_synch.c References: <20020716235216.B6785@iguana.icir.org> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG Luigi Rizzo wrote: > Hi, > we have implemented a weight-based process scheduler > for FreeBSD-stable, and are trying to port it to -current (in the > description below replace "process" with "thread/kse" as appropriate > for the -current case). Are you going to make this new scheduler generally available? What is the weighting? Is it weighted fair share? -- Terry To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Wed Jul 17 0:44:50 2002 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.FreeBSD.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 7C0EF37B400 for ; Wed, 17 Jul 2002 00:44:45 -0700 (PDT) Received: from canning.wemm.org (canning.wemm.org [192.203.228.65]) by mx1.FreeBSD.org (Postfix) with ESMTP id 9873443E58 for ; Wed, 17 Jul 2002 00:43:38 -0700 (PDT) (envelope-from peter@wemm.org) Received: from fw.wemm.org (canning.wemm.org [192.203.228.65]) by canning.wemm.org (Postfix) with ESMTP id ADA5F2A7D6 for ; Wed, 17 Jul 2002 00:43:35 -0700 (PDT) (envelope-from peter@wemm.org) Received: from overcee.wemm.org (overcee.wemm.org [10.0.0.3]) by fw.wemm.org (Postfix) with ESMTP id 6D8864C26C for ; Wed, 17 Jul 2002 00:43:35 -0700 (PDT) (envelope-from peter@wemm.org) Received: from wemm.org (localhost [127.0.0.1]) by overcee.wemm.org (Postfix) with ESMTP id 1DA923926; Wed, 17 Jul 2002 00:43:35 -0700 (PDT) (envelope-from peter@wemm.org) X-Mailer: exmh version 2.5 07/13/2001 with nmh-1.0.4 To: Luigi Rizzo Cc: arch@freebsd.org Subject: Re: proposed changes to kern_switch.c and kern_synch.c In-Reply-To: <20020716235216.B6785@iguana.icir.org> Date: Wed, 17 Jul 2002 00:43:35 -0700 From: Peter Wemm Message-Id: <20020717074335.1DA923926@overcee.wemm.org> Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG Luigi Rizzo wrote: > In order to make this work, it is convenient to have all > scheduler-specific functions and data structures in a > single file (kern_switch*.c), and generic support in > another one (kern_synch.c). I believe this was also the original > BSD design in partitioning the code between the two files. You would be mistaken there. kern_switch.c is new and has only existed very recently. kern_switch.c came about as a C implementation of code that used to be embedded inside i386/swtch.s. It has taken on a life of its own now though. :-/ Cheers, -Peter -- Peter Wemm - peter@wemm.org; peter@FreeBSD.org; peter@yahoo-inc.com "All of this is for nothing if we don't go to the stars" - JMS/B5 To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Wed Jul 17 0:49:22 2002 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.FreeBSD.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id E413F37B496 for ; Wed, 17 Jul 2002 00:49:18 -0700 (PDT) Received: from iguana.icir.org (iguana.icir.org [192.150.187.36]) by mx1.FreeBSD.org (Postfix) with ESMTP id B82D643E6D for ; Wed, 17 Jul 2002 00:49:06 -0700 (PDT) (envelope-from rizzo@iguana.icir.org) Received: (from rizzo@localhost) by iguana.icir.org (8.11.6/8.11.3) id g6H7luB07387; Wed, 17 Jul 2002 00:47:56 -0700 (PDT) (envelope-from rizzo) Date: Wed, 17 Jul 2002 00:47:51 -0700 From: Luigi Rizzo To: Peter Wemm Cc: arch@freebsd.org Subject: Re: proposed changes to kern_switch.c and kern_synch.c Message-ID: <20020717004750.A7375@iguana.icir.org> References: <20020716235216.B6785@iguana.icir.org> <20020717074335.1DA923926@overcee.wemm.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2.5.1i In-Reply-To: <20020717074335.1DA923926@overcee.wemm.org>; from peter@wemm.org on Wed, Jul 17, 2002 at 12:43:35AM -0700 Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG On Wed, Jul 17, 2002 at 12:43:35AM -0700, Peter Wemm wrote: > Luigi Rizzo wrote: > > > In order to make this work, it is convenient to have all > > scheduler-specific functions and data structures in a > > single file (kern_switch*.c), and generic support in > > another one (kern_synch.c). I believe this was also the original > > BSD design in partitioning the code between the two files. > > You would be mistaken there. kern_switch.c is new and has only existed > very recently. kern_switch.c came about as a C implementation of code that > used to be embedded inside i386/swtch.s. It has taken on a life of its own > now though. :-/ good to know! Anyways, does the partitioning of functionalities sound reasonable ? cheers luigi To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Wed Jul 17 0:54:16 2002 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.FreeBSD.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 58E9F37B400 for ; Wed, 17 Jul 2002 00:54:14 -0700 (PDT) Received: from iguana.icir.org (iguana.icir.org [192.150.187.36]) by mx1.FreeBSD.org (Postfix) with ESMTP id 1599343E3B for ; Wed, 17 Jul 2002 00:54:14 -0700 (PDT) (envelope-from rizzo@iguana.icir.org) Received: (from rizzo@localhost) by iguana.icir.org (8.11.6/8.11.3) id g6H7gik07335; Wed, 17 Jul 2002 00:42:44 -0700 (PDT) (envelope-from rizzo) Date: Wed, 17 Jul 2002 00:42:44 -0700 From: Luigi Rizzo To: Terry Lambert Cc: arch@FreeBSD.ORG Subject: Re: proposed changes to kern_switch.c and kern_synch.c Message-ID: <20020717004244.B7218@iguana.icir.org> References: <20020716235216.B6785@iguana.icir.org> <3D351B99.311431F@mindspring.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2.5.1i In-Reply-To: <3D351B99.311431F@mindspring.com>; from tlambert2@mindspring.com on Wed, Jul 17, 2002 at 12:24:09AM -0700 Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG On Wed, Jul 17, 2002 at 12:24:09AM -0700, Terry Lambert wrote: > Luigi Rizzo wrote: > > Hi, > > we have implemented a weight-based process scheduler > > for FreeBSD-stable, and are trying to port it to -current (in the > > description below replace "process" with "thread/kse" as appropriate > > for the -current case). > > Are you going to make this new scheduler generally available? yes of course! At least the version for -stable is basically working, for -current i still have to understand the relation between KSEG's and threads etc. > What is the weighting? it is basically the WF2Q+ algorithm used in dummynet (i am the king of recycling :). The OS community calls this "Proportional Share" or something like that. Same as for dummynet queues, active processes get each a share of the CPU which is proportional to their weight. The absolute share of course depends on the number of active processes and the total sum of weights. The complexity of the algorithm is O(log N) where N is the number of active processes. The code has been developed by a PhD student of mine, Paolo Valente, who is also the guy behind the WF2Q+ implementation in dummynet. cheers luigi To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Wed Jul 17 0:56: 1 2002 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.FreeBSD.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 5F18937B405 for ; Wed, 17 Jul 2002 00:55:56 -0700 (PDT) Received: from gw.catspoiler.org (217-ip-163.nccn.net [209.79.217.163]) by mx1.FreeBSD.org (Postfix) with ESMTP id 16EED43E64 for ; Wed, 17 Jul 2002 00:55:54 -0700 (PDT) (envelope-from dl-freebsd@catspoiler.org) Received: from mousie.catspoiler.org (mousie.catspoiler.org [192.168.101.2]) by gw.catspoiler.org (8.12.5/8.12.5) with ESMTP id g6H7t1wr029257; Wed, 17 Jul 2002 00:55:06 -0700 (PDT) (envelope-from dl-freebsd@catspoiler.org) Message-Id: <200207170755.g6H7t1wr029257@gw.catspoiler.org> Date: Wed, 17 Jul 2002 00:55:01 -0700 (PDT) From: Don Lewis Subject: Re: wiring the sysctl output buffer To: bde@zeta.org.au Cc: arch@FreeBSD.ORG In-Reply-To: <20020717135901.F3656-100000@gamplex.bde.org> MIME-Version: 1.0 Content-Type: TEXT/plain; charset=US-ASCII Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG On 17 Jul, Bruce Evans wrote: > On Tue, 16 Jul 2002, Don Lewis wrote: > >> On 14 Jul, Bruce Evans wrote: >> > ... >> > `oldlen' is known ahead of time, so a large enough buffer could be >> > allocated in most cases. Perhaps vslock() iff oldlen is very large. >> > The data could still change while it is being copied to a malloc()'ed >> > buffer (or earlier) if the kernel is preemptible. >> >> I basically like this approach. In most cases the data will be a small >> and of fixed size, so the handler could even store the temporary copy on >> the stack. If the data is a type that can be copied atomically, like an >> int, we don't have to worry about the data changing in mid-copy. If it > > It's not clear that ints can be copied atomically unless a suitable lock > is held. Longs can't be on some arches (i386's with 64-bit longs for > example :-). I believe that ints can be copied atomically by assignment if they are naturally aligned and nobody is doing byte writes to them. It's would be likely that longs could be copied atomically as long as neither the the reader nor the writer of the variable in question were preempted by interrupts in the middle of the two halves of their operations. I wouldn't want to make any guarantees without a lot of careful study. It's not likely that you could get an atomic snapshot with bcopy() or copyout() without doing explicit locking. >> is larger and we care about getting an atomic snapshot, the handler can >> lock the source for the duration of the copy to the temporary buffer. >> >> For large amounts of data, we probably want to avoid copying to a >> temporary buffer. PHK raised the issue of the user specifying a buffer >> size that is way too large. Allocating a huge buffer in wired down >> kernel memory sounds even worse than wiring down a huge buffer in user >> space. > > I think this is a small problem provided you only make the buffer large > enough to hold what the kernel is supplying in the current SYSCTL_OUT() > and not what the application is willing to accept for the whole sysctl. > There may be no suppliers of huge kernel data except KERN_VNODE, and > KERN_VNODE was turned off because it was too hard to supply that much > data correctly even years ago when there were fewer vnodes and less > concurrency than now. Any suppliers of huge kernel data may have the > same problems. As long as we know the amount of data ahead of time ... >> If we don't want to allocate a temporary buffer of length oldlen because >> it is much larger than what looks to be reasonable, but the amount of > > OK, but allocating if it is <= PAGE_SIZE or small enough to put on the > stack should handle most cases and hopefully not signficantly complicate > the problem of keeping track of what you allocated. That sounds about right since that's probably about the crossover point between the expense of the extra copy versus the call to vslock(). >> data is not easy to calculate ahead of time, we have another potential >> problem. We might have to lock a data structure, walk through this >> data structure and calculate the total data size, and unlock the data >> structure. After we malloc() the buffer, relock the data structure, and >> start traversing the data structure again to do the copy, we may >> discover that the amount of data to be returned has increased, so we >> would have to drop the lock, free the buffer, and start over again. > > Also, the size might turn out to be too huge to allocate. You would then > need to backtrack and recalulate the size using a less ambitious algorithm. > >> > I think the callers of SYSCTL_OUT() don't need a new API, but they >> > should supply the data in a form suitable for copying out directly. >> > This probably involves them making a copy while holding suitable >> > domain-specific locks. They can't just point to an active kernel >> > struct since it might change. >> > >> > I would just make a copy at a high level for now. >> >> I'd vote for a hybrid approach. I would remove the vslock() call from >> sysctl_old_user() and provide the new API for invoking it from the >> handler if appropriate. Instead, I would call WITNESS_SLEEP() if >> vslock() had not been called to wire the buffer to guard against >> blocking while locks are held. I would keep the call to vsunlock() in >> the sysctl cleanup code. > > It's never really appropriate, but I guess you need it for a few sysctls > because they are too large to fix all at once. I hope the new API won't > be used much. Sysctls with fixed locking can keep using the current code > since they will have copied the data to a safe place and not hold any > locks when they call SYSCTL_OUT() or care if SYSCTL_OUT() makes another > copy or blocks. I suspect that there are very few places where this would be used. A very large percentage of the sysctl calls use the standard handlers and small data types whose size is known ahead of time. All of these can be modified to make a temporary copy of the data and can avoid the vslock() overhead. >> I'd also like to provide a second argument to the interface to vslock() >> to allow the handler to specify a maximum, kernel enforced, buffer >> length to potentially limit the amount of wired memory to something less >> than oldlen, but unless we add a member to the sysctl_req structure to >> hold it, we would need to overwrite oldlen so that we can pass the >> proper value to vsunlock() in the cleanup code. What kind impact would >> changing sysctl_req have on binary compatibility? I doesn't look like a >> problem to me, since this structure seems to be mostly treated as an >> opaque type. > > Changing its size would probably not be good for some modules. > > I think no time should be spent changing interfaces for this. The > following hack might work well: use the current code in sysctl_old_user() > if req->oldlen is not huge. Otherwise, vslock() only the region being > copied out to and vsunlock() it after the copy so that we don't have to > keep track of what is vslock()ed. That doesn't fix sysctl_kern_function_list(), which grabs a lock and then walks a linked list, calling SYSCTL_OUT() on each element. In this case we can't allow SYSCTL_OUT() to block in either vslock() or copyout(). The incremental vslock() solution also doesn't really solve any problems if the calls to vslock() and vsunlock() are done in sysctl_old_user(). They neither guarantee us a coherent copy if the source buffer is dynamic and unlocked (though they might increase the probability), and they also prevent SYSCTL_OUT() being called while a lock is held. My initial solution was to allocate a temporary buffer, but that complicates the code and the similar code might have to be added to other parts of the kernel where similar situations might arise. It was not until later that I thought of pre-wiring the buffer, which looks like a much cleaner solution. In the latter case, I decided that it would be best to abstact the interface so that the handler doesn't need to become intimate with the internals of the sysctl system. If this solution allows us to avoid the call to vslock() in the majority of cases, then so much the better. The initial implementation of the interface to vslock() would probably just ignore the size argument, but defining the API this way would allow it to be used later if necessary without the necessity of retrofitting the handlers that use it. To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Wed Jul 17 0:57:21 2002 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.FreeBSD.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 8991137B400 for ; Wed, 17 Jul 2002 00:57:19 -0700 (PDT) Received: from smtp.noos.fr (zola.noos.net [212.198.2.76]) by mx1.FreeBSD.org (Postfix) with ESMTP id 9C5B743E6E for ; Wed, 17 Jul 2002 00:57:18 -0700 (PDT) (envelope-from root@gits.dyndns.org) Received: (qmail 44033808 invoked by uid 0); 17 Jul 2002 07:57:16 -0000 Received: from unknown (HELO gits.gits.dyndns.org) ([212.198.229.153]) (envelope-sender ) by 212.198.2.76 (qmail-ldap-1.03) with SMTP for ; 17 Jul 2002 07:57:16 -0000 Received: from gits.gits.dyndns.org (6g9omydzms07lczo@localhost [127.0.0.1]) by gits.gits.dyndns.org (8.12.5/8.12.5) with ESMTP id g6H7vGpP010632; Wed, 17 Jul 2002 09:57:16 +0200 (CEST) (envelope-from root@gits.dyndns.org) Received: (from root@localhost) by gits.gits.dyndns.org (8.12.5/8.12.5/Submit) id g6H7vFNg010631; Wed, 17 Jul 2002 09:57:15 +0200 (CEST) (envelope-from root) Date: Wed, 17 Jul 2002 09:57:15 +0200 From: Cyrille Lefevre To: Mark Valentine Cc: Poul-Henning Kamp , Garance A Drosihn , freebsd-arch@FreeBSD.ORG Subject: Re: scripting language in base system? Message-ID: <20020717075715.GA10590@gits.dyndns.org> Mail-Followup-To: Cyrille Lefevre , Mark Valentine , Poul-Henning Kamp , Garance A Drosihn , freebsd-arch@FreeBSD.ORG References: <20020716223107.GC29859@gits.dyndns.org> <200207162331.g6GNVsCa053104@dotar.thuvia.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <200207162331.g6GNVsCa053104@dotar.thuvia.org> User-Agent: Mutt/1.3.99i Organization: ACME X-Face: V|+c;4!|B?E%BE^{E6);aI.[< List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG On Wed, Jul 17, 2002 at 12:31:54AM +0100, Mark Valentine wrote: > > From: Cyrille Lefevre > > Date: Wed 17 Jul, 2002 > > Subject: Re: scripting language in base system? > > > On Tue, Jul 16, 2002 at 08:24:38PM +0100, Mark Valentine wrote: > > > Hm, Evil Thought: dynamically loadable shell builtins, anyone? > > > > > > Maybe not very pretty (see wksh for an example of extending ksh > > > to embrace the X Toolkit API, albeit statically), but perhaps with > > > the addition of a real list data type (which is about all Tcl needs), > > > it might be useful. > > > > http://www.cs.princeton.edu/~jlk/tksh/ > > Neat. It looks like it's annoyingly close to being free, too. yes, I know. it depends on AT&T ksh implementation. > > Q: in the mean time, how about to switch to pdksh as OpenBSD does ? > > Is this going to become necessary to get a more standards-conformant shell? some month ago, I've workded a lot on sh code to make it more SUSv3 compliant in some part but I don't have submitted my changes yet, I I need to validate them before... > Is pdksh the best implementation available to us? I've worked on both code. a little on pdksh one and a lot on sh one. in one word, pdksh code really looks less ugly than our sh code :( Cyrille. -- Cyrille Lefevre mailto:cyrille.lefevre@laposte.net To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Wed Jul 17 1:10:44 2002 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.FreeBSD.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 9350737B400 for ; Wed, 17 Jul 2002 01:10:38 -0700 (PDT) Received: from boreas.isi.edu (boreas.isi.edu [128.9.160.161]) by mx1.FreeBSD.org (Postfix) with ESMTP id 3F6BE43E31 for ; Wed, 17 Jul 2002 01:10:38 -0700 (PDT) (envelope-from larse@ISI.EDU) Received: from isi.edu (ras02.isi.edu [128.9.176.102]) by boreas.isi.edu (8.11.6/8.11.2) with ESMTP id g6H8AXr20740; Wed, 17 Jul 2002 01:10:33 -0700 (PDT) Message-ID: <3D36072C.3080908@isi.edu> Date: Wed, 17 Jul 2002 17:09:16 -0700 From: Lars Eggert User-Agent: Mozilla/5.0 (X11; U; Linux i386; en-US; rv:1.0.0) Gecko/20020529 X-Accept-Language: en-us, de-de MIME-Version: 1.0 To: Luigi Rizzo Cc: Terry Lambert , arch@FreeBSD.ORG Subject: Re: proposed changes to kern_switch.c and kern_synch.c References: <20020716235216.B6785@iguana.icir.org> <3D351B99.311431F@mindspring.com> <20020717004244.B7218@iguana.icir.org> Content-Type: multipart/signed; protocol="application/x-pkcs7-signature"; micalg=sha1; boundary="------------ms050402010500050509000001" Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG This is a cryptographically signed message in MIME format. --------------ms050402010500050509000001 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Luigi Rizzo wrote: > it is basically the WF2Q+ algorithm used in dummynet (i am the king > of recycling :). The OS community calls this "Proportional Share" > or something like that. Since it sounds similar, what are the key differences to the lottery scheduler (http://www.csua.berkeley.edu/computing/software/lottery-sched.html)? Lars -- Lars Eggert USC Information Sciences Institute --------------ms050402010500050509000001 Content-Type: application/x-pkcs7-signature; name="smime.p7s" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="smime.p7s" Content-Description: S/MIME Cryptographic Signature MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIIrjCC ArUwggIeoAMCAQICAwWBRzANBgkqhkiG9w0BAQIFADCBkjELMAkGA1UEBhMCWkExFTATBgNV BAgTDFdlc3Rlcm4gQ2FwZTESMBAGA1UEBxMJQ2FwZSBUb3duMQ8wDQYDVQQKEwZUaGF3dGUx HTAbBgNVBAsTFENlcnRpZmljYXRlIFNlcnZpY2VzMSgwJgYDVQQDEx9QZXJzb25hbCBGcmVl bWFpbCBSU0EgMjAwMC44LjMwMB4XDTAxMDgyNDE2NDAwMFoXDTAyMDgyNDE2NDAwMFowVDEP MA0GA1UEBBMGRWdnZXJ0MQ0wCwYDVQQqEwRMYXJzMRQwEgYDVQQDEwtMYXJzIEVnZ2VydDEc MBoGCSqGSIb3DQEJARYNbGFyc2VAaXNpLmVkdTCBnzANBgkqhkiG9w0BAQEFAAOBjQAwgYkC gYEA0AvLBsD78nxcUHeHkaMgl3b4qYPnfgbf8Lh+HQP8RgGMRG/Yb+vTpkGezlwt9pkJxiD1 1uZDy4CNNJUu3gKxKSb+zRV70O+lkwwftuHoLHoH4xwo3LcQ2LGDpd+I95tUN4dfJ3TmeEcU SF50dC/SuUI4w8AlhXQ8IxrhgdayTpECAwEAAaNWMFQwKgYFK2UBBAEEITAfAgEAMBowGAIB BAQTTDJ1TXlmZkJOVWJOSkpjZFoyczAYBgNVHREEETAPgQ1sYXJzZUBpc2kuZWR1MAwGA1Ud EwEB/wQCMAAwDQYJKoZIhvcNAQECBQADgYEAheZhn0pQA8zI7U2K1ZIAl11j0a1DKxnp3GtT vOUrGRB3WvYxidvdZ1kizhEsWeXU81TkNDH0DaRqtOEeu6Q2OhB+jeKEqY7IDAJE4/fI0e+d 6PnG1hd+vEvYmsKHkmzBhPc94XUOKNWO+qVNP2NGyNI3QIDy5wX4fdcOo1S34r4wggK1MIIC HqADAgECAgMFgUcwDQYJKoZIhvcNAQECBQAwgZIxCzAJBgNVBAYTAlpBMRUwEwYDVQQIEwxX ZXN0ZXJuIENhcGUxEjAQBgNVBAcTCUNhcGUgVG93bjEPMA0GA1UEChMGVGhhd3RlMR0wGwYD VQQLExRDZXJ0aWZpY2F0ZSBTZXJ2aWNlczEoMCYGA1UEAxMfUGVyc29uYWwgRnJlZW1haWwg UlNBIDIwMDAuOC4zMDAeFw0wMTA4MjQxNjQwMDBaFw0wMjA4MjQxNjQwMDBaMFQxDzANBgNV BAQTBkVnZ2VydDENMAsGA1UEKhMETGFyczEUMBIGA1UEAxMLTGFycyBFZ2dlcnQxHDAaBgkq hkiG9w0BCQEWDWxhcnNlQGlzaS5lZHUwgZ8wDQYJKoZIhvcNAQEBBQADgY0AMIGJAoGBANAL ywbA+/J8XFB3h5GjIJd2+KmD534G3/C4fh0D/EYBjERv2G/r06ZBns5cLfaZCcYg9dbmQ8uA jTSVLt4CsSkm/s0Ve9DvpZMMH7bh6Cx6B+McKNy3ENixg6XfiPebVDeHXyd05nhHFEhedHQv 0rlCOMPAJYV0PCMa4YHWsk6RAgMBAAGjVjBUMCoGBStlAQQBBCEwHwIBADAaMBgCAQQEE0wy dU15ZmZCTlViTkpKY2RaMnMwGAYDVR0RBBEwD4ENbGFyc2VAaXNpLmVkdTAMBgNVHRMBAf8E AjAAMA0GCSqGSIb3DQEBAgUAA4GBAIXmYZ9KUAPMyO1NitWSAJddY9GtQysZ6dxrU7zlKxkQ d1r2MYnb3WdZIs4RLFnl1PNU5DQx9A2karThHrukNjoQfo3ihKmOyAwCROP3yNHvnej5xtYX frxL2JrCh5JswYT3PeF1DijVjvqlTT9jRsjSN0CA8ucF+H3XDqNUt+K+MIIDODCCAqGgAwIB AgIQZkVyt8x09c9jdkWE0C6RATANBgkqhkiG9w0BAQQFADCB0TELMAkGA1UEBhMCWkExFTAT BgNVBAgTDFdlc3Rlcm4gQ2FwZTESMBAGA1UEBxMJQ2FwZSBUb3duMRowGAYDVQQKExFUaGF3 dGUgQ29uc3VsdGluZzEoMCYGA1UECxMfQ2VydGlmaWNhdGlvbiBTZXJ2aWNlcyBEaXZpc2lv bjEkMCIGA1UEAxMbVGhhd3RlIFBlcnNvbmFsIEZyZWVtYWlsIENBMSswKQYJKoZIhvcNAQkB FhxwZXJzb25hbC1mcmVlbWFpbEB0aGF3dGUuY29tMB4XDTAwMDgzMDAwMDAwMFoXDTA0MDgy NzIzNTk1OVowgZIxCzAJBgNVBAYTAlpBMRUwEwYDVQQIEwxXZXN0ZXJuIENhcGUxEjAQBgNV BAcTCUNhcGUgVG93bjEPMA0GA1UEChMGVGhhd3RlMR0wGwYDVQQLExRDZXJ0aWZpY2F0ZSBT ZXJ2aWNlczEoMCYGA1UEAxMfUGVyc29uYWwgRnJlZW1haWwgUlNBIDIwMDAuOC4zMDCBnzAN BgkqhkiG9w0BAQEFAAOBjQAwgYkCgYEA3jMypmPHCSVFPtJueCdngcXaiBmClw7jRCmKYzUq bXA8+tyu9+50bzC8M5B/+TRxoKNtmPHDT6Jl2w36S/HW3WGl+YXNVZo1Gp2Sdagnrthy+boC 9tewkd4c6avgGAOofENCUFGHgzzwObSbVIoTh/+zm51JZgAtCYnslGvpoWkCAwEAAaNOMEww KQYDVR0RBCIwIKQeMBwxGjAYBgNVBAMTEVByaXZhdGVMYWJlbDEtMjk3MBIGA1UdEwEB/wQI MAYBAf8CAQAwCwYDVR0PBAQDAgEGMA0GCSqGSIb3DQEBBAUAA4GBADGxS0dd+QFx5fVTbF15 1j2YwCYTYoEipxL4IpXoG0m3J3sEObr85vIk65H6vewNKjj3UFWobPcNrUwbvAP0teuiR59s ogxYjTFCCRFssBpp0SsSskBdavl50OouJd2K5PzbDR+dAvNa28o89kTqJmmHf0iezqWf54TY yWJirQXGMYICpjCCAqICAQEwgZowgZIxCzAJBgNVBAYTAlpBMRUwEwYDVQQIEwxXZXN0ZXJu IENhcGUxEjAQBgNVBAcTCUNhcGUgVG93bjEPMA0GA1UEChMGVGhhd3RlMR0wGwYDVQQLExRD ZXJ0aWZpY2F0ZSBTZXJ2aWNlczEoMCYGA1UEAxMfUGVyc29uYWwgRnJlZW1haWwgUlNBIDIw MDAuOC4zMAIDBYFHMAkGBSsOAwIaBQCgggFhMBgGCSqGSIb3DQEJAzELBgkqhkiG9w0BBwEw HAYJKoZIhvcNAQkFMQ8XDTAyMDcxODAwMDkxNlowIwYJKoZIhvcNAQkEMRYEFIkn8vFBwRWj M+A7W5wNHqeRmOATMFIGCSqGSIb3DQEJDzFFMEMwCgYIKoZIhvcNAwcwDgYIKoZIhvcNAwIC AgCAMA0GCCqGSIb3DQMCAgFAMAcGBSsOAwIHMA0GCCqGSIb3DQMCAgEoMIGtBgsqhkiG9w0B CRACCzGBnaCBmjCBkjELMAkGA1UEBhMCWkExFTATBgNVBAgTDFdlc3Rlcm4gQ2FwZTESMBAG A1UEBxMJQ2FwZSBUb3duMQ8wDQYDVQQKEwZUaGF3dGUxHTAbBgNVBAsTFENlcnRpZmljYXRl IFNlcnZpY2VzMSgwJgYDVQQDEx9QZXJzb25hbCBGcmVlbWFpbCBSU0EgMjAwMC44LjMwAgMF gUcwDQYJKoZIhvcNAQEBBQAEgYCy1l0bJziYdXqm3PYN4Hq8Ro9bx8lXUe7d6DXLGczzoj32 Xvx4shrE7gS94SCvdWfVHwy8MnrhXYzVdVwfuMi5VnZYo0Dixu9GBGp11LGY0GozwHHbY+gH Tl1u3QGMFRKAmNWULOuFx1y0eyhfWS8XJ4uzAvDMa1IpLRk4HOYYLwAAAAAAAA== --------------ms050402010500050509000001-- To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Wed Jul 17 1:57:54 2002 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.FreeBSD.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 5906837B401 for ; Wed, 17 Jul 2002 01:57:52 -0700 (PDT) Received: from iguana.icir.org (iguana.icir.org [192.150.187.36]) by mx1.FreeBSD.org (Postfix) with ESMTP id 1424F43E42 for ; Wed, 17 Jul 2002 01:57:51 -0700 (PDT) (envelope-from rizzo@iguana.icir.org) Received: (from rizzo@localhost) by iguana.icir.org (8.11.6/8.11.3) id g6H8vmW07871; Wed, 17 Jul 2002 01:57:48 -0700 (PDT) (envelope-from rizzo) Date: Wed, 17 Jul 2002 01:57:48 -0700 From: Luigi Rizzo To: Lars Eggert Cc: Terry Lambert , arch@FreeBSD.ORG Subject: Re: proposed changes to kern_switch.c and kern_synch.c Message-ID: <20020717015748.B7719@iguana.icir.org> References: <20020716235216.B6785@iguana.icir.org> <3D351B99.311431F@mindspring.com> <20020717004244.B7218@iguana.icir.org> <3D36072C.3080908@isi.edu> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2.5.1i In-Reply-To: <3D36072C.3080908@isi.edu>; from larse@ISI.EDU on Wed, Jul 17, 2002 at 05:09:16PM -0700 Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG On Wed, Jul 17, 2002 at 05:09:16PM -0700, Lars Eggert wrote: > Luigi Rizzo wrote: > > it is basically the WF2Q+ algorithm used in dummynet (i am the king > > of recycling :). The OS community calls this "Proportional Share" > > or something like that. > > Since it sounds similar, what are the key differences to the lottery > scheduler > (http://www.csua.berkeley.edu/computing/software/lottery-sched.html)? i don't know much on the lottery scheduler, but it looks like it is probabilistic as opposed to deterministic. The paper on lottery scheduling claims to have the same O(log N) complexity as WF2Q+ (though i haven't looked at the implementation issues). In any case, one of the good things of our work is that it provides a framework for replacing the standard scheduler with another one with minimal interaction with the rest of the system, being all code and data structures confined in a private file/data structures. cheers luigi To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Wed Jul 17 2:22:14 2002 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.FreeBSD.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 3084837B400 for ; Wed, 17 Jul 2002 02:22:12 -0700 (PDT) Received: from canning.wemm.org (canning.wemm.org [192.203.228.65]) by mx1.FreeBSD.org (Postfix) with ESMTP id 9432F43E42 for ; Wed, 17 Jul 2002 02:22:11 -0700 (PDT) (envelope-from peter@wemm.org) Received: from fw.wemm.org (canning.wemm.org [192.203.228.65]) by canning.wemm.org (Postfix) with ESMTP id 279072A7D6 for ; Wed, 17 Jul 2002 02:22:11 -0700 (PDT) (envelope-from peter@wemm.org) Received: from overcee.wemm.org (overcee.wemm.org [10.0.0.3]) by fw.wemm.org (Postfix) with ESMTP id E76304C26C for ; Wed, 17 Jul 2002 02:22:10 -0700 (PDT) (envelope-from peter@wemm.org) Received: from wemm.org (localhost [127.0.0.1]) by overcee.wemm.org (Postfix) with ESMTP id 934423915; Wed, 17 Jul 2002 02:22:10 -0700 (PDT) (envelope-from peter@wemm.org) X-Mailer: exmh version 2.5 07/13/2001 with nmh-1.0.4 To: Luigi Rizzo Cc: arch@freebsd.org Subject: Re: proposed changes to kern_switch.c and kern_synch.c In-Reply-To: <20020717004750.A7375@iguana.icir.org> Date: Wed, 17 Jul 2002 02:22:10 -0700 From: Peter Wemm Message-Id: <20020717092210.934423915@overcee.wemm.org> Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG Luigi Rizzo wrote: > On Wed, Jul 17, 2002 at 12:43:35AM -0700, Peter Wemm wrote: > > Luigi Rizzo wrote: > > > > > In order to make this work, it is convenient to have all > > > scheduler-specific functions and data structures in a > > > single file (kern_switch*.c), and generic support in > > > another one (kern_synch.c). I believe this was also the original > > > BSD design in partitioning the code between the two files. > > > > You would be mistaken there. kern_switch.c is new and has only existed > > very recently. kern_switch.c came about as a C implementation of code that > > used to be embedded inside i386/swtch.s. It has taken on a life of its own > > now though. :-/ > > good to know! > Anyways, does the partitioning of functionalities sound reasonable ? Please be a little careful. There has been a lot of talk about splitting the current scheduler into an event driven system rather than a 'run once a second' thing. This was something along the lines of splitting the current schedcpu stuff and the various other glue into 3 parts. I think the gist of it was to have each process do a slight incremental priority adjustment each time it used up its entire quantum, a slight adjustment at wakeup time, and the periodic system load factoring code would still run periodically. You haven't given much details about how you're intending to implement your alternate scheduler, or even which part you're intending to provide an alternative for (I'm guessing a different kern_switch, ie: alterantive run queues, chooseproc^H^H^H^Hkse, setrunqueue(), etc) Some other comments.. You seem to have seperate the comments about the digital decay magic in schedcpu() from the code that does the magic. In fact, things like cexp[] etc are shared between kern_switch (schedcpu1) and kern_synch (loadav) etc. Are you sure the partitioning is correct? Maybe *everything* other than the msleep/wakeup/etc related stuff should move? ie: setrunnable() (which uses updatepri()) etc move as well? Cheers, -Peter -- Peter Wemm - peter@wemm.org; peter@FreeBSD.org; peter@yahoo-inc.com "All of this is for nothing if we don't go to the stars" - JMS/B5 To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Wed Jul 17 2:36:58 2002 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.FreeBSD.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id C669137B400 for ; Wed, 17 Jul 2002 02:36:54 -0700 (PDT) Received: from iguana.icir.org (iguana.icir.org [192.150.187.36]) by mx1.FreeBSD.org (Postfix) with ESMTP id 64DBA43E64 for ; Wed, 17 Jul 2002 02:36:54 -0700 (PDT) (envelope-from rizzo@iguana.icir.org) Received: (from rizzo@localhost) by iguana.icir.org (8.11.6/8.11.3) id g6H9arp08510; Wed, 17 Jul 2002 02:36:53 -0700 (PDT) (envelope-from rizzo) Date: Wed, 17 Jul 2002 02:36:53 -0700 From: Luigi Rizzo To: Peter Wemm Cc: arch@FreeBSD.ORG Subject: Re: proposed changes to kern_switch.c and kern_synch.c Message-ID: <20020717023653.B8351@iguana.icir.org> References: <20020717004750.A7375@iguana.icir.org> <20020717092210.934423915@overcee.wemm.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2.5.1i In-Reply-To: <20020717092210.934423915@overcee.wemm.org>; from peter@wemm.org on Wed, Jul 17, 2002 at 02:22:10AM -0700 Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG there is a lot of overload in the use of priority fields to store both process state and priority info. Decoupling them is non-trivial, and so we decided to leave the old scheduler unmodified, and our alternate scheduler right now tries to keep the relevant information updated in a consistent way (please be patient, the code will come out in a couple of days, but i thought the overall design is more important than the details, which besides might have bugs). Rather than proposing a massive change all at once, my idea was to play safe and just start with moving some code from one file to another, and then revise things one step at a time. You are right about the schedcpu1() thing, in -stable the block of code moved there is slightly smalle i think. Yes, the alternate scheduler is in kern_switch_*.c, providing replacement for the various chooseproc() and friends, which are accessed through function pointers initialized according to the scheduler being chosen. cheers luigi On Wed, Jul 17, 2002 at 02:22:10AM -0700, Peter Wemm wrote: ... > > Anyways, does the partitioning of functionalities sound reasonable ? > > Please be a little careful. There has been a lot of talk about splitting > the current scheduler into an event driven system rather than a 'run once a > second' thing. This was something along the lines of splitting the current > schedcpu stuff and the various other glue into 3 parts. I think the gist > of it was to have each process do a slight incremental priority adjustment > each time it used up its entire quantum, a slight adjustment at wakeup time, > and the periodic system load factoring code would still run periodically. > > You haven't given much details about how you're intending to implement your > alternate scheduler, or even which part you're intending to provide an > alternative for (I'm guessing a different kern_switch, ie: alterantive > run queues, chooseproc^H^H^H^Hkse, setrunqueue(), etc) > > Some other comments.. You seem to have seperate the comments about the > digital decay magic in schedcpu() from the code that does the magic. In > fact, things like cexp[] etc are shared between kern_switch (schedcpu1) and > kern_synch (loadav) etc. Are you sure the partitioning is correct? > > Maybe *everything* other than the msleep/wakeup/etc related stuff should move? > ie: setrunnable() (which uses updatepri()) etc move as well? > > Cheers, > -Peter > -- > Peter Wemm - peter@wemm.org; peter@FreeBSD.org; peter@yahoo-inc.com > "All of this is for nothing if we don't go to the stars" - JMS/B5 > > > To Unsubscribe: send mail to majordomo@FreeBSD.org > with "unsubscribe freebsd-arch" in the body of the message To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Wed Jul 17 3:54:48 2002 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.FreeBSD.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id A885B37B400 for ; Wed, 17 Jul 2002 03:54:46 -0700 (PDT) Received: from hardtime.linuxman.net (hardtime.linuxman.net [66.147.26.65]) by mx1.FreeBSD.org (Postfix) with ESMTP id 774FD43E42 for ; Wed, 17 Jul 2002 03:54:45 -0700 (PDT) (envelope-from fullermd@over-yonder.net) Received: from mortis.over-yonder.net (localhost [127.0.0.1]) by hardtime.linuxman.net (8.11.6/8.11.6) with ESMTP id g6HAB3J05589; Wed, 17 Jul 2002 05:11:08 -0500 Received: by mortis.over-yonder.net (Postfix, from userid 100) id 39C131F06; Wed, 17 Jul 2002 05:54:34 -0500 (CDT) Date: Wed, 17 Jul 2002 05:54:34 -0500 From: "Matthew D. Fuller" To: Luigi Rizzo Cc: arch@freebsd.org Subject: Re: proposed changes to kern_switch.c and kern_synch.c Message-ID: <20020717105433.GB29269@over-yonder.net> References: <20020716235216.B6785@iguana.icir.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20020716235216.B6785@iguana.icir.org> User-Agent: Mutt/1.4i-fullermd.1 X-Editor: vi X-OS: FreeBSD Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG On Tue, Jul 16, 2002 at 11:52:16PM -0700 I heard the voice of Luigi Rizzo, and lo! it spake thus: > > --- Re. the multi-scheduler architecture --- > > The general idea is to make the process/thread/kse scheduler > a replaceable piece of the kernel, requiring no modifications > to the "struct proc", and with the ability of switching from > one scheduler to another one at runtime (this both for testing > purposes and for whatever need may arise). Random related thoughts: 1) From the work you've done on this already, how difficult would you expect it to be to do this in such a way that you could have multiple (>2) schedulers around, loading and unloading at will (when disabled, of course) through KLD's? 2) How much unavoidable overhead is there in switching between the schedulers? Could it get low enough that it might be meaningful to, instead of writing One True Scheduler, instead write 3 or 4 different optimized ones, and have some intelligence to switch between them automagically as load demands? -- Matthew Fuller (MF4839) | fullermd@over-yonder.net Systems/Network Administrator | http://www.over-yonder.net/~fullermd/ "The only reason I'm burning my candle at both ends, is because I haven't figured out how to light the middle yet" To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Wed Jul 17 4:10:16 2002 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.FreeBSD.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id B562937B400 for ; Wed, 17 Jul 2002 04:10:12 -0700 (PDT) Received: from boreas.isi.edu (boreas.isi.edu [128.9.160.161]) by mx1.FreeBSD.org (Postfix) with ESMTP id 1D2CB43E42 for ; Wed, 17 Jul 2002 04:10:12 -0700 (PDT) (envelope-from larse@ISI.EDU) Received: from isi.edu (ras02.isi.edu [128.9.176.102]) by boreas.isi.edu (8.11.6/8.11.2) with ESMTP id g6HB9kr22237; Wed, 17 Jul 2002 04:09:47 -0700 (PDT) Message-ID: <3D36312D.1010606@isi.edu> Date: Wed, 17 Jul 2002 20:08:29 -0700 From: Lars Eggert User-Agent: Mozilla/5.0 (X11; U; Linux i386; en-US; rv:1.0.0) Gecko/20020529 X-Accept-Language: en-us, de-de MIME-Version: 1.0 To: Luigi Rizzo Cc: Terry Lambert , arch@FreeBSD.ORG Subject: Re: proposed changes to kern_switch.c and kern_synch.c References: <20020716235216.B6785@iguana.icir.org> <3D351B99.311431F@mindspring.com> <20020717004244.B7218@iguana.icir.org> <3D36072C.3080908@isi.edu> <20020717015748.B7719@iguana.icir.org> Content-Type: multipart/signed; protocol="application/x-pkcs7-signature"; micalg=sha1; boundary="------------ms020504070509040502040606" Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG This is a cryptographically signed message in MIME format. --------------ms020504070509040502040606 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Luigi Rizzo wrote: >>Since it sounds similar, what are the key differences to the lottery >>scheduler >>(http://www.csua.berkeley.edu/computing/software/lottery-sched.html)? > > > i don't know much on the lottery scheduler, but it looks like it > is probabilistic as opposed to deterministic. The paper on lottery > scheduling claims to have the same O(log N) complexity as WF2Q+ > (though i haven't looked at the implementation issues). Ah, so it's more like Waldspurger's later stride scheduler than his lottery scheduler. > In any case, one of the good things of our work is that it provides > a framework for replacing the standard scheduler with another one > with minimal interaction with the rest of the system, being all > code and data structures confined in a private file/data structures. I agree; that sounds useful. Thanks for the info, Lars -- Lars Eggert USC Information Sciences Institute --------------ms020504070509040502040606 Content-Type: application/x-pkcs7-signature; name="smime.p7s" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="smime.p7s" Content-Description: S/MIME Cryptographic Signature MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIIrjCC ArUwggIeoAMCAQICAwWBRzANBgkqhkiG9w0BAQIFADCBkjELMAkGA1UEBhMCWkExFTATBgNV BAgTDFdlc3Rlcm4gQ2FwZTESMBAGA1UEBxMJQ2FwZSBUb3duMQ8wDQYDVQQKEwZUaGF3dGUx HTAbBgNVBAsTFENlcnRpZmljYXRlIFNlcnZpY2VzMSgwJgYDVQQDEx9QZXJzb25hbCBGcmVl bWFpbCBSU0EgMjAwMC44LjMwMB4XDTAxMDgyNDE2NDAwMFoXDTAyMDgyNDE2NDAwMFowVDEP MA0GA1UEBBMGRWdnZXJ0MQ0wCwYDVQQqEwRMYXJzMRQwEgYDVQQDEwtMYXJzIEVnZ2VydDEc MBoGCSqGSIb3DQEJARYNbGFyc2VAaXNpLmVkdTCBnzANBgkqhkiG9w0BAQEFAAOBjQAwgYkC gYEA0AvLBsD78nxcUHeHkaMgl3b4qYPnfgbf8Lh+HQP8RgGMRG/Yb+vTpkGezlwt9pkJxiD1 1uZDy4CNNJUu3gKxKSb+zRV70O+lkwwftuHoLHoH4xwo3LcQ2LGDpd+I95tUN4dfJ3TmeEcU SF50dC/SuUI4w8AlhXQ8IxrhgdayTpECAwEAAaNWMFQwKgYFK2UBBAEEITAfAgEAMBowGAIB BAQTTDJ1TXlmZkJOVWJOSkpjZFoyczAYBgNVHREEETAPgQ1sYXJzZUBpc2kuZWR1MAwGA1Ud EwEB/wQCMAAwDQYJKoZIhvcNAQECBQADgYEAheZhn0pQA8zI7U2K1ZIAl11j0a1DKxnp3GtT vOUrGRB3WvYxidvdZ1kizhEsWeXU81TkNDH0DaRqtOEeu6Q2OhB+jeKEqY7IDAJE4/fI0e+d 6PnG1hd+vEvYmsKHkmzBhPc94XUOKNWO+qVNP2NGyNI3QIDy5wX4fdcOo1S34r4wggK1MIIC HqADAgECAgMFgUcwDQYJKoZIhvcNAQECBQAwgZIxCzAJBgNVBAYTAlpBMRUwEwYDVQQIEwxX ZXN0ZXJuIENhcGUxEjAQBgNVBAcTCUNhcGUgVG93bjEPMA0GA1UEChMGVGhhd3RlMR0wGwYD VQQLExRDZXJ0aWZpY2F0ZSBTZXJ2aWNlczEoMCYGA1UEAxMfUGVyc29uYWwgRnJlZW1haWwg UlNBIDIwMDAuOC4zMDAeFw0wMTA4MjQxNjQwMDBaFw0wMjA4MjQxNjQwMDBaMFQxDzANBgNV BAQTBkVnZ2VydDENMAsGA1UEKhMETGFyczEUMBIGA1UEAxMLTGFycyBFZ2dlcnQxHDAaBgkq hkiG9w0BCQEWDWxhcnNlQGlzaS5lZHUwgZ8wDQYJKoZIhvcNAQEBBQADgY0AMIGJAoGBANAL ywbA+/J8XFB3h5GjIJd2+KmD534G3/C4fh0D/EYBjERv2G/r06ZBns5cLfaZCcYg9dbmQ8uA jTSVLt4CsSkm/s0Ve9DvpZMMH7bh6Cx6B+McKNy3ENixg6XfiPebVDeHXyd05nhHFEhedHQv 0rlCOMPAJYV0PCMa4YHWsk6RAgMBAAGjVjBUMCoGBStlAQQBBCEwHwIBADAaMBgCAQQEE0wy dU15ZmZCTlViTkpKY2RaMnMwGAYDVR0RBBEwD4ENbGFyc2VAaXNpLmVkdTAMBgNVHRMBAf8E AjAAMA0GCSqGSIb3DQEBAgUAA4GBAIXmYZ9KUAPMyO1NitWSAJddY9GtQysZ6dxrU7zlKxkQ d1r2MYnb3WdZIs4RLFnl1PNU5DQx9A2karThHrukNjoQfo3ihKmOyAwCROP3yNHvnej5xtYX frxL2JrCh5JswYT3PeF1DijVjvqlTT9jRsjSN0CA8ucF+H3XDqNUt+K+MIIDODCCAqGgAwIB AgIQZkVyt8x09c9jdkWE0C6RATANBgkqhkiG9w0BAQQFADCB0TELMAkGA1UEBhMCWkExFTAT BgNVBAgTDFdlc3Rlcm4gQ2FwZTESMBAGA1UEBxMJQ2FwZSBUb3duMRowGAYDVQQKExFUaGF3 dGUgQ29uc3VsdGluZzEoMCYGA1UECxMfQ2VydGlmaWNhdGlvbiBTZXJ2aWNlcyBEaXZpc2lv bjEkMCIGA1UEAxMbVGhhd3RlIFBlcnNvbmFsIEZyZWVtYWlsIENBMSswKQYJKoZIhvcNAQkB FhxwZXJzb25hbC1mcmVlbWFpbEB0aGF3dGUuY29tMB4XDTAwMDgzMDAwMDAwMFoXDTA0MDgy NzIzNTk1OVowgZIxCzAJBgNVBAYTAlpBMRUwEwYDVQQIEwxXZXN0ZXJuIENhcGUxEjAQBgNV BAcTCUNhcGUgVG93bjEPMA0GA1UEChMGVGhhd3RlMR0wGwYDVQQLExRDZXJ0aWZpY2F0ZSBT ZXJ2aWNlczEoMCYGA1UEAxMfUGVyc29uYWwgRnJlZW1haWwgUlNBIDIwMDAuOC4zMDCBnzAN BgkqhkiG9w0BAQEFAAOBjQAwgYkCgYEA3jMypmPHCSVFPtJueCdngcXaiBmClw7jRCmKYzUq bXA8+tyu9+50bzC8M5B/+TRxoKNtmPHDT6Jl2w36S/HW3WGl+YXNVZo1Gp2Sdagnrthy+boC 9tewkd4c6avgGAOofENCUFGHgzzwObSbVIoTh/+zm51JZgAtCYnslGvpoWkCAwEAAaNOMEww KQYDVR0RBCIwIKQeMBwxGjAYBgNVBAMTEVByaXZhdGVMYWJlbDEtMjk3MBIGA1UdEwEB/wQI MAYBAf8CAQAwCwYDVR0PBAQDAgEGMA0GCSqGSIb3DQEBBAUAA4GBADGxS0dd+QFx5fVTbF15 1j2YwCYTYoEipxL4IpXoG0m3J3sEObr85vIk65H6vewNKjj3UFWobPcNrUwbvAP0teuiR59s ogxYjTFCCRFssBpp0SsSskBdavl50OouJd2K5PzbDR+dAvNa28o89kTqJmmHf0iezqWf54TY yWJirQXGMYICpjCCAqICAQEwgZowgZIxCzAJBgNVBAYTAlpBMRUwEwYDVQQIEwxXZXN0ZXJu IENhcGUxEjAQBgNVBAcTCUNhcGUgVG93bjEPMA0GA1UEChMGVGhhd3RlMR0wGwYDVQQLExRD ZXJ0aWZpY2F0ZSBTZXJ2aWNlczEoMCYGA1UEAxMfUGVyc29uYWwgRnJlZW1haWwgUlNBIDIw MDAuOC4zMAIDBYFHMAkGBSsOAwIaBQCgggFhMBgGCSqGSIb3DQEJAzELBgkqhkiG9w0BBwEw HAYJKoZIhvcNAQkFMQ8XDTAyMDcxODAzMDgyOVowIwYJKoZIhvcNAQkEMRYEFL2pCROdjdsF sEBRFN4z9GfLSbfvMFIGCSqGSIb3DQEJDzFFMEMwCgYIKoZIhvcNAwcwDgYIKoZIhvcNAwIC AgCAMA0GCCqGSIb3DQMCAgFAMAcGBSsOAwIHMA0GCCqGSIb3DQMCAgEoMIGtBgsqhkiG9w0B CRACCzGBnaCBmjCBkjELMAkGA1UEBhMCWkExFTATBgNVBAgTDFdlc3Rlcm4gQ2FwZTESMBAG A1UEBxMJQ2FwZSBUb3duMQ8wDQYDVQQKEwZUaGF3dGUxHTAbBgNVBAsTFENlcnRpZmljYXRl IFNlcnZpY2VzMSgwJgYDVQQDEx9QZXJzb25hbCBGcmVlbWFpbCBSU0EgMjAwMC44LjMwAgMF gUcwDQYJKoZIhvcNAQEBBQAEgYAbtFPTTmaesLI4vzbK0DZPasP9LxXDnMb7rY3TvcK2NO/4 wQbhJz7D3ZQYU866UKdTm92pZ9Viy3yhCoXeC4bpXuIBPs7L+R9Lzaae5usg9FQ+umsM+npa HzyzNT3g5fqqA7NWSlEYvXGJBln9TTHgK9imlfsp5qnure0jjlltSwAAAAAAAA== --------------ms020504070509040502040606-- To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Wed Jul 17 4:30:56 2002 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.FreeBSD.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 6CE7337B400 for ; Wed, 17 Jul 2002 04:30:52 -0700 (PDT) Received: from gw.catspoiler.org (217-ip-163.nccn.net [209.79.217.163]) by mx1.FreeBSD.org (Postfix) with ESMTP id E4DB243E3B for ; Wed, 17 Jul 2002 04:30:51 -0700 (PDT) (envelope-from dl-freebsd@catspoiler.org) Received: from mousie.catspoiler.org (mousie.catspoiler.org [192.168.101.2]) by gw.catspoiler.org (8.12.5/8.12.5) with ESMTP id g6HBUhwr029921; Wed, 17 Jul 2002 04:30:47 -0700 (PDT) (envelope-from dl-freebsd@catspoiler.org) Message-Id: <200207171130.g6HBUhwr029921@gw.catspoiler.org> Date: Wed, 17 Jul 2002 04:30:43 -0700 (PDT) From: Don Lewis Subject: Re: wiring the sysctl output buffer To: bde@zeta.org.au Cc: arch@FreeBSD.ORG In-Reply-To: <200207170755.g6H7t1wr029257@gw.catspoiler.org> MIME-Version: 1.0 Content-Type: TEXT/plain; charset=US-ASCII Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG On 17 Jul, Don Lewis wrote: > On 17 Jul, Bruce Evans wrote: >> I think no time should be spent changing interfaces for this. The >> following hack might work well: use the current code in sysctl_old_user() >> if req->oldlen is not huge. Otherwise, vslock() only the region being >> copied out to and vsunlock() it after the copy so that we don't have to >> keep track of what is vslock()ed. > > That doesn't fix sysctl_kern_function_list(), which grabs a lock and > then walks a linked list, calling SYSCTL_OUT() on each element. Two other places that we have the similar problems are tcp_pcblist() and udp_pcblist(). This code appears to have other problems as well, such as the possibility of looking at stale data on the free list ... To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Wed Jul 17 4:45:44 2002 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.FreeBSD.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 67A2137B400 for ; Wed, 17 Jul 2002 04:45:41 -0700 (PDT) Received: from iguana.icir.org (iguana.icir.org [192.150.187.36]) by mx1.FreeBSD.org (Postfix) with ESMTP id 20FCC43E42 for ; Wed, 17 Jul 2002 04:45:41 -0700 (PDT) (envelope-from rizzo@iguana.icir.org) Received: (from rizzo@localhost) by iguana.icir.org (8.11.6/8.11.3) id g6HBjc809384; Wed, 17 Jul 2002 04:45:38 -0700 (PDT) (envelope-from rizzo) Date: Wed, 17 Jul 2002 04:45:38 -0700 From: Luigi Rizzo To: "Matthew D. Fuller" Cc: arch@FreeBSD.ORG Subject: Re: proposed changes to kern_switch.c and kern_synch.c Message-ID: <20020717044538.A9282@iguana.icir.org> References: <20020716235216.B6785@iguana.icir.org> <20020717105433.GB29269@over-yonder.net> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2.5.1i In-Reply-To: <20020717105433.GB29269@over-yonder.net>; from fullermd@over-yonder.net on Wed, Jul 17, 2002 at 05:54:34AM -0500 Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG On Wed, Jul 17, 2002 at 05:54:34AM -0500, Matthew D. Fuller wrote: ... > Luigi Rizzo, and lo! it spake thus: > > > > --- Re. the multi-scheduler architecture --- > > > > The general idea is to make the process/thread/kse scheduler > > a replaceable piece of the kernel, requiring no modifications > > to the "struct proc", and with the ability of switching from > > one scheduler to another one at runtime (this both for testing > > purposes and for whatever need may arise). > > Random related thoughts: > > 1) From the work you've done on this already, how difficult would you > expect it to be to do this in such a way that you could have multiple > (>2) schedulers around, loading and unloading at will (when disabled, of > course) through KLD's? it is basically already done -- schedulers are loaded and initialized via SYSINIT, and when you switch to another one all references to the old one are removed and data structures freed. It is just a matter of providing the module glue. > 2) How much unavoidable overhead is there in switching between the > schedulers? Could it get low enough that it might be meaningful to, > instead of writing One True Scheduler, instead write 3 or 4 different > optimized ones, and have some intelligence to switch between them > automagically as load demands? you need to move all processes from one scheduler to another, so it is O(n) overhead. Frankly i do not believe it is very interesting to switch between different schedulers at runtime other than for testing purposes. What _might_ be more interesting is to use multiple schedulers in a hierarchical fashion, e.g. WFQ within each priority class, or priority among threads or processes belonging to the same scheduling entity (KSEG or process group). Not that I am an advocate of that either, i generally prefer flat structures. cheers luigi To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Wed Jul 17 6:23:19 2002 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.FreeBSD.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 2593A37B400 for ; Wed, 17 Jul 2002 06:23:15 -0700 (PDT) Received: from mailman.zeta.org.au (mailman.zeta.org.au [203.26.10.16]) by mx1.FreeBSD.org (Postfix) with ESMTP id 1D0AA43E31 for ; Wed, 17 Jul 2002 06:23:14 -0700 (PDT) (envelope-from bde@zeta.org.au) Received: from bde.zeta.org.au (bde.zeta.org.au [203.2.228.102]) by mailman.zeta.org.au (8.9.3/8.8.7) with ESMTP id XAA12981; Wed, 17 Jul 2002 23:23:04 +1000 Date: Wed, 17 Jul 2002 23:26:49 +1000 (EST) From: Bruce Evans X-X-Sender: bde@gamplex.bde.org To: Luigi Rizzo Cc: arch@FreeBSD.ORG Subject: Re: proposed changes to kern_switch.c and kern_synch.c In-Reply-To: <20020716235216.B6785@iguana.icir.org> Message-ID: <20020717223904.J5141-100000@gamplex.bde.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG On Tue, 16 Jul 2002, Luigi Rizzo wrote: > we have implemented a weight-based process scheduler > for FreeBSD-stable, and are trying to port it to -current (in the > description below replace "process" with "thread/kse" as appropriate > for the -current case). > > In order to make this work, it is convenient to have all > scheduler-specific functions and data structures in a > single file (kern_switch*.c), and generic support in > another one (kern_synch.c). I believe this was also the original > BSD design in partitioning the code between the two files. > However, in our current code, there are some functions which > are scheduler-specific (see below) which are misplaced. This move would be sort of backwards IMO. As has been pointed out, the original BSD design is nothing like that. Almost everything related to scheduling was in kern_synch.c. I think of kern_synch.c is being mainly for scheduling (and sleeping), and wouldn't like to see the scheduling stuff moved out of it, especially to kern_switch.c. Moving mi_switch() to kern_switch.c would be OK. My viewpoint may be colored by having 902 lines of diffs (-c2) for kern_synch.c including about 100 lines of bug fixes and 300 lines of scheduler changes. (Latest breakage: running processes are classified as sleeping and not scheduled. Scheduling is not very important, so even huge bugs like this go unnoticed.) > So I would like to make the following, purely cosmetic, change > identified as step #1 below, both for consistency with what i > believe to be the original design, and to simplify further work > in this area. These would go to both -current and -stable; I > repeat, they are only cosmetic changes. > ... > Step #1 is basically a cosmetic change, requiring mostly a move of > some functions from kern_synch.c to kern_switch.c. These are > > roundrobin(); > > curpriority_cmp(); curpriority_cmp() doesn't exist in -current and shouldn't be reintroduced. It is part of maybe_resched() in -current. I think that this function and most of the things now in kern_switch.c should not be scheduler-dependent. Each scheduler can manage priorities and let common code choose the process with highest priority. If you want a different priority scheme then you would need to change much more, especially in -current where there is priority-propagation stuff that depends on linear priorities to work. > > resetpriority(); > > parts of schedcpu() and schedclock(); Why not all of these functions? schedcpu() is almost empty after the move. Each scheduler can easily duplicate this part (or maybe not have it). Bruce To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Wed Jul 17 7:12:37 2002 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.FreeBSD.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 52D1937B400 for ; Wed, 17 Jul 2002 07:12:34 -0700 (PDT) Received: from fledge.watson.org (fledge.watson.org [204.156.12.50]) by mx1.FreeBSD.org (Postfix) with ESMTP id 9BE5643E31 for ; Wed, 17 Jul 2002 07:12:33 -0700 (PDT) (envelope-from robert@fledge.watson.org) Received: from fledge.watson.org (fledge.pr.watson.org [192.0.2.3]) by fledge.watson.org (8.12.4/8.12.4) with SMTP id g6HECROo098312; Wed, 17 Jul 2002 10:12:27 -0400 (EDT) (envelope-from robert@fledge.watson.org) Date: Wed, 17 Jul 2002 10:12:26 -0400 (EDT) From: Robert Watson To: Luigi Rizzo Cc: arch@freebsd.org Subject: Re: proposed changes to kern_switch.c and kern_synch.c In-Reply-To: <20020716235216.B6785@iguana.icir.org> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG I'm very excited about the idea of a more flexible scheduler interface, as having that tends to encourage scheduler research. The main cautions I have are: (1) In introducing flexibility, be careful to avoid unnecessary costs. In particular, the notion of changing the scheduler at run-time is cool, but not if it involves introducing scary new locks. Being able to support introducing flexible new scheduling algorithms at boot time is very appealing, though. (2) Currently the MD code speaks a few specific MI-defined data structures in the scheduling code. In a move towards a more flexible mechanism for specifying scheduling behavior, hammering down the details of what exactly can and cannot be modified at the C level, and what the assembly code is and is not permitted to assume, is probably a useful piece of this work. Robert N M Watson FreeBSD Core Team, TrustedBSD Projects robert@fledge.watson.org Network Associates Laboratories To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Wed Jul 17 10:36:44 2002 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.FreeBSD.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 22BC537B400 for ; Wed, 17 Jul 2002 10:36:39 -0700 (PDT) Received: from mail.speakeasy.net (mail16.speakeasy.net [216.254.0.216]) by mx1.FreeBSD.org (Postfix) with ESMTP id AD85C43E3B for ; Wed, 17 Jul 2002 10:36:38 -0700 (PDT) (envelope-from jhb@FreeBSD.org) Received: (qmail 19441 invoked from network); 17 Jul 2002 17:36:37 -0000 Received: from unknown (HELO server.baldwin.cx) ([216.27.160.63]) (envelope-sender ) by mail16.speakeasy.net (qmail-ldap-1.03) with DES-CBC3-SHA encrypted SMTP for ; 17 Jul 2002 17:36:37 -0000 Received: from laptop.baldwin.cx (gw1.twc.weather.com [216.133.140.1]) by server.baldwin.cx (8.11.6/8.11.6) with ESMTP id g6HHaZ057524; Wed, 17 Jul 2002 13:36:35 -0400 (EDT) (envelope-from jhb@FreeBSD.org) Message-ID: X-Mailer: XFMail 1.5.2 on FreeBSD X-Priority: 3 (Normal) Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 8bit MIME-Version: 1.0 In-Reply-To: <20020716235216.B6785@iguana.icir.org> Date: Wed, 17 Jul 2002 13:36:40 -0400 (EDT) From: John Baldwin To: Luigi Rizzo Subject: RE: proposed changes to kern_switch.c and kern_synch.c Cc: arch@freebsd.org Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG On 17-Jul-2002 Luigi Rizzo wrote: > Hi, > we have implemented a weight-based process scheduler > for FreeBSD-stable, and are trying to port it to -current (in the > description below replace "process" with "thread/kse" as appropriate > for the -current case). Unfortunately, the differences between -current and -stable in this area aren't that simple. I think while developing on this on -stable is fine, trying to retrofit those changes into -current will be fairly hard. Mostly because you basically need to rewrite a lot of it. The current scheduler is still very much a work in progress and I think we need to not try to do too many things at once. Some remaining issues that I think need to be done first before we look into this: - Finish up KSE changes - Really make the scheduler fully preemptive - Get rid of schedcpu() in favor of event-driven priority updates somewhat similar to SVR4 as described in Unix Internals - A few other simplifications would be to have things like turnstiles and generic sleep queues shared between at least cv's and sleep/wakeup, possibly with turnstiles as well. Some other notes: I think kern_switch.c should just contain the code to support processor runqueues and possibly mi_switch(). > 3. implement a function which, under control of a sysctl call, > activate a new scheduler (by initialising all the function > pointers to appropriate values) and moves all processes from > the old to the new one. In SVR4 different processes could be in different scheduling classes that could cooperate with each other. I think that is a better long-term design goal then requiring switch-overs. -- John Baldwin <>< http://www.FreeBSD.org/~jhb/ "Power Users Use the Power to Serve!" - http://www.FreeBSD.org/ To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Wed Jul 17 11:26:46 2002 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.FreeBSD.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 6157A37B400 for ; Wed, 17 Jul 2002 11:26:44 -0700 (PDT) Received: from swan.mail.pas.earthlink.net (swan.mail.pas.earthlink.net [207.217.120.123]) by mx1.FreeBSD.org (Postfix) with ESMTP id 08CDB43E31 for ; Wed, 17 Jul 2002 11:26:44 -0700 (PDT) (envelope-from tlambert2@mindspring.com) Received: from pool0460.cvx22-bradley.dialup.earthlink.net ([209.179.199.205] helo=mindspring.com) by swan.mail.pas.earthlink.net with esmtp (Exim 3.33 #1) id 17UtVZ-0004gw-00; Wed, 17 Jul 2002 11:26:42 -0700 Message-ID: <3D35B6B4.1333B46C@mindspring.com> Date: Wed, 17 Jul 2002 11:25:56 -0700 From: Terry Lambert X-Mailer: Mozilla 4.79 [en] (Win98; U) X-Accept-Language: en MIME-Version: 1.0 To: Lars Eggert Cc: Luigi Rizzo , arch@FreeBSD.ORG Subject: Re: proposed changes to kern_switch.c and kern_synch.c References: <20020716235216.B6785@iguana.icir.org> <3D351B99.311431F@mindspring.com> <20020717004244.B7218@iguana.icir.org> <3D36072C.3080908@isi.edu> <20020717015748.B7719@iguana.icir.org> <3D36312D.1010606@isi.edu> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG Lars Eggert wrote: > > i don't know much on the lottery scheduler, but it looks like it > > is probabilistic as opposed to deterministic. The paper on lottery > > scheduling claims to have the same O(log N) complexity as WF2Q+ > > (though i haven't looked at the implementation issues). > > Ah, so it's more like Waldspurger's later stride scheduler than his > lottery scheduler. Just a "head's up": The thing you are comparing to the stride scheduler is Luigi's first impression of the lottery scheduler, not his weighted scheduler. Luigi's comment was directed to the lottery scheduler you referenced, not to the weighted scheduler. His comparative perspective is based on his own work, not on the work you are referencing. Next time you ask someone to "compare A and B", you may want to be careful to make sure that they are telling you how A differs from B, rather than how B differs from A. ;^). -- Terry To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Thu Jul 18 11: 5:13 2002 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.FreeBSD.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 92FB837B400 for ; Thu, 18 Jul 2002 11:05:07 -0700 (PDT) Received: from whale.sunbay.crimea.ua (whale.sunbay.crimea.ua [212.110.138.65]) by mx1.FreeBSD.org (Postfix) with ESMTP id 0AA3743E31 for ; Thu, 18 Jul 2002 11:05:04 -0700 (PDT) (envelope-from ru@whale.sunbay.crimea.ua) Received: (from ru@localhost) by whale.sunbay.crimea.ua (8.11.6/8.11.2) id g6II50b71736 for arch@FreeBSD.org; Thu, 18 Jul 2002 21:05:00 +0300 (EEST) (envelope-from ru) Date: Thu, 18 Jul 2002 21:05:00 +0300 From: Ruslan Ermilov To: arch@FreeBSD.org Subject: [POLL] need a good name for share/mk API versioning Message-ID: <20020718180500.GC63636@sunbay.com> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="eRtJSFbw+EEWtPj3" Content-Disposition: inline User-Agent: Mutt/1.3.99i Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG --eRtJSFbw+EEWtPj3 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Hi! Recently, some backwards incompatible changes were purposedly introduced to the share/mk API, particularly to bsd.lib.mk and bsd.incs.mk (include files). Everything is OK with src/, but some ports/ that rely on FreeBSD's share/mk API should be able to use both old (currently in -STABLE, soon to be upgraded to the new API) and new versions of the API, so we need to somehow differentiate these APIs. The solution is to add the FreeBSD specific make(1) variable that could be used to differentiate different API versions. So I would like to run a short pool as to what the name of this variable should be (the draft is _FREEBSD_MK_API_VERSION), and what should be its numbering scheme. I would like this numbering scheme to be a monotonically increasing natural number. This would be good because the same numbering scheme would be used in both -STABLE and -CURRENT, and would be bad because specific backwards incompatible API changes should be merged in the same order as they were applied to -CURRENT. The variable must be put into sys.mk (so that it's always defined), and one option it to use an already existing =2EFreeBSD variable. Your bet please. I will wait until Monday for poll to settle. Thanks, --=20 Ruslan Ermilov Sysadmin and DBA, ru@sunbay.com Sunbay Software AG, ru@FreeBSD.org FreeBSD committer, +380.652.512.251 Simferopol, Ukraine http://www.FreeBSD.org The Power To Serve http://www.oracle.com Enabling The Information Age --eRtJSFbw+EEWtPj3 Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.0.7 (FreeBSD) iD8DBQE9NwNMUkv4P6juNwoRAkNRAJ4wO5eDpG9vz9RhOZDXMB8Lxdh+nQCfZQT7 fpCwLeeaEmn5b7tZ4C++33k= =rFKZ -----END PGP SIGNATURE----- --eRtJSFbw+EEWtPj3-- To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Thu Jul 18 12:44: 4 2002 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.FreeBSD.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 617F037B400; Thu, 18 Jul 2002 12:44:01 -0700 (PDT) Received: from thuvia.demon.co.uk (thuvia.demon.co.uk [193.237.34.248]) by mx1.FreeBSD.org (Postfix) with ESMTP id 5463543E3B; Thu, 18 Jul 2002 12:43:59 -0700 (PDT) (envelope-from mark@thuvia.demon.co.uk) Received: from dotar.thuvia.org (dotar.thuvia.org [10.0.0.4]) by phaidor.thuvia.org (8.11.6/8.11.6) with ESMTP id g6IJhvb67564; Thu, 18 Jul 2002 20:43:57 +0100 (BST) (envelope-from mark@thuvia.demon.co.uk) Received: from dotar.thuvia.org (localhost [IPv6:::1]) by dotar.thuvia.org (8.12.3/8.12.3) with ESMTP id g6IJhuKC016232; Thu, 18 Jul 2002 20:43:56 +0100 (BST) (envelope-from mark@dotar.thuvia.org) Received: (from mark@localhost) by dotar.thuvia.org (8.12.3/8.12.3/Submit) id g6IJhu5A016231; Thu, 18 Jul 2002 20:43:56 +0100 (BST) Date: Thu, 18 Jul 2002 20:43:56 +0100 (BST) From: Mark Valentine Message-Id: <200207181943.g6IJhu5A016231@dotar.thuvia.org> In-Reply-To: X-Disclaimer: tequila X-Mailer: Mail User's Shell (7.2.6 beta(5) 10/07/98) To: ru@freebsd.org (Ruslan Ermilov), arch@freebsd.org Subject: Re: [POLL] need a good name for share/mk API versioning Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG > From: ru@freebsd.org (Ruslan Ermilov) > Date: Thu 18 Jul, 2002 > Subject: [POLL] need a good name for share/mk API versioning > Recently, some backwards incompatible changes were purposedly > introduced to the share/mk API, particularly to bsd.lib.mk and > bsd.incs.mk (include files). Everything is OK with src/, but > some ports/ that rely on FreeBSD's share/mk API should be able > to use both old (currently in -STABLE, soon to be upgraded to > the new API) and new versions of the API, so we need to somehow > differentiate these APIs. The solution is to add the FreeBSD > specific make(1) variable that could be used to differentiate > different API versions. If this were only for ports, the existing OSVERSION in bsd.port.mk would suffice, no? > So I would like to run a short pool as to what the name of this > variable should be (the draft is _FREEBSD_MK_API_VERSION), and > what should be its numbering scheme. I would like this numbering > scheme to be a monotonically increasing natural number. This > would be good because the same numbering scheme would be used in > both -STABLE and -CURRENT, and would be bad because specific > backwards incompatible API changes should be merged in the same > order as they were applied to -CURRENT. It sounds like you do want to have a version number visible beyond ports, which is quite reasonable (third party software may choose to use the "standard" bsd.*.mk rules, without using the ports infrastructure). > The variable must be put into sys.mk (so that it's always > defined), and one option it to use an already existing > FreeBSD variable. sys.mk is the wrong place for things specific to bsd.*.mk (all of the includes at the end of sys.mk are bugs). sys.mk is for make(1) defaults; bsd.*.mk comprises the FreeBSD build system built _on top of_ make(1), which (should) know nothing about bsd.*.mk - currently make(1) pollutes all other build systems with stuff from make.conf and other FreeBSD build system makefiles. bsd.sys.mk is a better place; _FREEBSD_MK_VERSION is a better name; I don't see any need for anything more complex than values of "1", "2", "3"... Cheers, Mark. -- Mark Valentine, Thuvia Labs "Tigers will do ANYTHING for a tuna fish sandwich." Mark Valentine uses "We're kind of stupid that way." *munch* *munch* and endorses FreeBSD -- To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Thu Jul 18 16:32:41 2002 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.FreeBSD.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 0AF6437B400; Thu, 18 Jul 2002 16:31:58 -0700 (PDT) Received: from iguana.icir.org (iguana.icir.org [192.150.187.36]) by mx1.FreeBSD.org (Postfix) with ESMTP id 66BA643E67; Thu, 18 Jul 2002 16:31:57 -0700 (PDT) (envelope-from rizzo@iguana.icir.org) Received: (from rizzo@localhost) by iguana.icir.org (8.11.6/8.11.3) id g6INVvf27426; Thu, 18 Jul 2002 16:31:57 -0700 (PDT) (envelope-from rizzo) Date: Thu, 18 Jul 2002 16:31:57 -0700 From: Luigi Rizzo To: arch@FreeBSD.ORG Subject: NEW SCHEDULER available (was Re: proposed changes to kern_switch.c and kern_synch.c) Message-ID: <20020718163157.A27017@iguana.icir.org> References: <20020716235216.B6785@iguana.icir.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2.5.1i In-Reply-To: <20020716235216.B6785@iguana.icir.org>; from rizzo@icir.org on Tue, Jul 16, 2002 at 11:52:16PM -0700 Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG [crossposted to -stable as relevant there] Hi, as promised, a first version of the Proportional Share scheduler that we developed is available at http://info.iet.unipi.it/~luigi/ps_sched.20020719a.diff These are for a recent -STABLE (i think any version from 4.4 should work; the only 3 files modified are kern_synch.c, kern_switch.c and proc.h, plus a one-line change to kern_exit.c). I have tested it a little bit on a diskless system, and it seems to survive running a full X session with the usual set of xterm, netscape etc. while i do a "renice" of the processes and even switch back and forth between schedulers. But do not trust this yet for a production system! The sysctl variable kern.scheduler controls the scheduler in use. kern.scheduler=1 (default) selects Proportional Share kern.scheduler=0 selects the Feedback Priority ("classic BSD") You can switch between the two schedulers by changing the value of the sysctl variable. To change the "weight" associated to a process, use the "nice" and "renice" commands. As usual, positive values (+1..+20) mean the process will get a smaller share of the CPU, negative values (-1..-20) will get a larger share. The actual weights (which control the relative fraction of CPU cycles given to each process) are assigned through a lookup table which maps the value to the range 1 ... 100 ... 1000 (min, normal, max weight). The "old" scheduler (Feedback Priority) should be as robust and stable as always, whereas there are still a few things to cleanup in the Proportional Share scheduler, namely: * I have not looked in detail at the SMP case, so do not run it on an SMP kernel; * given that there are no priorities, processes woken up by a tsleep() are not guaranteed to run before all processes with p_priority >= PUSER; however, they are given a shorter deadline so they are likely to run first. * RTPRI and IDLEPRI processes do not have yet any special treatment (they all end up together with normal processes; this is trivial to fix, i just haven't had time to look at that). Have fun, and please if you use it let me know of any bugs and possibly suggestions to adapt it to -current. cheers luigi On Tue, Jul 16, 2002 at 11:52:16PM -0700, Luigi Rizzo wrote: > Hi, > we have implemented a weight-based process scheduler > for FreeBSD-stable, and are trying to port it to -current (in the > description below replace "process" with "thread/kse" as appropriate > for the -current case). > > In order to make this work, it is convenient to have all > scheduler-specific functions and data structures in a > single file (kern_switch*.c), and generic support in > another one (kern_synch.c). I believe this was also the original > BSD design in partitioning the code between the two files. > However, in our current code, there are some functions which > are scheduler-specific (see below) which are misplaced. > > So I would like to make the following, purely cosmetic, change > identified as step #1 below, both for consistency with what i > believe to be the original design, and to simplify further work > in this area. These would go to both -current and -stable; I > repeat, they are only cosmetic changes. > > Comments ? I already spoke to julian who has no objections. > > For those interested, a patch for -current is attached, and the one > for -stable is very similar (for the records, in -stable we have a > fully functional weight-based process scheduler, where you still > control the weights using "nice", and you can switch between the > old and the new scheduler at any time using a sysctl variable). > > --- Re. the multi-scheduler architecture --- > > The general idea is to make the process/thread/kse scheduler > a replaceable piece of the kernel, requiring no modifications > to the "struct proc", and with the ability of switching from > one scheduler to another one at runtime (this both for testing > purposes and for whatever need may arise). > > > The way to achieve this would be the following: > > 1. identify all scheduler-specific functions, put all of them > in one file (e.g. kern_switch.c for the default scheduler), > and define function pointers for all of them; > > 2. use one field in "struct proc" as a link to whatever > scheduler-specific data structures are necessary. > In -stable, we have the p_pad3 field that can be used > for this purpose, much like the if_index field in struct ifnet. > > 3. implement a function which, under control of a sysctl call, > activate a new scheduler (by initialising all the function > pointers to appropriate values) and moves all processes from > the old to the new one. > > Step #1 is basically a cosmetic change, requiring mostly a move of > some functions from kern_synch.c to kern_switch.c. These are > > roundrobin(); > > curpriority_cmp(); > > resetpriority(); > > parts of schedcpu() and schedclock(); > > cheers > luigi > ---------------------------------- > > Index: kern_switch.c > =================================================================== > RCS file: /home/ncvs/src/sys/kern/kern_switch.c,v > retrieving revision 1.33 > diff -u -r1.33 kern_switch.c > --- kern_switch.c 14 Jul 2002 03:43:33 -0000 1.33 > +++ kern_switch.c 16 Jul 2002 22:15:06 -0000 > @@ -97,6 +97,7 @@ > #include > #include > #include > +#include > #include > > CTASSERT((RQB_BPW * RQB_LEN) == RQ_NQS); > @@ -107,6 +108,9 @@ > static struct runq runq; > SYSINIT(runq, SI_SUB_RUN_QUEUE, SI_ORDER_FIRST, runq_init, &runq) > > +static struct callout roundrobin_callout; > + > +static void roundrobin(void *arg); > static void runq_readjust(struct runq *rq, struct kse *ke); > /************************************************************************ > * Functions that manipulate runnability from a thread perspective. * > @@ -442,9 +446,13 @@ > { > int i; > > + callout_init(&roundrobin_callout, 0); > + > bzero(rq, sizeof *rq); > for (i = 0; i < RQ_NQS; i++) > TAILQ_INIT(&rq->rq_queues[i]); > + > + roundrobin(NULL); > } > > /* > @@ -719,3 +727,207 @@ > } > #endif > > +/* > + * -- formerly in kern_synch.c > + */ > + > +int curpriority_cmp(struct thread *td); > + > +int > +curpriority_cmp(struct thread *td) > +{ > + return (td->td_priority - curthread->td_priority); > +} > + > +/* > + * Force switch among equal priority processes every 100ms. > + * We don't actually need to force a context switch of the current process. > + * The act of firing the event triggers a context switch to softclock() and > + * then switching back out again which is equivalent to a preemption, thus > + * no further work is needed on the local CPU. > + */ > +/* ARGSUSED */ > +static void > +roundrobin(arg) > + void *arg; > +{ > + > +#ifdef SMP > + mtx_lock_spin(&sched_lock); > + forward_roundrobin(); > + mtx_unlock_spin(&sched_lock); > +#endif > + > + callout_reset(&roundrobin_callout, sched_quantum, roundrobin, NULL); > +} > + > +/* calculations for digital decay to forget 90% of usage in 5*loadav sec */ > +#define loadfactor(loadav) (2 * (loadav)) > +#define decay_cpu(loadfac, cpu) (((loadfac) * (cpu)) / ((loadfac) + FSCALE)) > +extern fixpt_t ccpu; > +#define CCPU_SHIFT 11 /* XXX dup from kern_synch.c! */ > + > +/* > + * Recompute process priorities, every hz ticks. > + * MP-safe, called without the Giant mutex. > + */ > +void schedcpu1(struct ksegrp *kg); > +/* ARGSUSED */ > +void > +schedcpu1(struct ksegrp *kg) > +{ > + register fixpt_t loadfac = loadfactor(averunnable.ldavg[0]); > + struct thread *td; > + struct kse *ke; > + int realstathz; > + int awake; > + > + realstathz = stathz ? stathz : hz; > + > + awake = 0; > + FOREACH_KSE_IN_GROUP(kg, ke) { > + /* > + * Increment time in/out of memory and sleep > + * time (if sleeping). We ignore overflow; > + * with 16-bit int's (remember them?) > + * overflow takes 45 days. > + */ > + /* XXXKSE **WRONG***/ > + /* > + * the kse slptimes are not touched in wakeup > + * because the thread may not HAVE a KSE > + */ > + if ((ke->ke_state == KES_ONRUNQ) || > + ((ke->ke_state == KES_THREAD) && > + (ke->ke_thread->td_state == TDS_RUNNING))) { > + ke->ke_slptime++; > + } else { > + ke->ke_slptime = 0; > + awake = 1; > + } > + > + /* > + * pctcpu is only for ps? > + * Do it per kse.. and add them up at the end? > + * XXXKSE > + */ > + ke->ke_pctcpu = (ke->ke_pctcpu * ccpu) >> FSHIFT; > + /* > + * If the kse has been idle the entire second, > + * stop recalculating its priority until > + * it wakes up. > + */ > + if (ke->ke_slptime > 1) { > + continue; > + } > + > +#if (FSHIFT >= CCPU_SHIFT) > + ke->ke_pctcpu += (realstathz == 100) ? > + ((fixpt_t) ke->ke_cpticks) << > + (FSHIFT - CCPU_SHIFT) : > + 100 * (((fixpt_t) ke->ke_cpticks) << > + (FSHIFT - CCPU_SHIFT)) / realstathz; > +#else > + ke->ke_pctcpu += ((FSCALE - ccpu) * > + (ke->ke_cpticks * FSCALE / realstathz)) >> > + FSHIFT; > +#endif > + ke->ke_cpticks = 0; > + } /* end of kse loop */ > + if (awake == 0) { > + kg->kg_slptime++; > + } else { > + kg->kg_slptime = 0; > + } > + kg->kg_estcpu = decay_cpu(loadfac, kg->kg_estcpu); > + resetpriority(kg); > + FOREACH_THREAD_IN_GROUP(kg, td) { > + int changedqueue; > + if (td->td_priority >= PUSER) { > + /* > + * Only change the priority > + * of threads that are still at their > + * user priority. > + * XXXKSE This is problematic > + * as we may need to re-order > + * the threads on the KSEG list. > + */ > + changedqueue = > + ((td->td_priority / RQ_PPQ) != > + (kg->kg_user_pri / RQ_PPQ)); > + > + td->td_priority = kg->kg_user_pri; > + if (changedqueue && > + td->td_state == TDS_RUNQ) { > + /* this could be optimised */ > + remrunqueue(td); > + td->td_priority = > + kg->kg_user_pri; > + setrunqueue(td); > + } else { > + td->td_priority = kg->kg_user_pri; > + } > + } > + } > +} > + > +/* > + * Compute the priority of a process when running in user mode. > + * Arrange to reschedule if the resulting priority is better > + * than that of the current process. > + */ > +void > +resetpriority(kg) > + register struct ksegrp *kg; > +{ > + register unsigned int newpriority; > + struct thread *td; > + > + mtx_lock_spin(&sched_lock); > + if (kg->kg_pri_class == PRI_TIMESHARE) { > + newpriority = PUSER + kg->kg_estcpu / INVERSE_ESTCPU_WEIGHT + > + NICE_WEIGHT * (kg->kg_nice - PRIO_MIN); > + newpriority = min(max(newpriority, PRI_MIN_TIMESHARE), > + PRI_MAX_TIMESHARE); > + kg->kg_user_pri = newpriority; > + } > + FOREACH_THREAD_IN_GROUP(kg, td) { > + maybe_resched(td); /* XXXKSE silly */ > + } > + mtx_unlock_spin(&sched_lock); > +} > + > +#if 0 > +/* > + * We adjust the priority of the current process. The priority of > + * a process gets worse as it accumulates CPU time. The cpu usage > + * estimator (p_estcpu) is increased here. resetpriority() will > + * compute a different priority each time p_estcpu increases by > + * INVERSE_ESTCPU_WEIGHT > + * (until MAXPRI is reached). The cpu usage estimator ramps up > + * quite quickly when the process is running (linearly), and decays > + * away exponentially, at a rate which is proportionally slower when > + * the system is busy. The basic principle is that the system will > + * 90% forget that the process used a lot of CPU time in 5 * loadav > + * seconds. This causes the system to favor processes which haven't > + * run much recently, and to round-robin among other processes. > + */ > +void > +schedclock(td) > + struct thread *td; > +{ > + struct kse *ke; > + struct ksegrp *kg; > + > + KASSERT((td != NULL), ("schedlock: null thread pointer")); > + ke = td->td_kse; > + kg = td->td_ksegrp; > + ke->ke_cpticks++; > + kg->kg_estcpu = ESTCPULIM(kg->kg_estcpu + 1); > + if ((kg->kg_estcpu % INVERSE_ESTCPU_WEIGHT) == 0) { > + resetpriority(kg); > + if (td->td_priority >= PUSER) > + td->td_priority = kg->kg_user_pri; > + } > +} > +#endif > Index: kern_synch.c > =================================================================== > RCS file: /home/ncvs/src/sys/kern/kern_synch.c,v > retrieving revision 1.185 > diff -u -r1.185 kern_synch.c > --- kern_synch.c 14 Jul 2002 03:43:33 -0000 1.185 > +++ kern_synch.c 16 Jul 2002 22:16:46 -0000 > @@ -67,6 +67,8 @@ > > #include > > +void schedcpu1(struct ksegrp *kg); /* XXX in kern_switch.c */ > + > static void sched_setup(void *dummy); > SYSINIT(sched_setup, SI_SUB_KICK_SCHEDULER, SI_ORDER_FIRST, sched_setup, NULL) > > @@ -76,7 +78,6 @@ > > static struct callout loadav_callout; > static struct callout schedcpu_callout; > -static struct callout roundrobin_callout; > > struct loadavg averunnable = > { {0, 0, 0}, FSCALE }; /* load average, of runnable procs */ > @@ -92,7 +93,6 @@ > > static void endtsleep(void *); > static void loadav(void *arg); > -static void roundrobin(void *arg); > static void schedcpu(void *arg); > > static int > @@ -135,28 +135,6 @@ > } > > /* > - * Force switch among equal priority processes every 100ms. > - * We don't actually need to force a context switch of the current process. > - * The act of firing the event triggers a context switch to softclock() and > - * then switching back out again which is equivalent to a preemption, thus > - * no further work is needed on the local CPU. > - */ > -/* ARGSUSED */ > -static void > -roundrobin(arg) > - void *arg; > -{ > - > -#ifdef SMP > - mtx_lock_spin(&sched_lock); > - forward_roundrobin(); > - mtx_unlock_spin(&sched_lock); > -#endif > - > - callout_reset(&roundrobin_callout, sched_quantum, roundrobin, NULL); > -} > - > -/* > * Constants for digital decay and forget: > * 90% of (p_estcpu) usage in 5 * loadav time > * 95% of (p_pctcpu) usage in 60 seconds (load insensitive) > @@ -225,7 +203,7 @@ > #define decay_cpu(loadfac, cpu) (((loadfac) * (cpu)) / ((loadfac) + FSCALE)) > > /* decay 95% of `p_pctcpu' in 60 seconds; see CCPU_SHIFT before changing */ > -static fixpt_t ccpu = 0.95122942450071400909 * FSCALE; /* exp(-1/20) */ > +fixpt_t ccpu = 0.95122942450071400909 * FSCALE; /* exp(-1/20) */ > SYSCTL_INT(_kern, OID_AUTO, ccpu, CTLFLAG_RD, &ccpu, 0, ""); > > /* kernel uses `FSCALE', userland (SHOULD) use kern.fscale */ > @@ -255,105 +233,15 @@ > schedcpu(arg) > void *arg; > { > - register fixpt_t loadfac = loadfactor(averunnable.ldavg[0]); > - struct thread *td; > struct proc *p; > - struct kse *ke; > struct ksegrp *kg; > - int realstathz; > - int awake; > > - realstathz = stathz ? stathz : hz; > sx_slock(&allproc_lock); > FOREACH_PROC_IN_SYSTEM(p) { > mtx_lock_spin(&sched_lock); > p->p_swtime++; > FOREACH_KSEGRP_IN_PROC(p, kg) { > - awake = 0; > - FOREACH_KSE_IN_GROUP(kg, ke) { > - /* > - * Increment time in/out of memory and sleep > - * time (if sleeping). We ignore overflow; > - * with 16-bit int's (remember them?) > - * overflow takes 45 days. > - */ > - /* XXXKSE **WRONG***/ > - /* > - * the kse slptimes are not touched in wakeup > - * because the thread may not HAVE a KSE > - */ > - if ((ke->ke_state == KES_ONRUNQ) || > - ((ke->ke_state == KES_THREAD) && > - (ke->ke_thread->td_state == TDS_RUNNING))) { > - ke->ke_slptime++; > - } else { > - ke->ke_slptime = 0; > - awake = 1; > - } > - > - /* > - * pctcpu is only for ps? > - * Do it per kse.. and add them up at the end? > - * XXXKSE > - */ > - ke->ke_pctcpu = (ke->ke_pctcpu * ccpu) >> FSHIFT; > - /* > - * If the kse has been idle the entire second, > - * stop recalculating its priority until > - * it wakes up. > - */ > - if (ke->ke_slptime > 1) { > - continue; > - } > - > -#if (FSHIFT >= CCPU_SHIFT) > - ke->ke_pctcpu += (realstathz == 100) ? > - ((fixpt_t) ke->ke_cpticks) << > - (FSHIFT - CCPU_SHIFT) : > - 100 * (((fixpt_t) ke->ke_cpticks) << > - (FSHIFT - CCPU_SHIFT)) / realstathz; > -#else > - ke->ke_pctcpu += ((FSCALE - ccpu) * > - (ke->ke_cpticks * FSCALE / realstathz)) >> > - FSHIFT; > -#endif > - ke->ke_cpticks = 0; > - } /* end of kse loop */ > - if (awake == 0) { > - kg->kg_slptime++; > - } else { > - kg->kg_slptime = 0; > - } > - kg->kg_estcpu = decay_cpu(loadfac, kg->kg_estcpu); > - resetpriority(kg); > - FOREACH_THREAD_IN_GROUP(kg, td) { > - int changedqueue; > - if (td->td_priority >= PUSER) { > - /* > - * Only change the priority > - * of threads that are still at their > - * user priority. > - * XXXKSE This is problematic > - * as we may need to re-order > - * the threads on the KSEG list. > - */ > - changedqueue = > - ((td->td_priority / RQ_PPQ) != > - (kg->kg_user_pri / RQ_PPQ)); > - > - td->td_priority = kg->kg_user_pri; > - if (changedqueue && > - td->td_state == TDS_RUNQ) { > - /* this could be optimised */ > - remrunqueue(td); > - td->td_priority = > - kg->kg_user_pri; > - setrunqueue(td); > - } else { > - td->td_priority = kg->kg_user_pri; > - } > - } > - } > + schedcpu1(kg); > } /* end of ksegrp loop */ > mtx_unlock_spin(&sched_lock); > } /* end of process loop */ > @@ -944,32 +832,6 @@ > } > > /* > - * Compute the priority of a process when running in user mode. > - * Arrange to reschedule if the resulting priority is better > - * than that of the current process. > - */ > -void > -resetpriority(kg) > - register struct ksegrp *kg; > -{ > - register unsigned int newpriority; > - struct thread *td; > - > - mtx_lock_spin(&sched_lock); > - if (kg->kg_pri_class == PRI_TIMESHARE) { > - newpriority = PUSER + kg->kg_estcpu / INVERSE_ESTCPU_WEIGHT + > - NICE_WEIGHT * (kg->kg_nice - PRIO_MIN); > - newpriority = min(max(newpriority, PRI_MIN_TIMESHARE), > - PRI_MAX_TIMESHARE); > - kg->kg_user_pri = newpriority; > - } > - FOREACH_THREAD_IN_GROUP(kg, td) { > - maybe_resched(td); /* XXXKSE silly */ > - } > - mtx_unlock_spin(&sched_lock); > -} > - > -/* > * Compute a tenex style load average of a quantity on > * 1, 5 and 15 minute intervals. > * XXXKSE Needs complete rewrite when correct info is available. > @@ -1022,11 +884,9 @@ > { > > callout_init(&schedcpu_callout, 1); > - callout_init(&roundrobin_callout, 0); > callout_init(&loadav_callout, 0); > > /* Kick off timeout driven events by calling first time. */ > - roundrobin(NULL); > schedcpu(NULL); > loadav(NULL); > } > > ----- End forwarded message ----- > > To Unsubscribe: send mail to majordomo@FreeBSD.org > with "unsubscribe freebsd-arch" in the body of the message To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Fri Jul 19 0:45:32 2002 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.FreeBSD.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 8B27037B400 for ; Fri, 19 Jul 2002 00:45:27 -0700 (PDT) Received: from whale.sunbay.crimea.ua (whale.sunbay.crimea.ua [212.110.138.65]) by mx1.FreeBSD.org (Postfix) with ESMTP id A0C4043E65 for ; Fri, 19 Jul 2002 00:45:23 -0700 (PDT) (envelope-from ru@whale.sunbay.crimea.ua) Received: (from ru@localhost) by whale.sunbay.crimea.ua (8.11.6/8.11.2) id g6J7j4v41320; Fri, 19 Jul 2002 10:45:04 +0300 (EEST) (envelope-from ru) Date: Fri, 19 Jul 2002 10:45:04 +0300 From: Ruslan Ermilov To: Mark Valentine Cc: arch@freebsd.org Subject: Re: [POLL] need a good name for share/mk API versioning Message-ID: <20020719074504.GC39934@sunbay.com> References: <200207181943.g6IJhu5A016231@dotar.thuvia.org> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="VywGB/WGlW4DM4P8" Content-Disposition: inline In-Reply-To: <200207181943.g6IJhu5A016231@dotar.thuvia.org> User-Agent: Mutt/1.3.99i Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG --VywGB/WGlW4DM4P8 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Thu, Jul 18, 2002 at 08:43:56PM +0100, Mark Valentine wrote: > > From: ru@freebsd.org (Ruslan Ermilov) > > Date: Thu 18 Jul, 2002 > > Subject: [POLL] need a good name for share/mk API versioning >=20 > > Recently, some backwards incompatible changes were purposedly > > introduced to the share/mk API, particularly to bsd.lib.mk and > > bsd.incs.mk (include files). Everything is OK with src/, but > > some ports/ that rely on FreeBSD's share/mk API should be able > > to use both old (currently in -STABLE, soon to be upgraded to > > the new API) and new versions of the API, so we need to somehow > > differentiate these APIs. The solution is to add the FreeBSD > > specific make(1) variable that could be used to differentiate > > different API versions. >=20 > If this were only for ports, the existing OSVERSION in bsd.port.mk > would suffice, no? >=20 No, this is for third party software (well, some ports) that use bsd.lib.mk. > > So I would like to run a short pool as to what the name of this > > variable should be (the draft is _FREEBSD_MK_API_VERSION), and > > what should be its numbering scheme. I would like this numbering > > scheme to be a monotonically increasing natural number. This > > would be good because the same numbering scheme would be used in > > both -STABLE and -CURRENT, and would be bad because specific > > backwards incompatible API changes should be merged in the same > > order as they were applied to -CURRENT. >=20 > It sounds like you do want to have a version number visible beyond ports, > which is quite reasonable (third party software may choose to use the > "standard" bsd.*.mk rules, without using the ports infrastructure). >=20 Yes. > > The variable must be put into sys.mk (so that it's always > > defined), and one option it to use an already existing > > FreeBSD variable. >=20 > sys.mk is the wrong place for things specific to bsd.*.mk (all of the > includes at the end of sys.mk are bugs). sys.mk is for make(1) defaults; > bsd.*.mk comprises the FreeBSD build system built _on top of_ make(1), wh= ich > (should) know nothing about bsd.*.mk - currently make(1) pollutes all oth= er > build systems with stuff from make.conf and other FreeBSD build system > makefiles. >=20 > bsd.sys.mk is a better place; _FREEBSD_MK_VERSION is a better name; I don= 't > see any need for anything more complex than values of "1", "2", "3"... >=20 Yes, sys.mk is not the best place, but technically it's the only place sufficient for what I need, and I need some variable defined before we include bsd.lib.mk; this is only possible with sys.mk. I think a working example would be handy: %%% Index: src.wnb.Makefile =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D RCS file: /home/ncvs/ports/textproc/wordnet/files/src.wnb.Makefile,v retrieving revision 1.3 diff -u -r1.3 src.wnb.Makefile --- src.wnb.Makefile 31 Oct 2001 05:04:29 -0000 1.3 +++ src.wnb.Makefile 18 Jul 2002 17:47:12 -0000 @@ -10,12 +10,15 @@ =20 LDADD=3D -L../lib -lwn1 -L${PREFIX}/lib -ltcl${TCL_VER} -ltk${TCL_VER} =20 +.if defined(_CURRENT_) +SHLIB_NAME=3D libtclwn1.so.7 +.else LIB=3D tclwn1 SHLIB_MAJOR=3D 7 SHLIB_MINOR=3D 0 -SRCS=3D stubs.c - INTERNALLIB=3D True # To avoid building the useless static library +.endif +SRCS=3D stubs.c =20 all: ${SHLIB_NAME} pkgIndex.tcl =20 %%% Cheers, --=20 Ruslan Ermilov Sysadmin and DBA, ru@sunbay.com Sunbay Software AG, ru@FreeBSD.org FreeBSD committer, +380.652.512.251 Simferopol, Ukraine http://www.FreeBSD.org The Power To Serve http://www.oracle.com Enabling The Information Age --VywGB/WGlW4DM4P8 Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.0.7 (FreeBSD) iD8DBQE9N8OAUkv4P6juNwoRAogQAJ9tVlN/QbW2AhqaQ50KUPU1ajA7VACggIVk xwJnodJkHEr3I8Hjlcd2Nb4= =O5u4 -----END PGP SIGNATURE----- --VywGB/WGlW4DM4P8-- To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Fri Jul 19 4:18:17 2002 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.FreeBSD.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 75A8137B400 for ; Fri, 19 Jul 2002 04:18:14 -0700 (PDT) Received: from mail.nsu.ru (mx.nsu.ru [193.124.215.71]) by mx1.FreeBSD.org (Postfix) with ESMTP id 2133343E5E for ; Fri, 19 Jul 2002 04:18:13 -0700 (PDT) (envelope-from danfe@regency.nsu.ru) Received: from drweb by mail.nsu.ru with drweb-scanned (Exim 3.20 #1) id 17VViw-0004uD-00; Fri, 19 Jul 2002 18:15:02 +0700 Received: from regency.nsu.ru ([193.124.210.26]) by mail.nsu.ru with esmtp (Exim 3.20 #1) id 17VViv-0004ty-00; Fri, 19 Jul 2002 18:15:01 +0700 Received: from regency.nsu.ru (localhost [127.0.0.1]) by regency.nsu.ru (8.12.4/8.12.4) with ESMTP id g6JBEvqJ096490; Fri, 19 Jul 2002 18:14:57 +0700 (NOVST) (envelope-from danfe@regency.nsu.ru) Received: (from danfe@localhost) by regency.nsu.ru (8.12.4/8.12.4/Submit) id g6JBEOXP096435; Fri, 19 Jul 2002 18:14:24 +0700 (NOVST) Date: Fri, 19 Jul 2002 18:14:23 +0700 From: Alexey Dokuchaev To: Terry Lambert Cc: Luigi Rizzo , arch@freebsd.org Subject: Re: proposed changes to kern_switch.c and kern_synch.c Message-ID: <20020719181423.A87152@regency.nsu.ru> References: <20020716235216.B6785@iguana.icir.org> <3D351B99.311431F@mindspring.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2.5.1i In-Reply-To: <3D351B99.311431F@mindspring.com>; from tlambert2@mindspring.com on Wed, Jul 17, 2002 at 12:24:09AM -0700 X-Envelope-To: tlambert2@mindspring.com, rizzo@icir.org, arch@freebsd.org Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG On Wed, Jul 17, 2002 at 12:24:09AM -0700, Terry Lambert wrote: > Luigi Rizzo wrote: > > Hi, > > we have implemented a weight-based process scheduler > > for FreeBSD-stable, and are trying to port it to -current (in the > > description below replace "process" with "thread/kse" as appropriate > > for the -current case). > > Are you going to make this new scheduler generally available? > > What is the weighting? > > Is it weighted fair share? And how does it compare to what we have now? ./danfe To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Fri Jul 19 5: 7:15 2002 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.FreeBSD.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id E5FEE37B401; Fri, 19 Jul 2002 05:06:57 -0700 (PDT) Received: from whale.sunbay.crimea.ua (whale.sunbay.crimea.ua [212.110.138.65]) by mx1.FreeBSD.org (Postfix) with ESMTP id A629943E42; Fri, 19 Jul 2002 05:06:40 -0700 (PDT) (envelope-from ru@whale.sunbay.crimea.ua) Received: (from ru@localhost) by whale.sunbay.crimea.ua (8.11.6/8.11.2) id g6JC5bY83112; Fri, 19 Jul 2002 15:05:37 +0300 (EEST) (envelope-from ru) Date: Fri, 19 Jul 2002 15:05:37 +0300 From: Ruslan Ermilov To: Bruce Evans Cc: arch@FreeBSD.org, current@FreeBSD.org Subject: install -d -C (was: Re: cvs commit: src/share/man/man5 make.conf.5 src/share/examples/etc make.conf) Message-ID: <20020719120537.GA76558@sunbay.com> References: <200207181207.g6IC7nbZ050557@freefall.freebsd.org> <20020719210904.Q12629-100000@gamplex.bde.org> <200207181254.g6ICsubF065254@freefall.freebsd.org> <20020719212215.G12629-100000@gamplex.bde.org> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="MGYHOYXEY6WxJCY8" Content-Disposition: inline In-Reply-To: <20020719210904.Q12629-100000@gamplex.bde.org> <20020719212215.G12629-100000@gamplex.bde.org> User-Agent: Mutt/1.3.99i Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG --MGYHOYXEY6WxJCY8 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Fri, Jul 19, 2002 at 09:28:09PM +1000, Bruce Evans wrote: > On Thu, 18 Jul 2002, Ruslan Ermilov wrote: >=20 > > ru 2002/07/18 05:54:56 PDT > > > > Modified files: > > share/man/man5 make.conf.5 > > share/examples/etc make.conf > > Log: > > To force install(1) to always compare files before installing, one > > now needs to set COPY=3D-C as -C is no longer compatible with the -d > > option. It is also likely to be renamed to INSTALL_COPY soon. > > Update documentation to reflect this change. > > > > PR: bin/40724 >=20 > The bug is that -C is no longer compatible with -d. See also misc/40414. >=20 This PR is already closed. On Fri, Jul 19, 2002 at 09:21:14PM +1000, Bruce Evans wrote: > On Thu, 18 Jul 2002, Ruslan Ermilov wrote: >=20 > > ru 2002/07/18 05:07:49 PDT > > > > Modified files: > > etc Makefile [...] > > usr.sbin/ypserv Makefile > > Log: > > s/${INSTALL} -c/${INSTALL} ${COPY}/ >=20 > Strongly unapproved by: bde. >=20 > This change is to help work around the foot-shooting of making -d > incompatible with -C and -p in install(1)'s flags. It abuses the old > poorly named COPY variable which had become a no-op. Now COPY is still > poorly named but has different semantics. All this is like breaking > cc to reject combinations of flags that don't really go together (e.g., > -I doesn't go with linking) instead of just ignoring the flags that > don't apply to the current operation, and then working around this > foot-shooting by splitting up CFLAGS and changing many Makefiles to > only use the part of CFLAGS that is relevant. >=20 Since its first revision (install.1,v 1.7 and install.c,v 1.16 they were incompatible). Later on, in rev. 1.26, it was made a no-op, just to support "INSTALL=3Dinstall -C" in /etc/make.conf. OpenBSD merged these changes and since then they still have them incompatible. There are two ways to proceed: 1. Rename COPY to INSTALL_COPY (that was my plan), optionally giving it by default an empty value. This shouldn't harm third-party makefiles as -c is now an effective no-op. But this would make us even more compatible with OpenBSD that has: : INSTALL_COPY The old usage of this flag is obsolescent since install(1) : now copies by default. However, it can also be used to : specify that a file not be copied unless it is different : (via the -p option). See install(1) for details. This : is to be used when building our own install script so : that the entire system can either be installed with copie= s, : or copy-if-different using a single knob. [-c] 2. Make again -C an allowed (ignored) option in the -d case. This would make us again incompatible with OpenBSD. I do not have a technical problem doing either, I'd just like to know what do others think about this. Thanks, --=20 Ruslan Ermilov Sysadmin and DBA, ru@sunbay.com Sunbay Software AG, ru@FreeBSD.org FreeBSD committer, +380.652.512.251 Simferopol, Ukraine http://www.FreeBSD.org The Power To Serve http://www.oracle.com Enabling The Information Age --MGYHOYXEY6WxJCY8 Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.0.7 (FreeBSD) iD8DBQE9OACRUkv4P6juNwoRAq/BAJ9rZICy8ikMjCIbTtgmqRRJSp88HgCfQoeK R0J0A3l3FqnSuKH+dHLu5KQ= =XJct -----END PGP SIGNATURE----- --MGYHOYXEY6WxJCY8-- To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Fri Jul 19 5:19: 2 2002 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.FreeBSD.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 3FE0237B400; Fri, 19 Jul 2002 05:18:51 -0700 (PDT) Received: from whale.sunbay.crimea.ua (whale.sunbay.crimea.ua [212.110.138.65]) by mx1.FreeBSD.org (Postfix) with ESMTP id 0403843E31; Fri, 19 Jul 2002 05:18:44 -0700 (PDT) (envelope-from ru@whale.sunbay.crimea.ua) Received: (from ru@localhost) by whale.sunbay.crimea.ua (8.11.6/8.11.2) id g6JCIQv84864; Fri, 19 Jul 2002 15:18:26 +0300 (EEST) (envelope-from ru) Date: Fri, 19 Jul 2002 15:18:26 +0300 From: Ruslan Ermilov To: Bruce Evans Cc: arch@FreeBSD.org, current@FreeBSD.org Subject: Re: install -d -C (was: Re: cvs commit: src/share/man/man5 make.conf.5 src/share/examples/etc make.conf) Message-ID: <20020719121826.GA83942@sunbay.com> References: <200207181207.g6IC7nbZ050557@freefall.freebsd.org> <20020719210904.Q12629-100000@gamplex.bde.org> <200207181254.g6ICsubF065254@freefall.freebsd.org> <20020719212215.G12629-100000@gamplex.bde.org> <20020719120537.GA76558@sunbay.com> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="W/nzBZO5zC0uMSeA" Content-Disposition: inline In-Reply-To: <20020719120537.GA76558@sunbay.com> User-Agent: Mutt/1.3.99i Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG --W/nzBZO5zC0uMSeA Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Fri, Jul 19, 2002 at 03:05:37PM +0300, Ruslan Ermilov wrote: > On Fri, Jul 19, 2002 at 09:28:09PM +1000, Bruce Evans wrote: > > On Thu, 18 Jul 2002, Ruslan Ermilov wrote: > >=20 > > > ru 2002/07/18 05:54:56 PDT > > > > > > Modified files: > > > share/man/man5 make.conf.5 > > > share/examples/etc make.conf > > > Log: > > > To force install(1) to always compare files before installing, one > > > now needs to set COPY=3D-C as -C is no longer compatible with the -d > > > option. It is also likely to be renamed to INSTALL_COPY soon. > > > Update documentation to reflect this change. > > > > > > PR: bin/40724 > >=20 > > The bug is that -C is no longer compatible with -d. See also misc/4041= 4. > >=20 > This PR is already closed. >=20 > On Fri, Jul 19, 2002 at 09:21:14PM +1000, Bruce Evans wrote: > > On Thu, 18 Jul 2002, Ruslan Ermilov wrote: > >=20 > > > ru 2002/07/18 05:07:49 PDT > > > > > > Modified files: > > > etc Makefile > [...] > > > usr.sbin/ypserv Makefile > > > Log: > > > s/${INSTALL} -c/${INSTALL} ${COPY}/ > >=20 > > Strongly unapproved by: bde. > >=20 > > This change is to help work around the foot-shooting of making -d > > incompatible with -C and -p in install(1)'s flags. It abuses the old > > poorly named COPY variable which had become a no-op. Now COPY is still > > poorly named but has different semantics. All this is like breaking > > cc to reject combinations of flags that don't really go together (e.g., > > -I doesn't go with linking) instead of just ignoring the flags that > > don't apply to the current operation, and then working around this > > foot-shooting by splitting up CFLAGS and changing many Makefiles to > > only use the part of CFLAGS that is relevant. > >=20 > Since its first revision (install.1,v 1.7 and install.c,v 1.16 they > were incompatible). Later on, in rev. 1.26, it was made a no-op, > just to support "INSTALL=3Dinstall -C" in /etc/make.conf. >=20 > OpenBSD merged these changes and since then they still have them > incompatible. >=20 > There are two ways to proceed: >=20 > 1. Rename COPY to INSTALL_COPY (that was my plan), optionally giving > it by default an empty value. This shouldn't harm third-party > makefiles as -c is now an effective no-op. But this would make > us even more compatible with OpenBSD that has: >=20 > : INSTALL_COPY The old usage of this flag is obsolescent since install= (1) > : now copies by default. However, it can also be used to > : specify that a file not be copied unless it is different > : (via the -p option). See install(1) for details. This > : is to be used when building our own install script so > : that the entire system can either be installed with cop= ies, > : or copy-if-different using a single knob. [-c] >=20 OTOH, if we go this way we can get rid of ugly ${COPY} completely. > 2. Make again -C an allowed (ignored) option in the -d case. This would > make us again incompatible with OpenBSD. >=20 > I do not have a technical problem doing either, I'd just like to know > what do others think about this. I'm not sure what's the better way, please help me out. :-) Cheers, --=20 Ruslan Ermilov Sysadmin and DBA, ru@sunbay.com Sunbay Software AG, ru@FreeBSD.org FreeBSD committer, +380.652.512.251 Simferopol, Ukraine http://www.FreeBSD.org The Power To Serve http://www.oracle.com Enabling The Information Age --W/nzBZO5zC0uMSeA Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.0.7 (FreeBSD) iD8DBQE9OAOSUkv4P6juNwoRAkd6AJ0R2Q3Rt7jvlFxoekLV0aKf4mk/4wCcDF8o Tg1DGiYPUiQ12B8X4f3SMF0= =D3ME -----END PGP SIGNATURE----- --W/nzBZO5zC0uMSeA-- To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Fri Jul 19 5:52:15 2002 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.FreeBSD.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 79EA237B400; Fri, 19 Jul 2002 05:52:07 -0700 (PDT) Received: from mailman.zeta.org.au (mailman.zeta.org.au [203.26.10.16]) by mx1.FreeBSD.org (Postfix) with ESMTP id 150D643E31; Fri, 19 Jul 2002 05:52:06 -0700 (PDT) (envelope-from bde@zeta.org.au) Received: from bde.zeta.org.au (bde.zeta.org.au [203.2.228.102]) by mailman.zeta.org.au (8.9.3/8.8.7) with ESMTP id WAA12677; Fri, 19 Jul 2002 22:52:03 +1000 Date: Fri, 19 Jul 2002 22:55:56 +1000 (EST) From: Bruce Evans X-X-Sender: bde@gamplex.bde.org To: Ruslan Ermilov Cc: arch@FreeBSD.org, Subject: Re: install -d -C (was: Re: cvs commit: src/share/man/man5 make.conf.5 src/share/examples/etc make.conf) In-Reply-To: <20020719121826.GA83942@sunbay.com> Message-ID: <20020719223359.W12927-100000@gamplex.bde.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG On Fri, 19 Jul 2002, Ruslan Ermilov wrote: > On Fri, Jul 19, 2002 at 03:05:37PM +0300, Ruslan Ermilov wrote: > > ... > > On Fri, Jul 19, 2002 at 09:21:14PM +1000, Bruce Evans wrote: > > > On Thu, 18 Jul 2002, Ruslan Ermilov wrote: > > > > > > > ru 2002/07/18 05:07:49 PDT > > > > > > > > Modified files: > > > > etc Makefile > > [...] > > > > usr.sbin/ypserv Makefile > > > > Log: > > > > s/${INSTALL} -c/${INSTALL} ${COPY}/ > > > > > > Strongly unapproved by: bde. > > > > > > This change is to help work around the foot-shooting of making -d > > > incompatible with -C and -p in install(1)'s flags. It abuses the old > > > ... > > Since its first revision (install.1,v 1.7 and install.c,v 1.16 they > > were incompatible). Later on, in rev. 1.26, it was made a no-op, I think this makes -c vs -d moot. > > just to support "INSTALL=install -C" in /etc/make.conf. -C is not really like -c. It really means "unbreak the default of !-c, and preserve certain metadata". Preserving the metadata is the main point of this option, but IIRC it was made as much like -c as possible just as a first attempt to kill -c. > > OpenBSD merged these changes and since then they still have them > > incompatible. That was probably a mistake. Certainly merging it all back was. -C is our (half my) flag, so we should know its intended use :-). > > There are two ways to proceed: > > > > 1. Rename COPY to INSTALL_COPY (that was my plan), optionally giving > > it by default an empty value. This shouldn't harm third-party > > makefiles as -c is now an effective no-op. But this would make > > us even more compatible with OpenBSD that has: > > > > : INSTALL_COPY The old usage of this flag is obsolescent since install(1) > > : now copies by default. However, it can also be used to > > : specify that a file not be copied unless it is different > > : (via the -p option). See install(1) for details. This > > : is to be used when building our own install script so > > : that the entire system can either be installed with copies, > > : or copy-if-different using a single knob. [-c] > > > OTOH, if we go this way we can get rid of ugly ${COPY} completely. I'd like to get rid of it too. But not in RELENG_4. -c has been the default for long enough now in -current. As you know, there are various problems in using the correctly named variable for install(1)'s flags (INSTALLFLAGS) to actually hold install's flags in a general way (mainly, this variable already exists and is used in a non-general way). However, the old hack of putting the flags in the same variable as the command still works well except for the -[Cp] vs -d conflict. This depends on the flags not being order-dependent. > > 2. Make again -C an allowed (ignored) option in the -d case. This would > > make us again incompatible with OpenBSD. > > > > I do not have a technical problem doing either, I'd just like to know > > what do others think about this. > > I'm not sure what's the better way, please help me out. :-) I've complained to you before about this :-). Now I'm interested in getting other ideas about it. Bruce To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Fri Jul 19 9: 2:19 2002 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.FreeBSD.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id BBD8137B400; Fri, 19 Jul 2002 09:02:16 -0700 (PDT) Received: from harmony.village.org (rover.bsdimp.com [204.144.255.66]) by mx1.FreeBSD.org (Postfix) with ESMTP id 8EE7443E31; Fri, 19 Jul 2002 09:02:15 -0700 (PDT) (envelope-from imp@bsdimp.com) Received: from localhost (warner@rover2.village.org [10.0.0.1]) by harmony.village.org (8.12.3/8.12.3) with ESMTP id g6JG2C1f031551; Fri, 19 Jul 2002 10:02:13 -0600 (MDT) (envelope-from imp@bsdimp.com) Date: Fri, 19 Jul 2002 10:01:51 -0600 (MDT) Message-Id: <20020719.100151.96158427.imp@bsdimp.com> To: mark@thuvia.demon.co.uk Cc: ru@FreeBSD.ORG, arch@FreeBSD.ORG Subject: Re: [POLL] need a good name for share/mk API versioning From: "M. Warner Losh" In-Reply-To: <200207181943.g6IJhu5A016231@dotar.thuvia.org> References: <200207181943.g6IJhu5A016231@dotar.thuvia.org> X-Mailer: Mew version 2.1 on Emacs 21.1 / Mule 5.0 (SAKAKI) Mime-Version: 1.0 Content-Type: Text/Plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG In message: <200207181943.g6IJhu5A016231@dotar.thuvia.org> Mark Valentine writes: : > From: ru@freebsd.org (Ruslan Ermilov) : > Date: Thu 18 Jul, 2002 : > Subject: [POLL] need a good name for share/mk API versioning : : > Recently, some backwards incompatible changes were purposedly : > introduced to the share/mk API, particularly to bsd.lib.mk and : > bsd.incs.mk (include files). Everything is OK with src/, but : > some ports/ that rely on FreeBSD's share/mk API should be able : > to use both old (currently in -STABLE, soon to be upgraded to : > the new API) and new versions of the API, so we need to somehow : > differentiate these APIs. The solution is to add the FreeBSD : > specific make(1) variable that could be used to differentiate : > different API versions. : : If this were only for ports, the existing OSVERSION in bsd.port.mk : would suffice, no? No. The OS version and the .mk files may (and often are) unrelated. This is especially true in the case of cross build environments, which are often used to insulate products from whatever version of FreeBSD they happen to be running on. At Timing solutions, we often build 4.5 based products on a 4.3 machine, and vice versa. Warner To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Sat Jul 20 9:45:41 2002 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.FreeBSD.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id E7A4B37B400; Sat, 20 Jul 2002 09:45:38 -0700 (PDT) Received: from thuvia.demon.co.uk (thuvia.demon.co.uk [193.237.34.248]) by mx1.FreeBSD.org (Postfix) with ESMTP id D84CE43E3B; Sat, 20 Jul 2002 09:45:36 -0700 (PDT) (envelope-from mark@thuvia.demon.co.uk) Received: from dotar.thuvia.org (dotar.thuvia.org [10.0.0.4]) by phaidor.thuvia.org (8.11.6/8.11.6) with ESMTP id g6KGjVb76184; Sat, 20 Jul 2002 17:45:31 +0100 (BST) (envelope-from mark@thuvia.demon.co.uk) Received: from dotar.thuvia.org (localhost [IPv6:::1]) by dotar.thuvia.org (8.12.3/8.12.3) with ESMTP id g6KGjVKC085420; Sat, 20 Jul 2002 17:45:31 +0100 (BST) (envelope-from mark@dotar.thuvia.org) Received: (from mark@localhost) by dotar.thuvia.org (8.12.3/8.12.3/Submit) id g6KGjUvS085419; Sat, 20 Jul 2002 17:45:30 +0100 (BST) Date: Sat, 20 Jul 2002 17:45:30 +0100 (BST) From: Mark Valentine Message-Id: <200207201645.g6KGjUvS085419@dotar.thuvia.org> In-Reply-To: <20020719074504.GC39934@sunbay.com> X-Disclaimer: tequila X-Mailer: Mail User's Shell (7.2.6 beta(5) 10/07/98) To: Ruslan Ermilov Subject: Re: [POLL] need a good name for share/mk API versioning Cc: arch@freebsd.org Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG > From: Ruslan Ermilov > Date: Fri 19 Jul, 2002 > Subject: Re: [POLL] need a good name for share/mk API versioning > Yes, sys.mk is not the best place, but technically it's the only place > sufficient for what I need, and I need some variable defined before we > include bsd.lib.mk; this is only possible with sys.mk. The real problem is a design issue with the FreeBSD build system, in that the Makefile doesn't know which version of the bsd.*.mk files it's going to get before it includes one of them. PARAMETER = some value .include A better design might have been to include build system default parameters seperately: .include .if defined(_WHATEVER_) PARAMETER = some value .else OTHERPARAMETER = other value .endif .include The present design assumes that make(1)'s ``?='' feature allowing defaults to be set as "fallback" values is sufficient, but we now see that this isn't good enough to cope with architectural changes (but supporting multiple build system versions probably wasn't a design goal). However, it's obviously too late to change every Makefile in the world to correct that design issue now. Putting information about bsd.*.mk in sys.mk, however, is also insufficient, since sys.mk doesn't know which versions of bsd.*.mk the makefile will pick up, if any (consider using ``-m'' or ``-I'' options to pick up bsd.*.mk from a directory other than the one containing sys.mk, which should be legitimate). One possible workaround might be to have sys.mk include a file defined by a variable (if defined, perhaps in the environment), which for the FreeBSD build system could be set to "bsd.sys.mk" or similar. This allows any build system to have a component included before the Makefile is read. .if defined(__BUILD_SYS_MK) .include "${__BUILD_SYS_MK}" .endif If really needed for compatibility/POLA reasons, sys.mk could default the variable: __BUILD_SYS_MK?=bsd.sys.mk The user could set the variable to /dev/null to avoid the existing pollution to his own build system. This mechanism would allow the make.conf stuff, etc. to be moved out of sys.mk. Cheers, Mark. -- Mark Valentine, Thuvia Labs "Tigers will do ANYTHING for a tuna fish sandwich." Mark Valentine uses "We're kind of stupid that way." *munch* *munch* and endorses FreeBSD -- To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Sat Jul 20 9:48:45 2002 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.FreeBSD.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id AA4F337B400; Sat, 20 Jul 2002 09:48:43 -0700 (PDT) Received: from thuvia.demon.co.uk (thuvia.demon.co.uk [193.237.34.248]) by mx1.FreeBSD.org (Postfix) with ESMTP id C77A743E5E; Sat, 20 Jul 2002 09:48:41 -0700 (PDT) (envelope-from mark@thuvia.demon.co.uk) Received: from dotar.thuvia.org (dotar.thuvia.org [10.0.0.4]) by phaidor.thuvia.org (8.11.6/8.11.6) with ESMTP id g6KGmab76202; Sat, 20 Jul 2002 17:48:36 +0100 (BST) (envelope-from mark@thuvia.demon.co.uk) Received: from dotar.thuvia.org (localhost [IPv6:::1]) by dotar.thuvia.org (8.12.3/8.12.3) with ESMTP id g6KGmZKC085463; Sat, 20 Jul 2002 17:48:35 +0100 (BST) (envelope-from mark@dotar.thuvia.org) Received: (from mark@localhost) by dotar.thuvia.org (8.12.3/8.12.3/Submit) id g6KGmZJE085462; Sat, 20 Jul 2002 17:48:35 +0100 (BST) Date: Sat, 20 Jul 2002 17:48:35 +0100 (BST) From: Mark Valentine Message-Id: <200207201648.g6KGmZJE085462@dotar.thuvia.org> In-Reply-To: <20020719.100151.96158427.imp@bsdimp.com> X-Disclaimer: tequila X-Mailer: Mail User's Shell (7.2.6 beta(5) 10/07/98) To: "M. Warner Losh" Subject: Re: [POLL] need a good name for share/mk API versioning Cc: ru@FreeBSD.ORG, arch@FreeBSD.ORG Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG > From: "M. Warner Losh" > Date: Fri 19 Jul, 2002 > Subject: Re: [POLL] need a good name for share/mk API versioning > In message: <200207181943.g6IJhu5A016231@dotar.thuvia.org> > Mark Valentine writes: > : If this were only for ports, the existing OSVERSION in bsd.port.mk > : would suffice, no? > > No. The OS version and the .mk files may (and often are) unrelated. > This is especially true in the case of cross build environments, which > are often used to insulate products from whatever version of FreeBSD > they happen to be running on. At Timing solutions, we often build 4.5 > based products on a 4.3 machine, and vice versa. Does this work for the ports system? I recognise the need for cross builds, but didn't consider it for ports. After all, they already use OSVERSION... Cheers, Mark. -- Mark Valentine, Thuvia Labs "Tigers will do ANYTHING for a tuna fish sandwich." Mark Valentine uses "We're kind of stupid that way." *munch* *munch* and endorses FreeBSD -- To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Sat Jul 20 9:52:55 2002 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.FreeBSD.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 0F9DE37B400; Sat, 20 Jul 2002 09:52:53 -0700 (PDT) Received: from thuvia.demon.co.uk (thuvia.demon.co.uk [193.237.34.248]) by mx1.FreeBSD.org (Postfix) with ESMTP id 5A77843E31; Sat, 20 Jul 2002 09:52:51 -0700 (PDT) (envelope-from mark@thuvia.demon.co.uk) Received: from dotar.thuvia.org (dotar.thuvia.org [10.0.0.4]) by phaidor.thuvia.org (8.11.6/8.11.6) with ESMTP id g6KGqnb76209; Sat, 20 Jul 2002 17:52:50 +0100 (BST) (envelope-from mark@thuvia.demon.co.uk) Received: from dotar.thuvia.org (localhost [IPv6:::1]) by dotar.thuvia.org (8.12.3/8.12.3) with ESMTP id g6KGqnKC085492; Sat, 20 Jul 2002 17:52:49 +0100 (BST) (envelope-from mark@dotar.thuvia.org) Received: (from mark@localhost) by dotar.thuvia.org (8.12.3/8.12.3/Submit) id g6KGqnFK085491; Sat, 20 Jul 2002 17:52:49 +0100 (BST) Date: Sat, 20 Jul 2002 17:52:49 +0100 (BST) From: Mark Valentine Message-Id: <200207201652.g6KGqnFK085491@dotar.thuvia.org> X-Mailer: Mail User's Shell (7.2.6 beta(5) 10/07/98) To: Ruslan Ermilov Subject: Re: [POLL] need a good name for share/mk API versioning Cc: arch@freebsd.org Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG > From: mark@dotar.thuvia.org (Mark Valentine) > Date: Sat 20 Jul, 2002 > Subject: Re: [POLL] need a good name for share/mk API versioning > One possible workaround might be to have sys.mk include a file defined by a > variable (if defined, perhaps in the environment), which for the FreeBSD build > system could be set to "bsd.sys.mk" or similar. This allows any build system > to have a component included before the Makefile is read. > > .if defined(__BUILD_SYS_MK) > .include "${__BUILD_SYS_MK}" > .endif Of course, no newly design build system should use this feature; use an early Makefile .include instead... Cheers, Mark. -- Mark Valentine, Thuvia Labs "Tigers will do ANYTHING for a tuna fish sandwich." Mark Valentine uses "We're kind of stupid that way." *munch* *munch* and endorses FreeBSD -- To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Sat Jul 20 11:14:52 2002 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.FreeBSD.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 600FE37B401; Sat, 20 Jul 2002 11:14:49 -0700 (PDT) Received: from harmony.village.org (rover.bsdimp.com [204.144.255.66]) by mx1.FreeBSD.org (Postfix) with ESMTP id C952F43E31; Sat, 20 Jul 2002 11:14:47 -0700 (PDT) (envelope-from imp@bsdimp.com) Received: from localhost (warner@rover2.village.org [10.0.0.1]) by harmony.village.org (8.12.3/8.12.3) with ESMTP id g6KIEk1f040124; Sat, 20 Jul 2002 12:14:46 -0600 (MDT) (envelope-from imp@bsdimp.com) Date: Sat, 20 Jul 2002 12:14:07 -0600 (MDT) Message-Id: <20020720.121407.17240249.imp@bsdimp.com> To: mark@thuvia.demon.co.uk Cc: ru@FreeBSD.ORG, arch@FreeBSD.ORG Subject: Re: [POLL] need a good name for share/mk API versioning From: "M. Warner Losh" In-Reply-To: <200207201648.g6KGmZJE085462@dotar.thuvia.org> References: <20020719.100151.96158427.imp@bsdimp.com> <200207201648.g6KGmZJE085462@dotar.thuvia.org> X-Mailer: Mew version 2.1 on Emacs 21.1 / Mule 5.0 (SAKAKI) Mime-Version: 1.0 Content-Type: Text/Plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG In message: <200207201648.g6KGmZJE085462@dotar.thuvia.org> Mark Valentine writes: : > From: "M. Warner Losh" : > Date: Fri 19 Jul, 2002 : > Subject: Re: [POLL] need a good name for share/mk API versioning : : > In message: <200207181943.g6IJhu5A016231@dotar.thuvia.org> : > Mark Valentine writes: : > : If this were only for ports, the existing OSVERSION in bsd.port.mk : > : would suffice, no? : > : > No. The OS version and the .mk files may (and often are) unrelated. : > This is especially true in the case of cross build environments, which : > are often used to insulate products from whatever version of FreeBSD : > they happen to be running on. At Timing solutions, we often build 4.5 : > based products on a 4.3 machine, and vice versa. : : Does this work for the ports system? I recognise the need for cross builds, : but didn't consider it for ports. After all, they already use OSVERSION... Using OSVERSION may be sufficient for the ports. Using a __FREEBSD_MK_API_VERSION variable would also be sufficient, as well as allowing for people that use the bsd.*.mk files externally to the project to continue to do so and cope with the different APIs. I've not looked at porting out .mk files over to the new scheme, so I don't know how necessary this would be. Warner To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Sat Jul 20 11:33:35 2002 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.FreeBSD.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id B130D37B401; Sat, 20 Jul 2002 11:33:32 -0700 (PDT) Received: from thuvia.demon.co.uk (thuvia.demon.co.uk [193.237.34.248]) by mx1.FreeBSD.org (Postfix) with ESMTP id 5E85243E3B; Sat, 20 Jul 2002 11:33:29 -0700 (PDT) (envelope-from mark@thuvia.demon.co.uk) Received: from dotar.thuvia.org (dotar.thuvia.org [10.0.0.4]) by phaidor.thuvia.org (8.11.6/8.11.6) with ESMTP id g6KIX1b76526; Sat, 20 Jul 2002 19:33:02 +0100 (BST) (envelope-from mark@thuvia.demon.co.uk) Received: from dotar.thuvia.org (localhost [IPv6:::1]) by dotar.thuvia.org (8.12.3/8.12.3) with ESMTP id g6KIX1KC087738; Sat, 20 Jul 2002 19:33:01 +0100 (BST) (envelope-from mark@dotar.thuvia.org) Received: (from mark@localhost) by dotar.thuvia.org (8.12.3/8.12.3/Submit) id g6KIX1FC087737; Sat, 20 Jul 2002 19:33:01 +0100 (BST) Date: Sat, 20 Jul 2002 19:33:01 +0100 (BST) From: Mark Valentine Message-Id: <200207201833.g6KIX1FC087737@dotar.thuvia.org> In-Reply-To: <20020720.121407.17240249.imp@bsdimp.com> X-Disclaimer: tequila X-Mailer: Mail User's Shell (7.2.6 beta(5) 10/07/98) To: "M. Warner Losh" Subject: Re: [POLL] need a good name for share/mk API versioning Cc: ru@FreeBSD.ORG, arch@FreeBSD.ORG Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG > From: "M. Warner Losh" > Date: Sat 20 Jul, 2002 > Subject: Re: [POLL] need a good name for share/mk API versioning > Using OSVERSION may be sufficient for the ports. Using a > __FREEBSD_MK_API_VERSION variable would also be sufficient, as well as > allowing for people that use the bsd.*.mk files externally to the > project to continue to do so and cope with the different APIs. Indeed, if we have __FREEBSD_MK_API_VERSION or equivalent, then ports should also use that where appropriate rather than OSVERSION. Cheers, Mark. -- Mark Valentine, Thuvia Labs "Tigers will do ANYTHING for a tuna fish sandwich." Mark Valentine uses "We're kind of stupid that way." *munch* *munch* and endorses FreeBSD -- To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Sat Jul 20 11:35:40 2002 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.FreeBSD.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id BFBB137B400; Sat, 20 Jul 2002 11:35:36 -0700 (PDT) Received: from harmony.village.org (rover.bsdimp.com [204.144.255.66]) by mx1.FreeBSD.org (Postfix) with ESMTP id 0988A43E5E; Sat, 20 Jul 2002 11:35:36 -0700 (PDT) (envelope-from imp@bsdimp.com) Received: from localhost (warner@rover2.village.org [10.0.0.1]) by harmony.village.org (8.12.3/8.12.3) with ESMTP id g6KIZY1f040229; Sat, 20 Jul 2002 12:35:34 -0600 (MDT) (envelope-from imp@bsdimp.com) Date: Sat, 20 Jul 2002 12:34:56 -0600 (MDT) Message-Id: <20020720.123456.79140066.imp@bsdimp.com> To: mark@thuvia.demon.co.uk Cc: ru@FreeBSD.ORG, arch@FreeBSD.ORG Subject: Re: [POLL] need a good name for share/mk API versioning From: "M. Warner Losh" In-Reply-To: <200207201833.g6KIX1FC087737@dotar.thuvia.org> References: <20020720.121407.17240249.imp@bsdimp.com> <200207201833.g6KIX1FC087737@dotar.thuvia.org> X-Mailer: Mew version 2.1 on Emacs 21.1 / Mule 5.0 (SAKAKI) Mime-Version: 1.0 Content-Type: Text/Plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG In message: <200207201833.g6KIX1FC087737@dotar.thuvia.org> Mark Valentine writes: : > From: "M. Warner Losh" : > Date: Sat 20 Jul, 2002 : > Subject: Re: [POLL] need a good name for share/mk API versioning : : > Using OSVERSION may be sufficient for the ports. Using a : > __FREEBSD_MK_API_VERSION variable would also be sufficient, as well as : > allowing for people that use the bsd.*.mk files externally to the : > project to continue to do so and cope with the different APIs. : : Indeed, if we have __FREEBSD_MK_API_VERSION or equivalent, then ports : should also use that where appropriate rather than OSVERSION. Yes. Exactly my point :-) Warner To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Sat Jul 20 14:39:47 2002 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.FreeBSD.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 92C1737B400 for ; Sat, 20 Jul 2002 14:39:39 -0700 (PDT) Received: from gw.catspoiler.org (217-ip-163.nccn.net [209.79.217.163]) by mx1.FreeBSD.org (Postfix) with ESMTP id C158143E3B for ; Sat, 20 Jul 2002 14:39:38 -0700 (PDT) (envelope-from dl-slicex@catspoiler.org) Received: from mousie.catspoiler.org (mousie.catspoiler.org [192.168.101.2]) by gw.catspoiler.org (8.12.5/8.12.5) with ESMTP id g6KLdRwr041480; Sat, 20 Jul 2002 14:39:32 -0700 (PDT) (envelope-from dl-slicex@catspoiler.org) Message-Id: <200207202139.g6KLdRwr041480@gw.catspoiler.org> Date: Sat, 20 Jul 2002 14:39:27 -0700 (PDT) From: Don Lewis Subject: Re: wiring the sysctl output buffer To: bde@zeta.org.au Cc: arch@FreeBSD.ORG In-Reply-To: <200207170755.g6H7t1wr029257@gw.catspoiler.org> MIME-Version: 1.0 Content-Type: TEXT/plain; charset=US-ASCII Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG On 17 Jul, Don Lewis wrote: > My initial solution was to allocate a temporary buffer, but that > complicates the code and the similar code might have to be added to > other parts of the kernel where similar situations might arise. It was > not until later that I thought of pre-wiring the buffer, which looks > like a much cleaner solution. In the latter case, I decided that it > would be best to abstact the interface so that the handler doesn't need > to become intimate with the internals of the sysctl system. If this > solution allows us to avoid the call to vslock() in the majority of > cases, then so much the better. > > The initial implementation of the interface to vslock() would probably > just ignore the size argument, but defining the API this way would allow > it to be used later if necessary without the necessity of retrofitting > the handlers that use it. Attached is another iteration of the patch. I added a length parameter to the vslock() wrapper, though this parameter is currently ignored. I modified the standard handlers to either make a temporary copy of the data or to wire the buffer as appropriate. I also added an assertion to sysctl_old_user() to catch calls to SYSCTL_OUT() that may block while locks are held. This assertion has already turned up some other calls to SYSCTL_OUT() where locks are held. This patch also includes the sysctl_kern_function_list() fix. What's the scoop with CTLFLAG_NOLOCK? Should it be used to retrieve static data? If so, should the default handlers skip the extra copy if this flag is set? Index: kern_sysctl.c =================================================================== RCS file: /home/ncvs/src/sys/kern/kern_sysctl.c,v retrieving revision 1.127 diff -u -r1.127 kern_sysctl.c --- kern_sysctl.c 15 Jul 2002 17:28:34 -0000 1.127 +++ kern_sysctl.c 20 Jul 2002 21:10:42 -0000 @@ -57,6 +57,7 @@ static MALLOC_DEFINE(M_SYSCTL, "sysctl", "sysctl internal magic"); static MALLOC_DEFINE(M_SYSCTLOID, "sysctloid", "sysctl dynamic oids"); +static MALLOC_DEFINE(M_SYSCTLTMP, "sysctltmp", "sysctl temp output buffer"); /* * Locking - this locks the sysctl tree in memory. @@ -751,12 +752,16 @@ int sysctl_handle_int(SYSCTL_HANDLER_ARGS) { - int error = 0; + int tmpout, error = 0; + /* + * Attempt to get a coherent snapshot by making a copy of the data. + */ if (arg1) - error = SYSCTL_OUT(req, arg1, sizeof(int)); + tmpout = *(int *)arg1; else - error = SYSCTL_OUT(req, &arg2, sizeof(int)); + tmpout = arg2; + error = SYSCTL_OUT(req, &tmpout, sizeof(int)); if (error || !req->newptr) return (error); @@ -776,10 +781,15 @@ sysctl_handle_long(SYSCTL_HANDLER_ARGS) { int error = 0; + long tmpout; + /* + * Attempt to get a coherent snapshot by making a copy of the data. + */ if (!arg1) return (EINVAL); - error = SYSCTL_OUT(req, arg1, sizeof(long)); + tmpout = *(long *)arg1; + error = SYSCTL_OUT(req, &tmpout, sizeof(long)); if (error || !req->newptr) return (error); @@ -799,8 +809,23 @@ sysctl_handle_string(SYSCTL_HANDLER_ARGS) { int error=0; + char *tmparg; + size_t outlen; - error = SYSCTL_OUT(req, arg1, strlen((char *)arg1)+1); + /* + * Attempt to get a coherent snapshot by copying to a + * temporary kernel buffer. + */ +retry: + outlen = strlen((char *)arg1)+1; + tmparg = malloc(outlen, M_SYSCTLTMP, M_WAITOK); + strncpy(tmparg, (char *)arg1, outlen); + if (tmparg[outlen-1] != '\0') { + free(tmparg, M_SYSCTLTMP); + goto retry; + } + error = SYSCTL_OUT(req, tmparg, outlen); + free(tmparg, M_SYSCTLTMP); if (error || !req->newptr) return (error); @@ -825,8 +850,23 @@ sysctl_handle_opaque(SYSCTL_HANDLER_ARGS) { int error; + void *tmparg; - error = SYSCTL_OUT(req, arg1, arg2); + /* + * Attempt to get a coherent snapshot, either by wiring the + * user space buffer or copying to a temporary kernel buffer + * depending on the size of the data. + */ + if (arg2 > PAGE_SIZE) { + sysctl_wire_old_buffer(req, arg2); + error = SYSCTL_OUT(req, arg1, arg2); + } else { + tmparg = malloc(arg2, M_SYSCTLTMP, M_WAITOK); + strcpy(tmparg, (char *)arg1); + bcopy(arg1, tmparg, arg2); + error = SYSCTL_OUT(req, tmparg, arg2); + free(tmparg, M_SYSCTLTMP); + } if (error || !req->newptr) return (error); @@ -953,10 +993,8 @@ int error = 0; size_t i = 0; - if (req->lock == 1 && req->oldptr) { - vslock(req->oldptr, req->oldlen); - req->lock = 2; - } + if (req->lock == 1 && req->oldptr) + WITNESS_SLEEP(1, NULL); if (req->oldptr) { i = l; if (req->oldlen <= req->oldidx) @@ -988,6 +1026,23 @@ error = copyin((char *)req->newptr + req->newidx, p, l); req->newidx += l; return (error); +} + +/* + * Wire the user space destination buffer. If set to a value greater than + * zero, the len parameter limits the maximum amount of wired memory. + * + * XXX - The len parameter is currently ignored due to the lack of + * a place to save it in the sysctl_req structure so that the matching + * amount of memory can be unwired in the sysctl exit code. + */ +void +sysctl_wire_old_buffer(struct sysctl_req *req, size_t len) +{ + if (req->lock == 1 && req->oldptr && req->oldfunc == sysctl_old_user) { + vslock(req->oldptr, req->oldlen); + req->lock = 2; + } } int Index: kern_linker.c =================================================================== RCS file: /home/ncvs/src/sys/kern/kern_linker.c,v retrieving revision 1.91 diff -u -r1.91 kern_linker.c --- kern_linker.c 7 Jul 2002 22:35:47 -0000 1.91 +++ kern_linker.c 17 Jul 2002 10:37:50 -0000 @@ -1794,6 +1794,7 @@ linker_file_t lf; int error; + sysctl_wire_old_buffer(req, 0); mtx_lock(&kld_mtx); TAILQ_FOREACH(lf, &linker_files, link) { error = LINKER_EACH_FUNCTION_NAME(lf, To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message