From owner-freebsd-arch@FreeBSD.ORG Sun Mar 30 01:37:23 2003 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id E7E8F37B401 for ; Sun, 30 Mar 2003 01:37:23 -0800 (PST) Received: from flood.ping.uio.no (flood.ping.uio.no [129.240.78.31]) by mx1.FreeBSD.org (Postfix) with ESMTP id DD5EA43FBD for ; Sun, 30 Mar 2003 01:37:22 -0800 (PST) (envelope-from des@ofug.org) Received: by flood.ping.uio.no (Postfix, from userid 2602) id 18BED5308; Sun, 30 Mar 2003 11:37:19 +0200 (CEST) X-URL: http://www.ofug.org/~des/ X-Disclaimer: The views expressed in this message do not necessarily coincide with those of any organisation or company with which I am or have been affiliated. To: Dan Nelson From: des@ofug.org (Dag-Erling =?iso-8859-1?q?Sm=F8rgrav?=) Date: Sun, 30 Mar 2003 11:37:18 +0200 In-Reply-To: <20030329204104.GF74971@dan.emsphone.com> (Dan Nelson's message of "Sat, 29 Mar 2003 14:41:04 -0600") Message-ID: User-Agent: Gnus/5.090015 (Oort Gnus v0.15) Emacs/21.2 References: <20030329204104.GF74971@dan.emsphone.com> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable cc: arch@freebsd.org Subject: Re: Allow underscores in DNS names X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 30 Mar 2003 09:37:27 -0000 Dan Nelson writes: > I thought proxy autodetect used wpad.domainname.com or looked up > http://domainname.com/wpad.dat ? All the XP machines here do that. You're right, it does. I don't know where I got the idea that WPAD used names with underscores. DES --=20 Dag-Erling Sm=F8rgrav - des@ofug.org From owner-freebsd-arch@FreeBSD.ORG Sun Mar 30 01:38:01 2003 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 2378637B401 for ; Sun, 30 Mar 2003 01:38:01 -0800 (PST) Received: from flood.ping.uio.no (flood.ping.uio.no [129.240.78.31]) by mx1.FreeBSD.org (Postfix) with ESMTP id 7DFFA43FCB for ; Sun, 30 Mar 2003 01:38:00 -0800 (PST) (envelope-from des@ofug.org) Received: by flood.ping.uio.no (Postfix, from userid 2602) id AC0C55308; Sun, 30 Mar 2003 11:37:59 +0200 (CEST) X-URL: http://www.ofug.org/~des/ X-Disclaimer: The views expressed in this message do not necessarily coincide with those of any organisation or company with which I am or have been affiliated. To: "M. Warner Losh" From: des@ofug.org (Dag-Erling =?iso-8859-1?q?Sm=F8rgrav?=) Date: Sun, 30 Mar 2003 11:37:58 +0200 In-Reply-To: <20030329.164403.54601077.imp@bsdimp.com> ("M. Warner Losh"'s message of "Sat, 29 Mar 2003 16:44:03 -0700 (MST)") Message-ID: User-Agent: Gnus/5.090015 (Oort Gnus v0.15) Emacs/21.2 References: <20030329.164403.54601077.imp@bsdimp.com> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable cc: arch@freebsd.org Subject: Re: Allow underscores in DNS names X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 30 Mar 2003 09:38:02 -0000 "M. Warner Losh" writes: > When this has come up in the past, it was decreed that _ is a bad bad > bad bad idea, even though people want it. You might want to check the > ancient archives (1998?) for all the reasons why. Arguments presented in ancient archives are not necessarily relevant five years later. DES --=20 Dag-Erling Sm=F8rgrav - des@ofug.org From owner-freebsd-arch@FreeBSD.ORG Sun Mar 30 05:06:52 2003 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 921BD37B401 for ; Sun, 30 Mar 2003 05:06:52 -0800 (PST) Received: from harmony.village.org (rover.bsdimp.com [204.144.255.66]) by mx1.FreeBSD.org (Postfix) with ESMTP id A9B7C43F3F for ; Sun, 30 Mar 2003 05:06:51 -0800 (PST) (envelope-from imp@bsdimp.com) Received: from localhost (warner@rover2.village.org [10.0.0.1]) by harmony.village.org (8.12.8/8.12.3) with ESMTP id h2UD6oA7019526; Sun, 30 Mar 2003 06:06:50 -0700 (MST) (envelope-from imp@bsdimp.com) Date: Sun, 30 Mar 2003 06:05:34 -0700 (MST) Message-Id: <20030330.060534.18864762.imp@bsdimp.com> To: des@ofug.org From: "M. Warner Losh" In-Reply-To: References: <20030329.164403.54601077.imp@bsdimp.com> X-Mailer: Mew version 2.1 on Emacs 21.2 / Mule 5.0 (SAKAKI) Mime-Version: 1.0 Content-Type: Text/Plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable cc: arch@freebsd.org Subject: Re: Allow underscores in DNS names X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 30 Mar 2003 13:06:54 -0000 In message: des@ofug.org (Dag-Erling Sm=F8rgrav) writes: : "M. Warner Losh" writes: : > When this has come up in the past, it was decreed that _ is a bad b= ad : > bad bad idea, even though people want it. You might want to check = the : > ancient archives (1998?) for all the reasons why. : = : Arguments presented in ancient archives are not necessarily relevant : five years later. True. However, they are still relevant today. '_' is illegal in DNS names, is rejected by the majority of hosts on the internet and generally is a bad idea. If you do it, make it optional. Warner From owner-freebsd-arch@FreeBSD.ORG Sun Mar 30 06:16:40 2003 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id C3E7E37B401 for ; Sun, 30 Mar 2003 06:16:40 -0800 (PST) Received: from flood.ping.uio.no (flood.ping.uio.no [129.240.78.31]) by mx1.FreeBSD.org (Postfix) with ESMTP id A2A7C43FA3 for ; Sun, 30 Mar 2003 06:16:39 -0800 (PST) (envelope-from des@ofug.org) Received: by flood.ping.uio.no (Postfix, from userid 2602) id A8B035308; Sun, 30 Mar 2003 16:16:36 +0200 (CEST) X-URL: http://www.ofug.org/~des/ X-Disclaimer: The views expressed in this message do not necessarily coincide with those of any organisation or company with which I am or have been affiliated. To: "M. Warner Losh" From: des@ofug.org (Dag-Erling =?iso-8859-1?q?Sm=F8rgrav?=) Date: Sun, 30 Mar 2003 16:16:35 +0200 In-Reply-To: <20030330.060534.18864762.imp@bsdimp.com> ("M. Warner Losh"'s message of "Sun, 30 Mar 2003 06:05:34 -0700 (MST)") Message-ID: User-Agent: Gnus/5.090015 (Oort Gnus v0.15) Emacs/21.2 References: <20030329.164403.54601077.imp@bsdimp.com> <20030330.060534.18864762.imp@bsdimp.com> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable cc: arch@freebsd.org Subject: Re: Allow underscores in DNS names X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 30 Mar 2003 14:16:42 -0000 "M. Warner Losh" writes: > True. However, they are still relevant today. '_' is illegal in DNS > names Says the RFC. IIRC, BIND traditionally did not enforce this, though it does now for A records in master zones unless you change the "check-names" setting (it seems to allow it for TXT records though). > is rejected by the majority of hosts on the internet Wrong. We (*BSD) are pretty much the only ones not to accept underscores in host names. I've tested Windows XP, Solaris 8 and Linux 2.4.18; feel free to try 'ping under_score.ofug.org' on other systems and report your findings here. > and > generally is a bad idea. I don't see why, and I've never heard any other argument against it than "the RFC says so". DES --=20 Dag-Erling Sm=F8rgrav - des@ofug.org From owner-freebsd-arch@FreeBSD.ORG Sun Mar 30 08:14:13 2003 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id D984D37B401; Sun, 30 Mar 2003 08:14:13 -0800 (PST) Received: from smtp-relay.omnis.com (smtp-relay.omnis.com [216.239.128.27]) by mx1.FreeBSD.org (Postfix) with ESMTP id 0F46543FBF; Sun, 30 Mar 2003 08:14:13 -0800 (PST) (envelope-from wes@softweyr.com) Received: from softweyr.homeunix.net (66-91-236-204.san.rr.com [66.91.236.204]) by smtp-relay.omnis.com (Postfix) with ESMTP id F3470435DE; Sun, 30 Mar 2003 08:14:09 -0800 (PST) From: Wes Peters Organization: Softweyr To: "Poul-Henning Kamp" Date: Sun, 30 Mar 2003 08:14:09 -0800 User-Agent: KMail/1.5 References: <10261.1048975354@critter.freebsd.dk> In-Reply-To: <10261.1048975354@critter.freebsd.dk> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200303300814.09180.wes@softweyr.com> cc: David Schultz cc: freebsd-arch@FreeBSD.ORG Subject: Re: Patch to protect process from pageout killing X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 30 Mar 2003 16:14:17 -0000 On Saturday 29 March 2003 14:02, Poul-Henning Kamp wrote: > In message <200303280910.32307.wes@softweyr.com>, Wes Peters writes: > >I've reworked my patch to use the madvise(2) syscall, like the > > original 4.x patch did. I've even documented it, in a man page of > > all places. Please see attached patch. If nobody objects, I'll > > commit sometime this weekend. > > I'm still not certain about the inheritance of this, do we want/is it > inherited ? In my case, I don't want the property to be inherited because I don't want all of the squid worker processes to be "immortal," just the parent. I'm not sure about other facilities. Children of inetd are a good gedanken but seem a mixed bag to me; you might not want to kill off a POP or IMAP server, but interactive logins probably are expendable. That seems to call for non-inheritance. This is easily tunable after the fact, if someone comes up with a compelling reason to have this flag be inherited. > Also, thinking about it, on at least a handful of machines I would > have more use for MADV_KILLMEFIRST having the exact opposite > behaviour. Hmm, that's an interesting viewpoint. I'm approaching this very much from the embedded server appliance viewpoint, and given my past few years experience am not able to see much beyond that and a programmers workstation. ;^) -- Where am I, and what am I doing in this handbasket? Wes Peters wes@softweyr.com From owner-freebsd-arch@FreeBSD.ORG Sun Mar 30 08:23:25 2003 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 466DA37B401 for ; Sun, 30 Mar 2003 08:23:25 -0800 (PST) Received: from whizzo.transsys.com (whizzo.TransSys.COM [144.202.42.10]) by mx1.FreeBSD.org (Postfix) with ESMTP id 835E643FA3 for ; Sun, 30 Mar 2003 08:23:24 -0800 (PST) (envelope-from louie@whizzo.transsys.com) Received: from whizzo.transsys.com (#6@localhost [127.0.0.1]) by whizzo.transsys.com (8.12.8/8.12.7) with ESMTP id h2UGNHDN042824; Sun, 30 Mar 2003 11:23:17 -0500 (EST) (envelope-from louie@whizzo.transsys.com) Message-Id: <200303301623.h2UGNHDN042824@whizzo.transsys.com> X-Mailer: exmh version 2.5 07/13/2001 with nmh-1.0.4 To: "M. Warner Losh" X-Image-URL: http://www.transsys.com/louie/images/louie-mail.jpg From: "Louis A. Mamakos" References: <20030329.164403.54601077.imp@bsdimp.com> <20030330.060534.18864762.imp@bsdimp.com> In-reply-to: Your message of "Sun, 30 Mar 2003 06:05:34 MST." <20030330.060534.18864762.imp@bsdimp.com> Mime-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable Date: Sun, 30 Mar 2003 11:23:17 -0500 Sender: louie@TransSys.COM cc: arch@freebsd.org cc: des@ofug.org Subject: Re: Allow underscores in DNS names X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 30 Mar 2003 16:23:26 -0000 > In message: > des@ofug.org (Dag-Erling Sm=F8rgrav) writes: > : "M. Warner Losh" writes: > : > When this has come up in the past, it was decreed that _ is a bad b= ad > : > bad bad idea, even though people want it. You might want to check = the > : > ancient archives (1998?) for all the reasons why. > : = > : Arguments presented in ancient archives are not necessarily relevant > : five years later. > = > True. However, they are still relevant today. '_' is illegal in DNS > names, is rejected by the majority of hosts on the internet and > generally is a bad idea. If you do it, make it optional. Strictly speaking, the '_' is illegal in HOSTNAMES. The DNS can contains= objects other than those used as hostnames, and the protocols support arbitrary strings of octets which can be used as labels in DNS names. It's the application of looking up host names using the DNS which is in question. And if underscore characters are so toxic in hostnames, then why are they allowed in /etc/hosts or NIS-dervied lookups? louie From owner-freebsd-arch@FreeBSD.ORG Sun Mar 30 08:36:26 2003 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 1012F37B401 for ; Sun, 30 Mar 2003 08:36:26 -0800 (PST) Received: from whizzo.transsys.com (whizzo.TransSys.COM [144.202.42.10]) by mx1.FreeBSD.org (Postfix) with ESMTP id 30FF643FB1 for ; Sun, 30 Mar 2003 08:36:25 -0800 (PST) (envelope-from louie@whizzo.transsys.com) Received: from whizzo.transsys.com (#6@localhost [127.0.0.1]) by whizzo.transsys.com (8.12.8/8.12.7) with ESMTP id h2UGaODN042934; Sun, 30 Mar 2003 11:36:24 -0500 (EST) (envelope-from louie@whizzo.transsys.com) Message-Id: <200303301636.h2UGaODN042934@whizzo.transsys.com> X-Mailer: exmh version 2.5 07/13/2001 with nmh-1.0.4 To: Terry Lambert X-Image-URL: http://www.transsys.com/louie/images/louie-mail.jpg From: "Louis A. Mamakos" References: <3E864AD1.6C1C3656@mindspring.com> <200303300205.h2U25vDN037209@whizzo.transsys.com> <3E8653EA.BAF9D765@mindspring.com> In-reply-to: Your message of "Sat, 29 Mar 2003 18:18:18 PST." <3E8653EA.BAF9D765@mindspring.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Sun, 30 Mar 2003 11:36:24 -0500 Sender: louie@TransSys.COM cc: arch@freebsd.org cc: Dag-Erling =?iso-8859-1?Q?Sm=F8rgrav?= Subject: Re: Allow underscores in DNS names X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 30 Mar 2003 16:36:27 -0000 > "Louis A. Mamakos" wrote: > > > There was a better patch that made it an option in resolv.conf, > > > rather than turning it on all the time. > > > > This is great, except that you'd don't need to have a resolv.conf > > on your system at all; the resolver will default to using a local > > caching nameserver. > > By this argument, it should do that anyway, if the only option > is this one. > > My own argument is that there should be an "allow_chars" option > in the resolv.conf, so that the Tuesday after this is committed, > and someone now wants "#" in domain names to support their idea > of mapping phone numbers to domain names, we don't have to go > through this whole dumb "let's violate RFC-952, just this once!" > argument yet againt. Sure, fine. Note that RFC-952 was written in a time where most host names were looked up in a file that was periodically FTP'ed from SRI-NIC. > > > > FreeBSD should be standards compliant, by default, and take work > > > to make it possible to give bogus data to other hosts on the > > > Internet who can not handle "_" or other characters because they > > > *are* standars compliant. > > > > Since this is a resolver option, you're not handing out names to > > other hosts using the DNS infrastructure. > > You are if you are a caching DNS server, which uses the resolver > code to look up data on the global DNS, caches it, and returns > it to local DNS querants. No, I don't think so, otherwise you break the DNS's ability to to have any strings of octets be DNS labels. A caching DNS server does not do gethostbyname(), nor does it use a stub resolver to resolve queries. [...] > > > > "Be conservative in what you send." > > > > And liberal in what you receive, which is exactly what modifing > > the resolver to not cause gethostbyname() and it's ilk to barf > > on these types of names. > > And liberal in what you resend? As I mentioned above, this discussion is irrelevant when it comes to "resending" anything within the context of the DNS. It's a policy decision on the part of the stub resolver that back-ends gethostby*(). > Reading the 1998 discussion, as was previously suggested, is a > good idea. > > > > There are lots of things in ancient RFCs which probably do not > > make as much sense these days as they once did. > > There is a fix for that: join an IETF group, and create a > "supercedes" RFC. Having served as the first chair of the DNS working group in the IETF more than a decade ago, I'm somewhat familiar with the process and the DNS, thanks. > The standards are the standards, as they are. Let's just review the first few paragraphs of this "standard:" STATUS OF THIS MEMO This RFC is the official specification of the format of the Internet Host Table. This edition of the specification includes minor revisions to RFC-810 which brings it up to date. Distribution of this memo is unlimited. INTRODUCTION The DoD Host Table is utilized by the DoD Hostname Server maintained by the DDN Network Information Center (NIC) on behalf of the Defense Communications Agency (DCA) [See RFC-953]. LOCATION OF THE STANDARD DOD ONLINE HOST TABLE A machine-translatable ASCII text version of the DoD Host Table is online in the file NETINFO:HOSTS.TXT on the SRI-NIC host. It can be obtained via FTP from your local host by connecting to host SRI-NIC.ARPA (26.0.0.73 or 10.0.0.51), logging in as user = ANONYMOUS, password = GUEST, and retrieving the file "NETINFO:HOSTS.TXT". The same table may also be obtained via the NIC Hostname Server, as described in RFC-953. The latter method is faster and easier, but requires a user program to make the necessary connection to the Name Server. Everyone that's still usign the DOD ONLINE HOST TABLE, please raise your hand? Thanks. > > > If there is a security issue in applications, they should get > > fixed regardless. > > OK. > > So you are advocating getting rid of the stupid "This program > uses gets(), which is unsafe" messages, right? I dunno, do the program do DNS queries? > Because the programs where the API that is being used lead to a > security isseu in applications, when people do not know how to > use the API properly. These programs are still broken and code in the stub resolver won't protect them from bogus names in /etc/hosts or NIS, so don't sleep too soundly at night, thinking you're safe from the evil underscores. > > > All this heartburn over what the gethostbyname() library function > > chooses to believe from the DNS still doesn't address getting > > hostnames out of NIS or /etc/hosts. > > NIS and /etc/hosts should *NEVER* contain a host name with an > "_". *NEVER*. Really. Using the awesome power of emacs, I caused this to happen on my system. Further, please do a 'man hosts' and see the last sentence of the DESCRIPTION section: Host names may contain any printable character other than a field delimiter, newline, or comment character. louie From owner-freebsd-arch@FreeBSD.ORG Sun Mar 30 09:46:28 2003 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 5C51037B401 for ; Sun, 30 Mar 2003 09:46:28 -0800 (PST) Received: from harmony.village.org (rover.bsdimp.com [204.144.255.66]) by mx1.FreeBSD.org (Postfix) with ESMTP id 8BB6243F93 for ; Sun, 30 Mar 2003 09:46:27 -0800 (PST) (envelope-from imp@bsdimp.com) Received: from localhost (warner@rover2.village.org [10.0.0.1]) by harmony.village.org (8.12.8/8.12.3) with ESMTP id h2UHkQA7020780; Sun, 30 Mar 2003 10:46:26 -0700 (MST) (envelope-from imp@bsdimp.com) Date: Sun, 30 Mar 2003 10:45:01 -0700 (MST) Message-Id: <20030330.104501.49852624.imp@bsdimp.com> To: des@ofug.org From: "M. Warner Losh" In-Reply-To: References: <20030330.060534.18864762.imp@bsdimp.com> X-Mailer: Mew version 2.1 on Emacs 21.2 / Mule 5.0 (SAKAKI) Mime-Version: 1.0 Content-Type: Text/Plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable cc: arch@freebsd.org Subject: Re: Allow underscores in DNS names X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 30 Mar 2003 17:46:29 -0000 In message: des@ofug.org (Dag-Erling Sm=F8rgrav) writes: : "M. Warner Losh" writes: : > True. However, they are still relevant today. '_' is illegal in D= NS : > names : = : Says the RFC. IIRC, BIND traditionally did not enforce this, though : it does now for A records in master zones unless you change the : "check-names" setting (it seems to allow it for TXT records though). Bind has enforced this for a long time. : > is rejected by the majority of hosts on the internet : = : Wrong. We (*BSD) are pretty much the only ones not to accept : underscores in host names. I've tested Windows XP, Solaris 8 and : Linux 2.4.18; feel free to try 'ping under_score.ofug.org' on other : systems and report your findings here. This must be new because bind has enforced this for a long time. : > and : > generally is a bad idea. : = : I don't see why, and I've never heard any other argument against it : than "the RFC says so". It makes it harder for the script kiddies to write eggs for buffer overflow exploits in the DNS system. That's the whole reason that the bind folks started adding the restrictive character set. Also, if you produce characters outside the character set, then you are generating illegal packets, and there is (used to be) a lot of software that would choke in subtle ways. Warner From owner-freebsd-arch@FreeBSD.ORG Sun Mar 30 09:49:20 2003 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id DF77C37B401 for ; Sun, 30 Mar 2003 09:49:20 -0800 (PST) Received: from harmony.village.org (rover.bsdimp.com [204.144.255.66]) by mx1.FreeBSD.org (Postfix) with ESMTP id 2217843F3F for ; Sun, 30 Mar 2003 09:49:20 -0800 (PST) (envelope-from imp@bsdimp.com) Received: from localhost (warner@rover2.village.org [10.0.0.1]) by harmony.village.org (8.12.8/8.12.3) with ESMTP id h2UHnJA7020791; Sun, 30 Mar 2003 10:49:19 -0700 (MST) (envelope-from imp@bsdimp.com) Date: Sun, 30 Mar 2003 10:47:54 -0700 (MST) Message-Id: <20030330.104754.78365345.imp@bsdimp.com> To: des@ofug.org From: "M. Warner Losh" In-Reply-To: References: <20030330.060534.18864762.imp@bsdimp.com> X-Mailer: Mew version 2.1 on Emacs 21.2 / Mule 5.0 (SAKAKI) Mime-Version: 1.0 Content-Type: Text/Plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable cc: arch@freebsd.org Subject: Re: Allow underscores in DNS names X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 30 Mar 2003 17:49:24 -0000 In message: des@ofug.org (Dag-Erling Sm=F8rgrav) writes: : "M. Warner Losh" writes: : > True. However, they are still relevant today. '_' is illegal in D= NS : > names : = : Says the RFC. IIRC, BIND traditionally did not enforce this, though : it does now for A records in master zones unless you change the : "check-names" setting (it seems to allow it for TXT records though). Bind 4 didn't enforce this until about 1998 or so. Like I explained in the other post, the reason it was changed was so that bind would only accept welll formed packets so that it could help reduce the liklihood that one could write an 'egg' for the payload for a buffer overflow. Bind 8 and bind 9 do enforce it for those RR that it is well defined. TXT records are well defined as allowing anything in them. Warner From owner-freebsd-arch@FreeBSD.ORG Sun Mar 30 09:50:36 2003 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 21B0237B401 for ; Sun, 30 Mar 2003 09:50:36 -0800 (PST) Received: from harmony.village.org (rover.bsdimp.com [204.144.255.66]) by mx1.FreeBSD.org (Postfix) with ESMTP id 30D8043F75 for ; Sun, 30 Mar 2003 09:50:35 -0800 (PST) (envelope-from imp@bsdimp.com) Received: from localhost (warner@rover2.village.org [10.0.0.1]) by harmony.village.org (8.12.8/8.12.3) with ESMTP id h2UHoYA7020813; Sun, 30 Mar 2003 10:50:34 -0700 (MST) (envelope-from imp@bsdimp.com) Date: Sun, 30 Mar 2003 10:49:09 -0700 (MST) Message-Id: <20030330.104909.01577314.imp@bsdimp.com> To: louie@TransSys.COM From: "M. Warner Losh" In-Reply-To: <200303301623.h2UGNHDN042824@whizzo.transsys.com> References: <20030330.060534.18864762.imp@bsdimp.com> <200303301623.h2UGNHDN042824@whizzo.transsys.com> X-Mailer: Mew version 2.1 on Emacs 21.2 / Mule 5.0 (SAKAKI) Mime-Version: 1.0 Content-Type: Text/Plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable cc: arch@freebsd.org cc: des@ofug.org Subject: Re: Allow underscores in DNS names X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 30 Mar 2003 17:50:37 -0000 In message: <200303301623.h2UGNHDN042824@whizzo.transsys.com> "Louis A. Mamakos" writes: : > In message: : > des@ofug.org (Dag-Erling Sm=F8rgrav) writes: : > : "M. Warner Losh" writes: : > : > When this has come up in the past, it was decreed that _ is a b= ad bad : > : > bad bad idea, even though people want it. You might want to ch= eck the : > : > ancient archives (1998?) for all the reasons why. : > : = : > : Arguments presented in ancient archives are not necessarily relev= ant : > : five years later. : > = : > True. However, they are still relevant today. '_' is illegal in D= NS : > names, is rejected by the majority of hosts on the internet and : > generally is a bad idea. If you do it, make it optional. : = : Strictly speaking, the '_' is illegal in HOSTNAMES. The DNS can cont= ains : objects other than those used as hostnames, and the protocols support= : arbitrary strings of octets which can be used as labels in DNS names.= : = : It's the application of looking up host names using the DNS which is : in question. And if underscore characters are so toxic in hostnames,= : then why are they allowed in /etc/hosts or NIS-dervied lookups? The question is one of standards conformance and what implementations of DNS do when they get an illegal character. Warner From owner-freebsd-arch@FreeBSD.ORG Sun Mar 30 12:47:40 2003 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 74CCD37B401 for ; Sun, 30 Mar 2003 12:47:40 -0800 (PST) Received: from smtp-relay.omnis.com (smtp-relay.omnis.com [216.239.128.27]) by mx1.FreeBSD.org (Postfix) with ESMTP id B2E3443FBD for ; Sun, 30 Mar 2003 12:47:39 -0800 (PST) (envelope-from wes@softweyr.com) Received: from softweyr.homeunix.net (66-91-236-204.san.rr.com [66.91.236.204]) by smtp-relay.omnis.com (Postfix) with ESMTP id 9654442FDD; Sun, 30 Mar 2003 12:47:35 -0800 (PST) From: Wes Peters Organization: Softweyr To: Jeff Roberson , Marc Olzheim Date: Sun, 30 Mar 2003 12:47:34 -0800 User-Agent: KMail/1.5 References: <20030326042114.H64602-100000@mail.chesapeake.net> In-Reply-To: <20030326042114.H64602-100000@mail.chesapeake.net> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200303301247.34629.wes@softweyr.com> cc: arch@freebsd.org Subject: Re: 1:1 Threading implementation. X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 30 Mar 2003 20:47:41 -0000 On Wednesday 26 March 2003 01:23, Jeff Roberson wrote: > On Wed, 26 Mar 2003, Marc Olzheim wrote: > > On Wed, Mar 26, 2003 at 03:36:57AM -0500, Jeff Roberson wrote: > > > First, if your application has more threads than cpus it is written > > > incorrectly. For people who are doing thread pools instead of > > > event driven IO models they will encounter the same overhead with > > > M:N as 1:1. I'm not sure what applications are entirely compute and > > > have more threads than cpus. These are the only ones which really > > > theoretically benefit. I don't think our threading model should be > > > designed to optimize poorly thought out applications. > > > > Might I suggest that there are 'nice' C++ ways of using > > thread-classes where both the usual C++ dogmas of readability and > > reuseability make you easily end up with more threads than cpus... > > I think that from a userland's point of view, most programmers > > shouldn't be caring less about how many cpus the machine has their > > core is running on. > > Sure, but in these cases you're not likely to be using them in > performance critical code. Which means you're not likely to be using > all of the cpu.. Which means you're going to have to go block in the > kernel anyway. And so, really what we're talking about is wasted > memory here. Not even many cpu cycles. > > I think people who actually care about performance don't want the M:N > overhead. 1:1 will be faster for them. > > For the rest, well, they didn't care about performance and so why > should we work so hard to make it marginally faster for them? If the gains are marginal it's probably not worth pursuing, especially not up front. Make it work, then make it work right, then make it work faster, right? This does not mean your statement "if your application has more threads than cpus it is written incorrectly" is correct. It is true only if threads are being use to accelerate the application. On the contrary, it is quite simple to postulate applications that are designed with threads to accomplish discrete tasks that must run in parallel and which may or may not compete with one another for resources (cpu, i/o, etc) depending on the current task load. -- Where am I, and what am I doing in this handbasket? Wes Peters wes@softweyr.com From owner-freebsd-arch@FreeBSD.ORG Sun Mar 30 14:07:06 2003 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 6CC0537B401 for ; Sun, 30 Mar 2003 14:07:06 -0800 (PST) Received: from dragon.nuxi.com (trang.nuxi.com [66.93.134.19]) by mx1.FreeBSD.org (Postfix) with ESMTP id 6E04F43FBF for ; Sun, 30 Mar 2003 14:07:05 -0800 (PST) (envelope-from obrien@NUXI.com) Received: from dragon.nuxi.com (obrien@localhost [127.0.0.1]) by dragon.nuxi.com (8.12.8/8.12.7) with ESMTP id h2UM6rAm035544; Sun, 30 Mar 2003 14:06:57 -0800 (PST) (envelope-from obrien@dragon.nuxi.com) Received: (from obrien@localhost) by dragon.nuxi.com (8.12.8/8.12.8/Submit) id h2UM6ro1035543; Sun, 30 Mar 2003 14:06:53 -0800 (PST) Date: Sun, 30 Mar 2003 14:06:53 -0800 From: "David O'Brien" To: "Louis A. Mamakos" Message-ID: <20030330220653.GA20921@dragon.nuxi.com> Mail-Followup-To: David O'Brien , "Louis A. Mamakos" , arch@freebsd.org References: <3E864AD1.6C1C3656@mindspring.com> <200303300205.h2U25vDN037209@whizzo.transsys.com> <3E8653EA.BAF9D765@mindspring.com> <200303301636.h2UGaODN042934@whizzo.transsys.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <200303301636.h2UGaODN042934@whizzo.transsys.com> User-Agent: Mutt/1.4i X-Operating-System: FreeBSD 5.0-CURRENT Organization: The NUXI BSD Group X-Pgp-Rsa-Fingerprint: B7 4D 3E E9 11 39 5F A3 90 76 5D 69 58 D9 98 7A X-Pgp-Rsa-Keyid: 1024/34F9F9D5 cc: arch@freebsd.org Subject: Re: Allow underscores in DNS names X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list Reply-To: arch@freebsd.org List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 30 Mar 2003 22:07:07 -0000 On Sun, Mar 30, 2003 at 11:36:24AM -0500, Louis A. Mamakos wrote: > Having served as the first chair of the DNS working group in the > IETF more than a decade ago, I'm somewhat familiar with the > process and the DNS, thanks. Then you are more than capable of taking this up with the BIND authors. Please take this discussion there. I personally want FreeBSD to match what they think is the Right Thing to do. I trust them. From owner-freebsd-arch@FreeBSD.ORG Sun Mar 30 16:11:45 2003 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 4FB9837B401 for ; Sun, 30 Mar 2003 16:11:45 -0800 (PST) Received: from mail.chesapeake.net (chesapeake.net [205.130.220.14]) by mx1.FreeBSD.org (Postfix) with ESMTP id 6AFD043F75 for ; Sun, 30 Mar 2003 16:11:44 -0800 (PST) (envelope-from jroberson@chesapeake.net) Received: from localhost (jroberson@localhost) by mail.chesapeake.net (8.11.6/8.11.6) with ESMTP id h2V0BXv60666; Sun, 30 Mar 2003 19:11:35 -0500 (EST) (envelope-from jroberson@chesapeake.net) Date: Sun, 30 Mar 2003 19:11:33 -0500 (EST) From: Jeff Roberson To: Wes Peters In-Reply-To: <200303301247.34629.wes@softweyr.com> Message-ID: <20030330190104.T64602-100000@mail.chesapeake.net> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: Marc Olzheim cc: arch@freebsd.org Subject: Re: 1:1 Threading implementation. X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 31 Mar 2003 00:12:01 -0000 On Sun, 30 Mar 2003, Wes Peters wrote: > On Wednesday 26 March 2003 01:23, Jeff Roberson wrote: > > On Wed, 26 Mar 2003, Marc Olzheim wrote: > > > On Wed, Mar 26, 2003 at 03:36:57AM -0500, Jeff Roberson wrote: > > > > First, if your application has more threads than cpus it is written > > > > incorrectly. For people who are doing thread pools instead of > > > > event driven IO models they will encounter the same overhead with > > > > M:N as 1:1. I'm not sure what applications are entirely compute and > > > > have more threads than cpus. These are the only ones which really > > > > theoretically benefit. I don't think our threading model should be > > > > designed to optimize poorly thought out applications. > > > > > > Might I suggest that there are 'nice' C++ ways of using > > > thread-classes where both the usual C++ dogmas of readability and > > > reuseability make you easily end up with more threads than cpus... > > > I think that from a userland's point of view, most programmers > > > shouldn't be caring less about how many cpus the machine has their > > > core is running on. > > > > Sure, but in these cases you're not likely to be using them in > > performance critical code. Which means you're not likely to be using > > all of the cpu.. Which means you're going to have to go block in the > > kernel anyway. And so, really what we're talking about is wasted > > memory here. Not even many cpu cycles. > > > > I think people who actually care about performance don't want the M:N > > overhead. 1:1 will be faster for them. > > > > For the rest, well, they didn't care about performance and so why > > should we work so hard to make it marginally faster for them? > > If the gains are marginal it's probably not worth pursuing, especially not > up front. Make it work, then make it work right, then make it work > faster, right? > > This does not mean your statement "if your application has more threads > than cpus it is written incorrectly" is correct. It is true only if > threads are being use to accelerate the application. On the contrary, it > is quite simple to postulate applications that are designed with threads > to accomplish discrete tasks that must run in parallel and which may or > may not compete with one another for resources (cpu, i/o, etc) depending > on the current task load. > The assertion is that performance critical cpu bound applications which use more than one thread per cpu are probably written incorrectly. I'm sort of tired of debating this. I think most people have a reasonable handle on the issue. I realize I didn't give enough context on my comment above. It sounds a bit like 64k is enough memory for everyone. ;-) What I'm arguing is that for most applications having the kernel do context switches isn't so expensive. And having an extra 8k of kernel memory per thread is probably ok. There are even ways to optimize this out without going through all the work of upcalls. Cheers, Jeff From owner-freebsd-arch@FreeBSD.ORG Sun Mar 30 19:09:59 2003 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id EEC1B37B401 for ; Sun, 30 Mar 2003 19:09:58 -0800 (PST) Received: from smtp-relay.omnis.com (smtp-relay.omnis.com [216.239.128.27]) by mx1.FreeBSD.org (Postfix) with ESMTP id 29E0F43F93 for ; Sun, 30 Mar 2003 19:09:58 -0800 (PST) (envelope-from wes@softweyr.com) Received: from softweyr.homeunix.net (66-91-236-204.san.rr.com [66.91.236.204]) by smtp-relay.omnis.com (Postfix) with ESMTP id 9446A42FC1; Sun, 30 Mar 2003 19:09:57 -0800 (PST) From: Wes Peters Organization: Softweyr To: "Louis A. Mamakos" , Terry Lambert Date: Sun, 30 Mar 2003 19:09:56 -0800 User-Agent: KMail/1.5 References: <3E864AD1.6C1C3656@mindspring.com> <200303300205.h2U25vDN037209@whizzo.transsys.com> In-Reply-To: <200303300205.h2U25vDN037209@whizzo.transsys.com> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Content-Disposition: inline Message-Id: <200303301909.56776.wes@softweyr.com> cc: arch@freebsd.org Subject: Re: Allow underscores in DNS names X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 31 Mar 2003 03:10:02 -0000 On Saturday 29 March 2003 18:05, Louis A. Mamakos wrote: > > Dag-Erling Sm=F8rgrav wrote: > > > The attached patch, inspired by a discussion on -STABLE, modifies > > > our resolver library to allow underscores in host names, by > > > classifying the underscore as a hyphen character. Even though > > > RFC952 forbids them, underscores are becoming increasingly common > > > in DNS, and they are sometimes used for mechanisms (such as > > > Microsoft's automatic proxy configuration scheme) which we might > > > want to support in FreeBSD. > > > > There was a better patch that made it an option in resolv.conf, > > rather than turning it on all the time. > > This is great, except that you'd don't need to have a resolv.conf > on your system at all; the resolver will default to using a local > caching nameserver. In this case, you WILL need a resolv.conf if you want to use underscores,=20 then, won't you? > > FreeBSD should be standards compliant, by default, and take work > > to make it possible to give bogus data to other hosts on the > > Internet who can not handle "_" or other characters because they > > *are* standars compliant. > > Since this is a resolver option, you're not handing out names to > other hosts using the DNS infrastructure. > > > "Be conservative in what you send." > > And liberal in what you receive, which is exactly what modifing > the resolver to not cause gethostbyname() and it's ilk to barf > on these types of names. > > There are lots of things in ancient RFCs which probably do not > make as much sense these days as they once did.=20 I strongly suspect that this discussion, like many in the networking=20 arena, are caused by a pack of fools not bothering to read the RFCs=20 before plunging off on a tangent and then later calling their stupidity a=20 'feature' rather than admitting they made a mistake. Nothing about the advisability of using wild character sets in DNS names=20 has changed except for the widespread misuse of it by a certain=20 implementation that fails to enforce the RFC requirements. This is not=20 necessarily a good reason to adulterate FreeBSD. I'm not arguing for or against any position, just making sure the=20 conversation stays on track. This is not a matter of FreeBSD being=20 wrong, it's a matter of whether we want to follow Microsofts breakage. > If there is a > security issue in applications, they should get fixed regardless. > All this heartburn over what the gethostbyname() library function > chooses to believe from the DNS still doesn't address getting > hostnames out of NIS or /etc/hosts. Especially since we have a new implementation of gethostbyname on the way,= =20 from a programmer who doesn't suck. That doesn't mean we won't have to=20 fix the old one in 4.x, but it does mean we won't have to keep patching=20 the old one with every other hairbrained DNS naming scheme (i.e. the Big5=20 vs. UTF argument) some other batch of morons comes up with. =2D-=20 Where am I, and what am I doing in this handbasket? Wes Peters wes@softweyr.com From owner-freebsd-arch@FreeBSD.ORG Sun Mar 30 20:26:55 2003 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 4760437B401 for ; Sun, 30 Mar 2003 20:26:55 -0800 (PST) Received: from whale.sunbay.crimea.ua (whale.sunbay.crimea.ua [212.110.138.65]) by mx1.FreeBSD.org (Postfix) with ESMTP id 7FFF443F75 for ; Sun, 30 Mar 2003 20:26:43 -0800 (PST) (envelope-from ru@whale.sunbay.crimea.ua) Received: from whale.sunbay.crimea.ua (ru@localhost [127.0.0.1]) h2V4QSAP066125 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Mon, 31 Mar 2003 07:26:28 +0300 (EEST) (envelope-from ru@whale.sunbay.crimea.ua) Received: (from ru@localhost) by whale.sunbay.crimea.ua (8.12.8/8.12.8/Submit) id h2V4QSHZ066120; Mon, 31 Mar 2003 07:26:28 +0300 (EEST) (envelope-from ru) Date: Mon, 31 Mar 2003 07:26:28 +0300 From: Ruslan Ermilov To: "M. Warner Losh" Message-ID: <20030331042628.GA65700@sunbay.com> References: <20030329.163343.53040416.imp@bsdimp.com> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="WhfpMioaduB5tiZL" Content-Disposition: inline In-Reply-To: <20030329.163343.53040416.imp@bsdimp.com> User-Agent: Mutt/1.5.4i cc: arch@freebsd.org Subject: Re: depend + all vs dependall X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 31 Mar 2003 04:26:56 -0000 --WhfpMioaduB5tiZL Content-Type: multipart/mixed; boundary="gBBFr7Ir9EOA20Yy" Content-Disposition: inline --gBBFr7Ir9EOA20Yy Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sat, Mar 29, 2003 at 04:33:43PM -0700, M. Warner Losh wrote: > NetBSD created a dependall target some time ago. This target does a > make depend and then a make all so they only have to traverse the tree > once for these two stages rather than twice. The time of a buildworld > came up in a discussion recently and I thought I'd see how hard it > would be to do something similar in FreeBSD. Here are my preliminary > results. >=20 > Machine: Dell Inspiron 8000, 256M RAM, P3-700 > time make buildworld > (2:04:34 wall time, didn't save the actual output :-(. >=20 > Machine: Dual Athlon XP2000+ 1.5G RAM aac controller. >=20 > time make buildworld -j 8 -s >=20 > run 0: did the above to 'flush the caches/load the sources in ram' >=20 > Pre-change: >=20 > Run 1: > 1941.458u 723.640s 32:23.67 137.1% 2747+2215k 1447+145802io 465pf+0w > Run 2: > 1942.160u 729.972s 31:45.84 140.2% 2748+2212k 1423+145755io 465pf+0w >=20 > After Changes: >=20 > Run 1: > 1922.767u 723.847s 30:48.64 143.1% 2785+2201k 1312+148256io 465pf+0w > Run 2: > 1922.661u 725.477s 30:49.99 143.1% 2788+2201k 1378+148489io 465pf+0w >=20 > So it looks like it saves a little over a minute out of 32 (1925s > average vs 1849s average, or almost a 4% reduction) on my big build > box. >=20 Where are these 1925 and 1849 averages? > My only concern with the patches is that they might interact badly > with a bug I remember from the FreeBSD 1.1R days, but can't reproduce, > in make. Once upon a time, 'make depend all' was different than 'make > depend && make all' because the .depend files weren't re-read after > the depend phase, but before the all phase, whereas two makes this > would be the case. Since this change combines the two, I'm a little > worried about that. Is that still a bug in FreeBSD's make? It won't > matter for a pure, virgin tree, but might for incremental builds... >=20 =2Edepend files are read at the time makefiles are read, once at startup, so calling "make depend all" is a bug. > =3D=3D=3D=3D //depot/user/imp/freebsd-imp/Makefile.inc1#18 (text+ko) =3D= =3D=3D=3D >=20 > @@ -319,18 +319,12 @@ > @echo ">>> stage 4: building libraries" > @echo "--------------------------------------------------------------" > cd ${.CURDIR}; ${WMAKE} -DNOHTML -DNOINFO -DNOMAN -DNOFSCHG libraries > -_depend: > - @echo > - @echo "--------------------------------------------------------------" > - @echo ">>> stage 4: make dependencies" > - @echo "--------------------------------------------------------------" > - cd ${.CURDIR}; ${WMAKE} par-depend > everything: > @echo > @echo "--------------------------------------------------------------" > @echo ">>> stage 4: building everything.." > @echo "--------------------------------------------------------------" > - cd ${.CURDIR}; ${WMAKE} all > + cd ${.CURDIR}; ${WMAKE} dependall > =20 This pessimizes "everything" that I often use standalone. > =3D=3D=3D=3D //depot/user/imp/freebsd-imp/share/mk/bsd.subdir.mk#2 (text+= ko) =3D=3D=3D=3D >=20 > @@ -25,8 +25,8 @@ > # put the stuff into the right "distribution". > # > # afterinstall, all, all-man, beforeinstall, checkdpadd, > -# clean, cleandepend, cleandir, depend, install, lint, maninstall, > -# obj, objlink, realinstall, regress, tags > +# clean, cleandepend, cleandir, depend, dependall, install, lint, > +# maninstall, obj, objlink, realinstall, regress, tags > # > =20 > .include > @@ -67,7 +67,7 @@ > =20 > =20 > .for __target in all all-man checkdpadd clean cleandepend cleandir \ > - depend distribute lint maninstall \ > + depend dependall distribute lint maninstall \ > obj objlink realinstall regress tags > ${__target}: _SUBDIR > .endfor If you want to really try dependall, you should implement it in bsd.subdir.mk like the distribute target, similar to this: =2Eif !target(dependall) dependall: cd ${.CURDIR}; \ ${MAKE} depend -DNO_SUBDIR; \ ${MAKE} all -DNO_SUBDIR =2Eendif Also, your test is not honest because original Makefile.inc1 did not parallelize the "all" stage of "buildworld", by not implementing par-all. Last time I tried par-all, it saved me 16% of time from the -j8 buildworld: 26m8.74s real 26m13.08s user 11m9.70s sys (old) 21m48.52s real 26m20.60s user 11m4.95s sys (new) Attached is the message with the patch. It has some Russian, but also includes a patch. Note that par-all only parallelizes top-level bsd.subdir.mk makefiles, as we depend on the ordering of traversing SUBDIRs in a few places. The plan is to drop this assumption in places that don't need this ordering. Cheers, --=20 Ruslan Ermilov Sysadmin and DBA, ru@sunbay.com Sunbay Software AG, ru@FreeBSD.org FreeBSD committer, +380.652.512.251 Simferopol, Ukraine http://www.FreeBSD.org The Power To Serve http://www.oracle.com Enabling The Information Age --gBBFr7Ir9EOA20Yy Content-Type: message/rfc822 Content-Disposition: inline Date: Fri, 4 Oct 2002 17:44:10 +0300 From: Ruslan Ermilov To: unisol@Me-262.ua.net Cc: freebsd@FreeBSDDiary.org.ua Subject: Re: [freebsd] FreeBSD upgrade Message-ID: <20021004144410.GB82605@sunbay.com> References: <20021002171801.D45096@free.ukr.net> <20021002142042.GA54439@freeman.ukrtelecom.net> <20021002143602.GA96604@burka.carrier.kiev.ua> <20021002144251.GB54730@freeman.ukrtelecom.net> <20021002211915.A91783@phantom.cris.net> <20021003073011.GK40754@netch.kiev.ua> <20021003111001.GA8724@sunbay.com> <20021004171349.B30990@Me-262.ua.net> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="ZoaI/ZTpAVc4A5k6" Content-Disposition: inline In-Reply-To: <20021004171349.B30990@Me-262.ua.net> User-Agent: Mutt/1.3.99i --ZoaI/ZTpAVc4A5k6 Content-Type: text/plain; charset=koi8-r Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Fri, Oct 04, 2002 at 05:13:49PM +0300, unisol@Me-262.ua.net wrote: > On Thu, Oct 03, 2002 at 02:10:01PM +0300, Ruslan Ermilov wrote: > > On Thu, Oct 03, 2002 at 10:30:11AM +0300, Valentin Nechayev wrote: > > > Wed, Oct 02, 2002 at 21:19:15, phantom wrote about "Re: [freebsd] Fr= eeBSD upgrade":=20 > > >=20 > > > > > > > =EE=C1 PIII 1G + ATA100 make buildworld =D0=D2=CF=C8=CF=C4=C9= =D4 =DA=C1 ~50 =CD=C9=CE=D5=D4 > > > > > > =E8=CD. Celeron 1G + ata100 make world, =C9 =D0=D2=C1=D7=CC=C5= =CE=CE=D9=CA /etc/make.conf ~ 37 min. > > > > > =E8=CD. Dual PIII 1.4G + SCSI RAID10 + =C4=C5=C6=CF=CC=D4=CE=D9= =CA /etc/make.conf=20 > > > > > make -j100 buildworld - 23 =CD=C9=CE=D5=D4=D9 > > > > PIV 1.4G + ATA66 + default /etc/make.conf > > > > make -j4 buildworld -- 24 minutes > > > > PS: Dolgo pupkami budem meryatsya ? :-) > > >=20 > > > =EE=C1=D7=C5=D2=CE=CF, =C4=CF=CC=C7=CF. =E5=D3=CC=C9 =CE=C5 =D0=CF=CE= =C9=CD=C1=D4=D8, =DE=D4=CF =DB=D4=C1=D4=CE=D9=CA make buildworld (=C2=C5=DA= -pipe, > > > =C2=C5=DA -j) =CD=C5=D2=D1=C5=D4 =D3=CB=CF=D2=CF=D3=D4=D8 =C4=C9=D3= =CB=CF=D7=CF=CA =D0=CF=C4=D3=C9=D3=D4=C5=CD=D9, =C1 =CE=C5 =D0=D2=CF=C3=C5= =D3=D3=CF=D2=C1 ;) > =EE=C5=D4 - =CE=C5=D0=D2=C1=D7=C4=C1. =E4=C9=D3=CB =D7=CC=C9=D1=C5=D4, = =CE=CF =CF=CF=CF=DE=C5=CE=D8 =CD=C1=CC=CF, =CD=CF=D6=C5=D4 =D4=CF=CC=D8=CB= =CF =D5 dual athlon > =DC=D4=CF =C2=D5=C4=C5=D4 =D3=D5=DD=C5=D3=D4=D7=C5=CE=CE=CF. =F0=D2=CF=D7= =C5=D2=C5=CE=CF =CE=C1 =D3=C2=CF=D2=CB=C5 world =CE=C1 2xPIII - -j4/-j8 =C4= =C1=CC=CF > =D0=D2=CF=D3=D4=CF =C4=D7=D5=CB=D2=C1=D4=CE=D9=CA =D0=D2=C9=D2=CF=D3=D4, = -j16 - =D4=CF=D2=CD=CF=DA=CE=D5=CC=CF =D5=D6=C5 =CE=C5=CD=CE=CF=C7=CF. >=20 > > =F0=CF=DE=C5=CD=D5 "=C2=C5=DA"? =FA=C4=C5=D3=D8-=D4=CF =CB=C1=CB =D2= =C1=DA =D2=C5=DE=D8 =C9=C4=A3=D4 =CF "=D3". :-) > >=20 > > =F1 =C4=D5=CD=C1=C0, =DE=D4=CF -j100 -- =DC=D4=CF =D3=CC=C9=DB=CB=CF=CD= . =F1 =C2=D9 =D0=CF=D3=CF=D7=C5=D4=CF=D7=C1=CC =CE=C1=DE=C1=D4=D8 =D3 > -j100 - =DC=D4=CF =D5=D6=C5 =D0=CF=DE=D4=C9 "=D0=D2=CF=C9=DA=D7=CF=C4=C9= =D4=C5=CC=D8=CE=CF=D3=D4=D8 =C4=C9=D3=CB=CF=D7=CF=CA =D3=C9=D3=D4=C5=CD=D9". > =F7=C9=C4=C5=CC =CC=C9 =CB=D4=CF, =DE=D4=CF =D0=D2=CF=C9=D3=C8=CF=C4=C9= =D4 =D3 P4-1.7GHz/1G DDR/100G IDE, =CB=CF=C7=C4=C1 > =CE=C1 =CE=A3=CD =D7=C9=D3=C9=D4 3000-4000 =D0=D2=CF=C3=C5=D3=D3=CF=D7, = =C7=C4=C5 1/4 =C9=DA =CE=C9=C8 =CF=D4=CB=D2=D9=D7=C1=C5=D4 ~5 =C6=C1=CA=CC= =CF=D7 > =C9 =C4=C5=CC=C1=C5=D4 fflush =CB=C1=D6=C4=D5=C0 =D3=C5=CB=D5=CE=C4=D5 = =D0=CF=D3=CC=C5 =D7=D9=D7=CF=C4=C1 ~6-10 =C2=C1=CA=D4? > =F7=CF=D4 =D4=CF =F4=CF=D2=CD=CF=DA=C1. =F0=D2=C9=DE=A3=CD, =D7=CE=C1=DE= =C1=CC=C5 =D7=D3=A3 =DC=D4=CF =D7=D0=CF=CC=CE=C5 =CE=CF=D2=CD=C1=CC=D8=CE= =CF =D2=C1=C2=CF=D4=C1=C5=D4, > =D0=CF=D4=CF=CD LA =D7=DA=CC=C5=D4=C1=C5=D4, =D7=D3=A3 =D4=CF=D2=CD=CF=DA= =C9=D4... =E1 =D7=D9=D7=CF=C4=C1 - =CB=CF=D0=C5=CA=CB=C9, ~1MB/s, > 160-200 transactions/s. > > -j4 =C9 =D5=D7=C5=CC=C9=DE=C9=D7=C1=D4=D8 =D7 =C4=D7=C1 =D2=C1=DA=C1 = =C4=CF =D4=C5=C8 =D0=CF=D2, =D0=CF=CB=C1 =C2=D5=C4=C5=CD =D3=CD=D9=D3=CC -- > > =DA=C1=CB=CF=CE =E1=CD=C4=C1=CC=D1. =E4=CC=D1 =C2=CF=CC=D8=DB=C9=CE=D3= =D4=D7=C1 =CD=C1=DB=C9=CE =CE=C1 =CB=CF=D4=CF=D2=D9=C8 =CD=CE=C5 =D0=D2=C9= =C8=CF=C4=C9=CC=CF=D3=D8 > > =C4=C5=CC=C1=D4=D8 -j world'=D9 =C9 release'=D9, =CF=D0=D4=C9=CD=C1=CC= =D8=CE=D9=CD =CF=CB=C1=DA=D9=D7=C1=CC=CF=D3=D8 -j4, > > =C4=C1=D6=C5 =CE=C1 2xCPU SMP =C9 1xCPU =CD=C1=DB=C9=CE=C1=C8. > =E5=D3=D4=C5=D3=D4=D7=C5=CE=CE=CF - parallel =D4=CF =CF=CE=CF =C4=C1, =CE= =CF =D7=D3=A3 =D2=C1=D7=CE=CF =CE=C5 =D7=D3=A3 =D2=C1=D3=D0=C1=D2=C1=CC=C5= =CC=C9=D7=C1=C5=D4=D3=D1 - > =D0=CF=CC=CF=D7=C9=CE=D5 =D7=D2=C5=CD=C5=CE=C9 =D7=D3=A3 =D2=C1=D7=CE=CF = "=CE=C9=DE=C5=C7=CF =CE=C5 =D0=D2=CF=C9=D3=C8=CF=C4=C9=D4", -j=D3=CB=CF=CC= =D8=CB=CF=C8=CF=DE=C5=DB=D8 > =D0=C9=DB=C9 - =D0=D2=CF=D4=C9=D7 dependencies&so on =CF=CE=CF =CE=C5 =D0= =D9=D4=C1=C5=D4=D3=D1 "=D0=C5=D2=C5=D4=D8"... >=20 =EB=D3=D4=C1=D4=C9 =CF =D0=C1=D2=C1=CC=CC=C5=CC=D8=CE=CF=D3=D4=C9. =F1 =D7= =D3=A3 =DA=C1=C2=D9=D7=C1=C0 =D0=CF=D0=D2=CF=C2=CF=D7=C1=D4=D8 =DC=D4=CF=D4= =D0=C1=D4=DE. =E9=CE=D4=C5=D2=C5=D3=CE=CF, =CB=C1=CB =CB =CE=C5=CD=D5 =CF=D4=CE=C5=D3=A3=D4=D3=D1 buildworld (RELENG_4= ). =EB=D4=CF-=CE=C9=C2=D5=C4=D8 =C8=CF=DE=C5=D4 "=DE=C5=D3=D4=CE=CF" =D0=CF=CD=C5=D2=D1=D4=D8 =DC=D4=CF =CE=C1 idle =CD=C1=DB=C9=CE=CB=C5 "=D3" = =C9 "=C2=C5=DA"? =F0=CF=C4 "=DE=C5=D3=D4=CE=CF=D3=D4=D8=C0" =D1 =D0=CF=CE= =C9=CD=C1=C0 =D7 =D0=C5=D2=D7=D5=C0 =CF=DE=C5=D2=C5=C4=D8 =CF=C4=C9=CE=C1=CB=CF=D7=D9=C5= =D5=D3=CC=CF=D7=C9=D1 =DA=C1=D0=D5=D3=CB=C1 (=CF=D4=D3=D5=D4=D3=D4=D7=C9= =C5 =D0=CF=D3=D4=CF=D2=CF=CE=CE=C5=CA =DA=C1=C7=D2=D5=DA=CB=C9, =CF=C4=C9=CE=C1=CB=CF=D7=CF=C5 =DE=C9=D3=CC=CF ma= ke(1) =D0=D2=CF=C3=C5=D3=D3=CF=D7 (=DA=CE=C1=DE=C5=CE=C9=C5 -j), =C9 =C9=DA= =CE=C1=DE=C1=CC=D8=CE=CF=C5 =CF=D4=D3=D5=D4=D3=D4=D7=C9=C5 /usr/obj (=CD=CF=D6=CE=CF =C4=CC=D1 =D5=D3= =CB=CF=D2=C5=CE=C9=D1 =D4=CF=C7=C4=C1 =C9 =D3 -DNOCLEAN). =F0=CF =C9=C4=C5=C5 =C4=CF=CC=D6=CE=CF =D2=C1=C2=CF=D4=C1=D4=D8 =C2=C5=DA = =D0=D2=CF=C2=CC=C5=CD (=C5=D3=CC=C9 =CE=C5 =D2=C1=C2=CF=D4=C1=C5=D4, =DA=CE= =C1=DE=C9=D4 =C7=C4=C5-=D4=CF =C2=C1=C7=C1), =C9 =D1 =D4=C1=CB =D0=CF=CC=C1=C7=C1=C0 =DE=D4=CF =D7=D9=C9= =C7=D2=D9=DB =C4=CF=CC=D6=C5=CE =C2=D9=D4=D8 =CE=C1=CC=C9=C3=CF. =E9=CE=D4= =C5=D2=C5=D3=CE=CF =C2=D9=CC=CF =C2=D9 =D5=D3=CC=D9=DB=C1=D4=D8 =CF =D2=C5=DA=D5=CC=D8=D4=C1=D4=C1=C8. =E5= =D3=CC=C9 =D2=C5=DA=D5=CC=D8=D4=C1=D4=D9 =C2=D5=C4=D5=D4 =D7=D0=C5=DE=C1=D4= =CC=D1=C0=DD=C9=CD=C9, =CF=C2=C5=DD=C1=C0 =DC=D4=CF =DA=C1=CB=CF=CD=CD=C9=D4=C9=D4=D8 (=D7 =CB=C1= =CB=CF=CA-=CE=C9=C2=D5=C4=D8 =C6=CF=D2=CD=C5). :-) %%% Index: Makefile.inc1 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D RCS file: /home/ncvs/src/Makefile.inc1,v retrieving revision 1.141.2.56 diff -u -r1.141.2.56 Makefile.inc1 --- Makefile.inc1 30 Aug 2002 18:26:49 -0000 1.141.2.56 +++ Makefile.inc1 4 Oct 2002 14:38:02 -0000 @@ -317,7 +317,7 @@ @echo "--------------------------------------------------------------" @echo ">>> stage 4: building everything.." @echo "--------------------------------------------------------------" - cd ${.CURDIR}; ${WMAKE} all + cd ${.CURDIR}; ${WMAKE} par-all =20 =20 WMAKE_TGTS=3D @@ -748,7 +748,7 @@ _prebuild_libs: ${_prebuild_libs:S/$/__L/} _generic_libs: ${_generic_libs:S/$/__L/} =20 -.for __target in clean cleandepend cleandir depend includes obj +.for __target in all clean cleandepend cleandir depend includes obj .for entry in ${SUBDIR} ${entry}.${__target}__D: .PHONY @if test -d ${.CURDIR}/${entry}.${MACHINE_ARCH}; then \ %%% Cheers, --=20 Ruslan Ermilov Sysadmin and DBA, ru@sunbay.com Sunbay Software AG, ru@FreeBSD.org FreeBSD committer, +380.652.512.251 Simferopol, Ukraine http://www.FreeBSD.org The Power To Serve http://www.oracle.com Enabling The Information Age --ZoaI/ZTpAVc4A5k6 Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.0.7 (FreeBSD) iD8DBQE9nak6Ukv4P6juNwoRAtPOAJ0ebBsXZyzXcwi4E6TpgieQ7bGGjwCfV9nj hec1ZLIllEJXSSpsf9Sb9Mc= =5tGi -----END PGP SIGNATURE----- --ZoaI/ZTpAVc4A5k6-- --gBBFr7Ir9EOA20Yy-- --WhfpMioaduB5tiZL Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.1 (FreeBSD) iD8DBQE+h8N0Ukv4P6juNwoRAqNXAJ9JLM/XJn34G8TyFDs3wyzhLNId3ACeJYnP FcqW/YDrV4UV0wkgHcfNEYY= =RAnI -----END PGP SIGNATURE----- --WhfpMioaduB5tiZL-- From owner-freebsd-arch@FreeBSD.ORG Sun Mar 30 23:04:49 2003 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id BAA2037B408; Sun, 30 Mar 2003 23:04:49 -0800 (PST) Received: from mailman.zeta.org.au (mailman.zeta.org.au [203.26.10.16]) by mx1.FreeBSD.org (Postfix) with ESMTP id 4C8B443FA3; Sun, 30 Mar 2003 23:04:46 -0800 (PST) (envelope-from bde@zeta.org.au) Received: from katana.zip.com.au (katana.zip.com.au [61.8.7.246]) by mailman.zeta.org.au (8.9.3/8.8.7) with ESMTP id RAA02723; Mon, 31 Mar 2003 17:04:43 +1000 Date: Mon, 31 Mar 2003 17:04:42 +1000 (EST) From: Bruce Evans X-X-Sender: bde@gamplex.bde.org To: Ruslan Ermilov In-Reply-To: <20030331042628.GA65700@sunbay.com> Message-ID: <20030331165732.Q17731@gamplex.bde.org> References: <20030329.163343.53040416.imp@bsdimp.com> <20030331042628.GA65700@sunbay.com> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: arch@freebsd.org cc: "M. Warner Losh" Subject: Re: depend + all vs dependall X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 31 Mar 2003 07:04:52 -0000 On Mon, 31 Mar 2003, Ruslan Ermilov wrote: > On Sat, Mar 29, 2003 at 04:33:43PM -0700, M. Warner Losh wrote: > ... > > @@ -67,7 +67,7 @@ > > > > > > .for __target in all all-man checkdpadd clean cleandepend cleandir \ > > - depend distribute lint maninstall \ > > + depend dependall distribute lint maninstall \ > > obj objlink realinstall regress tags > > ${__target}: _SUBDIR > > .endfor > > If you want to really try dependall, you should implement it > in bsd.subdir.mk like the distribute target, similar to this: > > .if !target(dependall) > dependall: > cd ${.CURDIR}; \ > ${MAKE} depend -DNO_SUBDIR; \ > ${MAKE} all -DNO_SUBDIR > .endif That won't work, since it gives a double tree traversal for the subdirs and thus defeats the point of dependall. Any splitting up of dependall must not be passed down. > Also, your test is not honest because original Makefile.inc1 > did not parallelize the "all" stage of "buildworld", by not > implementing par-all. Last time I tried par-all, it saved > me 16% of time from the -j8 buildworld: > > 26m8.74s real 26m13.08s user 11m9.70s sys (old) > 21m48.52s real 26m20.60s user 11m4.95s sys (new) > > Attached is the message with the patch. It has some Russian, > but also includes a patch. Note that par-all only parallelizes > top-level bsd.subdir.mk makefiles, as we depend on the ordering > of traversing SUBDIRs in a few places. The plan is to drop > this assumption in places that don't need this ordering. Please benchmark mainly for the usual case of non-SMP :-). Bruce From owner-freebsd-arch@FreeBSD.ORG Sun Mar 30 23:57:20 2003 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 9D50437B401 for ; Sun, 30 Mar 2003 23:57:20 -0800 (PST) Received: from whale.sunbay.crimea.ua (whale.sunbay.crimea.ua [212.110.138.65]) by mx1.FreeBSD.org (Postfix) with ESMTP id 31E7943F85 for ; Sun, 30 Mar 2003 23:56:58 -0800 (PST) (envelope-from ru@whale.sunbay.crimea.ua) Received: from whale.sunbay.crimea.ua (ru@localhost [127.0.0.1]) h2V7uSAP085158 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Mon, 31 Mar 2003 10:56:28 +0300 (EEST) (envelope-from ru@whale.sunbay.crimea.ua) Received: (from ru@localhost) by whale.sunbay.crimea.ua (8.12.8/8.12.8/Submit) id h2V7uNqC085135; Mon, 31 Mar 2003 10:56:23 +0300 (EEST) (envelope-from ru) Date: Mon, 31 Mar 2003 10:56:23 +0300 From: Ruslan Ermilov To: Bruce Evans Message-ID: <20030331075623.GA82512@sunbay.com> References: <20030329.163343.53040416.imp@bsdimp.com> <20030331042628.GA65700@sunbay.com> <20030331165732.Q17731@gamplex.bde.org> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="6TrnltStXW4iwmi0" Content-Disposition: inline In-Reply-To: <20030331165732.Q17731@gamplex.bde.org> User-Agent: Mutt/1.5.4i cc: arch@freebsd.org cc: "M. Warner Losh" Subject: Re: depend + all vs dependall X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 31 Mar 2003 07:57:22 -0000 --6TrnltStXW4iwmi0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Mon, Mar 31, 2003 at 05:04:42PM +1000, Bruce Evans wrote: > On Mon, 31 Mar 2003, Ruslan Ermilov wrote: >=20 > > On Sat, Mar 29, 2003 at 04:33:43PM -0700, M. Warner Losh wrote: > > ... > > > @@ -67,7 +67,7 @@ > > > > > > > > > .for __target in all all-man checkdpadd clean cleandepend cleandir \ > > > - depend distribute lint maninstall \ > > > + depend dependall distribute lint maninstall \ > > > obj objlink realinstall regress tags > > > ${__target}: _SUBDIR > > > .endfor > > > > If you want to really try dependall, you should implement it > > in bsd.subdir.mk like the distribute target, similar to this: > > > > .if !target(dependall) > > dependall: > > cd ${.CURDIR}; \ > > ${MAKE} depend -DNO_SUBDIR; \ > > ${MAKE} all -DNO_SUBDIR > > .endif >=20 > That won't work, since it gives a double tree traversal for the > subdirs and thus defeats the point of dependall. Any splitting > up of dependall must not be passed down. >=20 > > Also, your test is not honest because original Makefile.inc1 > > did not parallelize the "all" stage of "buildworld", by not > > implementing par-all. Last time I tried par-all, it saved > > me 16% of time from the -j8 buildworld: > > > > 26m8.74s real 26m13.08s user 11m9.70s sys (old) > > 21m48.52s real 26m20.60s user 11m4.95s sys (new) > > > > Attached is the message with the patch. It has some Russian, > > but also includes a patch. Note that par-all only parallelizes > > top-level bsd.subdir.mk makefiles, as we depend on the ordering > > of traversing SUBDIRs in a few places. The plan is to drop > > this assumption in places that don't need this ordering. >=20 > Please benchmark mainly for the usual case of non-SMP :-). >=20 You mean for the non-SMP case but with -j? On a Celeron 800 system with ATA100 disk using -j4 -DNOCLEAN buildworld of RELENG_4 a friend reported the following times: Without patch With patch real 69m43.271s 69m26.722s user 38m22.009s 38m19.384s sys 10m45.273s 10m41.596s Further reports show that on single-CPU systems with large CPU cache the real time win was near what I have reported for 2-CPU box, and it had no effect on small cache single-CPU systems and -j builds. Cheers, --=20 Ruslan Ermilov Sysadmin and DBA, ru@sunbay.com Sunbay Software AG, ru@FreeBSD.org FreeBSD committer, +380.652.512.251 Simferopol, Ukraine http://www.FreeBSD.org The Power To Serve http://www.oracle.com Enabling The Information Age --6TrnltStXW4iwmi0 Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.1 (FreeBSD) iD8DBQE+h/SnUkv4P6juNwoRAncVAJ9s43OcueEcqlcyZcji1X+ghi6NvACfXw3E ZIQ2HN9MoqBVvWBB8ym4LPI= =749t -----END PGP SIGNATURE----- --6TrnltStXW4iwmi0-- From owner-freebsd-arch@FreeBSD.ORG Mon Mar 31 01:05:29 2003 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 3459C37B401 for ; Mon, 31 Mar 2003 01:05:29 -0800 (PST) Received: from HAL9000.homeunix.com (12-233-57-131.client.attbi.com [12.233.57.131]) by mx1.FreeBSD.org (Postfix) with ESMTP id 8B04D43F93 for ; Mon, 31 Mar 2003 01:05:28 -0800 (PST) (envelope-from das@FreeBSD.ORG) Received: from HAL9000.homeunix.com (localhost [127.0.0.1]) by HAL9000.homeunix.com (8.12.6/8.12.5) with ESMTP id h2V95Fah032538; Mon, 31 Mar 2003 01:05:15 -0800 (PST) (envelope-from das@FreeBSD.ORG) Received: (from das@localhost) by HAL9000.homeunix.com (8.12.6/8.12.5/Submit) id h2V95EvX032537; Mon, 31 Mar 2003 01:05:14 -0800 (PST) (envelope-from das@FreeBSD.ORG) Date: Mon, 31 Mar 2003 01:05:14 -0800 From: David Schultz To: Poul-Henning Kamp Message-ID: <20030331090514.GB32360@HAL9000.homeunix.com> Mail-Followup-To: Poul-Henning Kamp , Wes Peters , Garance A Drosihn , Dan Nelson , freebsd-arch@FreeBSD.ORG References: <200303280910.32307.wes@softweyr.com> <10261.1048975354@critter.freebsd.dk> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <10261.1048975354@critter.freebsd.dk> cc: freebsd-arch@FreeBSD.ORG Subject: Re: Patch to protect process from pageout killing X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 31 Mar 2003 09:05:30 -0000 Thus spake Poul-Henning Kamp : > In message <200303280910.32307.wes@softweyr.com>, Wes Peters writes: > > >I've reworked my patch to use the madvise(2) syscall, like the original > >4.x patch did. I've even documented it, in a man page of all places. > >Please see attached patch. If nobody objects, I'll commit sometime this > >weekend. > > I'm still not certain about the inheritance of this, do we want/is it > inherited ? > > Also, thinking about it, on at least a handful of machines I would > have more use for MADV_KILLMEFIRST having the exact opposite > behaviour. KILLMEFIRST is more of an advisory mechanism. See relevant discussions of SIGDANGER, priority systems, etc. These are all good ideas, but I think they belong in a domain of cooperation and altruism, whereas MADV_PROTECT is the memory equivalent of rtprio. At some level, there is generally a set of processes that should never, ever be killed, e.g. postgres, sshd, apache. So you're right that KILLMEFIRST would be useful, but MADV_PROTECT is a very nice first step. From owner-freebsd-arch@FreeBSD.ORG Mon Mar 31 03:06:13 2003 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id EC14F37B401; Mon, 31 Mar 2003 03:06:12 -0800 (PST) Received: from mailman.zeta.org.au (mailman.zeta.org.au [203.26.10.16]) by mx1.FreeBSD.org (Postfix) with ESMTP id 560C543F3F; Mon, 31 Mar 2003 03:06:11 -0800 (PST) (envelope-from bde@zeta.org.au) Received: from katana.zip.com.au (katana.zip.com.au [61.8.7.246]) by mailman.zeta.org.au (8.9.3/8.8.7) with ESMTP id VAA30213; Mon, 31 Mar 2003 21:06:08 +1000 Date: Mon, 31 Mar 2003 21:06:07 +1000 (EST) From: Bruce Evans X-X-Sender: bde@gamplex.bde.org To: Ruslan Ermilov In-Reply-To: <20030331075623.GA82512@sunbay.com> Message-ID: <20030331203247.F18282@gamplex.bde.org> References: <20030329.163343.53040416.imp@bsdimp.com> <20030331042628.GA65700@sunbay.com> <20030331075623.GA82512@sunbay.com> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: arch@freebsd.org cc: "M. Warner Losh" Subject: Re: depend + all vs dependall X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 31 Mar 2003 11:06:17 -0000 On Mon, 31 Mar 2003, Ruslan Ermilov wrote: > On Mon, Mar 31, 2003 at 05:04:42PM +1000, Bruce Evans wrote: > > On Mon, 31 Mar 2003, Ruslan Ermilov wrote: > > > ... > > > Also, your test is not honest because original Makefile.inc1 > > > did not parallelize the "all" stage of "buildworld", by not > > > implementing par-all. Last time I tried par-all, it saved > > > me 16% of time from the -j8 buildworld: > > > > > > 26m8.74s real 26m13.08s user 11m9.70s sys (old) > > > 21m48.52s real 26m20.60s user 11m4.95s sys (new) > > > > > > Attached is the message with the patch. It has some Russian, > > > but also includes a patch. Note that par-all only parallelizes > > > top-level bsd.subdir.mk makefiles, as we depend on the ordering > > > of traversing SUBDIRs in a few places. The plan is to drop > > > this assumption in places that don't need this ordering. > > > > Please benchmark mainly for the usual case of non-SMP :-). > > > You mean for the non-SMP case but with -j? Mainly the non-SMP case without -j. -j is currently too broken to give anything except pessimizations for the non-SMP case, and it would not be clear if optimizations for it are just side effects of not going near the bug. After it is fixed then it will double the number of combinations to test. > On a Celeron 800 system with ATA100 disk using -j4 -DNOCLEAN buildworld > of RELENG_4 a friend reported the following times: > > Without patch With patch > real 69m43.271s 69m26.722s > user 38m22.009s 38m19.384s > sys 10m45.273s 10m41.596s > > Further reports show that on single-CPU systems with large CPU > cache the real time win was near what I have reported for 2-CPU > box, and it had no effect on small cache single-CPU systems and > -j builds. I think I understand why it often makes little difference: it saves a tree traversal, but costs an extra make process for each leaf directory. Bruce From owner-freebsd-arch@FreeBSD.ORG Mon Mar 31 03:18:21 2003 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 03BB437B401 for ; Mon, 31 Mar 2003 03:18:21 -0800 (PST) Received: from mailsrv.otenet.gr (mailsrv.otenet.gr [195.170.0.5]) by mx1.FreeBSD.org (Postfix) with ESMTP id CDABD43FA3 for ; Mon, 31 Mar 2003 03:18:19 -0800 (PST) (envelope-from keramida@freebsd.org) Received: from gothmog.gr (patr530-a138.otenet.gr [212.205.215.138]) by mailsrv.otenet.gr (8.12.9/8.12.8) with ESMTP id h2VBIGTw019907 for ; Mon, 31 Mar 2003 14:18:17 +0300 (EEST) Received: from gothmog.gr (gothmog [127.0.0.1]) by gothmog.gr (8.12.8/8.12.8) with ESMTP id h2VBIFVG087648 for ; Mon, 31 Mar 2003 14:18:15 +0300 (EEST) (envelope-from keramida@freebsd.org) Received: (from giorgos@localhost) by gothmog.gr (8.12.8/8.12.8/Submit) id h2VBIFfw087647; Mon, 31 Mar 2003 14:18:15 +0300 (EEST) (envelope-from keramida@freebsd.org) Date: Mon, 31 Mar 2003 14:18:15 +0300 (EEST) From: Giorgos Keramidas X-X-Sender: giorgos@gothmog To: arch@freebsd.org In-Reply-To: Message-ID: <20030331141452.F79544@gothmog> References: <20030329.164403.54601077.imp@bsdimp.com> <20030330.060534.18864762.imp@bsdimp.com> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Subject: Re: Allow underscores in DNS names X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 31 Mar 2003 11:18:22 -0000 On 2003-03-30 16:16, Dag-Erling Sm?rgrav wrote: >"M. Warner Losh" writes: >> True. However, they are still relevant today. '_' is illegal in DNS >> names > > Says the RFC. IIRC, BIND traditionally did not enforce this, though > it does now for A records in master zones unless you change the > "check-names" setting (it seems to allow it for TXT records though). > >> is rejected by the majority of hosts on the internet > > Wrong. We (*BSD) are pretty much the only ones not to accept > underscores in host names. I've tested Windows XP, Solaris 8 and > Linux 2.4.18; feel free to try 'ping under_score.ofug.org' on other > systems and report your findings here. If everyone else supports underscores (I've only tested Solaris and Linux 2.4.20), then perhaps, just for the sake of interoperability with other systems, we should too. Even in Linux (where it's accepted as a valid hostname by the resolver) it's annoying when one sees the warning it causes: : igloo# host under_score.ofug.org : under_score.ofug.org A 127.0.0.1 : !!! under_score.ofug.org A record has illegal name Giorgos. From owner-freebsd-arch@FreeBSD.ORG Mon Mar 31 03:24:47 2003 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 40B2937B401 for ; Mon, 31 Mar 2003 03:24:47 -0800 (PST) Received: from whale.sunbay.crimea.ua (whale.sunbay.crimea.ua [212.110.138.65]) by mx1.FreeBSD.org (Postfix) with ESMTP id 60FA243FB1 for ; Mon, 31 Mar 2003 03:24:37 -0800 (PST) (envelope-from ru@whale.sunbay.crimea.ua) Received: from whale.sunbay.crimea.ua (ru@localhost [127.0.0.1]) h2VBOKAP010385 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Mon, 31 Mar 2003 14:24:20 +0300 (EEST) (envelope-from ru@whale.sunbay.crimea.ua) Received: (from ru@localhost) by whale.sunbay.crimea.ua (8.12.8/8.12.8/Submit) id h2VBOKuH010380; Mon, 31 Mar 2003 14:24:20 +0300 (EEST) (envelope-from ru) Date: Mon, 31 Mar 2003 14:24:20 +0300 From: Ruslan Ermilov To: Bruce Evans Message-ID: <20030331112420.GA9806@sunbay.com> References: <20030329.163343.53040416.imp@bsdimp.com> <20030331042628.GA65700@sunbay.com> <20030331165732.Q17731@gamplex.bde.org> <20030331075623.GA82512@sunbay.com> <20030331203247.F18282@gamplex.bde.org> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="gBBFr7Ir9EOA20Yy" Content-Disposition: inline In-Reply-To: <20030331203247.F18282@gamplex.bde.org> User-Agent: Mutt/1.5.4i cc: arch@freebsd.org cc: "M. Warner Losh" Subject: Re: depend + all vs dependall X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 31 Mar 2003 11:24:48 -0000 --gBBFr7Ir9EOA20Yy Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Mon, Mar 31, 2003 at 09:06:07PM +1000, Bruce Evans wrote: > On Mon, 31 Mar 2003, Ruslan Ermilov wrote: >=20 > > On Mon, Mar 31, 2003 at 05:04:42PM +1000, Bruce Evans wrote: > > > On Mon, 31 Mar 2003, Ruslan Ermilov wrote: > > > > ... > > > > Also, your test is not honest because original Makefile.inc1 > > > > did not parallelize the "all" stage of "buildworld", by not > > > > implementing par-all. Last time I tried par-all, it saved > > > > me 16% of time from the -j8 buildworld: > > > > > > > > 26m8.74s real 26m13.08s user 11m9.70s sys (old) > > > > 21m48.52s real 26m20.60s user 11m4.95s sys (new) > > > > > > > > Attached is the message with the patch. It has some Russian, > > > > but also includes a patch. Note that par-all only parallelizes > > > > top-level bsd.subdir.mk makefiles, as we depend on the ordering > > > > of traversing SUBDIRs in a few places. The plan is to drop > > > > this assumption in places that don't need this ordering. > > > > > > Please benchmark mainly for the usual case of non-SMP :-). > > > > > You mean for the non-SMP case but with -j? >=20 > Mainly the non-SMP case without -j. -j is currently too broken to give > anything except pessimizations for the non-SMP case, and it would not be > clear if optimizations for it are just side effects of not going near > the bug. After it is fixed then it will double the number of combinations > to test. >=20 Without -j, my patch obviously has no effect on either [non-]SMP systems, as "par-all" becomes a plain "all". See also below. > > On a Celeron 800 system with ATA100 disk using -j4 -DNOCLEAN buildworld > > of RELENG_4 a friend reported the following times: > > > > Without patch With patch > > real 69m43.271s 69m26.722s > > user 38m22.009s 38m19.384s > > sys 10m45.273s 10m41.596s > > > > Further reports show that on single-CPU systems with large CPU > > cache the real time win was near what I have reported for 2-CPU > > box, and it had no effect on small cache single-CPU systems and > > -j builds. >=20 > I think I understand why it often makes little difference: it saves > a tree traversal, but costs an extra make process for each leaf > directory. >=20 Hardly so. My patch doesn't affect leaf directories; only level 1 bsd.subdir.mk makefiles (*bin*/Makefile, etc.) are affected by this parallelization. Cheers, --=20 Ruslan Ermilov Sysadmin and DBA, ru@sunbay.com Sunbay Software AG, ru@FreeBSD.org FreeBSD committer, +380.652.512.251 Simferopol, Ukraine http://www.FreeBSD.org The Power To Serve http://www.oracle.com Enabling The Information Age --gBBFr7Ir9EOA20Yy Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.1 (FreeBSD) iD8DBQE+iCVkUkv4P6juNwoRAtgMAJwL6H7rFkk3aQ064dg/YJe2JZSMswCfTYWb 8e4WCAS8/H+SPVSNrOkOiGw= =u+VW -----END PGP SIGNATURE----- --gBBFr7Ir9EOA20Yy-- From owner-freebsd-arch@FreeBSD.ORG Mon Mar 31 04:19:48 2003 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 5BE2737B401; Mon, 31 Mar 2003 04:19:48 -0800 (PST) Received: from mailman.zeta.org.au (mailman.zeta.org.au [203.26.10.16]) by mx1.FreeBSD.org (Postfix) with ESMTP id 0559C43FBD; Mon, 31 Mar 2003 04:19:47 -0800 (PST) (envelope-from bde@zeta.org.au) Received: from katana.zip.com.au (katana.zip.com.au [61.8.7.246]) by mailman.zeta.org.au (8.9.3/8.8.7) with ESMTP id WAA04330; Mon, 31 Mar 2003 22:19:44 +1000 Date: Mon, 31 Mar 2003 22:19:43 +1000 (EST) From: Bruce Evans X-X-Sender: bde@gamplex.bde.org To: Ruslan Ermilov In-Reply-To: <20030331112420.GA9806@sunbay.com> Message-ID: <20030331221454.S18507@gamplex.bde.org> References: <20030329.163343.53040416.imp@bsdimp.com> <20030331042628.GA65700@sunbay.com> <20030331075623.GA82512@sunbay.com><20030331112420.GA9806@sunbay.com> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: arch@freebsd.org cc: "M. Warner Losh" Subject: Re: depend + all vs dependall X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 31 Mar 2003 12:19:51 -0000 On Mon, 31 Mar 2003, Ruslan Ermilov wrote: > On Mon, Mar 31, 2003 at 09:06:07PM +1000, Bruce Evans wrote: > > On Mon, 31 Mar 2003, Ruslan Ermilov wrote: > > > On a Celeron 800 system with ATA100 disk using -j4 -DNOCLEAN buildworld > > > of RELENG_4 a friend reported the following times: > > > > > > Without patch With patch > > > real 69m43.271s 69m26.722s > > > user 38m22.009s 38m19.384s > > > sys 10m45.273s 10m41.596s > > > > > > Further reports show that on single-CPU systems with large CPU > > > cache the real time win was near what I have reported for 2-CPU > > > box, and it had no effect on small cache single-CPU systems and > > > -j builds. > > > > I think I understand why it often makes little difference: it saves > > a tree traversal, but costs an extra make process for each leaf > > directory. > > > Hardly so. My patch doesn't affect leaf directories; only > level 1 bsd.subdir.mk makefiles (*bin*/Makefile, etc.) are > affected by this parallelization. I thought that the above times were for dependall and was trying to explain why the optimization was so small. Bruce From owner-freebsd-arch@FreeBSD.ORG Mon Mar 31 05:27:05 2003 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 8EDAD37B401 for ; Mon, 31 Mar 2003 05:27:05 -0800 (PST) Received: from whale.sunbay.crimea.ua (whale.sunbay.crimea.ua [212.110.138.65]) by mx1.FreeBSD.org (Postfix) with ESMTP id E8C4043F3F for ; Mon, 31 Mar 2003 05:27:01 -0800 (PST) (envelope-from ru@whale.sunbay.crimea.ua) Received: from whale.sunbay.crimea.ua (ru@localhost [127.0.0.1]) h2VDQjAP023293 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Mon, 31 Mar 2003 16:26:46 +0300 (EEST) (envelope-from ru@whale.sunbay.crimea.ua) Received: (from ru@localhost) by whale.sunbay.crimea.ua (8.12.8/8.12.8/Submit) id h2VDQgTg023284; Mon, 31 Mar 2003 16:26:42 +0300 (EEST) (envelope-from ru) Date: Mon, 31 Mar 2003 16:26:42 +0300 From: Ruslan Ermilov To: Bruce Evans Message-ID: <20030331132642.GA21700@sunbay.com> References: <20030329.163343.53040416.imp@bsdimp.com> <20030331042628.GA65700@sunbay.com> <20030331165732.Q17731@gamplex.bde.org> <20030331075623.GA82512@sunbay.com> <20030331203247.F18282@gamplex.bde.org> <20030331112420.GA9806@sunbay.com> <20030331221454.S18507@gamplex.bde.org> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="dDRMvlgZJXvWKvBx" Content-Disposition: inline In-Reply-To: <20030331221454.S18507@gamplex.bde.org> User-Agent: Mutt/1.5.4i cc: arch@freebsd.org cc: "M. Warner Losh" Subject: Re: depend + all vs dependall X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 31 Mar 2003 13:27:06 -0000 --dDRMvlgZJXvWKvBx Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Mon, Mar 31, 2003 at 10:19:43PM +1000, Bruce Evans wrote: > On Mon, 31 Mar 2003, Ruslan Ermilov wrote: >=20 > > On Mon, Mar 31, 2003 at 09:06:07PM +1000, Bruce Evans wrote: > > > On Mon, 31 Mar 2003, Ruslan Ermilov wrote: > > > > On a Celeron 800 system with ATA100 disk using -j4 -DNOCLEAN buildw= orld > > > > of RELENG_4 a friend reported the following times: > > > > > > > > Without patch With patch > > > > real 69m43.271s 69m26.722s > > > > user 38m22.009s 38m19.384s > > > > sys 10m45.273s 10m41.596s > > > > > > > > Further reports show that on single-CPU systems with large CPU > > > > cache the real time win was near what I have reported for 2-CPU > > > > box, and it had no effect on small cache single-CPU systems and > > > > -j builds. > > > > > > I think I understand why it often makes little difference: it saves > > > a tree traversal, but costs an extra make process for each leaf > > > directory. > > > > > Hardly so. My patch doesn't affect leaf directories; only > > level 1 bsd.subdir.mk makefiles (*bin*/Makefile, etc.) are > > affected by this parallelization. >=20 > I thought that the above times were for dependall and was trying to > explain why the optimization was so small. >=20 No, they were for my 2-lines patch to Makefile.inc1 that adds par-all. Warner's change uses par-dependall, so to make the time comparison fair, I asked him to re-test with my patch (par-depend + par-all). Cheers, --=20 Ruslan Ermilov Sysadmin and DBA, ru@sunbay.com Sunbay Software AG, ru@FreeBSD.org FreeBSD committer, +380.652.512.251 Simferopol, Ukraine http://www.FreeBSD.org The Power To Serve http://www.oracle.com Enabling The Information Age --dDRMvlgZJXvWKvBx Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.1 (FreeBSD) iD8DBQE+iEISUkv4P6juNwoRAq8SAJ4sULXD3LvljHygFYQg3OGd0c6DxQCfZyqC JkaJyxkxh6F0lHRO7sn6F6E= =3E1L -----END PGP SIGNATURE----- --dDRMvlgZJXvWKvBx-- From owner-freebsd-arch@FreeBSD.ORG Mon Mar 31 09:00:31 2003 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 2B5FE37B401 for ; Mon, 31 Mar 2003 09:00:30 -0800 (PST) Received: from Daffy.timing.com (daffy.timing.com [206.168.13.218]) by mx1.FreeBSD.org (Postfix) with ESMTP id 4E85043F3F for ; Mon, 31 Mar 2003 09:00:29 -0800 (PST) (envelope-from ben@timing.com) Received: from piglet.timing.com (oink@piglet.timing.com [206.168.13.178]) by Daffy.timing.com (8.11.3/8.11.3) with ESMTP id h2VH0SH10016 for ; Mon, 31 Mar 2003 10:00:28 -0700 (MST) (envelope-from ben@timing.com) Received: from piglet.timing.com (oink@localhost.timing.com [127.0.0.1]) by piglet.timing.com (8.12.6/8.12.6) with ESMTP id h2VH0SKS006906 for ; Mon, 31 Mar 2003 10:00:28 -0700 (MST) (envelope-from ben@piglet.timing.com) Received: (from ben@localhost) by piglet.timing.com (8.12.6/8.12.6/Submit) id h2VH0S4X006903; Mon, 31 Mar 2003 10:00:28 -0700 (MST) From: Ben Mesander MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <16008.29740.33859.344866@piglet.timing.com> Date: Mon, 31 Mar 2003 10:00:28 -0700 To: arch@freebsd.org In-Reply-To: <20030331141452.F79544@gothmog> References: <20030329.164403.54601077.imp@bsdimp.com> <20030330.060534.18864762.imp@bsdimp.com> <20030331141452.F79544@gothmog> X-Mailer: VM 7.00 under Emacs 21.2.95.2 Subject: Re: Allow underscores in DNS names X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 31 Mar 2003 17:00:52 -0000 Giorgos Keramidas writes: > If everyone else supports underscores (I've only tested Solaris and > Linux 2.4.20), then perhaps, just for the sake of interoperability > with other systems, we should too. I do not know if this is still the case, but in the past, some MTA implementations refused to deliver mail to hosts with '_' in the name. --Ben From owner-freebsd-arch@FreeBSD.ORG Mon Mar 31 09:05:23 2003 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id E97AB37B401 for ; Mon, 31 Mar 2003 09:05:23 -0800 (PST) Received: from duke.cs.duke.edu (duke.cs.duke.edu [152.3.140.1]) by mx1.FreeBSD.org (Postfix) with ESMTP id 443A243FA3 for ; Mon, 31 Mar 2003 09:05:23 -0800 (PST) (envelope-from gallatin@cs.duke.edu) Received: from grasshopper.cs.duke.edu (grasshopper.cs.duke.edu [152.3.145.30]) by duke.cs.duke.edu (8.12.9/8.12.9) with ESMTP id h2VH5MMS019097 (version=TLSv1/SSLv3 cipher=EDH-RSA-DES-CBC3-SHA bits=168 verify=NO) for ; Mon, 31 Mar 2003 12:05:22 -0500 (EST) Received: (from gallatin@localhost) by grasshopper.cs.duke.edu (8.11.6/8.9.1) id h2VH5H227344; Mon, 31 Mar 2003 12:05:17 -0500 (EST) (envelope-from gallatin@cs.duke.edu) From: Andrew Gallatin MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <16008.30029.620721.433243@grasshopper.cs.duke.edu> Date: Mon, 31 Mar 2003 12:05:17 -0500 (EST) To: freebsd-arch@freebsd.org X-Mailer: VM 6.75 under 21.1 (patch 12) "Channel Islands" XEmacs Lucid Subject: cv_timedwait() & exiting procs X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 31 Mar 2003 17:05:25 -0000 FreeBSD's cv_timedwait() function helpfully notices that a process is exiting and returns EWOULDBLOCK if it is. However, if you call cv_timedwait() in the context of a process which is already exiting, you always get back EWOULDBLOCK, regardless of whether or not the timeout expired. Similarly for the cv_wait_sig() and cv_timedwait_sig(), except they set EINTR. Does anyone else consider this behaviour to be a bug? I think it should only return EWOULDBLOCK/EINTR because a process is exiting if the process wasn't already exiting when it entered the cv_*wait* routine, but perhaps I'm misguided... Drew From owner-freebsd-arch@FreeBSD.ORG Mon Mar 31 09:45:28 2003 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 7065E37B401 for ; Mon, 31 Mar 2003 09:45:28 -0800 (PST) Received: from mail.nsu.ru (mx.nsu.ru [193.124.215.71]) by mx1.FreeBSD.org (Postfix) with ESMTP id C9ADC43FCB for ; Mon, 31 Mar 2003 09:45:25 -0800 (PST) (envelope-from danfe@regency.nsu.ru) Received: from drweb by mail.nsu.ru with drweb-scanned (Exim 3.20 #1) id 1903LE-0003uE-00 for arch@freebsd.org; Tue, 01 Apr 2003 00:45:04 +0700 Received: from regency.nsu.ru ([193.124.210.26]) by mail.nsu.ru with esmtp (Exim 3.20 #1) id 1903LC-0003rE-00 for arch@freebsd.org; Tue, 01 Apr 2003 00:45:02 +0700 Received: from regency.nsu.ru (localhost [127.0.0.1]) by regency.nsu.ru (8.12.8/8.12.8) with ESMTP id h2VHit1Q051731 for ; Tue, 1 Apr 2003 00:44:55 +0700 (NOVST) (envelope-from danfe@regency.nsu.ru) Received: (from danfe@localhost) by regency.nsu.ru (8.12.8/8.12.8/Submit) id h2VHitJW051730 for arch@freebsd.org; Tue, 1 Apr 2003 00:44:55 +0700 (NOVST) Date: Tue, 1 Apr 2003 00:44:54 +0700 From: Alexey Dokuchaev To: arch@freebsd.org Message-ID: <20030331174454.GA51622@regency.nsu.ru> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="6c2NcOVqGQ03X4Wi" Content-Disposition: inline User-Agent: Mutt/1.4i X-Envelope-To: arch@freebsd.org Subject: itimerfix() fix for time validity check X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 31 Mar 2003 17:45:29 -0000 --6c2NcOVqGQ03X4Wi Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Hello there, While porting some apps written for Linux to FreeBSD, I've encountered the fact that, while under Linux it's quite appropriate (and possible) to pass million and more milliseconds as tv_usec value in struct timeval { long tv_sec; long tv_usec; }; used in functions like setitimer(), it's not like that in FreeBSD. Brief investigation showed that itimerfix() treats tv_usec >= 1000000 invalid. On contrary, Linux recalculates the values (and eventually, tv_usec is indeed less than 1000000. Since I'm unaware of what does any particular standard say on this issue, but since the dominant number of apps are being written (and designed) for Linux, this seems to cause run-time compatibility problem; quite a few apps in out ports collection might need special patching. As a probable solution to this problem, I'm posting a little patch that fixes it, by recalculating and rebalancing values in tv structure, for your further review. ./danfe --6c2NcOVqGQ03X4Wi Content-Type: text/plain; charset=us-ascii Content-Disposition: attachment; filename="kern_time.c.diff" --- kern_time.c.orig Tue Apr 1 00:25:02 2003 +++ kern_time.c Tue Apr 1 00:26:42 2003 @@ -559,6 +559,9 @@ itimerfix(struct timeval *tv) { + tv->tv_sec += tv->tv_usec / 1000000; + tv->tv_usec %= 1000000; + if (tv->tv_sec < 0 || tv->tv_sec > 100000000 || tv->tv_usec < 0 || tv->tv_usec >= 1000000) return (EINVAL); --6c2NcOVqGQ03X4Wi-- From owner-freebsd-arch@FreeBSD.ORG Mon Mar 31 09:55:15 2003 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id BBACD37B401 for ; Mon, 31 Mar 2003 09:55:15 -0800 (PST) Received: from abel.math.ntnu.no (abel.math.ntnu.no [129.241.15.50]) by mx1.FreeBSD.org (Postfix) with SMTP id C96B143F3F for ; Mon, 31 Mar 2003 09:55:13 -0800 (PST) (envelope-from perhov@math.ntnu.no) Received: (qmail 28175 invoked by uid 29119); 31 Mar 2003 17:55:12 -0000 Date: Mon, 31 Mar 2003 19:55:12 +0200 (MEST) From: Per Kristian Hove To: arch@freebsd.org In-Reply-To: <20030331141452.F79544@gothmog> Message-ID: References: <20030329.164403.54601077.imp@bsdimp.com> <20030331141452.F79544@gothmog> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Subject: Re: Allow underscores in DNS names X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 31 Mar 2003 17:55:17 -0000 [Giorgos Keramidas, 2003-03-31] | If everyone else supports underscores (I've only tested Solaris and | Linux 2.4.20), then perhaps, just for the sake of interoperability | with other systems, we should too. Just some data points (my opinion on this matter is irrelevant): In addition to Linux 2.4, Windows XP and Solaris 8 (as reported by DES), all of the following OSes allow (and, in some cases, have been allowing for quite a while:-) underscores: # HP-UX 10 $ uname -a HP-UX xxxxxxxx B.10.10 A 9000/712 unknown $ ping under_score.ofug.org -n 1 PING under_score.ofug.org: 64 byte packets 64 bytes from 127.0.0.1: icmp_seq=0. time=0. ms ----under_score.ofug.org PING Statistics---- 1 packets transmitted, 1 packets received, 0% packet loss round-trip (ms) min/avg/max = 0/0/0 # Digtal Unix/OSF1 4 (Tru64/OSF1 5.1 behaves differently: "Unknown host") $ uname -a OSF1 xxxxxxxx V4.0 1229 alpha alpha $ ping -c 1 under_score.ofug.org PING under_score.ofug.org (127.0.0.1): 56 data bytes 64 bytes from 127.0.0.1: icmp_seq=0 ttl=64 time=1 ms ----under_score.ofug.org PING Statistics---- 1 packets transmitted, 1 packets received, 0% packet loss round-trip (ms) min/avg/max = 1/1/1 ms # Ultrix 4.3 $ uname -a ULTRIX xxxxxxxx 4.3 0 VAX unknown $ /usr/etc/ping under_score.ofug.org under_score.ofug.org is alive # IRIX 5 $ uname -a IRIX xxxxxxxx 5.3 02091401 IP22 mips $ /usr/etc/ping -c 1 under_score.ofug.org PING under_score.ofug.org (127.0.0.1): 56 data bytes 64 bytes from 127.0.0.1: icmp_seq=0 ttl=255 time=0 ms ----under_score.ofug.org PING Statistics---- 1 packets transmitted, 1 packets received, 0% packet loss round-trip min/avg/max = 0/0/0 ms # Solaris 2.6/SPARC $ uname -a SunOS xxxxxxxx 5.6 Generic_105181-29 sun4u sparc $ ping under_score.ofug.org under_score.ofug.org is alive This probably only goes to show that the sanity checks were absent in these releases, not that it was a deliberate decision to allow underscores. -- Per Kristian Hove Dept. of Mathematical Sciences Norwegian University of Science and Technology From owner-freebsd-arch@FreeBSD.ORG Mon Mar 31 12:14:56 2003 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 2AE4E37B401 for ; Mon, 31 Mar 2003 12:14:56 -0800 (PST) Received: from dan.emsphone.com (dan.emsphone.com [199.67.51.101]) by mx1.FreeBSD.org (Postfix) with ESMTP id 7939E43FDD for ; Mon, 31 Mar 2003 12:14:55 -0800 (PST) (envelope-from dan@dan.emsphone.com) Received: (from dan@localhost) by dan.emsphone.com (8.12.9/8.12.9) id h2VKEsHV057176; Mon, 31 Mar 2003 14:14:54 -0600 (CST) (envelope-from dan) Date: Mon, 31 Mar 2003 14:14:54 -0600 From: Dan Nelson To: Alexey Dokuchaev Message-ID: <20030331201454.GA6000@dan.emsphone.com> References: <20030331174454.GA51622@regency.nsu.ru> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20030331174454.GA51622@regency.nsu.ru> X-OS: FreeBSD 5.0-CURRENT X-message-flag: Outlook Error User-Agent: Mutt/1.5.4i cc: arch@freebsd.org Subject: Re: itimerfix() fix for time validity check X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 31 Mar 2003 20:14:57 -0000 In the last episode (Apr 01), Alexey Dokuchaev said: > While porting some apps written for Linux to FreeBSD, I've > encountered the fact that, while under Linux it's quite appropriate > (and possible) to pass million and more milliseconds as tv_usec value > in > > struct timeval { > long tv_sec; > long tv_usec; > }; > > used in functions like setitimer(), it's not like that in FreeBSD. > > Brief investigation showed that itimerfix() treats tv_usec >= 1000000 > invalid. On contrary, Linux recalculates the values (and eventually, > tv_usec is indeed less than 1000000. Then it is violating the SUSV3/POSIX 1003.1-2001 standard: The setitimer() function shall fail if: [EINVAL] The value argument is not in canonical form. (In canonical form, the number of microseconds is a non-negative integer less than 1000000 and the number of seconds is a non-negative integer.) Free registration to access the standard is at http://www.unix.org/online.html > Since I'm unaware of what does any particular standard say on this > issue, but since the dominant number of apps are being written (and > designed) for Linux, this seems to cause run-time compatibility problem; > quite a few apps in out ports collection might need special patching. Those programs will also fail on Solaris, which fails on invalid timeval values. -- Dan Nelson dnelson@allantgroup.com From owner-freebsd-arch@FreeBSD.ORG Mon Mar 31 12:51:00 2003 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 8D6A337B435 for ; Mon, 31 Mar 2003 12:51:00 -0800 (PST) Received: from mail.speakeasy.net (mail15.speakeasy.net [216.254.0.215]) by mx1.FreeBSD.org (Postfix) with ESMTP id EC01F43FB1 for ; Mon, 31 Mar 2003 12:50:59 -0800 (PST) (envelope-from jhb@FreeBSD.org) Received: (qmail 16395 invoked from network); 31 Mar 2003 20:51:04 -0000 Received: from unknown (HELO server.baldwin.cx) ([216.27.160.63]) (envelope-sender )encrypted SMTP for ; 31 Mar 2003 20:51:04 -0000 Received: from laptop.baldwin.cx (gw1.twc.weather.com [216.133.140.1]) by server.baldwin.cx (8.12.8/8.12.8) with ESMTP id h2VKojOv016783; Mon, 31 Mar 2003 15:50:47 -0500 (EST) (envelope-from jhb@FreeBSD.org) Message-ID: X-Mailer: XFMail 1.5.4 on FreeBSD X-Priority: 3 (Normal) Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 8bit MIME-Version: 1.0 In-Reply-To: <16008.30029.620721.433243@grasshopper.cs.duke.edu> Date: Mon, 31 Mar 2003 15:50:46 -0500 (EST) From: John Baldwin To: Andrew Gallatin cc: freebsd-arch@freebsd.org Subject: RE: cv_timedwait() & exiting procs X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 31 Mar 2003 20:51:01 -0000 On 31-Mar-2003 Andrew Gallatin wrote: > > > FreeBSD's cv_timedwait() function helpfully notices that a process is > exiting and returns EWOULDBLOCK if it is. > > However, if you call cv_timedwait() in the context of a process which > is already exiting, you always get back EWOULDBLOCK, regardless of > whether or not the timeout expired. Similarly for the cv_wait_sig() > and cv_timedwait_sig(), except they set EINTR. > > Does anyone else consider this behaviour to be a bug? I think it > should only return EWOULDBLOCK/EINTR because a process is exiting if > the process wasn't already exiting when it entered the cv_*wait* > routine, but perhaps I'm misguided... I think you are referring to these changes? revision 1.23 date: 2002/06/29 17:26:18; author: julian; state: Exp; lines: +76 -13 Part 1 of KSE-III The ability to schedule multiple threads per process (one one cpu) by making ALL system calls optionally asynchronous. to come: ia64 and power-pc patches, patches for gdb, test program (in tools) Reviewed by: Almost everyone who counts (at various times, peter, jhb, matt, alfred, mini, bernd, and a cast of thousands) NOTE: this is still Beta code, and contains lots of debugging stuff. expect slight instability in signals.. Things like: @@ -293,6 +328,8 @@ rval = ERESTART; } PROC_UNLOCK(p); + if (p->p_flag & P_WEXIT) + rval = EINTR; #ifdef KTRACE if (KTRPOINT(td, KTR_CSW)) @@ -363,6 +400,8 @@ mi_switch(); } + if (td->td_proc->p_flag & P_WEXIT) + rval = EWOULDBLOCK; mtx_unlock_spin(&sched_lock); #ifdef KTRACE if (KTRPOINT(td, KTR_CSW)) @@ -450,6 +488,9 @@ } PROC_UNLOCK(p); + if (p->p_flag & P_WEXIT) + rval = EINTR; + #ifdef KTRACE if (KTRPOINT(td, KTR_CSW)) ktrcsw(0, 0); I guess this has something to do with single threading? If so it seems broken since it should only affect the singlethreading case while you are actually doing singlethreading. Maybe Julian can help fix it? -- John Baldwin <>< http://www.FreeBSD.org/~jhb/ "Power Users Use the Power to Serve!" - http://www.FreeBSD.org/ From owner-freebsd-arch@FreeBSD.ORG Mon Mar 31 13:05:46 2003 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 69E5337B404 for ; Mon, 31 Mar 2003 13:05:46 -0800 (PST) Received: from fledge.watson.org (fledge.watson.org [204.156.12.50]) by mx1.FreeBSD.org (Postfix) with ESMTP id B928F43FAF for ; Mon, 31 Mar 2003 13:05:45 -0800 (PST) (envelope-from robert@fledge.watson.org) Received: from fledge.watson.org (fledge.pr.watson.org [192.0.2.3]) by fledge.watson.org (8.12.9/8.12.9) with SMTP id h2VL66YY013318; Mon, 31 Mar 2003 16:06:06 -0500 (EST) (envelope-from robert@fledge.watson.org) Date: Mon, 31 Mar 2003 16:06:06 -0500 (EST) From: Robert Watson X-Sender: robert@fledge.watson.org To: Andrew Gallatin In-Reply-To: <16008.30029.620721.433243@grasshopper.cs.duke.edu> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: freebsd-arch@freebsd.org Subject: Re: cv_timedwait() & exiting procs X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 31 Mar 2003 21:05:47 -0000 On Mon, 31 Mar 2003, Andrew Gallatin wrote: > FreeBSD's cv_timedwait() function helpfully notices that a process is > exiting and returns EWOULDBLOCK if it is. > > However, if you call cv_timedwait() in the context of a process which is > already exiting, you always get back EWOULDBLOCK, regardless of whether > or not the timeout expired. Similarly for the cv_wait_sig() and > cv_timedwait_sig(), except they set EINTR. > > Does anyone else consider this behaviour to be a bug? I think it should > only return EWOULDBLOCK/EINTR because a process is exiting if the > process wasn't already exiting when it entered the cv_*wait* routine, > but perhaps I'm misguided... Hmm. It has always struck me that the nice thing about the SMPng synchronization primitives is that they implement well-defined anc consistent semantics in a manner consistent with other implementations of the same primitives. I'd rather see the caller test the P flags and change the arguments to cv_timedwait() than see cv_timedwait() implement unusual semantics based on process conditions. Process exiting stuff is a special case, but only from the perspective of the high level code: I think I'd be upset to find cv_timedwait() change behavior at a much lower level (i.e., in vnode handling or last reference socket handling) just because it happens to run at exit()-time. Robert N M Watson FreeBSD Core Team, TrustedBSD Projects robert@fledge.watson.org Network Associates Laboratories From owner-freebsd-arch@FreeBSD.ORG Mon Mar 31 13:30:12 2003 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 287EF37B481 for ; Mon, 31 Mar 2003 13:30:12 -0800 (PST) Received: from mail.nsu.ru (mx.nsu.ru [193.124.215.71]) by mx1.FreeBSD.org (Postfix) with ESMTP id 14EAE43F85 for ; Mon, 31 Mar 2003 13:30:11 -0800 (PST) (envelope-from danfe@regency.nsu.ru) Received: from drweb by mail.nsu.ru with drweb-scanned (Exim 3.20 #1) id 1906qk-0006w8-00; Tue, 01 Apr 2003 04:29:50 +0700 Received: from regency.nsu.ru ([193.124.210.26]) by mail.nsu.ru with esmtp (Exim 3.20 #1) id 1906qZ-0006nW-00; Tue, 01 Apr 2003 04:29:39 +0700 Received: from regency.nsu.ru (localhost [127.0.0.1]) by regency.nsu.ru (8.12.8/8.12.8) with ESMTP id h2VLTP1Q053604; Tue, 1 Apr 2003 04:29:25 +0700 (NOVST) (envelope-from danfe@regency.nsu.ru) Received: (from danfe@localhost) by regency.nsu.ru (8.12.8/8.12.8/Submit) id h2VLTMet053603; Tue, 1 Apr 2003 04:29:22 +0700 (NOVST) Date: Tue, 1 Apr 2003 04:29:22 +0700 From: Alexey Dokuchaev To: Dan Nelson Message-ID: <20030331212922.GA53585@regency.nsu.ru> References: <20030331174454.GA51622@regency.nsu.ru> <20030331201454.GA6000@dan.emsphone.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20030331201454.GA6000@dan.emsphone.com> User-Agent: Mutt/1.4i X-Envelope-To: dnelson@allantgroup.com, arch@freebsd.org cc: arch@freebsd.org Subject: Re: itimerfix() fix for time validity check X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 31 Mar 2003 21:30:14 -0000 On Mon, Mar 31, 2003 at 02:14:54PM -0600, Dan Nelson wrote: > In the last episode (Apr 01), Alexey Dokuchaev said: > > While porting some apps written for Linux to FreeBSD, I've > > encountered the fact that, while under Linux it's quite appropriate > > (and possible) to pass million and more milliseconds as tv_usec value > > in > > > > struct timeval { > > long tv_sec; > > long tv_usec; > > }; > > > > used in functions like setitimer(), it's not like that in FreeBSD. > > > > Brief investigation showed that itimerfix() treats tv_usec >= 1000000 > > invalid. On contrary, Linux recalculates the values (and eventually, > > tv_usec is indeed less than 1000000. > > Then it is violating the SUSV3/POSIX 1003.1-2001 standard: > > The setitimer() function shall fail if: > > [EINVAL] > The value argument is not in canonical form. (In canonical form, > the number of microseconds is a non-negative integer less than > 1000000 and the number of seconds is a non-negative integer.) > > Free registration to access the standard is at > http://www.unix.org/online.html Thanks for clarification. I think the issue can now be withdrawn. > > > Since I'm unaware of what does any particular standard say on this > > issue, but since the dominant number of apps are being written (and > > designed) for Linux, this seems to cause run-time compatibility problem; > > quite a few apps in out ports collection might need special patching. > > Those programs will also fail on Solaris, which fails on invalid > timeval values. Yes, indeed, I've just checked it. ./danfe From owner-freebsd-arch@FreeBSD.ORG Mon Mar 31 13:35:51 2003 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 6620637B401 for ; Mon, 31 Mar 2003 13:35:51 -0800 (PST) Received: from cultdeadsheep.org (charon.cultdeadsheep.org [80.65.226.72]) by mx1.FreeBSD.org (Postfix) with ESMTP id 3201F43F75 for ; Mon, 31 Mar 2003 13:35:49 -0800 (PST) (envelope-from sheepkiller@cultdeadsheep.org) Received: (qmail 64301 invoked from network); 31 Mar 2003 21:35:47 -0000 Received: from unknown (HELO lucifer.cultdeadsheep.org) (192.168.0.2) by goofy.cultdeadsheep.org with SMTP; 31 Mar 2003 21:35:47 -0000 Date: Mon, 31 Mar 2003 23:35:52 +0200 From: Clement Laforet To: Terry Lambert Message-Id: <20030331233552.16859546.sheepkiller@cultdeadsheep.org> In-Reply-To: <3E864AD1.6C1C3656@mindspring.com> References: <3E864AD1.6C1C3656@mindspring.com> Organization: tH3 cUlt 0f tH3 d3@d sH33p X-Mailer: Sylpheed version 0.8.10 (GTK+ 1.2.10; i386-portbld-freebsd4.6) X-Face: ._cVVRDn#-2((lnfi^P7CoD4htI$4+#G/G)!w|,}H5yK~%(3-C.JlEYbOjJGFwJkt*7N^%z jYeu[;}]}F"3}l5R'l"X0HbvT^D\Q&%deCo)MayY`);TO Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit cc: arch@freebsd.org cc: des@ofug.org Subject: Re: Allow underscores in DNS names X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 31 Mar 2003 21:35:55 -0000 Hi guys, Currently underscore naming is commonly used in LAN/WAN environment. As far as I known, there is only a few (none ?) internet wide host names with underscore. Allowing underscore is useful, but NOT mandatory, but for sysadmins it would be like ssh SUID or ppp SUID : an option when you "build world". I think that "non-vital" options should be supported but they MUST be switched by sysops at build world stage. > > There was a better patch that made it an option in resolv.conf, > rather than turning it on all the time. maybe a variable in make.conf would be better. Ex: make.conf: NOT_REALLY_RFC_COMPLIANT=YES In realworld: #ifndef NOT_REALLY_RFC_COMPLIANT #define hyphenchar(c) ((c) == 0x2d) #else #define hyphenchar(c) ((c) == 0x2d || (c) == 0x5f) #endif To conclude, if BIND authors say "support underscore is a reality we can't eclipse", let's follow them... regards, clem PS : sorry for my poor english ;-) From owner-freebsd-arch@FreeBSD.ORG Mon Mar 31 13:48:47 2003 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 1C0D537B401; Mon, 31 Mar 2003 13:48:47 -0800 (PST) Received: from salmon.maths.tcd.ie (salmon.maths.tcd.ie [134.226.81.11]) by mx1.FreeBSD.org (Postfix) with SMTP id 57B7043FD7; Mon, 31 Mar 2003 13:48:45 -0800 (PST) (envelope-from dwmalone@maths.tcd.ie) Received: from walton.maths.tcd.ie by salmon.maths.tcd.ie with SMTP id ; 31 Mar 2003 22:48:44 +0100 (BST) To: arch@freebsd.org X-Request-Do: Date: Mon, 31 Mar 2003 22:48:44 +0100 From: David Malone Message-ID: <200303312248.aa21293@salmon.maths.tcd.ie> cc: iedowse@freebsd.org Subject: Splitting up sendit X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 31 Mar 2003 21:48:49 -0000 I wonder if anyone has any comments on this patch. It splits sendit into a part which just does the copyin of arguments and then calls a new function, kern_sendit which does the real work. The point of this is to allow the linux emulation code to avoid using the stackgap for send, sendto and sendmsg. The patch also moves the grabbing of giant into kern_sendit 'cos, as I understand it, neither copyin or the allocation of mbufs requires giant. David. Index: sys/kern/uipc_syscalls.c =================================================================== RCS file: /cvs/FreeBSD-CVS/src/sys/kern/uipc_syscalls.c,v retrieving revision 1.145 diff -u -r1.145 uipc_syscalls.c --- sys/kern/uipc_syscalls.c 31 Mar 2003 06:25:42 -0000 1.145 +++ sys/kern/uipc_syscalls.c 31 Mar 2003 20:38:45 -0000 @@ -619,47 +619,18 @@ register struct msghdr *mp; int flags; { - struct uio auio; - register struct iovec *iov; - register int i; struct mbuf *control; - struct sockaddr *to = NULL; - int len, error; - struct socket *so; -#ifdef KTRACE - struct iovec *ktriov = NULL; - struct uio ktruio; - int iovlen; -#endif - - if ((error = fgetsock(td, s, &so, NULL)) != 0) - return (error); - -#ifdef MAC - error = mac_check_socket_send(td->td_ucred, so); - if (error) - goto bad; -#endif + struct sockaddr *to; + int error; - auio.uio_iov = mp->msg_iov; - auio.uio_iovcnt = mp->msg_iovlen; - auio.uio_segflg = UIO_USERSPACE; - auio.uio_rw = UIO_WRITE; - auio.uio_td = td; - auio.uio_offset = 0; /* XXX */ - auio.uio_resid = 0; - iov = mp->msg_iov; - for (i = 0; i < mp->msg_iovlen; i++, iov++) { - if ((auio.uio_resid += iov->iov_len) < 0) { - error = EINVAL; - goto bad; - } - } - if (mp->msg_name) { + if (mp->msg_name != NULL) { error = getsockaddr(&to, mp->msg_name, mp->msg_namelen); if (error) - goto bad; - } + return error; + mp->msg_name = to; + } else + to = NULL; + if (mp->msg_control) { if (mp->msg_controllen < sizeof(struct cmsghdr) #ifdef COMPAT_OLDSOCK @@ -690,7 +661,59 @@ } #endif } else { - control = 0; + control = NULL; + } + + error = kern_sendit(td, s, mp, flags, control); + +bad: + if (to) + FREE(to, M_SONAME); + return (error); +} + +int +kern_sendit(td, s, mp, flags, control) + struct thread *td; + int s; + struct msghdr *mp; + int flags; + struct mbuf *control; +{ + struct uio auio; + struct iovec *iov; + struct socket *so; + int i; + int len, error; +#ifdef KTRACE + struct iovec *ktriov = NULL; + struct uio ktruio; + int iovlen; +#endif + + mtx_lock(&Giant); + if ((error = fgetsock(td, s, &so, NULL)) != 0) + goto bad2; + +#ifdef MAC + error = mac_check_socket_send(td->td_ucred, so); + if (error) + goto bad; +#endif + + auio.uio_iov = mp->msg_iov; + auio.uio_iovcnt = mp->msg_iovlen; + auio.uio_segflg = UIO_USERSPACE; + auio.uio_rw = UIO_WRITE; + auio.uio_td = td; + auio.uio_offset = 0; /* XXX */ + auio.uio_resid = 0; + iov = mp->msg_iov; + for (i = 0; i < mp->msg_iovlen; i++, iov++) { + if ((auio.uio_resid += iov->iov_len) < 0) { + error = EINVAL; + goto bad; + } } #ifdef KTRACE if (KTRPOINT(td, KTR_GENIO)) { @@ -701,8 +724,9 @@ } #endif len = auio.uio_resid; - error = so->so_proto->pr_usrreqs->pru_sosend(so, to, &auio, 0, control, - flags, td); + error = so->so_proto->pr_usrreqs->pru_sosend(so, mp->msg_name, &auio, + 0, control, flags, td); + if (error) { if (auio.uio_resid != len && (error == ERESTART || error == EINTR || error == EWOULDBLOCK)) @@ -728,8 +752,8 @@ #endif bad: fputsock(so); - if (to) - FREE(to, M_SONAME); +bad2: + mtx_unlock(&Giant); return (error); } @@ -762,9 +786,7 @@ #endif aiov.iov_base = uap->buf; aiov.iov_len = uap->len; - mtx_lock(&Giant); error = sendit(td, uap->s, &msg, uap->flags); - mtx_unlock(&Giant); return (error); } @@ -794,9 +816,7 @@ aiov.iov_len = uap->len; msg.msg_control = 0; msg.msg_flags = 0; - mtx_lock(&Giant); error = sendit(td, uap->s, &msg, uap->flags); - mtx_unlock(&Giant); return (error); } @@ -816,7 +836,6 @@ struct iovec aiov[UIO_SMALLIOV], *iov; int error; - mtx_lock(&Giant); error = copyin(uap->msg, &msg, sizeof (struct omsghdr)); if (error) goto done2; @@ -842,7 +861,6 @@ if (iov != aiov) FREE(iov, M_IOV); done2: - mtx_unlock(&Giant); return (error); } #endif @@ -863,7 +881,6 @@ struct iovec aiov[UIO_SMALLIOV], *iov; int error; - mtx_lock(&Giant); error = copyin(uap->msg, &msg, sizeof (msg)); if (error) goto done2; @@ -891,7 +908,6 @@ if (iov != aiov) FREE(iov, M_IOV); done2: - mtx_unlock(&Giant); return (error); } Index: sys/sys/syscallsubr.h =================================================================== RCS file: /cvs/FreeBSD-CVS/src/sys/sys/syscallsubr.h,v retrieving revision 1.6 diff -u -r1.6 syscallsubr.h --- sys/sys/syscallsubr.h 3 Feb 2003 17:36:52 -0000 1.6 +++ sys/sys/syscallsubr.h 8 Feb 2003 20:47:07 -0000 @@ -32,6 +32,8 @@ #include struct sockaddr; +struct msghdr; +struct mbuf; int kern___getcwd(struct thread *td, u_char *buf, enum uio_seg bufseg, u_int buflen); @@ -70,6 +72,8 @@ int kern_rmdir(struct thread *td, char *path, enum uio_seg pathseg); int kern_select(struct thread *td, int nd, fd_set *fd_in, fd_set *fd_ou, fd_set *fd_ex, struct timeval *tvp); +int kern_sendit(struct thread *td, int s, struct msghdr *mp, int flags, + struct mbuf *control); int kern_sigaction(struct thread *td, int sig, struct sigaction *act, struct sigaction *oact, int flags); int kern_sigaltstack(struct thread *td, stack_t *ss, stack_t *oss); From owner-freebsd-arch@FreeBSD.ORG Mon Mar 31 13:51:58 2003 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 62C0F37B401; Mon, 31 Mar 2003 13:51:58 -0800 (PST) Received: from rwcrmhc51.attbi.com (rwcrmhc51.attbi.com [204.127.198.38]) by mx1.FreeBSD.org (Postfix) with ESMTP id 9B71343F93; Mon, 31 Mar 2003 13:51:57 -0800 (PST) (envelope-from julian@elischer.org) Received: from interjet.elischer.org (12-232-168-4.client.attbi.com[12.232.168.4]) by rwcrmhc51.attbi.com (rwcrmhc51) with ESMTP id <20030331215155051001dc7ge>; Mon, 31 Mar 2003 21:51:57 +0000 Received: from localhost (localhost.elischer.org [127.0.0.1]) by InterJet.elischer.org (8.9.1a/8.9.1) with ESMTP id NAA99127; Mon, 31 Mar 2003 13:51:53 -0800 (PST) Date: Mon, 31 Mar 2003 13:51:52 -0800 (PST) From: Julian Elischer To: John Baldwin In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: Andrew Gallatin cc: freebsd-arch@freebsd.org Subject: RE: cv_timedwait() & exiting procs X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 31 Mar 2003 21:52:00 -0000 (sounds of all massive context switch while head reloads) On Mon, 31 Mar 2003, John Baldwin wrote: > > On 31-Mar-2003 Andrew Gallatin wrote: > > > > > > FreeBSD's cv_timedwait() function helpfully notices that a process is > > exiting and returns EWOULDBLOCK if it is. > > > > However, if you call cv_timedwait() in the context of a process which > > is already exiting, you always get back EWOULDBLOCK, regardless of > > whether or not the timeout expired. Similarly for the cv_wait_sig() > > and cv_timedwait_sig(), except they set EINTR. > > > > Does anyone else consider this behaviour to be a bug? I think it > > should only return EWOULDBLOCK/EINTR because a process is exiting if > > the process wasn't already exiting when it entered the cv_*wait* > > routine, but perhaps I'm misguided... OK, the thing we are trying to take into account is the fact that a multithreaded program may call exit() in one thread while another thread is in, or about to enter a sleep (either the CV-kind or the normal one). In that situation the thread calling exit() sets a 'single threading' condition in the process and tries to wake up all threads of that process that are in interruptible sleeps. it then (assuming that it is not the only remaining thread already) suspends itself and awaits the other threads to suicide. (no point in letting then go back to userland as it is calling exit()..) Thus it is pointless continuing the syscall so if we are exiting, we always abort the syscall. The sleeps are probably called from some function that can cope with signals and will (we hope) unwind correctly on receiving one. One would think that it would return EINTR in this case. When it unwinds to the user boundary, it will notice that the process is exiting and fall on its sword. Some comments to your thoughts.. It would be a valid thought to allow the syscall to complete if the sleep completed. but why? Also, Maybe the test should also test if we are: 1/ threading and 2/ not the thread that is actually DOING the exit() (though exit doesn't do any sleeps that I KNOW of) > > I think you are referring to these changes? > > revision 1.23 > date: 2002/06/29 17:26:18; author: julian; state: Exp; lines: +76 -13 > Part 1 of KSE-III > > The ability to schedule multiple threads per process > (one one cpu) by making ALL system calls optionally asynchronous. > to come: ia64 and power-pc patches, patches for gdb, test program (in tools) > > Reviewed by: Almost everyone who counts > (at various times, peter, jhb, matt, alfred, mini, bernd, > and a cast of thousands) > > NOTE: this is still Beta code, and contains lots of debugging stuff. > expect slight instability in signals.. > > Things like: > > @@ -293,6 +328,8 @@ > rval = ERESTART; > } > PROC_UNLOCK(p); > + if (p->p_flag & P_WEXIT) > + rval = EINTR; > > #ifdef KTRACE > if (KTRPOINT(td, KTR_CSW)) > @@ -363,6 +400,8 @@ > mi_switch(); > } > > + if (td->td_proc->p_flag & P_WEXIT) > + rval = EWOULDBLOCK; > mtx_unlock_spin(&sched_lock); > #ifdef KTRACE > if (KTRPOINT(td, KTR_CSW)) > @@ -450,6 +488,9 @@ > } > PROC_UNLOCK(p); > > + if (p->p_flag & P_WEXIT) > + rval = EINTR; > + > #ifdef KTRACE > if (KTRPOINT(td, KTR_CSW)) > ktrcsw(0, 0); For the life of me I can't see why I thought it needed to return EWOULDBLOCK in that case, unless there is something special about EWOULDBLOCK processing further downstream. Hmm looks like the original code is in the non-interruptable case and responds with EWOULDBLOCK if it times out so I just did the same. Assuming that the caller would know how to deal with that.. In fact this is the WRONG THING to do.. the testr should be removed becasue returning early may cause the caller to do things like reset hardware and enter timeout routines.. better to let it complete So, what happens in msleep()? if (p->p_flag & P_THREADED) { /* * Just don't bother if we are exiting * and not the exiting thread or thread was marked as * interrupted. */ if (catch && (((p->p_flag & P_WEXIT) && (p->p_singlethread != td)) || (td->td_flags & TDF_INTERRUPT))) { td->td_flags &= ~TDF_INTERRUPT; return (EINTR); } } this is just before we lock the schedlock (and that may be a bug.. maybe it should be after).. but kkkbefore the mi_switch(). After the mi_switch() there is: if (td->td_flags & TDF_TIMEOUT) { td->td_flags &= ~TDF_TIMEOUT; if (sig == 0) rval = EWOULDBLOCK; } else if (td->td_flags & TDF_TIMOFAIL) { td->td_flags &= ~TDF_TIMOFAIL; } else if (timo && callout_stop(&td->td_slpcallout) == 0) { /* * This isn't supposed to be pretty. If we are here, then * the endtsleep() callout is currently executing on another * CPU and is either spinning on the sched_lock or will be * soon. If we don't synchronize here, there is a chance * that this process may msleep() again before the callout * has a chance to run and the callout may end up waking up * the wrong msleep(). Yuck. */ TD_SET_SLEEPING(td); p->p_stats->p_ru.ru_nivcsw++; mi_switch(); td->td_flags &= ~TDF_TIMOFAIL; } if ((td->td_flags & TDF_INTERRUPT) && (priority & PCATCH) && (rval == 0)) { td->td_flags &= ~TDF_INTERRUPT; rval = EINTR; } mtx_unlock_spin(&sched_lock); if (rval == 0 && catch) { PROC_LOCK(p); /* XXX: shouldn't we always be calling cursig() */ if (sig != 0 || (sig = cursig(td))) { if (SIGISMEMBER(p->p_sigacts->ps_sigintr, sig)) rval = EINTR; else rval = ERESTART; } PROC_UNLOCK(p); } We seem to rely on the fact that if another thread aborted us, we should end up returning EINTR anyhow.. But that will not be true unless TDF_INTERRUPT is also set. so that looks a bit like a bug to me... The result would be that the caller will get returned to early, but if it is written well, it should call msleep again, which will then return EINTR, so its probably benign. I think there are bugs in both msleep and cv_XXX in msleep, I think it may return to the caller without letting the caller know that it was aborted, and in cv_xxx it may return to the caller having aborted in a non threaded app which would be 'bad' and it may also return a bad result in the case of a non-interruptable wait rather than letting it return it's result. BTW I think it would be more readble if the cv_wait stuff were consulidated like the msleep() code.. one gets confused following all those twisty little tunnels.... (*) > > I guess this has something to do with single threading? If so it > seems broken since it should only affect the singlethreading case > while you are actually doing singlethreading. Maybe Julian can help > fix it? > > -- > > John Baldwin <>< http://www.FreeBSD.org/~jhb/ > "Power Users Use the Power to Serve!" - http://www.FreeBSD.org/ > _______________________________________________ > freebsd-arch@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-arch > To unsubscribe, send any mail to "freebsd-arch-unsubscribe@freebsd.org" > From owner-freebsd-arch@FreeBSD.ORG Mon Mar 31 14:21:06 2003 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 4BC2037B401; Mon, 31 Mar 2003 14:21:06 -0800 (PST) Received: from duke.cs.duke.edu (duke.cs.duke.edu [152.3.140.1]) by mx1.FreeBSD.org (Postfix) with ESMTP id 7619F43F75; Mon, 31 Mar 2003 14:21:05 -0800 (PST) (envelope-from gallatin@cs.duke.edu) Received: from grasshopper.cs.duke.edu (grasshopper.cs.duke.edu [152.3.145.30]) by duke.cs.duke.edu (8.12.9/8.12.9) with ESMTP id h2VMKvMS020892 (version=TLSv1/SSLv3 cipher=EDH-RSA-DES-CBC3-SHA bits=168 verify=NO); Mon, 31 Mar 2003 17:20:58 -0500 (EST) Received: (from gallatin@localhost) by grasshopper.cs.duke.edu (8.11.6/8.9.1) id h2VMKqk27630; Mon, 31 Mar 2003 17:20:52 -0500 (EST) (envelope-from gallatin@cs.duke.edu) From: Andrew Gallatin MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <16008.48964.858472.638793@grasshopper.cs.duke.edu> Date: Mon, 31 Mar 2003 17:20:52 -0500 (EST) To: Julian Elischer In-Reply-To: References: X-Mailer: VM 6.75 under 21.1 (patch 12) "Channel Islands" XEmacs Lucid cc: John Baldwin cc: freebsd-arch@FreeBSD.org Subject: RE: cv_timedwait() & exiting procs X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 31 Mar 2003 22:21:08 -0000 Julian Elischer writes: > > Some comments to your thoughts.. It would be a valid thought to allow > the syscall to complete if the sleep completed. but why? Also, Maybe the > test should also test if we are: > > 1/ threading > and > 2/ not the thread that is actually DOING the exit() > (though exit doesn't do any sleeps that I KNOW of) That would be nice, if I understand what you're saying. The scenario that I'm in is that I have character driver. At certain points, we send "commands" to the the device and sleep on a cv. When the device completes the command, it interrupts the host and the interrupt handler does a cv_signal. If the device does not respond after some seconds, we consider the device dead. One of these commands is issued when the device is closed. The problem occurs when an application does not call close himself, rather he just exits, and our driver's close entry point is called from the exit() code path. The "command" associated with device closes is issued in the context of this (single threaded) process. All is well, and the device responds. The interrupt happens. cv_signal() is called. But: cv_timedwait() then returns with EWOULDBLOCK. Because EWOULDBLOCK is returned, the driver assumes the timeout has been exceeded and device has gone dead. Sure, this is easy enough for me to fix in the driver code, but it really makes FreeBSD's condvar interface look busted.. I was thinking that we could only return an error because of P_WEXIT being set if P_WEXIT was not set when the cv call was entered. Drew From owner-freebsd-arch@FreeBSD.ORG Mon Mar 31 14:37:52 2003 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id A194E37B401; Mon, 31 Mar 2003 14:37:52 -0800 (PST) Received: from rwcrmhc53.attbi.com (rwcrmhc53.attbi.com [204.127.198.39]) by mx1.FreeBSD.org (Postfix) with ESMTP id 2687F43FB1; Mon, 31 Mar 2003 14:37:52 -0800 (PST) (envelope-from julian@elischer.org) Received: from interjet.elischer.org (12-232-168-4.client.attbi.com[12.232.168.4]) by rwcrmhc53.attbi.com (rwcrmhc53) with ESMTP id <2003033122375105300f98d6e>; Mon, 31 Mar 2003 22:37:51 +0000 Received: from localhost (localhost.elischer.org [127.0.0.1]) by InterJet.elischer.org (8.9.1a/8.9.1) with ESMTP id OAA99466; Mon, 31 Mar 2003 14:37:49 -0800 (PST) Date: Mon, 31 Mar 2003 14:37:48 -0800 (PST) From: Julian Elischer To: Andrew Gallatin In-Reply-To: <16008.48964.858472.638793@grasshopper.cs.duke.edu> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: John Baldwin cc: freebsd-arch@FreeBSD.org Subject: RE: cv_timedwait() & exiting procs X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 31 Mar 2003 22:37:58 -0000 easiest answer.. Rip out the EWOULD BLOCK clause.. it should NOT be there.. (teh one in John's email) I'm about to check in that change anyhow.. On Mon, 31 Mar 2003, Andrew Gallatin wrote: > > Julian Elischer writes: > > > > Some comments to your thoughts.. It would be a valid thought to allow > > the syscall to complete if the sleep completed. but why? Also, Maybe the > > test should also test if we are: > > > > 1/ threading > > and > > 2/ not the thread that is actually DOING the exit() > > (though exit doesn't do any sleeps that I KNOW of) > > That would be nice, if I understand what you're saying. > > The scenario that I'm in is that I have character driver. At certain > points, we send "commands" to the the device and sleep on a cv. When > the device completes the command, it interrupts the host and the > interrupt handler does a cv_signal. If the device does not respond > after some seconds, we consider the device dead. One of these commands > is issued when the device is closed. > > The problem occurs when an application does not call close himself, > rather he just exits, and our driver's close entry point is called > from the exit() code path. The "command" associated with device > closes is issued in the context of this (single threaded) process. > All is well, and the device responds. The interrupt happens. > cv_signal() is called. > > But: cv_timedwait() then returns with EWOULDBLOCK. Because > EWOULDBLOCK is returned, the driver assumes the timeout has been > exceeded and device has gone dead. > > Sure, this is easy enough for me to fix in the driver code, but it > really makes FreeBSD's condvar interface look busted.. > > I was thinking that we could only return an error because of P_WEXIT > being set if P_WEXIT was not set when the cv call was entered. > > Drew > > > From owner-freebsd-arch@FreeBSD.ORG Mon Mar 31 18:02:48 2003 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 03EE737B401 for ; Mon, 31 Mar 2003 18:02:48 -0800 (PST) Received: from duke.cs.duke.edu (duke.cs.duke.edu [152.3.140.1]) by mx1.FreeBSD.org (Postfix) with ESMTP id 3D10143FB1 for ; Mon, 31 Mar 2003 18:02:47 -0800 (PST) (envelope-from gallatin@cs.duke.edu) Received: from grasshopper.cs.duke.edu (grasshopper.cs.duke.edu [152.3.145.30]) by duke.cs.duke.edu (8.12.9/8.12.9) with ESMTP id h3122kMS005442 (version=TLSv1/SSLv3 cipher=EDH-RSA-DES-CBC3-SHA bits=168 verify=NO) for ; Mon, 31 Mar 2003 21:02:46 -0500 (EST) Received: (from gallatin@localhost) by grasshopper.cs.duke.edu (8.11.6/8.9.1) id h3122f727891; Mon, 31 Mar 2003 21:02:41 -0500 (EST) (envelope-from gallatin@cs.duke.edu) From: Andrew Gallatin MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <16008.62273.582001.784737@grasshopper.cs.duke.edu> Date: Mon, 31 Mar 2003 21:02:41 -0500 (EST) To: freebsd-arch@freebsd.org X-Mailer: VM 6.75 under 21.1 (patch 12) "Channel Islands" XEmacs Lucid Subject: uiomove TDF_DEADLKTREAT X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 01 Apr 2003 02:02:48 -0000 While glancing through some other code, I noticed that each time uiomove is called, the sched_lock spinlock is aquired/released twice to set and then clear some state in td_flags (TDF_DEADLOCKTREAT). Are these locks/unlocks in the critical path required for non-vfs callers of uiomove()? (ie, sosend()/soreceive()?) Drew From owner-freebsd-arch@FreeBSD.ORG Tue Apr 1 00:22:34 2003 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 03E2637B407 for ; Tue, 1 Apr 2003 00:22:34 -0800 (PST) Received: from critter.freebsd.dk (critter.freebsd.dk [212.242.86.163]) by mx1.FreeBSD.org (Postfix) with ESMTP id 2FF2243F3F for ; Tue, 1 Apr 2003 00:22:33 -0800 (PST) (envelope-from phk@phk.freebsd.dk) Received: from critter.freebsd.dk (localhost [127.0.0.1]) by critter.freebsd.dk (8.12.8/8.12.8) with ESMTP id h318MUSM031465 for ; Tue, 1 Apr 2003 10:22:31 +0200 (CEST) (envelope-from phk@phk.freebsd.dk) To: arch@freebsd.org From: Poul-Henning Kamp Date: Tue, 01 Apr 2003 10:22:30 +0200 Message-ID: <31464.1049185350@critter.freebsd.dk> Subject: #include and X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 01 Apr 2003 08:22:34 -0000 As we progress down the path of SMPng we will need to include and in more and more files. The current score, not counting nested include cases, they currently are included approx 16% and 19% of all .c files under /sys. My present predicament is that I will probably put a mutex in the bio queue which is defined in , and so far, I've found 20 .c files where I need to add and and I am not yet at a point where LINT compiles. Do we have a plan for these in the future ? I can see three obvious options: A) define them leaf #includes, and #include them from the majority of our .c files. B) Include them nested from other .h files which need them, in my case C) Include them nested from a central .h file like -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk@FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From owner-freebsd-arch@FreeBSD.ORG Tue Apr 1 00:40:49 2003 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id ED87337B401 for ; Tue, 1 Apr 2003 00:40:49 -0800 (PST) Received: from storm.FreeBSD.org.uk (storm.FreeBSD.org.uk [194.242.157.42]) by mx1.FreeBSD.org (Postfix) with ESMTP id 0ABC443FB1 for ; Tue, 1 Apr 2003 00:40:49 -0800 (PST) (envelope-from mark@grondar.org) Received: from storm.FreeBSD.org.uk (Ugrondar@localhost [127.0.0.1]) by storm.FreeBSD.org.uk (8.12.7/8.12.7) with ESMTP id h318elpT087197; Tue, 1 Apr 2003 09:40:47 +0100 (BST) (envelope-from mark@grondar.org) Received: (from Ugrondar@localhost)h318elOD087196; Tue, 1 Apr 2003 09:40:47 +0100 (BST) X-Authentication-Warning: storm.FreeBSD.org.uk: Ugrondar set sender to mark@grondar.org using -f Received: from grondar.org (localhost [127.0.0.1])h318bZ4j060918; Tue, 1 Apr 2003 09:37:35 +0100 (BST) (envelope-from mark@grondar.org) From: Mark Murray Message-Id: <200304010837.h318bZ4j060918@grimreaper.grondar.org> To: Poul-Henning Kamp In-Reply-To: Your message of "Tue, 01 Apr 2003 10:22:30 +0200." <31464.1049185350@critter.freebsd.dk> Date: Tue, 01 Apr 2003 09:37:35 +0100 Sender: mark@grondar.org cc: arch@freebsd.org Subject: Re: #include and X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 01 Apr 2003 08:40:50 -0000 Poul-Henning Kamp writes: > My present predicament is that I will probably put a mutex in the > bio queue which is defined in , and so far, I've found > 20 .c files where I need to add and and > I am not yet at a point where LINT compiles. > > Do we have a plan for these in the future ? I can see three obvious > options: > > A) define them leaf #includes, and #include them from the majority of > our .c files. > > B) Include them nested from other .h files which need them, in my > case > > C) Include them nested from a central .h file like Do you need the whole sys/lock.h and sys/mutex.h? Can you get by with #including sys/_lock.h and/or sys/_mutex.h in sys/bio.h? And possibly following up by adding the non-underscore variants in the hopefully few places where they are actually needed. M -- Mark Murray iumop ap!sdn w,I idlaH From owner-freebsd-arch@FreeBSD.ORG Tue Apr 1 00:44:16 2003 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 5940B37B401 for ; Tue, 1 Apr 2003 00:44:16 -0800 (PST) Received: from critter.freebsd.dk (critter.freebsd.dk [212.242.86.163]) by mx1.FreeBSD.org (Postfix) with ESMTP id 7761443FBF for ; Tue, 1 Apr 2003 00:44:15 -0800 (PST) (envelope-from phk@phk.freebsd.dk) Received: from critter.freebsd.dk (localhost [127.0.0.1]) by critter.freebsd.dk (8.12.8/8.12.8) with ESMTP id h318iASM031615; Tue, 1 Apr 2003 10:44:14 +0200 (CEST) (envelope-from phk@phk.freebsd.dk) To: Mark Murray From: "Poul-Henning Kamp" In-Reply-To: Your message of "Tue, 01 Apr 2003 09:37:35 BST." <200304010837.h318bZ4j060918@grimreaper.grondar.org> Date: Tue, 01 Apr 2003 10:44:10 +0200 Message-ID: <31614.1049186650@critter.freebsd.dk> cc: arch@freebsd.org Subject: Re: #include and X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 01 Apr 2003 08:44:16 -0000 In message <200304010837.h318bZ4j060918@grimreaper.grondar.org>, Mark Murray wr ites: >Poul-Henning Kamp writes: >> My present predicament is that I will probably put a mutex in the >> bio queue which is defined in , and so far, I've found >> 20 .c files where I need to add and and >> I am not yet at a point where LINT compiles. >> >> Do we have a plan for these in the future ? I can see three obvious >> options: >> >> A) define them leaf #includes, and #include them from the majority of >> our .c files. >> >> B) Include them nested from other .h files which need them, in my >> case >> >> C) Include them nested from a central .h file like > >Do you need the whole sys/lock.h and sys/mutex.h? Can you get by with >#including sys/_lock.h and/or sys/_mutex.h in sys/bio.h? And possibly >following up by adding the non-underscore variants in the hopefully >few places where they are actually needed. I can probably get away with the _* versions, but I'd prefer to know what our plans for this sort of situation actually is... -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk@FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From owner-freebsd-arch@FreeBSD.ORG Tue Apr 1 04:26:23 2003 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 001D537B401 for ; Tue, 1 Apr 2003 04:26:22 -0800 (PST) Received: from mailman.zeta.org.au (mailman.zeta.org.au [203.26.10.16]) by mx1.FreeBSD.org (Postfix) with ESMTP id 9EBCB43F75 for ; Tue, 1 Apr 2003 04:26:21 -0800 (PST) (envelope-from bde@zeta.org.au) Received: from katana.zip.com.au (katana.zip.com.au [61.8.7.246]) by mailman.zeta.org.au (8.9.3/8.8.7) with ESMTP id WAA00610; Tue, 1 Apr 2003 22:25:58 +1000 Date: Tue, 1 Apr 2003 22:25:57 +1000 (EST) From: Bruce Evans X-X-Sender: bde@gamplex.bde.org To: Poul-Henning Kamp In-Reply-To: <31464.1049185350@critter.freebsd.dk> Message-ID: <20030401222002.S22396@gamplex.bde.org> References: <31464.1049185350@critter.freebsd.dk> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: arch@freebsd.org Subject: Re: #include and X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 01 Apr 2003 12:26:23 -0000 On Tue, 1 Apr 2003, Poul-Henning Kamp wrote: > As we progress down the path of SMPng we will need to include > and in more and more files. > > The current score, not counting nested include cases, they currently > are included approx 16% and 19% of all .c files under /sys. > > My present predicament is that I will probably put a mutex in the > bio queue which is defined in , and so far, I've found > 20 .c files where I need to add and and > I am not yet at a point where LINT compiles. > > Do we have a plan for these in the future ? I can see three obvious > options: We've mostly followed this plan for the last 23 months: % RCS file: /home/ncvs/src/sys/sys/_mutex.h,v % Working file: _mutex.h % head: 1.9 % ... % ---------------------------- % revision 1.1 % date: 2001/05/01 08:13:17; author: markm; state: Exp; % Undo part of the tangle of having sys/lock.h and sys/mutex.h included in % other "system" header files. % % Also help the deprecation of lockmgr.h by making it a sub-include of % sys/lock.h and removing sys/lockmgr.h form kernel .c files. % % Sort sys/*.h includes where possible in affected files. % % OK'ed by: bde (with reservations) % ---------------------------- *.h should include and its prerequisite if any mutexes are declared, and *.c should include and its prerequisite if any mutext is used (other than to assign it). *.h may hide some of the details using macros like PROC_LOCK(), but should not include primary headers like to do so. Bruce From owner-freebsd-arch@FreeBSD.ORG Tue Apr 1 08:04:52 2003 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id B4A0237B401 for ; Tue, 1 Apr 2003 08:04:52 -0800 (PST) Received: from mail.speakeasy.net (mail13.speakeasy.net [216.254.0.213]) by mx1.FreeBSD.org (Postfix) with ESMTP id 1F08A43F75 for ; Tue, 1 Apr 2003 08:04:52 -0800 (PST) (envelope-from jhb@FreeBSD.org) Received: (qmail 4500 invoked from network); 1 Apr 2003 16:04:54 -0000 Received: from unknown (HELO server.baldwin.cx) ([216.27.160.63]) (envelope-sender )encrypted SMTP for ; 1 Apr 2003 16:04:54 -0000 Received: from laptop.baldwin.cx (gw1.twc.weather.com [216.133.140.1]) by server.baldwin.cx (8.12.8/8.12.8) with ESMTP id h31G4mOv019578; Tue, 1 Apr 2003 11:04:48 -0500 (EST) (envelope-from jhb@FreeBSD.org) Message-ID: X-Mailer: XFMail 1.5.4 on FreeBSD X-Priority: 3 (Normal) Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 8bit MIME-Version: 1.0 In-Reply-To: <31614.1049186650@critter.freebsd.dk> Date: Tue, 01 Apr 2003 11:04:49 -0500 (EST) From: John Baldwin To: Poul-Henning Kamp cc: arch@freebsd.org cc: Mark Murray Subject: Re: #include and X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 01 Apr 2003 16:04:53 -0000 On 01-Apr-2003 Poul-Henning Kamp wrote: > In message <200304010837.h318bZ4j060918@grimreaper.grondar.org>, Mark Murray wr > ites: >>Poul-Henning Kamp writes: >>> My present predicament is that I will probably put a mutex in the >>> bio queue which is defined in , and so far, I've found >>> 20 .c files where I need to add and and >>> I am not yet at a point where LINT compiles. >>> >>> Do we have a plan for these in the future ? I can see three obvious >>> options: >>> >>> A) define them leaf #includes, and #include them from the majority of >>> our .c files. >>> >>> B) Include them nested from other .h files which need them, in my >>> case >>> >>> C) Include them nested from a central .h file like >> >>Do you need the whole sys/lock.h and sys/mutex.h? Can you get by with >>#including sys/_lock.h and/or sys/_mutex.h in sys/bio.h? And possibly >>following up by adding the non-underscore variants in the hopefully >>few places where they are actually needed. > > I can probably get away with the _* versions, but I'd prefer to know > what our plans for this sort of situation actually is... The _lock.h and _mutex.h were the plan and are suitable for nesting in other headers such as sys/bio.h when needed. sys/lock.h and sys/mutex.h should only be included when you need the actual API's rather than just the structure definitions. As another argument, I wouldn't mind having sys/mutex.h and sys/sx.h include sys/lock.h but I'm not sure bde@ would like that. -- John Baldwin <>< http://www.FreeBSD.org/~jhb/ "Power Users Use the Power to Serve!" - http://www.FreeBSD.org/ From owner-freebsd-arch@FreeBSD.ORG Tue Apr 1 08:26:33 2003 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id DFC4737B401; Tue, 1 Apr 2003 08:26:33 -0800 (PST) Received: from storm.FreeBSD.org.uk (storm.FreeBSD.org.uk [194.242.157.42]) by mx1.FreeBSD.org (Postfix) with ESMTP id E6EC143FB1; Tue, 1 Apr 2003 08:26:32 -0800 (PST) (envelope-from mark@grondar.org) Received: from storm.FreeBSD.org.uk (Ugrondar@localhost [127.0.0.1]) by storm.FreeBSD.org.uk (8.12.7/8.12.7) with ESMTP id h31GQWpT091916; Tue, 1 Apr 2003 17:26:32 +0100 (BST) (envelope-from mark@grondar.org) Received: (from Ugrondar@localhost)h31GQWH8091915; Tue, 1 Apr 2003 17:26:32 +0100 (BST) X-Authentication-Warning: storm.FreeBSD.org.uk: Ugrondar set sender to mark@grondar.org using -f Received: from grondar.org (localhost [127.0.0.1])h31GMM4j036254; Tue, 1 Apr 2003 17:22:22 +0100 (BST) (envelope-from mark@grondar.org) From: Mark Murray Message-Id: <200304011622.h31GMM4j036254@grimreaper.grondar.org> To: John Baldwin In-Reply-To: Your message of "Tue, 01 Apr 2003 11:04:49 CDT." Date: Tue, 01 Apr 2003 17:22:22 +0100 Sender: mark@grondar.org cc: arch@FreeBSD.org Subject: Re: #include and X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 01 Apr 2003 16:26:34 -0000 John Baldwin writes: > >>Do you need the whole sys/lock.h and sys/mutex.h? Can you get by with > >>#including sys/_lock.h and/or sys/_mutex.h in sys/bio.h? And possibly > >>following up by adding the non-underscore variants in the hopefully > >>few places where they are actually needed. > > > > I can probably get away with the _* versions, but I'd prefer to know > > what our plans for this sort of situation actually is... > > The _lock.h and _mutex.h were the plan and are suitable for nesting > in other headers such as sys/bio.h when needed. sys/lock.h and sys/mutex.h > should only be included when you need the actual API's rather than just > the structure definitions. As another argument, I wouldn't mind having > sys/mutex.h and sys/sx.h include sys/lock.h but I'm not sure bde@ would > like that. Aren't we trying to remove sys/lock.h? If so, cant we move its contents elsewhere? M -- Mark Murray iumop ap!sdn w,I idlaH From owner-freebsd-arch@FreeBSD.ORG Tue Apr 1 08:44:09 2003 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id C5C7137B401 for ; Tue, 1 Apr 2003 08:44:09 -0800 (PST) Received: from mail.speakeasy.net (mail15.speakeasy.net [216.254.0.215]) by mx1.FreeBSD.org (Postfix) with ESMTP id 32AA443FA3 for ; Tue, 1 Apr 2003 08:44:09 -0800 (PST) (envelope-from jhb@FreeBSD.org) Received: (qmail 30228 invoked from network); 1 Apr 2003 16:44:12 -0000 Received: from unknown (HELO server.baldwin.cx) ([216.27.160.63]) (envelope-sender )encrypted SMTP for ; 1 Apr 2003 16:44:12 -0000 Received: from laptop.baldwin.cx (gw1.twc.weather.com [216.133.140.1]) by server.baldwin.cx (8.12.8/8.12.8) with ESMTP id h31Gi5Ov019692; Tue, 1 Apr 2003 11:44:05 -0500 (EST) (envelope-from jhb@FreeBSD.org) Message-ID: X-Mailer: XFMail 1.5.4 on FreeBSD X-Priority: 3 (Normal) Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 8bit MIME-Version: 1.0 In-Reply-To: <200304011622.h31GMM4j036254@grimreaper.grondar.org> Date: Tue, 01 Apr 2003 11:44:06 -0500 (EST) From: John Baldwin To: Mark Murray cc: arch@FreeBSD.org Subject: Re: #include and X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 01 Apr 2003 16:44:10 -0000 On 01-Apr-2003 Mark Murray wrote: > John Baldwin writes: >> >>Do you need the whole sys/lock.h and sys/mutex.h? Can you get by with >> >>#including sys/_lock.h and/or sys/_mutex.h in sys/bio.h? And possibly >> >>following up by adding the non-underscore variants in the hopefully >> >>few places where they are actually needed. >> > >> > I can probably get away with the _* versions, but I'd prefer to know >> > what our plans for this sort of situation actually is... >> >> The _lock.h and _mutex.h were the plan and are suitable for nesting >> in other headers such as sys/bio.h when needed. sys/lock.h and sys/mutex.h >> should only be included when you need the actual API's rather than just >> the structure definitions. As another argument, I wouldn't mind having >> sys/mutex.h and sys/sx.h include sys/lock.h but I'm not sure bde@ would >> like that. > > Aren't we trying to remove sys/lock.h? If so, cant we move its contents > elsewhere? No. sys/lock.h was renamed to sys/lockmgr.h. Files using lockmgr() should be including sys/lockmgr.h and not depending on the nested include in sys/lock.h as that was only for backwards compat and should go away. sys/lock.h defines things like struct lock_class and flags for struct lock_object as well as prototypes for witness, etc. Basically things that are shared across lock primitive such as mutexes, sx, and eventually lockmgr. -- John Baldwin <>< http://www.FreeBSD.org/~jhb/ "Power Users Use the Power to Serve!" - http://www.FreeBSD.org/ From owner-freebsd-arch@FreeBSD.ORG Tue Apr 1 09:16:29 2003 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id B0B8E37B401; Tue, 1 Apr 2003 09:16:29 -0800 (PST) Received: from mailman.zeta.org.au (mailman.zeta.org.au [203.26.10.16]) by mx1.FreeBSD.org (Postfix) with ESMTP id 5095C43FA3; Tue, 1 Apr 2003 09:16:28 -0800 (PST) (envelope-from bde@zeta.org.au) Received: from katana.zip.com.au (katana.zip.com.au [61.8.7.246]) by mailman.zeta.org.au (8.9.3/8.8.7) with ESMTP id DAA19696; Wed, 2 Apr 2003 03:16:25 +1000 Date: Wed, 2 Apr 2003 03:16:24 +1000 (EST) From: Bruce Evans X-X-Sender: bde@gamplex.bde.org To: John Baldwin In-Reply-To: Message-ID: <20030402031028.F23618@gamplex.bde.org> References: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: arch@freebsd.org cc: Poul-Henning Kamp cc: Mark Murray Subject: Re: #include and X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 01 Apr 2003 17:16:30 -0000 On Tue, 1 Apr 2003, John Baldwin wrote: > ... As another argument, I wouldn't mind having > sys/mutex.h and sys/sx.h include sys/lock.h but I'm not sure bde@ would > like that. Of course I wouldn't. Esepecially sys/sx.h. Users of its interfaces need little more than the LOCK_LINE and LOCK_FILE from sys/lock.h. Bruce From owner-freebsd-arch@FreeBSD.ORG Tue Apr 1 22:59:24 2003 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id AF91B37B401; Tue, 1 Apr 2003 22:59:24 -0800 (PST) Received: from whale.sunbay.crimea.ua (whale.sunbay.crimea.ua [212.110.138.65]) by mx1.FreeBSD.org (Postfix) with ESMTP id 44E0E43F75; Tue, 1 Apr 2003 22:59:21 -0800 (PST) (envelope-from ru@whale.sunbay.crimea.ua) Received: from whale.sunbay.crimea.ua (ru@localhost [127.0.0.1]) h326wvAP000531 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Wed, 2 Apr 2003 09:58:57 +0300 (EEST) (envelope-from ru@whale.sunbay.crimea.ua) Received: (from ru@localhost) by whale.sunbay.crimea.ua (8.12.8/8.12.8/Submit) id h326wtAH000517; Wed, 2 Apr 2003 09:58:55 +0300 (EEST) (envelope-from ru) Date: Wed, 2 Apr 2003 09:58:55 +0300 From: Ruslan Ermilov To: Bruce Evans Message-ID: <20030402065855.GA161@sunbay.com> References: <20030329.163343.53040416.imp@bsdimp.com> <20030330150957.M13638@gamplex.bde.org> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="LZvS9be/3tNcYl/X" Content-Disposition: inline In-Reply-To: <20030330150957.M13638@gamplex.bde.org> User-Agent: Mutt/1.5.4i cc: arch@FreeBSD.org cc: Jonathan Lemon cc: "M. Warner Losh" Subject: Re: depend + all vs dependall X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 02 Apr 2003 06:59:25 -0000 --LZvS9be/3tNcYl/X Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sun, Mar 30, 2003 at 04:12:09PM +1000, Bruce Evans wrote: [...] > > time make buildworld -j 8 -s >=20 > Note that all benchmarks using -j are invalid because of nondeterministic > wait times of up to 100 msec for each job. This pessimizes makeworld -j 4 > times by about 20% on a non-dual Athlon XP1600, and can't do good things > for the variance. The pessimization is larger on faster machines of > course. This is fixed in NetBSD. FreeBSD only has the hackaround of > reducing the timeout from 500 msec to 100 msec. >=20 FreeBSD also provides disabled at the moment kqueue(2) based implementation of the above wait. Now that Jonathan is back, perhaps he could look into the issues if they still exist? Cheers, --=20 Ruslan Ermilov Sysadmin and DBA, ru@sunbay.com Sunbay Software AG, ru@FreeBSD.org FreeBSD committer, +380.652.512.251 Simferopol, Ukraine http://www.FreeBSD.org The Power To Serve http://www.oracle.com Enabling The Information Age --LZvS9be/3tNcYl/X Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.1 (FreeBSD) iD8DBQE+ioouUkv4P6juNwoRAr5WAKCAq2DAuabnCe2sTwcS+ic1nXI4IACeJ8Yc aZARHoQS3Z57lvLCnWWr+wQ= =B2l8 -----END PGP SIGNATURE----- --LZvS9be/3tNcYl/X-- From owner-freebsd-arch@FreeBSD.ORG Tue Apr 1 23:55:31 2003 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 0E9F037B401; Tue, 1 Apr 2003 23:55:31 -0800 (PST) Received: from mailman.zeta.org.au (mailman.zeta.org.au [203.26.10.16]) by mx1.FreeBSD.org (Postfix) with ESMTP id 4078343FA3; Tue, 1 Apr 2003 23:55:27 -0800 (PST) (envelope-from bde@zeta.org.au) Received: from katana.zip.com.au (katana.zip.com.au [61.8.7.246]) by mailman.zeta.org.au (8.9.3/8.8.7) with ESMTP id RAA00948; Wed, 2 Apr 2003 17:55:24 +1000 Date: Wed, 2 Apr 2003 17:55:23 +1000 (EST) From: Bruce Evans X-X-Sender: bde@gamplex.bde.org To: Ruslan Ermilov In-Reply-To: <20030402065855.GA161@sunbay.com> Message-ID: <20030402174522.X25929@gamplex.bde.org> References: <20030329.163343.53040416.imp@bsdimp.com> <20030330150957.M13638@gamplex.bde.org> <20030402065855.GA161@sunbay.com> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: arch@FreeBSD.org cc: Jonathan Lemon cc: "M. Warner Losh" Subject: Re: depend + all vs dependall X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 02 Apr 2003 07:55:31 -0000 On Wed, 2 Apr 2003, Ruslan Ermilov wrote: > On Sun, Mar 30, 2003 at 04:12:09PM +1000, Bruce Evans wrote: > [...] > > > time make buildworld -j 8 -s > > > > Note that all benchmarks using -j are invalid because of nondeterministic > > wait times of up to 100 msec for each job. This pessimizes makeworld -j 4 > > times by about 20% on a non-dual Athlon XP1600, and can't do good things > > for the variance. The pessimization is larger on faster machines of > > course. This is fixed in NetBSD. FreeBSD only has the hackaround of > > reducing the timeout from 500 msec to 100 msec. > > > FreeBSD also provides disabled at the moment kqueue(2) based > implementation of the above wait. Now that Jonathan is back, > perhaps he could look into the issues if they still exist? I tried this first. It fixed make and didn't cause any kernel problems here, but is not necessary with correct programming of select(). It might be a micro-optimization relative to select though -- a slopy benchmark showed that it was 0.2% faster for makeworld. References: %%% Index: main.c =================================================================== RCS file: /home/ncvs/src/usr.bin/make/main.c,v retrieving revision 1.81 diff -u -2 -r1.81 main.c --- main.c 17 Dec 2002 04:26:22 -0000 1.81 +++ main.c 21 Mar 2003 23:41:47 -0000 @@ -411,4 +411,8 @@ } +void +catch_child(int sig) +{ +} /*- @@ -452,4 +456,20 @@ /* avoid faults on read-only strings */ static char syspath[] = _PATH_DEFSYSPATH; + + { + /* + * Catch SIGCHLD so that we get kicked out of select() when we + * need to look at a child. This is only known to matter for the + * -j case (perhaps without -P). + * + * XXX this is intentionally misplaced. + */ + struct sigaction sa; + + sigemptyset(&sa.sa_mask); + sa.sa_flags = SA_RESTART | SA_NOCLDSTOP; + sa.sa_handler = catch_child; + sigaction(SIGCHLD, &sa, NULL); + } #ifdef WANT_ENV_MKLVL %%% %%% RCS file: /home/NetBSD/NetBSD-cvs/src/usr.bin/make/job.c,v Working file: job.c head: 1.75 ... ---------------------------- revision 1.73 date: 2002/11/16 22:22:23; author: gson; state: Exp; lines: +98 -57 Fixed race condition that would cause make -j to pause for five seconds if a SIGCHLD arrived while make was not blocked in poll(), by making the SIGCHLD handler write to a pipe included in the poll. Avoided the need to implement a duplicate fix for the USE_SELECT case by emulating poll() in terms of select() when USE_SELECT is defined. Fixes bin/18895. ---------------------------- ... ---------------------------- revision 1.37 date: 2000/12/05 15:20:10; author: sommerfeld; state: Exp; lines: +35 -4 correct performance regression of recent change from select() to poll() for parallel make: - Make the poll() code behave more like the select() code: sleep for a bit waiting for output rather than busy-wait (eww). - Install a no-op SIGCHLD handler so that poll/select wake up early (with -1/EINTR) when a child exits. - Change the default sleep time from 500ms to 5 seconds since we now wake up promptly when a child exits. ---------------------------- %%% %%% RCS file: /home/ncvs/src/usr.bin/make/job.h,v Working file: job.h head: 1.20 ... ---------------------------- revision 1.8 date: 1997/04/21 20:32:11; author: phk; state: Exp; lines: +2 -2 branches: 1.8.2; In these XXX MHz days, waiting 500ms for a process to do something is really far too long. Let us try 100ms instead, if you have a PP200, maybe that's even too long. This should speed up make -j# builds. I wonder why SIGCHLD isn't used... ---------------------------- %%% Bruce From owner-freebsd-arch@FreeBSD.ORG Thu Apr 3 01:57:17 2003 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 550F237B401 for ; Thu, 3 Apr 2003 01:57:17 -0800 (PST) Received: from park.rambler.ru (park.rambler.ru [81.19.64.101]) by mx1.FreeBSD.org (Postfix) with ESMTP id A67DD43F3F for ; Thu, 3 Apr 2003 01:57:14 -0800 (PST) (envelope-from is@rambler-co.ru) Received: from is.park.rambler.ru (is.park.rambler.ru [81.19.64.102]) by park.rambler.ru (8.12.6/8.12.6) with ESMTP id h339v6mF099450; Thu, 3 Apr 2003 13:57:06 +0400 (MSD) Date: Thu, 3 Apr 2003 13:57:06 +0400 (MSD) From: Igor Sysoev X-Sender: is@is To: freebsd-arch@freebsd.org Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Subject: Re: libthr and 1:1 threading. X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 03 Apr 2003 09:57:17 -0000 Terry Lambert wrote: >> Well, they're both fixes. Another issue for applications that are >> threaded and may be bumping up against the system memory limits is whether >> or not the whole process stalls on a page fault or memory mapping fault, >> or whether it's just the thread. >This is what I meant by "deeper in the system calll layer". IMO, >if you are stalled on something like this on an async fd, then it >should queue the fault anyway, and return to user space for the >next request. This may just be a bug in the kernel processing of >demand faults on vnodes associated with async fd's (FWIW, System V >and Solaris both queue the fault for kernel processing, and then >return to user space). If a process caused a page fault or memory mapping fault at user level where do you suppose to return in user space after a fault was just queued ? To the same instruction that caused this fault ? With threads you can run another thread in such situation. BTW what do you mean by 'async fd' in Solaris ? O_ASYNC ? I do not see it in Solaris 8. O_NONBLOCK ? It does not matter for disk files. aioread() or aio_read() ? They are library calls that implemented via additional LWP for regular disk files. >> Certainly, you can argue that the application should be structured >> to make all I/O explicit and asynchronous, but for various reasons, >> that's not the case :-). >The mmap'ed file case is obviously not something that can be >handled without an explicit contract between user and kernel >for notification of the pagein temporary failure (I would use >a signal for that, probably, as a gross first approximation, >but per-process signal handling is currently not happy...). And what do you suppose to do in a signal handler ? Using some non-reenterant library functions ? Igor Sysoev http://sysoev.ru/en/ From owner-freebsd-arch@FreeBSD.ORG Thu Apr 3 03:21:36 2003 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id A92C737B401 for ; Thu, 3 Apr 2003 03:21:36 -0800 (PST) Received: from stork.mail.pas.earthlink.net (stork.mail.pas.earthlink.net [207.217.120.188]) by mx1.FreeBSD.org (Postfix) with ESMTP id 17C2843FAF for ; Thu, 3 Apr 2003 03:21:36 -0800 (PST) (envelope-from tlambert2@mindspring.com) Received: from pool0015.cvx22-bradley.dialup.earthlink.net ([209.179.198.15] helo=mindspring.com) by stork.mail.pas.earthlink.net with asmtp (SSLv3:RC4-MD5:128) (Exim 3.33 #1) id 1912mh-0005sn-00; Thu, 03 Apr 2003 03:21:32 -0800 Message-ID: <3E8C18CC.AF2C6B7F@mindspring.com> Date: Thu, 03 Apr 2003 03:19:40 -0800 From: Terry Lambert X-Mailer: Mozilla 4.79 [en] (Win98; U) X-Accept-Language: en MIME-Version: 1.0 To: Igor Sysoev References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-ELNK-Trace: b1a02af9316fbb217a47c185c03b154d40683398e744b8a4718c3e02cdc50a5e8ae02ba12f90e767548b785378294e88350badd9bab72f9c350badd9bab72f9c cc: freebsd-arch@freebsd.org Subject: Re: libthr and 1:1 threading. X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 03 Apr 2003 11:21:37 -0000 Igor Sysoev wrote: > If a process caused a page fault or memory mapping fault at user level > where do you suppose to return in user space after a fault was just queued ? > To the same instruction that caused this fault ? Yes. And then return as if the fault had failed. Only it's not fatal, because you only return out of the code if the fd was marked async; otherwise you go to sleep on the buffer getting filled in by an I/O initiated by the fault. If you do return, you don't crash the process, you return with EAGAIN. The trap handler provides the context for the delayed operation. > With threads you can run another thread in such situation. Yes. And save the 20ms per fault that Robert Watson estimated was the reason for the performance difference between libc_r ("user space threads") and libthr ("1:1 kernel threads"). > BTW what do you mean by 'async fd' in Solaris ? > O_ASYNC ? I do not see it in Solaris 8. > O_NONBLOCK ? It does not matter for disk files. O_NONBLOCK. Examining the Solaris 8 sources, it seems to have been removed from the disk I/O modules, and applies only to socktpi.c, ptm.c, audio_mc.c, ecpp.c, and envctrl.c. Apparently, it's also missing from the tty code, which goes against what Matt claimed, actually. This is unfortunate, in that it leaves me without an easily accessible example, unless you have a USL UNIX source license? Is there any chance you have legal access to the Solaris 2.2 or even the 2.4 source code, which is immediately following the project for integration between USL and SunSoft? Or the USL SVR4.0.2 or SVR4.2 source code? > aioread() or aio_read() ? They are library calls that implemented > via additional LWP for regular disk files. I know this. This was basically what Julian and Matt had discussed as a means of implementing AIO in FreeBSD, rather than using system calls. > >> Certainly, you can argue that the application should be structured > >> to make all I/O explicit and asynchronous, but for various reasons, > >> that's not the case :-). > > > >The mmap'ed file case is obviously not something that can be > >handled without an explicit contract between user and kernel > >for notification of the pagein temporary failure (I would use > >a signal for that, probably, as a gross first approximation, > >but per-process signal handling is currently not happy...). > > And what do you suppose to do in a signal handler ? > Using some non-reenterant library functions ? No. Call the user thread scheduler as a result of a fault that is normally not trappable because it resulted from a memory access to an mmap()'ed region of the address space, rather than resulting from an explicit system call. There is no system call context when a trap like that occurs, there is only a trap context. A signal would allow you to force a user threads context switch for a thread whose only reason it can't run is that running it would result in a page fault and delay all the other runnable threads that aren't waiting on a condition that would result in a page fault. The signal is just to get back to user space so you can force the faulting thread to yield and restart the operation by being rescheduled later, after the fault has been satisfied by the kernel's I/O subsystem. -- Terry From owner-freebsd-arch@FreeBSD.ORG Thu Apr 3 05:32:10 2003 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id B562537B401 for ; Thu, 3 Apr 2003 05:32:10 -0800 (PST) Received: from park.rambler.ru (park.rambler.ru [81.19.64.101]) by mx1.FreeBSD.org (Postfix) with ESMTP id 4EC8643F85 for ; Thu, 3 Apr 2003 05:32:09 -0800 (PST) (envelope-from is@rambler-co.ru) Received: from is.park.rambler.ru (is.park.rambler.ru [81.19.64.102]) by park.rambler.ru (8.12.6/8.12.6) with ESMTP id h33DW7mF003564; Thu, 3 Apr 2003 17:32:07 +0400 (MSD) Date: Thu, 3 Apr 2003 17:32:07 +0400 (MSD) From: Igor Sysoev X-Sender: is@is To: Terry Lambert In-Reply-To: <3E8C18CC.AF2C6B7F@mindspring.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: freebsd-arch@freebsd.org Subject: Re: libthr and 1:1 threading. X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 03 Apr 2003 13:32:11 -0000 On Thu, 3 Apr 2003, Terry Lambert wrote: > Igor Sysoev wrote: > > If a process caused a page fault or memory mapping fault at user level > > where do you suppose to return in user space after a fault was just queued ? > > To the same instruction that caused this fault ? > > Yes. And then return as if the fault had failed. > > Only it's not fatal, because you only return out of the code if > the fd was marked async; otherwise you go to sleep on the buffer > getting filled in by an I/O initiated by the fault. If you do > return, you don't crash the process, you return with EAGAIN. I did not say about read() operation. I said about some user land CPU instruction that accessed a page that is not in a memory. This instruction causes page fault. > > BTW what do you mean by 'async fd' in Solaris ? > > O_ASYNC ? I do not see it in Solaris 8. > > O_NONBLOCK ? It does not matter for disk files. > > O_NONBLOCK. > > Examining the Solaris 8 sources, it seems to have been removed from > the disk I/O modules, and applies only to socktpi.c, ptm.c, audio_mc.c, > ecpp.c, and envctrl.c. Apparently, it's also missing from the tty > code, which goes against what Matt claimed, actually. If Solaris 8's stdin handled by tty code then tty supports O_NONBLOCK: ----- >uname -imprsv SunOS 5.8 Generic_108529-17 i86pc i386 i86pc >cat t1.c #include #include #include main() { int rc; char buf[1024]; if ((rc = fcntl(0, F_GETFL)) == -1) { printf("fcntl(F_GETFL) failed: %d: %s\n", errno, strerror(errno)); exit(1); } if ((rc = fcntl(0, F_SETFL, rc | O_NONBLOCK)) == -1) { printf("fcntl(F_SETFL) failed: %d: %s\n", errno, strerror(errno)); exit(1); } if ((rc = read(0, buf, 1024)) == -1) { printf("read() failed: %d: %s\n", errno, strerror(errno)); } } >gcc -o t1 t1.c >./t1 read() failed: 11: Resource temporarily unavailable ----- > This is unfortunate, in that it leaves me without an easily > accessible example, unless you have a USL UNIX source license? > > Is there any chance you have legal access to the Solaris 2.2 or > even the 2.4 source code, which is immediately following the project > for integration between USL and SunSoft? This Solaris 2.4's readv() man http://www.doc.ic.ac.uk/~mac/manuals/solaris-manual-pages/solaris/usr/man/man2/read.2.html states that O_NONBLOCKed regular disk file can return EAGAIN only if there is a mandatory file/record lock. Solaris 8's man states the same. > Or the USL SVR4.0.2 or SVR4.2 source code? No, I have not an access to this source code. But if you have an access to these systems it's easy to check whethear these systems supports O_NONBLOCK on disk files or not. > > >> Certainly, you can argue that the application should be structured > > >> to make all I/O explicit and asynchronous, but for various reasons, > > >> that's not the case :-). > > > > > >The mmap'ed file case is obviously not something that can be > > >handled without an explicit contract between user and kernel > > >for notification of the pagein temporary failure (I would use > > >a signal for that, probably, as a gross first approximation, > > >but per-process signal handling is currently not happy...). > > > > And what do you suppose to do in a signal handler ? > > Using some non-reenterant library functions ? > > No. Call the user thread scheduler as a result of a fault that > is normally not trappable because it resulted from a memory access > to an mmap()'ed region of the address space, rather than resulting > from an explicit system call. There is no system call context when > a trap like that occurs, there is only a trap context. This is the same thing that KSE M:N model does with page faults. It upcalls UTS when a thread blocked on a pagein operation. Igor Sysoev http://sysoev.ru/en/ From owner-freebsd-arch@FreeBSD.ORG Thu Apr 3 06:23:06 2003 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 320B937B401 for ; Thu, 3 Apr 2003 06:23:06 -0800 (PST) Received: from stork.mail.pas.earthlink.net (stork.mail.pas.earthlink.net [207.217.120.188]) by mx1.FreeBSD.org (Postfix) with ESMTP id 8F1FB43FCB for ; Thu, 3 Apr 2003 06:23:05 -0800 (PST) (envelope-from tlambert2@mindspring.com) Received: from pool0015.cvx22-bradley.dialup.earthlink.net ([209.179.198.15] helo=mindspring.com) by stork.mail.pas.earthlink.net with asmtp (SSLv3:RC4-MD5:128) (Exim 3.33 #1) id 1915cL-00019U-00; Thu, 03 Apr 2003 06:23:02 -0800 Message-ID: <3E8C42EE.C3602A39@mindspring.com> Date: Thu, 03 Apr 2003 06:19:26 -0800 From: Terry Lambert X-Mailer: Mozilla 4.79 [en] (Win98; U) X-Accept-Language: en MIME-Version: 1.0 To: Igor Sysoev References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-ELNK-Trace: b1a02af9316fbb217a47c185c03b154d40683398e744b8a4e0dfef6ba63c346251571c943064c03a350badd9bab72f9c350badd9bab72f9c350badd9bab72f9c cc: freebsd-arch@freebsd.org Subject: Re: libthr and 1:1 threading. X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 03 Apr 2003 14:23:06 -0000 Igor Sysoev wrote: > I did not say about read() operation. I said about some user land CPU > instruction that accessed a page that is not in a memory. This instruction > causes page fault. I treated this case seperately, lower down in the message. > > O_NONBLOCK. > > > > Examining the Solaris 8 sources, it seems to have been removed from > > the disk I/O modules, and applies only to socktpi.c, ptm.c, audio_mc.c, > > ecpp.c, and envctrl.c. Apparently, it's also missing from the tty > > code, which goes against what Matt claimed, actually. > > If Solaris 8's stdin handled by tty code then tty supports O_NONBLOCK: [ ... test program ... ] Now try redirecting input from a file. O_NONBLOCK is a flag; the system doesn't care if you are setting it on a file, a tty, the parallel port, or the PTY master; it's just a flag. Matt's first argument is that the flag has no effect on disk files in FreeBSD; I don't dispute that, and never did. His second argument is that if it starts having an effect, there is code that people have written that depends on it not having an effect, and where they handle every possible error return listed on the man page and in the POSIX.1 standard, *except* EAGAIN. And this code will break, and that breakage is a violation of POLA. I happen to disagree with this second argument. > This Solaris 2.4's readv() man > http://www.doc.ic.ac.uk/~mac/manuals/solaris-manual-pages/solaris/usr/man/man2/read.2.html > states that O_NONBLOCKed regular disk file can return EAGAIN only > if there is a mandatory file/record lock. Solaris 8's man states the same. Actually, ity does not say "only". It also says: EAGAIN Total amount of system memory available when reading using raw I/O is temporarily insuffi- cient. Now let me point you to the POSIX standard and the Single UNIX Specification, which says: http://www.opengroup.org/onlinepubs/007908799/xsh/read.html When attempting to read a file (other than a pipe or FIFO) that supports non-blocking reads and has no data currently available: o If O_NONBLOCK is set, read() will return a -1 and set errno to [EAGAIN]. o If O_NONBLOCK is clear, read() will block the calling thread until some data becomes available. o The use of the O_NONBLOCK flag has no effect if there is some data available. Now we are just arguing about "supports non-blocking reads". But it's gets better later in the man page: ERRORS The read(), pread() and readv() functions will fail if: [EAGAIN] The O_NONBLOCK flag is set for the file descriptor and the process would be delayed. This confirms that it is a descriptor attribute, not an underlying object attribute, in the general sense, and that the *intent* is to permit detection and avoidance of delays. > > Or the USL SVR4.0.2 or SVR4.2 source code? > > No, I have not an access to this source code. > But if you have an access to these systems it's easy to check whethear > these systems supports O_NONBLOCK on disk files or not. Try redirecting your test program from a file. It's just a flag. My Solaris 8 source code and Solaris 8 SPARC box both allow it to be set on a file, even if it has no effect. > > > And what do you suppose to do in a signal handler ? > > > Using some non-reenterant library functions ? > > > > No. Call the user thread scheduler as a result of a fault that > > is normally not trappable because it resulted from a memory access > > to an mmap()'ed region of the address space, rather than resulting > > from an explicit system call. There is no system call context when > > a trap like that occurs, there is only a trap context. > > This is the same thing that KSE M:N model does with page faults. > It upcalls UTS when a thread blocked on a pagein operation. Yes. It's another way to leverage the existing kernel KSE code into getting higher performance on a UP box, without going to using libthr instead of libc_r. -- Terry From owner-freebsd-arch@FreeBSD.ORG Thu Apr 3 06:25:28 2003 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 6F14537B401 for ; Thu, 3 Apr 2003 06:25:28 -0800 (PST) Received: from stork.mail.pas.earthlink.net (stork.mail.pas.earthlink.net [207.217.120.188]) by mx1.FreeBSD.org (Postfix) with ESMTP id 0BEE743F85 for ; Thu, 3 Apr 2003 06:25:28 -0800 (PST) (envelope-from tlambert2@mindspring.com) Received: from pool0015.cvx22-bradley.dialup.earthlink.net ([209.179.198.15] helo=mindspring.com) by stork.mail.pas.earthlink.net with asmtp (SSLv3:RC4-MD5:128) (Exim 3.33 #1) id 1915ef-0001hx-00; Thu, 03 Apr 2003 06:25:26 -0800 Message-ID: <3E8C437F.2C81306D@mindspring.com> Date: Thu, 03 Apr 2003 06:21:52 -0800 From: Terry Lambert X-Mailer: Mozilla 4.79 [en] (Win98; U) X-Accept-Language: en MIME-Version: 1.0 To: Igor Sysoev References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-ELNK-Trace: b1a02af9316fbb217a47c185c03b154d40683398e744b8a4e0dfef6ba63c3462089ef6d124c34f6e350badd9bab72f9c350badd9bab72f9c350badd9bab72f9c cc: freebsd-arch@freebsd.org Subject: Re: libthr and 1:1 threading. X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 03 Apr 2003 14:25:28 -0000 I have to ask: Why is it so important to people that the libthr performance gains be impossible to achieve without use of the 1:1 model, rather than a modification of libc_r, or avoidance of existing kernel latencies? -- Terry From owner-freebsd-arch@FreeBSD.ORG Thu Apr 3 07:17:27 2003 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id B665637B401 for ; Thu, 3 Apr 2003 07:17:27 -0800 (PST) Received: from park.rambler.ru (park.rambler.ru [81.19.64.101]) by mx1.FreeBSD.org (Postfix) with ESMTP id 5EF0A43FBF for ; Thu, 3 Apr 2003 07:17:26 -0800 (PST) (envelope-from is@rambler-co.ru) Received: from is.park.rambler.ru (is.park.rambler.ru [81.19.64.102]) by park.rambler.ru (8.12.6/8.12.6) with ESMTP id h33FHKmF005318; Thu, 3 Apr 2003 19:17:24 +0400 (MSD) Date: Thu, 3 Apr 2003 19:17:20 +0400 (MSD) From: Igor Sysoev X-Sender: is@is To: Terry Lambert In-Reply-To: <3E8C42EE.C3602A39@mindspring.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: freebsd-arch@freebsd.org Subject: Re: libthr and 1:1 threading. X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 03 Apr 2003 15:17:28 -0000 On Thu, 3 Apr 2003, Terry Lambert wrote: > > > O_NONBLOCK. > > > > > > Examining the Solaris 8 sources, it seems to have been removed from > > > the disk I/O modules, and applies only to socktpi.c, ptm.c, audio_mc.c, > > > ecpp.c, and envctrl.c. Apparently, it's also missing from the tty > > > code, which goes against what Matt claimed, actually. > > > > If Solaris 8's stdin handled by tty code then tty supports O_NONBLOCK: > > [ ... test program ... ] > > Now try redirecting input from a file. O_NONBLOCK is a flag; the > system doesn't care if you are setting it on a file, a tty, the > parallel port, or the PTY master; it's just a flag. And what do you suppose to see in this case ? ---------- >cat t1.c #include #include #include #include void print_time(char *string, struct timeval *tp0, struct timeval *tp1) { long sec, usec; sec = tp1->tv_sec - tp0->tv_sec; if ((usec = tp1->tv_usec - tp0->tv_usec) < 0) { usec += 1000000; sec--; } printf("%s time: %u.%06u\n", string, sec, usec); } main() { int rc; char buf[1024]; struct timeval tv0, tv1; if ((rc = fcntl(0, F_GETFL)) == -1) { printf("fcntl(F_GETFL) failed: %d: %s\n", errno, strerror(errno)); exit(1); } if ((rc = fcntl(0, F_SETFL, rc | O_NONBLOCK)) == -1) { printf("fcntl(F_SETFL) failed: %d: %s\n", errno, strerror(errno)); exit(1); } gettimeofday(&tv0, NULL); if ((rc = read(0, buf, 1024)) == -1) { printf("read() failed: %d: %s\n", errno, strerror(errno)); } gettimeofday(&tv1, NULL); print_time("read", &tv0, &tv1); } >./t1 read() failed: 11: Resource temporarily unavailable read time: 0.000351 >./t1 < /kernel/genunix read time: 0.005157 >./t1 < /kernel/genunix read time: 0.000019 ---------- When /kernel/genunix was not cached a read() takes 5157 usec's. And no EAGAIN error. > Matt's first argument is that the flag has no effect on disk files > in FreeBSD; I don't dispute that, and never did. Not only in FreeBSD. Linux and Solaris do the same. > His second argument is that if it starts having an effect, there > is code that people have written that depends on it not having an > effect, and where they handle every possible error return listed > on the man page and in the POSIX.1 standard, *except* EAGAIN. And > this code will break, and that breakage is a violation of POLA. > > I happen to disagree with this second argument. The most Unices ignore O_NONBLOCK for disk files. So I think programmers never use this flag and select()/etc for disk files (because it's useless) and I think supporting this flag for disk files should not break existent software. But of course it creates portability problems. > > This Solaris 2.4's readv() man > > http://www.doc.ic.ac.uk/~mac/manuals/solaris-manual-pages/solaris/usr/man/man2/read.2.html > > states that O_NONBLOCKed regular disk file can return EAGAIN only > > if there is a mandatory file/record lock. Solaris 8's man states the same. > > Actually, ity does not say "only". It also says: > > EAGAIN Total amount of system memory available when > reading using raw I/O is temporarily insuffi- > cient. Yes, but there's no word about EAGAIN while a reading from a disk. > > > Or the USL SVR4.0.2 or SVR4.2 source code? > > > > No, I have not an access to this source code. > > But if you have an access to these systems it's easy to check whethear > > these systems supports O_NONBLOCK on disk files or not. > > Try redirecting your test program from a file. It's just a flag. > My Solaris 8 source code and Solaris 8 SPARC box both allow it > to be set on a file, even if it has no effect. See above. Igor Sysoev http://sysoev.ru/en/ From owner-freebsd-arch@FreeBSD.ORG Thu Apr 3 08:05:52 2003 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 0954537B401 for ; Thu, 3 Apr 2003 08:05:52 -0800 (PST) Received: from park.rambler.ru (park.rambler.ru [81.19.64.101]) by mx1.FreeBSD.org (Postfix) with ESMTP id D107643FDD for ; Thu, 3 Apr 2003 08:05:50 -0800 (PST) (envelope-from is@rambler-co.ru) Received: from is.park.rambler.ru (is.park.rambler.ru [81.19.64.102]) by park.rambler.ru (8.12.6/8.12.6) with ESMTP id h33G5nmF006490; Thu, 3 Apr 2003 20:05:49 +0400 (MSD) Date: Thu, 3 Apr 2003 20:05:49 +0400 (MSD) From: Igor Sysoev X-Sender: is@is To: Terry Lambert In-Reply-To: <3E8C437F.2C81306D@mindspring.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: freebsd-arch@freebsd.org Subject: Re: libthr and 1:1 threading. X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 03 Apr 2003 16:05:52 -0000 On Thu, 3 Apr 2003, Terry Lambert wrote: > I have to ask: > > Why is it so important to people that the libthr performance > gains be impossible to achieve without use of the 1:1 model, > rather than a modification of libc_r, or avoidance of existing > kernel latencies? If you mean that "people" is my humble person then I want to say that I am not against 1:1 model libthr, KSE's M:N model, current or improved libc_r, rfork()ed LinuxThreads and any possible future threads implementation. My last emails were caused only by your incorrect statements about a non-blocking behaviour of Solaris disk files. Igor Sysoev http://sysoev.ru/en/ From owner-freebsd-arch@FreeBSD.ORG Thu Apr 3 12:44:37 2003 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 8AC3C37B404 for ; Thu, 3 Apr 2003 12:44:37 -0800 (PST) Received: from canning.wemm.org (canning.wemm.org [192.203.228.65]) by mx1.FreeBSD.org (Postfix) with ESMTP id 396CC43FBF for ; Thu, 3 Apr 2003 12:44:37 -0800 (PST) (envelope-from peter@wemm.org) Received: from wemm.org (localhost [127.0.0.1]) by canning.wemm.org (Postfix) with ESMTP id 22D902A8A5; Thu, 3 Apr 2003 12:44:37 -0800 (PST) (envelope-from peter@wemm.org) X-Mailer: exmh version 2.5 07/13/2001 with nmh-1.0.4 To: Terry Lambert In-Reply-To: <3E8C437F.2C81306D@mindspring.com> Date: Thu, 03 Apr 2003 12:44:37 -0800 From: Peter Wemm Message-Id: <20030403204437.22D902A8A5@canning.wemm.org> cc: freebsd-arch@freebsd.org Subject: Re: libthr and 1:1 threading. X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 03 Apr 2003 20:44:37 -0000 Terry Lambert wrote: > I have to ask: > > Why is it so important to people that the libthr performance > gains be impossible to achieve without use of the 1:1 model, > rather than a modification of libc_r, or avoidance of existing > kernel latencies? Cost vs. Benefit is a big factor. As Jeff has shown, writing a functional 1:1 libthr is significantly easier than M:N or the other alternatives. Easier to write, easier to maintain, easier to understand, easier to debug, and *good enough*. Its all very well for somebody to sit on the sidelines and say what we should be doing, but when it comes down to actually doing it, we're the ones in the hot seat and actually have to write it and maintain it and debug it. kse based threads or async call gates or whatever are almost certainly more efficient at runtime, but we've got to get to the 'running' point first. A mathematical proof is no use to somebody who wants threaded mysql or java or mozilla to work well. Neither is an unstable thread library. At the end of the day working code is what counts, and the more highly ambitious threading systems have to pay the development, debugging and maintenence cost to get there. May the best code win! Cheers, -Peter -- Peter Wemm - peter@wemm.org; peter@FreeBSD.org; peter@yahoo-inc.com "All of this is for nothing if we don't go to the stars" - JMS/B5 From owner-freebsd-arch@FreeBSD.ORG Thu Apr 3 13:54:51 2003 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 7FE0137B401 for ; Thu, 3 Apr 2003 13:54:51 -0800 (PST) Received: from heron.mail.pas.earthlink.net (heron.mail.pas.earthlink.net [207.217.120.189]) by mx1.FreeBSD.org (Postfix) with ESMTP id E0E3843FAF for ; Thu, 3 Apr 2003 13:54:50 -0800 (PST) (envelope-from tlambert2@mindspring.com) Received: from pool0138.cvx21-bradley.dialup.earthlink.net ([209.179.192.138] helo=mindspring.com) by heron.mail.pas.earthlink.net with asmtp (SSLv3:RC4-MD5:128) (Exim 3.33 #1) id 191CfP-0004Oo-00; Thu, 03 Apr 2003 13:54:40 -0800 Message-ID: <3E8CAD52.531E0A9@mindspring.com> Date: Thu, 03 Apr 2003 13:53:22 -0800 From: Terry Lambert X-Mailer: Mozilla 4.79 [en] (Win98; U) X-Accept-Language: en MIME-Version: 1.0 To: Igor Sysoev References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-ELNK-Trace: b1a02af9316fbb217a47c185c03b154d40683398e744b8a48128cbf58f5ba7950a444668b65b6823666fa475841a1c7a350badd9bab72f9c350badd9bab72f9c cc: freebsd-arch@freebsd.org Subject: Re: libthr and 1:1 threading. X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 03 Apr 2003 21:54:51 -0000 Igor Sysoev wrote: > > Now try redirecting input from a file. O_NONBLOCK is a flag; the > > system doesn't care if you are setting it on a file, a tty, the > > parallel port, or the PTY master; it's just a flag. > > And what do you suppose to see in this case ? You don't get an error, attempting to set the flag. When you read the flags, O_NONBLOCK will still be there. > When /kernel/genunix was not cached a read() takes 5157 usec's. > And no EAGAIN error. The point is not that you are not getting the error: we all know that you will *NOT* get the error, because the code does not support non-blocking operations returning EAGAIN in this context. The point is that the flag is not explicitly repudiated by the OS when you attempt to set it on disk files: you do not get an error from the OS when attempting to set the flag. > > Matt's first argument is that the flag has no effect on disk files > > in FreeBSD; I don't dispute that, and never did. > > Not only in FreeBSD. Linux and Solaris do the same. Solaris used to, when it was more SVR4, before Sun forked the code. SVR4 does. I don't care about what Linux does; if they miss an optimization, it's no skin off my nose. This whole thread is a thinly veiled attempt to say that FreeBSD should be more like Linux: it should adopt Linux's threading model, and, now, Linux's other deficiencies. > > His second argument is that if it starts having an effect, there > > is code that people have written that depends on it not having an > > effect, and where they handle every possible error return listed > > on the man page and in the POSIX.1 standard, *except* EAGAIN. And > > this code will break, and that breakage is a violation of POLA. > > > > I happen to disagree with this second argument. > > The most Unices ignore O_NONBLOCK for disk files. Cool with me. So they miss the optimization that's obviously possible in this area. > So I think programmers never use this flag and select()/etc for disk > files (because it's useless) and I think supporting this flag for disk > files should not break existent software. > But of course it creates portability problems. For FreeBSD's libc_r? You have got to be kidding, when you talk about portability of FreeBSD's libc_r! The effect in the old sigsched threads implementation from the comp.sources.unix archive would merely be to provide a latency optimization on the OS's which support it. I think that there is no portability problem; only that it would cause some software to run faster on FreeBSD than on other OS's, were it to be used. > > > states that O_NONBLOCKed regular disk file can return EAGAIN only > > > if there is a mandatory file/record lock. Solaris 8's man states the same. > > > > Actually, it does not say "only". It also says: [ ... ] > Yes, but there's no word about EAGAIN while a reading from a disk. You miss the point. The word "only" has very specific connotations in standards documents: it restricts permissable implementations. Because it did not actually use the word "only", it means that the implementation is not restricted from returning it where the behaviour is not specifically defined. As we see from the POSIX.1 standard quote, which you removed from your reply, it is *permissable* for a conforming UNIX implementation to return the error on disk files. For certain networking and storage subsystems (e.g. HSM), I would expect it to even be a *necessity*. -- Terry From owner-freebsd-arch@FreeBSD.ORG Thu Apr 3 14:03:59 2003 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id BE8C437B401 for ; Thu, 3 Apr 2003 14:03:59 -0800 (PST) Received: from heron.mail.pas.earthlink.net (heron.mail.pas.earthlink.net [207.217.120.189]) by mx1.FreeBSD.org (Postfix) with ESMTP id 0E83143FD7 for ; Thu, 3 Apr 2003 14:03:59 -0800 (PST) (envelope-from tlambert2@mindspring.com) Received: from pool0138.cvx21-bradley.dialup.earthlink.net ([209.179.192.138] helo=mindspring.com) by heron.mail.pas.earthlink.net with asmtp (SSLv3:RC4-MD5:128) (Exim 3.33 #1) id 191CoK-0006J8-00; Thu, 03 Apr 2003 14:03:53 -0800 Message-ID: <3E8CAF7C.7729D532@mindspring.com> Date: Thu, 03 Apr 2003 14:02:36 -0800 From: Terry Lambert X-Mailer: Mozilla 4.79 [en] (Win98; U) X-Accept-Language: en MIME-Version: 1.0 To: Igor Sysoev References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-ELNK-Trace: b1a02af9316fbb217a47c185c03b154d40683398e744b8a403c6c5176e1190e1c6de8bb145b57f9893caf27dac41a8fd350badd9bab72f9c350badd9bab72f9c cc: freebsd-arch@freebsd.org Subject: Re: libthr and 1:1 threading. X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 03 Apr 2003 22:04:00 -0000 Igor Sysoev wrote: > On Thu, 3 Apr 2003, Terry Lambert wrote: > > I have to ask: > > > > Why is it so important to people that the libthr performance > > gains be impossible to achieve without use of the 1:1 model, > > rather than a modification of libc_r, or avoidance of existing > > kernel latencies? > > If you mean that "people" is my humble person then I want to say > that I am not against 1:1 model libthr, KSE's M:N model, current > or improved libc_r, rfork()ed LinuxThreads and any possible future > threads implementation. I didn't mean you specifically. > My last emails were caused only by your incorrect statements about > a non-blocking behaviour of Solaris disk files. When you referenced Solaris 8, rather than contradicting you, I looked at the source code and immediately admitted that I was wrong and you were right, so I don't see why this has continued. Can you do the same with regard to the fact that, even if the current implementation does not use the flag in its implementation of disk I/O, that it does not prohibit setting the flag, and it does not prohibit a third party, such as Veritas or some other disk FS provider, from implementing it? As I said before, I used this flag to great effect on SVR4.0.2 and SVR4.3 in the NetWare for UNIX product in 1994: it provided a 12% performance improvement, by prefaulting the first 9K of Windows executable files stored on Novell servers, and accessed from Windows (Windows repeatedly re-references the first 9K in loading executables, or used to in the WIN32 and Windows95 era). At the time, the effect was measurable on Solaris as well, and it was designed with the source code to the OS in hand for both SVR4 and Solaris, which had just been finished being jointly integrated at the time. That Solaris can't do this any more in more recent versions of Solaris is a loss of functionality, not a feature. -- Terry From owner-freebsd-arch@FreeBSD.ORG Thu Apr 3 14:21:30 2003 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id ED2C937B401 for ; Thu, 3 Apr 2003 14:21:29 -0800 (PST) Received: from puffin.mail.pas.earthlink.net (puffin.mail.pas.earthlink.net [207.217.120.139]) by mx1.FreeBSD.org (Postfix) with ESMTP id 620E043F93 for ; Thu, 3 Apr 2003 14:21:29 -0800 (PST) (envelope-from tlambert2@mindspring.com) Received: from pool0138.cvx21-bradley.dialup.earthlink.net ([209.179.192.138] helo=mindspring.com) by puffin.mail.pas.earthlink.net with asmtp (SSLv3:RC4-MD5:128) (Exim 3.33 #1) id 191D58-0002NE-00; Thu, 03 Apr 2003 14:21:16 -0800 Message-ID: <3E8CB38D.A2C4C2DF@mindspring.com> Date: Thu, 03 Apr 2003 14:19:57 -0800 From: Terry Lambert X-Mailer: Mozilla 4.79 [en] (Win98; U) X-Accept-Language: en MIME-Version: 1.0 To: Peter Wemm References: <20030403204437.22D902A8A5@canning.wemm.org> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-ELNK-Trace: b1a02af9316fbb217a47c185c03b154d40683398e744b8a4f082fe5af17d3672917c59bb067d19dc350badd9bab72f9c350badd9bab72f9c350badd9bab72f9c cc: freebsd-arch@freebsd.org Subject: Re: libthr and 1:1 threading. X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 03 Apr 2003 22:21:30 -0000 Peter Wemm wrote: > Terry Lambert wrote: > > I have to ask: > > > > Why is it so important to people that the libthr performance > > gains be impossible to achieve without use of the 1:1 model, > > rather than a modification of libc_r, or avoidance of existing > > kernel latencies? > > Cost vs. Benefit is a big factor. As Jeff has shown, writing a functional > 1:1 libthr is significantly easier than M:N or the other alternatives. > Easier to write, easier to maintain, easier to understand, easier to debug, > and *good enough*. Its all very well for somebody to sit on the sidelines > and say what we should be doing, but when it comes down to actually doing > it, we're the ones in the hot seat and actually have to write it and > maintain it and debug it. kse based threads or async call gates or > whatever are almost certainly more efficient at runtime, but we've got to > get to the 'running' point first. A mathematical proof is no use to > somebody who wants threaded mysql or java or mozilla to work well. Neither > is an unstable thread library. At the end of the day working code is what > counts, and the more highly ambitious threading systems have to pay the > development, debugging and maintenence cost to get there. The libthr code would not have been possible, without the KSE kernel work which has already been done. Jeff acknowledged this at the time he posted the code for the implementation. At that point, all we are arguing is about who won the race to a userland implementation that at least works. But whatever userland library is there is only possible because of the KSE project efforts so far; e.g. the relationship is: ,-------------------------------------------------------. | | | | | libc_r | KSE pthreads | libthr | | | | |U |-------------------------------------------------------|- | | |K | Old kernel | KSE kernel | | | | `-------------------------------------------------------' > May the best code win! Of course. As it should be. 8-). -- Terry From owner-freebsd-arch@FreeBSD.ORG Thu Apr 3 14:50:25 2003 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 5A99B37B401 for ; Thu, 3 Apr 2003 14:50:25 -0800 (PST) Received: from mail.speakeasy.net (mail13.speakeasy.net [216.254.0.213]) by mx1.FreeBSD.org (Postfix) with ESMTP id 6C54F43F3F for ; Thu, 3 Apr 2003 14:50:24 -0800 (PST) (envelope-from jhb@FreeBSD.org) Received: (qmail 30759 invoked from network); 3 Apr 2003 22:50:32 -0000 Received: from unknown (HELO server.baldwin.cx) ([216.27.160.63]) (envelope-sender )encrypted SMTP for ; 3 Apr 2003 22:50:32 -0000 Received: from laptop.baldwin.cx (gw1.twc.weather.com [216.133.140.1]) by server.baldwin.cx (8.12.8/8.12.8) with ESMTP id h33MoIOv028558; Thu, 3 Apr 2003 17:50:18 -0500 (EST) (envelope-from jhb@FreeBSD.org) Message-ID: X-Mailer: XFMail 1.5.4 on FreeBSD X-Priority: 3 (Normal) Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 8bit MIME-Version: 1.0 In-Reply-To: <3E8C437F.2C81306D@mindspring.com> Date: Thu, 03 Apr 2003 17:50:18 -0500 (EST) From: John Baldwin To: Terry Lambert cc: freebsd-arch@freebsd.org Subject: Re: libthr and 1:1 threading. X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 03 Apr 2003 22:50:25 -0000 On 03-Apr-2003 Terry Lambert wrote: > I have to ask: > > Why is it so important to people that the libthr performance > gains be impossible to achieve without use of the 1:1 model, > rather than a modification of libc_r, or avoidance of existing > kernel latencies? I have to ask: Why it is so important to you that the libthr performance gains be possible to achieve in some other obscure fashion? So FreeBSD finally has a working kernel-based threading implementation and it's not your pet async call gate design. Get over it! It's not rfork processes masquerading as threads, it's real threads. In the kernel. End of story. Please stop spreading FUD and drop this thread. Your contribution to it has ceased to be productive. -- John Baldwin <>< http://www.FreeBSD.org/~jhb/ "Power Users Use the Power to Serve!" - http://www.FreeBSD.org/ From owner-freebsd-arch@FreeBSD.ORG Fri Apr 4 11:22:50 2003 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id BA75837B401 for ; Fri, 4 Apr 2003 11:22:50 -0800 (PST) Received: from mail.redlinenetworks.com (mail.redlinenetworks.com [216.136.145.172]) by mx1.FreeBSD.org (Postfix) with ESMTP id 1522243F75 for ; Fri, 4 Apr 2003 11:22:50 -0800 (PST) (envelope-from sreekanth@redlinenetworks.com) Received: from SREELAPTOP (dhcp-170.redlinenetworks.com [192.168.40.170]) by mail.redlinenetworks.com (8.11.6/8.11.1) with ESMTP id h34JMnw05990 for ; Fri, 4 Apr 2003 11:22:50 -0800 (PST) (envelope-from sreekanth@redlinenetworks.com) From: "Sreekanth" To: Date: Fri, 4 Apr 2003 11:22:51 -0800 Message-ID: <00b101c2fadf$959eab90$aa28a8c0@SREELAPTOP> MIME-Version: 1.0 Content-Type: text/plain; charset="US-ASCII" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook, Build 10.0.2616 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2800.1106 Importance: Normal Subject: Mmap and malloc X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 04 Apr 2003 19:22:51 -0000 Hi, I have a situation i am not sure i understand.I am running a 4.6.2 FreeBSD system with a Dual Xeon board with 2GB of Physical memory.I have a test program which continuoulsy calls mmap(with larger size each time) till it receives MAP_FAILED return value. I then call malloc(with bigger memory size each time) continuosly till it fails.Here are my results. Max mem mapped = 714 MB Max mem alloc = 308 MB My MAXDMSIZ is 1200*1024*1024 and KVA_PAGE is 512 (2GB) If i swap the order in wich mmap and malloc are called, then i get the following result. Max mem alloc = 1200MB Max mem mapped = 711MB My questions are as follows 1) Why is the difference between two approaches. ? 2) What exactly is the relationship between mmap malloc and KVA_PAGES. 3) I found that the Maximum mmapped memory = 4GB - KVA_PAGES*PAGE_SIZE - MAXDSIZ - 134 What is the 134 value..? It is very consistent for me for various values of KVA_PAGES and MAXDSIZ, so i am ruling it being just a coincidence. I hope somebody has already seen this kind of behaviour and therefore can help me out.. Thanks in advance, Sreekanth