From owner-freebsd-arch Sun Jan 5 10: 7:11 2003 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 0B12437B409; Sun, 5 Jan 2003 10:07:09 -0800 (PST) Received: from snickers.greenwood.net (snickers.greenwood.net [205.247.45.3]) by mx1.FreeBSD.org (Postfix) with ESMTP id 1B4B743EDC; Sun, 5 Jan 2003 10:07:08 -0800 (PST) (envelope-from jamieclarcksong@yahoo.com) Received: from yahoo.co.uk (skor.greenwood.net [205.247.45.7]) by snickers.greenwood.net (8.12.5/8.12.5) with SMTP id h05GEIGT007332; Sun, 5 Jan 2003 13:11:06 -0500 Date: Sun, 5 Jan 2003 13:11:06 -0500 Message-Id: <200301051811.h05GEIGT007332@snickers.greenwood.net> From: jamieclarcksong@yahoo.com Reply-To: jamieclarcksong@yahoo.com To: jamieclarcksong@yahoo.com Subject: Examples of School Web Sites X-MailScanner: Found to be clean X-MailScanner-SpamCheck: not spam, SpamAssassin (score=4, required 5, FORGED_YAHOO_RCVD, FROM_AND_TO_SAME_1, NO_REAL_NAME, SPAM_PHRASE_00_01) X-MailScanner-SpamScore: ssss Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG Found some other school web sites that we may want to compare ours too! http://bearcat.ubly.k12.mi.us/links/links.html To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Mon Jan 6 3:42:36 2003 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 1221537B401 for ; Mon, 6 Jan 2003 03:42:35 -0800 (PST) Received: from mailman.zeta.org.au (mailman.zeta.org.au [203.26.10.16]) by mx1.FreeBSD.org (Postfix) with ESMTP id 6BDA043EB2 for ; Mon, 6 Jan 2003 03:42:33 -0800 (PST) (envelope-from bde@zeta.org.au) Received: from katana.zip.com.au (katana.zip.com.au [61.8.7.246]) by mailman.zeta.org.au (8.9.3/8.8.7) with ESMTP id WAA30703 for ; Mon, 6 Jan 2003 22:42:26 +1100 Date: Mon, 6 Jan 2003 22:42:50 +1100 (EST) From: Bruce Evans X-X-Sender: bde@gamplex.bde.org To: arch@freebsd.org Subject: uarea/kstack/pcb/swapout sillyness Message-ID: <20030106220745.Q2670-100000@gamplex.bde.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG Kernel stack pages don't seem to be handled by vm_proc_swap{in,out}. Only the upages are swapped. This makes having a user area separate from the proc struct even less useful than it used to be. We have considerable complications to support the dubious feature of swapping the user area. Swapping it out saves a whole 1 page per process (UAREA_PAGE is 1 for all arches). OTOH, we don't bother swapping out the much larger kernel stacks. There are 4 pages per stack on some arches and there may be multiple stacks per process (1 per thread). Having a separate user area actually wastes memory in the usual (?) case where the user area is not swapped. The user area now contains only struct sigacts, which has size 2688, and struct pstats, which has size 224, so 4096 - 2688 - 224 = 1184 bytes per process is wasted for the struct (not counting costs for indirections). There seems to be even less point to keeping the pcb separate from the thread struct. It saves space only by stealing stack space, and costs indirections. Some PS_INMEM checks are now bogus because the pcb is not in the user area so it doesn't get swapped. I use "options NO_SWAPPING" to prevent swapping of upages, and would like to make this the default and later remove its code and merge the structs. Swapping of upages seems to be least useful on large systems where it would save the most memory, since such systems might actually want to spend time running the processes instead of swapping the them see the long message for the commit that introduced NO_SWAPPING in 1996). With today's larger memories and bloated processes and kernel stacks, I think the space/time savings (positive or negative) don't matter and we just have useless complications to swap upages. Bruce To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Mon Jan 6 5:21:23 2003 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id E99C937B401 for ; Mon, 6 Jan 2003 05:21:22 -0800 (PST) Received: from critter.freebsd.dk (critter.freebsd.dk [212.242.86.163]) by mx1.FreeBSD.org (Postfix) with ESMTP id 0970943EC5 for ; Mon, 6 Jan 2003 05:21:22 -0800 (PST) (envelope-from phk@freebsd.org) Received: from critter.freebsd.dk (localhost [127.0.0.1]) by critter.freebsd.dk (8.12.6/8.12.6) with ESMTP id h06DL6Hj018804; Mon, 6 Jan 2003 14:21:13 +0100 (CET) (envelope-from phk@freebsd.org) To: Bruce Evans Cc: arch@freebsd.org Subject: Re: uarea/kstack/pcb/swapout sillyness From: phk@freebsd.org In-Reply-To: Your message of "Mon, 06 Jan 2003 22:42:50 +1100." <20030106220745.Q2670-100000@gamplex.bde.org> Date: Mon, 06 Jan 2003 14:21:06 +0100 Message-ID: <18803.1041859266@critter.freebsd.dk> Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG In message <20030106220745.Q2670-100000@gamplex.bde.org>, Bruce Evans writes: >I use "options NO_SWAPPING" to prevent swapping of upages, and would >like to make this the default and later remove its code and merge the >structs. You got my vote. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk@FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Mon Jan 6 8:55:41 2003 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id D701937B401 for ; Mon, 6 Jan 2003 08:55:40 -0800 (PST) Received: from HAL9000.homeunix.com (12-233-57-224.client.attbi.com [12.233.57.224]) by mx1.FreeBSD.org (Postfix) with ESMTP id 5BD8543E4A for ; Mon, 6 Jan 2003 08:55:40 -0800 (PST) (envelope-from dschultz@uclink.Berkeley.EDU) Received: from HAL9000.homeunix.com (localhost [127.0.0.1]) by HAL9000.homeunix.com (8.12.6/8.12.5) with ESMTP id h06GtdcA000993; Mon, 6 Jan 2003 08:55:39 -0800 (PST) (envelope-from dschultz@uclink.Berkeley.EDU) Received: (from das@localhost) by HAL9000.homeunix.com (8.12.6/8.12.5/Submit) id h06GtcOJ000992; Mon, 6 Jan 2003 08:55:38 -0800 (PST) (envelope-from dschultz@uclink.Berkeley.EDU) Date: Mon, 6 Jan 2003 08:55:38 -0800 From: David Schultz To: Bruce Evans Cc: arch@FreeBSD.ORG Subject: Re: uarea/kstack/pcb/swapout sillyness Message-ID: <20030106165538.GB721@HAL9000.homeunix.com> Mail-Followup-To: Bruce Evans , arch@FreeBSD.ORG References: <20030106220745.Q2670-100000@gamplex.bde.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20030106220745.Q2670-100000@gamplex.bde.org> Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG Thus spake Bruce Evans : > I use "options NO_SWAPPING" to prevent swapping of upages, and would > like to make this the default and later remove its code and merge the > structs. Please! Handling UPAGES was a PITA for implementing swapoff(), and there's far too much code for swapping than swapping is worth. To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Mon Jan 6 10: 0:27 2003 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 9C2F137B401 for ; Mon, 6 Jan 2003 10:00:25 -0800 (PST) Received: from k6.locore.ca (k6.locore.ca [198.96.117.170]) by mx1.FreeBSD.org (Postfix) with ESMTP id 8F2CF43E4A for ; Mon, 6 Jan 2003 10:00:24 -0800 (PST) (envelope-from jake@k6.locore.ca) Received: from k6.locore.ca (jake@localhost.locore.ca [127.0.0.1]) by k6.locore.ca (8.12.6/8.12.6) with ESMTP id h06I1EuP099153; Mon, 6 Jan 2003 13:01:14 -0500 (EST) (envelope-from jake@k6.locore.ca) Received: (from jake@localhost) by k6.locore.ca (8.12.6/8.12.6/Submit) id h06I0xDx099152; Mon, 6 Jan 2003 13:00:59 -0500 (EST) Date: Mon, 6 Jan 2003 13:00:58 -0500 From: Jake Burkholder To: Bruce Evans Cc: arch@FreeBSD.ORG Subject: Re: uarea/kstack/pcb/swapout sillyness Message-ID: <20030106130058.B96446@locore.ca> References: <20030106220745.Q2670-100000@gamplex.bde.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2.5i In-Reply-To: <20030106220745.Q2670-100000@gamplex.bde.org>; from bde@zeta.org.au on Mon, Jan 06, 2003 at 10:42:50PM +1100 Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG Apparently, On Mon, Jan 06, 2003 at 10:42:50PM +1100, Bruce Evans said words to the effect of; > Kernel stack pages don't seem to be handled by vm_proc_swap{in,out}. > Only the upages are swapped. This makes having a user area separate > from the proc struct even less useful than it used to be. We have > considerable complications to support the dubious feature of swapping > the user area. Swapping it out saves a whole 1 page per process > (UAREA_PAGE is 1 for all arches). OTOH, we don't bother swapping out > the much larger kernel stacks. There are 4 pages per stack on some > arches and there may be multiple stacks per process (1 per thread). > > Having a separate user area actually wastes memory in the usual (?) > case where the user area is not swapped. The user area now contains > only struct sigacts, which has size 2688, and struct pstats, which > has size 224, so 4096 - 2688 - 224 = 1184 bytes per process is wasted > for the struct (not counting costs for indirections). The kernel stack pages are still handled in MD code, due to the fact that ia64 uses contigmalloc to allocate them, but they are/will still be swapped. pmap_swapout_thread is called by swapout() on each thread in a process after swapping out the uarea. > > There seems to be even less point to keeping the pcb separate from > the thread struct. It saves space only by stealing stack space, > and costs indirections. Some PS_INMEM checks are now bogus because > the pcb is not in the user area so it doesn't get swapped. > > I use "options NO_SWAPPING" to prevent swapping of upages, and would > like to make this the default and later remove its code and merge the > structs. Swapping of upages seems to be least useful on large systems > where it would save the most memory, since such systems might actually > want to spend time running the processes instead of swapping the them > see the long message for the commit that introduced NO_SWAPPING in 1996). > With today's larger memories and bloated processes and kernel stacks, > I think the space/time savings (positive or negative) don't matter and > we just have useless complications to swap upages. Agree. The stuff in the uarea should be allocated normally and hung off of struct proc. Wether or not swapping kernel stacks is still useful I don't really know. Jake To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Mon Jan 6 10:25:10 2003 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 0691F37B401 for ; Mon, 6 Jan 2003 10:25:08 -0800 (PST) Received: from rwcrmhc51.attbi.com (rwcrmhc51.attbi.com [204.127.198.38]) by mx1.FreeBSD.org (Postfix) with ESMTP id 3B1A443EC2 for ; Mon, 6 Jan 2003 10:25:07 -0800 (PST) (envelope-from julian@elischer.org) Received: from InterJet.elischer.org (12-232-168-4.client.attbi.com[12.232.168.4]) by rwcrmhc51.attbi.com (rwcrmhc51) with ESMTP id <2003010618250505100jjpn7e>; Mon, 6 Jan 2003 18:25:06 +0000 Received: from localhost (localhost.elischer.org [127.0.0.1]) by InterJet.elischer.org (8.9.1a/8.9.1) with ESMTP id KAA04931; Mon, 6 Jan 2003 10:24:59 -0800 (PST) Date: Mon, 6 Jan 2003 10:24:58 -0800 (PST) From: Julian Elischer To: Jake Burkholder Cc: Bruce Evans , arch@FreeBSD.ORG Subject: Re: uarea/kstack/pcb/swapout sillyness In-Reply-To: <20030106130058.B96446@locore.ca> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG I didn't see the beginning of this in -arch.. was it there? the u-area handling ahs not changed since threads were introduced. whether they are swapped /not-swapped is the same. it is supposedly swapped out in: vm/vm_glue.c in vm_proc_swapout(struct proc *p) the thread is swapped out as mentionned below in: pmap_swapout_thread. It is a pitty that they are not in the same location, but as Jake said. the thread stack (including the PCB) turns out to be somewhat MD. On Mon, 6 Jan 2003, Jake Burkholder wrote: > Apparently, On Mon, Jan 06, 2003 at 10:42:50PM +1100, > Bruce Evans said words to the effect of; > > > Kernel stack pages don't seem to be handled by vm_proc_swap{in,out}. > > Only the upages are swapped. This makes having a user area separate > > from the proc struct even less useful than it used to be. We have > > considerable complications to support the dubious feature of swapping > > the user area. Swapping it out saves a whole 1 page per process > > (UAREA_PAGE is 1 for all arches). OTOH, we don't bother swapping out > > the much larger kernel stacks. There are 4 pages per stack on some > > arches and there may be multiple stacks per process (1 per thread). > > > > Having a separate user area actually wastes memory in the usual (?) > > case where the user area is not swapped. The user area now contains > > only struct sigacts, which has size 2688, and struct pstats, which > > has size 224, so 4096 - 2688 - 224 = 1184 bytes per process is wasted > > for the struct (not counting costs for indirections). > > The kernel stack pages are still handled in MD code, due to the fact > that ia64 uses contigmalloc to allocate them, but they are/will still > be swapped. pmap_swapout_thread is called by swapout() on each thread > in a process after swapping out the uarea. > > > > > There seems to be even less point to keeping the pcb separate from > > the thread struct. It saves space only by stealing stack space, > > and costs indirections. Some PS_INMEM checks are now bogus because > > the pcb is not in the user area so it doesn't get swapped. > > > > I use "options NO_SWAPPING" to prevent swapping of upages, and would > > like to make this the default and later remove its code and merge the > > structs. Swapping of upages seems to be least useful on large systems > > where it would save the most memory, since such systems might actually > > want to spend time running the processes instead of swapping the them > > see the long message for the commit that introduced NO_SWAPPING in 1996). > > With today's larger memories and bloated processes and kernel stacks, > > I think the space/time savings (positive or negative) don't matter and > > we just have useless complications to swap upages. > > Agree. The stuff in the uarea should be allocated normally and hung > off of struct proc. That is not a bad idea. It just depends on whether we want to swap that 1 page or not. We left it "as it was" specifically because it was swapped before so we swap it now.. > > Wether or not swapping kernel stacks is still useful I don't really know. Well they are bigger, and a process might have quite a few of them. A process with 10 suspended threads, each or 3 pages would be 30 pages or 120KB. It's a judgement call I guess. > > Jake > > To Unsubscribe: send mail to majordomo@FreeBSD.org > with "unsubscribe freebsd-arch" in the body of the message > To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Mon Jan 6 11:26: 1 2003 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 49D6737B401 for ; Mon, 6 Jan 2003 11:26:00 -0800 (PST) Received: from bluejay.mail.pas.earthlink.net (bluejay.mail.pas.earthlink.net [207.217.120.218]) by mx1.FreeBSD.org (Postfix) with ESMTP id D5C3F43EA9 for ; Mon, 6 Jan 2003 11:25:59 -0800 (PST) (envelope-from tlambert2@mindspring.com) Received: from pool0398.cvx40-bradley.dialup.earthlink.net ([216.244.43.143] helo=mindspring.com) by bluejay.mail.pas.earthlink.net with asmtp (SSLv3:RC4-MD5:128) (Exim 3.33 #1) id 18Vcsk-00076z-00; Mon, 06 Jan 2003 11:25:55 -0800 Message-ID: <3E19D7F2.2BB71EFE@mindspring.com> Date: Mon, 06 Jan 2003 11:24:34 -0800 From: Terry Lambert X-Mailer: Mozilla 4.79 [en] (Win98; U) X-Accept-Language: en MIME-Version: 1.0 To: David Schultz Cc: Bruce Evans , arch@FreeBSD.ORG Subject: Re: uarea/kstack/pcb/swapout sillyness References: <20030106220745.Q2670-100000@gamplex.bde.org> <20030106165538.GB721@HAL9000.homeunix.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-ELNK-Trace: b1a02af9316fbb217a47c185c03b154d40683398e744b8a46c9f1a74f6c0b748abf5afb11f0e63d793caf27dac41a8fd350badd9bab72f9c350badd9bab72f9c Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG David Schultz wrote: > Thus spake Bruce Evans : > > I use "options NO_SWAPPING" to prevent swapping of upages, and would > > like to make this the default and later remove its code and merge the > > structs. > > Please! Handling UPAGES was a PITA for implementing swapoff(), > and there's far too much code for swapping than swapping is worth. It occurs to me that all this code can be reused, should anyone implement PAE or PSE36-based use of more than 4G of RAM in 32 bit processors for FreeBSD... -- Terry To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Mon Jan 6 12: 7:32 2003 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id BF3BF37B401 for ; Mon, 6 Jan 2003 12:07:30 -0800 (PST) Received: from mailman.zeta.org.au (mailman.zeta.org.au [203.26.10.16]) by mx1.FreeBSD.org (Postfix) with ESMTP id 4FCD843E4A for ; Mon, 6 Jan 2003 12:07:29 -0800 (PST) (envelope-from bde@zeta.org.au) Received: from katana.zip.com.au (katana.zip.com.au [61.8.7.246]) by mailman.zeta.org.au (8.9.3/8.8.7) with ESMTP id HAA27132; Tue, 7 Jan 2003 07:07:02 +1100 Date: Tue, 7 Jan 2003 07:07:28 +1100 (EST) From: Bruce Evans X-X-Sender: bde@gamplex.bde.org To: Julian Elischer Cc: Jake Burkholder , Subject: Re: uarea/kstack/pcb/swapout sillyness In-Reply-To: Message-ID: <20030107061643.F4486-100000@gamplex.bde.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG On Mon, 6 Jan 2003, Julian Elischer wrote: > I didn't see the beginning of this in -arch.. was it there? Yes. Jake quoted all of my original message. > the u-area handling ahs not changed since threads were introduced. > whether they are swapped /not-swapped is the same. > it is supposedly swapped out in: > vm/vm_glue.c > in vm_proc_swapout(struct proc *p) > > the thread is swapped out as mentionned below > in: pmap_swapout_thread. > > It is a pitty that they are not in the same location, but as Jake said. > the thread stack (including the PCB) turns out to be somewhat MD. Yes. I missed the thread swapout call, having been misled partly by the rss calculation not counting swapped out stack pages. From kern_proc.c: % kp->ki_rssize = vmspace_resident_count(vm); /*XXX*/ % if (p->p_sflag & PS_INMEM) % kp->ki_rssize += UAREA_PAGES; % FOREACH_THREAD_IN_PROC(p, td) /* XXXKSE: thread swapout check */ % kp->ki_rssize += KSTACK_PAGES; There seems to be no need for the XXXKSE, except we should only count the stack pages if we aren't swapped out. PS_INMEM tells us if the stack pages are swapped out too, since we swap out the stack pages for all threads iff we swap out the upages (more precisely, it tells us if all these pages are swappable). > On Mon, 6 Jan 2003, Jake Burkholder wrote: > > > Apparently, On Mon, Jan 06, 2003 at 10:42:50PM +1100, > > Bruce Evans said words to the effect of; > > > I use "options NO_SWAPPING" to prevent swapping of upages, and would > > > like to make this the default and later remove its code and merge the > > > structs. Swapping of upages seems to be least useful on large systems > > > where it would save the most memory, since such systems might actually > > > want to spend time running the processes instead of swapping the them > > > see the long message for the commit that introduced NO_SWAPPING in 1996). > > > With today's larger memories and bloated processes and kernel stacks, > > > I think the space/time savings (positive or negative) don't matter and > > > we just have useless complications to swap upages. > > > > Agree. The stuff in the uarea should be allocated normally and hung > > off of struct proc. > > That is not a bad idea. It just depends on whether we want to swap that > 1 page or not. We left it "as it was" specifically because it was > swapped before so we swap it now.. Unfortunately for my argument, it's relatively easy to swap that 1 pages if we swap the stack pages (the infrastructure is much the same for both). Allocating the uarea normally simplifies some things but mostly not swapping. > > Wether or not swapping kernel stacks is still useful I don't really know. > > Well they are bigger, and a process might have quite a few of them. > A process with 10 suspended threads, each or 3 pages would be 30 pages > or 120KB. It's a judgement call I guess. I'm sure there are strange process mixes that could benefit from swapping 120KB per process, but I think they must be very uncommon now. Bruce To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Mon Jan 6 12:17:12 2003 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 310DD37B401 for ; Mon, 6 Jan 2003 12:17:11 -0800 (PST) Received: from canning.wemm.org (canning.wemm.org [192.203.228.65]) by mx1.FreeBSD.org (Postfix) with ESMTP id E2F1F43EB2 for ; Mon, 6 Jan 2003 12:17:10 -0800 (PST) (envelope-from peter@wemm.org) Received: from wemm.org (localhost [127.0.0.1]) by canning.wemm.org (Postfix) with ESMTP id 98CDB2A8A0; Mon, 6 Jan 2003 12:17:05 -0800 (PST) (envelope-from peter@wemm.org) X-Mailer: exmh version 2.5 07/13/2001 with nmh-1.0.4 To: Bruce Evans Cc: Julian Elischer , Jake Burkholder , arch@FreeBSD.ORG Subject: Re: uarea/kstack/pcb/swapout sillyness In-Reply-To: <20030107061643.F4486-100000@gamplex.bde.org> Date: Mon, 06 Jan 2003 12:17:05 -0800 From: Peter Wemm Message-Id: <20030106201705.98CDB2A8A0@canning.wemm.org> Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG Bruce Evans wrote: [..] > > > Wether or not swapping kernel stacks is still useful I don't really know. > > > > Well they are bigger, and a process might have quite a few of them. > > A process with 10 suspended threads, each or 3 pages would be 30 pages > > or 120KB. It's a judgement call I guess. > > I'm sure there are strange process mixes that could benefit from swapping > 120KB per process, but I think they must be very uncommon now. > > Bruce FWIW, the i386 has a relatively small stack. Other platforms are not so lucky and they add up.. 32K or 64K per thread is more painful than 8K per thread. Cheers, -Peter -- Peter Wemm - peter@wemm.org; peter@FreeBSD.org; peter@yahoo-inc.com "All of this is for nothing if we don't go to the stars" - JMS/B5 To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Mon Jan 6 19:11:49 2003 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id C75C037B401 for ; Mon, 6 Jan 2003 19:11:47 -0800 (PST) Received: from canning.wemm.org (canning.wemm.org [192.203.228.65]) by mx1.FreeBSD.org (Postfix) with ESMTP id 81BB343ED1 for ; Mon, 6 Jan 2003 19:11:47 -0800 (PST) (envelope-from peter@wemm.org) Received: from wemm.org (localhost [127.0.0.1]) by canning.wemm.org (Postfix) with ESMTP id 666D12A8A0; Mon, 6 Jan 2003 19:11:47 -0800 (PST) (envelope-from peter@wemm.org) X-Mailer: exmh version 2.5 07/13/2001 with nmh-1.0.4 To: Bruce Evans Cc: Julian Elischer , Jake Burkholder , arch@FreeBSD.ORG Subject: Re: uarea/kstack/pcb/swapout sillyness In-Reply-To: <20030107061643.F4486-100000@gamplex.bde.org> Date: Mon, 06 Jan 2003 19:11:47 -0800 From: Peter Wemm Message-Id: <20030107031147.666D12A8A0@canning.wemm.org> Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG Bruce Evans wrote: > On Mon, 6 Jan 2003, Julian Elischer wrote: > > the u-area handling ahs not changed since threads were introduced. > > whether they are swapped /not-swapped is the same. > > it is supposedly swapped out in: > > vm/vm_glue.c > > in vm_proc_swapout(struct proc *p) > > > > the thread is swapped out as mentionned below > > in: pmap_swapout_thread. > > > > It is a pitty that they are not in the same location, but as Jake said. > > the thread stack (including the PCB) turns out to be somewhat MD. > > Yes. I missed the thread swapout call, having been misled partly by the > rss calculation not counting swapped out stack pages. I wouldn't be too sad to see struct user being broken up and go away.. ie: something like moving the sigacts to a seperate zone, and the other parts attached to struct proc and/or struct thread. The pcb could move to an MD part of thread, but this increases kvm utilization since it currently lives in the kstack, not the user struct. ie: struct thread would get bigger, but the kstack would not. We already do some odd things with sharing the sigacts in the rfork()/ clone() case, that moves it right out of struct user for those processes, and obviously it the 2.5K (for 32 bit systems) of struct sigacts isn't swappable at that point as it lives in malloc somewhere. But I'm kinda reluctant to kill kstack swapping yet.. they can be damn big. Clean up: yes.. Outright remove: please not yet. Cheers, -Peter -- Peter Wemm - peter@wemm.org; peter@FreeBSD.org; peter@yahoo-inc.com "All of this is for nothing if we don't go to the stars" - JMS/B5 To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Tue Jan 7 9:19:25 2003 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id F02BB37B401 for ; Tue, 7 Jan 2003 09:19:21 -0800 (PST) Received: from khavrinen.lcs.mit.edu (khavrinen.lcs.mit.edu [18.24.4.193]) by mx1.FreeBSD.org (Postfix) with ESMTP id 3B19F43ED8 for ; Tue, 7 Jan 2003 09:19:21 -0800 (PST) (envelope-from wollman@khavrinen.lcs.mit.edu) Received: from khavrinen.lcs.mit.edu (localhost [IPv6:::1]) by khavrinen.lcs.mit.edu (8.12.6/8.12.6) with ESMTP id h07HJKCJ098682 (version=TLSv1/SSLv3 cipher=EDH-RSA-DES-CBC3-SHA bits=168 verify=OK) for ; Tue, 7 Jan 2003 12:19:20 -0500 (EST) (envelope-from wollman@khavrinen.lcs.mit.edu) Received: (from wollman@localhost) by khavrinen.lcs.mit.edu (8.12.6/8.12.6/Submit) id h07HJK7k098679; Tue, 7 Jan 2003 12:19:20 -0500 (EST) (envelope-from wollman) Date: Tue, 7 Jan 2003 12:19:20 -0500 (EST) From: Garrett Wollman Message-Id: <200301071719.h07HJK7k098679@khavrinen.lcs.mit.edu> To: arch@FreeBSD.org Subject: CFR: etc/periodic/daily/150.hoststat Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG Some time after I wrote the first version of this script, sendmail itself grew the ability to do what it does -- and sendmail does a much better job of it than this script ever could, since sendmail understands when it would itself consider the file stale. Since I wrote the script, it has grown some rather pointless excrescences. The following patch removes those and calls sendmail in the appropriate way to make it clean up after itself. As a result of this change, the pathname of the host status directory is no longer bogusly hard-coded in the script, so people who actually follow the sendmail documentation will be able to take advantage of it. Unfortunately, there is no easy way to tell whether this feature is enabled in sendmail or not. -GAWollman Index: etc/defaults/periodic.conf =================================================================== RCS file: /home/ncvs/src/etc/defaults/periodic.conf,v retrieving revision 1.22 diff -u -r1.22 periodic.conf --- etc/defaults/periodic.conf 25 Oct 2002 15:16:54 -0000 1.22 +++ etc/defaults/periodic.conf 7 Jan 2003 17:13:04 -0000 @@ -60,9 +60,8 @@ daily_clean_rwho_verbose="YES" # Mention files deleted # 150.clean-hoststat -daily_clean_hoststat_enable="YES" # Delete .hoststat daily -daily_clean_hoststat_days=3 # If not modified for -daily_clean_hoststat_verbose="YES" # Mention files deleted +daily_clean_hoststat_enable="YES" # Purge sendmail MX + # host cache daily # 200.backup-passwd daily_backup_passwd_enable="YES" # Backup passwd & group Index: etc/periodic/daily/150.clean-hoststat =================================================================== RCS file: /home/ncvs/src/etc/periodic/daily/150.clean-hoststat,v retrieving revision 1.6 diff -u -r1.6 150.clean-hoststat --- etc/periodic/daily/150.clean-hoststat 12 Aug 2002 11:09:01 -0000 1.6 +++ etc/periodic/daily/150.clean-hoststat 7 Jan 2003 17:13:05 -0000 @@ -14,34 +14,13 @@ case "$daily_clean_hoststat_enable" in [Yy][Ee][Ss]) - if [ -z "$daily_clean_hoststat_days" ]; then - echo '$daily_clean_hoststat_enable is enabled but' \ - '$daily_clean_hoststat_days is not set' - rc=2 - elif [ ! -d /var/spool/.hoststat ]; then - echo '$daily_clean_hoststat_enable is enabled but' \ - "/var/spool/.hoststat doesn't exist" + if [ -z "$(sendmail -bh 2>&1)" ]; then rc=2 else echo "" - echo "Removing stale files from /var/spool/.hoststat:" - - case "$daily_clean_hoststat_verbose" in - [Yy][Ee][Ss]) - print=-print;; - *) - print=;; - esac - - if [ -d /var/spool/.hoststat ]; then - cd /var/spool/.hoststat - rc=$(find . ! -name . -mtime +$daily_clean_hoststat_days \ - -delete $print | tee /dev/stderr | wc -l) - [ -z "$print" ] && rc=0 - [ $rc -gt 1 ] && rc=1 - else - rc=3 - fi + echo "Removing stale entries from sendmail host status cache:" + rc=0 + sendmail -bH || rc=1 fi;; *) rc=0;; Index: share_man_man5/periodic.conf.5 =================================================================== RCS file: /home/ncvs/src/share/man/man5/periodic.conf.5,v retrieving revision 1.44 diff -u -r1.44 periodic.conf.5 --- share_man_man5/periodic.conf.5 12 Dec 2002 17:25:57 -0000 1.44 +++ share_man_man5/periodic.conf.5 7 Jan 2003 17:13:05 -0000 @@ -236,18 +236,16 @@ .Pq Vt bool Set to .Dq YES -if you wish old files in -.Pa /var/spool/.hoststat -to be purged. -.It Va daily_clean_hoststat_days -.Pq Vt num -Set to the number of days that files must not have been modified before -they are deleted. -.It Va daily_clean_hoststat_verbose -.Pq Vt bool -Set to -.Dq YES -if you want the removed files to be reported in your daily output. +to run +.Ic sendmail -bH +to automatically purge stale entries from +.Xr sendmail 8 Ns 's +host status cache. +Files will be deleted using the same criteria as +.Nm sendmail +would normally use when determining whether to use them, +which are configured in +.Pa /etc/mail/sendmail.cf . .It Va daily_backup_passwd_enable .Pq Vt bool Set to To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Thu Jan 9 2:56:50 2003 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id A51EC37B401 for ; Thu, 9 Jan 2003 02:56:48 -0800 (PST) Received: from flood.ping.uio.no (flood.ping.uio.no [129.240.78.31]) by mx1.FreeBSD.org (Postfix) with ESMTP id 337F743E4A for ; Thu, 9 Jan 2003 02:56:48 -0800 (PST) (envelope-from des@ofug.org) Received: by flood.ping.uio.no (Postfix, from userid 2602) id 5117F5374; Thu, 9 Jan 2003 11:56:45 +0100 (CET) X-URL: http://www.ofug.org/~des/ X-Disclaimer: The views expressed in this message do not necessarily coincide with those of any organisation or company with which I am or have been affiliated. To: arch@freebsd.org Subject: small bsd.prog.mk change From: Dag-Erling Smorgrav Date: Thu, 09 Jan 2003 11:56:45 +0100 Message-ID: Lines: 11 User-Agent: Gnus/5.090007 (Oort Gnus v0.07) Emacs/21.2 (i386--freebsd) MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG --=-=-= The attached patch prevents the following warning which occurs when building the loader: "/usr/share/mk/bsd.prog.mk", line 37: warning: duplicate script for target "loader" ignored DES -- Dag-Erling Smorgrav - des@ofug.org --=-=-= Content-Type: text/x-patch Content-Disposition: attachment; filename=loader.diff Index: bsd.prog.mk =================================================================== RCS file: /home/ncvs/src/share/mk/bsd.prog.mk,v retrieving revision 1.129 diff -u -r1.129 bsd.prog.mk --- bsd.prog.mk 17 Oct 2002 13:48:13 -0000 1.129 +++ bsd.prog.mk 9 Jan 2003 10:54:47 -0000 @@ -30,11 +30,13 @@ OBJS+= ${SRCS:N*.h:R:S/$/.o/g} +.if !target(${PROG}) ${PROG}: ${OBJS} .if defined(PROG_CXX) ${CXX} ${CXXFLAGS} ${LDFLAGS} -o ${.TARGET} ${OBJS} ${LDADD} .else ${CC} ${CFLAGS} ${LDFLAGS} -o ${.TARGET} ${OBJS} ${LDADD} +.endif .endif .else !defined(SRCS) --=-=-=-- To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Thu Jan 9 4:28:57 2003 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 414DF37B401 for ; Thu, 9 Jan 2003 04:28:56 -0800 (PST) Received: from mailman.zeta.org.au (mailman.zeta.org.au [203.26.10.16]) by mx1.FreeBSD.org (Postfix) with ESMTP id 1FB0943ED8 for ; Thu, 9 Jan 2003 04:28:55 -0800 (PST) (envelope-from bde@zeta.org.au) Received: from katana.zip.com.au (katana.zip.com.au [61.8.7.246]) by mailman.zeta.org.au (8.9.3/8.8.7) with ESMTP id XAA03119; Thu, 9 Jan 2003 23:28:39 +1100 Date: Thu, 9 Jan 2003 23:29:18 +1100 (EST) From: Bruce Evans X-X-Sender: bde@gamplex.bde.org To: Dag-Erling Smorgrav Cc: arch@FreeBSD.ORG Subject: Re: small bsd.prog.mk change In-Reply-To: Message-ID: <20030109225032.R16405-100000@gamplex.bde.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG On Thu, 9 Jan 2003, Dag-Erling Smorgrav wrote: > The attached patch prevents the following warning which occurs when > building the loader: > > "/usr/share/mk/bsd.prog.mk", line 37: warning: duplicate script for target "loader" ignored % Index: bsd.prog.mk % =================================================================== % RCS file: /home/ncvs/src/share/mk/bsd.prog.mk,v % retrieving revision 1.129 % diff -u -r1.129 bsd.prog.mk % --- bsd.prog.mk 17 Oct 2002 13:48:13 -0000 1.129 % +++ bsd.prog.mk 9 Jan 2003 10:54:47 -0000 % @@ -30,11 +30,13 @@ % % OBJS+= ${SRCS:N*.h:R:S/$/.o/g} % % +.if !target(${PROG}) % ${PROG}: ${OBJS} % .if defined(PROG_CXX) % ${CXX} ${CXXFLAGS} ${LDFLAGS} -o ${.TARGET} ${OBJS} ${LDADD} % .else % ${CC} ${CFLAGS} ${LDFLAGS} -o ${.TARGET} ${OBJS} ${LDADD} % +.endif % .endif % % .else !defined(SRCS) Seomthing like this has been tried before, but apparently not finished. See revs.1.109-1.110. The above has a smaller scope than rev.1.109 so I think it doesn't have the problems of rev.1.109, but it leaves revs.1-109-1.110 unfinished and creates a style bug by not removing vestiges of rev.1.110. To fix the style bug, back out revs.1.109-1.110 before adding the ifdef as above, so that there is only 1 copy of the rule. Bruce To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Thu Jan 9 4:38:15 2003 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 2EE0437B401 for ; Thu, 9 Jan 2003 04:38:13 -0800 (PST) Received: from flood.ping.uio.no (flood.ping.uio.no [129.240.78.31]) by mx1.FreeBSD.org (Postfix) with ESMTP id EA91F43F13 for ; Thu, 9 Jan 2003 04:38:11 -0800 (PST) (envelope-from des@ofug.org) Received: by flood.ping.uio.no (Postfix, from userid 2602) id D287F5374; Thu, 9 Jan 2003 13:38:08 +0100 (CET) X-URL: http://www.ofug.org/~des/ X-Disclaimer: The views expressed in this message do not necessarily coincide with those of any organisation or company with which I am or have been affiliated. To: Bruce Evans Cc: arch@FreeBSD.ORG Subject: Re: small bsd.prog.mk change References: <20030109225032.R16405-100000@gamplex.bde.org> From: Dag-Erling Smorgrav Date: Thu, 09 Jan 2003 13:38:07 +0100 In-Reply-To: <20030109225032.R16405-100000@gamplex.bde.org> (Bruce Evans's message of "Thu, 9 Jan 2003 23:29:18 +1100 (EST)") Message-ID: Lines: 17 User-Agent: Gnus/5.090007 (Oort Gnus v0.07) Emacs/21.2 (i386--freebsd) MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG --=-=-= Bruce Evans writes: > Seomthing like this has been tried before, but apparently not finished. > See revs.1.109-1.110. The above has a smaller scope than rev.1.109 so > I think it doesn't have the problems of rev.1.109, but it leaves > revs.1-109-1.110 unfinished and creates a style bug by not removing > vestiges of rev.1.110. To fix the style bug, back out revs.1.109-1.110 > before adding the ifdef as above, so that there is only 1 copy of the > rule. How's this? DES -- Dag-Erling Smorgrav - des@ofug.org --=-=-= Content-Type: text/x-patch Content-Disposition: attachment; filename=loader.diff Index: bsd.prog.mk =================================================================== RCS file: /home/ncvs/src/share/mk/bsd.prog.mk,v retrieving revision 1.129 diff -u -r1.129 bsd.prog.mk --- bsd.prog.mk 17 Oct 2002 13:48:13 -0000 1.129 +++ bsd.prog.mk 9 Jan 2003 12:37:29 -0000 @@ -30,13 +30,6 @@ OBJS+= ${SRCS:N*.h:R:S/$/.o/g} -${PROG}: ${OBJS} -.if defined(PROG_CXX) - ${CXX} ${CXXFLAGS} ${LDFLAGS} -o ${.TARGET} ${OBJS} ${LDADD} -.else - ${CC} ${CFLAGS} ${LDFLAGS} -o ${.TARGET} ${OBJS} ${LDADD} -.endif - .else !defined(SRCS) .if !target(${PROG}) @@ -52,7 +45,9 @@ # the name of a variable temporary object. # - it's useful to keep objects around for crunching. OBJS= ${PROG}.o +.endif +.if !target(${PROG}) ${PROG}: ${OBJS} .if defined(PROG_CXX) ${CXX} ${CXXFLAGS} ${LDFLAGS} -o ${.TARGET} ${OBJS} ${LDADD} --=-=-=-- To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Thu Jan 9 18:15:42 2003 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 2236537B407 for ; Thu, 9 Jan 2003 18:15:37 -0800 (PST) Received: from debaser.madridwireless.net (26.Red-80-34-212.pooles.rima-tde.net [80.34.212.26]) by mx1.FreeBSD.org (Postfix) with ESMTP id 998C543ED8 for ; Thu, 9 Jan 2003 18:15:36 -0800 (PST) (envelope-from one-@madridwireless.net) Received: from irie.one-land.geored.org (82.Red-80-33-65.pooles.rima-tde.net [80.33.65.82]) by debaser.madridwireless.net (Postfix) with ESMTP id 31B98BBF1 for ; Fri, 10 Jan 2003 03:15:35 +0100 (CET) Date: Fri, 10 Jan 2003 03:15:35 +0100 From: Victor Sanchez To: freebsd-arch@freebsd.org Message-Id: <20030110031535.682f6535.one-@madridwireless.net> X-Mailer: Sylpheed version 0.8.8 (GTK+ 1.2.10; i386-portbld-freebsd5.0) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG subscribe To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Fri Jan 10 4: 3:18 2003 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 59EC037B401 for ; Fri, 10 Jan 2003 04:03:17 -0800 (PST) Received: from mailman.zeta.org.au (mailman.zeta.org.au [203.26.10.16]) by mx1.FreeBSD.org (Postfix) with ESMTP id 0C04943F1E for ; Fri, 10 Jan 2003 04:03:16 -0800 (PST) (envelope-from bde@zeta.org.au) Received: from katana.zip.com.au (katana.zip.com.au [61.8.7.246]) by mailman.zeta.org.au (8.9.3/8.8.7) with ESMTP id XAA02966; Fri, 10 Jan 2003 23:03:08 +1100 Date: Fri, 10 Jan 2003 23:03:52 +1100 (EST) From: Bruce Evans X-X-Sender: bde@gamplex.bde.org To: Dag-Erling Smorgrav Cc: arch@FreeBSD.ORG Subject: Re: small bsd.prog.mk change In-Reply-To: Message-ID: <20030110224932.A20648-100000@gamplex.bde.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG On Thu, 9 Jan 2003, Dag-Erling Smorgrav wrote: > Bruce Evans writes: > > Seomthing like this has been tried before, but apparently not finished. > > See revs.1.109-1.110. The above has a smaller scope than rev.1.109 so > > I think it doesn't have the problems of rev.1.109, but it leaves > > revs.1-109-1.110 unfinished and creates a style bug by not removing > > vestiges of rev.1.110. To fix the style bug, back out revs.1.109-1.110 > > before adding the ifdef as above, so that there is only 1 copy of the > > rule. > > How's this? [Context lost to attachment] OK with me. Also consider zapping the _EXTRADEPEND rule. It only works with the default rule for ${PROG}. In boot/i386/loader, this currently just gives a bogus dependency on ${LIBC}. Bruce To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Fri Jan 10 15:39:24 2003 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 82A1B37B401; Fri, 10 Jan 2003 15:39:22 -0800 (PST) Received: from ns1.xcllnt.net (209-128-86-226.BAYAREA.NET [209.128.86.226]) by mx1.FreeBSD.org (Postfix) with ESMTP id DA15C43EB2; Fri, 10 Jan 2003 15:39:21 -0800 (PST) (envelope-from marcel@xcllnt.net) Received: from athlon.pn.xcllnt.net (athlon.pn.xcllnt.net [192.168.4.3]) by ns1.xcllnt.net (8.12.6/8.12.6) with ESMTP id h0ANdL2G024366; Fri, 10 Jan 2003 15:39:21 -0800 (PST) (envelope-from marcel@piii.pn.xcllnt.net) Received: from athlon.pn.xcllnt.net (localhost [127.0.0.1]) by athlon.pn.xcllnt.net (8.12.6/8.12.6) with ESMTP id h0ANdLBR001425; Fri, 10 Jan 2003 15:39:21 -0800 (PST) (envelope-from marcel@athlon.pn.xcllnt.net) Received: (from marcel@localhost) by athlon.pn.xcllnt.net (8.12.6/8.12.6/Submit) id h0ANdLJ5001424; Fri, 10 Jan 2003 15:39:21 -0800 (PST) (envelope-from marcel) Date: Fri, 10 Jan 2003 15:39:21 -0800 From: Marcel Moolenaar To: arch@FreeBSD.org Cc: ru@FreeBSD.org Subject: Infinite loop in make(1) Message-ID: <20030110233921.GA1381@athlon.pn.xcllnt.net> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.5.1i Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG Gang, During release building we hit upon an infinite loop in make(1). This was cause by a substitution of which the LHS and RHS were both empty and the substitution was to be applied globally. For example (from ports/devel/m6811-gcc/Makefile): CFLAGS := ${CFLAGS:S/${_CPUCFLAGS}//g} If ${_CPUCFLAGS} is empty, the expansion obviously yields S///g. This causes make(1) to enter an infinite loop. Below a patch to prevent the infinite loop by forcing non-global substitution. This still allows S/^/foo/ and S/$/bar/ and even S///, but avoids the danger zone. Is this correct? \begin{patch} Index: var.c =================================================================== RCS file: /home/ncvs/src/usr.bin/make/var.c,v retrieving revision 1.40 diff -u -r1.40 var.c --- var.c 8 Nov 2002 16:59:11 -0000 1.40 +++ var.c 10 Jan 2003 23:30:58 -0000 @@ -1349,6 +1349,17 @@ cp++; } + /* + * Replacing the empty string for something else when + * done globally causes an infinite loop. The only + * meaningful substitution of the empty string would + * be those anchored by '^' or '$'. Thus, we can + * safely turn the substitution into a non-global one + * if the LHS is the empty string. + */ + if (pattern.leftLen == 0) + pattern.flags &= ~VAR_SUB_GLOBAL; + termc = *cp; newStr = VarModify(str, VarSubstitute, (void *)&pattern); \end{patch} -- Marcel Moolenaar USPA: A-39004 marcel@xcllnt.net To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Fri Jan 10 18:15: 5 2003 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 51A4737B401 for ; Fri, 10 Jan 2003 18:15:04 -0800 (PST) Received: from InterJet.dellroad.org (adsl-63-194-81-26.dsl.snfc21.pacbell.net [63.194.81.26]) by mx1.FreeBSD.org (Postfix) with ESMTP id 463ED43F65 for ; Fri, 10 Jan 2003 18:15:03 -0800 (PST) (envelope-from archie@dellroad.org) Received: from arch20m.dellroad.org (arch20m.dellroad.org [10.1.1.20]) by InterJet.dellroad.org (8.9.1a/8.9.1) with ESMTP id SAA91920 for ; Fri, 10 Jan 2003 18:01:09 -0800 (PST) Received: from arch20m.dellroad.org (localhost [127.0.0.1]) by arch20m.dellroad.org (8.12.6/8.12.6) with ESMTP id h0B20sHg024726 for ; Fri, 10 Jan 2003 18:00:54 -0800 (PST) (envelope-from archie@arch20m.dellroad.org) Received: (from archie@localhost) by arch20m.dellroad.org (8.12.6/8.12.6/Submit) id h0B20rUC024725 for freebsd-arch@freebsd.org; Fri, 10 Jan 2003 18:00:53 -0800 (PST) From: Archie Cobbs Message-Id: <200301110200.h0B20rUC024725@arch20m.dellroad.org> Subject: Virtual memory question To: freebsd-arch@freebsd.org Date: Fri, 10 Jan 2003 18:00:53 -0800 (PST) X-Mailer: ELM [version 2.4ME+ PL99b (25)] MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=US-ASCII Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG Dear freebsd-arch, Hope it's OK to ask a simple question here, it wasn't clear where else would be the right place to ask. The question is: how does the performance of various FreeBSD system calls (especially mmap() and munmap()) degrade when a process has lots and lots of tiny regions mapped into memory? I'm working on a memory allocator that would allocate say a few pages at a time, using mmap() (instead of sbrk()). So over time a process may end up with hundreds or even thousands of short, mmap()'d regions of memory. Is this going to cause any weird problems or slowness? BTW this idea was spawned by this text in the sbrk(3) man page: The brk() and sbrk() functions are legacy interfaces from before the advent of modern virtual memory management. Along those lines, why does our malloc(3) library use sbrk(3) instead of mmap(), which would enable returning free pages back to the system more readily (since the pages would not have to be contiguous)? Thanks, -Archie __________________________________________________________________________ Archie Cobbs * Packet Design * http://www.packetdesign.com To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Fri Jan 10 18:48:22 2003 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 731F537B401 for ; Fri, 10 Jan 2003 18:48:21 -0800 (PST) Received: from HAL9000.homeunix.com (12-233-57-224.client.attbi.com [12.233.57.224]) by mx1.FreeBSD.org (Postfix) with ESMTP id BBE8A43F1E for ; Fri, 10 Jan 2003 18:48:20 -0800 (PST) (envelope-from dschultz@uclink.Berkeley.EDU) Received: from HAL9000.homeunix.com (localhost [127.0.0.1]) by HAL9000.homeunix.com (8.12.6/8.12.5) with ESMTP id h0B2mHIZ002750; Fri, 10 Jan 2003 18:48:17 -0800 (PST) (envelope-from dschultz@uclink.Berkeley.EDU) Received: (from das@localhost) by HAL9000.homeunix.com (8.12.6/8.12.5/Submit) id h0B2mHRx002749; Fri, 10 Jan 2003 18:48:17 -0800 (PST) (envelope-from dschultz@uclink.Berkeley.EDU) Date: Fri, 10 Jan 2003 18:48:17 -0800 From: David Schultz To: Archie Cobbs Cc: freebsd-arch@FreeBSD.ORG Subject: Re: Virtual memory question Message-ID: <20030111024817.GA2615@HAL9000.homeunix.com> Mail-Followup-To: Archie Cobbs , freebsd-arch@FreeBSD.ORG References: <200301110200.h0B20rUC024725@arch20m.dellroad.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <200301110200.h0B20rUC024725@arch20m.dellroad.org> Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG Thus spake Archie Cobbs : > The question is: how does the performance of various FreeBSD system > calls (especially mmap() and munmap()) degrade when a process has > lots and lots of tiny regions mapped into memory? > > I'm working on a memory allocator that would allocate say a few > pages at a time, using mmap() (instead of sbrk()). So over time a > process may end up with hundreds or even thousands of short, mmap()'d > regions of memory. Is this going to cause any weird problems or > slowness? FreeBSD will combine adjacent mappings of the same type into a single object, so you shouldn't see a performance degradation in that case. I believe the vm_map entries are now laid out in a tree, so even if the map does become fragmented, performance should still be acceptable. > BTW this idea was spawned by this text in the sbrk(3) man page: > > The brk() and sbrk() functions are legacy interfaces from > before the advent of modern virtual memory management. > > Along those lines, why does our malloc(3) library use sbrk(3) instead > of mmap(), which would enable returning free pages back to the system > more readily (since the pages would not have to be contiguous)? malloc(3) returns pages to the system using madvise(2) (but only if the appropriate option is enabled). This fragments the map anyway, so I don't know why sbrk(3) is used. Perhaps sbrk() winds up having slightly lower overhead. To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Fri Jan 10 23:39:41 2003 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id DDC1E37B401 for ; Fri, 10 Jan 2003 23:39:39 -0800 (PST) Received: from critter.freebsd.dk (critter.freebsd.dk [212.242.86.163]) by mx1.FreeBSD.org (Postfix) with ESMTP id 1C7F543ED8 for ; Fri, 10 Jan 2003 23:39:39 -0800 (PST) (envelope-from phk@freebsd.org) Received: from critter.freebsd.dk (localhost [127.0.0.1]) by critter.freebsd.dk (8.12.6/8.12.6) with ESMTP id h0B7ZSFt003789; Sat, 11 Jan 2003 08:35:29 +0100 (CET) (envelope-from phk@freebsd.org) To: Archie Cobbs Cc: freebsd-arch@freebsd.org Subject: Re: Virtual memory question From: phk@freebsd.org In-Reply-To: Your message of "Fri, 10 Jan 2003 18:00:53 PST." <200301110200.h0B20rUC024725@arch20m.dellroad.org> Date: Sat, 11 Jan 2003 08:35:28 +0100 Message-ID: <3788.1042270528@critter.freebsd.dk> Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG In message <200301110200.h0B20rUC024725@arch20m.dellroad.org>, Archie Cobbs wri tes: >The question is: how does the performance of various FreeBSD system >calls (especially mmap() and munmap()) degrade when a process has >lots and lots of tiny regions mapped into memory? Badly. At least it used to do: When I wrote phkmalloc(3) I tried using malloc instead of sbrk(2) and it suffered because the VM system couldn't collapse all the individual allocations (ie: N lines in /proc/$pid/map Poul-Henning > >I'm working on a memory allocator that would allocate say a few >pages at a time, using mmap() (instead of sbrk()). So over time a >process may end up with hundreds or even thousands of short, mmap()'d >regions of memory. Is this going to cause any weird problems or >slowness? > >BTW this idea was spawned by this text in the sbrk(3) man page: > > The brk() and sbrk() functions are legacy interfaces from > before the advent of modern virtual memory management. > >Along those lines, why does our malloc(3) library use sbrk(3) instead >of mmap(), which would enable returning free pages back to the system >more readily (since the pages would not have to be contiguous)? > >Thanks, >-Archie > >__________________________________________________________________________ >Archie Cobbs * Packet Design * http://www.packetdesign.com > >To Unsubscribe: send mail to majordomo@FreeBSD.org >with "unsubscribe freebsd-arch" in the body of the message > -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk@FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Sat Jan 11 3:35:55 2003 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 1513237B401; Sat, 11 Jan 2003 03:35:54 -0800 (PST) Received: from HAL9000.homeunix.com (12-233-57-224.client.attbi.com [12.233.57.224]) by mx1.FreeBSD.org (Postfix) with ESMTP id 99E6143F18; Sat, 11 Jan 2003 03:35:53 -0800 (PST) (envelope-from dschultz@uclink.Berkeley.EDU) Received: from HAL9000.homeunix.com (localhost [127.0.0.1]) by HAL9000.homeunix.com (8.12.6/8.12.5) with ESMTP id h0BBZpIZ005026; Sat, 11 Jan 2003 03:35:51 -0800 (PST) (envelope-from dschultz@uclink.Berkeley.EDU) Received: (from das@localhost) by HAL9000.homeunix.com (8.12.6/8.12.5/Submit) id h0BBZp8G005025; Sat, 11 Jan 2003 03:35:51 -0800 (PST) (envelope-from dschultz@uclink.Berkeley.EDU) Date: Sat, 11 Jan 2003 03:35:51 -0800 From: David Schultz To: phk@FreeBSD.ORG Cc: Archie Cobbs , freebsd-arch@FreeBSD.ORG Subject: Re: Virtual memory question Message-ID: <20030111113551.GC3961@HAL9000.homeunix.com> Mail-Followup-To: phk@FreeBSD.ORG, Archie Cobbs , freebsd-arch@FreeBSD.ORG References: <200301110200.h0B20rUC024725@arch20m.dellroad.org> <3788.1042270528@critter.freebsd.dk> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <3788.1042270528@critter.freebsd.dk> Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG Thus spake phk@FreeBSD.ORG : > In message <200301110200.h0B20rUC024725@arch20m.dellroad.org>, Archie Cobbs wri > tes: > > >The question is: how does the performance of various FreeBSD system > >calls (especially mmap() and munmap()) degrade when a process has > >lots and lots of tiny regions mapped into memory? > > Badly. At least it used to do: When I wrote phkmalloc(3) I tried > using malloc instead of sbrk(2) and it suffered because the VM system ^^^^^^ you mean mmap(2)? > couldn't collapse all the individual allocations (ie: N lines in > /proc/$pid/map The following program does 1000 mmaps of various sizes, then prints its own memory map. It shows that the resulting vm_map has only a few entries in both -CURRENT and -STABLE. The log for src/sys/vm/vm_map.c suggests that the collapse code may have been broken/non-optimal at several times during the past six years. #include #include #include #include #include #include #define NUM_MMAPS 1000 int main() { int i, psize, fd, len; char *p; char buf[20]; psize = getpagesize(); for (i = 0; i < NUM_MMAPS; i++) { p = mmap(NULL, psize * (i % 4 + 1), PROT_READ|PROT_WRITE, MAP_ANON|MAP_SHARED, -1, 0); if (p == NULL) err(1, "mmap"); } snprintf(buf, 20, "/proc/%u/map", getpid()); if ((fd = open(buf, O_RDONLY)) < 0) err(1, "open"); if ((len = read(fd, p, psize - 1)) < 0) err(1, "read"); p[len] = '\0'; printf("%s\n", p); return 0; } To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Sat Jan 11 3:46:21 2003 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 930F737B401 for ; Sat, 11 Jan 2003 03:46:20 -0800 (PST) Received: from critter.freebsd.dk (critter.freebsd.dk [212.242.86.163]) by mx1.FreeBSD.org (Postfix) with ESMTP id C3C1343E4A for ; Sat, 11 Jan 2003 03:46:19 -0800 (PST) (envelope-from phk@freebsd.org) Received: from critter.freebsd.dk (localhost [127.0.0.1]) by critter.freebsd.dk (8.12.6/8.12.6) with ESMTP id h0BBk0Ft005171; Sat, 11 Jan 2003 12:46:02 +0100 (CET) (envelope-from phk@freebsd.org) To: David Schultz Cc: Archie Cobbs , freebsd-arch@freebsd.org Subject: Re: Virtual memory question From: phk@freebsd.org In-Reply-To: Your message of "Sat, 11 Jan 2003 03:35:51 PST." <20030111113551.GC3961@HAL9000.homeunix.com> Date: Sat, 11 Jan 2003 12:46:00 +0100 Message-ID: <5170.1042285560@critter.freebsd.dk> Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG In message <20030111113551.GC3961@HAL9000.homeunix.com>, David Schultz writes: >Thus spake phk@FreeBSD.ORG : >> In message <200301110200.h0B20rUC024725@arch20m.dellroad.org>, Archie Cobbs wri >> tes: >> >> >The question is: how does the performance of various FreeBSD system >> >calls (especially mmap() and munmap()) degrade when a process has >> >lots and lots of tiny regions mapped into memory? >> >> Badly. At least it used to do: When I wrote phkmalloc(3) I tried >> using malloc instead of sbrk(2) and it suffered because the VM system > ^^^^^^ you mean mmap(2)? >> couldn't collapse all the individual allocations (ie: N lines in >> /proc/$pid/map > >The following program does 1000 mmaps of various sizes, then >prints its own memory map. It shows that the resulting vm_map has >only a few entries in both -CURRENT and -STABLE. The log for >src/sys/vm/vm_map.c suggests that the collapse code may have been >broken/non-optimal at several times during the past six years. Could be, I have not revisited the issue since 1995 or so. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk@FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Sat Jan 11 10:32:55 2003 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id A926337B401 for ; Sat, 11 Jan 2003 10:32:54 -0800 (PST) Received: from dragon.nuxi.com (trang.nuxi.com [66.93.134.19]) by mx1.FreeBSD.org (Postfix) with ESMTP id 33EE743E4A for ; Sat, 11 Jan 2003 10:32:54 -0800 (PST) (envelope-from obrien@NUXI.com) Received: from dragon.nuxi.com (obrien@localhost [127.0.0.1]) by dragon.nuxi.com (8.12.6/8.12.2) with ESMTP id h0BIWdR6071765; Sat, 11 Jan 2003 10:32:44 -0800 (PST) (envelope-from obrien@dragon.nuxi.com) Received: (from obrien@localhost) by dragon.nuxi.com (8.12.6/8.12.6/Submit) id h0BIVMre071752; Sat, 11 Jan 2003 10:31:22 -0800 (PST) Date: Sat, 11 Jan 2003 10:31:21 -0800 From: "David O'Brien" To: Archie Cobbs Cc: freebsd-arch@freebsd.org Subject: Re: Virtual memory question Message-ID: <20030111183121.GA71528@dragon.nuxi.com> Reply-To: freebsd-arch@freebsd.org References: <200301110200.h0B20rUC024725@arch20m.dellroad.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <200301110200.h0B20rUC024725@arch20m.dellroad.org> User-Agent: Mutt/1.4i X-Operating-System: FreeBSD 5.0-CURRENT Organization: The NUXI BSD Group X-Pgp-Rsa-Fingerprint: B7 4D 3E E9 11 39 5F A3 90 76 5D 69 58 D9 98 7A X-Pgp-Rsa-Keyid: 1024/34F9F9D5 Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG On Fri, Jan 10, 2003 at 06:00:53PM -0800, Archie Cobbs wrote: > BTW this idea was spawned by this text in the sbrk(3) man page: > > The brk() and sbrk() functions are legacy interfaces from > before the advent of modern virtual memory management. > > Along those lines, why does our malloc(3) library use sbrk(3) instead > of mmap(), which would enable returning free pages back to the system > more readily (since the pages would not have to be contiguous)? See the glibc malloc(3) [which started with libdlmalloc] or the libdlmalloc port. It uses the heap (sbrk(3)) for small allocations, and mmap() for larger ones. The threashold is tunable. To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Sat Jan 11 14:29: 3 2003 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id EBC3D37B401 for ; Sat, 11 Jan 2003 14:29:01 -0800 (PST) Received: from eumenes.hosting.swbell.net (eumenes.hosting.swbell.net [216.100.98.7]) by mx1.FreeBSD.org (Postfix) with ESMTP id 336C443F13 for ; Sat, 11 Jan 2003 14:29:01 -0800 (PST) (envelope-from alc@imimic.com) Received: from imimic.com (adsl-216-63-78-18.dsl.hstntx.swbell.net [216.63.78.18]) by eumenes.hosting.swbell.net id RAA06417; Sat, 11 Jan 2003 17:28:54 -0500 (EST) [ConcentricHost SMTP Relay 1.14] Message-ID: <3E209AA5.ED387618@imimic.com> Date: Sat, 11 Jan 2003 16:28:53 -0600 From: "Alan L. Cox" Organization: iMimic Networking, Inc. X-Mailer: Mozilla 4.8 [en] (X11; U; Linux 2.4.2 i386) X-Accept-Language: en MIME-Version: 1.0 To: arch@freebsd.org Subject: Re: Virtual memory question Content-Type: text/plain; charset=x-user-defined Content-Transfer-Encoding: 7bit Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG >The question is: how does the performance of various FreeBSD system >calls (especially mmap() and munmap()) degrade when a process has >lots and lots of tiny regions mapped into memory? 5.x is much, much better at handling this than 4.x. That said, it's time for a list of disclaimers. 1. Lookups are accelerated in 5.x, but finding the right amount of free virtual address space isn't. The free space hint often works well, but can fail miserably. 2. Coalescing of mappings is complicated. The program that was posted here is a little too simplistic in its "workload". Specifically, it never touches any of the memory and never munmaps any of the memory. If it did both, you would see fragmentation in the vm_map and situations where you would say, "Gosh, why aren't these adjacent map entries coalesced?" The reason is that distinct anonymous memory objects got allocated. Currently, the only "work-around" is not to use anonymous memory. Create a "heap" file, use MAP_NOSYNC for performance, and make sure that you pass the right file offset to mmap(). Regards, Alan To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Sat Jan 11 14:44:46 2003 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id F050937B401 for ; Sat, 11 Jan 2003 14:44:44 -0800 (PST) Received: from canning.wemm.org (canning.wemm.org [192.203.228.65]) by mx1.FreeBSD.org (Postfix) with ESMTP id B01CD43F18 for ; Sat, 11 Jan 2003 14:44:44 -0800 (PST) (envelope-from peter@wemm.org) Received: from wemm.org (localhost [127.0.0.1]) by canning.wemm.org (Postfix) with ESMTP id 94D102A89E; Sat, 11 Jan 2003 14:44:44 -0800 (PST) (envelope-from peter@wemm.org) X-Mailer: exmh version 2.5 07/13/2001 with nmh-1.0.4 To: "Alan L. Cox" Cc: arch@freebsd.org Subject: Re: Virtual memory question In-Reply-To: <3E209AA5.ED387618@imimic.com> Date: Sat, 11 Jan 2003 14:44:44 -0800 From: Peter Wemm Message-Id: <20030111224444.94D102A89E@canning.wemm.org> Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG "Alan L. Cox" wrote: > >The question is: how does the performance of various FreeBSD system > >calls (especially mmap() and munmap()) degrade when a process has > >lots and lots of tiny regions mapped into memory? > > 5.x is much, much better at handling this than 4.x. That said, it's > time for a list of disclaimers. > > 1. Lookups are accelerated in 5.x, but finding the right amount of free > virtual address space isn't. The free space hint often works well, but > can fail miserably. > > 2. Coalescing of mappings is complicated. The program that was posted > here is a little too simplistic in its "workload". Specifically, it > never touches any of the memory and never munmaps any of the memory. If > it did both, you would see fragmentation in the vm_map and situations > where you would say, "Gosh, why aren't these adjacent map entries > coalesced?" The reason is that distinct anonymous memory objects got > allocated. Currently, the only "work-around" is not to use anonymous > memory. Create a "heap" file, use MAP_NOSYNC for performance, and make > sure that you pass the right file offset to mmap(). Speaking of which, I've been thinking about some sort of non-fs persistent object handle for doing mmaps. (Note: thinking about, not actually getting around to doing :-). Imagine mmapping /dev/zero, but having a persistent object per fd that was opened. This would be to enable things like fd passing to get access to another process's store and so on. Basically a replacement for the MAP_NOSYNC files, and it would be swap backed instead of vfs backed. This would be very damn useful at work. It could be ideal for malloc too since it would be a single object with coherent ordering etc. At work, we have lots of problems with complications of having a file system attached to things that we just want to shove data into and otherwise forget about. Cheers, -Peter -- Peter Wemm - peter@wemm.org; peter@FreeBSD.org; peter@yahoo-inc.com "All of this is for nothing if we don't go to the stars" - JMS/B5 To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Sat Jan 11 15:42:48 2003 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id CEDB737B401 for ; Sat, 11 Jan 2003 15:42:46 -0800 (PST) Received: from apollo.backplane.com (apollo.backplane.com [216.240.41.2]) by mx1.FreeBSD.org (Postfix) with ESMTP id 7BB1443E4A for ; Sat, 11 Jan 2003 15:42:45 -0800 (PST) (envelope-from dillon@apollo.backplane.com) Received: from apollo.backplane.com (localhost [127.0.0.1]) by apollo.backplane.com (8.12.6/8.12.6) with ESMTP id h0BNgjYp048597; Sat, 11 Jan 2003 15:42:45 -0800 (PST) (envelope-from dillon@apollo.backplane.com) Received: (from dillon@localhost) by apollo.backplane.com (8.12.6/8.12.6/Submit) id h0BNgj9a048596; Sat, 11 Jan 2003 15:42:45 -0800 (PST) Date: Sat, 11 Jan 2003 15:42:45 -0800 (PST) From: Matthew Dillon Message-Id: <200301112342.h0BNgj9a048596@apollo.backplane.com> To: Peter Wemm Cc: "Alan L. Cox" , arch@FreeBSD.ORG Subject: Re: Virtual memory question References: <20030111224444.94D102A89E@canning.wemm.org> Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG :Speaking of which, I've been thinking about some sort of non-fs persistent :object handle for doing mmaps. (Note: thinking about, not actually getting :around to doing :-). : :Imagine mmapping /dev/zero, but having a persistent object per fd that was :opened. This would be to enable things like fd passing to get access to :another process's store and so on. Basically a replacement for the :MAP_NOSYNC files, and it would be swap backed instead of vfs backed. : :This would be very damn useful at work. It could be ideal for malloc too :since it would be a single object with coherent ordering etc. At work, we :have lots of problems with complications of having a file system attached :to things that we just want to shove data into and otherwise forget about. : :Cheers, :-Peter :-- :Peter Wemm - peter@wemm.org; peter@FreeBSD.org; peter@yahoo-inc.com This is basically how shared memory works, except that shared memory is managed outside the file descriptor framework. I would love to see a shared memory object that is managed inside the file descriptor framework, sort of like 'pipe()'. I do not see any need to use /dev/zero to implement the feature, though, because it will not improve portability. How about something like: getmemfd(). You would then be able to mmap() to your heart's content. We wouldn't even need to implement ftruncate(), we could just default it to a 64 bit object space. fd = getmemfd(); mmap(...) pass the descriptor around, etc. BTW, this would be utterly trivial to implement. I don't even think we need a VFS. Just a fileops set. -Matt Matthew Dillon To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Sat Jan 11 16:31:30 2003 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 7BF0A37B401 for ; Sat, 11 Jan 2003 16:31:29 -0800 (PST) Received: from philotas.hosting.swbell.net (philotas.hosting.swbell.net [216.100.99.7]) by mx1.FreeBSD.org (Postfix) with ESMTP id 8695D43EB2 for ; Sat, 11 Jan 2003 16:31:28 -0800 (PST) (envelope-from alc@imimic.com) Received: from imimic.com (adsl-216-63-78-18.dsl.hstntx.swbell.net [216.63.78.18]) by philotas.hosting.swbell.net id TAA16453; Sat, 11 Jan 2003 19:31:05 -0500 (EST) [ConcentricHost SMTP Relay 1.14] Message-ID: <3E20B747.1FCA3B36@imimic.com> Date: Sat, 11 Jan 2003 18:31:03 -0600 From: "Alan L. Cox" Organization: iMimic Networking, Inc. X-Mailer: Mozilla 4.8 [en] (X11; U; Linux 2.4.2 i386) X-Accept-Language: en MIME-Version: 1.0 To: Matthew Dillon Cc: Peter Wemm , arch@FreeBSD.ORG Subject: Re: Virtual memory question References: <20030111224444.94D102A89E@canning.wemm.org> <200301112342.h0BNgj9a048596@apollo.backplane.com> Content-Type: text/plain; charset=x-user-defined Content-Transfer-Encoding: 7bit Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG Matthew Dillon wrote: > ... > How about something like: > > getmemfd(). > Roughly speaking, this is shm_open(3), which we currently implement using files. Alan To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Sat Jan 11 16:53:18 2003 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id C4D9F37B401 for ; Sat, 11 Jan 2003 16:53:16 -0800 (PST) Received: from apollo.backplane.com (apollo.backplane.com [216.240.41.2]) by mx1.FreeBSD.org (Postfix) with ESMTP id 5CC9C43F75 for ; Sat, 11 Jan 2003 16:53:12 -0800 (PST) (envelope-from dillon@apollo.backplane.com) Received: from apollo.backplane.com (localhost [127.0.0.1]) by apollo.backplane.com (8.12.6/8.12.6) with ESMTP id h0C0rBYp072259; Sat, 11 Jan 2003 16:53:11 -0800 (PST) (envelope-from dillon@apollo.backplane.com) Received: (from dillon@localhost) by apollo.backplane.com (8.12.6/8.12.6/Submit) id h0C0rBR1072243; Sat, 11 Jan 2003 16:53:11 -0800 (PST) Date: Sat, 11 Jan 2003 16:53:11 -0800 (PST) From: Matthew Dillon Message-Id: <200301120053.h0C0rBR1072243@apollo.backplane.com> To: "Alan L. Cox" Cc: Peter Wemm , arch@FreeBSD.ORG Subject: Re: Virtual memory question References: <20030111224444.94D102A89E@canning.wemm.org> <200301112342.h0BNgj9a048596@apollo.backplane.com> <3E20B747.1FCA3B36@imimic.com> Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG : :Matthew Dillon wrote: :> ... :> How about something like: :> :> getmemfd(). :> : :Roughly speaking, this is shm_open(3), which we currently implement :using files. : :Alan Except without the path rendezvous (which in sysv land may not even represent a real filesystem path). I do not think we would want to make this compatible with the brokekn SysV SHM API. It's just too broken. Interesting. This brings to mind the 'fifo' filesystem object type. You know, ala 'mkfifo'. There might be an advantage to that, then a normal open() could be used to get the shared memory object. mkshmem - make a shared memory object that can be open()'d. (ala mkfifo). -Matt Matthew Dillon To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Sat Jan 11 17: 3:43 2003 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 099C537B401 for ; Sat, 11 Jan 2003 17:03:43 -0800 (PST) Received: from cleitus.hosting.swbell.net (cleitus.hosting.swbell.net [216.100.99.4]) by mx1.FreeBSD.org (Postfix) with ESMTP id 384D843E4A for ; Sat, 11 Jan 2003 17:03:42 -0800 (PST) (envelope-from alc@imimic.com) Received: from imimic.com (adsl-216-63-78-18.dsl.hstntx.swbell.net [216.63.78.18]) by cleitus.hosting.swbell.net id UAA05967; Sat, 11 Jan 2003 20:03:37 -0500 (EST) [ConcentricHost SMTP Relay 1.14] Message-ID: <3E20BEE8.61F8D9CD@imimic.com> Date: Sat, 11 Jan 2003 19:03:36 -0600 From: "Alan L. Cox" Organization: iMimic Networking, Inc. X-Mailer: Mozilla 4.8 [en] (X11; U; Linux 2.4.2 i386) X-Accept-Language: en MIME-Version: 1.0 To: Matthew Dillon Cc: Peter Wemm , arch@FreeBSD.ORG Subject: Re: Virtual memory question References: <20030111224444.94D102A89E@canning.wemm.org> <200301112342.h0BNgj9a048596@apollo.backplane.com> <3E20B747.1FCA3B36@imimic.com> <200301120053.h0C0rBR1072243@apollo.backplane.com> Content-Type: text/plain; charset=x-user-defined Content-Transfer-Encoding: 7bit Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG Matthew Dillon wrote: > ... > Except without the path rendezvous (which in sysv land may not even > represent a real filesystem path). I do not think we would want to > make this compatible with the brokekn SysV SHM API. It's just too > broken. shm_open(3) is for use with mmap(2). It really has nothing to do with System V SHM. Alan To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Sat Jan 11 17: 9:35 2003 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id DC47E37B401 for ; Sat, 11 Jan 2003 17:09:33 -0800 (PST) Received: from apollo.backplane.com (apollo.backplane.com [216.240.41.2]) by mx1.FreeBSD.org (Postfix) with ESMTP id 83C0543EB2 for ; Sat, 11 Jan 2003 17:09:33 -0800 (PST) (envelope-from dillon@apollo.backplane.com) Received: from apollo.backplane.com (localhost [127.0.0.1]) by apollo.backplane.com (8.12.6/8.12.6) with ESMTP id h0C19XYp000212; Sat, 11 Jan 2003 17:09:33 -0800 (PST) (envelope-from dillon@apollo.backplane.com) Received: (from dillon@localhost) by apollo.backplane.com (8.12.6/8.12.6/Submit) id h0C19Xch000211; Sat, 11 Jan 2003 17:09:33 -0800 (PST) Date: Sat, 11 Jan 2003 17:09:33 -0800 (PST) From: Matthew Dillon Message-Id: <200301120109.h0C19Xch000211@apollo.backplane.com> To: "Alan L. Cox" Cc: Peter Wemm , arch@FreeBSD.ORG Subject: Re: Virtual memory question References: <20030111224444.94D102A89E@canning.wemm.org> <200301112342.h0BNgj9a048596@apollo.backplane.com> <3E20B747.1FCA3B36@imimic.com> <200301120053.h0C0rBR1072243@apollo.backplane.com> <3E20BEE8.61F8D9CD@imimic.com> Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG : :Matthew Dillon wrote: :> ... :> Except without the path rendezvous (which in sysv land may not even :> represent a real filesystem path). I do not think we would want to :> make this compatible with the brokekn SysV SHM API. It's just too :> broken. : :shm_open(3) is for use with mmap(2). It really has nothing to do with :System V SHM. : :Alan I meant this comment in the manual page: "The path argument does not necessarily represent a pathname (although it does in this and most other implementations). Two processes opening the same path are guaranteed to access the same shared memory object if and only if path begins with a slash (`/') character." Which implies that the code must support non-path string identifiers to be POSIX compliant. I should have said POSIX breakage instead of SysV breakage I suppose. -Matt To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message