From owner-freebsd-smp Sun Nov 19 1: 1:55 2000 Delivered-To: freebsd-smp@freebsd.org Received: from mail.interware.hu (mail.interware.hu [195.70.32.130]) by hub.freebsd.org (Postfix) with ESMTP id 5F3C537B479 for ; Sun, 19 Nov 2000 01:01:47 -0800 (PST) Received: from victoria-088.budapest.interware.hu ([195.70.63.88] helo=elischer.org) by mail.interware.hu with esmtp (Exim 3.16 #1 (Debian)) id 13xQM5-0003ph-00 for ; Sun, 19 Nov 2000 10:01:45 +0100 Message-ID: <3A1796F7.7A02ED18@elischer.org> Date: Sun, 19 Nov 2000 01:01:43 -0800 From: Julian Elischer X-Mailer: Mozilla 4.7 [en] (X11; U; FreeBSD 5.0-CURRENT i386) X-Accept-Language: en, hu MIME-Version: 1.0 To: smp@freebsd.org Subject: [Fwd: Threads (KSE etc) comments] Content-Type: multipart/mixed; boundary="------------E4DCA88212AA8612B72C91F6" Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org This is a multi-part message in MIME format. --------------E4DCA88212AA8612B72C91F6 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit I'm forwarding this to SMP as David obrien tells me all the threads discussion happens here now.. I WAS on smp but haven;t heard anything for a while. I think I'll go check if I'm still subscribed.. -- __--_|\ Julian Elischer / \ julian@elischer.org ( OZ ) World tour 2000 ---> X_.---._/ presently in: Budapest v --------------E4DCA88212AA8612B72C91F6 Content-Type: message/rfc822 Content-Transfer-Encoding: 7bit Content-Disposition: inline X-Mozilla-Status2: 00000000 Message-ID: <3A15A2C1.1F3FB6CD@elischer.org> Date: Fri, 17 Nov 2000 13:27:29 -0800 From: Julian Elischer X-Mailer: Mozilla 4.7 [en] (X11; U; FreeBSD 5.0-CURRENT i386) X-Accept-Language: en MIME-Version: 1.0 To: jasone@freebsd.org, arch@freebsd.org Subject: Threads (KSE etc) comments Content-Type: text/plain; charset=iso-8859-15 Content-Transfer-Encoding: 7bit Hi jason. I read your Nov 7 doc on the threading model. I have a few comments.. I've been thinking about the scheduling queues, and how to make sure that the process (KSEG actually) acts fairly with respect to other processes. I was confised for a while by your description. I think part of my confusion came from something that we specified in the meeting but has not been written in your document directly. let me see if we are agreed on what we decided.. A KSEG can only have as a maximum N KSEs associated with it, where N is the number of processors, (unless artificially reduced by a lower concurency declaration). (you said this but only indirectly). In general, KSEs are each assigned to a processor. They do not in general move between processors unless some explicit adjustment is being made(*), and as a general rule, two KSEs will not be assigned to the same processor. (in some transitional moments this may be allowed to briefly happen) This in general if you run a KSEC on the same KSE it was run on last time, you should be on the same processor, (and get any affinity advantages that might exist). (*)I am inclined to make the requirement of binding KSEs to processors HARD, as this allows us to simplify some later decisions. For example, if we hard bind KSEs to procesors then since we assign a different communications mailbox for each KSE we create, we can be sure that different KSEs will never preempt each other when writing out to their mailboxes. this also means that since there can only be one UTS incarnation active per KSE (or one KSE per UTS incarnation), that we can not have a UTS preempted by another incarnation on the same processor. We can therefore make sure that there needs to be no locking on mailboxes, or even any checking. I think this is what we decided.. is this correct? The binding is not really mentioned in your document. When we were talking about it, (at least in my memory) Each KSE had a mailbox. My memory of this was that we called a KSE creation call with a different argument, thus each KSE had a different return stack frame when it made upcalls. In the version you have outlined, there is no KSE creation call only KSEG creation calls. Thus all upcalls have the same frame, and there is the danger of colliding upcalls for different processors. My memory (where is that photo of the whiteboard that Nicole was supposed to send us) is that each KSE is assigned a differnt mailbox address in userland, which is associated with the frame that it will do upcalls on. One of the fields of the mailbox contains a pointer to a userland context structure which contains apece where the KSE should dump the user context should it need to, and a pointer to other such structures. This structure is defined by the kernel, but included in the UTS's 'per thread info'. Since there is one per thread, there is never a problem in running out of them when the kernel links them together in a linked list of completed operations. When the thread makes a system call, the KSE looks in the mailbox for the context structure for this thread, and if the thread blocks or resumes, it can save any information it needs to tell teh UTS there. The UTS sets the pointer into the mailbox when it schedules the thread, so even involintary blockages (e.g. page faults) have the pointer available. When the UTS is running it's own work, it ZERO's this pointer, which lets the kernel know that it is not really in a safe state for pre-emmpting. I think that we decided that a page fault in the UTS simply blocked until it was satisfied. When an upcall occurs, the stack frame it occurs on, and hence the mailbox pointed to are automatically correct, so the UTS doesn't even have to look it up. (the mailbox is allocated as a local variable in the frame of the KSE creation call and is this in the local frame of the upcall. this is something like I imagined the UTS would do to fire off a new KSE. The reason I was thinking of it this way was so that each KSE has a UTS supplied mailbox and (smallish) stack. /* * use make_new_kse() to do exactly that. * Returns -1 on failure and 1 on success. * * cookie allows the UTS to have it's own way of identifying the KSE/thread. * This stack is effectively lost to us so we first switch * to a small throw-away stack. It need only have enough space in it for the * upcalls to call the UTS, and whatever the UTS will need. * Some time after creation, there will be an upcall on the new KSE looking for work. * I could imagine wiring this UTS stack down.. */ void start_new_kse(void * cookie, jmp_buf *jb) /*XXX correct definition for jb? */ { struct kse_mailbox; int return_value; bzero(kse_mailbox, sizeof(kse_mailbox)); return_value = kse_new(&kse_mailbox); switch (return_value) { case -1:  perror("make_new_kse() failed"); _longjmp(jb, -1); case 0: printf ("successfully created kse %d\n", kse_mailbox.kse_id); _longjmp(jb, 1); exit (1); /* not reached */ default: printf(" An upcall of type %d occured\n", return_value); uts_scheduler(cookie, &kse_mailbox, return_value); /* must never return */ printf ("it returned!\n"); exit (1); } } make_new_kse(void * cookie) { int retval; jmp_buf env; if ((retval = _setjmp(env)) == 0) { load_new_stack() /* load a new smaller stack, but copy the top 100 bytes or so from the old stack so that our local variables appear to be the same. */ start_new_kse(cookie, env); /* not reached */ } return (retval) } When we have per-processor scheduling queues, there is only at most ONE KSE from any given KSEG in the scheduling queues for any given processor. With the single scheduling queue we have now do we allow N to be in the queues at once? (or do we put the KSEG in instead?) The terms KSE etc. have probably served their useful life. It's time to think of or find names that really describe them better KSE -- a per process processor.. Scheduler slot? openning? (a-la CAM/SCSI) KSEC ---- stack plus context... KSC.. it's trying to do somethign (a task?) KSEG ---- a class of schedulable entities.. A slot cluster? :-) PROC ---- probably needs to stay the same. -- __--_|\ Julian Elischer / \ julian@elischer.org ( OZ ) World tour 2000 ---> X_.---._/ presently in: Budapest v --------------E4DCA88212AA8612B72C91F6-- To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Sun Nov 19 5:31:42 2000 Delivered-To: freebsd-smp@freebsd.org Received: from spooky.eis.net.au (spooky.eis.net.au [203.12.171.2]) by hub.freebsd.org (Postfix) with ESMTP id 9AB4E37B479 for ; Sun, 19 Nov 2000 05:31:38 -0800 (PST) Received: (from ernie@localhost) by spooky.eis.net.au (8.11.1/8.9.3) id eAJDVaN00542 for freebsd-smp@freebsd.org; Sun, 19 Nov 2000 23:31:36 +1000 (EST) (envelope-from ernie) From: Ernie Elu Message-Id: <200011191331.eAJDVaN00542@spooky.eis.net.au> Subject: Abit BP6 failures To: freebsd-smp@freebsd.org Date: Sun, 19 Nov 2000 23:31:34 +1000 (EST) X-Mailer: ELM [version 2.4ME+ PL40 (25)] MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org Two of my Abit BP6 motherboards have developed an anoying fault. It takes an average 10 cycles of the power switch before they will boot, the rest of the time they just sit there dormant, no speaker beep or flash of the keyboard leds. Both have worked fine for about a year. Both are running 433Mhz celerons. It looks like a weird hardware problem, both were fine up until running 4.1.1-STABLE. Anyone had a simiar experience with the BP6? Also I cant seem to boot from the 4.1.1-RELEASE installer CD, the kernel loads up to the point where it's doing the APIC_IO test and freezes. 4.1.1-STABLE kernels seem to be fine. Any ideas? - Ernie. To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Sun Nov 19 6:16:24 2000 Delivered-To: freebsd-smp@freebsd.org Received: from mail.utfors.se (mail.utfors.se [195.58.103.125]) by hub.freebsd.org (Postfix) with ESMTP id 2B5A337B479 for ; Sun, 19 Nov 2000 06:16:17 -0800 (PST) Received: from ludd.luth.se (md4692003.utfors.se [212.105.32.3]) by mail.utfors.se (8.8.8/8.8.8) with ESMTP id PAA26122; Sun, 19 Nov 2000 15:15:51 +0100 (MET) Message-ID: <3A17E088.A24F5C05@ludd.luth.se> Date: Sun, 19 Nov 2000 15:15:36 +0100 From: Joachim =?iso-8859-1?Q?Str=F6mbergson?= Organization: Acne X-Mailer: Mozilla 4.75 [en] (X11; U; FreeBSD 4.1-STABLE i386) X-Accept-Language: en-US MIME-Version: 1.0 To: Ernie Elu Cc: freebsd-smp@FreeBSD.ORG Subject: Re: Abit BP6 failures References: <200011191331.eAJDVaN00542@spooky.eis.net.au> Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: 8bit Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org Aloha! Ernie Elu wrote: > > Two of my Abit BP6 motherboards have developed an anoying fault. > It takes an average 10 cycles of the power switch before they will boot, > the rest of the time they just sit there dormant, no speaker beep or flash > of the keyboard leds. Both have worked fine for about a year. > Both are running 433Mhz celerons. It looks like a weird hardware problem, > both were fine up until running 4.1.1-STABLE. > > Anyone had a simiar experience with the BP6? > > Also I cant seem to boot from the 4.1.1-RELEASE installer CD, the kernel > loads up to the point where it's doing the APIC_IO test and freezes. > > 4.1.1-STABLE kernels seem to be fine. > > Any ideas? Very strange, but we migyt need a more detail here. Do these machines boot at all, or does it take 10 power cycles to get them to POST? Can they boot any other systems? I'm using a BP6 board w. two celeron 533 MHz, and would *not* want it to stop booting after moving from 4.1-STABLE to 4.2-STABLE (the plan right now). Also, could you extract dmesg, kernel configs and so on? -- Cheers! Joachim - Alltid i harmonisk svängning --- FairLight ------ FairLight ------ FairLight ------ FairLight --- Joachim Strömbergson ASIC SoC designer, nice to CUTE animals Phone: +46(0)31 - 27 98 47 Web: http://www.ludd.luth.se/~watchman --------------- Spamfodder: regeringen@regeringen.se --------------- To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Sun Nov 19 12: 2:23 2000 Delivered-To: freebsd-smp@freebsd.org Received: from mailout01.sul.t-online.com (mailout01.sul.t-online.com [194.25.134.80]) by hub.freebsd.org (Postfix) with ESMTP id 14A6637B479 for ; Sun, 19 Nov 2000 12:02:20 -0800 (PST) Received: from fwd03.sul.t-online.com by mailout01.sul.t-online.com with smtp id 13xaeV-0007Ax-04; Sun, 19 Nov 2000 21:01:27 +0100 Received: from kim.sixpack.de (520030847883-0001@[193.159.140.13]) by fwd03.sul.t-online.com with esmtp id 13xaeI-1sZuWuC; Sun, 19 Nov 2000 21:01:14 +0100 Received: from kiwi.sixpack.de (kiwi.sixpack.de [192.168.2.32]) by kim.sixpack.de (8.11.1/8.11.1) with ESMTP id eAJJrmG21632 for ; Sun, 19 Nov 2000 20:53:48 +0100 (CET) (envelope-from oliver@sixpack.de) Received: from localhost (oliver@localhost) by kiwi.sixpack.de (8.11.1/8.9.3) with ESMTP id eAJJqjQ26998 for ; Sun, 19 Nov 2000 20:52:45 +0100 (CET) (envelope-from oliver@kiwi.sixpack.de) Date: Sun, 19 Nov 2000 20:52:44 +0100 (CET) From: Oliver Schneider To: FreeBSD SMP Subject: Re: Abit BP6 failures In-Reply-To: <3A17E088.A24F5C05@ludd.luth.se> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=ISO-8859-1 Content-Transfer-Encoding: QUOTED-PRINTABLE X-Sender: 520030847883-0001@t-dialin.net Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org Hi, Joachim Str=F6mbergson schrieb am Sun, 19 Nov 2000: > Ernie Elu wrote: > >=20 > > Two of my Abit BP6 motherboards have developed an anoying fault. [...] > > both were fine up until running 4.1.1-STABLE. > >=20 > > Also I cant seem to boot from the 4.1.1-RELEASE installer CD, the kerne= l > > loads up to the point where it's doing the APIC_IO test and freezes. I have no such CD here, but usually it is a single cpu version, or did you make your own release? Does a 4.0 release boot still? > Very strange, but we migyt need a more detail here. Do these machines > boot at all, or does it take 10 power cycles to get them to POST? Can > they boot any other systems?=20 I dont think so...=20 What BIOS version? The ru version works fine here. My machine boot with something like "BIOS NOT FOR SALE, PREVERSION!". Do you use SMP(?) 1.4 - (1.1|1.4). I dont know the the correct name, and I dont wanna boot right now, to have a look at the BIOS. With ru I have no trouble with all versions from 4.0 release to 4.2 stable beta. Finally have a look at www.bp6.com, i think there are problems with a power suplly part at some revisions. Cheers Oliver=20 To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Sun Nov 19 17:33:24 2000 Delivered-To: freebsd-smp@freebsd.org Received: from gtei2.bellatlantic.net (gtei2.bellatlantic.net [199.45.39.161]) by hub.freebsd.org (Postfix) with ESMTP id 1B09837B479 for ; Sun, 19 Nov 2000 17:33:22 -0800 (PST) Received: from seth-f1pgytg8r3 (client-64-223-145-91.bellatlantic.net [64.223.145.91]) by gtei2.bellatlantic.net (8.9.1/8.9.1) with SMTP id UAA15497; Sun, 19 Nov 2000 20:33:06 -0500 (EST) Message-Id: <3.0.6.32.20001119203634.00c689f8@hobbiton.shire.net> X-Sender: seth-pc@hobbiton.shire.net X-Mailer: QUALCOMM Windows Eudora Light Version 3.0.6 (32) Date: Sun, 19 Nov 2000 20:36:34 -0800 To: Ernie Elu From: Seth Leigh Subject: Re: Abit BP6 failures Cc: freebsd-smp@FreeBSD.ORG In-Reply-To: <200011192209.eAJM9Vq01712@spooky.eis.net.au> References: <3.0.6.32.20001119100352.00c41248@hobbiton.shire.net> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org Yeah, in fact I think it may have something to do with the chipset, since it didn't do it when it was new, but it sometimes does it now. I have been overclocking this machine since day 1, and I have noticed a bit of a degradation in its ability to be overclocked. For a long time I ran the Celeron 400s at 552 MHz, but now it isn't stable at 552, and I have to run it at 480 MHz, where it is rock solid. Why won't it run at 552 anymore? I believe it is actual degradation of either one or both of the CPUs, or else the chipset. I am going to pull the motherboard out, remove the heatsink from the BX chip on the motherboard, and put some heatsink compound underneath it, and mount a fan onto the heatsink. I will then try going back up to 552 MHz. The motherboard is a year and a half old as it is, so I don't really care if I risk frying it, since if I fry it I will just replace it with an 850 MHz or 900 MHz Athlon. If I don't fry it I will hold onto it for another 6 or 8 months and replace it with a dual Athlon, which I assume will be shipping by then. Seth At 08:09 AM 11/20/2000 +1000, you wrote: >> Actually yes, I have had this problem with my BP6. It isn't running >> FreeBSD though, this machine is running just Win2K and Solaris 2.8. >> >> What will happen is that sometimes, after shutting it down, is that when I >> go to reboot it it will just do *nothing* after I power it on, no display >> on the screen or anything. When that happens (which isn't all the time) I >> will just cycle the switch on the back of the case (the switch that >> directly turns off the power supply), and then cycle it back on and hit the >> On button on the front panel of my machine again. When this problem occurs >> it usually takes between 1 and 4 such cyclings to get it to boot up. >> >> I thought I was the only one with this problem. Oh well. >> >> The nice thing is that I hardly ever shut this machine down. It runs >> either Win2K or Solaris 24/7, so the problem isn't too annoying. Also, it >> doesn't always fail to boot up when I have shut it down, perhaps it does it >> a third of the time. >> >> Other than that, I have had zero problems with my BP6 under those two OSes. >> I love it, it is a year and a half old and has been running 2 Celeron 400s >> at either 552 MHz or 480 MHz ever since I built it. >> >> Oh yeah, I water cooled the CPUs. :-) >> >> (www.2coolcomputer.com is where I got the water blocks to do this) >> >> Seth Leigh >> > >Oh well it's good to see I am not the only one with the problem, looks like >a design fault in the BP6, I noticed that the problem almost goes away if >you only have a single CPU fitted. I wonder if that means the BX chip has >cooked itself over time. > >- Ernie. > > To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Sun Nov 19 23:55:19 2000 Delivered-To: freebsd-smp@freebsd.org Received: from io.yi.org (h24-69-199-88.gv.shawcable.net [24.69.199.88]) by hub.freebsd.org (Postfix) with ESMTP id 82F3037B4C5 for ; Sun, 19 Nov 2000 23:55:17 -0800 (PST) Received: from io.yi.org (localhost.gvcl1.bc.wave.home.com [127.0.0.1]) by io.yi.org (Postfix) with ESMTP id 93B73BA7A for ; Sun, 19 Nov 2000 23:55:16 -0800 (PST) X-Mailer: exmh version 2.1.1 10/15/1999 To: smp@freebsd.org Subject: patch: allproc_lock Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Sun, 19 Nov 2000 23:55:16 -0800 From: Jake Burkholder Message-Id: <20001120075516.93B73BA7A@io.yi.org> Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org Attached is a patch that adds allproc_lock. This is a lockmgr lock which protectes the following: allproc zombproc pidhashtbl proc.p_list proc.p_hash nextpid Please review it. To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Mon Nov 20 0: 7:13 2000 Delivered-To: freebsd-smp@freebsd.org Received: from io.yi.org (h24-69-199-88.gv.shawcable.net [24.69.199.88]) by hub.freebsd.org (Postfix) with ESMTP id 2AC5C37B4C5 for ; Mon, 20 Nov 2000 00:06:57 -0800 (PST) Received: from io.yi.org (localhost.gvcl1.bc.wave.home.com [127.0.0.1]) by io.yi.org (Postfix) with ESMTP id 5BEA4BA7A for ; Mon, 20 Nov 2000 00:06:55 -0800 (PST) X-Mailer: exmh version 2.1.1 10/15/1999 To: smp@FreeBSD.ORG Subject: Re: patch: allproc_lock In-Reply-To: Message from Jake Burkholder of "Sun, 19 Nov 2000 23:55:16 PST." <20001120075516.93B73BA7A@io.yi.org> Mime-Version: 1.0 Content-Type: multipart/mixed ; boundary="==_Exmh_-14298859040" Date: Mon, 20 Nov 2000 00:06:54 -0800 From: Jake Burkholder Message-Id: <20001120080655.5BEA4BA7A@io.yi.org> Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org This is a multipart MIME message. --==_Exmh_-14298859040 Content-Type: text/plain; charset=us-ascii > > Attached is a patch that adds allproc_lock. This is a lockmgr > lock which protectes the following: > > allproc > zombproc > pidhashtbl > proc.p_list > proc.p_hash > nextpid > > Please review it. Sorry, too quick with the send... Here's the patch: --==_Exmh_-14298859040 Content-Type: text/plain ; name="allproc2.diff"; charset=us-ascii Content-Description: allproc2.diff Content-Disposition: attachment; filename="allproc2.diff" Index: alpha/alpha/pmap.c =================================================================== RCS file: /home/ncvs/src/sys/alpha/alpha/pmap.c,v retrieving revision 1.44 diff -u -r1.44 pmap.c --- alpha/alpha/pmap.c 2000/10/17 10:05:46 1.44 +++ alpha/alpha/pmap.c 2000/11/19 10:12:35 @@ -723,12 +723,14 @@ printf("pmap_get_asn: generation rollover\n"); #endif PCPU_GET(current_asngen) = 1; + allproc_lock(LK_SHARED); LIST_FOREACH(p, &allproc, p_list) { if (p->p_vmspace) { tpmap = vmspace_pmap(p->p_vmspace); tpmap->pm_asn[PCPU_GET(cpuno)].gen = 0; } } + allproc_unlock(); } /* @@ -1553,12 +1555,14 @@ newlev1 = pmap_phys_to_pte(pa) | PG_V | PG_ASM | PG_KRE | PG_KWE; + allproc_lock(LK_SHARED); LIST_FOREACH(p, &allproc, p_list) { if (p->p_vmspace) { pmap = vmspace_pmap(p->p_vmspace); *pmap_lev1pte(pmap, kernel_vm_end) = newlev1; } } + allproc_unlock(); *pte = newlev1; pmap_invalidate_all(kernel_pmap); } @@ -3057,6 +3061,7 @@ struct proc *p; int npte = 0; int index; + allproc_lock(LK_SHARED); LIST_FOREACH(p, &allproc, p_list) { if (p->p_pid != pid) continue; @@ -3079,6 +3084,7 @@ index = 0; printf("\n"); } + allproc_unlock(); return npte; } pte = pmap_pte_quick( pmap, va); @@ -3103,6 +3109,7 @@ } } } + allproc_unlock(); return npte; } #endif Index: i386/i386/pmap.c =================================================================== RCS file: /home/ncvs/src/sys/i386/i386/pmap.c,v retrieving revision 1.264 diff -u -r1.264 pmap.c --- i386/i386/pmap.c 2000/11/07 18:31:16 1.264 +++ i386/i386/pmap.c 2000/11/19 10:12:35 @@ -3324,6 +3324,7 @@ struct proc *p; int npte = 0; int index; + allproc_lock(LK_SHARED); LIST_FOREACH(p, &allproc, p_list) { if (p->p_pid != pid) continue; @@ -3346,6 +3347,7 @@ index = 0; printf("\n"); } + allproc_unlock(); return npte; } pte = pmap_pte_quick( pmap, va); @@ -3370,6 +3372,7 @@ } } } + allproc_unlock(); return npte; } #endif Index: ia64/ia64/pmap.c =================================================================== RCS file: /home/ncvs/src/sys/ia64/ia64/pmap.c,v retrieving revision 1.8 diff -u -r1.8 pmap.c --- ia64/ia64/pmap.c 2000/10/17 10:05:49 1.8 +++ ia64/ia64/pmap.c 2000/11/19 10:12:36 @@ -2244,6 +2244,7 @@ struct proc *p; int npte = 0; int index; + allproc_lock(LK_SHARED); LIST_FOREACH(p, &allproc, p_list) { if (p->p_pid != pid) continue; @@ -2266,6 +2267,7 @@ index = 0; printf("\n"); } + allproc_unlock(); return npte; } pte = pmap_pte_quick( pmap, va); @@ -2290,6 +2292,7 @@ } } } + allproc_unlock(); return npte; } #endif Index: kern/imgact_elf.c =================================================================== RCS file: /home/ncvs/src/sys/kern/imgact_elf.c,v retrieving revision 1.84 diff -u -r1.84 imgact_elf.c --- kern/imgact_elf.c 2000/11/09 08:25:47 1.84 +++ kern/imgact_elf.c 2000/11/19 10:12:36 @@ -153,10 +153,12 @@ { struct proc *p; + allproc_lock(LK_SHARED); LIST_FOREACH(p, &allproc, p_list) { if (p->p_sysent == entry->sysvec) return TRUE; } + allproc_unlock(); return FALSE; } Index: kern/init_main.c =================================================================== RCS file: /home/ncvs/src/sys/kern/init_main.c,v retrieving revision 1.146 diff -u -r1.146 init_main.c --- kern/init_main.c 2000/11/05 10:41:34 1.146 +++ kern/init_main.c 2000/11/19 10:12:37 @@ -405,10 +405,12 @@ * Now we can look at the time, having had a chance to verify the * time from the file system. Pretend that proc0 started now. */ + allproc_lock(LK_SHARED); LIST_FOREACH(p, &allproc, p_list) { microtime(&p->p_stats->p_start); p->p_runtime = 0; } + allproc_unlock(); microuptime(&switchtime); PCPU_SET(switchticks, ticks); Index: kern/kern_exit.c =================================================================== RCS file: /home/ncvs/src/sys/kern/kern_exit.c,v retrieving revision 1.103 diff -u -r1.103 kern_exit.c --- kern/kern_exit.c 2000/10/20 07:58:02 1.103 +++ kern/kern_exit.c 2000/11/20 06:17:32 @@ -264,11 +264,13 @@ * Remove proc from allproc queue and pidhash chain. * Place onto zombproc. Unlink from parent's child list. */ + allproc_lock(LK_EXCLUSIVE); LIST_REMOVE(p, p_list); LIST_INSERT_HEAD(&zombproc, p, p_list); p->p_stat = SZOMB; LIST_REMOVE(p, p_hash); + allproc_unlock(); q = LIST_FIRST(&p->p_children); if (q) /* only need this if any child is S_ZOMB */ Index: kern/kern_fork.c =================================================================== RCS file: /home/ncvs/src/sys/kern/kern_fork.c,v retrieving revision 1.83 diff -u -r1.83 kern_fork.c --- kern/kern_fork.c 2000/10/20 07:58:03 1.83 +++ kern/kern_fork.c 2000/11/20 06:13:57 @@ -287,6 +287,7 @@ * If RFHIGHPID is set (used during system boot), do not allocate * low-numbered pids. */ + allproc_lock(LK_EXCLUSIVE); trypid = nextpid + 1; if (flags & RFHIGHPID) { if (trypid < 10) { @@ -343,12 +344,6 @@ } } - p2 = newproc; - p2->p_stat = SIDL; /* protect against others */ - p2->p_pid = trypid; - LIST_INSERT_HEAD(&allproc, p2, p_list); - LIST_INSERT_HEAD(PIDHASH(p2->p_pid), p2, p_hash); - /* * RFHIGHPID does not mess with the nextpid counter during boot. */ @@ -356,6 +351,13 @@ pidchecked = 0; else nextpid = trypid; + + p2 = newproc; + p2->p_stat = SIDL; /* protect against others */ + p2->p_pid = trypid; + LIST_INSERT_HEAD(&allproc, p2, p_list); + LIST_INSERT_HEAD(PIDHASH(p2->p_pid), p2, p_hash); + allproc_unlock(); /* * Make a proc table entry for the new process. Index: kern/kern_ktrace.c =================================================================== RCS file: /home/ncvs/src/sys/kern/kern_ktrace.c,v retrieving revision 1.43 diff -u -r1.43 kern_ktrace.c --- kern/kern_ktrace.c 2000/10/27 11:45:33 1.43 +++ kern/kern_ktrace.c 2000/11/19 10:12:37 @@ -278,6 +278,7 @@ * Clear all uses of the tracefile */ if (ops == KTROP_CLEARFILE) { + allproc_lock(LK_SHARED); LIST_FOREACH(p, &allproc, p_list) { if (p->p_tracep == vp) { if (ktrcanset(curp, p)) { @@ -289,6 +290,7 @@ error = EPERM; } } + allproc_unlock(); goto done; } /* @@ -494,6 +496,7 @@ */ log(LOG_NOTICE, "ktrace write failed, errno %d, tracing stopped\n", error); + allproc_lock(LK_SHARED); LIST_FOREACH(p, &allproc, p_list) { if (p->p_tracep == vp) { p->p_tracep = NULL; @@ -501,6 +504,7 @@ vrele(vp); } } + allproc_unlock(); } /* Index: kern/kern_proc.c =================================================================== RCS file: /home/ncvs/src/sys/kern/kern_proc.c,v retrieving revision 1.75 diff -u -r1.75 kern_proc.c --- kern/kern_proc.c 2000/09/07 01:32:51 1.75 +++ kern/kern_proc.c 2000/11/20 06:05:52 @@ -72,6 +72,7 @@ u_long pgrphash; struct proclist allproc; struct proclist zombproc; +struct lock __allproc_lock; vm_zone_t proc_zone; vm_zone_t ithread_zone; @@ -82,6 +83,7 @@ procinit() { + lockinit(&__allproc_lock, PZERO, "allproc", 0, 0); LIST_INIT(&allproc); LIST_INIT(&zombproc); pidhashtbl = hashinit(maxproc / 4, M_PROC, &pidhash); @@ -113,10 +115,12 @@ { register struct proc *p; + allproc_lock(LK_SHARED); LIST_FOREACH(p, PIDHASH(pid), p_hash) if (p->p_pid == pid) - return (p); - return (NULL); + break; + allproc_unlock(); + return (p); } /* @@ -470,6 +474,7 @@ if (error) return (error); } + allproc_lock(LK_SHARED); for (doingzomb=0 ; doingzomb < 2 ; doingzomb++) { if (!doingzomb) p = LIST_FIRST(&allproc); @@ -525,10 +530,13 @@ continue; error = sysctl_out_proc(p, req, doingzomb); - if (error) + if (error) { + allproc_unlock(); return (error); + } } } + allproc_unlock(); return (0); } Index: kern/kern_resource.c =================================================================== RCS file: /home/ncvs/src/sys/kern/kern_resource.c,v retrieving revision 1.64 diff -u -r1.64 kern_resource.c --- kern/kern_resource.c 2000/09/18 17:03:03 1.64 +++ kern/kern_resource.c 2000/11/19 10:12:37 @@ -119,11 +119,13 @@ case PRIO_USER: if (uap->who == 0) uap->who = curp->p_ucred->cr_uid; + allproc_lock(LK_SHARED); LIST_FOREACH(p, &allproc, p_list) if (!p_can(curp, p, P_CAN_SEE, NULL) && p->p_ucred->cr_uid == uap->who && p->p_nice < low) low = p->p_nice; + allproc_unlock(); break; default: @@ -185,12 +187,14 @@ case PRIO_USER: if (uap->who == 0) uap->who = curp->p_ucred->cr_uid; + allproc_lock(LK_SHARED); LIST_FOREACH(p, &allproc, p_list) if (p->p_ucred->cr_uid == uap->who && !p_can(curp, p, P_CAN_SEE, NULL)) { error = donice(curp, p, uap->prio); found++; } + allproc_unlock(); break; default: Index: kern/kern_sig.c =================================================================== RCS file: /home/ncvs/src/sys/kern/kern_sig.c,v retrieving revision 1.92 diff -u -r1.92 kern_sig.c --- kern/kern_sig.c 2000/11/17 18:09:15 1.92 +++ kern/kern_sig.c 2000/11/19 10:12:37 @@ -850,10 +850,11 @@ struct pgrp *pgrp; int nfound = 0; - if (all) + if (all) { /* * broadcast */ + allproc_lock(LK_SHARED); LIST_FOREACH(p, &allproc, p_list) { if (p->p_pid <= 1 || p->p_flag & P_SYSTEM || p == cp || !CANSIGNAL(cp, p, sig)) @@ -862,7 +863,8 @@ if (sig) psignal(p, sig); } - else { + allproc_unlock(); + } else { if (pgid == 0) /* * zero pgid means send to my process group. Index: kern/kern_synch.c =================================================================== RCS file: /home/ncvs/src/sys/kern/kern_synch.c,v retrieving revision 1.109 diff -u -r1.109 kern_synch.c --- kern/kern_synch.c 2000/11/17 18:09:15 1.109 +++ kern/kern_synch.c 2000/11/19 10:12:37 @@ -281,6 +281,7 @@ register int realstathz, s; realstathz = stathz ? stathz : hz; + allproc_lock(LK_SHARED); LIST_FOREACH(p, &allproc, p_list) { /* * Increment time in/out of memory and sleep time @@ -340,6 +341,7 @@ mtx_exit(&sched_lock, MTX_SPIN); splx(s); } + allproc_unlock(); vmmeter(); wakeup((caddr_t)&lbolt); timeout(schedcpu, (void *)0, hz); Index: kern/vfs_syscalls.c =================================================================== RCS file: /home/ncvs/src/sys/kern/vfs_syscalls.c,v retrieving revision 1.172 diff -u -r1.172 vfs_syscalls.c --- kern/vfs_syscalls.c 2000/11/18 21:01:02 1.172 +++ kern/vfs_syscalls.c 2000/11/19 10:12:38 @@ -366,6 +366,7 @@ return; if (VFS_ROOT(olddp->v_mountedhere, &newdp)) panic("mount: lost mount"); + allproc_lock(LK_SHARED); LIST_FOREACH(p, &allproc, p_list) { fdp = p->p_fd; if (fdp->fd_cdir == olddp) { @@ -379,6 +380,7 @@ fdp->fd_rdir = newdp; } } + allproc_unlock(); if (rootvnode == olddp) { vrele(rootvnode); VREF(newdp); Index: sys/proc.h =================================================================== RCS file: /home/ncvs/src/sys/sys/proc.h,v retrieving revision 1.123 diff -u -r1.123 proc.h --- sys/proc.h 2000/11/17 18:09:18 1.123 +++ sys/proc.h 2000/11/19 10:12:38 @@ -45,6 +45,7 @@ #include /* Machine-dependent proc substruct. */ #include /* For struct callout_handle. */ #include +#include /* For lockmgr. */ #include #include /* For struct rtprio. */ #include @@ -483,6 +484,20 @@ struct vm_zone; extern struct vm_zone *proc_zone; + +extern struct lock __allproc_lock; + +static __inline void +allproc_lock(int how) +{ + lockmgr(&__allproc_lock, how, NULL, CURPROC); +} + +static __inline void +allproc_unlock(void) +{ + lockmgr(&__allproc_lock, LK_RELEASE, NULL, CURPROC); +} int enterpgrp __P((struct proc *p, pid_t pgid, int mksess)); void fixjobc __P((struct proc *p, struct pgrp *pgrp, int entering)); Index: vm/vm_glue.c =================================================================== RCS file: /home/ncvs/src/sys/vm/vm_glue.c,v retrieving revision 1.100 diff -u -r1.100 vm_glue.c --- vm/vm_glue.c 2000/10/25 00:04:16 1.100 +++ vm/vm_glue.c 2000/11/19 10:14:55 @@ -360,6 +360,7 @@ pp = NULL; ppri = INT_MIN; + allproc_lock(LK_SHARED); for (p = allproc.lh_first; p != 0; p = p->p_list.le_next) { if (p->p_stat == SRUN && (p->p_flag & (P_INMEM | P_SWAPPING)) == 0) { @@ -381,6 +382,7 @@ } } + allproc_unlock(); /* * Nothing to do, back to sleep. */ @@ -439,6 +441,7 @@ outp = outp2 = NULL; outpri = outpri2 = INT_MIN; + allproc_lock(LK_SHARED); retry: for (p = allproc.lh_first; p != 0; p = p->p_list.le_next) { struct vmspace *vm; @@ -504,6 +507,7 @@ } } } + allproc_unlock(); /* * If we swapped something out, and another process needed memory, * then wakeup the sched process. Index: vm/vm_meter.c =================================================================== RCS file: /home/ncvs/src/sys/vm/vm_meter.c,v retrieving revision 1.41 diff -u -r1.41 vm_meter.c --- vm/vm_meter.c 2000/09/15 22:00:22 1.41 +++ vm/vm_meter.c 2000/11/19 10:12:38 @@ -78,6 +78,7 @@ register int i, nrun; register struct proc *p; + allproc_lock(LK_SHARED); for (nrun = 0, p = allproc.lh_first; p != 0; p = p->p_list.le_next) { switch (p->p_stat) { case SSLEEP: @@ -92,6 +93,7 @@ nrun++; } } + allproc_unlock(); for (i = 0; i < 3; i++) avg->ldavg[i] = (cexp[i] * avg->ldavg[i] + nrun * FSCALE * (FSCALE - cexp[i])) >> FSHIFT; @@ -149,6 +151,7 @@ /* * Calculate process statistics. */ + allproc_lock(LK_SHARED); for (p = allproc.lh_first; p != 0; p = p->p_list.le_next) { if (p->p_flag & P_SYSTEM) continue; @@ -199,6 +202,7 @@ if (paging) totalp->t_pw++; } + allproc_unlock(); /* * Calculate object memory usage statistics. */ Index: vm/vm_object.c =================================================================== RCS file: /home/ncvs/src/sys/vm/vm_object.c,v retrieving revision 1.177 diff -u -r1.177 vm_object.c --- vm/vm_object.c 2000/05/29 22:40:54 1.177 +++ vm/vm_object.c 2000/11/19 10:12:38 @@ -1620,12 +1620,16 @@ vm_object_t object; { struct proc *p; + allproc_lock(LK_SHARED); for (p = allproc.lh_first; p != 0; p = p->p_list.le_next) { if( !p->p_vmspace /* || (p->p_flag & (P_SYSTEM|P_WEXIT)) */) continue; - if( _vm_object_in_map(&p->p_vmspace->vm_map, object, 0)) + if( _vm_object_in_map(&p->p_vmspace->vm_map, object, 0)) { + allproc_unlock(); return 1; + } } + allproc_unlock(); if( _vm_object_in_map( kernel_map, object, 0)) return 1; if( _vm_object_in_map( kmem_map, object, 0)) Index: vm/vm_pageout.c =================================================================== RCS file: /home/ncvs/src/sys/vm/vm_pageout.c,v retrieving revision 1.162 diff -u -r1.162 vm_pageout.c --- vm/vm_pageout.c 2000/11/18 23:06:26 1.162 +++ vm/vm_pageout.c 2000/11/19 10:16:05 @@ -1129,6 +1129,7 @@ if ((vm_swap_size < 64 || swap_pager_full) && vm_page_count_min()) { bigproc = NULL; bigsize = 0; + allproc_lock(LK_SHARED); for (p = allproc.lh_first; p != 0; p = p->p_list.le_next) { /* * if this is a system process, skip it @@ -1158,6 +1159,7 @@ bigsize = size; } } + allproc_unlock(); if (bigproc != NULL) { killproc(bigproc, "out of swap space"); bigproc->p_estcpu = 0; @@ -1442,6 +1444,7 @@ * process is swapped out -- deactivate pages */ + allproc_lock(LK_SHARED); for (p = allproc.lh_first; p != 0; p = p->p_list.le_next) { vm_pindex_t limit, size; @@ -1480,6 +1483,7 @@ &p->p_vmspace->vm_map, limit); } } + allproc_unlock(); } } #endif --==_Exmh_-14298859040-- To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Mon Nov 20 5:33:41 2000 Delivered-To: freebsd-smp@freebsd.org Received: from mail.interware.hu (mail.interware.hu [195.70.32.130]) by hub.freebsd.org (Postfix) with ESMTP id C36EB37B479; Mon, 20 Nov 2000 05:33:29 -0800 (PST) Received: from victoria-203.budapest.interware.hu ([195.70.63.203] helo=elischer.org) by mail.interware.hu with esmtp (Exim 3.16 #1 (Debian)) id 13xr4X-0004nJ-00; Mon, 20 Nov 2000 14:33:25 +0100 Message-ID: <3A192821.13463950@elischer.org> Date: Mon, 20 Nov 2000 05:33:21 -0800 From: Julian Elischer X-Mailer: Mozilla 4.7 [en] (X11; U; FreeBSD 5.0-CURRENT i386) X-Accept-Language: en, hu MIME-Version: 1.0 To: jasone@freebsd.org, arch@freebsd.org, smp@freebsd.org Subject: Re: Threads (KSE etc) comments References: <3A15A2C1.1F3FB6CD@elischer.org> Content-Type: text/plain; charset=iso-8859-15 Content-Transfer-Encoding: 7bit Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org Netscape made an unholy mess of this. I'm going to try send it again and see if I can unscramble the mess. I also clarified a few points. julian. Julian Elischer wrote: > Hi jason. > I read your Nov 7 doc on the threading model. I have a few comments.. I've been thinking about the scheduling queues, and how to make sure that the process (KSEG actually) acts fairly with respect to other processes. I was confised for a while by your description. I think part of my confusion came from something that we specified in the meeting but has not been written in your document directly. Let me see if we are agreed on what we decided.. A KSEG can only have as a maximum N KSEs associated with it, where N is the number of processors, (unless artificially reduced by a lower concurency declaration). (you said this but only indirectly). In general, KSEs are each assigned to a processor. They do not in general move between processors unless some explicit adjustment is being made(*), and as a general rule, two KSEs will not be assigned to the same processor. (in some transitional moments this may be allowed to briefly happen) This in general if you run a KSEC on the same KSE it was run on last time, you should be on the same processor, (and get any affinity advantages that might exist). (*)I am inclined to make the requirement of binding KSEs to processors HARD,as this allows us to simplify some later decisions. For example, if we hard bind KSEs to procesors then since we assign a different communications mailbox for each KSE we create, we can be sure that different KSEs will never preempt each other when writing out to their mailboxes. this also means that since there can only be one UTS incarnation active per KSE (or one KSE per UTS incarnation), that we can not have a UTS preempted by another incarnation on the same processor. We can therefore make sure that there needs to be no locking on mailboxes, or even any checking. I think this is what we decided.. is this correct? The binding is not really mentioned in your document. When we were talking about it, (at least in my memory) Each KSE had a mailbox. My memory of this was that we called a KSE creation call with a different argument, thus each KSE had a different return stack frame when it made upcalls. In the version you have outlined, there is no KSE creation call only KSEG creation calls. Thus all upcalls have the same frame, and there is the danger of colliding upcalls for different processors. I think it works more naturally with everything just 'falling into place' if we have calls to create KSEs rather than KSEGs. The "make KSEG" call is simply a version of the "make KSE" call that also puts it into the new different group. You are left with teh very first 'original' thread being different in my shceme, but my answer to this would be to simply make the first "make KSE" call reuse the current stack etc. and not return a new one. My memory (where is that photo of the whiteboard that Nicole was supposed to send us) is that each KSE is assigned a differnt mailbox address in userland, which is associated with the frame that it will do upcalls on. One of the fields of the mailbox contains a pointer to a userland context structure which contains apece where the KSE should dump the user context should it need to, and a pointer to other such structures. This structure is defined by the kernel, but included in the UTS's 'per thread info'. Since there is one per thread, there is never a problem in running out of them when the kernel links them together in a linked list of completed operations. This is the structure you were talking about as if it were on the KSE upcall stack. In my memory it was preallocated as part of the thread information and completed (or suspended threads (KSECs) are simply linked together inj a list by the kernel, but it doesn;t have to allocate them. (so you discussion about how many need to be provided for is short circuited. When the thread makes a system call, the KSE looks in the mailbox for the context structure for this thread, and if the thread blocks or resumes, it can save any information it needs to tell the UTS there. The UTS sets the pointer into the mailbox when it schedules the thread, so even involintary blockages (e.g. page faults) have the pointer available. When the UTS is running it's own work, it ZERO's this pointer, which lets the kernel know that it is not really in a safe state for pre-emmpting. I think that we decided that a page fault in the UTS simply blocked until it was satisfied. When an upcall occurs, the stack frame it occurs on, and hence the mailbox pointed to are automatically correct, so the UTS doesn't even have to look it up. (the mailbox is allocated as a local variable in the frame of the KSE creation call and is this in the local frame of the upcall. This is something like I imagined the UTS would do to fire off a new KSE. The reason I was thinking of it this way was so that each KSE has a UTS supplied mailbox and (smallish) stack. /* * use make_new_kse() to do exactly that. * Returns -1 on failure and 1 on success. * * cookie allows the UTS to have it's own way of identifying the * KSE/thread. * This stack is effectively lost to us so we first switch * to a small throw-away stack. It need only have enough space in it for * the upcalls to call the UTS, and whatever the UTS will need. * Some time after creation, there will be an upcall on the new KSE * looking for work. * I could imagine wiring this UTS stack down.. */ void start_new_kse(void * cookie, jmp_buf *jb) /*XXX is jb right? */ { struct kse_mailbox; int return_value; bzero(kse_mailbox, sizeof(kse_mailbox)); return_value = kse_new(&kse_mailbox); switch (return_value) { case -1:  perror("make_new_kse() failed"); _longjmp(jb, -1); case 0: printf ("successfully created kse %d\n", kse_mailbox.kse_id); _longjmp(jb, 1); exit (1); /* not reached */ default: printf(" An upcall of type %d occured\n", return_value); uts_scheduler(cookie, &kse_mailbox, return_value); /* must never return */ printf ("it returned!\n"); exit (1); } } make_new_kse(void * cookie) { int retval; jmp_buf env; if ((retval = _setjmp(env)) == 0) { load_new_stack() /* load a new smaller stack, but copy the top 100 bytes or so from the old stack so that our local variables appear to be the same. */ start_new_kse(cookie, env); /* not reached */ } return (retval) } When we have per-processor scheduling queues, there is only at most ONE KSE from any given KSEG in the scheduling queues for any given processor. With the single scheduling queue we have now do we allow N to be in the queues at once? (or do we put the KSEG in instead?) The terms KSE etc. have probably served their useful life. It's time to think of or find names that really describe them better KSE -- a per process processor.. slot? openning? (a-la CAM/SCSI) KSEC ---- stack plus context... KSC..trying to do something (task?) KSEG ---- a class of schedulable entities.. A slot cluster? :-) PROC ---- probably needs to stay the same. -- __--_|\ Julian Elischer / \ julian@elischer.org ( OZ ) World tour 2000 ---> X_.---._/ presently in: Budapest v To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Mon Nov 20 7:57: 2 2000 Delivered-To: freebsd-smp@freebsd.org Received: from zibbi.icomtek.csir.co.za (zibbi.icomtek.csir.co.za [146.64.24.58]) by hub.freebsd.org (Postfix) with ESMTP id BD9A637B479; Mon, 20 Nov 2000 07:56:37 -0800 (PST) Received: (from jhay@localhost) by zibbi.icomtek.csir.co.za (8.11.0/8.11.0) id eAKFtYS92502; Mon, 20 Nov 2000 17:55:34 +0200 (SAT) (envelope-from jhay) From: John Hay Message-Id: <200011201555.eAKFtYS92502@zibbi.icomtek.csir.co.za> Subject: Re: cvs commit: src/sys/kern kern_timeout.c In-Reply-To: <200011171523.eAHFNEV94801@zibbi.icomtek.csir.co.za> from John Hay at "Nov 17, 2000 05:23:14 pm" To: jburkhol@home.com Date: Mon, 20 Nov 2000 17:55:34 +0200 (SAT) Cc: jhb@FreeBSD.ORG, smp@FreeBSD.ORG, cp@bsdi.com X-Mailer: ELM [version 2.4ME+ PL54 (25)] MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org > > > > Here's a patch that seems to work ok here. It works on my UP box running > > X and stuff, I don't notice any hung processes. It works on the SMP box, > > I built a kernel over NFS and it seemed ok, except that I can't > > get a kernel to boot using the serial console with WITNESS enabled. It > > stops at the twiddle thing before the copyright is printed and just hangs. > > This also happens without the patch and even with a UP kernel, so I don't > > really know what's going on. Please try and let me know if you still > > see the hung processes. > > > > Well I have tried your patch and so far it looks good. I have finished > a "make -j13 world" and a "make release NODOC=YES WORLD_FLAGS=-j4" with > no problems on a dual 266MHz PII. It looks like I spoke too soon. After those 2 runs without problems, I haven't been able to get a world or release to finish again. Always some process that gets stuck somewhere with no error or anything. If I kill it it dies and everything seems ok again. One thing that I have noticed the last few times is that there is always a zombie process when this happens and it looks like it is the zombie's parent that gets stuck. If I kill the parent, the zombie goes away and the rest of the processes that was waiting for it to finish untangle themselves. I have also tried allproc2.diff, but it did not seem to make a difference. John -- John Hay -- John.Hay@icomtek.csir.co.za To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Mon Nov 20 9:41:14 2000 Delivered-To: freebsd-smp@freebsd.org Received: from sampnt500.avantgo.com (unknown [209.220.59.68]) by hub.freebsd.org (Postfix) with ESMTP id 46A5837B479; Mon, 20 Nov 2000 09:41:10 -0800 (PST) Received: by sampnt500.avantgo.com with Internet Mail Service (5.5.2650.21) id ; Mon, 20 Nov 2000 09:42:06 -0800 Received: from river.avantgo.com ([10.1.30.114]) by sampnt500.avantgo.com with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2650.21) id W0LZ2H5X; Mon, 20 Nov 2000 09:41:56 -0800 From: Scott Hess To: Julian Elischer Cc: jasone@FreeBSD.ORG, arch@FreeBSD.ORG, smp@FreeBSD.ORG Date: Mon, 20 Nov 2000 09:45:53 -0800 (PST) Subject: Re: Threads (KSE etc) comments In-Reply-To: <3A192821.13463950@elischer.org> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org I have seen the notion of limiting KSE's to the number of CPUs mentioned a number of times on the arch and smp lists. This is presumably done to maximize performance of CPU-bound and network-I/O-bound programs. Please keep disk-I/O-bound programs in mind. The current (3.x/4.x) pthreads implementation is horrid for programs bound by disk I/O. If you have a single multithreaded process doing lots of disk I/Os (let's call it MySQL), then it effectively becomes seek constrained. Converting from single disk to a 6 disk hardware RAID only gains perhaps %20 in throughput, even though throughput should at least double or triple. My informal testing under 3.x with a 6 disk striped set indicated that maximum throughput occurred with 8 seperate processes (running 8 seperate loads), so long as the CPU didn't become a bottleneck. With 1 disk, the maximum was more like three processes - but it was still more than the single process your posting would restrict things to. Presumably more disks would want more processes, with some upper limit where more processes don't help. [No, I don't have the bandwidth to contribute work at this time. I might be able to scrape together some time to characterize the issue more precisely, though, if I knew that my results were going to get folded in. My main goal is that after everything is said and done, there isn't a huge hole intentionally designed into the system!] Later, scott On Mon, 20 Nov 2000, Julian Elischer wrote: > A KSEG can only have as a maximum N KSEs associated with it, where N is > the number of processors, (unless artificially reduced by a lower > concurency declaration). (you said this but only indirectly). In > general, KSEs are each assigned to a processor. They do not in general > move between processors unless some explicit adjustment is being > made(*), and as a general rule, two KSEs will not be assigned to the > same processor. (in some transitional moments this may be allowed to > briefly happen) This in general if you run a KSEC on the same KSE it was > run on last time, you should be on the same processor, > (and get any affinity advantages that might exist). To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Mon Nov 20 10:10:39 2000 Delivered-To: freebsd-smp@freebsd.org Received: from pcnet1.pcnet.com (pcnet1.pcnet.com [204.213.232.3]) by hub.freebsd.org (Postfix) with ESMTP id 6A41C37B479; Mon, 20 Nov 2000 10:10:33 -0800 (PST) Received: (from eischen@localhost) by pcnet1.pcnet.com (8.8.7/PCNet) id NAA21043; Mon, 20 Nov 2000 13:10:11 -0500 (EST) Date: Mon, 20 Nov 2000 13:10:10 -0500 (EST) From: Daniel Eischen To: Scott Hess Cc: Julian Elischer , jasone@FreeBSD.ORG, arch@FreeBSD.ORG, smp@FreeBSD.ORG Subject: Re: Threads (KSE etc) comments In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org On Mon, 20 Nov 2000, Scott Hess wrote: > I have seen the notion of limiting KSE's to the number of CPUs mentioned a > number of times on the arch and smp lists. This is presumably done to > maximize performance of CPU-bound and network-I/O-bound programs. > > Please keep disk-I/O-bound programs in mind. The current (3.x/4.x) > pthreads implementation is horrid for programs bound by disk I/O. If you > have a single multithreaded process doing lots of disk I/Os (let's call it > MySQL), then it effectively becomes seek constrained. Converting from > single disk to a 6 disk hardware RAID only gains perhaps %20 in > throughput, even though throughput should at least double or triple. > > My informal testing under 3.x with a 6 disk striped set indicated that > maximum throughput occurred with 8 seperate processes (running 8 seperate > loads), so long as the CPU didn't become a bottleneck. With 1 disk, the > maximum was more like three processes - but it was still more than the > single process your posting would restrict things to. Presumably more > disks would want more processes, with some upper limit where more > processes don't help. You can't compare one multi-threaded process with our current threads implementation to one KSE/KSEG. With one CPU, and therefore 1 KSE within the KSEG, a thread really blocks in the kernel waiting for disk I/O. Unlike our current threads implementation, when that happens under the KSE implementation, the KSE continues to run by issuing an upcall to the threads library to schedule another thread. This can repeat itself, allowing many threads to be blocked in the kernel waiting for disk I/O, but the KSE will not block unless it runs out of resources (needed to save kernel state for threads blocked in the kernel), is preempted by a higher priority process or kernel thread, or exhausts its quantum. With 2 CPUs, you could have 2 KSEs within the KSEG. Nothing changes except that the KSEG workload is spread across 2 KSEs running on different CPUs. The quantum for the KSEG, regardless of whether there are 1 or more KSEs associated with it, remains the same (I would advocate a separate quantum in this case, but I think I'm outvoted on that front). This is all for PTHREAD_SCOPE_PROCESS threads. PTHREAD_SCOPE_SYSTEM threads will be bound to their own KSEG with one KSE. These will act more like LinuxThreads; when one of these threads block in the kernel, another one will not be executed (because there are no other threads allocated for that KSEG/KSE by the threads library). -- Dan Eischen To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Mon Nov 20 11:18:54 2000 Delivered-To: freebsd-smp@freebsd.org Received: from io.yi.org (h24-69-199-88.gv.shawcable.net [24.69.199.88]) by hub.freebsd.org (Postfix) with ESMTP id 34A5637B479; Mon, 20 Nov 2000 11:18:51 -0800 (PST) Received: from io.yi.org (localhost.gvcl1.bc.wave.home.com [127.0.0.1]) by io.yi.org (Postfix) with ESMTP id 74C17BA7A; Mon, 20 Nov 2000 11:18:47 -0800 (PST) X-Mailer: exmh version 2.1.1 10/15/1999 To: John Hay Cc: jhb@FreeBSD.ORG, smp@FreeBSD.ORG, cp@bsdi.com Subject: Re: cvs commit: src/sys/kern kern_timeout.c In-Reply-To: Message from John Hay of "Sat, 20 Nov 2000 17:55:34 +0200." <200011201555.eAKFtYS92502@zibbi.icomtek.csir.co.za> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Mon, 20 Nov 2000 11:18:47 -0800 From: Jake Burkholder Message-Id: <20001120191847.74C17BA7A@io.yi.org> Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org > > > > > > Here's a patch that seems to work ok here. It works on my UP box running > > > X and stuff, I don't notice any hung processes. It works on the SMP box, > > > I built a kernel over NFS and it seemed ok, except that I can't > > > get a kernel to boot using the serial console with WITNESS enabled. It > > > stops at the twiddle thing before the copyright is printed and just hangs. > > > This also happens without the patch and even with a UP kernel, so I don't > > > really know what's going on. Please try and let me know if you still > > > see the hung processes. > > > > > > > Well I have tried your patch and so far it looks good. I have finished > > a "make -j13 world" and a "make release NODOC=YES WORLD_FLAGS=-j4" with > > no problems on a dual 266MHz PII. > > It looks like I spoke too soon. After those 2 runs without problems, I > haven't been able to get a world or release to finish again. Always > some process that gets stuck somewhere with no error or anything. If I > kill it it dies and everything seems ok again. > > One thing that I have noticed the last few times is that there is always > a zombie process when this happens and it looks like it is the zombie's > parent that gets stuck. If I kill the parent, the zombie goes away and > the rest of the processes that was waiting for it to finish untangle > themselves. I noticed this too. I'm going to start another build world here to see if I can find anything else out. One problem is that moving the system calls out from under Giant and adding the context stealing light-weight interrupt threads will change things a lot. I suspect that some of these problems will just go away. > > I have also tried allproc2.diff, but it did not seem to make a difference. I didn't expect it to. Its not a bug fix, its just to get moving on making the proc layer mp safe, which so far hasn't been happening. Thanks, Jake To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Mon Nov 20 12:14: 4 2000 Delivered-To: freebsd-smp@freebsd.org Received: from pike.osd.bsdi.com (pike.osd.bsdi.com [204.216.28.222]) by hub.freebsd.org (Postfix) with ESMTP id 90E8D37B4C5 for ; Mon, 20 Nov 2000 12:14:01 -0800 (PST) Received: from laptop.baldwin.cx (john@jhb-laptop.osd.bsdi.com [204.216.28.241]) by pike.osd.bsdi.com (8.11.0/8.9.3) with ESMTP id eAKKDuv82593; Mon, 20 Nov 2000 12:13:56 -0800 (PST) (envelope-from jhb@FreeBSD.org) Message-ID: X-Mailer: XFMail 1.4.0 on FreeBSD X-Priority: 3 (Normal) Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 8bit MIME-Version: 1.0 In-Reply-To: <20001120080655.5BEA4BA7A@io.yi.org> Date: Mon, 20 Nov 2000 12:14:29 -0800 (PST) From: John Baldwin To: Jake Burkholder Subject: Re: patch: allproc_lock Cc: smp@FreeBSD.org Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org On 20-Nov-00 Jake Burkholder wrote: >> >> Attached is a patch that adds allproc_lock. This is a lockmgr >> lock which protectes the following: >> >> allproc >> zombproc >> pidhashtbl >> proc.p_list >> proc.p_hash >> nextpid >> >> Please review it. > > Sorry, too quick with the send... Well, I kind of don't like having allproc_lock/unlock functions. I would prefer explicit lockmgr() calls with on an allproc_lock variable. Also, we might consider having a very early SI_SUB_LOCKS_INIT that initializes lots of locks such as allproc_lock and the callout wheel mutex. Also, did you see my lock list document? Looks like I need to update it so it can handle lockmgr locks. :-P > Here's the patch: -- John Baldwin -- http://www.FreeBSD.org/~jhb/ PGP Key: http://www.baldwin.cx/~john/pgpkey.asc "Power Users Use the Power to Serve!" - http://www.FreeBSD.org/ To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Mon Nov 20 12:14:19 2000 Delivered-To: freebsd-smp@freebsd.org Received: from pike.osd.bsdi.com (pike.osd.bsdi.com [204.216.28.222]) by hub.freebsd.org (Postfix) with ESMTP id 083C737B4C5 for ; Mon, 20 Nov 2000 12:14:17 -0800 (PST) Received: from laptop.baldwin.cx (john@jhb-laptop.osd.bsdi.com [204.216.28.241]) by pike.osd.bsdi.com (8.11.0/8.9.3) with ESMTP id eAKKDrv82588; Mon, 20 Nov 2000 12:13:55 -0800 (PST) (envelope-from jhb@FreeBSD.org) Message-ID: X-Mailer: XFMail 1.4.0 on FreeBSD X-Priority: 3 (Normal) Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 8bit MIME-Version: 1.0 In-Reply-To: <200011201555.eAKFtYS92502@zibbi.icomtek.csir.co.za> Date: Mon, 20 Nov 2000 12:14:25 -0800 (PST) From: John Baldwin To: John Hay Subject: Re: cvs commit: src/sys/kern kern_timeout.c Cc: cp@bsdi.com, smp@FreeBSD.org, jburkhol@home.com Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org On 20-Nov-00 John Hay wrote: >> > >> > Here's a patch that seems to work ok here. It works on my UP box running >> > X and stuff, I don't notice any hung processes. It works on the SMP box, >> > I built a kernel over NFS and it seemed ok, except that I can't >> > get a kernel to boot using the serial console with WITNESS enabled. It >> > stops at the twiddle thing before the copyright is printed and just hangs. >> > This also happens without the patch and even with a UP kernel, so I don't >> > really know what's going on. Please try and let me know if you still >> > see the hung processes. >> > >> >> Well I have tried your patch and so far it looks good. I have finished >> a "make -j13 world" and a "make release NODOC=YES WORLD_FLAGS=-j4" with >> no problems on a dual 266MHz PII. > > It looks like I spoke too soon. After those 2 runs without problems, I > haven't been able to get a world or release to finish again. Always > some process that gets stuck somewhere with no error or anything. If I > kill it it dies and everything seems ok again. > > One thing that I have noticed the last few times is that there is always > a zombie process when this happens and it looks like it is the zombie's > parent that gets stuck. If I kill the parent, the zombie goes away and > the rest of the processes that was waiting for it to finish untangle > themselves. This is the bug involving CURSIG() in msleep(). > I have also tried allproc2.diff, but it did not seem to make a difference. > > John > -- > John Hay -- John.Hay@icomtek.csir.co.za -- John Baldwin -- http://www.FreeBSD.org/~jhb/ PGP Key: http://www.baldwin.cx/~john/pgpkey.asc "Power Users Use the Power to Serve!" - http://www.FreeBSD.org/ To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Mon Nov 20 12:30:49 2000 Delivered-To: freebsd-smp@freebsd.org Received: from io.yi.org (h24-69-199-88.gv.shawcable.net [24.69.199.88]) by hub.freebsd.org (Postfix) with ESMTP id DE28237B479; Mon, 20 Nov 2000 12:30:45 -0800 (PST) Received: from io.yi.org (localhost.gvcl1.bc.wave.home.com [127.0.0.1]) by io.yi.org (Postfix) with ESMTP id 3DAB8BA7A; Mon, 20 Nov 2000 12:30:45 -0800 (PST) X-Mailer: exmh version 2.1.1 10/15/1999 To: John Baldwin Cc: smp@FreeBSD.ORG Subject: Re: patch: allproc_lock In-Reply-To: Message from John Baldwin of "Mon, 20 Nov 2000 12:14:29 PST." Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Mon, 20 Nov 2000 12:30:45 -0800 From: Jake Burkholder Message-Id: <20001120203045.3DAB8BA7A@io.yi.org> Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org > > On 20-Nov-00 Jake Burkholder wrote: > >> > >> Attached is a patch that adds allproc_lock. This is a lockmgr > >> lock which protectes the following: > >> > >> allproc > >> zombproc > >> pidhashtbl > >> proc.p_list > >> proc.p_hash > >> nextpid > >> > >> Please review it. > > > > Sorry, too quick with the send... > > Well, I kind of don't like having allproc_lock/unlock functions. > I would prefer explicit lockmgr() calls with on an allproc_lock > variable. Also, we might consider having a very early Sure, how come? I did it this way because lockmgr takes so many parameters which I expect will always be the same; the lock should always be gotten on behalf of curproc, etc. > SI_SUB_LOCKS_INIT that initializes lots of locks such as allproc_lock > and the callout wheel mutex. Also, did you see my lock list document? > Looks like I need to update it so it can handle lockmgr locks. :-P Yeah, I figured you were eating turkey this weekend :) I'll write up an entry for the callout_lock if you want. It locks the callwheel, ticks, softticks and nextsoftcheck. You might also want to account for stuff that's not locked, but is safe by virtue of atomicity. I think astpending falls into this category since its per-cpu. > > > Here's the patch: > > -- > > John Baldwin -- http://www.FreeBSD.org/~jhb/ > PGP Key: http://www.baldwin.cx/~john/pgpkey.asc > "Power Users Use the Power to Serve!" - http://www.FreeBSD.org/ > > > To Unsubscribe: send mail to majordomo@FreeBSD.org > with "unsubscribe freebsd-smp" in the body of the message To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Mon Nov 20 12:55:51 2000 Delivered-To: freebsd-smp@freebsd.org Received: from pike.osd.bsdi.com (pike.osd.bsdi.com [204.216.28.222]) by hub.freebsd.org (Postfix) with ESMTP id 17B4537B479 for ; Mon, 20 Nov 2000 12:55:45 -0800 (PST) Received: from laptop.baldwin.cx (john@jhb-laptop.osd.bsdi.com [204.216.28.241]) by pike.osd.bsdi.com (8.11.0/8.9.3) with ESMTP id eAKKtgi84155; Mon, 20 Nov 2000 12:55:42 -0800 (PST) (envelope-from jhb@FreeBSD.org) Message-ID: X-Mailer: XFMail 1.4.0 on FreeBSD X-Priority: 3 (Normal) Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 8bit MIME-Version: 1.0 In-Reply-To: <20001120203045.3DAB8BA7A@io.yi.org> Date: Mon, 20 Nov 2000 12:56:15 -0800 (PST) From: John Baldwin To: Jake Burkholder Subject: Re: patch: allproc_lock Cc: smp@FreeBSD.org Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org On 20-Nov-00 Jake Burkholder wrote: >> >> On 20-Nov-00 Jake Burkholder wrote: >> >> >> >> Attached is a patch that adds allproc_lock. This is a lockmgr >> >> lock which protectes the following: >> >> >> >> allproc >> >> zombproc >> >> pidhashtbl >> >> proc.p_list >> >> proc.p_hash >> >> nextpid >> >> >> >> Please review it. >> > >> > Sorry, too quick with the send... >> >> Well, I kind of don't like having allproc_lock/unlock functions. >> I would prefer explicit lockmgr() calls with on an allproc_lock >> variable. Also, we might consider having a very early > > Sure, how come? I did it this way because lockmgr takes so many > parameters which I expect will always be the same; the lock should > always be gotten on behalf of curproc, etc. I just don't want us to end up with lots of foo_lock/unlock functions. It is a bad trend to start. Imagine if each mutex had two such functions associated with it. One slightly cleaner solution might be to create macros read_lock(lock), write_lock(lock) and init_rwlock(lock) that map to lockmgr calls underneath. >> SI_SUB_LOCKS_INIT that initializes lots of locks such as allproc_lock >> and the callout wheel mutex. Also, did you see my lock list document? >> Looks like I need to update it so it can handle lockmgr locks. :-P > > Yeah, I figured you were eating turkey this weekend :) Actually, I was, but I have more to eat on Thursday. :) > I'll write up an entry for the callout_lock if you want. It > locks the callwheel, ticks, softticks and nextsoftcheck. Ok. I guess I should go ahead and commit the first version then. It is in DocBook (looks kind of HTML-ish) but it shouldn't be that hard to pick up. It will end up in doc/en_US.ISO-8859.1/articles/mutex_list/. > You might also want to account for stuff that's not locked, but > is safe by virtue of atomicity. I think astpending falls into this > category since its per-cpu. Yeah. Hmm. That's why I had question marks by astpending and friends. Also, many of the things currently listed as protected by sched_lock may not be in all cases, and many of them shouldn't be protectd by sched_lcok (I think p_stat is one that should, but most of the rest of them shouldn't be.) Getting process stuff locked down alongside of the KSE work is going to be interesting though. :-P -- John Baldwin -- http://www.FreeBSD.org/~jhb/ PGP Key: http://www.baldwin.cx/~john/pgpkey.asc "Power Users Use the Power to Serve!" - http://www.FreeBSD.org/ To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Mon Nov 20 13:57:58 2000 Delivered-To: freebsd-smp@freebsd.org Received: from smtp02.primenet.com (smtp02.primenet.com [206.165.6.132]) by hub.freebsd.org (Postfix) with ESMTP id 0C8D137B4D7; Mon, 20 Nov 2000 13:57:49 -0800 (PST) Received: (from daemon@localhost) by smtp02.primenet.com (8.9.3/8.9.3) id OAA02759; Mon, 20 Nov 2000 14:53:37 -0700 (MST) Received: from usr08.primenet.com(206.165.6.208) via SMTP by smtp02.primenet.com, id smtpdAAAm6aate; Mon Nov 20 14:52:34 2000 Received: (from tlambert@localhost) by usr08.primenet.com (8.8.5/8.8.5) id OAA02067; Mon, 20 Nov 2000 14:56:36 -0700 (MST) From: Terry Lambert Message-Id: <200011202156.OAA02067@usr08.primenet.com> Subject: Re: Threads (KSE etc) comments To: scott@avantgo.com (Scott Hess) Date: Mon, 20 Nov 2000 21:56:36 +0000 (GMT) Cc: julian@elischer.org (Julian Elischer), jasone@FreeBSD.ORG, arch@FreeBSD.ORG, smp@FreeBSD.ORG In-Reply-To: from "Scott Hess" at Nov 20, 2000 09:45:53 AM X-Mailer: ELM [version 2.5 PL2] MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org > I have seen the notion of limiting KSE's to the number of CPUs mentioned a > number of times on the arch and smp lists. This is presumably done to > maximize performance of CPU-bound and network-I/O-bound programs. A KSE is a "reservation for quantum". It is effectively a quantum holder. The quantum can be spread against multiple threads in user space. Except on multiple CPUs, which would permit multiple KSEs, all CPU utilization is serialized through the scheduler anyway. What a KSE buys you is thread group locality for context switcthes, which is something that is computationally difficult to achieve satisfactorily in a Linux or SVR4 approach to kernel threading, since it can quickly lead to starvation deadlock, where a threaded process gets affinity preference for its ready-to-run threads in the scheduler, thus starving everything else in the scheduler queue which is not the threaded process. In other words, it's incredibly hard to achieve affinity after you have broken the quantum-process accounting connection. KSEs preserve this connection. This typically means that, if you are attempting to use multiple threads to unfairly compete for quantum, you will have to follow the scheduler priority rules instead (many threads programmers use threads in an attempt to subvert the scheduler policy without resorting to increased priviledges; this is wrong). So your examples are all workable, merely by providing multiple user space threads. > On Mon, 20 Nov 2000, Julian Elischer wrote: > > A KSEG can only have as a maximum N KSEs associated with it, where N is > > the number of processors, (unless artificially reduced by a lower > > concurency declaration). (you said this but only indirectly). In > > general, KSEs are each assigned to a processor. They do not in general > > move between processors unless some explicit adjustment is being > > made(*), and as a general rule, two KSEs will not be assigned to the > > same processor. (in some transitional moments this may be allowed to > > briefly happen) This in general if you run a KSEC on the same KSE it was > > run on last time, you should be on the same processor, > > (and get any affinity advantages that might exist). I really, really disagree with Julian's statement about not assigning multiple KSEs to the same processor. There are perfectly valid loading reasons (e.g. two KSEs that each take 30% of the CPU, cooperating, with one CPU with a process that takse 90% of the CPU, while the other CPU is only at 60% utilization, etc.). There is also the PTHREAD_SCOPE_SYSTEM scheduling; one can easily conceive of a multithreading application, where one thread in the total list of threads needs to run with a real time priority to insure rapid interacative response, with metered intervals. One example would be a DVD player, where the rendering needs to occur at fixed intervals with no latency, whereas the fuffer-refill from the DVDROM, and the control functons for the program, overall, can run in normal priority threads. So I think there is a good argument for needing to support in the kernel the idea of multiple KSEs on a single processor; I understand Julian's anxiety about what this could do to the complexity of the sleep queues, but with respect to everyone, if the task's too complex, then stay out of the kernel (it's not too complex for Julian, it's just I don't think that the kernel should have to be written on the computer science equivalent of a 6th grade reading leveli: if it can't be understood by everyone, too bad: most people couldn't tell you how the ignition system in their car works, either). Terry Lambert terry@lambert.org --- Any opinions in this posting are my own and not those of my present or previous employers. To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Mon Nov 20 14: 5:28 2000 Delivered-To: freebsd-smp@freebsd.org Received: from smtp05.primenet.com (smtp05.primenet.com [206.165.6.135]) by hub.freebsd.org (Postfix) with ESMTP id 00F6337B4E5; Mon, 20 Nov 2000 14:05:22 -0800 (PST) Received: (from daemon@localhost) by smtp05.primenet.com (8.9.3/8.9.3) id PAA09966; Mon, 20 Nov 2000 15:05:57 -0700 (MST) Received: from usr08.primenet.com(206.165.6.208) via SMTP by smtp05.primenet.com, id smtpdAAAuVaGAt; Mon Nov 20 15:05:52 2000 Received: (from tlambert@localhost) by usr08.primenet.com (8.8.5/8.8.5) id PAA02205; Mon, 20 Nov 2000 15:05:14 -0700 (MST) From: Terry Lambert Message-Id: <200011202205.PAA02205@usr08.primenet.com> Subject: Re: Threads (KSE etc) comments To: eischen@vigrid.com (Daniel Eischen) Date: Mon, 20 Nov 2000 22:05:14 +0000 (GMT) Cc: scott@avantgo.com (Scott Hess), julian@elischer.org (Julian Elischer), jasone@FreeBSD.ORG, arch@FreeBSD.ORG, smp@FreeBSD.ORG In-Reply-To: from "Daniel Eischen" at Nov 20, 2000 01:10:10 PM X-Mailer: ELM [version 2.5 PL2] MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org > With 2 CPUs, you could have 2 KSEs within the KSEG. Nothing changes > except that the KSEG workload is spread across 2 KSEs running on > different CPUs. The quantum for the KSEG, regardless of whether > there are 1 or more KSEs associated with it, remains the same > (I would advocate a separate quantum in this case, but I think > I'm outvoted on that front). I have to disagree. Doing inter-CPU quantum accounting is a _big_ mistake. The reason that it would happen this way in the current code is that there is not the concept of per CPU run queues, something which is absolutely necesary, going forward. I think the only policy decision that should require priviledges in this case is M KSEs in an N CPU system, where M>N. For the most part, I would argue that this limitation can be enforced transparently, in the user space portion of the threads scheduler, and that overriding it requires priviledge. I might go so far as to say that the only place it is reasonable to override this is the PTHREAD_SCOPE_SYSTEM case, and/or for RT priority for one or more threads (which already requires priviledges). Competing for quantum on a single CPU as if you were more than one process, well.... that should take more than one process, unless you have priviledges. One gross hack might be to limit the number of total KSEs to the number of permitted child processes for a given process, but I'd hesitate to encourage people to use the not generally exposed and non (POSIX) standard interfaces that would be needed for a process to try to do this automatically. Terry Lambert terry@lambert.org --- Any opinions in this posting are my own and not those of my present or previous employers. To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Mon Nov 20 14:32:13 2000 Delivered-To: freebsd-smp@freebsd.org Received: from pcnet1.pcnet.com (pcnet1.pcnet.com [204.213.232.3]) by hub.freebsd.org (Postfix) with ESMTP id 23F7D37B657; Mon, 20 Nov 2000 14:32:05 -0800 (PST) Received: (from eischen@localhost) by pcnet1.pcnet.com (8.8.7/PCNet) id RAA02819; Mon, 20 Nov 2000 17:31:42 -0500 (EST) Date: Mon, 20 Nov 2000 17:31:42 -0500 (EST) From: Daniel Eischen To: Terry Lambert Cc: Scott Hess , Julian Elischer , jasone@FreeBSD.ORG, arch@FreeBSD.ORG, smp@FreeBSD.ORG Subject: Re: Threads (KSE etc) comments In-Reply-To: <200011202205.PAA02205@usr08.primenet.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org On Mon, 20 Nov 2000, Terry Lambert wrote: > > With 2 CPUs, you could have 2 KSEs within the KSEG. Nothing changes > > except that the KSEG workload is spread across 2 KSEs running on > > different CPUs. The quantum for the KSEG, regardless of whether > > there are 1 or more KSEs associated with it, remains the same > > (I would advocate a separate quantum in this case, but I think > > I'm outvoted on that front). > > I have to disagree. Doing inter-CPU quantum accounting is a > _big_ mistake. The reason that it would happen this way in > the current code is that there is not the concept of per CPU > run queues, something which is absolutely necesary, going > forward. So you agree with me then? > I think the only policy decision that should require priviledges > in this case is M KSEs in an N CPU system, where M>N. For the > most part, I would argue that this limitation can be enforced > transparently, in the user space portion of the threads scheduler, > and that overriding it requires priviledge. I might go so far > as to say that the only place it is reasonable to override this > is the PTHREAD_SCOPE_SYSTEM case, and/or for RT priority for one > or more threads (which already requires priviledges). I agree also. > Competing for quantum on a single CPU as if you were more than > one process, well.... that should take more than one process, > unless you have priviledges. With the exception of PTHREAD_SCOPE_PROCESS, where the thread scheduler (UTS hereafter) allocates/requests one KSEG with exactly one KSE. > One gross hack might be to limit the number of total KSEs to > the number of permitted child processes for a given process, > but I'd hesitate to encourage people to use the not generally > exposed and non (POSIX) standard interfaces that would be > needed for a process to try to do this automatically. You have to be careful with terminology. If we're going by what Jason has defined in his paper, the KSEG is the entity that has the quantum, not the KSE. So the KSEGs would be limited to the permitted number of child processes. If you want to have a separate quantum for each KSE, then you can probably eliminate the KSEG. I've made this comment also. -- Dan Eischen To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Mon Nov 20 14:39:33 2000 Delivered-To: freebsd-smp@freebsd.org Received: from pcnet1.pcnet.com (pcnet1.pcnet.com [204.213.232.3]) by hub.freebsd.org (Postfix) with ESMTP id 67E3737B479; Mon, 20 Nov 2000 14:39:28 -0800 (PST) Received: (from eischen@localhost) by pcnet1.pcnet.com (8.8.7/PCNet) id RAA03845; Mon, 20 Nov 2000 17:39:06 -0500 (EST) Date: Mon, 20 Nov 2000 17:39:06 -0500 (EST) From: Daniel Eischen To: Terry Lambert Cc: Scott Hess , Julian Elischer , jasone@FreeBSD.ORG, arch@FreeBSD.ORG, smp@FreeBSD.ORG Subject: Re: Threads (KSE etc) comments In-Reply-To: <200011202156.OAA02067@usr08.primenet.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org On Mon, 20 Nov 2000, Terry Lambert wrote: > > I have seen the notion of limiting KSE's to the number of CPUs mentioned a > > number of times on the arch and smp lists. This is presumably done to > > maximize performance of CPU-bound and network-I/O-bound programs. > > A KSE is a "reservation for quantum". It is effectively a quantum > holder. The quantum can be spread against multiple threads in user > space. Except on multiple CPUs, which would permit multiple KSEs, > all CPU utilization is serialized through the scheduler anyway. This isn't what has been defined. The KSEG is the "reservation for quantum", but I do tend to agree that it should in fact be the KSE. > > On Mon, 20 Nov 2000, Julian Elischer wrote: > > > A KSEG can only have as a maximum N KSEs associated with it, where N is > > > the number of processors, (unless artificially reduced by a lower > > > concurency declaration). (you said this but only indirectly). In > > > general, KSEs are each assigned to a processor. They do not in general > > > move between processors unless some explicit adjustment is being > > > made(*), and as a general rule, two KSEs will not be assigned to the > > > same processor. (in some transitional moments this may be allowed to > > > briefly happen) This in general if you run a KSEC on the same KSE it was > > > run on last time, you should be on the same processor, > > > (and get any affinity advantages that might exist). > > I really, really disagree with Julian's statement about not > assigning multiple KSEs to the same processor. There are > perfectly valid loading reasons (e.g. two KSEs that each take > 30% of the CPU, cooperating, with one CPU with a process that > takse 90% of the CPU, while the other CPU is only at 60% > utilization, etc.). > > There is also the PTHREAD_SCOPE_SYSTEM scheduling; one can > easily conceive of a multithreading application, where one thread > in the total list of threads needs to run with a real time > priority to insure rapid interacative response, with metered > intervals. One example would be a DVD player, where the > rendering needs to occur at fixed intervals with no latency, > whereas the fuffer-refill from the DVDROM, and the control > functons for the program, overall, can run in normal priority > threads. We have an application that has very similar needs. Right now we use Solaris, with some threads in RT and some not in RT. I intend on making this work in FreeBSD also. -- Dan Eischen To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Mon Nov 20 20:15:17 2000 Delivered-To: freebsd-smp@freebsd.org Received: from io.yi.org (h24-69-199-88.gv.shawcable.net [24.69.199.88]) by hub.freebsd.org (Postfix) with ESMTP id 69D0837B4C5; Mon, 20 Nov 2000 20:15:13 -0800 (PST) Received: from io.yi.org (localhost.gvcl1.bc.wave.home.com [127.0.0.1]) by io.yi.org (Postfix) with ESMTP id B3BADBA7A; Mon, 20 Nov 2000 20:15:12 -0800 (PST) X-Mailer: exmh version 2.1.1 10/15/1999 To: John Baldwin Cc: smp@FreeBSD.ORG Subject: Re: patch: allproc_lock In-Reply-To: Message from John Baldwin of "Mon, 20 Nov 2000 12:56:15 PST." Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Mon, 20 Nov 2000 20:15:12 -0800 From: Jake Burkholder Message-Id: <20001121041512.B3BADBA7A@io.yi.org> Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org > > On 20-Nov-00 Jake Burkholder wrote: > >> > >> On 20-Nov-00 Jake Burkholder wrote: > >> >> > >> >> Attached is a patch that adds allproc_lock. This is a lockmgr > >> >> lock which protectes the following: > >> >> > >> >> allproc > >> >> zombproc > >> >> pidhashtbl > >> >> proc.p_list > >> >> proc.p_hash > >> >> nextpid > >> >> > >> >> Please review it. > >> > > >> > Sorry, too quick with the send... > >> > >> Well, I kind of don't like having allproc_lock/unlock functions. > >> I would prefer explicit lockmgr() calls with on an allproc_lock > >> variable. Also, we might consider having a very early > > > > Sure, how come? I did it this way because lockmgr takes so many > > parameters which I expect will always be the same; the lock should > > always be gotten on behalf of curproc, etc. > > I just don't want us to end up with lots of foo_lock/unlock functions. It is a > bad trend to start. Imagine if each mutex had two such functions associated > with it. One slightly cleaner solution might be to create macros > read_lock(lock), write_lock(lock) and init_rwlock(lock) that map to lockmgr > calls underneath. Yeah, ok. I think I'll just use lockmgr explicitly. Its possible that in some cases we'll want to use the interlock in the future. > > >> SI_SUB_LOCKS_INIT that initializes lots of locks such as allproc_lock > >> and the callout wheel mutex. Also, did you see my lock list document? > >> Looks like I need to update it so it can handle lockmgr locks. :-P > > > > Yeah, I figured you were eating turkey this weekend :) > > Actually, I was, but I have more to eat on Thursday. :) > > > I'll write up an entry for the callout_lock if you want. It > > locks the callwheel, ticks, softticks and nextsoftcheck. > > Ok. I guess I should go ahead and commit the first version then. It is in > DocBook (looks kind of HTML-ish) but it shouldn't be that hard to pick up. It > will end up in doc/en_US.ISO-8859.1/articles/mutex_list/. Yeah, please commit it. I'm vaguely familiar with docbook, I'm sure I can figure it out. > > > You might also want to account for stuff that's not locked, but > > is safe by virtue of atomicity. I think astpending falls into this > > category since its per-cpu. > > Yeah. Hmm. That's why I had question marks by astpending and friends. Also, > many of the things currently listed as protected by sched_lock may not be in > all cases, and many of them shouldn't be protectd by sched_lcok (I think p_stat > is one that should, but most of the rest of them shouldn't be.) This needs careful review, but of course it'll all change with KSEs :) > > Getting process stuff locked down alongside of the KSE work is going to be > interesting though. :-P > Yeah, I don't know how this is going to work. Based on how long it took to get smpng off the ground, and the state that its in now, it seems nuts to me to try to get both these things done and well tested before 5.0 goes out, but I guess we'll see. > -- > > John Baldwin -- http://www.FreeBSD.org/~jhb/ > PGP Key: http://www.baldwin.cx/~john/pgpkey.asc > "Power Users Use the Power to Serve!" - http://www.FreeBSD.org/ > > > To Unsubscribe: send mail to majordomo@FreeBSD.org > with "unsubscribe freebsd-smp" in the body of the message To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Mon Nov 20 23:35:47 2000 Delivered-To: freebsd-smp@freebsd.org Received: from mail.interware.hu (mail.interware.hu [195.70.32.130]) by hub.freebsd.org (Postfix) with ESMTP id 8A6A037B479; Mon, 20 Nov 2000 23:35:42 -0800 (PST) Received: from pretoria-36.budapest.interware.hu ([195.70.53.100] helo=elischer.org) by mail.interware.hu with esmtp (Exim 3.16 #1 (Debian)) id 13y7xr-00015F-00; Tue, 21 Nov 2000 08:35:39 +0100 Message-ID: <3A1A25C6.99B1B8F3@elischer.org> Date: Mon, 20 Nov 2000 23:35:34 -0800 From: Julian Elischer X-Mailer: Mozilla 4.7 [en] (X11; U; FreeBSD 5.0-CURRENT i386) X-Accept-Language: en, hu MIME-Version: 1.0 To: Daniel Eischen Cc: Scott Hess , jasone@FreeBSD.ORG, arch@FreeBSD.ORG, smp@FreeBSD.ORG Subject: Re: Threads (KSE etc) comments References: Content-Type: text/plain; charset=iso-8859-15 Content-Transfer-Encoding: 7bit Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org Daniel Eischen wrote: > > With 2 CPUs, you could have 2 KSEs within the KSEG. Nothing changes > except that the KSEG workload is spread across 2 KSEs running on > different CPUs. The quantum for the KSEG, regardless of whether > there are 1 or more KSEs associated with it, remains the same > (I would advocate a separate quantum in this case, but I think > I'm outvoted on that front). not entirely I think a process using two KSEs in a KSEG should be charged for 2. > > This is all for PTHREAD_SCOPE_PROCESS threads. PTHREAD_SCOPE_SYSTEM > threads will be bound to their own KSEG with one KSE. These will > act more like LinuxThreads; when one of these threads block in > the kernel, another one will not be executed (because there are no > other threads allocated for that KSEG/KSE by the threads library). well, it WILL do the upcall but the UTS will respond with "No new thread to schedule, please yield" > > -- > Dan Eischen > > To Unsubscribe: send mail to majordomo@FreeBSD.org > with "unsubscribe freebsd-arch" in the body of the message -- __--_|\ Julian Elischer / \ julian@elischer.org ( OZ ) World tour 2000 ---> X_.---._/ presently in: Budapest v To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Mon Nov 20 23:41:34 2000 Delivered-To: freebsd-smp@freebsd.org Received: from mail.interware.hu (mail.interware.hu [195.70.32.130]) by hub.freebsd.org (Postfix) with ESMTP id EEA2837B4D7; Mon, 20 Nov 2000 23:41:22 -0800 (PST) Received: from pretoria-36.budapest.interware.hu ([195.70.53.100] helo=elischer.org) by mail.interware.hu with esmtp (Exim 3.16 #1 (Debian)) id 13y83I-0001GV-00; Tue, 21 Nov 2000 08:41:17 +0100 Message-ID: <3A1A2717.3270D40F@elischer.org> Date: Mon, 20 Nov 2000 23:41:11 -0800 From: Julian Elischer X-Mailer: Mozilla 4.7 [en] (X11; U; FreeBSD 5.0-CURRENT i386) X-Accept-Language: en, hu MIME-Version: 1.0 To: Terry Lambert Cc: Scott Hess , jasone@FreeBSD.ORG, arch@FreeBSD.ORG, smp@FreeBSD.ORG Subject: Re: Threads (KSE etc) comments References: <200011202156.OAA02067@usr08.primenet.com> Content-Type: text/plain; charset=iso-8859-15 Content-Transfer-Encoding: 7bit Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org Terry Lambert wrote: > > > I have seen the notion of limiting KSE's to the number of CPUs mentioned a > > number of times on the arch and smp lists. This is presumably done to > > maximize performance of CPU-bound and network-I/O-bound programs. > > A KSE is a "reservation for quantum". It is effectively a quantum > holder. The quantum can be spread against multiple threads in user > space. Except on multiple CPUs, which would permit multiple KSEs, > all CPU utilization is serialized through the scheduler anyway. > > What a KSE buys you is thread group locality for context switcthes, > which is something that is computationally difficult to achieve > satisfactorily in a Linux or SVR4 approach to kernel threading, > since it can quickly lead to starvation deadlock, where a threaded > process gets affinity preference for its ready-to-run threads in > the scheduler, thus starving everything else in the scheduler queue > which is not the threaded process. > > In other words, it's incredibly hard to achieve affinity after > you have broken the quantum-process accounting connection. KSEs > preserve this connection. > > This typically means that, if you are attempting to use multiple > threads to unfairly compete for quantum, you will have to follow > the scheduler priority rules instead (many threads programmers > use threads in an attempt to subvert the scheduler policy without > resorting to increased priviledges; this is wrong). > > So your examples are all workable, merely by providing multiple > user space threads. > > > On Mon, 20 Nov 2000, Julian Elischer wrote: > > > A KSEG can only have as a maximum N KSEs associated with it, where N is > > > the number of processors, (unless artificially reduced by a lower > > > concurency declaration). (you said this but only indirectly). In > > > general, KSEs are each assigned to a processor. They do not in general > > > move between processors unless some explicit adjustment is being > > > made(*), and as a general rule, two KSEs will not be assigned to the > > > same processor. (in some transitional moments this may be allowed to > > > briefly happen) This in general if you run a KSEC on the same KSE it was > > > run on last time, you should be on the same processor, > > > (and get any affinity advantages that might exist). > > I really, really disagree with Julian's statement about not > assigning multiple KSEs to the same processor. There are > perfectly valid loading reasons (e.g. two KSEs that each take > 30% of the CPU, cooperating, with one CPU with a process that > takse 90% of the CPU, while the other CPU is only at 60% > utilization, etc.). NOT within the same KSEG... don't forget that to some extent the amount of CPU each KSE will get will depend on how much work it has.. hopefully the KSEs with 30% loads will have done 'yields'. > > There is also the PTHREAD_SCOPE_SYSTEM scheduling; one can > easily conceive of a multithreading application, where one thread > in the total list of threads needs to run with a real time > priority to insure rapid interacative response, with metered > intervals. One example would be a DVD player, where the > rendering needs to occur at fixed intervals with no latency, > whereas the fuffer-refill from the DVDROM, and the control > functons for the program, overall, can run in normal priority > threads. This is ONE KSE PER CPU PER KSEG...... you can have several KSE's competing for a processor. put them in differnt KSEGs. > > So I think there is a good argument for needing to support in > the kernel the idea of multiple KSEs on a single processor; I > understand Julian's anxiety about what this could do to the > complexity of the sleep queues, but with respect to everyone, > if the task's too complex, then stay out of the kernel (it's > not too complex for Julian, it's just I don't think that the > kernel should have to be written on the computer science > equivalent of a 6th grade reading leveli: if it can't be > understood by everyone, too bad: most people couldn't tell > you how the ignition system in their car works, either). you misunderstood my comment > > Terry Lambert > terry@lambert.org > --- > Any opinions in this posting are my own and not those of my present > or previous employers. -- __--_|\ Julian Elischer / \ julian@elischer.org ( OZ ) World tour 2000 ---> X_.---._/ presently in: Budapest v To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Mon Nov 20 23:54:27 2000 Delivered-To: freebsd-smp@freebsd.org Received: from mail.interware.hu (mail.interware.hu [195.70.32.130]) by hub.freebsd.org (Postfix) with ESMTP id 6C40037B479; Mon, 20 Nov 2000 23:54:20 -0800 (PST) Received: from pretoria-36.budapest.interware.hu ([195.70.53.100] helo=elischer.org) by mail.interware.hu with esmtp (Exim 3.16 #1 (Debian)) id 13y8Fu-0001sg-00; Tue, 21 Nov 2000 08:54:19 +0100 Message-ID: <3A1A2A26.4CF0B849@elischer.org> Date: Mon, 20 Nov 2000 23:54:14 -0800 From: Julian Elischer X-Mailer: Mozilla 4.7 [en] (X11; U; FreeBSD 5.0-CURRENT i386) X-Accept-Language: en, hu MIME-Version: 1.0 To: arch@FreeBSD.ORG, smp@FreeBSD.ORG Subject: Re: Threads (KSE etc) comments References: Content-Type: text/plain; charset=iso-8859-15 Content-Transfer-Encoding: 7bit Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org CC's trimmed, which group should stay? SMP or ARCH? Daniel Eischen wrote: > > > With the exception of where the thread > scheduler (UTS hereafter) allocates/requests one KSEG with > exactly one KSE. there is no reason why ANY KSEG need sto be limitted to one KSE by the kernel unless there is only one CPU. The PTHREAD_SCOPE_PROCESS, KSEG may have one or more KSEs assigned to it depending on whether the UTS wants to create more or not.... Thinking about what Terry said, I guess you could have M>N (M KSEs, N CPUs) in a KSEG, but I don't think it would be useful to do so. > > > One gross hack might be to limit the number of total KSEs to > > the number of permitted child processes for a given process, > > but I'd hesitate to encourage people to use the not generally > > exposed and non (POSIX) standard interfaces that would be > > needed for a process to try to do this automatically. > > You have to be careful with terminology. If we're going by > what Jason has defined in his paper, the KSEG is the entity > that has the quantum, not the KSE. So the KSEGs would be > limited to the permitted number of child processes. yes, but that gives the ability to use M times as much CPU as a nonthreaded process. > > If you want to have a separate quantum for each KSE, then > you can probably eliminate the KSEG. I've made this comment > also. It's possible that we may be able to do so, but not yet.. > > -- > Dan Eischen -- __--_|\ Julian Elischer / \ julian@elischer.org ( OZ ) World tour 2000 ---> X_.---._/ presently in: Budapest v To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Mon Nov 20 23:56:23 2000 Delivered-To: freebsd-smp@freebsd.org Received: from mail.interware.hu (mail.interware.hu [195.70.32.130]) by hub.freebsd.org (Postfix) with ESMTP id 65A2237B4C5; Mon, 20 Nov 2000 23:56:19 -0800 (PST) Received: from pretoria-36.budapest.interware.hu ([195.70.53.100] helo=elischer.org) by mail.interware.hu with esmtp (Exim 3.16 #1 (Debian)) id 13y8Hq-0001zC-00; Tue, 21 Nov 2000 08:56:18 +0100 Message-ID: <3A1A2A9D.BA6A8943@elischer.org> Date: Mon, 20 Nov 2000 23:56:13 -0800 From: Julian Elischer X-Mailer: Mozilla 4.7 [en] (X11; U; FreeBSD 5.0-CURRENT i386) X-Accept-Language: en, hu MIME-Version: 1.0 To: arch@FreeBSD.ORG, smp@FreeBSD.ORG Subject: Re: Threads (KSE etc) comments References: Content-Type: text/plain; charset=iso-8859-15 Content-Transfer-Encoding: 7bit Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org Daniel Eischen wrote: > > We have an application that has very similar needs. Right now > we use Solaris, with some threads in RT and some not in RT. > I intend on making this work in FreeBSD also. that's what KSEGs are for... > > -- > Dan Eischen > > To Unsubscribe: send mail to majordomo@FreeBSD.org > with "unsubscribe freebsd-arch" in the body of the message -- __--_|\ Julian Elischer / \ julian@elischer.org ( OZ ) World tour 2000 ---> X_.---._/ presently in: Budapest v To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Tue Nov 21 3:59:51 2000 Delivered-To: freebsd-smp@freebsd.org Received: from pcnet1.pcnet.com (pcnet1.pcnet.com [204.213.232.3]) by hub.freebsd.org (Postfix) with ESMTP id 7FDFF37B4C5; Tue, 21 Nov 2000 03:59:46 -0800 (PST) Received: (from eischen@localhost) by pcnet1.pcnet.com (8.8.7/PCNet) id GAA13723; Tue, 21 Nov 2000 06:59:22 -0500 (EST) Date: Tue, 21 Nov 2000 06:59:22 -0500 (EST) From: Daniel Eischen To: Julian Elischer Cc: Scott Hess , jasone@FreeBSD.ORG, arch@FreeBSD.ORG, smp@FreeBSD.ORG Subject: Re: Threads (KSE etc) comments In-Reply-To: <3A1A25C6.99B1B8F3@elischer.org> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org On Mon, 20 Nov 2000, Julian Elischer wrote: > Daniel Eischen wrote: > > > > > With 2 CPUs, you could have 2 KSEs within the KSEG. Nothing changes > > except that the KSEG workload is spread across 2 KSEs running on > > different CPUs. The quantum for the KSEG, regardless of whether > > there are 1 or more KSEs associated with it, remains the same > > (I would advocate a separate quantum in this case, but I think > > I'm outvoted on that front). > > not entirely > I think a process using two KSEs in a KSEG should be charged for 2. I agree if each KSE gets its own quantum. The way Jason explained it to me (in the case that the KSEG has the quantum), is that if 2 KSEs (in the same KSEG) run concurrently on 2 CPUs, then they would run for 1 quantum each. But the resource usage in the KSEG would be incremented by 2 quantum. > > This is all for PTHREAD_SCOPE_PROCESS threads. PTHREAD_SCOPE_SYSTEM > > threads will be bound to their own KSEG with one KSE. These will > > act more like LinuxThreads; when one of these threads block in > > the kernel, another one will not be executed (because there are no > > other threads allocated for that KSEG/KSE by the threads library). > > well, it WILL do the upcall but the UTS will respond with > "No new thread to schedule, please yield" Yes, exactly because the kernel doesn't know which KSEG is scope process and which KSEGs are scope system. I suppose the kernel could easily know, but there's no reason for it to care. -- Dan Eischen To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Tue Nov 21 10:25:39 2000 Delivered-To: freebsd-smp@freebsd.org Received: from smtp05.primenet.com (smtp05.primenet.com [206.165.6.135]) by hub.freebsd.org (Postfix) with ESMTP id AD36A37B4C5; Tue, 21 Nov 2000 10:25:31 -0800 (PST) Received: (from daemon@localhost) by smtp05.primenet.com (8.9.3/8.9.3) id LAA24260; Tue, 21 Nov 2000 11:26:07 -0700 (MST) Received: from usr08.primenet.com(206.165.6.208) via SMTP by smtp05.primenet.com, id smtpdAAAIbaq_U; Tue Nov 21 11:25:40 2000 Received: (from tlambert@localhost) by usr08.primenet.com (8.8.5/8.8.5) id LAA27595; Tue, 21 Nov 2000 11:24:59 -0700 (MST) From: Terry Lambert Message-Id: <200011211824.LAA27595@usr08.primenet.com> Subject: Re: Threads (KSE etc) comments To: julian@elischer.org (Julian Elischer) Date: Tue, 21 Nov 2000 18:24:59 +0000 (GMT) Cc: arch@FreeBSD.ORG, smp@FreeBSD.ORG In-Reply-To: <3A1A2A26.4CF0B849@elischer.org> from "Julian Elischer" at Nov 20, 2000 11:54:14 PM X-Mailer: ELM [version 2.5 PL2] MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org > CC's trimmed, > > which group should stay? SMP or ARCH? ARCH is chartered for the discussion, SMP is where the work is actually geting done. I think that's the reason for the original poster to cross-post. I'll let clearer heads prevail by picking where to sent their response. > there is no reason why ANY KSEG need sto be limitted to one KSE > by the kernel unless there is only one CPU. > The PTHREAD_SCOPE_PROCESS, KSEG may have one or more KSEs > assigned to it depending > on whether the UTS wants to create more or not.... > > Thinking about what Terry said, I guess you could have M>N > (M KSEs, N CPUs) in a KSEG, but I don't think it would be > useful to do so. It lets me have M simultaneous blocking operations outstanding; it's only not useful in the context of KSEGs. I guess we could arrange it this way: (1 KSE, 1 KSEG), (M KSEGs, N CPUs) This should make it more obvious: I want M quanta from N CPUs. Or in other words, I want to compete as M porcesses on this N CPU system. The reason for this is to allow me to control my own interleave; for example, if we look at the "team" or "ddd" programs, both of which use interleaved I/O using multiple processes to improve overall throughput, we see an application where I _do not_ want 2 quanta for 3 threads, with 2 of the threads bound to only one of the quanta: doing that will lose me 25% of my throughput. If you want to visualize the scheduling order, think of a three element braid (braids are neat: the simplest is a three element, and you can think of it as a graph of FIFO queue, two elements at a time). If there is equal work, then the work on each of two elements is going to complete in alternating order, which means that if you have a third element waiting, it will be alternately assigned (ideally; KSEGs break this). I think that perhaps the best use of threading is to try to interleave I/O to ensure against I/O binding. KSEGs seem to be designed with the idea that the threaded application is CPU, not I/O bound, which is where I think the mistake lies. > > > One gross hack might be to limit the number of total KSEs to > > > the number of permitted child processes for a given process, > > > but I'd hesitate to encourage people to use the not generally > > > exposed and non (POSIX) standard interfaces that would be > > > needed for a process to try to do this automatically. > > > > You have to be careful with terminology. If we're going by > > what Jason has defined in his paper, the KSEG is the entity > > that has the quantum, not the KSE. So the KSEGs would be > > limited to the permitted number of child processes. > > yes, but that gives the ability to use M times as much CPU as a > nonthreaded process. If you won't give it to me, I'll just take it, instead, either by using rfork() and a shared memory segment for my global data, which gets me the equivalenet of a threaded environment, for all practical purposes, or I'll just fork() and run multiple instances of myself. Either way, you don't get to tell me not to compete as multiple processes, and I can throw a KSEG based policy out the window, without needing your permission to do it. Worse, there is additional context switch overhead introduced by this (the same reason Linux kernel threads are a bad idea), and everyone gets to pay the penalty for my application. Terry Lambert terry@lambert.org --- Any opinions in this posting are my own and not those of my present or previous employers. To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Tue Nov 21 13:23:27 2000 Delivered-To: freebsd-smp@freebsd.org Received: from pcnet1.pcnet.com (pcnet1.pcnet.com [204.213.232.3]) by hub.freebsd.org (Postfix) with ESMTP id BF69137B4CF; Tue, 21 Nov 2000 13:23:22 -0800 (PST) Received: (from eischen@localhost) by pcnet1.pcnet.com (8.8.7/PCNet) id QAA09525; Tue, 21 Nov 2000 16:23:01 -0500 (EST) Date: Tue, 21 Nov 2000 16:23:00 -0500 (EST) From: Daniel Eischen To: Terry Lambert Cc: Julian Elischer , arch@FreeBSD.ORG, smp@FreeBSD.ORG Subject: Re: Threads (KSE etc) comments In-Reply-To: <200011211824.LAA27595@usr08.primenet.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org On Tue, 21 Nov 2000, Terry Lambert wrote: > > yes, but that gives the ability to use M times as much CPU as a > > nonthreaded process. > > If you won't give it to me, I'll just take it, instead, either > by using rfork() and a shared memory segment for my global data, > which gets me the equivalenet of a threaded environment, for all > practical purposes, or I'll just fork() and run multiple instances > of myself. Either way, you don't get to tell me not to compete > as multiple processes, and I can throw a KSEG based policy out the > window, without needing your permission to do it. Worse, there > is additional context switch overhead introduced by this (the > same reason Linux kernel threads are a bad idea), and everyone > gets to pay the penalty for my application. I have to agree with Terry. You can't dictate what application writers will do. If they want more CPU and they can't get it with the threading model we provide, then they will get it another way. Limiting PTHREAD_SCOPE_PROCESS threads to 1 KSEG with only 1 quantum doesn't stop someone from [r]fork()'ing to get more CPU. Let's admit that this is what some applications will want to do and allow them to do it within our threading model. No, we won't do it by default, but we can provide simple hooks to allow an application to request more "CPU reservations" (a KSEG under the current definition). Note that I am only talking about scope process threads. Scope system threads are bound to their own KSEG/KSE pair. -- Dan Eischen To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Tue Nov 21 13:28:58 2000 Delivered-To: freebsd-smp@freebsd.org Received: from alpha.netaccess.on.ca (alpha.netaccess.on.ca [199.243.225.10]) by hub.freebsd.org (Postfix) with ESMTP id 7502237B479; Tue, 21 Nov 2000 13:28:52 -0800 (PST) Received: from schizo.controlq.com (dial162.nas.net [207.176.144.162]) by alpha.netaccess.on.ca (8.9.0/8.9.0) with ESMTP id QAA11149; Tue, 21 Nov 2000 16:28:50 -0500 (EST) Date: Tue, 21 Nov 2000 16:34:57 -0500 (EST) From: "Robert S. Sciuk" Cc: arch@FreeBSD.ORG, smp@FreeBSD.ORG Subject: Re: Threads (KSE etc) comments In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org This may be out of line, but in line with the cpu allocation question, is there (will there be) a mechanism for processor affinity determination, and changes?? This is important wrt cache coherency and producer/consumer type processes/threads. Just wondering. On Tue, 21 Nov 2000, Daniel Eischen wrote: > On Tue, 21 Nov 2000, Terry Lambert wrote: > > > yes, but that gives the ability to use M times as much CPU as a > > > nonthreaded process. > > > > If you won't give it to me, I'll just take it, instead, either > > by using rfork() and a shared memory segment for my global data, > > which gets me the equivalenet of a threaded environment, for all > > practical purposes, or I'll just fork() and run multiple instances > > of myself. Either way, you don't get to tell me not to compete > > as multiple processes, and I can throw a KSEG based policy out the > > window, without needing your permission to do it. Worse, there > > is additional context switch overhead introduced by this (the > > same reason Linux kernel threads are a bad idea), and everyone > > gets to pay the penalty for my application. > > I have to agree with Terry. You can't dictate what application > writers will do. If they want more CPU and they can't get it > with the threading model we provide, then they will get it > another way. Limiting PTHREAD_SCOPE_PROCESS threads to 1 KSEG > with only 1 quantum doesn't stop someone from [r]fork()'ing > to get more CPU. > > Let's admit that this is what some applications will want to > do and allow them to do it within our threading model. No, > we won't do it by default, but we can provide simple hooks to > allow an application to request more "CPU reservations" > (a KSEG under the current definition). > > Note that I am only talking about scope process threads. > Scope system threads are bound to their own KSEG/KSE pair. > > -- > Dan Eischen > > > To Unsubscribe: send mail to majordomo@FreeBSD.org > with "unsubscribe freebsd-smp" in the body of the message > -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= Robert S. Sciuk http://www.controlq.com 1032 Howard Rd. RR#2 Control-Q Research tel: 905.632.2466 Burlington, Ont. rob@controlq.com fax: 905.632.7417 Canada, L7R 3X5 To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Tue Nov 21 15:55:43 2000 Delivered-To: freebsd-smp@freebsd.org Received: from mail.interware.hu (mail.interware.hu [195.70.32.130]) by hub.freebsd.org (Postfix) with ESMTP id EE2F737B4CF; Tue, 21 Nov 2000 15:55:33 -0800 (PST) Received: from mogadishu-42.budapest.interware.hu ([195.70.52.106] helo=elischer.org) by mail.interware.hu with esmtp (Exim 3.16 #1 (Debian)) id 13yNFx-0002On-00; Wed, 22 Nov 2000 00:55:21 +0100 Message-ID: <3A1B0B64.6D694248@elischer.org> Date: Tue, 21 Nov 2000 15:55:16 -0800 From: Julian Elischer X-Mailer: Mozilla 4.7 [en] (X11; U; FreeBSD 5.0-CURRENT i386) X-Accept-Language: en, hu MIME-Version: 1.0 To: arch@FreeBSD.ORG Cc: smp@FreeBSD.ORG Subject: Re: Threads (KSE etc) comments References: Content-Type: text/plain; charset=iso-8859-15 Content-Transfer-Encoding: 7bit Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org Note to all on FreeBSD-SMP.. this is to ARCH, and CC'd to SMP I'll trim SMP next time around so if you are reading this in SMP and are not on 'arch'...... Daniel Eischen wrote: > > On Tue, 21 Nov 2000, Terry Lambert wrote: > > > yes, but that gives the ability to use M times as much CPU as a > > > nonthreaded process. > > > > If you won't give it to me, I'll just take it, instead, either > > by using rfork() and a shared memory segment for my global data, > > which gets me the equivalenet of a threaded environment, for all > > practical purposes, or I'll just fork() and run multiple instances > > of myself. Either way, you don't get to tell me not to compete > > as multiple processes, and I can throw a KSEG based policy out the > > window, without needing your permission to do it. Worse, there > > is additional context switch overhead introduced by this (the > > same reason Linux kernel threads are a bad idea), and everyone > > gets to pay the penalty for my application. > > I have to agree with Terry. You can't dictate what application > writers will do. If they want more CPU and they can't get it > with the threading model we provide, then they will get it > another way. Limiting PTHREAD_SCOPE_PROCESS threads to 1 KSEG > with only 1 quantum doesn't stop someone from [r]fork()'ing > to get more CPU. At this stage I'm having the following thoughts.. 1/ I think we've almost talked ourselves out of the need for a KSEG. 2/ I don't want to actually say that until I've thought about it a bit.. there were some REALLY GOOD REASONS for it before, I just can't think what they were. I'll probably remember at 3AM this morning while I'm mostly asleep. My remaining use for KSEGs is for housekeeping: 3/ My KSEG thought was: The user can take as many quanta as he wants by having many KSEGs. If he uses default actions and simply fires off a lot of threads, he gets N quanta, per 'round-robin cycle, where N is the number of CPUs, because by default the threads package only fires up a max of N KSEs to run the threads on in each KSEG. The KSEG is an 'official' way for him to be more greedy. but he has to ask for it.. One KSEG gives a single timeslot per round-robin, on each CPU. but KSEs in the same KSEG don't compete with each other in the KSEG. 4/ I am not sure that #3 is completely useful these days either, I'm thinking about it. 5/ it's possible that if KSEs in the same KSEG can never be on the same CPU then they can never 'pre-empt' each other. This means that locking between the UTS instances in the same KSE can predict that locks on structures held private within the KSE can never 'block', but will always 'spin-out' in a short time (assuming the UTS is written correctly). This means that (for example) threads assigned to the same KSEG can be migrated around easily within the KSEG with very simple locking. If there are two KSE's in teh same KSEG on teh same CPU, there is always the danger that one may pre-empt the other while it holds a lock on a KSEG internal structure, and then we might deadlock. (this thought is also rather vague) Basically, KSE's are to provide concurrency across processors, where KSEGs are to provide competition in the scheduler. I see the two as being orthogonal. KSEGs may go away, or may be simply a 'virtual element' that never actually exists. but I still see the use of confining 'normal threads' to some bounded box. The fact that we give you the ability to make more boxes at you leasure shouldn't detract from the usefulness of the box. Scheduling within the box (and migration therin) needs to be fast and may happen very frequently. Moving between boxes is probably less frequent. For that reason I like having the boxes well defined in a way that ensures that they are well behaved. making each box contain only one KSE from each CPU allows as much concurrency as you can get, yet makes sure that we don't have to deal with KSEs tripping over each other. I think it will make the use of simple (tiny) spinlocks the most that will be needed for such things as process migration within the box. If you start having KSEs (there is one UTS 'incarnation' per KSE) tripping over each other's CPUs it starts to get more complicated. Having said that, I'm still thinking about whether KSEGs can go away. > Let's admit that this is what some applications will want to > do and allow them to do it within our threading model. No, > we won't do it by default, but we can provide simple hooks to > allow an application to request more "CPU reservations" > (a KSEG under the current definition). exactly We allow them to be greedy. but we make them do slightly more work for it. At least they KNOW they are being greedy. > > Note that I am only talking about scope process threads. > Scope system threads are bound to their own KSEG/KSE pair. KSEG/KSEs If A SCOPE_SYSTEM thread creates a new thread, that thread competes with it on however many KSEs it has, within it's KSEG. > > -- > Dan Eischen Was my explanation of how KSEs communicate with the UTS clear enough? It's hard to decide that for your own writing.... -- __--_|\ Julian Elischer / \ julian@elischer.org ( OZ ) World tour 2000 ---> X_.---._/ presently in: Budapest v To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Wed Nov 22 11:20: 7 2000 Delivered-To: freebsd-smp@freebsd.org Received: from enterprise.gullo.com.br (gullo-gw.widesoft.com.br [200.246.206.70]) by hub.freebsd.org (Postfix) with ESMTP id D32AE37B4D7 for ; Wed, 22 Nov 2000 11:19:56 -0800 (PST) Received: from mtv.gullo.com.br (mtv.gullo.com.br [192.168.1.101]) by enterprise.gullo.com.br (8.9.3/8.8.7) with SMTP id RAA15980 for ; Wed, 22 Nov 2000 17:25:47 -0200 Date: Wed, 22 Nov 2000 17:21:02 -0200 From: Marcos Tischer Vallim To: freebsd-smp@freebsd.org Subject: HELP ME PLEASE!!! Message-Id: <3A1C1C9EFA.3196MTV@smtp.gullo.com.br> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: Quoted-Printable X-Mailer: Becky! ver 1.26.05 Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org I have Netfinity 3500 dual processor PIII 677 server 512MRam and two Hd SCSI on a FreeBSD 4.1 and a SCSI CD room. My question is, how can compily the Kernel to run on SMP. I have already look at FreeBSD sites and docs, faq, everything about this, and I didn`t get my answer. I compily the Kernel with the parameters found in these docs,but my problem still here at kernel`s boot the HD SCSI, I mean at the FS`s mounting. These are the paramenters that I got with an mptable command # SMP kernel config file options: # Required: options SMP # Symmetric MultiProcessor Kernel options APIC_IO # Symmetric (APIC) I/O # Optional (built-in defaults will work in most cases): #options NCPU=3d2 # number of CPUs #options NBUS=3d3 # number of busses #options NAPIC=3d2 # number of IO APICs #options NINTR=3d24 # number of INTs So, if you can help me, I`d be glad. Tk's ----------------------------------- Irm=e3os Gullo SA Artefatos de Metais CPD - Administrador Marcos Tischer Vallim UIN : 63731454 Tel : +55 019 451 6966 ramal 249 +55 019 9149 3332 ----------------------------------- To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Wed Nov 22 11:25: 5 2000 Delivered-To: freebsd-smp@freebsd.org Received: from wormhole.bluestar.net (wormhole.bluestar.net [208.53.1.61]) by hub.freebsd.org (Postfix) with ESMTP id 6491C37B4CF for ; Wed, 22 Nov 2000 11:25:02 -0800 (PST) Received: from planetwe.com (admin.planetwe.com [64.182.69.146]) by wormhole.bluestar.net (8.10.1/8.10.1) with ESMTP id eAMJOPQ14334; Wed, 22 Nov 2000 13:24:26 -0600 (CST) Message-ID: <3A1C1D69.1010905@planetwe.com> Date: Wed, 22 Nov 2000 13:24:25 -0600 From: Drew Sanford User-Agent: Mozilla/5.0 (X11; U; Linux 2.2.12 i386; en-GB; m18) Gecko/20001107 Netscape6/6.0 X-Accept-Language: en MIME-Version: 1.0 To: Marcos Tischer Vallim Cc: freebsd-smp@freebsd.org Subject: Re: HELP ME PLEASE!!! References: <3A1C1C9EFA.3196MTV@smtp.gullo.com.br> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 8bit Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org Marcos, did your server boot ok with the GENERIC kernel? If so, all you should have to do, is uncomment the SMP and APIC_IO options on a GERNERIC kernel. Marcos Tischer Vallim wrote: > I have Netfinity 3500 dual processor PIII 677 server 512MRam and > two Hd SCSI on a FreeBSD 4.1 and a SCSI CD room. My question is, how can > compily the Kernel to run on SMP. > I have already look at FreeBSD sites and docs, faq, everything > about this, and I didn`t get my answer. I compily the Kernel with the > parameters found in these docs,but my problem still here at kernel`s > boot the HD SCSI, I mean at the FS`s mounting. > > These are the paramenters that I got with an mptable command > > # SMP kernel config file options: > > > # Required: > options SMP # Symmetric MultiProcessor Kernel > options APIC_IO # Symmetric (APIC) I/O > > # Optional (built-in defaults will work in most cases): > #options NCPU=2 # number of CPUs > #options NBUS=3 # number of busses > #options NAPIC=2 # number of IO APICs > #options NINTR=24 # number of INTs > > So, if you can help me, I`d be glad. > > Tk's > > ----------------------------------- > Irmãos Gullo SA Artefatos de Metais > CPD - Administrador > Marcos Tischer Vallim > UIN : 63731454 > Tel : +55 019 451 6966 ramal 249 > +55 019 9149 3332 > ----------------------------------- > > > To Unsubscribe: send mail to majordomo@FreeBSD.org > with "unsubscribe freebsd-smp" in the body of the message -- Drew Sanford Systems Administrator Planetwe.com Email: drew@planetwe.com To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Thu Nov 23 11: 4:49 2000 Delivered-To: freebsd-smp@freebsd.org Received: from mail.interware.hu (mail.interware.hu [195.70.32.130]) by hub.freebsd.org (Postfix) with ESMTP id EA85737B4C5 for ; Thu, 23 Nov 2000 11:04:47 -0800 (PST) Received: from pretoria-31.budapest.interware.hu ([195.70.53.95] helo=elischer.org) by mail.interware.hu with esmtp (Exim 3.16 #1 (Debian)) id 13z1fp-0004Ie-00 for ; Thu, 23 Nov 2000 20:04:46 +0100 Message-ID: <3A1D35A4.EEE78A42@elischer.org> Date: Thu, 23 Nov 2000 07:20:04 -0800 From: Julian Elischer X-Mailer: Mozilla 4.7 [en] (X11; U; FreeBSD 5.0-CURRENT i386) X-Accept-Language: en, hu MIME-Version: 1.0 To: smp@freebsd.org Subject: duplicated fields in proc.c Content-Type: text/plain; charset=iso-8859-15 Content-Transfer-Encoding: 7bit Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org The following lines appear twice in proc.c in the same structure. (once commented out). I presume this is someone's typo? LIST_HEAD(ihhead, intrhand) it_ihhead; LIST_HEAD(srchead, isrc) it_isrchead; /* Fields used by all interrupt threads */ LIST_ENTRY(ithd) it_list; /* All interrupt threads */ -- __--_|\ Julian Elischer / \ julian@elischer.org ( OZ ) World tour 2000 ---> X_.---._/ presently in: Budapest v To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Thu Nov 23 12:40:30 2000 Delivered-To: freebsd-smp@freebsd.org Received: from enterprise.gullo.com.br (gullo-gw.widesoft.com.br [200.246.206.70]) by hub.freebsd.org (Postfix) with ESMTP id 6BE5437B479 for ; Thu, 23 Nov 2000 12:40:26 -0800 (PST) Received: from mtv.gullo.com.br (mtv.gullo.com.br [192.168.1.101]) by enterprise.gullo.com.br (8.9.3/8.8.7) with SMTP id SAA29958 for ; Thu, 23 Nov 2000 18:46:34 -0200 Date: Thu, 23 Nov 2000 18:41:27 -0200 From: Marcos Tischer Vallim To: freebsd-smp@freebsd.org Subject: Netfinity 8500 M20 Dual Processor PIII with SCSI Message-Id: <3A1D80F723A.1883MTV@smtp.gullo.com.br> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: Quoted-Printable X-Mailer: Becky! ver 1.26.05 Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org I have Netfinity 3500 dual processor PIII 677 server 512MRam and two Hd SCSI (controler SCSI is Adaptec Ultra 60) on a FreeBSD 4.1 and a SCSI CD room. My question is, how can compily the Kernel to run on SMP. I have already look at FreeBSD sites and docs, faq, everything about this, and I didn`t get my answer. I compily the Kernel with the parameters found in these docs,but my problem still here at kernel`s boot the HD SCSI, I mean at the FS`s mounting. These are the paramenters that I got with an mptable command # SMP kernel config file options: # Required: options SMP # Symmetric MultiProcessor Kernel options APIC_IO # Symmetric (APIC) I/O # Optional (built-in defaults will work in most cases): #options NCPU=3d2 # number of CPUs #options NBUS=3d3 # number of busses #options NAPIC=3d2 # number of IO APICs #options NINTR=3d24 # number of INTs So, if you can help me, I`d be glad. Tk's ----------------------------------- Irm=e3os Gullo SA Artefatos de Metais CPD - Administrador Marcos Tischer Vallim UIN : 63731454 Tel : +55 019 451 6966 ramal 249 +55 019 9149 3332 ----------------------------------- To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Thu Nov 23 13:58: 6 2000 Delivered-To: freebsd-smp@freebsd.org Received: from enterprise.gullo.com.br (gullo-gw.widesoft.com.br [200.246.206.70]) by hub.freebsd.org (Postfix) with ESMTP id 0ED5437B4CF for ; Thu, 23 Nov 2000 13:58:01 -0800 (PST) Received: from mtv.gullo.com.br (mtv.gullo.com.br [192.168.1.101]) by enterprise.gullo.com.br (8.9.3/8.8.7) with SMTP id UAA30688 for ; Thu, 23 Nov 2000 20:04:16 -0200 Date: Thu, 23 Nov 2000 19:59:08 -0200 From: Marcos Tischer Vallim To: freebsd-smp@freebsd.org Subject: Adaptec Ultra160 (AIC-7892) Message-Id: <3A1D932C32.1884MTV@smtp.gullo.com.br> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: Quoted-Printable X-Mailer: Becky! ver 1.26.05 Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org Some body know, some drive for this SCSI controler?(ahc, aic, aha...) What function for SMP?? TK's ----------------------------------- Irm=e3os Gullo SA Artefatos de Metais CPD - Administrador Marcos Tischer Vallim UIN : 63731454 Tel : +55 019 451 6966 ramal 249 +55 019 9149 3332 ----------------------------------- To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Thu Nov 23 15: 6: 4 2000 Delivered-To: freebsd-smp@freebsd.org Received: from mail.interware.hu (mail.interware.hu [195.70.32.130]) by hub.freebsd.org (Postfix) with ESMTP id 41EC637B4E5 for ; Thu, 23 Nov 2000 15:06:00 -0800 (PST) Received: from kairo-29.budapest.interware.hu ([195.70.50.93] helo=elischer.org) by mail.interware.hu with esmtp (Exim 3.16 #1 (Debian)) id 13z5RF-0000Ay-00 for ; Fri, 24 Nov 2000 00:05:58 +0100 Message-ID: <3A1DA2CD.6E819769@elischer.org> Date: Thu, 23 Nov 2000 15:05:49 -0800 From: Julian Elischer X-Mailer: Mozilla 4.7 [en] (X11; U; FreeBSD 5.0-CURRENT i386) X-Accept-Language: en, hu MIME-Version: 1.0 To: smp@freebsd.org Subject: Re: duplicated fields in proc.c References: <3A1D35A4.EEE78A42@elischer.org> Content-Type: text/plain; charset=iso-8859-15 Content-Transfer-Encoding: 7bit Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org Julian Elischer wrote: > > The following lines appear twice in proc.c in the same that was of course proc.h > structure. (once commented out). I presume this is someone's typo? > > LIST_HEAD(ihhead, intrhand) it_ihhead; > LIST_HEAD(srchead, isrc) it_isrchead; > > /* Fields used by all interrupt threads */ > LIST_ENTRY(ithd) it_list; /* All interrupt threads */ > -- __--_|\ Julian Elischer / \ julian@elischer.org ( OZ ) World tour 2000 ---> X_.---._/ presently in: Budapest v To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Sat Nov 25 4:17:53 2000 Delivered-To: freebsd-smp@freebsd.org Received: from fw.wintelcom.net (ns1.wintelcom.net [209.1.153.20]) by hub.freebsd.org (Postfix) with ESMTP id 2CFAF37B4D7 for ; Sat, 25 Nov 2000 04:17:48 -0800 (PST) Received: (from bright@localhost) by fw.wintelcom.net (8.10.0/8.10.0) id eAPCHlp20299 for smp@freebsd.org; Sat, 25 Nov 2000 04:17:47 -0800 (PST) Date: Sat, 25 Nov 2000 04:17:47 -0800 From: Alfred Perlstein To: smp@freebsd.org Subject: uidinfo patches, review please. Message-ID: <20001125041747.I8051@fw.wintelcom.net> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2.5i Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org Protect the uidinfo structures. My only concern is that I must now include sys/mutex.h in sys/resourcevar.h, I didn't notice any userland utils using this file, I'm going to try to do a buildworld. A kernel with this patch boots and seems to run multiuser fine. Index: kern/kern_resource.c =================================================================== RCS file: /home/ncvs/src/sys/kern/kern_resource.c,v retrieving revision 1.65 diff -u -r1.65 kern_resource.c --- kern/kern_resource.c 2000/11/22 07:41:58 1.65 +++ kern/kern_resource.c 2000/11/25 12:16:02 @@ -64,6 +64,7 @@ static MALLOC_DEFINE(M_UIDINFO, "uidinfo", "uidinfo structures"); #define UIHASH(uid) (&uihashtbl[(uid) & uihash]) +static struct mtx uidhash_mtx; static LIST_HEAD(uihashhead, uidinfo) *uihashtbl; static u_long uihash; /* size of hash table - 1 */ @@ -663,9 +664,15 @@ void uihashinit() { + uihashtbl = hashinit(maxproc / 16, M_UIDINFO, &uihash); + mtx_init(&uidhash_mtx, "uidinfo hash", MTX_DEF); } +/* + * lookup a uidinfo struct for the parameter uid. + * uidhash_mtx must be locked. + */ static struct uidinfo * uilookup(uid) uid_t uid; @@ -673,6 +680,7 @@ struct uihashhead *uipp; struct uidinfo *uip; + mtx_assert(&uidhash_mtx, MA_OWNED); uipp = UIHASH(uid); LIST_FOREACH(uip, uipp, ui_hash) if (uip->ui_uid == uid) @@ -681,43 +689,42 @@ return (uip); } +/* + * Create a uidinfo struct for the parameter uid. + * uidhash_mtx must be locked. + */ static struct uidinfo * uicreate(uid) uid_t uid; { - struct uidinfo *uip, *norace; + struct uidinfo *uip; - MALLOC(uip, struct uidinfo *, sizeof(*uip), M_UIDINFO, M_NOWAIT); - if (uip == NULL) { - MALLOC(uip, struct uidinfo *, sizeof(*uip), M_UIDINFO, M_WAITOK); - /* - * if we M_WAITOK we must look afterwards or risk - * redundant entries - */ - norace = uilookup(uid); - if (norace != NULL) { - FREE(uip, M_UIDINFO); - return (norace); - } - } + mtx_assert(&uidhash_mtx, MA_OWNED); + MALLOC(uip, struct uidinfo *, sizeof(*uip), M_UIDINFO, M_WAITOK); LIST_INSERT_HEAD(UIHASH(uid), uip, ui_hash); uip->ui_uid = uid; uip->ui_proccnt = 0; uip->ui_sbsize = 0; uip->ui_ref = 0; + mtx_init(&uip->ui_mtx, "uidinfo struct", MTX_DEF); return (uip); } +/* + * find or allocate a struct uidinfo for a particular uid + */ struct uidinfo * uifind(uid) uid_t uid; { struct uidinfo *uip; + mtx_enter(&uidhash_mtx, MTX_DEF); uip = uilookup(uid); if (uip == NULL) uip = uicreate(uid); - uip->ui_ref++; + uihold(uip); + mtx_exit(&uidhash_mtx, MTX_DEF); return (uip); } @@ -726,6 +733,8 @@ struct uidinfo *uip; { + mtx_enter(&uidhash_mtx, MTX_DEF); + mtx_enter(&uip->ui_mtx, MTX_DEF); if (--uip->ui_ref == 0) { if (uip->ui_sbsize != 0) /* XXX no %qd in kernel. Truncate. */ @@ -735,9 +744,13 @@ printf("freeing uidinfo: uid = %d, proccnt = %ld\n", uip->ui_uid, uip->ui_proccnt); LIST_REMOVE(uip, ui_hash); + mtx_destroy(&uip->ui_mtx); + mtx_exit(&uidhash_mtx, MTX_DEF); FREE(uip, M_UIDINFO); return (1); } + mtx_exit(&uip->ui_mtx, MTX_DEF); + mtx_exit(&uidhash_mtx, MTX_DEF); return (0); } @@ -751,12 +764,17 @@ int diff; int max; { + + mtx_enter(&uip->ui_mtx, MTX_DEF); /* don't allow them to exceed max, but allow subtraction */ - if (diff > 0 && uip->ui_proccnt + diff > max && max != 0) + if (diff > 0 && uip->ui_proccnt + diff > max && max != 0) { + mtx_exit(&uip->ui_mtx, MTX_DEF); return (0); + } uip->ui_proccnt += diff; if (uip->ui_proccnt < 0) printf("negative proccnt for uid = %d\n", uip->ui_uid); + mtx_exit(&uip->ui_mtx, MTX_DEF); return (1); } @@ -773,11 +791,14 @@ rlim_t new; int s; + mtx_enter(&uip->ui_mtx, MTX_DEF); + s = splnet(); new = uip->ui_sbsize + to - *hiwat; /* don't allow them to exceed max, but allow subtraction */ if (to > *hiwat && new > max) { splx(s); + mtx_exit(&uip->ui_mtx, MTX_DEF); return (0); } uip->ui_sbsize = new; @@ -785,5 +806,6 @@ if (uip->ui_sbsize < 0) printf("negative sbsize for uid = %d\n", uip->ui_uid); splx(s); + mtx_exit(&uip->ui_mtx, MTX_DEF); return (1); } Index: sys/resourcevar.h =================================================================== RCS file: /home/ncvs/src/sys/sys/resourcevar.h,v retrieving revision 1.17 diff -u -r1.17 resourcevar.h --- sys/resourcevar.h 2000/09/05 22:11:12 1.17 +++ sys/resourcevar.h 2000/11/25 12:16:03 @@ -40,6 +40,9 @@ #include #include +#include /* XXX */ + + /* * Kernel per-process accounting / statistics * (not necessarily resident except when running). @@ -90,10 +93,17 @@ long ui_proccnt; /* number of processes */ uid_t ui_uid; /* uid */ u_short ui_ref; /* reference count */ + struct mtx ui_mtx; /* protect counts */ }; #ifdef _KERNEL -#define uihold(uip) (uip)->ui_ref++ +#define uihold(uip) \ + do { \ + mtx_enter(&(uip)->ui_mtx, MTX_DEF); \ + (uip)->ui_ref++; \ + mtx_exit(&(uip)->ui_mtx, MTX_DEF); \ + } while(0) + struct proc; void addupc_intr __P((struct proc *p, u_long pc, u_int ticks)); -- -Alfred Perlstein - [bright@wintelcom.net|alfred@freebsd.org] "I have the heart of a child; I keep it in a jar on my desk." To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Sat Nov 25 11:28:55 2000 Delivered-To: freebsd-smp@freebsd.org Received: from fw.wintelcom.net (ns1.wintelcom.net [209.1.153.20]) by hub.freebsd.org (Postfix) with ESMTP id D1A6D37B4D7; Sat, 25 Nov 2000 11:28:53 -0800 (PST) Received: (from bright@localhost) by fw.wintelcom.net (8.10.0/8.10.0) id eAPJSrt00545; Sat, 25 Nov 2000 11:28:53 -0800 (PST) Date: Sat, 25 Nov 2000 11:28:53 -0800 From: Alfred Perlstein To: smp@freebsd.org Cc: jhb@freebsd.org Subject: disk IO terrible in smpng? Message-ID: <20001125112853.P8051@fw.wintelcom.net> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2.5i Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org I've been trying to generate some patches on a SMPng box, but the IO is killing me, anything disk bound seems to hog the CPU and make the machine terribly sluggish. Is this to be expected? -- -Alfred Perlstein - [bright@wintelcom.net|alfred@freebsd.org] "I have the heart of a child; I keep it in a jar on my desk." To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Sat Nov 25 11:30:56 2000 Delivered-To: freebsd-smp@freebsd.org Received: from pike.osd.bsdi.com (pike.osd.bsdi.com [204.216.28.222]) by hub.freebsd.org (Postfix) with ESMTP id 76DAC37B479 for ; Sat, 25 Nov 2000 11:30:53 -0800 (PST) Received: from foo.osd.bsdi.com (root@foo.osd.bsdi.com [204.216.28.137]) by pike.osd.bsdi.com (8.11.1/8.9.3) with ESMTP id eAPJUgC82704; Sat, 25 Nov 2000 11:30:42 -0800 (PST) (envelope-from jhb@foo.osd.bsdi.com) Received: (from jhb@localhost) by foo.osd.bsdi.com (8.11.1/8.11.0) id eAPJRpT33690; Sat, 25 Nov 2000 11:27:51 -0800 (PST) (envelope-from jhb) Message-ID: X-Mailer: XFMail 1.4.0 on FreeBSD X-Priority: 3 (Normal) Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 8bit MIME-Version: 1.0 In-Reply-To: <20001125041747.I8051@fw.wintelcom.net> Date: Sat, 25 Nov 2000 11:27:51 -0800 (PST) Organization: BSD, Inc. From: John Baldwin To: Alfred Perlstein Subject: RE: uidinfo patches, review please. Cc: smp@FreeBSD.ORG Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org On 25-Nov-00 Alfred Perlstein wrote: > Protect the uidinfo structures. > > My only concern is that I must now include sys/mutex.h in > sys/resourcevar.h, I didn't notice any userland utils using > this file, I'm going to try to do a buildworld. > > A kernel with this patch boots and seems to run multiuser > fine. I'll try and test this on Monday. Have you compiled a kernel with WITNESS turned on? You probably want to add an explicit lock order of: "uidinfo hash", "uidinfo struct", NULL to the order_list in the witness code to make sure those locks aren't gotten out of order as well. Looks ok to me though for the most part (see comments below). The nested include of sys/mutex.h does bite, but I think it may be something we may have to live with. Either that or stick pointers to struct mtx in structures instead with a forward declaration of struct mtx and malloc all the mutexes (Yuck). > Index: kern/kern_resource.c > =================================================================== > RCS file: /home/ncvs/src/sys/kern/kern_resource.c,v > retrieving revision 1.65 > diff -u -r1.65 kern_resource.c > --- kern/kern_resource.c 2000/11/22 07:41:58 1.65 > +++ kern/kern_resource.c 2000/11/25 12:16:02 > + mtx_assert(&uidhash_mtx, MA_OWNED); > + MALLOC(uip, struct uidinfo *, sizeof(*uip), M_UIDINFO, M_WAITOK); > LIST_INSERT_HEAD(UIHASH(uid), uip, ui_hash); > uip->ui_uid = uid; > uip->ui_proccnt = 0; > uip->ui_sbsize = 0; > uip->ui_ref = 0; > + mtx_init(&uip->ui_mtx, "uidinfo struct", MTX_DEF); Why not use M_ZERO with the call to MALLOC() so that you only have to initialize ui_uid and ui_mtx? > @@ -735,9 +744,13 @@ > printf("freeing uidinfo: uid = %d, proccnt = %ld\n", > uip->ui_uid, uip->ui_proccnt); > LIST_REMOVE(uip, ui_hash); > + mtx_destroy(&uip->ui_mtx); > + mtx_exit(&uidhash_mtx, MTX_DEF); You should mtx_exit(&uip->ui_mtx) before destroying it here. > FREE(uip, M_UIDINFO); > return (1); > } > + mtx_exit(&uip->ui_mtx, MTX_DEF); > + mtx_exit(&uidhash_mtx, MTX_DEF); > return (0); > } -- John Baldwin -- http://www.FreeBSD.org/~jhb/ PGP Key: http://www.Baldwin.cx/~john/pgpkey.asc "Power Users Use the Power to Serve!" - http://www.FreeBSD.org/ To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Sat Nov 25 12: 8: 2 2000 Delivered-To: freebsd-smp@freebsd.org Received: from io.yi.org (h24-69-199-88.gv.shawcable.net [24.69.199.88]) by hub.freebsd.org (Postfix) with ESMTP id B9CED37B479; Sat, 25 Nov 2000 12:08:00 -0800 (PST) Received: from io.yi.org (localhost.gvcl1.bc.wave.home.com [127.0.0.1]) by io.yi.org (Postfix) with ESMTP id CC0B8BA7A; Sat, 25 Nov 2000 12:07:54 -0800 (PST) X-Mailer: exmh version 2.1.1 10/15/1999 To: John Baldwin Cc: Alfred Perlstein , smp@FreeBSD.ORG Subject: Re: uidinfo patches, review please. In-Reply-To: Message from John Baldwin of "Sat, 25 Nov 2000 11:27:51 PST." Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Sat, 25 Nov 2000 12:07:54 -0800 From: Jake Burkholder Message-Id: <20001125200754.CC0B8BA7A@io.yi.org> Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org > > On 25-Nov-00 Alfred Perlstein wrote: > > Protect the uidinfo structures. > > > > My only concern is that I must now include sys/mutex.h in > > sys/resourcevar.h, I didn't notice any userland utils using > > this file, I'm going to try to do a buildworld. > > > > A kernel with this patch boots and seems to run multiuser > > fine. Looks ok to me. > > I'll try and test this on Monday. Have you compiled a kernel with WITNESS > turned on? You probably want to add an explicit lock order of: > > "uidinfo hash", "uidinfo struct", NULL > > to the order_list in the witness code to make sure those locks aren't gotten > out of order as well. Looks ok to me though for the most part (see comments Kernel boots with WITNESS enabled, I also added this to the order list; make buildworld is chugging away on my SMP box. > > @@ -735,9 +744,13 @@ > > printf("freeing uidinfo: uid = %d, proccnt = %ld\n", > > uip->ui_uid, uip->ui_proccnt); > > LIST_REMOVE(uip, ui_hash); > > + mtx_destroy(&uip->ui_mtx); > > + mtx_exit(&uidhash_mtx, MTX_DEF); > > You should mtx_exit(&uip->ui_mtx) before destroying it here. This isn't strictly necessary, mtx_destroy handles destroying a held mutex. I suppose it saves a locked bus op. To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Sat Nov 25 12:11:45 2000 Delivered-To: freebsd-smp@freebsd.org Received: from io.yi.org (h24-69-199-88.gv.shawcable.net [24.69.199.88]) by hub.freebsd.org (Postfix) with ESMTP id F37F937B4CF; Sat, 25 Nov 2000 12:11:43 -0800 (PST) Received: from io.yi.org (localhost.gvcl1.bc.wave.home.com [127.0.0.1]) by io.yi.org (Postfix) with ESMTP id D6886BA7A; Sat, 25 Nov 2000 12:11:42 -0800 (PST) X-Mailer: exmh version 2.1.1 10/15/1999 To: Alfred Perlstein Cc: John Baldwin , smp@FreeBSD.ORG Subject: Re: uidinfo patches, review please. In-Reply-To: Message from Jake Burkholder of "Sat, 25 Nov 2000 12:07:54 PST." <20001125200754.CC0B8BA7A@io.yi.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Sat, 25 Nov 2000 12:11:42 -0800 From: Jake Burkholder Message-Id: <20001125201142.D6886BA7A@io.yi.org> Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org > > > @@ -735,9 +744,13 @@ > > > printf("freeing uidinfo: uid = %d, proccnt = %ld\n", > > > uip->ui_uid, uip->ui_proccnt); > > > LIST_REMOVE(uip, ui_hash); > > > + mtx_destroy(&uip->ui_mtx); > > > + mtx_exit(&uidhash_mtx, MTX_DEF); > > > > You should mtx_exit(&uip->ui_mtx) before destroying it here. > > This isn't strictly necessary, mtx_destroy handles destroying a > held mutex. I suppose it saves a locked bus op. > No, I take this back. You should exit the mutex, it could screw up witness because the lock will still be in the p_heldmtx list. > > > To Unsubscribe: send mail to majordomo@FreeBSD.org > with "unsubscribe freebsd-smp" in the body of the message To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Sat Nov 25 12:23:17 2000 Delivered-To: freebsd-smp@freebsd.org Received: from io.yi.org (h24-69-199-88.gv.shawcable.net [24.69.199.88]) by hub.freebsd.org (Postfix) with ESMTP id 580FF37B4CF; Sat, 25 Nov 2000 12:23:15 -0800 (PST) Received: from io.yi.org (localhost.gvcl1.bc.wave.home.com [127.0.0.1]) by io.yi.org (Postfix) with ESMTP id CC8E8BA7A; Sat, 25 Nov 2000 12:23:14 -0800 (PST) X-Mailer: exmh version 2.1.1 10/15/1999 To: Alfred Perlstein Cc: John Baldwin , smp@FreeBSD.ORG Subject: Re: uidinfo patches, review please. In-Reply-To: Message from Jake Burkholder of "Sat, 25 Nov 2000 12:11:42 PST." <20001125201142.D6886BA7A@io.yi.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Sat, 25 Nov 2000 12:23:14 -0800 From: Jake Burkholder Message-Id: <20001125202314.CC8E8BA7A@io.yi.org> Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org > > > > @@ -735,9 +744,13 @@ > > > > printf("freeing uidinfo: uid = %d, proccnt = %ld\n", > > > > uip->ui_uid, uip->ui_proccnt); > > > > LIST_REMOVE(uip, ui_hash); > > > > + mtx_destroy(&uip->ui_mtx); > > > > + mtx_exit(&uidhash_mtx, MTX_DEF); > > > > > > You should mtx_exit(&uip->ui_mtx) before destroying it here. > > > > This isn't strictly necessary, mtx_destroy handles destroying a > > held mutex. I suppose it saves a locked bus op. > > > > No, I take this back. You should exit the mutex, it could screw > up witness because the lock will still be in the p_heldmtx list. > Bah, I was right the first time, its not necessary, witness_destroy handles it. To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Sat Nov 25 12:35:28 2000 Delivered-To: freebsd-smp@freebsd.org Received: from pike.osd.bsdi.com (pike.osd.bsdi.com [204.216.28.222]) by hub.freebsd.org (Postfix) with ESMTP id 7C03037B4CF for ; Sat, 25 Nov 2000 12:35:25 -0800 (PST) Received: from foo.osd.bsdi.com (root@foo.osd.bsdi.com [204.216.28.137]) by pike.osd.bsdi.com (8.11.1/8.9.3) with ESMTP id eAPKZ8C83726; Sat, 25 Nov 2000 12:35:08 -0800 (PST) (envelope-from jhb@foo.osd.bsdi.com) Received: (from jhb@localhost) by foo.osd.bsdi.com (8.11.1/8.11.0) id eAPKWGN34219; Sat, 25 Nov 2000 12:32:16 -0800 (PST) (envelope-from jhb) Message-ID: X-Mailer: XFMail 1.4.0 on FreeBSD X-Priority: 3 (Normal) Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 8bit MIME-Version: 1.0 In-Reply-To: <20001125200754.CC0B8BA7A@io.yi.org> Date: Sat, 25 Nov 2000 12:32:16 -0800 (PST) Organization: BSD, Inc. From: John Baldwin To: Jake Burkholder Subject: Re: uidinfo patches, review please. Cc: smp@FreeBSD.ORG, Alfred Perlstein Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org On 25-Nov-00 Jake Burkholder wrote: >> >> On 25-Nov-00 Alfred Perlstein wrote: >> > Protect the uidinfo structures. >> > >> > My only concern is that I must now include sys/mutex.h in >> > sys/resourcevar.h, I didn't notice any userland utils using >> > this file, I'm going to try to do a buildworld. >> > >> > A kernel with this patch boots and seems to run multiuser >> > fine. > > Looks ok to me. > >> >> I'll try and test this on Monday. Have you compiled a kernel with WITNESS >> turned on? You probably want to add an explicit lock order of: >> >> "uidinfo hash", "uidinfo struct", NULL >> >> to the order_list in the witness code to make sure those locks aren't gotten >> out of order as well. Looks ok to me though for the most part (see comments > > Kernel boots with WITNESS enabled, I also added this to the order list; > make buildworld is chugging away on my SMP box. Cool. >> > @@ -735,9 +744,13 @@ >> > printf("freeing uidinfo: uid = %d, proccnt = %ld\n", >> > uip->ui_uid, uip->ui_proccnt); >> > LIST_REMOVE(uip, ui_hash); >> > + mtx_destroy(&uip->ui_mtx); >> > + mtx_exit(&uidhash_mtx, MTX_DEF); >> >> You should mtx_exit(&uip->ui_mtx) before destroying it here. > > This isn't strictly necessary, mtx_destroy handles destroying a > held mutex. I suppose it saves a locked bus op. Well, I'd rather have it for correctness' sake. -- John Baldwin -- http://www.FreeBSD.org/~jhb/ PGP Key: http://www.Baldwin.cx/~john/pgpkey.asc "Power Users Use the Power to Serve!" - http://www.FreeBSD.org/ To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message