From owner-freebsd-smp Sun Nov 12 2:59: 3 2000 Delivered-To: freebsd-smp@freebsd.org Received: from peorth.iteration.net (peorth.iteration.net [208.190.180.178]) by hub.freebsd.org (Postfix) with ESMTP id 8EB3037B479 for ; Sun, 12 Nov 2000 02:58:59 -0800 (PST) Received: by peorth.iteration.net (Postfix, from userid 1001) id 283865730B; Sun, 12 Nov 2000 04:58:58 -0600 (CST) Date: Sun, 12 Nov 2000 04:58:58 -0600 From: "Michael C . Wu" To: Chris Dillon Cc: Jeremiah Gowdy , KT Sin , freebsd-smp@FreeBSD.ORG Subject: Re: via chipset and SMP Message-ID: <20001112045858.A7123@peorth.iteration.net> Reply-To: "Michael C . Wu" Mail-Followup-To: "Michael C . Wu" , Chris Dillon , Jeremiah Gowdy , KT Sin , freebsd-smp@FreeBSD.ORG References: <006301c04ab3$b70121c0$aa240018@cx443070b> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2.5i In-Reply-To: ; from cdillon@wolves.k12.mo.us on Fri, Nov 10, 2000 at 01:28:47PM -0600 X-PGP-Fingerprint: 5025 F691 F943 8128 48A8 5025 77CE 29C5 8FA1 2E20 X-PGP-Key-ID: 0x8FA12E20 Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org On Fri, Nov 10, 2000 at 01:28:47PM -0600, Chris Dillon scribbled: | On Thu, 9 Nov 2000, Jeremiah Gowdy wrote: | > > > I am thinking of getting a dual coppermine board with VIA chipset. | > > > Has anyone tried running FreeBSD SMP on such boards? Is the VIA's | > > > SMP chipset any good? | | > > Even their non-SMP chipsets are crap. It blew my mind that they had | > > the ability to create an SMP chipset at all, let alone make one that | > > might work. | | > My counter opinion would simply be, I have ran FreeBSD on my Tyan | > Tiger 133A motherboard which uses the Via Apollo Pro 133A chipset | > with Dual Pentium III 450s. They aren't the best performing | > boards out there, however, they are pretty stable in my | > experience, both Athlon and PIII. Add that to the fact that they | > support the latest tech like AGP4X and UDMA100. Their chipsets do | > lack a little performance, but not so much as to make me wish I | > had another board. Dual Pentium IIIs are still Dual Pentium IIIs. VIA chipsets have compatibility problems with many PCI/ISA cards. | Sure, the chipsets do work. Sometimes. Maybe. If you hold your | breath. | recently the 760 to speak of). Even some of our (FreeBSD's) own I think the AMD 760 will be much nicer than VIA. | driver writers have kvetched about how VIA or | insert-other-lowball-chipset-maker-here has not followed this or that | PCI spec or things of that nature. Intel apparently puts quite a bit | more compatibility testing into their chipsets, and follows | specifications closely (usually). I'm not sure if VIA, SiS, or ALi's | inability to create decent chipsets is because of a lack of access to | important standards information, lack of engineering ability, or for From what I understand by talking to VIA/D-Link/Realtek developers that I went to school with in Taiwan, their development (and especially testing/verification) models are not the envy of the world. Sometimes, it's the management pushing deadlines too hard, and sometimes it's just that they lack the years of design experience/technology of companies like AMD/Motorola/IBM/Intel. Or it is the pointy-haired bosses, and maybe a combination of the above. | other reasons (like, they're low-end, low-price chipsets, and you get | what you pay for). D-Link's company idea is to go for low-end, low-price. See Bill Paul's if_rl.c comments. :) -- +------------------------------------------------------------------+ | keichii@peorth.iteration.net | keichii@bsdconspiracy.net | | http://peorth.iteration.net/~keichii | Yes, BSD is a conspiracy. | +------------------------------------------------------------------+ To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Sun Nov 12 8: 9: 1 2000 Delivered-To: freebsd-smp@freebsd.org Received: from gndrsh.dnsmgr.net (GndRsh.dnsmgr.net [198.145.92.4]) by hub.freebsd.org (Postfix) with ESMTP id B63A337B479 for ; Sun, 12 Nov 2000 08:08:58 -0800 (PST) Received: (from freebsd@localhost) by gndrsh.dnsmgr.net (8.9.3/8.9.3) id IAA97211; Sun, 12 Nov 2000 08:08:10 -0800 (PST) (envelope-from freebsd) From: "Rodney W. Grimes" Message-Id: <200011121608.IAA97211@gndrsh.dnsmgr.net> Subject: Re: via chipset and SMP In-Reply-To: <20001112045858.A7123@peorth.iteration.net> from "Michael C . Wu" at "Nov 12, 2000 04:58:58 am" To: keichii@peorth.iteration.net (Michael C . Wu) Date: Sun, 12 Nov 2000 08:08:10 -0800 (PST) Cc: cdillon@wolves.k12.mo.us (Chris Dillon), jgowdy@home.com (Jeremiah Gowdy), ktsin@acm.org (KT Sin), freebsd-smp@FreeBSD.ORG X-Mailer: ELM [version 2.4ME+ PL54 (25)] MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org ... > D-Link's company idea is to go for low-end, low-price. > See Bill Paul's if_rl.c comments. :) Again, broad strokes applied to a vendor. Yes, D-Link does make some real cheap shit like 81x9 NIC cards, based on the cheapest NIC chip you can buy. _BUT_, and this is a big but, they also make some fine NIC cards. Like we 21040 based DFE-500, then the 21143/21150 four port DFE-770TX. These used DEC (now Intel) chips, infact some of the better NIC chips on the market. There ``low cost'' 10/100 switches are junk, they make some nicer switches using BroadCOM chips, again, probably some of the better 10/100 switch chips on the market. -- Rod Grimes - KD7CAX @ CN85sl - (RWG25) rgrimes@gndrsh.dnsmgr.net To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Mon Nov 13 10:43: 8 2000 Delivered-To: freebsd-smp@freebsd.org Received: from pike.osd.bsdi.com (pike.osd.bsdi.com [204.216.28.222]) by hub.freebsd.org (Postfix) with ESMTP id 630F737B69E; Mon, 13 Nov 2000 10:43:02 -0800 (PST) Received: from laptop.baldwin.cx (john@dhcp245.osd.bsdi.com [204.216.28.245]) by pike.osd.bsdi.com (8.11.0/8.9.3) with ESMTP id eADIgjB57242; Mon, 13 Nov 2000 10:42:45 -0800 (PST) (envelope-from jhb@FreeBSD.org) Message-ID: X-Mailer: XFMail 1.4.0 on FreeBSD X-Priority: 3 (Normal) Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 8bit MIME-Version: 1.0 In-Reply-To: Date: Mon, 13 Nov 2000 10:43:11 -0800 (PST) From: John Baldwin To: Zhiui Zhang Subject: RE: simple lock and the lost wakeup problem Cc: freebsd-smp@FreeBSD.org, freebsd-hackers@FreeBSD.org Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org On 12-Nov-00 Zhiui Zhang wrote: > > I am new to SMP subject and have some questions to ask: > > Is the simplelock() really needed since FreeBSD is using the big giant > lock and the kernel is non preemptive? Or has FreeBSD changed the big > giant lock and made kernel thread preemptive? Uresha Vahalia talks about > Lost Wakeup Problem (page 196), the test of the resource and sleep() has > to be done atomically. Which correct mechanism should I use on FreeBSD to > achieve this (avoid the lost-wakeup problem)? simplelocks are used in interrupt handlers and a few other places that do not run while holding the big giant lock. In -current all of this has changed, however, as we now have mutexes. Most of the kernel is still under the Giant mutex at this point in time, but in the future Giant will be replaced by many other locks that protect data structures within the kernel. To avoid the lost wakeup problem I think you are referring to (I don't have Unix Internals here with me @ home) in -current, we have a replacment for tsleep() called msleep(). msleep() takes an additional parameter which is a mutex to release when going to sleep. From the kernel's perspective, the mutex is released and the process is put to sleep as an atomic operation. Hope this helps. > Any help is appreciated. > > -Zhihui > > > > To Unsubscribe: send mail to majordomo@FreeBSD.org > with "unsubscribe freebsd-smp" in the body of the message -- John Baldwin -- http://www.FreeBSD.org/~jhb/ PGP Key: http://www.baldwin.cx/~john/pgpkey.asc "Power Users Use the Power to Serve!" - http://www.FreeBSD.org/ To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Wed Nov 15 16:48:59 2000 Delivered-To: freebsd-smp@freebsd.org Received: from pike.osd.bsdi.com (pike.osd.bsdi.com [204.216.28.222]) by hub.freebsd.org (Postfix) with ESMTP id 3408537B4C5; Wed, 15 Nov 2000 16:48:55 -0800 (PST) Received: from laptop.baldwin.cx (john@jhb-laptop.osd.bsdi.com [204.216.28.241]) by pike.osd.bsdi.com (8.11.0/8.9.3) with ESMTP id eAG0mlB39691; Wed, 15 Nov 2000 16:48:47 -0800 (PST) (envelope-from jhb@FreeBSD.org) Message-ID: X-Mailer: XFMail 1.4.0 on FreeBSD X-Priority: 3 (Normal) Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 8bit MIME-Version: 1.0 Date: Wed, 15 Nov 2000 16:49:26 -0800 (PST) From: John Baldwin To: smp@FreeBSD.org Subject: Patches to make witness work.. Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org [ Bcc'd to -current, send followups to -smp ] I've been spending my time recently trying to get the kernel to boot with witnees turned on. The witness code itself was not broken (with a tiny exception), but our kernel code did have some bugs. However, while fixing the problems witness pointed out, I ended up with a kernel that is not stable on SMP systems (go figure). Part of this may be due to a broken atkbd driver, whose interrupt thread seems to chew up a lot more CPU time than it should be. I've tried to commit as many of the safe changes as I could w/o actually breaking SMP systems. The rest of the patches that are needed are at http://www.FreeBSD.org/~jhb/patches/giant.patch. This patchset does two primary things: 1) It fixes msleep() and mawait() to release the sched_lock before calling CURSIG() and to pick the lock back up again afterwards. 2) It moves the dropping/reacquiring of Giant out of mi_switch() and out into the code that calls mi_switch(). This is needed because Giant needs to be released/reacquired when it is not under the sched_lock Any feedback, etc. is welcome. At the very least, UP systems (both x86 and alpha) appear to be fine with this patch, and can run kernels with witness enabled without any problems. Thus this patch can be used to debug deadlocks, etc. on UP systems at least. -- John Baldwin -- http://www.FreeBSD.org/~jhb/ PGP Key: http://www.baldwin.cx/~john/pgpkey.asc "Power Users Use the Power to Serve!" - http://www.FreeBSD.org/ To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Wed Nov 15 17:29:52 2000 Delivered-To: freebsd-smp@freebsd.org Received: from pike.osd.bsdi.com (pike.osd.bsdi.com [204.216.28.222]) by hub.freebsd.org (Postfix) with ESMTP id 23ABF37B4C5; Wed, 15 Nov 2000 17:29:49 -0800 (PST) Received: from laptop.baldwin.cx (john@jhb-laptop.osd.bsdi.com [204.216.28.241]) by pike.osd.bsdi.com (8.11.0/8.9.3) with ESMTP id eAG1TeB40635; Wed, 15 Nov 2000 17:29:40 -0800 (PST) (envelope-from jhb@FreeBSD.org) Message-ID: X-Mailer: XFMail 1.4.0 on FreeBSD X-Priority: 3 (Normal) Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 8bit MIME-Version: 1.0 In-Reply-To: Date: Wed, 15 Nov 2000 17:30:16 -0800 (PST) From: John Baldwin To: John Baldwin Subject: RE: Patches to make witness work.. Cc: smp@FreeBSD.org Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org On 16-Nov-00 John Baldwin wrote: > [ Bcc'd to -current, send followups to -smp ] > > I've been spending my time recently trying to get the kernel to boot with > witnees turned on. The witness code itself was not broken (with a tiny > exception), but our kernel code did have some bugs. However, while fixing > the > problems witness pointed out, I ended up with a kernel that is not stable on > SMP systems (go figure). Part of this may be due to a broken atkbd driver, > whose interrupt thread seems to chew up a lot more CPU time than it should > be. > I've tried to commit as many of the safe changes as I could w/o actually > breaking SMP systems. The rest of the patches that are needed are at > http://www.FreeBSD.org/~jhb/patches/giant.patch. This patchset does two > primary things: > > 1) It fixes msleep() and mawait() to release the sched_lock before calling > CURSIG() and to pick the lock back up again afterwards. > 2) It moves the dropping/reacquiring of Giant out of mi_switch() and out > into the code that calls mi_switch(). This is needed because Giant > needs to be released/reacquired when it is not under the sched_lock > > Any feedback, etc. is welcome. At the very least, UP systems (both x86 and > alpha) appear to be fine with this patch, and can run kernels with witness > enabled without any problems. Thus this patch can be used to debug > deadlocks, > etc. on UP systems at least. Erm, well, wouldn't you know it, I found a missing mtx_exit() in mawait() right after committing 1), and now my SMP testbox seems fine. I'm trying another kernel with witness turned on (as well as on another SMP machine), and if those work I'll commit the changes. -- John Baldwin -- http://www.FreeBSD.org/~jhb/ PGP Key: http://www.baldwin.cx/~john/pgpkey.asc "Power Users Use the Power to Serve!" - http://www.FreeBSD.org/ To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Wed Nov 15 18:19:30 2000 Delivered-To: freebsd-smp@freebsd.org Received: from pike.osd.bsdi.com (pike.osd.bsdi.com [204.216.28.222]) by hub.freebsd.org (Postfix) with ESMTP id 2867637B4CF; Wed, 15 Nov 2000 18:19:26 -0800 (PST) Received: from laptop.baldwin.cx (john@jhb-laptop.osd.bsdi.com [204.216.28.241]) by pike.osd.bsdi.com (8.11.0/8.9.3) with ESMTP id eAG2JHB42159; Wed, 15 Nov 2000 18:19:17 -0800 (PST) (envelope-from jhb@FreeBSD.org) Message-ID: X-Mailer: XFMail 1.4.0 on FreeBSD X-Priority: 3 (Normal) Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 8bit MIME-Version: 1.0 In-Reply-To: <200011160216.SAA29000@freefall.freebsd.org> Date: Wed, 15 Nov 2000 18:19:53 -0800 (PST) From: John Baldwin To: current@FreeBSD.org, smp@FreeBSD.org Subject: RE: cvs commit: src/sys/alpha/alpha trap.c src/sys/i386/i386 tra Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org On 16-Nov-00 John Baldwin wrote: > jhb 2000/11/15 18:16:45 PST > > Modified files: > sys/alpha/alpha trap.c > sys/i386/i386 trap.c > sys/ia64/ia64 trap.c > sys/kern kern_mutex.c kern_shutdown.c kern_sig.c > kern_subr.c kern_synch.c > Log: > Don't release and acquire Giant in mi_switch(). Instead, release and > acquire Giant as needed in functions that call mi_switch(). The releases > need to be done outside of the sched_lock to avoid potential deadlocks > from trying to acquire Giant while interrupts are disabled. > > Submitted by: witness It is now safe to turn on WITNESS in -current kernels without having the machine panic or lockup.... At least it is on my set of test boxes. As such, I plan to turn on the various debugging options in -current's GENERIC sometime tomorrow (including WITNESS, but without DIAGNOSTIC). -- John Baldwin -- http://www.FreeBSD.org/~jhb/ PGP Key: http://www.baldwin.cx/~john/pgpkey.asc "Power Users Use the Power to Serve!" - http://www.FreeBSD.org/ To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Wed Nov 15 18:28:42 2000 Delivered-To: freebsd-smp@freebsd.org Received: from adsl-63-192-211-186.dsl.snfc21.pacbell.net (adsl-63-192-211-186.dsl.snfc21.pacbell.net [63.192.211.186]) by hub.freebsd.org (Postfix) with ESMTP id 4C00037B4E5 for ; Wed, 15 Nov 2000 18:28:40 -0800 (PST) Received: from localhost.minions.com (bifrost@localhost.minions.com [127.0.0.1]) by adsl-63-192-211-186.dsl.snfc21.pacbell.net (8.9.3/8.9.3) with ESMTP id SAA35225; Wed, 15 Nov 2000 18:28:28 -0800 (PST) Date: Wed, 15 Nov 2000 18:28:28 -0800 (PST) From: Tom To: Koen Schreel Cc: KT Sin , freebsd-smp@FreeBSD.ORG Subject: Re: via chipset and SMP In-Reply-To: <3A0BD201.70E5D8C4@tue.nl> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org > > I am thinking of getting a dual coppermine board with VIA chipset. > > Has anyone tried running FreeBSD SMP on such boards? Is the VIA's > > SMP chipset any good? > > I'm running an MSI 694D-pro now pretty stable with two PIII-800's. Had > some problems with memory initially, but now it runs OK for some time. I had the same problem. I initially bought PC-133 ECC Registered DIMMS, and the 694D board didn't work at all. > Not the highest performing chipset, but it has a reasonable > price/performance ratio. yah, the board is $150 and supports a decent amount of stuff. I haven't had the box crash on me yet after I got the ram stuff sorted out. Its running 4.2-BETA right now, doing pretty speedily. --- Tom bifrost@minions.com "Where am I and why am I in this handbasket?" To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Thu Nov 16 1:19:19 2000 Delivered-To: freebsd-smp@freebsd.org Received: from viger.playstos.com (viger.playstos.com [194.79.208.10]) by hub.freebsd.org (Postfix) with ESMTP id C1BC837B4CF for ; Thu, 16 Nov 2000 01:19:14 -0800 (PST) Received: from sunshine (gw2.playstos.com [194.79.208.2]) by viger.playstos.com (Postfix) with SMTP id 4983ED60F9 for ; Thu, 16 Nov 2000 10:19:07 +0100 (CET) From: "Alessandro de Manzano" To: "freebsd-smp@freebsd.org" Date: Thu, 16 Nov 2000 10:20:41 +0100 Reply-To: "Alessandro de Manzano" X-Mailer: PMMail 2000 Professional (2.10.2010) For Windows 2000 (5.0.2195) MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Subject: dual celeron Message-Id: <20001116091907.4983ED60F9@viger.playstos.com> Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org Hello! (sorry for bad english!) I'm assembling a little FreeBSD box using spare, second hand, parts. I've a couple of Intel Celeron 433Mhz processor (for socket 370) which were used on a dual celeron machine with 2 riser cards for Slot 1. I would put them on a motherboard ABit BP6 (dual socket 370), just for have a little SMP FBSD machine :-) Is there something I should know about this experiment ? (I'm already running a 3.5 FBSD dual PII 266 machine at work, and is a rock solid web server). I plan to install 4.1.1-release and CVSup to 4.2 as soon as it will be released. Thanks a lot in advance! Alessandro de Manzano Playstos - TIMA S.p.A. Corso Sempione 63 20149 Milano, Italy tel.: +39-023314153 fax: +39-02315678 email: demanzano@playstos.com http://www.playstos.com To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Thu Nov 16 1:28:35 2000 Delivered-To: freebsd-smp@freebsd.org Received: from penguin-ext.wise.edt.ericsson.se (penguin-ext.wise.edt.ericsson.se [194.237.142.110]) by hub.freebsd.org (Postfix) with ESMTP id AE2D637B4C5 for ; Thu, 16 Nov 2000 01:28:32 -0800 (PST) Received: from poem.emw.ericsson.se (poem.emw.ericsson.se [136.225.49.25]) by penguin.wise.edt.ericsson.se (8.11.0/8.10.1/WIREfire-1.3) with ESMTP id eAG9STZ23196; Thu, 16 Nov 2000 10:28:31 +0100 (MET) Received: from Ericsson.com (balvenie.mo.emw.ericsson.se [132.196.48.29]) by poem.emw.ericsson.se (8.9.3/8.9.3) with ESMTP id KAA08554; Thu, 16 Nov 2000 10:28:27 +0100 (MET) Message-ID: <3A13A8AF.B43195A0@Ericsson.com> Date: Thu, 16 Nov 2000 10:28:15 +0100 From: Joachim Strombergson Organization: Ericsson Microwave Systems AB X-Mailer: Mozilla 4.75C-EMW [en] (X11; U; SunOS 5.7 sun4u) X-Accept-Language: en MIME-Version: 1.0 To: Alessandro de Manzano Cc: "freebsd-smp@freebsd.org" Subject: Re: dual celeron References: <20001116091907.4983ED60F9@viger.playstos.com> Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: 8bit Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org Aloha! Alessandro de Manzano wrote: > I've a couple of Intel Celeron 433Mhz processor (for socket 370) which > were used on a dual celeron machine with 2 riser cards for Slot 1. > > I would put them on a motherboard ABit BP6 (dual socket 370), just for > have a little SMP FBSD machine :-) > > Is there something I should know about this experiment ? Only that it probably will work like a charm! I've had my dual Celeron FreeBSD SMP box for 1.5 years now. I'm using the BP6 motherboard and the only changes during the time has been CPU upgrades. Stability is great, performance is great, the sun is shining,... Today I probably would buy a super high MHz single CPU machine to get that single program performane (Unreal Tournament, Emacs and other apps), but as a small home server this machine has been and is great. -- Med vänlig hälsning, Yours Joachim Strömbergson - Alltid i harmonisk svängning ---------------- Ericsson Microwave Systems AB ----------------- Joachim Strömbergson http://www.ericsson.se/microwave ASIC System on Silicon engineer, nice to CUTE animals. * Opinions above, expressed or implicit, are strictly personal * ------------- Spamfodder: regeringen@regeringen.se ------------- To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Thu Nov 16 1:51:18 2000 Delivered-To: freebsd-smp@freebsd.org Received: from viger.playstos.com (viger.playstos.com [194.79.208.10]) by hub.freebsd.org (Postfix) with ESMTP id A036137B4CF for ; Thu, 16 Nov 2000 01:51:11 -0800 (PST) Received: from sunshine (gw2.playstos.com [194.79.208.2]) by viger.playstos.com (Postfix) with SMTP id 9FB45D60F9; Thu, 16 Nov 2000 10:51:08 +0100 (CET) From: "Alessandro de Manzano" To: "Joachim Strombergson" Cc: "freebsd-smp@freebsd.org" Date: Thu, 16 Nov 2000 10:52:42 +0100 Reply-To: "Alessandro de Manzano" X-Mailer: PMMail 2000 Professional (2.10.2010) For Windows 2000 (5.0.2195) In-Reply-To: <3A13A8AF.B43195A0@Ericsson.com> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Subject: Re: dual celeron Message-Id: <20001116095108.9FB45D60F9@viger.playstos.com> Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org On Thu, 16 Nov 2000 10:28:15 +0100, Joachim Strombergson wrote: >Only that it probably will work like a charm! I've had my dual Celeron >FreeBSD SMP box for 1.5 years now. I'm using the BP6 motherboard and the >only changes during the time has been CPU upgrades. Stability is great, >performance is great, the sun is shining,... .. and the daemon is running ! :-) I'm very happy to hear such good information! >a small home server this machine has been and is great. yes, it will be exactly this, a little home server machine. thanks a lot! :-)) Alessandro de Manzano Playstos - TIMA S.p.A. Corso Sempione 63 20149 Milano, Italy tel.: +39-023314153 fax: +39-02315678 email: demanzano@playstos.com http://www.playstos.com To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Thu Nov 16 13:26:32 2000 Delivered-To: freebsd-smp@freebsd.org Received: from pike.osd.bsdi.com (pike.osd.bsdi.com [204.216.28.222]) by hub.freebsd.org (Postfix) with ESMTP id 1472837B4D7 for ; Thu, 16 Nov 2000 13:26:30 -0800 (PST) Received: from laptop.baldwin.cx (john@jhb-laptop.osd.bsdi.com [204.216.28.241]) by pike.osd.bsdi.com (8.11.0/8.9.3) with ESMTP id eAGLQJB67356; Thu, 16 Nov 2000 13:26:19 -0800 (PST) (envelope-from jhb@FreeBSD.org) Message-ID: X-Mailer: XFMail 1.4.0 on FreeBSD X-Priority: 3 (Normal) Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 8bit MIME-Version: 1.0 In-Reply-To: <200011162120.NAA57277@freefall.freebsd.org> Date: Thu, 16 Nov 2000 13:26:57 -0800 (PST) From: John Baldwin To: smp@FreeBSD.org, cp@bsdi.com Subject: RE: cvs commit: src/sys/kern kern_timeout.c Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org On 16-Nov-00 John Baldwin wrote: > jhb 2000/11/16 13:20:53 PST > > Modified files: > sys/kern kern_timeout.c > Log: > The recent changes to msleep() and mawait() resulted in timeout() and > untimeout() not being called with Giant in those functions. For now, > use the sched_lock to protect the callout wheel in softclock() and in > the various timeout and callout functions. > > Noticed by: tegge Big Kudos to Tor for finding this! Using the sched_lock here is more or less a bandaid for now. A possible long-term solution might be to use a dedicated sleep lock to protect the callout wheel. However, if you do this, then you can't be holding a spinlock when you call into timeout and friends. If we add in a lock to protect teh sleep queues, then we might be able to push sched_lock further down into msleep() and mawait(), so that we don't need sched_lock until after we call timeout and before we call untimeout, but I'm still thinking about this. Does anyone else have any ideas? One thing I am worried about is whether the overhead of the extra locks involved (sleepqueue locks, probably process locks (but we will need those anyway), and a callout wheel lock) will end up negating the performance gain from not having sched_lock protect so many different things. -- John Baldwin -- http://www.FreeBSD.org/~jhb/ PGP Key: http://www.baldwin.cx/~john/pgpkey.asc "Power Users Use the Power to Serve!" - http://www.FreeBSD.org/ To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Thu Nov 16 14:43: 7 2000 Delivered-To: freebsd-smp@freebsd.org Received: from io.yi.org (h24-69-199-88.gv.shawcable.net [24.69.199.88]) by hub.freebsd.org (Postfix) with ESMTP id 9F70937B479; Thu, 16 Nov 2000 14:43:04 -0800 (PST) Received: from io.yi.org (localhost.gvcl1.bc.wave.home.com [127.0.0.1]) by io.yi.org (Postfix) with ESMTP id DC1E6BA7A; Thu, 16 Nov 2000 14:42:59 -0800 (PST) X-Mailer: exmh version 2.1.1 10/15/1999 To: John Baldwin Cc: smp@FreeBSD.ORG, cp@bsdi.com, jake@io.yi.org Subject: Re: cvs commit: src/sys/kern kern_timeout.c In-Reply-To: Message from John Baldwin of "Thu, 16 Nov 2000 13:26:57 PST." Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Thu, 16 Nov 2000 14:42:59 -0800 From: Jake Burkholder Message-Id: <20001116224259.DC1E6BA7A@io.yi.org> Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org > > On 16-Nov-00 John Baldwin wrote: > > jhb 2000/11/16 13:20:53 PST > > > > Modified files: > > sys/kern kern_timeout.c > > Log: > > The recent changes to msleep() and mawait() resulted in timeout() and > > untimeout() not being called with Giant in those functions. For now, > > use the sched_lock to protect the callout wheel in softclock() and in > > the various timeout and callout functions. > > > > Noticed by: tegge > > Big Kudos to Tor for finding this! Using the sched_lock here is more or less a > bandaid for now. A possible long-term solution might be to use a dedicated > sleep lock to protect the callout wheel. However, if you do this, then you > can't be holding a spinlock when you call into timeout and friends. If we add > in a lock to protect teh sleep queues, then we might be able to push sched_lock > further down into msleep() and mawait(), so that we don't need sched_lock until > after we call timeout and before we call untimeout, but I'm still thinking > about this. Does anyone else have any ideas? One thing I am worried about is > whether the overhead of the extra locks involved (sleepqueue locks, probably > process locks (but we will need those anyway), and a callout wheel lock) will > end up negating the performance gain from not having sched_lock protect so many > different things. I think we need a separate spin lock for the callout wheel, ala BSD/OS's callout_mtx. Hardclock looks at the callout wheel and is now a fast interrupt, so it can't acquire a sleep mutex. Its a little paranoid because hardclock doesn't actually traverse any lists, it just checks if the current callout bucket is empty, and potentially schedules softclock, but you could miss a very short timeout on an smp system. ticks could also get incremented in the middle of softclock's test for if the callout's time has come. I have patches that do this and make softclock INTR_MPSAFE, I just need to test them. There's actually another major problem with this. The run queue and sleep queue use the same list linkage in struct proc, so its not safe to release sched_lock while you're on the sleep queue. If the process blocks on giant in CURSIG, the sleep queue will get corrupted. We really need to split the run queue/sleep queue linkage. Jake > > -- > > John Baldwin -- http://www.FreeBSD.org/~jhb/ > PGP Key: http://www.baldwin.cx/~john/pgpkey.asc > "Power Users Use the Power to Serve!" - http://www.FreeBSD.org/ > > > To Unsubscribe: send mail to majordomo@FreeBSD.org > with "unsubscribe freebsd-smp" in the body of the message To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Thu Nov 16 14:54:34 2000 Delivered-To: freebsd-smp@freebsd.org Received: from pike.osd.bsdi.com (pike.osd.bsdi.com [204.216.28.222]) by hub.freebsd.org (Postfix) with ESMTP id E528937B4C5 for ; Thu, 16 Nov 2000 14:54:31 -0800 (PST) Received: from laptop.baldwin.cx (john@jhb-laptop.osd.bsdi.com [204.216.28.241]) by pike.osd.bsdi.com (8.11.0/8.9.3) with ESMTP id eAGMs9B70127; Thu, 16 Nov 2000 14:54:09 -0800 (PST) (envelope-from jhb@FreeBSD.org) Message-ID: X-Mailer: XFMail 1.4.0 on FreeBSD X-Priority: 3 (Normal) Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 8bit MIME-Version: 1.0 In-Reply-To: <20001116224259.DC1E6BA7A@io.yi.org> Date: Thu, 16 Nov 2000 14:54:48 -0800 (PST) From: John Baldwin To: Jake Burkholder Subject: Re: cvs commit: src/sys/kern kern_timeout.c Cc: jake@io.yi.org, cp@bsdi.com, smp@FreeBSD.org Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org On 16-Nov-00 Jake Burkholder wrote: >> >> On 16-Nov-00 John Baldwin wrote: >> > jhb 2000/11/16 13:20:53 PST >> > >> > Modified files: >> > sys/kern kern_timeout.c >> > Log: >> > The recent changes to msleep() and mawait() resulted in timeout() and >> > untimeout() not being called with Giant in those functions. For now, >> > use the sched_lock to protect the callout wheel in softclock() and in >> > the various timeout and callout functions. >> > >> > Noticed by: tegge >> >> Big Kudos to Tor for finding this! Using the sched_lock here is more or >> less a >> bandaid for now. A possible long-term solution might be to use a dedicated >> sleep lock to protect the callout wheel. However, if you do this, then you >> can't be holding a spinlock when you call into timeout and friends. If we >> add >> in a lock to protect teh sleep queues, then we might be able to push >> sched_lock >> further down into msleep() and mawait(), so that we don't need sched_lock >> until >> after we call timeout and before we call untimeout, but I'm still thinking >> about this. Does anyone else have any ideas? One thing I am worried about >> is >> whether the overhead of the extra locks involved (sleepqueue locks, probably >> process locks (but we will need those anyway), and a callout wheel lock) >> will >> end up negating the performance gain from not having sched_lock protect so >> many >> different things. > > I think we need a separate spin lock for the callout wheel, ala BSD/OS's > callout_mtx. Hardclock looks at the callout wheel and is now a fast > interrupt, so it can't acquire a sleep mutex. Its a little paranoid > because hardclock doesn't actually traverse any lists, it just checks > if the current callout bucket is empty, and potentially schedules > softclock, but you could miss a very short timeout on an smp system. > ticks could also get incremented in the middle of softclock's test > for if the callout's time has come. > > I have patches that do this and make softclock INTR_MPSAFE, I just need > to test them. Ok. I was about to check the BSD/OS code to see how this was done there. > There's actually another major problem with this. The run queue and > sleep queue use the same list linkage in struct proc, so its not > safe to release sched_lock while you're on the sleep queue. If > the process blocks on giant in CURSIG, the sleep queue will get > corrupted. We really need to split the run queue/sleep queue > linkage. Ugh, ok. I'll do this next then. Grrrr. > Jake -- John Baldwin -- http://www.FreeBSD.org/~jhb/ PGP Key: http://www.baldwin.cx/~john/pgpkey.asc "Power Users Use the Power to Serve!" - http://www.FreeBSD.org/ To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Thu Nov 16 14:59: 5 2000 Delivered-To: freebsd-smp@freebsd.org Received: from pike.osd.bsdi.com (pike.osd.bsdi.com [204.216.28.222]) by hub.freebsd.org (Postfix) with ESMTP id 1A7E237B4D7; Thu, 16 Nov 2000 14:59:01 -0800 (PST) Received: from laptop.baldwin.cx (john@jhb-laptop.osd.bsdi.com [204.216.28.241]) by pike.osd.bsdi.com (8.11.0/8.9.3) with ESMTP id eAGMwnB70239; Thu, 16 Nov 2000 14:58:49 -0800 (PST) (envelope-from jhb@FreeBSD.org) Message-ID: X-Mailer: XFMail 1.4.0 on FreeBSD X-Priority: 3 (Normal) Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 8bit MIME-Version: 1.0 In-Reply-To: Date: Thu, 16 Nov 2000 14:59:28 -0800 (PST) From: John Baldwin To: John Baldwin Subject: Re: cvs commit: src/sys/kern kern_timeout.c Cc: smp@FreeBSD.org, cp@bsdi.com, jake@io.yi.org, Jake Burkholder Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org On 16-Nov-00 John Baldwin wrote: >> I think we need a separate spin lock for the callout wheel, ala BSD/OS's >> callout_mtx. Hardclock looks at the callout wheel and is now a fast >> interrupt, so it can't acquire a sleep mutex. Its a little paranoid >> because hardclock doesn't actually traverse any lists, it just checks >> if the current callout bucket is empty, and potentially schedules >> softclock, but you could miss a very short timeout on an smp system. >> ticks could also get incremented in the middle of softclock's test >> for if the callout's time has come. >> >> I have patches that do this and make softclock INTR_MPSAFE, I just need >> to test them. > > Ok. I was about to check the BSD/OS code to see how this was done there. > >> There's actually another major problem with this. The run queue and >> sleep queue use the same list linkage in struct proc, so its not >> safe to release sched_lock while you're on the sleep queue. If >> the process blocks on giant in CURSIG, the sleep queue will get >> corrupted. We really need to split the run queue/sleep queue >> linkage. > > Ugh, ok. I'll do this next then. Grrrr. Grr, wouldn't you know it, bar just died with a double fault because panic: cpu_switch has wchan Happened when I Ctrl-C'd a process. :-P *sigh* -- John Baldwin -- http://www.FreeBSD.org/~jhb/ PGP Key: http://www.baldwin.cx/~john/pgpkey.asc "Power Users Use the Power to Serve!" - http://www.FreeBSD.org/ To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Thu Nov 16 15: 8:46 2000 Delivered-To: freebsd-smp@freebsd.org Received: from wantadilla.lemis.com (wantadilla.lemis.com [192.109.197.80]) by hub.freebsd.org (Postfix) with ESMTP id 1804137B4C5 for ; Thu, 16 Nov 2000 15:08:42 -0800 (PST) Received: (from grog@localhost) by wantadilla.lemis.com (8.11.0/8.9.3) id eAGN8Ej44698; Fri, 17 Nov 2000 09:38:14 +1030 (CST) (envelope-from grog) Date: Fri, 17 Nov 2000 09:38:14 +1030 From: Greg Lehey To: Alessandro de Manzano Cc: "freebsd-smp@freebsd.org" Subject: Re: dual celeron Message-ID: <20001117093813.B44342@wantadilla.lemis.com> References: <20001116091907.4983ED60F9@viger.playstos.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2.5i In-Reply-To: <20001116091907.4983ED60F9@viger.playstos.com>; from demanzano@playstos.com on Thu, Nov 16, 2000 at 10:20:41AM +0100 Organization: LEMIS, PO Box 460, Echunga SA 5153, Australia Phone: +61-8-8388-8286 Fax: +61-8-8388-8725 Mobile: +61-418-838-708 WWW-Home-Page: http://www.lemis.com/~grog X-PGP-Fingerprint: 6B 7B C3 8C 61 CD 54 AF 13 24 52 F8 6D A4 95 EF Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org On Thursday, 16 November 2000 at 10:20:41 +0100, Alessandro de Manzano wrote: > Hello! > > (sorry for bad english!) > > I'm assembling a little FreeBSD box using spare, second hand, parts. > > I've a couple of Intel Celeron 433Mhz processor (for socket 370) which > were used on a dual celeron machine with 2 riser cards for Slot 1. > > I would put them on a motherboard ABit BP6 (dual socket 370), just for > have a little SMP FBSD machine :-) > > Is there something I should know about this experiment ? This is pretty much exactly the configuration I'm using for SMP development. As such, even if there are glitches (I don't know of any), you can be sure somebody else will see them before you do :-) Greg -- Finger grog@lemis.com for PGP public key See complete headers for address and phone numbers To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Thu Nov 16 15:34:19 2000 Delivered-To: freebsd-smp@freebsd.org Received: from pike.osd.bsdi.com (pike.osd.bsdi.com [204.216.28.222]) by hub.freebsd.org (Postfix) with ESMTP id D1B9637B4C5; Thu, 16 Nov 2000 15:34:16 -0800 (PST) Received: from laptop.baldwin.cx (john@jhb-laptop.osd.bsdi.com [204.216.28.241]) by pike.osd.bsdi.com (8.11.0/8.9.3) with ESMTP id eAGNY5B71332; Thu, 16 Nov 2000 15:34:05 -0800 (PST) (envelope-from jhb@FreeBSD.org) Message-ID: X-Mailer: XFMail 1.4.0 on FreeBSD X-Priority: 3 (Normal) Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 8bit MIME-Version: 1.0 In-Reply-To: Date: Thu, 16 Nov 2000 15:34:43 -0800 (PST) From: John Baldwin To: John Baldwin Subject: Re: cvs commit: src/sys/kern kern_timeout.c Cc: Jake Burkholder , jake@io.yi.org, cp@bsdi.com, smp@FreeBSD.org Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org On 16-Nov-00 John Baldwin wrote: > > On 16-Nov-00 John Baldwin wrote: >>> I think we need a separate spin lock for the callout wheel, ala BSD/OS's >>> callout_mtx. Hardclock looks at the callout wheel and is now a fast >>> interrupt, so it can't acquire a sleep mutex. Its a little paranoid >>> because hardclock doesn't actually traverse any lists, it just checks >>> if the current callout bucket is empty, and potentially schedules >>> softclock, but you could miss a very short timeout on an smp system. >>> ticks could also get incremented in the middle of softclock's test >>> for if the callout's time has come. >>> >>> I have patches that do this and make softclock INTR_MPSAFE, I just need >>> to test them. >> >> Ok. I was about to check the BSD/OS code to see how this was done there. >> >>> There's actually another major problem with this. The run queue and >>> sleep queue use the same list linkage in struct proc, so its not >>> safe to release sched_lock while you're on the sleep queue. If >>> the process blocks on giant in CURSIG, the sleep queue will get >>> corrupted. We really need to split the run queue/sleep queue >>> linkage. >> >> Ugh, ok. I'll do this next then. Grrrr. > > Grr, wouldn't you know it, bar just died with a double fault because > > panic: cpu_switch has wchan > > Happened when I Ctrl-C'd a process. :-P > > *sigh* I actually don't like the concept of CURSIG() forcing a context switch due to needing to grab Giant. For one thing, it breaks the nice assertion of running processes not having p->p_wchan != NULL that caused my machine to panic. I'm trying a patch right now that grabs Giant in msleep() before we grab the sched_lock so that the call to CURSIG() before mi_switch() won't need to block. It then releases Giant after CURSIG(). For the CURSIG() after mi_switch(), doing another context switch due to blocking on Giant isn't a problem, so it doesn't mess with it. (Not that there is anything one could do to work around it.) -- John Baldwin -- http://www.FreeBSD.org/~jhb/ PGP Key: http://www.baldwin.cx/~john/pgpkey.asc "Power Users Use the Power to Serve!" - http://www.FreeBSD.org/ To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Thu Nov 16 18:12:30 2000 Delivered-To: freebsd-smp@freebsd.org Received: from pike.osd.bsdi.com (pike.osd.bsdi.com [204.216.28.222]) by hub.freebsd.org (Postfix) with ESMTP id B928137B479 for ; Thu, 16 Nov 2000 18:12:27 -0800 (PST) Received: from foo.osd.bsdi.com (root@foo.osd.bsdi.com [204.216.28.137]) by pike.osd.bsdi.com (8.11.0/8.9.3) with ESMTP id eAH2C8B76224; Thu, 16 Nov 2000 18:12:08 -0800 (PST) (envelope-from jhb@foo.osd.bsdi.com) Received: (from jhb@localhost) by foo.osd.bsdi.com (8.11.1/8.11.0) id eAH28iJ94107; Thu, 16 Nov 2000 18:08:44 -0800 (PST) (envelope-from jhb) Message-ID: X-Mailer: XFMail 1.4.0 on FreeBSD X-Priority: 3 (Normal) Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 8bit MIME-Version: 1.0 In-Reply-To: Date: Thu, 16 Nov 2000 18:08:44 -0800 (PST) Organization: BSD, Inc. From: John Baldwin To: smp@FreeBSD.org Subject: Re: cvs commit: src/sys/kern kern_timeout.c Cc: cp@bsdi.com, jake@io.yi.org, Jake Burkholder Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org On 16-Nov-00 John Baldwin wrote: > > On 16-Nov-00 John Baldwin wrote: >> >> On 16-Nov-00 John Baldwin wrote: >>>> I think we need a separate spin lock for the callout wheel, ala BSD/OS's >>>> callout_mtx. Hardclock looks at the callout wheel and is now a fast >>>> interrupt, so it can't acquire a sleep mutex. Its a little paranoid >>>> because hardclock doesn't actually traverse any lists, it just checks >>>> if the current callout bucket is empty, and potentially schedules >>>> softclock, but you could miss a very short timeout on an smp system. >>>> ticks could also get incremented in the middle of softclock's test >>>> for if the callout's time has come. >>>> >>>> I have patches that do this and make softclock INTR_MPSAFE, I just need >>>> to test them. >>> >>> Ok. I was about to check the BSD/OS code to see how this was done there. >>> >>>> There's actually another major problem with this. The run queue and >>>> sleep queue use the same list linkage in struct proc, so its not >>>> safe to release sched_lock while you're on the sleep queue. If >>>> the process blocks on giant in CURSIG, the sleep queue will get >>>> corrupted. We really need to split the run queue/sleep queue >>>> linkage. >>> >>> Ugh, ok. I'll do this next then. Grrrr. >> >> Grr, wouldn't you know it, bar just died with a double fault because >> >> panic: cpu_switch has wchan >> >> Happened when I Ctrl-C'd a process. :-P >> >> *sigh* > > I actually don't like the concept of CURSIG() forcing a context switch due to > needing to grab Giant. For one thing, it breaks the nice assertion of > running > processes not having p->p_wchan != NULL that caused my machine to panic. I'm > trying a patch right now that grabs Giant in msleep() before we grab the > sched_lock so that the call to CURSIG() before mi_switch() won't need to > block. > It then releases Giant after CURSIG(). For the CURSIG() after mi_switch(), > doing another context switch due to blocking on Giant isn't a problem, so it > doesn't mess with it. (Not that there is anything one could do to work > around > it.) Well, when I tried this the machine hung on the first fast interrupt handler it ran, so it doesn't look like this approach works either. :-/ I've tried splitting up p_procq into p_runq, p_sleepq, and p_mtxq (for processes blocked on a mutex), but while those kernels boot ok and seem to sort of run, I end up with hung processes. If I (or anyone else) don't have a good solution for this, then I think I will back out the changes to move Giant out of mi_switch() tomorrow afternoon until we have a solution for this. -- John Baldwin -- http://www.FreeBSD.org/~jhb/ PGP Key: http://www.Baldwin.cx/~john/pgpkey.asc "Power Users Use the Power to Serve!" - http://www.FreeBSD.org/ To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Thu Nov 16 21:26: 7 2000 Delivered-To: freebsd-smp@freebsd.org Received: from io.yi.org (h24-69-199-88.gv.shawcable.net [24.69.199.88]) by hub.freebsd.org (Postfix) with ESMTP id 679A837B479; Thu, 16 Nov 2000 21:25:57 -0800 (PST) Received: from io.yi.org (localhost.gvcl1.bc.wave.home.com [127.0.0.1]) by io.yi.org (Postfix) with ESMTP id 5F927BA7A; Thu, 16 Nov 2000 21:25:56 -0800 (PST) X-Mailer: exmh version 2.1.1 10/15/1999 To: John Baldwin Cc: smp@FreeBSD.ORG, cp@bsdi.com, Jake Burkholder Subject: Re: cvs commit: src/sys/kern kern_timeout.c In-Reply-To: Message from John Baldwin of "Thu, 16 Nov 2000 18:08:44 PST." Mime-Version: 1.0 Content-Type: multipart/mixed ; boundary="==_Exmh_15084400340" Date: Thu, 16 Nov 2000 21:25:56 -0800 From: Jake Burkholder Message-Id: <20001117052556.5F927BA7A@io.yi.org> Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org This is a multipart MIME message. --==_Exmh_15084400340 Content-Type: text/plain; charset=us-ascii > > On 16-Nov-00 John Baldwin wrote: > > > > On 16-Nov-00 John Baldwin wrote: > >> > >> On 16-Nov-00 John Baldwin wrote: > >>>> I think we need a separate spin lock for the callout wheel, ala BSD/OS's > >>>> callout_mtx. Hardclock looks at the callout wheel and is now a fast > >>>> interrupt, so it can't acquire a sleep mutex. Its a little paranoid > >>>> because hardclock doesn't actually traverse any lists, it just checks > >>>> if the current callout bucket is empty, and potentially schedules > >>>> softclock, but you could miss a very short timeout on an smp system. > >>>> ticks could also get incremented in the middle of softclock's test > >>>> for if the callout's time has come. > >>>> > >>>> I have patches that do this and make softclock INTR_MPSAFE, I just need > >>>> to test them. > >>> > >>> Ok. I was about to check the BSD/OS code to see how this was done there. > >>> > >>>> There's actually another major problem with this. The run queue and > >>>> sleep queue use the same list linkage in struct proc, so its not > >>>> safe to release sched_lock while you're on the sleep queue. If > >>>> the process blocks on giant in CURSIG, the sleep queue will get > >>>> corrupted. We really need to split the run queue/sleep queue > >>>> linkage. > >>> > >>> Ugh, ok. I'll do this next then. Grrrr. > >> > >> Grr, wouldn't you know it, bar just died with a double fault because > >> > >> panic: cpu_switch has wchan > >> > >> Happened when I Ctrl-C'd a process. :-P > >> > >> *sigh* > > > > I actually don't like the concept of CURSIG() forcing a context switch due to > > needing to grab Giant. For one thing, it breaks the nice assertion of > > running > > processes not having p->p_wchan != NULL that caused my machine to panic. I'm > > trying a patch right now that grabs Giant in msleep() before we grab the > > sched_lock so that the call to CURSIG() before mi_switch() won't need to > > block. > > It then releases Giant after CURSIG(). For the CURSIG() after mi_switch(), > > doing another context switch due to blocking on Giant isn't a problem, so it > > doesn't mess with it. (Not that there is anything one could do to work > > around > > it.) > > Well, when I tried this the machine hung on the first fast interrupt handler it > ran, so it doesn't look like this approach works either. :-/ I've tried > splitting up p_procq into p_runq, p_sleepq, and p_mtxq (for processes blocked > on a mutex), but while those kernels boot ok and seem to sort of run, I end up > with hung processes. If I (or anyone else) don't have a good solution for this, > then I think I will back out the changes to move Giant out of mi_switch() > tomorrow afternoon until we have a solution for this. Hmm. This is unfortunate because the change to mi_switch is absolutely necessary. I don't see why changing the queues would cause problems. The p_runq/p_mtxq isn't really necessary because they are mutually exclusive, like the runq/sleepq used to be. Looking at the use of DROP_GIANT_NOSWITCH, I think that it should be after the mtx_enter(&sched_lock, MTX_SPIN) instead of before. The idea with the noswitch flag is that it allows you to release a sleep mutex with interrupts disabled, otherwise there's no point. If interrupts are enabled you're supposed to be able to block. Here's a patch that seems to work ok here. It works on my UP box running X and stuff, I don't notice any hung processes. It works on the SMP box, I built a kernel over NFS and it seemed ok, except that I can't get a kernel to boot using the serial console with WITNESS enabled. It stops at the twiddle thing before the copyright is printed and just hangs. This also happens without the patch and even with a UP kernel, so I don't really know what's going on. Please try and let me know if you still see the hung processes. I think we'll just have to settle with no longer being able to assert that a process has no wchan in cpu_switch. There's bound to be alot of contention with Giant, but eventually signals should be protected by a per-process mutex, so a context switch at that point is less likely. Jake > > -- > > John Baldwin -- http://www.FreeBSD.org/~jhb/ > PGP Key: http://www.Baldwin.cx/~john/pgpkey.asc > "Power Users Use the Power to Serve!" - http://www.FreeBSD.org/ > > > To Unsubscribe: send mail to majordomo@FreeBSD.org > with "unsubscribe freebsd-smp" in the body of the message --==_Exmh_15084400340 Content-Type: text/plain ; name="slpq.diff"; charset=us-ascii Content-Description: slpq.diff Content-Disposition: attachment; filename="slpq.diff" Index: i386/i386/swtch.s =================================================================== RCS file: /home/ncvs/src/sys/i386/i386/swtch.s,v retrieving revision 1.98 diff -u -r1.98 swtch.s --- i386/i386/swtch.s 2000/10/13 22:03:29 1.98 +++ i386/i386/swtch.s 2000/11/17 03:23:56 @@ -184,8 +184,10 @@ andl $~AST_RESCHED,_astpending #ifdef DIAGNOSTIC +#ifdef notanymore cmpl %eax,P_WCHAN(%ecx) jne badsw1 +#endif cmpb $SRUN,P_STAT(%ecx) jne badsw2 #endif Index: i386/i386/trap.c =================================================================== RCS file: /home/ncvs/src/sys/i386/i386/trap.c,v retrieving revision 1.160 diff -u -r1.160 trap.c --- i386/i386/trap.c 2000/11/16 02:16:43 1.160 +++ i386/i386/trap.c 2000/11/17 03:20:24 @@ -192,8 +192,8 @@ * our priority. */ s = splhigh(); - DROP_GIANT_NOSWITCH(); mtx_enter(&sched_lock, MTX_SPIN); + DROP_GIANT_NOSWITCH(); setrunqueue(p); p->p_stats->p_ru.ru_nivcsw++; mi_switch(); Index: ia64/ia64/trap.c =================================================================== RCS file: /home/ncvs/src/sys/ia64/ia64/trap.c,v retrieving revision 1.6 diff -u -r1.6 trap.c --- ia64/ia64/trap.c 2000/11/16 02:16:43 1.6 +++ ia64/ia64/trap.c 2000/11/17 04:34:19 @@ -102,8 +102,8 @@ * indicated by our priority. */ s = splstatclock(); - DROP_GIANT_NOSWITCH(); mtx_enter(&sched_lock, MTX_SPIN); + DROP_GIANT_NOSWITCH(); setrunqueue(p); p->p_stats->p_ru.ru_nivcsw++; mi_switch(); Index: alpha/alpha/trap.c =================================================================== RCS file: /home/ncvs/src/sys/alpha/alpha/trap.c,v retrieving revision 1.36 diff -u -r1.36 trap.c --- alpha/alpha/trap.c 2000/11/16 02:16:42 1.36 +++ alpha/alpha/trap.c 2000/11/17 04:34:06 @@ -120,8 +120,8 @@ * indicated by our priority. */ s = splstatclock(); - DROP_GIANT_NOSWITCH(); mtx_enter(&sched_lock, MTX_SPIN); + DROP_GIANT_NOSWITCH(); setrunqueue(p); p->p_stats->p_ru.ru_nivcsw++; mi_switch(); Index: sys/proc.h =================================================================== RCS file: /home/ncvs/src/sys/sys/proc.h,v retrieving revision 1.122 diff -u -r1.122 proc.h --- sys/proc.h 2000/10/29 16:06:54 1.122 +++ sys/proc.h 2000/11/17 03:30:57 @@ -127,7 +127,8 @@ struct ithd; struct proc { - TAILQ_ENTRY(proc) p_procq; /* run/sleep queue. */ + TAILQ_ENTRY(proc) p_procq; /* run/mutex queue. */ + TAILQ_ENTRY(proc) p_slpq; /* sleep queue. */ LIST_ENTRY(proc) p_list; /* List of all processes. */ /* substructures: */ Index: kern/kern_mutex.c =================================================================== RCS file: /home/ncvs/src/sys/kern/kern_mutex.c,v retrieving revision 1.16 diff -u -r1.16 kern_mutex.c --- kern/kern_mutex.c 2000/11/16 02:16:44 1.16 +++ kern/kern_mutex.c 2000/11/17 03:22:42 @@ -718,6 +718,7 @@ }; static char *sleep_list[] = { + "Giant", NULL }; Index: kern/kern_sig.c =================================================================== RCS file: /home/ncvs/src/sys/kern/kern_sig.c,v retrieving revision 1.91 diff -u -r1.91 kern_sig.c --- kern/kern_sig.c 2000/11/16 02:16:44 1.91 +++ kern/kern_sig.c 2000/11/17 03:21:45 @@ -1283,8 +1283,8 @@ psignal(p->p_pptr, SIGCHLD); do { stop(p); - DROP_GIANT_NOSWITCH(); mtx_enter(&sched_lock, MTX_SPIN); + DROP_GIANT_NOSWITCH(); mi_switch(); mtx_exit(&sched_lock, MTX_SPIN); PICKUP_GIANT(); @@ -1356,8 +1356,8 @@ stop(p); if ((p->p_pptr->p_procsig->ps_flag & PS_NOCLDSTOP) == 0) psignal(p->p_pptr, SIGCHLD); - DROP_GIANT_NOSWITCH(); mtx_enter(&sched_lock, MTX_SPIN); + DROP_GIANT_NOSWITCH(); mi_switch(); mtx_exit(&sched_lock, MTX_SPIN); PICKUP_GIANT(); Index: kern/kern_subr.c =================================================================== RCS file: /home/ncvs/src/sys/kern/kern_subr.c,v retrieving revision 1.37 diff -u -r1.37 kern_subr.c --- kern/kern_subr.c 2000/11/16 02:16:44 1.37 +++ kern/kern_subr.c 2000/11/17 03:21:59 @@ -377,8 +377,8 @@ p = curproc; s = splhigh(); - DROP_GIANT_NOSWITCH(); mtx_enter(&sched_lock, MTX_SPIN); + DROP_GIANT_NOSWITCH(); p->p_priority = p->p_usrpri; setrunqueue(p); p->p_stats->p_ru.ru_nivcsw++; Index: kern/kern_synch.c =================================================================== RCS file: /home/ncvs/src/sys/kern/kern_synch.c,v retrieving revision 1.108 diff -u -r1.108 kern_synch.c --- kern/kern_synch.c 2000/11/16 02:16:44 1.108 +++ kern/kern_synch.c 2000/11/17 03:29:16 @@ -420,9 +420,9 @@ if (p && KTRPOINT(p, KTR_CSW)) ktrcsw(p->p_tracep, 1, 0); #endif - DROP_GIANT_NOSWITCH(); WITNESS_SLEEP(0, mtx); mtx_enter(&sched_lock, MTX_SPIN); + DROP_GIANT_NOSWITCH(); if (mtx != NULL) { mtx_assert(mtx, MA_OWNED | MA_NOTRECURSED); @@ -461,7 +461,7 @@ p->p_nativepri = p->p_priority; CTR4(KTR_PROC, "msleep: proc %p (pid %d, %s), schedlock %p", p, p->p_pid, p->p_comm, (void *) sched_lock.mtx_lock); - TAILQ_INSERT_TAIL(&slpque[LOOKUP(ident)], p, p_procq); + TAILQ_INSERT_TAIL(&slpque[LOOKUP(ident)], p, p_slpq); if (timo) thandle = timeout(endtsleep, (void *)p, timo); /* @@ -583,7 +583,7 @@ p->p_slptime = 0; p->p_asleep.as_priority = priority; p->p_asleep.as_timo = timo; - TAILQ_INSERT_TAIL(&slpque[LOOKUP(ident)], p, p_procq); + TAILQ_INSERT_TAIL(&slpque[LOOKUP(ident)], p, p_slpq); } mtx_exit(&sched_lock, MTX_SPIN); @@ -612,9 +612,9 @@ int s; WITNESS_SAVE_DECL(mtx); - DROP_GIANT_NOSWITCH(); WITNESS_SLEEP(0, mtx); mtx_enter(&sched_lock, MTX_SPIN); + DROP_GIANT_NOSWITCH(); if (mtx != NULL) { mtx_assert(mtx, MA_OWNED | MA_NOTRECURSED); WITNESS_SAVE(mtx, mtx); @@ -778,7 +778,7 @@ s = splhigh(); mtx_enter(&sched_lock, MTX_SPIN); if (p->p_wchan) { - TAILQ_REMOVE(&slpque[LOOKUP(p->p_wchan)], p, p_procq); + TAILQ_REMOVE(&slpque[LOOKUP(p->p_wchan)], p, p_slpq); p->p_wchan = 0; } mtx_exit(&sched_lock, MTX_SPIN); @@ -800,9 +800,9 @@ mtx_enter(&sched_lock, MTX_SPIN); qp = &slpque[LOOKUP(ident)]; restart: - TAILQ_FOREACH(p, qp, p_procq) { + TAILQ_FOREACH(p, qp, p_slpq) { if (p->p_wchan == ident) { - TAILQ_REMOVE(qp, p, p_procq); + TAILQ_REMOVE(qp, p, p_slpq); p->p_wchan = 0; if (p->p_stat == SSLEEP) { /* OPTIMIZED EXPANSION OF setrunnable(p); */ @@ -846,9 +846,9 @@ mtx_enter(&sched_lock, MTX_SPIN); qp = &slpque[LOOKUP(ident)]; - TAILQ_FOREACH(p, qp, p_procq) { + TAILQ_FOREACH(p, qp, p_slpq) { if (p->p_wchan == ident) { - TAILQ_REMOVE(qp, p, p_procq); + TAILQ_REMOVE(qp, p, p_slpq); p->p_wchan = 0; if (p->p_stat == SSLEEP) { /* OPTIMIZED EXPANSION OF setrunnable(p); */ --==_Exmh_15084400340-- To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Fri Nov 17 7:23:48 2000 Delivered-To: freebsd-smp@freebsd.org Received: from zibbi.icomtek.csir.co.za (zibbi.icomtek.csir.co.za [146.64.24.58]) by hub.freebsd.org (Postfix) with ESMTP id 9BF4637B4C5; Fri, 17 Nov 2000 07:23:38 -0800 (PST) Received: (from jhay@localhost) by zibbi.icomtek.csir.co.za (8.11.0/8.11.0) id eAHFNEV94801; Fri, 17 Nov 2000 17:23:14 +0200 (SAT) (envelope-from jhay) From: John Hay Message-Id: <200011171523.eAHFNEV94801@zibbi.icomtek.csir.co.za> Subject: Re: cvs commit: src/sys/kern kern_timeout.c In-Reply-To: <20001117052556.5F927BA7A@io.yi.org> from Jake Burkholder at "Nov 16, 2000 09:25:56 pm" To: jburkhol@home.com (Jake Burkholder) Date: Fri, 17 Nov 2000 17:23:14 +0200 (SAT) Cc: jhb@FreeBSD.ORG (John Baldwin), smp@FreeBSD.ORG, cp@bsdi.com X-Mailer: ELM [version 2.4ME+ PL54 (25)] MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org > > Hmm. This is unfortunate because the change to mi_switch is absolutely > necessary. I don't see why changing the queues would cause problems. > The p_runq/p_mtxq isn't really necessary because they are mutually > exclusive, like the runq/sleepq used to be. > > Looking at the use of DROP_GIANT_NOSWITCH, I think that it should be > after the mtx_enter(&sched_lock, MTX_SPIN) instead of before. The idea > with the noswitch flag is that it allows you to release a sleep mutex > with interrupts disabled, otherwise there's no point. If interrupts are > enabled you're supposed to be able to block. > > Here's a patch that seems to work ok here. It works on my UP box running > X and stuff, I don't notice any hung processes. It works on the SMP box, > I built a kernel over NFS and it seemed ok, except that I can't > get a kernel to boot using the serial console with WITNESS enabled. It > stops at the twiddle thing before the copyright is printed and just hangs. > This also happens without the patch and even with a UP kernel, so I don't > really know what's going on. Please try and let me know if you still > see the hung processes. > > I think we'll just have to settle with no longer being able to assert > that a process has no wchan in cpu_switch. There's bound to be alot of > contention with Giant, but eventually signals should be protected by > a per-process mutex, so a context switch at that point is less likely. > Well I have tried your patch and so far it looks good. I have finished a "make -j13 world" and a "make release NODOC=YES WORLD_FLAGS=-j4" with no problems on a dual 266MHz PII. John -- John Hay -- John.Hay@icomtek.csir.co.za To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Fri Nov 17 9:30:10 2000 Delivered-To: freebsd-smp@freebsd.org Received: from pike.osd.bsdi.com (pike.osd.bsdi.com [204.216.28.222]) by hub.freebsd.org (Postfix) with ESMTP id 7F25B37B479 for ; Fri, 17 Nov 2000 09:30:07 -0800 (PST) Received: from laptop.baldwin.cx (john@jhb-laptop.osd.bsdi.com [204.216.28.241]) by pike.osd.bsdi.com (8.11.0/8.9.3) with ESMTP id eAHHTnB94337; Fri, 17 Nov 2000 09:29:50 -0800 (PST) (envelope-from jhb@FreeBSD.org) Message-ID: X-Mailer: XFMail 1.4.0 on FreeBSD X-Priority: 3 (Normal) Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 8bit MIME-Version: 1.0 In-Reply-To: <20001117052556.5F927BA7A@io.yi.org> Date: Fri, 17 Nov 2000 09:30:31 -0800 (PST) From: John Baldwin To: Jake Burkholder Subject: Re: cvs commit: src/sys/kern kern_timeout.c Cc: cp@bsdi.com, smp@FreeBSD.org Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org On 17-Nov-00 Jake Burkholder wrote: >> Well, when I tried this the machine hung on the first fast interrupt handler >> it >> ran, so it doesn't look like this approach works either. :-/ I've tried >> splitting up p_procq into p_runq, p_sleepq, and p_mtxq (for processes >> blocked >> on a mutex), but while those kernels boot ok and seem to sort of run, I end >> up >> with hung processes. If I (or anyone else) don't have a good solution for >> this, >> then I think I will back out the changes to move Giant out of mi_switch() >> tomorrow afternoon until we have a solution for this. > > Hmm. This is unfortunate because the change to mi_switch is absolutely > necessary. I don't see why changing the queues would cause problems. > The p_runq/p_mtxq isn't really necessary because they are mutually > exclusive, like the runq/sleepq used to be. I was merely doing it for completeness, but my SMP machines would end up with hung processes, though the machines are running ok. > Looking at the use of DROP_GIANT_NOSWITCH, I think that it should be > after the mtx_enter(&sched_lock, MTX_SPIN) instead of before. The idea > with the noswitch flag is that it allows you to release a sleep mutex > with interrupts disabled, otherwise there's no point. If interrupts are > enabled you're supposed to be able to block. That would be ok. I also used NOSWITCH because we are about to call mi_switch() anyways. However, go ahead and do that change. > Here's a patch that seems to work ok here. It works on my UP box running > X and stuff, I don't notice any hung processes. It works on the SMP box, > I built a kernel over NFS and it seemed ok, except that I can't > get a kernel to boot using the serial console with WITNESS enabled. It > stops at the twiddle thing before the copyright is printed and just hangs. > This also happens without the patch and even with a UP kernel, so I don't > really know what's going on. Please try and let me know if you still > see the hung processes. > > I think we'll just have to settle with no longer being able to assert > that a process has no wchan in cpu_switch. There's bound to be alot of > contention with Giant, but eventually signals should be protected by > a per-process mutex, so a context switch at that point is less likely. Yes. I have a few other patches for swtch.s, but the rest of your patch looks great, so go ahead and commit it. I'll commit my swtch.s fixes in a sec. Glad to see that school isn't keeping you too busy now. :) > Jake -- John Baldwin -- http://www.FreeBSD.org/~jhb/ PGP Key: http://www.baldwin.cx/~john/pgpkey.asc "Power Users Use the Power to Serve!" - http://www.FreeBSD.org/ To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Fri Nov 17 9:33:51 2000 Delivered-To: freebsd-smp@freebsd.org Received: from solaris.matti.ee (solaris.matti.ee [194.126.98.135]) by hub.freebsd.org (Postfix) with ESMTP id 11F3B37B479 for ; Fri, 17 Nov 2000 09:33:40 -0800 (PST) Received: from myhakas.matti.ee (myhakas.matti.ee [194.126.114.87]) by solaris.matti.ee (Postfix) with ESMTP id 7C5122CE3A; Fri, 17 Nov 2000 19:33:30 +0200 (EET) Received: (from vallo@localhost) by myhakas.matti.ee (8.11.1/8.11.1) id eAHHWxV80835; Fri, 17 Nov 2000 19:32:59 +0200 (EET) (envelope-from vallo) Date: Fri, 17 Nov 2000 19:32:59 +0200 From: Vallo Kallaste To: "Rodney W. Grimes" Cc: "Michael C . Wu" , freebsd-smp@FreeBSD.ORG Subject: 10/100 low-end switches [was: Re: via chipset and SMP] Message-ID: <20001117193259.B49176@myhakas.matti.ee> Reply-To: vallo@matti.ee References: <20001112045858.A7123@peorth.iteration.net> <200011121608.IAA97211@gndrsh.dnsmgr.net> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.3.11i In-Reply-To: <200011121608.IAA97211@gndrsh.dnsmgr.net>; from freebsd@gndrsh.dnsmgr.net on Sun, Nov 12, 2000 at 08:08:10AM -0800 Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org On Sun, Nov 12, 2000 at 08:08:10AM -0800, "Rodney W. Grimes" wrote: > There ``low cost'' 10/100 switches are junk, they make some nicer > switches using BroadCOM chips, again, probably some of the better > 10/100 switch chips on the market. Can you, as hardware geek, advise us what low-end 10/100 switches are worth of buy? It took me several weeks to shout down the lowend D-Link switches we had in sight some time ago. We're now running some 12-port SMC's, they're cheap.. -- Vallo Kallaste vallo@matti.ee To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Fri Nov 17 16: 7:15 2000 Delivered-To: freebsd-smp@freebsd.org Received: from pike.osd.bsdi.com (pike.osd.bsdi.com [204.216.28.222]) by hub.freebsd.org (Postfix) with ESMTP id 69B2037B479 for ; Fri, 17 Nov 2000 16:07:12 -0800 (PST) Received: from laptop.baldwin.cx (john@jhb-laptop.osd.bsdi.com [204.216.28.241]) by pike.osd.bsdi.com (8.11.0/8.9.3) with ESMTP id eAI06wB09146 for ; Fri, 17 Nov 2000 16:06:58 -0800 (PST) (envelope-from jhb@FreeBSD.org) Message-ID: X-Mailer: XFMail 1.4.0 on FreeBSD X-Priority: 3 (Normal) Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 8bit MIME-Version: 1.0 Date: Fri, 17 Nov 2000 16:07:40 -0800 (PST) From: John Baldwin To: smp@FreeBSD.org Subject: Mutex List Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org To attempt to make life a little easier, I started working today on some docco to help keep things more organized in the SMPng world. Specifically, I started on making a list of all the mutexes we have so far and what each one protects, etc. It is far from complete at this point (only sched_lock, vm86pcb_lock, and Giant are done at this point), but it is a proof of concept at least. It is written in DocBook, so that it can be checked into teh documentation tree and easily maintained by developers. (Well, easier than having 1 person be a bottleneck to receive all submissions). You can see the current HTML version at http://www.FreeBSD.org/~jhb/mutex_list/article.html. Feedback (especially the necessary bits needed to document any mutexes you may have added) is welcome. -- John Baldwin -- http://www.FreeBSD.org/~jhb/ PGP Key: http://www.baldwin.cx/~john/pgpkey.asc "Power Users Use the Power to Serve!" - http://www.FreeBSD.org/ To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Fri Nov 17 16:28:18 2000 Delivered-To: freebsd-smp@freebsd.org Received: from pike.osd.bsdi.com (pike.osd.bsdi.com [204.216.28.222]) by hub.freebsd.org (Postfix) with ESMTP id 4D72C37B479 for ; Fri, 17 Nov 2000 16:28:17 -0800 (PST) Received: from laptop.baldwin.cx (john@jhb-laptop.osd.bsdi.com [204.216.28.241]) by pike.osd.bsdi.com (8.11.0/8.9.3) with ESMTP id eAI0S3B09680 for ; Fri, 17 Nov 2000 16:28:03 -0800 (PST) (envelope-from jhb@FreeBSD.org) Message-ID: X-Mailer: XFMail 1.4.0 on FreeBSD X-Priority: 3 (Normal) Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 8bit MIME-Version: 1.0 In-Reply-To: <200011180021.QAA61821@freefall.freebsd.org> Date: Fri, 17 Nov 2000 16:28:45 -0800 (PST) From: John Baldwin To: smp@FreeBSD.org Subject: RE: cvs commit: src/sys/kern kern_timeout.c Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org On 18-Nov-00 John Baldwin wrote: > jhb 2000/11/17 16:21:01 PST > > Modified files: > sys/kern kern_timeout.c > Log: > Release sched_lock very briefly to give interrupts a chance to fire if we > are in softclock() for a long time. The old code already did an > splx()/slphigh() pair here, I just missed adding in the equivalent mutex > operations on sched_lock earlier. Before this I could get processes to hang (the box was still up, but one process was stuck, tying up its pty) though the box was still fine (I could login and do stuff) if I ran a buildworld -j X where X > 1. With this I built a kernel with -j 8 and am currently chugging through a -j 64 world. -- John Baldwin -- http://www.FreeBSD.org/~jhb/ PGP Key: http://www.baldwin.cx/~john/pgpkey.asc "Power Users Use the Power to Serve!" - http://www.FreeBSD.org/ To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Fri Nov 17 16:47: 4 2000 Delivered-To: freebsd-smp@freebsd.org Received: from pike.osd.bsdi.com (pike.osd.bsdi.com [204.216.28.222]) by hub.freebsd.org (Postfix) with ESMTP id E0AE037B479; Fri, 17 Nov 2000 16:47:01 -0800 (PST) Received: from laptop.baldwin.cx (john@jhb-laptop.osd.bsdi.com [204.216.28.241]) by pike.osd.bsdi.com (8.11.0/8.9.3) with ESMTP id eAI0klB10380; Fri, 17 Nov 2000 16:46:47 -0800 (PST) (envelope-from jhb@FreeBSD.org) Message-ID: X-Mailer: XFMail 1.4.0 on FreeBSD X-Priority: 3 (Normal) Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 8bit MIME-Version: 1.0 In-Reply-To: Date: Fri, 17 Nov 2000 16:47:30 -0800 (PST) From: John Baldwin To: John Baldwin Subject: RE: cvs commit: src/sys/kern kern_timeout.c Cc: smp@FreeBSD.org Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org On 18-Nov-00 John Baldwin wrote: > > On 18-Nov-00 John Baldwin wrote: >> jhb 2000/11/17 16:21:01 PST >> >> Modified files: >> sys/kern kern_timeout.c >> Log: >> Release sched_lock very briefly to give interrupts a chance to fire if we >> are in softclock() for a long time. The old code already did an >> splx()/slphigh() pair here, I just missed adding in the equivalent mutex >> operations on sched_lock earlier. > > Before this I could get processes to hang (the box was still up, but one > process was stuck, tying up its pty) though the box was still fine (I could > login and do stuff) if I ran a buildworld -j X where X > 1. With this I > built > a kernel with -j 8 and am currently chugging through a -j 64 world. Grrrr. Spoke too soon. :( It took longer, but I have one hung process (though the rest of the buildworld is still chugging along). I notice the problem because I have one sh process that has been a zombie for a way too long. It's parent is a make doing a make cleandir, and the parent is in SSLEEP on "pipewr". I notice that when the kernel tsleep()'s with "pipewr", it does a tsleep() with PCATCH, so I am betting that there it is still related to the CURSIG() stuff in msleep(). Since this doesn't happen very often, I'm guessing that this comes about when a process is switched out because it blocked on Giant. -- John Baldwin -- http://www.FreeBSD.org/~jhb/ PGP Key: http://www.baldwin.cx/~john/pgpkey.asc "Power Users Use the Power to Serve!" - http://www.FreeBSD.org/ To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Fri Nov 17 18:59:30 2000 Delivered-To: freebsd-smp@freebsd.org Received: from io.yi.org (h24-69-199-88.gv.shawcable.net [24.69.199.88]) by hub.freebsd.org (Postfix) with ESMTP id B9EBC37B479; Fri, 17 Nov 2000 18:59:12 -0800 (PST) Received: from io.yi.org (localhost.gvcl1.bc.wave.home.com [127.0.0.1]) by io.yi.org (Postfix) with ESMTP id 158D4BA7A; Fri, 17 Nov 2000 18:59:12 -0800 (PST) X-Mailer: exmh version 2.1.1 10/15/1999 To: John Baldwin Cc: smp@FreeBSD.ORG Subject: Re: cvs commit: src/sys/kern kern_timeout.c In-Reply-To: Message from John Baldwin of "Fri, 17 Nov 2000 16:47:30 PST." Mime-Version: 1.0 Content-Type: multipart/mixed ; boundary="==_Exmh_18918598640" Date: Fri, 17 Nov 2000 18:59:12 -0800 From: Jake Burkholder Message-Id: <20001118025912.158D4BA7A@io.yi.org> Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org This is a multipart MIME message. --==_Exmh_18918598640 Content-Type: text/plain; charset=us-ascii > > On 18-Nov-00 John Baldwin wrote: > > > > On 18-Nov-00 John Baldwin wrote: > >> jhb 2000/11/17 16:21:01 PST > >> > >> Modified files: > >> sys/kern kern_timeout.c > >> Log: > >> Release sched_lock very briefly to give interrupts a chance to fire if we > >> are in softclock() for a long time. The old code already did an > >> splx()/slphigh() pair here, I just missed adding in the equivalent mutex > >> operations on sched_lock earlier. > > > > Before this I could get processes to hang (the box was still up, but one > > process was stuck, tying up its pty) though the box was still fine (I could > > login and do stuff) if I ran a buildworld -j X where X > 1. With this I > > built > > a kernel with -j 8 and am currently chugging through a -j 64 world. > > Grrrr. Spoke too soon. :( It took longer, but I have one hung process (though > the rest of the buildworld is still chugging along). I notice the problem > because I have one sh process that has been a zombie for a way too long. It's > parent is a make doing a make cleandir, and the parent is in SSLEEP on > "pipewr". I notice that when the kernel tsleep()'s with "pipewr", it does a > tsleep() with PCATCH, so I am betting that there it is still related to the > CURSIG() stuff in msleep(). Since this doesn't happen very often, I'm guessing > that this comes about when a process is switched out because it blocked on > Giant. Hmm. Do you know if its a timed sleep? If so, its possible that you're missing a timeout because hardclock and softclock are racing. If its not a timeout, and the wakeup happens while we're blocked on Giant, it should be OK because wakeup will set p_wchan to 0 even if the process is in state SMTX and we'll goto resume when the process gets unblocked. Anyway, here's the callout_lock patch I mentioned, it might help. It will conflict with the above commit because I don't have it locally yet, but its small. Before this gets committed I'd like to move the callout initialization to its own sysinit in kern_timeout.c, so we can initialize the lock there instead of in all the machdep.c's. I think the APM_FIXUP_CALLTODO thing has rotted, its based on the old delta queue timeout implementation. Jake > > -- > > John Baldwin -- http://www.FreeBSD.org/~jhb/ > PGP Key: http://www.baldwin.cx/~john/pgpkey.asc > "Power Users Use the Power to Serve!" - http://www.FreeBSD.org/ > > > To Unsubscribe: send mail to majordomo@FreeBSD.org > with "unsubscribe freebsd-smp" in the body of the message --==_Exmh_18918598640 Content-Type: text/plain ; name="timo2.diff"; charset=us-ascii Content-Description: timo2.diff Content-Disposition: attachment; filename="timo2.diff" Index: sys/callout.h =================================================================== RCS file: /home/ncvs/src/sys/sys/callout.h,v retrieving revision 1.17 diff -u -r1.17 callout.h --- sys/callout.h 2000/05/26 02:06:53 1.17 +++ sys/callout.h 2000/11/18 02:00:18 @@ -61,6 +61,7 @@ #define CALLOUT_LOCAL_ALLOC 0x0001 /* was allocated from callfree */ #define CALLOUT_ACTIVE 0x0002 /* callout is currently active */ #define CALLOUT_PENDING 0x0004 /* callout is waiting for timeout */ +#define CALLOUT_MPSAFE 0x0008 /* callout handler is mp safe */ struct callout_handle { struct callout *callout; @@ -72,6 +73,7 @@ extern int ncallout; extern struct callout_tailq *callwheel; extern int callwheelsize, callwheelbits, callwheelmask, softticks; +extern struct mtx callout_lock; #define callout_active(c) ((c)->c_flags & CALLOUT_ACTIVE) #define callout_deactivate(c) ((c)->c_flags &= ~CALLOUT_ACTIVE) Index: kern/kern_clock.c =================================================================== RCS file: /home/ncvs/src/sys/kern/kern_clock.c,v retrieving revision 1.115 diff -u -r1.115 kern_clock.c --- kern/kern_clock.c 2000/10/25 05:19:38 1.115 +++ kern/kern_clock.c 2000/11/18 02:05:45 @@ -154,6 +154,7 @@ register struct clockframe *frame; { register struct proc *p; + int need_softclock = 0; p = curproc; if (p != idleproc) { @@ -187,16 +188,25 @@ statclock(frame); tc_windup(); - ticks++; /* * Process callouts at a very low cpu priority, so we don't keep the * relatively high clock interrupt priority any longer than necessary. */ + mtx_enter(&callout_lock, MTX_SPIN); + ticks++; if (TAILQ_FIRST(&callwheel[ticks & callwheelmask]) != NULL) { - sched_swi(softclock_ih, SWI_NOSWITCH); + need_softclock = 1; } else if (softticks + 1 == ticks) ++softticks; + mtx_exit(&callout_lock, MTX_SPIN); + + /* + * sched_swi acquires sched_lock, so we don't want to call it with + * callout_lock held; incorrect locking order. + */ + if (need_softclock) + sched_swi(softclock_ih, SWI_NOSWITCH); } /* Index: kern/kern_intr.c =================================================================== RCS file: /home/ncvs/src/sys/kern/kern_intr.c,v retrieving revision 1.31 diff -u -r1.31 kern_intr.c --- kern/kern_intr.c 2000/11/15 22:05:23 1.31 +++ kern/kern_intr.c 2000/11/18 02:06:07 @@ -258,7 +258,7 @@ { net_ih = sinthand_add("net", NULL, swi_net, NULL, SWI_NET, 0); softclock_ih = - sinthand_add("clock", &clk_ithd, softclock, NULL, SWI_CLOCK, 0); + sinthand_add("clock", &clk_ithd, softclock, NULL, SWI_CLOCK, INTR_MPSAFE); vm_ih = sinthand_add("vm", NULL, swi_vm, NULL, SWI_VM, 0); } Index: kern/kern_mutex.c =================================================================== RCS file: /home/ncvs/src/sys/kern/kern_mutex.c,v retrieving revision 1.17 diff -u -r1.17 kern_mutex.c --- kern/kern_mutex.c 2000/11/17 18:09:14 1.17 +++ kern/kern_mutex.c 2000/11/18 02:09:55 @@ -703,6 +703,7 @@ #ifdef __i386__ "clk", #endif + "callout", /* * leaf locks */ Index: kern/kern_timeout.c =================================================================== RCS file: /home/ncvs/src/sys/kern/kern_timeout.c,v retrieving revision 1.61 diff -u -r1.61 kern_timeout.c --- kern/kern_timeout.c 2000/11/16 21:20:52 1.61 +++ kern/kern_timeout.c 2000/11/18 02:18:10 @@ -56,6 +56,7 @@ int callwheelsize, callwheelbits, callwheelmask; struct callout_tailq *callwheel; int softticks; /* Like ticks, but for softclock(). */ +MUTEX_DECLARE(, callout_lock); static struct callout *nextsoftcheck; /* Next callout to be checked. */ @@ -90,7 +91,7 @@ steps = 0; s = splhigh(); - mtx_enter(&sched_lock, MTX_SPIN); + mtx_enter(&callout_lock, MTX_SPIN); while (softticks != ticks) { softticks++; /* @@ -107,8 +108,10 @@ if (steps >= MAX_SOFTCLOCK_STEPS) { nextsoftcheck = c; /* Give interrupts a chance. */ + mtx_exit(&callout_lock, MTX_SPIN); splx(s); s = splhigh(); + mtx_enter(&callout_lock, MTX_SPIN); c = nextsoftcheck; steps = 0; } @@ -129,18 +132,22 @@ c->c_flags = (c->c_flags & ~CALLOUT_PENDING); } - mtx_exit(&sched_lock, MTX_SPIN); + mtx_exit(&callout_lock, MTX_SPIN); + if (!(c->c_flags & CALLOUT_MPSAFE)) + mtx_enter(&Giant, MTX_DEF); splx(s); c_func(c_arg); s = splhigh(); - mtx_enter(&sched_lock, MTX_SPIN); + if (!(c->c_flags & CALLOUT_MPSAFE)) + mtx_exit(&Giant, MTX_DEF); + mtx_enter(&callout_lock, MTX_SPIN); steps = 0; c = nextsoftcheck; } } } nextsoftcheck = NULL; - mtx_exit(&sched_lock, MTX_SPIN); + mtx_exit(&callout_lock, MTX_SPIN); splx(s); } @@ -171,7 +178,7 @@ struct callout_handle handle; s = splhigh(); - mtx_enter(&sched_lock, MTX_SPIN); + mtx_enter(&callout_lock, MTX_SPIN); /* Fill in the next free callout structure. */ new = SLIST_FIRST(&callfree); @@ -183,7 +190,7 @@ callout_reset(new, to_ticks, ftn, arg); handle.callout = new; - mtx_exit(&sched_lock, MTX_SPIN); + mtx_exit(&callout_lock, MTX_SPIN); splx(s); return (handle); } @@ -205,10 +212,10 @@ return; s = splhigh(); - mtx_enter(&sched_lock, MTX_SPIN); + mtx_enter(&callout_lock, MTX_SPIN); if (handle.callout->c_func == ftn && handle.callout->c_arg == arg) callout_stop(handle.callout); - mtx_exit(&sched_lock, MTX_SPIN); + mtx_exit(&callout_lock, MTX_SPIN); splx(s); } @@ -242,7 +249,7 @@ int s; s = splhigh(); - mtx_enter(&sched_lock, MTX_SPIN); + mtx_enter(&callout_lock, MTX_SPIN); if (c->c_flags & CALLOUT_PENDING) callout_stop(c); @@ -260,7 +267,7 @@ c->c_time = ticks + to_ticks; TAILQ_INSERT_TAIL(&callwheel[c->c_time & callwheelmask], c, c_links.tqe); - mtx_exit(&sched_lock, MTX_SPIN); + mtx_exit(&callout_lock, MTX_SPIN); splx(s); } @@ -271,13 +278,13 @@ int s; s = splhigh(); - mtx_enter(&sched_lock, MTX_SPIN); + mtx_enter(&callout_lock, MTX_SPIN); /* * Don't attempt to delete a callout that's not on the queue. */ if (!(c->c_flags & CALLOUT_PENDING)) { c->c_flags &= ~CALLOUT_ACTIVE; - mtx_exit(&sched_lock, MTX_SPIN); + mtx_exit(&callout_lock, MTX_SPIN); splx(s); return; } @@ -292,7 +299,7 @@ if (c->c_flags & CALLOUT_LOCAL_ALLOC) { SLIST_INSERT_HEAD(&callfree, c, c_links.sle); } - mtx_exit(&sched_lock, MTX_SPIN); + mtx_exit(&callout_lock, MTX_SPIN); splx(s); } @@ -354,7 +361,7 @@ /* don't collide with softclock() */ s = splhigh(); - mtx_enter(&sched_lock, MTX_SPIN); + mtx_enter(&callout_lock, MTX_SPIN); for (p = calltodo.c_next; p != NULL; p = p->c_next) { p->c_time -= delta_ticks; @@ -365,7 +372,7 @@ /* take back the ticks the timer didn't use (p->c_time <= 0) */ delta_ticks = -p->c_time; } - mtx_exit(&sched_lock, MTX_SPIN); + mtx_exit(&callout_lock, MTX_SPIN); splx(s); return; Index: i386/i386/machdep.c =================================================================== RCS file: /home/ncvs/src/sys/i386/i386/machdep.c,v retrieving revision 1.420 diff -u -r1.420 machdep.c --- i386/i386/machdep.c 2000/10/27 11:45:23 1.420 +++ i386/i386/machdep.c 2000/11/18 01:59:51 @@ -409,6 +409,8 @@ TAILQ_INIT(&callwheel[i]); } + mtx_init(&callout_lock, "callout", MTX_SPIN | MTX_COLD); + #if defined(USERCONFIG) userconfig(); cninit(); /* the preferred console may have changed */ --==_Exmh_18918598640-- To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Sat Nov 18 3: 2:32 2000 Delivered-To: freebsd-smp@freebsd.org Received: from freebsd.dk (freebsd.dk [212.242.42.178]) by hub.freebsd.org (Postfix) with ESMTP id 2E2FB37B479 for ; Sat, 18 Nov 2000 03:02:30 -0800 (PST) Received: (from sos@localhost) by freebsd.dk (8.9.3/8.9.1) id MAA14834; Sat, 18 Nov 2000 12:01:24 +0100 (CET) (envelope-from sos) From: Soren Schmidt Message-Id: <200011181101.MAA14834@freebsd.dk> Subject: Re: 10/100 low-end switches [was: Re: via chipset and SMP] In-Reply-To: <20001117193259.B49176@myhakas.matti.ee> from Vallo Kallaste at "Nov 17, 2000 07:32:59 pm" To: vallo@matti.ee Date: Sat, 18 Nov 2000 12:01:24 +0100 (CET) Cc: freebsd@gndrsh.dnsmgr.net (Rodney W. Grimes), keichii@peorth.iteration.net (Michael C . Wu), freebsd-smp@FreeBSD.ORG X-Mailer: ELM [version 2.4ME+ PL54 (25)] MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 8bit Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org It seems Vallo Kallaste wrote: > On Sun, Nov 12, 2000 at 08:08:10AM -0800, "Rodney W. Grimes" wrote: > > > There ``low cost'' 10/100 switches are junk, they make some nicer > > switches using BroadCOM chips, again, probably some of the better > > 10/100 switch chips on the market. > > Can you, as hardware geek, advise us what low-end 10/100 switches > are worth of buy? It took me several weeks to shout down the lowend > D-Link switches we had in sight some time ago. We're now running > some 12-port SMC's, they're cheap.. I just bought a RubyTech SH8008RM 8 port 10/100 switch for ~ 90US$ works like a charm, nicely built, bargain if you ask me... -Søren To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Sat Nov 18 8:31:45 2000 Delivered-To: freebsd-smp@freebsd.org Received: from mailman.zeta.org.au (mailman.zeta.org.au [203.26.10.16]) by hub.freebsd.org (Postfix) with ESMTP id 3ADB637B479; Sat, 18 Nov 2000 08:31:40 -0800 (PST) Received: from bde.zeta.org.au (bde.zeta.org.au [203.2.228.102]) by mailman.zeta.org.au (8.8.7/8.8.7) with ESMTP id DAA16353; Sun, 19 Nov 2000 03:18:50 +1100 Date: Sun, 19 Nov 2000 03:18:48 +1100 (EST) From: Bruce Evans X-Sender: bde@besplex.bde.org To: Jake Burkholder Cc: John Baldwin , smp@FreeBSD.ORG Subject: Re: cvs commit: src/sys/kern kern_timeout.c In-Reply-To: <20001118025912.158D4BA7A@io.yi.org> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org On Fri, 17 Nov 2000, Jake Burkholder wrote: > > > > On 18-Nov-00 John Baldwin wrote: > > > > > > On 18-Nov-00 John Baldwin wrote: > > >> jhb 2000/11/17 16:21:01 PST > > >> > > >> Modified files: > > >> sys/kern kern_timeout.c > > >> Log: > > >> Release sched_lock very briefly to give interrupts a chance to fire if we > > >> are in softclock() for a long time. The old code already did an > > >> splx()/slphigh() pair here, I just missed adding in the equivalent mutex > > >> operations on sched_lock earlier. I fixed this too. > > > Before this I could get processes to hang (the box was still up, but one > > > process was stuck, tying up its pty) though the box was still fine (I could > > > login and do stuff) if I ran a buildworld -j X where X > 1. With this I > > > built > > > a kernel with -j 8 and am currently chugging through a -j 64 world. > > > > Grrrr. Spoke too soon. :( It took longer, but I have one hung process (though The fix should only affect interrupt latency. When sched_lock isn't released, softclock() will exit after checking the buckets for each tick value between the initial value of softticks and the final value of `ticks'. Then interrupts are unlocked and the next call to softclock() will handle any new timeout expiries. Losing the race to check for all timeout expiries before returning is unimportant. > I think the APM_FIXUP_CALLTODO thing has rotted, its based on the > old delta queue timeout implementation. It rotted 3 months before it was committed in -current. I didn't remove it because the bug (?) that it fixes are still there. Bruce To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Sat Nov 18 9:15:32 2000 Delivered-To: freebsd-smp@freebsd.org Received: from prism.flugsvamp.com (cb58709-a.mdsn1.wi.home.com [24.17.241.9]) by hub.freebsd.org (Postfix) with ESMTP id CF58D37B479 for ; Sat, 18 Nov 2000 09:15:27 -0800 (PST) Received: (from jlemon@localhost) by prism.flugsvamp.com (8.11.0/8.11.0) id eAIHF9367073; Sat, 18 Nov 2000 11:15:09 -0600 (CST) (envelope-from jlemon) Date: Sat, 18 Nov 2000 11:15:09 -0600 (CST) From: Jonathan Lemon Message-Id: <200011181715.eAIHF9367073@prism.flugsvamp.com> To: jburkhol@home.com, smp@freebsd.org Subject: Re: cvs commit: src/sys/kern kern_timeout.c X-Newsgroups: local.mail.freebsd-smp In-Reply-To: References: Organization: Cc: Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org In article you write: >@@ -129,18 +132,22 @@ > c->c_flags = > (c->c_flags & ~CALLOUT_PENDING); > } >- mtx_exit(&sched_lock, MTX_SPIN); >+ mtx_exit(&callout_lock, MTX_SPIN); >+ if (!(c->c_flags & CALLOUT_MPSAFE)) >+ mtx_enter(&Giant, MTX_DEF); > splx(s); > c_func(c_arg); > s = splhigh(); >- mtx_enter(&sched_lock, MTX_SPIN); >+ if (!(c->c_flags & CALLOUT_MPSAFE)) >+ mtx_exit(&Giant, MTX_DEF); >+ mtx_enter(&callout_lock, MTX_SPIN); > steps = 0; > c = nextsoftcheck; You'll have to cache c->c_flags (in the same fashion as c_func/c_arg), since the callout may be freed immediately prior to this bit of code. -- Jonathan To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Sat Nov 18 9:34:39 2000 Delivered-To: freebsd-smp@freebsd.org Received: from io.yi.org (h24-69-199-88.gv.shawcable.net [24.69.199.88]) by hub.freebsd.org (Postfix) with ESMTP id B9EC537B479 for ; Sat, 18 Nov 2000 09:34:37 -0800 (PST) Received: from io.yi.org (localhost.gvcl1.bc.wave.home.com [127.0.0.1]) by io.yi.org (Postfix) with ESMTP id D3879BA7A; Sat, 18 Nov 2000 09:34:37 -0800 (PST) X-Mailer: exmh version 2.1.1 10/15/1999 To: Jonathan Lemon Cc: smp@FreeBSD.ORG Subject: Re: cvs commit: src/sys/kern kern_timeout.c In-Reply-To: Message from Jonathan Lemon of "Sat, 18 Nov 2000 11:15:09 CST." <200011181715.eAIHF9367073@prism.flugsvamp.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Sat, 18 Nov 2000 09:34:37 -0800 From: Jake Burkholder Message-Id: <20001118173437.D3879BA7A@io.yi.org> Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org > In article you write: > >@@ -129,18 +132,22 @@ > > c->c_flags = > > (c->c_flags & ~CALLOUT_PENDING); > > } > >- mtx_exit(&sched_lock, MTX_SPIN); > >+ mtx_exit(&callout_lock, MTX_SPIN); > >+ if (!(c->c_flags & CALLOUT_MPSAFE)) > >+ mtx_enter(&Giant, MTX_DEF); > > splx(s); > > c_func(c_arg); > > s = splhigh(); > >- mtx_enter(&sched_lock, MTX_SPIN); > >+ if (!(c->c_flags & CALLOUT_MPSAFE)) > >+ mtx_exit(&Giant, MTX_DEF); > >+ mtx_enter(&callout_lock, MTX_SPIN); > > steps = 0; > > c = nextsoftcheck; > > You'll have to cache c->c_flags (in the same fashion as c_func/c_arg), > since the callout may be freed immediately prior to this bit of code. Right, thanks :) > -- > Jonathan > > > To Unsubscribe: send mail to majordomo@FreeBSD.org > with "unsubscribe freebsd-smp" in the body of the message To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Sat Nov 18 23:50: 6 2000 Delivered-To: freebsd-smp@freebsd.org Received: from io.yi.org (h24-69-199-88.gv.shawcable.net [24.69.199.88]) by hub.freebsd.org (Postfix) with ESMTP id 643A837B479 for ; Sat, 18 Nov 2000 23:50:01 -0800 (PST) Received: from io.yi.org (localhost.gvcl1.bc.wave.home.com [127.0.0.1]) by io.yi.org (Postfix) with ESMTP id 84FA1BA7A for ; Sat, 18 Nov 2000 23:50:00 -0800 (PST) X-Mailer: exmh version 2.1.1 10/15/1999 To: smp@freebsd.org Subject: Re: cvs commit: src/sys/alpha/alpha machdep.c src/sys/i386/i386 machdep.c src/sys/ia64/ia64 machdep.c src/sys/kern kern_clock.c kern_intr.c kern_mutex.c kern_timeout.c src/sys/sys callout.h In-Reply-To: Message from Jake Burkholder of "Sat, 18 Nov 2000 22:02:33 PST." <200011190602.WAA98532@freefall.freebsd.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Sat, 18 Nov 2000 23:50:00 -0800 From: Jake Burkholder Message-Id: <20001119075000.84FA1BA7A@io.yi.org> Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org > jake 2000/11/18 22:02:33 PST > > Modified files: > sys/alpha/alpha machdep.c > sys/i386/i386 machdep.c > sys/ia64/ia64 machdep.c > sys/kern kern_clock.c kern_intr.c kern_mutex.c > kern_timeout.c > sys/sys callout.h > Log: > - Protect the callout wheel with a separate spin mutex, callout_lock. > - Use the mutex in hardclock to ensure no races between it and > softclock. > - Make softclock be INTR_MPSAFE and provide a flag, > CALLOUT_MPSAFE, which specifies that a callout handler does not > need giant. There is still no way to set this flag when > regstering a callout. make -j 64 buildworld on SMP still ends up with a stuck process here. All the makes were in "select" last time, which passes PCATCH to tsleep, so its probably the msleep/CURSIG thing. grr. I didn't move the callout initialization because it would end up wasting some amount of memory if callwheel and callfree were just malloc(9)ed. With maxusers at 64 the callfree pool is about 75K and the callwheel is 32K, so malloc will allocate 128K and 32K respectively. Is this worth worrying about? I'd like opinions on the best way to allow MPSAFE callouts to be registered. There's alot of code that uses the timeout and callout_{reset,stop} interfaces, so I guess it'll be another rename the function and provide a compatibility macro. If possible new code should use the callout_* functions which allows the callout structure to be allocated by the caller. This is a pain: panic("timeout table full"); BSD/OS does some nasty lock gyrations to deal with dynamically allocating callouts in the timeout function, which is not allowed to fail, so I don't know if we want to go there. sched_cpu(), roundrobin() and endtsleep() are some timeout driven functions that could probably be made MPSAFE relatively easily. sched_cpu() is almost safe because of sched_lock, the traversal of allproc is the main reason for still needing Giant. BSD/OS uses a lockmgr lock to protect allproc, which seems like a good idea to me, despite the controversy over reader/writer locks. Many of the traversals are read-only, so a shared/exclusive lock would let them run concurrently. Jake To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message