From owner-freebsd-smp Sun Feb 9 7:24:16 2003 Delivered-To: freebsd-smp@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 41D3337B401 for ; Sun, 9 Feb 2003 07:24:15 -0800 (PST) Received: from HAL9000.homeunix.com (12-233-57-224.client.attbi.com [12.233.57.224]) by mx1.FreeBSD.org (Postfix) with ESMTP id D1D1643F3F for ; Sun, 9 Feb 2003 07:24:14 -0800 (PST) (envelope-from dschultz@uclink.Berkeley.EDU) Received: from HAL9000.homeunix.com (localhost [127.0.0.1]) by HAL9000.homeunix.com (8.12.6/8.12.5) with ESMTP id h19FOEoH002373 for ; Sun, 9 Feb 2003 07:24:14 -0800 (PST) (envelope-from dschultz@uclink.Berkeley.EDU) Received: (from das@localhost) by HAL9000.homeunix.com (8.12.6/8.12.5/Submit) id h19FOESY002372 for smp@FreeBSD.ORG; Sun, 9 Feb 2003 07:24:14 -0800 (PST) (envelope-from dschultz@uclink.Berkeley.EDU) Date: Sun, 9 Feb 2003 07:24:14 -0800 From: David Schultz To: smp@FreeBSD.ORG Subject: machine/critical.h and !__GNUC__, interrupt nesting level, etc. Message-ID: <20030209152414.GA1390@HAL9000.homeunix.com> Mail-Followup-To: smp@FreeBSD.ORG Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org It seems that each src/sys/$arch/$arch/critical.h contains the following: #else /* !__GNUC__ */ void cpu_critical_enter(void) void cpu_critical_exit(void) #endif /* __GNUC__ */ Am I losing my mind or will that cause a compile-time error without semicolons at the ends of the lines? There doesn't even seem to be a !__GNUC__ implementation at all, or any irreparable gccisms in the normal version for that matter. Another random question that came up while I was browsing the source entirely too late at night: Why isn't td_intr_nesting_level bounded? Isn't there a danger of overflowing the kernel stack? To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Sun Feb 9 9:32:57 2003 Delivered-To: freebsd-smp@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id F389C37B401; Sun, 9 Feb 2003 09:32:55 -0800 (PST) Received: from eagle.sharma-home.net (cpe-66-1-147-119.ca.sprintbbd.net [66.1.147.119]) by mx1.FreeBSD.org (Postfix) with ESMTP id 1A22743F3F; Sun, 9 Feb 2003 09:32:54 -0800 (PST) (envelope-from adsharma@eagle.sharma-home.net) Received: by eagle.sharma-home.net (Postfix, from userid 500) id 7328084F4; Sun, 9 Feb 2003 09:37:49 -0800 (PST) Date: Sun, 9 Feb 2003 09:37:49 -0800 From: Arun Sharma To: John Baldwin , smp@freebsd.org Subject: Re: kern/18524: The current kernel doesn't keep stats on a per c Message-ID: <20030209173749.GA11072@sharma-home.net> References: <3E420FBA.90504@sharma-home.net> <20030206165049.GA11373@sharma-home.net> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20030206165049.GA11373@sharma-home.net> User-Agent: Mutt/1.4i Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org I didn't get any feedback on this patch. Should I be posting this somewhere else (-arch ?) ? I think this feature is essential to figure out how well the new scheduler is load-balancing across cpus/hyperthreads. -Arun On Thu, Feb 06, 2003 at 08:50:49AM -0800, Arun Sharma wrote: > On Thu, Feb 06, 2003 at 11:13:15AM -0500, John Baldwin wrote: > > > and submit a new patch: > > > > > > http://www.sharma-home.net/~adsharma/misc/pcpu-cptime.patch > > > > [...] > > > > Why not stick the cp_time stuff in struct pcpu instead of using an > > array? > > The new patch _is_ putting cp_time in struct pcpu. The old patch in the PR > predates struct pcpu. > > I also chose to leave the existing cp_time alone. One could argue that a user > level tool could sum up the pcpu cp_times to derive the cp_time and > the kernel can avoid dirtying an extra cache line. If people feel > strongly about it, I can skip touching cp_time in the SMP case. It's a > choice between compatibility with UP vs performance. > > -Arun To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Sun Feb 9 10:49:17 2003 Delivered-To: freebsd-smp@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id C438B37B401 for ; Sun, 9 Feb 2003 10:49:15 -0800 (PST) Received: from mailman.zeta.org.au (mailman.zeta.org.au [203.26.10.16]) by mx1.FreeBSD.org (Postfix) with ESMTP id 8B50443F93 for ; Sun, 9 Feb 2003 10:49:14 -0800 (PST) (envelope-from bde@zeta.org.au) Received: from katana.zip.com.au (katana.zip.com.au [61.8.7.246]) by mailman.zeta.org.au (8.9.3/8.8.7) with ESMTP id FAA09887; Mon, 10 Feb 2003 05:49:09 +1100 Date: Mon, 10 Feb 2003 05:49:12 +1100 (EST) From: Bruce Evans X-X-Sender: bde@gamplex.bde.org To: David Schultz Cc: smp@FreeBSD.ORG Subject: Re: machine/critical.h and !__GNUC__, interrupt nesting level, etc. In-Reply-To: <20030209152414.GA1390@HAL9000.homeunix.com> Message-ID: <20030210045401.I2223-100000@gamplex.bde.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org On Sun, 9 Feb 2003, David Schultz wrote: > It seems that each src/sys/$arch/$arch/critical.h contains the > following: > > #else /* !__GNUC__ */ > > void cpu_critical_enter(void) > void cpu_critical_exit(void) > > #endif /* __GNUC__ */ > > Am I losing my mind or will that cause a compile-time error > without semicolons at the ends of the lines? There doesn't even As you can see, the !__GNUC__ case is rarely tested. > seem to be a !__GNUC__ implementation at all, or any irreparable > gccisms in the normal version for that matter. Rev.1.1 of the i386 version was more complicated and had intr_disable() calls. Even that is not a gccism since intr_disable()'s interface is not gcc-specific; the function is just implemented using gccism in the __GNUC__ case, and the details for that are elsewhere. Disabling all interrupts in these functions is basically a mistake, but may be needed for unpolished implementations or recalcitrant arches. > Another random question that came up while I was browsing the > source entirely too late at night: > > Why isn't td_intr_nesting_level bounded? Isn't there a danger of > overflowing the kernel stack? In RELENG_4, the corresponding variable (intr_nesting_level) is bounded by the number of bits in intrmask_t (since every nested interrupt must set an unset bit in cpl), and more strictly by the number and arrangement of splfoo()'s (since bits in cpl are mostly set more than 1 at a time by splfoo()). That was intended to give a small enough bound but probably didn't for the worst case (especially before the kernel stack size was increased a year or two ago). In -current, interrupts are rarely if ever nested, since most interrupt handling is done in interrupt tasks on separate stacks and switches to the separate stacks are mostly done with interrupts disabled (perhaps only in software). I think interrupt nesting essentially _never_ occurs. There is one special setup where I think it does: if we are running an interrupt handler with interrupts only masked in software, then a hardware interrupt may occur of course, and fast interrupts and maybe some IPIs are allowed to proceed, but there can be no further nesting since interrupts are then masked in hardware. td_intr_nesting_level ends up being almost useless. It is needed to completely classify interrupt context in statclock() (most cases are classified by (td->td_ithd != NULL), but the special setup mentioned above is tricky). It is used to check for malloc(M_WAITOK) in interrupt context, but this is bogus -- normal interrupt context is classify (td->td_ithd != NULL), and just calling malloc() is an error if td_intr_nesting_level is nonzero (this means that the context is fast interrupt handler or context-switching context). Old versions of SMPng fudged p_intr_nesting_level by setting it to 1 in ithread context, so the malloc() check was once less bogus. Bruce To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Sun Feb 9 14: 6:43 2003 Delivered-To: freebsd-smp@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 89C4F37B401; Sun, 9 Feb 2003 14:06:41 -0800 (PST) Received: from eagle.sharma-home.net (cpe-66-1-147-119.ca.sprintbbd.net [66.1.147.119]) by mx1.FreeBSD.org (Postfix) with ESMTP id 721E543F85; Sun, 9 Feb 2003 14:06:40 -0800 (PST) (envelope-from adsharma@sharma-home.net) Received: from astra.mirabella.net (astra.mirabella.net [192.168.1.3]) by eagle.sharma-home.net (Postfix) with ESMTP id B632980D8; Sun, 9 Feb 2003 14:11:42 -0800 (PST) Received: by astra.mirabella.net (Postfix, from userid 1001) id A55C02E; Sun, 9 Feb 2003 14:06:39 -0800 (PST) To: FreeBSD-gnats-submit@freebsd.org Subject: SMP machine hang during boot related to idle proc and sched_lock From: Arun Sharma Reply-To: Arun Sharma Cc: smp@freebsd.org X-send-pr-version: 3.113 X-GNATS-Notify: Message-Id: <20030209220639.A55C02E@astra.mirabella.net> Date: Sun, 9 Feb 2003 14:06:39 -0800 (PST) Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org >Submitter-Id: current-users >Originator: Arun Sharma >Organization: >Confidential: no >Synopsis: SMP machine hang during boot related to idle proc and sched_lock >Severity: serious >Priority: medium >Category: kern >Class: sw-bug >Release: FreeBSD 5.0-CURRENT i386 >Environment: System: FreeBSD astra.mirabella.net 5.0-CURRENT FreeBSD 5.0-CURRENT #16: Sat Feb 8 09:08:58 PST 2003 root@astra.mirabella.net:/usr/src/sys/i386/compile/astra i386 >Description: The machine hangs randomly during bootup on a 2 way SMP box. In some of those hangs, it gets into ddb and I could collect the following info: db> show pcpu cpuid = 0 curthread = 0xc0d19380: pid 46 "sh" curpcb = 0xcad54da0 fpcurthread = none idlethread = 0xc0d18b60: pid 12 "idle: cpu0" currentldt = 0x28 db> tr Debugger(c0364696,0,c036423d,cad54a64,1) at Debugger+0x55 panic(c036423d,c036426b,c0d18a80,0,cad54af8) at panic+0x11f _mtx_lock_spin(c038b6c0,2,0,0,c1fc4dc8) at _mtx_lock_spin+0x93 hardclock_process(cad54af8,0,c02f682b,20,0) at hardclock_process+0x76 hardclock(cad54af8,c0cf239c,c0334d57,c0829000,c1fc8b28) at hardclock+0x18 clkintr(0) at clkintr+0xec Xfastintr0() at Xfastintr0+0xba --- interrupt, eip = 0xc01cc580, esp = 0xcad54b3c, ebp = 0xcad54b58 --- _mtx_lock_spin(c038b6c0,0,0,0,0) at _mtx_lock_spin+0x50 vm_fault(c0d1f114,80f8000,2,8,c0d19380) at vm_fault+0x1379 trap_pfault(cad54d48,1,80f8a78,202,80f8a78) at trap_pfault+0x125 trap(2f,2f,2f,2f,80fc000) at trap+0x2a3 calltrap() at calltrap+0x5 --- trap 0xc, eip = 0x8052653, esp = 0xbfbff304, ebp = 0xbfbff308 --- db> show pcpu 1 cpuid = 1 curthread = 0xc0d18a80: pid 11 "idle: cpu1" curpcb = 0xcad36da0 fpcurthread = none idlethread = 0xc0d18a80: pid 11 "idle: cpu1" currentldt = 0x28 db > show msgbuf [...] panic: spin lock sched lock held by 0xc0d18a80 for > 5 seconds cpuid = 0; lapic.id = 00000000 The only piece not captured above is the stack of the idle process - which was in mi_switch(). Invariants and witness code were not configured-in. >How-To-Repeat: Boot the SMP kernel repeatedly. >Fix: Not clear. Need to figure out why the idle proc (cpu1) was sitting in mi_switch() for more than 5 secs. To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Mon Feb 10 0:20:34 2003 Delivered-To: freebsd-smp@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 1A51737B405 for ; Mon, 10 Feb 2003 00:20:32 -0800 (PST) Received: from HAL9000.homeunix.com (12-233-57-224.client.attbi.com [12.233.57.224]) by mx1.FreeBSD.org (Postfix) with ESMTP id E821043FBF for ; Mon, 10 Feb 2003 00:19:10 -0800 (PST) (envelope-from dschultz@uclink.Berkeley.EDU) Received: from HAL9000.homeunix.com (localhost [127.0.0.1]) by HAL9000.homeunix.com (8.12.6/8.12.5) with ESMTP id h1A8J9oH005115; Mon, 10 Feb 2003 00:19:09 -0800 (PST) (envelope-from dschultz@uclink.Berkeley.EDU) Received: (from das@localhost) by HAL9000.homeunix.com (8.12.6/8.12.5/Submit) id h1A8J80b005114; Mon, 10 Feb 2003 00:19:08 -0800 (PST) (envelope-from dschultz@uclink.Berkeley.EDU) Date: Mon, 10 Feb 2003 00:19:08 -0800 From: David Schultz To: Bruce Evans Cc: smp@FreeBSD.ORG Subject: Re: machine/critical.h and !__GNUC__, interrupt nesting level, etc. Message-ID: <20030210081908.GA4772@HAL9000.homeunix.com> Mail-Followup-To: Bruce Evans , smp@FreeBSD.ORG References: <20030209152414.GA1390@HAL9000.homeunix.com> <20030210045401.I2223-100000@gamplex.bde.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20030210045401.I2223-100000@gamplex.bde.org> Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org Thus spake Bruce Evans : > On Sun, 9 Feb 2003, David Schultz wrote: > > > It seems that each src/sys/$arch/$arch/critical.h contains the > > following: > > > > #else /* !__GNUC__ */ > > > > void cpu_critical_enter(void) > > void cpu_critical_exit(void) > > > > #endif /* __GNUC__ */ > > > > Am I losing my mind or will that cause a compile-time error > > without semicolons at the ends of the lines? There doesn't even > > As you can see, the !__GNUC__ case is rarely tested. > > > seem to be a !__GNUC__ implementation at all, or any irreparable > > gccisms in the normal version for that matter. > > Rev.1.1 of the i386 version was more complicated and had intr_disable() > calls. Even that is not a gccism since intr_disable()'s interface is > not gcc-specific; the function is just implemented using gccism in the > __GNUC__ case, and the details for that are elsewhere. Disabling all > interrupts in these functions is basically a mistake, but may be needed > for unpolished implementations or recalcitrant arches. In the source tree I'm looking at right now (-CURRENT from about three months ago), i386 is the only architecture that doesn't just disable interrupts in critical sections. That's a shame, especially for sparc64 where disabling interrupts is rather expensive. The i386 port also seems to be able to get away with other things for reasons I don't understand; for instance, td_intr_nesting_level is updated without using a lock prefix. I guess that's okay since it's per-CPU state, as long as it's accessed within a critical section. > > Another random question that came up while I was browsing the > > source entirely too late at night: > > > > Why isn't td_intr_nesting_level bounded? Isn't there a danger of > > overflowing the kernel stack? ... > In -current, interrupts are rarely if ever nested, since most interrupt > handling is done in interrupt tasks on separate stacks and switches > to the separate stacks are mostly done with interrupts disabled (perhaps > only in software). I was worried about context stealing, but it looks like that isn't implemented. Thanks for your response. I've picked up a few things to add to the end of my rather long ``ooh, shiny'' list. (You may be relieved to know that context stealing isn't one of them.) To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Mon Feb 10 5:37:33 2003 Delivered-To: freebsd-smp@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 34B4237B401 for ; Mon, 10 Feb 2003 05:37:31 -0800 (PST) Received: from mailman.zeta.org.au (mailman.zeta.org.au [203.26.10.16]) by mx1.FreeBSD.org (Postfix) with ESMTP id 2D1D543F85 for ; Mon, 10 Feb 2003 05:37:30 -0800 (PST) (envelope-from bde@zeta.org.au) Received: from katana.zip.com.au (katana.zip.com.au [61.8.7.246]) by mailman.zeta.org.au (8.9.3/8.8.7) with ESMTP id AAA08386; Tue, 11 Feb 2003 00:37:26 +1100 Date: Tue, 11 Feb 2003 00:37:28 +1100 (EST) From: Bruce Evans X-X-Sender: bde@gamplex.bde.org To: David Schultz Cc: smp@FreeBSD.ORG Subject: Re: machine/critical.h and !__GNUC__, interrupt nesting level, etc. In-Reply-To: <20030210081908.GA4772@HAL9000.homeunix.com> Message-ID: <20030211002023.C1011-100000@gamplex.bde.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org On Mon, 10 Feb 2003, David Schultz wrote: > Thus spake Bruce Evans : > > On Sun, 9 Feb 2003, David Schultz wrote: > > > [cpu_crtical_enter...] > > Rev.1.1 of the i386 version was more complicated and had intr_disable() > > calls. Even that is not a gccism since intr_disable()'s interface is > > not gcc-specific; the function is just implemented using gccism in the > > __GNUC__ case, and the details for that are elsewhere. Disabling all > > interrupts in these functions is basically a mistake, but may be needed > > for unpolished implementations or recalcitrant arches. > > In the source tree I'm looking at right now (-CURRENT from about > three months ago), i386 is the only architecture that doesn't just > disable interrupts in critical sections. That's a shame, > especially for sparc64 where disabling interrupts is rather > expensive. AFAIK (not far) sparc64 has 2 layers of hardware interrupt disabling which gives it some similar benefits to the i386 hardware/software interrupts (disabling interrupts at a high level doesn't actually disable _all_ interrupts). > The i386 port also seems to be able to get away with > other things for reasons I don't understand; for instance, > td_intr_nesting_level is updated without using a lock prefix. > I guess that's okay since it's per-CPU state, as long as it's > accessed within a critical section. Yes, it is really per-cpu and only kept in the thread structure as an optimization. It doesn't even need to be accessed (even for write) only withing critical sections, since it has the property of not being changed significantly by interrupt handlers -- intr handlers may change it up and down, but they must restore it to the value that it had when interrupt context was entered. This it is non-volatile as far as non-interrupt code can tell. Bruce To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Tue Feb 11 23:36:41 2003 Delivered-To: freebsd-smp@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id C2EDE37B401; Tue, 11 Feb 2003 23:36:39 -0800 (PST) Received: from fledge.watson.org (fledge.watson.org [204.156.12.50]) by mx1.FreeBSD.org (Postfix) with ESMTP id F1AE443F93; Tue, 11 Feb 2003 23:36:38 -0800 (PST) (envelope-from arr@watson.org) Received: from fledge.watson.org (localhost [127.0.0.1]) by fledge.watson.org (8.12.6/8.12.5) with ESMTP id h1C7aSP3021320; Wed, 12 Feb 2003 02:36:28 -0500 (EST) (envelope-from arr@watson.org) Received: from localhost (arr@localhost) by fledge.watson.org (8.12.6/8.12.6/Submit) with SMTP id h1C7aSQR021300; Wed, 12 Feb 2003 02:36:28 -0500 (EST) X-Authentication-Warning: fledge.watson.org: arr owned process doing -bs Date: Wed, 12 Feb 2003 02:36:27 -0500 (EST) From: "Andrew R. Reiter" To: freebsd-smp@freebsd.org Cc: arr@watson.org, sam@freebsd.org Subject: accounting fix (or something) Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org hi, I have a patch which, I believe, should help eliminate some (note there are more) issues in the acounting code in terms of the usage of the subsystem lock. The patch can be found at: http://www.watson.org/~arr/kern_acct.somempfix.diff It basically removes holding the acct_mtx over a vnode operation and unlocks the acct_mtx in acct_process() alittle earlier than it was. I welcome comments as Im sure people here can point stuff out :) However, in looking at acctwatch() which is the routine that is called via a timer (callout_ api), to do a similar "hack" that was done in the above listed patch (ie, fix holding acct_mtx over some vnode operations), would cause like 4 or something bus locks ... which sucks IMO since this routine would be being called on a "regular" basis (when accounting is on). I'm starting to think this area of code needs to be reworked to be done cleanly. Anyone have any thoughts on this? I have some ideas kicking around but should actually take the time to put them into an email. Anyway, as I said above, comments are appreciated (annoying ones from certain people will be read but most likely not responded to). Cheers, Andrew -- Andrew R. Reiter arr@watson.org To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Wed Feb 12 5:30:59 2003 Delivered-To: freebsd-smp@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 95B6237B401 for ; Wed, 12 Feb 2003 05:30:58 -0800 (PST) Received: from sdf-eu.org (sdf-eu.org [216.162.208.198]) by mx1.FreeBSD.org (Postfix) with ESMTP id 250E943FAF for ; Wed, 12 Feb 2003 05:30:58 -0800 (PST) (envelope-from nvg@sdf-eu.org) Received: (from nvg@localhost) by sdf-eu.org (8.11.3/8.11.6) id h1CDUsq14763 for freebsd-smp@FreeBSD.org; Wed, 12 Feb 2003 13:30:54 GMT Date: Wed, 12 Feb 2003 14:30:54 +0100 From: Robert Barten To: freebsd-smp@FreeBSD.org Subject: subscribe Message-ID: <20030212143054.B27687@droog.sdf-eu.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2.5i Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Thu Feb 13 22: 2:44 2003 Delivered-To: freebsd-smp@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id A9B5D37B401; Thu, 13 Feb 2003 22:02:42 -0800 (PST) Received: from eagle.sharma-home.net (cpe-66-1-147-119.ca.sprintbbd.net [66.1.147.119]) by mx1.FreeBSD.org (Postfix) with ESMTP id A2D7543F85; Thu, 13 Feb 2003 22:02:39 -0800 (PST) (envelope-from adsharma@eagle.sharma-home.net) Received: by eagle.sharma-home.net (Postfix, from userid 500) id 678728509; Thu, 13 Feb 2003 22:08:08 -0800 (PST) Date: Thu, 13 Feb 2003 22:08:08 -0800 From: Arun Sharma To: smp@freebsd.org Cc: jeff@freebsd.org Subject: SMP vulnerable to panics early on ? Message-ID: <20030214060808.GA15893@sharma-home.net> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.4i Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org It sounds like the time when FreeBSD first comes up and execs "sh" is the most vulnerable time for panics. Here's my latest one. I was using the experimental SCHED_ULE option. If my SMP kernel boots up ok, it normally is pretty solid. But right now, I have a roughly 50% failure rate at boot up. -Arun PS: What does ULE stand for ? Is it just a pun on "schedule" ? Fatal trap 12: page fault while in kernel mode cpuid = 1; lapic.id = 01000000 fault virtual address = 0x14c fault code = supervisor write, page not present instruction pointer = 0x8:0xc01dcee3 stack pointer = 0x10:0xcad36c48 frame pointer = 0x10:0xcad36c54 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, def32 1, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 11 (idle: cpu1) db> tr runq_remove(0,c0d1abb0,1ebe4ca,0,c0d17bd8) at runq_remove+0x33 sched_choose(c038cd80,0,c0d17bd8,cad36cdc,c03378a4) at sched_choose+0xa2 choosethread(c01de441,c0d18a80,c349aab8,c99ef33b,ff803014) at choosethread+0xd sw1b(7,800,87,800,47) at sw1b idle_proc(0,cad36d48,900,9f,754) at idle_proc+0xa4 fork_exit(c01c1370,0,cad36d48) at fork_exit+0xb3 fork_trampoline() at fork_trampoline+0x1a --- trap 0x1, eip = 0, esp = 0xcad36d7c, ebp = 0 --- db> show pcpu cpuid = 1 curthread = 0xc0d18a80: pid 11 "idle: cpu1" curpcb = 0xcad36da0 fpcurthread = none idlethread = 0xc0d18a80: pid 11 "idle: cpu1" currentldt = 0x28 db> show pcpu 0 cpuid = 0 curthread = 0xc0d19620: pid 40 "sh" curpcb = 0xcad5dda0 fpcurthread = none idlethread = 0xc0d18b60: pid 12 "idle: cpu0" currentldt = 0x28 db> tr 40 mi_switch(10018,c0d10010,c0d19620,c0d18a80,19b93) at mi_switch+0x161 stoppcbs(ab0ff000,392b0005,5a30fc0,c0392afc,ff0f773) at stoppcbs To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Fri Feb 14 0:16:58 2003 Delivered-To: freebsd-smp@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 11FE237B401 for ; Fri, 14 Feb 2003 00:16:58 -0800 (PST) Received: from Trade-IT.Ru (schaize.gazinter.net [212.44.64.27]) by mx1.FreeBSD.org (Postfix) with SMTP id 4749C43F75 for ; Fri, 14 Feb 2003 00:16:51 -0800 (PST) (envelope-from oleg@trade-it.ru) From: "Oleg V.Ewsiukov" To: Subject: subscribe Date: Fri, 14 Feb 2003 10:15:31 +0200 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="koi8-r" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2600.0000 Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org _________________________________________ ICQ#: 132296402 More ways to contact me: http://wwp.icq.com/132296402 _________________________________________ To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Fri Feb 14 2:38:13 2003 Delivered-To: freebsd-smp@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 68BC837B401; Fri, 14 Feb 2003 02:38:09 -0800 (PST) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.FreeBSD.org (Postfix) with ESMTP id E17E843FBD; Fri, 14 Feb 2003 02:38:08 -0800 (PST) (envelope-from bright@elvis.mu.org) Received: by elvis.mu.org (Postfix, from userid 1192) id AE437AE147; Fri, 14 Feb 2003 02:38:08 -0800 (PST) Date: Fri, 14 Feb 2003 02:38:08 -0800 From: Alfred Perlstein To: current@freebsd.org Cc: smp@freebsd.org Subject: fix: lock order reversal proc/filedesc. Message-ID: <20030214103808.GE93252@elvis.mu.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.4i Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org Thanks to Paul Saab's work on fixing twe(4) I was able to get a crash dump from my box and figure out why my filedesc locking patch was panic'ing. kevent_close was dereferencing the proc's filedesc pointer during close, that doesn't work so well when you need to do what I had to do. :) The gist of the fix is that the mutex 'fdesc_mutex' is used as a global barrier against losing hold of the proc->p_fd reference. So, basically, if you are not the process that owns a filedesc you must grab the mutex over _all_ usage of it, and you probably want the allproc_lock as well to make sure you don't lose your process as well. :) (see sysctl_kern_file() for an example.) I've also fixed kqueue_close to use the stashed filedesc pointer inside the f_data pointer rather than trying to get at it via p->p_fd. please review/test: Index: kern/kern_descrip.c =================================================================== RCS file: /home/ncvs/src/sys/kern/kern_descrip.c,v retrieving revision 1.185 diff -u -r1.185 kern_descrip.c --- kern/kern_descrip.c 11 Feb 2003 07:20:52 -0000 1.185 +++ kern/kern_descrip.c 14 Feb 2003 10:11:43 -0000 @@ -1366,6 +1366,10 @@ return (newfdp); } +/* A mutex to protect the association between a proc and filedesc. */ +struct mtx fdesc_mtx; +MTX_SYSINIT(fdesc, &fdesc_mtx, "fdesc", MTX_DEF); + /* * Release a filedesc structure. */ @@ -1382,6 +1386,10 @@ if (fdp == NULL) return; + mtx_lock(&fdesc_mtx); + td->td_proc->p_fd = NULL; + mtx_unlock(&fdesc_mtx); + FILEDESC_LOCK(fdp); if (--fdp->fd_refcnt > 0) { FILEDESC_UNLOCK(fdp); @@ -1398,7 +1406,6 @@ if (*fpp) (void) closef(*fpp, td); } - td->td_proc->p_fd = NULL; if (fdp->fd_nfiles > NDFILE) FREE(fdp->fd_ofiles, M_FILEDESC); if (fdp->fd_cdir) @@ -2105,7 +2112,9 @@ xf.xf_pid = p->p_pid; xf.xf_uid = p->p_ucred->cr_uid; PROC_UNLOCK(p); + mtx_lock(&fdesc_mtx); if ((fdp = p->p_fd) == NULL) { + mtx_unlock(&fdesc_mtx); continue; } FILEDESC_LOCK(fdp); @@ -2125,6 +2134,7 @@ break; } FILEDESC_UNLOCK(fdp); + mtx_unlock(&fdesc_mtx); if (error) break; } Index: kern/kern_event.c =================================================================== RCS file: /home/ncvs/src/sys/kern/kern_event.c,v retrieving revision 1.54 diff -u -r1.54 kern_event.c --- kern/kern_event.c 21 Jan 2003 08:55:53 -0000 1.54 +++ kern/kern_event.c 14 Feb 2003 09:59:11 -0000 @@ -839,7 +840,7 @@ kqueue_close(struct file *fp, struct thread *td) { struct kqueue *kq = fp->f_data; - struct filedesc *fdp = td->td_proc->p_fd; + struct filedesc *fdp = kq->kq_fdp; struct knote **knp, *kn, *kn0; int i; Index: kern/kern_exit.c =================================================================== RCS file: /home/ncvs/src/sys/kern/kern_exit.c,v retrieving revision 1.193 diff -u -r1.193 kern_exit.c --- kern/kern_exit.c 8 Feb 2003 02:58:16 -0000 1.193 +++ kern/kern_exit.c 13 Feb 2003 09:15:30 -0000 @@ -263,7 +263,7 @@ * Close open files and release open-file table. * This may block! */ - fdfree(td); /* XXXKSE *//* may not be the one in proc */ + fdfree(td); /* * Remove ourself from our leader's peer list and wake our leader. Index: kern/vfs_mount.c =================================================================== RCS file: /home/ncvs/src/sys/kern/vfs_mount.c,v retrieving revision 1.97 diff -u -r1.97 vfs_mount.c --- kern/vfs_mount.c 21 Jan 2003 08:55:55 -0000 1.97 +++ kern/vfs_mount.c 14 Feb 2003 10:09:16 -0000 @@ -1141,10 +1141,10 @@ return; sx_slock(&allproc_lock); LIST_FOREACH(p, &allproc, p_list) { - PROC_LOCK(p); + mtx_lock(&fdesc_mtx); fdp = p->p_fd; if (fdp == NULL) { - PROC_UNLOCK(p); + mtx_unlock(&fdesc_mtx); continue; } nrele = 0; @@ -1160,7 +1160,7 @@ nrele++; } FILEDESC_UNLOCK(fdp); - PROC_UNLOCK(p); + mtx_unlock(&fdesc_mtx); while (nrele--) vrele(olddp); } Index: sys/filedesc.h =================================================================== RCS file: /home/ncvs/src/sys/sys/filedesc.h,v retrieving revision 1.49 diff -u -r1.49 filedesc.h --- sys/filedesc.h 1 Jan 2003 01:56:19 -0000 1.49 +++ sys/filedesc.h 14 Feb 2003 10:12:59 -0000 @@ -139,6 +139,8 @@ return ((u_int)fd >= (u_int)fdp->fd_nfiles ? NULL : fdp->fd_ofiles[fd]); } +extern struct mtx fdesc_mtx; + #endif /* _KERNEL */ #endif /* !_SYS_FILEDESC_H_ */ To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Fri Feb 14 2:52: 3 2003 Delivered-To: freebsd-smp@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id EFF8E37B401; Fri, 14 Feb 2003 02:52:02 -0800 (PST) Received: from flood.ping.uio.no (flood.ping.uio.no [129.240.78.31]) by mx1.FreeBSD.org (Postfix) with ESMTP id EE65143FA3; Fri, 14 Feb 2003 02:52:01 -0800 (PST) (envelope-from des@ofug.org) Received: by flood.ping.uio.no (Postfix, from userid 2602) id 7934E536E; Fri, 14 Feb 2003 11:51:59 +0100 (CET) X-URL: http://www.ofug.org/~des/ X-Disclaimer: The views expressed in this message do not necessarily coincide with those of any organisation or company with which I am or have been affiliated. To: Alfred Perlstein Cc: current@freebsd.org, smp@freebsd.org Subject: Re: fix: lock order reversal proc/filedesc. From: Dag-Erling Smorgrav Date: Fri, 14 Feb 2003 11:51:58 +0100 In-Reply-To: <20030214103808.GE93252@elvis.mu.org> (Alfred Perlstein's message of "Fri, 14 Feb 2003 02:38:08 -0800") Message-ID: User-Agent: Gnus/5.090014 (Oort Gnus v0.14) Emacs/21.2 (i386--freebsd) References: <20030214103808.GE93252@elvis.mu.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org Alfred Perlstein writes: > Thanks to Paul Saab's work on fixing twe(4) I was able to get a > crash dump from my box How? I can't get a crash dump in -CURRENT, even on a plain jane ata disk, and it's been months since I last managed to get one. DES -- Dag-Erling Smorgrav - des@ofug.org To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Fri Feb 14 20: 7:54 2003 Delivered-To: freebsd-smp@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id ABE8A37B401 for ; Fri, 14 Feb 2003 20:07:52 -0800 (PST) Received: from dns1.vizion2000.net (dns1.vizion2000.net [64.58.171.82]) by mx1.FreeBSD.org (Postfix) with ESMTP id D9DCB43F93 for ; Fri, 14 Feb 2003 20:07:51 -0800 (PST) (envelope-from vizion@ixpres.com) Received: from vizion (vizion.vizion2000.net [64.58.171.92]) by dns1.vizion2000.net (8.11.6/8.11.6) with SMTP id h1F4X5S17169 for ; Fri, 14 Feb 2003 20:33:09 -0800 (PST) (envelope-from vizion@ixpres.com) Message-ID: <017501c2d4a5$dc73e3b0$15b55042@vizion2000.net> From: "vizion communication" To: Subject: SMP on Proliant 5500 Date: Fri, 14 Feb 2003 19:53:51 -0800 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2800.1106 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2800.1106 Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org Hi Here is the system: Compaq Proliant 5500 Quad Xeon 500Mhz Booting from Compaq Smart Raid configured to give 3 virtual drives JBOD on seperate adaptec SCSI 2 PCI card Fibre Channel Array 1.2T Adaptec AHA 6944A/TX 4 port PCI 10/100 NVidia PCI TNT2 M64 32M Video card - working fine after a struggle - configured as vesa vesa! Built in ATI Rage IIc (DISABLED) on standard peripheral interface FreeBSD 4.7 My first attempt at configuring this system for SMP was a total failure!! I lost my original configuration as I was unable to re-start successfully with kernel.old. I have successfully rebuilt the configuration as single processor and am ready to have another go to build as SMP. Before doing so some handholding advice would be very welcome. I have not been able to find guidance for building an SMP kernel but I have no doubt not been looking in the right place!! I have looked at the LINT file & commented out: cpu I386_CPU cpu I486_CPU and then added the lines options SMP options APIC_IO is there anything else I need to do prior to making the kernel? Also if there is someone with a similar system who could send me a copy of a successful kernel conf file I would be most grateful Thanks David To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Fri Feb 14 20:40:40 2003 Delivered-To: freebsd-smp@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id AAB9337B401 for ; Fri, 14 Feb 2003 20:40:32 -0800 (PST) Received: from dilbert.robbins.dropbear.id.au (005.a.008.mel.iprimus.net.au [210.50.86.5]) by mx1.FreeBSD.org (Postfix) with ESMTP id 8BC4F43FBD for ; Fri, 14 Feb 2003 20:40:27 -0800 (PST) (envelope-from tim@robbins.dropbear.id.au) Received: from dilbert.robbins.dropbear.id.au (5l6s42hmbfvpkzl2@localhost [127.0.0.1]) by dilbert.robbins.dropbear.id.au (8.12.6/8.12.6) with ESMTP id h1F4eO0n053644 for ; Sat, 15 Feb 2003 15:40:24 +1100 (EST) (envelope-from tim@dilbert.robbins.dropbear.id.au) Received: (from tim@localhost) by dilbert.robbins.dropbear.id.au (8.12.6/8.12.6/Submit) id h1F4eO6n053643 for freebsd-smp@freebsd.org; Sat, 15 Feb 2003 15:40:24 +1100 (EST) (envelope-from tim) Date: Sat, 15 Feb 2003 15:40:24 +1100 From: Tim Robbins To: freebsd-smp@freebsd.org Subject: Locking down itimers Message-ID: <20030215154024.A53601@dilbert.robbins.dropbear.id.au> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2.5.1i Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org This patch protects p_realtimer with the proc mutex instead of Giant, and obtains sched_lock around accesses to p_stats->p_timer[] to avoid a potential race with hardclock. It makes getitimer(), setitimer() and the realitexpire() callout MP-safe. It was necessary to restructure setitimer() a bit to guarantee that the swap: ovalue = p_realtimer, p_realtimer = value; occurs atomically with respect to other threads in the same process calling setitimer() and the callout. Does the patch seem reasonable? Any comments? Tim Index: sys/proc.h =================================================================== RCS file: /x/freebsd/src/sys/sys/proc.h,v retrieving revision 1.293 diff -U5 -p -r1.293 proc.h --- sys/proc.h 9 Feb 2003 08:49:02 -0000 1.293 +++ sys/proc.h 12 Feb 2003 07:44:15 -0000 @@ -565,11 +565,11 @@ struct proc { /* The following fields are all zeroed upon creation in fork. */ #define p_startzero p_oppid pid_t p_oppid; /* (c + e) Save ppid in ptrace. XXX */ struct vmspace *p_vmspace; /* (b) Address space. */ u_int p_swtime; /* (j) Time swapped in or out. */ - struct itimerval p_realtimer; /* (h?/k?) Alarm timer. */ + struct itimerval p_realtimer; /* (c) Alarm timer. */ struct bintime p_runtime; /* (j) Real time. */ int p_profthreads; /* (c) Num threads in addupc_task */ int p_traceflag; /* (o) Kernel trace points. */ struct vnode *p_tracep; /* (c + o) Trace to vnode. */ sigset_t p_siglist; /* (c) Sigs arrived, not delivered. */ Index: kern/kern_time.c =================================================================== RCS file: /x/freebsd/src/sys/kern/kern_time.c,v retrieving revision 1.96 diff -U5 -p -r1.96 kern_time.c --- kern/kern_time.c 3 Feb 2003 19:49:34 -0000 1.96 +++ kern/kern_time.c 12 Feb 2003 14:07:00 -0000 @@ -406,45 +406,42 @@ struct getitimer_args { }; #endif /* * MPSAFE */ -/* ARGSUSED */ int getitimer(struct thread *td, struct getitimer_args *uap) { struct proc *p = td->td_proc; struct timeval ctv; struct itimerval aitv; - int s; if (uap->which > ITIMER_PROF) return (EINVAL); - mtx_lock(&Giant); - - s = splclock(); /* XXX still needed ? */ if (uap->which == ITIMER_REAL) { /* * Convert from absolute to relative time in .it_value * part of real time timer. If time for real time timer * has passed return 0, else return difference between * current time and time for the timer to go off. */ + PROC_LOCK(p); aitv = p->p_realtimer; + PROC_UNLOCK(p); if (timevalisset(&aitv.it_value)) { getmicrouptime(&ctv); if (timevalcmp(&aitv.it_value, &ctv, <)) timevalclear(&aitv.it_value); else timevalsub(&aitv.it_value, &ctv); } } else { + mtx_lock_spin(&sched_lock); aitv = p->p_stats->p_timer[uap->which]; + mtx_unlock_spin(&sched_lock); } - splx(s); - mtx_unlock(&Giant); return (copyout(&aitv, uap->itv, sizeof (struct itimerval))); } #ifndef _SYS_SYSPROTO_H_ struct setitimer_args { @@ -453,63 +450,61 @@ struct setitimer_args { }; #endif /* * MPSAFE */ -/* ARGSUSED */ int setitimer(struct thread *td, struct setitimer_args *uap) { struct proc *p = td->td_proc; - struct itimerval aitv; + struct itimerval aitv, oitv; struct timeval ctv; - struct itimerval *itvp; - int s, error = 0; + int error; + + if (uap->itv == NULL) { + uap->itv = uap->oitv; + return (getitimer(td, (struct getitimer_args *)uap)); + } if (uap->which > ITIMER_PROF) return (EINVAL); - itvp = uap->itv; - if (itvp && (error = copyin(itvp, &aitv, sizeof(struct itimerval)))) + if ((error = copyin(uap->itv, &aitv, sizeof(struct itimerval)))) return (error); - - mtx_lock(&Giant); - - if ((uap->itv = uap->oitv) && - (error = getitimer(td, (struct getitimer_args *)uap))) { - goto done2; - } - if (itvp == 0) { - error = 0; - goto done2; - } - if (itimerfix(&aitv.it_value)) { - error = EINVAL; - goto done2; - } - if (!timevalisset(&aitv.it_value)) { + if (itimerfix(&aitv.it_value)) + return (EINVAL); + if (!timevalisset(&aitv.it_value)) timevalclear(&aitv.it_interval); - } else if (itimerfix(&aitv.it_interval)) { - error = EINVAL; - goto done2; - } - s = splclock(); /* XXX: still needed ? */ + else if (itimerfix(&aitv.it_interval)) + return (EINVAL); + if (uap->which == ITIMER_REAL) { + PROC_LOCK(p); if (timevalisset(&p->p_realtimer.it_value)) callout_stop(&p->p_itcallout); if (timevalisset(&aitv.it_value)) callout_reset(&p->p_itcallout, tvtohz(&aitv.it_value), realitexpire, p); getmicrouptime(&ctv); timevaladd(&aitv.it_value, &ctv); + oitv = p->p_realtimer; p->p_realtimer = aitv; + PROC_UNLOCK(p); + if (timevalisset(&oitv.it_value)) { + if (timevalcmp(&oitv.it_value, &ctv, <)) + timevalclear(&oitv.it_value); + else + timevalsub(&oitv.it_value, &ctv); + } } else { + mtx_lock_spin(&sched_lock); + oitv = p->p_stats->p_timer[uap->which]; p->p_stats->p_timer[uap->which] = aitv; + mtx_unlock_spin(&sched_lock); } - splx(s); -done2: - mtx_unlock(&Giant); - return (error); + if (uap->oitv == NULL) + return (0); + return (copyout(&oitv, uap->oitv, sizeof(struct itimerval))); } /* * Real interval timer expired: * send process whose timer expired an alarm signal. @@ -525,35 +520,31 @@ done2: void realitexpire(void *arg) { struct proc *p; struct timeval ctv, ntv; - int s; p = (struct proc *)arg; PROC_LOCK(p); psignal(p, SIGALRM); if (!timevalisset(&p->p_realtimer.it_interval)) { timevalclear(&p->p_realtimer.it_value); PROC_UNLOCK(p); return; } for (;;) { - s = splclock(); /* XXX: still neeeded ? */ timevaladd(&p->p_realtimer.it_value, &p->p_realtimer.it_interval); getmicrouptime(&ctv); if (timevalcmp(&p->p_realtimer.it_value, &ctv, >)) { ntv = p->p_realtimer.it_value; timevalsub(&ntv, &ctv); callout_reset(&p->p_itcallout, tvtohz(&ntv) - 1, realitexpire, p); - splx(s); PROC_UNLOCK(p); return; } - splx(s); } /*NOTREACHED*/ } /* Index: kern/init_main.c =================================================================== RCS file: /x/freebsd/src/sys/kern/init_main.c,v retrieving revision 1.224 diff -U5 -p -r1.224 init_main.c --- kern/init_main.c 4 Feb 2003 18:47:17 -0000 1.224 +++ kern/init_main.c 12 Feb 2003 08:18:51 -0000 @@ -384,11 +384,11 @@ proc0_init(void *dummy __unused) p->p_leader = p; bcopy("swapper", p->p_comm, sizeof ("swapper")); - callout_init(&p->p_itcallout, 0); + callout_init(&p->p_itcallout, 1); callout_init(&td->td_slpcallout, 1); /* Create credentials. */ p->p_ucred = crget(); p->p_ucred->cr_ngroups = 1; /* group 0 */ Index: compat/linux/linux_misc.c =================================================================== RCS file: /x/freebsd/src/sys/compat/linux/linux_misc.c,v retrieving revision 1.134 diff -U5 -p -r1.134 linux_misc.c --- compat/linux/linux_misc.c 17 Oct 2002 22:00:30 -0000 1.134 +++ compat/linux/linux_misc.c 13 Feb 2003 05:06:13 -0000 @@ -162,11 +162,10 @@ linux_sysinfo(struct thread *td, struct int linux_alarm(struct thread *td, struct linux_alarm_args *args) { struct itimerval it, old_it; struct timeval tv; - int s; #ifdef DEBUG if (ldebug(alarm)) printf(ARGS(alarm, "%u"), args->secs); #endif @@ -176,22 +175,22 @@ linux_alarm(struct thread *td, struct li it.it_value.tv_sec = (long)args->secs; it.it_value.tv_usec = 0; it.it_interval.tv_sec = 0; it.it_interval.tv_usec = 0; - s = splsoftclock(); + PROC_LOCK(td->td_proc); old_it = td->td_proc->p_realtimer; getmicrouptime(&tv); if (timevalisset(&old_it.it_value)) callout_stop(&td->td_proc->p_itcallout); if (it.it_value.tv_sec != 0) { callout_reset(&td->td_proc->p_itcallout, tvtohz(&it.it_value), realitexpire, td->td_proc); timevaladd(&it.it_value, &tv); } td->td_proc->p_realtimer = it; - splx(s); + PROC_UNLOCK(td->td_proc); if (timevalcmp(&old_it.it_value, &tv, >)) { timevalsub(&old_it.it_value, &tv); if (old_it.it_value.tv_usec != 0) old_it.it_value.tv_sec++; td->td_retval[0] = old_it.it_value.tv_sec; To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Fri Feb 14 22: 8:44 2003 Delivered-To: freebsd-smp@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id A0DC637B401; Fri, 14 Feb 2003 22:08:41 -0800 (PST) Received: from wantadilla.lemis.com (wantadilla.lemis.com [192.109.197.80]) by mx1.FreeBSD.org (Postfix) with ESMTP id D010C43FAF; Fri, 14 Feb 2003 22:08:39 -0800 (PST) (envelope-from grog@lemis.com) Received: by wantadilla.lemis.com (Postfix, from userid 1004) id 81664519CA; Sat, 15 Feb 2003 16:38:37 +1030 (CST) Date: Sat, 15 Feb 2003 16:38:37 +1030 From: Greg 'groggy' Lehey To: vizion communication Cc: FreeBSD Questions Subject: Re: SMP on Proliant 5500 Message-ID: <20030215060837.GI83043@wantadilla.lemis.com> References: <017501c2d4a5$dc73e3b0$15b55042@vizion2000.net> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="nEsDIrWrg+hrB7l1" Content-Disposition: inline In-Reply-To: <017501c2d4a5$dc73e3b0$15b55042@vizion2000.net> User-Agent: Mutt/1.4i Organization: The FreeBSD Project Phone: +61-8-8388-8286 Fax: +61-8-8388-8725 Mobile: +61-418-838-708 WWW-Home-Page: http://www.FreeBSD.org/ X-PGP-Fingerprint: 9A1B 8202 BCCE B846 F92F 09AC 22E6 F290 507A 4223 Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org --nEsDIrWrg+hrB7l1 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline On Friday, 14 February 2003 at 19:53:51 -0800, vizion communication wrote: > Hi > > Here is the system: > > Compaq Proliant 5500 > Quad Xeon 500Mhz > Booting from Compaq Smart Raid configured to give 3 virtual > drives > JBOD on seperate adaptec SCSI 2 PCI card > Fibre Channel Array 1.2T > Adaptec AHA 6944A/TX 4 port PCI 10/100 > NVidia PCI TNT2 M64 32M Video card - working fine after a > struggle - configured as vesa vesa! > Built in ATI Rage IIc (DISABLED) on standard peripheral > interface > FreeBSD 4.7 > > My first attempt at configuring this system for SMP was a > total failure!! I lost my original configuration > as I was unable to re-start successfully with kernel.old. I > have successfully rebuilt the configuration as single > processor and am ready to have another go to build as SMP. > Before doing so some handholding advice would be very > welcome. I have not been able to find guidance for building > an SMP kernel but I have no doubt not been looking in the > right place!! I have looked at the LINT file & commented > out: > > cpu I386_CPU > cpu I486_CPU > > and then added the lines > options SMP > options APIC_IO > > is there anything else I need to do prior to making the > kernel? That should be enough, but you should start from the GENERIC config file, not the LINT config file. As it says at the top of LINT: # NB: You probably don't want to try running a kernel built from this # file. Instead, you should start from GENERIC, and add options from # this file as required. > Also if there is someone with a similar system who could send me a > copy of a successful kernel conf file I would be most grateful You've pretty much defined it. Greg -- See complete headers for address and phone numbers --nEsDIrWrg+hrB7l1 Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.0 (FreeBSD) iD8DBQE+TdllIubykFB6QiMRAjjzAJ9CtRDP3Bw1Iaja1U4va26vxmNbxgCgn3Dr M4n8DWObTkTyWD4niTjdhXU= =hXjw -----END PGP SIGNATURE----- --nEsDIrWrg+hrB7l1-- To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Fri Feb 14 22:56:41 2003 Delivered-To: freebsd-smp@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 52A4237B401 for ; Fri, 14 Feb 2003 22:56:39 -0800 (PST) Received: from eagle.sharma-home.net (cpe-66-1-147-119.ca.sprintbbd.net [66.1.147.119]) by mx1.FreeBSD.org (Postfix) with ESMTP id 82F3843FBD for ; Fri, 14 Feb 2003 22:56:35 -0800 (PST) (envelope-from adsharma@eagle.sharma-home.net) Received: by eagle.sharma-home.net (Postfix, from userid 500) id AA7198509; Fri, 14 Feb 2003 23:02:11 -0800 (PST) Date: Fri, 14 Feb 2003 23:02:11 -0800 From: Arun Sharma To: smp@freebsd.org Subject: Re: SMP vulnerable to panics early on ? Message-ID: <20030215070211.GA24479@sharma-home.net> References: <20030214060808.GA15893@sharma-home.net> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20030214060808.GA15893@sharma-home.net> User-Agent: Mutt/1.4i Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org On Thu, Feb 13, 2003 at 10:08:08PM -0800, Arun Sharma wrote: > It sounds like the time when FreeBSD first comes up and execs "sh" is the > most vulnerable time for panics. Here's my latest one. I was using the > experimental SCHED_ULE option. > > If my SMP kernel boots up ok, it normally is pretty solid. But right now, I > have a roughly 50% failure rate at boot up. Here's another one. -Arun panic: lockmgr: locking against myself cpuid = 1; lapic.id = 01000000 db> tr Debugger(c0364696,1000000,c03633a2,cad35f14,1) at Debugger+0x55 panic(c03633a2,0,0,0,b) at panic+0x11f lockmgr(c03c033c,2,0,c0d18a80,b8) at lockmgr+0x531 _vm_map_lock_read(c03c0300,0,0,200024c,0) at _vm_map_lock_read+0x5a vm_map_lookup(cad36074,0,2,cad36078,cad36068) at vm_map_lookup+0x38 vm_fault(1,0,2,8,c03c033c) at vm_fault+0xc4 _mtx_lock_spin(c038b6c0,0,0,0,0) at _mtx_lock_spin+0x6e msleep(c03c033c,c03d0c38,44,c0371b53,0) at msleep+0xf4 acquire(cad361bc,1000000,600,cad361a0,c0d17b10) at acquire+0xa0 lockmgr(c03c033c,2,0,c0d18a80,61) at lockmgr+0x3e4 _vm_map_lock_read(c03c0300,0,0,1000101,cad764ec) at _vm_map_lock_read+0x5a vm_map_lookup(cad362e4,cad764ec,1,cad362e8,cad362d8) at vm_map_lookup+0x38 vm_fault(c03c0300,cad764ec,0,0,0) at vm_fault+0xc4 _end() at 0xcad764ec db> show pcpu cpuid = 1 curthread = 0xc0d18a80: pid 11 "idle: cpu1" curpcb = 0xcad36da0 fpcurthread = none idlethread = 0xc0d18a80: pid 11 "idle: cpu1" currentldt = 0x28 db> show pcpu 0 cpuid = 0 curthread = 0xc0d18a80: pid 11 "idle: cpu1" curpcb = 0xcad36da0 fpcurthread = none idlethread = 0xc0d18b60: pid 12 "idle: cpu0" currentldt = 0x28 To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Sat Feb 15 2:37:49 2003 Delivered-To: freebsd-smp@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id B0BA137B401 for ; Sat, 15 Feb 2003 02:37:48 -0800 (PST) Received: from as9-2-1.bi.s.bonet.se (as9-2-1.bi.s.bonet.se [217.215.7.177]) by mx1.FreeBSD.org (Postfix) with ESMTP id 1189643F3F for ; Sat, 15 Feb 2003 02:37:48 -0800 (PST) (envelope-from peo@intersonic.se) Received: by as9-2-1.bi.s.bonet.se (Postfix, from userid 1003) id 75CA349697A; Sat, 15 Feb 2003 11:37:46 +0100 (CET) Message-ID: <3E4E1873.5060306@intersonic.se> Date: Sat, 15 Feb 2003 11:37:39 +0100 From: Per olof Ljungmark Organization: Intersonic AB User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.2.1) Gecko/20030126 X-Accept-Language: en-us, en, sv MIME-Version: 1.0 To: vizion communication Cc: freebsd-smp@freebsd.org Subject: Re: SMP on Proliant 5500 References: <017501c2d4a5$dc73e3b0$15b55042@vizion2000.net> In-Reply-To: <017501c2d4a5$dc73e3b0$15b55042@vizion2000.net> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Status: No, hits=-102.0 required=5.7 tests=IN_REP_TO,NOSPAM_INC,REFERENCES,SPAM_PHRASE_00_01, USER_AGENT,USER_AGENT_MOZILLA_UA,USER_IN_WHITELIST, X_ACCEPT_LANG version=2.44 X-Spam-Level: X-Sanitizer: Advosys mail filter Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org vizion communication wrote: > Hi > > Here is the system: > > Compaq Proliant 5500 > Quad Xeon 500Mhz > Booting from Compaq Smart Raid configured to give 3 virtual > drives > JBOD on seperate adaptec SCSI 2 PCI card > Fibre Channel Array 1.2T > Adaptec AHA 6944A/TX 4 port PCI 10/100 > NVidia PCI TNT2 M64 32M Video card - working fine after a > struggle - configured as vesa vesa! > Built in ATI Rage IIc (DISABLED) on standard peripheral > interface > FreeBSD 4.7 > Just my $0.02: Did you a) update the BIOS to latest and b) check the OS settings in setup? My experience from cpq servers told me you got do do this 1st. /per olof To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Sat Feb 15 15: 7:34 2003 Delivered-To: freebsd-smp@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 9AF1B37B401; Sat, 15 Feb 2003 15:07:33 -0800 (PST) Received: from k6.locore.ca (k6.locore.ca [198.96.117.170]) by mx1.FreeBSD.org (Postfix) with ESMTP id BDB7F43F75; Sat, 15 Feb 2003 15:07:32 -0800 (PST) (envelope-from jake@k6.locore.ca) Received: from k6.locore.ca (jake@localhost.locore.ca [127.0.0.1]) by k6.locore.ca (8.12.6/8.12.6) with ESMTP id h1FNAfNk074596; Sat, 15 Feb 2003 18:10:41 -0500 (EST) (envelope-from jake@k6.locore.ca) Received: (from jake@localhost) by k6.locore.ca (8.12.6/8.12.6/Submit) id h1FNAfBH074595; Sat, 15 Feb 2003 18:10:41 -0500 (EST) Date: Sat, 15 Feb 2003 18:10:41 -0500 From: Jake Burkholder To: Tim Robbins Cc: freebsd-smp@FreeBSD.ORG Subject: Re: Locking down itimers Message-ID: <20030215181041.K63597@locore.ca> References: <20030215154024.A53601@dilbert.robbins.dropbear.id.au> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2.5i In-Reply-To: <20030215154024.A53601@dilbert.robbins.dropbear.id.au>; from tjr@FreeBSD.ORG on Sat, Feb 15, 2003 at 03:40:24PM +1100 Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org Apparently, On Sat, Feb 15, 2003 at 03:40:24PM +1100, Tim Robbins said words to the effect of; > This patch protects p_realtimer with the proc mutex instead of Giant, > and obtains sched_lock around accesses to p_stats->p_timer[] to avoid > a potential race with hardclock. It makes getitimer(), setitimer() > and the realitexpire() callout MP-safe. > > It was necessary to restructure setitimer() a bit to guarantee that > the swap: > ovalue = p_realtimer, p_realtimer = value; > occurs atomically with respect to other threads in the same process > calling setitimer() and the callout. > > Does the patch seem reasonable? Any comments? Looks reasonable to me. Jake To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message