From owner-freebsd-hackers@FreeBSD.ORG Thu Oct 16 09:58:21 2003 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id E72E516A4B3; Thu, 16 Oct 2003 09:58:20 -0700 (PDT) Received: from xorpc.icir.org (xorpc.icir.org [192.150.187.68]) by mx1.FreeBSD.org (Postfix) with ESMTP id 2C30E43FBD; Thu, 16 Oct 2003 09:58:20 -0700 (PDT) (envelope-from rizzo@xorpc.icir.org) Received: from xorpc.icir.org (localhost [127.0.0.1]) by xorpc.icir.org (8.12.9p1/8.12.3) with ESMTP id h9GGwJsd034611; Thu, 16 Oct 2003 09:58:19 -0700 (PDT) (envelope-from rizzo@xorpc.icir.org) Received: (from rizzo@localhost) by xorpc.icir.org (8.12.9p1/8.12.3/Submit) id h9GGwJIi034548; Thu, 16 Oct 2003 09:58:19 -0700 (PDT) (envelope-from rizzo) Date: Thu, 16 Oct 2003 09:58:19 -0700 From: "'Luigi Rizzo'" To: Michael Marchetti Message-ID: <20031016095819.A31251@xorpc.icir.org> References: Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2.5.1i In-Reply-To: ; from mmarchetti@sandvine.com on Thu, Oct 16, 2003 at 12:05:25PM -0400 cc: "'stable@freebsd.org'" cc: "'hackers@freebsd.org'" Subject: Re: hardclock interrupt deadlock X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 16 Oct 2003 16:58:21 -0000 On Thu, Oct 16, 2003 at 12:05:25PM -0400, Michael Marchetti wrote: > I have enabled DEVICE_POLLING. It does work with SMP (disabled the check). You can remove code and pretend the remaining code works, but that does not mean that it _actually_ does what is expected to do. cheers luigi > -----Original Message----- > From: Luigi Rizzo [mailto:rizzo@icir.org] > Sent: Thursday, October 16, 2003 11:49 AM > To: Michael Marchetti > Cc: 'hackers@freebsd.org'; 'stable@freebsd.org' > Subject: Re: hardclock interrupt deadlock > > > On Thu, Oct 16, 2003 at 11:17:50AM -0400, Michael Marchetti wrote: > > Hi, > > > > We have encountered a problem where the system hangs. We are running a > 4.7 > > SMP kernel using kernel polling on a Dual Xeon with hyperthreading enabled > > puzzled on what you mean by "kernel polling" ... DEVICE_POLLING, > if that is what you mean, cannot work with SMP -- it should not even > build unless you manually disabled the check. > > luigi > > > (essentially a 4 processor system). As a result, the only HW interrupts > in > > the system are hardclock (8254), the rtc, serial console and scsi. The > > synchronous interrupts are (8254 and rtc). When the system is hung, I > have > > found that the ipending and iactive bits for the 8254 and rtc are set > > (meaning the interrupt is pending and active) although giant lock is not > > held and all processors are idle (and halted). This lead me to believe > that > > somehow the ipending bit was set "just before" the last interrupt > returned. > > The only way the system would be able to run that interrupt again is if > > another interrupt would run and it would notice that ipending is set, and > it > > would run (an interrupt delay would be seen). In a non-polling system, I > > imagine the ethernet interrupts would wake it up. I believe I found a > > potential hole where this could happen. > > > > In i386/isa/ipl.s: > > > > #ifdef SMP > > cli /* early to prevent INT deadlock */ > > doreti_next2: > > #endif > > movl %eax,%ecx > > notl %ecx /* set bit = unmasked level */ > > #ifndef SMP > > cli > > #endif > > andl _ipending,%ecx /* set bit = unmasked pending INT */ > > jne doreti_unpend > > movl %eax,_cpl > > > > I'm concerned in the instance the ipending is checked and deemed to be not > > set, but just after another interrupt occurs causing ipending to be set. > > Because CPL is not yet unmasked, that interrupt is not forwarded. In > > Particular, in i386/isa/apic_vector.s: > > > > 3: ; /* other cpu has isr lock */ \ > > APIC_ITRACE(apic_itrace_noisrlock, irq_num, APIC_ITRACE_NOISRLOCK) > > ;\ > > lock ; \ > > orl $IRQ_BIT(irq_num), _ipending ; \ > > testl $IRQ_BIT(irq_num), _cpl ; \ > > jne 4f ; /* this INT masked */ \ > > call forward_irq ; /* forward irq to lock holder */ \ > > POP_FRAME ; /* and return */ \ > > iret ; \ > > ALIGN_TEXT ; \ > > > > The check for _cpl occurs right after the ipending, thus causing a > potential > > race for checking/modifying the cpl. > > > > One quick solution that I thought might correct this would be in ipl.s, > > right after modifying the cpl, recheck the ipending again to see if it > > changed, such as: > > > > > > #ifdef SMP > > cli /* early to prevent INT deadlock */ > > doreti_next2: > > #endif > > movl %eax,%ecx > > notl %ecx /* set bit = unmasked level */ > > #ifndef SMP > > cli > > #endif > > andl _ipending,%ecx /* set bit = unmasked pending INT */ > > jne doreti_unpend > > movl %eax,_cpl > > andl _ipending,%ecx /* set bit = unmasked pending INT */ > > jne doreti_unpend > > > > > > Any opinions/insight? > > > > thanks. > > _______________________________________________ > > freebsd-hackers@freebsd.org mailing list > > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org"