From owner-freebsd-current@FreeBSD.ORG Tue Sep 9 05:56:48 2003 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 36A6716A4BF; Tue, 9 Sep 2003 05:56:48 -0700 (PDT) Received: from mailman.zeta.org.au (mailman.zeta.org.au [203.26.10.16]) by mx1.FreeBSD.org (Postfix) with ESMTP id B5C3843FEC; Tue, 9 Sep 2003 05:56:43 -0700 (PDT) (envelope-from bde@zeta.org.au) Received: from gamplex.bde.org (katana.zip.com.au [61.8.7.246]) by mailman.zeta.org.au (8.9.3p2/8.8.7) with ESMTP id WAA29966; Tue, 9 Sep 2003 22:56:33 +1000 Date: Tue, 9 Sep 2003 22:56:32 +1000 (EST) From: Bruce Evans X-X-Sender: bde@gamplex.bde.org To: current@freebsd.org Message-ID: <20030909211212.L600@gamplex.bde.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Subject: ATAng interrupt storm and bogus interrupt stormproofing X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 09 Sep 2003 12:56:48 -0000 ATAng generates an interrupt storm deterministically on 1 system here. This should lock up the system, but bugs in interrupt handler scheduling prevent the lockup except in my version of -current where the bugs are different. ATAng interrupt storm: The ATA_ATAPI_IDENTIFY command generates a second "spurious" interrupt. This interrupt is not properly handled. It is completely ignored, at least on the system with the problem. This system has only 1 atapi slave on the channel that generates the interrupt, and no interrupt sharing for this interrupt. Ignoring the interrupt gives an interrupt storm since the interrupt is level sensitive. The interrupt really is generated by the atapi slave, since reading the non-alternate status register of that device clears it. I use this read as a quick fix to limit the storm to 10 interrupts. This may be the correct non-quick fix too. Bugs in interrupt handler scheduling: Ignoring level-sensitive interrupts should result in their thread being switched to endlessly (except for switches to higher priority threads and possibly for switches to equal priority threads). But there is some bug in interrupt handler scheduling that results in low priority threads running while the high priority interrupt storm thread is runnable. This allows the thread that issued the ATA_ATAPI_IDENTIFY command to run eventually, and it or another thread clears the storm as a side effect of issuing another command. I suspect that the scheduling bug is related to the old one of just setting the TDF_NEEDRESCHED flag when a switch can't be done immediately. TDF_NEEDRESCHED has no effect on kernel threads since it is only examined on return to user mode. % int % ithread_schedule(struct ithd *ithread, int do_switch) % { % ... % ithread->it_need = 1; % mtx_lock_spin(&sched_lock); % if (TD_AWAITING_INTR(td)) { % CTR2(KTR_INTR, "%s: setrunqueue %d", __func__, p->p_pid); % TD_CLR_IWAIT(td); % setrunqueue(td); % if (do_switch && % (ctd->td_critnest == 1) ) { ^^^^^^^^^^^^^^^^^^^^^^^ Sometimes the interrupt will occur when ctd (curthread) is in a critical region. Then ctd->td_critnest >= 2 here and we don't switch. % KASSERT((TD_IS_RUNNING(ctd)), % ("ithread_schedule: Bad state for curthread.")); % ctd->td_proc->p_stats->p_ru.ru_nivcsw++; % if (ctd->td_flags & TDF_IDLETD) % ctd->td_state = TDS_CAN_RUN; /* XXXKSE */ % mi_switch(); % } else { % curthread->td_flags |= TDF_NEEDRESCHED; ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ When we don't switch, we set this, but the setting has no effect on kernel threads, so we will only switch to them as a side effect of handling another interrupt. This explains why we may keep running a lower priority thread but now why we switch to a lower priority thread on switching back from handling other interrupts. Annotated debugging session: % total=0x437cc8 entry point=0x129750 % WARNING: loader(8) metadata is missing! % [ preserving 815752 bytes of kernel symbol table ] % Debugger("Boot flags requested debugger") % Stopped at Debugger+0x45: xchgl %ebx,in_Debugger.0 % db> b Xintr11 % db> c Set a breakpoit after booting with -dh. I know from previous debugging sessions that there will be no problems until soon after Xintr11 is hit. % Copyright (c) 1992-2003 The FreeBSD Project. % Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 % The Regents of the University of California. All rights reserved. % FreeBSD 5.1-CURRENT #2863: Tue Sep 9 20:32:22 EST 2003 % bde@besplex.bde.org:/c/sysc/i386/compile/BESPLEX % Timecounter "i8254" frequency 1193182 Hz quality 0 % CPU: Pentium II/Pentium II Xeon/Celeron (400.91-MHz 686-class CPU) % Origin = "GenuineIntel" Id = 0x665 Stepping = 5 % Features=0x183fbff % real memory = 268435456 (256 MB) % avail memory = 256163840 (244 MB) % Pentium Pro MTRR support enabled % npx0: [FAST] % npx0: on motherboard % npx0: flags 0x80 npx0: INT 16 interface % pcibios: BIOS version 2.10 % Using $PIR table, 8 entries at 0xc00fdef0 % pcib0: at pcibus 0 on motherboard % pci0: on pcib0 % pci_cfgintr: 0:11 INTA BIOS irq 11 % pci_cfgintr: 0:11 INTA BIOS irq 11 % pci_cfgintr: 0:13 INTA BIOS irq 9 % pci_cfgintr: 0:17 INTA BIOS irq 5 % pci_cfgintr: 0:19 INTA BIOS irq 11 % pci_cfgintr: 0:19 INTB BIOS irq 11 % pcib1: at device 1.0 on pci0 % pci1: on pcib1 % pci_cfgintr: 0:1 INTA routed to irq 5 % pcib1: slot 0 INTA is routed to irq 5 % pci1: at device 0.0 (no driver attached) % isab0: at device 7.0 on pci0 % isa0: on isab0 % atapci0: port 0xf000-0xf00f at device 7.1 on pci0 % ata0: at 0x1f0 irq 14 on atapci0 % ata0: [MPSAFE] % ata1: at 0x170 irq 15 on atapci0 % ata1: [MPSAFE] % pci0: at device 7.2 (no driver attached) % pci0: at device 7.3 (no driver attached) % pci0: at device 11.0 (no driver attached) % pci0: at device 11.1 (no driver attached) % fxp0: port 0xa400-0xa43f mem 0xea000000-0xea0fffff,0xea104000-0xea104fff irq 9 at device 13.0 on pci0 % fxp0: Ethernet address 00:90:27:99:02:99 % miibus0: on fxp0 % inphy0: on miibus0 % inphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto % puc0: port 0xb000-0xb01f,0xac00-0xac07,0xa800-0xa807 mem 0xea103000-0xea103fff,0xea102000-0xea102fff irq 5 at device 17.0 on pci0 % stray irq 7 % sio4: on puc0 % sio4: type 16550A % stray irq 7 % sio5: on puc0 % sio5: type 16550A % atapci1: port 0xbc00-0xbcff,0xb800-0xb803,0xb400-0xb407 irq 11 at device 19.0 on pci0 % atapci1: [MPSAFE] % ata2: at 0xb400 on atapci1 % ata2: [MPSAFE] % atapci2: port 0xc800-0xc8ff,0xc400-0xc403,0xc000-0xc007 irq 11 at device 19.1 on pci0 % atapci2: [MPSAFE] % ata3: at 0xc000 on atapci2 % ata3: [MPSAFE] % orm0: