Date: Fri, 16 Jul 2004 07:38:51 +0200 From: "Daniel Eriksson" <daniel_k_eriksson@telia.com> To: <noackjr@alumni.rice.edu> Cc: freebsd-current@freebsd.org Subject: RE: consistent hard lockup with recent kernels Message-ID: <!~!UENERkVCMDkAAQACAAAAAAAAAAAAAAAAABgAAAAAAAAA0VcX9IoJqUaXPS8MjT1PdsKAAAAQAAAASwXyhQrhq0KvaGmo0/NC3gEAAAAA@telia.com> In-Reply-To: <40F748F3.6020702@alumni.rice.edu>
next in thread | previous in thread | raw e-mail | index | archive | help
[-- Attachment #1 --]
Jon Noack wrote:
> Yeah, it is most likely bugs uncovered by native preemption. For now
> you can remask then by commenting out the "#define PREEMPTION" in
> src/sys/<ARCH>/include/param.h and rebuilding your kernel. Robert
> Watson's comments on this:
> "This results in a quite usable kernel, although quite probably more
> poor interrupt handling latency, etc."
I just tried a new kernel (2004.07.16.01.00.00) with preemption turned on,
and my problems with interrupt storms and SATA still remains. With
preemption turned off the machine seems stable, but somewhat sluggish.
With preemption enabled I get interrupt storms and DMA problems with the
SATA devices like below when trying to access all discs at once (running
fsck on all 11 file systems simultaneously). ad20 is one of two SATA devices
in the system.
Interrupt storm detected on "irq20: atapci4"; throttling interrupt source
ad20: FAILURE - already active DMA on this device
ad20: setting up DMA failed
Attached is the dmesg.boot from the old kernel (2004.07.12.22.00.00) with
preemption turned off that seems to be stable.
I'm using HZ=2000, DEVICE_POLLING and ZERO_COPY_SOCKETS, plus device apic. I
also have net.isr enabled.
'vmstat -i' gives the following output:
interrupt total rate
irq1: atkbd0 248 0
irq0: clk 8176853 1993
irq4: sio0 12 0
irq6: fdc0 6 0
irq8: rtc 524902 127
irq13: npx0 1 0
irq14: ata0 33148 8
irq15: ata1 24641 6
irq16: atapci0+ 867398 211
irq17: atapci2+ 296877 72
irq18: re0 2 0
irq19: ahc0 94035 22
irq20: atapci4 184381 44
irq23: vr0 2 0
Total 10202506 2487
/Daniel Eriksson
[-- Attachment #2 --]
Copyright (c) 1992-2004 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD 5.2-CURRENT #0: Tue Jul 13 03:17:24 CEST 2004
daniel@xxx.xxx.xxx:/usr/obj/usr/src/sys/FORTIFY
WARNING: Kernel preemption is disabled, expect reduced performance.
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: AMD Athlon(TM) XP 2500+ (1999.78-MHz 686-class CPU)
Origin = "AuthenticAMD" Id = 0x6a0 Stepping = 0
Features=0x383fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE>
AMD Features=0xc0400000<AMIE,DSP,3DNow!>
real memory = 1342156800 (1279 MB)
avail memory = 1304948736 (1244 MB)
ACPI APIC Table: <ASUS A7V600-X>
ioapic0: Changing APIC ID to 2
ioapic0 <Version 0.3> irqs 0-23 on motherboard
npx0: [FAST]
npx0: <math processor> on motherboard
npx0: INT 16 interface
acpi0: <ASUS A7V600-X> on motherboard
acpi0: [GIANT-LOCKED]
acpi0: Power Button (fixed)
Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
acpi_timer0: <32-bit timer at 3.579545MHz> port 0xe408-0xe40b on acpi0
cpu0: <ACPI CPU> on acpi0
acpi_button0: <Power Button> on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
agp0: <VIA 8377 (Apollo KT400/KT400A/KT600) host to PCI bridge> mem 0xf8000000-0xfbffffff at device 0.0 on pci0
pcib1: <ACPI PCI-PCI bridge> at device 1.0 on pci0
pci1: <ACPI PCI bus> on pcib1
pci1: <display, VGA> at device 0.0 (no driver attached)
atapci0: <HighPoint HPT374 (channel 0+1) UDMA133 controller> port 0xb400-0xb4ff,0xb800-0xb803,0xd000-0xd007,0xd400-0xd403,0xd800-0xd807 irq 16 at device 10.0 on pci0
ata2: at 0xd800 on atapci0
ata3: at 0xd000 on atapci0
atapci1: <HighPoint HPT374 (channel 2+3) UDMA133 controller> port 0x9800-0x98ff,0xa000-0xa003,0xa400-0xa407,0xa800-0xa803,0xb000-0xb007 irq 16 at device 10.1 on pci0
ata4: at 0xb000 on atapci1
ata5: at 0xa400 on atapci1
ahc0: <Adaptec 29160 Ultra160 SCSI adapter> port 0x9400-0x94ff mem 0xed800000-0xed800fff irq 19 at device 12.0 on pci0
ahc0: [GIANT-LOCKED]
aic7892: Ultra160 Wide Channel A, SCSI Id=7, 32/253 SCBs
atapci2: <HighPoint HPT374 (channel 0+1) UDMA133 controller> port 0x7800-0x78ff,0x8000-0x8003,0x8400-0x8407,0x8800-0x8803,0x9000-0x9007 irq 17 at device 14.0 on pci0
ata6: at 0x9000 on atapci2
ata7: at 0x8400 on atapci2
atapci3: <HighPoint HPT374 (channel 2+3) UDMA133 controller> port 0x6000-0x60ff,0x6400-0x6403,0x6800-0x6807,0x7000-0x7003,0x7400-0x7407 irq 17 at device 14.1 on pci0
ata8: at 0x7400 on atapci3
ata9: at 0x6800 on atapci3
atapci4: <VIA 6420 SATA150 controller> port 0x4000-0x40ff,0x4400-0x440f,0x4800-0x4803,0x5000-0x5007,0x5400-0x5403,0x5800-0x5807 irq 20 at device 15.0 on pci0
ata10: at 0x5800 on atapci4
ata11: at 0x5000 on atapci4
atapci5: <VIA 8237 UDMA133 controller> port 0x3800-0x380f,0x376,0x170-0x177,0x3f6,0x1f0-0x1f7 irq 20 at device 15.1 on pci0
ata0: at 0x1f0 irq 14 on atapci5
ata1: at 0x170 irq 15 on atapci5
uhci0: <VIA 83C572 USB controller> port 0x3400-0x341f irq 21 at device 16.0 on pci0
uhci0: [GIANT-LOCKED]
usb0: <VIA 83C572 USB controller> on uhci0
usb0: USB revision 1.0
uhub0: VIA UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub0: 2 ports with 2 removable, self powered
uhci1: <VIA 83C572 USB controller> port 0x3000-0x301f irq 21 at device 16.1 on pci0
uhci1: [GIANT-LOCKED]
usb1: <VIA 83C572 USB controller> on uhci1
usb1: USB revision 1.0
uhub1: VIA UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub1: 2 ports with 2 removable, self powered
uhci2: <VIA 83C572 USB controller> port 0x2800-0x281f irq 21 at device 16.2 on pci0
uhci2: [GIANT-LOCKED]
usb2: <VIA 83C572 USB controller> on uhci2
usb2: USB revision 1.0
uhub2: VIA UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub2: 2 ports with 2 removable, self powered
uhci3: <VIA 83C572 USB controller> port 0x2400-0x241f irq 21 at device 16.3 on pci0
uhci3: [GIANT-LOCKED]
usb3: <VIA 83C572 USB controller> on uhci3
usb3: USB revision 1.0
uhub3: VIA UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub3: 2 ports with 2 removable, self powered
ehci0: <EHCI (generic) USB 2.0 controller> mem 0xed000000-0xed0000ff irq 21 at device 16.4 on pci0
ehci0: [GIANT-LOCKED]
ehci_pci_attach: companion usb0
ehci_pci_attach: companion usb1
ehci_pci_attach: companion usb2
ehci_pci_attach: companion usb3
usb4: EHCI version 1.0
usb4: companion controllers, 2 ports each: usb0 usb1 usb2 usb3
usb4: <EHCI (generic) USB 2.0 controller> on ehci0
usb4: USB revision 2.0
uhub4: (0x1106) EHCI root hub, class 9/0, rev 2.00/1.00, addr 1
uhub4: 8 ports with 8 removable, self powered
isab0: <PCI-ISA bridge> at device 17.0 on pci0
isa0: <ISA bus> on isab0
vr0: <VIA VT6102 Rhine II 10/100BaseTX> port 0x2000-0x20ff mem 0xec800000-0xec8000ff irq 23 at device 18.0 on pci0
miibus0: <MII bus> on vr0
rlphy0: <RTL8201L 10/100 media interface> on miibus0
rlphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
vr0: Ethernet address: 00:0e:a6:1f:29:1e
vr0: [GIANT-LOCKED]
re0: <RealTek 8110S Single-chip Gigabit Ethernet> port 0x1800-0x18ff mem 0xec000000-0xec0000ff irq 18 at device 19.0 on pci0
miibus1: <MII bus> on re0
rgephy0: <RTL8169S/8110S media interface> on miibus1
rgephy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseTX, 1000baseTX-FDX, auto
re0: Ethernet address: 00:50:fc:f8:c6:81
re0: [GIANT-LOCKED]
fdc0: <Enhanced floppy controller (i82077, NE72065 or clone)> port 0x3f7,0x3f2-0x3f5 irq 6 drq 2 on acpi0
fdc0: FIFO enabled, 8 bytes threshold
fd0: <1440-KB 3.5" drive> on fdc0 drive 0
sio0: configured irq 4 not in bitmap of probed irqs 0
sio0: port may not be enabled
sio0 port 0x3f8-0x3ff irq 4 on acpi0
sio0: type 16550A
atkbdc0: <Keyboard controller (i8042)> port 0x64,0x60 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
orm0: <ISA Option ROM> at iomem 0xc0000-0xcafff on isa0
pmtimer0 on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
sio1: configured irq 3 not in bitmap of probed irqs 0
sio1: port may not be enabled
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
Timecounter "TSC" frequency 1999782834 Hz quality 800
Timecounters tick every 1.000 msec
ipfw2 initialized, divert enabled, rule-based forwarding enabled, default to accept, logging unlimited
ad0: 114473MB <ST3120022A> [232581/16/63] at ata0-master UDMA100
ad1: 114473MB <ST3120022A> [232581/16/63] at ata0-slave UDMA100
ad2: 117800MB <IC35L120AVV207-0> [239340/16/63] at ata1-master UDMA100
ad3: 117800MB <IC35L120AVV207-0> [239340/16/63] at ata1-slave UDMA100
ad4: 238475MB <WDC WD2500JB-00FUA0> [484521/16/63] at ata2-master UDMA100
ad5: 238475MB <WDC WD2500JB-00FUA0> [484521/16/63] at ata2-slave UDMA100
ad6: 239372MB <Maxtor 4A250J0> [486344/16/63] at ata3-master UDMA133
ad7: 239372MB <Maxtor 5A250J0> [486344/16/63] at ata3-slave UDMA133
ad8: 194481MB <Maxtor 6Y200P0> [395136/16/63] at ata4-master UDMA133
ad9: 194481MB <Maxtor 6Y200P0> [395136/16/63] at ata4-slave UDMA133
ad10: 194481MB <Maxtor 6Y200P0> [395136/16/63] at ata5-master UDMA133
ad11: 239372MB <Maxtor 7Y250P0> [486344/16/63] at ata5-slave UDMA133
ad12: 117800MB <IC35L120AVV207-0> [239340/16/63] at ata6-master UDMA100
ad13: 117800MB <HDS722512VLAT20> [239340/16/63] at ata6-slave UDMA100
ad14: 117246MB <Maxtor 6Y120L0> [238216/16/63] at ata7-master UDMA133
ad15: 117246MB <Maxtor 6Y120L0> [238216/16/63] at ata7-slave UDMA133
ad16: 114473MB <WDC WD1200JB-00DUA3> [232581/16/63] at ata8-master UDMA100
ad18: 26059MB <Maxtor 92732U8> [52946/16/63] at ata9-master UDMA66
ad20: 239372MB <Maxtor 7Y250M0> [486344/16/63] at ata10-master SATA150
ad22: 238475MB <WDC WD2500JD-00FYB0> [484521/16/63] at ata11-master SATA150
ar0: 476950MB <ATA RAID0 array> [60802/255/63] status: READY subdisks:
disk0 READY on ad4 at ata2-master
disk1 READY on ad5 at ata2-slave
ar1: 478744MB <ATA RAID0 array> [61031/255/63] status: READY subdisks:
disk0 READY on ad6 at ata3-master
disk1 READY on ad7 at ata3-slave
ar2: 388962MB <ATA RAID0 array> [49585/255/63] status: READY subdisks:
disk0 READY on ad9 at ata4-slave
disk1 READY on ad8 at ata4-master
Waiting 5 seconds for SCSI devices to settle
sa0 at ahc0 bus 0 target 5 lun 0
sa0: <Seagate STT20000N 6A51> Removable Sequential Access SCSI-2 device
sa0: 10.000MB/s transfers (10.000MHz, offset 15)
da0 at ahc0 bus 0 target 0 lun 0
da0: <QUANTUM ATLAS 10K 9WLS UCH0> Fixed Direct Access SCSI-3 device
da0: 40.000MB/s transfers (20.000MHz, offset 31, 16bit), Tagged Queueing Enabled
da0: 8759MB (17938986 512 byte sectors: 255H 63S/T 1116C)
da1 at ahc0 bus 0 target 1 lun 0
da1: <QUANTUM ATLAS 10K 9WLS UCH0> Fixed Direct Access SCSI-3 device
da1: 40.000MB/s transfers (20.000MHz, offset 31, 16bit), Tagged Queueing Enabled
da1: 8759MB (17938986 512 byte sectors: 255H 63S/T 1116C)
da2 at ahc0 bus 0 target 10 lun 0
da2: <IBM DDYS-T36950N S93E> Fixed Direct Access SCSI-3 device
da2: 160.000MB/s transfers (80.000MHz, offset 63, 16bit), Tagged Queueing Enabled
da2: 35003MB (71687340 512 byte sectors: 255H 63S/T 4462C)
da3 at ahc0 bus 0 target 11 lun 0
da3: <IBM IC35L036UWD210-0 S5CQ> Fixed Direct Access SCSI-3 device
da3: 160.000MB/s transfers (80.000MHz, offset 63, 16bit), Tagged Queueing Enabled
da3: 35003MB (71687340 512 byte sectors: 255H 63S/T 4462C)
da4 at ahc0 bus 0 target 12 lun 0
da4: <IBM DDYS-T36950N S93E> Fixed Direct Access SCSI-3 device
da4: 160.000MB/s transfers (80.000MHz, offset 63, 16bit), Tagged Queueing Enabled
da4: 35003MB (71687340 512 byte sectors: 255H 63S/T 4462C)
da5 at ahc0 bus 0 target 13 lun 0
da5: <IBM DDYS-T36950N S93E> Fixed Direct Access SCSI-3 device
da5: 160.000MB/s transfers (80.000MHz, offset 63, 16bit), Tagged Queueing Enabled
da5: 35003MB (71687340 512 byte sectors: 255H 63S/T 4462C)
Mounting root from ufs:/dev/ad0s1a
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?!~!UENERkVCMDkAAQACAAAAAAAAAAAAAAAAABgAAAAAAAAA0VcX9IoJqUaXPS8MjT1PdsKAAAAQAAAASwXyhQrhq0KvaGmo0/NC3gEAAAAA>