Date: Thu, 6 Jan 2011 13:31:45 +0300 From: Lev Serebryakov <lev@FreeBSD.org> To: "freebsd-stable@freebsd.org" <freebsd-stable@freebsd.org> Subject: FreeBSD 8.2-PRERELEASE hangs under load with "live" kernel Message-ID: <1608571029.20110106133145@serebryakov.spb.ru>
next in thread | raw e-mail | index | archive | help
[-- Attachment #1 --]
Hello, Freebsd-stable.
I've added torrent client (transmission) to software on my home
server and it starts to hang in very unusual way: kernel works but
userland doesn't.
I can ping it (and it answers). I can scroll console with
"scrolllock" button and keys. I can break into debugger with
Ctrl+SysReq and it shows, that one CPU is occupied by idle process and
other by "Giant tasq", but no userland processes answer: I can not
ssh to it, I cannot login on console, samba is dead, etc.
"ps" in kernel debugger shows, that many of processes in "pfault"
state, and noting more "special".
memtest86+ doesn't show any errors after 8 passes of tests (about
10 hours), so RAM looks Ok.
What should I do in kdb to understand what happens?
Kernel config and /var/run/dmesg.boot is attached.
--
// Black Lion AKA Lev Serebryakov <lev@FreeBSD.org>
[-- Attachment #2 --]
cpu HAMMER
ident BLOB
makeoptions DEBUG=-g # Build kernel with gdb(1) debug symbols
options PRINTF_BUFR_SIZE=1024
options DDB
options KDB
options KDB_UNATTENDED
options BREAK_TO_DEBUGGER
options SCHED_ULE # ULE scheduler
options PREEMPTION # Enable kernel thread preemption
options INET # InterNETworking
options FFS # Berkeley Fast Filesystem
options SOFTUPDATES # Enable FFS soft updates support
options UFS_ACL # Support for access control lists
options UFS_DIRHASH # Improve performance on big directories
options UFS_GJOURNAL # Enable gjournal-based UFS journaling
options MD_ROOT # MD is a potential root device
options NFSCLIENT # Network Filesystem Client
options NFSSERVER # Network Filesystem Server
options NFSLOCKD # Network Lock Manager
options PROCFS # Process filesystem (requires PSEUDOFS)
options PSEUDOFS # Pseudo-filesystem framework
options GEOM_LABEL # Provides labelization
options COMPAT_43TTY # BSD 4.3 TTY compat [KEEP THIS!]
options COMPAT_IA32 # Compatible with i386 binaries
options COMPAT_FREEBSD4 # Compatible with FreeBSD4
options COMPAT_FREEBSD5 # Compatible with FreeBSD5
options COMPAT_FREEBSD6 # Compatible with FreeBSD6
options COMPAT_FREEBSD7 # Compatible with FreeBSD7
options SCSI_DELAY=5000 # Delay (in ms) before probing SCSI
options KTRACE # ktrace(1) support
options STACK # stack(9) support
options SYSVSHM # SYSV-style shared memory
options SYSVMSG # SYSV-style message queues
options SYSVSEM # SYSV-style semaphores
options _KPOSIX_PRIORITY_SCHEDULING # POSIX P1003_1B real-time extensions
options KBD_INSTALL_CDEV # install a CDEV entry in /dev
options AUDIT # Security event auditing
# Make an SMP-capable kernel by default
options SMP # Symmetric MultiProcessor Kernel
# Device polling
options DEVICE_POLLING
# CPU frequency control
device cpufreq
device coretemp
device smb
device smbus
device ichsmb
device iicbus
device iicsmb
device iic
# Bus support.
device acpi
device pci
# SCSI
device scbus
device pass
device da
# ATA and ATAPI devices
#device ata
#device atadisk # ATA disk drives
#options ATA_STATIC_ID # Static device numbering
device ahci
# atkbdc0 controls both the keyboard and the PS/2 mouse
device atkbdc # AT keyboard controller
device atkbd # AT keyboard
device psm # PS/2 mouse
device kbdmux # keyboard multiplexer
device vga # VGA video card driver
# syscons is the default console driver, resembling an SCO console
device sc
# Serial (COM) ports
device uart # Generic UART driver
# Parallel port
device ppc
device ppbus # Parallel port bus (required)
device em # Intel PRO/1000 adapter Gigabit Ethernet Card
# Pseudo devices.
device loop # Network loopback
device random # Entropy device
device ether # Ethernet support
device tun # Packet tunnel.
device pty # Pseudo-ttys (telnet etc)
device md # Memory "disks"
device firmware # firmware assist module
# The `bpf' device enables the Berkeley Packet Filter.
# Be aware of the administrative consequences of enabling this!
# Note that 'bpf' is required for DHCP.
device bpf # Berkeley packet filter
# USB support
device uhci # UHCI PCI->USB interface
device ehci # EHCI PCI->USB interface (USB 2.0)
device usb # USB Bus (required)
device ucom # Disks/Mass storage - Requires scbus and da
device umass # Disks/Mass storage - Requires scbus and da
# FireWire support
device firewire # FireWire bus code
[-- Attachment #3 --]
Copyright (c) 1992-2011 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 8.2-PRERELEASE #3: Mon Jan 3 18:46:36 MSK 2011
lev@blob.home.serebryakov.spb.ru:/usr/obj/usr/src/sys/BLOB amd64
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: Intel(R) Core(TM)2 Duo CPU E4500 @ 2.20GHz (2200.09-MHz K8-class CPU)
Origin = "GenuineIntel" Id = 0x6fd Family = 6 Model = f Stepping = 13
Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
Features2=0xe39d<SSE3,DTES64,MON,DS_CPL,EST,TM2,SSSE3,CX16,xTPR,PDCM>
AMD Features=0x20100800<SYSCALL,NX,LM>
AMD Features2=0x1<LAHF>
TSC: P-state invariant
real memory = 2147483648 (2048 MB)
avail memory = 2053398528 (1958 MB)
ACPI APIC Table: <A_M_I_ OEMAPIC >
FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
FreeBSD/SMP: 1 package(s) x 2 core(s)
cpu0 (BSP): APIC ID: 0
cpu1 (AP): APIC ID: 1
ioapic0 <Version 2.0> irqs 0-23 on motherboard
kbd1 at kbdmux0
umcs7840: Version 1.0.20101210.57 (rev 2bfca65f4623)
acpi0: <A_M_I_ OEMXSDT> on motherboard
acpi0: [ITHREAD]
acpi0: Power Button (fixed)
acpi0: reservation of 0, a0000 (3) failed
acpi0: reservation of 100000, 7fc00000 (3) failed
Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0
cpu0: <ACPI CPU> on acpi0
ACPI Warning: Incorrect checksum in table [OEMB] - 0xCB, should be 0xC6 (20101013/tbutils-354)
cpu1: <ACPI CPU> on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
vgapci0: <VGA-compatible display> port 0xec00-0xec07 mem 0xfea80000-0xfeafffff,0xd0000000-0xdfffffff,0xfe900000-0xfe9fffff irq 16 at device 2.0 on pci0
pci0: <simple comms> at device 3.0 (no driver attached)
pci0: <mass storage, ATA> at device 3.2 (no driver attached)
pci0: <simple comms, UART> at device 3.3 (no driver attached)
em0: <Intel(R) PRO/1000 Network Connection 7.1.8> port 0xdc00-0xdc1f mem 0xfea40000-0xfea5ffff,0xfea79000-0xfea79fff irq 20 at device 25.0 on pci0
em0: No MSI/MSIX using a Legacy IRQ
em0: [FILTER]
em0: Ethernet address: 00:1e:8c:75:03:0d
uhci0: <Intel 82801I (ICH9) USB controller> port 0xd480-0xd49f irq 16 at device 26.0 on pci0
uhci0: [ITHREAD]
usbus0: <Intel 82801I (ICH9) USB controller> on uhci0
uhci1: <Intel 82801I (ICH9) USB controller> port 0xd800-0xd81f irq 21 at device 26.1 on pci0
uhci1: [ITHREAD]
usbus1: <Intel 82801I (ICH9) USB controller> on uhci1
uhci2: <Intel 82801I (ICH9) USB controller> port 0xd880-0xd89f irq 18 at device 26.2 on pci0
uhci2: [ITHREAD]
usbus2: <Intel 82801I (ICH9) USB controller> on uhci2
ehci0: <Intel 82801I (ICH9) USB 2.0 controller> mem 0xfea7a800-0xfea7abff irq 18 at device 26.7 on pci0
ehci0: [ITHREAD]
usbus3: EHCI version 1.0
usbus3: <Intel 82801I (ICH9) USB 2.0 controller> on ehci0
uhci3: <Intel 82801I (ICH9) USB controller> port 0xd000-0xd01f irq 23 at device 29.0 on pci0
uhci3: [ITHREAD]
usbus4: <Intel 82801I (ICH9) USB controller> on uhci3
uhci4: <Intel 82801I (ICH9) USB controller> port 0xd080-0xd09f irq 19 at device 29.1 on pci0
uhci4: [ITHREAD]
usbus5: <Intel 82801I (ICH9) USB controller> on uhci4
uhci5: <Intel 82801I (ICH9) USB controller> port 0xd400-0xd41f irq 18 at device 29.2 on pci0
uhci5: [ITHREAD]
usbus6: <Intel 82801I (ICH9) USB controller> on uhci5
ehci1: <Intel 82801I (ICH9) USB 2.0 controller> mem 0xfea7a400-0xfea7a7ff irq 23 at device 29.7 on pci0
ehci1: [ITHREAD]
usbus7: EHCI version 1.0
usbus7: <Intel 82801I (ICH9) USB 2.0 controller> on ehci1
pcib1: <ACPI PCI-PCI bridge> at device 30.0 on pci0
pci1: <ACPI PCI bus> on pcib1
fwohci0: <Lucent FW322/323> mem 0xfebff000-0xfebfffff irq 20 at device 2.0 on pci1
fwohci0: [ITHREAD]
fwohci0: OHCI version 1.0 (ROM=1)
fwohci0: No. of Isochronous channels is 8.
fwohci0: EUI64 00:1e:8c:00:00:1a:5f:da
fwohci0: Phy 1394a available S400, 2 ports.
fwohci0: Link S400, max_rec 2048 bytes.
firewire0: <IEEE1394(FireWire) bus> on fwohci0
fwohci0: Initiate bus reset
fwohci0: fwohci_intr_core: BUS reset
fwohci0: fwohci_intr_core: node_id=0x00000000, SelfID Count=1, CYCLEMASTER mode
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
ahci0: <Intel ICH9 AHCI SATA controller> port 0xc880-0xc887,0xc800-0xc803,0xc480-0xc487,0xc400-0xc403,0xc080-0xc09f mem 0xfea78800-0xfea78fff irq 22 at device 31.2 on pci0
ahci0: [ITHREAD]
ahci0: AHCI v1.20 with 6 3Gbps ports, Port Multiplier supported
ahcich0: <AHCI channel> at channel 0 on ahci0
ahcich0: [ITHREAD]
ahcich1: <AHCI channel> at channel 1 on ahci0
ahcich1: [ITHREAD]
ahcich2: <AHCI channel> at channel 2 on ahci0
ahcich2: [ITHREAD]
ahcich3: <AHCI channel> at channel 3 on ahci0
ahcich3: [ITHREAD]
ahcich4: <AHCI channel> at channel 4 on ahci0
ahcich4: [ITHREAD]
ahcich5: <AHCI channel> at channel 5 on ahci0
ahcich5: [ITHREAD]
ichsmb0: <Intel 82801I (ICH9) SMBus controller> port 0x400-0x41f mem 0xfea7a000-0xfea7a0ff irq 18 at device 31.3 on pci0
ichsmb0: [ITHREAD]
smbus0: <System Management Bus> on ichsmb0
smb0: <SMBus generic I/O> on smbus0
pci0: <dasp> at device 31.6 (no driver attached)
acpi_button0: <Power Button> on acpi0
atrtc0: <AT realtime clock> port 0x70-0x71 irq 8 on acpi0
ppc0: <Parallel port> port 0x378-0x37f,0x778-0x77f irq 7 drq 3 on acpi0
ppc0: SMC-like chipset (ECP/EPP/PS2/NIBBLE) in COMPATIBLE mode
ppc0: FIFO with 16/16/9 bytes threshold
ppc0: [ITHREAD]
ppbus0: <Parallel port bus> on ppc0
acpi_hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
Timecounter "HPET" frequency 14318180 Hz quality 900
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
uart0: [FILTER]
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
atkbd0: [ITHREAD]
orm0: <ISA Option ROM> at iomem 0xcb800-0xcc7ff on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
coretemp0: <CPU On-Die Thermal Sensors> on cpu0
est0: <Enhanced SpeedStep Frequency Control> on cpu0
p4tcc0: <CPU Frequency Thermal Control> on cpu0
coretemp1: <CPU On-Die Thermal Sensors> on cpu1
est1: <Enhanced SpeedStep Frequency Control> on cpu1
p4tcc1: <CPU Frequency Thermal Control> on cpu1
Timecounters tick every 1.000 msec
firewire0: 1 nodes, maxhop <= 0 cable IRM irm(0) (me)
firewire0: bus manager 0
GEOM_RAID5: Module loaded, version 1.0.20101126.49 (rev 8f2366016dd1)
usbus0: 12Mbps Full Speed USB v1.0
usbus1: 12Mbps Full Speed USB v1.0
usbus2: 12Mbps Full Speed USB v1.0
usbus3: 480Mbps High Speed USB v2.0
usbus4: 12Mbps Full Speed USB v1.0
usbus5: 12Mbps Full Speed USB v1.0
usbus6: 12Mbps Full Speed USB v1.0
usbus7: 480Mbps High Speed USB v2.0
ugen0.1: <Intel> at usbus0
uhub0: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0
ugen1.1: <Intel> at usbus1
uhub1: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus1
ugen2.1: <Intel> at usbus2
uhub2: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus2
ugen3.1: <Intel> at usbus3
uhub3: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus3
ugen4.1: <Intel> at usbus4
uhub4: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus4
ugen5.1: <Intel> at usbus5
uhub5: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus5
ugen6.1: <Intel> at usbus6
uhub6: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus6
ugen7.1: <Intel> at usbus7
uhub7: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus7
ada0 at ahcich0 bus 0 scbus0 target 0 lun 0
ada0: <SAMSUNG HD321KJ CP100-10> ATA-8 SATA 2.x device
ada0: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada0: Command Queueing enabled
ada0: 305245MB (625142448 512 byte sectors: 16H 63S/T 16383C)
ada1 at ahcich1 bus 0 scbus1 target 0 lun 0
ada1: <WDC WD5000AAKS-00YGA0 12.01C02> ATA-8 SATA 2.x device
ada1: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada1: Command Queueing enabled
ada1: 476940MB (976773168 512 byte sectors: 16H 63S/T 16383C)
ada2 at ahcich2 bus 0 scbus2 target 0 lun 0
ada2: <WDC WD5000AAKS-00A7B2 01.03B01> ATA-8 SATA 2.x device
ada2: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada2: Command Queueing enabled
ada2: 476940MB (976773168 512 byte sectors: 16H 63S/T 16383C)
ada3 at ahcich3 bus 0 scbus3 target 0 lun 0
ada3: <WDC WD5000AAKS-00YGA0 12.01C02> ATA-8 SATA 2.x device
ada3: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada3: Command Queueing enabled
ada3: 476940MB (976773168 512 byte sectors: 16H 63S/T 16383C)
ada4 at ahcich4 bus 0 scbus4 target 0 lun 0
ada4: <WDC WD5000AAKS-00YGA0 12.01C02> ATA-8 SATA 2.x device
ada4: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada4: Command Queueing enabled
ada4: 476940MB (976773168 512 byte sectors: 16H 63S/T 16383C)
ada5 at ahcich5 bus 0 scbus5 target 0 lun 0
ada5: <WDC WD5000AAKS-00YGA0 12.01C02> ATA-8 SATA 2.x device
ada5: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada5: Command Queueing enabled
ada5: 476940MB (976773168 512 byte sectors: 16H 63S/T 16383C)
SMP: AP CPU #1 Launched!
uhub0: 2 ports with 2 removable, self powered
uhub1: 2 ports with 2 removable, self powered
uhub2: 2 ports with 2 removable, self powered
uhub4: 2 ports with 2 removable, self powered
uhub5: 2 ports with 2 removable, self powered
uhub6: 2 ports with 2 removable, self powered
GEOM_RAID5: storage: device created (stripesize=131072).
GEOM_RAID5: storage: ada1(4): disk attached.
GEOM_RAID5: storage: ada2(3): disk attached.
GEOM_RAID5: storage: ada3(2): disk attached.
GEOM_RAID5: storage: ada4(0): disk attached.
GEOM_RAID5: storage: ada5(1): disk attached.
GEOM_RAID5: storage: activated (need about 95MiB kmem (max)).
Root mount waiting for: usbus7 usbus3
Root mount waiting for: usbus7 usbus3
uhub3: 6 ports with 6 removable, self powered
uhub7: 6 ports with 6 removable, self powered
ugen3.2: <vendor 0x9710> at usbus3
umcs78400: <vendor 0x9710 product 0x7840, rev 2.00/0.01, addr 2> on usbus3
umcs78400: Chip mcs7840, found 2 active ports
umcs78400: On-die confguration: RST: active low, HRD: yes, PLL: avail, POR: avail, Ports: 4, EEPROM write disabled, IrDA is not available
Root mount waiting for: usbus7
ugen7.2: <Generic> at usbus7
umass0: <Generic Mass Storage Device, class 0/0, rev 2.00/1.00, addr 2> on usbus7
Root mount waiting for: usbus7
Trying to mount root from ufs:/dev/ada0s1a
da0 at umass-sim0 bus 0 scbus6 target 0 lun 0
da0: <Generic USB SD Reader 1.00> Removable Direct Access SCSI-0 device
da0: 40.000MB/s transfers
da0: Attempt to query device size failed: NOT READY, Medium not present
da1 at umass-sim0 bus 0 scbus6 target 0 lun 1
da1: <Generic USB CF Reader 1.01> Removable Direct Access SCSI-0 device
da1: 40.000MB/s transfers
da1: Attempt to query device size failed: NOT READY, Medium not present
da2 at umass-sim0 bus 0 scbus6 target 0 lun 2
da2: <Generic USB SM Reader 1.02> Removable Direct Access SCSI-0 device
da2: 40.000MB/s transfers
da2: Attempt to query device size failed: NOT READY, Medium not present
da3 at umass-sim0 bus 0 scbus6 target 0 lun 3
da3: <Generic USB MS Reader 1.03> Removable Direct Access SCSI-0 device
da3: 40.000MB/s transfers
da3: Attempt to query device size failed: NOT READY, Medium not present
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?1608571029.20110106133145>