From owner-freebsd-stable@FreeBSD.ORG Tue Apr 8 08:41:07 2008 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 53F16106566B for ; Tue, 8 Apr 2008 08:41:07 +0000 (UTC) (envelope-from ari@ish.com.au) Received: from fish.ish.com.au (eth5921.nsw.adsl.internode.on.net [59.167.240.32]) by mx1.freebsd.org (Postfix) with ESMTP id 757008FC16 for ; Tue, 8 Apr 2008 08:41:06 +0000 (UTC) (envelope-from ari@ish.com.au) Received: from delish.ish.com.au ([203.29.62.201] helo=ish.com.au) by fish.ish.com.au with esmtps (SSLv3:DES-CBC3-SHA:168) (Exim 4.43) id 1Jj8ka-0000c9-LG; Tue, 08 Apr 2008 18:00:16 +1000 Received: from ip-182.ish.com.au ([203.29.62.182] verified) by ish.com.au (CommuniGate Pro SMTP 5.2.1) with ESMTP id 3427134; Tue, 08 Apr 2008 18:06:24 +1000 Message-Id: <77E81AD6-FBCC-4D30-A5CB-A9B918D4793F@ish.com.au> From: Aristedes Maniatis To: FreeBSD Stable Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes Content-Transfer-Encoding: 7bit Mime-Version: 1.0 (Apple Message framework v919.2) Date: Tue, 8 Apr 2008 18:06:24 +1000 X-Mailer: Apple Mail (2.919.2) Cc: bzeeb+freebsd+lor@zabbadoz.net, Jurgen Weber Subject: LOR sleepq/scrlock X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Apr 2008 08:41:07 -0000 FreeBSD dash.ish.com.au 7.0-RELEASE FreeBSD 7.0-RELEASE #10 i386 with PAE (5Gb RAM) We've had fairly reproducible freezes. After several hours of stress testing or even overnight not doing anything, everything locks up including the console. We then installed a debugging kernel (without INVARIANTS since that prevented the kernel from compiling at all) and obtained this LOR when it froze: LOR: 1st 0x807d3d90 sleepq chain (sleepq chain) @/usr/src/sys/kern/ subr_sleepqueue.c:773 2nd 0x807c8110 scrlock (scrlock) @/usr/src/sys/dev/syscons/syscons.c: 2526 I have taken photographs of the KDB output following this but have not transcribed it until someone says that it will be useful to them. I could put it up as slightly fuzzy screen photographs on our web site. I am somewhat concerned about the combination of PAE (being slightly old technology now) and zfs (being cutting edge), not having a huge amount of testing against each other. However nothing in this lock seems to suggest zfs to me. The only unusual thing on this box is this: hw.physmem: 1063911424 actual memory is half that, but I thought this might be a side effect of PAE. Then, restarting the machine and it hit a couple of LOR errors in quick succession. I can't see any exact matches to http://sources.zabbadoz.net/freebsd/lor.html However the box boots normally after these LOR, so they may not be fatal and may not be related to the above, but just in case... lock order reversal: 1st 0x862f9204 inp (udpinp) @ /usr/src/sys/netinet/udp_usrreq.c:843 2nd 0x8081f498 PFil hook read/write mutex (PFil hook read/write mutex) @ /usr/src/sys/net/pfil.c:73 KDB: stack backtrace: db_trace_self_wrapper(807430c0,c3d639fc, 80438ff5,80744463,8081f498,...) at db_trace_self_wrapper+0x26 kdb_backtrace(80744463,8081f498,8074a8c5,8074a8c5,8074a8ad,...) at kdb_backtrace+0x29 witness_checkorder(8081f498,1,8074a8ad,49,807528d7,...) at witness_checkorder+0x5e5 _rw_rlock(8081f498,8074a8ad,49,0,c3d63ab8,...) at _rw_rlock+0x2a pfil_run_hooks(8081f480,c3d63ad8,83931800,2,862f9168,...) at pfil_run_hooks+0x35 ip_output(867ea700,0,c3d63a9c,0,0,...) at ip_output+0x86f udp_send(862f77bc,0,867ea700,83c91bb0,0,...) at udp_send+0x57b sosend_dgram(862f77bc,83c91bb0,c3d63bd4,867ea700,0,...) at sosend_dgram +0x356 sosend(862f77bc,83c91bb0,c3d63bd4,0,0,...) at sosend+0x3f kern_sendit(84cad660,1e,c3d63c58,0,0,...) at kern_sendit+0x106 sendit(0,1,c3d63c54,28,83c91c60,...) at sendit+0xb1 sendmsg(84cad660,c3d63cfc,c,84cad660,807845c0,...) at sendmsg+0x71 syscall(c3d63d38) at syscall+0x2b3 Xint0x80_syscall() at Xint0x80_syscall+0x20 --- syscall (28, FreeBSD ELF32, sendmsg), eip = 0x2842415b, esp = 0x7f3fc7fc, ebp = 0x7f3fc818 --- lock order reversal: 1st 0x8082010c tcp (tcp) @ /usr/src/sys/netinet/tcp_input.c:400 2nd 0x8081f498 PFil hook read/write mutex (PFil hook read/write mutex) @ /usr/src/sys/net/pfil.c:73 KDB: stack backtrace: db_trace_self_wrapper(807430c0,c16039ec, 80438ff5,80744463,8081f498,...) at db_trace_self_wrapper+0x26 kdb_backtrace(80744463,8081f498,8074a8c5,8074a8c5,8074a8ad,...) at kdb_backtrace+0x29 witness_checkorder(8081f498,1,8074a8ad,49,807528d7,...) at witness_checkorder+0x5e5 _rw_rlock(8081f498,8074a8ad,49,0,c1603aa8,...) at _rw_rlock+0x2a pfil_run_hooks(8081f480,c1603ac8,83935000,2,0,...) at pfil_run_hooks +0x35 ip_output(8394ed00,0,c1603a8c, 0,0,0,80796f90,0,0,0,804b2971,80796f94,80796f9c,c8) at ip_output+0x86f tcp_respond(0,83984830,83984844,8394ed00,46ca580,...) at tcp_respond +0x395 tcp_dropwithreset(1,3,99e2,873e1dcb,1600,...) at tcp_dropwithreset+0x126 tcp_input(8394ed00,14,83935000,1,0,...) at tcp_input+0xcf9 ip_input(8394ed00,14e,800,83935000,800,...) at ip_input+0x64a netisr_dispatch(2,8394ed00,10,3,0,...) at netisr_dispatch+0x55 ether_demux(83935000,8394ed00,3,0,3,...) at ether_demux+0x1c1 ether_input(83935000,8394ed00,8072d03c,6a9,83922014,...) at ether_input +0x323 fxp_intr(83922000,0,8073e924,471,8384a764,...) at fxp_intr+0x237 ithread_loop(839219b0,c1603d38,8073e756,305,838fc804,...) at ithread_loop+0x145 fork_exit(803ee250,839219b0,c1603d38) at fork_exit+0x94 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc1603d70, ebp = 0 --- lock order reversal: 1st 0x86c5809c inp (tcpinp) @ /usr/src/sys/netinet/tcp_usrreq.c:470 2nd 0x8081f498 PFil hook read/write mutex (PFil hook read/write mutex) @ /usr/src/sys/net/pfil.c:73 KDB: stack backtrace: db_trace_self_wrapper (807430c0,c39d7a30,80438ff5,80744463,8081f498,...) at db_trace_self_wrapper+0x26 kdb_backtrace(80744463,8081f498,8074a8c5,8074a8c5,8074a8ad,...) at kdb_backtrace+0x29 witness_checkorder(8081f498,1,8074a8ad,49,807528d7,...) at witness_checkorder+0x5e5 _rw_rlock(8081f498,8074a8ad,49,0,c39d7aec,...) at _rw_rlock+0x2a pfil_run_hooks(8081f480,c39d7b0c,839d3400,2,86c58000,...) at pfil_run_hooks+0x35 ip_output(867ce300,0,c39d7ad0,0,0,...) at ip_output+0x86f tcp_output(86c59000,0,8074feb4,1d6,86c5809c,...) at tcp_output+0x12dc tcp_usr_connect(862f7ad4,86572bd0,83d5b000,25,c39d7c60,...) at tcp_usr_connect+0xf4 soconnect(862f7ad4,86572bd0,83d5b000,10,16,...) at soconnect+0x52 kern_connect(83d5b000,5a,86572bd0,86572bd0,282cd72c,...) at kern_connect+0x59 connect(83d5b000,c39d7cfc,c,80744e40,80784c50,...) at connect+0x46 syscall(c39d7d38) at syscall+0x2b3 Xint0x80_syscall() at Xint0x80_syscall+0x20 --- syscall (98, FreeBSD ELF32, connect), eip = 0x2838a11b, esp = 0x7f7fda4c, ebp = 0x7f7fda68 --- dmesg: Copyright (c) 1992-2008 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 7.0-RELEASE #10: Tue Apr 8 12:54:38 EST 2008 root@dash.ish.com.au:/usr/obj/usr/src/sys/PAE WARNING: WITNESS option enabled, expect reduced performance. Timecounter "i8254" frequency 1193182 Hz quality 0 CPU: Intel(R) XEON(TM) CPU 2.40GHz (2395.92-MHz 686-class CPU) Origin = "GenuineIntel" Id = 0xf24 Stepping = 4 Features = 0x3febfbff < FPU ,VME ,DE ,PSE ,TSC ,MSR ,PAE ,MCE ,CX8 ,APIC ,SEP ,MTRR ,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM> Logical CPUs per core: 2 real memory = 1073741824 (1024 MB) avail memory = 1038790656 (990 MB) MPTable: < Kings Canyon> FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 1 cpu2 (AP): APIC ID: 6 cpu3 (AP): APIC ID: 7 WITNESS: spin lock intrcnt not in order list ioapic0: Assuming intbase of 0 ioapic1: Assuming intbase of 24 ioapic2: Assuming intbase of 48 ioapic0 irqs 0-23 on motherboard ioapic1 irqs 24-47 on motherboard ioapic2 irqs 48-71 on motherboard registered firmware set registered firmware set registered firmware set registered firmware set registered firmware set registered firmware set registered firmware set registered firmware set registered firmware set registered firmware set registered firmware set kbd1 at kbdmux0 cryptosoft0: on motherboard cpu0 on motherboard cpu1 on motherboard cpu2 on motherboard cpu3 on motherboard pcib0: pcibus 0 on motherboard pci0: on pcib0 pci0: at device 0.1 (no driver attached) pcib1: at device 2.0 on pci0 pci1: on pcib1 pcib2: at device 29.0 on pci1 pci2: on pcib2 pci2: at device 1.0 (no driver attached) pcib3: at device 31.0 on pci1 pci3: on pcib3 ahc0: port 0x4000-0x40ff mem 0xfc300000-0xfc300fff irq 28 at device 2.0 on pci3 ahc0: [ITHREAD] aic7899: Ultra160 Wide Channel A, SCSI Id=7, 32/253 SCBs ahc1: port 0x4400-0x44ff mem 0xfc301000-0xfc301fff irq 29 at device 2.1 on pci3 ahc1: [ITHREAD] aic7899: Ultra160 Wide Channel B, SCSI Id=7, 32/253 SCBs em0: port 0x4800-0x481f mem 0xfc340000-0xfc35ffff,0xfc320000-0xfc33ffff irq 31 at device 4.0 on pci3 em0: Ethernet address: 00:30:48:24:aa:6f em0: [FILTER] pci0: at device 29.0 (no driver attached) pci0: at device 29.1 (no driver attached) pci0: at device 29.2 (no driver attached) pcib4: at device 30.0 on pci0 pci4: on pcib4 vgapci0: port 0x5000-0x50ff mem 0xfd000000-0xfdffffff,0xfc400000-0xfc400fff irq 16 at device 1.0 on pci4 fxp0: port 0x5400-0x543f mem 0xfc401000-0xfc401fff,0xfc420000-0xfc43ffff irq 17 at device 2.0 on pci4 miibus0: on fxp0 inphy0: PHY 1 on miibus0 inphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto fxp0: Ethernet address: 00:30:48:24:aa:70 fxp0: [ITHREAD] isab0: at device 31.0 on pci0 isa0: on isab0 atapci0: port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x2060-0x206f at device 31.1 on pci0 ata0: on atapci0 ata0: [ITHREAD] ata1: on atapci0 ata1: [ITHREAD] pci0: at device 31.3 (no driver attached) pmtimer0 on isa0 orm0: at iomem 0xc0000-0xc7fff,0xc8000-0xc8fff, 0xdc000-0xdffff,0xe0000-0xe3fff pnpid ORM0000 on isa0 atkbdc0: at port 0x60,0x64 on isa0 atkbd0: irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] atkbd0: [ITHREAD] fdc0: at port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on isa0 fdc0: [FILTER] fd0: <1440-KB 3.5" drive> on fdc0 drive 0 sc0: at flags 0x100 on isa0 sc0: VGA <16 virtual consoles, flags=0x300> sio0 at port 0x3f8-0x3ff irq 4 flags 0x10 on isa0 sio0: type 16550A sio0: [FILTER] sio1 at port 0x2f8-0x2ff irq 3 on isa0 sio1: type 16550A sio1: [FILTER] vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 unknown: can't assign resources (memory) unknown: can't assign resources (memory) unknown: can't assign resources (port) unknown: can't assign resources (memory) unknown: can't assign resources (memory) unknown: can't assign resources (memory) unknown: can't assign resources (port) unknown: can't assign resources (port) unknown: can't assign resources (port) WARNING: ZFS is considered to be an experimental feature in FreeBSD. Timecounters tick every 1.000 msec Fast IPsec: Initialized Security Association Processing. ZFS filesystem version 6 ZFS storage pool version 6 acd0: CDROM at ata1-master UDMA33 Waiting 5 seconds for SCSI devices to settle ses0 at ahc0 bus 0 target 6 lun 0 ses0: Fixed Processor SCSI-2 device ses0: 3.300MB/s transfers ses0: SAF-TE Compliant Device da0 at ahc0 bus 0 target 0 lun 0 da0: Fixed Direct Access SCSI-3 device da0: 160.000MB/s transfers (80.000MHz DT, offset 127, 16bit) da0: Command Queueing Enabled da0: 70136MB (143638992 512 byte sectors: 255H 63S/T 8941C) da1 at ahc0 bus 0 target 1 lun 0 da1: Fixed Direct Access SCSI-3 device da1: 160.000MB/s transfers (80.000MHz DT, offset 127, 16bit) da1: Command Queueing Enabled da1: 70136MB (143638992 512 byte sectors: 255H 63S/T 8941C) da2 at ahc0 bus 0 target 2 lun 0 da2: Fixed Direct Access SCSI-3 device da2: 160.000MB/s transfers (80.000MHz DT, offset 127, 16bit) da2: Command Queueing Enabled da2: 35046MB (71775284 512 byte sectors: 255H 63S/T 4467C) SMP: AP CPU #2 Launched! SMP: AP CPU #1 Launched! SMP: AP CPU #3 Launched! WARNING: WITNESS option enabled, expect reduced performance. Trying to mount root from zfs:tank/root WARNING: /mnt/otherhdd was not properly dismounted --------------------------> ish http://www.ish.com.au Level 1, 30 Wilson Street Newtown 2042 Australia phone +61 2 9550 5001 fax +61 2 9550 4001 GPG fingerprint CBFB 84B4 738D 4E87 5E5C 5EFA EF6A 7D2E 3E49 102A