Date: Wed, 17 Sep 2008 08:33:23 -0400 (EDT) From: Geoffrey Mainland <mainland@apeiron.net> To: FreeBSD-gnats-submit@FreeBSD.org Subject: kern/127439: deadlock in pf Message-ID: <20080917123323.EFF8F22CCB@zeno.apeiron.net> Resent-Message-ID: <200809171250.m8HCo1Tt076838@freefall.freebsd.org>
next in thread | raw e-mail | index | archive | help
>Number: 127439
>Category: kern
>Synopsis: deadlock in pf
>Confidential: no
>Severity: critical
>Priority: high
>Responsible: freebsd-bugs
>State: open
>Quarter:
>Keywords:
>Date-Required:
>Class: sw-bug
>Submitter-Id: current-users
>Arrival-Date: Wed Sep 17 12:50:01 UTC 2008
>Closed-Date:
>Last-Modified:
>Originator: Geoffrey Mainland
>Release: FreeBSD 7.1-PRERELEASE i386
>Organization:
>Environment:
System: FreeBSD zeno.apeiron.net 7.1-PRERELEASE FreeBSD 7.1-PRERELEASE #7: Tue Sep 16 09:28:16 EDT 2008 toor@zeno.apeiron.net:/usr/obj/usr/src/sys/ZENO i386
>Description:
This happens reliably every night. I'm not sure what's running that triggers it.
ifconfig:
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=9b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM>
ether 00:0e:0c:5f:c1:f8
inet6 fe80::20e:cff:fe5f:c1f8%em0 prefixlen 64 scopeid 0x1
inet 192.168.0.10 netmask 0xffffff00 broadcast 192.168.0.255
inet 192.168.0.1 netmask 0xffffffff broadcast 192.168.0.1
inet 192.168.0.2 netmask 0xffffffff broadcast 192.168.0.2
media: Ethernet autoselect (100baseTX <full-duplex>)
status: active
fxp0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=8<VLAN_MTU>
ether 00:90:27:62:87:4d
inet6 fe80::290:27ff:fe62:874d%fxp0 prefixlen 64 scopeid 0x2
inet 68.164.219.98 netmask 0xfffffff8 broadcast 68.164.219.103
inet 68.164.219.99 netmask 0xffffffff broadcast 68.164.219.99
inet 68.164.219.100 netmask 0xffffffff broadcast 68.164.219.100
inet 68.164.219.101 netmask 0xffffffff broadcast 68.164.219.101
media: Ethernet autoselect (100baseTX <full-duplex>)
status: active
vr0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=2808<VLAN_MTU,WOL_UCAST,WOL_MAGIC>
ether 00:15:f2:43:48:7b
inet6 fe80::215:f2ff:fe43:487b%vr0 prefixlen 64 scopeid 0x3
inet 192.168.1.1 netmask 0xffffff00 broadcast 192.168.1.255
inet 192.168.1.2 netmask 0xffffffff broadcast 192.168.1.2
media: Ethernet autoselect (none)
status: no carrier
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x4
inet 127.0.0.1 netmask 0xff000000
pfsync0: flags=0<> metric 0 mtu 1460
syncpeer: 224.0.0.240 maxupd: 128
pflog0: flags=0<> metric 0 mtu 33204
gif0: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1280
tunnel inet 68.164.219.98 --> 66.55.128.25
inet6 fe80::20e:cff:fe5f:c1f8%gif0 prefixlen 64 scopeid 0x7
inet6 2001:4830:1200:10b::2 --> 2001:4830:1200:10b::1 prefixlen 128
tun0: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1500
inet6 fe80::20e:cff:fe5f:c1f8%tun0 prefixlen 64 scopeid 0x8
inet 192.168.2.1 --> 192.168.2.2 netmask 0xffffffff
Opened by PID 1454
Kernel config:
cpu I686_CPU
ident ZENO
options SCHED_ULE
options SMP
options PREEMPTION
options DEVICE_POLLING
options HZ=2000
options _KPOSIX_PRIORITY_SCHEDULING
options P1003_1B_MQUEUE
options KDB
options KDB_TRACE
options DDB
options WITNESS
options INVARIANTS
options INVARIANT_SUPPORT
makeoptions DEBUG=-g #Build kernel with gdb(1) debug symbols
options COMPAT_FREEBSD4
options COMPAT_FREEBSD5
options COMPAT_FREEBSD6
options SYSVSHM
options SYSVSEM
options SYSVMSG
options STACK
options INET #Internet communications protocols
options INET6 #IPv6 communications protocols
options IPSEC #IP security (requires device crypto)
options NETATALK #Appletalk communications protocols
options NETSMB #SMB/CIFS requester
options LIBMCHAIN
options SCTP
options NETGRAPH # netgraph(4) system
device ether #Generic Ethernet
device loop #Network loopback device
device bpf #Berkeley packet filter
device tap #Virtual Ethernet driver
device tun #Tunnel driver (ppp(8), nos-tun(8))
device gre #IP over IP tunneling
device pf #PF OpenBSD packet-filter firewall
device pflog #logging support interface for PF
device pfsync #synchronization interface for PF
device gif #IPv6 and IPv4 tunneling
device faith #for IPv6 and IPv4 translation
device stf #6to4 IPv6 over IPv4 encapsulation
options FFS #Fast filesystem
options NFSCLIENT #Network File System client
options CD9660 #ISO 9660 filesystem
options MSDOSFS #MS DOS File System (FAT, FAT32)
options NFSSERVER #Network File System server
options NFSLOCKD #Network Lock Manager
options NTFS #NT File System
options PROCFS #Process filesystem (requires PSEUDOFS)
options PSEUDOFS #Pseudo-filesystem framework
options SMBFS #SMB/CIFS filesystem
options UDF #Universal Disk Format
options NFS_ROOT #NFS usable as root device
options SOFTUPDATES
options UFS_ACL
options UFS_DIRHASH
device random
device mem
options AUDIT
device scbus #base SCSI code
device da #SCSI direct access devices (aka disks)
device cd #SCSI CD-ROMs
device pt #SCSI processor
device pass #CAM passthrough driver
device pty #Pseudo ttys
device md #Memory/malloc disk
options LIBICONV
options KBD_INSTALL_CDEV # install a CDEV entry in /dev
device splash # Splash screen and screen saver support
device sc
options SC_DISABLE_KDBKEY # disable `debug' key
device ata
device atadisk # ATA disk drives
device ataraid # ATA RAID drives
device atapicd # ATAPI CDROM drives
device atapifd # ATAPI floppy drives
device atapicam # emulate ATAPI devices as SCSI ditto via CAM
options ATA_STATIC_ID
device fdc
device sound
device ppc
device ppbus
device lpt
device ppi
device uhci
device ehci
device usb
device crypto # core crypto support
device cryptodev # /dev/crypto for access to h/w
device apic # I/O apic
device nvram # Access to rtc cmos via /dev/nvram
device sio
device eisa
device pci
options VESA
device psm
device atkbdc
device atkbd
device vga
options COMPAT_LINUX
options COMPAT_AOUT
options LINPROCFS
options LINSYSFS
dmesg output (after crash):
Copyright (c) 1992-2008 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 7.1-PRERELEASE #7: Tue Sep 16 09:28:16 EDT 2008
toor@zeno.apeiron.net:/usr/obj/usr/src/sys/ZENO
WARNING: WITNESS option enabled, expect reduced performance.
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: AMD Sempron(tm) Processor 3100+ (1800.09-MHz 686-class CPU)
Origin = "AuthenticAMD" Id = 0x10fc0 Stepping = 0
Features=0x78bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,MMX,FXSR,SSE,SSE2>
AMD Features=0xc2500800<SYSCALL,NX,MMX+,FFXSR,3DNow!+,3DNow!>
AMD Features2=0x1<LAHF>
real memory = 1073414144 (1023 MB)
avail memory = 1040887808 (992 MB)
WITNESS: spin lock cpuset not in order list
WITNESS: spin lock intrcnt not in order list
netsmb_dev: loaded
cryptosoft0: <software crypto> on motherboard
acpi0: <A M I OEMRSDT> on motherboard
acpi0: [ITHREAD]
acpi0: Power Button (fixed)
acpi0: reservation of 0, a0000 (3) failed
acpi0: reservation of 100000, 3fef0000 (3) failed
Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib1: <ACPI PCI-PCI bridge> at device 1.0 on pci0
pci1: <ACPI PCI bus> on pcib1
vgapci0: <VGA-compatible display> mem
0xfb000000-0xfbffffff,0xf0000000-0xf7ffffff irq 11 at device 0.0 on pci1
em0: <Intel(R) PRO/1000 Network Connection 6.9.5> port 0xe800-0xe83f mem
0xfae00000-0xfae1ffff,0xfad00000-0xfad1ffff irq 11 at device 11.0 on pci0
em0: [FILTER]
em0: Ethernet address: 00:0e:0c:5f:c1:f8
fxp0: <Intel 82559 Pro/100 Ethernet> port 0xe400-0xe43f mem
0xfab00000-0xfab00fff,0xfaa00000-0xfaafffff irq 10 at device 12.0 on pci0
miibus0: <MII bus> on fxp0
inphy0: <i82555 10/100 media interface> PHY 1 on miibus0
inphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
fxp0: Ethernet address: 00:90:27:62:87:4d
fxp0: [ITHREAD]
atapci0: <VIA 6420 SATA150 controller> port
0xe000-0xe007,0xd800-0xd803,0xd400-0xd407,0xd000-0xd003,0xc800-0xc80f,0xc400-0xc4ff
irq 10 at device 15.0 on pci0
atapci0: [ITHREAD]
ata2: <ATA channel 0> on atapci0
ata2: [ITHREAD]
ata3: <ATA channel 1> on atapci0
ata3: [ITHREAD]
atapci1: <VIA 8237 UDMA133 controller> port
0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xfc00-0xfc0f at device 15.1 on pci0
ata0: <ATA channel 0> on atapci1
ata0: [ITHREAD]
ata1: <ATA channel 1> on atapci1
ata1: [ITHREAD]
uhci0: <VIA 83C572 USB controller> port 0xb000-0xb01f irq 11 at device 16.0 on
pci0
uhci0: [GIANT-LOCKED]
uhci0: [ITHREAD]
usb0: <VIA 83C572 USB controller> on uhci0
usb0: USB revision 1.0
uhub0: <VIA UHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb0
uhub0: 2 ports with 2 removable, self powered
uhci1: <VIA 83C572 USB controller> port 0xb400-0xb41f irq 11 at device 16.1 on
pci0
uhci1: [GIANT-LOCKED]
uhci1: [ITHREAD]
usb1: <VIA 83C572 USB controller> on uhci1
usb1: USB revision 1.0
uhub1: <VIA UHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb1
uhub1: 2 ports with 2 removable, self powered
uhci2: <VIA 83C572 USB controller> port 0xb800-0xb81f irq 10 at device 16.2 on
pci0
uhci2: [GIANT-LOCKED]
uhci2: [ITHREAD]
usb2: <VIA 83C572 USB controller> on uhci2
usb2: USB revision 1.0
uhub2: <VIA UHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb2
uhub2: 2 ports with 2 removable, self powered
uhci3: <VIA 83C572 USB controller> port 0xc000-0xc01f irq 10 at device 16.3 on
pci0
uhci3: [GIANT-LOCKED]
uhci3: [ITHREAD]
usb3: <VIA 83C572 USB controller> on uhci3
usb3: USB revision 1.0
uhub3: <VIA UHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb3
uhub3: 2 ports with 2 removable, self powered
ehci0: <VIA VT6202 USB 2.0 controller> mem 0xfa700000-0xfa7000ff irq 5 at device
16.4 on pci0
ehci0: [GIANT-LOCKED]
ehci0: [ITHREAD]
usb4: EHCI version 1.0
usb4: companion controllers, 2 ports each: usb0 usb1 usb2 usb3
usb4: <VIA VT6202 USB 2.0 controller> on ehci0
usb4: USB revision 2.0
uhub4: <VIA EHCI root hub, class 9/0, rev 2.00/1.00, addr 1> on usb4
uhub4: 8 ports with 8 removable, self powered
isab0: <PCI-ISA bridge> at device 17.0 on pci0
isa0: <ISA bus> on isab0
pci0: <multimedia, audio> at device 17.5 (no driver attached)
vr0: <VIA VT6102 Rhine II 10/100BaseTX> port 0xa400-0xa4ff mem
0xfa600000-0xfa6000ff irq 11 at device 18.0 on pci0
vr0: Quirks: 0x0
vr0: Revision: 0x78
miibus1: <MII bus> on vr0
rlphy0: <RTL8201L 10/100 media interface> PHY 1 on miibus1
rlphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
vr0: Ethernet address: 00:15:f2:43:48:7b
vr0: [ITHREAD]
cpu0: <ACPI CPU> on acpi0
acpi_button0: <Power Button> on acpi0
acpi_button1: <Sleep Button> on acpi0
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
atkbd0: [ITHREAD]
fdc0: <floppy drive controller (FDE)> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on
acpi0
fdc0: [FILTER]
fd0: <1440-KB 3.5" drive> on fdc0 drive 0
sio0: <16550A-compatible COM port> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
sio0: type 16550A
sio0: [FILTER]
orm0: <ISA Option ROMs> at iomem 0xcd000-0xcdfff,0xce000-0xcefff,0xcf000-0xd3fff
pnpid ORM0000 on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
ppc0: <Parallel port> at port 0x378-0x37f irq 7 on isa0
ppc0: SMC-like chipset (ECP/EPP/PS2/NIBBLE) in COMPATIBLE mode
ppc0: FIFO with 16/16/9 bytes threshold
ppbus0: <Parallel port bus> on ppc0
ppbus0: [ITHREAD]
lpt0: <Printer> on ppbus0
lpt0: Interrupt-driven port
ppi0: <Parallel I/O> on ppbus0
ppc0: [GIANT-LOCKED]
ppc0: [ITHREAD]
sio1: configured irq 3 not in bitmap of probed irqs 0
sio1: port may not be enabled
Timecounter "TSC" frequency 1800086355 Hz quality 800
Timecounters tick every 1.000 msec
IPsec: Initialized Security Association Processing.
ad0: 194481MB <Maxtor 6B200P0 BAH41BM0> at ata0-master UDMA133
acd0: DVDR <NEC DVD RW ND-3550A/1.05> at ata1-master UDMA33
ad4: 239372MB <Maxtor 7L250S0 BANC1G10> at ata2-master SATA150
cd0 at ata1 bus 0 target 0 lun 0
cd0: <_NEC DVD_RW ND-3550A 1.05> Removable CD-ROM SCSI-0 device
cd0: 33.000MB/s transfers
cd0: Attempt to query device size failed: NOT READY, Medium not present
WARNING: WITNESS option enabled, expect reduced performance.
Trying to mount root from ufs:/dev/ad4s1a
WARNING: / was not properly dismounted
lock order reversal:
1st 0xc0907fcc pf task mtx (pf task mtx) @
/usr/src/sys/contrib/pf/net/pf_ioctl.c:1394
2nd 0xc0973488 ifnet (ifnet) @ /usr/src/sys/net/if.c:1558
KDB: stack backtrace:
db_trace_self_wrapper(c088cf61,e658ba3c,c05eb7b6,c088f4ad,c0973488,...) at
db_trace_self_wrapper+0x26
kdb_backtrace(c088f4ad,c0973488,c0896cfd,c0896cfd,c0896b56,...) at
kdb_backtrace+0x29
witness_checkorder(c0973488,9,c0896b56,616,0,...) at witness_checkorder+0x6d6
_mtx_lock_flags(c0973488,0,c0896b56,616,c3f37a70,...) at _mtx_lock_flags+0xbc
ifunit(c3f37a70,0,c08711f2,572,c05e958e,...) at ifunit+0x2f
pfioctl(c3d2d800,c0104414,c3f37a70,3,c3f48690,...) at pfioctl+0x23b5
devfs_ioctl_f(c3f49c2c,c0104414,c3f37a70,c3b2c000,c3f48690,...) at
devfs_ioctl_f+0xe5
kern_ioctl(c3f48690,3,c0104414,c3f37a70,1000000,...) at kern_ioctl+0x243
ioctl(c3f48690,e658bcfc,c,c08bade8,c08d3630,...) at ioctl+0x134
syscall(e658bd38) at syscall+0x274
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (54, FreeBSD ELF32, ioctl), eip = 0x281aac4b, esp = 0xbfbfde5c, ebp
= 0xbfbfde88 ---
lock order reversal:
1st 0xc097830c tcp (tcp) @ /usr/src/sys/netinet/tcp_input.c:400
2nd 0xc09775d8 PFil hook read/write mutex (PFil hook read/write mutex) @
/usr/src/sys/net/pfil.c:73
KDB: stack backtrace:
db_trace_self_wrapper(c088cf61,e42579ac,c05eb7b6,c088f4ad,c09775d8,...) at
db_trace_self_wrapper+0x26
kdb_backtrace(c088f4ad,c09775d8,c0897dab,c0897dab,c0897d93,...) at
kdb_backtrace+0x29
witness_checkorder(c09775d8,1,c0897d93,49,c08a1d09,...) at
witness_checkorder+0x6d6
_rw_rlock(c09775d8,c0897d93,49,e4257a6c,0,...) at _rw_rlock+0x8e
pfil_run_hooks(c09775c0,e4257a8c,c3c31c00,2,0,...) at pfil_run_hooks+0x35
ip_output(c3c46100,0,e4257a50,0,0,0,c08e7c90,0,0,0,c067c807,c08e7c94,c08e7c9c,c8)
at ip_output+0x90f
tcp_respond(0,c3c87020,c3c87034,c3c46100,2da9088c,...) at tcp_respond+0x3e7
tcp_dropwithreset(1,3,c089c953,353,1900,...) at tcp_dropwithreset+0x152
tcp_input(c3c46100,14,c3c31c00,1,0,...) at tcp_input+0xe45
ip_input(c3c46100,c3c46100,800,c3c31c00,800,...) at ip_input+0x686
netisr_dispatch(2,c3c46100,10,3,0,...) at netisr_dispatch+0x72
ether_demux(c3c31c00,c3c46100,3,0,3,...) at ether_demux+0x2e5
ether_input(c3c31c00,c3c46100,c0aa0a74,6a9,ffffffff,...) at ether_input+0x37f
fxp_intr_body(ffffffff,0,c0aa0a74,5db,c3c33014,...) at fxp_intr_body+0x1c4
fxp_intr(c3c33000,0,c08866ae,4b6,c3b3c268,...) at fxp_intr+0xa0
ithread_loop(c3c1fa50,e4257d38,c0886453,31c,c3bef2b8,...) at ithread_loop+0x1c5
fork_exit(c0590660,c3c1fa50,e4257d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe4257d70, ebp = 0 ---
lock order reversal:
1st 0xc4013d44 udpinp (udpinp) @ /usr/src/sys/netinet/udp_usrreq.c:878
2nd 0xc09775d8 PFil hook read/write mutex (PFil hook read/write mutex) @
/usr/src/sys/net/pfil.c:73
KDB: stack backtrace:
db_trace_self_wrapper(c088cf61,e658ba14,c05eb7b6,c088f4ad,c09775d8,...) at
db_trace_self_wrapper+0x26
kdb_backtrace(c088f4ad,c09775d8,c0897dab,c0897dab,c0897d93,...) at
kdb_backtrace+0x29
witness_checkorder(c09775d8,1,c0897d93,49,c08a1d09,...) at
witness_checkorder+0x6d6
_rw_rlock(c09775d8,c0897d93,49,e658bad4,c4013ca8,...) at _rw_rlock+0x8e
pfil_run_hooks(c09775c0,e658baf4,c3d44000,2,c4013ca8,...) at pfil_run_hooks+0x35
ip_output(c3ef6100,0,e658bab8,0,0,...) at ip_output+0x90f
udp_send(c42454e0,0,c3ef6100,0,0,...) at udp_send+0x8cd
sosend_dgram(c42454e0,0,e658bbec,c3ef6100,0,...) at sosend_dgram+0x351
sosend(c42454e0,0,e658bbec,0,0,...) at sosend+0x54
kern_sendit(c3f48690,4,e658bc68,0,0,...) at kern_sendit+0xdb
sendit(0,8143023,0,0,0,...) at sendit+0xb1
sendto(c3f48690,e658bcfc,18,c08a5d78,c08d3d98,...) at sendto+0x48
syscall(e658bd38) at syscall+0x274
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (133, FreeBSD ELF32, sendto), eip = 0x2816bc83, esp = 0xbfbfd73c,
ebp = 0xbfbfd768 ---
lock order reversal:
1st 0xc423f150 tcpinp (tcpinp) @ /usr/src/sys/netinet/tcp_usrreq.c:472
2nd 0xc09775d8 PFil hook read/write mutex (PFil hook read/write mutex) @
/usr/src/sys/net/pfil.c:73
KDB: stack backtrace:
db_trace_self_wrapper(c088cf61,e65a3a30,c05eb7b6,c088f4ad,c09775d8,...) at
db_trace_self_wrapper+0x26
kdb_backtrace(c088f4ad,c09775d8,c0897dab,c0897dab,c0897d93,...) at
kdb_backtrace+0x29
witness_checkorder(c09775d8,1,c0897d93,49,c08a1d09,...) at
witness_checkorder+0x6d6
_rw_rlock(c09775d8,c0897d93,49,e65a3af0,c423f0b4,...) at _rw_rlock+0x8e
pfil_run_hooks(c09775c0,e65a3b10,c3d44000,2,c423f0b4,...) at pfil_run_hooks+0x35
ip_output(c3c94e00,0,e65a3ad4,0,0,...) at ip_output+0x90f
tcp_output(c42421d0,c3d2bc50,1d8,c423f150,c4259000,...) at tcp_output+0x140c
tcp_usr_connect(c4259000,c3d2bc50,c3d2f8c0,25,e65a3c64,...) at
tcp_usr_connect+0x11c
soconnect(c4259000,c3d2bc50,c3d2f8c0,10,16,...) at soconnect+0x52
kern_connect(c3d2f8c0,9,c3d2bc50,c3d2bc50,0,...) at kern_connect+0x59
connect(c3d2f8c0,e65a3cfc,c,c088ff65,c08d3a50,...) at connect+0x46
syscall(e65a3d38) at syscall+0x274
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (98, FreeBSD ELF32, connect), eip = 0x28161e9b, esp = 0xbfbfe71c,
ebp = 0xbfbfe868 ---
lock order reversal:
1st 0xc3eda524 tcp_sc_head (tcp_sc_head) @
/usr/src/sys/netinet/tcp_syncache.c:494
2nd 0xc09775d8 PFil hook read/write mutex (PFil hook read/write mutex) @
/usr/src/sys/net/pfil.c:73
KDB: stack backtrace:
db_trace_self_wrapper(c088cf61,e4257854,c05eb7b6,c088f4ad,c09775d8,...) at
db_trace_self_wrapper+0x26
kdb_backtrace(c088f4ad,c09775d8,c0897dab,c0897dab,c0897d93,...) at
kdb_backtrace+0x29
witness_checkorder(c09775d8,1,c0897d93,49,c08a1d09,...) at
witness_checkorder+0x6d6
_rw_rlock(c09775d8,c0897d93,49,e4257914,0,...) at _rw_rlock+0x8e
pfil_run_hooks(c09775c0,e4257934,c3c31c00,2,0,...) at pfil_run_hooks+0x35
ip_output(c3ef7a00,0,e42578f8,0,0,...) at ip_output+0x90f
syncache_respond(c426ad70,c40c0834,0,0,c40c0834,...) at syncache_respond+0x3a2
_syncache_add(c42400b4,e4257ba8,c40b3700,0,0,...) at _syncache_add+0x2b0
syncache_add(e4257b68,e4257b90,c40c0834,c42400b4,e4257ba8,...) at
syncache_add+0x38
tcp_input(c40b3700,14,c3c31c00,1,0,...) at tcp_input+0xd6b
ip_input(c40b3700,c40b3700,800,c3c31c00,800,...) at ip_input+0x686
netisr_dispatch(2,c40b3700,10,3,0,...) at netisr_dispatch+0x72
ether_demux(c3c31c00,c40b3700,3,0,3,...) at ether_demux+0x2e5
ether_input(c3c31c00,c40b3700,c0aa0a74,6a9,ffffffff,...) at ether_input+0x37f
fxp_intr_body(ffffffff,0,c0aa0a74,5db,c3c33014,...) at fxp_intr_body+0x1c4
fxp_intr(c3c33000,0,c08866ae,4b6,c3b3c268,...) at fxp_intr+0xa0
ithread_loop(c3c1fa50,e4257d38,c0886453,31c,c3bef2b8,...) at ithread_loop+0x1c5
fork_exit(c0590660,c3c1fa50,e4257d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe4257d70, ebp = 0 ---
lock order reversal:
1st 0xc09786cc udp (udp) @ /usr/src/sys/netinet/udp_usrreq.c:395
2nd 0xc09775d8 PFil hook read/write mutex (PFil hook read/write mutex) @
/usr/src/sys/net/pfil.c:73
KDB: stack backtrace:
db_trace_self_wrapper(c088cf61,e42579b8,c05eb7b6,c088f4ad,c09775d8,...) at
db_trace_self_wrapper+0x26
kdb_backtrace(c088f4ad,c09775d8,c0897dab,c0897dab,c0897d93,...) at
kdb_backtrace+0x29
witness_checkorder(c09775d8,1,c0897d93,49,c08a1d09,...) at
witness_checkorder+0x6d6
_rw_rlock(c09775d8,c0897d93,49,e4257a78,0,...) at _rw_rlock+0x8e
pfil_run_hooks(c09775c0,e4257a98,c3c31c00,2,0,...) at pfil_run_hooks+0x35
ip_output(c3efae00,0,e4257a5c,0,0,...) at ip_output+0x90f
icmp_reflect(c40c6020,c3efaec8,14,c3efaf00,c40c6020,...) at icmp_reflect+0x3df
icmp_error(c40b4d00,3,3,0,0,...) at icmp_error+0x3bd
udp_input(c40b4d00,14,c3c31c00,1,0,...) at udp_input+0x5ea
ip_input(c40b4d00,c40b4d00,800,c3c31c00,800,...) at ip_input+0x686
netisr_dispatch(2,c40b4d00,10,3,0,...) at netisr_dispatch+0x72
ether_demux(c3c31c00,c40b4d00,3,0,3,...) at ether_demux+0x2e5
ether_input(c3c31c00,c40b4d00,c0aa0a74,6a9,ffffffff,...) at ether_input+0x37f
fxp_intr_body(ffffffff,0,c0aa0a74,5db,c3c33014,...) at fxp_intr_body+0x1c4
fxp_intr(c3c33000,0,c08866ae,4b6,c3b3c268,...) at fxp_intr+0xa0
ithread_loop(c3c1fa50,e4257d38,c0886453,31c,c3bef2b8,...) at ithread_loop+0x1c5
fork_exit(c0590660,c3c1fa50,e4257d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe4257d70, ebp = 0 ---
kernel backtrace:
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "i386-marcel-freebsd"...
Unread portion of the kernel message buffer:
panic: _rw_rlock (tcp): wlock already held @
/usr/src/sys/contrib/pf/net/pf.c:3016
cpuid = 0
KDB: stack backtrace:
db_trace_self_wrapper(c088cf61,e6846220,c05ae7df,c08b659d,0,...) at
db_trace_self_wrapper+0x26
kdb_backtrace(c08b659d,0,c0889c7e,e684622c,0,...) at kdb_backtrace+0x29
panic(c0889c7e,c085a754,c088f55e,c087092d,bc8,...) at panic+0x10f
_rw_rlock(c097830c,c087092d,bc8,c08d9624,c087092d,...) at _rw_rlock+0x73
pf_socket_lookup(2,e68463dc,0,cc4,3,...) at pf_socket_lookup+0x208
pf_test_tcp(e6846444,e6846440,2,c3efee00,c3c8e900,...) at pf_test_tcp+0x142
pf_test6(2,c3d44000,e68464a0,0,0,...) at pf_test6+0x8a0
pf_check6_out(0,e68464a0,c3d44000,2,0,...) at pf_check6_out+0x47
pfil_run_hooks(c097ad00,e6846638,c3d44000,2,0,...) at pfil_run_hooks+0x88
ip6_output(c3c8e900,0,e6846618,0,0,...) at ip6_output+0x122e
pf_send_tcp(c4fcfe00,c41259b4,1c,c4fcfe5c,c4fcfe4c,...) at pf_send_tcp+0x6dd
pf_test_tcp(e68468e8,e68468e4,2,c3f20900,c4fcfe00,...) at pf_test_tcp+0xcef
pf_test6(2,c3f06400,e6846944,0,c446b7bc,...) at pf_test6+0x8a0
pf_check6_out(0,e6846944,c3f06400,2,c446b7bc,...) at pf_check6_out+0x47
pfil_run_hooks(c097ad00,e6846adc,c3f06400,2,c446b7bc,...) at pfil_run_hooks+0x88
ip6_output(c4fcfe00,0,e6846abc,0,0,...) at ip6_output+0x122e
tcp_output(c45553a0,c447e7c0,201,c446b858,c45553a0,...) at tcp_output+0x137e
tcp6_usr_connect(c50cd340,c447e7c0,c4eed690,25,e6846c64,...) at
tcp6_usr_connect+0x171
soconnect(c50cd340,c447e7c0,c4eed690,1c,16,...) at soconnect+0x52
kern_connect(c4eed690,3,c447e7c0,c447e7c0,0,...) at kern_connect+0x59
connect(c4eed690,e6846cfc,c,c08a288e,c08d3a50,...) at connect+0x46
syscall(e6846d38) at syscall+0x274
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (98, FreeBSD ELF32, connect), eip = 0x282e6e9b, esp = 0xbfbfe7ec,
ebp = 0xbfbfe848 ---
KDB: enter: panic
shared rw PFil hook read/write mutex r = 1 (0xc097ad18) locked @
/usr/src/sys/net/pfil.c:73
exclusive rw tcpinp r = 0 (0xc446b858) locked @
/usr/src/sys/netinet/tcp_usrreq.c:513
exclusive rw tcp r = 0 (0xc097830c) locked @
/usr/src/sys/netinet/tcp_usrreq.c:510
exclusive sx so_rcv_sx r = 0 (0xc452fbec) locked @
/usr/src/sys/kern/uipc_sockbuf.c:148
exclusive sx so_rcv_sx r = 0 (0xc483cbec) locked @
/usr/src/sys/kern/uipc_sockbuf.c:148
exclusive sx so_rcv_sx r = 0 (0xc4e89bec) locked @
/usr/src/sys/kern/uipc_sockbuf.c:148
exclusive sx so_rcv_sx r = 0 (0xc4e8970c) locked @
/usr/src/sys/kern/uipc_sockbuf.c:148
exclusive sx so_rcv_sx r = 0 (0xc483c22c) locked @
/usr/src/sys/kern/uipc_sockbuf.c:148
exclusive sx so_rcv_sx r = 0 (0xc480d70c) locked @
/usr/src/sys/kern/uipc_sockbuf.c:148
exclusive sx so_rcv_sx r = 0 (0xc4e8a08c) locked @
/usr/src/sys/kern/uipc_sockbuf.c:148
exclusive sx so_rcv_sx r = 0 (0xc4e8a56c) locked @
/usr/src/sys/kern/uipc_sockbuf.c:148
exclusive sx so_rcv_sx r = 0 (0xc41a456c) locked @
/usr/src/sys/kern/uipc_sockbuf.c:148
exclusive sx so_rcv_sx r = 0 (0xc41c156c) locked @
/usr/src/sys/kern/uipc_sockbuf.c:148
exclusive sx so_rcv_sx r = 0 (0xc41c18ac) locked @
/usr/src/sys/kern/uipc_sockbuf.c:148
exclusive sx so_rcv_sx r = 0 (0xc41c1bec) locked @
/usr/src/sys/kern/uipc_sockbuf.c:148
exclusive sx so_rcv_sx r = 0 (0xc41f108c) locked @
/usr/src/sys/kern/uipc_sockbuf.c:148
shared rw udpinp r = 0 (0xc400f63c) locked @
/usr/src/sys/netinet/udp_usrreq.c:878
Uptime: 16h23m36s
Physical memory: 1015 MB
Dumping 166 MB: 151 135 119 103 87 71 55 39 23 7
Reading symbols from /boot/kernel/if_em.ko...Reading symbols from
/boot/kernel/if_em.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/if_em.ko
Reading symbols from /boot/kernel/if_fxp.ko...Reading symbols from
/boot/kernel/if_fxp.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/if_fxp.ko
Reading symbols from /boot/kernel/miibus.ko...Reading symbols from
/boot/kernel/miibus.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/miibus.ko
Reading symbols from /boot/kernel/if_vr.ko...Reading symbols from
/boot/kernel/if_vr.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/if_vr.ko
Reading symbols from /boot/kernel/ulpt.ko...Reading symbols from
/boot/kernel/ulpt.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/ulpt.ko
Reading symbols from /boot/kernel/accf_http.ko...Reading symbols from
/boot/kernel/accf_http.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/accf_http.ko
Reading symbols from /boot/kernel/acpi.ko...Reading symbols from
/boot/kernel/acpi.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/acpi.ko
#0 doadump () at pcpu.h:196
196 pcpu.h: No such file or directory.
in pcpu.h
(kgdb) bt
#0 doadump () at pcpu.h:196
#1 0xc05ae54c in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:418
#2 0xc05ae816 in panic (fmt=Variable "fmt" is not available.
) at /usr/src/sys/kern/kern_shutdown.c:572
#3 0xc05acf63 in _rw_rlock (rw=0xc097830c, file=0xc087092d
"/usr/src/sys/contrib/pf/net/pf.c", line=3016)
at /usr/src/sys/kern/kern_rwlock.c:253
#4 0xc0473e58 in pf_socket_lookup (direction=2, pd=0xe68463dc, inp_arg=0x0) at
/usr/src/sys/contrib/pf/net/pf.c:3016
#5 0xc047dd62 in pf_test_tcp (rm=0xe6846444, sm=0xe6846440, direction=2,
kif=0xc3efee00, m=0xc3c8e900, off=40,
h=0xc3c8e944, pd=0xe68463dc, am=0xe6846448, rsm=0xe684643c, ifq=0x0,
inp=0x0)
at /usr/src/sys/contrib/pf/net/pf.c:3270
#6 0xc04816c0 in pf_test6 (dir=2, ifp=0xc3d44000, m0=0xe68464a0, eh=0x0,
inp=0x0)
at /usr/src/sys/contrib/pf/net/pf.c:7368
#7 0xc0484e37 in pf_check6_out (arg=0x0, m=0xe68464a0, ifp=0xc3d44000, dir=2,
inp=0x0)
at /usr/src/sys/contrib/pf/net/pf_ioctl.c:3739
#8 0xc0657618 in pfil_run_hooks (ph=0xc097ad00, mp=0xe6846638, ifp=0xc3d44000,
dir=2, inp=0x0)
at /usr/src/sys/net/pfil.c:78
#9 0xc07034fe in ip6_output (m0=0xc3c8e900, opt=0x0, ro=0xe6846618,
flags=Variable "flags" is not available.
) at /usr/src/sys/netinet6/ip6_output.c:853
#10 0xc0477dad in pf_send_tcp (replyto=0xc4fcfe00, r=0xc41259b4, af=28 '\034',
saddr=0xc4fcfe5c, daddr=0xc4fcfe4c,
sport=20480, dport=46591, seq=0, ack=1170313007, flags=20 '\024', win=0,
mss=0, ttl=0 '\0', tag=1, rtag=0, eh=0x0,
ifp=0xc3f06400) at /usr/src/sys/contrib/pf/net/pf.c:1978
#11 0xc047e90f in pf_test_tcp (rm=0xe68468e8, sm=0xe68468e4, direction=2,
kif=0xc3f20900, m=0xc4fcfe00, off=40,
h=0xc4fcfe44, pd=0xe6846880, am=0xe68468ec, rsm=0xe68468e0, ifq=0x0,
inp=0xc446b7bc)
at /usr/src/sys/contrib/pf/net/pf.c:3424
#12 0xc04816c0 in pf_test6 (dir=2, ifp=0xc3f06400, m0=0xe6846944, eh=0x0,
inp=0xc446b7bc)
at /usr/src/sys/contrib/pf/net/pf.c:7368
#13 0xc0484e37 in pf_check6_out (arg=0x0, m=0xe6846944, ifp=0xc3f06400, dir=2,
inp=0xc446b7bc)
at /usr/src/sys/contrib/pf/net/pf_ioctl.c:3739
#14 0xc0657618 in pfil_run_hooks (ph=0xc097ad00, mp=0xe6846adc, ifp=0xc3f06400,
dir=2, inp=0xc446b7bc)
at /usr/src/sys/net/pfil.c:78
#15 0xc07034fe in ip6_output (m0=0xc4fcfe00, opt=0x0, ro=0xe6846abc,
flags=Variable "flags" is not available.
) at /usr/src/sys/netinet6/ip6_output.c:853
#16 0xc06debbe in tcp_output (tp=0xc45553a0) at
/usr/src/sys/netinet/tcp_output.c:1114
#17 0xc06ea5d1 in tcp6_usr_connect (so=0xc50cd340, nam=0xc447e7c0,
td=0xc4eed690) at tcp_offload.h:257
#18 0xc060b002 in soconnect (so=0xc50cd340, nam=0xc447e7c0, td=0xc4eed690) at
/usr/src/sys/kern/uipc_socket.c:771
#19 0xc06129e9 in kern_connect (td=0xc4eed690, fd=3, sa=0xc447e7c0) at
/usr/src/sys/kern/uipc_syscalls.c:570
#20 0xc0612b56 in connect (td=0xc4eed690, uap=0xe6846cfc) at
/usr/src/sys/kern/uipc_syscalls.c:534
#21 0xc083a2d4 in syscall (frame=0xe6846d38) at
/usr/src/sys/i386/i386/trap.c:1090
#22 0xc0821220 in Xint0x80_syscall () at /usr/src/sys/i386/i386/exception.s:255
#23 0x00000033 in ?? ()
Previous frame inner to this frame (corrupt stack?)
(kgdb)
>How-To-Repeat:
>Fix:
>Release-Note:
>Audit-Trail:
>Unformatted:
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20080917123323.EFF8F22CCB>