From owner-freebsd-bugs@FreeBSD.ORG Fri Aug 31 11:30:02 2012 Return-Path: Delivered-To: freebsd-bugs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B7D50106568A for ; Fri, 31 Aug 2012 11:30:02 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 895968FC1E for ; Fri, 31 Aug 2012 11:30:02 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.5/8.14.5) with ESMTP id q7VBU2O9099978 for ; Fri, 31 Aug 2012 11:30:02 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.5/8.14.5/Submit) id q7VBU2cX099975; Fri, 31 Aug 2012 11:30:02 GMT (envelope-from gnats) Resent-Date: Fri, 31 Aug 2012 11:30:02 GMT Resent-Message-Id: <201208311130.q7VBU2cX099975@freefall.freebsd.org> Resent-From: FreeBSD-gnats-submit@FreeBSD.org (GNATS Filer) Resent-To: freebsd-bugs@FreeBSD.org Resent-Reply-To: FreeBSD-gnats-submit@FreeBSD.org, "Eugene M. Zheganin" Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id AD0041065674 for ; Fri, 31 Aug 2012 11:21:06 +0000 (UTC) (envelope-from nobody@FreeBSD.org) Received: from red.freebsd.org (red.freebsd.org [IPv6:2001:4f8:fff6::22]) by mx1.freebsd.org (Postfix) with ESMTP id 969548FC25 for ; Fri, 31 Aug 2012 11:21:06 +0000 (UTC) Received: from red.freebsd.org (localhost [127.0.0.1]) by red.freebsd.org (8.14.4/8.14.4) with ESMTP id q7VBL60Q027167 for ; Fri, 31 Aug 2012 11:21:06 GMT (envelope-from nobody@red.freebsd.org) Received: (from nobody@localhost) by red.freebsd.org (8.14.4/8.14.4/Submit) id q7VBL6LJ027166; Fri, 31 Aug 2012 11:21:06 GMT (envelope-from nobody) Message-Id: <201208311121.q7VBL6LJ027166@red.freebsd.org> Date: Fri, 31 Aug 2012 11:21:06 GMT From: "Eugene M. Zheganin" To: freebsd-gnats-submit@FreeBSD.org X-Send-Pr-Version: www-3.1 Cc: Subject: kern/171202: [lor] multiple LOR's during the startup X-BeenThere: freebsd-bugs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Bug reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 31 Aug 2012 11:30:02 -0000 >Number: 171202 >Category: kern >Synopsis: [lor] multiple LOR's during the startup >Confidential: no >Severity: non-critical >Priority: low >Responsible: freebsd-bugs >State: open >Quarter: >Keywords: >Date-Required: >Class: sw-bug >Submitter-Id: current-users >Arrival-Date: Fri Aug 31 11:30:02 UTC 2012 >Closed-Date: >Last-Modified: >Originator: Eugene M. Zheganin >Release: 9.1-PRERELEASE >Organization: Vivat-Trade LLC >Environment: FreeBSD taiga 9.1-PRERELEASE FreeBSD 9.1-PRERELEASE #0: Fri Aug 17 13:55:19 YEKT 2012 emz@taiga:/usr/obj/usr/src/sys/TAIGADBG amd64 >Description: During the investigation of the possible reasons of periodic hangups of the server, I build the WITNESS/INVARIANTS kernel. I receive multiple LORs during the startup of the server. bz@ kindly told me it's worth to report. So I'm doing this. I'm not aware if these LOR's actually are connected with my server's hangups. Target machine is a IBM x3650 server. It runs IPv4, IPV6 via gif tunnel to the tunnelbroker, pf with a hundred of rules, pf nat, OSPFv6 with net/bird6, OSPFv4 with net/quagga, zfs. # dmesg Copyright (c) 1992-2012 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 9.1-PRERELEASE #0: Fri Aug 17 13:55:19 YEKT 2012 emz@taiga:/usr/obj/usr/src/sys/TAIGADBG amd64 WARNING: WITNESS option enabled, expect reduced performance. CPU: Intel(R) Xeon(R) CPU E5405 @ 2.00GHz (1995.04-MHz K8-class CPU) Origin = "GenuineIntel" Id = 0x10676 Family = 6 Model = 17 Stepping = 6 Features=0xbfebfbff Features2=0xce33d AMD Features=0x20100800 AMD Features2=0x1 TSC: P-state invariant, performance statistics real memory = 4294967296 (4096 MB) avail memory = 4089982976 (3900 MB) Event timer "LAPIC" quality 400 ACPI APIC Table: FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs FreeBSD/SMP: 1 package(s) x 4 core(s) cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 1 cpu2 (AP): APIC ID: 2 cpu3 (AP): APIC ID: 3 ioapic0 irqs 0-23 on motherboard kbd1 at kbdmux0 acpi0: on motherboard acpi0: Power Button (fixed) ipmi0: KCS mode found at io 0xca8 on acpi cpu0: on acpi0 cpu1: on acpi0 cpu2: on acpi0 cpu3: on acpi0 attimer0: port 0x40-0x43 irq 0 on acpi0 Timecounter "i8254" frequency 1193182 Hz quality 0 Event timer "i8254" frequency 1193182 Hz quality 100 atrtc0: port 0x70-0x73 irq 8 on acpi0 Event timer "RTC" frequency 32768 Hz quality 0 hpet0: iomem 0xfed00000-0xfed003ff on acpi0 Timecounter "HPET" frequency 14318180 Hz quality 950 Event timer "HPET" frequency 14318180 Hz quality 450 Event timer "HPET1" frequency 14318180 Hz quality 440 Event timer "HPET2" frequency 14318180 Hz quality 440 Timecounter "ACPI-fast" frequency 3579545 Hz quality 900 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x588-0x58b on acpi0 pcib0: on acpi0 pcib0: Length mismatch for 3 range: c0000000 vs ffffffffc0000000 pci0: on pcib0 pcib1: at device 2.0 on pci0 pci26: on pcib1 pcib2: at device 0.0 on pci26 pci27: on pcib2 pcib3: at device 0.0 on pci27 pci28: on pcib3 pcib4: irq 17 at device 1.0 on pci27 pci36: on pcib4 pcib5: at device 0.3 on pci26 pci37: on pcib5 pcib6: at device 3.0 on pci0 pci4: on pcib6 aac0: port 0x5000-0x50ff mem 0xc9e00000-0xc9ffffff,0xc7fe0000-0xc7ffffff irq 17 at device 0.0 on pci4 aac0: Enabling 64-bit address support aac0: Enable Raw I/O aac0: Enable 64-bit array aac0: New comm. interface enabled aac0: ServeRAID 8k , aac driver 2.1.9-1 pcib7: at device 4.0 on pci0 pci16: on pcib7 pcib8: at device 5.0 on pci0 pci69: on pcib8 pcib9: at device 6.0 on pci0 pci7: on pcib9 pcib10: at device 7.0 on pci0 pci68: on pcib10 pci0: at device 8.0 (no driver attached) pcib11: at device 28.0 on pci0 pci2: on pcib11 pcib12: at device 0.0 on pci2 pci3: on pcib12 bce0: mem 0xce000000-0xcfffffff irq 16 at device 0.0 on pci3 miibus0: on bce0 brgphy0: PHY 1 on miibus0 brgphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow bce0: Ethernet address: 00:1a:64:c6:a8:7c bce0: ASIC (0x57081020); Rev (B2); Bus (PCI-X, 64-bit, 133MHz); B/C (4.6.1); Bufs (RX:2;TX:2;PG:0); Flags (MFW); MFW (ipms 1.6.0) Coal (RX:6,6,18,18; TX:20,20,80,80) pcib13: at device 28.1 on pci0 pci5: on pcib13 pcib14: at device 0.0 on pci5 pci6: on pcib14 bce1: mem 0xca000000-0xcbffffff irq 17 at device 0.0 on pci6 miibus1: on bce1 brgphy1: PHY 1 on miibus1 brgphy1: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow bce1: Ethernet address: 00:1a:64:c6:a8:7e bce1: ASIC (0x57081020); Rev (B2); Bus (PCI-X, 64-bit, 133MHz); B/C (4.6.1); Bufs (RX:2;TX:2;PG:0); Flags (MFW); MFW (ipms 1.6.0) Coal (RX:6,6,18,18; TX:20,20,80,80) uhci0: port 0x2200-0x221f irq 23 at device 29.0 on pci0 usbus0 on uhci0 uhci1: port 0x2600-0x261f irq 22 at device 29.1 on pci0 usbus1 on uhci1 uhci2: port 0x2a00-0x2a1f irq 23 at device 29.2 on pci0 usbus2 on uhci2 uhci3: port 0x2e00-0x2e1f irq 22 at device 29.3 on pci0 usbus3 on uhci3 ehci0: mem 0xf9000000-0xf90003ff irq 23 at device 29.7 on pci0 usbus4: EHCI version 1.0 usbus4 on ehci0 pcib15: at device 30.0 on pci0 pci1: on pcib15 vgapci0: port 0x4000-0x40ff mem 0xd0000000-0xd7ffffff,0xdfff0000-0xdfffffff irq 22 at device 6.0 on pci1 isab0: at device 31.0 on pci0 isa0: on isab0 atapci0: port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x480-0x48f at device 31.2 on pci0 ata0: at channel 0 on atapci0 ata1: at channel 1 on atapci0 pci0: at device 31.3 (no driver attached) uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 uart0: console (19200,n,8,1) ipmi1: on isa0 device_attach: ipmi1 attach returned 16 ipmi1: on isa0 device_attach: ipmi1 attach returned 16 orm0: at iomem 0xc0000-0xcafff,0xcb000-0xd07ff on isa0 sc0: at flags 0x100 on isa0 sc0: VGA <16 virtual consoles, flags=0x100> vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 atkbdc0: at port 0x60,0x64 on isa0 atkbd0: irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] ppc0: cannot reserve I/O port range p4tcc0: on cpu0 p4tcc1: on cpu1 p4tcc2: on cpu2 p4tcc3: on cpu3 ZFS NOTICE: Prefetch is disabled by default if less than 4GB of RAM is present; to enable, add "vfs.zfs.prefetch_disable=0" to /boot/loader.conf. ZFS filesystem version 5 ZFS storage pool version 28 Timecounters tick every 1.000 msec usbus0: 12Mbps Full Speed USB v1.0 usbus1: 12Mbps Full Speed USB v1.0 ugen0.1: at usbus0 uhub0: on usbus0 ugen1.1: at usbus1 uhub1: on usbus1 usbus2: 12Mbps Full Speed USB v1.0 usbus3: 12Mbps Full Speed USB v1.0 ugen2.1: at usbus2 uhub2: on usbus2 ugen3.1: at usbus3 uhub3: on usbus3 usbus4: 480Mbps High Speed USB v2.0 ugen4.1: at usbus4 uhub4: on usbus4 uhub0: 2 ports with 2 removable, self powered uhub1: 2 ports with 2 removable, self powered uhub2: 2 ports with 2 removable, self powered uhub3: 2 ports with 2 removable, self powered ipmi0: IPMI device rev. 0, firmware rev. 1.49, version 2.0 ipmi0: Number of channels 3 ipmi0: Attached watchdog aacd0 on aac0 aacd0: 279800MB (573030400 sectors) cd0 at ata1 bus 0 scbus1 target 0 lun 0 SMP: AP CPU #3 Launched! SMP: AP CPU #1 Launched! cd0: Removable CD-ROM SCSI-0 device SMP: AP CPU #2 Launched! cd0: 33.300MB/s transfers (UDMA2, ATAPI 12bytes, PIO 65534bytes) cd0: cd present [1512048 x 2048 byte records] Timecounter "TSC-low" frequency 15586274 Hz quality 1000 WARNING: WITNESS option enabled, expect reduced performance. Root mount waiting for: usbus4 uhub4: 8 ports with 8 removable, self powered Root mount waiting for: usbus4 Trying to mount root from zfs:zfsroot []... ugen2.2: at usbus2 ukbd0: on usbus2 kbd2 at ukbd0 uhid0: on usbus2 bce0: Gigabit link up! WARNING: attempt to domain_add(netgraph) after domainfinalize() lock order reversal: 1st 0xffffffff80b34620 pf task mtx (pf task mtx) @ /usr/src/sys/contrib/pf/net/pf_ioctl.c:1581 2nd 0xffffffff80d4ac00 ifnet_rw (ifnet_rw) @ /usr/src/sys/net/if.c:2022 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a kdb_backtrace() at kdb_backtrace+0x37 _witness_debugger() at _witness_debugger+0x2c witness_checkorder() at witness_checkorder+0x853 _rw_rlock() at _rw_rlock+0x81 ifunit() at ifunit+0x22 pfioctl() at pfioctl+0x129b devfs_ioctl_f() at devfs_ioctl_f+0x7a kern_ioctl() at kern_ioctl+0xcd sys_ioctl() at sys_ioctl+0xfd amd64_syscall() at amd64_syscall+0x2fa Xfast_syscall() at Xfast_syscall+0xf7 --- syscall (54, FreeBSD ELF64, sys_ioctl), rip = 0x800d81cbc, rsp = 0x7fffffffa3d8, rbp = 0x7fffffffa530 --- altq: emulate 256000000Hz cpu clock lock order reversal: 1st 0xfffffe00079aabd8 zfs (zfs) @ /usr/src/sys/kern/vfs_mount.c:849 2nd 0xfffffe0007465bd8 devfs (devfs) @ /usr/src/sys/kern/vfs_subr.c:2158 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a kdb_backtrace() at kdb_backtrace+0x37 _witness_debugger() at _witness_debugger+0x2c witness_checkorder() at witness_checkorder+0x853 __lockmgr_args() at __lockmgr_args+0x1184 vop_stdlock() at vop_stdlock+0x39 VOP_LOCK1_APV() at VOP_LOCK1_APV+0x9b _vn_lock() at _vn_lock+0x47 vget() at vget+0x7b devfs_allocv() at devfs_allocv+0x13f devfs_root() at devfs_root+0x4d vfs_donmount() at vfs_donmount+0xb02 sys_nmount() at sys_nmount+0x66 amd64_syscall() at amd64_syscall+0x2fa Xfast_syscall() at Xfast_syscall+0xf7 --- syscall (378, FreeBSD ELF64, sys_nmount), rip = 0x800a8d6fc, rsp = 0x7fffffffccc8, rbp = 0x801009048 --- lock order reversal: 1st 0xffffffff80d50d48 PFil hook read/write mutex (PFil hook read/write mutex) @ /usr/src/sys/net/pfil.c:77 2nd 0xffffffff80d4e4f8 pcbinfohash (pcbinfohash) @ /usr/src/sys/netinet6/in6_pcb.c:1110 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a kdb_backtrace( ) at kdb_backtrace+0x37 _witness_debugger() at _witness_debugger+0x2c witness_checkorder() at witness_checkorder+0x853 _rw_rlock() at _rw_rlock+0x81 in6_pcblookup_hash() at in6_pcblookup_hash+0x47 pf_socket_lookup() at pf_socket_lookup+0x191 pf_test_rule() at pf_test_rul e+0x9fb pf_test6() at pf_test6+0xf8a pf_check6_in() at pf_check6_in+0x35 pfil_run_hooks() at pfil_run_hooks+0xd2 ip6_input() at ip6_input+0x7f9 swi_net() at swi_net+0x161 intr_eve nt_execute_handlers() at intr_event_execute_handlers+0x6a ithread_loop() at ithread_loop+0xb4 fork_exit() at fork_exit+0x135 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff800024ccf0, rbp = 0 --- lock order reversal: 1st 0xffffffff80d50d48 PFil hook read/write mutex (PFil hook read/write mutex) @ /usr/src/sys/net/pfil.c:77 2nd 0xfffffe00074ed168 udpinp (udpinp) @ /usr/src/sys/netinet6/in6_pcb.c:1121 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a kdb_backtrace() at kdb_backtrace+0x37 _witness_debugger() at _witness_debugger+0x2c witness _ che ckor d e r ( ) at witness_checkorder+0x853 _rw_r lock () at _rw_rlo c k+0x81 in6_pcblookup_hash() at in6_pcblookup_hash+0xb7 pf_socket_l o okup() at pf_socket_lookup+0x24e pf_test_rule() at pf_test_rule+0x9fb pf_test6() at pf_test6+0xf8a pf_check6_in() at pf_check6_in+0x35 pfil_run_hooks() at pfil_run_hooks+0xd2 ip6_input() at ip6_input+0x7f9 swi_net() at swi_net+0x161 intr_event_execute_handlers() at intr_event_execute_handlers+0x6a ithread_loop() at ithread_loop+0xb4 fork_exit() at fork_exit+0x135 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff800024ccf0, rbp = 0 --- lock order reversal: 1st 0xffffffff80d50d48 PFil hook read/write mutex (PFil hook read/write mutex) @ /usr/src/sys/net/pfil.c:77 2nd 0xfffffe0007b6af30 tcpinp (tcpinp) @ /usr/src/sys/netinet6/in6_pcb.c:1121 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a kdb_backtrace() at kdb_backtrace+0x37 _witness_debugger() at _witness_debugger+0x2c witness_checkorder() at witness_checkorder+0x853 _rw_rlock() at _rw_rlock+0x81 in6_pcblookup_hash() at in6_pcblookup_hash+0xb7 pf_socket_lookup() at pf_socket_lookup+0x191 pf_test_rule() at pf_test_rule+0x9fb pf_test6() at pf_test6+0xf8a pf_check6_in() at pf_check6_in+0x35 pfil_run_hooks() at pfil_run_hooks+0xd2 ip6_input() at ip6_input+0x7f9 swi_net() at swi_net+0x161 intr_event_execute_handlers() at intr_event_execute_handlers+0x6a ithread_loop() at ithread_loop+0xb4 fork_exit() at fork_exit+0x135 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff800024ccf0, rbp = 0 --- lock order reversal: 1st 0xffffffff80d4cde0 in_ifaddr_lock (in_ifaddr_lock) @ /usr/src/sys/netinet/if_ether.c:569 2nd 0xfffffe0005acb720 carp_if (carp_if) @ /usr/src/sys/netinet/ip_carp.c:1195 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a kdb_backtrace() at kdb_backtrace+0x37 _witness_debugger() at _witness_debugger+0x2c witness_checkorder() at witness_checkorder+0x853 _mtx_lock_flags() at _mtx_lock_flags+0x85 carp_iamatch() at carp_iamatch+0x39 arpintr() at arpintr+0x368 netisr_dispatch_src() at netisr_dispatch_src+0x152 ether_demux() at ether_demux+0x17d ether_nh_input() at ether_nh_input+0x20e netisr_dispatch_src() at netisr_dispatch_src+0x152 ether_demux() at ether_demux+0x86 ether_nh_input() at ether_nh_input+0x20e netisr_dispatch_src() at netisr_dispatch_src+0x152 bce_intr() at bce_intr+0x46a intr_event_execute_handlers() at intr_event_execute_handlers+0x6a ithread_loop() at ithread_loop+0xb4 fork_exit() at fork_exit+0x135 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff811a901cf0, rbp = 0 --- vlan1: promiscuous mode enabled carp0: INIT -> BACKUP carp0: link state changed to DOWN carp0: link state changed to DOWN carp0: INIT -> BACKUP carp0: link state changed to DOWN carp0: link state changed to DOWN carp0: INIT -> BACKUP carp0: link state changed to DOWN carp0: BACKUP -> MASTER (preempting a slower master) carp0: link state changed to UP vlan15: promiscuous mode enabled carp2: INIT -> BACKUP carp2: link state changed to DOWN carp2: INIT -> BACKUP carp2: link state changed to DOWN carp2: link state changed to DOWN carp2: BACKUP -> MASTER (preempting a slower master) carp2: link state changed to UP lock order reversal: 1st 0xfffffe00a835d458 zfs (zfs) @ /usr/src/sys/kern/vfs_mount.c:849 2nd 0xfffffe00096febd8 newnfs (newnfs) @ /usr/src/sys/kern/vfs_subr.c:2158 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a kdb_backtrace() at kdb_backtrace+0x37 _witness_debugger() at _witness_debugger+0x2c witness_checkorder() at witness_checkorder+0x853 __lockmgr_args() at __lockmgr_args+0x1184 nfs_lock1() at nfs_lock1+0x4a VOP_LOCK1_APV() at VOP_LOCK1_APV+0x9b _vn_lock() at _vn_lock+0x47 vget() at vget+0x7b vfs_hash_get() at vfs_hash_get+0xd5 ncl_nget() at ncl_nget+0xd5 nfs_root() at nfs_root+0x3d vfs_donmount() at vfs_donmount+0xb02 sys_nmount() at sys_nmount+0x66 amd64_syscall() at amd64_syscall+0x2fa Xfast_syscall() at Xfast_syscall+0xf7 --- syscall (378, FreeBSD ELF64, sys_nmount), rip = 0x80087c6fc, rsp = 0x7fffffffd1b8, rbp = 0x7fffffffd26c --- arp: 192.168.6.57 moved from 14:8f:c6:bc:e0:b7 to 00:08:ca:65:7a:c5 on vlan10 arp: 192.168.6.40 moved from 40:30:04:4b:ed:ed to 14:8f:c6:bc:e0:b7 on vlan10 arp: 192.168.6.44 moved from 40:30:04:4b:ed:ed to b8:ff:61:31:1e:d4 on vlan10 cannot forward src fe80:b::9cb9:2755:f0a0:488b, dst fec0:0:0:ffff::3, nxt 17, rcvif vlan1, outif gif0 cannot forward src fe80:12::617a:1076:42b2:68a2, dst fec0:0:0:ffff::3, nxt 17, rcvif vlan15, outif gif0 cannot forward src fe80:12::617a:1076:42b2:68a2, dst fec0:0:0:ffff::2, nxt 17, rcvif vlan15, outif gif0 cannot forward src fe80:12::617a:1076:42b2:68a2, dst fec0:0:0:ffff::1, nxt 17, rcvif vlan15, outif gif0 cannot forward src fe80:12::95b6:d17:e8fc:4e39, dst fec0:0:0:ffff::3, nxt 17, rcvif vlan15, outif gif0 cannot forward src fe80:b::f10a:ed10:819f:59d6, dst fec0:0:0:ffff::3, nxt 17, rcvif vlan1, outif gif0 cannot forward src fe80:12::1455:1c89:e03d:ac90, dst fec0:0:0:ffff::3, nxt 17, rcvif vlan15, outif gif0 cannot forward src fe80:12::fd85:e73d:10b9:b225, dst fec0:0:0:ffff::3, nxt 17, rcvif vlan15, outif gif0 cannot forward src fe80:12::4cae:74c6:6a20:eb7, dst fec0:0:0:ffff::3, nxt 17, rcvif vlan15, outif gif0 cannot forward src fe80:12::f552:7bc3:5cd2:e591, dst fec0:0:0:ffff::3, nxt 17, rcvif vlan15, outif gif0 arp: 192.168.8.235 moved from 5c:b5:24:13:ee:ba to 94:63:d1:c6:02:9d on vlan11 cannot forward src fe80:12::b576:9a8a:fbe2:17e8, dst fec0:0:0:ffff::3, nxt 17, rcvif vlan15, outif gif0 cannot forward src fe80:12::4cae:74c6:6a20:eb7, dst fec0:0:0:ffff::3, nxt 17, rcvif vlan15, outif gif0 arp: 192.168.8.212 moved from 00:23:6c:00:e8:b9 to 38:16:d1:c6:e6:eb on vlan11 lock order reversal: 1st 0xffffffff80d50d48 PFil hook read/write mutex (PFil hook read/write mutex) @ /usr/src/sys/net/pfil.c:77 2nd 0xfffffe0007467e08 gif softc (gif softc) @ /usr/src/sys/net/if_gif.c:430 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a kdb_backtrace() at kdb_backtrace+0x37 _witness_debugger() at _witness_debugger+0x2c witness_checkorder() at witness_checkorder+0x853 _mtx_lock_flags() at _mtx_lock_flags+0x85 gif_output() at gif_output+0xcb nd6_output_lle() at nd6_output_lle+0x457 nd6_output() at nd6_output+0x18 ip6_output() at ip6_output+0x122f pf_send_tcp() at pf_send_tcp+0x3ff pf_tcp_track_full() at pf_tcp_track_full+0xb15 pf_test_state_tcp() at pf_test_state_tcp+0x167 pf_test6() at pf_test6+0xda5 pf_check6_in() at pf_check6_in+0x35 pfil_run_hooks() at pfil_run_hooks+0xd2 ip6_input() at ip6_input+0x7f9 netisr_dispatch_src() at netisr_dispatch_src+0x152 gif_input() at gif_input+0xf5 in_gif_input() at in_gif_input+0x13e encap4_input() at encap4_input+0x139 ip_input() at ip_input+0xbd netisr_dispatch_src() at netisr_dispatch_src+0x152 ether_demux() at ether_demux+0x17d ng_ether_rcv_upper() at ng_ether_rcv_upper+0xa4 ng_apply_item() at ng_apply_item+0x281 ng_snd_item() at ng_snd_item+0x2af ng_apply_item() at ng_apply_item+0x281 ng_snd_item() at ng_snd_item+0x2af ng_apply_item() at ng_apply_item+0x281 ng_snd_item() at ng_snd_item+0x2af ng_ether_input() at ng_ether_input+0x7a ether_nh_input() at ether_nh_input+0x17c netisr_dispatch_src() at netisr_dispatch_src+0x152 ether_demux() at ether_demux+0x86 ether_nh_input() at ether_nh_input+0x20e netisr_dispatch_src() at netisr_dispatch_src+0x152 bce_intr() at bce_intr+0x46a intr_event_execute_handlers() at intr_event_execute_handlers+0x6a ithread_loop() at ithread_loop+0xb4 fork_exit() at fork_exit+0x135 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff811a901cf0, rbp = 0 --- >How-To-Repeat: >Fix: >Release-Note: >Audit-Trail: >Unformatted: