From owner-freebsd-net@FreeBSD.ORG Wed May 21 19:43:18 2014 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 23351E96 for ; Wed, 21 May 2014 19:43:18 +0000 (UTC) Received: from mail.ignoranthack.me (ignoranthack.me [199.102.79.106]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 055BC2916 for ; Wed, 21 May 2014 19:43:17 +0000 (UTC) Received: from [10.12.72.220] (unknown [69.164.56.1]) (using SSLv3 with cipher DHE-RSA-AES128-SHA (128/128 bits)) (No client certificate requested) (Authenticated sender: sbruno@ignoranthack.me) by mail.ignoranthack.me (Postfix) with ESMTPSA id 3A8B6194138 for ; Wed, 21 May 2014 19:43:16 +0000 (UTC) Subject: re(4) stalls, crashes(supposed patch exists?) From: Sean Bruno Reply-To: sbruno@freebsd.org To: FreeBSD Net Content-Type: text/plain; charset="us-ascii" Date: Wed, 21 May 2014 12:43:15 -0700 Message-ID: <1400701395.1848.16.camel@bruno> Mime-Version: 1.0 X-Mailer: Evolution 2.32.1 FreeBSD GNOME Team Port Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 May 2014 19:43:18 -0000 The Bytemark Site of freebsd.org is experiencing periodic stalls and crashes on the machines being used as routers. I have heard of a rumored patch that exists "somewhere" to resolve this, but when I asked at BSDCan, I got no takers. Any thoughts? FreeBSD igw0.bme.freebsd.org 11.0-CURRENT FreeBSD 11.0-CURRENT #0 r265337: Sun May 4 22:05:35 UTC 2014 peter@igw0.bme.freebsd.org:/usr/obj/usr/src/sys/IGW0 amd64 re0@pci0:3:0:0: class=0x020000 card=0x85051043 chip=0x816810ec rev=0x09 hdr=0x00 vendor = 'Realtek Semiconductor Co., Ltd.' device = 'RTL8111/8168B PCI Express Gigabit Ethernet controller' class = network subclass = ethernet re0: port 0xe800-0xe8ff mem 0xfdfff000-0xfdffffff,0xfdff8000-0xfdffbfff irq 18 at device 0.0 on pci3 re0: Using 1 MSI-X message re0: turning off MSI enable bit. re0: Chip rev. 0x48000000 re0: MAC rev. 0x00000000 miibus0: on re0 rgephy0: PHY 1 on miibus0 rgephy0: none, 10baseT, 10baseT-FDX, 10baseT-FDX-flow, 100baseTX, 100baseTX-FDX, 100baseTX-FDX-flow, 1000baseT-FDX, 1000baseT-FDX-master, 1000baseT-FDX-flow, 1000baseT-FDX-flow-master, auto, auto-flow re0: Ethernet address: 08:60:6e:d7:31:d2 Panic: panic: _mtx_lock_sleep: recursed on non-recursive mutex pf_idhash @ /usr/src/sys/netpfil/pf/pf.c:922 cpuid = 0 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe00003c5d70 kdb_backtrace() at kdb_backtrace+0x39/frame 0xfffffe00003c5e20 vpanic() at vpanic+0x126/frame 0xfffffe00003c5e60 kassert_panic() at kassert_panic+0x136/frame 0xfffffe00003c5ed0 __mtx_lock_sleep() at __mtx_lock_sleep+0x369/frame 0xfffffe00003c5f50 __mtx_lock_flags() at __mtx_lock_flags+0xf5/frame 0xfffffe00003c5fa0 pf_state_insert() at pf_state_insert+0x37d/frame 0xfffffe00003c6040 pf_test_rule() at pf_test_rule+0x26ce/frame 0xfffffe00003c6520 pf_test6() at pf_test6+0x11ef/frame 0xfffffe00003c66c0 pf_check6_in() at pf_check6_in+0x36/frame 0xfffffe00003c66e0 pfil_run_hooks() at pfil_run_hooks+0x93/frame 0xfffffe00003c6770 ip6_input() at ip6_input+0x619/frame 0xfffffe00003c6950 netisr_dispatch_src() at netisr_dispatch_src+0x90/frame 0xfffffe00003c69c0 ether_demux() at ether_demux+0x13c/frame 0xfffffe00003c69f0 ether_nh_input() at ether_nh_input+0x32a/frame 0xfffffe00003c6a20 netisr_dispatch_src() at netisr_dispatch_src+0x90/frame 0xfffffe00003c6a90 re_rxeof() at re_rxeof+0x539/frame 0xfffffe00003c6af0 re_intr_msi() at re_intr_msi+0xcc/frame 0xfffffe00003c6b30 intr_event_execute_handlers() at intr_event_execute_handlers+0x93/frame 0xfffffe00003c6b70 ithread_loop() at ithread_loop+0xa6/frame 0xfffffe00003c6bb0 fork_exit() at fork_exit+0x84/frame 0xfffffe00003c6bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00003c6bf0 --- trap 0, rip = 0, rsp = 0xfffffe00003c6cb0, rbp = 0 --- KDB: enter: panic Panic: Kernel page fault with the following non-sleepable locks held: shared rw ipsec request (ipsec request) r = 0 (0xfffff80005dea160) locked @ /usr/src/sys/netipsec/ipsec_output.c:802 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a/frame 0xfffffe011ad60f40 kdb_backtrace() at kdb_backtrace+0x37/frame 0xfffffe011ad61000 _witness_debugger() at _witness_debugger+0x2c/frame 0xfffffe011ad61020 witness_warn() at witness_warn+0x2d4/frame 0xfffffe011ad61160 trap_pfault() at trap_pfault+0x6a/frame 0xfffffe011ad611f0 trap() at trap+0x41a/frame 0xfffffe011ad613f0 calltrap() at calltrap+0x8/frame 0xfffffe011ad613f0 --- trap 0xc, rip = 0xffffffff806d5ada, rsp = 0xfffffe011ad614b0, rbp = 0xfffffe011ad61580 --- ipsec6_output_tunnel() at ipsec6_output_tunnel+0xda/frame 0xfffffe011ad61580 ip6_forward() at ip6_forward+0x69a/frame 0xfffffe011ad61710 ip6_input() at ip6_input+0xd06/frame 0xfffffe011ad618a0 netisr_dispatch_src() at netisr_dispatch_src+0x15d/frame 0xfffffe011ad61910 ether_demux() at ether_demux+0x1a9/frame 0xfffffe011ad61940 ether_nh_input() at ether_nh_input+0x209/frame 0xfffffe011ad61980 netisr_dispatch_src() at netisr_dispatch_src+0x15d/frame 0xfffffe011ad619f0 re_rxeof() at re_rxeof+0x4a9/frame 0xfffffe011ad61a60 re_int_task() at re_int_task+0x1ea/frame 0xfffffe011ad61aa0 taskqueue_run_locked() at taskqueue_run_locked+0x93/frame 0xfffffe011ad61b00 taskqueue_run() at taskqueue_run+0x3d/frame 0xfffffe011ad61b20 intr_event_execute_handlers() at intr_event_execute_handlers+0x6a/frame 0xfffffe011ad61b50 ithread_loop() at ithread_loop+0x9b/frame 0xfffffe011ad61ba0 fork_exit() at fork_exit+0x139/frame 0xfffffe011ad61bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe011ad61bf0 --- trap 0, rip = 0, rsp = 0xfffffe011ad61cb0, rbp = 0 ---