From owner-freebsd-net@freebsd.org Sat Oct 3 21:26:02 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 58D37A0FFB9 for ; Sat, 3 Oct 2015 21:26:02 +0000 (UTC) (envelope-from nvass@gmx.com) Received: from mout.gmx.net (mout.gmx.net [212.227.17.21]) (using TLSv1.2 with cipher DHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mout.gmx.net", Issuer "TeleSec ServerPass DE-1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 1536E1277; Sat, 3 Oct 2015 21:26:02 +0000 (UTC) (envelope-from nvass@gmx.com) Received: from moby.local ([91.140.35.215]) by mail.gmx.com (mrgmx101) with ESMTPSA (Nemesis) id 0M7CRe-1aWc6I0VDq-00x4O4; Sat, 03 Oct 2015 23:26:00 +0200 Subject: Re: carp on if_bridge deadlock To: =?UTF-8?Q?Ermal_Lu=c3=a7i?= References: <560F00E6.5010503@gmx.com> Cc: freebsd-net From: Nikos Vassiliadis Message-ID: <561047D2.1010303@gmx.com> Date: Sun, 4 Oct 2015 00:25:38 +0300 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:38.0) Gecko/20100101 Thunderbird/38.2.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 8bit X-Provags-ID: V03:K0:ylH4RAf2xxwTQn8qfcaFUVUIWuR6Uv3/0oiMvjxmBgn1RyXePvn tUD5UZNrZOQVBME5+WYRHzXvz5mCP0EAMms5naLqHJZm7FuKJ/P4u29vHAT+NG7QZKT1Ty6 PZyZq4RRb6OjoO5k1YA50KSSItE+/SisOijdJR8kpX5qudyfRMq+qJxzBbjC69baIBZBTM7 KT7s1Q2EmO6tCvlsnYpGw== X-UI-Out-Filterresults: notjunk:1;V01:K0:vttxo0xr10Q=:2C56oJC86qt8eCWL2zE4l/ eJjM45lzxYPaUySsYXDPPx17dOmM5odwSz7UyYRtK9GTsolkgurWX6txFhPqFd+zONtRcBtIc hz1cMuMNE4VhHONGSKWZxa0CxQKgRx30YzBRw2uYaT4Q9VTfdwM2ab0votE5BaFvJvQaKMs7Z fVZz9wTtAezwBP2Vn6BIXX3S49gKXb6VS+hwnRdHWekOkz+2EjMgVQ3awufX50fV/5QTiX4wR 55s3/qpaimZXovoHuo8uoKtr7Ica3cAzEboapho5FI6S7MvHRXNLueVpmY2jvdbHqfBnHp2/V 1mp/Mc0xHLS0Rf7URNv9Qo/Meav6hkBMpAYQAWPF6RU3gtiZB9d6ME3n1cVklwFiNHmmJ0P5n Qt1UeH1EUpKfAXE+KlCxJmF/qGtAZYAkPZUn8TAOWjyHnjxNOaWZNq0+6lPJq66yx+hetqIiH u3gJfIa9gsuEPPqtDZZdquiM+TUp5fdRrXLnSY+sHZertiInVFoTEGlLLrY/kMS309/N0ogWR bRN2CXt3CwMojneTt5cBf+VProNve4CMCbqFH/2gJyCRSOMjGhmfAqYRRFt+B8pWSGX8+GER3 guzHZDfVVHBpKM4smSylbGbqH8RDGIhP+yIr8ubaLCtw7q8nsUifCDQhGkp2fq1fLENn5IguT 65BmNS1TwCHYdno80O2dEsgV7hR/1LDHb/oHiF8+270wX1Rh+N2+zHGbCQguUoDuCYHYXVjxH HUmnSO8qOf0h+JcFzSF2RCuIHmSYrEZOWxdddcWrMPJSWZnAY68cN1+VwpTvrIV9NIjwou4az DnQsjZ0 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 03 Oct 2015 21:26:02 -0000 Hi Ermal, Thanks for the pointer. I just tried and I get this panic: > bridge0: flags=8943 metric 0 mtu 1500 > ether 00:a0:98:27:8d:55 > inet 10.0.0.11 netmask 0xff000000 broadcast 10.255.255.255 > inet 10.0.0.2 netmask 0xff000000 broadcast 10.25panic: Lock carp_if not exclusively locked @ /usr/src/sys/modules/carp/../../netinet/ip_carp.c:1744 > > cpuid = 2 > KDB: stack backtrace: > db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe0092cd56f0 > vpanic() at vpanic+0x189/frame 0xfffffe0092cd5770 > panic() at panic+0x43/frame 0xfffffe0092cd57d0 > __rw_assert() at __rw_assert+0xa1/frame 0xfffffe0092cd57e0 > carp_ioctl() at carp_ioctl+0x334/frame 0xfffffe0092cd5860 > kern_ioctl() at kern_ioctl+0x230/frame 0xfffffe0092cd58c0 > sys_ioctl() at sys_ioctl+0x153/frame 0xfffffe0092cd59a0 > amd64_syscall() at amd64_syscall+0x2e1/frame 0xfffffe0092cd5ab0 > Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0092cd5ab0 > --- syscall (54, FreeBSD ELF64, sys_ioctl), rip = 0x8011e927a, rsp = 0x7fffffffbd08, rbp = 0x7fffffffe500 --- > KDB: enter: panic > [ thread pid 295 tid 100060 ] > Stopped at kdb_enter+0x3e: movq $0,kdb_why > db> On 10/03/15 20:52, Ermal Luçi wrote: > This should apply https://reviews.freebsd.org/D3133 > > Somehow it is still pending on gnn@ for some reason! > > On Sat, Oct 3, 2015 at 12:10 AM, Nikos Vassiliadis wrote: > >> Hi, >> >> I am trying to use carp over an if_bridge and am getting >> this LOR: >> >>> login: lock order reversal: >>> 1st 0xfffff8000848a018 if_bridge (if_bridge) @ >>> /usr/src/sys/modules/if_bridge/../../net/if_bridge.c:2315 >>> 2nd 0xfffff80003a58778 carp_if (carp_if) @ >>> /usr/src/sys/modules/carp/../../netinet/ip_carp.c:1118 >>> KDB: stack backtrace: >>> db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame >>> 0xfffffe0092c3b6a0 >>> witness_checkorder() at witness_checkorder+0xe7a/frame 0xfffffe0092c3b720 >>> __mtx_lock_flags() at __mtx_lock_flags+0xa8/frame 0xfffffe0092c3b770 >>> carp_forus() at carp_forus+0x7a/frame 0xfffffe0092c3b7b0 >>> bridge_input() at bridge_input+0x338/frame 0xfffffe0092c3b820 >>> ether_nh_input() at ether_nh_input+0x2ab/frame 0xfffffe0092c3b860 >>> netisr_dispatch_src() at netisr_dispatch_src+0x86/frame 0xfffffe0092c3b8d0 >>> ether_input() at ether_input+0x4f/frame 0xfffffe0092c3b900 >>> vtnet_rxq_eof() at vtnet_rxq_eof+0x845/frame 0xfffffe0092c3b9b0 >>> vtnet_rx_vq_intr() at vtnet_rx_vq_intr+0x4e/frame 0xfffffe0092c3b9e0 >>> intr_event_execute_handlers() at intr_event_execute_handlers+0xe4/frame >>> 0xfffffe0092c3ba20 >>> ithread_loop() at ithread_loop+0xa6/frame 0xfffffe0092c3ba70 >>> fork_exit() at fork_exit+0x84/frame 0xfffffe0092c3bab0 >>> fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0092c3bab0 >>> --- trap 0, rip = 0, rsp = 0, rbp = 0 --- >>> >> >> Eventually all network activity will stop and this will happen: >> >>> load: 0.33 cmd: ifconfig 665 [*carp_if] 228.29r 0.00u 0.01s 0% 2716k >>> >> Or: >> >>> load: 2.86 cmd: ifconfig 720 [running] 104.92r 0.00u 36.53s 100% 2716k >>> >> >> A single ping to the carp address is enough to trigger the problem. >> The debugger says: >> >>> db> show all locks >>> Process 669 (sysctl) thread 0xfffff800084269a0 (100082) >>> exclusive sleep mutex Giant (Giant) r = 0 (0xffffffff81cc4e90) locked @ >>> /usr/src/sys/kern/kern_sysctl.c:164 >>> Process 668 (ifconfig) thread 0xfffff8000842e9a0 (100079) >>> exclusive sx carp_sx (carp_sx) r = 0 (0xffffffff820b0ac0) locked @ >>> /usr/src/sys/modules/carp/../../netinet/ip_carp.c:1644 >>> Process 12 (intr) thread 0xfffff800038a89a0 (100010) >>> exclusive sleep mutex carp_softc (carp_softc) r = 0 (0xfffff80008266508) >>> locked @ /usr/src/sys/kern/kern_mutex.c:158 >>> Process 12 (intr) thread 0xfffff80003a679a0 (100031) >>> exclusive sleep mutex carp_if (carp_if) r = 0 (0xfffff8000825b078) locked >>> @ /usr/src/sys/modules/carp/../../netinet/ip_carp.c:1118 >>> exclusive sleep mutex if_bridge (if_bridge) r = 0 (0xfffff80008467018) >>> locked @ /usr/src/sys/modules/if_bridge/../../net/if_bridge.c:2315 >>> db> >>> >> >> There is a related PR >> https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=200319 >> But I couldn't apply cleanly the patch to HEAD or 10-STABLE. >> >> Thanks for any insights, >> Nikos >> >> _______________________________________________ >> freebsd-net@freebsd.org mailing list >> https://lists.freebsd.org/mailman/listinfo/freebsd-net >> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" >> >> -- >> Ermal >> > _______________________________________________ > freebsd-net@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" >