From owner-freebsd-net@FreeBSD.ORG Sat Jan 24 23:07:41 2015 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 39CBBDF1; Sat, 24 Jan 2015 23:07:41 +0000 (UTC) Received: from mail-lb0-x230.google.com (mail-lb0-x230.google.com [IPv6:2a00:1450:4010:c04::230]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id C2E31DE7; Sat, 24 Jan 2015 23:07:40 +0000 (UTC) Received: by mail-lb0-f176.google.com with SMTP id z12so2803296lbi.7; Sat, 24 Jan 2015 15:07:38 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=content-type:mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=Ew7sgCS8PWaMaGm0wDV17+3qKqkUe7gw8o6gMsAzm5c=; b=BO2/fqAk/FAgUiswszII0jaBwJg4IMQN3l0HfYeoqbCvwS2MLgbcSNoCNM2O/NXfVR 6/xSEMzayhydRota08dcu/9Uy1HDD2NJOr8c4nzNZiGYuusI+sCy4hA6Y81/aEraIMSK s1m3pHoUoUbHUq2gyn2Mft3y2PhJ0Zqz4hZWlc8fGgwPS6JKbpLw2yyUc6xdyNIzZ8sV OU24msuWnFct2NSQOOuEs+Tep7RKF4Vic1WRGcsgGbspLPNrCA8JBF7IPaFRI/XLgkwJ cGtRpn5kF+tHo9wbzybIMHnQE38Ia3vM5F/U88KSpowpe/OXhKCTPkpILRlIUKxNtF8h hw6Q== X-Received: by 10.152.161.168 with SMTP id xt8mr14232742lab.35.1422140858839; Sat, 24 Jan 2015 15:07:38 -0800 (PST) Received: from [10.0.1.7] (broadband-5-228-253-40.nationalcablenetworks.ru. [5.228.253.40]) by mx.google.com with ESMTPSA id dx2sm1460975lbc.49.2015.01.24.15.07.36 (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Sat, 24 Jan 2015 15:07:37 -0800 (PST) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 8.1 \(1993\)) Subject: Re: network locks up with udp traffic From: Dmitry Sivachenko In-Reply-To: <6B125546-E095-4A4C-9B6B-7E80D1D6E8EF@gmail.com> Date: Sun, 25 Jan 2015 02:07:35 +0300 Content-Transfer-Encoding: quoted-printable Message-Id: <81FCF115-3EE7-4902-9F9A-4852B59348AC@gmail.com> References: <5B08402C-67A7-49E7-ADF8-390C94DCF1D7@gmail.com> <6B125546-E095-4A4C-9B6B-7E80D1D6E8EF@gmail.com> To: Adrian Chadd X-Mailer: Apple Mail (2.1993) Cc: FreeBSD Net X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 24 Jan 2015 23:07:41 -0000 > On 25 =D1=8F=D0=BD=D0=B2. 2015 =D0=B3., at 1:45, Dmitry Sivachenko = wrote: >=20 >=20 >> On 25 =D1=8F=D0=BD=D0=B2. 2015 =D0=B3., at 0:10, Adrian Chadd = wrote: >>=20 >> Ah, try adding STACK to your config? >>=20 >=20 >=20 > It's already there. Ah, sorry, it is options DDB that was missing. Now we have full stack: lock order reversal: 1st 0xffffffff80ea5588 pcbinfohash (pcbinfohash) @ = /opt/WRK/src/sys/netinet6/ud p6_usrreq.c:1202 2nd 0xffffffff80ea5530 udp (udp) @ = /opt/WRK/src/sys/netinet6/in6_pcb.c:614 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame = 0xfffffe0c581c1270 kdb_backtrace() at kdb_backtrace+0x39/frame 0xfffffe0c581c1320 witness_checkorder() at witness_checkorder+0xc04/frame = 0xfffffe0c581c13a0 _rw_wlock_cookie() at _rw_wlock_cookie+0x45/frame 0xfffffe0c581c13e0 in6_pcbnotify() at in6_pcbnotify+0x12e/frame 0xfffffe0c581c1480 udp6_common_ctlinput() at udp6_common_ctlinput+0x111/frame = 0xfffffe0c581c14e0 pfctlinput2() at pfctlinput2+0x7d/frame 0xfffffe0c581c1520 ip6_output() at ip6_output+0x15b8/frame 0xfffffe0c581c1780 udp6_send() at udp6_send+0x75c/frame 0xfffffe0c581c1910 sosend_dgram() at sosend_dgram+0x30b/frame 0xfffffe0c581c1980 kern_sendit() at kern_sendit+0x191/frame 0xfffffe0c581c1a30 sendit() at sendit+0x225/frame 0xfffffe0c581c1a80 sys_sendmsg() at sys_sendmsg+0x68/frame 0xfffffe0c581c1ae0 amd64_syscall() at amd64_syscall+0x244/frame 0xfffffe0c581c1bf0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0c581c1bf0 --- syscall (28, FreeBSD ELF64, sys_sendmsg), rip =3D 0x803ec5eaa, rsp =3D= 0x7fffdf5f6848, rbp =3D 0x7fffdf5f6880 --- lock order reversal: 1st 0xffffffff80ea5588 pcbinfohash (pcbinfohash) @ = /opt/WRK/src/sys/netinet6/udp6_usrreq.c:1202 2nd 0xffffffff80ea52d8 tcp (tcp) @ = /opt/WRK/src/sys/netinet6/in6_pcb.c:614 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame = 0xfffffe0c581c1240 kdb_backtrace() at kdb_backtrace+0x39/frame 0xfffffe0c581c12f0 witness_checkorder() at witness_checkorder+0xc04/frame = 0xfffffe0c581c1370 _rw_wlock_cookie() at _rw_wlock_cookie+0x45/frame 0xfffffe0c581c13b0 in6_pcbnotify() at in6_pcbnotify+0x12e/frame 0xfffffe0c581c1450 tcp6_ctlinput() at tcp6_ctlinput+0x1a5/frame 0xfffffe0c581c14e0 pfctlinput2() at pfctlinput2+0x7d/frame 0xfffffe0c581c1520 ip6_output() at ip6_output+0x15b8/frame 0xfffffe0c581c1780 udp6_send() at udp6_send+0x75c/frame 0xfffffe0c581c1910 sosend_dgram() at sosend_dgram+0x30b/frame 0xfffffe0c581c1980 kern_sendit() at kern_sendit+0x191/frame 0xfffffe0c581c1a30 sendit() at sendit+0x225/frame 0xfffffe0c581c1a80 sys_sendmsg() at sys_sendmsg+0x68/frame 0xfffffe0c581c1ae0 amd64_syscall() at amd64_syscall+0x244/frame 0xfffffe0c581c1bf0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0c581c1bf0 --- syscall (28, FreeBSD ELF64, sys_sendmsg), rip =3D 0x803ec5eaa, rsp =3D= 0x7fffdf5f6848, rbp =3D 0x7fffdf5f6880 --- lock order reversal: 1st 0xffffffff80ea5588 pcbinfohash (pcbinfohash) @ = /opt/WRK/src/sys/netinet6/udp6_usrreq.c:1202 2nd 0xffffffff80ea4740 rip (rip) @ = /opt/WRK/src/sys/netinet6/in6_pcb.c:614 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame = 0xfffffe0c581c12b0 kdb_backtrace() at kdb_backtrace+0x39/frame 0xfffffe0c581c1360 witness_checkorder() at witness_checkorder+0xc04/frame = 0xfffffe0c581c13e0 _rw_wlock_cookie() at _rw_wlock_cookie+0x45/frame 0xfffffe0c581c1420 in6_pcbnotify() at in6_pcbnotify+0x12e/frame 0xfffffe0c581c14c0 rip6_ctlinput() at rip6_ctlinput+0x70/frame 0xfffffe0c581c14e0 pfctlinput2() at pfctlinput2+0x7d/frame 0xfffffe0c581c1520 ip6_output() at ip6_output+0x15b8/frame 0xfffffe0c581c1780 udp6_send() at udp6_send+0x75c/frame 0xfffffe0c581c1910 sosend_dgram() at sosend_dgram+0x30b/frame 0xfffffe0c581c1980 kern_sendit() at kern_sendit+0x191/frame 0xfffffe0c581c1a30 sendit() at sendit+0x225/frame 0xfffffe0c581c1a80 sys_sendmsg() at sys_sendmsg+0x68/frame 0xfffffe0c581c1ae0 amd64_syscall() at amd64_syscall+0x244/frame 0xfffffe0c581c1bf0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0c581c1bf0 --- syscall (28, FreeBSD ELF64, sys_sendmsg), rip =3D 0x803ec5eaa, rsp =3D= 0x7fffdf5f6848, rbp =3D 0x7fffdf5f6880 ---=