From owner-freebsd-bugs@FreeBSD.ORG Sun Dec 22 13:10:01 2013 Return-Path: Delivered-To: freebsd-bugs@smarthost.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 1468018D for ; Sun, 22 Dec 2013 13:10:01 +0000 (UTC) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id E37E71B4E for ; Sun, 22 Dec 2013 13:10:00 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.7/8.14.7) with ESMTP id rBMDA0cp023039 for ; Sun, 22 Dec 2013 13:10:00 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.7/8.14.7/Submit) id rBMDA0Gf023033; Sun, 22 Dec 2013 13:10:00 GMT (envelope-from gnats) Resent-Date: Sun, 22 Dec 2013 13:10:00 GMT Resent-Message-Id: <201312221310.rBMDA0Gf023033@freefall.freebsd.org> Resent-From: FreeBSD-gnats-submit@FreeBSD.org (GNATS Filer) Resent-To: freebsd-bugs@FreeBSD.org Resent-Reply-To: FreeBSD-gnats-submit@FreeBSD.org, Nikolay Denev Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 2CAD115E for ; Sun, 22 Dec 2013 13:04:53 +0000 (UTC) Received: from oldred.freebsd.org (oldred.freebsd.org [IPv6:2001:1900:2254:206a::50:4]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 184C21B32 for ; Sun, 22 Dec 2013 13:04:53 +0000 (UTC) Received: from oldred.freebsd.org ([127.0.1.6]) by oldred.freebsd.org (8.14.5/8.14.7) with ESMTP id rBMD4q7I060417 for ; Sun, 22 Dec 2013 13:04:52 GMT (envelope-from nobody@oldred.freebsd.org) Received: (from nobody@localhost) by oldred.freebsd.org (8.14.5/8.14.5/Submit) id rBMD4q38060416; Sun, 22 Dec 2013 13:04:52 GMT (envelope-from nobody) Message-Id: <201312221304.rBMD4q38060416@oldred.freebsd.org> Date: Sun, 22 Dec 2013 13:04:52 GMT From: Nikolay Denev To: freebsd-gnats-submit@FreeBSD.org X-Send-Pr-Version: www-3.1 Subject: misc/185092: panic: rtfree 2 (using RADIX_MPATH in a VNET jail) X-BeenThere: freebsd-bugs@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Bug reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 22 Dec 2013 13:10:01 -0000 >Number: 185092 >Category: misc >Synopsis: panic: rtfree 2 (using RADIX_MPATH in a VNET jail) >Confidential: no >Severity: non-critical >Priority: low >Responsible: freebsd-bugs >State: open >Quarter: >Keywords: >Date-Required: >Class: sw-bug >Submitter-Id: current-users >Arrival-Date: Sun Dec 22 13:10:00 UTC 2013 >Closed-Date: >Last-Modified: >Originator: Nikolay Denev >Release: 10.0-PRERELEASE >Organization: >Environment: FreeBSD mars 10.0-PRERELEASE FreeBSD 10.0-PRERELEASE #5 r259547: Sat Dec 21 21:48:57 UTC 2013 ndenev@mars:/usr/obj/usr/src/sys/MARS i386 >Description: I'm running with VIMAGE and RADIX_MPATH enabled on a Soekris Net5501 running 10.0-PRERELASE (r259547). I have a VNET jail, liked with epair(4) interface to the main stack, and inside I have two OpenVPN instances running, that both install a default route (the way OpenVPN does this as 0.0.0.0/1) and the ECMP works fine. However when I wanted to stop the OpenVPN processes and did a "killall openvpn" inside the VNET jail I got the following panic : root@mars:/usr/obj/usr/src/sys/MARS # kgdb /boot/kernel/kernel.debug /var/crash/vmcore.2 GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i386-marcel-freebsd"... Unread portion of the kernel message buffer: panic: rtfree 2 KDB: stack backtrace: #0 0xc0639ec2 at kdb_backtrace+0x52 #1 0xc0607ef5 at panic+0x85 #2 0xc06df879 at rtfree+0x139 #3 0xc06e2acd at route_output+0x48d #4 0xc06df3cf at raw_usend+0x3f #5 0xc066fef9 at sosend_generic+0x429 #6 0xc0670217 at sosend+0x57 #7 0xc0655d44 at soo_write+0x44 #8 0xc064dab6 at dofilewrite+0x86 #9 0xc064d7b6 at kern_writev+0x96 #10 0xc064d70c at sys_write+0x5c #11 0xc0864233 at syscall+0x353 #12 0xc084fa01 at Xint0x80_syscall+0x21 Uptime: 14h35m12s Physical memory: 503 MB Dumping 90 MB: 75 59 43 27 11 No symbol "stopped_cpus" in current context. #0 doadump (textdump=-1011979520) at pcpu.h:233 233 __asm("movl %%fs:%1,%0" : "=r" (td) (kgdb) bt #0 doadump (textdump=-1011979520) at pcpu.h:233 #1 0xc0607c02 in kern_reboot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:447 #2 0xc0607f5e in panic (fmt=) at /usr/src/sys/kern/kern_shutdown.c:754 #3 0xc06df879 in rtfree (rt=) at /usr/src/sys/net/route.c:464 #4 0xc06e2acd in route_output (m=) at /usr/src/sys/net/rtsock.c:951 #5 0xc06df3cf in raw_usend (so=, flags=0, m=, nam=0x0, control=, td=0xc3db8310) at /usr/src/sys/net/raw_usrreq.c:238 #6 0xc066fef9 in sosend_generic (so=0xc3e35d40, uio=, top=, control=0x0, flags=, td=) at /usr/src/sys/kern/uipc_socket.c:1271 #7 0xc0670217 in sosend (so=0xc3e35d40, addr=0x0, uio=0xd9bc1c10, top=0x0, control=0x0, flags=0, td=0xc3db8310) at /usr/src/sys/kern/uipc_socket.c:1315 #8 0xc0655d44 in soo_write (fp=0xc3c0c428, uio=0xd9bc1c10, active_cred=0xc448ff00, flags=0, td=0xc3db8310) at /usr/src/sys/kern/sys_socket.c:103 #9 0xc064dab6 in dofilewrite (td=0xc3db8310, fd=3, fp=0xc3c0c428, auio=0xd9bc1c10, offset=-1, flags=0) at file.h:303 #10 0xc064d7b6 in kern_writev (td=0xc3db8310, fd=3, auio=) at /usr/src/sys/kern/sys_generic.c:467 #11 0xc064d70c in sys_write (td=, uap=) at /usr/src/sys/kern/sys_generic.c:382 #12 0xc0864233 in syscall (frame=) at subr_syscall.c:134 #13 0xc084fa01 in Xint0x80_syscall () at /usr/src/sys/i386/i386/exception.s:270 #14 0x00000033 in ?? () Code in sys/net/rtsock.c:951 looks responsible for this, and is marked with XXX. if (rt) /* XXX can this be true? */ RTFREE(rt); >How-To-Repeat: Setup OpenVPN instances to install the same route using RADIX_MPATH inside a VNET jail. Kill the instances. >Fix: >Release-Note: >Audit-Trail: >Unformatted: