From owner-freebsd-bugs@FreeBSD.ORG Wed Jan 1 13:30:01 2014 Return-Path: Delivered-To: freebsd-bugs@smarthost.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 7FB9CDC7 for ; Wed, 1 Jan 2014 13:30:01 +0000 (UTC) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 6B3B7161F for ; Wed, 1 Jan 2014 13:30:01 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.7/8.14.7) with ESMTP id s01DU1am016169 for ; Wed, 1 Jan 2014 13:30:01 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.7/8.14.7/Submit) id s01DU1hK016168; Wed, 1 Jan 2014 13:30:01 GMT (envelope-from gnats) Date: Wed, 1 Jan 2014 13:30:01 GMT Message-Id: <201401011330.s01DU1hK016168@freefall.freebsd.org> To: freebsd-bugs@FreeBSD.org Cc: From: Nikolay Denev Subject: Re: misc/185092: panic: rtfree 2 (using RADIX_MPATH in a VNET jail) X-BeenThere: freebsd-bugs@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list Reply-To: Nikolay Denev List-Id: Bug reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 01 Jan 2014 13:30:01 -0000 The following reply was made to PR misc/185092; it has been noted by GNATS. From: Nikolay Denev To: FreeBSD-gnats-submit@freebsd.org, freebsd-bugs@freebsd.org, "freebsd-net@freebsd.org" Cc: Subject: Re: misc/185092: panic: rtfree 2 (using RADIX_MPATH in a VNET jail) Date: Wed, 1 Jan 2014 13:21:13 +0000 --001a11c1ba200f4f8504eee88df5 Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable On Wed, Jan 1, 2014 at 1:10 PM, Nikolay Denev wrote: > On Sun, Dec 22, 2013 at 1:10 PM, wrote= : > >> Thank you very much for your problem report. >> It has the internal identification `misc/185092'. >> The individual assigned to look at your >> report is: freebsd-bugs. >> >> You can access the state of your problem report at any time >> via this link: >> >> http://www.freebsd.org/cgi/query-pr.cgi?pr=3D185092 >> >> >Category: misc >> >Responsible: freebsd-bugs >> >Synopsis: panic: rtfree 2 (using RADIX_MPATH in a VNET jail) >> >Arrival-Date: Sun Dec 22 13:10:00 UTC 2013 >> > > I'm trying to understand exactly what is happening here, and examining a > core dump with kgdb I'm getting some output that confuses me : > > (kgdb) bt > #0 doadump (textdump=3D-1011569920) at pcpu.h:233 > #1 0xc06069b2 in kern_reboot (howto=3D260) at > /usr/src/sys/kern/kern_shutdown.c:447 > #2 0xc0606d0e in panic (fmt=3D) at > /usr/src/sys/kern/kern_shutdown.c:754 > #3 0xc06de639 in rtfree (rt=3D) at > /usr/src/sys/net/route.c:464 > #4 0xc06e188d in route_output (m=3D) at > /usr/src/sys/net/rtsock.c:951 > #5 0xc06de18f in raw_usend (so=3D, flags=3D0, m=3D<= value > optimized out>, nam=3D0x0, control=3D, > td=3D0xc3bd2000) at /usr/src/sys/net/raw_usrreq.c:238 > #6 0xc066eca9 in sosend_generic (so=3D0xc3e9c1a8, uio=3D out>, top=3D, control=3D0x0, > flags=3D, td=3D) at > /usr/src/sys/kern/uipc_socket.c:1271 > #7 0xc066efc7 in sosend (so=3D0xc3e9c1a8, addr=3D0x0, uio=3D0xd9b9cc10, > top=3D0x0, control=3D0x0, flags=3D0, td=3D0xc3bd2000) > at /usr/src/sys/kern/uipc_socket.c:1315 > #8 0xc0654af4 in soo_write (fp=3D0xc3c0c818, uio=3D0xd9b9cc10, > active_cred=3D0xc3f1dd00, flags=3D0, td=3D0xc3bd2000) > at /usr/src/sys/kern/sys_socket.c:103 > #9 0xc064c866 in dofilewrite (td=3D0xc3bd2000, fd=3D3, fp=3D0xc3c0c818, > auio=3D0xd9b9cc10, offset=3D-1, flags=3D0) at file.h:303 > #10 0xc064c566 in kern_writev (td=3D0xc3bd2000, fd=3D3, auio=3D out>) at /usr/src/sys/kern/sys_generic.c:467 > #11 0xc064c4bc in sys_write (td=3D, uap=3D optimized out>) at /usr/src/sys/kern/sys_generic.c:382 > #12 0xc08614d3 in syscall (frame=3D) at > subr_syscall.c:134 > #13 0xc084cca1 in Xint0x80_syscall () at > /usr/src/sys/i386/i386/exception.s:270 > #14 0x281975b7 in ?? () > Previous frame inner to this frame (corrupt stack?) > Current language: auto; currently minimal > (kgdb) fr 3 > #3 0xc06de639 in rtfree (rt=3D) at > /usr/src/sys/net/route.c:464 > 464 panic("rtfree 2"); > (kgdb) print *rt > $1 =3D {rt_nodes =3D {{rn_mklist =3D 0xc3b4ab30, rn_parent =3D 0x1, rn_bi= t =3D 0, > rn_bmask =3D 0 '\0', rn_flags =3D 0 '\0', rn_u =3D {rn_leaf =3D { > rn_Key =3D 0xc0882687 "shutdown_post_sync", rn_Mask =3D 0x10300= 00 >
, rn_Dupedkey =3D 0x0}, rn_node =3D { > rn_Off =3D -1064819065, rn_L =3D 0x1030000, rn_R =3D 0x0}}}, > {rn_mklist =3D 0x0, rn_parent =3D 0x4, rn_bit =3D -18048, rn_bmask =3D -9= 4 '?', > rn_flags =3D 195 '?', rn_u =3D {rn_leaf =3D {rn_Key =3D 0xc3a545e0 = "", > rn_Mask =3D 0xc3a4e440 " ??(???\020'", rn_Dupedkey =3D 0xc3a4e880}, > rn_node =3D {rn_Off =3D -1012578848, rn_L =3D 0xc3a4e440, rn_R = =3D > 0xc3a4e880}}}}, rt_gateway =3D 0x74756873, rt_flags =3D 1853321060, > rt_refcnt =3D 1936683103, rt_ifp =3D 0x79735f74, rt_ifa =3D 0x636e, rt_= rmx =3D > {rmx_mtu =3D 0, rmx_expire =3D 0, rmx_pksent =3D 0, rmx_weight =3D 0}, > rt_fibnum =3D 0, rt_mtx =3D {lock_object =3D {lo_name =3D 0x0, lo_flags= =3D 0, > lo_data =3D 0, lo_witness =3D 0x0}, mtx_lock =3D 0}} > > > > rn_Key with value of =93shutdown_post_sync=94 ? > > It=92s visible also in the raw_usend() frame: > > (kgdb) fr 5 > #5 0xc06de18f in raw_usend (so=3D, flags=3D0, m=3D<= value > optimized out>, nam=3D0x0, control=3D, > td=3D0xc3bd2000) at /usr/src/sys/net/raw_usrreq.c:238 > 238 return ((*so->so_proto->pr_output)(m, so)); > (kgdb) print *m > $2 =3D {m_hdr =3D {mh_next =3D 0xc3b4ab30, mh_nextpkt =3D 0x1, mh_data = =3D 0x0, > mh_len =3D -1064819065, mh_type =3D 0, mh_flags =3D 66304, mh_pad =3D 0}, > M_dat =3D {MH =3D {MH_pkthdr =3D {rcvif =3D 0x0, tags =3D {slh_first = =3D 0x4}, len =3D > -1012745856, flowid =3D 3282388448, > csum_flags =3D 14097648373312316480, fibnum =3D 26739, cosqos =3D= 117 > 'u', rsstype =3D 116 't', l2hlen =3D 100 'd', l3hlen =3D 111 'o', > l4hlen =3D 119 'w', l5hlen =3D 110 'n', PH_per =3D {eigth =3D "_p= ost_sy", > sixteen =3D {28767, 29551, 24436, 31091}, thirtytwo =3D {1936683103, > 2037604212}, sixtyfour =3D {8751443454668533855}, unintptr = =3D > {1936683103}, ptr =3D 0x736f705f}, PH_loc =3D { > eigth =3D "nc\000\000\000\000\000", sixteen =3D {25454, 0, 0, 0= }, > thirtytwo =3D {25454, 0}, sixtyfour =3D {25454}, unintptr =3D {25454}, > ptr =3D 0x636e}}, MH_dat =3D {MH_ext =3D {ref_cnt =3D 0x0, ext_= buf =3D > 0x0, ext_size =3D 0, ext_type =3D 0, ext_flags =3D 0, ext_free =3D 0, > ext_arg1 =3D 0x0, ext_arg2 =3D 0x0}, > MH_databuf =3D '\0' , "file", '\0' times>, > "\006\000\000\000\020\000\000\000??\215?\000\000C\001\000\000\000\000\000= \000\000\000\004\000\000\000\000\000\000\00000Y?", > '\0' , > "`2Y?\000\000\000\000\000\000\000\000T\211\223?\022\000\000\000\000\203??= \000\000\000\000\000???", > '\0' }}, > M_databuf =3D > "\000\000\000\000\004\000\000\000\200????E??@??\200??shutdown_post_sync", > '\0' , "file", '\0' , > "\006\000\000\000\020\000\000\000??\215?\000\000C\001\000\000\000\000\000= \000\000\000\004\000\000\000\000\000\000\00000Y?", > '\0' , > "`2Y?\000\000\000\000\000\000\000\000T\211\223?\022\000\000\000\000\203??= \000\000\000\000\000???", > '\0' }} > > > This is 10.0-PRERELEASE r259547M (with applied the recent nd6_nbr.c rtfre= e > patch, which I thought earlier might be the cause of the panics I'm > seeing). > > The machine is Soekris Net5501-70 with this kernel config : > > cpu I586_CPU > cpu I686_CPU > ident MARS > options CPU_GEODE > options CPU_SOEKRIS > > options HZ=3D2000 > options DEVICE_POLLING > options BPF_JITTER > > makeoptions DEBUG=3D-g # Build kernel with gdb(1) debug symbols > > options SCHED_ULE # ULE scheduler > options PREEMPTION # Enable kernel thread preemption > options INET # InterNETworking > options INET6 # IPv6 communications protocols > options TCP_OFFLOAD # TCP offload > options FFS # Berkeley Fast Filesystem > options SOFTUPDATES # Enable FFS soft updates support > options UFS_DIRHASH # Improve performance on big directories > options PROCFS # Process filesystem (requires PSEUDOFS) > options PSEUDOFS # Pseudo-filesystem framework > options GEOM_PART_GPT # GUID Partition Tables. > options GEOM_LABEL # Provides labelization > options COMPAT_FREEBSD4 # Compatible with FreeBSD4 > options COMPAT_FREEBSD5 # Compatible with FreeBSD5 > options COMPAT_FREEBSD6 # Compatible with FreeBSD6 > options COMPAT_FREEBSD7 # Compatible with FreeBSD7 > options SCSI_DELAY=3D500 # Delay (in ms) before probing SCSI > options KTRACE # ktrace(1) support > options STACK # stack(9) support > options SYSVSHM # SYSV-style shared memory > options SYSVMSG # SYSV-style message queues > options SYSVSEM # SYSV-style semaphores > options _KPOSIX_PRIORITY_SCHEDULING # POSIX P1003_1B real-time extensions > options PRINTF_BUFR_SIZE=3D128 # Prevent printf output being interspersed= . > options KBD_INSTALL_CDEV # install a CDEV entry in /dev > options HWPMC_HOOKS # Necessary kernel hooks for hwpmc(4) > options CAPABILITY_MODE # Capsicum capability mode > options CAPABILITIES # Capsicum capabilities > options PROCDESC # Support for process descriptors > options INCLUDE_CONFIG_FILE # Include this file in kernel > > # Debugging support. Always need this: > options KDB # Enable kernel debugger support. > options KDB_TRACE # Print a stack trace for a panic. > options KDB_UNATTENDED > > options TEXTDUMP_PREFERRED > options TEXTDUMP_VERBOSE > > device pci > device ata # Legacy ATA/SATA controllers > options ATA_STATIC_ID # Static device numbering > > # ATA/SCSI peripherals > device scbus # SCSI bus (required for ATA/SCSI) > device da # Direct Access (disks) > device pass # Passthrough device (direct ATA/SCSI access) > > # Add suspend/resume support for the i8254. > device pmtimer > > # Serial (COM) ports > device uart # Generic UART driver > > device miibus # MII bus support > device vr # VIA Rhine, Rhine II > > # Wireless NIC cards > device wlan # 802.11 support > options IEEE80211_DEBUG # enable debug msgs > options IEEE80211_AMPDU_AGE # age frames in AMPDU reorder q's > options IEEE80211_SUPPORT_MESH # enable 802.11s draft support > device wlan_wep # 802.11 WEP support > device wlan_ccmp # 802.11 CCMP support > device wlan_tkip # 802.11 TKIP support > device wlan_amrr # AMRR transmit rate control algorithm > device ath # Atheros NICs > device ath_pci # Atheros pci/cardbus glue > device ath_hal # pci/cardbus chip support > options AH_SUPPORT_AR5416 # enable AR5416 tx/rx descriptors > options AH_AR5416_INTERRUPT_MITIGATION # AR5416 interrupt mitigation > options ATH_ENABLE_11N # Enable 802.11n support for AR5416 and later > device ath_rate_sample # SampleRate tx rate control for ath > > # Pseudo devices. > device loop # Network loopback > device random # Entropy device > device ether # Ethernet support > device vlan # 802.1Q VLAN support > device tun # Packet tunnel. > device md # Memory "disks" > device gif # IPv6 and IPv4 tunneling > device gre > device faith # IPv6-to-IPv4 relaying (translation) > device firmware # firmware assist module > device if_bridge > > options VIMAGE > options ROUTETABLES=3D8 > options RADIX_MPATH > > options SW_WATCHDOG > > device crypto > device cryptodev > device glxsb > > options BOOTVERBOSE=3D1 > > #device pf > #device pflog > #device pfsync > device carp > device enc > device lagg > device epair > > #options ALTQ > #options ALTQ_CBQ > #options ALTQ_RED > #options ALTQ_RIO > #options ALTQ_HFSC > #options ALTQ_PRIQ > > options IPFIREWALL > options IPFIREWALL_DEFAULT_TO_ACCEPT > options IPFIREWALL_NAT > options LIBALIAS > options IPDIVERT > options DUMMYNET > > device bpf # Berkeley packet filter > > # USB support > options USB_DEBUG # enable debug msgs > device uhci # UHCI PCI->USB interface > device ohci # OHCI PCI->USB interface > device ehci # EHCI PCI->USB interface (USB 2.0) > device usb # USB Bus (required) > device umass # Disks/Mass storage - Requires scbus and da > > > Also src.conf and make.conf : > > root@vpn_vrf:[VNET(x)]:/usr/src/sys # cat /etc/src.conf > WITHOUT_ACCT=3Dyes > WITHOUT_ACPI=3Dyes > WITHOUT_AMD=3Dyes > WITHOUT_APM=3Dyes > WITHOUT_ASSERT_DEBUG=3Dyes > WITHOUT_AT=3Dyes > WITHOUT_ATF=3Dyes > WITHOUT_ATM=3Dyes > WITHOUT_AUDIT=3Dyes > WITHOUT_BLUETOOTH=3Dyes > WITHOUT_CALENDAR=3Dyes > WITHOUT_CDDL=3Dyes > WITHOUT_CTM=3Dyes > WITHOUT_DICT=3Dyes > WITHOUT_FLOPPY=3Dyes > WITHOUT_GAMES=3Dyes > WITHOUT_HTML=3Dyes > WITHOUT_INFO=3Dyes > WITHOUT_IPFILTER=3Dyes > WITHOUT_IPX=3Dyes > #WITHOUT_KERNEL_SYMBOLS=3Dyes > WITHOUT_LEGACY_CONSOLE=3Dyes > WITHOUT_LOCALES=3Dyes > WITHOUT_LPR=3Dyes > WITHOUT_MAIL=3Dyes > WITHOUT_NDIS=3Dyes > WITHOUT_QUOTAS=3Dyes > WITHOUT_ROUTED=3Dyes > WITHOUT_SENDMAIL=3Dyes > WITH_SVN=3Dyes > WITHOUT_ZFS=3Dyes > > root@vpn_vrf:[VNET(x)]:/usr/src/sys # cat /etc/make.conf > CFLAGS=3D-O2 > COPTFLAGS=3D -O -pipe > CPUTYPE=3Dgeode > KERNCONF=3DMARS > NO_MODULES=3Dyes > BOOTWAIT=3D0 > DOC_LANG=3Den_US.ISO8859-1 > > > > --Nikolay > > Also, originally I thought that the panic is when a multi path route is being deleted, however again from the coredump it seems that the panic happens when openvpn deletes the host route it installs for the remote openvpn server pointed to the default gw (before openvpn installs the new default gw pointing to the vpn tunnel) : (kgdb) fr 12 (kgdb) x/12sb td->td_proc->p_args 0xc4269780: "\001" 0xc4269782: "" 0xc4269783: "" 0xc4269784: "B" 0xc4269786: "" 0xc4269787: "" 0xc4269788: "/sbin/route" 0xc4269794: "delete" 0xc426979b: "-net" 0xc42697a0: "78.90.222.xxx" 0xc42697ad: "10.255.255.0" 0xc42697ba: "255.255.255.255" (kgdb) I'm trying to reproduce this on a VirtualBox instance now, however so far no luck (no OpenVPN running, just adding and removing routes). --Nikolay --001a11c1ba200f4f8504eee88df5 Content-Type: text/html; charset=windows-1252 Content-Transfer-Encoding: quoted-printable
On Wed, Jan 1, 2014 at 1:10 PM, Nikolay Denev <ndenev@gmai= l.com> wrote:
On Sun, Dec 22, 20= 13 at 1:10 PM, <FreeBSD-gnats-submit@freebsd.org> wrote:
Thank you very much for your problem report.
It has the internal identification `misc/185092'.
The individual assigned to look at your
report is: freebsd-bugs.

You can access the state of your problem report at any time
via this link:

http://www.freebsd.org/cgi/query-pr.cgi?pr=3D185092

>Category: =A0 =A0 =A0 misc
>Responsible: =A0 =A0freebsd-bugs
>Synopsis: =A0 =A0 =A0 panic: rtfree 2 (using RADIX_MPATH in a VNET jail= )
>Arrival-Date: =A0 Sun Dec 22 13:10:00 UTC 2013

I'm= trying to understand exactly what is happening here, and examining a core = dump with kgdb I'm getting some output that confuses me :

(kgdb) bt
<= /div>
#0 =A0doadump (t= extdump=3D-1011569920) at pcpu.h:233
#1 =A00xc06069b= 2 in kern_reboot (howto=3D260) at /usr/src/sys/kern/kern_shutdown.c:447
#2 =A00xc0606= d0e in panic (fmt=3D<value optimized out>) at /usr/src/sys/kern/kern_= shutdown.c:754
#3 =A00xc06de63= 9 in rtfree (rt=3D<value optimized out>) at /usr/src/sys/net/route.c:= 464
#4 =A0= 0xc06e188d in route_output (m=3D<value optimized out>) at /usr/src/sy= s/net/rtsock.c:951
#5 =A00xc06de18= f in raw_usend (so=3D<value optimized out>, flags=3D0, m=3D<value = optimized out>, nam=3D0x0, control=3D<value optimized out>,=A0
=A0 =A0 td=3D0xc3bd2000) at /usr/src/sys/net/raw= _usrreq.c:238
#6 =A00xc066eca9 in sosend_generic (so=3D0xc3e9c1a8, uio=3D<value op= timized out>, top=3D<value optimized out>, control=3D0x0,=A0
=A0 =A0 flags= =3D<value optimized out>, td=3D<value optimized out>) at /usr/s= rc/sys/kern/uipc_socket.c:1271
#7 =A00xc066efc7 in sosend (so=3D0xc3e9c1a8, addr=3D0x0, uio=3D0xd9b9cc10, = top=3D0x0, control=3D0x0, flags=3D0, td=3D0xc3bd2000)
=A0 =A0 at /usr/src/sys/kern/ui= pc_socket.c:1315
#8 =A00xc0654af= 4 in soo_write (fp=3D0xc3c0c818, uio=3D0xd9b9cc10, active_cred=3D0xc3f1dd00= , flags=3D0, td=3D0xc3bd2000)
=A0 =A0 at /usr/src/sys/kern/sys_socket.c:103
#9 =A00xc064c866 in dofilewrite (td=3D0= xc3bd2000, fd=3D3, fp=3D0xc3c0c818, auio=3D0xd9b9cc10, offset=3D-1, flags= =3D0) at file.h:303
#10 0xc064c566 = in kern_writev (td=3D0xc3bd2000, fd=3D3, auio=3D<value optimized out>= ) at /usr/src/sys/kern/sys_generic.c:467
#11 0xc064c4bc in sys_write (td=3D<value optimized out>, uap=3D<va= lue optimized out>) at /usr/src/sys/kern/sys_generic.c:382
#12 0xc08614d3 in sysca= ll (frame=3D<value optimized out>) at subr_syscall.c:134
#13 0xc084cca1 = in Xint0x80_syscall () at /usr/src/sys/i386/i386/exception.s:270
#14 0x281975b7 in ??= ()
Previous frame = inner to this frame (corrupt stack?)
=
Current language: =A0auto; currently minimal
(kgdb) fr 3
#3 =A00xc06de639 in= rtfree (rt=3D<value optimized out>) at /usr/src/sys/net/route.c:464<= /div>
464 panic("rtfree 2");
(kgdb) print *rt
$1 =3D {rt_nodes =3D = {{rn_mklist =3D 0xc3b4ab30, rn_parent =3D 0x1, rn_bit =3D 0, rn_bmask =3D 0= '\0', rn_flags =3D 0 '\0', rn_u =3D {rn_leaf =3D {
=A0 =A0 =A0 =A0 =A0 rn_Key =3D 0xc0882687 "= shutdown_post_sync", rn_Mask =3D 0x1030000 <Address 0x1030000 out o= f bounds>, rn_Dupedkey =3D 0x0}, rn_node =3D {
=A0 =A0 =A0 =A0 =A0 rn_Off =3D -1064819065, rn_L =3D 0x1030000, rn_R =3D 0x= 0}}}, {rn_mklist =3D 0x0, rn_parent =3D 0x4, rn_bit =3D -18048, rn_bmask = =3D -94 '?',=A0
=A0 =A0 =A0 rn_flags =3D 195 '?', rn_u =3D {rn_leaf = =3D {rn_Key =3D 0xc3a545e0 "", rn_Mask =3D 0xc3a4e440 " ??(?= ??\020'", rn_Dupedkey =3D 0xc3a4e880},=A0
=A0 =A0 =A0 =A0= rn_node =3D {rn_Off =3D -1012578848, rn_L =3D 0xc3a4e440, rn_R =3D 0xc3a4e= 880}}}}, rt_gateway =3D 0x74756873, rt_flags =3D 1853321060,=A0
=
=A0 rt_refcnt =3D 1936683103, rt_ifp =3D 0x79735= f74, rt_ifa =3D 0x636e, rt_rmx =3D {rmx_mtu =3D 0, rmx_expire =3D 0, rmx_pk= sent =3D 0, rmx_weight =3D 0},=A0
=A0 rt_fibnum =3D 0, rt_mtx =3D {lock_object =3D {lo_name =3D 0x0, lo_flags= =3D 0, lo_data =3D 0, lo_witness =3D 0x0}, mtx_lock =3D 0}}

<= div class=3D"gmail_extra">
rn_Key with value of =93shutdown_post_= sync=94 ?

It=92s visible also in the raw_usend() frame:

(kgdb) fr 5
#5 =A00xc06d= e18f in raw_usend (so=3D<value optimized out>, flags=3D0, m=3D<val= ue optimized out>, nam=3D0x0, control=3D<value optimized out>,=A0<= /div>
=A0 =A0 td=3D0x= c3bd2000) at /usr/src/sys/net/raw_usrreq.c:238
238 return ((*so->so_proto->pr_output)(m, so));
(kgdb) print *m=
$2 =3D {m= _hdr =3D {mh_next =3D 0xc3b4ab30, mh_nextpkt =3D 0x1, mh_data =3D 0x0, mh_l= en =3D -1064819065, mh_type =3D 0, mh_flags =3D 66304, mh_pad =3D 0},=A0
=A0 M_dat =3D {= MH =3D {MH_pkthdr =3D {rcvif =3D 0x0, tags =3D {slh_first =3D 0x4}, len =3D= -1012745856, flowid =3D 3282388448,=A0
=A0 =A0 =A0 =A0 csum_flags =3D 14097648373312316480, fibnum =3D 26739, cosq= os =3D 117 'u', rsstype =3D 116 't', l2hlen =3D 100 'd&= #39;, l3hlen =3D 111 'o',=A0
=
=A0 =A0 =A0 =A0 l4hlen =3D 119 'w', l5hlen =3D 110 'n', PH_= per =3D {eigth =3D "_post_sy", sixteen =3D {28767, 29551, 24436, = 31091}, thirtytwo =3D {1936683103,=A0
=A0 =A0 =A0 =A0 =A0 =A0 2037604212}, sixtyfour =3D {8751443454668533855}, unin= tptr =3D {1936683103}, ptr =3D 0x736f705f}, PH_loc =3D {
=A0 =A0 =A0 =A0 =A0 eigth =3D "nc\000\000\000\000\000", sixteen = =3D {25454, 0, 0, 0}, thirtytwo =3D {25454, 0}, sixtyfour =3D {25454}, unin= tptr =3D {25454},=A0
=A0 =A0 =A0 =A0= =A0 ptr =3D 0x636e}}, MH_dat =3D {MH_ext =3D {ref_cnt =3D 0x0, ext_buf =3D= 0x0, ext_size =3D 0, ext_type =3D 0, ext_flags =3D 0, ext_free =3D 0,=A0
=A0 =A0 =A0 =A0 =A0 ext_arg1 =3D 0x0, ext_arg2 = =3D 0x0},=A0
=A0 =A0 =A0 =A0 MH_databuf =3D '\0' <repeats 56 times>, &q= uot;file", '\0' <repeats 20 times>, "\006\000\000\0= 00\020\000\000\000??\215?\000\000C\001\000\000\000\000\000\000\000\000\004\= 000\000\000\000\000\000\00000Y?", '\0' <repeats 12 times>= ;, "`2Y?\000\000\000\000\000\000\000\000T\211\223?\022\000\000\000\000= \203??\000\000\000\000\000???", '\0' <repeats 23 times>}= },=A0
=A0 =A0 M_datab= uf =3D "\000\000\000\000\004\000\000\000\200????E??@??\200??shutdown_p= ost_sync", '\0' <repeats 62 times>, "file", &#= 39;\0' <repeats 20 times>, "\006\000\000\000\020\000\000\000= ??\215?\000\000C\001\000\000\000\000\000\000\000\000\004\000\000\000\000\00= 0\000\00000Y?", '\0' <repeats 12 times>, "`2Y?\000\= 000\000\000\000\000\000\000T\211\223?\022\000\000\000\000\203??\000\000\000= \000\000???", '\0' <repeats 23 times>}}
This is 10.0-PRERELEASE r259547M (with a= pplied the recent nd6_nbr.c rtfree patch, which I =A0thought earlier might = be the cause of the panics I'm seeing).

The machine= is Soekris Net5501-70 with this kernel config :

cpu I586_CPU
=
cpu I686_CPU
=
ident MARS
options CPU_GEODE<= /div>
options =A0 =A0 CPU_SOEKRIS

options HZ=3D2000
options DEVI= CE_POLLING
options BPF_JITTER

makeoptions DEBUG=3D-g # Build kernel with gdb(1) debug symbols

options SCHED_ULE # ULE sc= heduler
options PREE= MPTION # Enable kernel thread = preemption
options INET= # InterNETworking
options INET6 # IPv6 communications protocols
options TCP_= OFFLOAD # TCP offload
options FFS # Berkeley Fast Filesystem
options SOFTUPDATE= S # Enable FFS soft updates su= pport
options UFS_= DIRHASH # Improve performance = on big directories
options PROC= FS # Process filesystem (requ= ires PSEUDOFS)
options PSEU= DOFS # Pseudo-filesystem frame= work
options GEOM= _PART_GPT # GUID Partition Tab= les.
options GEOM= _LABEL # Provides labelization=
options COMP= AT_FREEBSD4 # Compatible with = FreeBSD4
options COMP= AT_FREEBSD5 # Compatible with = FreeBSD5
options COMP= AT_FREEBSD6 # Compatible with = FreeBSD6
options COMP= AT_FREEBSD7 # Compatible with = FreeBSD7
options SCSI= _DELAY=3D500 # Delay (in ms) b= efore probing SCSI
options KTRA= CE # ktrace(1) support
options STAC= K # stack(9) support
options SYSVSHM # SYSV-style shared memory
options SYSV= MSG # SYSV-style message queu= es
options SYSV= SEM # SYSV-style semaphores
options _KPO= SIX_PRIORITY_SCHEDULING # POSIX P1003_1B real-time extensions
options PRINTF_BUFR_SIZE= =3D128 # Prevent printf output = being interspersed.
options KBD_= INSTALL_CDEV # install a CDEV e= ntry in /dev
options HWPM= C_HOOKS # Necessary kernel hoo= ks for hwpmc(4)
options CAPA= BILITY_MODE # Capsicum capabil= ity mode
options CAPA= BILITIES # Capsicum capabiliti= es
options PROC= DESC # Support for process des= criptors
options INCL= UDE_CONFIG_FILE =A0 =A0 # Include this file in kernel

# Debugging support. =A0Always need this:
options KDB = # Enable kernel debugger support.
options KDB_TRACE # Print a stack trace for a panic.
options KDB_UNATTENDED

=
options TEXT= DUMP_PREFERRED
options TEXTDUMP_VERBOSE

device pci
device ata= # Legacy ATA/SATA controllers
options ATA_= STATIC_ID # Static device numbe= ring

# ATA/SCSI peripherals
<= /div>
device scbus # SCSI bus (required for ATA/SCSI)
device da # Direct Access (disks)
device pass # Passthrough device (direct ATA/SCS= I access)

# Add suspend/resume support = for the i8254.
device pmtim= er
<= div class=3D"gmail_extra">
# Serial (COM) ports
dev= ice uart # Generic UART driver

device miibus = # MII bus support
device vr # VIA Rhine, Rhine II

# Wireless NIC cards
device wlan <= /span># 802.11 support
options IEEE80211_DEBUG # enable d= ebug msgs
options IEEE80211_AMPDU_AGE # age frames in AMPDU reorder q's
<= /div>
options IEEE80211_SUPPORT_MESH= # enable 802.11s draft support=
device wlan_wep # 802.11 WEP suppo= rt
device wlan_ccmp # 802.11 CCMP sup= port
device wlan_tkip # 802.11 TKIP sup= port
device wlan_amrr # AMRR transmit r= ate control algorithm
device ath # Atheros NICs
device ath_pci # Atheros pci/cardbus glue
<= div class=3D"gmail_extra">
device ath_hal # pci/cardbus chip support
<= div class=3D"gmail_extra">
options AH_SUPPORT_AR5416 # enable AR5416 tx/rx descriptors
options AH_AR5416_INTERRUPT_MI= TIGATION # AR5416 interrupt mit= igation
options ATH_ENABLE_11N # Enable 80= 2.11n support for AR5416 and later
device ath_rate_sample # SampleRate tx rate control for ath

# Pseudo devices.
device loop # Network loopback
device random # Entropy device
devi= ce ether # Ethernet support
device vlan<= span style=3D"white-space:pre-wrap"> # 802.1Q VLAN support
device tun # Packet tunnel.
device md # Memory "disks"
device gif # IPv6 and IPv4 tunneling
device =A0 =A0 =A0gre=
device faith # IPv6-to-IPv4 relaying= (translation)
device firmw= are # firmware assist module
device =A0 =A0 =A0if_bridge

=
options =A0 =A0 VIMAGE
options =A0 =A0 ROUTETA= BLES=3D8
options =A0 =A0 RADIX_MPATH

options =A0 =A0 SW_WA= TCHDOG

device =A0 =A0 =A0crypto
= device =A0 =A0 =A0cryptodev
device =A0 =A0 =A0glxsb
<= br>
=
options =A0 =A0 BOOTVERBOSE=3D1

#device =A0 =A0 =A0pf
#device =A0 =A0 =A0pflog
#device = =A0 =A0 =A0pfsync
device =A0 =A0 =A0 carp
device =A0= =A0 =A0 enc
device =A0 =A0 =A0 lagg
device =A0 =A0 = =A0 epair

#options =A0 =A0 ALTQ
#options =A0 =A0 ALTQ= _CBQ
#options =A0 =A0 ALTQ_RED
#options =A0 =A0 ALTQ= _RIO
#options =A0 =A0 ALTQ_HFSC
#options =A0 =A0 ALTQ= _PRIQ

options =A0 =A0 IPFIREWALL
options =A0 =A0 IPFIREWALL_DEFAULT_TO_ACCEPT
options =A0 =A0 IPFIR= EWALL_NAT
options =A0 =A0 LIBALIAS
=
options =A0 =A0 IPDIV= ERT
=
options =A0 =A0 DUMMYNET

device bpf # Berkeley packet filter

# USB support
options USB_DEBUG # enable debug msgs
device uhci <= /span># UHCI PCI->USB interface
device ohci <= /span># OHCI PCI->USB interface
device ehci <= /span># EHCI PCI->USB interface (USB 2.0)
device usb =A0 =A0 # USB Bus (required)
device umass = # Disks/Mass storage - Requires scbus and da

A= lso src.conf and make.conf :

root@vpn_vrf:[VNET(x)]:/usr/src/sys # = cat /etc/src.conf=A0
WITHOUT_ACCT=3Dyes
WITHOUT_ACPI=3Dyes
WITHOUT_AMD=3Dyes
=
WITH= OUT_APM=3Dyes
WITH= OUT_ASSERT_DEBUG=3Dyes
WITHOUT_AT=3Dyes
WITHOUT_ATF=3Dyes
WITHOUT_ATM=3Dyes
<= /div>
WITHO= UT_AUDIT=3Dyes
WITH= OUT_BLUETOOTH=3Dyes
WITHOUT_CALENDAR=3Dyes
WITHOUT_CDDL=3Dyes
WITHOUT_CTM=3Dyes
=
WITH= OUT_DICT=3Dyes
WITH= OUT_FLOPPY=3Dyes
WITHOUT_GAMES=3Dyes
WITHOUT_HTML=3Dyes
WITHOUT_INFO=3Dyes
WITHOUT_IPFILTER=3Dyes
WITHOUT_IPX=3Dye= s
#WITHOUT_KERNEL_SYMBOLS=3Dyes
WITHOUT_LEGACY_CONSOLE=3Dyes
WITHOUT_LOCALES=3Dyes
<= div class=3D"gmail_extra">
WITHOUT_LPR=3Dyes=
WITHOUT_MA= IL=3Dyes
WITHOUT_NDIS=3Dyes
WITHOUT_QUOTAS=3Dyes
WITHOUT_ROUTED=3Dyes
WITHOUT_SENDMAIL=3Dyes
WITH_SVN=3Dyes
= WITHOUT_ZFS=3Dyes

root@vpn_vrf:[VNET(x)]:/usr/src/sys # cat /etc/make.conf
CFLAGS= =3D-O2
COPT= FLAGS=3D -O -pipe
CPUTYPE=3Dgeode
KERNCONF=3DMARS
NO_MODULES=3Dyes
BOOTWAIT=3D0
DOC_LANG=3Den_US.ISO8859-1


=
--Nikolay


Also, originally I = thought that the panic is when a multi path route is being deleted, however= again from the coredump it seems that the panic happens when openvpn delet= es the host route it installs for the remote openvpn server pointed to the = default gw (before openvpn installs the new default gw pointing to the vpn = tunnel) :

(kgdb) fr 12
(kgdb) x/12sb td->td_proc->= ;p_args
0xc4269780: =A0"\001"
0xc4269782: =A0""
0xc4269783: =A0"= ;"
0xc4269784: =A0"B"
0xc4269786: =A0""
0xc4269787: =A0"= ;"
0xc4269788: =A0"/sbin/route"
0xc4269794: =A0"delete"
0xc426979b: =A0"= ;-net"
0xc42697a0: =A0"78.90.222.xxx"
0xc42697ad: =A0"10.255.255.0"
0xc42697ba: =A0"= ;255.255.255.255"
(kgdb)=A0

I= 'm trying to reproduce this on a VirtualBox instance now, however so fa= r no luck (no OpenVPN running, just adding and removing routes).


--Nikolay
--001a11c1ba200f4f8504eee88df5--