Date: Wed, 1 Jan 2014 17:39:42 +0000 From: Nikolay Denev <ndenev@gmail.com> To: FreeBSD-gnats-submit@freebsd.org, freebsd-bugs@freebsd.org, "freebsd-net@freebsd.org" <freebsd-net@freebsd.org> Subject: Re: misc/185092: panic: rtfree 2 (using RADIX_MPATH in a VNET jail) Message-ID: <CA%2BP_MZF4pMYqXV-BbrVfUjirfD7=qzr5Nz=k1NvOz6_wmT4GrQ@mail.gmail.com> In-Reply-To: <CA%2BP_MZEqn4iK%2BWC-ouVNFEhMrq2tpH91qgcT5zJyaUnX%2BkVn1g@mail.gmail.com> References: <201312221304.rBMD4q38060416@oldred.freebsd.org> <201312221310.rBMDA0KH022980@freefall.freebsd.org> <CA%2BP_MZHrGB0OdddCmhpOptA-sBUHOERCdjSfUFh20pJOQsZ7Kw@mail.gmail.com> <CA%2BP_MZEqn4iK%2BWC-ouVNFEhMrq2tpH91qgcT5zJyaUnX%2BkVn1g@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Ok, killing openvpn with -9 leaves the routes around, and particularly
interesting are the following routes :
78.90.222.xx 10.255.255.0 UGHS 0 5841 epair0 =3D=
>
78.90.222.xx/32 10.255.255.0 UGS 0 0 epair0
Now, if I do :
route delete 78.90.222.xx 10.255.255.0
The route, with the H flag is deleted. If I repeat the command the
second route is deleted as well, even if the second command specifies
a netmask no panic.
However the first delete command specifies the /32 mask like this :
route delete 78.90.222.xx 10.255.255.0 255.255.255.255
Then I get "rtfree 2" kernel panic immediately.
This seems to be happening as I'm manually installing static routes in
the vnet jail for the VPN remote endpoints , however OpenVPN adds such
routes too however differently, which results in two routing entries.
For example :
route add $IP $GW
and
route add $IP $GW 255.255.255.255
add to different route entries, one is /32 network, the other is a host rou=
te.
--Nikolay
On Wed, Jan 1, 2014 at 1:21 PM, Nikolay Denev <ndenev@gmail.com> wrote:
> On Wed, Jan 1, 2014 at 1:10 PM, Nikolay Denev <ndenev@gmail.com> wrote:
>>
>> On Sun, Dec 22, 2013 at 1:10 PM, <FreeBSD-gnats-submit@freebsd.org> wrot=
e:
>>>
>>> Thank you very much for your problem report.
>>> It has the internal identification `misc/185092'.
>>> The individual assigned to look at your
>>> report is: freebsd-bugs.
>>>
>>> You can access the state of your problem report at any time
>>> via this link:
>>>
>>> http://www.freebsd.org/cgi/query-pr.cgi?pr=3D185092
>>>
>>> >Category: misc
>>> >Responsible: freebsd-bugs
>>> >Synopsis: panic: rtfree 2 (using RADIX_MPATH in a VNET jail)
>>> >Arrival-Date: Sun Dec 22 13:10:00 UTC 2013
>>
>>
>> I'm trying to understand exactly what is happening here, and examining a
>> core dump with kgdb I'm getting some output that confuses me :
>>
>> (kgdb) bt
>> #0 doadump (textdump=3D-1011569920) at pcpu.h:233
>> #1 0xc06069b2 in kern_reboot (howto=3D260) at
>> /usr/src/sys/kern/kern_shutdown.c:447
>> #2 0xc0606d0e in panic (fmt=3D<value optimized out>) at
>> /usr/src/sys/kern/kern_shutdown.c:754
>> #3 0xc06de639 in rtfree (rt=3D<value optimized out>) at
>> /usr/src/sys/net/route.c:464
>> #4 0xc06e188d in route_output (m=3D<value optimized out>) at
>> /usr/src/sys/net/rtsock.c:951
>> #5 0xc06de18f in raw_usend (so=3D<value optimized out>, flags=3D0, m=3D=
<value
>> optimized out>, nam=3D0x0, control=3D<value optimized out>,
>> td=3D0xc3bd2000) at /usr/src/sys/net/raw_usrreq.c:238
>> #6 0xc066eca9 in sosend_generic (so=3D0xc3e9c1a8, uio=3D<value optimize=
d
>> out>, top=3D<value optimized out>, control=3D0x0,
>> flags=3D<value optimized out>, td=3D<value optimized out>) at
>> /usr/src/sys/kern/uipc_socket.c:1271
>> #7 0xc066efc7 in sosend (so=3D0xc3e9c1a8, addr=3D0x0, uio=3D0xd9b9cc10,
>> top=3D0x0, control=3D0x0, flags=3D0, td=3D0xc3bd2000)
>> at /usr/src/sys/kern/uipc_socket.c:1315
>> #8 0xc0654af4 in soo_write (fp=3D0xc3c0c818, uio=3D0xd9b9cc10,
>> active_cred=3D0xc3f1dd00, flags=3D0, td=3D0xc3bd2000)
>> at /usr/src/sys/kern/sys_socket.c:103
>> #9 0xc064c866 in dofilewrite (td=3D0xc3bd2000, fd=3D3, fp=3D0xc3c0c818,
>> auio=3D0xd9b9cc10, offset=3D-1, flags=3D0) at file.h:303
>> #10 0xc064c566 in kern_writev (td=3D0xc3bd2000, fd=3D3, auio=3D<value op=
timized
>> out>) at /usr/src/sys/kern/sys_generic.c:467
>> #11 0xc064c4bc in sys_write (td=3D<value optimized out>, uap=3D<value
>> optimized out>) at /usr/src/sys/kern/sys_generic.c:382
>> #12 0xc08614d3 in syscall (frame=3D<value optimized out>) at
>> subr_syscall.c:134
>> #13 0xc084cca1 in Xint0x80_syscall () at
>> /usr/src/sys/i386/i386/exception.s:270
>> #14 0x281975b7 in ?? ()
>> Previous frame inner to this frame (corrupt stack?)
>> Current language: auto; currently minimal
>> (kgdb) fr 3
>> #3 0xc06de639 in rtfree (rt=3D<value optimized out>) at
>> /usr/src/sys/net/route.c:464
>> 464 panic("rtfree 2");
>> (kgdb) print *rt
>> $1 =3D {rt_nodes =3D {{rn_mklist =3D 0xc3b4ab30, rn_parent =3D 0x1, rn_b=
it =3D 0,
>> rn_bmask =3D 0 '\0', rn_flags =3D 0 '\0', rn_u =3D {rn_leaf =3D {
>> rn_Key =3D 0xc0882687 "shutdown_post_sync", rn_Mask =3D 0x1030=
000
>> <Address 0x1030000 out of bounds>, rn_Dupedkey =3D 0x0}, rn_node =3D {
>> rn_Off =3D -1064819065, rn_L =3D 0x1030000, rn_R =3D 0x0}}},
>> {rn_mklist =3D 0x0, rn_parent =3D 0x4, rn_bit =3D -18048, rn_bmask =3D -=
94 '?',
>> rn_flags =3D 195 '?', rn_u =3D {rn_leaf =3D {rn_Key =3D 0xc3a545e0=
"",
>> rn_Mask =3D 0xc3a4e440 " ??(???\020'", rn_Dupedkey =3D 0xc3a4e880},
>> rn_node =3D {rn_Off =3D -1012578848, rn_L =3D 0xc3a4e440, rn_R =
=3D
>> 0xc3a4e880}}}}, rt_gateway =3D 0x74756873, rt_flags =3D 1853321060,
>> rt_refcnt =3D 1936683103, rt_ifp =3D 0x79735f74, rt_ifa =3D 0x636e, rt=
_rmx =3D
>> {rmx_mtu =3D 0, rmx_expire =3D 0, rmx_pksent =3D 0, rmx_weight =3D 0},
>> rt_fibnum =3D 0, rt_mtx =3D {lock_object =3D {lo_name =3D 0x0, lo_flag=
s =3D 0,
>> lo_data =3D 0, lo_witness =3D 0x0}, mtx_lock =3D 0}}
>>
>>
>>
>> rn_Key with value of =93shutdown_post_sync=94 ?
>>
>> It=92s visible also in the raw_usend() frame:
>>
>> (kgdb) fr 5
>> #5 0xc06de18f in raw_usend (so=3D<value optimized out>, flags=3D0, m=3D=
<value
>> optimized out>, nam=3D0x0, control=3D<value optimized out>,
>> td=3D0xc3bd2000) at /usr/src/sys/net/raw_usrreq.c:238
>> 238 return ((*so->so_proto->pr_output)(m, so));
>> (kgdb) print *m
>> $2 =3D {m_hdr =3D {mh_next =3D 0xc3b4ab30, mh_nextpkt =3D 0x1, mh_data =
=3D 0x0,
>> mh_len =3D -1064819065, mh_type =3D 0, mh_flags =3D 66304, mh_pad =3D 0}=
,
>> M_dat =3D {MH =3D {MH_pkthdr =3D {rcvif =3D 0x0, tags =3D {slh_first =
=3D 0x4}, len =3D
>> -1012745856, flowid =3D 3282388448,
>> csum_flags =3D 14097648373312316480, fibnum =3D 26739, cosqos =
=3D 117
>> 'u', rsstype =3D 116 't', l2hlen =3D 100 'd', l3hlen =3D 111 'o',
>> l4hlen =3D 119 'w', l5hlen =3D 110 'n', PH_per =3D {eigth =3D "_=
post_sy",
>> sixteen =3D {28767, 29551, 24436, 31091}, thirtytwo =3D {1936683103,
>> 2037604212}, sixtyfour =3D {8751443454668533855}, unintptr =
=3D
>> {1936683103}, ptr =3D 0x736f705f}, PH_loc =3D {
>> eigth =3D "nc\000\000\000\000\000", sixteen =3D {25454, 0, 0, =
0},
>> thirtytwo =3D {25454, 0}, sixtyfour =3D {25454}, unintptr =3D {25454},
>> ptr =3D 0x636e}}, MH_dat =3D {MH_ext =3D {ref_cnt =3D 0x0, ext=
_buf =3D
>> 0x0, ext_size =3D 0, ext_type =3D 0, ext_flags =3D 0, ext_free =3D 0,
>> ext_arg1 =3D 0x0, ext_arg2 =3D 0x0},
>> MH_databuf =3D '\0' <repeats 56 times>, "file", '\0' <repeats 20
>> times>,
>> "\006\000\000\000\020\000\000\000??\215?\000\000C\001\000\000\000\000\00=
0\000\000\000\004\000\000\000\000\000\000\00000Y?",
>> '\0' <repeats 12 times>,
>> "`2Y?\000\000\000\000\000\000\000\000T\211\223?\022\000\000\000\000\203?=
?\000\000\000\000\000???",
>> '\0' <repeats 23 times>}},
>> M_databuf =3D
>> "\000\000\000\000\004\000\000\000\200????E??@??\200??shutdown_post_sync"=
,
>> '\0' <repeats 62 times>, "file", '\0' <repeats 20 times>,
>> "\006\000\000\000\020\000\000\000??\215?\000\000C\001\000\000\000\000\00=
0\000\000\000\004\000\000\000\000\000\000\00000Y?",
>> '\0' <repeats 12 times>,
>> "`2Y?\000\000\000\000\000\000\000\000T\211\223?\022\000\000\000\000\203?=
?\000\000\000\000\000???",
>> '\0' <repeats 23 times>}}
>>
>>
>> This is 10.0-PRERELEASE r259547M (with applied the recent nd6_nbr.c rtfr=
ee
>> patch, which I thought earlier might be the cause of the panics I'm
>> seeing).
>>
>> The machine is Soekris Net5501-70 with this kernel config :
>>
>> cpu I586_CPU
>> cpu I686_CPU
>> ident MARS
>> options CPU_GEODE
>> options CPU_SOEKRIS
>>
>> options HZ=3D2000
>> options DEVICE_POLLING
>> options BPF_JITTER
>>
>> makeoptions DEBUG=3D-g # Build kernel with gdb(1) debug symbols
>>
>> options SCHED_ULE # ULE scheduler
>> options PREEMPTION # Enable kernel thread preemption
>> options INET # InterNETworking
>> options INET6 # IPv6 communications protocols
>> options TCP_OFFLOAD # TCP offload
>> options FFS # Berkeley Fast Filesystem
>> options SOFTUPDATES # Enable FFS soft updates support
>> options UFS_DIRHASH # Improve performance on big directories
>> options PROCFS # Process filesystem (requires PSEUDOFS)
>> options PSEUDOFS # Pseudo-filesystem framework
>> options GEOM_PART_GPT # GUID Partition Tables.
>> options GEOM_LABEL # Provides labelization
>> options COMPAT_FREEBSD4 # Compatible with FreeBSD4
>> options COMPAT_FREEBSD5 # Compatible with FreeBSD5
>> options COMPAT_FREEBSD6 # Compatible with FreeBSD6
>> options COMPAT_FREEBSD7 # Compatible with FreeBSD7
>> options SCSI_DELAY=3D500 # Delay (in ms) before probing SCSI
>> options KTRACE # ktrace(1) support
>> options STACK # stack(9) support
>> options SYSVSHM # SYSV-style shared memory
>> options SYSVMSG # SYSV-style message queues
>> options SYSVSEM # SYSV-style semaphores
>> options _KPOSIX_PRIORITY_SCHEDULING # POSIX P1003_1B real-time extension=
s
>> options PRINTF_BUFR_SIZE=3D128 # Prevent printf output being intersperse=
d.
>> options KBD_INSTALL_CDEV # install a CDEV entry in /dev
>> options HWPMC_HOOKS # Necessary kernel hooks for hwpmc(4)
>> options CAPABILITY_MODE # Capsicum capability mode
>> options CAPABILITIES # Capsicum capabilities
>> options PROCDESC # Support for process descriptors
>> options INCLUDE_CONFIG_FILE # Include this file in kernel
>>
>> # Debugging support. Always need this:
>> options KDB # Enable kernel debugger support.
>> options KDB_TRACE # Print a stack trace for a panic.
>> options KDB_UNATTENDED
>>
>> options TEXTDUMP_PREFERRED
>> options TEXTDUMP_VERBOSE
>>
>> device pci
>> device ata # Legacy ATA/SATA controllers
>> options ATA_STATIC_ID # Static device numbering
>>
>> # ATA/SCSI peripherals
>> device scbus # SCSI bus (required for ATA/SCSI)
>> device da # Direct Access (disks)
>> device pass # Passthrough device (direct ATA/SCSI access)
>>
>> # Add suspend/resume support for the i8254.
>> device pmtimer
>>
>> # Serial (COM) ports
>> device uart # Generic UART driver
>>
>> device miibus # MII bus support
>> device vr # VIA Rhine, Rhine II
>>
>> # Wireless NIC cards
>> device wlan # 802.11 support
>> options IEEE80211_DEBUG # enable debug msgs
>> options IEEE80211_AMPDU_AGE # age frames in AMPDU reorder q's
>> options IEEE80211_SUPPORT_MESH # enable 802.11s draft support
>> device wlan_wep # 802.11 WEP support
>> device wlan_ccmp # 802.11 CCMP support
>> device wlan_tkip # 802.11 TKIP support
>> device wlan_amrr # AMRR transmit rate control algorithm
>> device ath # Atheros NICs
>> device ath_pci # Atheros pci/cardbus glue
>> device ath_hal # pci/cardbus chip support
>> options AH_SUPPORT_AR5416 # enable AR5416 tx/rx descriptors
>> options AH_AR5416_INTERRUPT_MITIGATION # AR5416 interrupt mitigation
>> options ATH_ENABLE_11N # Enable 802.11n support for AR5416 and later
>> device ath_rate_sample # SampleRate tx rate control for ath
>>
>> # Pseudo devices.
>> device loop # Network loopback
>> device random # Entropy device
>> device ether # Ethernet support
>> device vlan # 802.1Q VLAN support
>> device tun # Packet tunnel.
>> device md # Memory "disks"
>> device gif # IPv6 and IPv4 tunneling
>> device gre
>> device faith # IPv6-to-IPv4 relaying (translation)
>> device firmware # firmware assist module
>> device if_bridge
>>
>> options VIMAGE
>> options ROUTETABLES=3D8
>> options RADIX_MPATH
>>
>> options SW_WATCHDOG
>>
>> device crypto
>> device cryptodev
>> device glxsb
>>
>> options BOOTVERBOSE=3D1
>>
>> #device pf
>> #device pflog
>> #device pfsync
>> device carp
>> device enc
>> device lagg
>> device epair
>>
>> #options ALTQ
>> #options ALTQ_CBQ
>> #options ALTQ_RED
>> #options ALTQ_RIO
>> #options ALTQ_HFSC
>> #options ALTQ_PRIQ
>>
>> options IPFIREWALL
>> options IPFIREWALL_DEFAULT_TO_ACCEPT
>> options IPFIREWALL_NAT
>> options LIBALIAS
>> options IPDIVERT
>> options DUMMYNET
>>
>> device bpf # Berkeley packet filter
>>
>> # USB support
>> options USB_DEBUG # enable debug msgs
>> device uhci # UHCI PCI->USB interface
>> device ohci # OHCI PCI->USB interface
>> device ehci # EHCI PCI->USB interface (USB 2.0)
>> device usb # USB Bus (required)
>> device umass # Disks/Mass storage - Requires scbus and da
>>
>>
>> Also src.conf and make.conf :
>>
>> root@vpn_vrf:[VNET(x)]:/usr/src/sys # cat /etc/src.conf
>> WITHOUT_ACCT=3Dyes
>> WITHOUT_ACPI=3Dyes
>> WITHOUT_AMD=3Dyes
>> WITHOUT_APM=3Dyes
>> WITHOUT_ASSERT_DEBUG=3Dyes
>> WITHOUT_AT=3Dyes
>> WITHOUT_ATF=3Dyes
>> WITHOUT_ATM=3Dyes
>> WITHOUT_AUDIT=3Dyes
>> WITHOUT_BLUETOOTH=3Dyes
>> WITHOUT_CALENDAR=3Dyes
>> WITHOUT_CDDL=3Dyes
>> WITHOUT_CTM=3Dyes
>> WITHOUT_DICT=3Dyes
>> WITHOUT_FLOPPY=3Dyes
>> WITHOUT_GAMES=3Dyes
>> WITHOUT_HTML=3Dyes
>> WITHOUT_INFO=3Dyes
>> WITHOUT_IPFILTER=3Dyes
>> WITHOUT_IPX=3Dyes
>> #WITHOUT_KERNEL_SYMBOLS=3Dyes
>> WITHOUT_LEGACY_CONSOLE=3Dyes
>> WITHOUT_LOCALES=3Dyes
>> WITHOUT_LPR=3Dyes
>> WITHOUT_MAIL=3Dyes
>> WITHOUT_NDIS=3Dyes
>> WITHOUT_QUOTAS=3Dyes
>> WITHOUT_ROUTED=3Dyes
>> WITHOUT_SENDMAIL=3Dyes
>> WITH_SVN=3Dyes
>> WITHOUT_ZFS=3Dyes
>>
>> root@vpn_vrf:[VNET(x)]:/usr/src/sys # cat /etc/make.conf
>> CFLAGS=3D-O2
>> COPTFLAGS=3D -O -pipe
>> CPUTYPE=3Dgeode
>> KERNCONF=3DMARS
>> NO_MODULES=3Dyes
>> BOOTWAIT=3D0
>> DOC_LANG=3Den_US.ISO8859-1
>>
>>
>>
>> --Nikolay
>>
>
> Also, originally I thought that the panic is when a multi path route is
> being deleted, however again from the coredump it seems that the panic
> happens when openvpn deletes the host route it installs for the remote
> openvpn server pointed to the default gw (before openvpn installs the new
> default gw pointing to the vpn tunnel) :
>
> (kgdb) fr 12
> (kgdb) x/12sb td->td_proc->p_args
> 0xc4269780: "\001"
> 0xc4269782: ""
> 0xc4269783: ""
> 0xc4269784: "B"
> 0xc4269786: ""
> 0xc4269787: ""
> 0xc4269788: "/sbin/route"
> 0xc4269794: "delete"
> 0xc426979b: "-net"
> 0xc42697a0: "78.90.222.xxx"
> 0xc42697ad: "10.255.255.0"
> 0xc42697ba: "255.255.255.255"
> (kgdb)
>
> I'm trying to reproduce this on a VirtualBox instance now, however so far=
no
> luck (no OpenVPN running, just adding and removing routes).
>
>
> --Nikolay
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CA%2BP_MZF4pMYqXV-BbrVfUjirfD7=qzr5Nz=k1NvOz6_wmT4GrQ>