From owner-freebsd-net@FreeBSD.ORG Wed Jan 1 17:39:46 2014 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 568EB76E; Wed, 1 Jan 2014 17:39:46 +0000 (UTC) Received: from mail-lb0-x22f.google.com (mail-lb0-x22f.google.com [IPv6:2a00:1450:4010:c04::22f]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 678541672; Wed, 1 Jan 2014 17:39:45 +0000 (UTC) Received: by mail-lb0-f175.google.com with SMTP id w6so6739657lbh.6 for ; Wed, 01 Jan 2014 09:39:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type:content-transfer-encoding; bh=kZvagYU709K9yZsz2Pe+lQ+hSIQH14qASTUFA3w2Hcg=; b=I3ICXmwx+ODDr6YyvL1WTd5bGGTn8e5M4/L6gGQIzRyVYynI8FD1+SRMVi+mBsy0o6 Yzymh7zK92qLKa87OWrFGvNCDpn2AOcklge787qXH+u2qZyq1BcgUF/OLG4EwY8gUf4G 8HEZQaQjUhLNnUyknN9oEO+dE/8IcBgjMPU3Z/23On2kLrD3a0yrQFosCDufb4SyUlIQ +vvKq4T4fAdUTShWpPy33RkBvg67WjMcjESHkR+goPkPq5blgKIHuyUJcdrcDzE0hXNf fOvMnpikZ9N2BBnXazIoEHfI9vGd9DOcwfaw0HKndMoMaioUhIhsOD4Uz+OrdpPAzu0c 7T3w== MIME-Version: 1.0 X-Received: by 10.152.23.39 with SMTP id j7mr19871082laf.28.1388597982231; Wed, 01 Jan 2014 09:39:42 -0800 (PST) Received: by 10.114.242.33 with HTTP; Wed, 1 Jan 2014 09:39:42 -0800 (PST) In-Reply-To: References: <201312221304.rBMD4q38060416@oldred.freebsd.org> <201312221310.rBMDA0KH022980@freefall.freebsd.org> Date: Wed, 1 Jan 2014 17:39:42 +0000 Message-ID: Subject: Re: misc/185092: panic: rtfree 2 (using RADIX_MPATH in a VNET jail) From: Nikolay Denev To: FreeBSD-gnats-submit@freebsd.org, freebsd-bugs@freebsd.org, "freebsd-net@freebsd.org" Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 01 Jan 2014 17:39:46 -0000 Ok, killing openvpn with -9 leaves the routes around, and particularly interesting are the following routes : 78.90.222.xx 10.255.255.0 UGHS 0 5841 epair0 =3D= > 78.90.222.xx/32 10.255.255.0 UGS 0 0 epair0 Now, if I do : route delete 78.90.222.xx 10.255.255.0 The route, with the H flag is deleted. If I repeat the command the second route is deleted as well, even if the second command specifies a netmask no panic. However the first delete command specifies the /32 mask like this : route delete 78.90.222.xx 10.255.255.0 255.255.255.255 Then I get "rtfree 2" kernel panic immediately. This seems to be happening as I'm manually installing static routes in the vnet jail for the VPN remote endpoints , however OpenVPN adds such routes too however differently, which results in two routing entries. For example : route add $IP $GW and route add $IP $GW 255.255.255.255 add to different route entries, one is /32 network, the other is a host rou= te. --Nikolay On Wed, Jan 1, 2014 at 1:21 PM, Nikolay Denev wrote: > On Wed, Jan 1, 2014 at 1:10 PM, Nikolay Denev wrote: >> >> On Sun, Dec 22, 2013 at 1:10 PM, wrot= e: >>> >>> Thank you very much for your problem report. >>> It has the internal identification `misc/185092'. >>> The individual assigned to look at your >>> report is: freebsd-bugs. >>> >>> You can access the state of your problem report at any time >>> via this link: >>> >>> http://www.freebsd.org/cgi/query-pr.cgi?pr=3D185092 >>> >>> >Category: misc >>> >Responsible: freebsd-bugs >>> >Synopsis: panic: rtfree 2 (using RADIX_MPATH in a VNET jail) >>> >Arrival-Date: Sun Dec 22 13:10:00 UTC 2013 >> >> >> I'm trying to understand exactly what is happening here, and examining a >> core dump with kgdb I'm getting some output that confuses me : >> >> (kgdb) bt >> #0 doadump (textdump=3D-1011569920) at pcpu.h:233 >> #1 0xc06069b2 in kern_reboot (howto=3D260) at >> /usr/src/sys/kern/kern_shutdown.c:447 >> #2 0xc0606d0e in panic (fmt=3D) at >> /usr/src/sys/kern/kern_shutdown.c:754 >> #3 0xc06de639 in rtfree (rt=3D) at >> /usr/src/sys/net/route.c:464 >> #4 0xc06e188d in route_output (m=3D) at >> /usr/src/sys/net/rtsock.c:951 >> #5 0xc06de18f in raw_usend (so=3D, flags=3D0, m=3D= > optimized out>, nam=3D0x0, control=3D, >> td=3D0xc3bd2000) at /usr/src/sys/net/raw_usrreq.c:238 >> #6 0xc066eca9 in sosend_generic (so=3D0xc3e9c1a8, uio=3D> out>, top=3D, control=3D0x0, >> flags=3D, td=3D) at >> /usr/src/sys/kern/uipc_socket.c:1271 >> #7 0xc066efc7 in sosend (so=3D0xc3e9c1a8, addr=3D0x0, uio=3D0xd9b9cc10, >> top=3D0x0, control=3D0x0, flags=3D0, td=3D0xc3bd2000) >> at /usr/src/sys/kern/uipc_socket.c:1315 >> #8 0xc0654af4 in soo_write (fp=3D0xc3c0c818, uio=3D0xd9b9cc10, >> active_cred=3D0xc3f1dd00, flags=3D0, td=3D0xc3bd2000) >> at /usr/src/sys/kern/sys_socket.c:103 >> #9 0xc064c866 in dofilewrite (td=3D0xc3bd2000, fd=3D3, fp=3D0xc3c0c818, >> auio=3D0xd9b9cc10, offset=3D-1, flags=3D0) at file.h:303 >> #10 0xc064c566 in kern_writev (td=3D0xc3bd2000, fd=3D3, auio=3D> out>) at /usr/src/sys/kern/sys_generic.c:467 >> #11 0xc064c4bc in sys_write (td=3D, uap=3D> optimized out>) at /usr/src/sys/kern/sys_generic.c:382 >> #12 0xc08614d3 in syscall (frame=3D) at >> subr_syscall.c:134 >> #13 0xc084cca1 in Xint0x80_syscall () at >> /usr/src/sys/i386/i386/exception.s:270 >> #14 0x281975b7 in ?? () >> Previous frame inner to this frame (corrupt stack?) >> Current language: auto; currently minimal >> (kgdb) fr 3 >> #3 0xc06de639 in rtfree (rt=3D) at >> /usr/src/sys/net/route.c:464 >> 464 panic("rtfree 2"); >> (kgdb) print *rt >> $1 =3D {rt_nodes =3D {{rn_mklist =3D 0xc3b4ab30, rn_parent =3D 0x1, rn_b= it =3D 0, >> rn_bmask =3D 0 '\0', rn_flags =3D 0 '\0', rn_u =3D {rn_leaf =3D { >> rn_Key =3D 0xc0882687 "shutdown_post_sync", rn_Mask =3D 0x1030= 000 >>
, rn_Dupedkey =3D 0x0}, rn_node =3D { >> rn_Off =3D -1064819065, rn_L =3D 0x1030000, rn_R =3D 0x0}}}, >> {rn_mklist =3D 0x0, rn_parent =3D 0x4, rn_bit =3D -18048, rn_bmask =3D -= 94 '?', >> rn_flags =3D 195 '?', rn_u =3D {rn_leaf =3D {rn_Key =3D 0xc3a545e0= "", >> rn_Mask =3D 0xc3a4e440 " ??(???\020'", rn_Dupedkey =3D 0xc3a4e880}, >> rn_node =3D {rn_Off =3D -1012578848, rn_L =3D 0xc3a4e440, rn_R = =3D >> 0xc3a4e880}}}}, rt_gateway =3D 0x74756873, rt_flags =3D 1853321060, >> rt_refcnt =3D 1936683103, rt_ifp =3D 0x79735f74, rt_ifa =3D 0x636e, rt= _rmx =3D >> {rmx_mtu =3D 0, rmx_expire =3D 0, rmx_pksent =3D 0, rmx_weight =3D 0}, >> rt_fibnum =3D 0, rt_mtx =3D {lock_object =3D {lo_name =3D 0x0, lo_flag= s =3D 0, >> lo_data =3D 0, lo_witness =3D 0x0}, mtx_lock =3D 0}} >> >> >> >> rn_Key with value of =93shutdown_post_sync=94 ? >> >> It=92s visible also in the raw_usend() frame: >> >> (kgdb) fr 5 >> #5 0xc06de18f in raw_usend (so=3D, flags=3D0, m=3D= > optimized out>, nam=3D0x0, control=3D, >> td=3D0xc3bd2000) at /usr/src/sys/net/raw_usrreq.c:238 >> 238 return ((*so->so_proto->pr_output)(m, so)); >> (kgdb) print *m >> $2 =3D {m_hdr =3D {mh_next =3D 0xc3b4ab30, mh_nextpkt =3D 0x1, mh_data = =3D 0x0, >> mh_len =3D -1064819065, mh_type =3D 0, mh_flags =3D 66304, mh_pad =3D 0}= , >> M_dat =3D {MH =3D {MH_pkthdr =3D {rcvif =3D 0x0, tags =3D {slh_first = =3D 0x4}, len =3D >> -1012745856, flowid =3D 3282388448, >> csum_flags =3D 14097648373312316480, fibnum =3D 26739, cosqos = =3D 117 >> 'u', rsstype =3D 116 't', l2hlen =3D 100 'd', l3hlen =3D 111 'o', >> l4hlen =3D 119 'w', l5hlen =3D 110 'n', PH_per =3D {eigth =3D "_= post_sy", >> sixteen =3D {28767, 29551, 24436, 31091}, thirtytwo =3D {1936683103, >> 2037604212}, sixtyfour =3D {8751443454668533855}, unintptr = =3D >> {1936683103}, ptr =3D 0x736f705f}, PH_loc =3D { >> eigth =3D "nc\000\000\000\000\000", sixteen =3D {25454, 0, 0, = 0}, >> thirtytwo =3D {25454, 0}, sixtyfour =3D {25454}, unintptr =3D {25454}, >> ptr =3D 0x636e}}, MH_dat =3D {MH_ext =3D {ref_cnt =3D 0x0, ext= _buf =3D >> 0x0, ext_size =3D 0, ext_type =3D 0, ext_flags =3D 0, ext_free =3D 0, >> ext_arg1 =3D 0x0, ext_arg2 =3D 0x0}, >> MH_databuf =3D '\0' , "file", '\0' > times>, >> "\006\000\000\000\020\000\000\000??\215?\000\000C\001\000\000\000\000\00= 0\000\000\000\004\000\000\000\000\000\000\00000Y?", >> '\0' , >> "`2Y?\000\000\000\000\000\000\000\000T\211\223?\022\000\000\000\000\203?= ?\000\000\000\000\000???", >> '\0' }}, >> M_databuf =3D >> "\000\000\000\000\004\000\000\000\200????E??@??\200??shutdown_post_sync"= , >> '\0' , "file", '\0' , >> "\006\000\000\000\020\000\000\000??\215?\000\000C\001\000\000\000\000\00= 0\000\000\000\004\000\000\000\000\000\000\00000Y?", >> '\0' , >> "`2Y?\000\000\000\000\000\000\000\000T\211\223?\022\000\000\000\000\203?= ?\000\000\000\000\000???", >> '\0' }} >> >> >> This is 10.0-PRERELEASE r259547M (with applied the recent nd6_nbr.c rtfr= ee >> patch, which I thought earlier might be the cause of the panics I'm >> seeing). >> >> The machine is Soekris Net5501-70 with this kernel config : >> >> cpu I586_CPU >> cpu I686_CPU >> ident MARS >> options CPU_GEODE >> options CPU_SOEKRIS >> >> options HZ=3D2000 >> options DEVICE_POLLING >> options BPF_JITTER >> >> makeoptions DEBUG=3D-g # Build kernel with gdb(1) debug symbols >> >> options SCHED_ULE # ULE scheduler >> options PREEMPTION # Enable kernel thread preemption >> options INET # InterNETworking >> options INET6 # IPv6 communications protocols >> options TCP_OFFLOAD # TCP offload >> options FFS # Berkeley Fast Filesystem >> options SOFTUPDATES # Enable FFS soft updates support >> options UFS_DIRHASH # Improve performance on big directories >> options PROCFS # Process filesystem (requires PSEUDOFS) >> options PSEUDOFS # Pseudo-filesystem framework >> options GEOM_PART_GPT # GUID Partition Tables. >> options GEOM_LABEL # Provides labelization >> options COMPAT_FREEBSD4 # Compatible with FreeBSD4 >> options COMPAT_FREEBSD5 # Compatible with FreeBSD5 >> options COMPAT_FREEBSD6 # Compatible with FreeBSD6 >> options COMPAT_FREEBSD7 # Compatible with FreeBSD7 >> options SCSI_DELAY=3D500 # Delay (in ms) before probing SCSI >> options KTRACE # ktrace(1) support >> options STACK # stack(9) support >> options SYSVSHM # SYSV-style shared memory >> options SYSVMSG # SYSV-style message queues >> options SYSVSEM # SYSV-style semaphores >> options _KPOSIX_PRIORITY_SCHEDULING # POSIX P1003_1B real-time extension= s >> options PRINTF_BUFR_SIZE=3D128 # Prevent printf output being intersperse= d. >> options KBD_INSTALL_CDEV # install a CDEV entry in /dev >> options HWPMC_HOOKS # Necessary kernel hooks for hwpmc(4) >> options CAPABILITY_MODE # Capsicum capability mode >> options CAPABILITIES # Capsicum capabilities >> options PROCDESC # Support for process descriptors >> options INCLUDE_CONFIG_FILE # Include this file in kernel >> >> # Debugging support. Always need this: >> options KDB # Enable kernel debugger support. >> options KDB_TRACE # Print a stack trace for a panic. >> options KDB_UNATTENDED >> >> options TEXTDUMP_PREFERRED >> options TEXTDUMP_VERBOSE >> >> device pci >> device ata # Legacy ATA/SATA controllers >> options ATA_STATIC_ID # Static device numbering >> >> # ATA/SCSI peripherals >> device scbus # SCSI bus (required for ATA/SCSI) >> device da # Direct Access (disks) >> device pass # Passthrough device (direct ATA/SCSI access) >> >> # Add suspend/resume support for the i8254. >> device pmtimer >> >> # Serial (COM) ports >> device uart # Generic UART driver >> >> device miibus # MII bus support >> device vr # VIA Rhine, Rhine II >> >> # Wireless NIC cards >> device wlan # 802.11 support >> options IEEE80211_DEBUG # enable debug msgs >> options IEEE80211_AMPDU_AGE # age frames in AMPDU reorder q's >> options IEEE80211_SUPPORT_MESH # enable 802.11s draft support >> device wlan_wep # 802.11 WEP support >> device wlan_ccmp # 802.11 CCMP support >> device wlan_tkip # 802.11 TKIP support >> device wlan_amrr # AMRR transmit rate control algorithm >> device ath # Atheros NICs >> device ath_pci # Atheros pci/cardbus glue >> device ath_hal # pci/cardbus chip support >> options AH_SUPPORT_AR5416 # enable AR5416 tx/rx descriptors >> options AH_AR5416_INTERRUPT_MITIGATION # AR5416 interrupt mitigation >> options ATH_ENABLE_11N # Enable 802.11n support for AR5416 and later >> device ath_rate_sample # SampleRate tx rate control for ath >> >> # Pseudo devices. >> device loop # Network loopback >> device random # Entropy device >> device ether # Ethernet support >> device vlan # 802.1Q VLAN support >> device tun # Packet tunnel. >> device md # Memory "disks" >> device gif # IPv6 and IPv4 tunneling >> device gre >> device faith # IPv6-to-IPv4 relaying (translation) >> device firmware # firmware assist module >> device if_bridge >> >> options VIMAGE >> options ROUTETABLES=3D8 >> options RADIX_MPATH >> >> options SW_WATCHDOG >> >> device crypto >> device cryptodev >> device glxsb >> >> options BOOTVERBOSE=3D1 >> >> #device pf >> #device pflog >> #device pfsync >> device carp >> device enc >> device lagg >> device epair >> >> #options ALTQ >> #options ALTQ_CBQ >> #options ALTQ_RED >> #options ALTQ_RIO >> #options ALTQ_HFSC >> #options ALTQ_PRIQ >> >> options IPFIREWALL >> options IPFIREWALL_DEFAULT_TO_ACCEPT >> options IPFIREWALL_NAT >> options LIBALIAS >> options IPDIVERT >> options DUMMYNET >> >> device bpf # Berkeley packet filter >> >> # USB support >> options USB_DEBUG # enable debug msgs >> device uhci # UHCI PCI->USB interface >> device ohci # OHCI PCI->USB interface >> device ehci # EHCI PCI->USB interface (USB 2.0) >> device usb # USB Bus (required) >> device umass # Disks/Mass storage - Requires scbus and da >> >> >> Also src.conf and make.conf : >> >> root@vpn_vrf:[VNET(x)]:/usr/src/sys # cat /etc/src.conf >> WITHOUT_ACCT=3Dyes >> WITHOUT_ACPI=3Dyes >> WITHOUT_AMD=3Dyes >> WITHOUT_APM=3Dyes >> WITHOUT_ASSERT_DEBUG=3Dyes >> WITHOUT_AT=3Dyes >> WITHOUT_ATF=3Dyes >> WITHOUT_ATM=3Dyes >> WITHOUT_AUDIT=3Dyes >> WITHOUT_BLUETOOTH=3Dyes >> WITHOUT_CALENDAR=3Dyes >> WITHOUT_CDDL=3Dyes >> WITHOUT_CTM=3Dyes >> WITHOUT_DICT=3Dyes >> WITHOUT_FLOPPY=3Dyes >> WITHOUT_GAMES=3Dyes >> WITHOUT_HTML=3Dyes >> WITHOUT_INFO=3Dyes >> WITHOUT_IPFILTER=3Dyes >> WITHOUT_IPX=3Dyes >> #WITHOUT_KERNEL_SYMBOLS=3Dyes >> WITHOUT_LEGACY_CONSOLE=3Dyes >> WITHOUT_LOCALES=3Dyes >> WITHOUT_LPR=3Dyes >> WITHOUT_MAIL=3Dyes >> WITHOUT_NDIS=3Dyes >> WITHOUT_QUOTAS=3Dyes >> WITHOUT_ROUTED=3Dyes >> WITHOUT_SENDMAIL=3Dyes >> WITH_SVN=3Dyes >> WITHOUT_ZFS=3Dyes >> >> root@vpn_vrf:[VNET(x)]:/usr/src/sys # cat /etc/make.conf >> CFLAGS=3D-O2 >> COPTFLAGS=3D -O -pipe >> CPUTYPE=3Dgeode >> KERNCONF=3DMARS >> NO_MODULES=3Dyes >> BOOTWAIT=3D0 >> DOC_LANG=3Den_US.ISO8859-1 >> >> >> >> --Nikolay >> > > Also, originally I thought that the panic is when a multi path route is > being deleted, however again from the coredump it seems that the panic > happens when openvpn deletes the host route it installs for the remote > openvpn server pointed to the default gw (before openvpn installs the new > default gw pointing to the vpn tunnel) : > > (kgdb) fr 12 > (kgdb) x/12sb td->td_proc->p_args > 0xc4269780: "\001" > 0xc4269782: "" > 0xc4269783: "" > 0xc4269784: "B" > 0xc4269786: "" > 0xc4269787: "" > 0xc4269788: "/sbin/route" > 0xc4269794: "delete" > 0xc426979b: "-net" > 0xc42697a0: "78.90.222.xxx" > 0xc42697ad: "10.255.255.0" > 0xc42697ba: "255.255.255.255" > (kgdb) > > I'm trying to reproduce this on a VirtualBox instance now, however so far= no > luck (no OpenVPN running, just adding and removing routes). > > > --Nikolay