From owner-freebsd-bugs@FreeBSD.ORG Thu Jul 5 14:40:08 2012 Return-Path: Delivered-To: freebsd-bugs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 813751065675 for ; Thu, 5 Jul 2012 14:40:08 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 314678FC1C for ; Thu, 5 Jul 2012 14:40:07 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.5/8.14.5) with ESMTP id q65Ee7u2090175 for ; Thu, 5 Jul 2012 14:40:07 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.5/8.14.5/Submit) id q65Ee7jX090174; Thu, 5 Jul 2012 14:40:07 GMT (envelope-from gnats) Resent-Date: Thu, 5 Jul 2012 14:40:07 GMT Resent-Message-Id: <201207051440.q65Ee7jX090174@freefall.freebsd.org> Resent-From: FreeBSD-gnats-submit@FreeBSD.org (GNATS Filer) Resent-To: freebsd-bugs@FreeBSD.org Resent-Reply-To: FreeBSD-gnats-submit@FreeBSD.org, Alexander Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 640FE106566B for ; Thu, 5 Jul 2012 14:33:21 +0000 (UTC) (envelope-from nobody@FreeBSD.org) Received: from red.freebsd.org (red.freebsd.org [IPv6:2001:4f8:fff6::22]) by mx1.freebsd.org (Postfix) with ESMTP id 4E9038FC0A for ; Thu, 5 Jul 2012 14:33:21 +0000 (UTC) Received: from red.freebsd.org (localhost [127.0.0.1]) by red.freebsd.org (8.14.4/8.14.4) with ESMTP id q65EXJIJ005674 for ; Thu, 5 Jul 2012 14:33:19 GMT (envelope-from nobody@red.freebsd.org) Received: (from nobody@localhost) by red.freebsd.org (8.14.4/8.14.4/Submit) id q65EXJFK005673; Thu, 5 Jul 2012 14:33:19 GMT (envelope-from nobody) Message-Id: <201207051433.q65EXJFK005673@red.freebsd.org> Date: Thu, 5 Jul 2012 14:33:19 GMT From: Alexander To: freebsd-gnats-submit@FreeBSD.org X-Send-Pr-Version: www-3.1 Cc: Subject: kern/169664: Wrongful replacement of interface connected net route X-BeenThere: freebsd-bugs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Bug reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 05 Jul 2012 14:40:08 -0000 >Number: 169664 >Category: kern >Synopsis: Wrongful replacement of interface connected net route >Confidential: no >Severity: critical >Priority: medium >Responsible: freebsd-bugs >State: open >Quarter: >Keywords: >Date-Required: >Class: sw-bug >Submitter-Id: current-users >Arrival-Date: Thu Jul 05 14:40:06 UTC 2012 >Closed-Date: >Last-Modified: >Originator: Alexander >Release: 9.x, 8.x >Organization: >Environment: FreeBSD 9.0-PRERELEASE FreeBSD 9.0-PRERELEASE #1: Sun Dec 25 15:43:40 MSK 2011 /usr/obj/usr/src/sys/CUSTOM_AMD64 amd64 and 9.0 RELEASE >Description: Hi, This problem led to serious failures in routing at several companies, using FreeBSD 9.0 RELEASE and 8.x too. Route of connected network (specified by ip and mask on interface) changed or vanished during working of BGP process. In different companies uses different BGP daemons (BIRD or OpenBGPD). Now it became clear, the reason of a part of problems (I is not sure that all) was reception from BGP of the announcement of the same subnet (as on the interface). The same occurs and when using OSPF. After that, it is impossible to remove a route. (also replacements of a route can achieve manually, without BGP, if interface is down state while try to add static route). Turning off/on of the correct interface helps only. Or only destroy and recreate vlan interface. But this action (down/up or recreate) in 9.0 RELEASE periodically leads to a kernel panic (Probably, and on 8.x, but is more rare). It is described in the block "How to repeat the problem", and here I give also diagnostics of some strange occurences of similar failures: Case 2 (Jun 2012), Server 1: Route to host from connected network (on vlan7) points to other interface. (route received by BGP). # route -n get 193.232.244.100 route to: 193.232.244.100 destination: 193.232.244.0 mask: 255.255.254.0 gateway: 194.226.x.x interface: vlan5 flags: recvpipe sendpipe ssthresh rtt,msec mtu weight expire 0 0 0 0 1500 1 0 (ifconfig vlan7 see on next case, it is same): Case 1 (March 2012), Server 2 (other): Route to host from connected network points to other interface, because route to connected network is disappeared and default wins: # route -n get 193.232.244.100 route to: 193.232.244.100 destination: default mask: default gateway: 83.137.50.254 interface: bge0 flags: recvpipe sendpipe ssthresh rtt,msec mtu weight expire 0 0 0 0 1500 1 0 Connected interface: # ifconfig vlan7 vlan7: flags=8843 metric 0 mtu 1500 options=3 ether xx:xx:xx:xx:xx:xx inet6 fe80::21b:21ff:fe8b:e93d%vlan7 prefixlen 64 scopeid 0xf inet 193.232.245.xxx netmask 0xfffffe00 broadcast 193.232.245.255 inet 193.232.247.xxx netmask 0xfffffe00 broadcast 193.232.247.255 inet6 2001:7f8:20:101::245:xxx prefixlen 64 inet6 2001:7f8:20:102::247:xxx prefixlen 64 nd6 options=3 media: Ethernet autoselect status: active vlan: 7 parent interface: lagg0 Thus ARP is ok: # arp -an | grep 193.232.244.100 ? (193.232.244.100) at 00:1a:64:99:87:9a on vlan7 expires in 299 seconds [vlan] Fixing: # ifconfig vlan7 down # ifconfig vlan7 up # # route -n get 193.232.244.100 route to: 193.232.244.100 destination: 193.232.244.0 mask: 255.255.254.0 interface: vlan7 flags: recvpipe sendpipe ssthresh rtt,msec mtu weight expire 0 0 0 0 1500 1 0 Case 3 (Jun 2012), server 1: Route is known as connected, but IP not assigned (disappear) on interface (earlier was). Route is present as connected: # route get 193.232.244.0/23 route to: msk-ix-network destination: msk-ix-network mask: 255.255.254.0 interface: vlan7 flags: recvpipe sendpipe ssthresh rtt,msec mtu weight expire 0 0 0 0 1500 1 0 but no IP from this subnet assigned on this interface: (see up in Case1 for good vlan7 configuration). # ifconfig vlan7 vlan7: flags=8843 metric 0 mtu 1500 options=3 ether xx:xx:xx:xx:xx:xx inet6 fe80::xxx:xxxx:xxxx:xxxx%vlan7 prefixlen 64 scopeid 0x11 inet 193.232.247.xxx netmask 0xfffffe00 broadcast 193.232.247.255 inet6 2001:7f8:20:101::245:xxx prefixlen 64 inet6 2001:7f8:20:102::247:xxx prefixlen 64 nd6 options=23 media: Ethernet autoselect status: active vlan: 7 parent interface: lagg0 >How-To-Repeat: Use BGP or OSPF with receive same route as connected interface network, and wait (couple of days). Or shut down the interface, add static route of same net, turn up the interface. After that, it is impossible to remove a route. Or, in some versions of FreeBSD, is possible, but connected route doesn't come back. Sample: Iface without IP: # ifconfig vlan4 vlan4: flags=8843 metric 0 mtu 1500 ether 00:1b:21:8b:e9:3c inet6 fe80::21b:21ff:fe8b:e93c%vlan4 prefixlen 64 scopeid 0x8 media: Ethernet autoselect (1000baseT ) status: active vlan: 4 parent interface: igb0 Adding a static route to anywhere: # route add 192.168.72.0/24 x.x.x.1 add net 192.168.72.0: gateway x.x.x.1 # Check: # route get 192.168.72.11 route to: 192.168.72.11 destination: 192.168.72.0 mask: 255.255.255.0 gateway: x.x.x.1 interface: vlan3 flags: recvpipe sendpipe ssthresh rtt,msec mtu weight expire 0 0 0 0 1500 1 0 Try to add IP, with subnet from added route: # ifconfig vlan4 192.168.72.2/24 alias ifconfig: ioctl (SIOCAIFADDR): File exists On 9.0 PRERELEASE IP wasn't established. On 8.x it is established that is shown: # ifconfig vlan4 vlan4: flags=8843 metric 0 mtu 1500 ether 00:1b:21:8b:e9:3c inet6 fe80::21b:21ff:fe8b:e93c%vlan4 prefixlen 64 scopeid 0x8 inet 192.168.72.2 netmask 0xffffff00 broadcast 192.168.72.255 media: Ethernet autoselect (1000baseT ) status: active vlan: 4 parent interface: igb0 Check now: # route get 192.168.72.11 route to: 192.168.72.11 destination: 192.168.72.0 mask: 255.255.255.0 gateway: x.x.x.1 interface: vlan3 flags: recvpipe sendpipe ssthresh rtt,msec mtu weight expire 0 0 0 0 1500 1 0 Static route are still active and not replace by connected route of added address! Remove static route: # route delete 192.168.72.0/24 delete net 192.168.72.0 # Check: # route get 192.168.72.11 route to: 192.168.72.11 destination: default mask: default gateway: x.x.x.x interface: vlan3 flags: recvpipe sendpipe ssthresh rtt,msec mtu weight expire 0 0 0 0 1500 1 0 Now the route isn't present in general, the default works! Fix by down/up # ifconfig vlan4 down # ifconfig vlan4 up # route get 192.168.72.1 route to: 192.168.72.1 destination: 192.168.72.0 mask: 255.255.255.0 interface: vlan4 flags: recvpipe sendpipe ssthresh rtt,msec mtu weight expire 0 0 0 0 1500 1 0 Now again try to add static: route add 192.168.72.0/24 x.x.x.1 route: writing to routing socket: File exists add net 192.168.72.0: gateway x.x.x.1: route already in table Check: # route get 192.168.72.1 route to: 192.168.72.1 destination: 192.168.72.0 mask: 255.255.255.0 interface: vlan4 flags: recvpipe sendpipe ssthresh rtt,msec mtu weight expire 0 0 0 0 1500 1 0 No successfull. Now with down interface: # ifconfig vlan4 down # route get 192.168.72.1 route to: 192.168.72.1 destination: default mask: default gateway: x.x.x.x interface: vlan3 flags: recvpipe sendpipe ssthresh rtt,msec mtu weight expire 0 0 0 0 1500 1 0 # route add 192.168.72.0/24 x.x.x.1 add net 192.168.72.0: gateway x.x.x.1 Check: # route get 192.168.72.1 route to: 192.168.72.1 destination: 192.168.72.0 mask: 255.255.255.0 gateway: x.x.x.1 interface: vlan3 flags: recvpipe sendpipe ssthresh rtt,msec mtu weight expire 0 0 0 0 1500 1 0 Added successfully. Now up interface: #ifconfig vlan4 up Check: # route get 192.168.72.1 route to: 192.168.72.1 destination: 192.168.72.0 mask: 255.255.255.0 gateway: sputnik interface: vlan3 flags: recvpipe sendpipe ssthresh rtt,msec mtu weight expire 0 0 0 0 1500 1 0 Trouble repeated. >Fix: Workaround: ifconfig X down ifconfig X up or (in same cases) ifconfig X destroy ifconfig X create Attention: On 9.0 RELEASE this actions periodically leads to a kernel panic! >Release-Note: >Audit-Trail: >Unformatted: