Skip site navigation (1)Skip section navigation (2)
Date:      Thu, 5 Jul 2012 14:33:19 GMT
From:      Alexander <xor@davydkovo.net>
To:        freebsd-gnats-submit@FreeBSD.org
Subject:   kern/169664: Wrongful replacement of interface connected net route
Message-ID:  <201207051433.q65EXJFK005673@red.freebsd.org>
Resent-Message-ID: <201207051440.q65Ee7jX090174@freefall.freebsd.org>

next in thread | raw e-mail | index | archive | help

>Number:         169664
>Category:       kern
>Synopsis:       Wrongful replacement of interface connected net route
>Confidential:   no
>Severity:       critical
>Priority:       medium
>Responsible:    freebsd-bugs
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Thu Jul 05 14:40:06 UTC 2012
>Closed-Date:
>Last-Modified:
>Originator:     Alexander
>Release:        9.x, 8.x
>Organization:
>Environment:
FreeBSD 9.0-PRERELEASE FreeBSD 9.0-PRERELEASE #1: Sun Dec 25 15:43:40 MSK 2011 /usr/obj/usr/src/sys/CUSTOM_AMD64  amd64

and 9.0 RELEASE

>Description:
Hi,

This problem led to serious failures in routing at several companies, using FreeBSD 9.0 RELEASE and 8.x too.

Route of connected network (specified by ip and mask on interface) changed or vanished during working of BGP process.

In different companies uses different BGP daemons (BIRD or OpenBGPD).

Now it became clear, the reason of a part of problems (I is not sure that all) was reception from BGP of the announcement of the same subnet (as on the interface).
The same occurs and when using OSPF.
After that, it is impossible to remove a route.

(also replacements of a route can achieve manually, without BGP, if interface is down state while try to add static route).

Turning off/on of the correct interface helps only. Or only destroy and recreate vlan interface.

But this action (down/up or recreate) in 9.0 RELEASE periodically leads to a kernel panic (Probably, and on 8.x, but is more rare).

It is described in the block "How to repeat the problem",
and here I give also diagnostics of some strange occurences of similar failures:


Case 2 (Jun 2012), Server 1:
Route to host from connected network (on vlan7) points to other interface.
(route received by BGP).

# route -n get 193.232.244.100
   route to: 193.232.244.100
destination: 193.232.244.0
       mask: 255.255.254.0
    gateway: 194.226.x.x
  interface: vlan5
      flags: <UP,GATEWAY,DONE,PROTO1>
 recvpipe  sendpipe  ssthresh  rtt,msec    mtu        weight    expire
       0         0         0         0      1500         1         0

(ifconfig vlan7 see on next case, it is same):



Case 1 (March 2012), Server 2 (other):

Route to host from connected network points to other interface, because
route to connected network is disappeared and default wins:

# route -n get 193.232.244.100
   route to: 193.232.244.100
destination: default
       mask: default
    gateway: 83.137.50.254
  interface: bge0
      flags: <UP,GATEWAY,DONE,STATIC>
 recvpipe  sendpipe  ssthresh  rtt,msec    mtu        weight    expire
       0         0         0         0      1500         1         0

Connected interface:

# ifconfig vlan7
vlan7: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=3<RXCSUM,TXCSUM>
        ether xx:xx:xx:xx:xx:xx
        inet6 fe80::21b:21ff:fe8b:e93d%vlan7 prefixlen 64 scopeid 0xf
        inet 193.232.245.xxx netmask 0xfffffe00 broadcast 193.232.245.255
        inet 193.232.247.xxx netmask 0xfffffe00 broadcast 193.232.247.255
        inet6 2001:7f8:20:101::245:xxx prefixlen 64
        inet6 2001:7f8:20:102::247:xxx prefixlen 64
        nd6 options=3<PERFORMNUD,ACCEPT_RTADV>
        media: Ethernet autoselect
        status: active
        vlan: 7 parent interface: lagg0

Thus ARP is ok:

# arp -an | grep 193.232.244.100
? (193.232.244.100) at 00:1a:64:99:87:9a on vlan7 expires in 299 seconds [vlan]

Fixing:

# ifconfig vlan7 down
# ifconfig vlan7 up
#
# route -n get 193.232.244.100
   route to: 193.232.244.100
destination: 193.232.244.0
       mask: 255.255.254.0
  interface: vlan7
      flags: <UP,DONE>
 recvpipe  sendpipe  ssthresh  rtt,msec    mtu        weight    expire
       0         0         0         0      1500         1         0



Case 3 (Jun 2012), server 1:

Route is known as connected, but IP not assigned (disappear) on interface (earlier was).

Route is present as connected:

# route get 193.232.244.0/23
   route to: msk-ix-network
destination: msk-ix-network
       mask: 255.255.254.0
  interface: vlan7
      flags: <UP,DONE>
 recvpipe  sendpipe  ssthresh  rtt,msec    mtu        weight    expire
       0         0         0         0      1500         1         0

but no IP from this subnet assigned on this interface:
(see up in Case1 for good vlan7 configuration).

# ifconfig vlan7
vlan7: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=3<RXCSUM,TXCSUM>
        ether xx:xx:xx:xx:xx:xx
        inet6 fe80::xxx:xxxx:xxxx:xxxx%vlan7 prefixlen 64 scopeid 0x11
        inet 193.232.247.xxx netmask 0xfffffe00 broadcast 193.232.247.255
        inet6 2001:7f8:20:101::245:xxx prefixlen 64
        inet6 2001:7f8:20:102::247:xxx prefixlen 64
        nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
        media: Ethernet autoselect
        status: active
        vlan: 7 parent interface: lagg0


>How-To-Repeat:
Use BGP or OSPF with receive same route as connected interface network, and wait (couple of days).

Or shut down the interface, add static route of same net, turn up the interface.
After that, it is impossible to remove a route.
Or, in some versions of FreeBSD, is possible, but connected route doesn't come back.

Sample:

Iface without IP:

# ifconfig vlan4
vlan4: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 00:1b:21:8b:e9:3c
        inet6 fe80::21b:21ff:fe8b:e93c%vlan4 prefixlen 64 scopeid 0x8
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        vlan: 4 parent interface: igb0

Adding a static route to anywhere:

# route add 192.168.72.0/24 x.x.x.1
add net 192.168.72.0: gateway x.x.x.1
#

Check:

# route get 192.168.72.11
   route to: 192.168.72.11
destination: 192.168.72.0
       mask: 255.255.255.0
    gateway: x.x.x.1
  interface: vlan3
      flags: <UP,GATEWAY,DONE,STATIC>
 recvpipe  sendpipe  ssthresh  rtt,msec    mtu        weight    expire
       0         0         0         0      1500         1         0


Try to add IP, with subnet from added route:

# ifconfig vlan4 192.168.72.2/24 alias
ifconfig: ioctl (SIOCAIFADDR): File exists

On 9.0 PRERELEASE IP wasn't established. On 8.x it is established that is shown:

# ifconfig vlan4
vlan4: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 00:1b:21:8b:e9:3c
        inet6 fe80::21b:21ff:fe8b:e93c%vlan4 prefixlen 64 scopeid 0x8
        inet 192.168.72.2 netmask 0xffffff00 broadcast 192.168.72.255
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        vlan: 4 parent interface: igb0

Check now:

# route get 192.168.72.11
   route to: 192.168.72.11
destination: 192.168.72.0
       mask: 255.255.255.0
    gateway: x.x.x.1
  interface: vlan3
      flags: <UP,GATEWAY,DONE,STATIC>
 recvpipe  sendpipe  ssthresh  rtt,msec    mtu        weight    expire
       0         0         0         0      1500         1         0


Static route are still active and not replace by connected route of added address!


Remove static route:

# route delete 192.168.72.0/24
delete net 192.168.72.0
#

Check:

# route get 192.168.72.11
   route to: 192.168.72.11
destination: default
       mask: default
    gateway: x.x.x.x
  interface: vlan3
      flags: <UP,GATEWAY,DONE,STATIC>
 recvpipe  sendpipe  ssthresh  rtt,msec    mtu        weight    expire
       0         0         0         0      1500         1         0

Now the route isn't present in general, the default works!


Fix by down/up


# ifconfig vlan4 down
# ifconfig vlan4 up

# route get 192.168.72.1
   route to: 192.168.72.1
destination: 192.168.72.0
       mask: 255.255.255.0
  interface: vlan4
      flags: <UP,DONE>
 recvpipe  sendpipe  ssthresh  rtt,msec    mtu        weight    expire
       0         0         0         0      1500         1         0


Now again try to add static:

route add 192.168.72.0/24 x.x.x.1
route: writing to routing socket: File exists
add net 192.168.72.0: gateway x.x.x.1: route already in table

Check:

# route get 192.168.72.1
   route to: 192.168.72.1
destination: 192.168.72.0
       mask: 255.255.255.0
  interface: vlan4
      flags: <UP,DONE>
 recvpipe  sendpipe  ssthresh  rtt,msec    mtu        weight    expire
       0         0         0         0      1500         1         0

No successfull.

Now with down interface:

# ifconfig vlan4 down
# route get 192.168.72.1
   route to: 192.168.72.1
destination: default
       mask: default
    gateway: x.x.x.x
  interface: vlan3
      flags: <UP,GATEWAY,DONE,STATIC>
 recvpipe  sendpipe  ssthresh  rtt,msec    mtu        weight    expire
       0         0         0         0      1500         1         0
# route add 192.168.72.0/24 x.x.x.1
add net 192.168.72.0: gateway x.x.x.1

Check:

# route get 192.168.72.1
   route to: 192.168.72.1
destination: 192.168.72.0
       mask: 255.255.255.0
    gateway: x.x.x.1
  interface: vlan3
      flags: <UP,GATEWAY,DONE,STATIC>
 recvpipe  sendpipe  ssthresh  rtt,msec    mtu        weight    expire
       0         0         0         0      1500         1         0

Added successfully.

Now up interface:

#ifconfig vlan4 up

Check:

# route get 192.168.72.1
   route to: 192.168.72.1
destination: 192.168.72.0
       mask: 255.255.255.0
    gateway: sputnik
  interface: vlan3
      flags: <UP,GATEWAY,DONE,STATIC>
 recvpipe  sendpipe  ssthresh  rtt,msec    mtu        weight    expire
       0         0         0         0      1500         1         0

Trouble repeated.




>Fix:
Workaround:

ifconfig X down
ifconfig X up

or (in same cases)

ifconfig X destroy
ifconfig X create

Attention: On 9.0 RELEASE this actions periodically leads to a kernel panic!

>Release-Note:
>Audit-Trail:
>Unformatted:



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?201207051433.q65EXJFK005673>