From owner-freebsd-net@freebsd.org Mon May 30 07:51:19 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 37B5CB4E044 for ; Mon, 30 May 2016 07:51:19 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from mailman.ysv.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 224871DA2 for ; Mon, 30 May 2016 07:51:19 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: by mailman.ysv.freebsd.org (Postfix) id 1DF4EB4E042; Mon, 30 May 2016 07:51:19 +0000 (UTC) Delivered-To: net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1D99BB4E041 for ; Mon, 30 May 2016 07:51:19 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from mail108.syd.optusnet.com.au (mail108.syd.optusnet.com.au [211.29.132.59]) by mx1.freebsd.org (Postfix) with ESMTP id C32FA1DA0 for ; Mon, 30 May 2016 07:51:18 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from besplex.bde.org (c110-21-42-169.carlnfd1.nsw.optusnet.com.au [110.21.42.169]) by mail108.syd.optusnet.com.au (Postfix) with ESMTPS id C9DB21A4FE7; Mon, 30 May 2016 17:51:09 +1000 (AEST) Date: Mon, 30 May 2016 17:51:09 +1000 (EST) From: Bruce Evans X-X-Sender: bde@besplex.bde.org To: Kevin Oberman cc: Bruce Evans , David DeSimone , "net@freebsd.org" Subject: Re: ifconfig creates a bogus(?) route In-Reply-To: Message-ID: <20160530144845.V1573@besplex.bde.org> References: <20160528154122.C1843@besplex.bde.org> <20160529111806.I1012@besplex.bde.org> <20160529163228.I1958@besplex.bde.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Optus-CM-Score: 0 X-Optus-CM-Analysis: v=2.1 cv=M8SwUHEs c=1 sm=1 tr=0 a=kDyANCGC9fy361NNEb9EQQ==:117 a=kDyANCGC9fy361NNEb9EQQ==:17 a=L9H7d07YOLsA:10 a=9cW_t1CCXrUA:10 a=s5jvgZ67dGcA:10 a=kj9zAlcOel0A:10 a=PO7r1zJSAAAA:8 a=kn6OvvWOlDjpCIAclQUA:9 a=jMuuPBF1GrfXECIa:21 a=ppoOvatWiB26iUAO:21 a=CjuIK1q_8ugA:10 a=Oa0T6EYmKFNB-xRHvYM1:22 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 30 May 2016 07:51:19 -0000 On Sun, 29 May 2016, Kevin Oberman wrote: > On Sun, May 29, 2016 at 12:12 AM, Bruce Evans wrote: > >> On Sun, 29 May 2016, Bruce Evans wrote: >> >> ... >>> I still haven't figured out how to set up the bogus(?) route using route >>> add/change, but can now complete the initialization using a "route change" >>> that appears to be null: >>> >>> Test script: >>> ... >>> X # Fix up the initialization with a null change: >>> X route change -iface 192.168.2.8 192.168.2.8 >>> X netstat -rn >>> >> >> Before: >> >>> ... >>> Y Destination Gateway Flags Refs Use Netif >>> Expire >>> Y 127.0.0.1 link#2 UH 0 0 lo0 >>> Y 192.168.2.0/24 link#1 U 0 0 em0 >>> Y 192.168.2.8 link#1 UHS 0 0 lo0 >>> Y route to: 192.168.2.8 >>> Y destination: 192.168.2.8 >>> Y interface: lo0 >>> Y flags: >>> ... >>> >> >> After: >> >>> ... >>> Y Destination Gateway Flags Refs Use Netif >>> Expire >>> Y 127.0.0.1 link#2 UH 0 0 lo0 >>> Y 192.168.2.0/24 link#1 U 0 0 em0 >>> Y 192.168.2.8 192.168.2.8 UHS 0 0 lo0 >>> Y route to: 192.168.2.8 >>> Y destination: 192.168.2.8 >>> Y interface: lo0 >>> Y flags: >>> ... >>> Note that the "null" route change just echos the current setting and it >>> doesn't change anything according to netstat -rn and route -n, but it >>> fixes up some internal state so that the ping works. When ping works, >>> tcpdump shows it going from 127.0.0.1 to 192.168.2.8 and coming back as >>> expected. When ping hangs, tcpdump shows it going from 0.0.0.0 to >>> 192.168.2.8 and never coming back. The default route doesn't exist in >>> either case. >>> >> >> Bah, it did change (from link#1 to 192.168.2.8 for Gateway in netstat -rn >> output only). -iface makes no difference (I thought it prevented this >> change). >> >> The initialization bug has something to do with the magic default route >> of 0.0.0.0). When lo0 is used uninitialized, it is still initialized >> to 0 and the 0 gets copied to somewhere not shown properly in the >> initial route from (or is it to?) 192.168.2.8. netstat -rn translates >> this 0 together with another address to #link1, but route -n get >> doesn't show it. I think this configuration is supposed to not occur. >> I couldn't find a way to get back it using route(8) after fixing it. >> >> When lo0 is brought up first, different magic obscures what is >> happening. E.g., when lo0 is misconfigured with "inet 127.0.0.127", >> the em0 route is auto-configured with this strange lo0 address instead >> of completely misconfigured to 0. Neither netstat -rn nor route -n >> get displays this clearly (they give the same display as in the >> uninitialized case except for the .127 route). Then if lo0 is >> ifconfig'ed again to fix this misconfiguration, the all routes related >> to the old lo0 (bogus or working) are deleted and not restored. But >> if lo0 is brought up for the first time after em0, the the 0.0.0.0 >> route related to the old lo0 is not deleted; it keeps breaking things. > > When it is in the "broken" condition, what is the output of "route -n get > 0"? It is an error: "route: writing to routing socket: No such process". This is as i should be since I try to delete all active routes in the test so as to get close enough to the boot-time condition. > So you have a default route? None appears in any of your messages. Normally > 0 should route to default. In fact 0/0 is a definition of default. (Well, I > suppose anything/0 is default, but 0/0 is the normal representation.) Since > most systems have a default gateway defined, this may relate to the issue > you are seeing. I'm very unclear what happens with 0/0 if default is not > defined. Normally I don't have a default route, except on 1 gateway system where it goes to the cable modem. This is normally configured by dhcp. On other systems on the LAN, I use only static routing and rarely have a default route. Sometimes I add and delete the default route on the other systems to the gateway system. It is certainly not there after deleting it. > FWIW, here is what I see on this system (10.3-STABLE r299096): > Routing tables > > Internet: > Destination Gateway Flags Netif Expire > default 192.168.1.1 UGS re0 > 127.0.0.1 link#2 UH lo0 > 192.168.1.0/24 link#1 U re0 > 192.168.1.16 link#1 UHS lo0 My userland is mostly 5.2-CURRENT, and its rc never generates a default route like that. The cable modem gives a non-local default. My gateway default is 2.2 instead of 1.1 since when I invented static routes for my local machines I knew even less about routing than I do now and chose numbers starting at 002.002 instead of 1.1. (BTW, no one replied to my bug report on this list about breakage of parsing of numbers like 002 somewhere in libc/net*. I used the leading zeros to line up dotted quad numbers in tables.) I now understand the link#1 display a bit better: - in old versions of netstat -rn, it is what is displayed when the only available info in a sockaddr_dl, is the link number. This happens for the /24 link. route get shows nothing in this case (I hacked it to print the gateway when the gateway flag is not set, but it prints nothing starting with the same info). Otherwise, the ethernet address is printed for type IFT_ETHER. This happens for the local machine's address and for external machines on the LAN. Otherwise, a network address is printed. This happens for the localhost and default routes. - this is dumbed down in current versions of netstat -rn :-(. The strings are now generated mostly in the kernel, and this seems to require more complicated code in userland even without fixups to recover lost details or print more details for debugging. The kernel also generates link#N when there is little info, and seems to only generate the network address instead of the ethernet address for the local machine. Data for external machines on the LAN is no longer printed in any form. I finally got some useful info out of debugging printfs in route: - in the non-working version, the gateway address has sa_family = 18 (IF_LINK), sdl_family = 18 (IF_LINK), sdl_index = 1 (link#1), sdl_type = 6 (IFT_ETHER), and otherwise 0's. Non-old netstat -rn prints this as "link#1" and my debugging code prints this and some raw fields. - in the working version produced by bringing up lo0 before em0, the gateway address is unchanged in both non-old netstat -rn and debugging route -n get. - in the working version produced by the "null" route change, the gateway does change -- to sa_family = 2 (AF_INET) and s_addr = 0x0802a8c0 (otherwise 0's). Non-old netstat -rn and debugging route -n get display this consistently as 192.168.2.8. - the working version produced by old kernels is different again. It gives IF_LINK as in the non-working version, but now the link is for the em0 and the address is for em0 . Netif is still lo0. Old netstat -rn and debugging route -n get agree on this since the non-raw part of the debugging code was copied from old netstat. (The address is decided by the kernel and decoded by netstat depending on sdl_length -- the length is 12 for ethernet addresses and 4 for ip4 addresses.) I got no further trying to set the gateway address precisely so as to give an actual null change or to change to and from all of the working and non-working versions. While here, I will asks about firewalling of routing. I use a clone of rc.firewall created more than 10 years ago. rc.firewall hasn't changed much, and is still too simple to do what I want, but also too complicated to understand or change easily. It has fairly hard-coded support for not much more than 2 nontrivial configurations: workstation, for which the external link may be shared but every workstation must run a firewall; and simple (actually complicated), for which one machine is a gateway to the external link. I need the latter, , but it needs different NICs which I don't always have available. Plugging the external link into the shared switch fails first here: > # Stop spoofing > ${fwcmd} add deny all from ${inet}:${imask} to any in via ${oif} > ${fwcmd} add deny all from ${onet}:${omask} to any in via ${iif} since with only 1 NIC, the internal NIC iif cannot be different from the external NIC oif. inet and onet are of course different, but using inet to reach onet is exactly the spoofing that these rules disallow as far as I understand. So these rules apparently need to be relaxed a lot. A network expert should be able to do this without reducing security much, but no one did it for /etc/rc.firewall. This particular spoofing problem doesn't seem to be mentioned in the man page or old versions of the handbook. I had reduced security here for almost 10 years by forgetting to update across several changes of NICs and machines. It is not used for anything except this rule. Bruce