From owner-freebsd-net@FreeBSD.ORG Tue Apr 23 12:08:09 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id EBFAB406; Tue, 23 Apr 2013 12:08:09 +0000 (UTC) (envelope-from rrs@lakerest.net) Received: from lakerest.net (lakerest.net [70.155.160.98]) by mx1.freebsd.org (Postfix) with ESMTP id F1ADB1E13; Tue, 23 Apr 2013 12:08:08 +0000 (UTC) Received: from [10.1.1.213] (bsd4.lakerest.net [70.155.160.102]) (authenticated bits=0) by lakerest.net (8.14.4/8.14.3) with ESMTP id r3NC87db094072 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NOT); Tue, 23 Apr 2013 08:08:07 -0400 (EDT) (envelope-from rrs@lakerest.net) Subject: Re: Default route changes unexpectedly Mime-Version: 1.0 (Apple Message framework v1283) Content-Type: text/plain; charset=us-ascii From: Randall Stewart In-Reply-To: Date: Tue, 23 Apr 2013 08:08:07 -0400 Content-Transfer-Encoding: quoted-printable Message-Id: <556E6D18-15FD-4D89-8064-45B139C9C6E7@lakerest.net> References: <5136FD71.6000408@freebsd.org> To: Nick Rogers X-Mailer: Apple Mail (2.1283) Cc: "freebsd-net@freebsd.org" , Andre Oppermann X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Apr 2013 12:08:10 -0000 Ok I too have been struck by this *multiple* times on my base home router. I am not sure how its happening, but I have placed in my kernel a = special catch that if the default route is set via the normal route.c path and = it is *not* the default route to my ISP, I will crash the kernel. My thoughts are this is *not* going to happen and its probably some = other memory corruption, but I will start with the most obvious first ;-) Now, I have also put in a cron that checks every 10 min the default = route, and=20 if it finds it *not* correct, it will a) log it, and b) fix it. So if I get a log showing up, I will know its *not* some errant program = in 9.x=20 causing a routing change via a routing socket, and if not I should have = a crash revealing who the guilty party is ;-) I will keep you all informed as I try to gather more information ;-D R On Mar 7, 2013, at 12:07 PM, Nick Rogers wrote: > On Wed, Mar 6, 2013 at 12:25 AM, Andre Oppermann = wrote: >> On 05.03.2013 18:39, Nick Rogers wrote: >>>=20 >>> Hello, >>>=20 >>> I am attempting to create awareness of a serious issue affecting = users >>> of FreeBSD 9.x and PF. There appears to be a bug that allows the >>> kernel's routing table to be corrupted by traffic routing through = the >>> system. Under heavy traffic load, the default route can seemingly >>> randomly change to an IP address that is not directly connected to = the >>> network (i.e., is not configured anywhere). Dhclient is not in the >>> mix, nor is routed, bgpd, etc. Running `route monitor` shows no >>> evidence of the change in the default route. The one commonality >>> between all the systems experiencing this problem seems to be the = use >>> of PF. >>>=20 >>> Obviously this is a serious problem as it causes all Internet-bound >>> traffic to stop routing until the default route is corrected. Some >>> users, including myself, are working around this problem by = installing >>> a script that runs multiple times a second to check if the default >>> route is incorrect and fixing it if necessary, which mitigates the >>> amount of downtime caused by the bug. >>=20 >>=20 >> Can you describe your traffic forwarding setup in more detail? >> Is it only pf, or do you run netgraph, or other things as well? >> Do you use flow routing? >=20 > I use PF for NAT, filtering, and rdr rules. ALTQ for bandwidth > management. I do not use netgraph. I use vlans. PF redirects to squid > as a transproxy. I'm not familiar with flow routing so unless its > enabled in 9.1 by default I do not use it. >=20 >>=20 >> How frequent does this happen? > Every other day during periods of heavier Internet-bound traffic. >=20 >>=20 >> I'm trying to create a stack graph to see which parts of the network >> stack are involved in handling your packet. >>=20 >> -- >> Andre >>=20 >>> Please refer to these past posts for more examples and evidence of >>> other users experiencing this problem: >>>=20 >>> http://forums.freebsd.org/showthread.php?p=3D211610#post211610 >>>=20 >>>=20 >>> = http://freebsd.1045724.n5.nabble.com/Default-route-quot-random-quot-gatewa= y-modification-bug-td5750820.html >>>=20 >>> = http://lists.freebsd.org/pipermail/freebsd-net/2012-March/031879.html >>>=20 >>> = http://lists.freebsd.org/pipermail/freebsd-ipfw/2010-September/004361.html= >>>=20 >>> There is also a PR that was incorrectly labeled as an IPFW issue. >>> Myself and others believe this issue is not restricted to the use of >>> IPFW and that the PR should be relabeled. I am inclined to think it = is >>> strictly a PF issue since I am not using IPFW, however there is >>> evidence of the default route changing on people using IPFW for past >>> versions of FreeBSD (7.x/8.x), so perhaps this is related. >>>=20 >>> http://www.freebsd.org/cgi/query-pr.cgi?pr=3Dkern/174749 >>>=20 >>> Another PR for the same problem but specific to IPFW and 8.2-RELEASE >>>=20 >>> http://www.freebsd.org/cgi/query-pr.cgi?pr=3D157796 >>>=20 >>> I am hoping someone reading this can give the problem the attention = it >>> deserves. Thank you. >>>=20 >>> -Nick >>> _______________________________________________ >>> freebsd-net@freebsd.org mailing list >>> http://lists.freebsd.org/mailman/listinfo/freebsd-net >>> To unsubscribe, send any mail to = "freebsd-net-unsubscribe@freebsd.org" >>>=20 >>>=20 >>=20 > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" >=20 ------------------------------ Randall Stewart 803-317-4952 (cell)