From owner-freebsd-net@FreeBSD.ORG Fri Apr 1 16:00:37 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 458D0106566B for ; Fri, 1 Apr 2011 16:00:37 +0000 (UTC) (envelope-from jamesbrandongooch@gmail.com) Received: from mail-wy0-f182.google.com (mail-wy0-f182.google.com [74.125.82.182]) by mx1.freebsd.org (Postfix) with ESMTP id CCF228FC08 for ; Fri, 1 Apr 2011 16:00:36 +0000 (UTC) Received: by mail-wy0-f182.google.com with SMTP id 23so3656151wyf.13 for ; Fri, 01 Apr 2011 09:00:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=knllz0id8Rtd2dedthDUgIWpy9OkL3PYNA0aZLYhffs=; b=UdE2WTlRIfhr5u/DxjeXKzmjCKO+j0e//sdBFSJ6X+qd9nKSX00GOhyeSDV8NwQiRx TyoP79mCoKGGZKJUTWtbChtdDT0i4knoy+2hJ7zB3Vz2gfmvTCUyxZpKKLw6drzjYE8s GGiYoGmcn5kP3Ly+oauB7qIQShM0cFEBiE8zE= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; b=eTcrMoEwjWNp4Od/2gpsmqCXYiYSiLXCJBleNUsRJXYiHK/BZmBAPJz0nAOlJqRKB+ KDxkKRQNSXMAyGfdY18FwV4VbNVnKaaHDPtSolobDs+ZhjmSaAydooBnVPzCafHOnqlQ 256JVm9ybn3gJIKcgBZlMSxpqk56DKGNAkewQ= MIME-Version: 1.0 Received: by 10.216.144.223 with SMTP id n73mr4125630wej.37.1301673636421; Fri, 01 Apr 2011 09:00:36 -0700 (PDT) Received: by 10.216.0.205 with HTTP; Fri, 1 Apr 2011 09:00:36 -0700 (PDT) In-Reply-To: <4D95E62A.5000109@comcast.net> References: <20110401141655.GA5350@deta.isafeelin.org> <4D95E62A.5000109@comcast.net> Date: Fri, 1 Apr 2011 11:00:36 -0500 Message-ID: From: Brandon Gooch To: Steve Polyack Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: freebsd-net@freebsd.org, Frederique Rijsdijk Subject: Re: Network stack unstable after arp flapping X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 01 Apr 2011 16:00:37 -0000 On Fri, Apr 1, 2011 at 9:50 AM, Steve Polyack wrote: > On 04/01/11 10:16, Frederique Rijsdijk wrote: >> >> Hi, >> >> We (hosting provider) are in the process of implementing ipv6 in our >> network (yay). Yesterday one of the final steps in configuring and updat= ing >> our core routers were taken, which did not go entirely as planned. As a >> result, the default gateway mac addresses for all our machines changed a= bout >> 800 times in a time span of about 4 minutes. >> >> Here's a small piece of the logging: >> >> Mar 31 18:36:12 srv01 kernel: arp: x.x.x.1 moved from 00:00:0c:9f:f0:3d = to >> 00:00:0c:07:ac:3d on bge0 >> Mar 31 18:36:12 srv01 kernel: arp: x.x.x.1 moved from 00:00:0c:07:ac:3d = to >> 00:00:0c:9f:f0:3d on bge0 >> Mar 31 18:36:13 srv01 kernel: arp: x.x.x.1 moved from 00:00:0c:9f:f0:3d = to >> 00:00:0c:07:ac:3d on bge0 >> Mar 31 18:36:14 srv01 kernel: arp: x.x.x.1 moved from 00:00:0c:07:ac:3d = to >> 00:00:0c:9f:f0:3d on bge0 >> Mar 31 18:36:14 srv01 kernel: arp: x.x.x.1 moved from 00:00:0c:9f:f0:3d = to >> 00:00:0c:07:ac:3d on bge0 >> Mar 31 18:36:14 srv01 kernel: arp: x.x.x.1 moved from 00:00:0c:07:ac:3d = to >> 00:00:0c:9f:f0:3d on bge0 >> Mar 31 18:36:15 srv01 kernel: arp: x.x.x.1 moved from 00:00:0c:9f:f0:3d = to >> 00:00:0c:07:ac:3d on bge0 >> >> The x.x.x.1 is always the same IP, the gateway of the machine. >> >> The result of that, is that loads of FreeBSD machines (6.x, 7.x and 8.x) >> developed serious network issues, mainly being no or slow traffic betwee= n >> other (FreeBSD) machine accross different VLAN's in our own network. >> >> First thing that comes to mind is the network itself, but all Linux >> machines (Ubuntu, Red Hat and CentOS) had no issues at all. Only BSD. >> >> An arp -ad on both machines where problems occured, didn't solve anythin= g. >> What worked better was /etc/rc.d/netif restart and a /etc/rc.d/routing >> restart. Some machines even had to be rebooted in order to get networkin= g >> back to normal. >> >> This almost sounds like a bug in the network stack in BSD, but I can not >> imagine that I'm right. The BSD networking stack is considered to be one= of >> the best.. >> >> Any ideas anyone? > > We experienced a similar issue here, but IIRC only on our 8.x systems (we > don't have any 7.x). =A0Disabling flowtable cleared everything up immedia= tely. > =A0You can try that and see if it helps. =A0It seems like the flowtable = =A0caches > and associates the next-hop router MAC address with each flow, and > unfortunately this doesn't get purged when the kernel senses and logs an = ARP > change. =A0The only other solution I've seen was to stop all network traf= fic > on the machine until the flows/cache entries expired. > > http://www.freebsd.org/cgi/query-pr.cgi?pr=3D155604 has more details of m= y > run-in with this. =A0The title should be corrected though, as I found sho= rtly > after that all traffic is affected. > > - Steve FYI, the FLOWTABLE option has been removed from the DEFAULT kernel config on HEAD, a change which will be MFC'd in a couple of days to 8-STABLE... -Brandon