From owner-freebsd-stable@FreeBSD.ORG Tue Sep 30 13:04:58 2014 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id EAA21239 for ; Tue, 30 Sep 2014 13:04:57 +0000 (UTC) Received: from mail.ipfw.ru (mail.ipfw.ru [IPv6:2a01:4f8:120:6141::2]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 7EF8488C for ; Tue, 30 Sep 2014 13:04:57 +0000 (UTC) Received: from [2a02:6b8:0:401:222:4dff:fe50:cd2f] (helo=ptichko.yndx.net) by mail.ipfw.ru with esmtpsa (TLSv1:DHE-RSA-AES128-SHA:128) (Exim 4.82 (FreeBSD)) (envelope-from ) id 1XYt7g-0002gd-HR; Tue, 30 Sep 2014 12:49:28 +0400 Message-ID: <542AAA3C.1080803@ipfw.ru> Date: Tue, 30 Sep 2014 17:03:56 +0400 From: "Alexander V. Chernikov" User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:31.0) Gecko/20100101 Thunderbird/31.1.2 MIME-Version: 1.0 To: Rumen Telbizov , "freebsd-stable@freebsd.org" Subject: Re: 10.1-BETA2 possible kernel memory leak in routing table References: In-Reply-To: Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 30 Sep 2014 13:04:58 -0000 On 29.09.2014 22:58, Rumen Telbizov wrote: > Hello everyone, > > I have recently upgraded one of our main firewalls from FreeBSD 9.2 > to10.1-BETA2 (r271983). It has been running in production since last > Thursday and overall I am pretty happy with it and seeing significant > improvements in performance. > > However I do notice this morning that the *wired* portion of the memory has > been steadily growing over the weekend and is now over a few gigabytes. > Here's a graph: > > [image: Inline image 2] > > The switchover was done on late 25th when production traffic started > flowing through it. I checked vmstat -m and noticed that the *routetbl* is > pretty large - 2.8GB and keeps on growing. Here's a few minutes worth of > monitoring it: > > > # while true; do date; vmstat -m | grep routetbl; sleep 60; done > > Type InUse MemUse HighUse Requests Size(s) > Mon Sep 29 18:27:55 UTC 2014 > routetbl 5988792 2888491K - 14285826 32,64,128,256,512,2048 > Mon Sep 29 18:28:55 UTC 2014 > routetbl 5990120 2889131K - 14288972 32,64,128,256,512,2048 > Mon Sep 29 18:29:55 UTC 2014 > routetbl 5991448 2889771K - 14292352 32,64,128,256,512,2048 > Mon Sep 29 18:30:55 UTC 2014 > routetbl 5992776 2890411K - 14295464 32,64,128,256,512,2048 > Mon Sep 29 18:31:55 UTC 2014 > routetbl 5994104 2891051K - 14298576 32,64,128,256,512,2048 > Mon Sep 29 18:32:55 UTC 2014 > routetbl 5995432 2891691K - 14301904 32,64,128,256,512,2048 > Mon Sep 29 18:33:55 UTC 2014 > routetbl 5996096 2892011K - 14303624 32,64,128,256,512,2048 > Mon Sep 29 18:34:55 UTC 2014 > routetbl 5997422 2892650K - 14306980 32,64,128,256,512,2048 > Mon Sep 29 18:35:55 UTC 2014 > routetbl 5998750 2893290K - 14310092 32,64,128,256,512,2048 > Mon Sep 29 18:36:55 UTC 2014 > routetbl 6000078 2893930K - 14313204 32,64,128,256,512,2048 > Mon Sep 29 18:37:55 UTC 2014 > routetbl 6001406 2894570K - 14316532 32,64,128,256,512,2048 > Mon Sep 29 18:38:55 UTC 2014 > routetbl 6002734 2895210K - 14319644 32,64,128,256,512,2048 > Mon Sep 29 18:39:55 UTC 2014 > routetbl 6004062 2895850K - 14323024 32,64,128,256,512,2048 > Mon Sep 29 18:40:56 UTC 2014 > routetbl 6004726 2896170K - 14324745 32,64,128,256,512,2048 > Mon Sep 29 18:41:56 UTC 2014 > routetbl 6006054 2896810K - 14327857 32,64,128,256,512,2048 > Mon Sep 29 18:42:56 UTC 2014 > routetbl 6007382 2897450K - 14331185 32,64,128,256,512,2048 > Mon Sep 29 18:43:56 UTC 2014 > routetbl 6008710 2898090K - 14334297 32,64,128,256,512,2048 > > As you can see for this interval it grew 9599K (2898090-2888491). The > routing table does not change at all on this machine. netstat -rn | wc -l > reports 145 entries all the time. As a comparison on the old, inactive > FreeBSD 9.2 firewall I have: > > # vmstat -m | grep routetbl > routetbl 451 98K - 7336488 32,64,128,256,512,1024 Can you show route -Arn output ? Can you disable carp and see if this changes anything? > > Here are some configuration details of the system. > > loader.conf: > autoboot_delay="3" > zfs_load="YES" > vfs.root.mountfrom="zfs:zroot/freebsd10" > vfs.zfs.arc_max="500M" > kern.geom.label.gptid.enable="0" > hw.ix.num_queues="12" > hw.ix.txd="4096" > hw.ix.rxd="4096" > > sysctl.conf: > # CARP related > net.inet.carp.allow=1 > net.inet.carp.preempt=1 > net.inet.carp.log=2 > > # Stop sending ICMP redirects > net.inet.ip.redirect=0 > > # Fast Forwarding > net.inet.ip.fastforwarding=1 > > # Enable better lacp/lagg transmit distribution > net.link.lagg.default_use_flowid=0 Are you sure this is really "better" ? > > > I could provide additional debug information upon request. Your help is > appreciated. > > Regards,