From owner-freebsd-stable@FreeBSD.ORG Mon Sep 29 18:58:24 2014 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id D1FB44E7 for ; Mon, 29 Sep 2014 18:58:24 +0000 (UTC) Received: from mail-lb0-x22b.google.com (mail-lb0-x22b.google.com [IPv6:2a00:1450:4010:c04::22b]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 188F86B6 for ; Mon, 29 Sep 2014 18:58:23 +0000 (UTC) Received: by mail-lb0-f171.google.com with SMTP id z12so42386lbi.16 for ; Mon, 29 Sep 2014 11:58:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type; bh=T8/BsSYd0Vwnr8SuGEj5VVm/00idpu/xOiK7tkoE2As=; b=Ct+6LGzzjc+QFg/a7ltunwpM8piuCjkuDeTEORGQ2Qf5u14jC9pIwIX7qXEeBN2ZbD /6uNz2DLpWl+COkRahpqdDlP8yWPUcUnxqvt3IMAGB6UCpsVJ98YLXThjuASgJWuoM4h wvnCqxq1pgpXs5hm7p2YgDDxy5+np9zBFBa6NM+H03I8MtRx6PrbZTkydrB29Jd0R4D6 G0bZsxLZr3upFjGdWVFCh8q6echp47/C3aod3Y5o3Y1qU91gO9B0Hc3EaIPCNi43BZmv r+Mdc7v1zNUThlO3pW0UwftFP4UIv6d0hQXbPGwHmfa7OLBZDb1A+YXwb0kvi/8A2DNe hwFA== MIME-Version: 1.0 X-Received: by 10.152.5.40 with SMTP id p8mr42179973lap.32.1412017101414; Mon, 29 Sep 2014 11:58:21 -0700 (PDT) Received: by 10.114.216.37 with HTTP; Mon, 29 Sep 2014 11:58:20 -0700 (PDT) Date: Mon, 29 Sep 2014 11:58:20 -0700 Message-ID: Subject: 10.1-BETA2 possible kernel memory leak in routing table From: Rumen Telbizov To: "freebsd-stable@freebsd.org" Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 29 Sep 2014 18:58:24 -0000 Hello everyone, I have recently upgraded one of our main firewalls from FreeBSD 9.2 to10.1-BETA2 (r271983). It has been running in production since last Thursday and overall I am pretty happy with it and seeing significant improvements in performance. However I do notice this morning that the *wired* portion of the memory has been steadily growing over the weekend and is now over a few gigabytes. Here's a graph: [image: Inline image 2] The switchover was done on late 25th when production traffic started flowing through it. I checked vmstat -m and noticed that the *routetbl* is pretty large - 2.8GB and keeps on growing. Here's a few minutes worth of monitoring it: # while true; do date; vmstat -m | grep routetbl; sleep 60; done Type InUse MemUse HighUse Requests Size(s) Mon Sep 29 18:27:55 UTC 2014 routetbl 5988792 2888491K - 14285826 32,64,128,256,512,2048 Mon Sep 29 18:28:55 UTC 2014 routetbl 5990120 2889131K - 14288972 32,64,128,256,512,2048 Mon Sep 29 18:29:55 UTC 2014 routetbl 5991448 2889771K - 14292352 32,64,128,256,512,2048 Mon Sep 29 18:30:55 UTC 2014 routetbl 5992776 2890411K - 14295464 32,64,128,256,512,2048 Mon Sep 29 18:31:55 UTC 2014 routetbl 5994104 2891051K - 14298576 32,64,128,256,512,2048 Mon Sep 29 18:32:55 UTC 2014 routetbl 5995432 2891691K - 14301904 32,64,128,256,512,2048 Mon Sep 29 18:33:55 UTC 2014 routetbl 5996096 2892011K - 14303624 32,64,128,256,512,2048 Mon Sep 29 18:34:55 UTC 2014 routetbl 5997422 2892650K - 14306980 32,64,128,256,512,2048 Mon Sep 29 18:35:55 UTC 2014 routetbl 5998750 2893290K - 14310092 32,64,128,256,512,2048 Mon Sep 29 18:36:55 UTC 2014 routetbl 6000078 2893930K - 14313204 32,64,128,256,512,2048 Mon Sep 29 18:37:55 UTC 2014 routetbl 6001406 2894570K - 14316532 32,64,128,256,512,2048 Mon Sep 29 18:38:55 UTC 2014 routetbl 6002734 2895210K - 14319644 32,64,128,256,512,2048 Mon Sep 29 18:39:55 UTC 2014 routetbl 6004062 2895850K - 14323024 32,64,128,256,512,2048 Mon Sep 29 18:40:56 UTC 2014 routetbl 6004726 2896170K - 14324745 32,64,128,256,512,2048 Mon Sep 29 18:41:56 UTC 2014 routetbl 6006054 2896810K - 14327857 32,64,128,256,512,2048 Mon Sep 29 18:42:56 UTC 2014 routetbl 6007382 2897450K - 14331185 32,64,128,256,512,2048 Mon Sep 29 18:43:56 UTC 2014 routetbl 6008710 2898090K - 14334297 32,64,128,256,512,2048 As you can see for this interval it grew 9599K (2898090-2888491). The routing table does not change at all on this machine. netstat -rn | wc -l reports 145 entries all the time. As a comparison on the old, inactive FreeBSD 9.2 firewall I have: # vmstat -m | grep routetbl routetbl 451 98K - 7336488 32,64,128,256,512,1024 Here are some configuration details of the system. loader.conf: autoboot_delay="3" zfs_load="YES" vfs.root.mountfrom="zfs:zroot/freebsd10" vfs.zfs.arc_max="500M" kern.geom.label.gptid.enable="0" hw.ix.num_queues="12" hw.ix.txd="4096" hw.ix.rxd="4096" sysctl.conf: # CARP related net.inet.carp.allow=1 net.inet.carp.preempt=1 net.inet.carp.log=2 # Stop sending ICMP redirects net.inet.ip.redirect=0 # Fast Forwarding net.inet.ip.fastforwarding=1 # Enable better lacp/lagg transmit distribution net.link.lagg.default_use_flowid=0 I could provide additional debug information upon request. Your help is appreciated. Regards, -- Rumen Telbizov Unix Systems Administrator