From owner-freebsd-bugs@FreeBSD.ORG Wed Oct 1 18:48:43 2014 Return-Path: Delivered-To: freebsd-bugs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 0BECD93B for ; Wed, 1 Oct 2014 18:48:43 +0000 (UTC) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id E0D643FD for ; Wed, 1 Oct 2014 18:48:42 +0000 (UTC) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.14.9/8.14.9) with ESMTP id s91ImgOS087075 for ; Wed, 1 Oct 2014 18:48:42 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-bugs@FreeBSD.org Subject: [Bug 194078] New: 10.1-BETA2 kernel memory leak in routing table upon PF reload Date: Wed, 01 Oct 2014 18:48:43 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: new X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 10.0-STABLE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Many People X-Bugzilla-Who: telbizov@gmail.com X-Bugzilla-Status: Needs Triage X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-bugs@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: bug_id short_desc product version rep_platform op_sys bug_status bug_severity priority component assigned_to reporter Message-ID: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-bugs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Bug reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 01 Oct 2014 18:48:43 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=194078 Bug ID: 194078 Summary: 10.1-BETA2 kernel memory leak in routing table upon PF reload Product: Base System Version: 10.0-STABLE Hardware: Any OS: Any Status: Needs Triage Severity: Affects Many People Priority: --- Component: kern Assignee: freebsd-bugs@FreeBSD.org Reporter: telbizov@gmail.com Corresponding discussion over email is available at: http://lists.freebsd.org/pipermail/freebsd-stable/2014-September/080296.html We discovered that our newly upgraded 10.1-BETA2 (r271983) has started leaking kernel memory and the *wired* portion has been steadily growing over the last few days. The *routetbl* pool is getting bloated with time. # while true; do date; vmstat -m | grep routetbl; sleep 60; done Type InUse MemUse HighUse Requests Size(s) Mon Sep 29 18:27:55 UTC 2014 routetbl 5988792 2888491K - 14285826 32,64,128,256,512,2048 Mon Sep 29 18:28:55 UTC 2014 routetbl 5990120 2889131K - 14288972 32,64,128,256,512,2048 Mon Sep 29 18:29:55 UTC 2014 routetbl 5991448 2889771K - 14292352 32,64,128,256,512,2048 Mon Sep 29 18:30:55 UTC 2014 routetbl 5992776 2890411K - 14295464 32,64,128,256,512,2048 Mon Sep 29 18:31:55 UTC 2014 routetbl 5994104 2891051K - 14298576 32,64,128,256,512,2048 Mon Sep 29 18:32:55 UTC 2014 routetbl 5995432 2891691K - 14301904 32,64,128,256,512,2048 Mon Sep 29 18:33:55 UTC 2014 routetbl 5996096 2892011K - 14303624 32,64,128,256,512,2048 Mon Sep 29 18:34:55 UTC 2014 routetbl 5997422 2892650K - 14306980 32,64,128,256,512,2048 Mon Sep 29 18:35:55 UTC 2014 routetbl 5998750 2893290K - 14310092 32,64,128,256,512,2048 Mon Sep 29 18:36:55 UTC 2014 routetbl 6000078 2893930K - 14313204 32,64,128,256,512,2048 Mon Sep 29 18:37:55 UTC 2014 routetbl 6001406 2894570K - 14316532 32,64,128,256,512,2048 Mon Sep 29 18:38:55 UTC 2014 routetbl 6002734 2895210K - 14319644 32,64,128,256,512,2048 Mon Sep 29 18:39:55 UTC 2014 routetbl 6004062 2895850K - 14323024 32,64,128,256,512,2048 Mon Sep 29 18:40:56 UTC 2014 routetbl 6004726 2896170K - 14324745 32,64,128,256,512,2048 Mon Sep 29 18:41:56 UTC 2014 routetbl 6006054 2896810K - 14327857 32,64,128,256,512,2048 Mon Sep 29 18:42:56 UTC 2014 routetbl 6007382 2897450K - 14331185 32,64,128,256,512,2048 Mon Sep 29 18:43:56 UTC 2014 routetbl 6008710 2898090K - 14334297 32,64,128,256,512,2048 After some investigation we discovered that the memory leak is triggered by a reload of the PF rule set. A simple test case with 1 rule and 1 table reveals the bug and makes it easy to reproduce. Every time 'pfctl -f /etc/firewall/pf.conf' (or your corresponding pf.conf file) is run the output above shows an increase in memory. In between reloads the memory usage stays stable. The following DTrace script could be used to watch the same behavior: #!/usr/sbin/dtrace -s #pragma D option quiet BEGIN { printf("%-20s %20s %20s %20s\n", "FUNCTION", "ALLOCATED", "FREE", "TOTAL"); } dtmalloc::$1:malloc { @malloc[probefunc] = sum(args[3]); @total[probefunc] = sum(args[3]); } dtmalloc::$1:free { @free[probefunc] = sum(args[3]); @total[probefunc] = sum(-args[3]); } tick-1s { printa("%-20.20s %20@d %20@d %20@d\n", @malloc, @free, @total); } Output: # ./script2.d routetbl FUNCTION ALLOCATED FREE TOTAL routetbl 512 512 0 ... routetbl 46592 46592 0 routetbl 808960 444416 364544 ... routetbl 861184 496640 364544 routetbl 1623552 894464 729088 ... routetbl 1641984 912896 729088 Further diagnostics: #!/usr/sbin/dtrace -s fbt:kernel:rt_msg2:entry { @rt_msg2[stack()] = count(); } fbt:kernel:rn_addroute:entry { @rn_addroute[stack()] = count(); } Output: kernel`sysctl_rtsock+0x274 kernel`sysctl_root+0x214 kernel`userland_sysctl+0x1d8 kernel`sys___sysctl+0x74 kernel`amd64_syscall+0x334 kernel`0xffffffff80900e0b 2340 kernel`sysctl_rtsock+0x64c kernel`sysctl_root+0x214 kernel`userland_sysctl+0x1d8 kernel`sys___sysctl+0x74 kernel`amd64_syscall+0x334 kernel`0xffffffff80900e0b 4680 This problem was reported to Gleb Smirnoff and he managed to reproduce and confirm the problem. -- You are receiving this mail because: You are the assignee for the bug.