From owner-freebsd-stable@FreeBSD.ORG Thu Oct 2 22:25:04 2014 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 9D5F1246; Thu, 2 Oct 2014 22:25:04 +0000 (UTC) Received: from mail-ie0-x22f.google.com (mail-ie0-x22f.google.com [IPv6:2607:f8b0:4001:c03::22f]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4796CB9B; Thu, 2 Oct 2014 22:25:04 +0000 (UTC) Received: by mail-ie0-f175.google.com with SMTP id x19so75587ier.6 for ; Thu, 02 Oct 2014 15:25:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=orDgrK4Z+lH9tvwZL8mRWY+L/NVY/3CqcELZe0PKwF0=; b=B/MtquzVl+IJUhQ0eFp17fuuy0PG2atBo1hnWkDRAfxvVeIIMtXqXnk2eD9lWPxhCH y3jxz9WanAiv3qpzaIaZOqpQqm+SpJvF8K/8bJ5oaEsOYyiVPITQeLQi9Hu7GquF5++t XpZ48ceAz2dv4mMiNUGtmT+/uBl/dOZxAM640sIAt5TpOK8wwtWUmc7UNshmuUxJ7VVE SEQ6vMbEPEQkYtT5ji0PgnkLI03BOdkqvgNObnF9cqW+7CPXURXymOZsfwFfgHMuk7qF nrFRmi4hjOefpwI6rpIejAClzAuLlALEVPUns7eCM6Y3JadXDHqPJP7Uzi2FNAjc4vGa 74UA== MIME-Version: 1.0 X-Received: by 10.42.10.209 with SMTP id r17mr8713758icr.65.1412288702470; Thu, 02 Oct 2014 15:25:02 -0700 (PDT) Received: by 10.50.87.130 with HTTP; Thu, 2 Oct 2014 15:25:02 -0700 (PDT) In-Reply-To: References: <542AAA3C.1080803@ipfw.ru> <542AE376.6000003@FreeBSD.org> <542AFAE3.9030705@FreeBSD.org> <20141001135124.GM73266@glebius.int.ru> <542C20D7.3070606@sentex.net> <20141001171646.GQ73266@glebius.int.ru> <542C5529.9030800@FreeBSD.org> <7AB35136-89AA-48F2-8B0E-1BA3DCD4A6BA@ipfw.ru> Date: Thu, 2 Oct 2014 15:25:02 -0700 Message-ID: Subject: Re: 10.1-BETA2 possible kernel memory leak in routing table From: Rumen Telbizov To: "Alexander V. Chernikov" Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 Cc: brian@freebsd.org, Gleb Smirnoff , "Alexander V. Chernikov" , "freebsd-stable@freebsd.org" X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 02 Oct 2014 22:25:04 -0000 Hello everyone, Alexander, Gleb: good news guys. The problem seems to be resolved! Good job.Here's what I have. 1. I upgraded the backup firewall to 10-STABLE (r272435), applied Gleb's patch of pf_table.c and rebuilt. Just as reported by Gleb, the leak seemed to be reduced but it was still present and leaked a little bit of memory upon every PF reload. 2. In addition to 1) I applied the second patch from Alexander on radix.c and rebuilt the kernel. After the reboot it seems like the amount of memory consumed by the routetbl grows a little bit temporarily after reload and then drops back to the previous, stable amount shortly after. 3. This is how I tested it with: Just after boot I checked vmstat -m | grep routetbl routetbl 606 170K - 4334 32,64,128,256,512,2048 Then I ran a tight loop of reloading the ruleset (and all tables) for a while: while true; do pfctl -f /etc/firewall/pf.conf; done The memory output while the loop was still running looked something like this: ... routetbl 5166 2387K - 33177467 32,64,128,256,512,2048 routetbl 5358 2483K - 33190813 32,64,128,256,512,2048 routetbl 5598 2603K - 33202561 32,64,128,256,512,2048 routetbl 5870 2739K - 33214365 32,64,128,256,512,2048 routetbl 5634 2638K - 33226099 32,64,128,256,512,2048 routetbl 6286 2947K - 33239443 32,64,128,256,512,2048 routetbl 6510 3059K - 33251175 32,64,128,256,512,2048 routetbl 6750 3179K - 33262925 32,64,128,256,512,2048 routetbl 4622 2115K - 33274645 32,64,128,256,512,2048 routetbl 4910 2259K - 33286441 32,64,128,256,512,2048 routetbl 3582 1658K - 33298175 32,64,128,256,512,2048 routetbl 5326 2467K - 33311519 32,64,128,256,512,2048 ... after I stopped the reload loop and waited a minute the routetbl memory pool went back to previous state routetbl 606 170K - 35615574 32,64,128,256,512,2048 I assume this delay is normal and is simply the kernel freeing memory after it gets "cold". I will leave this machine running and reloading the ruleset every 30 seconds as intended in carp backup mode and will report any potential problems back here. Aside from that I think we're all good. The problem seems solved. Gleb, Alexander, thank you very much for your help. I appreciate your quick response in providing patches for this problem. I hope those 2 patches get tested by more people and get MFC'ed into 10-STABLE soon. Regards, -- Rumen Telbizov Unix Systems Administrator