From owner-freebsd-net@freebsd.org Tue Jul 11 18:18:13 2017 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 61DDADA94F0 for ; Tue, 11 Jul 2017 18:18:13 +0000 (UTC) (envelope-from list1@gjunka.com) Received: from msa1.earth.yoonka.com (yoonka.com [88.98.225.149]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "msa1.earth.yoonka.com", Issuer "msa1.earth.yoonka.com" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 030A080F2B for ; Tue, 11 Jul 2017 18:18:11 +0000 (UTC) (envelope-from list1@gjunka.com) Received: from ultrabook.yoonka.com (p5DC0EB77.dip0.t-ipconnect.de [93.192.235.119]) (authenticated bits=0) by msa1.earth.yoonka.com (8.15.2/8.15.2) with ESMTPSA id v6BIDbc2011571 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Tue, 11 Jul 2017 18:13:38 GMT (envelope-from list1@gjunka.com) X-Authentication-Warning: msa1.earth.yoonka.com: Host p5DC0EB77.dip0.t-ipconnect.de [93.192.235.119] claimed to be ultrabook.yoonka.com From: Grzegorz Junka Subject: Re: memory leaks in 11.0? To: freebsd-net@freebsd.org References: <1771938.JUakJLQFm8@energia> Message-ID: <94cb502b-7c55-8feb-9ee8-e7f6b7f1f9e8@gjunka.com> Date: Tue, 11 Jul 2017 18:13:32 +0000 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Thunderbird/52.0.1 MIME-Version: 1.0 In-Reply-To: <1771938.JUakJLQFm8@energia> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit Content-Language: en-GB-large X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 11 Jul 2017 18:18:13 -0000 On 11/07/2017 13:56, Kajetan Staszkiewicz wrote: > Hello, > > I finally upgraded one of many of my routers to 11.0. > > Unfortunately after running fine for a month it ran out of memory. "wired" > memory slowly grows up to allocating all memory in system when no more memory > is left for other programs. Things first get swapped and eventually die. > > The router runs BIRD which has not much to do, it is for internal networks > only, pf, pfsync (currently disabled via `ifconfig pfsync0 down`), filebeat, > smokeping, ntp, nrpe and custom python cron job for sending data to Graphite. > > `vmstat -z` shows constantly increasing allocation of "512" and "UMA Slabs". > Memory allocated for all pf-related things seems fine. I have graphite graps > for every `vmstat -z` and the icrease on "512" grows in similar way as "wired" > memory. "512" has 2 917 392 used objects allocated at this moment, "UMA Slabs" > is 379 006, there is 2636MiB "wired" memory. > > How can I debug which part of kernel is responsible for this? I run GENERIC > kernel with ixl driver 1.7.11 from Intel, as the one in GENERIC had issues > detecting links on my x710 NIC. > > I ask here, because it is a router, mostly being busy with his network cards, > routing and pf. Please direct me to a better group if you can. > > I can crash this system if needed and dump memory (I hope that is possible on > GENERIC) for analysis. > That's interesting. Are those ARM-based routers or x86/amd64? This my server: Mem: 48M Active, 190M Inact, 28G Wired, 3075M Free ARC: 26G Total, 6118M MFU, 18G MRU, 32K Anon, 184M Header, 1294M Other Swap: 48G Total, 20M Used, 48G Free % uptime 5:12PM up 93 days, 1:58, 1 users, load averages: 0.23, 0.20, 0.16 uname -r 11.0-RELEASE-p8 vmstat -z 512: 512, 0, 2921, 1703,505278087, 0, 0 UMA Slabs: 80, 0, 551576, 51574, 1716599, 0, 0 It looks like the memory allocations for '512' constantly go up but I didn't check if they go down again at some point. However, 'UMA Slabs' seems to be not changing much. I believe the Wired memory as shown in top includes ZFS which the server is installed on? If that's the case then it's probably ZFS filling up the free memory with ZFS cache? Grzegorz