From owner-freebsd-net@freebsd.org Tue Jul 11 14:02:57 2017 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 35FD2DA0B93 for ; Tue, 11 Jul 2017 14:02:57 +0000 (UTC) (envelope-from hps@selasky.org) Received: from mail.turbocat.net (turbocat.net [IPv6:2a01:4f8:c17:6c4b::2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 02CF175492 for ; Tue, 11 Jul 2017 14:02:56 +0000 (UTC) (envelope-from hps@selasky.org) Received: from hps2016.home.selasky.org (unknown [62.141.129.119]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.turbocat.net (Postfix) with ESMTPSA id 20969260805; Tue, 11 Jul 2017 16:02:46 +0200 (CEST) Subject: Re: memory leaks in 11.0? To: Kajetan Staszkiewicz , freebsd-net@freebsd.org References: <1771938.JUakJLQFm8@energia> From: Hans Petter Selasky Message-ID: <611f7ba6-ee4a-51e8-38c1-7f8e84b1dc9e@selasky.org> Date: Tue, 11 Jul 2017 16:00:36 +0200 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Thunderbird/52.2.0 MIME-Version: 1.0 In-Reply-To: <1771938.JUakJLQFm8@energia> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 11 Jul 2017 14:02:57 -0000 On 07/11/17 15:56, Kajetan Staszkiewicz wrote: > Hello, > > I finally upgraded one of many of my routers to 11.0. > > Unfortunately after running fine for a month it ran out of memory. "wired" > memory slowly grows up to allocating all memory in system when no more memory > is left for other programs. Things first get swapped and eventually die. > > The router runs BIRD which has not much to do, it is for internal networks > only, pf, pfsync (currently disabled via `ifconfig pfsync0 down`), filebeat, > smokeping, ntp, nrpe and custom python cron job for sending data to Graphite. > > `vmstat -z` shows constantly increasing allocation of "512" and "UMA Slabs". > Memory allocated for all pf-related things seems fine. I have graphite graps > for every `vmstat -z` and the icrease on "512" grows in similar way as "wired" > memory. "512" has 2 917 392 used objects allocated at this moment, "UMA Slabs" > is 379 006, there is 2636MiB "wired" memory. > > How can I debug which part of kernel is responsible for this? I run GENERIC > kernel with ixl driver 1.7.11 from Intel, as the one in GENERIC had issues > detecting links on my x710 NIC. > > I ask here, because it is a router, mostly being busy with his network cards, > routing and pf. Please direct me to a better group if you can. > > I can crash this system if needed and dump memory (I hope that is possible on > GENERIC) for analysis. > Hi, Last time I traced memory leaks I used some dtrace scripts to trace all allocations and frees and then analayzed the result using a perl script which I found on the internet. --HPS