From owner-freebsd-current@freebsd.org Tue Sep 26 13:41:59 2017 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id ADDFAE0C099 for ; Tue, 26 Sep 2017 13:41:59 +0000 (UTC) (envelope-from o.hartmann@walstatt.org) Received: from mout.gmx.net (mout.gmx.net [212.227.15.15]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "mout.gmx.net", Issuer "TeleSec ServerPass DE-2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 2771966581; Tue, 26 Sep 2017 13:41:58 +0000 (UTC) (envelope-from o.hartmann@walstatt.org) Received: from freyja.zeit4.iv.bundesimmobilien.de ([87.138.105.249]) by mail.gmx.com (mrgmx001 [212.227.17.190]) with ESMTPSA (Nemesis) id 0LgHvY-1dc6Hw0S9M-00ng1W; Tue, 26 Sep 2017 15:41:56 +0200 Date: Tue, 26 Sep 2017 15:41:55 +0200 From: "O. Hartmann" To: Guido Falsi Cc: "O. Hartmann" , freebsd-current Subject: Re: net/asterisk13: memory leak under 12-CURRENT? Message-ID: <20170926154155.28deb2e1@freyja.zeit4.iv.bundesimmobilien.de> In-Reply-To: <979b6cfe-0e38-5df3-7bb5-cdb8de6677bf@FreeBSD.org> References: <20170926144522.21e59cfe@freyja.zeit4.iv.bundesimmobilien.de> <979b6cfe-0e38-5df3-7bb5-cdb8de6677bf@FreeBSD.org> Organization: Walstatt MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Provags-ID: V03:K0:+088Ni3wqUJaisiHvil2tMlUWXc1J/2xKPR6IcV02e1j7kvoMB1 aulDWSEq504AFFSGT42AQdZ38U6JDZ/nyrgvFSjXHfqy3nnHl/qcUWya061jnjlG/K6aHFu mP3legeFCsmmEPPEXNpZFixQu6npcFDK1PHAXPwAaTBRydXaOCk8IcuSQFiTBdPuQJye14p aS21VTrYScQQY84v7+MbQ== X-UI-Out-Filterresults: notjunk:1;V01:K0:BoFkb/MONhQ=:Qk1a/hP3ltaifHdJzGD6bC sQxXgSy+JXLoet3On4uGdXQcuI6odbuqe3kX5npk5W484FjalQNYD07NUO7R3NqmpcCe5LyH1 8gHfiL+CgY19TAQRbHFCvRz+qTEjW2zNCxrsDiGm+3vIOUi/ajWQMwxKk5/wInO6zsRX91a7t Ys8zakg7O+74os7thWfzVDRIieuJ/KCfNGNLVv1f2YpLvx9g3FXRXC+JtPAa342eYWRO4eiJH MN4yFiNPlLQDLrhCTqiedJujz1aevnkTV91f5JLEt6cCjbMKVRlZD4WHm+7T2/X57AwRY+5V+ HzV3KK0weGcVmgSqczYbTcpuimTsE1wcwp0HXBd8K2MbLIOF9u9rYclAP0mCpJMTaElcjFqHu CoukhgfxWqGupTUEGjPcNjUlZ3VzUfeLGqhp5to2lCp5tl1LOYOwM1DQn+6IlSbilMishb4Zw JuS/nT3v+T1qHWwFUQSth5S1KvBIjrj+7B1HqBpB+NokVpSCcMEyDj/Usk+bwOTUufVMcrK0s 7FSGShHytexzvpiQTqh+1kJAGTjZi4Pzx0x0mckJcO4yYGN5v03wTxD5bR39i/FDlBsrLvXS4 YwArD7+NbNQDAaHBidB+ac2paVA4QzDjVIMHXBVvtmfILwmjqW6Ibea+YjIXh+Nsu0so3EPb+ 1WflYt29UgZnvMbRo4TxJBD6bobhOgDD718Sonb1Q3UsZ5RHocdO9XsPgqivjnjUfg2qg841u cVVG4jeRvnw42bM55zXRQddpj5OYSm+9i8X1afyLQsHsi7gMGT/uxulmDorknZwu6ZxrHb+7B dXW8iOSkkN5hJ+UGReqp3T5mZY1JJUdV2TqWicmRxW03FsHE10= X-Mailman-Approved-At: Tue, 26 Sep 2017 17:23:59 +0000 X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 26 Sep 2017 13:41:59 -0000 On Tue, 26 Sep 2017 15:06:23 +0200 Guido Falsi wrote: > On 09/26/2017 14:45, O. Hartmann wrote: > > Befor starting a PR I'd liek to ask for some advice to document a supposedly > > existent memory leak in net/asterisk13 and 12-CURRENT. > > > > Background: > > > > Running recent 12-CURRENT (FreeBSD 12.0-CURRENT #58 r323999: Tue Sep 26 > > 06:18:27 CEST 2017 amd64) on a PCengine APU 2C4, equipted with 4GB of RAM > > and 4080 KB free after the most recent SeaBIOS has been started, I try to > > utilise a net/asterisk13 as PBX on that platform. Asterisk13 has been build > > via poudriere and is compiled with CLANG/LLVM, not GCC! > > > > I'm running asterisk on similar hardware and also building with clang, > and have it going for months without any problems. I have disabled many > modules in that build though. Problem could be caused by some ancillary > library pulled in by some module. Since I run net/asterisk with automatic module loading (I'm new to asterisk), this is very likely and might cause the problem somehow. > > > When FreeBSD (NanoBSD) has been started, depending on the recent revision > > of the OS/Kernel, top shows ~3680 - 3705 MBytes of free memory. Starting > > net/asterisk13 via "service asterisk onestart" indicates an overall usage > > of up to 100 MBytes of memory. After having now run the Asterisk 13.17.2 > > daemon for two days resulted in ~ 3100 MBytes of free memory, while the > > asterisk daemon was still running and doing nothing. But performing several > > restarts on a freshly started box gives the same result after ~ 10 restarts > > - and even after stopping asterisk and let the system run for a couple of > > days doesn't free the memory. I stopped after 15 restarts or so watching > > the loss of memory after each restart of asterisk, so i came to the > > conclusion that there must be a memory leak somewhere. now i try to provide > > more informations as needed for a PR. > > > > Can someone help? > > Not sure, restarting the daemon should free any leaked memory the daemon > has. If a killed process leaves memory locked at the system level there > should be some other cause. Even with no runnidng asterisk, memory level drops after the last shutdown of asterisk and keeps that low. Even for weeks! My router never shows that high memory consumption, even under load. > > Have you tried diagnosing this memory with more tools? top so far ... > > > Depending on how deep you want to investigate ps, top and vmstat are > simple to use and give you some indication. To go deeper you'll need to > use more complicated tools. Others are better suited than me for > suggesting those. DTrace could be a powerful but not easy to use instrument. Well, DTrace would be an overkill for me. I'd like to provide more informations in case any FreeBSD developer wants to look at this - in case there is a real bug in either FreeBSD CURRENT - or net/asterisk. > > Looking just at free memory (which under a UNIX OS can have various > valid definitions, depending on how you account for inactive memory, > buffers and laundry RAM) is definitely not enough. Have you seen > asterisk process allocate more and more memory? > > You should check also what other processes in the system are doing. A > computer software when running never really does "nothing"(except, > maybe, when swapped out) and some memory usage can accumulate in many > parts of the system. Well, since it is a well defined NanoBSD system which has run for weeks with the same memory consumptions ebven under load, it seems obvious that there must be a black hole for memory when it somes to loading/unloading asterisk. The memory drain can be reproduced. > The question would be: how to use vmstat to give hints for those familiar with memory subsystems to indicate a real bug? I tried to find some advices, but maybe my English isn't good enough to make google help. Kind regards, Oliver