From owner-freebsd-stable@freebsd.org Thu Feb 18 06:50:44 2016 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8FA96AAB63A for ; Thu, 18 Feb 2016 06:50:44 +0000 (UTC) (envelope-from bc979@lafn.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 73EC02C9 for ; Thu, 18 Feb 2016 06:50:44 +0000 (UTC) (envelope-from bc979@lafn.org) Received: by mailman.ysv.freebsd.org (Postfix) id 7173AAAB638; Thu, 18 Feb 2016 06:50:44 +0000 (UTC) Delivered-To: stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5707DAAB637; Thu, 18 Feb 2016 06:50:44 +0000 (UTC) (envelope-from bc979@lafn.org) Received: from mail.lafn.org (static-71-177-216-148.lsanca.fios.verizon.net [71.177.216.148]) by mx1.freebsd.org (Postfix) with ESMTP id 2DC012C7; Thu, 18 Feb 2016 06:50:43 +0000 (UTC) (envelope-from bc979@lafn.org) Received: from [10.0.1.12] (unknown [10.0.1.12]) by mail.lafn.org (Postfix) with ESMTPSA id 236E3114C4FD; Wed, 17 Feb 2016 22:50:43 -0800 (PST) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 9.2 \(3112\)) Subject: Re: intr using Swap From: Doug Hardie In-Reply-To: <447fi2sse9.fsf@lowell-desk.lan> Date: Wed, 17 Feb 2016 22:50:43 -0800 Cc: owner-freebsd-stable@freebsd.org, stable@freebsd.org, Efra?n D?ctor , hiren panchasara Content-Transfer-Encoding: quoted-printable Message-Id: References: <56C4AF81.3040202@motumweb.com> <87f6fb602e0ad11b7600c70a08d74c30@dweimer.net> <56C4C244.8070805@motumweb.com> <56C4F7E9.9090405@motumweb.com> <20160217230138.GJ89208@strugglingcoder.info> <44d1rusuxs.fsf@lowell-desk.lan> <49F794B1-937F-4AEA-90CF-7C19AFF7EFE2@lafn.org> <447fi2sse9.fsf@lowell-desk.lan> To: Lowell Gilbert X-Mailer: Apple Mail (2.3112) X-Virus-Scanned: clamav-milter 0.98.7 at mail X-Virus-Status: Clean X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 18 Feb 2016 06:50:44 -0000 > On 17 February 2016, at 17:45, Lowell Gilbert = wrote: >=20 > Doug Hardie writes: >=20 >>> On 17 February 2016, at 16:50, Lowell Gilbert = wrote: >>>=20 >>> Have you measured that paging (not swapping; that's a more extreme >>> measure where the whole process gets removed from memory) is a >>> significant load on your system in a specific case? If not, I doubt = that >>> it's actually the case, and you're mitigating a non-existent problem >>=20 >> I believe the question here is what is using up the swap space. From >> what I have been able to find with a similar situation is that malloc >> will allocate swap space to backup memory and mmap will also allocate >> swap if there is no backing file. procstat -v can be helpful in >> chasing down some of those issues. However, I ended up guessing = which >> process it was by sequentially restarting processes and watching >> swapinfo. I still have not been able to chase down what in that >> process is using the space. There are no mmaps that are not file >> backed so it must be a malloc. Finding the right one has been >> elusive. >=20 > Sure, but I'm pretty sure that the other worriers in this thread don't > actually have any problem at all. I tried to poke a (Socratically > limited) number of questions as a start of figuring out whether that's > really the case, but thus far, I'd bet that it is. If that turns out = to > be a losing bet, I *will* spend time on fixing the code. >=20 > Your observations are more useful, but I'm still not sure they = indicate > a problem that needs to be solved. There are clearly cases where > significant quantities of swap can get used up storing copies of clean > pages backing files on disk. Unless that slows down bringing in new > pages that need to be read or written, I don't think that's a problem. Well, the problem is quite significant for me in that eventually the = system runs out of swap and starts killing processes. Its not quite = random, but I haven't spent much time trying to figure out how it = selects those to kill. The specific system unfortunately is remote = (about a 3 hour drive) and when sshd gets killed, I have no option other = than having someone go on site to reboot it. I have had to start = monitoring swap usage with nagios and having it notifiy me when swap is = at 40% used. So far that has given me enough time to find an internet = connection and restart the process that is eating the swap. The = developer of that process claims that the problem must be in one of the = library functions it uses. That does seem reasonable as I am using that = process on a large number of systems and only one has the issue. = However, until I can track down which system call or malloc is eating up = swap space, I don't really see any way to fix the problem. I recently = rebuilt that system with an updated system and resized swap to be 10x = memory. That does raise the mean time between process kills, but = doesn't eliminate the problem. About every other week the alarm is = raised now. Before it was pretty much every other day.