From owner-freebsd-questions@FreeBSD.ORG Thu Mar 26 23:39:14 2015 Return-Path: Delivered-To: freebsd-questions@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id CA93B7E for ; Thu, 26 Mar 2015 23:39:14 +0000 (UTC) Received: from mail-ig0-x22a.google.com (mail-ig0-x22a.google.com [IPv6:2607:f8b0:4001:c05::22a]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 85648C94 for ; Thu, 26 Mar 2015 23:39:14 +0000 (UTC) Received: by igcau2 with SMTP id au2so22534058igc.1 for ; Thu, 26 Mar 2015 16:39:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=from:content-type:message-id:mime-version:subject:date:references :to:in-reply-to; bh=SoBp8O1qCSNL+Iy1/AHqiIVeXHWVRu1GJI924jK0naI=; b=n+FG+jVYxvZexx5HQ6t+hyVxBL4pxA/agn8BaOLejljqQZoBuDY9u/O8nQ4G1To8FA 63EWHurB2a0Yu9mnoS7yLFOgpaHLLfiqdyRg2frMJOavFym7S7M3TYAU0gmEt3tbxXKH fmRNFX3sDij3AEAY0FFH0t00sGs7ABwMfje7HiXbQ4JeQC+AkmIKDPmxciHToHcA1+mx nfLM5nfySvonP8q9A6KGDbJDZyzKbn7pnREj5lnkd21H6uB31WBI80IGYw3f+Bt5axav lUwJy8uPsSwoz93I1g++PZhoy8EmDo9ZYfHmbVOuKxtp+jPHbQoAvFoT96IRGlTC3NXW OUNw== X-Received: by 10.50.253.12 with SMTP id zw12mr40158857igc.24.1427413153942; Thu, 26 Mar 2015 16:39:13 -0700 (PDT) Received: from [192.168.89.100] (192-171-49-199.cpe.pppoe.ca. [192.171.49.199]) by mx.google.com with ESMTPSA id i20sm207876igh.16.2015.03.26.16.39.11 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Thu, 26 Mar 2015 16:39:12 -0700 (PDT) From: The Lost Admin Message-Id: <79371E33-B999-4CAC-A8E4-8D5DDBF043E6@gmail.com> Mime-Version: 1.0 (Mac OS X Mail 7.3 \(1878.6\)) Subject: Re: Significant memory leak in 9.3p10? Date: Thu, 26 Mar 2015 19:39:08 -0400 References: <20150316232404.GM2379@kib.kiev.ua> To: "freebsd-questions@freebsd.org" In-Reply-To: X-Mailer: Apple Mail (2.1878.6) Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 26 Mar 2015 23:39:14 -0000 The Lost Admin thelostadmin@gmail.com On Mar 26, 2015, at 3:46 PM, J David wrote: > On Mon, Mar 16, 2015 at 7:52 PM, J David = wrote: >> On Mon, Mar 16, 2015 at 7:24 PM, Konstantin Belousov >> wrote: >>> There are a lot of possibilities to create persistent anonymous = shared >>> memory objects. Not complete list is tmpfs mounts, swap-backed md = disks, >>> sysv shared memory, possibly posix shared memory (I do not remember = which >>> implementation is used in stable/9). >>=20 >> If that's the explanation, how could it be >> detected/measured/investigated/resolved/prevented? >>=20 >> Under ordinary circumstances, machines will go run like this for = days/weeks: >>=20 >> Mem: 549M Active, 3623M Inact, 567M Wired, 3484K Cache, 827M Buf, = 3156M Free >> Swap: 1024M Total, 1024M Free >>=20 >> Then, when this happens, it rapidly degrades from that to so bad that >> processes start getting killed for being out of swap space. >=20 > These FreeBSD machines running out of swap space and dying continues > to be a daily problem causing outages and unscheduled reboots. Is > there really no way to even research what might be causing the > problem? >=20 > (Widening the cross-posting in the hopes of eliciting more help, so > the brief summary of the problem orginally posted to freebsd-stable is > that an unknown actor consumes all the user-space memory in the > system, including swap space, to the point where processes are killed > for being out of swap space, but if every process on the machine is > stopped, very little of the user-space memory in use is freed. > Original message with more details is here: > = https://lists.freebsd.org/pipermail/freebsd-stable/2015-March/081986.html > .) >=20 > There are no tmpfs mounts or md disks, so it would have to be one of > the other causes. How can FreeBSD's use of persistent, anonymous > shared memory objects be investigated, measured, or controlled so we > can get a handle on this issue? In your initial thread, you said: $ sudo halt -p > Waiting (max 60 seconds) for system process `vnlru' to stop...done > Waiting (max 60 seconds) for system process `bufdaemon' to stop...done > Waiting (max 60 seconds) for system process `syncer' to stop=85 > Syncing disks, vnodes remaining...0 0 0 0 0 0 0 0 0 done > All buffers synced. <----- 10 MINUTE HANG AFTER PRINTING THIS > Uptime: 3d15h56m32s > usbus0: Controller shutdown > uhub0: at usbus0, port 1, addr 1 (disconnected) > usbus0: controller did not stop > usbus0: Controller shutdown complete > acpi0: Powering system off > Connection closed by foreign host. > So it seems like somewhere after "All buffers synced" and printing the > uptime, it's very slowly unwinding whatever is using up all that RAM > and swap. Have you looked through the system shutdown scripts (part of init/rc) to = see what happens after the uptime is printed? that might give you a = lead. The output from your PS seams to be much shorter than I would expect. = Are you sure it included everything? For example, I would expect to see = processes for cron, syslog, and normally sshd. I=92ve also got a few = more kernel processes that you don=92t appear to have. Most notably is = pagedaemon For what it=92s worth, I=92m running 9.3 RELEASE-P12 (the -p10 kernel) = on a system 24x7 (6 days since the last reboot) and I haven=92t had an = issue. It=92s a low volume NFS server.=