From owner-freebsd-stable@FreeBSD.ORG Thu Mar 26 19:46:05 2015 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id DF18D8A0; Thu, 26 Mar 2015 19:46:05 +0000 (UTC) Received: from mail-ig0-x230.google.com (mail-ig0-x230.google.com [IPv6:2607:f8b0:4001:c05::230]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id A3401E92; Thu, 26 Mar 2015 19:46:05 +0000 (UTC) Received: by igcxg11 with SMTP id xg11so1817664igc.0; Thu, 26 Mar 2015 12:46:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=v4CpH3nIi14MWPQHaE725rhotM9U81oHmOz4AkhSxKY=; b=03L1BGyCkd4tVQlBINvmUjXy6BTSyfAzIRKOqFZQABMdJtVsruYjFUh36azpwwr2U+ ESmjxEEC7J7MB3UVb8hG/6JF2JIFJCyhCD2Gxt6T2AgYsYgMH0zOCmmNwHaKoI3lNKZ0 i3fDT/kPuREnZmJyW9n9le4iwEnbZtRc7gqGaZUwf5ot/N5HHvVw1pRbFuFI/6lx8n0g d7A1vf71kS+VphnpjQ8N2vTjgNhjnkLC+BkRFiqsOgMkZyrNXmxRpkCKNukZ4hM4yoS1 RRqoq9tExCOhT/ZQVEQjb2zFW/klOzqRRgVnfsyapzaRkbV1Dr7hJ96Cc6+R9PRklCol b1eQ== MIME-Version: 1.0 X-Received: by 10.50.30.130 with SMTP id s2mr39698547igh.11.1427399165096; Thu, 26 Mar 2015 12:46:05 -0700 (PDT) Sender: jdavidlists@gmail.com Received: by 10.36.67.139 with HTTP; Thu, 26 Mar 2015 12:46:05 -0700 (PDT) In-Reply-To: References: <20150316232404.GM2379@kib.kiev.ua> Date: Thu, 26 Mar 2015 15:46:05 -0400 X-Google-Sender-Auth: vZFI7aRJztBZXJqUcqlojPg1MA4 Message-ID: Subject: Re: Significant memory leak in 9.3p10? From: J David To: freebsd-stable Content-Type: text/plain; charset=UTF-8 Cc: "freebsd-questions@freebsd.org" X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 26 Mar 2015 19:46:06 -0000 On Mon, Mar 16, 2015 at 7:52 PM, J David wrote: > On Mon, Mar 16, 2015 at 7:24 PM, Konstantin Belousov > wrote: >> There are a lot of possibilities to create persistent anonymous shared >> memory objects. Not complete list is tmpfs mounts, swap-backed md disks, >> sysv shared memory, possibly posix shared memory (I do not remember which >> implementation is used in stable/9). > > If that's the explanation, how could it be > detected/measured/investigated/resolved/prevented? > > Under ordinary circumstances, machines will go run like this for days/weeks: > > Mem: 549M Active, 3623M Inact, 567M Wired, 3484K Cache, 827M Buf, 3156M Free > Swap: 1024M Total, 1024M Free > > Then, when this happens, it rapidly degrades from that to so bad that > processes start getting killed for being out of swap space. These FreeBSD machines running out of swap space and dying continues to be a daily problem causing outages and unscheduled reboots. Is there really no way to even research what might be causing the problem? (Widening the cross-posting in the hopes of eliciting more help, so the brief summary of the problem orginally posted to freebsd-stable is that an unknown actor consumes all the user-space memory in the system, including swap space, to the point where processes are killed for being out of swap space, but if every process on the machine is stopped, very little of the user-space memory in use is freed. Original message with more details is here: https://lists.freebsd.org/pipermail/freebsd-stable/2015-March/081986.html .) There are no tmpfs mounts or md disks, so it would have to be one of the other causes. How can FreeBSD's use of persistent, anonymous shared memory objects be investigated, measured, or controlled so we can get a handle on this issue? Thanks!